hn4a-lipsync-check: is the mouth in sync, over time?

Give it a video of one person speaking and the audio it should match; get the mouth's lag every 0.2 s, the spans where it stays out of tolerance, the words spoken there, a plot, and a strip of mouth crops at the worst span. Because the lag is measured over time, a take that starts in sync and drifts is caught where it drifts, not averaged away.

Internal QA only. The SyncNet weights behind it are published for research use, with no commercial licence stated. Until that is settled, use the check to choose and fix takes; never put its output (plot, strip, numbers) in a deliverable or in front of a client.

Run it

exec("hn4a-lipsync-check", [
  "talking_head/chunk08_h3lipsync_a.mp4",
  "--audio", "talking_head/chunk08.wav",
  "--words", "talking_head/chunk08.stt.json",
  "--out-dir", "talking_head/lipsync_check",
  "--json"
], cwd="<job>")

Argument	What it does
`<video>`	the take; one face, any size, anywhere in frame (a 180 px face in 1080p is found)
`--audio`	the audio it must match; default: the video's own track. Pass the chunk's WAV for an H3 lip-sync take, `track.wav` for an assembled video
`--words`	a speech-to-text JSON on the same clock (`elevenlabs-speech-to-text`), to name the words in each flagged span
`--offset-s S`	add S to printed and plotted times, such as a chunk's start on the full track
`--out-dir D`	where the plot and strip go; default: next to the video
`--late-ms`, `--early-ms`	the tolerance band; leave at 90 and 150
`--json`	the full result as JSON on stdout; without it, a short summary

Exit status: 0 in sync, 2 flagged. Time: about 10 s per 10 s of video on Apple silicon, 111 s for a 163 s track. Features are cached per file, so re-checking the same take against new audio takes about 0.3 s. The first run fetches the weights.

What you get

The summary, one line per flagged span:

chunk08_h3lipsync_a.mp4: FLAGGED  global +40 ms (conf 7.9), lag -60..+300 ms
     2.90-  9.80s  mouth late 182 ms  "check who really sent it before you"
  plot: talking_head/lipsync_check/chunk08_h3lipsync_a_lipsync.png
  strip: talking_head/lipsync_check/chunk08_h3lipsync_a_lipsync_strip.jpg

With --json:

ok: true when nothing is flagged.
offset_ms: [t, lag_ms, conf] per 1 s window, every 0.2 s. Positive lag means the mouth is late (sound before picture).
flags: [t0, t1, mean_lag_ms] per flagged span; flag_words: the words in each.
global: {lag_ms, conf}, the single best offset over the whole clip.
min_lag_ms, max_lag_ms: the range over confident windows.
thresholds, plot, and strip when flagged.

The plot shows lag over time with the tolerance band shaded, confidence as colour, flagged spans in red and the words along the bottom. The strip shows mouth crops at 8 fps from 1 s before to 2 s after the worst point, each over a bar of the audio level, so a closed mouth on loud audio shows at a glance.

How to read it

A flag is a run of windows with lag over +90 ms (mouth late) or under −150 ms (mouth early), in windows with confidence 3 or more, sustained for 1 s or more. Low-confidence windows (pauses) inside a run don't break it. The band follows ITU-R BT.1359's acceptability limits (+90 / −185 ms), a little stricter on the early side.
Confidence: good takes score 6 to 12; pauses score low and are ignored.
Good H3 Max takes sit between −83 and +60 ms: the mouth runs 10 to 60 ms early, hence the asymmetric band.
Drift shows as a lag climbing through the clip (in sync at the start, +200 to +500 ms by the end of a 10 s chunk). Same slope on every seed: re-sync the mouth with veed-v2-resync-existing-take rather than re-rolling, then check the re-sync too.
A flat curve shifted out of the band (no climb) points to audio placed a few frames off rather than drift: check where the audio was cut before touching the take.
How far to trust it: it recovered injected offsets to a median 2 to 3 ms, flagged 126 of 127 out-of-tolerance cases with no false alarms on 68 in-tolerance ones, and flagged the drifting takes a person reported, plus one they hadn't noticed. The one miss went out of tolerance only for the last 0.8 s of a chunk, under the 1 s minimum.

Tips and traps

Run it on every lip-synced take, on every re-sync, and once more on the assembled video against the full track: on one assembled track it flagged exactly the two chunks that drifted and nothing else.
Check against the audio the take was rendered on. A take made on old audio after a re-voice will flag everywhere; re-render it.
Cartoon faces can't be measured. On drawn characters the confidence stayed under 3 for the whole clip and the offsets jumped ±800 ms, so it reported "ok" without measuring anything. Judge a cartoon's lip sync by eye on a review page.
It tracks one face, the first it finds. In a two-person shot, crop to the speaker before checking.
Don't replace it with lip aperture against the speech envelope: that scatters ±400 ms on good takes and can't measure an absolute offset.