Lip-sync video

Make someone speak on screen. Write the audio first (guide-voice-a-script, or the isolated voice from the source), then pick the route by the outcome wanted. Ask first: is there a take whose skin, light and performance must stay, or are you starting from a still, and how many people are in the shot?

• A still and one speaker (a presenter, an avatar, a line): lip-sync the still.
• A good take whose mouth is off or whose words changed: fix only the mouth.
• A scene with several people, blocking and voice-over: ref2vid with the audio.

Prices are as of 2026-10; get exact ones from describe(model) before a batch.

Routes

For a scene, render each shot on both ref2vid models and let the person pick; picks split about evenly. For a locked single-person talking head, Seedance holds best.

Preparing the audio

• Transcribe it with elevenlabs-speech-to-text, then python-read-word-timings for words, speaker turns and pauses. Diarization clusters voices, not people.
• From a mix, isolate the voice first (elevenlabs-voice-isolation). Cut where the next speaker starts by ear, from a ladder of trims a few hundredths of a second apart: STT and loudness minima were each ~0.1 s off.
• Cut windows with ffmpeg-cut-audio-window at pauses: 0.2 s before the shot's first word, ending 0.2 s before the next shot's first word, or halfway into a shorter pause. A ref2vid window under 2 s is padded to 3 s. H3 lip sync wants 5–15 s: pad with silence, never take a longer window.
• Chunks for a long track: 5–14 s, cut at the longest pause in range, preferring a pause after . ? !. At each cut the speaker returns to the pinned pose, which reads as a breath between sentences and a hitch mid-sentence. Cut every chunk on one frame grid for the whole track (round(start × fps) to round(end × fps)); trimming each chunk separately drifted 313 ms over 17 chunks.
• Plan silence into the track wherever the picture needs action time (an exit, a transition) before rendering any video.

Directing ref2vid scenes

• Cast only the people physically in the frame; a face on a screen gets no reference.
• Pin each speaker by screen position at every line ("the woman on the RIGHT").
• Screens are a fixed state ("a paused still"), never a replay.
• Voice-over: say nobody on screen talks.
• Time-cue the action to the transcript ("1.7–9.2 s: the man counts the tests").
• One seed for the whole job, so a one-line change is a true A/B.
• Seedance with a recorded line: give a clip-length recording with each line at its real time, and bind each line to its span in the prompt (seedance-2-5-prompt-format). With a short untimed reference it re-performs the line at its own pace: up to 1.3 s late and ~10% long. Takes rendered that way can be cut to the line with python-find-line-in-render.
• The models re-voice the line, and the mouth follows their version. Assemble each clip with its own audio; never lay the track under a re-voiced clip (Seedance drifted up to 0.44 s mid-shot). The words came back verbatim on every pick.

For H3 Max lip sync

• Put the spoken words in the prompt, or the prompt says the lips stay closed and H3 renders a silent face. A talking mouth crop moves 5–8 (mean frame difference); a silent one ~1.
• Render the still as 3:4 portrait at 768P for presenters: about the same pixels on the face as 16:9 1080P at half the price. 9:16 clips wide gestures.
• Keep gestures minimal unless asked. When asked, direct a beat per line, and ban finger counting outright: asked for fingers on "First / Second / Finally", H3 held up three on a line with no third item.

Checking sync

Run hn4a-lipsync-check on every lip-synced take. It reads the mouth's lag every 0.2 s and flags spans late by more than +90 ms or early by more than −150 ms, sustained for 1 s (~10 s per 10 s clip). Good H3 takes sit at −83 to +60 ms. H3's tempo drift (in sync at the start, 200–500 ms late by the end of a 10 s chunk) hit 3 of 17 chunks and nobody spotted it in a quick watch.

Gates, each a review page with sound on: voices → stills → video → export. The dialogue track has none: once the voices are picked, it goes on when its transcript checks out. Show the original and last round's take next to the new ones.

Revisions

Fix a chunk or a shot end to end: re-voice its audio, re-render it, re-assemble. Don't splice audio or patch video by hand. H3's lip timing comes from the audio and is the same on every seed, so a drifting chunk won't fix with re-rolls; re-sync its mouth with VEED instead.

Avoid

• Sync 3: it left the face unchanged (output ~identical to input).
• Re-rolling H3 drift: same slope on every seed.
• Attaching every character to every shot: wrong people in rooms, lines on wrong faces.
• Laying the track under re-voiced clips.
• Patching audio or video by hand on a revision.

Related

guide-voice-a-script (the audio), guide-choose-a-video-model, guide-animate-a-shot (silent shots), guide-recast-a-person (new people, then new voices).