HeyGen v3 Precision: re-sync the mouth of an existing take

Give it a take and the audio it must speak; get the same take back with only the mouth redone. Everything else (skin texture, light, the performance, the framing) is the take's own, which H3 Max lip sync can't offer because it re-renders the whole face.

Pick between the two mouth-only siblings by what the shot needs:

Sync 3 is not a sibling: on a 3/4-view face it returned the input nearly unchanged (PSNR 50 dB). Use h3-max-lipsync-a-still only when there is no usable take.

Inputs

• The take (MP4). Note its frame rate with ffprobe-inspect-media.
• The audio it must speak: mono WAV, the same length as the part of the take you keep, starting at the take's first frame. Cut it with ffmpeg-cut-audio-window: an isolated voice, silent from where the next speaker starts (picked by ear), so their words aren't lip-synced onto this mouth.

Run it

1. Trim the take to the audio's length on its own frame grid, with the audio muxed in. frames = round(audio seconds × fps); read both with ffprobe:

   exec ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate -of csv=p=0 shot03_seedance_c.mp4
   exec ffprobe -v error -show_entries format=duration -of csv=p=0 shot03_line.wav
   exec ffmpeg -v error -y -i shot03_seedance_c.mp4 -i shot03_line.wav -map 0:v -map 1:a \
     -vf "trim=end_frame=33,setpts=PTS-STARTPTS" \
     -c:v libx264 -crf 14 -preset slow -pix_fmt yuv420p -c:a aac -b:a 256k shot03_seedance_c_trim.mp4
   

2. Call describe("fal/fal-ai/heygen/v3/lipsync/precision") before the first run in a job, then:

   run("fal/fal-ai/heygen/v3/lipsync/precision", {
     "video_url": "<job>/video/shot03_seedance_c_trim.mp4",
     "audio_url": "<job>/audio/shot03_line.wav",
     "enable_dynamic_duration": false
   }, into="<job>/video/raw", unit="shot03_heygen", tag="shot03")
   

   enable_dynamic_duration: false matters: the default retimes the clip to the audio, and the shot's length is fixed.

3. Put it back on the take's frame grid with the audio muxed in again, so it drops into the cut frame for frame (24 fps here; use the rate ffprobe gave):

   exec ffmpeg -v error -y -i raw/shot03_heygen_a.mp4 -i shot03_line.wav -map 0:v -map 1:a \
     -vf "fps=24:round=near" -frames:v 33 \
     -c:v libx264 -crf 14 -preset slow -pix_fmt yuv420p -c:a aac -b:a 256k shot03_heygen_a.mp4
   

Keep the take you started from: the re-synced file is a new variant letter, never an overwrite.

What you get

The take with a new mouth, the audio's length, at the take's frame rate after step 3. Cost is $0.10 per second of video.

Tips and traps

• A re-sync fixes the mouth, not the delivery. If the performance is wrong (too calm, wrong beat), re-render the take first, then re-sync the best-acted one.
• Re-voiced audio that runs longer than the take can't fit: re-render the shot. Never speed a clip up or slow it down to fit; it reads badly.
• On a revision, re-sync from the take you started from, not from an earlier re-sync.

Check it

• hn4a-lipsync-check on the result against the audio: no flagged spans.
• Count frames (exec ffprobe -count_frames): it must equal frames from step 1.
• Put the take, the re-sync and the original shot on a review page and record the pick.