VEED v2: re-sync the mouth of an existing take, cheapest

Give it a take and the audio it must speak; get the take back with only the mouth redone, the take's skin, light and performance kept. Its main job is fixing H3 Max drift: on some chunks H3's lips run 3 to 5% slow, identically on every seed, so a re-roll can't fix it and a mouth-only re-sync can.

When to use this and when its sibling:

Inputs

• The take (MP4) and its frame rate (ffprobe-inspect-media).
• The audio it must speak, mono WAV starting at the take's first frame. For an H3 chunk it's the chunk's own WAV, the one the take was rendered on. Otherwise cut it with ffmpeg-cut-audio-window.

Run it

1. Trim the take to the audio's length, with the audio muxed in. frames = round(audio seconds × fps):

   exec ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate -of csv=p=0 chunk08_h3lipsync_a.mp4
   exec ffprobe -v error -show_entries format=duration -of csv=p=0 chunk08.wav
   exec ffmpeg -v error -y -i chunk08_h3lipsync_a.mp4 -i chunk08.wav -map 0:v -map 1:a \
     -vf "trim=end_frame=217,setpts=PTS-STARTPTS" \
     -c:v libx264 -crf 14 -preset slow -pix_fmt yuv420p -c:a aac -b:a 256k chunk08_trim.mp4
   

2. Call describe("fal/veed/lipsync/v2") before the first run in a job, then:

   run("fal/veed/lipsync/v2", {
     "video_url": "<job>/talking_head/chunk08_trim.mp4",
     "audio_url": "<job>/talking_head/chunk08.wav"
   }, into="<job>/talking_head/raw", unit="chunk08_veed", tag="chunk08")
   

3. Convert it back from 25 fps to the take's rate on the take's frame grid, cut to frames, with the audio muxed in again:

   exec ffmpeg -v error -y -i raw/chunk08_veed_a.mp4 -i chunk08.wav -map 0:v -map 1:a \
     -vf "fps=24:round=near" -frames:v 217 \
     -c:v libx264 -crf 14 -preset slow -pix_fmt yuv420p -c:a aac -b:a 256k chunk08_h3lipsync_b.mp4
   

   fps=24:round=near picks the nearest 25 fps frame for each 24 fps slot; a 217-frame take came back 217 frames, identical to a fix made by hand. Don't blend or interpolate frames.

Save the result as the chunk's next variant letter and record what it came from (the take, veed_v2, the audio), so a later round knows it's a re-sync.

What you get

The take with a re-synced mouth, at the take's frame rate and frame count. Cost is $0.07 per second of video.

Tips and traps

• If the chunk's audio changed (a re-voiced chunk), re-render the chunk on the new audio rather than re-syncing a take made on the old one.
• The accepted re-sync of a drifting chunk measured −4 to −127 ms over its length: inside the tolerance, with the mouth a little early, as good H3 takes are.

Check it

• Run hn4a-lipsync-check on the result against the same audio: it must come back with no flagged spans. Check every re-sync, not only the first.
• Count frames: equal to frames from step 1.
• Watch it once at playback size next to the flagged take.