Assemble lip-sync chunks

Give it the chunk plan (python-plan-lipsync-chunks), a take per chunk from h3-max-lipsync-a-still, the still they were pinned to and the track; get a report on every take, retimed copies of the ones whose mouth drifts, and one talking head with the track under it, checked against that track. A take whose mouth is out of sync in a way a retime can't fix gets a mouth-only re-sync (veed-v2-resync-existing-take) before it is picked.

Inputs

• chunks.json and the chunk WAVs from python-plan-lipsync-chunks.
• picks.json: each chunk's take, {"0": "talking_head/chunk00_a.mp4", …}. Any frame rate and size; the join converts to the plan's fps and the first take's size (a VEED re-sync comes back at 25 fps).
• The still every chunk was pinned to, and the mouth's box on the takes' px (read off a gridded frame, python-contact-sheet), for the pose and mouth checks.
• The track the chunks were cut from.

Run it

Write references/lipsync_assemble.md's script into the job's scripts/ as lipsync_assemble.py, as is, beside pinned_frames.py (python-check-pinned-frames), which it imports. It runs hn4a-lipsync-check itself.

exec python scripts/lipsync_assemble.py check  --chunks talking_head/chunks.json --picks talking_head/picks.json --still refs/presenter.png --mouth 920,235,100,60
exec python scripts/lipsync_assemble.py retime --chunks talking_head/chunks.json --chunk 13 --take talking_head/chunk13_a.mp4 --dest talking_head/chunk13_a_rt.mp4
exec python scripts/lipsync_assemble.py join   --chunks talking_head/chunks.json --picks talking_head/picks.json --track audio/track.wav --dest talking_head/talking_head.mp4

Check each take as it lands rather than all at the end, so a re-roll starts while the rest still render. A check takes about 17 s a chunk; the join about 3 min for 163 s at 1080p.

What you get

check prints a line a chunk and writes check.json: first_vs_still, cut_vs_first, mouth_motion, in_sync, lag_range_ms, drift and the advice:

retime plays the take faster by its drift and prints the slope before and after (on 17 real H3 Max chunks, two drifted: −44 → +265 ms at +3.3%, retimed to −49 → −35; −9 → +511 ms at +5.5%, to −9 → +22; both then in sync). join prints the frame count, the seams and the whole talking head's own check.

Tips and traps

• H3 re-renders the face, so a pinned first frame matches the still at only 0.94–0.98 here (greyscale SSIM at 480 px); the pose at the cut is measured against the take's own first frame instead (0.93–0.99 on a good set).
• Drift is the same on every seed: never re-roll it. A retime moves the picture 3–5% faster, which nobody sees, and also brings a slow take back to the pose sooner (the +5.5% chunk was the one still moving at its cut: 0.933, and 0.992 once retimed).
• Mouth motion follows the face's size: 7–11 for an H3 face at 1080P, 2–4 on a 720p webcam, about 1 for a closed mouth. Compare chunks with each other.
• One grid for the whole track. Chunk n fills frames round(start·fps) to round(end·fps); trimming each chunk on its own drifted 313 ms over 17 chunks (ffmpeg-frame-exact-trim, The frame grid).
• Dissolve, don't cut. Renders run 0.05–0.7 s long and differ by about 1% zoom: hard cuts stepped up to 4.7 at the seams against a median frame step of 0.37; the 5-frame dissolve over the outgoing tail kept every seam at 2.1 or under, the video's own p99.
• A take shorter than its chunk holds its last frame, the pinned pose.

Check it

• join's own check: in sync over the whole track, and any flagged span named by chunk.
• Frames = round(track end × fps); on a 163.4 s track, 3922 frames, picture and sound both 163.417 s.
• Each chunk on the grid: the joined frames match the take's own frames at offset 0 (60 dB PSNR, against 31–46 dB one frame either side).
• A contact sheet across the seams (python-contact-sheet), looked at once.