Find a recorded line in a render

Give it one or more renders and the recorded line they should speak; get, per render, the offset where the line starts, the correlation of the match, and the start frame. Cut each render from that frame to the shot's length, with the recorded line as its audio, and the line lands where the original's does.

It lines up the start only. A render that re-paced the line (spoke it slower, added a pause) still drifts after the start; fix that with a mouth-only re-sync (heygen-v3-resync-existing-take, veed-v2-resync-existing-take) or a render with the line timed in the prompt.

Inputs

• The renders: videos with their own soundtrack (the model's re-voiced line).
• The line: the original's recorded line as WAV, starting at the shot's first frame and ending with the line (ffmpeg-cut-audio-window). Each render must be at least as long as the line.
• The shot's length in frames and its fps, from the original.

Run it

Write references/find_line.md's script into the project folder as find_line.py, unchanged, and run it:

exec("python", ["find_line.py", "--line", "audio/shot03_line.wav", "--fps", "24",
  "video/raw/shot03_seedance_a.mp4", "video/raw/shot03_seedance_b.mp4"], cwd="<job>")

[
  {"render": "video/raw/shot03_seedance_a.mp4", "offset_s": 0.53, "envelope_corr": 0.874, "start_frame": 13},
  {"render": "video/raw/shot03_seedance_b.mp4", "offset_s": 1.31, "envelope_corr": 0.412, "start_frame": 31}
]

Then cut each render from start_frame for the shot's length plus a 0.5 s margin (here 33 + 12 frames), with the recorded line as its audio, so the review judges lip sync against what will air:

exec ffmpeg -v error -y -i video/raw/shot03_seedance_a.mp4 -i audio/shot03_line.wav \
  -filter_complex "[0:v]trim=start_frame=13:end_frame=58,setpts=PTS-STARTPTS[v]" \
  -map "[v]" -map 1:a -c:v libx264 -crf 14 -preset slow -pix_fmt yuv420p -c:a aac \
  video/shot03_seedance_a.mp4

Keep the untrimmed render in raw/, and record each take's offset and correlation next to the takes.

Reading the numbers

• offset_s: where the line starts in the render, to 10 ms.
• envelope_corr: the normalized correlation at that offset, −1 to 1. A low value usually means extra words or noise past the line in the render, or a line performed with a different rhythm; listen before trusting its offset.
• start_frame: round(offset_s × fps), the first frame to keep.

Tips and traps

• Check by ear. Play the cut with the recorded line under it; the offset is a measurement, not a verdict.
• The search slides the line over the whole render, one 10 ms step at a time; it takes seconds on a shot and grows with render length. Search a short render, not a full programme.
• A model that re-performs an untimed reference line can start up to 1.3 s late and run about 10% long. The cut fixes the start; the length difference shows as drift by the end of the line. Check the cut take with hn4a-lipsync-check against the line.
• Never pick where one speaker ends and the next begins from envelope minima: they were about 0.1 s off on a line that ran into the next speaker's. Pick that by ear (ffmpeg-cut-audio-window).