Cut phrases from takes

Give it the takes (a mic plus one or two video files recorded together, each with its own crop), each take's word timings and a cut list that names the kept words as text; get every track cut to the same frames, the voice, and timeline.json with each item's comp span and every kept word in comp time. The edit is text, so it is read and approved as text, and a change to it is a one-line edit and a re-run.

For one file with no transcript worth trusting at the word level (a single webcam full of retakes), the speech-island cut in recipe-talking-head-extra-edit (Stage 1) is the sibling. For phrase times on a finished track, python-read-word-timings.

Inputs

Word timings for each take, on the clock of the file you cut the mic from: whisper-cli first (whisper-cli -f take1.wav -l auto -ml 1 -sow -oj -of work/stt/take1, one segment per word, offsets in ms; offer model/whisper-large-v3-turbo for accuracy), or elevenlabs-speech-to-text only if the person agrees to the cost. Both shapes are read as they come.
The takes, probed (ffprobe-inspect-media): screen recordings are often VFR and ultrawide; the script resamples every track to the cut list's fps. A screen and its webcam must start together (numbered pairs); if they don't, trim one to the other first.
The cut list (references/cut-phrases.md has the format and a full example): per item the take, a second near where the words are, and the words exactly as the transcript spells them.

Run it

Write references/cut-phrases.md's script into the job folder as cut_phrases.py, unchanged, then:

exec python cut_phrases.py transcript --stt work/stt/take1.json          # lines broken at gaps > 0.8 s
exec python cut_phrases.py plan  --cutlist cutlist.json                  # the cut as text, with comp times
exec python cut_phrases.py cut   --cutlist cutlist.json --out work/cut   # --threads 8
exec python cut_phrases.py align --clip source/first_cut.mp4 --mic source/cam_4.mp4 --near 7 --search 20

(cwd the job folder.)

transcript each take. Retakes show up as runs of the same opening words; pick the cleanest complete take of every line (later takes are usually better).
Write the items. Copy the words from the transcript, not from memory: punctuation and case don't matter, spelling does ("gonna", "3" against "three").
plan until every phrase resolves; it stops on the first phrase it can't find.
cut. Then bring voice.wav to −16 LUFS with ffmpeg-normalize-loudness.

What it does to each phrase:

Finds the words in that take within 25 s of the approximate second, nearest first.
Pads each side by min(0.12 s, half the gap to the neighbouring word), so a dropped retake or "um" next to it never leaks in.
Splits it at every pause over 0.45 s and keeps 0.22 s of that pause.
Cuts each piece as exactly round(d × fps) frames on every track, with 15 ms audio fades, and joins them by stream copy (every piece shares one encode, -g 12 for fast seeks in the compositor).

What you get

In --out: one MP4 per track (screen.mp4, cam.mp4, same frame count), voice.wav (high-passed and lightly compressed; voice_raw.wav before that), an inline_NNNN.wav per INLINE item with a sound, and timeline.json:

fps, frames, duration, tracks, audio.
items: per item kind (speech, fast, inline, jump, insert), take, src_start, speed, start, end, frames, label, sound_wav, and for speech its words.
words: every kept word {text, start, end} in comp time, ready for captions(words, box) (hn4a-compositor-in-scene-titles) and for cueing zooms and highlights on phrases.

align prints {"at": seconds, "r": correlation}: where the clip's t=0 sits on the mic's clock, from 5 ms loudness envelopes. Use at as the INLINE item's at.

Tips and traps

Split a sentence into two items to drop an "um" between them; the pad stops at half the gap, so the filler stays out.
Cue graphics on phrases, never on typed seconds: read comp times from words, so an edit to the cut list moves every cue with its words.
The mic hears the computer. Anything played back while recording (the product's own videos, another tab) is in the mic and its transcript. Leave it out of speech items; show it as INLINE or INSERT with the original file's sound.
align: a clip played on the computer matched its mic copy at r 0.81 through a room mic. A clearly lower r, or an at far from where you heard it, means the template missed: widen --search, change --length, and listen to the result before using it.
Work in a big screen track (2560×1440 for a 1080p delivery) so the compositor's camera can zoom to 1.3x without softening; the webcam can be a small square face crop.
Long cuts: about 270 items cut in 51 s on 8 threads. Re-running rewrites everything; the cut is cheap, so change the list and re-run rather than patching segments.

Check it

Every track's frame count equals frames in timeline.json (exec ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of csv=p=0 work/cut/screen.mp4), and voice.wav lasts frames ÷ fps.
Transcribe voice.wav with whisper-cli and read it against plan: a clipped first or last word, or a leaked retake, shows up as a missing or extra word at an item edge.
Watch the cut once with the voice under it: the screen and the webcam change together at every item.

Hypernatural