Assemble a dialogue track

Give it the passages of a performed script (each one text-to-dialogue take with its line spans) and you get one track, track.wav and track.mp3, plus track.json holding every line's scene, shot, character, words, tags and span on the new track. Pause edits and per-character retiming happen here too, so the line times stay true after every edit. It does not voice anything (elevenlabs-text-to-dialogue does) and does not set loudness (ffmpeg-normalize-loudness, run on the finished track).

Inputs

Each passage's audio (group<N>.wav, pcm_48000 recommended) and its spans file (group<N>.json), in the formats in references/dialogue-track.md.
passages.json listing the passages in script order with their lines and the script's timings.
For pause edits by phrase and for line-times: a speech-to-text JSON of the audio (elevenlabs-speech-to-text).

Run it

Write the script in references/dialogue-track.md into the project folder as dialogue_track.py, as is, and run it with exec python. Keep its constants: the thresholds below depend on them.

exec python dialogue_track.py assemble --passages take_a/passages.json --out take_a/track
exec python dialogue_track.py retime --audio take_a/group2.wav --spans take_a/group2.json \
    --tempo PRESENTER=1.15 --out take_a/group2_tempo
exec python dialogue_track.py pause --audio take_a/track.wav --meta take_a/track.json \
    --after "Thanks for the help." --stt take_a/stt.json --delta 2.4 --out take_b/track
exec python dialogue_track.py line-times --track take_b/track.json --stt take_b/stt.json

assemble: each passage is preceded by the script's own silence before its first line, clamped to 0.4–2.5 s, then concatenated as mono at the first passage's sample rate (--sample-rate to force one; match the passages so nothing is resampled down). Writes <out>.wav, <out>.mp3 (192 kbps) and <out>.json.
retime: speeds (or slows) the named characters' lines in one passage with atempo, pitch kept. Each line owns the audio from its start to the next line's start, its trailing pause included. Writes <out>.wav and <out>.json (new spans); point the passage at those before assemble.
pause: inserts silence (--delta > 0) or cuts it (< 0) at --at seconds, or just after a phrase's last word (--after with --stt: 0.25 s after the word, or halfway to the next word if that is sooner). Every later line moves by the delta. Works on a passage (with its spans file) or the finished track (with track.json).
line-times: each line's first-word start and last-word end, by aligning the track's transcript to the script's words in order.

What you get

track.json is the source of truth for line times: shot windows for lip-sync, cue points for titles, the per-shot audio an editor gets. A line's span in track.json runs on into the pause after it; use line-times when you need where the speech itself starts and stops (cutting a shot's audio, cueing a word).

Tips and traps

Make every edit through this script, never by trimming audio by hand, so track.json moves with the audio. An edit is a new take folder (letters never reused); copy the take, edit, re-transcribe.
Tempo: ElevenLabs v3 and v4 ignore the speed voice setting, so a voice that reads slower than the scene needs is fixed here. Stay at 1.2 or below. Measure first: words per second per character from the transcript. On one job a voice read at 2.4 words/s on v3 however it was directed and took 1.2; on v4 both voices read at about 2.9 words/s and needed nothing.
Pauses: pause tags land only roughly. Measure the pause in the transcript, then set it here. Plan silence into the track wherever the picture needs action time (a character stands and walks out after a line) before any video is rendered.
Normalize loudness once, after the last edit, on the whole track; never per passage.

Check it

Transcribe the new track and run line-times: every line found, in order, with the words the script has.
track.json duration matches ffprobe on track.wav within a few milliseconds.
Listen across each passage join and each edited pause: no click, no clipped word.

Hypernatural