Give it the word times of a piece of speech, in whichever shape you have them; get
the timing facts an edit runs on, as JSON: words, phrase times, cue times, a check
against the script, speaker turns, pauses, and where a clip's own speech ends. To make
the word times in the first place, transcribe with
whisper-cli-transcribe-with-word-times (free) or elevenlabs-speech-to-text (paid,
diarized). To cut a voice track into lip-sync chunks, python-plan-lipsync-chunks,
which reads word times through this script.
Inputs
One JSON of word times, any of these (it tells them apart by their keys):
| Source | Shape | Use it when |
|---|---|---|
whisper-cli with -ml 1 -sow -oj (or -ojf) | transcription segments, ms | Free; any recorded or rendered speech |
elevenlabs-speech-to-text | words with speaker_id | Several speakers to tell apart (turns) |
A TTS take's alignment (elevenlabs-tts-v4 with timestamps) | characters with start and end times | Cueing a build on a generated voice: the times come free with the take |
Times estimated from a script (fake_timings.py) | words | The first look, before any voice exists |
Times are on the clock of the audio that was timed: time the exact file whose timeline
you will cut or cue against. For check, the script as plain text, audio tags and all.
Run it
Write references/word_timings.md's script into the job's scripts/ as word_timings.py,
unchanged, and run one command per question (--stt takes any of the shapes above):
exec("python", ["scripts/word_timings.py", "check", "--stt", "audio/voice_a.alignment.json", "--script", "script.txt"], cwd="<job>")
| Command | Gives | Use it to |
|---|---|---|
words --stt S | [{text, start, end, speaker_id}], spacing and sound events dropped | see the transcript with times |
phrase --stt S [--after T] "P1" "P2" … | [{phrase, start, end}], each found after the one before; null if not spoken | cue a title, a slide build or a graphic on the word |
cues --stt S [--lead 0.12] name="phrase" … | {lead, last_word_end, cues: {name: start − lead}}; stops with an error on a phrase not spoken | the cue table a build script runs on |
check --stt S --script T | {script_words, heard_words, matched, differences: [{at, script, heard, tag_spoken}]} | check a voice take or a mix against its script |
turns --stt S [--max-gap 0.7] [--name speaker_0=Nora] | [{speaker, start, end, text}]: one speaker's consecutive words, split at pauses over 0.7 s | find who speaks when; cut a shot's dialogue window |
pauses --stt S [--start A] [--end B] [--min-gap 0.2] | [{at, gap}]: midpoint and length of each silence of 0.2 s or more | find safe places to cut a track |
speech-end --stt S [--window W] [--tail 0.25] | {last_word_end, plays_for} | how long a clip's own re-voiced audio plays |
stitch --runs R --out W | one words JSON (the speech-to-text shape every command reads) from utterances transcribed one by one, each moved to its start | word times for a recording with long silences: a sung vocal, a webcam file of retakes |
A build script beside it in scripts/ imports the same functions rather than shelling
out:
from word_timings import read_words, phrase_time, cue_times
(hn4a-compositor, references/build-script.md).
Reading the results
- Phrases match case- and punctuation-blind, whole words in sequence. Pass a few
words that occur once ("Finally, report"), and pass several phrases in reading order
so each is found after the last.
nullmeans the voice didn't say it as written: readwordsand fix the phrase, never guess a time. - Cues are found in the order given, each after the one before, so a name said twice (in the title and on the end card) lands on the right one. A missing phrase stops the command: the script and the take have drifted apart, so fix the phrase or re-take, never type a time.
- Check: every difference is worth a look;
matchedequal toscript_wordsand no differences is a clean take.tag_spoken: truemeans words from an[audio tag]were read aloud: re-take that line. A transcript writes numbers as digits ("3" for "three") and can misspell names (whisper-cli-transcribe-with-word-times): those differences are the transcript's, not the take's, once you have heard them. - Speakers are voice clusters, not people.
speaker_0,speaker_1are numbered in order of first appearance; the ids follow how the speech sounds, not who is on screen. Listen, then map ids to names with--name. Only speech-to-text has them. - Pauses: cut at
at, the midpoint of the silence, never at a word's edge. - speech-end: run on the transcript of a clip's own soundtrack (a reference-audio
render re-voices the line, and its timing drifts from the track).
plays_foris the longer of the planned window and the last word plus 0.25 s, so a line that runs past its window isn't cut off. Room tone fools a silence detector; the transcript doesn't.
Tips and traps
- Cue a generated voice from its own alignment, not a transcript of it: the times are exact, free, and spelled as the script is. A transcript is for recorded speech, and for checking the take's words.
- Sentence ends come from the source's punctuation. whisper's base model turned 4
of 7 full stops into commas on a short read; the large model kept all 7. Anything that
splits on sentence ends (
python-plan-lipsync-chunks) needs the large model, or the alignment. - A cue phrase is matched against the transcript's spelling, not the script's: a
cue on "Over three million" stopped on a transcript that heard "Over 3 million". Read
wordsfirst when a cue word is a number, an acronym or a name, or cue on the words next to it. - whisper JSON with whole phrases per segment (run without
-ml 1) is refused: re-run with-ml 1 -sow -oj. - Word times are good for cueing, not for the exact frame where one
speaker ends and the next begins: there they were about 0.1 s off. Pick that cut by
ear from a ladder of trims (
ffmpeg-cut-audio-window). - After a re-voice or a retimed track, time it again; old times are wrong by however much the audio moved.


