Transcribe with word times (whisper-cli)

Give it speech (a recording, a voice take, a rendered mix); get a JSON with every word and its start and end, free and on this machine. Read it with python-read-word-timings for phrases, cues, pauses and a check against the script. Use elevenlabs-speech-to-text instead when several speakers must be told apart (it diarizes), and a TTS take's own alignment instead when you only need to cue on a generated voice (python-read-word-timings).

Inputs

• The audio as a 16 kHz mono WAV. whisper-cli reads WAV, FLAC, MP3 and OGG; an MP4, MOV or AIFF fails with "failed to read audio file". Pull the sound out first.
• The model: exec runs whisper-cli with the largest Whisper model installed. Base comes with it; offer the person model/whisper-large-v3-turbo (1.5 GB) when the words will be seen (captions, a script check) or base gets them wrong.
• For a movie, its audio stream's start_time (ffprobe-inspect-media).

Run it

exec ffmpeg -v error -y -i source/take1.mp4 -vn -ar 16000 -ac 1 work/stt/take1.wav
exec whisper-cli -f work/stt/take1.wav -l auto -ml 1 -sow -oj -of work/stt/take1

• -ml 1 -sow: one word a segment, split on words; -oj writes work/stt/take1.json (-of takes the path without .json). -l auto detects the language; pass it (-l de) when you know it. -ojf adds tokens with their probabilities; python-read-word-timings reads either.
• Several files at once: start each with exec(wait: false) and wait for them all with tasks(wait: true, execs: [...]) before reading any. Look at the pictures while they run.
• A movie's time is the WAV's time plus its audio's start. The extracted WAV starts at the first audio sample, so when ffprobe -show_entries stream=codec_type,start_time gives audio 0.079 against video 0, a word at 2.02 s in the WAV is at 2.10 s in the movie. Screen recordings often start their sound about 0.1 s in.

What you get

transcription: one segment per word with offsets (from, to) in milliseconds and the word in text with a leading space and its punctuation (" chat."); silence can come back as an empty segment or [BLANK_AUDIO]. python-read-word-timings words turns it into [{text, start, end}] in seconds and drops those.

Tips and traps

• Word times are for finding and cueing, not for cutting on a frame. On a full file, base-model times drifted up to 1 s. A 3 s window cut out and transcribed on its own got the words right and their times up to 0.7 s wrong. Transcribe a window to learn which words it holds; cut where the audio itself goes quiet (a 10 ms RMS envelope), and check the cut by transcribing it.
• Don't window with -ot/-d. A 3 s window asked for with -ot 4000 -d 3000 ran on to 9.4 s. Cut the window into its own WAV (ffmpeg -ss A -to B) and add A to its times.
• Base loses sentence ends and names. On the same read, base heard "Akmi" for a product called Acme and turned 4 of 7 full stops into commas; large-v3-turbo got both right. Punctuation drives sentence-based chunking and captions, so use the large model there.
• Numbers come back as digits ("2 million" for "two million"), and a cue phrase written from the script won't match: read words first.
• "Thank you" on silence: whisper can hallucinate a closing phrase on a quiet tail. Drop words that sit where the audio is silent.
• Names are misspelled the same way each time. Keep a list of the fixes with the job's notes and apply it before captions; never let a misspelled name reach a caption.
• Words start early after a long silence. On a vocal with 2–4 s gaps between its lines, whisper put the first word at 0.0 s (sung at 2.5 s) and other line starts up to 2.4 s early, with either model. The same vocal cut into utterances (python-find-speech-pauses, runs), each transcribed on its own and stitched back (python-read-word-timings, stitch), put every line start within 0.06 s. Do that wherever the gaps are long, and hear the track before you trust a transcript that disagrees with it.
• The mic hears the computer: anything played back while recording (a product's own video, a tab) is in the transcript. Mark it before picking takes.

Check it

Check a voice take, or a finished mix (pulled to WAV first), against its script:

exec python scripts/word_timings.py check --stt work/stt/voice_a.json --script source/script.txt

Every script word heard (matched equal to script_words), no tag_spoken: true (an [audio tag] read aloud), and each difference heard and explained (a digit, a misspelled name) before the take is used. python-read-word-timings has the command.