Give it an audio file (or a video's soundtrack) and you get a JSON transcript: each
word with start and end in seconds and a speaker_id, plus spacing entries and,
if asked, audio events. Use it to find speaker turns and pauses, to time graphics on
words, to cut lip-sync windows at pauses, and to check every voiced take. Turning the
JSON into turns, phrase times and pauses is python-read-word-timings.
Inputs
- An audio file. For a video, extract the soundtrack first, mono is enough:
exec ffmpeg -v error -y -i <video>.mp4 -vn -ac 1 <job>/analysis/<video>_audio.mp3
- Whether you want audio events (laughs, a phone chime) tagged in the word list.
Run it
Call describe("fal/fal-ai/elevenlabs/speech-to-text") before the first run in a job.
The proven fields:
run("fal/fal-ai/elevenlabs/speech-to-text",
{"audio_url": "<job>/analysis/<video>_audio.mp3", "diarize": true, "tag_audio_events": false},
into="<job>/analysis", unit="<video>_stt")
Save the result as stt.json next to the audio it came from, and keep it: every later
step reads it, and a re-transcription costs a new call. Re-transcribe whenever the
audio changes (a new take, a pause edit): timings shift with every edit, so re-cue
from the new file rather than nudging old numbers.
What you get
words: a list of entries with text, start, end, type and speaker_id.
type is word, spacing (the gaps, skip them) or audio_event (with
tag_audio_events). Speaker ids are speaker_0, speaker_1, … in order of first
appearance. A track with nobody speaking returns an empty word list, which is a valid
answer, not an error.
Tips and traps
- Diarization clusters voices, not people. A deepfake or an impersonation shares the real person's id. Map ids to characters yourself, by watching who speaks (mouth and voice), and record the map where you keep the job's notes.
- Word times are close, not frame-exact: at a hand-off between two speakers they were
about 0.1 s off where the cut belonged. For a cut that must land between two
speakers, find it by ear from a ladder of trims (
elevenlabs-voice-isolation). - Where a clip's speech ends: use the last word's
endplus about 0.25 s, not a silence detector; room tone fools it. - Checking a voiced take (TTS or text-to-dialogue): transcribe it and compare the words to the script, the speaking rate per character (words per second over each character's lines), and the length of any pause a note mentions.
Check it
- Word count roughly matches the script or what you hear; no long stretch of speech without words.
- The speaker ids change where the voices change; spot-check two or three turns by ear.


