ElevenLabs speech-to-text

Give it an audio file (or a video's soundtrack) and you get a JSON transcript: each word with start and end in seconds and a speaker_id, plus spacing entries and, if asked, audio events. Use it to find speaker turns and pauses, to time graphics on words, to cut lip-sync windows at pauses, and to check every voiced take. Turning the JSON into turns, phrase times and pauses is python-read-word-timings.

Inputs

An audio file. For a video, extract the soundtrack first, mono is enough:

exec ffmpeg -v error -y -i <video>.mp4 -vn -ac 1 <job>/analysis/<video>_audio.mp3

Whether you want audio events (laughs, a phone chime) tagged in the word list.

Run it

Call describe("fal/fal-ai/elevenlabs/speech-to-text") before the first run in a job. The proven fields:

run("fal/fal-ai/elevenlabs/speech-to-text",
    {"audio_url": "<job>/analysis/<video>_audio.mp3", "diarize": true, "tag_audio_events": false},
    into="<job>/analysis", unit="<video>_stt")

Save the result as stt.json next to the audio it came from, and keep it: every later step reads it, and a re-transcription costs a new call. Re-transcribe whenever the audio changes (a new take, a pause edit): timings shift with every edit, so re-cue from the new file rather than nudging old numbers.

What you get

words: a list of entries with text, start, end, type and speaker_id. type is word, spacing (the gaps, skip them) or audio_event (with tag_audio_events). Speaker ids are speaker_0, speaker_1, … in order of first appearance. A track with nobody speaking returns an empty word list, which is a valid answer, not an error.

Tips and traps

Diarization clusters voices, not people. A deepfake or an impersonation shares the real person's id. Map ids to characters yourself, by watching who speaks (mouth and voice), and write the map into AGENTS.md.
Word times are close, not frame-exact: at a hand-off between two speakers they were about 0.1 s off where the cut belonged. For a cut that must land between two speakers, find it by ear from a ladder of trims (elevenlabs-voice-isolation).
Where a clip's speech ends: use the last word's end plus about 0.25 s, not a silence detector; room tone fools it.
Checking a voiced take (TTS or text-to-dialogue): transcribe it and compare the words to the script, the speaking rate per character (words per second over each character's lines), and the length of any pause a note mentions.

Check it

Word count roughly matches the script or what you hear; no long stretch of speech without words.
The speaker ids change where the voices change; spot-check two or three turns by ear.

Hypernatural