Log in
TechniqueBy HypernaturalVersion 3, updated

ElevenLabs speech-to-text

hn/elevenlabs-speech-to-text

Transcribe a voice track or a video's soundtrack with ElevenLabs speech-to-text (on fal): every word with its start and end time, a diarized speaker id per word, and optional audio events like (laughs). The basis for speaker turns, pauses to cut at, cue times for titles, and checking a voiced take's words and pace. Use whenever you need to know when something is said.

SKILL.md

Give it an audio file (or a video's soundtrack) and you get a JSON transcript: each word with start and end in seconds and a speaker_id, plus spacing entries and, if asked, audio events. Use it to find speaker turns and pauses, to time graphics on words, to cut lip-sync windows at pauses, and to check every voiced take. Turning the JSON into turns, phrase times and pauses is python-read-word-timings.

Inputs

  • An audio file. For a video, extract the soundtrack first, mono is enough:
exec ffmpeg -v error -y -i <video>.mp4 -vn -ac 1 <job>/analysis/<video>_audio.mp3
  • Whether you want audio events (laughs, a phone chime) tagged in the word list.

Run it

Call describe("fal/fal-ai/elevenlabs/speech-to-text") before the first run in a job. The proven fields:

run("fal/fal-ai/elevenlabs/speech-to-text",
    {"audio_url": "<job>/analysis/<video>_audio.mp3", "diarize": true, "tag_audio_events": false},
    into="<job>/analysis", unit="<video>_stt")

Save the result as stt.json next to the audio it came from, and keep it: every later step reads it, and a re-transcription costs a new call. Re-transcribe whenever the audio changes (a new take, a pause edit): timings shift with every edit, so re-cue from the new file rather than nudging old numbers.

What you get

words: a list of entries with text, start, end, type and speaker_id. type is word, spacing (the gaps, skip them) or audio_event (with tag_audio_events). Speaker ids are speaker_0, speaker_1, … in order of first appearance. A track with nobody speaking returns an empty word list, which is a valid answer, not an error.

Tips and traps

  • Diarization clusters voices, not people. A deepfake or an impersonation shares the real person's id. Map ids to characters yourself, by watching who speaks (mouth and voice), and record the map where you keep the job's notes.
  • Word times are close, not frame-exact: at a hand-off between two speakers they were about 0.1 s off where the cut belonged. For a cut that must land between two speakers, find it by ear from a ladder of trims (elevenlabs-voice-isolation).
  • Where a clip's speech ends: use the last word's end plus about 0.25 s, not a silence detector; room tone fools it.
  • Checking a voiced take (TTS or text-to-dialogue): transcribe it and compare the words to the script, the speaking rate per character (words per second over each character's lines), and the length of any pause a note mentions.

Check it

  • Word count roughly matches the script or what you hear; no long stretch of speech without words.
  • The speaker ids change where the voices change; spot-check two or three turns by ear.

Workflows that use this skill

FAQs

Transcribe a voice track or a video's soundtrack with ElevenLabs speech-to-text (on fal): every word with its start and end time, a diarized speaker id per word, and optional audio events like (laughs). The basis for speaker turns, pauses to cut at, cue times for titles, and checking a voiced take's words and pace. Use whenever you need to know when something is said.

Install Hypernatural Desktop, a free Mac app, and connect it to Claude Code, Codex or Claude Desktop. Then describe the video you want and your agent finds the skill itself, or paste this page's link (https://hypernatural.ai/skill/hn/elevenlabs-speech-to-text) and it installs hn/elevenlabs-speech-to-text and follows it.

It runs fal/fal-ai/elevenlabs/speech-to-text. Hypernatural Desktop has every one of them built in, so there are no API keys to set up, though you can bring your own.

The skill is free. You only pay for the AI generations it runs, and your agent shows you the price of every batch before it spends anything. Editing, captions and titles run on your Mac at no cost.

Yes. Ask your agent to adjust it for your job, or to save its own version. You can publish your version under your own handle so anyone can install it.

Go make something cool.

Install the app, open your agent, describe the video. That's the whole onboarding.

Sticker HypernaturalSticker BannerSticker TVBoy