ElevenLabs text-to-dialogue

Give it a passage of a script as an ordered list of (voice, tagged line) and you get one take of the whole passage, every line in its character's voice, plus each line's span in that take. Run it once per passage, then join the passages with python-assemble-dialogue-track. For one voice reading alone (auditions, narration), elevenlabs-tts-v4 is simpler.

Inputs

• The script as scenes, shots and lines, each line with its character, parenthetical, emotion and script timing (the script.json the script leaves produce).
• The cast: per speaking character, a voice id and its performance guide (from casting), with the person's standing direction at the top of the guide.
• output_format: wav_48000 (lossless, a WAV as it comes; pcm_48000 is the same samples without a header).

Prompt

Load references/direction.md and direct each passage yourself: tags from each line's parenthetical and emotion, the beats around it and the guide; the words untouched and checked against the script with its word check. Group scenes into passages as it says (a scene is a passage unless a line runs across the cut). Save the tagged lines and their one-sentence intents as directed.json in the take folder.

Run it

Text-to-dialogue with timestamps is a model under run, elevenlabs/text-to-dialogue/with-timestamps; describe it before the first run. Its input is ElevenLabs' own body, run once per passage, with the take folder as into and the passage as unit:

run(model: "elevenlabs/text-to-dialogue/with-timestamps", cwd: "<job folder>",
    into: "take_a", unit: "group0", tag: "group0", input: {
      "inputs": [{"voice_id": "<manager voice>", "text": "[dry] You signed off on that?"},
                 {"voice_id": "<assistant voice>", "text": "[pause] [quietly] I did. [firmly] And I'll fix it before lunch."}],
      "model_id": "eleven_v4", "settings": {"stability": 0.5}, "seed": 7,
      "output_format": "wav_48000"})

It writes two files, lettered like every take (a re-run in the same folder is group0_b_…):

• take_a/group0_a_speech.wav: the passage's audio.
• take_a/group0_a_timestamps.json: voice_segments, each with dialogue_input_index, start_time_seconds and end_time_seconds, and the take's alignment per character.

Turn them into group0.wav and the spans file group0.json with the script in references/dialogue-response.md. Through Hypernatural it's billed a credit a character of every line's text, settled to what ElevenLabs reports; estimate: true prices a passage first. If models("elevenlabs text to dialogue") doesn't list it, voice each line with elevenlabs-tts-v4 instead, joined with python-assemble-dialogue-track.

Settings:

• stability: 0.5 by default; 0 is the most expressive. For more expression, set 0 on the passage that needs it only, not the whole script.
• seed: 7 throughout; keep it fixed across a take's passages and their re-runs.
• use_pvc_as_ivc: true uses a professional clone's instant-clone version instead.

A single input works too: a presenter's chunk was re-voiced this way (stability 0), which gives the line's span along with the audio.

What you get

Per passage: the audio and each line's start and end. Spans are approximate where a line's tail runs into the next line's pause; exact speech times come from transcribing the assembled track.

Tips and traps

• Takes are folders, take_a, take_b, … with letters never reused. To act on a director's notes for specific lines, write those lines' tags by hand into the new take's directed.json (for example [smug] [delighted] [almost whispering] Bingo.), re-run only the passages they're in and copy the rest from the old take. Re-directing the whole passage from scratch loses the notes.
• Fixes in order: hand tags, then a pause edit, then tempo, then stability. Pauses: tags land only roughly, so measure in the transcript and edit (python-assemble-dialogue-track pause). Tempo: the speed setting is ignored, so a slow voice is sped up after the fact (retime, at most 1.2).
• On eleven_v4, every take of a 52-line, two-variant script came back word-perfect and both lead voices read at about 2.9 words/s; on v3 one voice read at 2.4 words/s however it was directed. v4 also lands pause tags better.
• previous_text / future_text (continuity between passages) are rejected by v3; don't rely on them.
• Library professional voice clones are trained per model, and only the owner can train one for v4; an untrained one renders but can drift (Voice B "sounded wrong" on v3). Prefer voices verified for eleven_v4.

Check it

Assemble, then transcribe the track (elevenlabs-speech-to-text). Check each line's words against the script, each character's speaking rate (words per second), and every pause a note mentions. The voices were approved at casting, so a take that passes goes on without a listening gate; line notes come when the person hears it with the pictures.