Script from video

Give it a video, of any length, with or without speech, and optionally a line of context. You get script.json (scenes, then shots, then elements, timed), script.md to read, and script.fountain to open in a screenplay app. The work is split between two models. Speech-to-text gives the exact words, their timings and voice clusters. Gemini watches the picture with its sound, at 4 fps, and writes everything else. Code makes the shots from the edit cuts and files each line under its shot. When the person has a written script instead of a video, use llm-convert-written-script. It writes the same script.json.

Inputs

The video. Check it with ffprobe-inspect-media.
Optional context: what the video is, and who people are when the video doesn't say ("a security-awareness training drama; the woman in the blazer is a security expert"). It helps with roles and names.

Run it

Use a work folder in the project (<job>/script/). Write two files from references/ into the project folder, unchanged: script-tools.md's script as script_tools.py and prompt.md's template as script_prompt.txt.

Prepare. It probes the video, writes the soundtrack to audio.mp3 (mono, 128k), finds the edit cuts (scene score > 0.3 at 320 px, the way ffmpeg-scene-cuts does), and makes a shot at every cut. Cuts under 0.4 s apart count once. It splits the video into chunks of at most 300 s, each ending at a cut in its last 40% when there is one. It saves each shot's middle frame (shots/shotNN.jpg, 480 px) and a 720p proxy per chunk.
```
exec python script_tools.py prepare --video <video> --work <job>/script
```
Listen. Run speech-to-text on audio.mp3 (elevenlabs-speech-to-text): run("fal/fal-ai/elevenlabs/speech-to-text", {audio_url: "<job>/script/audio.mp3", diarize: true, tag_audio_events: true}, into: "<job>/script"). Save the result with its words list as <job>/script/stt.json. With no audio stream, prepare has already written an empty one.
Watch, one chunk at a time, in order. Each chunk's prompt carries the cast and the last scene found so far, so chunk N waits for chunk N−1.
```
exec python script_tools.py prompt --work <job>/script --chunk 1 --template script_prompt.txt --context "<context>"
```
It writes chunks/chunk01_parts.json: the proxy sampled at 4 fps with its sound, each shot's still after a Shot N, middle frame: label, then the prompt, as Gemini parts with local paths. Send them to Gemini 3.1 Pro through run. Find its id with models("gemini") and describe it before the first run (earlier jobs used gemini-3.1-pro-preview):
```
run("<gemini id>", {
  "contents": [{"role": "user", "parts": <the list in chunks/chunk01_parts.json>}],
  "generationConfig": {"responseMimeType": "application/json",
                       "responseSchema": <the schema in references/prompt.md>,
                       "maxOutputTokens": 65536}},
  into="<job>/script/chunks", unit="chunk01")
```
Save the JSON answer as chunks/chunk01.json. Repeat for chunk 2, and so on. The gateway's Gemini adapter is being built in hn4a; describe gives its exact fields.
Assemble.
```
exec python script_tools.py assemble --work <job>/script
```
It enforces gendered labels and merges the cast across chunks by label. Dialogue is filed into shots by its transcript time. Everything else goes to the shot Gemini named. Scenes are renumbered from 1. It writes script.json, script.md and script.fountain.

Every step is cached in the work folder. To redo one chunk, delete its chunkNN.json and rerun from prompt. To re-transcribe, delete stt.json.

What you get

script.json in the format in references/script-json.md, plus script.md (cast table, scene table, story beats, then the script by scene and shot) and script.fountain. Assemble prints the scene, shot and line counts, and how many lines lack a parenthetical or emotion.

Tips and traps

Who spoke comes from mouths, not diarization. Speaker ids cluster voices. They don't identify people. Two people can share an id, one person can be split across two, and an impostor voicing a deepfake shares the real man's id. The prompt has Gemini attribute each line by watching and listening, and keep the transcript's words and timings.
The labelled stills keep shots straight. Gemini's sense of time drifts a shot or two over a couple of minutes. Without stills its shot descriptions swapped two shots and ran one shot off for the last third of a 2:50 video. With them (about 25k image tokens) all 23 shots matched.
A scene is a sequence. Told "new location = new scene", Gemini made 16 scenes of that 2:50 video, one per cutaway. Defined as a run of shots that keeps its cutaways, inserts and B-roll, it made 5, which matched the person's own split.
Gendered labels are enforced in code. Gemini still returns a bare role (EXPERT) despite the prompt, so assemble prefixes FEMALE or MALE to any unnamed label without a gendered word. Action prose may still say "the Expert".
Delivery notes must be mandatory. A first prompt got a parenthetical on 5 of 27 lines and no beats. "Every line, even a calm one" brought it to 26 of 26.
Names only when spoken or shown. A spoken form of address counts: "Morning, rookie" makes ROOKIE.
Size and time: a 2:50 1080p video with four speakers was one chunk of about 78k prompt tokens (49k of video at 4 fps, 25k of stills) and took about 2 min. A frame costs about 70 tokens at the default media resolution. Raise the fps (prompt --fps 6) for rapid exchanges. A 1-hour video is 12 chunks, run one after another.
A request can hang on an open socket, once for about 3 h across a laptop sleep. If a chunk runs far past a few minutes, run it again: a model call can't be stopped, so leave the hung one and follow the new call's id.

Check it

Read script.md against the video. The shots should match the cuts (check a contact sheet from ffmpeg-scene-cuts when in doubt), every line should have the right speaker, and every line should have a parenthetical and an emotion. Fix a wrong attribution or label in chunks/chunkNN.json and rerun assemble. Don't edit script.json by hand.