Screen walkthrough from screen and webcam recordings

The person brings paired recordings, numbered so each screen capture starts with its webcam file (the screen has no sound; the webcam carries the mic), the brand kit (fonts, colours, logo, any mascot art) and the finished files the walkthrough shows off. You deliver a 1920×1080 cut of the best takes: the screen under a camera that zooms to and highlights what is being talked about, the presenter in a round corner bubble, karaoke captions, a brand motion graphic on every jump in time, and the finished videos full screen with their own sound.

Everything is local and free except speech-to-text, which is whisper-cli unless the person agrees to pay for elevenlabs-speech-to-text. Record the pairs, crops, bubble side and fps in job-notes.md, and the brand answers in brand-kit.md (ask for the kit as recipe-talking-head-extra-edit does, Stage 2).

Steps

1. Probe and transcribe (ffprobe-inspect-media). Probe every file: screen recordings come VFR and often ultrawide; webcams 1080p30. Extract each webcam's audio and transcribe all of them in parallel with whisper-cli (-l auto -ml 1 -sow -oj; offer model/whisper-large-v3-turbo), each started with exec(wait: false). Look at the screens (step 2) while they run, and wait for all of them (tasks(wait: true, execs: [...])) before the paper edit. The mic heard the computer: anything played back while recording (the product's own videos, a tab) is in the transcript, so mark it now.
2. Look at the screen. A 4×5 contact sheet per recording at one frame per 10 s, labelled in Pillow (the local ffmpeg may have no drawtext). Choose one 16:9 region per recording that holds everything shown, and check one crop at full resolution. Note which way the presenter looks (toward their screen): the bubble goes on that side.
3. Write the paper edit (python-cut-phrases-from-takes): transcript each take, pick the cleanest complete take of every line (later takes usually are), and write the cut list as text. Get to the UI within seconds. Add a JUMP with a label ("LATER THAT DAY", "A FEW MINUTES ON") wherever time passes, a 2x FAST item for any action that pays off a line (a line about asking for changes → the typing), an INLINE item for a file the presenter plays inside the UI (aligned with align), and an INSERT for a finished file shown full screen. plan until it resolves. Gate: the person reads the plan as text with its total length; a cut list change costs nothing now.
4. Cut the tracks (python-cut-phrases-from-takes, ffmpeg-normalize-loudness): the screen crop scaled to 2560×1440 (room to zoom), a 720×720 face crop of the webcam, the mic, on one 30 fps grid; the voice to −16 LUFS. Mux the screen track and voice into a plain rough cut. Gate: the rough cut on a review page, before any graphics.
5. Build the spec (guide-add-titles-and-motion-graphics, hn4a-compositor-motion-graphics, hn4a-compositor-in-scene-titles): one build script reads timeline.json and writes the spec, bottom first:
   • An ink background, then the screen track in a camera() group. Framings are cued on phrases from words ("So the first is"), never typed seconds, so a cut list change keeps them on their words. Each framing holds with a 4% creep, then eases 0.7 s to the next, landing 0.15 s before its phrase. Clamp framings to the screen, except one that frames cards under the bubble: frame those beside it and let the ink show past the screen edge.
   • Highlights parented to the camera: a transparent fill 10 px larger than the element, radius 14, a 5 px ring in the brand's accent plus a soft glow, popping in 0.1 s before its phrase and out at the next phrase or 0.15 s before its shot ends. Measure every box on a gridded full-resolution frame (Dead ends).
   • The webcam: full frame for the opening line, then on the presenter's own gesture its box, crop and radius keys shrink it into a 270 px bubble 44 px from the corner over 0.7 s, while the screen flies in from the side they point to (an x and a −6° rotate key). Then the bubble track for the rest, a 6 px accent ring, faded out during inserts.
   • Karaoke captions() on words in Brand Sans 800, 54 px, the spoken word on an accent pill, every word with a 4 px ink outline (a ring of text-shadows), in a box above the bottom edge clear of the bubble; skip the insert spans.
   • Each JUMP: a dotted panel in the brand's primary colour wipes across with an accent band leading it, the mascot runs through bobbing every 0.22 s, a clock spins beside the label in the brand's display face. 0.35 s lead and tail around the 1.6 s hold.
   • Each INSERT: the finished vertical video 1000 px tall centred on the primary colour, an accent ring, a short tag ("THE RESULT") and the mascot. An end card of 3.2 s.
   • audio: voice.wav, each inline_NNNN.wav at its item's start, each insert's file at its start at 0.9.
6. Stills, then the live spec. A still at every framing, highlight, jump and insert (still --at), run one after another. Fix boxes in the build script and rebuild. Gate: review_open with the spec as a {"compose": "cut/walkthrough.json"} candidate, the stills as its refs; render only after a yes.
7. Render and deliver (core skill): 8 workers, frame count against round(duration × fps), a contact sheet looked at. Gate: the render on a review page beside the rough cut or the previous round.

Lessons from the first round

The first delivery came back with these notes; build them in from the start.

• Get to the UI in seconds. A 15 s full-frame webcam opening was too long. Cue the switch on the presenter's gesture: they pointed up-left on "this", so the screen flew in from the left as they shrank into the bubble.
• Bubble on the side the presenter looks toward. They looked right, at their screen, so a bottom-right bubble looked away from the UI; bottom-left looks into it.
• Zoom less. 2.1x cut off text. Keep framings at 1.15–1.6x and frame the whole thing being talked about (the full card, the header with the price), not its middle.
• A highlight ends with its shot. One timed "+3 s" stayed up after the cut to the next recording; end highlights on phrases or on the shot's end.
• Keep the action that pays off a line. After a line about asking the app for a livelier read, the person wanted to see the typing: a 2x FAST item, keyboard sound at tempo.
• A clip shown inside the UI stays inside the UI. The first cut played in the app's review panel, so show it there (INLINE, zoomed on the player) with the file's own sound aligned to the mic by envelope cross-correlation (5 ms hops, r 0.81). Only the finished video at the end goes full screen.

Choices this recipe makes

• A text paper edit, not a waveform cut. Takes are many and split across files, and the person approves words; the cut list names them. For one webcam file and no screen, the island cut in recipe-talking-head-extra-edit is the route.
• Cues on phrases. Every zoom, highlight and caption reads comp times from timeline.json, so a re-cut rebuilds the spec without touching a number.
• A 2560×1440 screen track under a 1080p frame: framings up to 1.33x lose no detail, and 1.6x stays sharp enough for UI text.
• Outlined captions. Plain white captions vanished over a white UI; the 4 px ink outline reads on light and dark screens.
• The original files for anything played back. The mic's copy is roomy and late; the file's own sound is clean.

Dead ends

• Highlight boxes estimated from a contact sheet landed a whole row high. Draw 50 px grid lines on a 1920×1080 frame of the cut screen track and read boxes off that.
• Text layers with html and no style.font: the runtime checks the first family in style.font; without one it fell back to Times and the render aborted.
• The mic's copy of a played-back clip: it leaked into the transcript as speech and sounded like a room recording. Skip it; use the original file.
• A mascot PNG with a coloured drop shadow lost its shadow on a background of the same colour and read as a blob. A die-cut sticker version (ink body, accent screen, a white 14 px outline) reads on the brand colour and over screenshots.
• Stills in parallel: 16 at once hung. One at a time takes about 1.4 s on a 2,100-layer spec.

Worked example

Acme Co.'s desktop app walkthrough: 30 min of footage in 5 pairs (screens 3440×1440 VFR, webcams 1920×1080 at 30 fps) cut to 5:40.

• The cut list: 74 phrases, 4 jumps, 1 FAST, 1 INLINE, 1 INSERT; 271 segments after pause squeezing, cut in 51 s with 8 threads.
• The spec: about 2,100 layers (captions are two per word). Stills about 1.4 s each, one at a time. Rendered with 8 workers.
• Cost: nothing beyond speech-to-text; every other step ran locally.