Compose motion graphics

Give it a JSON layer spec (a canvas, fonts, an audio list and bottom-first layers of fills, images, video, text and SVG paths, each with a box, CSS, entrance and exit presets and keyframes); get back a frame-exact H.264 MP4 at the spec's size and rate with its audio mixed in. The same runtime plays the spec live in a browser page, so what you preview is what renders. Use it for overlay layouts: slides beside a talking head, lists that build on the words, logos, lower thirds. For titles that sit inside the footage (type behind people, glitches, typing, reflections), use hn4a-compositor-in-scene-titles, which writes the same spec.

Inputs

The voice track (the dialogue master) and its word timings: speech-to-text on that exact file (elevenlabs-speech-to-text), read with python-read-word-timings. Before there is a voice, word times estimated from the approved script stand in for it, for the first look only (references/first-scene-preview.md).
The footage (a talking head, a cut) and, for a cutout, a matted copy with a real alpha channel (birefnet-matte: VP9 WebM yuva420p; ProRes 4444 also works). Point the spec at the real files; long-GOP sources are proxied for you.
Fonts: a Google Fonts CSS URL with display=block, or local font files.
A still of the layout to match, if there is one, to lay boxes out against.

The spec

One file per slide layout (or run of slides) in the job folder, <job>/slides/<name>.json, asset paths relative to it. The shape:

{
  "size": [1920, 1080], "fps": 30, "duration": 55.8, "background": "#000",
  "fonts": [{"css": "https://fonts.googleapis.com/css2?family=Inter:wght@500;700&display=block"}],
  "audio": [{"src": "../audio/track.wav", "start": 0, "trim": 0, "volume": 1}],
  "layers": [
    {"id": "bg", "type": "fill", "fill": "#1B2E24"},
    {"id": "head", "type": "video", "src": "../footage/presenter_take.mp4", "trim": 0,
     "box": {"x": 1000, "y": 0, "w": 900, "h": 1080}},
    {"id": "heading", "type": "text",
     "runs": ["Three signs to ", {"text": "listen for.", "style": {"color": "#B8E62E"}}],
     "box": {"x": 120, "y": 170, "w": 1400, "h": 120},
     "style": {"font": "700 96px/1.05 Inter", "color": "#fff"},
     "start": 20.8, "end": 55.5, "in": {"fx": "fade", "dur": 0.6}, "out": {"fx": "fade", "dur": 0.4}}
  ]
}

Layers draw bottom first. start/end are comp seconds; box is canvas px; style is any CSS in camelCase.
Presets for in/out: fade, rise, pop, zoom, wipe, draw, cut.
anim keyframes per property, times relative to the layer's start: {"scale": [[0, 0.8], [0.3, 1, "out"]]}. Animatable: opacity, x, y, scale, rotate, blur, reveal, draw, radius, box, crop.
Video: trim is the source time at the layer's start; crop picks a source region (a round bubble is a square face crop with radius: "50%"). Video layers are muted: the sound goes in audio.

Every field, the per-type options and the eases: references/spec-format.md. A complete slide (title, presenter shrinking to a bubble, three points building on the words): references/worked-spec.md.

Show the first scene early

Right after the script is approved, before or while the voices are auditioned, build the opening scene alone in two or three looks, cued from word times estimated from the script, and put them on a review page as live compose candidates. The person sees motion within minutes and picks the look the whole video is built in; the full build waits for the voice. The estimate script, the looks and the page: references/first-scene-preview.md.

Cue from word times

Slides change on the words. Take each entrance from the word timings, a little before the word (the client's own slides landed about 0.1 s early): "First," spoken at 23.0 s puts its point at 22.9. When the voice track is re-taken, transcribe the new file and re-cue from it; never nudge numbers by hand.

Run it

Run hn4a-compositor through exec with these subcommands and flags.

exec hn4a-compositor page   slides/slide_b.json                    # slide_b.html: play, scrub, frame-step
exec hn4a-compositor still  slides/slide_b.json slides/b_22.9.png --at 22.9
exec hn4a-compositor render slides/slide_b.json [out.mp4]          # --workers 8 (default), --format jpeg|png, --no-audio, --no-proxy
exec hn4a-compositor render slides/slide_b.json --frames 690:990 --into slides/slide_b.mp4
exec hn4a-compositor proxy  mattes/presenter_cutout.webm           # short-GOP copy by hand; keeps alpha

Run with cwd set to the job folder. page writes the preview next to the spec; render, still and page rewrite it from the JSON, so edit the JSON, never the HTML.

Write the spec, then page and open it: Space plays with audio, , . step a frame, ?t=20.5 opens at a time.
A still per scene (still --at T), tiled on one sheet and looked at once. Check the longest copy: text boxes don't shrink text, so a long heading wraps or overflows (set "whiteSpace": "nowrap" to force one line). Fix what's broken and show it; the person's notes will move more than another pass of yours. Your own look catches mistakes; it doesn't approve the design.
Show it live before you render. review_open with the spec as a {"compose": "slides/slide_b.json"} candidate (it plays in its card, free, in seconds) and the key-moment stills beside it as the row's refs; get a yes on the caption and title design (type, colour, placement, entrances). Stills alone are your own check, not the person's. Only then render.
Render the picked spec.
Revisions: re-render only the span that changed. --frames A:B --into FULL.mp4 (A:B the frames the edit touches, the layer's start..end × fps) widens the span to FULL's keyframes (at most a second each side), renders it, splices it in by stream copy keeping FULL's audio, and checks every frame outside the span is unchanged. A 10 s span of a 163 s comp: about 15 s render plus 0.4 s splice, against minutes for a whole render. A change to duration, fps or audio needs a whole render (--into refuses a frame-count mismatch).

Review pages play a spec live: the compositor's JavaScript runtime is available in the page, so the person can watch a layout before it is rendered.

What you get

An H.264 MP4, CRF 16, BT.709-tagged, with a keyframe on every whole second (which is what lets --into splice by stream copy), and the spec's audio mixed by ffmpeg, never captured, so it can't drift. Speed: about 13 fps with one browser, 35 fps with 8, when every video layer seeks fast.

Tips and traps

Long-GOP video is proxied automatically. A video layer whose keyframes are more than 12 frames apart renders through a cached 12-frame-GOP proxy in .proxies/ next to the spec (re-made when the source changes). An H3 Max talking head with a keyframe every 250 frames went from 9 to 34 fps, SSIM 0.998. --no-proxy renders from the sources.
Fonts fail loudly. render and still abort on any warning, such as a font family with no loaded face (a typo would otherwise ship a fallback). The first family named in each text layer's font must be one loaded in fonts.
Sound lives in audio, not in video layers.
A cutout needs a real matte (birefnet-matte); a colour key leaves a fringe on skin and hair. Or keep the footage in a window or bubble where its own background shows.
Comparing a render to a source by pixels: the render is BT.709; decode both with their own matrices, or greens and skin look shifted.
The runtime seeks video half a frame in (frame k at (k + 0.5)/fps), which is why rendered frames match their source frames exactly; seekBias overrides it per layer and is rarely needed.

Check it

Frame count = round(duration × fps): exec ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of csv=p=0 <out.mp4>
A contact sheet of the render, looked at, and the first seconds listened to.
After an --into revision, stills either side of the span.

Hypernatural

hn4a-compositor-motion-graphics