Kinetic lyrics

Give it the cut on the song, the timed lyric sheet, the song's beats and the shot list with each shot's look; get a compositor spec with the lyrics as type over the cut, every word landing on its syllable and riding the beat. The spec format, the commands and the live-spec gate are hn4a-compositor's; how the cut and the lyric times are made is guide-time-a-video-to-a-song. Spoken captions are guide-caption-a-video; lyrics drawn into the picture itself are a still's job (guide-pick-a-stylized-look).

Inputs

• The cut with the song under it (python-cut-to-bar-grid), as the bottom video layer, and the song's WAV as the spec's audio.
• lines.json from python-time-lyrics: each line with a start and end per word, in the sheet's spelling. Caption what is sung; its ? words heard first.
• beats.json from python-map-song-bars: beats for the pulses, downbeats and drum_onsets for flashes and shakes.
• The shot list in song seconds, [{"start", "end", "look", "baked"}]: which look is on screen, and whether its lyric is already drawn into the picture.
• A look per picture look: its font as a local file (in the spec's fonts, and measured from the same file), size, colour and outline. Type that pairs with each generated look: guide-pick-a-stylized-look, references/looks.md, and style-3d-animation.

Run it

Write references/kinetic_lyrics.md's module into the job's scripts/ as kinetic_lyrics.py, beside titles_fx.py, and a build script beside them (the same reference has a complete one):

exec python scripts/build_lyrics.py --out lyrics/lyrics_v1.json
exec hn4a-compositor still lyrics/lyrics_v1.json lyrics/s_11.6.png --at 11.6
exec hn4a-compositor render lyrics/lyrics_v1.json

(cwd the job folder.) lyric_layers(lines, beats, shots, looks, box) is the whole caption pass; punch=, tilts= and boxes= pick the lines that do more.

The design

These are what worked on one 3:22 two-look cartoon video and a 2-minute four-look one; start from them and judge each on a still.

• Type follows the picture's look. The look on screen at a line's first word picks its type: in the ink-comic look, heavy capitals with an ink outline and magenta and cyan offset shadows; in a collage look, a spray marker in pink and lime; the two alternating word by word where the looks clash; gold where they merge. A line sung up to 0.25 s before a cut belongs to the shot after it.
• One layer per word on one shared layout. Every word's layer holds the whole line with the other words hidden, so the line never reflows as words land. Each word lands 0.04 s before its syllable with an overshoot and pulses on every beat after.
• Hits. Soft-light white flashes on the chorus downbeats; camera shake on the big hits (the cut's own layer, scaled 1.05 so the edges never show); sound-word bursts on the intro's downbeats (a star clipPath fill with the word over it).
• Punch words. The hook word of a line slammed in at 300 px or more at the centre for about a second, overshooting on scale and rotation. Score-style pop-ups ("+500 PTS") in the scene's comic type sold a contest beat.
• Lines that act out their words: a word about distance shrinks and blurs (scale, blur keys), a word about leaving drifts off (x), a word about rising rises (y), a curve bends (a tilt), the last line rides the picture's last big move; a path heart pulsed on the kick.
• Lines that lie in the scene: a line on a floor, a road or a wall, receding into the set, goes in a plane group pinned to that surface (on_plane, and hn4a-compositor-perspective-planes for the corners or a tracked surface). The word pops and pulses still work: they are on the layers inside the group.
• Skip every line already drawn into its shot (baked on the shot).
• Lines never overlap: each ends 0.25 s before the next line's first word, or the next entrance lands on the last one. A line holds 0.6 s after its last word otherwise.
• Readable on bright frames: a bottom gradient to 45% black under the lyrics.
• An end title as tiles, one row per word: stacked divs, since a <br> does nothing inside a flex row.

Tips and traps

• Measure, never estimate. With guessed widths the pulsing words slid into each other and read as one. Each word's centre comes from the font file (Pillow's getlength), so embed that same file in the spec. On the test line every word's measured centre was within 3 px of where the browser drew it.
• A long line shrinks to fit its box (to 94% of the width). A sheet line over about six words reads better split at its comma into two lines of the sheet.
• Tilted lines lift their ends: 4° on a 1740 px line lifts one end about 60 px. The module warns when a tilt would leave the box; give the line a taller box.
• Tilt goes on an inner div, which the module writes: the runtime owns each layer's own transform, and the pulses scale it.
• Place bursts against a still: one placed by eye covered a character's head.
• A plane group stops the render if x, y or scale is keyed on the group itself (hn4a-compositor-perspective-planes); keep the motion on the words.

Check it

• A still at every special moment, punch word, burst and look change, on one sheet (python-contact-sheet, images), looked at: no word off the frame, no burst on a face, the type in the look of the shot under it.
• The live spec, then the render (hn4a-compositor, Live spec, then render); watch it with sound at a few lines: each word lands on its syllable.
• Frame count = round(duration × fps); a spot fixed with --into.