The song is the timeline: the shot list, the cut and the captions all read the same times off it, so nothing drifts over three minutes. First question: does the picture change on the music or on the words? A cut on the downbeats suits a song with a steady beat and a picture that dances to it; a cut on the sung lines suits a story told line by line. Both start from the same map of the song and end in the same frame-exact cut.
Routes
| Route | Pick it when | Cost and time | What you get | Leaves |
|---|---|---|---|---|
| Bar-grid cut | The cut should land on the music: a cartoon or montage, a steady 4/4 beat | Stems and STT under $1 for a 3:22 song; the rest local, minutes | Every cut on a downbeat or half bar; one or two bars a shot | python-map-song-bars, python-cut-to-bar-grid |
| Lyric-slot cut | Each shot carries a sung line or half line; lip-synced lines; a song whose bars drift | The same | Shots of 3.5–6.5 s starting on their line, instrumental gaps given a transition shot | python-time-lyrics, python-cut-to-bar-grid (--seconds) |
Lip-synced lines go with either route: the shot's window of the vocal stem, at
least 2 s, is the audio the line is rendered against (guide-lipsync-video; the full
mix works passably when there is no stem).
Process
- The song as a WAV and its four stems (
demucs-split-song-stems). The WAV is the length every later step cuts to: a converted WAV ran 0.19 s shorter than its M4A said. - The sung words, from the vocal stem, never the mix. Free: cut the stem into sung
phrases (
python-find-speech-pauses,runs), transcribe each (whisper-cli-transcribe-with-word-times) and stitch them back onto the song's clock (python-read-word-timings,stitch); whisper on the whole stem started lines up to 2.4 s early on a vocal with gaps. Paid:elevenlabs-speech-to-text. - The bar grid (
python-map-song-bars, with the drum stem): tempo, beats, downbeats and every sung phrase printed with its bar. Set the sections yourself from the printout and the energy curve. - The lyric sheet, timed (
python-time-lyrics): every line with a time per word in the sheet's spelling. Hear every?, copy a clean chorus onto a garbled repeat (--repeat), put fast repeated words on the beats. Caption what is sung: change the sheet where the singer did. - The shot list on the grid, a table with one row per shot: its start and end (bars
or seconds), the sung words, the look, the picture, any lettering drawn in, any
lip sync. Bar grid: one to two bars a shot in verses and choruses, half bars for a
stutter. Lyric slots: a line or half line a shot, about 3.5–6.5 s. Check each sung
line against its shot: a vocal often starts a beat into its bar, and a repeated
section can be sung tighter. Gate: the shot list (
review_approve). - Shots long enough to trim: render each a little past its slot and let the cut
trim (
h3-max-image-to-video: a 0.92 s floor, and it ran 0.05–0.7 s long;kling-v3-image-to-video: a 3 s floor). A clip short of its slot holds its last frame, which reads as a freeze. - The cut (
python-cut-to-bar-grid, bars or--seconds,--check): each shot exactly its frames, the song under it. Gate: the whole cut with the song, before any captions or fixes: clips judged one by one, silent, made no sense to the person, who asked for the cut with music as soon as the clip review opened. - Lyrics on screen: composited over the cut (
hn4a-compositor-kinetic-lyrics), or drawn into the shot itself where the look wants it (guide-pick-a-stylized-look), never both on one line.
Choosing
- Bars or lines. One 3:22 cartoon cut every shot on the bars: 83 of 83 cuts within a frame. One 2-minute multi-look video slotted its 26 shots to sung lines, several of them lip-synced. A piece can mix them: cut on bars, with a lip-synced line given its line's span.
- Build the cut so changes are a re-run. Expect the person to drop whole scenes or the song's head at the cut gate (an intro that doesn't land is common); with the shot list as data, that is an edit to the list and a re-cut, not a re-render.
- Cheap shots that need no model fill an instrumental gap or a long outro: a flicker
through the looks (
guide-pick-a-stylized-look, Choosing), or a split screen of every look's take played forward then reversed. - 3/4 time or a tempo change: the bar numbers drift; take the lyric-slot route, or check the bars by ear at every section.
Avoid
- Timestamps embedded in a song file: one generator's were 5 s off.
- A model's section labels for the song: off by one, though its read of the singer was right.
- Word times from the mix, or from whisper on a whole vocal with long gaps.
- Trimming or retiming one stem on its own: the stems, the WAV and the cut share one clock.
- Cutting a piece of the song with
-tper piece: put every cut on the frame grid (ffmpeg-frame-exact-trim).
Related
workflow-animated-music-video,workflow-multi-style-music-video: the two music video jobs built on this.hn4a-compositor-kinetic-lyrics: the lyrics as type on the cut.guide-lipsync-video: lines sung on camera.
