Split a song into stems

Give it a song; get four WAVs, vocals, drums, bass and other, each as long as the song and on its clock. Transcribe the vocal stem, not the mix, to time the lyrics (python-time-lyrics), and pass the drum stem to python-map-song-bars to find the downbeats. To pull speech out of a noisy soundtrack (dialogue, not a song), use elevenlabs-voice-isolation instead.

Inputs

The song, any format ffmpeg reads (M4A, MP3, WAV). Convert it to 44.1 kHz stereo WAV first; that WAV is the song for every later step (stems, bars, the cut):

exec("ffmpeg", ["-v", "error", "-y", "-i", "source/song.m4a", "-ac", "2", "-ar", "44100", "analysis/song.wav"], cwd="<job>")

Its length, from the WAV (ffprobe-inspect-media), not from the source file: a converted WAV came out 0.19 s shorter than the M4A reported. Cut to the WAV.

Run it

Call describe("fal/fal-ai/demucs") before the first run in a job. The proven fields:

run("fal/fal-ai/demucs",
    {"audio_url": "<job>/analysis/song.wav", "model": "htdemucs_ft",
     "output_format": "wav", "stems": ["vocals", "drums", "bass", "other"]},
    into="<job>/analysis/stems", unit="song_stems")

Pass stems as exactly those four. Left out, the list defaults to six, adding guitar and piano, and htdemucs_ft refuses the request.
Save each stem as analysis/stems/<stem>.wav and keep them: every later step reads them, and nothing about the song changes between passes.
Analysing a 3:22 song (the stems plus speech-to-text of the vocal) cost under $1.

What you get

Four WAVs at the song's rate and length. The vocal stem carries the lead and any harmonies; other holds whatever isn't voice, drums or bass (keys, guitars, pads).

Transcribe the vocal stem next. whisper-cli is free and reads WAV at 16 kHz mono (resampling keeps the clock):

exec("ffmpeg", ["-v", "error", "-y", "-i", "analysis/stems/vocals.wav", "-ar", "16000", "-ac", "1", "analysis/vocals_16k.wav"], cwd="<job>")
exec("whisper-cli", ["-f", "analysis/vocals_16k.wav", "-l", "auto", "-ml", "1", "-sow", "-oj", "-of", "analysis/vocals_whisper"], cwd="<job>")

elevenlabs-speech-to-text on analysis/stems/vocals.wav is the paid route, and the one the worked job used. Both JSON shapes feed python-map-song-bars and python-time-lyrics.

Tips and traps

Speech-to-text of a sung vocal is close, not right. From the vocal stem the word times were good, but it misheard words, dropped short ones, and garbled the timing of one chorus out of three while the other two were clean. Time the real lyric sheet against it (python-time-lyrics); never caption the transcript as is.
Transcribing the full mix instead of the stem gives worse words and times; split first.
The drum stem's onsets pick which of every four beats is the downbeat far more reliably than the mix does (python-map-song-bars --drums).
A lip-sync shot on a song takes its window of the vocal stem as the audio, not the mix (the mix works passably when there is no stem).
Never trim or retime a stem on its own: all four, the WAV and the cut share one clock.

Check it

Every stem's duration equals analysis/song.wav's (ffprobe-inspect-media).
Listen to the vocal stem over a verse and a chorus, and check it is near silent in an instrumental stretch (ffmpeg volumedetect over that window) before transcribing.

Hypernatural