Log in
TechniqueBy HypernaturalVersion 2, updated

Split a song into stems

hn/demucs-split-song-stems

Split a song into four stems (vocals, drums, bass, other) with demucs htdemucs_ft on fal: give it the song, get four WAVs on the song's own clock. The vocal stem is what speech-to-text should hear to time the lyrics, and the drum stem sharpens downbeat finding; use it first when cutting a music video or lyric captions to a song.

SKILL.md

Give it a song; get four WAVs, vocals, drums, bass and other, each as long as the song and on its clock. Transcribe the vocal stem, not the mix, to time the lyrics (python-time-lyrics), and pass the drum stem to python-map-song-bars to find the downbeats. To pull speech out of a noisy soundtrack (dialogue, not a song), use elevenlabs-voice-isolation instead.

Inputs

  • The song, any format ffmpeg reads (M4A, MP3, WAV). Convert it to 44.1 kHz stereo WAV first; that WAV is the song for every later step (stems, bars, the cut):
exec("ffmpeg", ["-v", "error", "-y", "-i", "source/song.m4a", "-ac", "2", "-ar", "44100", "analysis/song.wav"], cwd="<job>")
  • Its length, from the WAV (ffprobe-inspect-media), not from the source file: a converted WAV came out 0.19 s shorter than the M4A reported. Cut to the WAV.

Run it

Call describe("fal/fal-ai/demucs") before the first run in a job. The proven fields:

run("fal/fal-ai/demucs",
    {"audio_url": "<job>/analysis/song.wav", "model": "htdemucs_ft",
     "output_format": "wav", "stems": ["vocals", "drums", "bass", "other"]},
    into="<job>/analysis/stems", unit="song_stems")
  • Pass stems as exactly those four. Left out, the list defaults to six, adding guitar and piano, and htdemucs_ft refuses the request.
  • Save each stem as analysis/stems/<stem>.wav and keep them: every later step reads them, and nothing about the song changes between passes.
  • Analysing a 3:22 song (the stems plus speech-to-text of the vocal) cost under $1.

What you get

Four WAVs at the song's rate and length. The vocal stem carries the lead and any harmonies; other holds whatever isn't voice, drums or bass (keys, guitars, pads).

Transcribe the vocal stem next. whisper-cli is free and reads WAV at 16 kHz mono (resampling keeps the clock). A vocal has long gaps between its lines, and whisper on the whole stem starts lines up to 2.4 s early, so cut it into sung phrases first and transcribe each (python-find-speech-pauses, runs), then stitch them back onto the song's clock (python-read-word-timings, stitch):

exec("ffmpeg", ["-v", "error", "-y", "-i", "analysis/stems/vocals.wav", "-ar", "16000", "-ac", "1", "analysis/vocals_16k.wav"], cwd="<job>")
exec("python", ["scripts/speech_map.py", "runs", "analysis/vocals_16k.wav", "--out-dir", "analysis/runs"], cwd="<job>")

Save the printed list as analysis/runs.json, run whisper-cli on each analysis/runs/run_NNN.wav with -of analysis/runs/run_NNN (whisper-cli-transcribe-with-word-times), then:

exec("python", ["scripts/word_timings.py", "stitch", "--runs", "analysis/runs.json", "--out", "analysis/vocals_words.json"], cwd="<job>")

elevenlabs-speech-to-text on analysis/stems/vocals.wav is the paid route, and the one the worked job used. Either JSON feeds python-map-song-bars and python-time-lyrics.

Tips and traps

  • Speech-to-text of a sung vocal is close, not right. From the vocal stem the word times were good, but it misheard words, dropped short ones, and garbled the timing of one chorus out of three while the other two were clean. Time the real lyric sheet against it (python-time-lyrics); never caption the transcript as is.
  • Transcribing the full mix instead of the stem gives worse words and times; split first.
  • The drum stem's onsets pick which of every four beats is the downbeat far more reliably than the mix does (python-map-song-bars --drums).
  • A lip-sync shot on a song takes its window of the vocal stem as the audio, not the mix (the mix works passably when there is no stem).
  • Never trim or retime a stem on its own: all four, the WAV and the cut share one clock.

Check it

  • Every stem's duration equals analysis/song.wav's (ffprobe-inspect-media).
  • Listen to the vocal stem over a verse and a chorus, and check it is near silent in an instrumental stretch (ffmpeg volumedetect over that window) before transcribing.

FAQs

Split a song into four stems (vocals, drums, bass, other) with demucs htdemucs_ft on fal: give it the song, get four WAVs on the song's own clock. The vocal stem is what speech-to-text should hear to time the lyrics, and the drum stem sharpens downbeat finding; use it first when cutting a music video or lyric captions to a song.

Install Hypernatural Desktop, a free Mac app, and connect it to Claude Code, Codex or Claude Desktop. Then describe the video you want and your agent finds the skill itself, or paste this page's link (https://hypernatural.ai/skill/hn/demucs-split-song-stems) and it installs hn/demucs-split-song-stems and follows it.

It runs fal/fal-ai/demucs. Hypernatural Desktop has every one of them built in, so there are no API keys to set up, though you can bring your own.

The skill is free. You only pay for the AI generations it runs, and your agent shows you the price of every batch before it spends anything. Editing, captions and titles run on your Mac at no cost.

Yes. Ask your agent to adjust it for your job, or to save its own version. You can publish your version under your own handle so anyone can install it.

Go make something cool.

Install the app, open your agent, describe the video. That's the whole onboarding.

Sticker HypernaturalSticker BannerSticker TVBoy