ElevenLabs voice isolation

Give it a video or audio file with speech under music or effects and you get the voice alone. Use it when an original line has to drive a model (H3 Max lip sync, a reference-audio video model, a mouth re-sync) and the source mix would leak music into the mouth. It only separates; for a denoise of an already clean voice it's the wrong tool, and it adds no top end to a band-limited voice.

Inputs

The source clip, whole. Isolate once per source and cut lines from the result, so every line comes from the same pass.

Run it

Call describe("fal/fal-ai/elevenlabs/audio-isolation") before the first run in a job. Proven with the source video as the input:

run("fal/fal-ai/elevenlabs/audio-isolation",
    {"video_url": "<job>/inputs/acme_office_spot.mp4"},
    into="<job>/audio", unit="acme_office_spot_voice_isolated")

The output is the isolated voice as an audio file (audio). If you only have audio, check describe for the audio input field.

Cut the line by ear: the ladder

The isolated voice still holds every speaker. To give one speaker's line to a model, cut a window that starts at the shot's first frame and goes silent where the next speaker starts, with a 30 ms fade before the silence so the next words aren't lip-synced. One trim, all times in seconds on the isolated voice (<fade_at> is <silent_from> - 0.03 - <start>); afade out leaves everything after it silent and apad keeps the length:

exec ffmpeg -hide_banner -loglevel error -y -ss <start> -t <duration> -i <voice> \
  -af "afade=t=out:st=<fade_at>:d=0.03,apad" -t <duration> -ac 1 -ar 48000 <trim>.wav

Where exactly to go silent is the hard part:

• Transcript timestamps and loudness minima were each about 0.1 s off at a hand-off where one speaker's last word ran into the next speaker's first.
• So make a ladder: five to seven trims of the same window with the silence point a few hundredths of a second apart (for example the transcript's estimate -0.06 s to +0.06 s in 0.02 s steps), named by their silence point (shot03_line_trims/3.19.wav, 3.21.wav, …).
• Put the ladder on a review page and let the person pick by ear: the earliest trim with no clipped last syllable and none of the next speaker's breath or onset.

What you get

One isolated voice file per source, and per line a trimmed mono 48 kHz WAV with its silence point in the file name. Record the picked point in AGENTS.md.

Tips and traps

• Pad a short line with silence to the model's minimum length; never take a longer window to reach it, or the extra words get lip-synced too.

Check it

Play the picked trim alone: the line complete, nothing of the next speaker, no music bed. Then transcribe it (elevenlabs-speech-to-text): only that line's words.