Cut an audio window for a lip-sync model

Give it a track and a window; get a mono WAV of just that window, padded with silence where a model needs a minimum length. Three cuts cover the jobs: a span of a track (a line, a chunk), a shot's dialogue window that goes silent where the next speaker starts, and a short line padded to a model's minimum.

Cut from an isolated voice (elevenlabs-voice-isolation), not a mix with music and effects: the models lip-sync whatever they hear.

Inputs

• The track: an isolated voice WAV or MP3, or a video (its first audio stream).
• The window: start and end in seconds on the track. Take span times from the word timings (python-read-word-timings): cut at the midpoint of a pause, never inside a word.

Run it

A span [start, end), mono 48 kHz PCM. -ss and -t before -i seek fast and are sample-accurate on audio:

exec ffmpeg -hide_banner -loglevel error -y -ss 41.2300 -t 9.8400 -i voice_isolated.wav \
  -ac 1 -ar 48000 -c:a pcm_s16le chunk03.wav

For chunks of a long track that will be joined again, keep the track's own sample rate (exec ffprobe -v error -select_streams a:0 -show_entries stream=sample_rate -of csv=p=0 track.wav) and pass it as -ar.

A short line, padded to a length the model accepts. Pad, never take a longer window: the extra words would be lip-synced too.

exec ffmpeg -hide_banner -loglevel error -y -ss 12.4000 -t 1.6000 -i voice_isolated.wav \
  -af "apad=whole_dur=5" -ac 1 -ar 48000 -c:a pcm_s16le shot05_line.wav

Use whole_dur=5 for H3 Max lip sync (5 s worked; under 2 s is rejected), and whole_dur=3 for a line under 2 s going to a reference-audio input (fal rejects clips under about 2 s: H3 Max 2 s, Seedance 1.8 s).

A shot's dialogue window: duration seconds from the shot's start, fully silent from silent_from (absolute time on the track) with a 30 ms fade before it, so the next speaker's words aren't lip-synced onto this mouth. The fade starts at silent_from − 0.03 − start, relative to the cut:

# start 1.708, duration 1.875 (the take's 45 frames at 24 fps), silent from 3.210
exec ffmpeg -hide_banner -loglevel error -y -ss 1.7080 -t 1.8750 -i voice_isolated.wav \
  -af "afade=t=out:st=1.4720:d=0.03,apad" -t 1.8750 -ac 1 -ar 48000 shot03_line.wav

apad with the second -t holds the length at duration after the fade.

Picking silent_from by ear

Transcript timestamps and loudness minima were both about 0.1 s off where one speaker's last word ran into the next speaker's first. Cut a ladder of windows a few hundredths apart (3.15, 3.18, 3.21, 3.24, 3.27), put them on a review page as audio, and let the person pick the one that ends cleanly on the last word with nothing of the next. Record the pick in AGENTS.md.

What you get

A mono PCM WAV of the window: the right input for target_audio_url, audio_url or a reference-audio field. Check its length with exec ffprobe -show_entries format=duration.

Tips and traps

• Lengths per model: H3 Max lip sync takes 5 to 15 s (cut a long track into chunks of 5 to 14 s at sentence pauses); H3 Max reference audio 2 to 15 s; a mouth-only re-sync wants the take's length exactly.
• Cut chunks of one track back to back (each chunk's end is the next one's start), so joining them rebuilds the track without a gap or an overlap.
• After a re-voice, cut the new audio; never splice the old and new by hand.