Track face and hands

Give it the cut (the exact file the compositor spec plays); get, frame for frame, where the person, their face and their hands are, and the cutouts that keep them in front of graphics. Free and local. For a cutout whose edges must hold up (hair against a busy room, furniture that must stay or go), birefnet-matte is the paid sibling; the skin-threshold tracker in hn4a-compositor-hand-interactive-titles is the no-model fallback for one hand.

Inputs

• The cut, after the picture is locked. Every key and cutout reads its frames, so a change to the cut means running this again.
• The three MediaPipe model files in refs/models/ (references/track.md lists them with their links).

Run it

Write references/track.md's script into the job's scripts/ as track.py, as is:

exec python scripts/track.py --video work/rough_cut.mp4 --models refs/models --out work/track

• Person matte: selfie_multiclass's background confidence, inverted, smoothed over time (0.65 this frame, 0.35 the last), a smoothstep from 0.3 to 0.7 and a 1.2 px blur.
• Hands: hand_landmarker, two hands, palm = the mean of the wrist and the four knuckles. Hands-only matte: the landmarks' hull, dilated 31 px, inside the person matte.
• Face: face_landmarker, its box, nose, chin and forehead points.

About 4–5 frames a second at 1280×720 on a laptop (the three models are most of it): a minute of 30 fps footage takes about 7 minutes. Start it with exec(wait: false).

What you get

In work/track/: track.json, matte/ and hands/ PNGs per frame, person_cutout.webm and hands_cutout.webm (VP9 yuva420p, a keyframe every 12 frames), freeze.png and freeze_cutout.png for an end hold of about 1.4 s. In a spec, a cutout layer sits over the titles it hides, same start as the cut and trim 0; the hands cutout goes over a title that sits on the face but under the hands.

Tips and traps

• The face is found only when it is big enough. On a webcam close-up it was found in 640 of 640 frames; on a full-body shot of the same person (face about 70 px tall at 720p) in 32 of 180. Use the hands and the matte there, or crop in first.
• The matte keeps what touches the person: a chair back and an air purifier beside them came through. Mask such things with a box on the matte PNGs, or use birefnet-matte where furniture must be decided right.
• side is MediaPipe's label, which assumes a mirrored selfie image. Tell hands apart by palm x when it matters, after one look at a frame.
• The hands hull is the hand, not what it holds: a microphone in the hand isn't in the hands-only cutout.
• A hand that blurs in a fast push drops out of tracking for a few frames; the move helpers carry it on at its last speed (hn4a-compositor-hand-interactive-titles).
• Decode the cutouts with -c:v libvpx-vp9 before -i when reading them in ffmpeg, or the alpha is dropped.

Check it

Composite a few frames of the cutout over magenta with the face box, nose and palms drawn on, tiled on a sheet (python-contact-sheet), and look: edges on the hair and fingers, nothing of the room left in, the points on the right places.