Perspective planes

Give it a surface in the picture (four corners on a frame, a camera's horizon and height, or footage to track) and the graphics to put on it; get a plane group that draws those graphics flat on that surface, in perspective, frame-exact. Use it for anything meant to lie on, recede along or tilt into the scene: a route running away down a street, type on a floor or a wall, a picture on a screen, a title on a tabletop. It fakes depth with flat surfaces; there are no 3D models, lights or occlusion by the scene except the mattes you put on top. For upright titles in the scene (type behind people, glitches, typing), use hn4a-compositor-in-scene-titles; the spec format and the commands are hn4a-compositor's.

Inputs

• The surface, one of:
  • four corners on the canvas (top-left, top-right, bottom-right, bottom-left), read off a gridded full-size frame (python-contact-sheet, --grid) for footage that holds still;
  • a level camera's horizon and lens, for a ground that runs to the horizon (a street, a floor, a field);
  • footage where the surface moves: its four corners on the first frame, then tracked.
• The graphics, laid out flat in the group's own px as for any group: fills, paths, text, images.
• For a person in front of the surface: a matted cutout of them over the same span (birefnet-matte, VP9 WebM yuva420p).

Run it

Write references/planes.md's module into the job's scripts/ as planes.py and call it from the build script beside it that writes the spec:

{"id": "poster", "type": "group", "box": {"x": 0, "y": 0, "w": 1200, "h": 675},
 "plane": [1.2549084, -0.010459171, 410, 0.12534644, 1.0401308, 232,
           0.00022540347, 0.000025963561, 1]},
{"id": "poster_bg", "type": "fill", "parent": "poster", "fill": "#E33",
 "box": {"x": 0, "y": 0, "w": 1200, "h": 675}},
{"id": "headline", "type": "text", "parent": "poster", "text": "Opening night",
 "box": {"x": 80, "y": 240, "w": 1040, "h": 200},
 "style": {"font": "700 150px Inter", "color": "#fff"}, "in": {"fx": "fade"}}

That pins the corners of a 1200 × 675 group to (410, 232), (1508, 301), (1482, 842) and (396, 918) on the canvas: plane_from_corners(1200, 675, those four). Order the layers footage, plane group and its children, then the person's cutout over the same span, so the person stands in front of the surface.

Then the commands and the live-spec gate in hn4a-compositor, judging every plane on a still (below).

The matrix

• 9 numbers, a 3×3 matrix row by row, taking a point (u, v) of the group's px to canvas px: x = (m0·u + m1·v + m2) / w and y = (m3·u + m4·v + m5) / w, with w = m6·u + m7·v + m8. Inside a camera group it maps to that group's px.
• A tracked surface takes anim.plane: one key per frame, [t, [9 numbers], "linear"], times relative to the group's start. Values interpolate number by number, so plane_keys puts neighbouring matrices on one scale.
• Footage px are not canvas px unless the video layer fills the canvas at its own size. Otherwise multiply each tracked point by the layer's placement first (scale by box size over source size, then move by the box's corner).
• The sign matters. w must be positive where the surface is in front of the camera. A matrix fitted to points you can see and scaled so its last number is 1 has that only while the group's top-left corner is in front of the camera too. For a surface that runs back past the camera (a street under your feet), plane_keys and street_plane fix the sign; by hand, multiply by the sign of w at a point that is on screen. With the wrong sign the surface shows upside down beyond its vanishing line, or render stops with "its plane never reaches the frame".
• A ground from a camera: street_plane puts the group's top edge at the far end, so type laid out upright reads upright from the camera; focal about the canvas width is a normal lens, and the horizon is where the footage's own horizon sits.

Size and sharpness

• The group's px are only its coordinates. Each frame the runtime draws the surface at the size it shows, so one group is sharp under the camera and steady in the distance whatever px it is laid out in: pick px that make the layout easy. A street laid out at 150 px per camera height, shown ten times that large at the foot of a 1080 × 1920 frame, rendered with edges 1.7 px soft; a perfect drawing has 1.6.
• Give the group the whole surface once (a street 160 camera heights long, running behind the camera and off the frame); the runtime draws the part that shows each frame. Don't trim it per frame.
• A render draws a surface that runs far away as strips by distance, each at its own resolution: one whose far end on the canvas is about three or more times as far off as its near end, such as a street to the horizon. Every layer inside the group is built once per strip, up to six. Solid fills and pictures join with nothing showing. A see-through fill, or a group that is fading, gets a stray px drawn twice where two strips meet, two or three single px a frame across a 1080 px street. A group with a video inside it, or a blur on it, is drawn as one strip, as in the preview: keep a long surface to paths, shapes, text and images.
• The preview draws one strip. Once a far end is in view the near end is soft in the preview (9 px edges where the render had 2). Judge sharpness on a still.

Tips and traps

• The matrix is the group's whole transform. x, y, scale, rotate, radius and reveal on a plane group, and the presets that use them (rise, pop, zoom, wipe), stop the render: put them on the layers inside, where they move along the surface (a draw on a path runs a route up the street; a rise lifts a word along the floor). opacity and blur work on the group. The x and y of its box are unused, and nothing outside its box is drawn.
• Give each child a box. A child's box defaults to the canvas, not the group, so a fill with no box covers the canvas's size in group px, not the group's.
• Tracking needs texture. track_corners follows features inside the corners and stops when fewer than 12 survive: a blank wall or a glossy screen gives it nothing. Track a textured area on the same surface, or key the corners by hand on a few frames. A synthetic card moving and turning over 30 frames tracked within 0.08 px.
• Judge a track on stills or by stepping frames. While the preview plays, a video runs up to 0.12 s off the clock before it is re-seeked, so a tracked surface can seem to slide on its plate. Paused frames and renders are exact.
• The old ways. A CSS perspective or skew on a text layer's inner html, or graphics drawn in Pillow per frame into an overlay video, both did this job before planes. Use a plane instead: a route drawn as Pillow polygons came out 1.1 px too wide and its width jumped 0.49 px between frames at the far end; the same route in a plane was 0.17 px off and jumped 0.24 px. CSS on the html still suits a small tilt that isn't tied to anything in the picture (a card swinging in on its hinge).
• An older app may not draw this. One from before planes ignores plane and draws the group flat at its box, with no warning; the first with planes drew the group at its own px, soft where it shows larger and missing its far end when the group is hundreds of millions of px. If a still shows either, the app needs updating; until then lay the group out at about the px it covers where it shows largest.

Check it

• apply_plane(m, u, v) on the group's four corners lands on the points you measured.
• A still at the first, middle and last frame of a tracked plane, its corners on the surface's corners; then the render stepped through where the motion is fastest.
• A still where the surface is closest to the camera, for sharpness, and one where a person crosses it, for the matte.