ffrwd/yolo26

YOLO26 detection and segmentation, hosted in wasm. One model family, one row shape, then composition: detect produces rows, segment_mask produces a matte, the utilities turn rows into mattes or drawn overlays, and everything downstream is native ffmpeg.

Requires ffrwd 0.29, whose ffrwd/wasm is 0.19.1.

License

This package is AGPL-3.0. The model weights are Ultralytics' YOLO26, released under AGPL-3.0, and the package follows them. Using the models over a network in a commercial product needs Ultralytics' commercial license; nothing here changes that.

If that license is in your way, use ffrwd/rfdetr instead: the same exports, utilities and recipes over Roboflow's RF-DETR, whose weights are Apache-2.0, so a query swaps the package name and nothing else moves. On the footage the two were compared on, its mattes agree with this package's at 0.95 IoU and it runs at about half the frame rate.

The weights are not in the archive: the manifest pins them - exact repo, revision, file and sha256 - and ffrwd install fetches and verifies them. Both are the medium end-to-end (NMS-free) ONNX exports at 640x640, about 176 MB together, run through wasi:nn on the machine's own ONNX Runtime.

Model exports

• detect(v, conf DEFAULT 0.25) returns STRUCT(class text, conf number, x number, y number, w number, h number)[]: one row per object per frame, the class as COCO label text, the box in the frame's own pixels, whole numbers each. The rows are all that leaves; the picture stays where it was.
• segment_mask(v, class DEFAULT NULL, conf DEFAULT 0.25) returns the found instances as one grayscale matte the size of v, optionally narrowed to one class name.

A reader takes the picture from the source and the rows from the detector, so both arguments name the same v: ffrwd.yolo26.draw_boxes(v, ffrwd.yolo26.detect(v)). Narrow detect's rows at run time with the gather spelling:

ARRAY(SELECT r FROM unnest(ffrwd.yolo26.detect(v)) r
      WHERE r.class = 'person' AND r.conf >= 0.5)

Both models take the picture as yuv420p, three eighths of the bytes of rgba, and convert it to RGB with ffrwd-frame in the range and matrix the stream declares; an untagged stream is read as tv-range BT.601, as ffmpeg reads it. They letterbox it into the 640 square with OpenCV's bilinear resize, which is what the models were trained on (Ultralytics' own preprocessing), rather than the Pillow resize ffrwd-frame carries for the models trained on that.

Utilities

• boxes_mask(v, boxes, grow DEFAULT 0, feather DEFAULT 0): the rows rasterized into a grayscale matte the size of v; grow pads each box in pixels, feather softens the edge. It reads x, y, w and h and nothing else, so any rows carrying a box will do. The picture is read for its times and size alone: it arrives in whatever format the query already has it in, and no pixels are carried for it.
• draw_boxes(v, boxes, thickness DEFAULT 2): the boxes drawn on the picture as green outlines. It reads whole pixels: a query handing it boxes in fractions is refused when it compiles. A frame with no box on it leaves as it arrived.

Composition

The composition layer lives in ffrwd/mask_tools - blur_where, mosaic_where, spotlight, cutout and the masked spelling they share - because it is model-agnostic: any grayscale matte beside any stream, all native ffmpeg. This package's recipes call it as ffrwd.mask_tools.blur_where(...) and so on, feeding it the mattes segment_mask and boxes_mask produce.

Recipes

blur-people, mosaic-people, spotlight, replace-background, draw, detections - run ffrwd list for each one's variables, or read the header of the recipe file.

ffrwd run ffrwd/yolo26:blur-people -v source=street.mp4 -v dest=blurred.mp4

From 1.0

1.0's detect handed the picture back with its rows beside it, and draw_boxes took that record: draw_boxes(detect(v)). 1.1's writes the rows alone, and a reader names the picture itself: draw_boxes(v, detect(v)). A query selecting detect(v).boxes selects detect(v). The mattes of segment_mask and boxes_mask are one byte a pixel in gray rather than the picture's own format.

1.0 read every yuv420p picture as full-range BT.601, whatever the stream said. A tv-range picture, which is nearly every video, reached the model at 219/255 of its contrast with its blacks lifted, and a BT.709 one through the wrong matrix as well. 1.1 reads the stream's own range and matrix, so the boxes move where that mattered:

• On a 1080p street clip, tv range and BT.709, fifty frames: 1065 rows before and 1064 after; 1039 are the same object in both (same class, overlap at least 0.5, mean 0.992), with confidences 0.014 apart on average. The 26 rows only 1.0 wrote and the 25 only 1.1 writes are the marginal ones, none surer than 0.43 and 0.55. Read through 1.0's conversion, the clip's first frame sits 8.5 counts from ffmpeg's own conversion on average, at 0.87 of its contrast.
• On a full-range BT.601 clip, where 1.0's reading was the right one, all 1324 rows are the same object in both, 904 of them to the pixel and the rest at least 0.98 overlapping, confidences 0.004 apart on average: what is left is 1.1 rounding the RGB to eight bits before the resize, as ffmpeg does, where 1.0 resized the unrounded values.

The recipes that draw or composite move with the boxes and the masks.

Building

cargo build --target wasm32-wasip2 --release
cargo test --release

Every module is a node written with ffrwd-node, which carries the world they are built against, so there is no build.rs, no wit of ours beyond wasi:nn's, and nothing to install first. common holds what the model nodes share: the colour conversion, the letterbox, and the graph run through wasi:nn. ffrwd-frame's yuv is not on a tag yet, so common finds the crate by path, in a checkout of ffrwd-frame on its yuv branch beside this one; the pin moves to a tag once that branch is merged.

The weights are fetched at ffrwd install time by whoever installs the published package, and ffrwd install in a checkout puts them beside the built modules.