Spatial Assembly AR

A web app for spatial recognition and AR overlay of complex mechanical assemblies, built for iPad and iPhone. It guides an operator through an assembly step by step, verifies each part as it goes in — position, orientation, seating, sequence, interference, and left/right swaps — and lets a remote expert join the same spatial session rather than watching a shaky screen-share.

Why it beats a screen-sharing tool (TeamViewer et al.): those stream pixels of one person's screen. Here, both people are in the same 3D scene. Poses, placements and annotations sync as a few hundred bytes per second, each device renders the model against its own viewpoint, and the expert can look around the workpiece independently. That survives a factory-floor cellular link where video collapses.

Compared with TeamViewer Assist AR, honestly

TeamViewer Assist AR is a native app built on ARKit and ARCore, with 3D annotations placed on a live video feed, OCR and session summaries, and support for industrial smart glasses. Two of those are structural advantages we cannot match from a browser, and the rest is a different product to a different problem.

	TeamViewer Assist AR	This app
Tracking	ARKit / ARCore visual-inertial odometry, native	WebXR (which is ARCore) on Android, Quest and visionOS Safari; on iPhone/iPad Safari there is no WebXR, so: gravity, tap-to-place, and frame-by-frame visual tracking of the recognised object
What it knows about the object	Nothing — a human expert draws on the video	The CAD assembly: parts, mates, tolerances, build sequence, keep-outs
What the overlay does	Shows where the expert pointed	Verifies fit as parts go in: position, tilt, roll, seating, sequence, interference, handed swaps
Anchoring	Feature map from ARKit/ARCore	Detected plane; or the object itself, recognised markerlessly with metric scale
Needs a remote expert	Yes, that is the product	No — it works solo; co-presence is an option on top
Install	App Store / Play, MDM rollout	A URL
Glasses	RealWear, Vuzix, Epson	Android-based glasses run the same URL; untested by us

The honest gap: on iPhone and iPad there is no way to reach ARKit from Safari. What we have instead is object tracking, not world tracking: while the recognised object is in shot the overlay follows it frame by frame (see above), which covers the case that matters for guided assembly. Look away from it and there is nothing holding the anchor — ARKit would keep it from the room's feature map. Closing that means a thin native wrapper around a WKWebView with an ARKit plugin. On Android and Quest, WebXR gives us the same ARCore tracking they use.

Where the trade goes the other way: an ARKit anchor knows where a surface is but not what is standing on it. Because the model, the mates and the tolerances are in the app, this can tell an operator that the left bearing cap is fitted on the right, or that step 7 was skipped — which is not something a drawing on a video feed can do, no matter how good the tracking underneath it is.

What it does

• Guided assembly — an ordered, dependency-aware step list with live status, per-step instructions, tools, cautions, and a time estimate.
• Snapping & fit verification — connectors are full 6-DoF frames, so every mate is checked for lateral/axial position, tilt, roll (with rotational symmetry), and seating depth against a tolerance band (ok / warn / fail).
• Error detection — interference (oriented-bounding-box SAT), out-of-sequence installs, keep-out intrusions, missing parts on a signed-off step, and the classic handed-part swap (left cap fitted in the right position).
• Animation — fly-in build animation in dependency order, per-step "show me", and an exploded view with a live spread slider.
• Background geometry — the bench and fixtures render as depth-only occluders so virtual parts correctly disappear behind real ones; keep-out volumes are drawn and enforced.
• Anchoring that finds the world, not the screen — WebXR hit-test on a real plane where the device has it; markerless object recognition of the assembly's own facade (see below); QR marker re-registration; and, on iOS, aim-and-tap on the floor plane implied by gravity. It never drops the model at a guessed standoff and calls that AR.
• Registration — 3-point (Horn's absolute orientation), 2-point + gravity, plane-drop, and automatic marker re-registration so the overlay re-locks onto the workpiece after the operator walks away and back.
• Spatial co-presence — shared session with participants, viewpoints and world-anchored annotations over a transport-agnostic protocol.

Architecture

 camera frame
   └─ OpenCV.js      sharpness gate → illumination normalize → crop / perspective correct
        └─ ONNX Runtime Web   detection · segmentation · classification  (WebGPU→WASM)
             └─ engine (pure TS)   snapping · collision · diagnostics · sequencing · registration
                  └─ Babylon.js    rendering + native WebXR (immersive-ar)
                       └─ React     HUD, panels, mode bar

• src/engine/ — the domain core, renderer-agnostic and fully unit-tested: math, snapping, collision, diagnostics, sequencer, animation, alignment (registration), plus tracking/ (capability detection, iOS camera passthrough, marker homography pose) and collab/ (the co-presence protocol and session). three is a dependency here — used headless purely as a well-tested linear-algebra library; it does no rendering.
• src/vision/ — the recognition pipeline: opencv (OpenCV.js loader + preprocessing, with pure-JS fallbacks), onnx (ONNX Runtime Web wrapper with YOLO-style detection decode, NMS, softmax classify, argmax segment), and pipeline (orchestration, re-entrancy-guarded, expectation fusion).
• src/render/babylon/ — Babylon.js SceneManager, mesh factory, and the native WebXR entry.
• src/components/ — the React shell and HUD.
• src/state/ — a Zustand store that derives diagnostics and the sequence view on every change.
• src/mendix/ — a Mendix pluggable widget (SpatialArViewer) plus its .xml manifest. Structurally typed to compile without the Mendix SDK and not part of the app's bundle; packaged on its own as an .mpk with every release (a preview: load-tested in a host page, not yet in Studio Pro — see docs/RELEASING.md).

Everything degrades gracefully

The heavy runtimes (Babylon aside) are loaded lazily and are optional. With neither OpenCV nor an ONNX model present, the app is still a complete guided-AR tool driven by geometry alone; each capability enriches the scene as it loads. This keeps first paint fast on a cold connection while ~10 MB of WASM streams in behind it. The status bar shows honest capability badges (HTTPS, WebGL2, WebXR, Camera, OpenCV, ONNX).

Performance & recognition

Snapping / diagnostics performance. The diagnostics engine re-runs on every placement, so its hot paths are optimised for large assemblies:

• Interference broadphase is backed by a uniform-grid spatial hash (src/engine/spatialHash.ts), replacing the O(n²) pairwise sweep with a near-linear one — proven equivalent to brute force in tests. This is what keeps the 108-part rack responsive as parts are placed.
• Connector local frames are memoised by connector identity, removing the repeated matrix work they incurred (twice per mate, every diagnostics run).

Object recognition accuracy & stability. The vision pipeline does more than run a model per frame:

• Aspect-correct preprocessing (src/vision/preprocess.ts): bilinear resampling and letterboxing instead of a nearest-neighbour squash, with detection boxes un-mapped back to the original frame. Squashing aspect ratio is one of the biggest silent accuracy killers for a detector; this fixes it.
• Temporal fusion (src/vision/tracking.ts): a SORT-style DetectionTracker associates detections across frames (IoU + EMA, hit/miss lifecycle) so boxes stop flickering and only confirmed parts drive the UI; a ClassificationVoter takes a rolling majority vote — important for the handed left/right pair, where a single frame flips. Temporal history resets on step change.
• Soft-NMS recovers a real overlapping part that hard NMS would delete — common in a dense assembly.
• Colour-coded discrepancy — on the affected part (src/vision/verdict.ts, SceneManager, RecognitionOverlay.tsx): a recognised object is highlighted on its own geometry in the registered overlay — green if it is the part the current step expects, red if it is a different known part (a wrong pick caught before it is even seated) — with a compact tag pinned to the part (via screen-space projection) and a verdict banner in words. The colour lands on the part area, not as a box over unrelated regions of the frame; unrecognised objects are left to the banner.

Device support

Device	Mode	How
Android Chrome / Quest / Vision Pro	WebXR immersive AR	Babylon native, hit-test anchor drop, real occlusion
iPhone / iPad (Safari)	Camera passthrough	Rear camera + motion sensors; aim-and-tap floor placement, object recognition, auto re-lock via QR marker
Any browser	3D preview	Turntable view; everything except the camera works
iOS with a single model	AR Quick Look	System AR viewer fallback

iOS Safari has no WebXR, so the camera-passthrough path is a first-class citizen, not an afterthought — including 6-DoF marker pose recovery (planar homography) so the overlay re-registers automatically.

Flexible / embeddable UI

The UI reshapes from a full workstation down to a bare embeddable canvas — same build, no separate embed bundle. A host picks the shape via a preset, then individual panel toggles, then look controls (density, brand accent). Config arrives three ways:

• React prop — <App config={{ preset: 'minimal', accent: '#ff7a00' }} /> (this is how the Mendix widget passes uiPreset / embedded / accent).
• URL params — for an <iframe> embed, no code: …/?ui=viewer&embedded=1&accent=%23ff7a00, or granular …/?panels=steps,diagnostics and …/?diagnostics=0&steps=1.
• Defaults — full workstation.

Presets: full (everything) · compact (denser, no drawers/picker) · minimal (viewport + active-step guide + recognition) · viewer (bare canvas with the on-part recognition tint only). See src/ui/config.ts.

Host integration and future Teamcenter assemblies

The iframe is the supported isolation boundary for multiple viewers on a host page: each iframe owns its store, renderer, and camera lifecycle. The React/Mendix shim still uses a singleton store and is not a multi-instance widget package.

An iframe host can opt into the version 1 messaging contract with an exact parentOrigin (scheme, hostname, and optional port; no path or trailing slash):

<iframe
  id="assembly-viewer"
  src="https://viewer.example/?ui=compact&amp;embedded=1&amp;parentOrigin=https%3A%2F%2Fplm.example.com"
  allow="camera; xr-spatial-tracking; accelerometer; gyroscope; magnetometer; fullscreen"
  title="Assembly viewer">
</iframe>

Both host and viewer must use HTTPS for AR. The host's Permissions Policy must also permit those features for the viewer origin; an iframe allow attribute cannot override a denial by an ancestor. AR entry remains a direct user gesture, not a message command.

Listen for ready, then send a command. Check both origin and source on responses, just as the viewer does on incoming requests:

const frame = document.getElementById('assembly-viewer');
const viewerOrigin = 'https://viewer.example';
window.addEventListener('message', (event) => {
  if (event.origin !== viewerOrigin || event.source !== frame.contentWindow) return;
  if (event.data?.channel !== 'spatial-ar-viewer' || event.data.version !== 1) return;
  if (event.data.type === 'ready') {
    frame.contentWindow.postMessage({
      channel: 'spatial-ar-viewer', version: 1, requestId: 'load-1',
      type: 'load-assembly', assembly: exportedAssembly,
    }, viewerOrigin);
  }
  if (event.data.type === 'response' && !event.data.ok) {
    console.error(event.data.error.code, event.data.error.message);
  }
});

ready means the host bridge is listening, not that external meshes or recognition models have finished downloading. Every command needs a unique requestId (1-128 characters). Responses echo it with ok and either a state snapshot or a structured error. The bridge is disabled without parentOrigin.

Command	Payload	Behavior
load-assembly	assembly: object or JSON string	Validates before replacing live state; returns field-level errors without losing the current assembly
get-state	none	Assembly identity/revision/source, part count, active step, selected occurrence, completed steps, progress, AR source
select-part	partId: occurrence id or null	Selects a known part or clears selection
set-step	stepId	Selects a known instruction step; does not sign it off
reset	none	Resets the current assembly

Exit AR before load-assembly or reset; otherwise the response is ar-active. Loading a revised manifest with the same assembly id rebuilds the scene and clears old recognition, placements, and animation state.

Teamcenter is an export boundary in this release, not a live connector. A trusted server-side adapter must authenticate to Teamcenter, resolve the BOM revision/configuration and occurrences, convert JT/STEP/CAD geometry to glTF 2.0 or GLB, and produce an AssemblyDef. Keep Teamcenter credentials out of the viewer and manifest. Resource URLs need normal browser access/CORS; arbitrary protected PLM endpoints cannot be read by the browser without a host-managed delivery path.

Teamcenter/export concept	Runtime field
Unique assembly occurrence	PartDef.id (never collapse repeated bolts into one id)
Item/part number and revision	sku, revision
Original PLM identifiers	Optional source: { system: "teamcenter", itemId, revisionId, occurrenceId } on assembly/parts
Occurrence transform	targetPose, normalized to metres and the viewer's left-handed Y-up assembly frame
Tessellated geometry	mesh: { type: "url", url, bounds, scale? }; glTF loader handles glTF's native handedness
Work instructions, mates, tolerances	steps, connectors, defaultTolerance; these must be authored, not inferred from a mesh

sourceUnits records CAD authoring units for display only; it does not convert transforms or dimensions. bounds are full local dimensions in runtime metres. Use unit quaternions [x,y,z,w]. A geometry-only export can use steps: [], background: [], and empty connector arrays until instructions are authored. Imports reject duplicate occurrence ids, non-finite transforms, invalid resource URLs, dangling part/connector/step references, and cyclic work instructions.

The offline worker caches the app shell and runtime bundles only. It does not intercept CAD/GLB files, ONNX model files, or PLM API responses; authenticated and no-store requests bypass it, and it leaves other host applications' caches alone. Reserve /assets/ for public build resources. Serve private CAD textures and other protected resources outside that directory with appropriate HTTP cache headers; the service worker is not a replacement for server-side access control.

Recognition is not inferred from CAD. Production recognition requires a catalogue-specific trained model, explicit class-to-part mapping, and a held-out camera dataset covering devices, lighting, occlusion, and visually similar revisions. A 2D class match does not establish an occurrence's 6-DoF pose, manufacturing tolerance, seating, or revision identity. Repeated occurrences of the same item must remain ambiguous unless separate spatial evidence resolves them. The built-in grid/marker trackers are specialized registration aids, not general Teamcenter part recognition.

React hosts can pass a stable recognitionConfig to App. Class names must match the ONNX output order; mapping values identify runtime occurrences, not SKUs:

const recognitionConfig = {
  detector: {
    url: '/models/validated-catalogue.onnx',
    inputSize: 640,
    format: 'yolov8' as const,
    labels: ['bracket', 'bolt'],
  },
  labelMapping: {
    bracket: 'bracket-occurrence-1',
    bolt: ['bolt-occurrence-1', 'bolt-occurrence-2'],
  },
};
// Define this config outside render, or memoize it when host inputs change.
<App config={{ embedded: true }} recognitionConfig={recognitionConfig} />

With a mapping, unmapped classes are unknown; multiple candidate occurrences are ambiguous and are not painted as a confirmed match. Classifier-only models do not create localized overlays. Configuration/model errors appear in AR settings, and changing a catalogue, step, or recognition configuration invalidates stale inference results.

The Mendix shim now forwards detectorModelUrl / classifierModelUrl, with detectorLabelsJson / classifierLabelsJson required for the corresponding model and optional labelMappingJson using the same mapping contract. Invalid host class JSON visibly disables inference instead of guessing a label order. The iframe message API does not yet replace models dynamically; deploy a catalogue-configured viewer or use the React/Mendix configuration API.

<!-- Drop the guided viewer into any page -->
<iframe src="https://your-host/?ui=minimal&embedded=1"
        style="width:100%;height:600px;border:0" allow="camera;xr-spatial-tracking"></iframe>

A live embed demo ships with the build at /embed-demo.html — it hosts the app in an iframe with switches for every preset, panel, density and accent, and shows the URL to copy. Run npm start (or npm run serve) and open http://localhost:8080/embed-demo.html.

Versioning & BOM

Every part carries a clean revision (revision, plus optional revisionDate / supersedes) — aerospace letter revisions (A < B < … < AA) or dotted numeric, ordered and compared by src/engine/versioning.ts. From that:

• a Bill of Materials (one line per part number and revision, quantities and mass rolled up), shown in-app under the BOM tab;
• an assembly content fingerprint — a stable build id derived from every part's revision, so an as-built record traces to exactly the versions fitted;
• a wrong-revision relation (revisionRelation) that distinguishes a superseded (older) part from a newer one — the dangerous shop-floor case.

Deploying

The repo ships server/serve.mjs for a container or a shop-floor box, and a vercel.json for a static host. They set the same things: Permissions-Policy for camera and sensors, immutable caching for the fingerprinted bundles, no-cache for the HTML shell and the service worker, and deliberately no X-Frame-Options, so the app stays embeddable.

A static host is not the same environment as the bundled server, and one failure only appears on the second deployment: a service worker that serves its cached HTML shell hands the browser an index.html from the previous build, whose fingerprinted bundles no longer exist. That is a blank screen, and nothing local reproduces it. So npm run deploy:check serves the real build from a directory it swaps underneath a headless browser and walks the sequence a user actually gets — first visit, offline visit, visit after a redeploy, offline again:

npm run build && npm run deploy:check

The service worker is split accordingly: /assets/* is cache-first (the names are fingerprinted, so a cached copy cannot be stale), everything else is network-first with a 3 s timeout and a cached fallback. It also precaches the bundles named by index.html at install, because on a first visit the worker is not yet controlling the page and would otherwise cache a shell whose scripts it never stored — offline that looks fine right up until it has to work.

Upgrading from an earlier deployment: a device that already registered the previous worker gets one stale load before the new one takes over. One reload fixes it; there is no way to avoid that from the new build, which is the point of not shipping the old policy in the first place.

Running

npm install
npm run dev         # http://localhost:5173

# On a device, AR needs a secure context (camera + sensors):
npm run dev -- --https   # or tunnel the port behind a trusted cert

npm test            # unit tests across engine + vision + versioning
npm run typecheck
npm run build       # production bundle

Run the packaged app

The build is served by a zero-dependency Node server — no framework, no extra install — so a built copy runs anywhere Node 20+ runs:

npm start                      # build, then serve on http://0.0.0.0:8080
npm run serve                  # serve an existing ./dist
PORT=3000 npm run serve
node server/serve.mjs --isolate   # COOP/COEP headers for multi-threaded ONNX WASM

Or as a container:

docker load -i spatial-ar-image-X.Y.Z-amd64.tar.gz   # from a release
docker run -p 8080:8080 spatial-ar:X.Y.Z
docker build -t spatial-ar . && docker run -p 8080:8080 spatial-ar   # or build it

Each release carries every component: the web app as .zip/.tar.gz, the server as an npm package (npm install -g ./lucadesimoni-spatial-ar-viewer-X.Y.Z.tgz, then spatial-ar-viewer), the container image for amd64 and arm64 (docker load), and the Mendix widget as an .mpk. None is published to a public registry. How releases are cut and checked: docs/RELEASING.md.

AR on a device needs a secure context — pass --https --cert cert.pem --key key.pem, or put the server behind a TLS-terminating proxy / tunnel.

Plugging in recognition models

No model weights are bundled — the app ships small and runs fully geometry-only until you supply a model. The best public starting point for industrial AR is Ultralytics YOLO (YOLO11 / YOLOv8) exported to ONNX: real-time, WebGPU- accelerated in ONNX Runtime Web, and the pipeline's decoder already handles its output (src/vision/defaultModels.ts, COCO-80 labels bundled; both YOLOv5 and YOLOv8/YOLO11 output formats are decoded, auto-detected).

It recognises generic objects, not your specific parts. For real part recognition, fine-tune on a few hundred labelled images and name the classes after your part IDs (so the on-part discrepancy overlay maps 1:1), export to ONNX, host it, and set VITE_DETECTOR_MODEL_URL (see .env.example). A stock model can be mapped to parts for a demo via remapLabels.

GPU acceleration — per device

• Rendering uses WebGL2 — then WebGL1, then a software rasteriser — so it is GPU-accelerated on every device with a GPU (src/render/babylon/engineFactory.ts, powerPreference: high-performance). The status bar shows the live backend (WebGL2 (GPU) / Software (no GPU) …). WebGPU is supported but opt-in with ?gpu=webgpu: the scene is a few dozen low-poly meshes that gain nothing measurable from it, and no headless browser in this repository's test environment can create a WebGPU context, so it is the one renderer the checks cannot cover. It falls back to WebGL if initialisation fails.
• ML inference uses the WebGPU execution provider where available (fast), falling back to multi-threaded WASM (CPU) — so recognition runs everywhere, just faster where WebGPU exists (modern Android/desktop, and iOS/iPadOS with Safari 26+).

Anchoring: floor, object, marker

The app tries these in order, and tells the operator which one it is using:

Source	Where it works	How
WebXR hit-test	Android, Quest, Vision Pro	The device reports a real plane; the reticle sits on the true floor and the anchor is world-locked. The session is entered by the app itself and the HUD is composited over the camera through dom-overlay; if the session is refused for any reason the app falls back to camera passthrough rather than showing a transparent canvas over a black page.
Object recognition	Everywhere the camera works	The assembly's own facade is found in the frame by src/vision/gridRecognition.ts, giving a full metric pose — no marker, no setup.
QR marker	Assemblies that ship a fiducial	Planar homography from four corners; the most precise of the four.
Aim and tap	iOS Safari (no WebXR)	Gravity plus an assumed eye height defines the floor; the tap lands the assembly at a true distance.

Everything hangs off one anchor node, so whichever source wins, the whole assembly moves with it and the parts stay aligned to each other and snapped to their mates.

Following it once it is recognised

Detection on its own updates the anchor once or twice a second, which is fine for a shelf that stays put and useless for an operator who moves: the overlay freezes between answers. So a lock is handed to vision/latticeTracker, which follows the lattice intersections from frame to frame by normalised cross-correlation — coarse pass at half resolution to cover fast motion, fine pass to land sub-pixel — and refits the homography every frame.

Three details are what make it hold rather than slide:

• after each frame the points are snapped back onto the fitted homography, so per-point error cannot accumulate — the object's own rigid geometry is the correction, and points briefly lost behind a hand come back on their own;
• points that moved unlike the rest are dropped before the fit, and a fit whose surviving points do not span three rows and three columns is refused outright, because points confined to two lines leave the perspective unconstrained and will happily produce a confident wrong pose on a repeating pattern;
• beyond its motion budget it lets go and re-detects instead of matching something that merely looks similar.

Cost is about 3 ms per frame at 480x360, against 2.4 ms for a full detection — so speed is not really the point. The point is continuity, and that tracked points feed a full homography: the detector needs the facade roughly square-on, whereas once the lock exists it survives the operator walking round to an oblique view.

Recognising an object without a marker

A generic object detector tells you that there is furniture in the frame, which is worth nothing to AR — you need the outline and the scale. A regular grid facade gives both: the openings are identical, so their pitch is a number straight out of the BOM. gridRecognition projects gradient energy onto each axis, peak-picks the board lines, and fits a periodic lattice by exhaustive search over spacing and phase, scored by inliers minus missing lines. Clutter in one bay adds a stray peak that simply does not fit the lattice. It is a few milliseconds of plain JavaScript, deterministic, and it needs no model download. The facade must be roughly square-on (within ~20°), which is how you stand in front of a shelf anyway.

Placing a part, and the snap

Drag a part in the 3D view and let go: the drag runs on a plane through the part's centre facing the camera, and only the release is committed, so the snap solver sees one decision rather than sixty intermediate ones. Within capture range (60 mm / 35°) the part seats itself on the joint; outside it, the part stays where it was dropped and the diagnostics say why — that is the difference between an assist and a lie. The step card's Place does the same thing for the whole step at once, from each part's standoff, so the residuals mean something; the first part of an assembly has nothing to snap to and is simply put where it belongs.

npm run place:check drives this with a real pointer: a part released 38 mm off its joint must end at 0 mm from nominal with a recorded snap, and one dropped 140 mm out must stay out and raise NOT_ENGAGED.

Checking a step from the camera

Once a shelf is recognised and anchored in AR, the app looks at each part of the active step in the camera picture and says whether it is in place: per part ("seen" / "not seen" / why it cannot check) and for the step ("every part looks in place", or "2 of 4 not in place yet"). It is a suggestion; the operator still ticks the step.

Two readers vote:

• an edge check, which asks whether the image has edges where the part's outline should be;
• a small trained model, which looks at the patch where the part should be, given its expected outline.

A vote counts when they do not contradict each other, and three agreeing looks in a row settle a part either way. The model (147 KB, ONNX, runs in the browser) was trained on synthetic renders of the assemblies. How it was made, how good it is, and how real device frames become training data are in docs/PRESENCE_MODEL.md.

What the guidance looks like

The step card tells you what to fit; the 3D view has to tell you which parts, and the two must not have to be read against each other:

• The active step's parts are highlighted and every other part is dimmed to a ghost.
• Each highlighted part carries its own name, pinned to it and projected every frame. Four identical shelves in a row would produce four overlapping labels, so a tag is dropped when it would land on one already placed and the remainder is reported as a count.
• Show me plays that step's parts flying in along their approach direction, staggered in the order they go in — which is what shows how a part goes in and which way round it is. It is a demonstration: placements are untouched and the view returns to exactly what it was.

npm run steps:check asserts all three against every step of every bundled assembly: that each label belongs to the step it is shown for, that "Show me" moves the step's own parts and no others, and that it changes nothing.

A part's approach is where it is brought in from — a board dropped from above is [0, 1, 0]. The opposite reading is just as natural and was in half the sample data, which is why parts used to fly in through the assembly and the exploded view pushed them into it. The type carries the definition now.

The AR view on a phone or tablet

In AR the camera image is the interface, so the desktop layout is put away entirely — no header, no side columns, no bottom bar. What is left is a single thumb-height HUD over the live view:

• the current step, always visible, with prev/next arrows;
• Steps, Errors and View open the corresponding panel in a sheet that covers under half the screen and closes with the same button;
• Move goes back to aiming at the floor, to re-place the assembly — because placement is a mode, not a permanent state. The reticle appears while you are placing and goes away once the assembly is down, and a tap after that is someone touching the screen rather than a request to move the workpiece. Whether AR asks for a placement at all on entry is a setting: turn it off and it opens where you left it; Leaving AR hands the camera back to the system — the tracks are stopped and the <video> element is detached, because an element still holding a dead stream is enough for the next getUserMedia on Android to come back NotReadableError: Could not start video source. Teardown is fault-tolerant step by step, so nothing can leave the app convinced it is still in a session it has left. And in AR the overlay is a reference registered to a real workpiece, not a model to manipulate: dragging and selecting parts are off there, because a stray touch was quietly placing parts and opening an inspector over the guidance.

When AR cannot start, the app says so. A denied camera used to make the button do nothing at all, which is indistinguishable from a broken build; now the reason is on screen, and the placement badge names which path is running — WebXR, or the camera fallback.

• Settings holds the placement behaviour above and the two numbers the browser will not tell us — the camera's field of view and how high the device is being held. Both are live: drag the FOV slider until the overlay matches the real object, drag the height until the reticle sits on the real floor. ?camfov=52 pre-sets the first.

Every control is at least 48 px tall, the bar is capped at 760 px wide so it stays under the thumb on a 13-inch iPad, and the canvas takes touch-action: none so a tap places the assembly instead of scrolling the page. The screen is held awake for the session — a tablet that dims mid-step drops the camera feed and the anchor with it.

Outside AR a phone gets one bottom bar — steps, errors, view modes, the register/collaborate/BOM drawers, and the way into AR — each opening the same sheet. It used to be three stacked bars plus floating drawer tabs over the model, which left the 3D view a letterbox on a 390 px screen.

Two layout rules exist specifically because of iOS Safari:

• The HUD is a normal flex child of a 100dvh column, never position: fixed. iOS resolves a fixed element against the layout viewport — the tall one that exists when the toolbars are hidden — so a bar anchored to its bottom sits behind the toolbar on a real iPhone while every desktop viewport looks perfect. npm run ar:verify asserts the computed position and that the HUD's bottom edge is inside the viewport.
• The way into AR lives in the bottom nav on a phone, not in the header. In the header it is one long assembly name away from being pushed off the edge, and the top of the screen is the worst place to reach one-handed.

The overlay is rendered at the field of view actually visible rather than the camera's own: the video is painted with object-fit: cover, so a 4:3 frame in a landscape tablet loses about 7% of its height, and drawing at the uncropped FOV would shrink the whole overlay by the same amount. It is recomputed on every resize, because rotating an iPad changes which axis gets cropped.

Sample assemblies

Three are bundled; switch between them with the picker at the top of the step guide, or open one directly with ?assembly=kallax-4x4.

• src/data/gearbox.ts — a small worked two-stage bench gearbox (11 parts) that exercises every diagnostic: a handed pair of bearing caps (the swap trap), a keyed shaft (roll matters), bolts that must follow their housing, a keep-out volume over the output shaft, and a strict six-step build order. The best place to read how the model works.
• src/data/kallax.ts — a 4x4 cube shelf (20 parts, 8 steps) modelled on the IKEA KALLAX 147 x 147 cm, so the AR path can be tested against a real object that is actually in people's homes. Its facade is the markerless recognition target. Dimensions are derived from the two published numbers (1470 mm across, 330 mm opening ⇒ 30 mm board) rather than guessed, so the model and the joints cannot drift apart.
• src/data/jetEngine.ts — a complex twin-spool turbofan on its transport stand (50 parts, 16-step dependency graph), colour-coded by module: blue fan case, titanium fan blades, teal booster, green compressor, copper combustor, turbines in heat colours, silver nozzle. It exercises what the others cannot: parts nested round a common axis (shafts through spools through casings), rings of identical parts installed as sets (16 balanced fan blades, 8 fuel nozzles), radial insertion, and an intake danger zone. Pass-throughs are derived from the real radii, so a genuine clash between concentric parts is still reported — the first draft's spinner reached 29 mm into the LP shaft and the check caught it. Open it with ?assembly=jet-engine.
• src/data/equipmentRack.ts — a large 14-bay modular equipment rack (108 parts, 46-step dependency graph) that stresses the app at scale: handed left/right rail pairs on every bay (swap detection ×14), a rear cable-channel keep-out, background floor/wall occluders, and enough geometry to exercise the renderer, the exploded view, and the full-build animation. It is generated by a joint helper so all ~110 parts fit cleanly at nominal, verified in tests.

Testing

docs/TESTING.md lays out the five layers — types, unit, browser, deployment, production — what each one can prove, and the checklist of things that need a human with a phone because a headless browser has no camera optics, no gyroscope and no WebXR device.

Where this is going, and what is missing

ROADMAP.md has the architecture, the rules that hold it together, and the staged plan. TODO.md is the honest list of gaps, stated approximations (assumed camera intrinsics, assumed eye height, axis-aligned recognition) and things that cannot be verified without hardware.

Requirements compliance

See COMPLIANCE.md for a traceability matrix mapping every requirement from the brief to where it is implemented and how it is verified.

Production & performance

Maximum performance on every device. On startup the renderer detects a device tier (cores, memory, GPU renderer, texture limits — src/render/perf.ts) and picks a profile: pixel-ratio cap, antialias, target FPS, and recognition cadence. On top of that, Babylon's SceneOptimizer auto-degrades under live load — if the scene can't hold the target frame rate it lowers internal resolution until it can, so a heavy assembly stays smooth on a weak tablet instead of stuttering. Static background geometry is frozen and per-move picking is off. Override for benchmarking/kiosk tuning: ?perf=low|mid|high|max, ?dpr=1.5, ?fps=30.

Production server. server/serve.mjs adds gzip (cached by mtime), a /healthz liveness probe, immutable caching for fingerprinted /assets/*, no-cache for HTML, and a Permissions-Policy granting the camera/AR sensor features (no X-Frame-Options, so it stays embeddable). Multi-threaded ONNX WASM via --isolate (COOP/COEP). The Dockerfile has a HEALTHCHECK.

Offline-capable PWA. A web app manifest makes it installable (standalone, full-screen — good for a tablet kiosk), and a service worker caches the app shell (stale-while-revalidate) so it opens and runs offline on flaky shop-floor Wi-Fi. OpenCV.js, ONNX Runtime's wasm and the Draco decoder are served with the app from its own origin (src/vision/runtimeUrls.ts): nothing is fetched from a third-party host, so the app works behind a firewall that blocks CDNs and the service worker caches the runtimes for offline use.

Error reports. A deployed build posts each uncaught error, unhandled rejection and render crash to /api/errors on its own origin (src/diagnostics/errorReport.ts): the message, the stack, the build, the page path and the user agent, at most five per page view, URLs scrubbed of their query strings. server/serve.mjs writes each one to stdout as a JSON line ("type":"client-error"), and on Vercel api/errors.js writes it to the function's log (the project ▸ Logs). Neither keeps the client's IP address. Set VITE_ERROR_REPORT_URL at build time to send them elsewhere, or to off.

Privacy notice. public/privacy.html, linked beside the build line in the Log drawer, says what the app stores and sends. It names Outthinkx as responsible and Vercel as host: a licensee running its own deployment must publish its own.

Going to production

• Serve over HTTPS (required for camera / sensors / WebXR): --https --cert cert.pem --key key.pem, or a TLS-terminating proxy / tunnel.
• Run with --isolate for threaded ONNX inference (the runtimes are already self-hosted).
• Supply your trained ONNX model URLs and map the model's class labels to part IDs so the on-part discrepancy overlay fires.
• CI (.github/workflows/ci.yml) runs typecheck + tests + build on every push.

License

Copyright (c) 2026 Outthinkx. Licensed under the PolyForm Noncommercial License 1.0.0 (see LICENSE).

• Free, with no time limit, for noncommercial use: personal use, hobby projects, study and research, and use by charities, schools, public research and health organisations and government institutions.
• Commercial use needs a paid licence from Outthinkx: using it in or for a business, selling it, bundling it into a product, or offering it as a service. Contact Luca De Simoni, Outthinkx (github.com/lucadesimoni).

The open source components the app includes keep their own licences; every distribution carries them in THIRD_PARTY_NOTICES.txt.

Versions 0.1.0 and 0.1.1 were published under GPL-3.0-or-later, and copies of those versions remain under it. Version 0.2.0, first published under an evaluation licence, is also available under these terms.