CLAVE
CLAVE sorts recyclable waste traveling on a conveyor belt through a learned
perception-action policy. A neural architecture fuses visual perception,
object tracking, and pick timing into an end-to-end system trained by
imitation of a scripted expert.
The policy runs against the clock with
hard safety guarantees: a Rust core handles interlocks and workspace limits,
while neural inference provides perception and decision-making. Models train
in Python against MuJoCo simulation in this repository. Everything through the
v1.x line runs in simulation.
CLAVE, which stands for Coleta de Lixo Auxiliada por Visão Embarcada, is also Portuguese for the musical clef, which places the project in the
same family as arco, bossa, fret, and luthier.
About
The hard problem is learning a policy that maps visual streams to physical
actions. Classifying an object on a belt is a solved exercise; predicting
what to pick, when to pick it, and where to route it under conveyor motion,
variable material properties, and hardware constraints is where the
engineering lives.
CLAVE addresses this by replacing handcoded scheduling rules with a learned
policy trained end to end:
- Neural perception: continuous spatial-temporal representations of the
conveyor state, replacing frame-by-frame snapshots.
- Learned tracking: tracking networks that hold an identity through
occlusion, illumination change, and overlapping objects without geometric
heuristics.
- Policy learning: a perception-action loop trained by imitation from a
scripted expert (behavior cloning), with reinforcement fine-tuning planned
so pick timing survives physical variation.
- Hybrid safety: inference proposes, and a Rust layer disposes. Every
action passes a deterministic check on workspace bounds and actuator
limits before it reaches the arm.
The split between learned policy and deterministic safety is where Rust
earns its place, and the perception stage becomes a replaceable component
behind a contract.
Installation
git clone --recurse-submodules https://github.com/alexandrelheinen/clave.git
cd clave
./scripts/setup.sh # reports missing tools, installs nothing silently
uv sync --locked --extra dev --extra world # Python deps, MuJoCo, dev tooling, as uv.lock pins them
. .venv/bin/activate
cargo build --release -p clave-sitl # Rust safety layer
# Object meshes the world spawns. A few megabytes, checked by digest.
python scripts/import_scene_assets.py --objects
# Optional: scene dressing (shelves, pallets, trash cans).
# Without it the belt uses a plain box on legs and the floor a gray plane.
uv sync --locked --extra dev --extra world --extra assets && python scripts/import_scene_assets.py --warehouse
Running it
Everything that runs the simulation lives under a single command:
clave sim
By default it runs the sorting-line simulation and annotates the rollout with
everything the tracker believes: markers, beliefs, queue decisions, arrival
errors and grasp lifts. A window opens when a display is available.
For example, to run a 30 seconds simulation, using ground truth perception, debug logging, without UI, but dumping the rendering video and telemetry under runs/debug/<identifier> (for PR validation, <identifier> is pr plus the PR number, e.g. pr80):
clave sim --seconds 30 --gt --log-level DEBUG --no-progress --no-window --video --telemetry --out runs/debug/pr80
--no-progress turns the tqdm bar off. Debug lines and the bar share the
terminal, and a bar that redraws a line erases the story of what the arm is
doing. The story itself is the debug log: why an object was chosen, how the
visit will be flown, whether the jaw closed, which chute it was released
into, and any collision, acceleration spike, or loss of control along the
way. A line is an event: a decision that changed, or a watch episode that
began. The physics loop and the capture loop do not each get a line. It is
specified in
docs/requirements/simulation-narrative.md.
Other commands
clave benchmark # compare every configuration in one table
clave record-dataset # record labeled rollouts for training
clave corpus # record the train and validation corpora and publish them
clave train --candidate <name> # train one candidate
clave validate --dataset <digest> --candidate <name> --checkpoint <file>
clave validate-run --outcomes <file> # score against gates
The benchmark runs every configuration in configs/benchmark/default.yml over
the same seeds and prints one comparison, with an evidence pack beside it. Read
docs/measurements.md before reading the table, because
two of the five headline metrics cannot be measured yet and that document says
why.
Simulation
Every number below is configuration in
configs/world/sorting_line.yml, and every one
of them is sized to the manipulator rather than to a photograph of a recovery
facility. docs/measurements.md carries the
measurements that set them.
| Part | Dimension |
|---|
| Belt | 3.00 m long, 0.50 m wide, surface at 0.90 m |
| Belt speed | 0.25 to 0.35 m/s, randomized per run |
| Conveyor modules | 10 modules of 0.300 m, spanning the 3.00 m belt |
| Arm | Universal Robots UR10e, on a pedestal 0.70 m off the belt centerline |
| Effector workspace | an annulus 0.25 m to 1.25 m about the base, over a 0.50 m vertical band, measured by sweeping the model with the tool held vertical |
| Reachable window | 2.071 m of belt on the centerline, 6.6 s per object at 0.31 m/s |
| Sensing | two detection cameras (gate_wide, pick_wide) and three barcode cameras, all nadir |
| Objects | 10 scanned packages, four material classes |
The conveyor modules
The belt is drawn as a row of identical modules from the CC BY 4.0 Open-RMF
model. The module is published at 0.500 by 0.504 by 0.502 m with its belt
surface on top, and CLAVE scales each axis independently to the belt it
configures:
| Axis | Published | Scaled to | Factor |
|---|
| Length | 0.500 m | 0.300 m, so four span 1.20 m | 0.600 |
| Width | 0.504 m | 0.50 m, the belt width | 0.992 |
| Height | 0.502 m | 0.90 m, so the module surface lands on the belt surface | 1.793 |
Change belt.modules and belt.length together: four modules of 0.30 m
make a 1.20 m belt, eight of 0.15 m make the same belt out of shorter sections,
and six modules on a 1.80 m belt lengthen the line. The module count changes
nothing the simulation measures, because the modules carry no collision
geometry: an object still rests on the box the belt has always been.
The objects
Every object is a scanned package. There are no parametric shapes on the
belt: a colored cylinder teaches a classifier shape rather than material, which
is the whole problem CLAVE exists to solve.
| Class | Objects | Source |
|---|
M-02 HDPE | 5 supplement tubs and toiletry bottles | Scanned Objects |
M-04 Other plastic | 1 sprinkles jar | Scanned Objects |
M-06 Ferrous metal | 1 food can | YCB |
M-09 Paperboard | 3 cartons | YCB and Scanned Objects |
The set is the ten meshes whose narrowest sides fit the 85 mm jaw.
master_chef_can is 102.5 mm across and stays off the belt.
Seven classes still have no object, and the gripper was never the reason.
Probing both pinned collections for them finds nothing: no PET bottle, no
individual aluminum can, no glass container, no corrugated box, no beverage
carton. Google Scanned Objects is a retail-shelf scan of toys, shoes,
supplements and electronics, and the pinned YCB subset is ten manipulation
benchmark items. Exactly one bottle exists among the 836 models that fit. Those
classes need a new asset source rather than a wider effector.
Submodule sizes
git submodule update --init --recursive fetches the guidelines, the warehouse
props and nothing else from those collections. The object meshes are a few
megabytes, downloaded by python scripts/import_scene_assets.py --objects
from the commits recorded in configs/assets/objects.yml.
| Submodule | Checked out | What it supplies |
|---|
third_party/scanned_objects | about 15 MB fetched | The meshes the world spawns, by digest, not the 2 GB collection |
third_party/ycb_sim | included above | The YCB meshes the world and the barcode fixtures name |
third_party/aws-robomaker-small-warehouse-world | 17 MB | Warehouse props and textures |
.guidelines | Under 10 MB | The shared guidelines |
Current version
The latest tag is v1.6.0. Everything in this line runs in simulation and
nothing has touched hardware.
| Part | State |
|---|
| Material taxonomy, corpus mappings | 11 classes, four corpora mapped |
| Corpora | TrashNet and ZeroWaste fetched, digested and measured |
| Simulated world | MuJoCo conveyor from Open-RMF modules, UR10e, parallel jaw, chutes, warehouse dressing, randomized per seed |
| Data pipeline | Labeled rollouts, ground-truth boxes, proprioception, digested splits |
| Candidates | Six benchmarked, four trained |
| Runtime | Frame to published decision, with a Rust safety layer that can override the model |
| Integration | The decision on a ROS 2 topic, specified for a consumer to implement against |
| Benchmark | Every runnable configuration compared under one protocol |
What is still open: identity in the proposal loop still comes from the
simulator, the jaw holds about one object in seven, and cycle time to a
chute waits on that hold. No model has been trained on real imagery. The
roadmap carries v1.7.0, v1.8.0 and v1.9.0.
Versioning
CLAVE follows semantic versioning, with each position given a fixed meaning
for this project.
| Position | Meaning |
|---|
| MAJOR | The delivery surface changes. v1.x is the simulation era; a hardware era would open at v2.0.0 and has no plan yet. |
| MINOR | One roadmap step lands and is tagged. A minor is the unit of planned work. |
| PATCH | A fix, a correction, or a change of mind inside a step already tagged. Unplanned by definition. |
v1.6.0 is the latest tag: the feed-rate loop holds a configured setpoint.
A minor is tagged only once its release criteria hold and
./scripts/validate.sh exits 0. The next steps are v1.7.0, v1.8.0 and
v1.9.0. None of them is a claim about hardware.
Ecosystem
CLAVE is one project in a family that shares a method and a set of
guidelines. Contracts point at siblings; source trees are never vendored
between them.
| Project | Role |
|---|
| CLAVE (this repo) | Learned perception and pick policy, plus the safety layer that checks it |
| ARCO | Python planning and control algorithms, including joint-space MPC |
| FRET | ROS 2 and MuJoCo SITL, the ROBOTIS manipulators, and the simulation CLAVE trains in |
| Luthier | Photogrammetry and point clouds |
| BOSSA | C++20 edge runtime on ARM Linux, publishing telemetry to SQLite |
FRET is the integration edge. CLAVE publishes a decision, FRET's planner
nodes consume it, and those nodes call ARCO as a synchronous library rather
than reaching CLAVE directly. FRET also supplies the MuJoCo world CLAVE
trains against, including the sibling ROBOTIS arms and the pinned mesh
submodules. BOSSA carries telemetry once a hardware era opens, which is
outside the v1.x line. What CLAVE takes from each sibling is itemized in the
roadmap.
Repository layout
| Path | Contents |
|---|
| CONTRIBUTING.md | The constitution: quality gates, merge policy, agent rules |
| docs/guidelines.md | Coding notes specific to CLAVE, on top of the shared baseline |
| .guidelines/ | Shared guidelines and the SDD method |
| docs/roadmap.md | The ladder to v1.0.0, its release criteria, and the spec dependency order |
| docs/architecture.md | What each block does, its inputs and outputs, and what is deliberately absent |
| docs/measurements.md | Every measured number a gate or a configuration value rests on |
| docs/trajectory-formulation.md | The mathematics the arm's motion is planned with, and the one place it was wrong |
crates/ | The decision contract, the routing policy, the publisher, the safety layer, the runtime |
src/clave/ | The world, the data pipeline, training, validation, the runtime, the benchmark, the demos |
configs/ | Every tunable. Nothing in Python or MJCF carries a numeric default |
third_party/ | Vendored UR10e and gripper, the warehouse submodule, and object meshes fetched by digest |
| docs/requirements/ |
License
MIT. See LICENSE.
