Multimodal Camouflaged Object Detection (COD)

Pixel-level segmentation of camouflaged objects with a DINOv2 RGB branch, optional estimated depth (Depth Anything V2) and four RGB↔depth fusion strategies, plus a FastAPI backend and a React/TypeScript UI.

Research question: can reliability-aware fusion of RGB features and estimated monocular depth improve COD over a strong RGB-only foundation-model baseline without unacceptable overhead? Depth here is predicted from the RGB image by a network — it is not sensor depth and not ground truth. Measured results live in docs/RESULTS.md (auto-generated) and are discussed, with caveats, in docs/REPORT.md. Nothing in this repo claims state-of-the-art performance.

Layout

cod/                 python package
  data/              COD10K + folder datasets, augmentation, depth-aligned samples
  depth/             Depth Anything V2 wrapper, robust normalisation, uint16 PNG cache
  models/            DINOv2 backbone, depth encoder, fusion modules, multiscale decoder, CODModel
  training/          losses, trainer (resume, early stop, NaN guard, warmup+cosine, progressive unfreezing)
  evaluation/        S/Fβw/E/MAE/mIoU/Dice, evaluation loop, depth degradations
  inference/         Predictor (per-stage timings, depth memoisation), overlay/mask helpers
  api/               FastAPI app + settings (env-driven)
configs/             YAML experiment configs (`_base_` inheritance, dotted CLI overrides)
scripts/             train / evaluate / predict / benchmark / precompute_depth / check_dataset / export_onnx / ...
frontend/            Vite + React + TypeScript UI
tests/               pytest suite (metrics, data, model, losses, inference, API)
docs/                REPORT.md, RESULTS.md (generated), STATUS.md

Setup

Tested: macOS 15 arm64 (M4, 24 GB), Python 3.14, PyTorch 2.14 (MPS). Node ≥ 20 for the UI.

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt            # add requirements-dev.txt for tests
cd frontend && npm install && cd ..

Device selection is automatic (cuda → mps → cpu; override with --device / COD_DEVICE). On CUDA install the matching torch wheel from pytorch.org first; training uses bf16 autocast there (train.amp: auto) and fp32 on MPS/CPU. Pretrained weights (facebook/dinov2-*, depth-anything/Depth-Anything-V2-*-hf) download from Hugging Face on first use.

Data

COD10K v3 must be downloaded by you from the official project page and its usage terms followed; set data.root (default ~/Downloads/COD10K-v3). Only the camouflaged images (COD10K-CAM-*; 3 040 train / 2 026 test) are used; NonCAM images are excluded. COD10K has no official validation split, so 10 % of train is held out with a seeded shuffle. CAMO, CHAMELEON and NC4K are not bundled or downloaded: pass them to scripts/evaluate.py --extra NAME IMAGE_DIR MASK_DIR after obtaining them from their authors (and cache depth for them under data.depth_cache/NAME).

python scripts/check_dataset.py             # pairing, corrupt files, empty masks, cross-split near-duplicates
python scripts/precompute_depth.py          # estimated-depth cache (uint16 PNG, ~10 img/s on M4)

Train / evaluate / infer

python scripts/train.py --config configs/vits_rgb.yaml train.epochs=30            # RGB-only baseline
python scripts/train.py --config configs/vits_reliability.yaml train.epochs=30    # reliability-aware fusion
python scripts/train.py --config configs/vits_gated.yaml model.boundary_head=true loss.boundary=0.5
python scripts/train.py --config configs/vits_rgb.yaml --resume                   # resume from experiments/<name>/last.pt
python scripts/run_ablation.sh 4 && scripts/run_eval_all.sh 4 && python scripts/make_report.py

python scripts/evaluate.py --checkpoint experiments/NAME/best.pt --out res.json   # [--no-depth] [--degrade shuffle|noise|...]
python scripts/predict.py image.jpg --checkpoint checkpoints/mm.pt                # mask + overlay PNGs
python scripts/benchmark.py --checkpoint ... --image ...                          # params / latency / memory
python scripts/export_checkpoint.py experiments/NAME/best.pt checkpoints/mm.pt    # strip optimiser state
python scripts/export_onnx.py --checkpoint checkpoints/mm.pt --out mm.onnx        # + onnxruntime parity check
tensorboard --logdir experiments

Fusion types (model.fusion): none (RGB only), concat, cross_attn, gated, reliability. Every run writes config.yaml, env.json (hardware), history.json, summary.json, last.pt, best.pt (selected by train.val_metric) under experiments/<name>/. Starting values for all hyper-parameters follow the project brief (AdamW, backbone 1e-5 / head 1e-4, wd 1e-4, warmup + cosine, BCE + Dice, grad-clip 1.0); they are configuration, not tuned optima.

Run the application

export COD_RGB_CHECKPOINT=checkpoints/rgb.pt COD_MM_CHECKPOINT=checkpoints/mm.pt   # either or both (.env.example)
uvicorn cod.api.app:create_app --factory --port 8000
cd frontend && npm run dev            # http://localhost:5173 (proxies /api to :8000)
# or: (cd frontend && npm run build) && COD_FRONTEND_DIR=frontend/dist uvicorn cod.api.app:create_app --factory

This is the beginning of the README. Read the rest at the original. Read the full README ↗