An ESP32-CAM pan/tilt turret that tracks motion on-device and serves a live web HUD
What it does
Void Sentry watches a room through an OV3660 camera on an ESP32, finds the moving object by frame differencing, and points a stepper-plus-servo turret at it. Everything runs on the board: no PC or cloud is involved. A browser HUD streams the live video over a WebSocket and shows tracking telemetry. The camera is fixed on the base and the turret aims absolutely: image position is mapped to pan/tilt angles through a two-point calibration you do with the joystick or from the HUD. The firmware is written in C++20 on the Arduino core, with the pure logic split into libraries that are unit-tested on the host.
Highlights
- Cheap detection path: each QVGA JPEG is decoded at 1/8 scale (DC coefficients only) to a 40x30 grey grid in about 7 ms, versus about 71 ms at 1/4 scale.
- Measured under load: with a person close to the camera and the turret following, the worst 5 s window costs 11.4 ms per frame for decode plus tracking (25-30 ms in Stage 3, very likely caused by a HUD request storm, see the engineering notes), and every captured frame is analysed (83-84 per 5 s).
- No per-frame allocation: the camera task copies frames into 4 fixed 32 KB PSRAM slots; readers hold RAII leases, and a leased slot is never rewritten.
- Dual-core by design: core 1 runs camera copy, detection and control; core 0 runs network, the camera driver and the motor task. A 20 min soak showed no reboots and about 147 KB heap free (min 137 KB).
- Smooth motion: trapezoidal stepper profile at a 1 kHz motor tick (max 400 steps/s, accel 800 steps/s^2), rate-limited servo (120 deg/s), soft limits and a 2 s hold after the target is lost before returning home.
- Tested off-device: the pure logic is unit-tested on the host (also under ASan + UBSan, with coverage tracked per library in CI), plus a regression bench over recorded real clips and synthetic clips.
- Hardware-driven fixes: a camera clock change (20 MHz to 12 MHz) took ping from 265 ms to 19 ms, and a HUD request-loop bug that saturated core 0 was found with a purpose-built idle counter. Both are written up in docs/engineering-notes.md.
- Browser-origin checks on the WebSockets: handshakes from another browser origin, or with a non-IPv4 Host header (DNS rebinding), are refused. Non-browser clients on the LAN are not authenticated yet.
Architecture
flowchart LR
CAM["OV3660 camera"] --> CT["camera task (core 1)"]
CT --> FS["FrameStore<br/>4 x 32 KB PSRAM slots"]
FS --> DT["detect task (core 1)<br/>JPEG decode, MotionDetector, Tracker"]
DT --> SNAP["track snapshot"]
SNAP --> CTRL["Controller<br/>AimPolicy, AimMap"]
JOY["Joystick"] --> CTRL
CTRL --> MT["motor task (core 0)"]
MT --> STEP["Stepper (yaw)"]
MT --> SERVO["Servo (pitch)"]
FS --> ST["stream task (core 1)"]
ST -->|"/stream WebSocket"| HUD["Browser HUD"]
HUD <-->|"/commands WebSocket"| CTRL
Frames flow one way: camera, slots, detection, snapshot, controller, motors. The stream task reads the same slots for the HUD. Controller depends only on interfaces (BaseDetectionModule, BaseTurret), and objects are wired by reference in src/main.cpp.
httpd is pinned to core 0 so network bursts cannot preempt detection. The full task table (cores, priorities, stack sizes) is in CLAUDE.md, and the reasoning in the engineering notes.
Performance
Measured on the board (QVGA JPEG, quality 18, camera clock 12 MHz, about 16.6 fps capture).
| Measurement | Result |
|---|
| Decode, 1/8 scale DC-only vs 1/4 scale | about 7 ms vs about 71 ms |
| Per-frame decode + track, empty scene | 7.7-9.1 ms + 0.49-0.59 ms |
| Per-frame decode + track, person close, turret following (worst 5 s window) | 11.4 ms (Stage 3: 25-30 ms) |
| Frames analysed per 5 s | 83-84 (every captured frame) |
| 20 min soak | no reboots, heap about 147 KB free (min 137 KB) |
| Camera clock 20 MHz vs 12 MHz: ping | 265 ms vs 19 ms (capture 27.8 vs 16.6 fps) |
A higher camera clock degrades WiFi on this board, so 12 MHz is the best frame rate that keeps the network healthy. The full sweep is in the engineering notes.
Hardware and wiring
Parts: Freenove ESP32-WROVER CAM (OV3660, PSRAM required) on the Freenove GPIO Extension Board; 28BYJ-48 stepper with a ULN2003 driver (yaw); SG90 servo (pitch); 2-axis joystick with push button.
| Function | GPIO |
|---|
| Joystick X / Y (ADC1, works with WiFi on) | 32 / 33 |
| Joystick button | 0 |
| Servo (pitch) | 14 |
| Stepper IN1-IN4 (yaw) | 13, 12, 15, 2 |
With the camera attached exactly 8 GPIOs are free, and the turret uses all 8. ESP32/include/pins.h is the only place pins are defined, and its static_asserts reject flash/PSRAM pins, camera collisions and duplicates. GPIO12 high at reset selects 1.8 V flash and causes a boot loop, so check the ULN2003 polarity before powering on.
여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗
