IRLSurfer: Play Subway Surfers With Your Body Using a Webcam
Webcam body-tracking game controller: MediaPipe pose estimation turns side-steps, jumps and ducks into keyboard input in real time.
Live Web Demo (Zero-Install) | Features | How It Works | Benchmarks | Quickstart | Contributing
Overview
IRLSurfer is an edge computer vision controller that translates whole-body kinematics from an ordinary webcam into low-latency keyboard inputs to control Subway Surfers in real time.
Most computer vision input projects suffer from spatial sensitivity (breaking when players shift relative to the lens), high-frequency landmark jitter, and false keystroke triggers caused by baseline drift or natural idle fidgeting. IRLSurfer eliminates these failure modes through an end-to-end deterministic pipeline: MediaPipe Pose landmark extraction, perspective-invariant body space normalization ($W_s$ scaling), adaptive speed-based low-pass filtering (One Euro Filter), and a Schmitt-trigger finite state machine with velocity gating.
Key Technical Achievements
- Sub-Millisecond Pipeline Latency: Core compute takes 0.81 ms per frame on an Apple Silicon CPU (>1,200 FPS throughput), decoupling input latency from inference overhead and running purely bound by camera hardware frame rates (30/60 FPS).
- Zero False Triggers (0.00 FTPM): 0 false triggers across both stationary idle baselines (30.0s) and natural distractor motions (20.0s of head-scratching, arm waving, and chair adjustment).
- 100% Macro F1 Detection Accuracy: Evaluated against a 110.0-second labeled ground-truth benchmark suite ($\pm 150\text{ ms}$ temporal acceptance window) across all 5 game gestures.
- Dual-Runtime Parity: Shared schema (
config/gestures.json) powers identical state machines in Python 3.12 (Desktop OS controller via pynput) and TypeScript (Zero-install browser demo), validated via automated cross-language parity test suites.
Features
- Full-body gesture control: side-step left or right to switch lanes, jump to jump, duck to roll, and touch your wrists together (held for 0.3 s) to fire the hoverboard.
- Works with any webcam: no depth camera, Kinect, or wearable needed. Pose detection uses Google MediaPipe Pose Landmarker (lite, full, or heavy models).
- Real Subway Surfers control: the Python desktop controller sends Left / Right / Up / Down / Space key presses via
pynput, so it drives Subway Surfers in a browser, emulator, or native app.
- Zero-install browser game: a TypeScript + Vite web build runs MediaPipe in the browser (WebAssembly) with a built-in 3-lane arcade runner. Play it here.
- Distance-invariant tracking: movements are measured in shoulder widths, so thresholds hold whether you stand close to or far from the camera.
- Jitter-free, low false-trigger input: One Euro adaptive filtering plus a Schmitt-trigger state machine with velocity gating and auto-recentering.
- Debug HUD overlay: OpenCV window shows the skeleton, displacement vectors, detector state, and latency.
- Record, label, evaluate, benchmark:
irlsurfer record, label, eval, and bench subcommands for building and scoring gesture datasets.
Demo
- Live browser demo: oj102.github.io/IRLSurfer (allow camera access, or play with the keyboard).
- Desktop controller: run
irlsurfer play, open Subway Surfers in another window, and step back until your shoulders and hips are in frame. The HUD shows STEP BACK - TOO CLOSE / STEP CLOSER - TOO FAR / DISTANCE OK guidance.
Tech Stack
| Layer | Technology |
|---|
| Pose estimation | MediaPipe Pose Landmarker (Python mediapipe, web @mediapipe/tasks-vision) |
| Video capture and HUD | OpenCV (opencv-python) / browser MediaStream + Canvas |
| Signal processing | One Euro Filter, NumPy |
| Gesture detection | Deterministic Schmitt-trigger finite state machine (shared config/gestures.json) |
| Keyboard output | pynput (macOS Accessibility / CoreGraphics) |
| Web runtime | TypeScript 5, Vite, Vitest |
| Quality | pytest + coverage, mypy (strict), Ruff, GitHub Actions CI, GitHub Pages deploy |
Live Demo vs. Desktop Controller
| Runtime | Environment | Input Mechanism | Primary Use Case |
|---|
| Web Runtime | Browser (WebGL / WASM) | Direct Canvas Keystroke Dispatch | Zero-install live demo in any browser |
| Desktop Controller | Native Python 3.12 CLI | OS Input Hooks (pynput / CoreGraphics) | Control Subway Surfers on web, emulator, or native desktop |
How It Works
Each webcam frame goes through six stages: capture, MediaPipe pose estimation, body-space normalization, One Euro filtering, the gesture state machine, and keyboard dispatch. A 2-second calibration at startup records your neutral standing position. Full design notes live in docs/architecture.md.
System Architecture
The pipeline strictly decouples asynchronous frame capture, pose estimation, coordinate transformation, temporal filtering, state machine detection, and OS input dispatch:
以上為 README 的開頭部分。其餘內容請閱讀原文。 檢視完整 README ↗
