EdgeSentry — robot-cell safety agent (OpenCV 5 + AWS)
Built for the OpenCV AI Competition 2026, powered by AWS, and entered for the Agentic Vision Award.
EdgeSentry watches the camera over an industrial robot cell and decides what to do about
every person who steps inside the hazard zone: stop the robot, ask a supervisor, or
keep monitoring. OpenCV 5 does the seeing. A tool-using agent on Amazon Bedrock does the
deciding. Code guardrails veto any unsafe or unproven action. Evidence goes to S3, alerts go
to SNS, and a supervisor closes the loop from the browser.
Live demo: https://jzyv7mljhphuteckul2lxqe77u0xbqev.lambda-url.us-east-1.on.aws/ (AWS Lambda, us-east-1; capped daily usage)
Technical report: docs/TECHNICAL_REPORT.md · Evaluation data: docs/EVALUATION.json · Live agent trace: docs/TRACE_EXAMPLE.json
What happens when you press "Run safety check"
- See: OpenCV 5 compares each frame of a 24-frame camera clip to the empty-cell
reference (after correcting camera drift and lighting), finds people, splits groups,
and tracks everyone across frames.
- Verify: an intrusion only counts after 1.5 s inside the zone, and a worker is
re-inspected up close before any stop is allowed.
- Decide: the Bedrock agent reads the OpenCV scene report, calls the close-up
inspection tool, and picks an action per worker. Guardrails block anything unsafe, and
a controller sends the agent back if it leaves anyone unresolved.
- Act: each action stores an evidence frame in S3. Stops and approval requests are
published to SNS. A supervisor approves the entry or stops the robot from the page.
Results (all measured; reproduce with python scripts/evaluate.py)
| What | Result |
|---|
| Bedrock agent, 10 live trials on the reference clip | 10/10 runs fully correct, 0 false stops, 0 missed stops, 0 fallbacks; median 8.7 s per decision run |
| Guardrails and controller in those trials | 4 unsafe actions blocked (logging a verified intrusion); 9/10 runs needed a nudge to resolve a skipped worker |
| Robustness sweep: noise, lighting ±30/+80, camera shift 3–8 px, 1° rotation, 2 fps, grouped/kneeling workers | 16 of 17 variants fully correct. The 1 failure is a vest at 8% brightness, which fails safe (an unneeded stop, never a missed one) |
| Ablation: no close-up re-inspection | 0.75 accuracy with 1 false stop on the reference clip |
| Ablation: no registration / lighting / group splitting | Misses a stop under lighting +30, camera shift 8 px, +80 lighting, grouped workers and rotation |
| Deployed Lambda (2048 MB), 5 warm runs per mode | OpenCV perception ~49 fps at 1280×720; fixed policy 0.2 s, Bedrock agent 7.3 s median; 4 evidence frames + 3 alerts per run |
The clip is synthetic, so ground truth is exact. That makes the evaluation reproducible, but
it says nothing about real-world accuracy. See Limitations.
Architecture
flowchart LR
U[Browser] -->|HTTPS| URL[Lambda Function URL]
URL --> L[AWS Lambda · Python 3.12<br/>OpenCV 5 perception + safety agent]
L -->|Converse API, tool use| BR[Amazon Bedrock<br/>Amazon Nova Pro]
L --> S3[(S3 evidence<br/>private · encrypted · 30-day expiry)]
L --> SNS[SNS alert topic<br/>KMS-encrypted]
L --> DDB[(DynamoDB<br/>daily usage cap)]
L --> CW[CloudWatch logs, EMF metrics,<br/>alarms, dashboard · X-Ray]
Agent loop
flowchart TD
R[get_scene_report<br/>OpenCV 5 tracks, dwell, coarse PPE] --> A{Bedrock agent}
A -->|inspect_worker| I[OpenCV close-up re-inspection<br/>2x upscale · HSV · morphology]
I -->|vest found / not found| A
A -->|stop · approval · log| G{Code guardrails}
G -->|allowed| X[S3 evidence + SNS alert]
G -->|blocked, with reason| A
A -->|stopped with workers unresolved| N[Controller nudge]
N --> A
X --> H[Supervisor: approve entry / stop robot]
A -.->|model unavailable, or it decided nothing| P[Fixed policy<br/>same tools, same guardrails]
A -.->|model left some tracks open| Q[Fixed policy finishes only those tracks]
Run it
Python 3.11+.
pip install -r requirements-dev.txt && pip install -e .
pytest -q # 54 tests, real OpenCV on synthetic frames
python scripts/serve_local.py # the web app at http://localhost:8080 (offline, fixed policy)
python scripts/evaluate.py # robustness sweep + ablations -> docs/EVALUATION.json
python scripts/evaluate.py --bedrock 10 # + live Bedrock agent trials (needs AWS credentials)
Deploy to your own AWS account (AWS CLI v2 and Python; no Docker, SAM or CDK needed):
AWS_REGION=us-east-1 ./infra/deploy.sh # builds the Linux package, deploys infra/template.yaml
AWS_REGION=us-east-1 [email protected] ./infra/deploy.sh # optional: email alerts
python scripts/live_check.py <AppUrl> --runs 5 # measure the deployed endpoint
AWS_REGION=us-east-1 ./infra/teardown.sh # delete everything
Amazon Bedrock model access for Amazon Nova Pro must be enabled in the account. Without
it, the app still works: the fixed policy decides, and the page says so.
Repository map
| Path | What it is |
|---|
edgesentry/detector.py | Clean-plate detector: phase correlation + ECC registration, lighting compensation, envelope differencing, Otsu, morphology, connected components, head-profile group splitting, close-up PPE inspection |
edgesentry/agent.py | Perception summary, agent tools, guardrails, Bedrock Converse loop with controller, fixed policy, evaluation |
edgesentry/service.py, lambda_function.py | Cloud service: S3/SNS delivery, CloudWatch EMF metrics, HTTP API, usage caps, supervisor decisions |
edgesentry/web/index.html | The web app (light/dark, no build step) |
edgesentry/synthetic.py | Industrial-cell scene renderer with exact ground truth |
infra/ | CloudFormation template, deploy and teardown scripts |
scripts/evaluate.py, scripts/live_check.py | Reproducible evaluation and live-endpoint measurement |
legacy-js/ | Earlier Node.js prototype, kept for history. Not part of this submission |
Responsible use
EdgeSentry is a decision-support and alerting layer. It is not a certified safety
function and does not replace guards, light curtains, or safety PLCs (ISO 10218 / ISO 13849).
It never fabricates a stop: every stop needs a verified intrusion plus a close-up confirmation.
It never silently drops presence: unrecognised shapes are treated as people. Faces are not
identified, and evidence frames expire after 30 days. More in the
technical report.
License
MIT
