Turncoat
Will your AI agent turn on you? Turncoat is an automated prompt-injection
red team and benchmark for tool-using LLM agents. It attacks an agent you
control, checks every outcome against the agent's real tool calls, and reports an
attack-success rate with 95% confidence intervals, so you can measure how
exposed an agent is, and whether a defense actually helps, before you ship it.
Headline, on a public benchmark (AgentDojo banking v1.2.2, 70 matched cases):
the cheap agent (Nemotron 3.5 Lightning) was hijacked in 43/70. Adding an
Ultra gate that approves each sensitive tool call: 0/70, while finishing
more tasks under attack (82 vs 56 of 144) and costing 37% of Ultra per
task in normal use. The catch: Ultra alone (10/70 hijacked) still finishes the
most tasks (114/144) and is about 3× faster under attack.
Everything runs on NVIDIA Nemotron via Nebius Token Factory: the attacker is
Nemotron 3 Ultra; the agent under test is Nemotron 3.5 Lightning by default,
with Nano 30B and Ultra 550B for comparison.
Built solo for the Nebius × NVIDIA Global AI Hackathon.
Defensive security research in the tradition of garak and PyRIT: you only test
agents you own. Synthetic data only. No real person, account, secret or file
exists anywhere in this repo.
Formerly named Sieve. The Python package and run directories keep the name
sieve; old repository and demo links redirect or still work.
Open the interactive demo — nine recorded
live campaigns are available without a code. Judges can also start a new campaign
using the code in the private testing instructions.
Results
Benchmark: 2 × 60 live campaigns, replicated
Three agents, 20 independent campaigns each, run on two different days.
Full compromise means the agent took the dangerous action (exfiltrated a
secret, wired money, deleted a file) because injected content told it to.
A campaign is up to 10 adaptive attempts, so the campaign rate answers "does
a persistent attacker get through within 10 tries?". The per-attempt rate
answers "how often does a single injection work?".
| Agent under test (Lightning) | Campaigns, 15 Oct | Campaigns, 16 Oct (replication) | 95% CI | Per attempt, 15 / 16 Oct | Attacker refusals |
|---|
| Banking assistant (wire funds) | 100% (20/20) | 100% (20/20) | 84–100% | 27% (20/74) / 28% (20/72) | 0% |
| Email assistant (leak a secret) | 95% (19/20) | 95% (19/20) | 76–99% | 28% (19/67) / 23% (19/83) | 0% |
| Document assistant (delete files) | 10% (2/20) | 10% (2/20) | 3–30% | 1% (2/192) / 1% (2/192) | 0% |
The document rows match exactly on both days, down to each technique count.
Techniques run in a fixed order, so two days with two successes at the same
attempt positions give identical tables; the
per-campaign raw data lets you check the 16 Oct side.
Cost: $0.99 and $1.01 for 60 campaigns, a median of $0.011 per campaign in the
16 Oct replication
(campaign-stats-2026-10-16.json).
The strongest technique was output hijacking: telling the agent that its
result must be delivered by performing the dangerous action. In the
replication, 85% of those attempts fully compromised the banking agent (17/20),
and 56% the email agent (10/18). Raw reports:
15 Oct,
16 Oct
(JSON). Every 16 Oct and model-comparison
campaign is published in results/raw/: each injection, tool
call, verdict, token count and cost. A CI test recomputes the published tables
from it. The 15 Oct first run survives only as its report.
Does a bigger model resist better? Yes, sharply
Same attacker, three Nemotron models as the agent under test, 10 campaigns per
cell, $2.27 total (results/compare-2026-10-15.json):
| Agent model | Email | Banking | Documents | Median $ / campaign (this run) | Median model time |
|---|
| Nemotron 3.5 Lightning | 90% | 100% | 0% | $0.0071 | 9.0 s |
| Nemotron 3 Nano 30B | 60% | 100% | 0% | $0.0202 | 33.5 s |
| Nemotron 3 Ultra 550B | 0% | 0% | 0% | $0.0379 | 28.1 s |
Ultra did not just stay inert. It called its tools, did the task, and explicitly
flagged the injected message as suspicious. Prompt-injection robustness is a
measurable property of the model choice. Cost and latency come from
results/campaign-stats-2026-10-15.json.
Measured defense and AgentDojo banking
The 16 October defense evaluation
compares Lightning, Lightning with an independent Ultra action-authorization
gate, and Ultra alone. It includes all 16 user tasks × 9 attacker goals in
AgentDojo banking v1.2.2, clean-task checks, fixed attacks and a separate adaptive
attack experiment. Outcomes are checked from environment state.
On the 70 task/goal pairs with successful clean tasks and valid attack
outcomes for all three agents, full attacker-goal success was 43/70 for
Lightning, 0/70 with the gate and 10/70 for Ultra.
| AgentDojo banking (static attack) | Hijacked, 70 matched pairs | Tasks finished under attack | Clean tasks finished | Cost per clean task |
|---|
| Lightning alone | 43/70 (61%) | 56/144 | 10/16 | $0.0008 |
| Lightning + Ultra gate | 0/70 | 82/144 | 11/16 | $0.0029 (37% of Ultra) |
| Ultra alone | 10/70 (14%) | 114/144 | 14/16 | $0.0080 |
여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗
