TollChat
Cost dashboard · Daily AWS and OpenAI billing; per-answer cost is not yet measured.
A deployed AI application with separate development and production environments, deterministic pricing tools, input and output guardrails, and PII redaction before trace export.
Live demo · Engineering evidence ·
Evaluation status · Technical guide
Experiment journal: how I test and improve the agent
Follow the hypotheses, baseline comparisons, measured results, costs, limitations, and decisions behind changes to TollChat.
For humans
TollChat answers a practical question: "What would this Northern Virginia commute cost me?" It turns fragmented toll feeds, operator maps, and rate schedules into current prices and annual commute estimates.
I built the data pipeline, directed PostgreSQL/PostGIS route model, two deterministic pricing tools, and the Strands agent that uses them. I deployed the application on Amazon Bedrock AgentCore with separate AWS development and production environments, reviewed release workflows, and safety checks around the conversation. Diagnostic traces redact detected personal information before export.
This reference implementation demonstrates how I build and operate an AI application, including the boundaries around the model. Six scheduled scenarios now exercise simulated conversations with deterministic tool-call checks and model-based judges. A locally authored suite of 50 training, 10 shadow and 25 external holdout cases scores whole conversations; findings are in the experiment journal. Exposed training results do not qualify production.
Engineering evidence
Each row links a claim to its implementation and a way to inspect or verify it. Tests describe checked behavior; workflow definitions describe delivery controls. Neither alone proves a successful live deployment. Recorded results below retain their original scope.
Architecture and safety
The agent resolves intent and helps users choose supported entrances and exits. It can call exactly two tools. The tools validate directed routes, apply schedules and freshness rules, and return pricing components and totals. The model cannot submit its own route plan or pricing components.
flowchart LR
U[User] --> I[Input guardrail]
I --> A[Strands agent on AgentCore]
A --> C[Current toll tool]
A --> B[Annual commute tool]
C --> O[(PostgreSQL / PostGIS routing and pricing)]
B --> O
V[VDOT feeds] --> S[S3 snapshots and loader] --> O
P[Operator maps and rate schedules] --> O
A --> G[Output guardrail] --> R[Checked snapshots and completed answer]
A -. Telemetry copies .-> T[PII redaction before export]
T --> L[CloudWatch traces and logs]
The public interface uses CloudFront and WAF. Inside the runtime, the input check runs before the agent. Each streamed assistant-text snapshot and the completed answer pass an output check before delivery; tool activity can also stream. Guardrail errors return a safe failure response.
Telemetry uses a separate guardrail. It scans copies of diagnostic content without changing the active conversation or tool results. If redaction times out, fails, or cannot process content safely, the exporter substitutes an omission marker. CloudWatch masking adds another layer, but does not erase underlying originals. See the privacy notice for processing and retention limits.
Observed trace redaction
Supplied trace capture: both addresses appear as {ADDRESS} in a gen_ai.user.message event.
The visible message is How do I get from {ADDRESS} to {ADDRESS}?. This is evidence of address redaction in this trace, not a measurement of detection coverage across PII types or environments. Automated detection can miss information; the telemetry runbook describes synthetic verification across export destinations.
Source-backed routing and pricing
Northern Virginia has four distinct pricing systems: I-66 Inside the Beltway, the I-95/I-395/I-495 Express Lanes, Dulles Toll Road, and Dulles Greenway. Their dynamic prices, directional schedules, ramp charges, and cross-road connections require different rules.
I reverse-engineered VDOT feed identifiers against public calculators and operator maps to recover directed ramps, aliases, pricing keys, and cross-road handoffs. The source mappings and data builder produce the routing oracle's checked-in SQL inputs. Coordinates help identify nearby ramps; proximity alone never creates a road connection.
That work exposed 330 pricing IDs in the I-95/I-495 route map but only 314 in retained VDOT history. The missing 16 affected 107 of 685 published routes. The proxy mapping uses source-overlap evidence to select available price proxies and labels those prices as modeled. The historical holdout summary documents their measured error and limitations.
Blue-green deployments and conversation cutover
An agent release changes both the application and the context in which a conversation runs. I built blue-green delivery to validate the next release while the active release continues serving traffic. Candidate checks verify the actual release identity, a grounded agent answer, guardrail behavior, and trace redaction. Production requires a separate human approval before cutover, followed by fresh validation before traffic switches.
Promotion changes routing to the validated candidate and retains the previous release for recovery. During post-cutover observation, two consecutive probe failures trigger one rollback attempt. The deployment runbook documents the validation gates and recovery procedure; public and private routing updates are not globally atomic.
Conversation state has an explicit release boundary. Each session carries a release ID. If a request reaches a different release—or uses a legacy session without an ID—the chat proxy rejects it before invoking or resetting AgentCore, clears the session cookie, and tells the user to start a new conversation. In-flight requests may finish, but prompts are never automatically replayed. I chose a visible restart so an existing conversation is not silently continued under a different agent release.
The user-facing restart message when a session does not match the serving release.
Evaluation status
Corpus 5.2.19 is active: 50 training and 10 shadow cases in the repo, 25 holdout
cases stored externally. Training hill-climbing defaults to one trial per case;
shadow runs are manual, not a CI gate. The holdout is not blind (it was read for an
October 10 diagnosis), and exposed development measurements do not qualify
production or establish whole-agent accuracy. The former production qualification
gate remains retired. Production delivery
uses the existing artifact provenance, protected human approvals, saved-plan,
migration, and canary checks described in the runbook.
These delivery checks do not establish independently measured agent accuracy.
The experiment journal records what was tried,
results, limitations, costs, and decisions. The evaluation guide
covers current usage. Granular experiment output stays in private storage.
The dashboard runbook covers scheduled operational
results, which are separate from local experiments.
Reproduce and inspect
| Area | Contents |
|---|
v2/agent/ and v2/agent_tools/ | Agent, frontend, versioned prompt/tool contracts, and deterministic tools |
v2/db/, v2/oracle/, and v2/lambdas/ | Database, migrations, source mappings, data ingestion, and chat proxy |
v2/tests/ and v2/eval/ | Executable checks, evaluation cases, and recorded experiments |
v2/infra/, infra/, and .github/workflows/ | Application/shared infrastructure and delivery workflows |
License
Unless otherwise noted, project-authored source code and documentation are available under the Apache License 2.0.
