shop-support-agent

A customer-support agent for a fake online shop that looks things up and files requests through tools, with guardrails, a human-approval gate for refunds, and an evaluation suite that measures how reliably it follows its rules.

Live demo: https://shop-support-agent-demo.vercel.app

The shop, its products, orders and customers are all fake sample data. The demo runs within Groq's rate limits, so it can answer "busy" when many people use it at once.

Screenshots

What it is and why

The agent answers questions about products, orders and refunds for a sample shop called Example Goods. It uses four tools, and its policy forbids stating a product, price or order fact that no tool returned. Anything risky is checked in code, not left to the prompt. Order data needs a matching order ID and email. Saving contact details needs the customer's consent. A refund request only goes into a queue for a human. Every reply comes with a trace: model calls, tool calls, guardrail events, tokens and an estimated cost. An evaluation suite of 36 tasks measures the agent over repeated trials, because one passing run says little about an agent.

How a request flows

flowchart TD
    A["Chat page"] -->|"POST /chat with the whole chat"| B["FastAPI server"]
    B --> C{"Rate limit"}
    C -->|"over the limit"| C1["429 with Retry-After"]
    C -->|"allowed"| D{"Input guardrails"}
    D -->|"attack pattern"| D1["Fixed refusal, no model call"]
    D -->|"clean"| E["Agent loop with the model"]
    E -->|"tool call"| F["Dispatcher and tools"]
    F -->|"tool result"| E
    F -->|"request_refund"| G["Pending approval in the store"]
    G -.->|"only when ADMIN_TOKEN is set"| H["Human approves or rejects"]
    E -->|"final reply"| I{"Output guardrails"}
    I --> J["Reply and trace"]
    D1 --> J
    J --> A

The server keeps no chat memory, so the page sends the whole conversation with every message. The rate limit allows 10 messages a minute per client address. It runs before the request body is validated, so invalid requests count too. The input guardrails check every customer message, including earlier ones, against the patterns in guardrails.toml. A match gets a fixed refusal and no model call. The agent loop sends the policy, the chat and the tool schemas to the model, runs any tool calls, and repeats, up to 5 model calls and a 40-second deadline per request. The dispatcher validates every tool call. It checks identity for order tools and consent for saving contact details, and returns an error code the model must explain. A refund request becomes a pending approval in the store. Only a human with the admin token can approve or reject it, and the agent can never do either. The output guardrails check the final reply for policy leaks, invented action claims, and emails or phone numbers the customer did not type.

Tools

Tool	What it does	Risk	Control around it (in code)
search_products	Searches the catalogue for products, prices and stock	low	Read only. Arguments validated.
get_order_status	Returns an order's status, items, dates, tracking and total	low	Read only. Needs the order ID and the email used for the order. An unknown order and a wrong email return the same no_matching_order, so order IDs cannot be probed.
capture_lead	Saves a name, email and message for the support team	medium	Write. Refused with consent_required unless a customer message in the chat clearly agrees (app/tools/consent.py). The email is validated. Contact details are not echoed back into results or traces.
request_refund	Files a refund request for a human to review	high	Write, but it only files a request and never moves money. Same identity check as orders. Eligibility is checked in code: delivered, within 30 days, no earlier request, a reason given. It returns no amount, so the model has nothing to promise.

여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗