Alpine Outdoor – Support Copilot
Course prototype of AI-assisted customer support for a fictional outdoor retailer.
Pattern: the system suggests, a human decides. The AI never sends replies or issues refunds.
Pipeline
- Extraction (LLM, run manually in an AI chat): customer message → structured JSON. The app makes no API calls.
- Rules (TypeScript, deterministic):
lib/rules.ts decides eligibility and escalation from data/policy.md.
- Draft reply (LLM, coming next): written only from the policy and the rules decision.
- Human review: agent approves, edits, or escalates.
Project layout
| Path | Purpose |
|---|
data/policy.md | Support policy (source of truth) |
data/orders.json | 15 mock orders |
data/test_cases.json | 25 customer messages with answer keys. Never upload this to the AI. |
lib/rules.ts | Rules engine (no LLM) |
lib/evaluate.ts | Runs rules on every test case and compares with the answer key |
lib/extractSchema.ts | Zod schema for the AI extraction |
lib/manualEval.ts | Scores pasted AI output (browser-safe): validation, hallucination checks, accuracy, consistency |
lib/manualResults.ts | Reads and writes saved results (server only) |
prompts/extract_v1.txt, prompts/extract_v2.txt | Prompts for the AI chat (simple and improved) |
manual/cases_for_ai.txt | The 25 messages for the AI, without answers |
scripts/eval-manual.ts | Scores one pasted run and saves it |
results/raw/<label>.json | AI output exactly as pasted |
results/<label>.json | Scored run |
app/page.tsx | Agent workspace: AI output vs. answer key |
app/run/page.tsx | Browser-only eval for non-technical users |
app/eval/page.tsx | Comparison of AI runs, plus rules evaluation |
docs/ai-log.md | Log of AI experiments and decisions |
Commands
npm install
npm run dev # http://localhost:3000
npm run test:rules # rules accuracy on the answer-key extraction
npm run eval:manual -- --file results/raw/v1-claude-run1.json --label v1-claude-run1
npm run build
Manual AI runs
Anyone can run an evaluation in the browser at /run, with no install: it provides the prompts and files, and scores the pasted AI reply in the browser. Nothing is stored, so download the CSV to keep results.
To publish a run on / and /eval, save the reply to results/raw/<label>.json, run npm run eval:manual, and commit results/. Those pages read results/*.json when the app is built.
Labels ending in -run1, -run2, ... form one group, which is how consistency is measured.
The answer key ships in the browser code so /run can score without a server. This is fine for a mock dataset, but anyone determined can read it.
