Water Boards Assistant (student prototype)

A mockup of the California Water Boards website with a chat box in the bottom-right corner. The assistant answers questions using only a saved copy of Water Board pages and files (its data store), links the source for every answer, and says so (and points to staff) when the answer isn't there.

Prototype built by UC Berkeley students. Not official Water Board guidance.

How it stays inside the data store

The model never browses. Four things keep it to our material:

1. The data store is a fixed snapshot. data/data-store.json says what goes in (see "Choosing what's in the data store" below). npm run sources turns the inventory export in data/inventory/ into data/sources.csv, and npm run corpus downloads those pages (and, if turned on, the Water Board pages they link to) and saves the text to data/corpus/. Nothing is fetched while the assistant answers.
2. Gemini gets no tools. We call it without Google Search grounding, URL context or code execution, so it can't look anything up. It only sees the passages we pass in with each question.
3. The prompt only allows answers from those passages. The model must return JSON with a status (answered, not_found, clarify, out_of_scope), the answer, and the ids of the passages it used.
4. Code checks every reply before it's shown. An "answered" reply that doesn't cite a passage it was given is replaced with a hand-off to staff, and any web address that isn't one of our sources is removed.

Gemini still has its training knowledge, so the guardrails make outside facts much less likely, not impossible. The eval's Grounded criterion checks for exactly this.

How a question is answered

visitor question
  -> query rewrite (small Gemini model turns "appeal" into "petition for review", Fresno into Central Valley Region 5, ...)
  -> search the snapshot with the question plus up to 3 rewritten queries (BM25 + optional embeddings)
     -> top 10 passages, at most 3 per page
  -> Gemini (no tools) with the rules in lib/prompt.ts -> JSON {status, answer, citations}
  -> guardrails in lib/agent.ts -> answer + source links shown in the chat
  -> question logged (no names or emails)

Rubric item	Where it's handled
Cited	Every answered reply must cite a passage; the chat shows the page link (PDFs open at the cited page).
Knows its limits / hands off	not_found status, then the program contact from the passages, or Contact Us and the regional board map.
Clarifies	clarify status: one question (county, type of business, drinking water vs. wastewater...).
Stays in scope	out_of_scope status for legal advice, predictions about someone's own case, and off-topic requests.
Plain language	Prompt rules: short sentences, explain acronyms, under ~120 words.
Current	Each passage carries the date printed on its page or file; the prompt prefers the newest dated version and gives today's date.
Visible disclaimer	Banner at the top of the page and a fixed line in the chat box.
Question log	lib/questionLog.ts; the notice is shown in the chat box and on the page.

Full-site search (whole waterboards.ca.gov)

The chat can also search a crawl of the whole site, kept in an Upstash Vector index. When the index is set up, every question searches both the snapshot in data/corpus/ and the index, and the results are merged. The index counts for more: the snapshot is a small, older subset of it, and at equal weight its pages crowded out newer ones that only the index has (such as the current wastewater fee schedule).

Search check: open /api/search?q=your+question on the site to see the passages the assistant would read for a question, whether each came from the index or the snapshot, and whether the index is reachable (stillIndexing is how many new passages Upstash has not finished indexing). It does not call the chat model or log the question.

1. Create the index at console.upstash.com → Vector → Create Index: name waterboards, region us-east-1, type Hybrid, dense model OpenAI text-embedding-3-small, sparse model BM25, plan Pay as You Go (the free tier holds only about 130,000 passages).
2. Add the keys UPSTASH_VECTOR_REST_URL and UPSTASH_VECTOR_REST_TOKEN to Vercel (Settings → Environment Variables) and to GitHub (Settings → Secrets and variables → Actions). Redeploy Vercel.
3. Start the crawl: Actions → Crawl full site → Run workflow. Each run works about 5 hours, commits its progress to data/crawl/ (summary.json shows the counts) and starts the next run until nothing is left. To stop it, disable the workflow. To start over, run it with "Start over" checked.

The crawler (scripts/crawl-site.ts) starts from the inventory pages, the home page and the sitemap, then follows links breadth-first on www.waterboards.ca.gov only: pages and PDFs, no search URLs, no other websites. It sends one request every 0.4 s at most, backs off when the site answers 403/429, and pauses for 10 minutes if refusals keep coming. Limits live in data/data-store.json under fullSite (maxDocs, maxPassages, maxPassagesPerDoc, maxPdfPages, skipUrlPatterns). Upstash pay-as-you-go allows about 1.3 million passages at 1,536 dimensions; maxPassages stops at 1.25 million. Each run first applies the current limits to what is already stored: pages that now match skipUrlPatterns are deleted from the index and long pages are cut to maxPassagesPerDoc, so changing the limits and running the workflow again (without "Start over") frees the room. The skip list leaves out the 2024 Integrated Report appendix fact sheets and the TMDL reference records, which had filled 80% of the index. Pages come before PDFs in each run, and forms, fee schedules, FAQs and guides before other PDFs.

Cost: storage is $0.25/GB and searches $0.40 per 100,000 on Upstash pay-as-you-go; each chat question makes up to four searches. Upstash's pricing page does not list a separate charge for its hosted embedding model.

Choosing what's in the data store

Edit data/data-store.json and push; the "Build data store" action rebuilds the snapshot and Vercel redeploys.

Setting	Data store	When to use it
"tiers": ["1"], "followLinks": false	the 184 Tier 1 inventory rows	The guidesheet's data store. Use this for the step 5 numbers.
"tiers": ["1", "2", "3"], "followLinks": false	all 450 inventory rows	Adds Tier 2 and 3, including the outdated files the team flagged.
"tiers": ["1", "2", "3"], "followLinks": true (current)	all inventory rows plus up to maxLinkedDocs Water Board pages and PDFs they link to	Widest coverage for the demo. Many test answers (agendas, regional office pages, the current wastewater fee schedule, petition instructions) are not in the inventory at all, only linked from it.

maxPdfPages caps how many pages of each PDF are read; maxPdfMegabytes skips very large files. Every passage the model sees carries the date printed on its page or file, and the prompt tells it to prefer the program's own page, official schedules and regulations, and the newest date when sources disagree, since the wider settings bring in old pages too. The eval summary records which data store it ran against.

Put it online (about 10 minutes)

1. Push this folder to GitHub (git push -u origin main).
2. Build the data store. The "Build data store" action (Actions tab) starts on its own after the first push. It downloads the pages, commits data/corpus/ and takes a few minutes (longer with followLinks). You can also run it locally (below).
3. Deploy on Vercel. Go to vercel.com -> Add New -> Project -> import waterboard-agent. Keep the defaults.
4. In the Vercel project, open Settings -> Environment Variables and add GEMINI_API_KEY (Rahnon has the key). Optional: ADMIN_TOKEN (any password) for downloading the question log.
5. Question log: in Vercel, Storage -> Create Database -> Upstash for Redis and connect it to the project. It adds the database variables for you. Without it, questions still show up in Vercel's runtime logs as [question] lines.
6. Redeploy. Share the *.vercel.app link.

If the data store isn't committed yet when Vercel builds, the build downloads it itself (npm run build runs the corpus step when data/corpus/ is missing). If the site can't be reached, the page still deploys and the chat says the data store isn't ready.

Run it on your laptop

Needs Node 22.

npm install
cp .env.example .env.local          # then paste the Gemini key into .env.local
npm run sources                     # inventory -> data/sources.csv, using data/data-store.json
npm run corpus                      # download the pages and files into data/corpus/
npm run embed                       # optional: add embeddings for better matching
npm run dev                         # http://localhost:3000
npm run ask -- "How much is the grade 2 drinking water operator exam?"   # one question in the terminal

Run the eval (Guidesheet step 5)

The two suites from our test sheet are in eval/suites/ (30 Answer Accuracy, 30 Conversation tests).

npm run eval                         # Answer Accuracy, 3 runs per question (rubric rule)
npm run eval -- --runs 1             # 1 run each (use this on a free-tier key)
npm run eval -- --suite conversation
npm run eval -- --ids P1,P3,W4
npm run eval -- --out eval/results/<folder> --resume   # continue after hitting a rate limit

Or from GitHub: Actions -> Run eval -> Run workflow (add GEMINI_API_KEY as a repository secret first). Results are committed to eval/results/<run>/:

• summary.md: pass count, every Critical failure, and a Not in data store table that shows whether the agent said it didn't know.
• grading_sheet_answer_accuracy.csv (and _conversation.csv): one row per run with the agent's reply, links, fact checks and draft grades per criterion, plus blank Final grade and Critical failure? columns for us to fill in.
• results.jsonl: everything, including the passages the agent cited.

A test is marked Not in data store when none of its answer source URLs are in the snapshot and none of its key facts (dollar amounts, emails, phone numbers, dates) appear anywhere in it. For those, the right behavior is "I couldn't find that" plus a staff contact.

The draft grades come from a second Gemini call that sees the answer key and applies our rubric (Correct, Current and Grounded are Critical; Cited and Knows its limits are Major; Direct is Minor; a question passes only if every run passes). Treat them as a first pass and confirm each one by hand before reporting.

Question log

Each entry keeps only the question text, the time, a random id per browser tab, the turn number and the answer status. Emails, phone numbers and "my name is ..." are masked before saving. No IP addresses, names or answers are stored. Download everything as CSV at /api/questions?token=<ADMIN_TOKEN>.

Known data store issues

• Tier 1 snapshot of 10/25/2026: 179 of 184 sources downloaded. Not usable: fee_increase.pdf and dwocp_payment_form.pdf (both now return "not found" on the live site), and three pages with no text (gama/wq_privatewells.html, water_recycling_policy/, diversion_use/water_use.html).
• Only 9 of the 30 Answer Accuracy tests have their key fact anywhere in Tier 1. The answer pages for most tests (meeting agendas, regional office contact pages, the current wastewater fee schedule ww-fee-schedule.pdf, petition filing instructions, basin plans) were never logged in the inventory.
• Conflicting files inside the data store. The drinking water Online Payments FAQ says cards are accepted while the program page and the ACH guide say they are not; the Water Rights Applications FAQ says the minimum annual fee is $100 while the FY 2025-26 fee schedule says $350. The prompt's source-priority rule is there for these.
• One Tier 1 row (water measurement regulation) has a blank URL in the export; scripts/make-sources.ts fills in the page it describes and prints a note.

Files

app/page.tsx                 mock Water Boards page (banner, header, topics, about section)
components/ChatWidget.tsx    bottom-right chat box
app/api/chat/route.ts        chat endpoint
app/api/questions/route.ts   question log CSV export (needs ADMIN_TOKEN)
app/api/search/route.ts      search check: what the assistant would read for a question
lib/agent.ts                 retrieval -> Gemini -> guardrails
lib/prompt.ts                the model's rules
lib/queryTerms.ts            adds the site's own words to a search (test -> examination, grade 2 -> Grade II)
lib/retrieval.ts             keyword + vector search over the snapshot
lib/questionLog.ts           question logging and scrubbing
data/data-store.json         what goes into the data store (tiers, link following, limits)
scripts/make-sources.ts      inventory -> data/sources.csv
scripts/build-corpus.ts      data/sources.csv -> data/corpus/ snapshot
scripts/crawl-site.ts        full-site crawl -> Upstash Vector index (progress in data/crawl/)
lib/vectorStore.ts           full-site index search
scripts/embed-corpus.ts      optional embeddings
scripts/run-eval.ts          eval runner and grader
eval/suites/                 our test suites
data/corpus/text/            readable text of every source after `npm run corpus` (not committed)