usaspending-agent

A tool-calling assistant for questions about USASpending.gov federal spending data. An LLM agent picks between retrieval (conceptual/definitional term lookups, and semantic NAICS/PSC/CFDA/county code lookups) and live USASpending API calls (actual numbers), across as many tool calls and turns as a question needs.

Live demo: usaspending-agent.vercel.app (frontend on Vercel, backend on Railway — see Deployment)

Architecture

• Conceptual questions ("what is a sub-award?", "what is an IDV?") are answered by hybrid retrieval over two sources — the Analyst's Guide to Federal Spending Data (a PDF, Q&A-chunked) and the live USASpending Glossary API (~150 terms, one chunk per term) — combined into one Chroma + Whoosh index: dense embeddings (Chroma) + BM25 keyword search (Whoosh), merged and reranked with a cross-encoder.
• Code-lookup questions ("what NAICS code is custom software development?") resolve a plain-English description to a NAICS/PSC/CFDA code or county FIPS code, each (except county, which calls the live location endpoint directly) via its own separate Chroma + Whoosh index, same hybrid-retrieval shape as the conceptual index above.
• Live-data questions ("how much did NSF spend on X?") are answered by calling the real USASpending.gov API.
• A tool-calling agent decides which tool(s) a question needs, including questions that need more than one. It runs on LangChain's create_agent (a LangGraph graph), via langgraph_tools.py's LangChain-compatible wrappers of the same tool functions), with a SQLite checkpointer keyed by conversation_id giving real multi-turn memory — a follow-up like "what about last year?" resolves against the prior turn's history.
• A cheap classifier gates obviously out-of-scope questions before the (more expensive) agent loop runs at all — a first-turn version (bare question + best retrieval passage) and a separate follow-up version that folds in prior conversation turns, so a continuation like "was that a lot?" isn't misjudged as off-topic just because it has no keywords of its own.
• Download requests ("can I download NSF's January 2024 awards as a CSV?") are ordinary tools in the tool-calling loop: download_records for filtered award, transaction, or sub-award CSVs via POST /api/v2/download/search/, and download_single_award for one award by its internal ID. Both poll the job and return the file link with a citation. A "Download this" button after a spending answer posts that answer's structured filters straight to the same pipeline (download_handler.py), with no LLM call.
• The model never does multi-number math (totals, percentages, ratios, before/after change, rankings) in its own prose — it calls one of six typed arithmetic tools, or code_execution as a fallback for calculations those six don't cover.

Tools available to the agent

Tool	Answers
search_guide	Definitions and concepts from the Analyst's Guide and the USASpending Glossary, with page or term citations
lookup_agency	An agency's basic profile (toptier code, mission, website)
resolve_naics_code	Plain-English industry/business description → matching NAICS code(s), for the naics_code filter below — semantic matches against the official code index, not a confirmed exact lookup
resolve_psc_code	Same idea for a product/service description → PSC (Product and Service Code) code(s), for the psc_code filter
resolve_cfda_program	Same idea for a federal grant/loan/assistance program description → CFDA/Assistance Listing program number(s), for the cfda_program filter
resolve_county_fips	A county name → its 3-digit FIPS code, for the performed_in_county/recipient_in_county filters — strips local suffixes (Louisiana parishes, Alaska boroughs/census areas) the live endpoint's own match otherwise fails on
resolve_budget_function	A plain-English budget category (e.g. "National Defense", "Medicare") → matching budget_function/budget_subfunction code(s), for get_spending_explorer_breakdown's filters
list_top_agencies_by_budget	Agencies ranked by budget authority, largest first, with each one's share of the total federal budget — always the current fiscal year/quarter, no historical range
get_agency_budget	An agency's actual appropriated budgetary resources, obligations, and outlays for a fiscal year range — the real answer to "what is X's budget," as opposed to the spending tools below, which report award-level spending (a different, non-interchangeable number). group_by="sub_component" breaks the same figures down by sub-component/bureau (e.g. NIH or CDC within HHS) for a single fiscal year instead, optionally scoped to one named bureau's own federal accounts
get_agency_award_breakdown	One agency's award obligations and transaction/new-award counts for a single fiscal year, broken down by sub-agency by default — query_spending's group_by aggregates have no count fields at all. group_by="award_category" breaks down by award category instead, and gives the agency's own single total with no breakdown

Setup

uv sync
cp .env.example .env   # fill in ANTHROPIC_API_KEY (and ANTHROPIC_WORKSPACE_ID if using an
                        # identity-linked key); LANGSMITH_* is optional, for tracing

Building the retrieval indexes (one-time, or whenever a source changes)

The conceptual/definitional index (Guide + Glossary):

uv run python -m backend.app.retrieval.pipeline.ingest \
  --pdf data/raw/analyst-guide.pdf --out data/chunks/analysts_guide_chunks.jsonl
uv run python -m backend.app.retrieval.pipeline.ingest_glossary   # fetches the live Glossary API
uv run python -m backend.app.retrieval.pipeline.vector_index \
  --chunks data/chunks/analysts_guide_chunks.jsonl data/chunks/glossary_chunks.jsonl
uv run python -m backend.app.retrieval.pipeline.bm25_index \
  --chunks data/chunks/analysts_guide_chunks.jsonl data/chunks/glossary_chunks.jsonl

The three code-lookup indexes (resolve_naics_code/resolve_psc_code/resolve_cfda_program), each its own Chroma collection/Whoosh index via env var overrides so they don't collide with the one above or each other — source files already in data/raw/ (Census NAICS files, the PSC manual, the SAM.gov CFDA bulk CSV):

uv run python -m backend.app.retrieval.pipeline.ingest_naics
uv run python -m backend.app.retrieval.pipeline.ingest_psc
uv run python -m backend.app.retrieval.pipeline.ingest_cfda

CHROMA_DB_DIR=./data/chroma_naics WHOOSH_INDEX_DIR=./data/whoosh_naics \
  uv run python -m backend.app.retrieval.pipeline.vector_index --chunks data/chunks/naics_chunks.jsonl
CHROMA_DB_DIR=./data/chroma_naics WHOOSH_INDEX_DIR=./data/whoosh_naics \
  uv run python -m backend.app.retrieval.pipeline.bm25_index --chunks data/chunks/naics_chunks.jsonl
# same pattern for psc (chroma_psc/whoosh_psc, psc_chunks.jsonl) and cfda (chroma_cfda/whoosh_cfda, cfda_chunks.jsonl)

Running

Two processes: the FastAPI backend, and the Next.js frontend that talks to it through a server-side proxy route.

uv run uvicorn backend.app.main:app --reload --host 127.0.0.1 --port 8000

cd web && npm install   # first time only
npm run dev

• Browser UI: http://localhost:3000 — the empty chat state shows 5 randomly-sampled quick-question buttons (from web/src/lib/demoQuestions.ts's hand-vetted set of 20, each verified live to trigger its expected tool call) for a fast first-touch demo
• API: POST /ask with {"question": "...", "conversation_id": "..."}, returns {answer_text, source_type, conversation_id, charts, citations, tool_citations, downloads} — omit conversation_id on the first call, then pass back the one returned to continue the same multi-turn thread
• Rate limited to ASK_RATE_LIMIT_PER_MINUTE (default 20) requests/minute per client IP
• Health check: GET /health

You can also run the agent directly from the CLI, without starting the server:

uv run python -m backend.app.agent --question "What is a prime award?"

Deployment

Backend on Railway, frontend on Vercel — two separate deploys, connected by one env var.

Backend (Railway):

• Connect this repo; Railway builds Dockerfile (see railway.json). The image bakes the Guide/Glossary/NAICS/PSC/CFDA indices in at build time (read-only at runtime, never rewritten), so only the conversation-history DB needs persistent storage.
• Add a Volume mounted at /data — CONVERSATIONS_DB_PATH already defaults to /data/conversations.db in the image.
• Env vars: ANTHROPIC_API_KEY (required), ANTHROPIC_WORKSPACE_ID (if using an identity-linked key), LANGSMITH_* (optional tracing), ASK_RATE_LIMIT_PER_MINUTE (optional, default 20).
• Single-instance/single-worker only — the SqliteSaver checkpointer is one sqlite connection in one process. Don't set --workers > 1 or scale replicas past 1 (see #72 for the Postgres swap that would allow that).
• Health check: GET /health (already wired into railway.json).

Frontend (Vercel):

• Deploy web/ as the project root.
• Set FASTAPI_BASE_URL to the Railway backend's public URL. The browser never calls the backend directly — web/src/app/api/ask/route.ts proxies server-side — so no CORS setup is needed on the FastAPI side.

Testing

uv run pytest -q      # unit tests: chunking/ingestion (Guide, Glossary, NAICS, PSC,
                       # CFDA), retrieval merge logic, chart/citation eligibility,
                       # arithmetic tools, API client parsing, conversation persistence,
                       # the LangGraph tool wrappers — no network calls, no API cost
uv run ruff check .

agent/dev_tools/ and retrieval/dev_tools/ hold separate, opt-in scripts (real billed LLM calls for most of them) not run by the above or by CI — see Project layout below.

Project layout

backend/app/
  main.py                  FastAPI app (POST /ask, GET /health)
  logging_config.py        Shared logging setup (server + CLI)
  agent/                   Tool-calling agent (package)
    singletons.py            Lazily-constructed clients: Anthropic/LangChain chat
                                models, the Guide+Glossary and NAICS/PSC/CFDA
                                retrievers, the USASpending API client, and the
                                LangGraph checkpointer + conversation graph
    tools/                   The 21 @beta_tool data tools, split by concern:
                                _shared.py (call recording, per-turn budget,
                                untrusted-data wrapping, + search_guide/
                                lookup_agency/get_agency_budget (also covers
                                the old get_agency_budget_by_subcomponent via
                                group_by)/list_top_agencies_by_budget/
                                get_agency_award_breakdown), spending/
                                (category.py/over_time.py/search.py/geography.py/
                                consolidated.py - query_spending, consolidating
                                get_spending_by_category/get_spending_over_time/
                                search_awards/search_transactions/search_subawards/
                                get_spending_by_geography (the six stay as
                                internal implementation, not bound to the agent),
                                all funneled through tool_filters._build_filters),
                                awards.py
                                (get_award_details/get_award_subawards/
                                get_award_funding_breakdown/
                                get_award_transaction_history), recipients.py
                                (search_recipients/get_recipient_details/
                                get_recipient_children), award_type_breakdown.py
                                (get_award_type_breakdown), disaster.py
                                (get_disaster_spending_overview),
                                spending_explorer.py
                                (get_spending_explorer_breakdown) +
                                business_type_labels.py
                                (its code->label table), naics.py/psc.py/cfda.py/
                                budget_function.py/location.py (the resolve_*_code /
                                resolve_budget_function / resolve_county_fips
                                semantic code-lookup tools). See docs/adding-a-tool.md
                                for the tool registration checklist.
    tool_filters.py           Shared filter-building layer for the spending tools
    arithmetic_tools.py       The six deterministic arithmetic tools
    langgraph_tools.py        LangChain-compatible wrappers of the same tool
                                functions, for the LangGraph agent path
    scope.py                  In-scope gate: a first-turn classifier plus a
                                separate follow-up classifier that folds in prior
                                conversation turns
    download_handler.py         CSV download pipeline shared by the download tools
                                and the "Download this" button
    response_shaping.py       Chart/citation/download-spec logic, fiscal-year math
    orchestrator.py           System prompt, AgentResult, ask() - runs the
                                LangGraph conversation path
    cli.py                    The --question CLI entry point
    dev_tools/                Manual, opt-in scripts (real billed LLM calls unless
                                noted, not in CI): red-team checks (data-injection,
                                jailbreak, prompt-extraction, resource-abuse),
                                tool-selection, code-lookup, and download-handler
                                accuracy evals (LangSmith Dataset + evaluate()),
                                scope-classifier calibration, live 3-turn conversation-path
                                verification, a code_execution wiring check, live
                                verification of the spending-tool filters
                                (verify_shared_filters.py), a free/local
                                coverage-diff check against the live API contract
                                (check_filter_coverage.py), and a LangSmith
                                trace-dump utility (fetch_trace.py)
  retrieval/
    hybrid.py                 Dense+sparse retriever with cross-encoder reranking (used at request time)
    pipeline/                  One-off scripts: Guide PDF/Glossary API/NAICS/PSC/CFDA source data -> chunks -> indexes
    dev_tools/                 Manual scripts: sanity_check.py, calibrate_threshold.py
  usaspending/
    client.py                 Typed client for the live USASpending.gov API
    models.py                 Award/recipient/agency response models
    filter_models.py          AdvancedFilterObject models + coverage tracking
    capture.py                 ContextVar-based live-request capture
    exceptions.py               USASpendingAPIError and HTTP error handling
web/                         Next.js frontend (separate process; proxies to the
                                FastAPI API through web/src/app/api/ask/route.ts)
tests/                       Unit tests
BACKLOG.md                   Known gaps and deferred work