Financial RAG — Verified SEC 10-K Question Answering

An agentic RAG system that answers precise financial questions from real SEC 10-K filings — and checks its own numbers against what the company reported to the SEC before it answers. Ask about revenue, R&D, net income, trends, segments, risks or legal proceedings; every figure is cited, and headline figures are verified against SEC XBRL data.

Live demo: https://financialrag-production-420e.up.railway.app/

Measured on 53 SEC-grounded questions	Original pipeline	This agent
Correct headline figure	41 / 53 (77%)	53 / 53 (100%) on the first run and on 3 further runs

See Evaluation for how that was measured and what it does and doesn't prove.

---

What it does

"What was Apple's revenue in FY2024?"
→ Apple's total net sales for FY2024 were $391,035 million (~$391 billion). [1]
  [1] Apple Inc. (AAPL) | FY2024 | Item 8: Financial Statements

"Compare Microsoft and Google R&D spending in 2024"
→ a cited side-by-side table: Microsoft $29,510M, Alphabet $49,326M

"How did JPMorgan's net income trend from 2023 to 2025?"
→ one sub-question per year, answered in parallel, merged into one cited trend

"What was Netflix's revenue in their latest 10-K?"
→ (first time asked: fetches and indexes Netflix's latest 10-K on the fly)

The agent:

• Plans — classifies each query (single company, comparison, trend, out of scope) and extracts companies, years and the metric it is about.
• Fetches what's missing — a company that isn't indexed is resolved via SEC's registry and its 10-K is downloaded, parsed and indexed mid-query.
• Verifies — the answer's headline figure is compared with the figure SEC XBRL reports for that company and fiscal year; every other dollar amount must appear in the retrieved text (or be a sum/difference of stated figures).
• Corrects itself, once per failure mode — see the graph below. Worst case it answers directly from the SEC data with a citation.
• Decomposes comparison and trend questions into per-company/per-year sub-questions, answered in parallel.

Coverage — 12 bundled companies, FY2023–2025 (AAPL · MSFT · GOOGL · AMZN · JPM · WFC · BAC · GS · BLK · STT · TROW · IVZ), plus any other SEC-listed company on demand (the live demo has also indexed e.g. NVDA, TSLA, NFLX, META, V, COST).

---

How it works

query.ask() runs a LangGraph agent (agent/graph.py).

plan ──► early exit (out of scope · unknown company · no company named · LLM outage)
     ├─► qa                                   (one company / year)
     └─► decompose ─► answer_sub × N (parallel, each runs `qa`) ─► synthesize
                                         │
                                         ▼
                                      caveats  (say so if some named companies couldn't be found)

qa:  retrieve ─► generate ─► verify ──ok─────────────────────────────────► finalize
                               │
                               ├─ headline figure wrong/missing ─► inject SEC XBRL fact ─► generate ─► verify
                               │                 (still wrong) ─► answer deterministically from XBRL
                               ├─ refusal ("not found") ────────► broaden retrieval ─► generate ─► verify
                               └─ unsupported figure ───────────► strict retry ─► generate ─► verify

Each corrective edge is guarded by a flag in the graph state, so every step runs at most once — bounded cost, no loops. The original linear pipeline is kept as ask_baseline() so improvements can be measured rather than asserted.

Retrieval

• Hybrid search — dense (BAAI/bge-base-en-v1.5, ONNX) + sparse BM25, fused with RRF, in Qdrant (one collection per ticker + fiscal year).
• Cross-encoder rerank (Xenova/ms-marco-MiniLM-L-12-v2, ONNX) with metric-aware boosts so the consolidated income-statement row beats footnote mentions.
• Parent–child chunking — ~1000-token child chunks are embedded; the full parent section goes to the LLM.

Structured facts (SEC XBRL) — facts/xbrl.py

Reading a number out of a noisy markdown table is where LLMs go wrong (wrong column, year or scale). The same filing's iXBRL tags are published by SEC as clean facts, so for revenue, net income, operating income, R&D, assets, cash and a few more, the agent knows the exact reported value.

• Taken from the fiscal year's own 10-K — a later filing's restated comparative (e.g. Bank of America's recast FY2024 net income) is not used, so the answer never contradicts the document it cites.
• "Net income" means the headline figure attributable to the company (the basis for EPS), not including noncontrolling interests.
• Banks and asset managers use different revenue tags; the module tries the known ones and takes the largest total.

Verification — verification/verify.py

Two checks: the key figure must match XBRL (rounding-aware, e.g. $391,035 million = ~$391.0 billion), and every other dollar amount must be supported by the retrieved text at any of $/$K/$M scale, or be a difference/sum of other stated figures. Percentages and per-share values aren't checked. It also normalises gpt-oss quirks (391 035 with a narrow no-break space).

LLM layer — llm.py

One entry point for routing, generation and synthesis, switchable between Groq (default) and OpenAI with llm_provider. It exists because of three real failure modes:

1. Groq's replacement models (openai/gpt-oss-20b / -120b) are reasoning models; some prompts burn the whole token budget thinking and return an empty reply. An empty reply is retried with a doubled budget.
2. Per-minute rate limits are retried after the wait the API asks for; daily limits and exhausted billing fail fast instead of being retried.
3. Citation markers like 【1】 are normalised to [1].

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗