Beacon Research
SEC 10-K RAG assistant with section-aware chunking, inline citations, and per-answer LLM-as-judge scoring — quality is measurable, not just plausible.
Real recorded session — a research question answered with inline citations and a confidence badge, then a follow-up question in the same thread
Answer with inline citations [1][2] and LLM-as-judge confidence badge (groundedness + relevance scores)
Corpus indicator — five 10-K filings loaded across Technology, Banking, Healthcare, Retail, and Energy
Follow-up question stays in the same conversation thread — multi-turn history persisted server-side
Summary
A RAG chat assistant over five real SEC 10-K filings (Apple, JPMorgan Chase, Johnson & Johnson, Walmart, Exxon Mobil). Every answer carries inline citations and a live confidence badge computed by two independent LLM-as-judge evaluators — so answer quality is displayed alongside the answer, not assumed.
Live demo: beacon-research.onrender.com — ask it a question about any of the five filings. It runs on a free tier that sleeps when idle, so the first request after a quiet spell takes 30–60 seconds to wake the instance; the UI says so while you wait rather than looking broken.
The Problem
Financial document Q&A demos are easy to make look good in cherry-picked screenshots. The harder problem: how do you know when the model is hallucinating versus when it’s genuinely grounded? Naive chunking produces vague citations like “page 42.” Fixed-size splits don’t preserve the section structure that makes 10-Ks intelligible. And without a quality signal, the user has no way to distinguish a well-grounded answer from a plausible-sounding fabrication.
Approach
Section-aware chunking: Rather than naive fixed-size splits, the ingestion pipeline splits on real 10-K Item headings (Business, Risk Factors, MD&A, Financial Statements). Every chunk carries its section metadata — so citations read “Apple 10-K, Item 7 MD&A (FY2025)” rather than an anonymous page reference.
Vector retrieval: OpenAI text-embedding-3-small embeddings stored in ChromaDB (file-based, no external infrastructure). The top-K retrieved chunks are numbered and injected into the generation context with instructions to cite by number.
Dual LLM-as-judge scoring: Two separate judge calls per answer — groundedness (is the answer actually supported by the retrieved context?) and relevance (does it address the question?). Both use a 1–5 rubric at temperature=0 for consistency. The average determines the confidence badge shown in the UI (high / medium / low). The same judges serve as a pytest quality gate: the eval harness runs a fixed 11-question set across all five companies and enforces a minimum average on both dimensions, plus a per-question floor and a check that retrieval didn’t wander to the wrong company. It is a deliberate pre-release run, not a CI step — every run makes real API calls against real filings and costs real money, so it is excluded from the default suite and run by hand before shipping a change.
A limitation that turned out to be a misdiagnosis: this write-up used to say that JPMorgan and Exxon “format their Item 7 and 8 headings differently,” and that the splitter recovered only Items 1 and 1A for them. That explanation was wrong, and the real cause is the more interesting result.
The regex matched fine. Both companies incorporate Items 7 and 8 by reference: the Item 7. heading is genuinely present in the document, but the text immediately after it is a short cross-reference paragraph — “…appears on pages 46–160…” — pointing at the real content elsewhere, under a heading that never says “Item 7” at all. The splitter’s own minimum-length guard, there to discard table-of-contents mentions, was throwing that short body away. The section was being found and then discarded, not missed. Believing the first explanation meant the fix would have been aimed at a regex that was never broken.
The fix falls back to searching the whole filing for the item’s plain-English heading (“Management’s Discussion and Analysis” for Item 7, “Report of Independent Registered Public Accounting Firm” for Item 8) whenever the literal heading yields too short a body, and takes the longest resulting match. Apple, Johnson & Johnson and Walmart state their items directly, so the fallback never runs for them. JPMorgan went from 70 indexed chunks to 639, Exxon from 12 to 195, and the corpus from 468 chunks to 1,220 — the MD&A and financial statements for two of the five companies had simply been absent from the index before.
Key Results
- Eval harness average: 4.73/5 groundedness, 4.64/5 relevance on a fixed 11-question set spanning all five companies, one of which is a multi-turn follow-up (threshold: 3.5; per-question floor: 2)
- 77 backend tests covering the full pipeline: EDGAR ingestion, section splitting, chunking, retrieval, query contextualization, generation, citation deduplication, judge score parsing, rate limiting, and every API endpoint
- 1,220 chunks indexed across five companies and five sectors — Tech, Banking, Healthcare, Retail, Energy
- Playwright E2E verification: the corpus indicator loads, an answer carries citations and a confidence badge, a follow-up stays in the same thread, and a session held open across a redeploy recovers instead of dead-ending
- Deployed and publicly reachable, with per-visitor rate limiting and a spend cap on the API key behind it