Beacon Research
SEC 10-K RAG assistant with section-aware chunking, inline citations, and per-answer LLM-as-judge scoring — quality is measurable, not just plausible.
Real recorded session — a research question answered with inline citations and a confidence badge, then a follow-up question in the same thread
Answer with inline citations [1][2] and LLM-as-judge confidence badge (groundedness + relevance scores)
Corpus indicator — five 10-K filings loaded across Technology, Banking, Healthcare, Retail, and Energy
Follow-up question stays in the same conversation thread — multi-turn history persisted server-side
Summary
A RAG chat assistant over five real SEC 10-K filings (Apple, JPMorgan Chase, Johnson & Johnson, Walmart, Exxon Mobil). Every answer carries inline citations and a live confidence badge computed by two independent LLM-as-judge evaluators — so answer quality is displayed alongside the answer, not assumed.
The Problem
Financial document Q&A demos are easy to make look good in cherry-picked screenshots. The harder problem: how do you know when the model is hallucinating versus when it’s genuinely grounded? Naive chunking produces vague citations like “page 42.” Fixed-size splits don’t preserve the section structure that makes 10-Ks intelligible. And without a quality signal, the user has no way to distinguish a well-grounded answer from a plausible-sounding fabrication.
Approach
Section-aware chunking: Rather than naive fixed-size splits, the ingestion pipeline splits on real 10-K Item headings (Business, Risk Factors, MD&A, Financial Statements). Every chunk carries its section metadata — so citations read “Apple 10-K, Item 7 MD&A (FY2025)” rather than an anonymous page reference.
Vector retrieval: OpenAI text-embedding-3-small embeddings stored in ChromaDB (file-based, no external infrastructure). The top-K retrieved chunks are numbered and injected into the generation context with instructions to cite by number.
Dual LLM-as-judge scoring: Two separate judge calls per answer — groundedness (is the answer actually supported by the retrieved context?) and relevance (does it address the question?). Both use a 1–5 rubric at temperature=0 for consistency. The average determines the confidence badge shown in the UI (high / medium / low). The same judges serve as an offline pytest quality gate: the eval harness runs a fixed 10-question set across all five companies and enforces a minimum threshold on every push.
Known limitation: JPMorgan and Exxon format their Item 7 and 8 headings differently than the other three filings. The section splitter only recovers Items 1 and 1A for those two (70 and 12 chunks respectively vs. 84–187 for the others). This is documented, not hidden.
Key Results
- Eval harness average: 4.80/5 groundedness, 4.40/5 relevance on a fixed 10-question set spanning all five companies (threshold: 3.5; per-question floor: 2)
- 44 backend tests covering the full pipeline: EDGAR ingestion, chunking, retrieval, generation, citation deduplication, judge score parsing, and all API endpoints
- Playwright E2E verification: corpus indicator loads, answer includes citations and confidence badge, follow-up question stays in the same conversation thread
- 468 chunks indexed across five companies, five sectors — Tech, Banking, Healthcare, Retail, Energy