Retrieval Over Two Centuries of Federal Financial Statements: OfficeQA Pro V2

Introduction

A question an analyst might ask an agent looks like this: between Massachusetts and Virginia, which state's customs collectors had the more unequal payments into the Treasury in 1797, and what was the Gini coefficient? The arithmetic is easy for a modern model. The hard part comes first. The answer lives in one table on one page of the Treasury's account for 1797, and the corpus holds an account for nearly every year from 1793 onward, each laid out in much the same way.

That is the retrieval problem in financial document work. The documents are long, tabular and repetitive. A question names a year, an office and a line item, and only one of many look-alike tables carries the right digits. Retrieve the 1798 account instead of the 1797 one and the model will compute a confident, well-formatted, wrong answer.

Most retrieval benchmarks are built on clean prose: Wikipedia articles, web pages, papers. OfficeQA Pro V2, released by Databricks as the harder companion to OfficeQA Pro, is built on two centuries of U.S. government financial publications parsed from the original PDFs [1, 2]. Its questions are written so that an agent has to find the right figures and then reason over them. The best agent harness on its published results reaches 54.4% answer accuracy [2]. This report isolates the step those agents depend on: putting the right document in front of the model.

We ran HydraDB over both the documents the questions cite and the full corpus, and scored retrieval against the benchmark's provenance labels. This report explains what the benchmark measures, how HydraDB was configured, and where the result comes from.

What OfficeQA Pro V2 Measures

The corpus combines two series of U.S. government financial publications: the Combined Statements of Receipts, Outlays, and Balances of the U.S. Government, and the earlier Congressional serial-set receipts documents [2].

Three properties make the corpus a demanding retrieval target. The same report structure repeats year after year, so a table from one year is a close match for the same table from any other. The content is overwhelmingly numeric, so the words that would distinguish one document from another are few: a year in a heading, a column label, a footnote. And the older documents carry the typography and page layout of their era, which survives parsing as irregular tables and broken lines.

Questions cite many documents

There are 90 questions. Each carries a gold answer and the list of documents and page numbers the answer draws on. Most questions require combining figures across several documents, for example a series over a run of years:

The median question cites 5.5 documents and the most demanding cites 24 [2]. The questions ask for computed results, such as a Gini coefficient across collectors or an exponential moving average over a decade of receipts, so an answer is only correct if every figure feeding it was retrieved and read correctly.

Scoring retrieval from provenance

OfficeQA scores answers by matching the final value against the gold answer, with no numeric tolerance by default [2]. It labels provenance at the level of documents and pages rather than golden passages, so we measure retrieval at the document level:

• Recall@k. The share of questions for which at least one chunk from a cited document appears among the top k results.
• MRR@10. The mean, over questions, of the reciprocal rank of the first chunk from a cited document within the top ten.

Ranks are counted over chunks, so several chunks from one document occupy several ranks. Because most questions cite several documents, recall@k here measures whether a question's evidence is reached, not whether every cited document is returned. We report both the cited set and the full corpus. The cited set isolates retrieval among the 249 documents that matter; the full corpus adds the 1,186 documents no question cites, which act as distractors drawn from every other year.

Retrieval Is the Hard Part of Financial Document QA

Vector similarity finds text that resembles the question. On this corpus that gets a system into the right neighbourhood and no further. Every receipts account resembles every other receipts account, and an embedding of a table dominated by digits says little about which year it covers. Keyword matching helps with exact terms such as a state or an office, but those terms appear in every year's edition too. The signal that separates the right document is small and structural: the year it reports, the period a table covers, how a line item relates to the totals around it.

HydraDB approaches this as a context problem rather than a similarity problem. Each document is ingested as knowledge with its tables preserved, and at ingest HydraDB extracts the entities and relationships inside it into a knowledge graph, so that a figure is connected to the office, the item and the reporting period it belongs to [3]. At query time retrieval is hybrid, combining keyword and embedding search, in HydraDB's thinking mode, which expands the query and adds a reranking pass. Graph paths contribute their own candidates, and the streams are fused into a single ranked list.

Because we evaluated the complete configuration rather than each signal separately, we attribute the result to the retrieval stack as a whole. We do not report a per-component ablation here.

Evaluation Setup

• Corpus. Two runs, each in its own database: the 249 documents cited by the questions, and all 1,435 documents.
• Ingestion. Each document was rendered from the benchmark's parsed JSON to Markdown with the benchmark's own transform script, keeping tables intact, and ingested as one knowledge source keyed by its file name. All 1,435 documents were indexed.
• Retrieval. Hybrid search, thinking mode, graph context on, top ten chunks per question. The same settings were used for both corpora, with no per-question tuning.
• Scoring. Document recall@k at 1, 3, 5 and 10, and MRR@10, against each question's cited source files, over all 90 questions.

Results

Retrieval depth

The ranking is front-loaded. On the cited set, 61 of the 73 questions that reach their evidence within ten results reach it at rank one. That matters for an agent: the strongest evidence arrives first, so less context has to be passed to the model to include it.

Adding the 1,186 distractors lowers the curve by nine to ten points at every cutoff without changing its shape. Question by question, of the 61 questions answered at rank one on the cited set, 43 are still answered at rank one on the full corpus, 9 move down within the top ten and 9 drop out of it. Overall, 12 questions reached on the cited set are missed on the full corpus, and 4 missed on the cited set are reached on it.

By source

Questions over the Combined Statements reach their evidence in the top ten for 47 of 52 on the cited set. Questions over the older serial-set receipts documents are harder, at 25 of 35.

By era

Grouping questions by the earliest year they cite shows where the difficulty sits. Statements from 1950 onward are retrieved almost every time, 27 of 28 and 8 of 9 on the cited set. The receipts accounts of 1790 to 1849, the oldest documents in the corpus, are the hardest at 18 of 26, falling to 13 of 26 on the full corpus. The smaller eras hold 9 to 14 questions, so a single question moves them by several points; on 1850 to 1899 the full corpus run scores one question higher than the cited run.

Where the gap remains

The earliest documents are the main weakness. Their tables are the least regular in the corpus and the hardest to tell apart from neighbouring years, and they account for half of the 12 questions that the cited set reaches and the full corpus misses.

The second gap is coverage. Recall@10 counts a question as reached when any of its cited documents is in the top ten, while a question citing a median of 5.5 documents needs all of them to be answered. Bringing every cited document into a short result list, for questions that span many years, is the harder target, and it is where end-to-end answering on this benchmark is decided. These two areas are the current focus of work.

Why This Matters for Financial AI Agents

A capable model can compute a Gini coefficient or a moving average from the right table. If retrieval hands it the account for the wrong year, it computes the right formula over the wrong figures and produces an answer that looks correct. On a corpus of look-alike documents the deciding question is not whether a system finds similar text, but whether it finds the right document.

HydraDB placed a cited document first for 67.8% of questions and within the top ten for 81.1% on the cited set, and held 72.2% recall@10 after the corpus grew almost six-fold with distractors from every other year. Because the ranking is front-loaded, an agent can pass a short context and still include the strongest evidence.

The benchmark does not cover every requirement of a production financial system. It is a static archive with a single user, so it does not test permissions or how quickly new filings become retrievable, and its official score measures the complete agent rather than retrieval alone.

Conclusion

On OfficeQA Pro V2, HydraDB reaches 81.1% recall@10 and 0.729 MRR@10 on the documents the questions cite, and 72.2% recall@10 with 0.630 MRR@10 across all 1,435 documents spanning 1793 to 2024. Most of that recall arrives at the first result. The oldest receipts accounts and the multi-document questions, where every cited year must land in a short list, are where the work is. The result is self-reported and measures retrieval only; OfficeQA's published results score end-to-end answers.

References

1. OfficeQA: A Grounded Reasoning Benchmark Suite. Databricks (2026). arXiv:2603.08655.
2. OfficeQA benchmark suite and OfficeQA Pro V2. Databricks, GitHub (2026). Dataset: databricks/officeqa-pro-v2, Hugging Face.
3. Beyond Flat Embeddings for Production AI Agents. HydraDB Research Team (2026).