Retrieval Over Company-Internal Knowledge for AI Agents: EnterpriseRAG-Bench

Introduction

Enterprise AI agents are only as useful as the context they can retrieve. But the knowledge inside a company rarely lives in one clean, searchable place. It is scattered across Slack threads, email chains, support tickets, meeting transcripts, CRM records, engineering discussions and internal documents. The same fact may appear in several systems, newer information may contradict older information, and important decisions are often buried inside conversations written with company-specific acronyms and codenames.

This makes enterprise retrieval fundamentally harder than searching a clean knowledge base. Given a question, an AI agent must not only find semantically similar text. It has to identify the right source, distinguish current information from outdated information, follow context across systems and retrieve enough evidence to answer correctly.

Most Retrieval benchmarks have not tested the retrieval of enterprise internal knowledge. They are built on Wikipedia, web pages, Stack Overflow and academic papers: clean, consistent and public. EnterpriseRAG-Bench, released this year by Onyx, is, by its authors’ account, the first public benchmark built on company-internal data: 511,958 documents across nine enterprise systems, deliberately misfiled, duplicated and contradicted, with 500 questions an employee might actually ask [1, 4].

We ran HydraDB through the full benchmark and scored it with the benchmark’s own evaluator. HydraDB scores 88.7 overall, against 80.3 for the highest system on the public leaderboard, and leads on answer correctness, answer completeness and document recall. This report explains what the benchmark measures, how HydraDB was configured, and where the result comes from.

What EnterpriseRAG-Bench Measures

A company’s internal knowledge data is the hardest to get, no real company will publish its internal documents and each employee conversations. EnterpriseRAG-Bench is evaluation over an synthetic dataset. The dataset simulates Redwood Inference, a company selling AI model inference as a service. what it reproduces is the shape of enterprise knowledge, not any one company’s facts. Its nine sources are in roughly the proportions a real company would have [1]:

Chat and email make up roughly four fifths of the corpus, creating a large conversational search space. Polished documentation accounts for only 1 %.

Onyx generated the corpus from a scaffold of the company, with its initiatives, an employee directory and the structure of each source, so that documents reference each other the way real ones do: a pull request points at the requirements document it implements. They then added noise on purpose [2]:

• 8 % of documents were moved to the wrong location, some at random and some to a plausible but wrong folder chosen by a model, the way a person misfiles things.
• Near-duplicates were added with key facts changed, sometimes in a different system from the original. Their own example is pricing that changes after a follow-up sales conversation, so the old figure and the new one both exist.
• Informal material was added: memes channels, hackathon repositories, draft notes in badly organised folders.
• Every document uses internal terminology: project codenames, product acronyms and team jargon with no meaning outside the company.

Ten kinds of question

There are 500 questions in ten categories. Each carries a reference (“gold”) answer, a list of atomic facts a complete answer should contain, and, for 470 of the 500, the gold documents the answer lives in [1].

The last two categories catch systems that always answer: “info not found” fails a system that invents an answer, and “high level” fails one that can only quote a single document.

Scoring gives no credit for nearly right

Onyx publishes the evaluator. Its judge defaults to GPT-5.4, and we did not change it. For each question it produces four numbers [2]:

• Correctness. A yes-or-no judgment of whether the answer agrees with the gold answer. The judge is lenient about style and extra detail and strict about substance: the core of the question must be addressed, nothing may contradict the gold answer, and any quantity mentioned in both must match.
• Completeness. The percentage of the gold answer facts the submitted answer states or implies, each fact checked by a separate judge call.
• Document recall. The percentage of gold documents that appear in the documents the system returned.
• Invalid extra documents. How many returned documents were neither gold nor judged relevant.

The leaderboard ranks on one combined score: the average over all 500 questions of the completeness percentage when the answer is judged correct, and zero when it is not. An answer that states nine facts out of ten but contradicts the gold answer on the tenth scores zero for that question.

Two further details of the protocol matter. Citations are stripped from answers before judging, so a system cannot earn credit by pointing at documents. And because a gold document set assembled when the benchmark was built is not guaranteed to be perfect, the evaluator runs a correction step: three independent judges classify the gold documents and the documents each system returned as required, valid or invalid, and update the gold set by majority vote when the evidence warrants it. When the set changes, the gold answer and its facts are regenerated before scoring. In our run this happened on 14 of the 500 questions. Scored strictly, with neither the correction step nor citation stripping, the same answers score 88.3.

Onyx sells a RAG product and excludes its own system from the leaderboard. Every listed system was verified by Onyx before listing [1].

Retrieval Is the Hard Part of Enterprise RAG

Vector similarity finds text that resembles the question. On a corpus like this one that is not enough. The right answer is often the third message in a thread that corrects the first. The document that matches the question best may be the misfiled duplicate with the old price. A question asked in plain English has to land on a document written in codenames. And a question about a project has to pull together a ticket, a design document, a pull request and a Slack decision that share no vocabulary.

HydraDB approaches this as a context problem rather than a similarity problem. HydraDB provides ingestion pipeline that loads Slack Jira, Linear and GitHub as tickets with their comments, Confluence and Drive as documents, HubSpot as CRM records. HydraDB’s inference step ran at ingest, extracting the people, thread relationships and cross-references from each item, so that a message knows what it replies to and a ticket knows who owns it [5]. Everything went into one collection with no per-source tuning.

At query time, retrieval is hybrid (keyword plus embedding) in HydraDB’s thinking mode, which adds a reranking pass over the candidate set. The top twelve documents are passed in full to the answer model; the top ten are submitted for scoring.

The combined setup performed strongly on conflicting-information, constrained and semantic questions. Passing whole documents may help preserve correcting messages in threads and qualifying clauses in tickets, while reranking may help distinguish the right document when several candidates look similar. Because we evaluated the complete setup rather than each component separately, these explanations remain hypotheses rather than measured component effects.

Evaluation Setup

• Corpus. All 511,958 documents, one HydraDB collection.
• Retrieval. Hybrid search, thinking mode, top twelve documents returned per question, top ten submitted. The benchmark does not fix how many documents a system may submit, so recall figures across systems depend partly on that choice. Ten was chosen because most questions have one or two gold documents and ten leaves room for the multi-document categories.
• Answering. GPT-5.4, given the full text of the twelve retrieved documents, with a self-check pass before the final answer. The judge is also GPT-5.4, the evaluator’s default, so correctness and completeness measure HydraDB’s retrieval plus GPT-5.4’s writing from it. Document recall is HydraDB’s alone.
• Scoring. The benchmark’s evaluator, full protocol, all 500 questions. Nothing was tuned per category or per question.

Results

Against the leaderboard

EnterpriseRAG-Bench, public leaderboard as of 28 August 2026, 22 systems [3]. Positive Δ is a HydraDB advantage except on invalid extra documents.

Reproduce this run: the reproduction package includes the artifacts, REPRODUCE.md, and METHODOLOGY.md. The repository is currently private and is planned for release alongside this report.

On invalid extra documents HydraDB is mid-pack: it returns ten documents for every question, and most questions have one or two gold documents, so most returned documents count as extra by construction. Systems that return fewer documents do better on that column, which the leaderboard does not rank on.

By category

Four categories were answered with 100 % correctness: intra-document reasoning, conflicting information, miscellaneous and info not found. Conflicting information and constrained questions are the ones built to punish a system that retrieves the right neighbourhood and then picks the wrong document; HydraDB answered all 20 of the former and 28 of 30 of the latter, and on all 20 info-not-found questions it said the answer was not there.

Retrieval depth

Across the 470 questions with gold documents, document recall (the share of gold documents found) was 81 % within the first result, 92 % within three and 97 % within ten. On 441 of the 470, every gold document was inside the top ten. This is the number that is HydraDB’s alone, and it is 10 points above the best document recall on the leaderboard.

By source

For questions whose answer lives in a single system:

Confluence, the polished 1 % of the corpus, was the weakest single source. We had expected Slack to be hardest; it was second-weakest at 50 of 57.

Where the gap remains

Project-related and completeness questions remain the hardest categories. Both need several documents, often across systems, and on them document recall fell to 82.5 % and 76.2 % against 98 to 100 % everywhere else: some of the required documents are ranked below tenth position. Their category-average completeness scores were 77.9 % and 71.2 %, respectively. These two categories, and Confluence, are the current focus of work.

Why This Matters for Enterprise AI Agents

A model may be capable of reasoning over a question, but if the relevant decision is buried in an old Slack thread, updated in a ticket, or contradicted by a newer document, the agent can still produce the wrong answer. This is what EnterpriseRAG-Bench tests: whether a system can retrieve the right context, not merely similar information.

HydraDB placed every gold document in the top ten results for 94% of the 470 questions with ground-truth documents. Using an off-the-shelf answer model over that retrieved context, it answered 92% of all questions correctly while capturing 92.5% of the required facts on average.

The benchmark does not cover every requirement of a production enterprise system. It models a single company and user, so it does not evaluate permissions or access control. It is also a static snapshot, so it does not measure how quickly new or updated information becomes available for retrieval.

Conclusion

HydraDB scores 88.7 on EnterpriseRAG-Bench, 8.4 points above the highest system on the public leaderboard, with 92.0 % answer correctness, 92.5% answer completeness and 96.7% document recall across 500 questions over half a million company-internal documents. The number that is HydraDB’s alone is document recall, 96.7% against a best listed 86.6%; the categories that punish picking the wrong document from the right neighbourhood, constrained and conflicting information, were answered at 28 of 30 and 20 of 20. Project-related and completeness questions, where several documents, often across systems, must all land in the top ten, are where the work is. The result is self-reported and has not yet been verified or listed by the benchmark authors. The benchmark, dataset and evaluator are open, and a run-specific reproduction package is prepared for release alongside this report.

References

1. EnterpriseRAG-Bench: A RAG Benchmark for Company Internal Knowledge. Sun, Y., Rahmfeld, J., Weaver, C., Desai, R., Huang, W., Butler, M. H. — Onyx (2026). arXiv:2605.05253.
2. EnterpriseRAG-Bench: data generation methodology and evaluation. Onyx (2026).
3. EnterpriseRAG-Bench Leaderboard. Onyx — Hugging Face Space, accessed 28 August 2026.
4. EnterpriseRAG-Bench dataset. Onyx — Hugging Face (2026).
5. Beyond Flat Embeddings for Production AI Agents. HydraDB Research Team (2026).
6. HydraDB on EnterpriseRAG-Bench: reproduction package. HydraDB Research Team (2026).