AI agents retrieve from knowledge where the wrong answer is often the most similar one: an outdated version of the right document, another customer's near-identical record, an injected note. We evaluate HydraDB on KyroBench, a context-correctness benchmark built by KyroDB Research in which every query is surrounded by stale, cross-tenant and poisoned look-alikes. On the public split of 3,634 documents and 864 cases, HydraDB reaches 0.917 recall@k and 0.776 nDCG@k, ahead of every baseline we ran. Recall is close to the strongest baseline; ranking is not: HydraDB's nDCG is 0.338 higher and its MRR 0.531 higher.
| Recall@k | nDCG@k | MRR | Cases with gold retrieved |
|---|---|---|---|
| 0.917 | 0.776 | 0.784 | 857 of 864 |
Self-reported. Scored with a reimplementation of KyroBench's recall and nDCG that matches its own scorer to three decimals on identical inputs. Baselines are KyroBench's open-source controls, run by us on the same corpus.
Introduction
Retrieval benchmarks usually reward a system for finding text that resembles the question. In production that is often the failure. A support agent asked about a customer's billing plan will find last quarter's plan as easily as this quarter's. A code agent asked how a service is deployed will find the runbook that was superseded last month. In a multi-tenant product, the most similar record may belong to a different customer. The model downstream cannot tell which of these is right; it uses what it is given.
KyroBench is built around that failure. Every query sits in a neighbourhood of documents that are similar to it by design, and only one is correct: the current, in-scope, authoritative one [1]. A system that ranks by similarity alone retrieves the trap as readily as the answer.
The benchmark was built by KyroDB Research, the team behind a competing system, so its traps are not tuned to HydraDB. Its published leaderboard reports a composite score produced by its authors, on which HydraDB is not listed. Rather than compare against that composite, we ran KyroBench's own open-source baseline controls ourselves, on the same corpus, with the same scorer. This report explains what the benchmark measures, how HydraDB was configured, and where the result comes from.
What KyroBench Measures
KyroBench is generated deterministically by an open-source tool, so anyone can rebuild the exact corpus used here [1]. It ships at several scales; we use the standard scale. Roughly half of the generated cases are released as the public split, with labels; the rest are held out by the authors.
| Property | Standard public split |
|---|---|
| Active documents | 3,634 |
| Scenarios | 288 |
| Retrieval cases | 864 |
| Domains | 8 |
| License | Apache-2.0 |
The eight domains cover support escalation, billing, public policy, code agents, legal contracts, CRM memory, healthcare and SRE runbooks. Each scenario places a small set of active documents under one tenant and namespace: the current gold document, a distractor and a poisoned item, alongside labelled traps.
Four kinds of trap
| Trap | What it is |
|---|---|
| Stale version | A superseded copy of the gold document, nearly identical |
| Cross-tenant look-alike | The same content, owned by another tenant |
| Injected note | Text written to match the query |
| Poisoned tool result | A misleading tool output that reads like context |
Every trap shares most of its text with the query or with the gold document, so a ranking built on similarity alone cannot separate it from the answer.
Each case lists its gold documents with a graded relevance, and separately lists the stale, forbidden, polluting and deleted items, so a retrieval run can be checked for what it should and should not return.
Scoring rewards the right document at the top
KyroBench scores each case within its scope: the tenant, namespace and entitlement boundary the request belongs to. A real application always knows which customer or user a request is for, and never searches every tenant's data at once. We report:
- Recall@k. The fraction of a case's gold documents returned in the top k, averaged over cases.
- nDCG@k. Ranking quality using KyroBench's graded gains, so placing the gold document first scores more than placing it fifth.
- MRR. The mean reciprocal rank of the first gold document.
- Full recall. The share of cases in which every gold document is returned.
Our scorer reproduces KyroBench's own recall and nDCG to three decimals on identical retrievals. KyroBench also reports a precision figure whose denominator differs from the standard definition; we do not report it.
Ranking Is the Hard Part of Context Correctness
On KyroBench, finding a gold document somewhere in the result list is not difficult. A system can return a large window and the gold will usually be inside it. The difficulty is ranking: putting the current, authoritative document above the stale copy, the cross-tenant twin and the injected note that all resemble it. An agent that receives the trap first will use the trap.
HydraDB treats this as a context problem rather than a similarity problem. Documents are ingested as app sources carrying their tenant, namespace and entitlement as metadata, so scope is enforced by a filter rather than left to similarity. Within scope, retrieval is hybrid, scoring each candidate on both embedding similarity and keyword match, which separates documents that share their meaning but differ in identifiers, versions or terms. Because we evaluated the complete configuration rather than each component separately, we attribute the result to the retrieval stack as a whole.
Evaluation Setup
- Corpus. The standard public split: 3,634 active documents across 288 scenarios, in one HydraDB database.
- Ingestion. Every document ingested as an app source through the context API, keeping its KyroBench id as the source id so returned results could be scored directly.
- Retrieval. Hybrid search in fast mode, top eight results per case, graph context disabled. Each query is scoped to its case's tenant, namespace and entitlement with a metadata filter, the same scope KyroBench uses to score its own systems. No per-domain tuning.
- Baselines. KyroBench's open-source controls, long_context, qdrant_raw, pgvector_raw and naive_rag, run by us on the same corpus and scored with the same scorer. A reference oracle scores 1.0 on recall and nDCG.
Results
Against the baselines
| System | Recall@k | nDCG@k |
|---|---|---|
| HydraDB | 0.917 | 0.776 |
| long_context | 0.914 | 0.438 |
| qdrant_raw | 0.717 | 0.366 |
| pgvector_raw | 0.649 | 0.316 |
| naive_rag | 0.434 | 0.217 |
long_context matches HydraDB on recall because it returns a large window and lets the gold document fall somewhere inside it. Its nDCG of 0.438 and MRR of 0.253 show where the gold lands: rarely near the top. HydraDB finds the gold as often and places it near the top, with an MRR of 0.784. The raw vector stores trail on both recall and ranking, because KyroBench's traps are built to be what pure similarity cannot tell apart.
Every number in this section was produced by us: HydraDB through its public API, the baselines through KyroBench's own controls. Nothing is copied from a leaderboard.
By domain
| Domain | Recall@k | nDCG@k |
|---|---|---|
| support_escalation | 0.979 | 0.784 |
| support_billing | 0.952 | 0.796 |
| public_policy | 0.942 | 0.750 |
| code_agent | 0.921 | 0.773 |
| crm_memory | 0.894 | 0.748 |
| legal_contract | 0.883 | 0.771 |
| healthcare | 0.881 | 0.789 |
| sre_runbook | 0.877 | 0.797 |
Recall stays between 0.877 and 0.979 in every domain, and nDCG between 0.748 and 0.797, so no small set of easy domains carries the average. CRM memory and public policy have the lowest nDCG, where the gold document is most often retrieved but not ranked first.
Scope is part of the test
Run the same queries without the scope filter, against every tenant's documents at once, and recall falls to 0.185.
This is the cross-tenant trap working as designed. Unscoped, near-identical documents from the 287 other scenarios outrank the correct one, and only 246 of 864 cases retrieve any gold document. The scoped figure is the one KyroBench defines, and it matches how applications run: a request is always made on behalf of one tenant or user. We report both so the effect of scope is visible rather than hidden in the headline.
Where the gap remains
Seven of the 864 scoped cases miss every gold document, where an in-scope near-neighbour scores closer to the query than the correct document. Full recall is 0.806: in about one case in five, at least one gold document is missing from the top eight. And an nDCG of 0.776 means the gold document is retrieved but not always placed first, most often in CRM memory and public policy. Ranking within scope is the current focus of work.
Why This Matters for AI Agents
Teams building agents do not get a clean corpus. They get versioned, contradictory, multi-tenant knowledge in which the wrong answer is the one that looks most similar. A system that returns everything similar hands the model the trap alongside the answer and leaves it to guess. KyroBench measures whether the right document comes first.
HydraDB retrieved a gold document in 857 of 864 cases and ranked it with an nDCG of 0.776, against 0.438 for the best baseline. That gap is the difference between context that looks right and context that is right, and in an agent it compounds on every turn.
The benchmark does not cover everything a production system needs. It measures retrieval rather than answer quality, and it is a static snapshot, so it does not test how quickly an update or a deletion takes effect in what an agent retrieves.
Conclusion
On KyroBench's standard public split, HydraDB reaches 0.917 recall@k, 0.776 nDCG@k and 0.784 MRR over 864 cases, ahead of every baseline control on both recall and ranking, and above 0.87 recall in all eight domains. The margin is largest where the benchmark is hardest: ranking the current, in-scope document above the look-alikes built to displace it. The result is self-reported, produced with KyroBench's open corpus and controls, and measures retrieval only.
References
- KyroBench. KyroDB Research, GitHub (2026). Apache-2.0.
- Beyond Flat Embeddings for Production AI Agents. HydraDB Research Team (2026).