Retrieving the Right Memory Across Four Years of Personal Context: ATM-Bench

Introduction

A question a personal agent has to answer looks like this: "Today is May 2, 2024. I had a formal in Peterhouse last week, find the photos." The model can reason about dates and places easily. The hard part comes first: the right two photos live among 4,292 other photos and videos of the same person's life, many of them also of Cambridge colleges, also in spring, also at dusk. Return a formal from a different term and the agent answers confidently about the wrong evening.

This is the retrieval problem in personal memory. A life is repetitive. The same people, places and routines recur for years, so any one memory has many near-twins, and the detail that separates the right one is small: the exact date, the specific location, a name on a sign, a line in an email. ATM-Bench, released with the paper According to Me: Long-Term Personalized Referential Memory QA [1], is built to test exactly this, multimodal, multi-source, referential recall over a privacy-preserving archive of one person's photos, videos and emails spanning roughly four years and several countries.

Most memory benchmarks test short conversational recall. ATM-Bench tests retrieval over a large, heterogeneous, look-alike corpus, then grounds every question in the specific items its answer draws on. This report isolates the step the answer depends on: putting the right memory in front of the model. We ran HydraDB over the full corpus and scored retrieval against the benchmark's evidence labels with its own code.

What ATM-Bench Measures

The corpus is one person's real, de-identified memory over roughly four years: email, photos and video, each item carrying its original timestamp and, for media, GPS-derived location [1].

Images and videos are represented by the dataset's own precomputed captions, OCR and tags, the "schema-guided memory" (SGM) setting that the standard memory-system rows on the leaderboard use, so the comparison holds the memory representation fixed and varies only the retrieval system.

Three properties make it a demanding retrieval target. The content is repetitive, a life produces many near-identical scenes, so one photo of a park is a close match for dozens of others. It is multimodal and lopsided, 61% of the items are text-dense emails, which dominate a naive index and bury the shorter media captions. And the questions are referential and temporal, they point at a memory by when and where it happened, so the signal that resolves them is often a date or place, not a topic.

Questions name specific memories

There are 1,013 questions: 514 open-ended, 360 numeric, and 139 list-recall. Each carries a gold answer and the evidence_ids of the memory items it draws on. About 75% of questions have a single gold item; the most demanding cite six. By modality, 632 questions are answered from media alone and 381 from email alone. A separate 31-question hard split stresses multi-evidence reasoning, with up to six items per question.

Scoring retrieval from evidence ids

ATM-Bench labels provenance at the item level, so retrieval is scored directly against each question's evidence_ids using the benchmark's own definitions [2]:

• Recall@k. The share of questions with at least one gold item in the top-k distinct items, the leaderboard column.
• Recall@kGT. The stricter share where all gold items are in the top-k.
• MRR. The reciprocal rank of the first gold item.

Because the item id equals the evidence id, a retrieved chunk maps back to gold with no lookup table. We report retrieval over the full corpus; there is no scoped or per-question subset.

Ranking the Right Memory Among Look-Alikes

Vector similarity finds memories that resemble the question. On a personal archive that is not enough: every walk in the park resembles every other walk, and an embedding of a short caption says little about which April it was taken. The signal that separates the right memory is structural and temporal, the exact timestamp, the place, the entities named in a caption or an email.

HydraDB treats this as a context problem. Each memory is ingested with its full metadata, its timestamp and, for media, its location, and HydraDB builds a connected, time-aware representation that ties a memory to the people, places and periods it belongs to [3]. At query time it combines complementary retrieval signals and returns a single ranked list of memories.

Because we evaluated the complete configuration rather than each signal separately, we attribute the result to HydraDB's retrieval as a whole and do not report a per-component ablation.

Evaluation Setup

• Corpus. All 11,034 memory items, in a single HydraDB database.
• Ingestion. Each item rendered to text exactly as ATM-Bench's own retriever renders it, id, type, timestamp, location, short caption, caption, OCR and tags for media; id, timestamp, summary and body for email, and ingested into HydraDB with that metadata. The item id is the evidence id. All 11,034 items indexed, 0 failed batches.
• Retrieval. HydraDB's retrieval API, up to 50 candidates per query deduplicated to distinct evidence items. We report the default production configuration (primary) and a lighter-weight configuration.
• Scoring. Recall@k at 1, 5, 10, 20, Recall@kGT and MRR, against each question's evidence_ids, with ATM-Bench's own scorer, over all 1,013 questions.

Results

Retrieval depth

Recall@10 is 0.918 and climbs to 0.940 at top 20, so the correct memory is almost always in the shortlist; all-evidence recall@10, which requires every gold item for a question, is 0.854. On a corpus that is 61% text-dense email, HydraDB keeps the media questions from being crowded out by look-alike emails. The lighter-weight configuration reaches 0.875 recall@10.

Against the leaderboard

Measured against the public leaderboard's memory and RAG rows, all scored on the same 1,013 questions [4]:

HydraDB is second on the full set, two thousandths behind Memexa v2, which spent a stronger model re-writing the media captions, and ahead of RE-call on the same 2B captions. On the hard split it leads every system that reports one, at 0.774 recall@10.

By modality and answer type

Recall@10 stays above 0.90 on every slice, including the hardest ones. Media questions hold at 0.907 even though they are the slice least served by text matching, and multi-item list-recall, where a question needs several gold memories at once, holds at 0.906. The media questions are where ATM-Bench is won or lost: they are 62% of the set, and they are what separated the systems on the leaderboard.

Where the gap remains

Eighty-two questions still have no gold item in the top ten, and 59 of them are media. These are not low-ranked misses that a deeper cut would catch, the gold item is absent from the top twenty. The cause is the memory representation, not the retriever: the dataset's 2B-model captions lack the referential detail the questions use ("the duck couple walking funny," "the Caterham car"), so there is no text that matches. The one system above HydraDB on the full set, Memexa v2, closed exactly this gap by re-captioning the media with a far stronger model. Re-captioning the 4,292 media items is the next retrieval lever, and it needs the raw media rather than a change to retrieval.

A note on answer quality

ATM-Bench also scores end-to-end answers (QS). Under the leaderboard's own answerer and judge, an 8B vision-language model with a minimal-reasoning GPT-5-mini judge, HydraDB's retrieval yields 57.4% QS on the full set. With all evidence present the 8B answerer still fails a large share of questions, so QS here measures the answerer, not retrieval; swapping in a stronger answerer raises the same retrieval by roughly thirteen points. We quote recall@10 for system comparison and name the answerer and judge next to any QS.

Why This Matters for Personal-Memory AI Agents

A capable model can answer a referential question once it has the right memory. If retrieval hands it a photo from the wrong week, it describes the wrong evening in fluent detail. On a four-year personal archive full of near-twins, the deciding question is not whether a system finds similar memories but whether it finds the right one, the exact item, at the exact time and place the question names.

HydraDB reached 0.918 recall@10 on the full set and 0.774 on the hard split by ingesting each memory with its full timestamped metadata into a connected, time-aware representation, and by retrieval that stays robust on a lopsided, text-heavy corpus. That is second on the public leaderboard, behind only a system that improved the captions, not the retrieval.

The benchmark does not cover every requirement of a production memory system. It is a single person's static archive, so it does not test permissions, cross-user isolation, or how quickly a new memory becomes retrievable, and its official score measures the complete agent rather than retrieval alone.

Conclusion

On ATM-Bench, HydraDB reaches 0.918 recall@10 with 0.781 MRR across 11,034 personal-memory items, and 0.774 recall@10 on the hard split, second on the public leaderboard and first among systems using the benchmark's own captions. The gain comes from metadata-rich, time-aware ingestion and from retrieval that holds up on a corpus where text-dense emails would otherwise bury the media. The remaining loss is a caption-quality ceiling in the dataset's media representation, not the retriever. The result is self-reported and measures retrieval; QS is bounded by the answerer model, not by HydraDB.

References

1. According to Me: Long-Term Personalized Referential Memory QA. Mei et al. (2026). arXiv:2603.01990.
2. ATM-Bench dataset and evaluation suite. Hugging Face (2026). Code: JingbiaoMei/ATM-Bench, GitHub.
3. Beyond Flat Embeddings for Production AI Agents. HydraDB Research Team (2026).
4. ATM-Bench live leaderboard. Retrieved October 2026.