Does Dense Retrieval Improve Mnemos Memory Answers?

TL;DR: Across sampled LongMemEval (100 questions) and LoCoMo (50 questions), simple BM25 retrieval produces better final answers than the dense and hybrid variants tested here.

Hybrid fusion did not recover the BM25 result on either sample.

1. Goal

Test whether replacing or augmenting Mnemos's lexical sentence retrieval with Nomic or LFM dense retrieval improves final memory-question answer quality.

2. Methodology

User questionA benchmark memory question.

→

Retrieve memoryA tested retrieval variant returns sentence candidates and local clusters.

→

Agent contextRanked clusters are packed into a shared 6,000-character context.

→

Gemma answerGemma 4 26B answers from that context.

→

Score answerCompare with reference using heuristics and GPT-OSS 20B.

Experiment details

Retrieval variants: BM25-only; Nomic vector-only; Nomic BM25 + vector; LFM vector-only; LFM BM25 + vector.
Fixed retrieval path: Mnemos splits history into sentence-level records. Each sentence has an FTS entry and, for vector variants, one dense vector. It retrieves up to 30 candidates per lane, RRF-fuses ranks for hybrid variants, expands candidates into local sentence clusters, and packs them into the answer context.
Benchmarks: LoCoMo-10 has 10 multi-session conversations and roughly 1,540 QA pairs; LongMemEval has 500 long-term-memory questions over multi-session chat histories. Both ask for facts from earlier dialogue.
Answer evaluation: token F1, exact match, and substring match compare with the dataset reference; GPT-OSS 20B independently judges semantic correctness.

3. Results

These are deliberately bounded shared samples: 100 LongMemEval questions and 50 LoCoMo questions. Running this fixed 150-question sample across all five variants, including Gemma generation and GPT-OSS judging, takes about 6-8 hours locally. This is not a full-dataset run.

Heuristic metrics: F1 measures token overlap; exact is normalized equality; substring counts when either answer contains the other.
GPT-OSS judge: independent semantic-correctness accuracy against the reference answer.

LongMemEval: 100 questions

Retrieval variant	F1 [0-1] ↑	Exact [0-1] ↑	Substring [0-1] ↑	GPT-OSS judge [0-1] ↑
BM25-only	0.341	0.170	0.410	0.520
Nomic vector-only	0.256	0.130	0.270	0.350
Nomic BM25 + vector	0.269	0.140	0.280	0.380
LFM vector-only	0.266	0.140	0.290	0.370
LFM BM25 + vector	0.278	0.160	0.280	0.360

LoCoMo: 50 questions

Retrieval variant	F1 [0-1] ↑	Exact [0-1] ↑	Substring [0-1] ↑	GPT-OSS judge [0-1] ↑
BM25-only	0.470	0.260	0.320	0.500
Nomic vector-only	0.322	0.220	0.220	0.340
Nomic BM25 + vector	0.346	0.220	0.260	0.380
LFM vector-only	0.371	0.140	0.240	0.360
LFM BM25 + vector	0.338	0.140	0.160	0.300

4. Unified Observation

BM25 is the consistent winner: it leads every reported final-answer metric on both samples. The evidence supports a narrow conclusion: for these conversation-centric benchmark slices and this Mnemos sentence-and-cluster route, off-the-shelf dense retrieval adds no measured value over BM25.

5. Qualitative Evidence

Across the completed 150-question slice, there are 18 strict BM25 wins: BM25 is judged correct while all four dense/hybrid variants are judged wrong. The cases below are the complete set, not a random sample.

Disclaimer: this qualitative audit and its failure labels were performed by Codex from the stored retrieval contexts, generated answers, and GPT-OSS verdicts. They are diagnostic interpretations, not independently human-validated root-cause annotations.

Recurring failure patterns

Pattern	Cases	What recurs
Multi-memory aggregation	4	Dense/hybrid variants answer from one matching event when the question requires counting or combining several events.
Competing related facts	3	They choose a plausible nearby fact, or ignore a key negation, rather than the requested value.
Direct-fact coverage	7	They answer that the fact is absent or incomplete even though BM25 supplies a usable memory context.
Temporal order or date arithmetic	2	They retrieve a related event but do not resolve which occurred first or calculate the elapsed time.
Compositional arithmetic	1	They retrieve part of the evidence but do not combine the values into the requested answer.
Multi-constraint preference	1	They focus on one accessory category rather than the user’s complete equipment and quality preferences.

All 18 paired scenarios

Dataset	Question	Pattern	BM25 result	Dense/hybrid result and diagnostic
LongMemEval	How often do I see Dr. Smith?	Competing related facts	Every week.	All four say “every two weeks.” Both facts are retrieved, but BM25 ranks weekly #1 while every other variant ranks biweekly #1: a confirmed context-ranking failure.
LongMemEval	How many bikes do I currently own?	Multi-memory aggregation	Four bikes.	All four say three, omitting the newer hybrid bike. The result is consistent with missing an update in the packed evidence.
LongMemEval	How many doctor appointments did I have in March?	Multi-memory aggregation	Two appointments.	All four return one appointment, selecting either the March 3 or March 20 event instead of combining both.
LongMemEval	How many graduation ceremonies did I attend in three months?	Multi-memory aggregation	Three ceremonies.	All four return one, centered on Rachel’s ceremony rather than aggregating all three events.
LongMemEval	What was the total distance of four road trips?	Multi-memory aggregation	3,000 miles.	All four report no information or one 160-mile trip; the answer needs coverage and composition across four trips.
LongMemEval	Which platform gained the most followers?	Competing related facts	TikTok.	All four choose Twitter, a plausible growth fact but not the maximum requested by the question.
LongMemEval	How many debut-album copies were released?	Direct-fact coverage	500 copies.	All four say the fact is unavailable. This is a direct evidence-coverage failure in the answer path.
LongMemEval	What time do I get home on weeknights?	Direct-fact coverage	Around 6:30 pm.	All four say no time is stated, despite a routine-fact answer available to BM25.
LongMemEval	What discount did I get on the book?	Compositional arithmetic	20%.	All four surface only an original or sale price and abstain; the requested result requires combining both values.
LongMemEval	How many autographed footballs did I add?	Competing related facts	Not enough information; the history is about baseballs, not footballs.	All four convert the nearby baseball fact into a football answer instead of respecting the false premise.
LongMemEval	What photography accessories would complement my setup?	Multi-constraint preference	Sony-compatible, high-quality accessories spanning the current setup.	All four narrow to Godox flash cases/pouches and omit the broader Sony and quality constraints.
LoCoMo	What hobby did Sam take up in May 2023?	Direct-fact coverage	Painting.	All four return no answer or “none,” while BM25 supplies the dated hobby fact.
LongMemEval	Which came first: The Crown or Game of Thrones?	Temporal order or date arithmetic	Game of Thrones.	All four retrieve a related show history but return an empty answer rather than resolve the ordering.
LongMemEval	How many days ago was the Maundy Thursday service?	Temporal order or date arithmetic	Four days.	All four say the service is absent; BM25 recovers enough date evidence to calculate the interval.
LoCoMo	What did Caroline research?	Direct-fact coverage	Adoption agencies.	All four produce a blank or broader counseling/career answer instead of the specific research topic.
LoCoMo	What did Mel and her kids paint in July 2023?	Direct-fact coverage	A sunset with a palm tree.	All four return blank or a different craft project, missing the dated visual detail.
LoCoMo	Which book did Tim recommend to John on Dec. 8?	Direct-fact coverage	A Dance with Dragons.	All four return a generic series description, another title, or no answer instead of the exact recommendation.
LoCoMo	What is Jolene’s favorite game with her partner?	Direct-fact coverage	It Takes Two.	All four give a vague preference, a different game, or no answer; the named game does not survive the answer path.

The direct-fact and aggregation labels describe observed answer/context behavior; they do not independently prove the exact internal cause for every row. The Dr. Smith case is the strongest one: inspection confirms that ranking order, rather than absence of the correct fact, drives the wrong answer. A temporal conflict-resolution variant is therefore a sensible next ablation.

Where BM25 still fails (62 universal failures)

There are 62 universal failures in the 150-question slice: 44/100 LongMemEval and 18/50 LoCoMo questions where all five variants are judged wrong. A token-overlap audit of the BM25 packed context finds likely gold evidence in 31/62 cases, likely missing evidence in 10/62, and an ambiguous result in 21/62. This is a diagnostic heuristic, not a causal proof: evidence may be present but poorly ranked, incomplete, or too difficult for the answer model to compose.

Failure pattern	Representative questions	What fails across all variants
Cross-memory counting and aggregation	Korean restaurants: 4, Crash Course videos: 15, antique family items: 5.	Answers commonly return a subset such as 3, 12, or 2. Retrieval/context coverage may contain only some relevant episodes, and Gemma does not reliably enumerate the complete set.
Arithmetic and composition	Boot-price difference: $750, coffee mugs: $12 each, commute plus preparation: 90 minutes.	The answer requires combining two values. All methods often cite one input, a range, or an abstention instead of calculating.
Temporal updates and relative dates	French-press ratio change, clinic arrival time, Caroline’s support-group date, Nate’s second tournament.	Related events are retrieved, but exact dates, relative-time resolution, or an updated value are lost or not reasoned through by the answer model.
Long multi-part facts	John’s basketball schools and high-school duration; Facebook plus Instagram reach; Gina’s business advice.	The model returns one fragment, a generic summary, or no answer when the reference requires several linked facts.
Direct evidence not used	Dog breed, pastries at the cafe, Tokyo location Shinjuku.	At least some cases have plausible answer evidence in the packed context, yet Gemma outputs “not mentioned,” an empty answer, or a broad location. These are likely generation/context-use failures, but need per-case trace inspection to separate them from ranking.

6. Appendix

High-level Mnemos memory system

MacPaw has confirmed this is a reasonably accurate high-level representation of the current Mnemos retrieval path. It is an architecture summary: optional components are shown for context, not as claims that every deployment enables them.

Mermaid diagram of the Mnemos ingest, retrieval, and answer paths

One real chat example: what is indexed, retrieved, and answered

This is a LongMemEval case from the evaluated corpus. The raw source is a multi-turn chat session, stored as one episode and split into sentence records for FTS/vector retrieval. At question time, ranked sentence hits expand into nearby sentence clusters; those clusters are packed into the answer-model context.

Raw chat text entering the episode: user: ... I've been keeping [my old sneakers] under my bed for storage, and they're starting to smell.; user: ... I'm looking forward to storing my old sneakers in a shoe rack in [my closet], they're currently taking up space.
Index records: The full session is retained as the raw episode. Its individual sentences become retrieval records, including one sentence about the original under-bed location and another about the later intended closet/shoe-rack location. Both receive an FTS entry and, for vector variants, a sentence embedding.
Question and packed context: Question: Where do I initially keep my old sneakers?; BM25 retrieves clusters containing both the older under my bed statement and the later closet plan. Mnemos passes the ranked, expanded clusters to the reader under the shared 6,000-character context budget; it does not pass a prewritten summary of the session.
Gold answer versus observed answer: Gold: under my bed; BM25 + Gemma answer in this run: They are currently taking up space in your closet.; This is an answer-generation/temporal-interpretation miss, not proof that the original fact was absent from the index: the relevant under-bed sentence was retrieved, but the reader chose the later closet statement. It illustrates why the report measures final answer quality rather than retrieval alone.

Scope: This does use Mnemos's sentence-level index, candidate clustering, context packing, answer model, and judge route. It excludes only the upstream heterogeneous-event ingestion and memory-extraction stages, because LoCoMo and LongMemEval provide conversational histories rather than files, calendar events, contacts, or OCR data. It holds the evaluated path fixed and changes only the lexical/vector retrieval variant.

Interpretation: final-answer quality is the primary metric because sentence-level Recall@k is not directly comparable once retrieved candidates are expanded into different local clusters for context assembly.

Proposed Next Direction: Agent-Maintained Memory

Diff 1: Let the agent maintain Markdown memory: during chat, the agent adds or edits concise topical Markdown files through tools. This replaces indexing the entire raw event stream, which otherwise inflates the index, adds noise, and makes retrieval harder.
Diff 2: Chunk and index Markdown memory: create an index over Markdown chunks rather than raw conversation sentences. As files grow beyond the practical local-model context budget, a separate maintenance workflow can compact, split, archive, and reindex them.
Diff 3: Read directly before retrieving: the system prompt lists available Markdown files with their topic and token length. The agent reads a small relevant file directly into context (for example, travel.md at 320 tokens); only large files require an index search. Start with FTS5/BM25 and validate whether dense or hybrid retrieval adds enough value to justify the extra model and complexity. The current ablation finds dense retrieval does not help on the tested LoCoMo and LongMemEval slices.
Diff 4: Evaluate against realistic memory use: LoCoMo is dialogue between two people, while LongMemEval is user-assistant chat but not agentic or tool-call based. LongMemEval remains a useful homogeneous chat baseline, but the next benchmark should reflect Eney-style user queries and decide whether tool responses belong in the memory input stream.

Proposed memory routing

The system prompt lists accessible notebooks with their topic and token length. Every notebook is indexed at file/section level; the agent reads a small relevant file whole, while a large file is searched for relevant sections before answering.

Open the interactive memory-routing walkthrough

TODOs to think about

Deduplication across repeated or overlapping memory records.
Conflicting records and how an agent should revise or preserve their history.
Timestamps and temporal validity for memory entries and updates.
Interactive upper-bound plots: add accuracy-versus-latency and accuracy-versus-context-token charts for exact gold evidence, gold plus ordinary distractors, and gold plus retrieved hard negatives / real chunks.
Clickable experiment details: make each plotted point expandable, showing a short description of the evidence condition, answer model, context budget, trial count, and latency definition, following the Pipette-style interaction.

Authored by: Akhilesh Gotmare, Liquid AI and Codex. Last updated: July 21, 2026.