02 / MacPaw - Mnemos
Notes on memory architecture, retrieval experiments, and Eney-style memory evaluation.
Primary benchmarks: HippoCamp-Chat v0.8.8 is the frozen cross-modal target, pairing HippoCamp's personal files with multi-session user-agent chats and tool trajectories. LongMemEval-v1 complements it with explicit chat-memory evaluation.
Metrics and diagnostics: final-answer accuracy is the headline metric, while all-evidence recall diagnoses retrieval. HippoCamp, MemConflict, HaluMem, and DynamicMem provide secondary checks for temporal updates, conflict handling, hallucination, and user modeling.
| Dataset | Size GB | Questions | Evidence units / question avg | median | Searchable units / question avg | median | Turns / session avg | median | Comments |
|---|---|---|---|---|---|---|
| LongMemEval-S | 0.28 | 500 | 1.90 | 2 sessions | 47.73 | 48 sessions | 10.34 | 12 | Released S split; supporting and distractor sessions share the same searchable timeline. |
| LongMemEval-M | 2.74 | 500 | 1.90 | 2 sessions | 475.31 | 476 sessions | 10.30 | 12 | Same questions and supporting sessions as S, with roughly 10× more distractor history. |
| HippoCamp Gold | 0.53 | 581 | 4.28 | 1 files | 643.67 | 711 files / persona | N/A | File-based benchmark; Victoria's finance corpus drives the high searchable mean. |
| HippoCamp-Chat-S v0.8.8 | 0.0014 | 60 50 answerable 10 no-answer | 2.48 | 2 1.36 chats + 1.12 durable | 621 | 689 49.53 chats + 571.02 files | 11.26 | 10 | One consolidated S benchmark across three persona timelines. Answerable cases require 2.48 annotated units on average. At each query timestamp, the memory searches about 621 visible units: 49.53 chats and 571.02 durable files on average. |
1 Sizes use the released cleaned JSON for LongMemEval, the parsed-text HippoCamp Gold corpus, and the locked HippoCamp-Chat v0.8.8 JSON payload. HippoCamp-Chat excludes its inherited files. A turn is one user or assistant message.
2 HippoCamp-Chat exceeds LongMemEval-S in total file + cutoff-visible-chat units and closely matches its conversational interference at 49.53 visible chats per question. Files and chat sessions are not equivalent retrieval units.
3 Early queries see less history; later queries approach all 90 chats in their persona timeline.
The frozen release contains 60 questions, 270 user-assistant chats, and 1,931 durable files. Of the chats, 68 are gold evidence and 202 are constraint-checked distractors.
| Dimension | Distribution |
|---|---|
| Personas | Adam: 22; Bei: 18; Victoria: 20 |
| Answerability | Answerable: 50; no-answer: 10 |
| Difficulty | Easy: 23; medium: 31; hard: 6 |
| Evidence shape | 1 chat + 1 source: 33; 2 chats + 1 source: 12; four or more evidence units: 5 |
| Evidence span | <1 day: 2; 2–7 days: 18; 7–31 days: 20; 1–3 months: 7; 6+ months: 3; no-answer: 10 |
| Memory tags | Fact recall: 47; personalisation: 38; temporal reasoning: 33; memory updating: 24; conflict resolution: 19; negative-space: 5; no-answer: 10 |
| Gold durable sources | Email: 18; calendar: 12; files: 21 |
Known limitations. Chats and tool outputs are synthetic approximations, and gold-chat assistant turns are less verbose than production assistants. The locked release should remain unchanged while memory designs are compared.
The main view is the chronological experiment record. Each dot is an indexed attempt; the most important repeated arms are highlighted with their mean and ±1 SD. Hover or select a dot for the result details, then open Archive when you need the full audit trail.
Select a point to inspect its configuration, verified metrics, and known gaps.
Architecture variants, lexical screens, and evidence ceilings share a 12K-character pack cap. Accuracy is over 50 answerable cases per trial; gold-supplied rows have no retrieval metrics. Experiment audit log.
Answer model: Qwen3.8-27B (thinking ON, medium). GPT-5.5 (medium) judges accuracy only. Qwen ingestion and query planning use thinking OFF. A generated summary paraphrases and condenses its source; key spans stay verbatim.
| Experiment | Variant / condition | Accuracy mean ± sample SD; trials | Reader context tokens p50; avg records/question | Index size | Latency breakdown evidence build; final-Qwen TTFT, generation and decode rate; E2E p50 / p95 / mean | Recall all / any | MRR |
|---|---|---|---|---|---|---|---|
| Evidence-quality upper bounds | |||||||
| EXP-085 | Exact gold evidence; retrieval bypassed | 91.3% ± 3.1 n=3; 88–94% | 549.5 [284–1,800]; ~2 records | N/A (no index) | Context: N/A (fixed pack supplied)TTFT: 0.122 s / 0.320 sDecode: 7.693 s / 27.543 sE2E: 7.811 s / 27.858 s | N/A (gold supplied) | N/A (retrieval bypassed) |
| EXP-092 | Gold + 8 random distractors | 85.3% ± 2.3 n=3; 84–88% | 2,931.5 [2,439–3,961]; ~14 records | N/A (no index) | Context: N/A (fixed pack supplied)TTFT: 0.343 s / 0.660 sDecode: 10.632 s / 43.040 sE2E: 10.967 s / 43.804 s | N/A (gold supplied) | N/A (retrieval bypassed) |
| EXP-083 | Gold + actual retriever candidate setRealistic upper bound (gold injected) | 77.3% ± 5.0 n=3; 72–82% | 2,992.5 [2,234–4,659]; ~12 records | 558 MB | Context: N/A (prebuilt candidate + gold pack)TTFT: 0.354 s / 0.733 sDecode: 11.010 s / 47.953 sE2E: 11.350 s / 48.665 s | N/A (gold injected) | N/A (gold injected) |
| EXP-098 | Gold + LLM-selected hard negatives | 82.7% ± 5.0 n=3; 78–88% | 2,819.5 [2,234–3,696]; ~11 records | N/A (no index) | Context: N/A (fixed pack supplied)TTFT: 0.335 s / 0.766 sDecode: 11.684 s / 49.375 sE2E: 11.988 s / 50.206 s | N/A (gold supplied) | N/A (retrieval bypassed) |
| Mnemos architecture progression | |||||||
| Dense-rawEXP-073 | Raw-source dense retrieval + BGE rerank | 41.3% ± 1.2 n=3; 40–42% | 2,969.5 [1,464–4,420]; ~8 records | 558 MB | Context: 1.788 s / 7.864 sTTFT: 0.363 s / 0.741 sDecode: 12.204 s / 54.140 sE2E: 16.478 s / 55.824 s | 42.0% / 70.2% | 0.820 |
| Summary-DenseEXP-120 | Abstractive source summaries + dense retrieval / rerank | 46.7% ± 2.3 n=3; 44–48% | 3,124 [2,462–3,883]; ~14 records | 40.6 MB | Evidence build p50: 0.22 sFinal Qwen TTFT p50: 0.332 sGeneration p50: 16.5 s; decode rate p50: 51.1 tok/sE2E p50 / p95 / mean: 16.7 s / 60.7 / 26.0 s | 58.0% / 79.7% | 0.861 |
| Query-DecomposeEXP-121 | Fixed 300-token summary + upfront query decomposition + dense retrieval / rerank | 54.0% ± 3.5 n=3; 50–56% | 3,095 [2,514–4,235]; ~24 records | 39.4 MB | Evidence build p50: 2.92 s (includes decomposition call)Final Qwen TTFT p50: 0.333 sGeneration p50: 15.1 s; decode rate p50: 51.5 tok/sE2E p50 / p95 / mean: 18.7 s / 60.2 / 25.5 s | 76.0% / 89.3% | 0.884 |
| Read-RetrieveEXP-119 | Hybrid summary + verbatim spans; dense + BGE rerank; one follow-up retrieval | 56.0% ± 3.5 n=3; 54–60% | 3,022 [2,591–3,828]; ~11–12 records | 40.9 MB | Evidence build p50: 1.15 s (includes controller)Final Qwen TTFT p50: 0.327 sGeneration p50: 15.2 s; decode rate p50: 51.5 tok/sE2E p50 / p95 / mean: 16.6 s / 57.4 / 22.3 s | 64.0% / 84.0% | 0.862 |
| Simplistic baselines (lexical) | |||||||
| BM25-rawEXP-125 | Raw documents; BM25 only | 4.0% n=1; SD N/A | 2,979 ~5 records | 533 MB | Evidence build p50: 1.80 sFinal Qwen TTFT p50: 0.32 sGeneration p50: 7.93 sE2E p50 / p95: 13.1 s / 32.9 s | 8.0% / 25.5% | 0.345 |
| BM25-SummaryEXP-126 | BM25 over fixed 300-token summaries | 26.0% n=1; SD N/A | 3,069 ~22 records | ~39 MB | Evidence build p50: 0.02 sFinal Qwen TTFT p50: 0.34 sGeneration p50: 13.26 sE2E p50 / p95: 13.3 s / 51.6 s | 34.0% / 60.0% | 0.627 |
| BM25-Summary+RerankEXP-127 | BM25 over summaries + BGE rerank; no dense search | 36.0% n=1; SD N/A | 3,054 ~23 records | ~39 MB | Evidence build p50: 0.13 sFinal Qwen TTFT p50: 0.33 sGeneration p50: 13.35 sE2E p50 / p95: 13.5 s / 57.5 s | 44.0% / 70.7% | 0.821 |
| Serving: Measured on 1 AMD GPU; vLLM 0.19.1, BF16, TP=1, 32K max context, 90% GPU memory. For EXP-119–121, evidence-build includes any planner/controller call; TTFT and generation are final-Qwen only. Decode is throughput (tok/s), not duration. New-run p95/mean shown are end-to-end aggregates. | |||||||
End-to-end p50 on a zero-based 0–25 second scale. The BM25 screens have one trial each; Mnemos variants have three. Horizontal references show EXP-083, the realistic upper bound with gold injected into retrieved candidates, and EXP-085, the exact-gold reader ceiling.
Verbatim prompt templates from cluster checkout mnemos-w10-compose-16c4ed5. Braced names are runtime substitutions, not literal prompt text.
scripts/build_persisted_source_index_qwen_compress.py · qwen_abstractive_300 · default_max_output_tokens=300
Write the densest possible factual summary of this source in UNDER 90 WORDS:
the shortest text that still preserves every fact a later question could
depend on (names, dates, numbers, decisions, preferences, events). If the
source contains more facts than fit in 90 words, keep the most specific,
answer-bearing ones and drop the rest rather than exceeding the budget.
Paraphrase freely for density, but do not add anything not present in the
text and do not drop a fact you have room for. Output plain text only, no
markdown headers, no preamble or commentary - just the summary itself.scripts/build_persisted_source_index_qwen_compress.py · qwen_abstractive
Write the densest possible factual summary of this source: the shortest
text that still preserves every fact a later question could depend on
(names, dates, numbers, decisions, preferences, events). Paraphrase freely
for density, but do not add anything not present in the text and do not
drop a fact to save space. Output plain text only, no markdown headers, no
preamble or commentary - just the summary itself.src/memory_arch_eval/iterative_native_retrieval.py · one_shot_planner_messages()
You plan the only opportunity to query a chronological personal-memory index for another model.
After this response, your searches run concurrently. You will not see their results, revise the plan,
inspect neighbors, or issue another query. Do not answer the user's question.
Return only JSON with exactly this schema:
{"searches":[{"query":string,"k":integer}]}
Rules:
- Produce between 1 and {max_queries} distinct searches.
- The first query must reproduce the user's exact original question, character for character.
- Choose k independently for every query from 1 to {max_k}.
- Use additional searches to cover independent answer facets, entities, aliases, dates, events, or lexical
bridges that the exact question may miss. Because this is your only shot, cover plausible facets now.
- Prefer small k for precise lookups and larger k for broad or ambiguous facets.
- Do not invent personal facts, names, dates, or relationships absent from the question.
User: Original question:
{question}scripts/iterative_retrieval_augmentations.py · _plan_case()
Plan focused retrieval queries for a locked memory benchmark. Return only JSON with
exactly this schema: {"queries":[string],"stop":boolean}. Emit at most {max_queries}
new queries. Set stop=true and queries=[] only when the evidence is sufficient. Do not
answer the question and do not include claims not present in the question or evidence.
User: Question:
{question}
Already used queries:
{used_queries}
Retrieved raw evidence:
{evidence}Longitudinal memory benchmark planProposed memory flowOne synthetic long-history eval episode
How to read the grey points:
k=40/60/80/100; the low points show what happens before enough evidence is recovered.The remaining coloured points are the retrieval, ingestion, and architecture sweep; failed and superseded attempts are in Archive.
These are the most useful non-oracle candidates from the exploration so far, ranked by confirmed accuracy and practical promise. The repeated arms matter more than the isolated 56% peaks from EXP-068/078, which were not independently confirmed.
Qwen builds a compact source summary plus two or three verbatim key spans. Mnemos performs dense retrieval and BGE reranking, then a Qwen controller writes one follow-up query for missing evidence.
The same compressed source index and one-hop controller, but the summary keeps query-relevant spans rather than a fixed span budget. It is the more selective, lower-context version of the hybrid recipe.
After the first retrieval, the controller sees the current evidence and emits both a follow-up query and a sufficient/stop decision. It can spend another hop only when the first pack appears incomplete.
At ingestion, Qwen extracts short cross-source tags and builds a lightweight link graph. At query time, linked partners of the top dense hits are added before BGE reranking, helping chat evidence reach its related file.
The cleanest test of the retrieval mechanism itself: raw parent records, dense retrieval and reranking, then one read-driven follow-up search. It works, but the uncompressed index and larger reader context make it the least efficient finalist.
flowchart TD Q[User question] --> I[Compressed source index] I --> R[Dense or lexical retrieval + BGE rerank] R --> L[EXP-106: linked partners before rerank] R --> A[EXP-100: adaptive follow-up query or stop] R --> H[EXP-119/104: hybrid summary span selection] L --> P[Source-aware evidence pack] A --> P H --> P P --> O[Qwen answerer] O --> J[GPT-5.5 judge] X[EXP-071: raw-index read-then-retrieve] --> P
Reading this shortlist: EXP-119 is the strongest completed recipe under the shared 12K-character cap; EXP-074 remains its historical baseline. EXP-104 is the closest selective variant. EXP-100 and EXP-106 target recall directly at lower context cost. EXP-071 is the useful raw-record control that proves read-then-retrieve is doing real work, while also exposing its latency and index-size cost.
These earlier point estimates fix Qwen3.8-27B as the answer model and GPT-5.5 as the independent judge. They remain useful for retrieval diagnostics, but the promoted three-trial study above is the more reliable performance comparison. Answer accuracy is over the 50 answerable tasks.
| System / arm | Memory path | Answer accuracy | Recall @20 all sources | Mean source recall | MRR | Median context build | Median Qwen answer | Median end-to-end | Comments |
|---|---|---|---|---|---|---|---|---|---|
| Hindsight iterative | Native Hindsight + Qwen controller | 52% 26/50 | 0% | 28.2% | 0.44 | 210 s | 39 s | 253 s | Up to six native searches and 20 retained records, with strict query-time filtering. |
| Mem0 iterative | Native Mem0 + Qwen controller | 42% 21/50 | 0% | 28.7% | 0.40 | 135 s | 33 s | 175 s | Up to six native searches and 20 retained records. |
| Mnemos iterative FTS | FTS5/BM25 + Qwen controller | 26% 13/50 | 62% | 81.7% | 0.84 | 165 s | 57 s | 242 s | Iteration improves source coverage, but often retains the wrong passage from a correct parent source. |
| Mnemos iterative FTS + pseudo-QA | Pseudo-QA ingestion + Qwen controller | 20% 10/50 | 38% | 57.5% | 0.72 | 137 s | 35 s | 177 s | Combining pseudo-QA with iteration did not compound their separate gains. |
| Hindsight | Native pipeline | 32% 16/50 | 18% | 46.5% | 0.63 | N/A | 129 s | N/A | Context-construction timing was not recorded. Semantic and keyword retrieval plus local reranking returns about 8.5 coherent items at the median. |
| Mem0 | Native extraction and retrieval | 26% 13/50 | 22% | 55.2% | 0.63 | N/A | 132 s | N/A | Context-construction timing was not recorded. Sonnet 5 extracts atomic chat memories; durable files remain overlapping raw chunks. |
| Mnemos FTS | Sentence FTS5/BM25 | 14% 7/50 | 44% | 68.3% | 0.61 | 0.1 s | 101 s | 101 s | Raw-record baseline with 20 retrieved records. Almost all measured latency is Qwen answer generation. |
| Exact gold evidence | Retrieval bypassed | 88% 44/50 | N/A | N/A | N/A | N/A | 35 s | 35 s | Reader upper bound using the same Qwen answerer and GPT-5.5 judge. The remaining errors are downstream synthesis or answer-quality failures. |
Timings are medians over the recorded evaluation queries. Context build includes controller inference and index search through the final packed evidence. Qwen answer is the wall-clock latency of the batch containing each query; end-to-end is computed per query as context build plus that batch latency, so it is not the sum of the displayed medians. GPT-5.5 judging is excluded. All current rows abstain on 10/10 no-answer tasks.
| Arm | Accuracy mean ± SD; range | Context build mean [95% CI] | Qwen answer mean [95% CI] | End-to-end mean [95% CI] |
|---|---|---|---|---|
| Exact gold evidence Sonnet reader: 88% (44/50) | 91.3% ± 3.1; 88–94% Qwen reader; Sonnet is similar | 0.0 s [0.0, 0.0] | 25.4 s [22.5, 28.8] | 25.4 s [22.5, 28.8] |
| Mem0 native | 27.3% ± 1.2; 26–28% | 5.7 s [4.9, 6.5] | 40.3 s [35.5, 45.3] | 46.0 s [41.1, 51.2] |
| Mem0 one-shot parallel | 22.0% ± 2.0; 20–24% | 39.8 s [34.6, 45.3] | 36.7 s [32.3, 41.3] | 76.5 s [69.0, 84.5] |
| Mem0 Planned-Select | 32.7% ± 2.3; 30–34% | 58.4 s [52.3, 64.9] | 33.5 s [29.2, 38.1] | 91.9 s [83.2, 100.9] |
| Mem0 iterative | 44.7% ± 1.2; 44–46% | 130.7 s [121.8, 139.9] | 24.6 s [20.6, 29.0] | 155.3 s [144.4, 166.6] |
| Mem0 parallel iterative | 35.3% ± 1.2; 34–36% | 181.4 s [166.1, 197.3] | 32.6 s [28.1, 37.7] | 214.0 s [196.5, 232.5] |
| Mem0 Qwen-ingested native | 21.3% ± 3.1; 18–24% | 5.7 s [4.9, 6.6] | 45.8 s [39.8, 52.1] | 51.6 s [45.4, 57.9] |
| Mem0 Qwen-ingested Planned-Select | 26.0% ± 4.0; 22–30% | 58.4 s [52.2, 64.6] | 34.1 s [29.8, 38.6] | 92.4 s [84.3, 100.8] |
| Mem0 Qwen-ingested iterative | 36.7% ± 1.2; 36–38% | 147.2 s [137.8, 156.9] | 37.9 s [33.0, 43.0] | 185.2 s [173.3, 197.1] |
| Hindsight native | 25.3% ± 5.0; 20–30% | 28.8 s [22.5, 32.6] | 30.0 s [23.2, 36.2] | 58.8 s [46.1, 68.5] |
| Hindsight one-shot parallel | 15.3% ± 9.5; 8–26% | 72.6 s [60.6, 83.9] | 22.7 s [15.5, 30.9] | 95.4 s [77.6, 112.4] |
| Hindsight iterative | 16.0% ± 2.0; 14–18% | 111.4 s [86.5, 137.8] | 13.7 s [11.1, 16.8] | 125.1 s [98.5, 153.0] |
| Hindsight parallel iterative | 16.7% ± 4.2; 12–20% | 206.5 s [156.5, 261.4] | 13.9 s [11.1, 17.1] | 220.4 s [168.5, 276.8] |
| Mnemos FTS | 16.0% ± 2.0; 14–18% | 0.5 s [0.4, 0.6] | 42.6 s [37.5, 48.3] | 43.0 s [38.0, 48.7] |
| Mnemos one-shot parallel | 14.0% ± 2.0; 12–16% | 20.2 s [16.8, 24.1] | 41.8 s [37.1, 46.7] | 62.0 s [56.6, 67.6] |
| Mnemos iterative FTS | 16.0% ± 2.0; 14–18% | 141.2 s [131.4, 151.3] | 24.8 s [20.8, 29.1] | 166.0 s [154.5, 177.7] |
| Mnemos iterative FTS + pseudo-QA | 21.3% ± 1.2; 20–22% | 155.8 s [145.2, 166.6] | 23.7 s [20.6, 27.1] | 179.5 s [167.6, 191.5] |
| Qwen generic synopsis | 14.0% ± 2.0; 12–16% | 0.5 s [0.4, 0.6] | 44.0 s [39.0, 49.1] | 44.5 s [39.5, 49.6] |
| Qwen temporal ledger | 14.7% ± 1.2; 14–16% | 0.4 s [0.3, 0.5] | 43.6 s [39.2, 48.4] | 44.1 s [39.6, 48.9] |
| Qwen atomic cards | 19.3% ± 2.3; 18–22% | 0.4 s [0.3, 0.6] | 41.6 s [37.1, 46.3] | 42.0 s [37.6, 46.7] |
| Sonnet atomic cards | 22.7% ± 1.2; 22–24% | 0.4 s [0.3, 0.5] | 48.4 s [42.6, 54.8] | 48.8 s [43.0, 55.2] |
| Qwen retrieval anchors | 20.0% ± 2.0; 18–22% | 0.4 s [0.3, 0.5] | 44.2 s [39.2, 49.4] | 44.6 s [39.6, 49.9] |
| Qwen pseudo-QA cards | 20.7% ± 3.1; 18–24% | 0.4 s [0.3, 0.5] | 40.8 s [36.5, 45.4] | 41.3 s [36.9, 45.8] |
Reproducibility note. The earlier 52% Hindsight-iterative point did not reproduce: these trials scored 14%, 16%, and 18%. Mem0 iterative was substantially more stable at 44%, 44%, and 46%, making it the strongest reproducible retrieval arm in this study. The new Qwen-ingested trials scored 18%, 24%, and 22% natively; 38%, 36%, and 36% with sequential iteration; and 26%, 22%, and 30% with Planned-Select. This isolates a real ingestion-quality advantage for the Sonnet-built Mem0 index while showing that iterative retrieval can recover part, but not all, of the gap.
These arms keep native Mnemos sentence FTS fixed and append one kind of derived record at ingestion. Qwen3.8-27B answers every row and GPT-5.5 judges it; only the ingestion treatment varies.
| Treatment | Ingestion model | Answer accuracy | Recall @20 all sources | Mean source recall | MRR | Median latency | Comments |
|---|---|---|---|---|---|---|---|
| Pseudo-QA cards | Qwen3.8-27B | 24% 12/50 | 42% | 71.5% | 0.66 | 117 s | Appends likely question-answer pairs. Best one-pass ingestion treatment despite slightly lower all-source recall than raw FTS. |
| Retrieval anchors | Qwen3.8-27B | 20% 10/50 | 44% | 68.8% | 0.62 | 106 s | Appends compact keywords and paraphrases intended to bridge likely query wording to each raw record. |
| Atomic memory cards | Claude Sonnet 5 | 20% 10/50 | 42% | 72.2% | 0.71 | 123 s | Uses Sonnet only at ingestion; Qwen remains the answerer. Highest MRR among these ingestion treatments. |
| Atomic memory cards | Qwen3.8-27B | 18% 9/50 | 40% | 68.8% | 0.69 | 117 s | Breaks each source into compact standalone facts while retaining the raw record. |
| Temporal state ledger | Qwen3.8-27B | 18% 9/50 | 46% | 71.7% | 0.65 | 70 s | Extracts entity, attribute, value, and validity time. Retrieval improves more than final answer accuracy. |
| Generic synopsis | Qwen3.8-27B | 14% 7/50 | 42% | 68.0% | 0.62 | 68 s | General summaries do not improve answer accuracy over raw records. |
| Raw records | None | 14% 7/50 | 44% | 68.3% | 0.61 | 103 s | Control arm. Derived representations help only when their structure matches likely queries. |
Latency is median end-to-end time per answerable query and excludes judging. Ingestion cost and one-time index construction are not included.
The candidate answers are identical in both columns. GPT-5.5 is the primary judge; Qwen self-judging was generally more permissive and is retained only as a diagnostic.
| System / treatment | GPT-5.5 judge | Qwen judge | Difference |
|---|---|---|---|
| Hindsight iterative | 52% | 58% | +6 |
| Mem0 iterative | 42% | 42% | 0 |
| Hindsight | 32% | 38% | +6 |
| Mem0 | 26% | 36% | +10 |
| Mnemos iterative FTS | 26% | 32% | +6 |
| Mnemos iterative FTS + pseudo-QA | 20% | 22% | +2 |
| Mnemos FTS | 14% | 16% | +2 |
| Qwen pseudo-QA cards | 24% | 26% | +2 |
| Qwen retrieval anchors | 20% | 26% | +6 |
| Sonnet atomic cards | 20% | 26% | +6 |
| Qwen atomic cards | 18% | 22% | +4 |
| Qwen temporal ledger | 18% | 18% | 0 |
| Qwen generic synopsis | 14% | 18% | +4 |
These earlier v0.8.8 runs use Claude Sonnet 5 as final reader and GPT-5.5 as judge. Because the judge is now fixed, they provide a direct answer-model sensitivity comparison with the current Qwen-reader results.
| Memory system | Retrieval / memory path | Answer model | Answer accuracy |
|---|---|---|---|
| Hindsight | Native pipeline | Claude Sonnet 5 | 26% 13/50 |
| Mem0 | Native pipeline | Claude Sonnet 5 | 22% 11/50 |
| Mnemos FTS | Sentence FTS5 / BM25 | Claude Sonnet 5 | 14% 7/50 |
| Honcho | Native pipeline | Claude Sonnet 5 | 0% 0/50 |
HippoCamp-Chat v0.8.8 contains 50 answerable and 10 no-answer tasks. Retrieval is capped at k=20 and answer context at 12,000 characters. All four systems abstained correctly on 10/10 no-answer tasks. Honcho retrieved relevant evidence in only 3/50 answerable cases, so its 0% result primarily reflects retrieval failure.
Appending model-generated memory records at ingestion improves the native 14% FTS baseline. Qwen pseudo-QA cards perform best among the completed treatments.
| Ingestion treatment | Ingestion model | Answer model | Answer accuracy |
|---|---|---|---|
| Raw records | None | Claude Sonnet 5 | 14% 7/50 |
| Atomic memory cards | Claude Sonnet 5 | Claude Sonnet 5 | 20% 10/50 |
| Pseudo-QA cards | Qwen3.8-27B | Claude Sonnet 5 | 22% 11/50 |
| Atomic memory cards | Qwen3.8-27B | Claude Sonnet 5 | 18% 9/50 |
| Retrieval anchors | Qwen3.8-27B | Claude Sonnet 5 | 18% 9/50 |
| Generic synopsis | Qwen3.8-27B | Claude Sonnet 5 | 14% 7/50 |
| Temporal state ledger | Qwen3.8-27B | Claude Sonnet 5 | 14% 7/50 |
All rows retain the same native Mnemos FTS retrieval and judging path. The best completed treatment raises answer accuracy by 8 points, from 14% to 22%; more expensive ingestion is therefore not sufficient by itself, and the representation format matters.
| Arm | Answer model | Purpose | Answer accuracy |
|---|---|---|---|
| Mnemos FTS | Claude Sonnet 5 | Headline retrieval run | 14% 7/50 |
| Mnemos FTS | LFM 24B, greedy | Answer-model sensitivity | 12% 6/50 |
| Mnemos FTS | LFM 24B, sampled | T=0.1, top_k=50, repetition penalty 1.05 | 10% 5/50 |
| Mnemos FTS | LFM2.5-2.6B | Model-card inference settings | 10% 5/50 |
| Exact gold evidence | Claude Sonnet 5 | Reader upper bound | 88% 44/50 |
| Exact gold evidence | LFM 24B, sampled | Reader upper bound | 50% 25/50 |
| Exact gold evidence | LFM 24B, greedy | Reader upper bound | 48% 24/50 |
| Exact gold evidence | LFM2.5-2.6B | Reader upper bound | 36% 18/50 |
| Exact gold chat only | Claude Sonnet 5 | Cross-modal leakage check | 4% 2/50 |
| Exact gold chat only | LFM 24B, sampled | Cross-modal leakage check | 4% 2/50 |
| Exact gold chat only | LFM 24B, greedy | Cross-modal leakage check | 2% 1/50 |
| Exact gold chat only | LFM2.5-2.6B | Cross-modal leakage check | 2% 1/50 |
| Exact gold non-chat only | Claude Sonnet 5 | Cross-modal leakage check | 4% 2/50 |
| Exact gold non-chat only | LFM 24B, sampled | Cross-modal leakage check | 4% 2/50 |
| Exact gold non-chat only | LFM 24B, greedy | Cross-modal leakage check | 4% 2/50 |
| Exact gold non-chat only | LFM2.5-2.6B | Cross-modal leakage check | 6% 3/50 |
| Mnemos FTS, no chat distractors | Claude Sonnet 5 | Retrieval control | 12% 6/50 |
| Mnemos FTS, no chat distractors | LFM 24B, sampled | Retrieval control | 8% 4/50 |
| Mnemos FTS, no chat distractors | LFM 24B, greedy | Retrieval control | 8% 4/50 |
| Mnemos FTS, no chat distractors | LFM2.5-2.6B | Retrieval control | 8% 4/50 |
The exact-gold arms bypass retrieval. Low chat-only and non-chat-only accuracy indicates that most answerable tasks genuinely require both modalities. The no-chat-distractor control is diagnostic, not the benchmark setting.
| Memory system / arm | Answer accuracy | Recall @20 (all) | Mean source recall | MRR | No-answer abstention |
|---|---|---|---|---|---|
| Hindsight | 26% 13/50 | 18% | 46.5% | 0.63 | 10/10 |
| Mem0 | 22% 11/50 | 22% | 55.2% | 0.63 | 10/10 |
| Mnemos FTS | 14% 7/50 | 44% | 68.3% | 0.61 | 10/10 |
| Honcho | 0% 0/50 | 0% | 2.0% | 0.06 | 10/10 |
| Mnemos FTS, no chat distractors | 12% 6/50 | 46% | N/A | 0.60 | 10/10 |
Recall @20 (all) means that at least one retrieved chunk represents every annotated gold parent source; it does not mean the complete source text was retrieved. MRR is based on the first represented gold source. These are retrieval diagnostics, not headline quality metrics.
| Ingestion treatment | Ingestion model | Answer accuracy | Recall @20 (all) | Mean source recall | MRR |
|---|---|---|---|---|---|
| Raw records | None | 14% | 44% | 68.3% | 0.61 |
| Atomic memory cards | Claude Sonnet 5 | 20% | 42% | 72.2% | 0.71 |
| Pseudo-QA cards | Qwen3.8-27B | 22% | 42% | 71.5% | 0.66 |
| Atomic memory cards | Qwen3.8-27B | 18% | 40% | 68.8% | 0.69 |
| Retrieval anchors | Qwen3.8-27B | 18% | 44% | 68.8% | 0.62 |
| Generic synopsis | Qwen3.8-27B | 14% | 42% | 68.0% | 0.62 |
| Temporal state ledger | Qwen3.8-27B | 14% | 46% | 71.7% | 0.65 |
These results are retained for provenance but should not be compared directly with the native v0.8.8 baseline because the custom Linux compatibility harness used different indexing and chunking.
| System / arm | Dataset / harness | Answer accuracy | Recall @20 (all) | MRR |
|---|---|---|---|---|
| Mnemos FTS control | Custom Linux compatibility | 10% | 22% | 0.28 |
| Mnemos dense | Custom Linux compatibility | 14% | 30% | 0.32 |
| Mnemos hybrid RRF | Custom Linux compatibility | 16% | 28% | 0.36 |
| Mnemos hybrid + reranking | Custom Linux compatibility | 20% | 34% | 0.62 |
| Mnemos hybrid + parent expansion | Custom Linux compatibility | 14% | 28% | 0.36 |
| Mnemos hybrid + multi-query | Custom Linux compatibility | 18% | 28% | 0.37 |
| Mnemos temporal graph | Custom Linux compatibility | 18% | 30% | 0.50 |
The first uniform Qwen run stopped answers at 1,024 generated tokens. The current tables above automatically retry limit-hitting outputs at 3,072 tokens. These earlier scores are retained only to show the effect of that implementation change.
| System / treatment | Earlier accuracy | Current accuracy | Change | Comment |
|---|---|---|---|---|
| Hindsight | 40% | 38% | -2 | Earlier score was manually recovered because its summary aggregation failed; it was never part of the published table. |
| Mem0 | 32% | 36% | +4 | Two additional answers pass after retry. |
| Mnemos iterative FTS | 32% | 32% | 0 | Two cases became correct and two became incorrect; strict grounding rose from 32% to 36%. |
| Mnemos FTS | 18% | 16% | -2 | Longer generation did not improve aggregate accuracy. |
| Qwen retrieval anchors | 26% | 26% | 0 | |
| Qwen pseudo-QA cards | 24% | 26% | +2 | |
| Qwen atomic cards | 24% | 22% | -2 | |
| Qwen generic synopsis | 18% | 18% | 0 | |
| Qwen temporal ledger | 22% | 18% | -4 | |
| Sonnet atomic cards | 24% | 26% | +2 | Sonnet is used only for the ingestion representation. |
Successive versions improved source independence, removed chat-only and non-chat-only leakage, corrected overly strict reference answers, strengthened conflict and temporal cases, normalized first-person questions and realistic user style, and added constrained distractors plus no-answer cases. The long synthesis cycle was necessary because each task must remain natural while still requiring the intended evidence at the correct point in time.
Acceptance criteria. A fixed strong model must answer correctly from the complete gold evidence, then fail or abstain when either chat or non-chat evidence is masked. Questions, answers, timelines, and distractor constraints are locked only after this three-arm check.
The raw source remains indexed. Each treatment appends a derived record with the same persona, timestamp, modality, parent source ID, and provenance.
Example chat: “I still want Friday lunch with the team. At least half of the remaining dining balance should stay untouched.”
| Treatment | Example derived record |
|---|---|
| Generic synopsis | Keep Friday's team lunch while preserving at least half of the remaining dining budget. |
| Atomic memory card | Friday's team lunch may use at most half of the remaining dining budget. |
| Retrieval anchors | Friday social lunch; team lunch; dining budget; spending limit; keep half untouched |
| Pseudo-QA card | Q: How much of the remaining dining budget can Friday lunch use? A: No more than half. |
| Temporal state ledger | Entity: Friday team lunch; attribute: spending rule; value: maximum half of remaining balance; valid from: source timestamp. |
Example spreadsheet: the October budget records $200 budgeted and $130 spent on dining, leaving $70. A pseudo-QA artifact can state: “How much remains in the dining budget? $70.” The final query, “How much can I spend at Friday's team lunch?”, still requires the chat's half-balance rule and the spreadsheet's current balance, producing $35.
Research direction: compare Mnemos, Mem0 OSS, Hindsight, and Honcho under one harness on HippoCamp-Chat, original HippoCamp, and LongMemEval-M. Hold the memory-processing model, answer model, chronological cutoffs, retrieval budget, prompt, and judge fixed so the first study isolates the memory architecture; then vary the processing and answer models on the leading systems.
Report final-answer and no-answer accuracy, category slices, evidence recall, ingestion time and cost, search latency, retrieved tokens, index size, exact configuration, and code revision. Mem0's native extraction pipeline and its raw-index control must remain separate. Honcho's retrieved context or representation should use the shared answer model; its native Dialectic response can be reported as a separate product-mode arm.
Existing efforts such as OmniMemEval, MemDelta, and vendor benchmark suites cover parts of this matrix, but not a controlled architecture × processing-model × answer-model study over realistic cross-modal personal data. A reproducible comparison with per-case traces could therefore be a useful research release.