02 / Mnemos - eval

Longitudinal memory benchmark

Seed weeks of user-agent history, swap memory backends, and score whether the final Eney-style agent answers correctly.

• Generate long traces: chat, tools, docs, calendar, Slack, stale facts, distractors.
• Feed the same trace into each memory design: MD-only, BM25/FTS, dense, hybrid.
• Ask fresh-session questions that require buried memory.
• Score three layers: stored fact, retrieved context, final answer.
• Use final task success as the main metric; retrieval/write scores are diagnostics.

flowchart LR
  A["Synthetic multi-week user history"] --> B["Memory ingestion"]
  B --> C1["MD-only"]
  B --> C2["BM25 / FTS"]
  B --> C3["Dense"]
  B --> C4["Hybrid"]
  C1 --> D["Fresh Eney-style task"]
  C2 --> D
  C3 --> D
  C4 --> D
  D --> E["Agent receives memory context"]
  E --> F["Final answer"]
  B --> G["Did it store the right fact?"]
  E --> H["Did it retrieve the right context?"]
  F --> I["Did the agent answer correctly?"]
    

The benchmark isolates memory architecture while keeping the model, harness, and task fixed.