MacPaw / Eney / Aug 10
Memory Evaluation Extension
A 32-case addition for evaluating whether an agent can maintain and use durable personal memory, not merely answer with preloaded context.
What Changed
| Earlier suites | Aug 10 extension |
|---|---|
| 34 non-MCP and 50 MCP tool-use tasks. | 32 general-user Mac memory tasks, organized as 22 core cases plus 10 coverage cases. |
| Measures tool selection, execution, and final outcome. | Adds durable write/update, restraint, retrieval, multi-note synthesis, fresh-session persistence, conflict/recency, expiry, forgetting, conditional facts, and conversation/tool-result recall. |
| Existing MacPaw task formats and graders. | Same sample.yaml contract and graders; lifecycle pairs use a portable patch to create separate writer and reader sessions. |
Execution Coverage
| Slice | Tasks | LFM 2.5 2.6B Tool Q4_K_M | Claude Sonnet 4.6 | Readout |
|---|---|---|---|---|
| Core calibration | 16 | 3/16 | 13/16 | Local memory-only comparison. |
| Coverage extension | 10 | 3/10 | 9/10 | Conflict, expiry, forgetting, conditional, and tool-result context. |
| Fresh-session lifecycle pairs | 2 | 0/2 | 2/2 | Writer and reader are separate sessions sharing memory only. |
| Comparable local total | 28 | 21.4% (6/28) | 85.7% (24/28) | Current memory benchmark readout. |
| VM-bound filesystem actions | 4 | Excluded | Excluded | Need MacPaw's VM path semantics for a valid score. |
Combined Benchmark
The MacPaw draft shared on August 5 is imported unchanged: 190 tasks. It is combined only at build time with the 32 Liquid-authored memory cases, producing a 222-task root. The 57 MCP cases select one or more of seven servers: Jira, Slack, Confluence, Spotify, Google Maps, Google Drive, and YouTube. The published mcp_base_set carries the matching 77 tools.
The combined model runs are prepared as the next evaluation step. For a score that is comparable with MacPaw's canonical reports, use their VM-backed path. Local mode is still useful for protocol debugging and memory-only smoke tests, but it cannot faithfully map ~/Downloads fixture paths into a case sandbox.