01 / MacPaw - Eney
Eney
Notes on the Eney harness and local evaluation of LFM tool-use models.
Benchmark Versions
This page contains results from multiple, non-equivalent benchmark releases. Only scores in the same row and version are comparable.
| Version | What it is | Size | Status |
|---|---|---|---|
| Legacy Export v0 | Original MacPaw split: general desktop tasks plus MCP tasks. | 84 | Archived exploratory suite |
| Liquid Memory Extension v1 | General-user memory tasks authored by Liquid to cover writing, maintenance, retrieval, restraint, conflict, expiry, forgetting, and long context. | 32 | Liquid augmentation |
| Liquid Draft v1 | MacPaw's Aug 10 draft plus Liquid's 32 memory tasks. This was a working union, not a canonical MacPaw release. | 222 190 + 32 | Superseded |
| Eney Refined v2 | MacPaw's reviewed, refined benchmark incorporating the selected Liquid memory set. | 204 172 + 32 | Current |
Current Readout: Eney Refined v2
Current benchmark: 172 reviewed base samples + 32 Liquid memory samples = 204 (Aug 14). MacPaw reports Claude Sonnet at 80% and Gemma 4 E4B at 39%. LFM 2.5 2.6B Q4_K_M is running locally; its final score will be added here once all 204 cases have graded. Do not compare this table with the archived suites below.
Liquid Memory Extension v1
The Aug 10 extension adds 32 general-user Mac memory cases: durable writes and updates, restraint, retrieval, multi-note synthesis, fresh-session persistence, conflict/recency, expiry, forgetting, conditional memory, conversation/tool-result recall, and longer context. The 28 cases that do not depend on host-home filesystem paths have clean, comparable local runs for both models.
| Aug 10 memory slice | Cases | LFM 2.5 2.6B Tool Q4_K_M | Claude Sonnet 4.6 | Interpretation |
|---|---|---|---|---|
| Core memory calibration | 16 | 3/16 | 13/16 | Memory-only local cases. |
| Coverage extension | 10 | 3/10 | 9/10 | Conflict, expiry, forgetting, conditional, and tool-result context. |
| Fresh-session lifecycle pairs | 2 | 0/2 | 2/2 | Write in one session; retrieve from memory in a new session. |
| Comparable local total | 28 | 21.4% (6/28) | 85.7% (24/28) | Draft-v1 memory benchmark readout. |
| VM-bound filesystem actions | 4 | Excluded | Excluded | Need MacPaw's VM path semantics for a valid score. |
What changed from Legacy Export v0: memory is evaluated as an explicit agent capability, including writes, maintenance, retrieval, restraint, and fresh-session use. The now-superseded 190-case MacPaw draft plus this 32-case extension formed the 222-task Liquid Draft v1 combined dataset.
Archived local tool-use runs: Legacy Export v0
These earlier 34/50-task results use Liquid's local-host fallback rather than MacPaw's private Tart VM environment. Docker/tool isolation differs modestly, but this is not considered a material blocker for early directional comparison.
Legacy Export v0 / llama.cpp
| Suite | 2.6B Tool Q8_0 | 2.6B Tool Q4_K_M | 2.6B RC1 Q8_0 | 2.6B RC1 Q4_K_M | 8B A1B Q8_0 | 8B A1B Q4_K_M | Hermes 3 8B Q4_K_M | Claude Sonnet 4.6 |
|---|---|---|---|---|---|---|---|---|
| Inference server | llama.cpp | llama.cpp | llama.cpp | llama.cpp | llama.cpp | llama.cpp | llama.cpp | Hosted |
| 34 tasks / non-MCP | 29% (10/34) | 24% (8/34) | 41.2% (14/34) | 35.3% (12/34) | 29.4% (10/34) | 29.4% (10/34) | 11.8% (4/34) | 82.4% |
| 50 tasks / MCP | 32% (16/50) | 28% (14/50) | 56.0% (28/50) | 46.0% (23/50) | 42.0% (21/50) | 40.0% (20/50) | 24.0% (12/50) | 80.0% |
Legacy Export v0 is the original 34-task general + 50-task MCP split, not the current refined benchmark. Q8_0 and Q4_K_M use the June 2.6B Tool GGUF checkpoint. RC1 uses the same engine through Q8_0 and Q4_K_M GGUFs. Hermes 3 uses its owner-specified ChatML and XML function-calling protocol, translated only at Eney's OpenAI boundary.
Diagnostic / non-comparable runs
| Suite | 2.6B RC1 BF16 | 8B A1B 8-bit | Qwen 3.5 9B Q4_K_M | Qwen 3.5 9B Q8_0 |
|---|---|---|---|---|
| Inference server | MLX | MLX | llama.cpp | llama.cpp |
| 34 tasks / non-MCP | 29.4% (10/34) | 26.5% (9/34) | 17.6% (6/34) | Stopped early |
| 50 tasks / MCP | 22.0% (11/50) | Invalid run | 32.0% (16/50) | Not run |
MLX results are retained for investigation but should not be compared directly: the 8B run hit the coordinator's 120-second no-harness-event watchdog before grading. Qwen is also retained as a diagnostic only: its low score and early Q8 behavior suggest a likely prompt/parser mismatch with Eney's generic tool surface.