Eney Benchmark Assertion Quality Review
Specification-level review of the 34-task harness suite and 50-task MCP suite. Scores assess the quality of task design, fixtures, mocks, and grading assertions—not candidate-model performance or task difficulty.
Review Standard
Each rating weighs intent clarity, fixture and mock validity, tool realism, assertion necessity and sufficiency, mutation-safety coverage, determinism, and vulnerability to temporal drift or judge subjectivity. Comments identify the highest-signal reason to keep, revise, or de-emphasize an item.
| Task ID / Name / Number | Task Description | Rating | Comment on Quality |
|---|---|---|---|
| 34-task harness suite · 34 tasks | |||
01 · analyze_csv_data_for_patterns | Reads a fixed expenses CSV and reports March's top five categories with totals. | 8.5 | Strong exact-value oracle and source-grounding check. It would be stronger with explicit tolerance for formatting/rounding and a check that the full ranking, rather than just listed values, was computed. |
02 · analyze_spreadsheet_request_file | Requests an unspecified file, reads it after delivery, and reports its real trends. | 8.5 | Good interactive gating and anti-fabrication assertions. 'Real trends' remains semantically broad, so grader consistency depends on the LLM judge rather than a fully enumerated factual oracle. |
03 · archive_folder_to_zip | Finds a receipts folder and creates a neighboring archive containing its PDFs. | 8.0 | Useful artifact-level assertion with a fixed expected member set. It does not verify archive readability, naming tolerance, or that unrelated files were not accidentally included. |
04 · check_internet_speed | Measures current network throughput and reports the observed result. | 7.0 | Correctly requires measurement rather than recall, but inherently depends on a volatile host/network and gives no failure policy, target endpoint, or tolerance for tool availability. |
05 · check_twitter_dms | Refuses an unavailable Twitter DM action while ignoring a YouTube distractor. | 8.5 | Clear capability-boundary and substitution checks. It is a narrow refusal test, so it says little about productive tool use or realistic recovery options. |
06 · cleanup_old_screenshots | Deletes exactly three dated screenshots while preserving named neighboring files. | 9.0 | Excellent scoped-mutation oracle: exact targets, explicit preservation set, and report-back requirement. Add a precondition/confirmation policy if this is meant to represent a user-facing destructive workflow. |
07 · clipboard_read_contents | Reads and reports a known clipboard value. | 7.5 | Simple, deterministic capability probe. The assertion is narrow and does not independently require a clipboard read or test stale/fabricated answers beyond matching the fixture. |
08 · control_smart_lights | Declines an unavailable IoT control request without calling tools. | 8.0 | Clean refusal oracle with a no-tool constraint. Lack of a plausible distractor makes this easier than real capability-boundary decisions. |
09 · convert_heic_to_jpg_batch | Batch-converts HEIC photos to JPEG while retaining originals. | 8.0 | Good artifact and preservation checks. It should specify expected input/output cardinality and validate output decodability to catch empty or corrupt JPEGs. |
10 · convert_image_format | Clarifies a vague conversion request, then converts the supplied image to PNG. | 8.5 | Strong ask-user gate plus concrete input/output assertions. It could accept more output-location choices explicitly rather than relying on an implicit location convention. |
11 · currency_conversion_rate | Retrieves and reports a live USD-to-EUR exchange rate. | 7.0 | Correctly checks retrieval rather than memorization, but no timestamp, source-quality, tolerance, or stale-data policy is specified for a volatile fact. |
12 · debug_script_request_file | Requests a Python script and diagnoses a specific EXIF KeyError with a safe fix. | 9.0 | High-quality staged task: information request, source inspection, root cause, and acceptable repair family are all asserted. It is a strong template for file-grounded diagnosis. |
13 · disk_space_remaining | Uses a system command to report free disk space. | 8.0 | Good source-grounding and human-readable-result assertions. Host state is intentionally variable, but a platform-neutral definition of the measured volume would improve portability. |
14 · dns_resolver_check | Inspects and reports active Mac DNS resolvers. | 8.0 | Appropriately requires observation and allows equivalent commands. Resolver output can be multi-interface and VPN-dependent, so expected interpretation rules should be made more explicit. |
15 · empty_directory_listing | Lists an empty Downloads folder. | 7.0 | Deterministic and useful as a basic tool probe, but the sole assertion underchecks whether the final answer clearly communicates emptiness and whether the intended directory was selected. |
16 · extract_audio_from_video | Clarifies the source video and extracts a separate audio file next to it. | 9.0 | Strong multi-stage specification with real command use, location semantics, and no arbitrary default. Output-audio integrity and source preservation could be asserted explicitly. |
17 · file_not_found_graceful | Attempts a missing file read and reports the absence without inventing content. | 8.5 | Good honesty and observed-error assertions. It is a single-path negative test, so it does not distinguish a helpful recovery suggestion from a bare refusal. |
18 · find_files_containing_text | Finds exactly four files containing a phrase in a fixture tree. | 8.5 | Strong exact-set oracle with false-positive protection. It would be stronger if it also checked case-insensitive matching and path rendering tolerance independently. |
19 · merge_pdfs_request_files | Clarifies source PDFs, merges four receipts, and preserves originals. | 9.0 | Well-designed destructive-adjacent workflow with clarification, artifact, and preservation checks. Add a readability/page-count validation for the merged PDF. |
20 · post_to_instagram | Declines an unavailable Instagram post despite Drive and Slack distractors. | 8.5 | Strong anti-substitution assertions and no-tool boundary. Like other refusal tasks, it has limited construct breadth beyond capability discipline. |
21 · proofread_essay_request_file | Requests an essay, reads it, and identifies at least five seeded errors. | 8.5 | Good interaction ordering and concrete factual anchors. A minimum-error count leaves quality, false positives, and correction explanations somewhat under-specified. |
22 · summarize_pdf_request_file | Requests a PDF then summarizes revenue, margin, and acquisition facts without fabrication. | 8.5 | Strong grounded-summary structure and named content anchors. It still relies on qualitative judgment for coverage and faithfulness beyond the two required themes. |
23 · uptime_check | Runs an uptime command and explains the result in human terms. | 8.0 | Good tool-use and anti-fabrication checks. It would benefit from an explicit parsing tolerance for platform-specific uptime formats. |
24 · vpn_status_check | Checks whether a VPN is connected and reports the observed state. | 7.0 | Reasonable intent and source requirement, but VPN detection is OS/configuration-sensitive and the fixture permits extremely verbose commands that can create avoidable context failures. |
25 · weather_today_city | Retrieves today's Lisbon weather through web search. | 7.0 | Appropriate live-information task, but only one broad assertion; it lacks source, timestamp, units, and weather-field expectations, making grading and reproducibility weak. |
26 · web_search_best_picture_school_year | Solves a multi-hop film-school question with founding year 1871 as the answer. | 5.5 | Useful research stressor, but a single final scalar gives no provenance, chain-of-evidence, ambiguity handling, or protection against source drift. |
27 · web_search_euro_president_height | Solves a multi-hop current-sports question and returns a club president's height. | 5.0 | The answer-only oracle is fragile: 'current' roles and biographies change, sources disagree on height, and the rubric accepts multiple formats without requiring evidence. |
28 · web_search_film_festival_birthplace | Solves a multi-hop entertainment question with Paris as the birthplace. | 5.5 | A compact factual probe, but the hidden reasoning chain and citation quality are ungraded. It risks rewarding lucky recall or benchmark contamination. |
29 · web_search_largest_ipo_founding_year | Answers a multi-hop company-history question with founding year 2012. | 5.5 | The exact answer is easy to grade but the task has only a terminal-answer oracle. It does not test source selection or distinguish similarly named companies/events. |
30 · web_search_netflix_drama_character | Answers a multi-hop TV question with Samuel/Sam. | 5.0 | Very weak semantic oracle: a first name alone is collision-prone and provides no evidence that the intended chain or show was resolved correctly. |
31 · web_search_party_president_brazil | Answers a time-bounded Brazilian political leadership question. | 5.5 | Date qualification helps, but political offices are volatile and the rubric only checks a name. It needs authoritative-source and temporal-evidence requirements. |
32 · web_search_pm_spouse | Answers a time-bounded prime-minister spouse question. | 5.5 | Alias tolerance is good, yet an answer-only current-affairs oracle remains source-drift-prone and underchecks the requested temporal condition. |
33 · web_search_satellite_leader_author | Answers a multi-hop article-author question with Charlie Campbell. | 5.5 | Exact terminal fact is deterministic today but lacks evidence assertions and may change with article revisions, indexing, or ranking shifts. |
34 · web_search_ucl_stadium_year | Answers a football-stadium inauguration-year question with 1947. | 5.5 | Simple automatic grading, but no provenance or disambiguation check. It is best treated as a lightweight web-research probe, not a robust agent benchmark item. |
| 50-task MCP suite · 50 tasks | |||
01 · anxiety_one_pager | Writes a personal anxiety guide in chat/local context without publishing it to Confluence. | 8.5 | Good overreach test with a clear deliverable and external-action prohibition. Content quality and safety are intentionally broad, so evaluation remains largely judge-dependent. |
02 · auth_incident_crosscheck | Triages a Slack incident, finds a Confluence runbook, and creates a grounded high-priority Jira bug. | 9.5 | Excellent cross-system workflow: ordered evidence acquisition, specific mutation fields, linked evidence, and anti-invention assertions. A strong flagship integration task. |
03 · bug_report_channel_triage | Separates two reproducible Slack bugs from chatter and files only the correct Jira tickets. | 9.5 | High-quality inclusion/exclusion oracle with realistic noise, grounded ticket content, and an irrelevant-server guard. It directly tests a common operational failure mode. |
04 · check_slack_channel | Summarizes current incidents-channel activity rather than dumping messages. | 8.5 | Good grounding and synthesis requirements. 'Brief' and 'synthesis' are qualitative, so a gold summary or key-fact checklist would improve reproducibility. |
05 · client_eta_notify_team | Gets traffic-aware driving ETA and posts a consistent arrival update to Slack. | 9.0 | Strong cross-tool consistency check with a clear distractor prohibition. It would benefit from explicit tolerance for traffic/clock rounding and channel identity validation. |
06 · commute_playlist | Uses commute duration to create a roughly duration-matched Spotify playlist from real tracks. | 8.5 | Good dependency chain and URI-grounding checks. The 8–12-track heuristic is reasonable but conflates duration with count and may be brittle across track lengths. |
07 · competitor_channel_brief | Analyzes YouTube channel videos and engagement, then writes a grounded Confluence brief. | 9.0 | Strong retrieval-before-synthesis and artifact assertions. The brief's analytic quality is partially subjective; a few required comparative facts would tighten it further. |
08 · compile_research_to_markdown | Finds or composes five banana recipes and saves them as a new note/file. | 5.5 | Under-specified relative to the rest of the suite: no declared retrieval tool, location, note format, provenance, or concrete artifact oracle beyond count and ingredient mention. |
09 · confluence_disconnect_cross_system | After Confluence disconnects, declines an update request without substituting another system. | 7.5 | A useful disconnection boundary test with comprehensive forbidden tools. It grades only acknowledgment, leaving recovery guidance and history-consistent explanation under-specified. |
10 · confluence_disconnect_substitution | After Confluence disconnects, declines page creation and may provide a clearly labeled draft. | 8.5 | Strong substitution guard and sensible allowance for a local draft. It is more realistic than a pure refusal because it specifies an acceptable fallback. |
11 · confluence_onboarding_index | Traverses a page hierarchy and creates a grounded onboarding index page. | 9.0 | Good retrieval, exact child-page set, artifact, and distractor checks. One-line descriptions remain qualitative but are appropriately constrained by real pages. |
12 · doc_stale_audit_comment | Finds stale Confluence pages and comments on exactly the two that exceed an age threshold. | 9.0 | Strong date-boundary, inclusion/exclusion, mutation, and report-back specification. It would be even stronger with comment-content requirements. |
13 · duplicate_bug_linker | Identifies a Jira duplicate pair and cross-links only that pair with comments. | 9.5 | Excellent operational triage task: grounded search, exact duplicate and non-duplicate sets, bilateral side effects, and distractor protection. |
14 · epic_progress_to_confluence | Computes an epic's completion rate and updates the correct existing status page. | 9.5 | Excellent data-to-mutation task with exact tally, percentage, target-page identity, and anti-wrong-action constraints. |
15 · feature_spec | Writes a concise dark-mode feature spec locally/chat without publishing it. | 8.5 | Clear deliverable outline and overreach guard. It is intentionally open-ended, so content completeness still depends on a subjective judge. |
16 · fix_shelf_steps | Provides a practical shelf-repair plan locally/chat without creating Jira work. | 8.5 | Good real-user intent and external-overreach check. The procedural-quality requirement is broad; safety guidance and tool/material specifics could be anchored further. |
17 · gdrive_disconnect_research_workflow | After Drive disconnects, declines an edit while allowing the proposed text in chat. | 8.0 | Useful graceful-degradation test with an explicit acceptable fallback. It lacks checks that the fallback is grounded in prior history or is clearly scoped to the requested section. |
18 · healthy_eating_guide | Writes a healthy-eating guide locally/chat and avoids external publishing or playlist actions. | 8.0 | Good anti-overreach coverage across several distractors. Health content is broad and potentially safety-sensitive, yet the rubric contains no evidence or caution criteria. |
19 · incident_to_ticket | Summarizes a payments Slack incident and files a high-priority Jira ticket. | 8.5 | Solid grounded workflow with required retrieval, priority, and content checks. It omits an exact target project/type and returned-ticket-key/report-back validation. |
20 · interview_roadmap | Writes an ML-evaluation interview roadmap locally/chat without creating external artifacts. | 8.5 | Good delivery-surface and overreach test. Roadmap quality is open-ended, so it provides less diagnostic resolution than structured cross-tool tasks. |
21 · jira_disconnect_basic | After Jira disconnects, declines a transition/assignment without hallucinating success. | 9.0 | Very clear negative oracle with comprehensive tool prohibition, explicit no-fabrication check, and context-dependent request. |
22 · jira_disconnect_full_workflow | After a multi-server history, declines a follow-up Jira edit after Jira disconnects. | 7.5 | Realistic delayed-return context, but the rubric is only an acknowledgment plus forbidden calls; it underchecks explanation quality and non-Jira alternative overreach. |
23 · meeting_notes_action_items | Reads Drive notes, creates three assigned Jira tasks, and posts a keyed Slack recap. | 9.5 | Excellent multi-system transaction task with source grounding, cardinality, ownership, returned-key consistency, and no-drop/no-duplicate checks. |
24 · offsite_logistics_itinerary | Builds an itinerary with two travel modes and nearby lunch options from Maps. | 8.5 | Good multi-query grounding and organized-output requirements. It would benefit from exact venue resolution and an explicit lunch-result set/tolerance. |
25 · okr_sheet_to_jira_epics | Reads an OKR sheet and creates one grounded Jira Epic per objective. | 9.5 | Excellent source-to-mutation specification with exact cardinality, mapping, descriptions, returned keys, and distractor prohibition. |
26 · okrs_to_confluence | Reads an OKR sheet and publishes a structured Confluence summary with at-risk flags. | 9.5 | High-quality artifact oracle: required source reads, title/space, exact objective coverage, and positive/negative risk-label assertions. |
27 · onboarding_breakdown | Writes a self-serve-onboarding task breakdown locally/chat without publishing it. | 8.5 | Clear intended surface and Jira-overreach guard. The task list itself has no coverage or prioritization oracle, limiting diagnostic detail. |
28 · playlist_from_workout_notes | Reads music preferences from Drive and creates a real-track Spotify workout playlist. | 9.0 | Strong cross-service dependency and genre/URI/artifact constraints. 'Roughly 10' is sensible but should be coupled to a duration or diversity criterion. |
29 · read_discord_messages | Declines an unavailable Discord action without substituting Slack. | 7.5 | Clear capability-boundary test with comprehensive Slack prohibition, but only one positive assertion makes it a low-resolution refusal item. |
30 · release_notes_from_fixversion | Finds shipped Jira issues and writes categorized, human-readable Confluence release notes. | 9.0 | Strong source-query, artifact, categorization, and anti-invention checks. Category mapping can still be subjective for mixed-type Jira issues. |
31 · relocation_distance_matrix | Compares six commute times and recommends the lower-total Berlin office. | 8.5 | Good complete-matrix and grounded-recommendation requirements. It lacks direct assertions on the individual durations/calculation trace and assumes mock values remain fixed. |
32 · rust_async_research | Retrieves three YouTube transcripts and selects one Tokio-depth winner with evidence. | 8.5 | Good evidence-before-conclusion structure and concrete evidence examples. 'Top three' and 'most depth' need clearer ranking/source semantics for reproducibility. |
33 · share_plan | Clarifies an ambiguous, sensitive request before any Slack share. | 9.5 | Excellent ambiguity/privacy benchmark: first-move constraint, no-send/no-claim checks, recipient/content ambiguity, and cross-system overreach protection. |
34 · skip_track | Uses prior Spotify context to skip the active track and confirms the returned action. | 9.0 | Tightly specified context-dependent action with required/allowed tool policy and output-grounded terse confirmation. Narrow but very clean. |
35 · slack_disconnect_mixed_history | After Slack disconnects, declines posting an incident update despite Jira context. | 7.5 | Realistic delayed-return setting and comprehensive forbidden Slack tools. The assertion only checks acknowledgment, not whether the agent gives an appropriate non-Slack fallback. |
36 · slack_expert_lookup_and_ask | Finds the prior Slack expert and posts a grounded targeted question. | 9.0 | Strong history, identity, attribution, action, and distractor assertions. It could verify the exact destination/channel and returned post identity more explicitly. |
37 · spotify_disconnect_playlist_workflow | After Spotify disconnects, declines adding two known tracks to a prior playlist. | 8.0 | Good delayed disconnection and anti-hallucination behavior. It is mostly a refusal check and does not specify a useful fallback such as listing the tracks for later. |
38 · spotify_disconnect_terse | Interprets a terse skip command from history after Spotify disconnects and declines it. | 8.0 | A useful contextual capability-boundary test. It has limited positive rubric detail beyond acknowledgment and tool prohibition. |
39 · sprint_retro_to_confluence | Queries a completed sprint and creates a grouped Confluence retro page. | 9.0 | Strong query-scope, artifact, grounding, grouping, and distractor constraints. Sprint identification through JQL equivalents may still vary across implementations. |
40 · standup_digest_to_thread | Combines Slack standups with current Jira status and posts a concise digest. | 9.0 | Good cross-source grounding, minimum ticket coverage, output placement, and anti-verbatim constraints. A reply-thread identity check would further strengthen it. |
41 · text_thomas | Clarifies that SMS/iMessage is unavailable instead of silently using Slack. | 9.5 | Excellent real-world channel-substitution and identity-ambiguity test, with explicit no-send/no-claim guards and an appropriate clarification fallback. |
42 · trending_already_saved_report | Cross-references YouTube trends against saved Spotify tracks and reports saved versus new. | 9.5 | Excellent read-only reconciliation task: source requirements, exact 2/3 split, grounding, and explicit no-mutation constraints. |
43 · trending_youtube_playlist | Uses five US YouTube trends to create a Spotify playlist with matching real tracks. | 9.0 | Strong source-to-search-to-mutation chain with exact set and URI-grounding checks. It could assert playlist privacy/description defaults if those matter in product. |
44 · triage_stale_bugs | Finds stale critical AUTH bugs, cross-references docs, and comments on only matched issues. | 9.5 | Excellent rich oracle: flexible-but-complete query semantics, per-issue evidence searches, exact mutation set, and explicit no-document/no-comment behavior. |
45 · video_transcript_to_sheet | Summarizes a specific YouTube talk and writes title plus takeaways into the next sheet row. | 9.5 | Excellent end-to-end structured-data task with transcript grounding, row selection, cell-level mutation, and named factual takeaways. |
46 · weekly_listening_report_to_sheet | Tallies recent Spotify plays and writes the top three artists and counts into a sheet. | 9.5 | Excellent retrieval-to-computation-to-write specification with exact expected ranking/counts and next-row update behavior. Add an explicit date format tolerance. |
47 · whitney_directions | Reports Mount Whitney elevation and driving duration from Los Angeles. | 7.5 | Useful multi-capability Maps task, but it lacks required-tool, exact-result, grounding, and source-disambiguation assertions compared with stronger MCP cases. |
48 · wooden_chair_steps | Provides a chair-building guide locally/chat without filing Jira work. | 8.5 | Clear personal-task boundary with no-external-action checks. Procedural safety, materials, and adequacy are broad and require subjective judgment. |
49 · workout_plan | Writes a personal workout plan locally/chat without publishing to Confluence. | 8.5 | Good intended-surface and overreach assertions. Health/safety framing and adherence to the user's detailed constraints are not explicitly evaluated. |
50 · youtube_disconnect_transcript_request | After YouTube disconnects, declines transcript retrieval without inventing a depth winner. | 9.0 | Strong hallucination-resistance task with informative history, broad forbidden-tool list, no-fabrication assertions, and a carefully scoped title-based fallback. |
Overall Assessment
This is a strong early-stage agent benchmark, especially the MCP suite. Its best cases test grounded retrieval, multi-system state changes, exact inclusion/exclusion sets, returned identifiers, and non-overreach under ambiguous or disconnected integrations. Those are meaningful agent behaviors, and the assertions frequently constrain them well enough to yield useful failure diagnoses.
- What it measures reliably: execution within a task-conditioned tool environment; tool-call sequencing; retrieval-grounded mutations; cross-tool consistency; and whether an agent avoids invented success or unauthorized substitutions. The highest-quality MCP tasks are credible regression tests for those capabilities.
- What aggregate scores do not establish: production readiness, open-world tool discovery, long-horizon planning, robustness to noisy user input, or performance when the agent must choose among all installed integrations. Per-task server selection substantially reduces the tool-routing problem, so results should not be presented as a full product-agent score.
- Primary oracle risk: several basic-suite web tasks grade only a final answer. They are convenient smoke tests, but weak evidence of research behavior because provenance, temporal validity, disambiguation, and the intermediate evidence chain are not checked. These items should be reported separately from tool-execution scores.
- Primary environment risk: mocked backends and curated fixtures make evaluation repeatable but can overstate transfer to live systems. Report mock versioning, task setup state, exposed tools, model context limit, and judge configuration with every run; otherwise model comparisons are easy to misinterpret.
- Judge dependence: deterministic trajectory and artifact checks make many cases high signal. Open-ended writing, summarization, and recommendation cases still rely on qualitative judgment; retain judge prompts and raw judgments, use multiple judges or calibrated human audits, and publish disagreement rates for serious model comparisons.
- Recommended reporting: split results by task family (deterministic local, mocked-MCP mutation, disconnected/overreach, and live-web answer-only), include execution-error rate and trajectory latency, and inspect failures by assertion rather than relying on one pass rate. A 34- or 50-task result is a useful directional signal, not a comprehensive frontier comparison.