Knowledge note
Agent Evaluation Benchmarks
Benchmarks and harnesses for evaluating coding agents, tool-use agents, and efficient model evaluation, including DeepSWE, the τ-bench family, AutoJudger, and EssenceBench.
Agent Evaluation Benchmarks
- Pier: Coding-Agent Evaluation Harness — Harbor-compatible coding-agent evaluation harness focused on installed agents in sandboxed tasks, stricter ATIF trajectory conversion,
mini-swe-agentintegration, a DeepSWE-style local-dataset execution path, and a visual architecture explainer. - DeepSWE: Long-Horizon Software Engineering Benchmark — Long-horizon coding benchmark with original tasks, 91 repositories across five languages, behavioral verifiers, a visual explainer, and limitations.
- DeepSWE v1.1: Execution and Scoring Changes — DeepSWE execution-and-grading update with committed-patch isolated verification, CTRF structured test reports, and v1 versus v1.1 impact.
- τ-bench: Tool-Agent-User Interaction Benchmark — Original benchmark: two customer-service domains (retail and airline), pass^k metric, empirical results across 12 models, and failure analysis.
- τ²-Bench: Mechanism and Design — Dec-POMDP formalism, dual-control domain, task generation, and evaluation methodology.
- τ-Voice: Full-Duplex Voice Agent Benchmark — Extends τ²-bench to voice: tick-based orchestrator, controllable voice user simulator, 278 tasks, empirical results, and error analysis.
- AutoJudger: Agent-Driven Efficient MLLM Benchmarking — Agent-driven framework for adaptive MLLM evaluation using IRT difficulty estimation and semantic-aware retrieval.
- EssenceBench: LLM Evaluation with 200x Less Data — Fixed benchmark coresets learned through semantic and behavioral redundancy filtering, genetic subset search, and attribution-guided refinement; includes a correction to the paper’s ambiguous 200x HellaSwag ranking claim.
- From Raw Experience to Skill Consumption: Model-Generated Agent Skills — SkillLens studies the full trajectory-to-skill lifecycle across five domains, identifies negative transfer and extractor-target asymmetry, and turns utility-grounded findings into a validated meta-skill.
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills — Treats one portable skill as the trainable state of a frozen agent, using bounded text edits, strict validation gating, rejected-edit feedback, and epoch-wise slow/meta updates.
- AutoSkill: Experience-Driven Lifelong Learning — Turns recurring user requests into retrieved, versioned
SKILL.mdartifacts through coupled response-generation and background skill-evolution loops; its WildChat study demonstrates extraction scale rather than downstream utility. - HORIZON: Agentic Hardware Design as Repository-Level Code Evolution — Git-traced RTL self-evolution from Markdown harness to executable evaluator, with 100% best-so-far completion but large convergence-cost and reward-hacking caveats.