Hyper-𝜏-bench: Evaluating agents that build agents

Synthesis

1. 01

   From customer service agent to agent builder

   The field has moved beyond evaluating whether models can act as reliable customer service agents (𝜏-bench's original focus) to evaluating whether models can themselves build such agents.

   Supporting evidence 1

   Original excerpt

   We built 𝜏-bench in 2024 to answer a question that felt novel at the time: Can a model act as a reliable customer service agent? That’s table stakes now. The harder question is, who’s building the agent in the first place? Increasingly, it’s the models themselves.

   Ben Shi, Keshav Dhandhania · Paragraph 1

   Hyper-𝜏-bench: Evaluating agents that build agents · Sep 8, 2026

   Read in source context →

   Continue exploring

   agent evaluation scope →
2. 02

   Reported success with engineer context

   The authors report 23.9% success on held-out tasks for their best standalone configuration, Claude Opus 5 with max reasoning in Claude Code. Paired with an engineer with deep context, the same class of model reaches 82.2% on those tasks.

   Supporting evidence 1

   Original excerpt

   Working alone, our best configuration — Claude Opus 5 (max reasoning) running in Claude Code — passes just 23.9% of the held-out evaluation tasks. Paired with an engineer with deep context, the same class of model reaches 82.2% on the same tasks.

   Ben Shi, Keshav Dhandhania · Paragraph 6

   Hyper-𝜏-bench: Evaluating agents that build agents · Sep 8, 2026

   Read in source context →

   Continue exploring

   human-AI collaboration performance →
3. 03

   Client questioning strongly correlates with agent-building success

   On tasks where the simulated client held sole context for 20–25 requirements, developer agents that asked zero questions scored 5%, one question 15%, two questions 25%, and so on — demonstrating a direct payoff from client interviews.

   Supporting evidence 1

   Original excerpt

   They don’t interview the client. Developers only asked four questions at most for tasks where the client had sole context for 20-25 requirements. Asking pays off directly. For tasks where reference agents (built by an engineer) scored 95–100% — the builds that asked zero questions scored 5%, one question 15%, two questions 25%, and so on.

   Ben Shi, Keshav Dhandhania · Paragraph 9

   Hyper-𝜏-bench: Evaluating agents that build agents · Sep 8, 2026

   Read in source context →

   Continue exploring

   client interviewing behavior →