Hyper-𝜏-bench: Evaluating agents that build agents

Sierra Blog · Publicado el 8 sept 2026

The article contrasts evaluating customer-service agents with evaluating models that build agents. It reports standalone and engineer-paired task performance, and relates client questioning to success on benchmark tasks. Lee 3 puntos de vista con sus evidencias y enlaces a las fuentes.

Ben Shi, Keshav Dhandhania

Leer los puntos clave Abrir Editor

De un vistazo

From customer service agent to agent builder
The field has moved beyond evaluating whether models can act as reliable customer service agents (𝜏-bench's original focus) to evaluating whether models can themselves build such agents.
Ver el momento de apoyo · Párrafo 1
Reported success with engineer context
The authors report 23.9% success on held-out tasks for their best standalone configuration, Claude Opus 5 with max reasoning in Claude Code. Paired with an engineer with deep context, the same class of model reaches 82.2% on those tasks.
Ver el momento de apoyo · Párrafo 6
Client questioning strongly correlates with agent-building success
On tasks where the simulated client held sole context for 20–25 requirements, developer agents that asked zero questions scored 5%, one question 15%, two questions 25%, and so on — demonstrating a direct payoff from client interviews.
Ver el momento de apoyo · Párrafo 9

Pasajes clave3

Pasajes atribuidos con contexto para verificarlos. Abra el texto original para comprobar la fuente.

Párrafo 1

agent evaluation scope

From customer service agent to agent builder

Extracto original

We built 𝜏-bench in 2024 to answer a question that felt novel at the time: Can a model act as a reliable customer service agent? That’s table stakes now. The harder question is, who’s building the agent in the first place? Increasingly, it’s the models themselves.

Ben Shi, Keshav Dhandhania·8 sept 2026

Párrafo 9

client interviewing behavior

Client questioning strongly correlates with agent-building success

Extracto original

They don’t interview the client. Developers only asked four questions at most for tasks where the client had sole context for 20-25 requirements. Asking pays off directly. For tasks where reference agents (built by an engineer) scored 95–100% — the builds that asked zero questions scored 5%, one question 15%, two questions 25%, and so on.

Ben Shi, Keshav Dhandhania·8 sept 2026

Párrafo 6

human-AI collaboration performance

Reported success with engineer context

Extracto original

Working alone, our best configuration — Claude Opus 5 (max reasoning) running in Claude Code — passes just 23.9% of the held-out evaluation tasks. Paired with an engineer with deep context, the same class of model reaches 82.2% on the same tasks.

Ben Shi, Keshav Dhandhania·8 sept 2026

Fuente y metodología

Estas perspectivas enlazan a sus fuentes originales. Las paráfrasis están identificadas y no son citas textuales.

Abrir transcripción o material de origen Reportar un problema