Rebuilding Playbook Review as a Multi-Agent System | Harvey

Harvey Blog · Veröffentlicht am 2. Sept. 2026

A technical review of a multi-agent system for contract playbook implementation, describing challenges in semantic comprehension of conditional logic, the 'lightest touch' standard for redline edits, an LLM-judged evaluation framework based on lawyer-defined rubrics, and an orchestrator-worker agent architecture for distributed rule review and reconciliation. Lies 4 Standpunkte mit Belegen und Links zu den Originalquellen.

Maharshi Patel, Pablo Felgueres, Zach Huang

Kernaussagen lesen Herausgeber öffnen

Auf einen Blick

Conditional logic in contracts requires semantic comprehension
Contracts contain complex conditional logic: clauses span pages, reference each other, and describe context-dependent triggers. AI agents must grasp meaning and consequences—not just match words—so they distinguish syntactic from semantic differences.
Unterstützendes Moment lesen · Absatz 13
Lightest-touch editing is a critical quality bar
The best redline edit is the smallest legally sufficient change. Rewriting entire clauses instead of making surgical three-word edits slows deals and increases human review burden—a standard called 'lightest touch' that off-the-shelf AI models typically miss.
Unterstützendes Moment lesen · Absatz 16
LLM judges score redline quality against lawyer-defined rubrics
Redline quality is evaluated using lawyer-sourced rubrics that assess minimality, correct placement, and legal soundness. A committee of three frontier LLMs acts as independent judges to score outputs; their votes are aggregated.
Unterstützendes Moment lesen · Absatz 33
Orchestrator-worker pattern balances quality, latency, and complexity
An orchestrator agent distributes rule-specific review tasks to parallel subagents—each functioning like a legal associate—and reconciles results, including conflict resolution. This balances output quality, system latency, and engineering complexity.
Unterstützendes Moment lesen · Absatz 43

Wichtige Passagen4

Zugeordnete Passagen mit dem Kontext zur Überprüfung. Öffnen Sie den Originaltext, um die Quelle zu prüfen.

Absatz 13

contract review complexity

Conditional logic in contracts requires semantic comprehension

Originalauszug

Contracts have complex conditional logic and clauses span pages, reference each other, and describe when certain conditions are triggered and not. And even after creating a playbook, a counterparty’s contract will almost never use your playbook’s exact words. Positions are often qualified with conditional logic. An agent reviewing a clause, therefore, requires comprehending meaning and repercussions of these conditional, syntatic vs semantic differences instead of word matching.

Maharshi Patel, Pablo Felgueres, Zach Huang·2. Sept. 2026

Kontext

Contracts include complex conditional logic:

Absatz 16

redline quality

Lightest-touch editing is a critical quality bar

Originalauszug

The best edit is usually the smallest one: Every redline you send gets scrutinized by an opposing counsel. If a clause needs three words changed and an agent rewrites the entire clause it will both slow down a deal and create extra work for human reviewers. Lawyers call this taking the "lightest touch," and it's a quality bar that out-of-the-box AI models and agents miss.

Maharshi Patel, Pablo Felgueres, Zach Huang·2. Sept. 2026

Absatz 33

evaluation methodology

LLM judges score redline quality against lawyer-defined rubrics

Originalauszug

Redline quality is more complex in that it requires judgment on whether an edit is minimal, correctly placed, and legally sound . After sourcing the rubrics with lawyers, we set up an evaluation framework that uses LLM judges to score against these rubrics. We then used a committee of three frontier models to score independently and aggregate the votes.

Maharshi Patel, Pablo Felgueres, Zach Huang·2. Sept. 2026

Kontext

Flagging risk is best understood and scored as a classification problem which is a straightforward task.

Absatz 43

multi-agent architecture

Orchestrator-worker pattern balances quality, latency, and complexity

Originalauszug

We proved with a single agent prototype that quality problems can be solved with an agent. Often the best way to improve on latency is to find ways to parallelize work. With that idea in mind, we implemented an orchestrator-worker pattern consisting of a lead agent in charge of distributing work between parallel agents. Each of these agents in turn focus on a specific task, in this case reviewing a rule, and finally the lead agent reconciles the results to ensure final output is of good quality.

Maharshi Patel, Pablo Felgueres, Zach Huang·2. Sept. 2026

Quelle & Methodik

Diese Standpunkte sind mit ihren Originalquellen verknüpft. Paraphrasen sind gekennzeichnet und keine wörtlichen Zitate.

Transkript oder Quellenmaterial öffnen Ein Problem melden