From Agents to Harnesses 2 - Domain Specific Harnesses

01 · The turnThe software is not the code. It is the harness.

We have worked in premier agent harnesses for some time: Claude Code, Codex, and Cursor. One conviction became strong. The future of domain software is not static legacy code. It is also not free-form “vibe” code. It is a harness around an LLM operating system for one domain. That harness holds natural-language procedures, rules, parallelization hints, and scripts that the agent composes on demand.

The new standards from Anthropic make this concrete. Agent Skills go into plugins. One plugin holds skills, slash-commands, sub-agents, scripts, MCP connectors, and deterministic hooks. With a plugin we can harness the harness itself. The result is a higher-order, domain-specific configuration of a general agent.

In Gradual Determinism we argued for the governance of a spectrum. That spectrum goes from soft natural language that needs judgement, to hard invariants that a machine enforces. We argued that you move rules along it deliberately, not all at once. This piece is the applied sequel.

τ²-bench gives us a laboratory with a scoreboard. It has three customer-service domains, real policies, real tools, and an evaluator that grades the trajectory. We will build its agents again as harnesses, and let the numbers speak.

02 · The motivating anecdote“The API does not check this for the agent.”

Open the policy of the airline domain. A confession appears three times, word for word:

“The API does not check that cancellation rules are met, so the agent must make sure the rules apply before calling the API!” data/tau2/domains/airline/policy.md

The policy is a small model of the full problem. Its 166 lines contain four different kinds of rule. Each kind needs a different treatment:

Now look at what the legacy agent does with all of it. It puts the full policy into one system prompt, one time. Then it tells the model to be careful:

# src/tau2/agent/llm_agent.py
SYSTEM_PROMPT = """
<instructions>{agent_instruction}</instructions>
<policy>{domain_policy}</policy>
"""

# every turn: system_prompt + history -> one generate() call.
# the model must re-derive the $100-vs-$50 matrix, the 24h window,
# and "basic economy cannot be modified" from memory, each time.

This is the determinism spectrum at one single point. An exact formula, a hard invariant, and a matter of taste all use the same medium: prose. The same mechanism enforces all three: the hope that the model remembers. The policy repeats its plea three times, and that repetition is the signal. The author knew that a guardrail was absent. The author could only ask the model, in English, to supply it.

A rule that the system “must make sure of” but never checks is not a rule. It is a wish.

03 · The reframeOne policy, four altitudes, five primitives.

A harness puts each rule where it belongs on the determinism spectrum. We no longer ask one prompt to be a judge, a calculator, and a guard at the same time. We route each concern to the plugin primitive that is made for it.

Here is how the policy of one domain becomes one Claude Code plugin:

The hook is the heart of it. The legacy agent fails when it calls cancel_reservation on a booking that is not eligible. A pre-tool hook changes the plea of the policy into a mechanism:

# hooks/precheck_cancel.py — PreToolUse on cancel_reservation
def precheck(tool_call):
    r = get_reservation(tool_call.args["reservation_id"])
    if any_segment_flown(r):
        return deny("flown segment — transfer required")
    if not (booked_within_24h(r) or r.airline_cancelled
            or r.cabin == "business" or insured_and_covered(r)):
        return deny("cancellation rules not met")
    return allow()   # now the API call is safe by construction

04 · The wagerWhy a harness should out-score a prompt.

τ²-bench does not grade vibes. Its evaluator scores each trajectory against the evaluation_criteria of the task. It asks three questions. Did the correct tool actions fire? Did the database end in the correct state? Did the run obey the policy?

So you can falsify our claim. We will report it as a pass-rate delta, with the same models and the same tasks. We think that four mechanisms will move that number:

Less to hold at one time. With progressive disclosure, the model reasons over the cancellation procedure during a cancellation. It does not hold the booking, baggage, and insurance rules at the same time. The live context is smaller, and there are fewer mistakes from cross-contamination.

Arithmetic leaves the token stream. Today the evaluator scores three things as failures: a confusion between $50 and $100, an incorrect refund delta, and an off-by-one baggage allowance. A script cannot make these errors.

The agent cannot violate an invariant. The most damaging failure is a write that the policy forbids. A hook stops that write before it gets to the tool. The model no longer must remember the rule. It cannot break it.

We do not promise a number in advance. This is a wager with a scoreboard. If the harness does not beat the monolith on pass-rate, the thesis is wrong, and we will say so. We think that the harness will win. We also think that the margin increases on the hard, conditional tasks that contain much arithmetic. A flat prompt is weakest on those tasks.

05 · The longer gameHarnesses that watch themselves, and evolve.

A harness is more accurate, and it is also observable. Each skill that loads is a structured event. So is each script that computes, and each hook that permits or denies. We add that observability from the start, then we complete the loop. τ² supplies the other half: a verdict on each run.

This is where the gradual-determinism thesis gives its dividend. A class of failures occurs again. The fix is not a new roll of a large prompt and a prayer. The fix is a local and clear edit at a known altitude. It is a new precondition in one hook, a clearer step in one skill, or a corrected constant in one script.

The harness becomes a live record of domain knowledge that we obtained with difficulty. The agent itself can propose the patch. It can justify the patch from the trace. It can validate the patch against the scoreboard before the change lands.

That is the destination: domain-specific harnesses that write and improve themselves. They add determinism exactly where the evidence demands it, and no sooner. The software improves itself. You can audit each improvement, and you can reverse it. Measured outcomes support it, not vibes.

06 · What we are buildingThe plan, in one breath.

We ship one Claude Code plugin for each τ²-bench domain: airline, retail, and telecom. The plugin has skills that divide the policy by altitude. It has scripts that own the arithmetic. It has sub-agents for isolated audit. It uses the existing MCP servers of the fork as the tool connector. It has slash-commands as entry points, and hooks that make the “must make sure” clauses true by construction.

Then we run the plugin against the legacy LLMAgent, with the same models and the same tasks. We report the pass-rate delta plainly. Then we instrument the plugin, and we let it start to improve itself.

We do not write better agents. We build the harness in which a capable agent becomes a domain expert, and then learns.