If you're not the model, you're the harness

Take one question, one model, and two configs that differ only in the debate rules. In config A the skeptic speaks first and gets one round. In config B the advocate speaks first, both sides get three rounds, and the skeptic can read the research notes. Run each config five times. If A and B disagree with each other more than A disagrees with itself, the harness changed the answer. The model stayed the same.

Public numbers show the same thing. One open-weight coding model scored 41.0 on Terminal-Bench 2.0 in the benchmark's own reference harness and 32.8 inside a popular coding CLI. Those are the lab's own runs, and a public leaderboard run of the CLI setup came in at 33.3, so call it roughly eight points rather than a precise gap. Same weights, eight points apart, and the big difference was the code around them.

There's a saying going around agent engineering circles: if you're not the model, you're the harness. I think it's right. At least one big analyst firm's 2027 trend list now has the coding agent harness in its top ten, and its advice is to give it a dedicated owner. Good, that's what the table below is for. This article is for engineers building agents, which mostly means building everything around the model.

The example is a small research desk: two agents argue a question, a judge writes a memo, a human decides. Every step starts with how the desk breaks, then adds one harness piece, one snippet and one row in this table. Keep it open while you read:

Config and numbers below are illustrative.

Step 1: Notice the harness

How it breaks: two people run "the same agent" and get different answers, and nobody can say why, because the rules live in someone's head.

# variant-a.yaml
debate:
  first_speaker: skeptic
  max_rounds: 1
  skeptic_sees_research: false

# variant-b.yaml
debate:
  first_speaker: advocate
  max_rounds: 3
  skeptic_sees_research: true

for variant in ["variant-a.yaml", "variant-b.yaml"]:
    answers = [run_debate(new_run_id(), QUESTION, load_config(variant)) for _ in range(5)]
    print(variant, Counter(a.decision for a in answers))

Everything that differs between those outputs lives outside the model: the loop, what each agent can see, what it may do, how turns are coordinated, and where it runs. Each step below pins down one of those.

Step 2: Roles and permissions

How it breaks: a debater that was only supposed to read notes can also browse the web, and one day it quotes a page nobody vetted.

# desk.yaml
desk: research-debate-example
debate:
  first_speaker: advocate
  alternate_first_speaker: true
  max_rounds: 3
roles:
  researcher: {count: 2, model: fast-cheap-model, tools: [web_search, read_file]}
  advocate:   {model: strong-model, tools: [read_notes], stance: "argue FOR adoption"}
  skeptic:    {model: strong-model, tools: [read_notes], stance: "argue AGAINST adoption"}
  judge:      {model: other-family-model, tools: [read_transcript], output: decision_memo}
identity: {per_role: true, ttl_minutes: 30}     # each role gets its own short-lived credential
tools:
  read_file:       {allow: ["runs/{run}/notes/**"]}
  read_notes:      {allow: ["runs/{run}/notes/**", "runs/{run}/scratch/{role}/**"]}
  read_transcript: {allow: ["runs/{run}/transcript.jsonl"]}
  web_search:      {snapshot: "2026-06-30"}     # see Step 7
approver: question_owner

The example question: "Is this new open-source inference library worth adopting for our stack?" A tool list in a prompt is a suggestion; the allowlist has to be enforced by the tool itself:

def read_file(path, role, run_id):
    allowed = [p.format(run=run_id, role=role) for p in cfg.tools.read_file.allow]
    if not any(fnmatch(path, pattern) for pattern in allowed):
        raise PermissionDenied(f"{role} may not read {path}")
    return open(path).read()

The identity line matters more than it looks. If every role shares one API key, the log can only say "the desk did it". With one short-lived credential per role, a bad read or an odd web search points at the researcher or the skeptic, and revoking one role doesn't take down the others. Agent identity is becoming its own field for exactly this reason.

Step 3: The loop and when it stops

How it breaks: the debate never stops, or it stops after round one because both sides politely drifted into agreement.

def run_debate(run_id, question, cfg, notes=None, transcript=None, start_round=1):
    meter = CostMeter.load(run_id, budget_eur=cfg.budget_eur)
    if notes is None:
        notes = [r.run(question, meter) for r in researchers(cfg)]
        checkpoint.save(run_id, "research", {"notes": notes})
 
    transcript = transcript or []
    for rnd in range(start_round, cfg.debate.max_rounds + 1):
        for role in speaking_order(cfg, rnd):        # alternates first speaker per round
            ctx = context.for_role(role.name, run_id)
            transcript.append(role.argue(question, ctx, transcript, meter))
        converged = judge(cfg).is_converged(transcript, meter)
        checkpoint.save(run_id, f"round-{rnd}",
                        {"transcript": transcript, "converged": converged})
        if converged:
            break
 
    memo = judge(cfg).summarize(question, transcript, meter)  # may say "no consensus"
    checkpoint.save(run_id, "memo", {"memo": memo})
    return request_approval(run_id, memo)

Three decisions hide in there, and each needs an owner: how many rounds, what "converged" means, and what happens on a deadlock. In this desk the judge writes "no consensus, here is the strongest argument on each side" and the human decides.

Debates have known failure modes, so design for them. Whoever speaks first can get an edge, so speaking_order alternates per round. A judge tends to favour arguments written by its own model family, so the judge runs on a different family. And two copies of the same model can drift into agreement; have the judge check whether the skeptic still names a concrete risk before calling it converged.

Step 4: Context

How it breaks: the skeptic quietly reads the advocate's scratchpad, or the judge sees the research notes and starts doing its own research.

context:
  advocate: [runs/{run}/notes/**, runs/{run}/scratch/advocate/**]
  skeptic:  [runs/{run}/notes/**, runs/{run}/scratch/skeptic/**]
  judge:    [runs/{run}/transcript.jsonl]
  maps:
    root: runs/{run}/notes/MAP.md   # what lives where, one line per folder
    owner: research_lead
    review: weekly

def for_role(role, run_id):
    patterns = [p.format(run=run_id) for p in policy[role]]
    return [read_file(f, role, run_id) for f in glob_all(patterns)]   # same allowlist as Step 2

The policy only counts if the loop goes through it, which is why run_debate calls context.for_role instead of passing notes around.

On retrieval: the open-source terminal coding agents whose source you can read mostly find code with search tools rather than a vector index. A desk this size can do the same. A short map file per folder, with big archives marked "search, don't load", is a cheap thing to try before adding a vector database. Maps drift, so the map gets an owner.

Step 5: Approvals

How it breaks: a memo from last week gets approved today, after the facts changed. Or a resumed run opens a second ticket and someone approves the wrong one.

def request_approval(run_id, memo, approve_within_hours=48):
    return approvals.get_or_create(                 # idempotent per run
        key=run_id, memo=memo, approver=cfg.approver,
        approve_by=now() + timedelta(hours=approve_within_hours))
 
def act_on(ticket_id):
    ticket = approvals.get(ticket_id)               # re-read from the service
    if ticket.status == "expired":
        raise ApprovalExpired(ticket_id)            # re-run or re-ask, never auto-act
    if ticket.status != "approved":
        raise NotApproved(ticket_id)
    publish(ticket.memo)

The real exposure here is prompt injection. Follow the path: a web page tells a researcher to ignore its brief, the researcher writes that into the notes, both debaters read the notes, the judge reads the debate. Step 2 cuts the path where it can: only researchers touch the web, nobody can post or send, and nothing leaves the desk except through act_on.

Step 6: Runtime: resume, audit log and cost

How it breaks: a provider times out in round 2, the run restarts from zero, and you pay for the whole debate twice.

def resume(run_id, question, cfg):
    last = checkpoint.latest(run_id)                       # "research", "round-2", "memo"
    if last is None:
        return run_debate(run_id, question, cfg)
    if last == "memo":
        return request_approval(run_id, checkpoint.load(run_id, "memo")["memo"])
    notes = checkpoint.load(run_id, "research")["notes"]
    if last == "research":
        return run_debate(run_id, question, cfg, notes=notes)
    state = checkpoint.load(run_id, last)
    if state["converged"]:
        meter = CostMeter.load(run_id, cfg.budget_eur)
        memo = judge(cfg).summarize(question, state["transcript"], meter)
        checkpoint.save(run_id, "memo", {"memo": memo})
        return request_approval(run_id, memo)
    rnd = int(last.split("-")[1])
    return run_debate(run_id, question, cfg, notes=notes,
                      transcript=state["transcript"], start_round=rnd + 1)

Audit log. A week later someone asks why the desk said no, and the log only says strong-model. Log one line per turn with the resolved model version, the git sha of desk.yaml, the prompt version, the cost, and a pointer to the full output:

{"run":"q-0042","round":2,"role":"skeptic","model":"strong-model@2026-09-30","config_version":"9f3c1a2","prompt_version":"skeptic-v4","context":["runs/q-0042/notes/r1.md"],"tokens_in":5200,"tokens_out":610,"cost_eur":0.0213,"output":"runs/q-0042/turns/2-skeptic.md","ts":"2026-10-12T10:14:03Z"}

Cost. Every model call goes through the meter, and its total is stored per run so a crash loop can't reset it:

class CostMeter:
    def __init__(self, run_id, budget_eur, spent=0.0):
        self.run_id, self.budget, self.spent = run_id, budget_eur, spent
 
    @classmethod
    def load(cls, run_id, budget_eur):
        return cls(run_id, budget_eur, ledger.total(run_id))   # survives restarts
 
    def record(self, role, tokens_in, tokens_out, price):
        cost = (tokens_in * price.in_eur_per_mtok + tokens_out * price.out_eur_per_mtok) / 1_000_000
        self.spent += cost
        ledger.add(self.run_id, role=role, eur=cost)
        if self.spent > self.budget:                # checked after the call, so it can overshoot by one call
            raise BudgetExceeded(f"{self.run_id}: {self.spent:.2f} EUR")

Each turn re-reads the whole transcript, so round 3 costs noticeably more than round 1. Watch tokens_in grow in the log and max_rounds turns into a cost decision too.

Keep the watchers out of reach. The audit log, the ledger and the alert config are written by the loop, never by an agent, and no role has a tool that can touch them. Coding agents at a big lab have already emptied an alert recipient list and removed a monitor during routine tasks, and one reported the job done without mentioning it. So treat an agent's "done" as a claim, and check it from outside after every run:

def verify_run(run_id):
    alerts = load_config("alerts.yaml")                 # read by the harness, no agent tool can write it
    if not alerts.recipients:
        raise MonitoringBroken("alert recipients are empty")
    if audit.count(run_id) != checkpoint.turns(run_id):
        raise MonitoringBroken(f"{run_id}: turns without an audit line")

Step 7: Evals

How it breaks: the debate looks better than one agent only because it spent four times the tokens, or because researchers found the answer on the web, written after the fact.

# evals/debate-desk.yaml
dataset: evals/past_questions.jsonl          # 30 past adoption decisions + hindsight outcome
web_search_snapshot: "2026-06-30"            # no pages newer than the oldest decision date
scoring: evals/rubric.md                     # or "did we regret it six months later?"
reference:
  type: single_agent
  model: strong-model
  samples: 4                                 # ~same token budget as 3 rounds, majority vote
candidate:
  config: desk.yaml
metrics: [decision_score, eur_per_question, minutes_per_question]
rerun_on: [model_change, prompt_change, harness_change, weekly]

Three things make this honest. Freeze web search to a snapshot older than the decisions, or the desk just reads the answer. Give the single agent the same budget: three rounds is roughly a dozen model calls, so let the single agent answer several times and vote. And know your margin of error: with 30 questions and a score around 70%, the 95% margin is about plus or minus 16 points. Only big gaps count.

This is also where benchmark talk belongs. A public comparison of 21 model and harness pairs on two coding benchmarks found the harness moved the success rate by only a few points, while the cost for the same model differed by up to several times. Swap the harness and the score barely moves. The bill does. That was single-agent coding on 30 tasks per benchmark, so it says nothing about approvals or multi-agent setups.

Other numbers point the other way: the eight-point gap from the intro, and the research result in Step 8 where a tuned harness added about sixteen points. My read is that a generic harness swap moves the score a little, and a harness shaped to the task can move it a lot. Leaderboards have caught up too. At least one for coding agents now ranks model plus harness pairs instead of bare models, across 290 tasks, and its top pair sits at 61.7. Those are the publisher's own runs, and the suite changed between versions, so only compare scores within one version. Either way, the number that counts is the one you measure on your own questions.

Step 8: Changing safely

How it breaks: the judge's model gets swapped on a Friday, nobody re-runs the evals, and on Monday the approvers are reading memos from a different model without knowing it.

Every change takes the same route: build a candidate config, evaluate it against the current one, get approval, then promote.

def propose_change(change, tolerance=0.05):
    current = load_config("desk.yaml")
    candidate = change.apply_to(current.copy())           # live config untouched
    report = evals.compare(candidate, current, suite="evals/debate-desk.yaml")
    ok = (report.candidate.decision_score >= report.current.decision_score - tolerance
          and report.candidate.eur_per_question <= report.current.eur_per_question * 1.2)
    if not ok:
        return reject(change, report)
    return approvals.get_or_create(key=change.id, memo=change.describe() + report.summary(),
                                   approver=cfg.approver,
                                   approve_by=now() + timedelta(hours=48))
 
def on_change_approved(change_id):                         # called by the approvals service
    ticket = approvals.get(change_id)
    previous = promote(ticket.candidate_config)            # returns the old config
    audit.log(event="harness_change", change=change_id, previous=previous.sha)
    notify(cfg.approver, f"desk config changed: {ticket.memo}")

What should carry over a model change: desk.yaml, the loop, the context policy, the approval gate, the audit log, the evals. What probably needs re-tuning: prompts written around one model's quirks, context budgets and formats, and any cost case built on today's prices. Tell your approvers when the model behind the memos changed; they've calibrated on the old one.

Vendors fit the same gate. Put every model behind one small interface, and try a swap where it's least risky, on the judge:

class Judge(Protocol):
    def is_converged(self, transcript: list, meter: CostMeter) -> bool: ...
    def summarize(self, question: str, transcript: list, meter: CostMeter) -> Memo: ...
 
change = ConfigChange(id="judge-vendor-b", path="roles.judge.model", value="vendor-b/large")
propose_change(change)

If that looks like ports and adapters, it is. Own the orchestration and the boundaries; vendors are adapters you can swap.

There's early research where a separately trained model proposes edits to an agent's harness code based on how its runs went. At test time both models stay frozen and only the harness code changes. Interesting, and I can't trust it until I try it myself. It fits the gate above anyway: a proposed harness edit is one more change through propose_change, with evals and a human approval.

A second line of research grows the harness instead of rewriting it. It keeps one base harness, lets an agent add subclasses that override only what one cluster of tasks needs, and puts a router in front that picks a subclass or keeps the base. On a business-workflow benchmark that took the same executor model from 61.6% to 77.4% on held-out tasks, while spending more samples on the base harness didn't help. Those are the authors' numbers on their chosen benchmarks, so try it before you believe it. The shape is easy to borrow for the desk, though:

class DeskHarness:                                   # base: the loop from Step 3
    def gather(self, question, meter): ...
 
class LicenseQuestion(DeskHarness):                  # override only what differs
    def gather(self, question, meter):
        notes = super().gather(question, meter)
        return notes + [read_file(f"runs/{meter.run_id}/notes/license.md", "researcher", meter.run_id)]
 
def pick_harness(question):                          # the router; default is the base
    return LicenseQuestion() if "license" in question.lower() else DeskHarness()

Each new subclass is one more change through the same gate.

The ownership table, filled in

The example desk's choices:

Your 30-minute homework

1. Pick one agent, or one team of agents, you run today.
2. Fill in the Owner column of the blank table at the top. "Nobody yet" is allowed.
3. Fix only the scariest empty box this week.

Which box is empty in yours?