How to Tell Whether the Agent Workflow Improved Delivery

Before any of the controls in this series, the adoption dashboard looked the way adoption dashboards are supposed to look. More pull requests created and merged, a shorter median time to merge, a growing share of changes with the agent as author. The one-click publish change was one of them. It merged quickly, with passing tests and a clean review, and it counted toward every line on the board.

Underneath, the publish log said something else. Memo M-104 reached clients with no sign-off record, and two of its claims, c3 and c4, pointed at no note. The rule existed elsewhere in the repo, but it was missing from the ticket and from the new endpoint's checks. Every check passed. The dashboard counted the result as throughput.

This is the illustrative research desk from the series. This part is for the person who gets asked "is the agent thing working?" and has to answer with something better than a chart going up.

Output counts rise either way

Pull-request counts, merge time and agent-authored share measure adoption. A change that skips a constraint merges exactly as fast as one that honours it, often faster, so every one of those numbers is blind to the publish change.

Any count has the same trap. It rises with volume whether or not the work got better. A team that ships more changes and more mistakes looks identical to one that ships more changes and fewer. Agents tend to amplify whatever a delivery system already does well or badly, so a single number is the wrong instrument. You want several, with tension between them, read together.

A note on vocabulary. When I say "delivery metrics" I mean the standard software delivery set: change lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate. I spell them out rather than use the acronym.

Who is accountable at each gate

Before any metric, a manager needs to know who decides what, and when something must escalate. This is the management view of the handoff model from the first article:

"Sometimes" is the honest answer for releases. The one-click flag can stop further distribution, and a queued memo can still be cancelled, but a memo clients were already notified about can only be corrected or retracted.

The dashboard contract

The dashboard I would set up is written as a contract, in two groups. Delivery tiles answer whether the coding agent improved how the team ships software. Publication-control tiles answer whether the app's own controls held. Both matter, but an unsigned memo does not by itself tell you whether agent-assisted delivery got better or worse, so they are read separately and linked by a trace. For each tile: what each tile means, who owns it, how often it is read, and what happens when it moves. The action rules matter more than the tiles. Every window, margin and minimum sample below is an illustrative policy setting written as a variable. Agree the real values during the pilot, from your own baseline. The one fixed rule is completeness: tier 3 constraint capture is 100% or the change is blocked.

Delivery tiles

Publication-control tiles

Every publication-control row carries a trace back to the change that produced it: software release, desk agent run, memo digest, sign-off, publication event. When a control fails, the trace tells you whether to look at a code change, a model change or a person.

I read it with two rules. Never show agent share or PR counts without review load and change fail rate next to them. And never show a missing value as zero. Incomplete data is "unknown".

The one rate I would start with

The leading tile, constraint capture, tells you whether intent is being written down at all. The intent-correction rate tells you, afterwards, whether intent reached the agent's inputs. A correction counts when the intent record, the criteria or a constraint had to change after merge, tagged as such in the tracker at the time. A code-only fix does not count.

The rate needs a fixed follow-up window per change, declared before anyone looks at the data, and it is reported only for mature cohorts, where every change has had its full window. The query counts a change once however many corrections it got, and leaves the rate empty when observation was incomplete:

# pseudocode: intent-correction rate per mature monthly cohort; all timestamps UTC
INCLUSION_RULE = "agent-assisted-v1"   # versioned: an agent run record linked to the change
 
def intent_correction_rate(changes, corrections, cutoff, follow_up_days):
    window = days(follow_up_days)
    # a monthly cohort is mature only when its last possible change has had its full window
    mature = [c for c in changes if end_of_month(c.merged_at) + window <= cutoff]
    rows = []
    for key, group in group_by(mature, "month", "risk_tier", "service",
                               "change_class", "agent_assisted"):
        corrected = [c for c in group if any(           # a change counts once
            k.kind == "intent" and c.merged_at <= k.corrected_at <= c.merged_at + window
            for k in corrections_of(c))]
        observed = all(c.observed_through >= c.merged_at + window for c in group)
        rate = len(corrected) / len(group) if observed else None   # "unknown", never zero
        rows.append({**key, "changes": len(group), "corrected": len(corrected),
                     "rate": rate, "inclusion_rule": INCLUSION_RULE})
    return rows

The real query also refuses duplicate change ids, unknown risk tiers and corrections for changes it does not know, because each of those quietly changes the denominator. Here is a small input, with illustrative data. All four changes are tier 3 features in the publishing service, observed through 2026-10-10, and the run uses a 30-day window and a cutoff of 2026-10-10:

 cohort  agent_assisted  changes  corrected  rate
2026-08           False        1          0   0.0
2026-08            True        2          1   0.5

The September change is left out because its cohort has not had its full window by the cutoff. ch-1 had two intent corrections and counts once. ch-2 had only a code fix, so it does not count.

Compare the agent-assisted rows with the non-agent rows for the same tier, service, change class and window. "Agent-assisted" means what the versioned inclusion rule says, and the rule is recorded on every row, so a changed definition shows up as a changed rule, not a changed rate. If there is no comparable non-agent group, use the first mature agent cohort as a baseline and call it a baseline, not a target.

The rate has limits. It is a local signal, not proof that the workflow caused anything. And it cannot tell a team that writes better intent records from a team that stopped tagging corrections. That recording bias is worst on teams that never tagged corrections before. So start tagging first, and treat the first period as calibration.

Valid at the moment of publication

"Published without sign-off" sounds simple and is not. A sign-off id on the publication record is not enough: the sign-off has to be for that exact memo digest, by an authorized researcher, not yet expired and not revoked, judged at the publication timestamp, not today. And an approval record that cannot be found is "unknown", which is a telemetry problem, not proof that nobody approved. The check reads the authoritative publication and sign-off records, and reuses signoff_status from the first part. Passing the publication time is not enough on its own: who counted as a researcher, and which policy version applied, also have to come from history, or a policy change today would mark every older publication as invalid:

# pseudocode: judge every publication against the sign-off as it stood at publication time
def publications_without_valid_signoff(publications, signoffs):
    for pub in publications:                     # the publishing service's own records
        at = pub.published_at
        researchers = researcher_roster_as_of(at)          # authority then, not today
        policy = policy_version_as_of(at)
        if pub.signoff_id is None:
            reason = "missing"
        elif pub.signoff_id not in signoffs or researchers is None or policy is None:
            reason = "unknown"                   # a record or its history is absent
        else:
            reason = signoff_status(signoffs[pub.signoff_id], pub.memo,   # memo as published
                                    researchers, policy, at=at)
        if reason != "valid":
            yield pub.publication_id, pub.memo.id, at, reason

With illustrative records, where the roster and policy history are complete and nothing changed in them:

publication_id memo_id              published_at          reason
            p1   M-104 2026-10-03 08:00:00+00:00         missing
            p3   M-106 2026-10-04 09:00:00+00:00 digest_mismatch
            p4   M-107 2026-10-05 10:00:00+00:00         unknown

p1 is the original incident. p3 was signed, but the memo was edited after sign-off. p4 points at a sign-off the record store cannot find, so it goes to the telemetry queue before anyone calls it an incident.

Every agent run leaves a record

None of the tiles above work without data, and the most useful data is a record of every agent run. It doubles as the audit trail, which is the question every reviewer, auditor or incident lead eventually asks: what exactly did the agent do, with which model and instructions, and who approved the result?

{
  "run_id": "<uuid>", "started_at": "<utc>", "agent_id": "<agent identity>",
  "requested_by": "<person>", "work_item": "<id>", "branch": "agent/<run_id>",
  "risk_tier": 3,
  "model": {"deployment": "<name>", "version": "<pinned version>"},
  "instructions_sha256": "<hash of system prompt and repo instructions>",
  "context": [{"doc": "intent-fast-publish.md", "sha256": "<hash>"}],
  "tool_calls": [
    {"tool": "write_file", "target": "src/api/memos.py", "decision": "allow"},
    {"tool": "approve_pull_request", "decision": "deny", "rule": "human-decision"}
  ],
  "usage": {"input_tokens": null, "output_tokens": null, "cost": null},
  "outputs": {"pull_request": "<id>", "commit": "<sha>"},
  "evals": [{"case": "B-ticket-silent", "runs": "<n>", "passes": "<x>",
             "runs_with_critical_failure": 0, "lower_bound_95": "<p>", "gate": "pass"}],
  "human_decisions": [{"gate": "software_release", "actor": "<person>", "decision": "approve",
                       "artifact_sha256": "<digest>", "at": "<utc>"}],
  "data_classification": "no client data in prompts or context",
  "retention_class": "<records policy id>"
}

The whole record is illustrative, and null means not measured, never zero. Leave out raw credentials, client data and the model's hidden reasoning. Record actions, sources, decisions and outcomes.

If you send these to a log table, say AgentRun_CL, the snake_case JSON has to be mapped to the table's columns on the way in. With a data collection rule, that is one transform, with model, tool_calls and human_decisions declared as dynamic in the incoming stream:

// transformKql in the data collection rule
source
| project TimeGenerated = todatetime(started_at), RunId = tostring(run_id),
          AgentId = tostring(agent_id), RequestedBy = tostring(requested_by),
          RiskTier = toint(risk_tier), ModelDeployment = tostring(model.deployment),
          ModelVersion = tostring(model.version), ToolCalls = tool_calls,
          HumanDecisions = human_decisions

Then two queries cover the most important tiles. The first is the separation-of-duties alert:

// Separation of duties, software gates only: approvals by the coding agent or the
// engineer who requested the change. Memo approval is a different gate: a researcher
// approving the memo they commissioned is expected, so it is never in this query.
AgentRun_CL
| where TimeGenerated > ago(30d)
| mv-expand d = HumanDecisions
| where tostring(d.decision) == "approve"
| where tostring(d.gate) in ("implementation", "software_release", "waiver")
| where tostring(d.actor) == AgentId or tostring(d.actor) == RequestedBy
| project TimeGenerated, RunId, Gate = tostring(d.gate), Actor = tostring(d.actor)

The second is the guardrail-denial rate:

// Guardrail denials per 100 tool calls, by tier and model version
AgentRun_CL
| where TimeGenerated > ago(30d)
| mv-expand t = ToolCalls
| summarize Calls = count(), Denied = countif(tostring(t.decision) == "deny")
    by RiskTier, ModelDeployment, ModelVersion
| extend DeniedPer100 = round(100.0 * Denied / Calls, 1)

Run the separation-of-duties check against the code host's own audit log too. The agent's own log cannot prove that something did not happen. The publication-control tile runs the sign-off check above against the publishing service's own records, not against telemetry, for the same reason.

The original failure leaves little trace until it fails. A constraint that never reached the intent record, and that no eval case or policy lookup asks about, produces no failing test, no failing eval, no rejected review, right up until the publish log shows a memo with no valid sign-off. Constraint capture is my best early signal, and it only measures whether people wrote something in a field, not whether they wrote the right thing. What a manager can do this week is smaller: open the last tier 3 release record and ask for the reversibility line, and ask when its eval cases were last confirmed. If nobody can answer, that is the measurement.

Traced is not the same as true

There is one row in the dashboard that none of this settles. After the controls, M-104 could only go out with every claim pointing at a sourced note. Claim c5, "Lead times fell in every region", pointed at note n4 from the start. It was structurally traced. It was also wrong: n4's data shows lead times fell in two of three regions. The source-fidelity tile counts findings like that, but only after a researcher reads the claim against the note.

The memo is agent-written prose. The last part asks what checks the sentences.