Agents Can Open the Pull Request, but They Cannot Sign the Release

Before any of the controls in this series, the publish pull request had everything a reviewer is trained to want: a clear description, a small diff, passing tests, a plan comment that matched the code line for line. The tech lead read it in one sitting, found nothing wrong, and approved. Nothing in it was wrong.

Nothing in it mentioned sign-off either. The diff added an endpoint in src/api/memos.py that called the delivery adapter. It never touched src/publishing/, so the PublishingPolicy module the old export used never appeared in the review. The rule R1, that every claim traces to a sourced note and a named researcher approves the exact memo, existed elsewhere in the repo, but it was missing from the ticket and from this endpoint's checks. The reviewer judged the code correctly and approved a change the product could not accept. Memo M-104 reached clients unsigned.

This is the illustrative research desk from the series. It shows what changes about review when the author is an agent, and why this app needs two different kinds of approval.

A clean diff is not evidence

A more careful reviewer, a second reviewer, or an AI reviewer next to the human all read the same diff. On their own they do not touch the failure, because the failure is not in the diff.

A human author carries evidence that never appears in a pull request. They know the old export goes through a policy module, they remember why, they know which module owns the rule. Reviewers lean on that without noticing. An agent carries none of it. Its diff is exactly as good as the artifacts it was given.

So for an agent-written change, review has to check what the agent was asked against the accepted artifacts: the links from the pull request back to the intent record, the acceptance criteria, the plan and the check results. If the pull request cannot say which constraints apply to this behaviour, the answer is "unknown", not "probably fine".

Two reviews, one signature, and a second gate

I split what most teams call "review" into two reviews and a decision, each with a different owner.

The intent review asks whether the change does what the product needs and whether every constraint on this behaviour was consulted. That is the product manager, with the security reviewer for the controls they own. In the original change this review never happened, because everybody assumed it had happened upstream.

The implementation review asks whether the code is sound, whether the test and eval evidence is real, and what breaks if it is wrong. That is the code owner. It did happen, and it was done well. One good review out of two is how correct code for the wrong product gets approved.

The signature is not a review. A named release authority decides, based on both reviews and the release record, and the agent can prepare everything that feeds that decision without being allowed to make it.

This app has a second gate that is easy to confuse with the first. The software release gate decides whether a new version of the app ships. The memo publication gate decides whether one memo snapshot reaches clients. They have different subjects, different approvers and different separation rules:

A researcher approving the memo they commissioned is the job, not a conflict. An engineer approving the agent change they requested is a conflict. One blanket rule for both would either block the desk or let the engineer through.

A rejection goes back to the earliest thing that failed: the intent for a product mismatch, the plan for a design error, the code for a defect, and not simply back to the agent's editor.

Here is who does what. The agent is never accountable, never an approver and never the signer.

Every A resolves to a person with a named deputy. If no qualified person is available, the release waits. A waiver needs a second named person to approve it, and nobody approves their own.

The template that turns an omission into a question

The cheapest mechanism is a change-request template that the author, human or agent, has to fill in before anyone reads a line:

## Change request
Risk tier: 3 (causes a tier-3 event: memo_published)
Intent: intent-fast-publish.md
Criteria covered: AC1, AC2, AC3
Constraints consulted (owner): R1a (security reviewer), R1b (researcher), R2 (tech lead)
Open items: Q1, owner security reviewer, due before release
 
### Agent provenance
Agent identity: <id>   Requested by: <engineer, from the broker's record>   Run: <run id>
Model deployment and version: <name>/<version>
Tool calls denied during the run: <count, link to run record>
 
### Evidence
Tests: <pipeline run>   Evals: <run id>, runs=<n>, passes=<x>, critical failures=<c>, lower bound <p> vs required <p*>
Surfaces changed: src/api/memos.py
Surfaces consulted, unchanged: PublishingService, PublishingPolicy
 
### Reversibility
Code: reversible by flag. Memo delivery: irreversible once clients are notified.
Rollback: memos.one-click off. Decided by: release authority. Executed by: operations.

In the original change the constraints line would have said "none" and the tier-3 event line would have said "no". The value of the template is that both are now claims a product manager has to confirm. It turns an omission into a sentence somebody signs.

Reviewable is not enforced

On GitHub, the usual first step is a code-ownership file:

# .github/CODEOWNERS
/src/publishing/**          @research-desk/platform-owners  @research-desk/security-reviewers
/src/publishing/policy/**   @research-desk/security-reviewers

That has two problems. First, an ownership file reacts to edits, not omissions. The original change touched src/api/ and left src/publishing/ untouched, so neither line fired. Second, when one line lists two owners, approval from any one of them satisfies it. The security review looks required and is not.

And the file only routes. It requests reviewers; whether their approval is required is a separate branch protection setting. Without that setting, the security review is a convention, and a convention is what the original change ran on.

The host I use, Azure Repos, has no ownership file. A required-reviewer branch policy with a path filter does both jobs there: it adds the reviewers and, when it is marked required, blocks completion until they approve. The first problem does not go away, because a path filter also fires only on edits. So enforce each group by policy, and make the omission fail CI instead of relying on routing. With the host's CLI, the policies for main look like this. The platform team runs them, never a squad or an agent:

ORG="https://dev.azure.com/<org>"; P="<project>"; R="<repo-id>"; B="main"   # replace the placeholders
TIER3="/src/publishing/*;/src/memos/*;/src/api/*;/src/*/policy/*;/evals/*;/fixtures/*;/docs/policy/*;/tiers.yaml;/azure-pipelines*;/agent-instructions.md;/skeptic-instructions.md"
 
# Two human approvals; the author's own vote does not count; new pushes reset votes
az repos policy approver-count create --org "$ORG" -p "$P" --repository-id "$R" --branch "$B" \
  --minimum-approver-count 2 --creator-vote-counts false --allow-downvotes false \
  --reset-on-source-push true --blocking true --enabled true
 
# Tier 3 paths wait for one approval from every listed group; no group can stand in for another.
# Pass each group's identity id.
az repos policy required-reviewer create --org "$ORG" -p "$P" --repository-id "$R" --branch "$B" \
  --required-reviewer-ids "<code-owners-id>;<product-managers-id>;<security-reviewers-id>" \
  --path-filter "$TIER3" --message "Tier 3 path: code owner, product and security review required" \
  --blocking true --enabled true
 
# Tests must pass, rerun on every merge-commit change
az repos policy build create --org "$ORG" -p "$P" --repository-id "$R" --branch "$B" \
  --build-definition-id "<pipeline-id>" --display-name "tests" \
  --queue-on-source-update-only false --manual-queue-only false --valid-duration 0 \
  --blocking true --enabled true
 
# The requester of an agent change may not be one of its approvers.
# The CLI has no dedicated command for a status-check policy, so it takes a config file.
cat > sod-status-policy.json <<EOF
{
  "isBlocking": true,
  "isEnabled": true,
  "type": { "id": "cbdc66da-9728-4af8-aada-9a5a32e4a226" },
  "settings": {
    "statusGenre": "sod",
    "statusName": "requester-not-approver",
    "invalidateOnSourceUpdate": true,
    "authorId": "<check-service-identity-id>",
    "scope": [{ "repositoryId": "$R", "refName": "refs/heads/$B", "matchKind": "Exact" }]
  }
}
EOF
az repos policy create --org "$ORG" -p "$P" --config sod-status-policy.json

The src/api/* path joined tier 3 after the original change, here and in tiers.yaml. The tier-3 event line catches behaviour, and the path list catches the place where that behaviour was added last time.

Then turn on "prohibit the most recent pusher from approving their own changes". The CLI has no flag for it, so set it in the branch policy settings. That rule and "the creator's vote does not count" stop the identities that pushed or opened the change from approving it. But the broker opened the pull request, not the engineer who asked for the change. The last policy closes most of that gap: the broker records who requested the agent run, and a small check service posts the sod/requester-not-approver status on each vote, against the current merge commit. authorId makes the policy count only that service's status, so a contributor cannot post a passing one themselves. The check is fast, not atomic: between a vote and the service's answer, an earlier success still stands for a moment. That is why the release pipeline repeats the check before promotion, as below. It applies to software gates only. Memo approval has its own rule in the publishing service:

# pseudocode: the sod/requester-not-approver status for one pull request
SOFTWARE_GATES = {"implementation", "software_release", "waiver"}
 
def requester_not_approver(gate, requested_by, agent_id, approvers):
    if gate not in SOFTWARE_GATES:
        return "not_applicable"               # a researcher approving their own memo is a different gate
    banned = {requested_by, agent_id}         # canonical identity ids, not display names
    return "failed" if any(a in banned for a in approvers) else "succeeded"
 
# eng-2 asked for the change, approvers [eng-2, tech-lead]               -> failed
# eng-2 asked for the change, approvers [tech-lead, security-reviewer]   -> succeeded
# memo gate, researcher-1 approves the memo they commissioned            -> not_applicable

Keep the publishing service tests from the first part in the suite, so a new path that reaches clients without a valid sign-off fails the build no matter who reviewed it.

The agent's identity is the real control

"Agents cannot sign" is a policy statement until it is a permission. Writing "never approve" in the agent's instructions does nothing if the identity it runs as can approve.

On the host I use, the permission to contribute to pull requests covers creating, commenting and voting together. So the coding identity should not have it at all. A repository setting, strict vote mode, changes that by also requiring the Contribute permission to vote, so check the effective permissions of the agent and the broker separately. My default setup:

• The coding agent runs as its own identity. It can push only to its own working branches. Contributing to main or release branches, contributing to pull requests, bypassing policies and editing policies are denied.
• A small broker service, with its own identity, opens the pull request on the agent's behalf. By default, creating a pull request needs the same permission as voting, so the broker could technically vote, but as the creator its vote never counts. It cannot push to main, bypass policies or edit them. It records who requested the agent run, from the authenticated session, not from the agent. Its credentials never reach the agent.
• The agent has no access to the production environment, its service connections, or the delivery credential.
• The production environment has an approval check for the release authority group, with approvers not allowed to approve their own runs, and a branch-control check. These checks live on the environment, outside the pipeline YAML, so an agent that edits the pipeline file cannot remove them.
• Environment checks only guard the stages that use the environment. So the production service connection gets the same approval and branch control, plus a required-template check, so only the trusted release pipeline can use it. An ordinary job that tries to deploy directly with that connection is stopped.
• Right before promotion, the release pipeline compares the digest it is about to deploy with the approved digest in the release record, and stops on any difference. It also runs the requester check again over every approval behind this artifact, the pull request approvals as well as the release and waiver approvals, using the broker's record of who requested the change. It stops if that engineer or the agent approved anything, or if a record is missing. This is where the race at the pull request is contained: a merge can slip through for a moment, a release cannot. "Approvers may not approve their own runs" does not cover this: the engineer who asked for the change is rarely the one who queued the release run.
• Inside the app, the desk agents cannot call the approval service or the publishing service. Only an authenticated researcher can create a sign-off.

Then test it. Try to approve as the agent, get a pull request through on the broker's vote, try to merge as the agent, push to main as the agent, approve as the engineer who requested the change, deploy from an ordinary pipeline with the production connection, and create a sign-off as a desk agent. Make sure each attempt fails. A control nobody has tried to break is a guess.

The signature depends on what can be undone

Who may sign follows from what can be undone, and how. I use four classes. Reversible by flag or config, no data changed. Reversible by redeploying the previous build, no data changed. Reversible with repair, where data changed and needs a migration or backfill. Irreversible as an event: a memo reached clients, an email was sent, a role was granted.

The publish change is the instructive case, because a memo goes through three states. Queued: still cancellable. Published: visible on the client portal, and a rollback can still stop further distribution. Notified: clients have it in their inbox. From there, nothing recalls it. What is left is a correction or retraction notice, which is incident response, not rollback. The code behind the button is flag-reversible; the memo is not. So the record has to name both, and the class decides the signer. Reversibility is not the only input, though. Data access, client impact and blast radius count too.

The release record also has to be bound to exactly what was approved. A record that lists test names but no commit can be approved for one build and deployed with another.

# Release decision: one-click memo publishing
Commit: <sha>   Artifact: sha256:<digest>   Approved at: <utc>
Eval inputs: cases <sha256>, scorers <sha256>, instructions <sha256>, fixtures <sha256>
Approval is void if the deployed artifact digest or any of these hashes differs.
 
Intent: intent-fast-publish.md   Criteria: AC1-AC3   Open items: none
Evidence: tests <run id>, evals <run id>, intent review <link>, code review <link>, security review <link>
Reversibility: code reversible by flag; queued memos cancellable; notified memos need a correction notice
Approver: release authority, research desk   Deputy: <name>
Rollback: memos.one-click off. Decided by: release authority. Executed by: operations on call.
Watch after release: publications without a valid sign-off at publication time must be zero; any row means flag off.

One page for the person who signs

The release authority is often not an engineer, and should not need to read the diff. Give them a card with six questions in plain language. Here it is for the original change, as it should have been filled in, and after the controls:

RELEASE DECISION CARD: one-click memo publishing, tier 3
 
BEFORE THESE CONTROLS (what the card would have shown)
1. What changes for clients?          Memos reach them faster.
2. Worst plausible result if wrong?   (blank)
3. Can it be undone?                  Flag off.
4. Which rules were checked, by whom? None listed.
5. Evidence I can open:               unit tests.
6. Who watches after release?         (blank)
 
AFTER
1. What changes for clients?          Signed memos reach them faster.
2. Worst plausible result if wrong?   An unsigned memo with untraced claims reaches clients.
3. Can it be undone?                  Code: flag off, minutes. Notified memos: no, correction notice only.
4. Which rules were checked, by whom? R1a security reviewer, R1b researcher, R2 tech lead (all signed).
5. Evidence I can open:               service tests, evals (runs, critical failures, lower bound), reviews.
6. Who watches after release?         Operations watches publications without a valid sign-off.
Decision: [ ] Approve  [ ] Return to: ____  [ ] Stop
Signed: <name, role>   Commit: <sha>   Artifact: sha256:<digest>   At: <utc>

If the person signing cannot answer questions 2 and 3 from the card, the card is not done. The blank lines on the first card were the whole incident.

The signal that becomes the next intent

When something goes wrong in production, the incident should not close when the fix ships. It closes when its lesson has become a new intent record with an owner, because a fix that does not change the artifacts upstream leaves the next agent free to repeat the omission. For the publish change, that means R1 gets written into the intent template for every path that reaches clients, and the "ticket silent, agent must route through the service or stop" eval case becomes a permanent regression case.

Where this is too much

Two reviews and a signature fit a change that can send an unsigned memo to clients. They do not fit most changes. A patch bump inside the dependency policy needs the normal pull request and one reviewer. A typo needs no release record.

On a small team, one person may hold several roles, and for low tiers that is fine. For tier 3 it is not. The author of a sensitive change, or the engineer who asked the agent for it, does not approve it. If there is truly only one qualified person, the release waits. A waiver needs a second named person to approve it, with an expiry date and the artifact it applies to, and nobody approves their own.

Test your controls

The hard case is the team where the release authority exists on paper and nobody holds it in practice. The on-call engineer signs because they are there. A deputy rule and a "release waits" rule help, but they also slow things down, and teams under pressure find ways around rules that slow them down. That is why the controls have to be tested, not just written. On the next tier 3 release, try each bypass in the list above, add the reversibility line and the two rollback names to the record, and see who objects.

Before these controls, the original change shipped and nothing alarmed. The dashboard even counted it as a win. The next part asks how a manager would have known.