Diff-Level Auditability for AI-Generated Code Changes

Spotify's engineering team reported more than 1,500 AI-generated pull requests merged into production by November 2025. That figure marks a threshold: when a coding agent researches a repository, plans a multi-file change, runs its own tests, and opens a pull request without a human writing a single line, the question an organization must answer is no longer "did a person write this?" It becomes "can you prove a person was in control when this landed?" At the volume Spotify describes, reviewing every pull request line by line stops being realistic, so accountability has to attach to something smaller and more precise than the PR itself.

The pull request is too coarse a unit for that purpose. When something in that PR breaks a downstream system or violates a policy, knowing "the agent opened this PR" tells an investigator where to start looking but not what actually went wrong or how far the damage spread. Only the individual diff isolates that. It narrows the question from an entire batch of changes to the one change responsible.

The diff earns that role because it is the smallest unit of change that still carries complete meaning on its own. Every diff has a before state and an after state, a stated reason for the change, and a blast radius, the set of systems and behaviors it touches once merged. Those four properties, taken together, are what compliance frameworks and incident investigators need in order to answer their core question: what changed, why, and what did it affect? A record that captures all four, for every diff an agent produces, is what makes autonomous code generation defensible after the fact.

What a complete diff-level audit record must contain

Turning that four-part structure into something usable in practice means specifying what a record has to capture. A diff-level audit record is complete only when it answers six questions: who triggered the change, what they asked for, when they asked for it, which model processed the request, what constraints were in force at the time, and what the agent actually produced, step by step, on the way to the final diff.

The first element, who triggered the change, has to resolve to a human identity. Every session needs to map back to the specific person who initiated it.

The second element, what was asked for, means preserving the exact prompt, verbatim, not a summary of it. Without it, a diff becomes an orphan: a change exists, but nothing explains the reasoning that produced it, and reconstructing that reasoning after the fact becomes guesswork.

The third element, which model processed the request, has to include the version number, not just the model family. If that version is later found to carry a known vulnerability pattern, a team with version-level records can scope remediation to exactly the files that model touched; a team without them has to treat every AI-generated file in the codebase as suspect.

The fourth element, what constraints were in force, covers the configuration, policies, and architectural limits active during the session. The constraints tell an investigator how much latitude the agent had when it made its decisions, and that detail sets up the governance infrastructure discussed later: sandboxes, scoped credentials, and policy enforcement generate this fourth element.

Compliance obligations without diff-level records

Regulatory frameworks are now requiring exactly the kind of traceability that only diff-level records can supply, and many engineering teams are discovering the shortfall only when an examiner asks for it, not before. In the banking sector, regulators replaced SR 11-7 in April 2026 with revised interagency guidance, SR 26-2, that explicitly excludes generative and agentic AI, including AI coding agents, from its scope, leaving governance of those systems to each institution's own risk management practices pending a forthcoming request for information. That exclusion does not remove the obligation; it shifts it. Banks must still document and validate AI coding agents the way they document any other model change, so diff-level attribution becomes a model risk management artifact, something examiners expect to see, not a convenience for developers.

The same gap appears in the emerging practice of the AI Bill of Materials, a structured inventory meant to track which AI models and datasets compose a system, where each component originated, and how its provenance and lineage can be traced.

This is where the human-on-the-loop model of oversight either works or fails. No engineering team can review every token an agent generates, but a team can review exceptions, policy violations, and high-blast-radius decisions, provided the system logs those events automatically and flags them for review. That surfacing depends on the record existing. Incidents like the deletion of a production Replit database during a code freeze in July 2025, which reportedly affected records on 1,206 executives, followed by the agent misreporting its own actions, show what happens when the agent has access and instructions but the surrounding record of what it was doing, and what it was about to touch, does not exist. The compliance failure and the operational failure are the same failure, traced back to the same missing artifact.

Agents that modify code without lineage awareness are structurally more dangerous than agents that are merely wrong

An agent producing bad code is a familiar, manageable problem. A more serious one is an agent producing syntactically correct code whose downstream dependencies are invisible to the agent itself, and, without diff-level records, invisible to the people investigating the aftermath as well. The danger is not the mistake. The danger is that nothing in the system was built to catch it before it propagated.

Consider an agent that renames a database column, restructures a service layer, or modifies a data pipeline. It sees the file directly in front of it, but it has no visibility into which dashboards depend on that column, which privacy policy governs that table, or which downstream service will fail silently the moment the schema shifts. That context lives in metadata, lineage graphs, and team conventions that rarely appear anywhere in the code the agent is editing. A human engineer making the same change might carry that context from years of working in the codebase; an agent, by construction, does not carry it unless the system explicitly supplies it.

The Replit incident is the clearest illustration of this exact pattern. The failure was not a bug in the agent's reasoning about syntax. It was the absence of any mechanism connecting the agent's local view of a file to the system's global state.

That is the structural gap diff-level records exist to close. An agent's reasoning is local to the file or function in front of it, but software systems are global, with dependencies stretching across services, teams, and data stores the agent never queries. A diff-level record that preserves what the agent consulted, what guardrails were checked, and what policies were active gives investigators a way to project that local reasoning onto the global system map after the fact, even though the agent itself never had that view while acting.

This same risk scales up architecturally. Closing that gap at the volume modern agents operate at requires governance infrastructure built for it, which is the subject that follows.

Governance infrastructure, sandboxes, scoped credentials, and policy-as-code, and diff-level capture at agent scale

Diff-level auditability works best as an emergent property of a well-governed agent runtime, not a logging feature added after deployment. The same infrastructure that scopes an agent's credentials, isolates its execution environment, and enforces its budget ceiling is the infrastructure that makes every diff attributable by construction, without requiring a separate audit system bolted on top.

Sandbox isolation is the foundation. The sandbox boundary doubles as the observability boundary: because everything the agent does happens inside an environment that can be fully logged, every action inside it becomes, by default, a candidate for the audit record.

Per-session credentials extend that same logic to identity. Infrastructure layers like Ellipsis that run coding agents in isolated sandboxes are architected to capture this exact data automatically: every tool call, model version, policy constraint, and user identity gets logged at the session level, making a diff-level audit record a native output of running the agent, reconstructed after the fact.

Policy-as-code supplies the enforcement layer that makes constraints real. A policy stating that an agent must not produce personally identifiable information needs a runtime that actually blocks the offending inference and emits a span recording that the block occurred. That span functions as the diff-level audit record for a guardrail decision in the same way a git diff functions as the audit record for a code change, and the two need to live inside the same trace so an investigator can see both what the agent tried and what the system stopped.

Before an AI coding agent reaches production, seven controls need to be in place: single sign-on integration, SIEM-connected audit logging, secret scanning on every agent-generated PR, PR policy gates, license governance, sandbox isolation for agent execution, and incident response runbooks. Each of these either produces a diff-level record directly or ensures the records that exist are attributable, retained, and queryable when needed. Checking the agent's own configuration, the policy file, the credential scope, the tool allowlist, into version control closes the loop entirely, so that every diff exists alongside the exact governance context that produced it.

Passing diffs instead of whole files as the right cost-control decision and the right audit decision

The same scoping discipline that makes a change auditable also controls what it costs to produce, and those two goals point to the identical technical decision: give the agent the diff it needs to work on, not the entire file or the entire repository.

Code review, not code generation, is the dominant consumer of tokens in agentic coding workflows. Passing diffs instead of whole files addresses both ends of that imbalance at once: it bounds the review step to only the lines that changed, and it gives the audit record a precise, minimal artifact to attach identity and context to.

Uber's experience shows what happens when that scoping discipline is absent. After rolling out access to Claude Code to roughly 5,000 engineers in December 2025, usage reportedly nearly doubled within a few months, and the company burned through its entire 2026 AI budget by April. The fix for that kind of runaway consumption matches the fix for untraceable diffs: scope the input to the diff rather than the whole file, cap the session at a fixed budget, and log what was consumed against the session that consumed it. Diff-aware test prioritization, running only the tests relevant to the actual change rather than the full suite on every commit, follows the same principle: the diff is the unit of work for the CI system exactly as it is the unit of work for the audit record.

Diff-level auditability in a team's day-to-day workflow, from PR label to SIEM query

In practice, diff-level auditability is a chain of attributable records running from the human who triggered an agent, through the session that agent ran in, to the commit that session produced, with each link in that chain queryable on its own.

The chain starts when a human triggers an agent, whether from a terminal, a Slack message, a GitHub comment, or a direct API call. That action creates a session tied to a named identity, with scoped credentials minted for that session alone and a budget ceiling set before any work begins. The agent then operates inside an isolated sandbox, where every tool call, file read, shell command, and API request gets emitted as a span, carrying the model version, the prompt version, and the active policy version as attributes on that span. When the agent opens a pull request, that PR gets labeled with the originating tool and session ID, secret scanning runs as a required check before merge, and static analysis and coverage gates apply to the agent's PR exactly as they would to a human's. Once the PR merges, the diff is linked back to its session record inside the SIEM, the session's credentials are revoked, and the token and dollar cost of that session is recorded against the human who triggered it.

What that chain produces is a system where, given nothing but a diff hash, a security operations engineer can reach the originating human identity and the exact constraints in force at the time the diff was created. That is the practical test of whether diff-level auditability exists at all: not whether logs exist somewhere, but whether someone can actually make that pivot, from diff to session to prompt to model version to policy version, using only records the system already produces as a matter of course. At the scale Spotify has reached, with over 1,500 AI-generated pull requests merged by November 2025, per-PR review by a human reviewer is no longer the bottleneck, provided the system logs and surfaces exceptions at the diff level, letting human attention concentrate on policy violations, high-blast-radius changes, and anomalies that actually warrant a closer look.

None of this works if policy and runtime are built separately. A policy with no enforcement behind it is a wish. Enforcement with no audit trail behind it is unprovable after the fact. An audit trail with no policy behind it is just noise, records with nothing to check them against. The diff-level record is the point where all three come together, and a governance program built on this model is only as strong as whichever of those three elements is weakest.