Using AI Agents to Investigate and Triage Production Incidents

AI agents can do much of the work of production incident investigation, from gathering context and correlating signals to ranking alerts. Their value is to turn the on-call engineer’s work from diagnosis toward decisions. That shift depends less on granting autonomy than on earning it through measured precision, bounded access, and evidence that engineers can inspect.

Why production incident investigation became a bottleneck AI agents can address

At 3 AM, the pager wakes an engineer who must reconstruct an incident across disconnected tools. The SIEM has one view, the EDR another, while identity providers and ticketing systems hold further clues. Each has its own format and query language. The engineer has to correlate the signals by hand, running on adrenaline when the work calls for careful judgment.

That arrangement does not scale with alert volume. Hiring alone cannot resolve a workflow that repeatedly requires a person to assemble context across fragmented systems. Correlation rules, playbook automation, and ML assistance have addressed parts of that work, but agentic triage now targets the investigation itself. Its reach still depends on detection coverage. An alert that never fires cannot enter either a human or an automated triage queue.

What AI triage agents actually do in a production incident investigation

The gains concentrate in four steps of the investigation lifecycle. First, context building brings information from the SIEM, EDR, threat intelligence feeds, CMDB, and ticketing into a structured context pack. The agent reasons over that pack. This is a deliberate reliability choice.

Next, the agent can run pivot queries across related alerts, translating natural-language questions into searches across an IP, user, host, or file hash. The analyst need not know KQL or SPL to ask the next question. Production deployments commonly restrict these tools to read-only access. The agent can investigate without acquiring permission to change the environment.

From that evidence, the agent produces a structured summary containing a disposition verdict, confidence level, investigation timeline, and suggested next steps. The reasoning trail is essential to review. A confident conclusion without supporting reasoning is harder to validate than an incorrect answer whose path is visible. Queue ranking also helps by weighing severity, confidence, asset criticality, and nearby alerts on the same identity together. For a small team, elevating a low-severity alert that appears near related alerts on the same user within an hour can be more useful than simply reducing volume.

Kiro’s published account makes the context-building mechanism concrete. Its operational procedures, alarm wiki, and post-incident notes feed the agent’s context. Each fix is saved once for future sessions, while closed tickets are aggregated and queried in later investigations. Agents also create or update playbooks that engineers manage like a codebase. The human handoff therefore moves from starting an investigation to reviewing one already assembled.

Where current AI triage agents fail and why the limits are not easily patched

Agents inherit the gaps in their evidence and detection coverage. Novel attack patterns and zero-day behavior can fall outside the patterns available to them, while MITRE ATT&CK mappings cannot keep pace with every change in adversary tradecraft. Local knowledge creates a separate constraint. An agent may not know that a particular engineer tests on Fridays or that finance performs a monthly batch export. Without that context, routine activity can support a misleading conclusion.

Ambiguity also creates room for unwarranted confidence. An agent can supply a half-answer that sounds complete, making the missing evidence harder for an analyst to notice. Interrupted sessions wrote raw ticket comments verbatim into the corrections store, creating a route for incorrect operational knowledge to enter future sessions. Schema validation before writes and nightly pruning were needed to address it.

Better models may reduce some reasoning errors. They cannot provide local facts that no one wrote down or detect activity that the underlying coverage misses. These are data and coverage problems. Their predictable nature makes deliberate deployment possible, but only if teams distinguish model performance from the quality of its inputs. Without tuned detection rules, centralized data, and documented runbooks, an agent amplifies existing problems at machine speed.

How teams should deploy AI triage agents in phases, and what gates each phase

The practical response is to expand responsibility only when measured analyst confidence supports it. Enrichment comes first, with agents building context packs while analysts still write their own summaries. This separates evidence collection from trust in conclusions. The gate is stable precision in what the agent retrieves and presents.

The summary phase transfers narrative construction to the agent. Analysts validate structured summaries and reasoning trails. Only after this work demonstrates stable precision should autonomous closure become available for known-good alert classes with low false-positive rates. Autonomy remains specific to demonstrated behavior. Logs should support promotion by showing stable precision, low false-positive rates, and controllable replication behavior.

Kiro’s operational loop illustrates the maintenance discipline required. Engineers review what the agent has learned weekly, validate schemas before corrections-store writes, and prune the knowledge base nightly. Presentation receives engineering attention too. Early agents posted detailed work steps in customer-visible threads, so the team moved to short key points there and retained detail in the work log. Trust depended on useful evidence.

Evaluation gates should run at the pre-merge and pre-promotion stages so that changes in how an agent behaves and what it is allowed to do follow the discipline of CI/CD. Each gate needs clear thresholds that auditors can trace to the autonomy granted. A phase transition is therefore more than a process milestone. It is a governance artifact explaining why a particular capability was allowed into production.

The security risks that are specific to agents with tool access in production environments

When agents have tool access, untrusted text can put production systems at risk. A log line, document, or tool result can contain instructions intended to redirect the model. Indirect prompt injection exploits that connection between reading and acting. A deceived agent may then use production credentials to act at machine speed as well as give a misleading answer.

Sai Varun Kodathala’s 2026 aiAuthZ research found that refusal behavior varied substantially across models and that price did not predict safety. The same attack could be refused by one model and executed by another. Documented incidents make the consequences concrete. The Replit DROP DATABASE event of July 2025 and the Claude Code infrastructure-destruction event of February 2026 both involved agents with write access taking irreversible actions.

Tool dependencies give attackers another way in. The postmark-mcp package shipped clean releases before adding email exfiltration code, so trusting an agent is no reason to trust its tools. Model refusal alone cannot secure either boundary. The aiAuthZ research places the decisive authorization check outside the agent process because an in-process permission system is compromised along with the model. That separation is the foundation for credential and policy design.

What authorization and credential design for production triage agents actually requires

Safe authorization requires a policy decision outside the agent host and credentials scoped to a single session and declared task. The aiAuthZ architecture uses an external gateway to verify caller identity through a per-message HMAC-SHA256 signature bound to a single-use nonce and a timestamp window. It then evaluates role-based and argument-level policy that the agent cannot read or modify. Every decision is written to a tamper-resistant audit log. The agent requests authority but does not control it.

The research reported residual attack success of 0% across all evaluated models with the gateway in place, at negligible added latency. That result describes attack success under the evaluated conditions, with its scope limited to those tests. The architectural requirement is the separate trust domain. A policy check inside the compromised agent’s process does not provide the same boundary, regardless of how explicitly the policy is written.

Containment complements that authorization boundary at the operating-system layer. NVIDIA’s OpenShell, launched at GTC 2026, wraps agents in kernel-level isolation governed by declarative YAML security policies. Microsoft’s Agent Governance Toolkit, released under MIT license in April 2026, provides deterministic, sub-millisecond policy enforcement addressing the OWASP Agentic AI Top 10. Its Agent SRE module covers SLOs, error budgets, circuit breakers, chaos engineering, and progressive delivery.

Kiro’s experience supports rules for each action in place of a single setting that makes the agent autonomous. Investigation should be read-only, any write should need explicit escalation, and administrator access should be blocked by default. These distinctions preserve the boundary between diagnosis and intervention.

Cost and observability controls that prevent agentic triage from becoming a liability at scale

Agentic workflows use far more tokens for each task than standard chat interactions, and multi-agent designs use even more. A single investigation can require many API calls. Reasoning loops in multi-agent pipelines have produced invoices exceeding tens of thousands of dollars over days to weeks. At enterprise scale, spend therefore requires a governed budget.

Budget enforcement should precede execution at the run, step, and tool levels. Separate limits should cover tokens, model calls, tool calls, retries, and dollars. Before a step runs, a budget manager estimates projected usage and denies the request if any limit would be exceeded. Monitoring after execution cannot prevent the spend. Model routing can cut costs the most, while prompt caching and context optimization can cut them further.

To control costs, teams also need to understand the execution path. Conventional APM and flat logs do not provide the full account of a multi-step, multi-agent investigation. Agent observability links calls into a causal tree, exposes actions inside connected tools, and binds each call to the human who authorized it. Infrastructure-level, auditable egress adds a tamper-resistant network record independent of agent-side logging. That independent evidence supports compliance claims about enforcement.

The production baseline extends beyond traces and budgets. It includes SSO integration, SIEM-connected audit logging, secret scanning on agent outputs, policy gates, sandbox isolation, and incident response runbooks. These controls do different jobs, and none can take the place of another. Together, they make the agent’s identity, access, execution, and resource use subject to operational oversight.

What changes for on-call engineers when agents handle the investigative work

Kiro describes on-call pagers becoming decision queues for work already performed by agents. The rotation remains, with engineers reviewing completed investigations for incidents that arise in the middle of the night. The reviewing engineer must still trace the reasoning, verify the evidence, and establish that the investigation involved no unauthorized action.

Documentation consequently becomes part of the operational capability of the agent. Procedures, alarm wikis, and post-incident notes provide the local knowledge that a model cannot infer reliably on its own. Teams that have underinvested in those records should expect weaker agent performance. Maintaining them is no longer only preparation for the next human responder. It also shapes the evidence available in future automated investigations.

Kiro describes a further expansion toward drafting remediation, including threshold adjustments, code reviews, and feature branches. Human reviewers would evaluate correctness, security, and production risk before those changes proceed. This moves beyond read-only investigation into write-capable work. Teams thus need to have the authorization boundaries, escalation rules, and audit controls in place before they grant write access.

A triage agent running on a laptop against staging is a prototype. In production, agents need session isolation, scoped credentials, cost caps, and a full audit trail before engineers can safely treat the pager as a queue of decisions. Governed infrastructure is what makes completed investigative work trustworthy enough to act on.