Agentic Development

AI Agent Monitoring: Running Coding Agents in Production

AI agent monitoring is what separates coding agents that investigate incidents and generate fixes safely from those that guess. Learn how to connect logs, traces, and verification evidence so production agent activity stays observable, prioritized, and trustworthy.

MMichael SwindellSeptember 8, 202615 min read

Running coding agents in production means operating software where agents can investigate incidents, use tools, generate changes, and assist with remediation against live systems and real operational data. The central challenge is not code generation. It is providing enough trusted context, constraints, and evidence for useful decisions without creating an unmanageable stream of changes. A harness that guarantees deterministic execution is what makes production monitoring trustworthy in the first place. Teams need production observability, service relationships, incident history, clear risk boundaries, and human approval paths. An agent-first control plane can connect work, acceptance criteria, tests, and accountability so agents act within an explicit delivery process rather than an isolated prompt.

At a Glance

Production agents need context, evidence, prioritization, and capacity models that include tool use.

Context first: Logs alone are rarely enough. Agents need code, incidents, service relationships, and recent changes.
Human attention is scarce: Automatically generated pull requests fail when they create review work without a clear business case.
Prioritize narrowly: Rank proposed changes by business impact, operational risk, and implementation size.
Benchmark real workflows: Tool calls create idle GPU time, so model-only load tests can underestimate supported concurrency.
Trace before guessing: A useful trace gives an agent evidence for investigating rare, hard-to-reproduce failures.

Why is AI agent observability critical in production?

Coding agents diagnose production problems best when they can correlate logs with code, incident records, service dependencies, recent changes, and runtime configuration. A log search can identify symptoms, but operational context helps an agent distinguish an isolated error from a change that affects connected systems and customers.

Production systems are not static. Services change, variables are renamed, infrastructure mappings drift, dependencies move, and security patches alter behavior. When an incident occurs, the useful question is not only, “What failed?” It is also, “What changed, what depends on it, and what is the likely blast radius?”

This is why a production-ready context layer should connect several sources of truth:

Application and infrastructure logs: The event record and its surrounding signals.
Source code and deployment history: The implementation and recent modifications.
Incident reports: Prior symptoms, mitigations, and known operational patterns.
Service topology: The relationships between services, accounts, endpoints, and dependencies.
Configuration context: Environment variables, network routes, identity boundaries, and versioned settings.

Distributed traces are particularly valuable because they represent a request as a connected path across systems, rather than isolated log entries. The OpenTelemetry tracing documentation describes traces as a way to follow requests through distributed systems, including the spans created by individual operations. That structure gives coding agents a better starting point for evidence-based diagnosis. This operational discipline builds on the harness itself — see harness engineering and the path to a software factory for how that foundation gets built.

See What Every Agent Verified Before It Acted

The Levr Dashboard tracks every agent's activity against its linked tests and workflow gates, so production fixes stay evidence-based rather than speculative. Each proposal traces back to the issue and acceptance criteria that triggered it.

Agentic Software Development | Control Plane | Manual & Automatic Verification Issue Tracking |Levr

Levr Dashboard: agent activity tied to verification gates.

How should teams use agents for production incident diagnosis?

Use agents to accelerate investigation, not to bypass operational judgment. Start with read-oriented access to observability data, incidents, and relevant code, then require agents to produce an evidence-backed diagnosis, likely impact, and proposed next action before granting any ability to modify production systems.

A reliable incident workflow separates discovery from action. The agent can search broadly, correlate records quickly, and summarize possibilities. The human operator still decides whether the evidence justifies a rollback, configuration change, patch, or escalation.

Use an evidence packet

Give the agent a bounded package of information rather than an unstructured request to “fix production.” For example, an incident packet might include:

The incident timestamp and affected environment.
Relevant traces, logs, and error-rate changes.
The last successful and first failing deployment.
Linked services and known upstream or downstream dependencies.
Recent incident records that match the same error signature.
The acceptance condition for recovery, such as restored endpoint behavior or passing checks.

This approach makes review easier. The agent should be able to point to the data that supports its conclusion, identify uncertainty, and state what additional evidence would change its recommendation.

Keep permissions proportional to the task

Production diagnosis does not require unrestricted production control. Begin with access that supports investigation, such as reading logs, traces, deployment metadata, and tickets. Treat write access, destructive operations, credential changes, and customer-impacting actions as separate approval boundaries.

The Google SRE guidance on managing incidents reinforces the value of clear incident roles, coordinated response, and documented processes. Coding agents can assist with analysis and coordination, but the operational system still needs accountable decision-making.

Why do automated remediation pull requests often go unread?

Automated pull requests go unread when they ask humans to assess too many changes with unclear priority, unclear ownership, and uncertain payoff. Even safe-looking patches consume review attention, so a large queue of agent-generated work can resemble an alert backlog that nobody has time to trust or triage.

The failure is not that the agent found possible improvements. The failure is treating every technically valid optimization as work that deserves immediate human attention.

A reviewer needs a concise answer to four questions:

Why now? What customer, revenue, reliability, security, or operational outcome is affected?
How large is the gain? State the likely improvement in terms the owning team can evaluate.
What could break? Describe operational risk, migration requirements, and affected dependencies.
What will review cost? Identify the code surface, tests, rollout plan, and rollback path.

A pull request is an implementation artifact, not automatically a prioritized unit of work. If the human case has not been made, opening more pull requests just increases cognitive load.

How to use AI agent monitoring for prioritization?

Prioritize agent-generated fixes by combining business impact, frequency of exposure, implementation risk, and change size. The best first candidates are narrow changes with meaningful upside, low operational risk, no difficult migration path, and a clear connection to a business-critical workflow.

This changes the agent’s job from “find every possible optimization” to “identify the changes most likely to justify human review.” A useful prioritization model is simple enough to explain in a ticket or incident follow-up.

A practical impact and risk filter

Business criticality: Does the change affect payments, authentication, or another path the organization already considers important?
Traffic or execution frequency: Does the affected endpoint or workflow run often enough for an improvement to matter?
Expected outcome: Can the agent estimate a measurable benefit, such as reduced latency or fewer failures?
Implementation risk: Does the work avoid data migrations, broad architectural changes, and fragile dependencies?
Reviewability: Can a developer understand the reasoning and validate the patch without a long investigation?

The output should be a ranked recommendation, not an unattended flood of changes. Start with a brief human-facing summary, then link to the evidence, the proposed patch, required tests, and rollback conditions.

In Levr, one way to make this operational is to define the fix as an issue with structured acceptance criteria, attach relevant test coverage, and hold the work behind workflow gates. Agents can update the same project objects that humans review, creating an attributable record of the proposed work and its verification state.

Give agent work a visible path from diagnosis to approval.

Levr connects agent-generated issues, acceptance criteria, tests, and review gates so high-priority production work can be evaluated as a complete operational change, not just another patch.

Explore agent-first project workflows

Why do tool calls change AI agent capacity planning?

Tool calls change capacity planning because an agentic request alternates between model inference and external work such as retrieval, API calls, or CPU-based processing. During those non-GPU phases, accelerator capacity may be available for other requests, making model-only benchmarks an incomplete representation of real concurrency.

A benchmark that assumes every request continuously occupies a GPU can produce an overly conservative capacity estimate when real agent workflows spend meaningful time waiting on tools. The effect depends on how often tools are called, how long they take, and whether requests can be scheduled efficiently while those calls are in progress. Much of what shows up in production traces traces back to tool binding and runtime interrupts decisions made at the harness level.

Model the full request lifecycle

Measure more than tokens per second. For each representative workflow, track:

Model inference duration.
Number of tool calls per request.
Tool latency and variance.
CPU activity during tool execution.
Queue time before and after inference.
End-to-end task completion time.

Consider a simplified case where inference takes one second and a tool call takes another second. The GPU does not need to remain occupied for the complete two-second lifecycle. If the serving system schedules another request during the tool phase, the same GPU can support more simultaneous user workflows than a continuous-inference benchmark suggests.

Do not translate this into a universal multiplier. Tool latency, batching behavior, state management, hardware, and scheduler design all matter. The key lesson is methodological: benchmark the workload your agents actually run, including waiting, tool execution, retries, and multi-step behavior.

How can traces help agents fix rare production bugs?

Traces help agents fix rare bugs by replacing speculation with a time-ordered record of what actually happened. When a bug is difficult to reproduce, a trace can show the execution path, timing gaps, failed branch, and related operations that an agent needs to form and test a concrete hypothesis.

Without a reproduction or trace, an agent may generate plausible fixes for the wrong failure mode. It might never trigger the condition locally, or it might optimize a nearby code path that has nothing to do with the reported behavior.

Build targeted tracing before broad automation

You do not need to connect an experimental agent directly to every production system to get value from tracing. A focused framework for test runs, performance investigations, or overnight jobs can capture the details needed to investigate intermittent failures.

Useful trace data can include request identifiers, operation order, timestamps, relevant parameters, state transitions, exceptions, dependency calls, and test environment details. The goal is not to record every byte forever. The goal is to capture enough information to explain why a specific run behaved differently.

Tracing adds overhead, which can distort performance measurements. Use that limitation explicitly. A tracing configuration intended for debugging may not be suitable for final performance benchmarking. Keep diagnostic and performance test modes separate, then compare their outputs with that instrumentation cost in mind.

What should an agent-ready production workflow include?

An agent-ready production workflow includes shared operational context, defined autonomy boundaries, evidence-based recommendations, automated verification, and accountable human approval. It should make it easy to answer what changed, what the agent did, which checks passed, who approved the action, and how to roll it back.

The workflow should work whether agents are only investigating or are also preparing patches and tests. The critical point is that every stage remains visible and connected.

Define the operational intent: Create a clear issue describing the incident, observed impact, and recovery target.
Attach trusted context: Link traces, logs, incident records, affected services, and relevant code changes.
Bound the agent task: Specify which data the agent can read, what it may propose, and which actions require approval.
Require a ranked recommendation: Ask for expected impact, risk, evidence, tests, and rollback conditions.
Verify before completion: Run relevant automated checks and record the results against the acceptance criteria.
Keep human review explicit: A responsible engineer decides whether to merge, deploy, defer, or reject the proposed change.
Preserve the activity trail: Retain the reasoning, generated artifacts, test results, approvals, and final outcome for future incidents.

Levr’s agentic workflow is designed around this progression: define intent, let agents take on scoped work, verify outcomes, and retain human review and approval where it matters.

Technical Deep Dive FAQ

What does it mean to run an agent in production?

Running an agent in production means using a coding agent in connection with live operational workflows, systems, or data rather than only in a local development environment. The agent may investigate incidents, inspect observability signals, prepare code changes, run approved tools, or summarize operational evidence. Production use requires more than a capable model. It requires clear access boundaries, trustworthy context, verification steps, and accountability for actions that affect customers, services, or sensitive systems.

What is AI observability for coding agents?

AI observability for coding agents is the ability to inspect what an agent was asked to do, which context and tools it used, what actions it attempted, and what outcome followed. In production, that also includes links to logs, traces, deployments, test results, and incident records. Observability does not make an agent correct by itself. It gives operators the evidence needed to assess agent conclusions, diagnose failures, audit actions, and improve the workflow over time.

Can coding agents diagnose production incidents?

Yes, coding agents can assist with production incident diagnosis when they have access to relevant, trusted context. Combining logs, service information, incident history, code, and traces lets an agent search and correlate evidence faster than a manual investigation alone. The agent should present a diagnosis as a reasoned recommendation, not as unquestioned fact. For consequential changes, a human operator should evaluate the evidence, approve remediation, and confirm that recovery conditions have been met.

What context should an incident-response agent receive?

An incident-response agent should receive only the context needed for the investigation: the incident timeframe, affected services, traces, relevant logs, recent deployments, linked code, known dependencies, and prior incident records. It should also receive a clear definition of the expected healthy state. Avoid handing the agent broad, unrelated access simply for convenience. Narrow context reduces ambiguity, limits exposure, and makes it easier for a human to assess whether the resulting diagnosis is well supported.

Why should agent-generated pull requests be prioritized?

Agent-generated pull requests should be prioritized because each one creates review, ownership, testing, and deployment work for a human team. A technically valid patch may still be low value, poorly timed, or too risky to justify attention. Before opening a pull request, an agent should explain the business impact, affected workflow, expected gain, risk level, required validation, and rollback path. That turns automated output into a decision-ready proposal instead of another unranked item in an already crowded queue.

How do you measure the value of an agent-proposed fix?

Measure an agent-proposed fix by connecting the change to a meaningful operational outcome. Start with the affected path, such as authentication, payment processing, or a frequently invoked endpoint. Then estimate the expected improvement, identify the population affected, and describe the implementation and rollout risk. The goal is not perfect forecasting. It is giving engineering and product owners enough information to compare the proposed work with competing priorities and decide whether review and implementation are justified.

Why do tool calls affect GPU utilization?

Tool calls affect GPU utilization because an agentic request does not spend its entire lifecycle performing language-model inference. When the agent waits for a database query, external API, retrieval system, or another CPU-driven tool, the GPU may be idle for that request. Capacity tests that model only inference can miss this behavior. A realistic benchmark measures model time, tool time, queueing, and end-to-end latency so the team can estimate concurrency from the actual workflow.

How should teams benchmark agentic workloads?

Benchmark agentic workloads with representative multi-step tasks rather than single model calls. Include the number and duration of tool calls, retries, external service latency, CPU work, model inference, queueing behavior, and final task-completion time. Run scenarios that reflect normal and high-latency tool behavior because variance can change both throughput and user-perceived latency. Capacity planning should then account for where hardware is active, where requests wait, and whether the scheduler can use otherwise idle inference capacity.

How do traces improve debugging for rare failures?

Traces improve debugging by recording the path of an actual failing execution. This matters for rare bugs because neither an engineer nor an agent may be able to reproduce the failure on demand. A trace can reveal operation order, timing relationships, failed dependencies, state transitions, and exceptions from the affected run. An agent can then investigate a specific hypothesis against evidence rather than proposing a patch based only on a bug report or an incomplete error message.

Should agents be allowed to fix production incidents automatically?

Automatic remediation should match the risk of the action. Low-risk, reversible, well-tested actions may be candidates for carefully bounded automation. High-impact actions, production configuration changes, destructive operations, and changes with uncertain blast radius should require human approval. The safer model is progressive autonomy: start with investigation and recommendations, add patch generation and automated verification, then expand execution rights only where evidence, controls, and rollback procedures support that level of trust.

How can engineering managers keep agent actions accountable?

Engineering managers can keep agent actions accountable by requiring every agent task to be linked to a defined issue, acceptance criteria, verification results, and an attributed activity record. The team should be able to identify what context informed the task, which tools were used, what artifacts were created, which tests ran, and who approved the final outcome. This makes agent work inspectable alongside human work and prevents operational knowledge from disappearing into disconnected prompts and private execution logs.

Key Takeaways

Production agents are most useful when they are treated as evidence-driven participants in an operational workflow, not as unattended sources of code changes.

The practical operating model for reliable coding agents

Start with context: Link logs, traces, code, incidents, dependencies, and recent changes before asking an agent to diagnose a problem.
Optimize for reviewer trust: Send ranked recommendations with impact, risk, evidence, and rollout conditions instead of opening every possible pull request.
Measure complete workflows: Include tool calls and waiting time in capacity tests, not only model inference.
Instrument rare failures: Traces give agents the evidence they need to investigate difficult bugs without guessing.
Scale autonomy carefully: Keep high-impact actions behind verification gates and accountable human approval.

Faster agent output only improves reliability when context, verification, and decision-making keep pace with it.