AgentOps: Run, Monitor & Improve

• Tracing from day one. Every step is traced and tagged with business context, so a problem can be traced to a user and a workflow rather than to a log line.
• Evaluation, offline and online. A curated set of known-good cases runs before every release, and live traffic is scored continuously against the same standards.
• Quality gates on change. No model, prompt or policy change ships without passing the harness. Failing production cases are folded back into the test set.
• Drift detection. Scheduled replays of the known-good set against current production, with rolling comparison windows rather than fixed thresholds.
• Cost and behaviour monitoring. Spend, latency, retries and tool usage watched per service, so a cost problem surfaces as a trend rather than an invoice.
• A rollback path. A known-good configuration to return to, tested rather than assumed.

• A managed run with agreed response times.
• A maintained evaluation suite that grows as real failures are found.
• Regular reporting on quality, adoption, cost and exceptions per service.
• Policy and model tuning as your rules and the underlying platforms change.

Most suppliers hand over at go-live, which is the point at which the real work starts. We price the run alongside the build, because an agent that nobody is watching will eventually do something nobody intended, and the organisation that answers for it is yours.

What is agent drift and how do you detect it?
Drift is a change in agent behaviour caused by something other than a deliberate design decision: a model version update, a prompt edit, a changed tool schema, or revised source content. We detect it by replaying a curated set of known-good cases against current production on a schedule and comparing results over rolling windows, rather than waiting for a complaint.

What usually goes wrong with AI agents in production?
Less often the model than people expect. The common failures are tool calls that fail or time out, context that gets truncated on long tasks, loops that do not terminate, and silent changes in an upstream system. All four are operational problems, which is why they need operational monitoring rather than a better prompt.

Can you run agents that another supplier built?
Usually yes. We assess the existing implementation, instrument it, build an evaluation suite around current behaviour and establish a rollback position before changing anything. Taking over an unmonitored agent without first establishing what “working” means is how handovers go wrong.

How is AI running cost controlled?
By monitoring spend per service and per workflow rather than as a single bill, setting budgets and alerts at the workflow level, and tuning retrieval, context size and model selection per task. Most unexpected AI cost comes from retries and oversized context rather than from volume.