How to Assign KPIs to AI Agents

A surprising number of marketing agents run in production with no KPI beyond "is it running." That's an availability metric, not a performance one, and it means a team can go months with an agent that's technically online but quietly doing a mediocre job — misrouting leads, drafting captions nobody uses, flagging the wrong competitor changes — without anyone having a number that would tell them.

Assigning real KPIs to an agent is different from assigning them to a campaign or a channel, because an agent has an intermediate layer of behavior (decisions, tool calls, output quality) sitting between "it ran" and "the business result changed." Good agent KPIs cover both layers.

Start with what the agent is actually for

Before picking metrics, write down the agent's job in one sentence with a clear success condition. "Enrich new leads with firmographic data before they reach a rep" is specific enough to measure. "Help with lead qualification" is not. If you can't state the job specifically, the KPI problem is actually a scoping problem — fix that first.

Layer 1: Task-level metrics (is it doing its job correctly)

Completion rate: the percentage of triggered runs that finish without erroring out. This is the baseline availability number most teams already track, and it's necessary but not sufficient.

Accuracy against a known-good sample: pull a set of the agent's outputs periodically and have a human judge them against a rubric — was the lead score reasonable, was the caption on-brand, was the flagged anomaly actually anomalous. This requires a defined rubric up front, which is worth the one-time effort because it turns "seems fine" into an actual number you can track over time.

Correction rate: how often a human has to edit or override the agent's output before it's usable. A rising correction rate over time is one of the earliest signals of model drift or an upstream data change, often visible weeks before it would show up in a downstream business metric.

Tool call success rate: for agents that call external systems (CRM, ad platforms, analytics), track how often those calls succeed cleanly versus fail, time out, or return unexpected data. This is infrastructure health, not model quality, but a marketing team is usually the first to notice it degrading because they're the ones consuming the output.

Layer 2: Outcome-level metrics (did it help)

Time saved per cycle: compare the time a task took before the agent existed to the time it takes now, including the human review step. Be honest about the review time — an agent that saves 2 hours of drafting but adds 90 minutes of correction has saved 30 minutes, not 2 hours.

Downstream lift on the metric it's meant to influence: a lead-enrichment agent should eventually show up in faster rep response time or better lead-to-opportunity conversion; a content-drafting agent should show up in publishing cadence or time-to-publish. Attribute cautiously — isolate the agent's contribution with a before/after comparison or a holdout group where feasible, rather than crediting every improvement in the metric to the agent.

Cost per completed task: token/API cost plus infrastructure cost plus human review time, divided by units of useful output. This number tends to be more volatile than people expect early on and worth tracking monthly rather than assuming it once at launch.

Layer 3: Drift and risk metrics (is it still the same agent it was)

Output variance over time: run the agent against a fixed, unchanging test set on a schedule (weekly or monthly) and compare outputs to the previous run. A meaningful shift with no corresponding change on your end (new prompt, new model version you approved) is a signal something changed upstream, often a silent model update from the vendor.

Escalation/override frequency: how often the agent hits a guardrail and defers to a human rather than acting. Both a rising and a falling trend are worth investigating — rising might mean the agent is encountering more edge cases than expected; falling might mean guardrails have been quietly loosened or the agent is getting overconfident.

Setting targets without guessing

Don't set a target before you have a baseline. Run the agent for two to four weeks logging every metric above without holding anyone to a number, then set targets against that observed baseline plus a reasonable improvement margin. Setting a target from a vendor's demo numbers or a competitor's case study almost always sets an unrealistic bar, because those numbers were measured on someone else's data and someone else's definition of success.

Reviewing the KPIs

A monthly review that looks at all three layers together — not just the outcome metric — is what actually catches problems early. An agent can hit its outcome KPI while its correction rate is quietly climbing, which means the outcome number is about to turn over once the drift compounds far enough. Reviewing task-level and drift metrics alongside outcomes is what gives you the warning before that happens, rather than after.