Measuring AI agents: evaluation methods that predict production reliability

Background

An AI agent does more than answer a question. It reads an input, decides what to do, calls tools such as a database query, an ERP API or an email sender, reads the results and decides again. A single task might take five or fifteen steps.

That makes agents useful for work such as reconciling payments against invoices, triaging support tickets or preparing purchase orders. It also makes them hard to measure. A wrong step early on can lead to a confident, well formatted and completely wrong result at the end.

Teams usually build some evaluation before launch. The problem is that many of these evaluations do not predict what happens next. The score looks good, the agent goes live, and reliability turns out to be lower than anyone expected.

Approach

This paper is based on analysis of public documents and the practices we use in delivery. We used the NIST AI Risk Management Framework 1.0, particularly its Measure function, ISO/IEC 42001:2023 for the expectation that AI systems are monitored across their life, and the OWASP Top 10 for Large Language Model Applications for the security failure modes an evaluation should cover. For systems that may fall within Regulation (EU) 2024/1689 (the EU AI Act), we noted the emphasis on accuracy, cybersecurity and human oversight for high risk systems.

The evaluation structure itself comes from how we test agents before and after release. No scores are reported, because a score without its cases and grading method is not meaningful to a reader.

Findings

1. Evaluation cases are too clean

Evaluation sets are often written by the team that built the agent, from examples that occurred to them. Production inputs are messier: incomplete records, duplicate entries, scanned documents, mixed languages, customers who change their request halfway through an email thread. An evaluation that does not contain these cases will overstate reliability.

2. Grading rewards plausibility

When outputs are graded by a person skimming them, or by another model with a vague rubric, fluent and confident answers score well whether or not they are right. For agents that act, the question is whether the right record was updated with the right value.

3. One run is not a measurement

Agents built on language models do not always take the same path through a task, even with the same input. A case that passes once may fail one time in five. A single run per case hides this variance completely.

4. Tool failures are not tested

In production, APIs time out, return errors, return empty results or return something in an unexpected format. An agent that has only ever seen clean tool responses will meet these for the first time with real users.

5. Adversarial input is ignored

Agents that read emails, documents or web pages can be given instructions hidden in that content. The OWASP list puts prompt injection first for good reason. An evaluation that does not include such inputs says nothing about how the agent will behave when it meets one.

6. Cost and time are left out

An agent that reaches the right answer in forty steps and at ten times the expected cost is not a success in production. Step count, latency and cost per task belong in the evaluation alongside correctness.

A framework

We use the Four Layer Evaluation Stack to decide what to test, and the Five Tests of a Predictive Evaluation to decide whether to trust the result.

The Four Layer Evaluation Stack

1. Step layer. Does each component behave correctly in isolation? Examples: does the extraction step pull the right GSTIN and invoice total from a document, does the classifier route a ticket to the right queue.
2. Task layer. Given a real input, does the agent reach the correct final state? Grade the end state directly where possible: the right ledger entry posted, the right reply drafted, the right exception raised.
3. Trajectory layer. Did the agent take an acceptable path? Check for forbidden actions, unnecessary tool calls, loops and steps that touched data they should not have.
4. Operational layer. Under realistic conditions, what are the cost per task, time per task, rate of escalation to a human and behaviour when tools fail?

The Five Tests of a Predictive Evaluation

Before trusting an evaluation's result, check that it passes all five.

1. Representative. Cases are drawn from real production inputs, or from a recent period of real work, including the awkward ones. A sample of production traffic should be added to the set regularly after launch.
2. Objectively graded. Each case has an expected outcome that can be checked by code or by a precise rubric. Where a model grades, its grading is itself checked against human judgement on a sample.
3. Repeated. Each case runs several times, and the result reported is the pass rate across all runs.
4. Stressed. The set includes tool failures, malformed tool responses, missing data and prompt injection attempts, with the expected safe behaviour defined for each.
5. Costed. Every run records steps, tokens, cost and latency, and the evaluation has thresholds for these as well as for correctness.

Using the results

• Set a release threshold on the task layer and hard limits on the trajectory layer. A single forbidden action in the evaluation should block release regardless of the overall score.
• Compare every change, whether a prompt, a model version or a tool, against the same set before release.
• After launch, compare production outcomes with evaluation results each week. If they diverge, the evaluation set is no longer representative and needs new cases.

Recommendations

1. Collect production cases from day one. In a pilot or shadow phase, save real inputs and the correct outcomes as decided by the people who do the work.
2. Grade end states. Where the agent changes a record, check the record.
3. Run each case at least several times. Report pass rates and look closely at cases that pass sometimes and fail sometimes.
4. Write the failure cases deliberately. Spend a day breaking your own tools and planting instructions in documents, then add those cases to the set.
5. Put cost in the release decision. Agree a cost and latency budget per task before the build and hold the agent to it.
6. Keep a human in front of risky actions. Evaluation reduces risk but does not remove it. Approval steps for payments, deletions or external messages remain sensible even when scores are high.

Limitations

This paper offers a structure. We do not claim that following it guarantees production reliability. Our claim is that evaluations which fail the five tests are unlikely to predict it. Building representative and stressed evaluation sets takes real effort from domain experts, which small teams may find hard to fund. Model based grading, which we accept with checks, carries its own biases. The framework is designed for task oriented business agents. Open ended conversational systems and creative generation need different measures, and we do not address them here.

Sources

• NIST AI Risk Management Framework 1.0
• ISO/IEC 42001:2023
• OWASP Top 10 for Large Language Model Applications
• Regulation (EU) 2024/1689 (EU AI Act)