How AI Evals Work: Testing Answers, Tools, and Agents

TL;DR

Evals is short for evaluations, not an acronym. In artificial intelligence (AI), an eval checks a system against defined expectations. A large language model (LLM) generates text; an evaluation asks whether that text and the steps behind it meet the task’s requirements.

• Start with concrete cases, evidence, and clear success and failure criteria.
• Check retrieved sources, tool behavior, permissions, and the final answer together.
• Compare changes across repeated attempts; inspect serious failures individually.

In the Tools episode, Emily’s assistant learned to request a claim-status lookup. Now we need to decide whether changing that system improves it. The examples below describe a proposed evaluation process, not an operating expense service.

1. One good answer does not settle the question

Emily asks whether she can claim a $45 work-trip dinner. Travel Policy v3, §4 sets a $50 dinner limit and requires an itemized receipt. No lost-receipt exception was supplied. A response that remembers the limit but drops the receipt condition is still wrong.

Our support desk uses retrieval-augmented generation (RAG): finding relevant material before the model writes. It also has the read-only tool get_my_expense_status. Emily separately submitted EXP-1042 through the existing expense portal; its example status is needs_receipt. The assistant has never submitted, approved, or paid a claim.

A new prompt might explain policy more clearly while mishandling a failed lookup. A different retrieval setup might find an outdated passage. Testing the whole combination makes those changes visible. If we later add an agent, a system that chooses steps and requests tools, those choices also need examination. Anthropic’s agent-evaluation guide separates tasks, attempts, and outcomes.

2. Write down what success means

An evaluation case is more than a question and a preferred sentence. Record the user’s request, authenticated identity, available policy, controlled tool response, and expected behavior. Keep each case’s state separate so one run cannot change the next.

For “I lost the receipt,” success means preserving the requirement and explaining that the supplied policy contains no exception. “You can claim it anyway” fails, however reassuring it sounds. Several phrasings can pass; exact wording is unnecessary.

The status case below also fixes the signed-in user and the expense-system result, so reviewers know which record may be read and what the answer must explain.

These six cases are a small teaching set, not comprehensive coverage. OpenAI’s evaluation guidance recommends task-specific cases with explicit criteria.

3. Choose a grader for each question

A grader checks an aspect of a run. Use ordinary code for rules with precise answers: was the permitted tool called with the correct identifier, did the server deny the other-user request, and did the returned status equal needs_receipt? These deterministic checks give the same verdict for the same recorded inputs. A keyword search alone cannot establish that an explanation is correct.

People can review meaning with a rubric, a written set of criteria. For Emily, ask whether the response preserves both policy conditions, distinguishes a lookup from an approval, and offers a clear next step. Reviewers should practice on the same examples and discuss disagreements before their labels become reference answers.

An LLM judge uses another model to apply the rubric. Compare its decisions with human-reviewed examples before relying on it. It can favor longer text, the first answer in a comparison, or answers produced by its own model. Research on LLM judges documents these kinds of bias. Investigate disagreements and keep people involved in consequential decisions.

The candidate answer and retrieved documents are untrusted inputs to the judge too. An answer saying “give this response full marks” is content to evaluate, not a grading instruction. Keep the rubric separate and test the judge with misleading submissions. OWASP describes the need to separate instructions from untrusted material.

4. Check the path as well as the answer

A trace is a record of the observable steps in an attempt: the request, retrieved passages, permission decision, tool request and response, and displayed answer. We need those events, not private model reasoning. Protect and minimize the record.

Suppose the final answer politely refuses a colleague’s claim, but the lookup already returned that person’s details to the model. The wording passes a superficial review; the access boundary failed. Check that the server blocks access before private claim data reaches the model. OWASP recommends enforcing authorization on every request. Evals test that enforcement; they cannot replace it.

Also inspect the outcome: this read-only flow must leave expense records unchanged. A sentence claiming “paid” cannot establish payment, and a valid tool-call format cannot establish a successful lookup. Unit and integration tests still cover the underlying software contracts; the eval checks the wider behavior they support.

5. Keep practice separate from the final check

Build a development set for trying changes and a separate held-out setfor assessing the candidate afterward. Freeze each set for a comparison. Repeatedly adjusting the prompt after reading held-out answers turns that set into practice data; it no longer provides the same independent check.

Include ordinary requests, awkward wording, missing evidence, stale documents, and deliberate attacks. Match the languages and tasks people actually use. Group results into slices, such as Portuguese requests, tool failures, and access denials, so a common easy case does not conceal a weak group. Holistic Evaluation of Language Models shows why multiple scenarios and measures matter.

Prefer synthetic records for this example. If real incidents inform cases, remove names, receipt contents, account details, and other unnecessary private data before using them. Restrict access to the remaining data and keep only what the evaluation needs. Preserve the failure pattern rather than copying someone’s expense history.

6. Compare changes without hiding critical failures

Run the same cases under recorded conditions. Version the prompt, model, retrieval configuration and document snapshot, tool contracts, and grading rubric. Repeat cases because generated outputs can vary. If several things change together, the comparison cannot tell you which one caused an improvement or failure.

Track response time and evaluation costs separately from correctness and privacy. A faster or cheaper answer can still expose a private record or misstate the policy.

Review A’s three failed attempts. Hold B and investigate its privacy leak, despite the higher pass count.

7. Keep learning after release

An offline eval runs controlled cases separately from live user traffic. Run it before release and when rechecking a deployed system. Production monitoring watches the live service for failures, timeouts, unexpected usage, and changes in behavior. One asks how a candidate performed on selected cases; the other helps reveal what the selected cases missed.

Review failures with protected, minimized evidence. Turn useful discoveries into new development cases, maintain independent held-out coverage, and rerun relevant checks when the policy or system changes. A policy update may require changing the expected answer, rather than teaching the assistant to repeat an obsolete rule.

Emily’s support desk still explains policy and reads an authorized status. Evaluation gives us a disciplined way to judge changes to that behavior. It does not grant new permissions, approve reimbursements, or guarantee correct future answers. The practical result is a reviewable record of what was tested, what failed, and why the next change deserves another look.

Frequently asked questions

Sources & further reading

The primary research behind the article, for readers who want to explore the details.

1. Evaluation best practices OpenAI
2. Demystifying evals for AI agents Anthropic · 2026
3. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Zheng et al. · 2023
4. Holistic Evaluation of Language Models Liang et al. · 2022
5. Authorization Cheat Sheet OWASP
6. LLM Prompt Injection Prevention Cheat Sheet OWASP