EveStudioDocs

Evals

An eval is a scored test case for your agent. You drive the agent through one or more turns and assert on what it did (the run completed, the right tool ran, the reply contains the right text) and the eval passes or fails. Evals are how you catch a regression when you change an instruction, a tool, or the model. Studio's Evals tab lists them and runs them for you.

What an eval is in Eve

Eve discovers evals under an app-root evals/ directory, in .eval.ts files. The file path is the eval's identity: evals/weather/brooklyn-forecast.eval.ts is the eval weather/brooklyn-forecast. Each is a single async test(t) function where t is both the driver and the assertion surface:

import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.calledTool("get_weather");
    t.check(t.reply, includes("Sunny"));
  },
});

The key thing to understand: evals exercise the same HTTP surface your users hit, against a live model. The runner boots (or targets) a real agent server, drives a real session, and grades what comes back. A passing eval means the agent actually booted, accepted the request, and produced the result you asserted. That realism is also why evals take time; every run makes live model calls.

Assertions come in two severities. Gates are hard: a failed gate fails the eval. Soft assertions (.soft()) are tracked as metrics and don't fail the eval unless you run in strict mode. Evals can also carry tags so you can run a fast subset.

The Evals tab

Open an agent and select the Evals top tab. It lists every eval discovered under evals/ (via eve eval --list --json): each one's id (its file path) and description.

From the list you can:

  • Run on a row: execute a single eval (eve eval <id> --json).
  • Run all: execute every discovered eval (eve eval --json).

To run the same suite against a deployment from a terminal, add --url: eve eval --url https://<your-agent>.vercel.app.

Because each run drives a real session against a live model, runs are not instant. Studio shows a running… badge with a Cancel button while a run is in flight and streams the runner's JSON into the Output console. Let it finish rather than assuming it stalled.

Reading results

When a run completes, the header shows all passed or failures, from the runner's exit code, and the Output console holds the per-eval detail:

  • Passed: every gate assertion in that eval held. This is the CLI's exit code 0.
  • Failed: a gate assertion failed, the run errored, or (in strict mode) a soft assertion missed its threshold. This is exit code 1. A configuration error is exit code 2.
  • An eval that calls t.skip(reason) is reported as skipped and counts as neither.

For the full detail behind a failure, Eve drops artifacts under .eve/evals/<timestamp>/ on the agent: a run summary.json, a results.jsonl index, and per-eval assertion results, verdicts, captured event streams, and t.log lines. The console tells you which eval failed; the artifact tells you why.

When to run them

Run the relevant evals after you change something that affects behavior (an instruction, a tool or subagent, or the model) and again before you deploy. A small set of smoke evals (assert the run succeeded, plus one or two content checks) catches most regressions. In CI, eve eval --strict --junit .eve/junit.xml turns soft misses into failures and gives your CI provider per-eval annotations.

Where to go next