Evals

An eval is a repeatable check that runs your agent against a fixed input and gates the recorded trajectory: the run completed, the right tool ran, the reply has the right shape. Evals are how you know a prompt tweak helped, a refactor didn't regress the agent, and last month's fix is still holding.

Evals exercise the same surface your users hit. The runner starts (or targets) a real agent server, drives sessions over the public API, and grades what comes back. A passing eval means the agent started, accepted a message, and did what you asserted.

NOTE

Import paths here use @cursor/july/evals. On projects still using @anysphere/agent-serve, swap the import and run agent-serve eval. See Run the CLI for the full rename table.

Define evals with defineEval

The Agent SDK discovers evals under the project-root evals/ directory, in .eval.ts or .eval.js files. That's a sibling of agent/, never inside it (agent/evals/ is silently ignored). TypeScript is the normal authoring format.

The file path is the eval's identity, so you don't author an id. Directories group related evals: evals/builds/api.eval.ts becomes id builds/api. An index filename collapses to its directory, so evals/builds/index.eval.ts becomes builds.

An eval is a single async test(t). You drive the agent with t and assert on the run with the same t:

// evals/readiness.eval.ts
import { defineEval, includes } from "@cursor/july/evals";

export default defineEval({
  description: "Inspects a PR without approving it.",
  tags: ["smoke"],
  timeoutMs: 120_000,
  async test(t) {
    await t.send(
      "Is https://github.com/acme/checkout/pull/42 ready to approve?"
    );
    t.succeeded();
    t.calledTool("inspect_pr");
    t.notCalledTool("approve_pr");
    t.check(t.reply, includes(/ready|approve/i));
  },
});

One file can also hold several datapoints through cases (provide either test or cases, not both). Each case id becomes <fileId>/<case.id>:

// evals/prs.eval.ts → prs/checkout, prs/search
export default defineEval({
  tags: ["smoke", "prs"],
  cases: [
    {
      id: "checkout",
      description: "Checkout PR readiness.",
      async test(t) {
        await t.send(
          "Is https://github.com/acme/checkout/pull/42 ready to approve?"
        );
        t.succeeded();
        t.calledTool("inspect_pr");
      },
    },
    {
      id: "search",
      async test(t) {
        await t.send(
          "Check https://github.com/acme/search/pull/7 before approval."
        );
        t.succeeded();
        t.calledTool("inspect_pr");
      },
    },
  ],
});

Case ids must be single path segments, unique within the file. Each case can set its own description, tags, timeoutMs, and iterations. A case-level value replaces the file-level value for that datapoint.

Iterations

iterations (file or case, default 1) runs a datapoint repeatedly. Discovery expands iterations: 3 on case nyc to runnable ids weather/nyc/1, weather/nyc/2, weather/nyc/3 (filter prefix weather/nyc still selects all three). Each expanded case exposes t.iteration / t.iterations on the test context. Cap is 100.

maxConcurrency counts authored datapoints, not expanded iterations: siblings …/1……/n share one concurrency slot and run sequentially. A suite with 11 cases × 3 iterations and maxConcurrency: 20 therefore has at most 11 cases in flight, not 33.

Configure eval runs

Each project with evals needs evals/evals.config.ts or evals/evals.config.js, and it must set maxConcurrency. Each case issues real model-provider requests, so concurrency is capped hard at 200. Existing projects use 20. Discovery with eval --list works without this file, but running a case does not.

import { defineEvalConfig } from "@cursor/july/evals";

export default defineEvalConfig({
  maxConcurrency: 20, // required
  // timeoutMs: 180_000, // optional project-wide default
  // judge: { model: "..." }, // default judge model for t.judge.*
  // reporters: [], // destinations that observe every case
  // maxPlaygroundRuns: 50, // playground /v1/dev/evals history only (default 20)
});

The timeout order is case or file timeoutMs, CLI --timeout-ms, project config timeoutMs, then the 180-second runner default.

The optional fields:

Option	Default	Meaning
timeoutMs	180_000	Project-wide per-case timeout
judge	unset	Default judge model for t.judge.*; see Judge free-form output
reporters	unset	Destinations that observe every case; --skip-report suppresses them
maxPlaygroundRuns	20	Max batches in the playground / /v1/dev/evals* history (not CLI eval)

Reporters come from @cursor/july/evals/reporters: JUnit writes a JUnit XML file for CI, Artifacts writes per-case files, and combineReporters merges several into one (renderJUnitXml renders the XML for a custom destination). A file or case can add its own reporters on top of the config list.

Playground batches survive restarts whenever agent/storage.ts exists with an evals table or a KV core providing delete and list (the table is derived over the core); see Storage. Without storage they live in process memory and disappear when serve exits — navigating away and back still works while the process is up.

Drive and assert with t

t is both the driver and the assertion surface. You write ordinary control flow, sending turns and asserting inline.

Drive the agent with t.send(message, options?). It runs one turn and waits for the session to park or fail. Multiple sends in one case share the session, which is how you write multi-turn evals.

Each t.send resolves to a turn result with message, sessionId, events, toolCalls, ok, and index. The turn carries the same assertion vocabulary as t, scoped to that turn, so you can grade an intermediate turn before the next send overwrites t.reply. turn.expectOk() throws when the turn failed, for later steps that depend on it.

Read the full case state with t.reply (the last assistant text), t.events (every captured session event across turns), t.turns (settled turns, oldest first), and t.sessionId. t.signal aborts when the case hits its timeout; pass it to your own async work.

Assert with the gates:

Gate	Checks
t.succeeded()	the run did not fail and is not parked on an unanswered approval
t.parked()	the run cleanly parked on an unanswered approval request
t.messageIncludes(token)	the joined assistant text matches a string or RegExp
t.calledTool(name, matcher?)	a matching call to name happened
t.notCalledTool(name)	no request for name, in any lifecycle state
t.loadedSkill(name)	the agent opened the skill's SKILL.md (read, grep, or shell cat)
t.toolOrder(names)	tool requests appear in this relative order (extra calls allowed)
t.usedNoTools()	no tool calls at all
t.maxToolCalls(max)	at most max tool calls
t.noFailedActions()	no tool call reported an error
t.calledSubagent(name, matcher?)	a matching subagent delegation happened
t.taggedArtifact(kind?, predicate?)	at least one artifact was tagged
t.event(type, matcher?)	at least one matching event of type occurred
t.notEvent(type, matcher?)	no matching event of type occurred
t.eventOrder(matchers)	matching event groups occur in this relative order
t.eventsSatisfy(label, predicate)	your predicate over the typed event stream
t.check(value, expectation)	any value, against a builder
t.score(name, value)	records a 0–1 score you computed; soft until you add a bar
t.requireToolCall(name, matcher?)	gates on a matching call and returns it, so later code can read its input and output
t.requireInputRequest(filter?)	gates on exactly one pending approval request and returns it

Every gate returns a handle: .soft() demotes it to tracked-only, .atLeast(0.7) adds a soft score bar, and .gate(0.8) promotes a scored assertion into a hard gate.

With no matcher, calledTool is request-based: a requested call counts even when its result has not arrived. Pass t.calledTool("inspect_pr", { status: "completed" }) to require the call to return. input, output, and count matcher fields accept a literal, a RegExp, or a predicate.

The expectation builders are includes(string | RegExp), equals(value), matches(schema), similarity(expected), and satisfies(predicate, label). includes stringifies its input, equals compares values deeply, matches validates against a Standard Schema (or anything with safeParse, like Zod), similarity scores normalized text similarity, and satisfies runs your predicate. The plain function normalizedSimilarity(actual, expected) returns the same 0–1 score for use with t.score.

A few more context members shape a case: t.require(value, expectation) records a gate and stops the test body when it fails, without a duplicate execution error. t.skip(reason) ends the case as skipped (reported separately, never changes the exit code; call it before sending messages). t.metric(name, value) records a structured score for the playground case card. t.log(message) records a debug line for the CLI and playground result.

Three t.send options apply on session create (first t.send only):

• workspaceFiles — { path: contents }, seeded into the local session workspace. Prefer this over machine-local paths.
• workspaceDir — absolute harness cwd (local runtime).
• cloud — per-session cloud options merged over the agent's static cloud config (repos / env / …). Use a pinned repos override to attach a fixture repo for cloud evals without putting it on the agent's default cloud.repos. Cloud ignores workspaceFiles seeds.

const toolResults = t.events.filter((e) => e.type === "action.result");
t.check(
  toolResults.length,
  satisfies((n) => (n as number) <= 4, "at most 4 tool calls")
);

A case with no explicit gates falls back to whether at least one turn completed successfully. Add t.succeeded() and behavior-specific gates anyway. They make the contract visible during review.

Judge free-form output

When wording matters and no regex captures it, t.judge grades the reply with an LLM. The built-in graders are factuality(expected), summarizes(expected), closedQA(criteria), and sql(expected). Each scores t.reply by default; pass { on } to grade another value.

t.judge.factuality("It is 54°F in NYC right now.").atLeast(0.7);

Judge assertions are soft by default, so a judge never fails a build until you give it a bar with .atLeast(0.7) or promote it with .gate(0.8). The judge model comes from defineEvalConfig({ judge }), defineEval({ judge }), a case-level judge, or a per-call { model } override; the nearest one wins. For a domain-specific judge whose verdict is not a single score, t.judge.model(prompt) sends a raw prompt to the same model and returns the reply. You then record the parsed result with t.score or t.check.

Run evals from the CLI

The eval command discovers, filters, and runs cases.

Run the CLI under Node 22.13 or newer. Do not use Bun. Its HTTP/2 client breaks tool-result streams and causes eval turns to fail.

agent-sdk eval --dir . --list                         # discover only
agent-sdk eval --dir .                                # run all
agent-sdk eval --dir . builds/checkout                # one datapoint
agent-sdk eval --dir . builds search                  # several ids or prefixes
agent-sdk eval --dir . --tag smoke --tag pull-request # any matching tag
agent-sdk eval --dir . --json --no-stream             # machine-readable results
agent-sdk eval --dir . --verbose                      # logs + reply snippets

Id filters use OR semantics. Each filter selects an exact id and its descendants. For example, builds selects builds, builds/checkout, and every other case below that path. Repeated tags also use OR semantics. When you provide both ids and tags, a case must match both groups.

eval boots an ephemeral server on port 0 with a temp state root outside the project, so cases don't inherit ambient monorepo rules and don't pollute .agent-serve/. Point --url at a running server to eval a live agent instead:

agent-sdk eval --dir . \
  --url http://127.0.0.1:3000/weather-agent \
  --bearer-token "$AGENT_TOKEN"

The eval definitions still come from --dir; --url only changes the agent that receives the turns. For a locally mounted multi-agent directory, --slug weather-agent chooses the target. Use --state-root to keep ephemeral session state at a chosen path, --timeout-ms to override the project timeout, and --no-stream to keep live progress off stderr. A TTY streams turn progress by default. --verbose still writes t.log lines to stderr and adds reply snippets to text results.

Model turns need a Cursor credential from agent-sdk login or CURSOR_API_KEY.

The exit code is 0 when every selected case passes, 1 when any case fails, and 2 when no case matches. --list exits 0, including when it finds no cases.

For a compact command index, see CLI: eval.

JSON results

Use --json --no-stream in scripts and CI. The top-level result carries the totals and one result per case:

{
  "ok": true,
  "passed": 1,
  "failed": 0,
  "results": [
    {
      "id": "readiness",
      "ok": true,
      "assertions": [{ "name": "succeeded", "passed": true }],
      "sessionId": "ses_123",
      "inputs": ["Is checkout pull request 42 ready to approve?"],
      "toolCalls": [{ "toolName": "inspect_pr", "isError": false }],
      "logs": [],
      "durationMs": 12340
    }
  ]
}

Each case result can also include description, finalText, tools, error, and tool arguments or output. This shape lets CI report the failed assertion without parsing terminal text.

Run evals in the playground

Start the server with --dev, open the playground, and choose Evals. You can run every case or one case, watch progress, and open the resulting session trace.

agent-sdk serve --dir . --dev

Playground runs target the live server instead of an ephemeral one. Their sessions appear in the session list. One eval batch can run at a time. Batches persist across restarts whenever agent/storage.ts provides an evals table or a KV core with delete and list (the table is derived over the core); without storage they are in-memory only (capped by maxPlaygroundRuns) — see Storage.

The UI uses the playground eval routes (available without --dev): GET /v1/dev/evals lists datapoints and config (includes maxPlaygroundRuns / durableRuns), GET /v1/dev/evals/runs rehydrates recent batches after navigation, POST /v1/dev/evals/runs starts a batch (returns an Eval ID / runId), GET /v1/dev/evals/runs/:runId polls it, and POST /v1/dev/evals/runs/:runId/cancel cancels a running batch. See Playground eval routes. The start request returns 202 while cases run in the background. Poll until the snapshot status becomes completed, failed, or cancelled. Configuration errors appear on a failed snapshot.

On --prod / --url, the CLI prints the Eval ID as soon as the batch is accepted (and a Playground deep link with ?view=evals&evalRunId=…):

agent-sdk eval --prod --slug vulnerability-scanner --tag deepsec
# Eval ID: evalrun_…
# Cancel:  agent-sdk eval cancel evalrun_… --prod --slug vulnerability-scanner
# Playground: https://…/playground?view=evals&evalRunId=evalrun_…

agent-sdk eval cancel evalrun_… --prod --slug vulnerability-scanner
agent-sdk eval status evalrun_… --prod --slug vulnerability-scanner

The Evals tab prefers the server’s in-flight batch (activeRunId) over a stale tab-local remembered id, so CLI / Slack kicks show up without an incognito window.

What good cases assert

Gate decisions and shape, not prose. Model wording varies run to run. Tool choice, tool avoidance, and output structure are the stable contract.

1. t.succeeded(): always, first.
2. The tool decision: calledTool for the intended path, notCalledTool for the likely wrong alternative. The pair is stronger than either alone.
3. Output shape: a regex for the contract (/ready|blocked/i, a JSON marker, a findings-block fence), never exact sentences.
4. For structured output, parse t.reply and check fields with satisfies instead of substring-matching JSON.

The common failure modes: asserting exact phrasing, packing more than about five gates into one case (split it), and cases that depend on live external state that drifts (pin the input; see fixtures).

Pick fixtures by agent type

The right fixture depends on the surface under test.

Agent surface	Fixture
Chat / domain assistant	A canonical prompt string, chosen once and frozen
Tool-heavy	Run agent-sdk call <tool> first to pin what the tool returns, then freeze the prompt that triggers it
GitHub webhook	agent-sdk github replay <pr> --events '*' --dry-run --out fixtures/github snapshots real payloads for offline replay (GitHub guide)
PR reviewer with host preparation	Diff, metadata, and gold labels pinned to commit SHAs; keep any live PR matrix small
Workspace-dependent	workspaceFiles in t.send options, never developer-machine paths

Tag the fast, reliably passing core smoke and run --tag smoke in the inner loop. Leave slow or flaky-prone cases untagged for explicit runs.

Materialize API-backed fixtures

An input that only points at external data, such as a pull request URL, snapshot id, or pair of commit SHAs, is not self-contained. Fetch it once and commit the rendered fixture before you expand the suite.

1. Save the diff, metadata, and labels under fixtures/ at pinned revisions.
2. Seed those files with workspaceFiles, or read them from the fixture directory.
3. Assert decisions and output shape against the saved evidence.
4. Keep a small smoke subset for any remaining live pipeline checks.

Read committed fixtures with @cursor/july/evals/loaders: loadJson, loadJsonl, and loadYaml resolve relative paths against the project root the runner discovered, not the cwd the CLI was invoked from (resolveFixturePath and evalFixtureRoot expose the same resolution for other file formats).

maxConcurrency limits parallel datapoints. It does not limit model or API fan-out inside one datapoint. Materialized fixtures prevent a large suite from exhausting provider and GitHub rate limits. The evals skill has the full fixture workflow.

Keep improvements with regression evals

Every hillclimb round that keeps a change must land an eval that would have failed before the change. If you can't express the improvement as a gate (a calledTool shift, a bounded action.result count, an output-shape regex), the improvement is unverified, and it'll regress silently.

The rule cuts the other way too: never weaken an existing gate to make a round pass. That's the freeze line moving, and it turns your regression suite into a list of checks that no longer protect anything.

Compare variants on live traffic

Use defineAB to compare variant metrics on live sessions. It is not a test runner and has no agent-sdk ab command. Keep defineEval as the regression ratchet. Eval sessions do not enroll or change live metrics. See Live A/B metrics for assignment, behavior, collection, and inspection.

What's next

Continue with these pages:

• Live A/B metrics: sticky variants and cumulative metrics on live sessions
• Hillclimbing: the loop evals make trustworthy
• Building agents with agents: have a coding agent write the first suite
• GitHub guide: deterministic webhook fixtures with github replay
• Sessions and streaming: the events t.events contains