Live A/B metrics

Use defineAB to compare variants on live agent sessions. New sessions receive a sticky assignment in each enrolled experiment. The Agent SDK folds their durable event streams into tool, token, failure, and wall-time metrics. You can send cumulative samples to your metrics backend and inspect aggregates in the playground.

defineAB compares live variants through sticky assignment, instruction overlays, optional tool branches, and cumulative metrics. Metric callbacks observe the result without approving, rejecting, or failing a turn. Use evals for pass/fail regression checks on fixed inputs.

NOTE

Import paths here use @cursor/july/ab. On projects still using @anysphere/agent-serve, swap the import. See Run the CLI for the full rename table.

Choose live A/B metrics or evals

Both features read the session event stream, but they answer different questions.

	Live A/B metrics	Evals
Question	How do variants compare on live sessions?	Does the agent still meet a fixed contract?
Location	agent/ab.ts or agent/ab/<name>.ts	evals/**/*.eval.ts
Input	Dev or production traffic	Frozen prompts and fixtures
Output	Cumulative metrics by session and arm	Pass/fail assertions
How it runs	Automatically on new live sessions	agent-sdk eval

There is no agent-sdk ab command or assertion API.

Define an experiment

Author one experiment in agent/ab.ts, add more under agent/ab/<name>.ts, or use both forms. Each file defines one experiment. The experiment name comes from name when set. Otherwise, The Agent SDK uses ab for agent/ab.ts and the file stem for files under agent/ab/.

// agent/ab/concise-weather.ts
import {
  defineAB,
  splitBySessionHash,
} from "@cursor/july/ab";

export default defineAB({
  name: "concise-weather",
  variants: {
    control: {
      label: "Baseline",
    },
    treatment: {
      label: "Short replies",
      description: "Adds a one-paragraph response limit.",
      instructions: "Keep weather replies to one short paragraph.",
    },
  },
  split: splitBySessionHash({
    weights: { control: 1, treatment: 1 },
    holdout: 0.1,
  }),
  derive: {
    weatherCalls: (event) =>
      event.type === "action.result" &&
      event.data.toolName === "get_weather"
        ? 1
        : null,
  },
  onSample(sample) {
    console.log(
      sample.experiment,
      sample.variant,
      sample.metrics.toolCalls,
      sample.metrics.wallTimeMs
    );
  },
});

Every definition needs:

• At least two variants. Variant keys cannot be empty or contain / or \.
• A split function that returns a variant key or null.
• An onSample callback for completed or failed turns.

label and description appear with the arm in result surfaces. instructions changes the prompt for sessions in that arm. derive adds custom counters.

Duplicate experiment names are validation errors. Check discovery before you serve:

agent-sdk validate --dir .
agent-sdk info --dir . --json

The abs field in info lists the discovered experiment names.

Assign sticky variants

Enrollment happens once, when a live session is created and before its first turn:

1. The Agent SDK records session.started.
2. Each experiment runs its split function.
3. The Agent SDK records one durable ab.assigned event per experiment.
4. The selected arms become available on session.abs.
5. Variant instruction overlays reach the first model turn.

A split can return a variant key or null. A null assignment is a sticky skip for that experiment. It increments the experiment's skipped total, still appears in the snapshot's sessions list with variant: null, and does not collect arm metrics or call onSample.

Use the split helper that matches your rollout:

Helper	Behavior
splitBySessionHash({ weights?, holdout?, salt? })	Hashes the session id into a reproducible arm; the recommended default
splitByRandom({ weights?, holdout? })	Draws once when the session starts, then persists the result
splitAlways("control")	Pins every new session to one arm
splitNone()	Skips every new session without deleting the experiment
splitIf(predicate, inner)	Runs inner only when the predicate passes
Custom split(ctx)	Returns a declared variant key or null

The split context includes the agent name, channel id, session info, experiment name, and declared variant keys. For example, enroll only Slack sessions:

split: splitIf(
  (ctx) => ctx.channel.id === "slack",
  splitBySessionHash()
),

Weights default to equal. Non-positive weights leave an arm out of the draw, and at least one arm must have a positive weight. holdout is the fraction of sessions assigned null, from 0 through 1. Change salt to reshuffle future hash assignments without renaming the experiment.

If a custom split throws or returns an unknown variant, the Agent SDK logs the error and records variant: null. The failed decision becomes a sticky skip instead of breaking the session.

Enrollment only applies to new sessions. Adding an experiment does not assign existing conversations. Follow-ups keep the session's original arms. Keep experiment names and variant keys stable while you collect and compare results.

Change behavior by variant

Variant instructions are appended to the agent's base instructions. Local sessions receive the merged instructions in AGENTS.md before every turn. Cloud sessions receive them in the first-turn preamble only. For cloud follow-ups, branch through session.abs when the arm must remain visible to deterministic behavior.

Tools can branch on the assignment through ctx.session.abs. Hooks can read the same field for logging or export:

const treatment =
  ctx.session.abs?.["concise-weather"] === "treatment";

if (treatment) {
  return conciseWeatherResult;
}

return baselineWeatherResult;

This makes the assignment available to deterministic code as well as the model prompt. Use both patterns together when one experiment must steer the prompt and host code at once.

defineAB does not select a different model or runtime for each arm. Keep those settings in agent/agent.ts, or write explicit host logic when your experiment needs another behavior lever.

The split and selected arm can affect agent behavior. derive and onSample only observe the resulting event stream. Errors in either callback are logged and never fail the turn.

Collect built-in and custom metrics

Metrics accumulate for each session and experiment. When one session joins several experiments, every enrolled experiment folds the same turn and tool events into its own counters.

Metric	How the Agent SDK calculates it
turns	Adds one on turn.completed or turn.failed
turnFailures	Adds one on turn.failed
toolCalls	Adds one for each action.result
toolErrors	Adds one when action.result.data.isError is true
inputTokens, outputTokens	Adds usage from completed turns
cacheReadTokens, cacheWriteTokens	Adds cache usage from completed turns
costUsd	Sums the estimated turn cost recorded on turn.completed (turns whose model has no known rates contribute 0)
wallTimeMs	Sums the time from turn.started to its completed or failed event
custom	Sums finite numeric deltas returned by derive

onSample fires after every turn.completed and turn.failed event for an enrolled arm. The sample contains:

Field	Value
experiment	Experiment name
variant, variantLabel?	Sticky arm and optional display label
sessionId, channelId	Source session
metrics	Cumulative metrics through this turn
reason	turn.completed or turn.failed
at	Terminal event timestamp

The metrics are cumulative, not per-turn deltas. A second sample from the same session includes the first turn's counts.

Each derive extractor runs on every session event for its enrolled experiment, including streamed message.appended events. Keep it synchronous and cheap. Return a finite number to add a delta, or null to skip the event. Send samples to your metrics service from onSample; do not perform network or disk work in derive.

Skipped sessions never call onSample. Errors from derive or onSample are logged, then metric collection continues.

Inspect assignments and results

Open the playground's A/Bs tab to see aggregate arm totals and per-session assignments. The tab reads GET /v1/abs.

The response has two views of the same durable data:

Field	Contents
experiments	Declared variants, skipped-session count, arm session counts, and aggregate metrics
sessions	Visible sessions with their assignments and cumulative metrics

GET /v1/abs returns sessions visible to the current principal by default. In --dev, loopback requests include every session. Add --allow-anonymous to include every session from non-loopback callers too. This include-all behavior can still apply to GET /v1/abs in dev when bearer or custom auth keeps GET /v1/sessions owner-scoped.

Session events.ndjson is the source of truth for assignment + fold. GET /v1/abs recomputes aggregates from those logs. Any agent/storage.ts exports samples and snapshots durably: an authored abs table when the backend has a native shape for it, or the table derived over the KV core otherwise. See Storage.

Configure the playground fold window

Assignments and foldable metrics already persist in each session's events.ndjson under --state-root. The optional agent/ab.config.ts only caps how many sessions the playground and GET /v1/abs fold:

import { defineABConfig } from "@cursor/july/ab";

export default defineABConfig({
  // Optional — defaults to 200. Only affects GET /v1/abs / A/Bs tab.
  maxPlaygroundSessions: 500,
});

maxPlaygroundSessions keeps the newest sessions in the fold. It does not prune session logs or change assignment. For export to S3, a DB, or your metrics vendor, send samples from onSample or declare a storage abs table.

Keep assignments durable

The append-only events.ndjson stream is the source of truth. Each ab.assigned event persists a variant key or null skip. Built-in metrics come from the turn and tool events that follow it.

After a server restart or a parked session resumes, the live collector replays the stream to rebuild cumulative counters. Replay does not call onSample (or write to the storage abs table) for historical turns. Only a new completed or failed turn emits another sample.

The snapshot API also replays derive across the full stream, so custom totals match the current extractor. Changing a derive function can change historical snapshot totals. Treat metric definitions as versioned experiment code.

Keep eval traffic separate

Sessions created by agent-sdk eval and the playground Evals runner use purpose: "eval". They skip A/B enrollment entirely:

• No split function runs.
• No ab.assigned event is recorded.
• No onSample callback fires.
• The session is omitted from GET /v1/abs.

Ordinary chat, agent-sdk run, Slack, GitHub, and other channel sessions use the live purpose. You do not need splitIf to exclude eval traffic.

Know the boundaries

defineAB provides sticky assignment, variant instructions, session.abs for tools, cumulative metrics, and local inspection. It does not provide:

• A test command, assertion API, or pass/fail result
• Statistical significance calculations
• An experiment rollout or lifecycle service
• Per-variant model or runtime configuration
• A built-in analytics warehouse (bring your own via onSample or the storage abs table)

Use evals to protect known behavior. Use onSample or a storage abs table when you need sample/snapshot exports beyond the session event log.

What's next

Continue with these pages:

• Evals: pass/fail regression checks on fixed inputs
• Hillclimbing: improve an agent against fixed fixtures
• Hooks: other event-stream consumers
• Sessions and streaming: the ab.assigned event and durable log
• Playground: the A/Bs tab
• HTTP API: GET /v1/abs
• Live A/B metrics skill: have a coding agent wire an experiment