Agent SDK

Appearance

Hillclimbing ​

Make an agent better on fixed inputs: measure, change one lever, remeasure, and lock every kept win with an eval.

What is hillclimbing? ​

Hillclimbing is a measured improvement loop. You pin a few fixtures, name the one dominant problem in the run, change one lever, and check the same fixtures again. Keep only what helps. Every kept change lands an eval so the win stays put.

You don't have to run the loop alone. The package ships a coding-agent skill that drives it with you.

mermaid
flowchart LR
  measure[Measure] --> change[Change one lever]
  change --> remeasure[Remeasure]
  remeasure --> measure

How do I hillclimb an agent with a coding agent? ​

Have Cursor read skills/hillclimb/SKILL.md.

Tell it:

  1. Which agent you're improving (path or slug)
  2. One to three fixtures you'll reuse every round: a PR URL, a saved webhook body, or a canonical chat prompt
  3. What "better" means this round: correct tool choice, fewer tools, lower latency, or output quality. Name the freeze line too: API shape, public output, and existing evals that must stay green

The skill serves the agent, hits your fixtures, reads the session trajectory, proposes one change, remeasures, and checks with you before the next round.

Other skills cover the edges:

When you need…Skill
The measured improvement loopskills/hillclimb/SKILL.md
An eval that locks a kept winskills/evals/SKILL.md
Repeatable GitHub webhook inputsskills/github/SKILL.md
A run that misbehavesskills/debug/SKILL.md

See Building agents with agents for every framework skill and a good first prompt.

What do I need before a hillclimb round? ​

Agree on four things before you edit:

  1. The target agent: the project you're improving
  2. Fixtures: one to three fixed inputs you can compare across runs
  3. Success criteria: what better means this round
  4. The freeze line: what must not change

Pin the input first. A moving fixture is noise. For GitHub agents, use agent-sdk github replay (see the GitHub guide). For a single tool without a model turn, use agent-sdk call. For a chat turn, use agent-sdk run --dir . --message "…".

How do I run one hillclimb round? ​

Measure. Hit the agent the way a user would: playground, channel HTTP, or Slack in --dev. Or ask the hillclimb skill to do it. agent-sdk run returns a JSON trajectory and writes a trace under .agent-serve/traces/.

Reflect. Score the trajectory, not impressions. Was the answer right? Did the model thrash (too many tools, fat evidence, grep loops)? Did it invent work the host should have prepared? Name the single dominant problem for this round in one sentence. Example: "Full-file dumps trigger grep loops."

Change one lever. Prefer the smallest change that addresses that problem:

  1. Host prep: seed what the model needs so it doesn't hunt
  2. Evidence shape: trim or reorder artifacts
  3. Instructions and skills: tighten the procedure
  4. Tool surface: remove or gate tools that invite wandering
  5. Framework changes: only when the agent can't express the fix

Remeasure. Same fixtures. Diff tools, wall time, and quality side by side. Keep the change only if the target metric improves and the freeze line holds.

How do I lock a hillclimb improvement with an eval? ​

Every kept change needs an eval that would have failed before the change: a tool-choice gate, an action.result count bound, or an output-shape check. Run agent-sdk eval --dir . --json between rounds. Never weaken an existing gate to pass the round.

Details live in Evals. The evals skill will author the case with you.

What habits help hillclimbing stay reliable? ​

  • One problem per round. Don't bundle "trim evidence and rewrite instructions" unless you chose that on purpose.
  • Keep fixtures fixed until you deliberately need a harder case.
  • Separate host work from model tools when you blame latency. Moving deterministic prep onto the host is often the biggest win. In one PR reviewer, host-prepared evidence cut turns from about 8 minutes to about 1 minute.
  • Spot-check quality on at least one fixture against a known-good answer. Efficiency-only climbs quietly drop findings.
  • Treat turn.failed with "turn interrupted" as expected when a follow-up or stop preempted the turn.
  • Don't deploy, post to real surfaces, or weaken evals as part of a climb.