Run staged security reviews from GitHub events

Security Reviewer turns a pull request into a staged host-side review. One tool prepares the diff and selects modules. A second fans out specialized reviewers and triages candidates as they arrive. A third deduplicates the confirmed findings, writes artifacts, and may publish a GitHub review.

Use this example when the workflow needs several model workers, but the host must own orchestration, progress, artifacts, and the final write.

Source lives under factory/security-reviewer/ (factory agent, not under examples/).

Want one model turn and one comment? Scaffold the security-reviewer template.

Browse the Security Reviewer source.

Run a three-stage host pipeline

Security Reviewer is a pipeline, not one long agent turn:

Stage	Tool	Result
Prepare	prepare_review	Fetch metadata and diff, create a runId, and select security modules.
Review and triage	run_reviewers	Run module reviewers in parallel and start triage as each candidate arrives.
Finalize	finalize_review	Apply thresholds, deduplicate findings, write artifacts, and optionally post a review.

run_triage remains available as a compatibility stage. In the normal flow, triage has already completed inside run_reviewers, so it reports existing results. If candidates exist without triage output, it starts triage workers and writes their state.

The configured root agent chooses and sequences tools in chat. The review workers use a model selected by the host pipeline. They are created programmatically with the agent SDK, not discovered from agent/subagents/.

Follow a GitHub wake

1. A non-draft pull_request.opened or pull_request.synchronize event arrives for an allowlisted repository.
2. The GitHub channel returns a host { task }, so the webhook gets a 202 response before the long review starts.
3. Host code tries to post a pending commit status and creates a playground session.
4. The root model sends one acknowledgement. It doesn't run review tools on this path.
5. The task calls prepare_review, run_reviewers, and finalize_review deterministically inside that session.
6. Reviewer candidates stream into duplicate gating and triage.
7. Finalization writes artifacts and tries to post the GitHub review.
8. The host tries to set a success commit status when no findings remain, a failure status when findings remain, or an error status when the pipeline throws.
9. The final response is appended to the session as an assistant message.

The session records each stage as a normal tool event, even though host code selected the tools.

Review and status posting are best-effort. The channel chooses its final status from the finding count even when the review posting result says posted: false.

Map the framework features

Capability	Source	Role
Root agent	agent/agent.ts, agent/instructions.md	Configure local chat and explain the three-stage contract.
Server tools	agent/tools/	Expose each review stage to chat and host orchestration.
GitHub channel	agent/channels/github.ts	Filter wakes, run background tasks, and publish status.
Progress channel	agent/channels/asr-progress.ts	Serve live reviewer and triage state by runId.
Playground renderer	agent/playground/tools/run_reviewers.tsx	Replace the generic tool chip with live module rows.
SDK review pipeline	review-stages.ts, @anysphere/security-review-lib	Select modules, call model workers, triage, deduplicate, and write artifacts.
Storage	agent/storage.ts	Persist framework sessions with cursorHostedStorage (lazy restore).
A/B	agent/ab.ts	Compare all-severity versus high-only GitHub comments.
Eval	evals/	Check stage-tool presence against a pinned sample.

There is no Slack channel, authored skill, discovered subagent, MCP connection, schedule, reminder, hook, tool approval, or cloud runtime.

Prepare the host

You need:

• Node 22.13 or newer.
• An agent-runtime credential for the root turn and review workers.
• GitHub read access for preparation.
• GitHub write access for webhook-driven reviews and commit statuses.

The pipeline exposes settings for:

• the worker model,
• reviewer and triage parallelism,
• reviewer, triage, duplicate-gate, and final-dedupe timeouts, and
• prior-comment loading.

The active names live beside the orchestration in review-stages.ts.

Validate the discovered agent

agent-sdk validate --dir ../../factory/security-reviewer
agent-sdk info --dir ../../factory/security-reviewer --json
agent-sdk eval --dir ../../factory/security-reviewer --list

The manifest should show four server tools, two authored channels, one storage definition, and one A/B experiment. The eval listing should show one case.

Know the chat path's write boundary

In chat, the root instructions ask the model to use this order:

prepare_review -> run_reviewers -> finalize_review

They also ask the model to set postComment: true only on request. This is prompt policy, not a deterministic safety gate. The model chooses tool arguments, and finalize_review has no human approval. Use the direct stage calls below when a no-post proof must be enforced.

Call stages directly without publishing

Call each stage and pass postComment: false yourself:

agent-sdk call prepare_review \
  --dir ../../factory/security-reviewer \
  --input '{"prUrl":"https://github.com/owner/repo/pull/123"}'

agent-sdk call run_reviewers \
  --dir ../../factory/security-reviewer \
  --input '{"runId":"<run-id>"}'

agent-sdk call finalize_review \
  --dir ../../factory/security-reviewer \
  --input '{"runId":"<run-id>","postComment":false}'

Review state lives under the project's run-artifact directory, so later stages can open the prepared runId.

CAUTION

finalize_review with postComment: true writes to GitHub. The webhook path always requests that write. Chat instructions alone don't prevent it.

Watch parallel work in the playground

Run the dev server:

agent-sdk dev ../../factory/security-reviewer

Open the printed playground and start a review. The custom run_reviewers renderer polls the progress channel's GET /:runId route.

It refreshes every 500 ms while the stage runs. Each row shows a reviewer module's state, candidates, reviewed areas, and failure. A second section shows triage jobs and confirmed or rejected counts.

This is an authored playground extension. The Agent SDK discovers it by the tool name, so the generic run_reviewers chip becomes a domain-specific view without changing the framework playground.

Fan out reviewers while triage starts

Module selection uses repository and path rules. The current module set covers:

• agent tooling trust boundaries,
• privileged service RPCs,
• product-specific security risks,
• dependency and supply-chain changes,
• deployment and infrastructure code,
• filesystem and workspace boundaries,
• privacy, and
• general security review.

Selected modules may run more than once. Candidates pass through a duplicate gate, then bounded triage. Reviewer or triage failures can produce partial results. A final dedupe failure stops finalization.

The pipeline writes JSONL journals as work completes. Final artifacts include the review bundle, patch, reviewer outputs, candidates, triage decisions, findings, accounting, and audit events.

Separate session storage from review artifacts

defineStorage + cursorHostedStorage sends Agent SDK session and event records to Cursor-hosted Bugbot storage through the control-plane proxy. Security Reviewer sets restore: "off" so startup doesn't load old review sessions in bulk. A continuation lookup can still fetch a needed session.

The staged review files are separate from session storage. Session-store durability doesn't preserve those files. All stages for one runId must see the same filesystem.

This split is useful when conversation history needs shared durability but large review artifacts belong on attached storage or an object store.

Compare live comment variants

The comment-severity experiment uses sticky session assignment with a 5% holdout:

• control posts every finding.
• treatment posts only high and critical findings.

Finalization enforces the comment filter. The treatment also adds an instruction overlay asking chat and playground summaries to lead with high and critical findings. Full artifacts, finalResponse, finding counts, and status still include every finding. Stage-tool counters appear in the playground A/B view. Local sample and snapshot files persist under .agent-serve/.

When a treatment session has only low or medium findings, the filtered review body currently says no vulnerabilities were found even though artifacts and status retain findings. Account for that mismatch before using this experiment as a publishing policy.

Eval sessions skip A/B enrollment.

Test the GitHub channel carefully

The channel currently accepts two configured repositories. It wakes on opened and synchronize, skips drafts, and requests pr-write access.

Inspect its event surface:

agent-sdk github events \
  --dir ../../factory/security-reviewer \
  --json

Replay reaches the full publishing path:

TEST_PR_URL=https://github.com/your-org/allowlisted-test-repo/pull/123
agent-sdk github replay \
  "$TEST_PR_URL" \
  --dir ../../factory/security-reviewer \
  --action opened

Set TEST_PR_URL to a PR in the channel's configured repository allowlist. Run the command only against a PR intended for test reviews. It posts a commit status and may post findings.

Inspect the eval before running it

agent-sdk eval --dir ../../factory/security-reviewer --list

The case reads pinned metadata from the committed fixture and checks for all three tool names. It doesn't assert their order. The current prepare_review still fetches the live PR, so the case needs GitHub access and isn't fully offline. It also doesn't assert finding location, severity, agreement with gold.json, or postComment: false.

Don't use this committed case as a no-post proof with write-capable GitHub credentials. The prompt asks for no comment, but the model can still pass postComment: true.

When you adapt the pipeline, add assertions for confirmed findings and make the prepare stage accept a materialized fixture if repeatable offline evals matter.

Build another staged pipeline

Use staged host orchestration when:

• each phase needs its own timeout and artifact,
• model workers should run in bounded parallel,
• later work can start as soon as partial results arrive,
• a webhook must acknowledge before the work finishes, or
• operators need live progress beyond one tool spinner.

Keep external writes in finalization. Pass a runId between stages, journal progress before publishing, and make partial-worker failures visible in the result.

Where to go next

• GitHub
• Tools
• Channels
• Playground
• Storage
• Live A/B metrics
• Evals