Investigate every alert in its own Slack thread

This agent is an on-call teammate. Alert feeds post into an alerts channel as bots. Each new alert dispatches an investigation session pinned to that post's thread: the agent reacts 👀 the moment it locks in, investigates immediately, and posts brief findings backed by evidence it observed. Replies in the thread reach it only after the thread has been quiet for about a minute, and reminder tools let it wake itself later to re-check a baseline or confirm an alert cleared.

Use this example when alerts land in Slack and you want one thread-scoped investigation per alert, with an agent that paces its own engagement instead of answering every message.

Browse the current alert-investigator source.

Follow an alert

1. An alert feed (Alertmanager, PagerDuty, Datadog) posts a new top-level message in the watched alerts channel.
2. The channel watch accepts it. includeBotPosts lets bot authors through; the agent's own posts always stay dropped.
3. The handler reacts 👀 on the alert post and sets "Investigating…" typing. The reaction is the lock-in signal: this alert has an owner.
4. The Agent SDK creates a session keyed to the alert's thread and dispatches immediately. New alerts get no debounce.
5. The agent reads the alert, gathers evidence, and posts findings to the thread once it has a hypothesis.
6. People discuss in the thread. Replies buffer per thread and dispatch as one coalesced follow-up after roughly a minute of quiet.
7. The agent arms reminders for anything that needs time and posts interim updates when new evidence changes the picture.

Mentions and DMs skip the watch entirely and behave like ordinary chat.

Map the files

File	Purpose
agent/agent.ts	Names the agent and keeps harness workspaces outside any monorepo checkout.
agent/instructions.md	Engagement rules, the investigation loop, and the message discipline.
agent/channels/slack-app.ts	Dedicated Socket Mode app: watch configuration and handler wiring.
agent/lib/alert-watch.ts	The engagement policy: lock in on new alerts, coalesce replies.
agent/lib/thread-debounce.ts	Per-thread quiet window.
agent/lib/alerts.ts	Dispatch classification, prompt building, and thread addressing.
agent/lib/slack-api.ts	Reactions and thread posts on this agent's own token pair.
agent/tools/reminders_create.ts	Self-scheduled wakes bound to the thread (plus reminders_list and reminders_cancel).
agent/tools/post_thread_update.ts	Interim updates to the thread mid-turn.
agent/storage.ts	Persists sessions and events with cursorHostedStorage.
evals/evals.config.ts	Caps eval run concurrency.
evals/smoke.eval.ts	Checks identity and the reminder-tool route.

Let bot posts through the watch

Channel watching drops bot-authored posts by default so two agents can never feed each other. Alert channels invert the assumption: the posts worth watching come from bots. channelPosts.includeBotPosts opts in per channel:

engagement: {
  channelPosts: {
    allow: ["#alerts"],
    posts: "all",
    includeBotPosts: true,
  },
},

Loop safety survives the opt-in. The pack matches the watching app's own posts by the bot_id and bot user id from auth.test and drops them, so the agent's findings never re-dispatch it. Posts that mention the bot stay on the mention path.

posts: "all" also delivers thread replies. The handler, not the pack, decides their pace.

Pace the engagement

The example runs two rhythms:

• A new alert dispatches immediately.
• Thread replies produce one engagement per lull.

The pack's debounceMs is per message; it exists to let edits settle. This agent needs a per-thread window instead, so the handler owns it (lib/thread-debounce.ts). Every reply restarts a 60-second timer keyed by thread. Superseded waiters resolve null and the handler returns null for them. When the thread goes quiet, the newest waiter receives the whole batch and dispatches one follow-up that lists every message with mentionable attribution.

Two details make the window matter. A follow-up that arrives while a turn runs preempts that turn (latest message wins), so engaging per message would keep cancelling the investigation. And @mentions bypass the window through Slack's mention path, so a person who needs the agent now still gets it now.

Schedule your own re-checks

Investigations rarely finish in one pass. A baseline comparison needs 20 minutes of data. An alert that cleared may re-fire. The example hands the model three tools over host.reminders:

• reminders_create arms a one-shot (delay: "20m") or recurring (every: "30m" with a plain-language stop condition) wake bound to the thread's conversation.
• reminders_list shows the thread's standing watches.
• reminders_cancel disarms one, and refuses ids that belong to another thread's conversation.

When a reminder fires, its prompt returns to the same session as a follow-up turn, and the reply lands in the alert thread. The instructions keep wake prompts generic (re-read live state instead of replaying stale numbers) and wake replies to one line, for example "re-checked p99 on api-gateway: 120ms, back at baseline, cancelling the watch."

Keep these tool filenames if you copy the design: the framework's reminder fire prompt tells the model to call reminders_cancel by name when a stop condition is set.

Alert people mid-investigation

The final reply of each turn posts to the thread on its own. post_thread_update covers evidence that shouldn't wait for the turn to finish: it posts a one-or-two-sentence update through the agent's token, with <@USERID> mentions for the people who need to act. The instructions restrict it to changes in hypothesis, severity, or blast radius. Progress narration doesn't qualify.

Connect the Slack app

Channel watching is Socket Mode only, so this example uses a dedicated app:

agent-sdk slack create --dir examples/oncall --name "Oncall" --channel-posts
agent-sdk slack doctor --prefix ONCALL

--channel-posts prefills channel-watch events (message.channels / message.groups). Invite the bot to each watched channel after the wizard finishes.

ONCALL_ALERTS_CHANNELS sets the watch list as comma-separated ids or #names. It defaults to #alerts.

Wire observability MCP servers under agent/mcp-connections/ so evidence gathering reaches your logs, metrics, and dashboards. The example ships none; without them the agent works from the alert text, its links, and the thread.

Validate and start the server

agent-sdk validate --dir examples/oncall
agent-sdk info --dir examples/oncall --json
agent-sdk dev examples/oncall

The info output lists four server tools and the watched channel on the slack-app channel. Missing tokens leave that channel idle without stopping the server.

In dev mode, reminder timers don't auto-fire. List and fire them by hand through the dev routes described in Schedules and reminders.

Test the policy without Slack

The engagement policy is plain code with unit tests:

pnpm exec vitest run examples/oncall

The integration test drives a synthetic Events API delivery through the real parse, watch, and dispatch plumbing. It asserts a bot alert dispatches pinned to its thread after the lock-in reaction, the agent's own posts never loop, and replies coalesce behind the quiet window.

The smoke eval spends a model turn:

agent-sdk eval --dir examples/oncall smoke --json

It checks identity and the reminder-tool route lexically. It doesn't prove Slack delivery or reaction behavior; the unit tests cover the dispatch side, and a live check needs the dedicated app connected.

Build an alert investigator

Use this structure when a bot feed should drive thread-scoped work:

1. Watch the feed channel with includeBotPosts: true and a narrow allowlist.
2. Acknowledge on the triggering post before dispatching, so people see ownership without opening the thread.
3. Dispatch new items immediately; coalesce thread chatter behind a per-thread quiet window.
4. Give the agent reminder tools for anything that needs time, and make cancel discipline part of the instructions.
5. Keep every posted message brief and tied to evidence the agent saw.

Where to go next

• Slack
• Schedules and reminders
• Tools
• Playbook router for the human-post variant of channel watching