Shopify's Spy Chatbot Centralizes Incident Response in Slack, Cutting Manual Coordination Overhead
Shopify built a ChatOps model around an Incident Manager on Call role and an internal bot called Spy, which automates channel creation, alert routing, and task reminders inside Slack — shrinking the feedback loop and eliminating fragmented communication during outages.
Background
Production engineers at Shopify are expected to handle incidents efficiently under pressure, yet incident handling is inherently messy and cognitively demanding. Before Spy, coordinators had to context-switch across multiple tools and rely on memory for best-practice steps, creating risk of inconsistency. Shopify modeled its incident command structure on the NIMS Incident Command System, assigning distinct roles to the IMOC (coordination), the Support Response Manager (public communication), and component experts (technical resolution).
What Was Implemented
- Internal chatbot Spy, built on open-source Lita (Ruby), integrated with Slack as the primary chat platform
- Automated `#war-room` Slack channel creation on incident start, binding all discussion to one place
- Webhook-based integrations with PagerDuty (alerting), StatusPage (public status updates), and GitHub (deploys and actions)
- In-channel commands for mitigation actions: traffic rebalancing, datacenter failover, deploy locking, job blackholing
- Automated war-room note collection and service disruption document generation at incident close
- Timed reminders for status-page updates and IMOC fatigue detection with automatic squad escalation
Results
Shopify's engineering team reports qualitative operational improvements: a shorter feedback loop, centralized communication, reduced post-incident toil, and more consistent adherence to incident-management best practices. Spy is described as enabling responders to "really effectively lead an incident response." No quantitative KPIs (MTTR reduction, incident count, etc.) are provided in the source. All outcome language is qualitative and self-reported.
Lessons
- Centralizing all incident communication in a single Slack channel prevents the parallel-discussion fragmentation that commonly slows response
- Automating the mechanical tasks of incident management (channel creation, page acknowledgment, reminder cadence, note capture) frees the IMOC to focus on decision-making rather than coordination overhead
- On-call fatigue is a measurable risk; building handoff triggers into the bot protects engineer well-being and sustains quality through long incidents
- Open-source chatbot frameworks (Lita) can be extended with incident-specific modules without requiring a dedicated vendor relationship
- A structured post-incident document generated automatically from tagged war-room messages improves postmortem quality and consistency