Datadog's Bits AI Drafts Postmortems from Incident Metadata and Slack Threads, Cutting Engineer Documentation Load
Datadog built an LLM-powered postmortem assistant inside its Bits AI product that combines structured incident metadata with unstructured Slack discussion, producing draft reports that give engineers a structured starting point rather than a blank page.
Background
Postmortem documentation is cognitively demanding work that occurs immediately after an incident — when engineers are fatigued and under pressure to return to normal operations. Blank-page syndrome slows postmortem completion and leads to inconsistent quality. Datadog recognized that LLMs could provide a structured starting draft, capturing the incident's mechanical facts (timeline, affected services, severity) before requiring human synthesis of root cause and lessons learned.
What Was Implemented
- LLM ensemble model applied to postmortem drafting within the Bits AI product
- Input combination: structured Datadog Incident Management metadata + unstructured Slack channel discussion
- Model selection: GPT-3.5 for lower-complexity sections; GPT-4 for higher-complexity analytical sections (cost/quality trade-off)
- Sensitive data stripping before external model submission
- AI-generated content explicitly labeled in the draft to prevent blind acceptance
- 100+ hours of prompt engineering and instruction tuning across diverse incident types
Results
Datadog's internal team reports that the system provides engineers a structured starting point for postmortems, reducing the blank-page problem and improving postmortem completion speed. The book's claim that the system "speeds the handoff between incident responders and follow-up owners" is confirmed by the source's description of automated field pre-population. No quantitative metric (e.g., time-to-postmortem, completion rate, or review cycle time) is published in the primary source. The team acknowledges LLMs "can't fully replace humans, at least at present" but states that GenAI-enhanced products "can greatly improve productivity."
Lessons
- Using different model tiers (GPT-3.5 vs. GPT-4) for different sections of a postmortem is an effective strategy for balancing cost, speed, and accuracy in production
- 100+ hours of prompt tuning is a realistic investment for incident-domain-specific postmortem generation; teams should plan for this rather than assuming out-of-the-box quality
- Clearly labeling AI-generated content is essential for trust and safety; engineers must know what has been automated
- Slack conversation data paired with structured incident metadata produces richer drafts than either input alone