Canva Uses GPT-4 to Auto-Generate Post-Incident Review Summaries, Reducing Engineer Toil
Canva's reliability engineering team built a production pipeline that fetches incident reports from Confluence, strips sensitive data, and passes them to GPT-4 for blameless summaries — with most AI-generated drafts accepted by engineers without modification.
Background
Maintaining consistent, high-quality post-incident documentation is a known challenge in fast-growing engineering organizations. Canva's 150 million monthly active user base means even small incidents carry business-level significance. Inconsistent summaries make it harder to spot recurring issues, brief leadership, or hand off ownership between response and remediation teams.
What Was Implemented
- GPT-4-chat (gpt-4-32k) selected after comparing three model approaches (fine-tune, completion, chat)
- Confluence integration to fetch PIR documents as raw HTML, parsed to plain text
- Pre-processing pipeline to strip PII (links, emails, Slack channel names) before external model access
- Prompt engineering using TELeR-aligned structure; two PIR/summary example pairs per prompt for format guidance; temperature set to 0 for deterministic output
- Blameless framing enforced in system prompt; focus on detection method, impacted groups, affected service, duration, root cause, trigger, and mitigation
- Archive of summaries in data warehouse with Jira webhook tracking for AI vs. human comparison
Results
After approximately two months in production, Canva's engineering team observed that the majority of AI-generated PIR summaries required no human modification , indicating high quality and consistency. The system reduced the operational toil of reliability engineers. The approximate per-summary cost was up to $0.60 using GPT-4-32k. No quantitative before/after comparison of time-to-summary or incident-recurrence rate was published in the source.
Lessons
- Stripping PII before sending incident data to external LLMs is non-negotiable for blameless, compliant summaries in enterprise environments
- Setting temperature to 0 and constraining output format via system prompts minimizes hallucination in incident summarization tasks
- Providing two PIR/summary example pairs per prompt balances format guidance with response variety — better than a single example, which causes rigid mimicry
- Comparing AI-generated vs. human-revised summaries via webhook creates a feedback loop for ongoing quality monitoring
- Fine-tuning with ~1,500 examples was insufficient for Canva's needs; few-shot prompting with GPT-4 outperformed the fine-tuned smaller model