Zalando Transforms Thousands of Postmortems into Strategic Infrastructure Intelligence Using a Multi-Stage LLM Pipeline
Zalando's SRE team built a multi-stage AI pipeline — now powered by Claude Sonnet 4 on AWS Bedrock — that analyzes its entire postmortem corpus to surface recurring datastore failure patterns, cutting analysis time from days to hours while requiring ongoing human curation to address a persistent ~10% attribution error rate.
Background
Zalando inherited Google's postmortem culture from the SRE book, requiring engineering leadership sign-off on every incident close. Over years, this produced a rich corpus of thousands of postmortems — but the institutional knowledge locked inside was inaccessible at scale. Connecting incidents across teams required "immense cognitive load and informal networking." Without AI, strategic questions about recurring failure modes could not be answered quickly.
What Was Implemented
- Multi-stage LLM pipeline: Summarization → Classification → Analyzer → Patterns → Opportunity
- Early prototype: Google NotebookLM (3x productivity gain but high hallucination rate)
- Production pipeline evolution: open-source 3B–27B models → Claude Sonnet 4 on AWS Bedrock
- Focus on five datastore technologies: Postgres, AWS DynamoDB, AWS ElastiCache, AWS S3, Elasticsearch
- Strict prompt constraints: no speculation, no inference, only explicitly stated causal connections
- Negative prompting to combat Surface Attribution Error (residual ~10% rate acknowledged)
- Human curation: 100% in early phases, relaxed to 10–20% random sampling as system matured
- Output: one-page failure pattern report per technology area, reviewed and edited by humans before distribution
Results
Over two years, the pipeline reduced analysis time from days to under 24 hours for a full year of postmortem data per technology area. Early prototype productivity improvement was ~3x (NotebookLM phase). The S3 insight led directly to automated change validation that prevented 25% of subsequent S3 datastore incidents. For ElastiCache, capacity planning changes addressed a persistent 80% CPU utilization pattern at peak traffic. The residual Surface Attribution Error rate is ~10% even with Claude Sonnet 4 — acknowledged as an ongoing limitation requiring human editorial oversight.
Lessons
- A "map-fold" pipeline architecture (independent document processing → aggregation) outperforms single large-context-window approaches for large postmortem corpora
- Surface Attribution Error — the model blaming technologies mentioned in text rather than causally responsible ones — persists even in frontier models and requires negative prompting plus human curation
- Compliance alignment (PII in postmortem documents) must precede any cloud-hosted LLM deployment for this use case
- Human curation is not optional: 10–20% random sampling per batch maintains trust and catches novel failure modes
- AI postmortem analysis is most valuable as a strategic input for infrastructure investment decisions, not just a summarization tool