Slack's Project Cornflake cuts flaky test job failures from 56.8% to 3.9% and saves 553 hours of triage time
Slack's Mobile Developer Experience team built an automated flaky test detection and suppression system — Project Cornflake — that reduced test job failure rates from 56.76% to 3.85% in under a year, saving 553 hours of manual triage time across Android and iOS mobile codebases with 27,000+ automated tests.
Background
Slack's mobile codebases scaled faster than the manual processes used to manage test quality. With over 27,000 automated tests and hundreds of PRs per week, flaky tests were causing the majority of CI build failures and requiring costly manual investigation. Manual triaging took ~28 minutes per PR, and the volume made it impossible for individual developers to investigate failures proactively.
What Was Implemented
- Automated flaky test detection system (Project Cornflake V2) operating on main branch test failures
- Detection logic: parses test results, identifies flaky vs. failing tests, excludes infrastructure/API/crash failures
- Suppression: automatically disables flaky test in source code, creates Jira ticket, opens and auto-merges a PR
- Test ownership mapping to route Jira tickets to the responsible feature team
- Weekly Slack channel notifications summarizing suppressed tests per team
- Quarantine monitoring: suppressed tests re-run periodically; automatically re-enabled if no longer flaky
- Covers both Android and iOS mobile platforms; supports E2E, Functional, and Unit test types
Results
Main branch stability: 19.82% → 96% (July 2020 to February 2021). Test job failure rate: 56.76% → 3.85% . 553 hours of triage time saved (693 Android PRs + 492 iOS PRs at ~28 minutes each). 74% of surveyed developers reported a positive impact on main branch stability. 64% reported reduced PR reruns. The system has required "little to no maintenance cost" in the year following V2 rollout.
Lessons
- Suppressing test *execution* (disabling the test in source) is more reliable than suppressing test *results* (filtering outputs): the latter allows genuinely broken tests to leak into the main branch undetected
- Automatic PR creation, approval, and merge for test suppression is essential at scale — manual PR processes create bottlenecks that defeat the purpose of automation
- Test ownership mapping is a prerequisite: without it, Jira tickets cannot be routed to the responsible team, and accountability disappears
- Quarantine re-runs with automatic re-enablement are necessary to prevent a permanent accumulation of disabled tests
- Developer sentiment surveys are a valuable complement to quantitative CI metrics — they reveal whether the improvement translates to actual productivity experience