Microsoft's flaky test management system spans 100 product teams, flags 49,000 flaky tests, and prevents 160,000 unnecessary session failures
Microsoft built a company-wide flaky test management service — embedded in CloudBuild and CloudTest — that supports over 100 product teams, has identified approximately 49,000 flaky tests, and has prevented more than 160,000 test sessions from failing unnecessarily, dramatically improving developer productivity across its global engineering organization.
Background
At Microsoft's scale — hundreds of product teams, diverse languages, and continuous CI/CD pipelines — flaky tests created pervasive noise in the development process. Engineers were spending time investigating failures that had nothing to do with their code changes, reducing confidence in CI signals and slowing releases. Building a single, language-agnostic system that could serve all product teams through existing infrastructure (CloudBuild and CloudTest) was the strategic approach.
What Was Implemented
- Flaky test management service embedded in CloudBuild (distributed build) and CloudTest (verification service)
- Three-phase system: inference (detecting flaky tests via retry telemetry) → reporting (auto-filing and assigning bug reports) → mitigation (quarantining known flaky tests)
- Default inference: retry-based detection on main branch rolling sessions; extensible with custom team-level logic
- Language-agnostic mitigation operating on test result files, not source code
- Automatic bug report creation, assignment via ownership metadata or git history heuristics
- Quarantine with continued test execution: tests still run but results are suppressed for known flaky cases
- Automatic quarantine removal when developer closes the bug report
- Cultural enforcement option: block PRs for developers with 10+ open flaky test bugs
Results
100+ product teams served across Microsoft. ~49,000 flaky tests identified and reported. 160,000+ test sessions prevented from failing unnecessarily . Telemetry confirms that many associated bug reports have been closed, demonstrating remediation (not just suppression). Teams that enforce the PR-blocking policy are incentivized to fix flaky tests proactively.
Lessons
- Language-agnosticism — operating on test result files rather than source code — is the key architectural choice that enables a single system to serve an entire enterprise with diverse technology stacks
- Continuing to run quarantined tests (rather than disabling them) preserves the ability to detect when flakiness resolves, enabling automatic re-enablement
- Closing the feedback loop between bug report lifecycle and quarantine state ensures flaky tests eventually get fixed rather than permanently suppressed
- Custom inference logic per team accommodates the diversity of failure patterns across a large organization
- Cultural mechanisms (PR blocking for developers with too many open flaky test bugs) translate automated detection into engineering accountability