GitHub reduces flaky builds by 18x — from 9% of commits to less than 0.5% — with automated detection and quarantine
GitHub built an internal flaky test management system for its Ruby on Rails monolith that reduced the percentage of commits with at least one flaky build from approximately 9% (1 in 11 commits) to less than 0.5% (1 in 200), an 18x improvement, by automatically detecting, containing, and delegating flaky failures to the engineers who introduced them.
Background
GitHub's own codebase — a large Ruby on Rails monolith — suffered from a 9% rate of commits affected by flaky builds, slowing developer velocity and eroding trust in CI signals. Existing detection methods (retry + same-code comparison) caught only 25% of flaky failures, leaving the rest to manual investigation.
What Was Implemented
- Multi-scenario test retry system: 3 targeted retries per failing test covering randomness/race conditions, time-based flakiness, and order-dependence/shared state
- Automatic flakiness confirmation: if test passes on any retry scenario, failure is marked flaky and build is kept green
- Impact scoring: tracks failures by branch, developer, and deploy to prioritize highest-impact flaky tests
- Automatic issue creation and assignment to engineers identified via git blame and recent commit history
- Historical failure context attached to each issue (failure patterns, likely cause category)
- Self-improving: system continuously learns from retry outcomes to better classify failure causes
Results
18x reduction in commits with flaky builds : from ~9% (1 in 11 commits) to less than 0.5% (1 in 200), achieved within approximately six weeks of system introduction. 90% of flaky failures now automatically identified without developer intervention (up from 25%). The system is described as GitHub's lowest flaky-build rate since tracking began in 2016.
Lessons
- Targeting the top 0.4% of highest-impact flaky tests — rather than trying to eliminate all flakiness — delivers the greatest developer-experience improvement per engineering effort
- Multi-scenario retries (same process, time-shifted, different host) are far more effective than single-retry approaches; each scenario targets a different root cause of flakiness
- Automatic issue assignment using git blame and commit history ensures that flaky test accountability lands with the engineer who most likely introduced the failure — preventing the "bystander effect" at scale
- Impact scoring and prioritization queues are essential: without them, high-volume flaky detection creates a backlog that teams cannot act on
- Keeping builds green while logging and triaging flaky failures prevents developer workflow disruption while still ensuring the problem reaches the right person