Meta's predictive test selection captures 99.9% of regressions while running only one-third of tests, doubling infrastructure efficiency
Meta (formerly Facebook) deployed a machine-learning predictive test selection system that catches more than 99.9% of regressions before they reach the trunk, while running just a third of all dependent tests — halving testing infrastructure costs.
Background
Meta's trunk-based development model — where all engineers work against the latest version of the codebase — requires every proposed code change to be thoroughly tested before acceptance. With thousands of daily changes and a monolithic codebase, build-dependency-based test selection (the conventional approach) selected up to 25% of all tests per change, creating unsustainable infrastructure costs. Meta needed a more precise approach that could cut test execution volume while maintaining regression detection fidelity.
What Was Implemented
- Gradient-boosted decision-tree ML model trained on historical code change and test outcome data
- Probabilistic test selection: each test's failure probability for a given change is estimated; tests below threshold are excluded
- Continuous model retraining as the codebase evolves — no manual tuning required
- Flakiness mitigation: aggressive retry logic during training data collection distinguishes genuine failures from flaky noise
- Integration into Meta's trunk-based CI system, operating at the scale of thousands of daily changes
Results
Meta's predictive test selection system catches more than 99.9% of all regressions before they reach the trunk, while running only one-third of dependent tests — cutting total testing infrastructure costs by a factor of two. More than 95% of individual test failures are still reported to developers. The system has operated in production for over a year as of the November 2018 publication.
Lessons
- Reframing test selection as a probability estimation problem (rather than a dependency coverage problem) unlocks dramatically more efficient test portfolios
- Gradient-boosted decision trees deliver explainable, efficiently trainable test selection models at scale
- Continuous retraining on recent code changes is essential for the model to remain accurate as the codebase evolves
- Aggressive retry logic during training data collection is the key to teaching the model to distinguish regressions from flakiness