Amazon uses A/B testing and causal ML to validate AI-enhanced catalog data improves customer experience
Amazon's Catalog team applies generative AI to enrich product data at scale and uses causal random forests and Bayesian structural time series to rigorously measure whether enrichments improve customer decisions and conversion.
Background
Amazon's catalog is the foundation of its customer search and purchase experience. Inaccurate, incomplete, or inconsistent product data leads to poor customer decisions and reduced conversion. Generative AI offers a way to enrich at scale, but without rigorous measurement, it is impossible to know whether enrichments help or harm the customer experience.
What Was Implemented
- Generative AI models synthesizing text and image data from seller listings, manufacturer sites, and customer reviews to enrich catalog attributes
- A/B experiments exposing cohorts of customers to enriched vs. baseline product information
- Causal random forest model to extrapolate A/B test learnings to new enrichment initiatives without running exhaustive experiments
- Bayesian structural time series for observational measurement when A/B testing is not feasible
- Attribute-level quality thresholds: enrichments are deployed to the live catalog only when model accuracy exceeds 90–95%+ depending on the attribute
Results
Amazon's measurement framework validates whether enrichments improve customer experience before full rollout. The approach allows large-scale catalog improvements (completing product information across millions of attributes) to be evaluated efficiently. No specific conversion rate or revenue metrics are disclosed publicly in the Amazon Science source. The system has been validated against actual A/B experiments for selected use cases.
Lessons
- Measuring the causal impact of catalog enrichment requires statistical rigor beyond observed sales trends — A/B experiments and observational modeling are both necessary tools
- Causal random forests enable efficient extrapolation from sparse experimental data, reducing the cost of running experiments on every enrichment type
- Setting accuracy thresholds (90–95%) before deploying enrichments to the live catalog is a practical quality control mechanism
- Product image enrichment (e.g., AI-generated lifestyle backgrounds) can be A/B tested with the same causal rigor as text enrichments