The question behind this project
Does a different page lead to more purchases?
Two groups see different versions of a page. One seems to work better—but could that be chance? This practice case uses an enriched dataset to examine the difference and how sure we can be.
Read the analysis
Product analytics An analytical case study
Evaluating a change in conversion with an A/B test
Is the treatment effect large and consistent enough to justify a rollout?
The finding
This constructed experiment demonstrates how an aggregate result can conceal different effects across user groups.
Analytical work
Bring effect size, statistical uncertainty, segment checks, and experiment design into one decision process.
Estimated change in conversion and its uncertainty
01
Estimate a difference, not just a verdict
Compare treatment and control conversion rates in percentage points. Read the uncertainty around that difference alongside the minimum improvement worth shipping. A small p-value alone says nothing about implementation cost or commercial relevance.
02
Check who contributes to the average
Device and returning-user comparisons illustrate heterogeneous effects. These patterns were deliberately introduced by modifying conversion outcomes and generating synthetic attributes. They demonstrate an analytical workflow, not a recovered finding from the original experiment.
03
Make the next test more decisive
Choose a primary metric, a meaningful minimum effect, and a stopping rule before collecting data. Frequentist intervals and Bayesian probabilities describe uncertainty under their respective assumptions; neither removes the need to account for repeated looks and exploratory segments.
Decision supported
What to do with this evidence
Use this case to specify a confirmatory experiment. Any segment-specific rollout requires fresh, unmodified evidence and a predeclared decision rule.
What this analysis cannot establish
- Conversion outcomes are changed by the enrichment pipeline. Revenue, device, and other added attributes are synthetic; published treatment effects cannot justify a real product rollout.
- Opposite effects in some segments do not by themselves establish Simpson’s paradox. Exploratory comparisons require multiplicity and interaction checks.
Continue the investigation