A/B Testing Mastery
Understand, measure, and scale. No statistics degree required.
What you will learn in this track
You will learn what A/B testing really is, when a result is valid, how long tests need to run, and how to assess your shop's test readiness.
- After completing this module, you'll be able to tell the difference between a real A/B test and a before-and-after comparison, explain why only parallel, randomized testing rules out external confounding factors, and write a falsifiable hypothesis that clearly names the variable, the expected effect, and the primary metric.
- After completing this module, you'll be able to correctly interpret the p-value, confidence interval, and sample size of a test result, distinguish statistical significance from practical significance, and use the MDE to decide whether a significant winner actually justifies a rollout.
- After completing this module, you'll be able to estimate the required sample size and minimum runtime of a test based on baseline conversion, MDE, traffic, and business cycle - and explain why stopping a test early makes the results worthless.
- After this module, you'll know the right next step for each of the four possible outcomes - winner, loser, no result, stopped early - including how to roll out a winner safely and monitor it afterward, how to break down losers and inconclusive results by segment and document them properly, and how to communicate any result to management in a single sentence.
- After completing this module, you'll be able to honestly assess your test readiness using the 21-point checklist covering technology, organization, culture, and tool stack, calculate your score, and turn your biggest gaps into a prioritized action plan.
Modules at a glance
A/B Testing for Non-Statisticians
After completing this module, you'll be able to tell the difference between a real A/B test and a before-and-after comparison, explain why only parallel, randomized testing rules out external confounding factors, and write a falsifiable hypothesis that clearly names the variable, the expected effect, and the primary metric.
Understanding Significance
After completing this module, you'll be able to correctly interpret the p-value, confidence interval, and sample size of a test result, distinguish statistical significance from practical significance, and use the MDE to decide whether a significant winner actually justifies a rollout.
Test Duration & Traffic
After completing this module, you'll be able to estimate the required sample size and minimum runtime of a test based on baseline conversion, MDE, traffic, and business cycle - and explain why stopping a test early makes the results worthless.
Winner, Loser, No Result
After this module, you'll know the right next step for each of the four possible outcomes - winner, loser, no result, stopped early - including how to roll out a winner safely and monitor it afterward, how to break down losers and inconclusive results by segment and document them properly, and how to communicate any result to management in a single sentence.
The A/B Testing Audit
After completing this module, you'll be able to honestly assess your test readiness using the 21-point checklist covering technology, organization, culture, and tool stack, calculate your score, and turn your biggest gaps into a prioritized action plan.
A/B Testing: How It Works Under the Hood
After completing this module, you'll be able to assess whether your shop should test client-side, server-side, or hybrid, and hold your own in conversations with developers about flicker, caching, hydration in SPAs, and the data layer, without writing a single line of code.
A/B Testing Tools: An Overview
After completing this module, you'll be able to categorize A/B testing platforms into WYSIWYG editors, feature flag systems, full-stack, and open-source solutions, evaluate them using the six selection criteria (traffic, complexity, data privacy, team size, integration, and total cost over three years), and spot red flags before signing a contract.
Ethical CRO & Avoiding Dark Patterns
After completing this module, you'll be able to recognize dark patterns like Roach Motel, confirmshaming, hidden costs, and false urgency in your shop, evaluate every test hypothesis using the ethics check, and consciously prioritize long-term brand value over short-term conversion gains.
Certificate on completion
Pass all module quizzes and the final exam to receive the "A/B Testing Mastery" certificate as a PDF.
8 modules to go before the exam.