What is A/B testing? A/B testing is the process of comparing two or more variants to see which one performs the best. Examples where A/B testing could be applied are when you want to compare the blood pressure of patients before and after taking a prescription or when you want to compare whether the changes of an online store would increase the store traffic or not.
How do you perform an A/B Test?
Let’s use the blood pressure of patients example from above to explain how to perform an A/B test.
- Research — Before performing an A/B test, we would need to research information relating to the patient’s medical history. In other words, collect data to help get a better understanding on the experiment that we will run.
- Observe and Formulate Hypothesis — Define the Null Hypothesis: H0. According to our example, the null hypothesis, H0, would represent that there is no effect on the blood pressure of the patients taking the prescription. While the alternative hypothesis, H1, would represent that there is an effect on the blood pressure of the patients taking the prescription.
- Formulate parameters — Before conducting A/B test we would need to figure out the appropriate 𝛼, power, effect size and sample size.
- Run Test — Depending on the type of problem we want to run an A/B test on, we would utilize various statistical test on it. For more information on choosing the right statistical test please refer to these two links. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3116565/ https://stats.idre.ucla.edu/other/mult-pkg/whatstat/
- Result Analysis — After completing the tests, we analyze the test results by seeing if we can reject our null hypothesis or not. If we are able to reject our null hypothesis then we can conclude that the prescription does have an effect on blood pressure.
I hope this help clarify what an A/B test is. If you have any questions feel free to reach out to me at andypeng93@gmail.com.
