Stages of A/B testing

The five stages of conducting an A/B experiment. They are:

1. Identifying the product feature that needs to be changed. In the case of A/B testing, it is generally recommended to work on one change at a time.
2. In the second step, the hypothesis for A/B testing is defined, which should include:
   • The statement of change, 
   • The impact that you expect to see with the change, and
   • The rationale for the expected change.
3. In the next step, the key success metric or metrics are chosen. The key success metric will help you in assessing the success or failure of the A/B experiment. It also helps in determining the sample size and the duration of the experiment.
4. The hypothesis is updated according to the chosen key success metric.
5. Finally, the experiment is taken live.

To read more about the concept of basis points, you can refer to this link.

To read more about the Minimum Detectable Effect (MDE), You can refer to this link.

• A/B Testing: Introduction
• A Product Manager’s notes on A/B Testing
• Machine Learning in A/B Testing:  How Machine learning can be used to accelerate A/B testing

You can access Evan’s Sample Size calculator through this link.

1. The minimum detectable effect has a big impact on the sample size required. For a smaller MDE (minimum detectable effect), you need a larger sample size and vice versa.
2. The magnitude of the baseline conversion rate also has an effect on the sample size required. The smaller the baseline conversion rate, the larger is the sample size required.

Learn the basics of Multivariate Testing

• Multivariate Vs A/B Testing: This article clearly explains the difference between multivariate and A/B testing. 
• Read about everything related to Multivariate testing in this interesting article. 
• Examples of multivariate testing