Experimental design

Current conversion rate is

This is the conversion rate of the control treatment. In medical trials, it is the success rate of the placebo

A new treatment needs to increase conversions by

Acceptable false conversion rate α is

Acceptable false rejection rate β is

Experiment design summary

With the suggested

we have two possible ways to make an error:

If both treatments have the same success rate of %, we incur the costs of changing for no benfit α% of the time.
If the alternative treatment has a success rate of %, we reject it and miss out on the benefits β% of the time.

It is suggested that you run your experiment N number of times.

How many trials do you wish to run:

Sample Number	Proportion A	Proportion B