tldr: the approximate channel ROI determines how much power your experiment will have. You should choose this number conservatively (i.e., a low ROI or a high CPA) to ensure that your experiment is well-powered to detect a true lift.
When designing an experiment, the GeoLift tool asks you to provide an approximate ROI//CPA of the channel. This may seem counterintuitive, isn’t the whole point of designing a GeoLift experiment because we don’t know the ROI? It is indeed. In this article we’ll go over how we use this information and tips for choosing a good value.
How GeoLift Uses Approximate ROI
Like many statistical tools, GeoLift’s Design tool is essentially a simulation engine. The engine works by simulating lots of different possible experiments, and then choosing the experimental designs that have the best statistical properties.
The design tool in GeoLift actually has two different simulation phases that are similar but somewhat distinct. The “approximate ROI” setting has large impacts on the results of both simulations.
At the core of simulation engine is a simulated experiment. In any given simulation we will have a treatment group of geographies and a set of control geographies. In the simulation, we will simulate a true lift (e.g., by adding a 10% lift in conversions to every day for every geography in the treatment group) then we will check to see how precisely and how frequently we can “detect” that true 10% lift using the synthetic control method . If a given set of treatment and controls allows us to consistently estimate the correct amount of lift, that is a “good” experiment design, whereas sets of treatment and control geographies that frequently over- or under-estimate the true lift are “worse” experiment designs.
It’s important to keep in mind that larger effect sizes are easier to detect, and the estimated ROI / CPA determines the size of the effect used in these simulations. So if you choose an ROI that is too high, your simulations won’t be realistic and your test will not have as much power as you think.
Simulation 1: narrowing down the geographies
The goal of the first simulation is to narrow down the number of possible experiments to analyze. As an example, there are over 10 billion possible ways to create an experiment of 10 treatment states out of 50 possible states (in the US). So to make the algorithm run in a reasonable amount of time, we need a shortcut to determine which of those 10 billion combinations yield plausible experimental designs.
In the first simulation we select geographies into potential test groups, then estimate the total lift driven by the experiment (by multiplying the money-to-invest parameter by the approximate ROI parameter) and then we estimate if the algorithm detects a “statistically significant” lift.
Groups that show statistically significant lift and had estimates close to the simulated value get recommended.
Simulation 2: determining power
In the second simulation we deep dive on a particular set of geographies and simulate multiple versions of the experiment over multiple starting dates in order to determine exactly how much you need to spend in order to measure the effect precisely.
Once again, we use the approximate ROI parameter to simulate the absolute size of the effect – larger effects from higher ROIs are “easier” to detect a lift so the algorithm will recommend smaller experiments in general. If the true ROI ends up being lower than your guess, your experiment will be under-powered!
Tips for Choosing
By looking at the consequences of choosing a too high value or a too low value, we can come up with practical advice for where to start.
If the ROI you choose is too low (or CPA too high):
-
It will be harder to determine statistical significance, so only the locations with the least noise will be offered as potentials
-
In extreme cases, no locations will meet the threshold and GeoLift will force you to change some of your design.
-
-
The recommendations of how much to spend will be higher because more spend will be required to reach the statistical power goals we set.
-
Once you run the experiment, more signal will be generated than you expected so you’ll have narrow confidence intervals.
If the ROI you choose is too high (or CPA too low):
-
Many different combinations of test locations will appear viable because the effect will be large and easy to measure
-
The spend recommendations will be lower since statistical power is easier to achieve.
-
Once you run the experiment, there will be less signal than we simulated so you’ll have less certainty in the results and wider confidence intervals.
From those consequences, we recommend starting with the lowest ROI that (a) is still believable and (b) would have practical business consequences. For example, you may think it’s possible that Channel X’s performance is <1, but if 1x is the lowest ROI that would cause you to invest any money in Channel X, that could be a candidate to use as the ROI number. If you run through the analysis with a low ROI and find the spend recommendations infeasibly large, you can use a higher ROI, but the tradeoff is if the ROI turns out to be on the lower end, we likely won’t be able to get a precise read on it.