ð How to design an optimal Recast GeoLift test?
The main goals of the Design tool is to select geographies to include in your test and control groups and get a spend recommendation for the test group.
Step 1: Ingest your data
First we need a dataset of the historical outcome variable for each day for each geography. Geographies can be at any level that you have the ability to target (e.g. states, DMAs, etc). If you use zip codes as the geography, Recast GeoLift can automatically convert these to commuting zones. To design an experiment, you will need at least 90 days of data, with closer to a year of data preferred. Your CSV should contain 3 columns:
Location (geography)
Date
KPI
You can use the tool to map the columns in your dataset to the date, outcome variable and location ID columns as well as specify the date format in your dataset. If a geography/date combination is missing, or if the KPI value is missing for a particular geography/date combination, GeoLift will assume the KPI was 0 on that day.
Once you have mapped your columns, you can ingest your dataset. You can see your KPI over the time period for your top 8 geographies by volume as well as click through your dataset. Use these visualizations to check for any data problems.
The historical dataset helps the experimentation tool analyze similarities and differences in geographical performance to select the best geographies to include in your test group. Comparison between geographies with similar historical performance results in more powerful statistical analysis as we can more accurately identify the differences as a result of the spend change.
Step 2: Configure your analysis
During the configuration phase, you will provide parameters for Recast GeoLift to work with. Recast GeoLift will use those parameters to run simulations in order to determine which geographies would make up the best test as well as a recommended spend amount to get usable results.
First select your experiment type.
There are two types of experiments:
Spend increase: This type of experiment makes sense when you have extra money and want to increase your KPI, you think you might be underspending in the channel and want to test increased spend, or when you are adding a brand new channel to your mix.
Spend decrease: This type of experiment makes sense when you want to save money (at the cost of some of your KPI), or when you think you might be overspending in a channel.
If youâre not sure which to do you can do the analysis twice and compare the recommended plans for each type.
Next, select the KPI type: revenue or conversions. If your KPI is not either, select conversions if you think in terms of CPA and revenue if you think in terms of ROI.
Then select your experiment parameters.
Approximate Channel ROI/CPA: Your expected ROI/CPA helps Recast calculate the amount of revenue/conversions you can drive in your test group. For example, if you want to spend $10,000 and think your CPAs are around $100, this means we can simulate 100 additional conversions in the test geographies and estimate whether that provides statistically meaningful lift. In general, using a conservative number (high CPA / low ROI) will result in a more conservative test, meaning GeoLift will recommend more dramatic changes and youâll have more statistical power.
Experiment length: Provide how many days you want to run the experiment. If youâre not getting good results with a smaller number of days, increasing the amount of days may help.
Cooldown period: The cooldown period after the experiment is a time period where you are no longer spending extra in the channel, but you are still observing the test geographies because revenue/conversions are still coming in from the previous spend. Choosing a short cooldown period might cause you to miss conversions/revenue, while choosing a long time period will cause additional noise that makes it harder to estimate precise ROIs.
Approximate $ for test (or to turn off): This is a starting point for how much money you would hope to spend on the test. Recast GeoLift will first take this spend amount and your efficiency expectations and try to find geographies for which this size of experiment produces a meaningful result. If it cannot find any, it will error and you may need to increase the size of this (or up your ROI expectations). Once it has found a set of geographies with good potential, step 3 will do additional simulations to refine the spend recommendation.
Optionally, you can select certain geos to include or exclude in your test geo. If you leave this blank Recast will select the optimal test geo for you. This can be useful if, for example, you cannot increase the spend in certain geos or you want to exclude certain geographies because other changes happening in the geography may confound the experiment. Alternatively, you can use the Exclude Completely to ensure locations are not used in the design of your experiment at all.
Number of test geos to consider: this allows you to place limitations on the number of geographies you test in. By default, Recast GeoLift allows between 2 and half the total geographies, but you can narrow this down. The simulation engine will try different combinations of geos between whatever high and low numbers you put in
Click âDetermine test marketsâ when you are ready. Recast GeoLift will analyze your data and provide options for various experiment configurations which you will be able to select from. Recast GeoLift ranks the experiment options in terms of the difference between the simulated lift and the lift estimated. If Recast GeoLift is unable to find test groups that result in low error and statistical significance, it will fail to generate the candidate test markets and instead recommend tweaking your settings. Increasing the spend amount, lengthening the number of days, and increasing the number of test markets are all ways to increase the statistical power.
Location shows the set of geographies that belong in the test group.
Baseline Revenue/Conversions is the expected amount of revenue/conversions in the test geography in the simulated âbusiness as usualâ time period.
Simulated Lift is the additional revenue/conversions the additional investment drove in the simulation.
Estimated Lift is the amount of revenue/conversions that Recast GeoLift attributed to the increase in spend.
Absolute % Error This is a measure of the percent difference between the true simulated lift and Recast GeoLiftâs lift estimate. A small absolute % error means that the estimated ROI is close to the true ROI in the simulation.
For each of the experiment configurations, you will be able to see a graph of the expected conversions over time in the test and control geos for the period of the experiment.
You can use this information provided to select a set of test geographies that meets your investment constraints and which minimizes bias.
To get a final spend recommendation for the selected locations and a deep dive into the power at different spend amounts, click âDeep Dive with these locations.â
Step 3: Deep dive power analysis
The results of your power analysis are two testing plans at different effect levels (and different spend levels), as well as an analysis of the likelihood that your experiment results in statistically significant lift.
The two testing plans are a Baseline Confidence Plan that requires less intervention while still meeting the baseline criteria for statistical significance and a High Confidence Plan that will result in smaller confidence intervals. The High Confidence Plan is calculated by multiplying the the baseline confidence numbers by 1.5.
The power analysis graphs below will help you assess the recommended plans. The power analysis runs many simulations for your selected geos in order to help determine how statistically useful the results will be.
Geo - A set of locations
Test Geo - The set of locations where we will implement a spend change for the duration of the experiment.
Synthetic Control Geo - These are weighted conversions in the control group. We use this to simulate the experiment and determine the probability of significant results.
Lift - The incremental effect of the spend change in the test geo.
Approximate CPA - this is a reasonable guess at the spend required to acquire a customer in the channel of interest. This is used to calculate the total spend required in the experiment to produce the number of conversions required for statistical significance in the test geo. A higher CPA means that you will need to spend more to drive the conversion.
Experiment length - The number of days during which we will implement the spend change in the test geo.
Bias - The simulated difference between the actual incrementality and the effect estimated by the experiment analysis.