❓ Frequently Asked Questions

When to run a lift test?

Recast GeoLift can be particularly helpful when you are seeing large confidence intervals in your model or when your in-platform estimates differ from your model’s estimates.

By conducting a lift test, you are able to provide your model with real information on the effect of your ad spend. This helps the model anchor its parameters to a ground truth. As a result you should see the confidence interval of your model’s estimates decrease following a GeoLift test.

Moreover, if you are seeing different estimates by your model and in-platform numbers, it can be difficult to take action on the numbers you are seeing. Using Recast GeoLift , you can provide your model with a snapshot of more precise incrementality data to train and improve your estimates. Through the cycle of learning more about the incrementality of each channel and improving your estimates, you can make more confident bets and improve your outcome.

What are the limitations of Recast Geolift?

Advertising buys must allow geographic targeting – if it’s hard to limit spend inside a geography and there’s spillage to surrounding geographies, the estimates won’t be as accurate.

Recast GeoLift provides point in time estimates – unlike MMM, the performance measured from a Recast GeoLift test applies to a particular point in time. Measuring different points in time would require running a new test. Our Bayesian MMM works well with the Recast GeoLift output as we estimate incrementality for each channel every day. This means we can apply the knowledge gained from the Recast GeoLift model accurately to the specific time when the test was run without making assumptions about marketing performance over time.

Heterogeneous geos- If different states have different demographics who respond differently to ads, the results will be less accurate

Outliers - eg. a major localized event during the test period could skew the results.

Beta Release - Recast GeoLift is currently in Beta. In particular our application may struggle with high levels of traffic. If the app is not responding to input, please try again later.

What can I do with Recast GeoLift ?

Recast GeoLift allows you to both analyze the results of lift tests as well as design optimal lift tests.

How does Recast GeoLift work statistically?

Recast GeoLift uses a statistical method called augmented synthetic controls to help estimate the incremental effect of the experimental change. When running an experiment, you typically want your test and control groups to be as similar as possible (except for the treatment). Synthetic control models help with this by analyzing the pre-test period and assigning each control geography a weight such that when you sum up the conveions in the control group, it is as close to the test group as possible. Then, these weights are used when analyzing the experiment to determine if the change in conversions in the test group was caused by the spend changes or by random fluctuations.

How long should my test be?

The tool will allow you to test different experiment lengths, and will warn you if it can’t find feasible designs for a particular length. However, we generally would not recommend going under two weeks and think a month is a good starting point for most businesses.

Can I run concurrent tests?

Recast GeoLift supports analyzing tests that were run concurrently, but is not currently set up to provide optimal designs in the case of concurrent tests.

When analyzing the tests, the important thing to do is when analyzing test #1, exclude all test locations from test #2, so that they’re not used in the control when building the model. Similarly, when analyzing test #2, exclude all test locations from test #1.

While you can still use our tool to design concurrent tests, they may not be optimal because there’s no way to precisely control it so that the other experiment’s test regions don’t appear in the controls of the second test, as each change to the exclusions will change the set of test locations recommended. If you are only testing in a small percentage of all locations, these problems will likely be minimal. These problems will be more pronounced if you’re testing in a large percentage of all your regions.