We have paused the ability to run REBA analyses while we investigate unacceptable levels of sampling variability in our Bayesian modeling procedure. We hope to make this available as soon as it meets the high bar you expect from Recast.
GeoLift by Recast supports two different modeling algorithms to produce estimates of the incremental effect: Recast Bayesian Analysis (REBA) and Basic (open source). REBA is our own custom, in-house Bayesian model while Basic is based off the open-source GeoLift and augsynth packages.
REBA
The Recast Bayesian Analysis model is a fully Bayesian approach to synthetic control modeling. In testing various approaches, we have found this model produces better point estimates and more narrow confidence intervals with better coverage than both the Basic algorithm and other algorithms found in open source packages. As a result, this is the default and our recommended approach, although we want to supply a more classical approach for those familiar with the open-source libraries.
For details on the REBA model, see REBA Model Technical Documentation
Basic (Open-Source)
This algorithm is based off Meta’s open-source GeoLift R package, which uses another package called augsynth to fit the synthetic control models. Uncertainty quantification is done using permutation testing. One downside of this approach is that the confidence interval on the treatment effect is not guaranteed to (and frequently does not) contain the point estimate of the treatment effect. This algorithm has the advantage of being computationally faster, so may be preferred for large datasets.
Generating the original test market recommendations on the Design tab is currently implemented only with the Basic algorithm, although we hope to make this selectable in the future.
FAQ
Is it a good idea to switch models between Step 2 and Step 3 of the Design process?
Usually not! We allow you to select the model you want whenever we fit models in the app, but we generally recommend using the same model throughout the Design process.
Why do the two methods recommend highly different amounts of spend?
If one model does not “like” the set of geographies chosen by the other model, it will recommend much more spend than the other model to compensate for the lack of good fit. This is why it’s usually better to stick with geo recommendations and spend recommendations from the same model.
Why am I getting a warning about a high rate of false positives?
To test for false positives, we run a simulation where we simulate no incremental conversions and then calculate the model’s predicted conversions. If this number crosses the statistical significance threshold, we flag it as a false positive. Although some false positives are inevitable, it can be a sign of a bad synthetic control, so we flag it as something you may want to avoid.
I ran the analysis using both models and they give different results. Which should I trust?
Through our simulation study, we have found that REBA gives less biased estimates (point estimate is closer to the truth) and narrower confidence intervals. Therefore, we expect this model to be more reliable on average.