Marketing has some problems that randomized controlled trial studies (RCTs) can solve, especially in the areas of media attribution and measurement and budgetary considerations. Which problems? What are the solutions? We cover those here.
Measuring the holistic impact of the digital advertising ecosystem—including search ads, SEO, social ads, programmatic display, CTV, and digital video ads—can be challenging. Several of these areas, along with offline to online measurement/attribution, have technical and budgetary problems that randomized controlled trial studies (aka. controlled experimentation, field experiments, clinical trials, or A/B tests) can solve. We outline these unique problems and solutions below.
Table of Contents:
Media Attribution & Measurement: Display & Video
CTV Attribution & CTV Measurement
Online-to-Offline Attribution & Measurement
Budgetary Issues: Justifying Spend & Validating ROI
When to Get Help Setting Up RCT Studies
Media Attribution & Measurement: Display & Video
Unique problems
In display & video advertising, even before all the cookie issues (which are getting more challenging by the day), an inherent measurement problem exists. The nature of display & video ads is almost like a TV commercial or a billboard—they show up when you’re already focusing on something else and compete for your attention. For example, you see video ads as a pre-roll when you’re about to watch a video on YouTube, or display ads show up as banners on a site where you’re already engaging with some other content. Since you’re already busy, it takes a lot to stop what you’re doing to engage with a display ad.
The loss of cookies exasperates this inherent problem. A prospect used to get a cookie just for viewing an ad, which meant that if they saw an ad then searched for and purchased the product later, the action would qualify as a view-through. Now, though, browsers block most of those cookies, causing increased difficulty when trying to attribute how well a display, CTV, or video campaign is performing.
How RCT studies help
In the simplest terms, media RCTs randomize ad viewers based on geography and compare them against a non-viewer control group. How does this solve attribution problems? When serving different ad sets in selected comparable geographies and holding others as control, you can then measure and compare the overall lift within markets. From these measurements and analyses—plus a hefty amount of data science work to ensure your geographies are truly comparable and results are statistically significant—you can track online and offline activities within the markets to better understand what’s working.
For example, say a brand with a significant offline brick and mortar presence runs a market analysis and finds three markets that are highly comparable in Baltimore, Cleveland, and Indianapolis. They then run one set of ads in Baltimore, the same set in Cleveland, and hold Indianapolis as a control group. If there’s a lift in online and offline store sales in Baltimore and Cleveland but not Indianapolis, you can determine the impact the ads had at a holistic level.
While it may seem reminiscent of traditional media measurement, this cookieless attribution technique is the gold standard solution for a reason: It can demonstrate causality and helps advertisers understand what their ads are doing without a complete chain of visible events.
CTV Attribution & CTV Measurement
Unique problems
Here’s a common situation with CTV: If you’re watching Hulu and you see an ad for a brand, you may get a prompt to ask Alexa to add a product to your shopping cart or to visit the brand’s site using a specific url, or even to click on a link. But you’re not going to ask Alexa right then—as you’re about to watch your show—and you’re not going to pause the ad to click the link or type that url into your phone manually. Instead, you’re going to continue watching your show and maybe Google the brand on your phone.
When people are watching TV, they’re commonly also on another device like a smartphone. To return to our example, if you Google the brand on your phone while watching that CTV ad on Hulu, the brand has almost no way of knowing the ad made you do that.
How RCT studies help
Using a geo-based randomized controlled trial study to block media markets into similar geographies, marketers can run CTV ads and look at every interaction: Did people Google the brand? Were direct visits up? When CTV is running in an area and there are many more direct visits, organic brand searches, paid brand clicks, and even store visits than in an area where CTV isn’t running, that helps quantify ad impact.
Online-to-offline Attribution & Measurement Problems
Unique problems
QSR, personal services, and other brick-and-mortar reliant brands also see significant difficulty evaluating the true impact of their online media efforts. These brands might market their hearts out online, but they’ll likely see the vast majority of their sales or event attendance in person. So how can these brands know their online ads are making a difference?
How RCT studies help
An RCT study can break up geographies, track all the ads that ran in, say, Topeka, and measure all the interactions there. Then the study can compare those interactions to those in, say, Tampa, where the brand didn’t run any ads. In this way, marketers can see the impact of their digital efforts, even if sales occur in non-digital spaces.
Budgetary issues: Justifying spend & Validating ROI
Another problem marketers face is “marketing by accounting.” While the 360-degree fallacy affects all channels, marketers still need to justify the return on their spending.
When marketers compare returns from channels that can still be tracked to cookieless media channels (like programmatic and CTV), the cookieless channels will always underperform in lower-funnel tactics like paid search. If you want to fight for funding for these channels and don’t have any data-ammo to fight with, you’re unlikely to receive an adequate budget.
Here’s an example of that challenge highlighting display advertising. Say the CFO of your company hands you a 20 million dollar budget to spend on advertising this year. Say you max out search on 10 million, allocate an additional 5 million to social, and have another 5 million left to spend on display.
In a cookie-reliant model at the end of the year, you can go to the CFO and report the following: For the 10 million I spent in search, I got 50 million back–it was great! I got 5:1 ROAS. In social, I got 3:1 back! And, in display….I spent 5 million.
While your display advertising probably influenced a portion of that 5:1 search return and even drove social, organic, and in-store, you can’t prove the ROI for that 5 million. Is it likely that you’ll get that money to spend next year? Not with data points like that.
Maybe when you turn off display, all your other channels do worse. This might give you a clue that display is important and needed for your team’s success, but if you don’t know the level of impact it’s having, and you can’t justify it to your executives, you’re in a bad place.
Conversely, there is no user-level tracking in a geo-based media RCT, and the impact measurement can be the actual business outcome of interest. Therefore, even without cookies, these studies can help you know what to spend and measure the display advertising impact.
When to get help setting up RCT Experiments
If you’ve made it this far, you likely recognize the critical impact RCT studies can have on your marketing program, but how do you go about actually setting up these studies? Can you set up RCTs yourself? Certainly. You might have the in-house data science resources and expertise to do this or to health check your media attribution models.
However, you might not have that expertise, or your data scientists might be overburdened and not have the time to ensure a successful RCT study. Typically, data scientists aren’t solely focused on marketing efforts. That’s where Further’s RCT experts can step in and help augment your team’s work.
RCT studies are our team’s focus. We know how to structure these studies. We take many factors into account that differentiate our studies from others’, including geography-based comparisons that we’ve already got at our fingertips. Remember, online behavior doesn’t always correspond with similar city sizes or demographics. Instead, we group geographies based on several complicated factors, including how people interact with various mediums.
