Choose the best experimentation method for your team and your business goals. Here are some tips to consider when evaluating client-side vs. server-side testing.
As the cookies crumble and sticky-identifier-loss makes traditional client-side testing less reliable, many of our clients are shifting to server-side testing. But there’s a lot of confusion about exactly what this shift means and its implications for both testing programs and workflow considerations.
At the root of the confusion is unfamiliarity and misperceptions with both client-side and server-side technology and processes, so this post will clarify those, offer some pros and cons of each, and offer recommendations to experimentation leaders faced with new industry-wide challenges. For deeper context and thought leadership about client-side testing vs. server side testing, stay tuned for a forthcoming blog post by Merritt Aho.
What’s the difference between client-side testing and server-side testing?
Client-side testing is the more traditional, common, frequent type of testing that optimization experts use. It’s tried and true and easy to implement but extremely cookie reliant—a fact that now requires caution.
What is client side testing?
Client-side testing means that the experimentation (the change to content) happens at the browser level. An optimizer sets up targeting rules and a visitor’s browser modifies the existing (default) page into the intended variation.
Client-side testing typically uses an off-the-shelf platform like Adobe Target, Optimizely, Google Optimize, etc. This kind of testing can arrive ready-to-go like a pre-built home: You can move right in, but you may want to renovate the kitchen if it doesn’t work for your needs.
Lots of people think of client-side testing like a JavaScript or WYSIWYG file you run on your website that doesn’t need to go through your standard IT processes. “You can have your marketers do all the tests!” they think. “You don’t need a developer!” they also think. But those thinkers are wrong (see below).
What is server-side testing?
Server-side testing means that the experimentation (the change to content) happens on the company’s web server. The server determines the version a visitor sees during the page load process, as opposed to after.
Server-side testing requires more investment in resources and infrastructure up-front than client-side testing. This kind of testing is like a home that you build from the ground up. It takes a long time before you can move into it, and when you do move in, you want it to work exactly as you’ve imagined it.
The primary difference between client-side and server-side testing
is NOT whether or not you need a developer to write code. Both sides can involve code. The difference is where the code lives and how the code is delivered to the user. Whether the code lives on the client-side or the server-side affects the stickiness of the test and the pace of testing, which we’ll cover in more detail below.
Server-side testing requires specific coding infrastructure on your website. Importantly, this code likely requires a team of specific internal developers working on an existing parallel deployment pipeline to make sure it’s performing properly. The biggest challenge for server-side experimentation is that those internal website developers may not have an understanding of how to make a/b tests that are viable. They’re trained to do pixel-perfect changes, which is not what testing is. Most of the time in testing, our goal is a minimally viable version (not pixel-perfect changes) that we can put in front of traffic to make sure that it works, which can be at odds for server-side processes.
What is the workflow difference between client-side testing vs. server side testing?
For a client-side test, the optimization team develops a hypothesis, creates a test spec, and the marketer or developer builds it out within a testing platform (or you contract out that work). There’s extra overhead on the site, but then anyone with access and appropriate skills can implement the targeting rules that will determine the visitors’ experience.
For a server-side test, the work involves more investment in internal resources and more risk, but it has the potential for greater reward. Server-side testing processes happen through a code system that’s likely managed by an internal website developer. This work isn’t likely to be contracted out.
Server-side testing requires this internal developer to spend a lot of time designing the test, so that only the final version of a test is ever served to a visitor, and the developer needs to have an understanding of how the code works with both the optimization platform and the analytics of the site. However, the developer working on the tests isn’t likely to just be working on the tests, and they’re working within a cycle with a standard QA process.
What are the pros and cons of client-side testing vs. server-side testing?
- You can run a lot of client-side tests in the time it takes you to run a few server-side tests. That’s because you can stand something up quicker. HOWEVER, you incur a performance tradeoff because the browser has to make the page different than what it was.
- For client-side testing, the integration of the platform can be handled by the vendor so IT can be hands-off if they choose, which isn’t an option with server-side.
- Client-side testing typically has a quicker experiment development cycle but can have a longer deployment cycle to implement the winning experience than server-side testing where you can often flip a switch to select the winning experience permanently.
- Client-side is a little more risk-averse, therefore less innovative than server-side.
- Client-side a/b testing is easy to debug because it’s done in the browser vs. server-side, where it may need to be addressed across the whole tech stack.
- On server-side, if the test wins, you have already built it. On the client side, if the test wins, you need to build the thing you validated into the site.
- You can set all your cookies on server-side vs. client-side. This helps you naturally mitigate browser privacy changes.
- For server-side testing, since you’re only giving a visitor the final version of a test, you’re shipping less code to them. This can result in a faster loading site, without the performance instability.
- One disadvantage to most server-side platforms is the effort and cost involved in initial setup vs. “one line of JS on page.” That’s true whether you’re implementing a vendor platform or adjusting an in-house solution to allow more optimization/marketing-driven testing. HOWEVER, once set up, not only can you run a lot more kinds of tests, a lot of testing also becomes much easier because it’s baked into the entire call stack.
Bottom line: Each method incurs a time loss somewhere, so each company needs to be clear on optimization program goals and the tradeoffs and benefits of both types of testing. Also, keep in mind that the chances for “success” with testing don’t change—there’s a 30% industry win rate average, so 70% of the time, a new feature won’t outperform an old one.
Every time you set up a test, you’re making an investment. On client-side, you’re not investing incrementally, so you’re not losing incrementally. On server-side, though, you’re making an investment and you’ll likely need to strip it out and try again. Removing code is an incremental effort that doesn’t exist for client-side.
Three recommendations for choosing client-side vs. server-side testing
Along with the advantages and disadvantages mentioned above, it is extremely important to keep in mind that each business and use case for testing is different.
1. You should consider having a good mix.
What kind of a mix? As long as it’s a mix it doesn’t matter as much. As optimizers, our expectation is that because you can move so much faster with client-side, your test volume could be 2-3x as much as you could do with a server-side only test program. So why not do that? Obviously, this depends on the goals of your organization and whether or not it’s easy for you to do client-side testing when the visitor state isn’t sticky.
To determine this, first determine whether you have a workaround, e.g., if you require authentication. If you have no way to ID the users and you want to test something that can’t be anything but visitor-based, such as retargeting (updating this session based on what you saw in prior sessions), then you need a way on a server to identify who someone is. If we have a way on the server to know who you are, we can use retargeting on the server.
When the server hosts information that is PII or info that they don’t want to make available to the user’s browser (like special offers or pricing information, deals, loyalty levels they might want hidden from competitors, coupon aggregators, etc.), they want to keep the info at the server side, not the browser side. So for example, you might target loyalty levels or run pricing elasticity tests only server-side.
Conversely, if you want to test a new page layout, header, CTA copy, the number of products in a comparison widget, optimal way to showcase reviews, etc., none of these tests should require server-side support if scoped correctly. Therefore, you should be able to run many of these experiments client-side—with much less effort—giving you more tests in less time. This will help increase the impact of your program and make it easier to “pay for” the slower, larger-lift server-side tests. A key consideration for client-side testing is whether the visitor-stickiness issue might add confounding variables to your analysis of the test impact. If so, consider same-session testing.
2. You should consider doing more same-session experimentation/optimization w/ client-side tests.
If you are testing across multiple visits or sessions, you need visitor-stickiness to effectively measure impact. If they get into a or b, they stay in their group on return to the site. But a non-sticky test says it’s okay if one person sees a or b, because we’re looking at the impact of the test in that session. Did this treatment increase this person’s likelihood to click this button IN THIS VISIT.
In marketing we have the concept of first in vs. last in. Standard experimentation is focused on first in. You’re always going to see that first experience until the experiment is over. But with same-session testing, we use last-in focus. Behavior gets tied to what you saw last. It’s irrelevant what you saw first. Instead, we focus on EACH visit – and what you see that visit.
In experimentation language, this means you don’t have a “clean control.” However, if your goal is to change the behavior same-session, you don’t need a visitor-based “control,” you need session-based “control,” which we have.
One important note – with increasing privacy impacts taking away our abilities to follow a visitor session over session, session-based testing may become even more important of a focus in the very near future.
3. Caveat your analysis in client-side tests.
A single visitor could get into both the control and treatment. Your testing platform may not be able to see it, but your analytics platform often can. When we run two tests at the same time and customers are allowed in both tests at the same time, we can and should do an interaction analysis.
We should test whether inclusion in test one affected behavior in test two and vice versa. We should do the same thing with any identified visitors who qualified for both treatment and control in the same test. This allows us to find a positive or negative interaction, and we can strip out interactions to get a cleaner hypothesis test to understand the pure impact of each treatment, while also getting a richer understanding of any positive (or negative) interactions.
