A/B testing is one of the best ways to level up your company’s digital marketing strategy. I’ll walk you through the top benefits of A/B testing and provide step-by-step instructions for implementing A/B tests below.
What is A/B Testing?
A/B testing, sometimes referred to as split testing, is a method used to compare two versions of a product or marketing campaign to determine which performs better. It is a valuable tool for businesses, organizations, and individuals looking to optimize their marketing efforts and improve their products or services.
With A/B testing, the “A” is your control, and the “B” is your variant. Traffic is split between the two versions, and the results are tracked. A winner is declared once a certain visitor threshold for testing is complete.
Only one element changes between the A and B versions when A/B tests are performed. For example, your call-to-action (CTA) button might be blue on your A version and red on your B version. If multiple changes are made, such as changing button colors, background images, and wording, and you’re testing all the different combinations of those elements, it’s referred to as “multivariate testing.” This is a form of split testing as well since visitors are split between the variations. However, multivariate testing is more complicated and requires greater traffic, so A/B testing is generally best for small businesses and those just starting out.
What Can be A/B Tested?
Just about any type of digital marketing collateral or digital product can be A/B tested.
Websites
In all, 77 percent of companies say they run A/B tests on their websites, according to Finances Online research. While there is a wealth of potential split tests that could be run, some basic examples include:
- Changing the headline
- Adjusting the navigation menu
- Adding trust signals
- Swapping accent colors
Landing Pages
Businesses use A/B testing on their landing pages 60 percent of the time per Finances Online. For clarification, a landing page is designed for conversion. You should be sending your ad traffic, email clickers, and other interested parties to these special pages rather than to your general website. A few examples of tests for landing pages include:
- Changing the CTA button color
- Changing the CTA wording
- Adding or removing videos
- Moving your form or CTA to a different position
Email A/B testing comes at a close third, with 59 percent of businesses reporting that they run tests. Some examples of email marketing A/B testing include:
- Using a different subject line
- Including an emoji in the subject line
- Personalizing the subject line or email body
- Using the business name as the sender vs. using a person’s name
Paid Search and PPC Ads
Surveys show 58 percent of businesses run tests on their paid search campaigns. It’s so effective that many pay-per-click (PPC) advertising platforms now require you to provide alternate assets and copy so that they can run the tests for you. Some options here include:
- Different headings
- Alternate images
- Unique CTAs
- Distinct benefit statements
Top Benefits of A/B Testing
You should now have a good feel for what A/B testing is. You may even have some ideas you’d like to try. Now, let’s go over the business case for A/B testing.
Improved Decision-Making
A/B testing allows you to make informed decisions about your marketing and product development efforts based on real data rather than assumptions or guesswork.
For example, the team at Bing had a feeling that slow load times impacted revenue, as recounted by Harvard Business Review. This is something developers and marketers are acutely aware of today. The problem was that they didn’t know what slow loads were costing them and how much to invest in development to address speed. They ran a series of A/B tests that showed every 100-millisecond difference in performance impacted revenue by 0.6 percent. This wouldn’t be a huge difference for a small company, but for Bing, even a 100-millisecond improvement was worth $18 million per year. Armed with these figures, the company created a task force to address speed.
Increased Conversion Rates
More than half of all marketers use A/B testing to boost conversions per Finances Online research. By testing different versions of a product or marketing campaign, you can identify the elements that are most effective at converting potential customers or users.
With that said, sometimes non-obvious elements can have a dramatic impact on conversions. For example, HubSpot knew that users who engaged with the site’s search bar were considerably more likely to convert. The company wanted to find out if small tweaks to the search bar could increase overall conversions. Through A/B testing, the company discovered a variant that increased search bar use by more than six percent and conversions by more than three percent.
Enhanced User Experience
Anything that creates friction for your visitors or customers diminishes the user experience. Even little disconnects between what a user expects to happen and what actually happens will send some people packing. A/B testing can help you identify and fix these problems, leading to a better experience for your customers or users.
For example, all businesses should have an onboarding process for customers. Giving people a smooth start is even more important when the product is virtual, such as an application. This is where Houseparty, the social networking app, ran into some kinks. Users had to grant the app permission to access the microphone so the people they were talking to could hear them. Other security approvals, such as access to contacts and permission to send notifications, were also crucial to the app’s use. The company’s first flow launched all the questions at once with a quick note on each explaining why it was necessary. Unfortunately, people didn’t stop to read. They denied access and then couldn’t use the app well. The company ran A/B tests on a new onboarding process and discovered that if the app paused to explain why something was necessary, access permissions spiked 15 percent, and friend requests doubled.
Increased Revenue
Most of the examples outlined here will also produce additional revenue, though sometimes, revenue is the ultimate goal. That’s why big companies like Amazon and Microsoft reportedly run 10,000 or more A/B tests per year.
For example, Amazon used to advertise its credit cards on its homepage. When the company ran an A/B test to move mentions of the card to the shopping cart, profits soared by tens of millions of dollars per year, as reported by Harvard Business Review.
Back over at Bing, a team member had an idea that tweaked how headlines were displayed. The company shelved it for more than six months before an engineer realized it could be tested rather quickly. The tiny change boosted revenue by 12 percent, amounting to more than $100 million in additional revenue in the United States alone.
How to Use A/B Testing Effectively
Ready to start running A/B tests? These tips will get you off to a good start.
Step 1: Identify the Goal of Your A/B Test
What are you trying to achieve? Consider goals like:
- Increasing conversions
- Improving user experience
- Boosting engagement
- Better budget allocation
Step 2: Choose a Metric to Measure Success:
How will you know if your A/B test was successful? Choose a metric to track, such as:
- Conversion rate
- Time on site
- Engagement
- ROI
- Cost per click
Step 3: Select the Right Technology for the Job
One thing that often prevents small businesses from conducting A/B experiments is a lack of technology. However, you may already have access to A/B testing software and not even realize it. Plus, you can sometimes conduct tests without any special software or pick-up tools that can help. Many are quite affordable, but remember that A/B testing helps improve your results. This means that regular A/B testing, though it may be an investment, will deliver ROI when done effectively, regardless.
For instance, if your business invested in a CRM like HubSpot or Salesforce and uses its email marketing tools, you can already run A/B tests. Furthermore, standalone email marketing platforms like Klaviyo offer testing tools too. Yet, even if your chosen platform doesn’t offer native A/B testing, you can still split your list on your own and send each half a different version of an email.
The same is true if you’re running website tests. Zoho One users, for instance, have access to Zoho PageSense, which allows you to split traffic and send people to different landing pages. If you have a WordPress site, there are lots of plugins that offer similar functionality.
Consider what you want to test and what your goals are, then see if your current tools will allow you to run A/B tests. If not, do a little research to see what’s available. While I have a few tools I prefer over others, what matters is that you choose something that works for your needs and gives you the data you require.
Step 4: Start with an A/A Test
It’s helpful to start with an A/A test before running A/B tests. In an A/A test, you test something against itself. In a well-run A/A test, both versions should produce roughly the same results. If your results aren’t close, further analysis is required to determine why. Don’t move forward with A/B testing until you identify and resolve the issue, or any results you have from the A/B tests will likely be skewed by the same errors.
For example, most A/B testing tools will automatically split your audience in a way that eliminates bias. The split is supposed to be random, and through this, the demographics of each group should be roughly the same. For instance, if you’re running an A/B test on a landing page, your A group should have approximately the same level of mobile traffic as your B group.
If you’re responsible for splitting up traffic, as you might be while running tests on email campaigns, ensure your A group and B group are randomized. Consider how your software presents your list. For instance, some platforms will deliver your list to you in the order in which people were added to your database. You do not want to split it down the middle. Your older contacts are likely to behave differently than your newer ones. You may also have blocks of contacts that came to you in a specific way – perhaps you ran a referral campaign or attended a conference. Those blocks of content may skew results for the full group, so they’d need to be split up as well.
You may need to change the tools you’re using or how you’re segmenting your audience if your results don’t align after an A/A test.
Step 5: Set Up Your A/B Test
Once you have clean A/A tests, you can start running A/B tests. Again, it’s better to only change one element per test to ensure which change is responsible for the result. Determine which elements of your product or marketing campaign you want to test and create two versions to compare. If you have lots of ideas, you’d like to test, create a list and plan to test them in sequence.
Step 6: Run Your A/B Test
Collect data from your A/B test and analyze the results. Look for patterns and trends in the data to determine which version performed better.
Some sources say that you need data from a minimum of 25,000 users before you can have conclusive results, while others say as few as 5,000 users is enough. If you’re already doing the math and realizing it might take you a month or more to get that kind of volume, stay confident in your efforts. Many modern A/B testing platforms have excellent predictive capabilities that can often identify the winner with high accuracy, even with much lower volume.
Step 7: Replicate
Remember the Bing story about how a small tweak to headlines boosted revenue by 12 percent? Unfortunately, Bing didn’t act on it right away. In fact, the results were so surprising that the team was skeptical. They thought something went wrong with the test.
You should be applying Twyman’s law too. It states: “Any figure that looks interesting or different is usually wrong.” It further notes that “the more unusual or interesting the data, the more likely they are to have been the result of an error of one kind or another.”
Errors sometimes happen, regardless of how cautious you are. Moreover, correlation does not equal causation. For instance, Yahoo once ran a test to see if ads influenced the number of branded searches. Their A/B test showed searches for brands increased by 871 to 1,198 percent while their ads ran on Yahoo sites. That sounds like a clear winner. However, the single experiment didn’t consider that other variables were changing during the test period. Later tests showed that, although searches increased, the growth was closer to five percent.
Run at least one more A/B test with the same variables to see if it produces the same results. Consider running more than one if your decision is going to impact your marketing budget, revenue, or ROI in a major way.
Step 8: Implement the Winning Version
“I have not failed,” Thomas Edison reportedly said. “I’ve just found 10,000 ways that won’t work.” This is what, statistically, you will take away from most of your A/B tests. The new variant only wins in one in eight tests per Finances Online research. That’s not necessarily a bad thing. It means you’re using the best strategy for your business in this moment. Future tests will help you improve incrementally, so stick with them and keep trying new tests.
When you’re sure you have a winner based on data, implement it. Then, again, keep running tests against it.
Step 9: Keep Going
A/B testing is just one tool among many that organizations can use to optimize their products and marketing efforts. Moreover, A/B testing has limitations, such as the sample size and the inability to account for external factors that may influence the results. It’s important to use a balanced approach with your marketing and product development for this reason. For example, traditional marketing analytics can tell you quite a bit about performance, which features people use, and where they leave your funnels. Polls, written surveys, heat maps, user testing, focus groups, and many other tools can help you improve digital marketing.
Get Help with A/B Testing or Improving Your Digital Marketing
As a seasoned digital marketing consultant with a background in business, I use a variety of tools and resources to ensure my clients are always getting the best possible results for their marketing investment. If you’re struggling to implement A/B testing effectively or aren’t getting the results you expect from your digital marketing efforts, I can help. Tell me a little bit about what you’re experiencing, and we’ll schedule a time to talk.
