Follow Datanami:
November 1, 2013

Using A/B Testing to Improve Your Business

Nick Robson

Many businesses, especially smaller businesses, are either unaware of what A/B testing is or they have erroneously dismissed it as too complicated for them to try.  This assumption can actually harm the potential growth of a business by neglecting one of the most useful tools for understanding exactly what your customers respond to.

A/B Testing involves collecting data for two groups (Group A and Group B) by supplying them with two different versions of what is essentially the same content and analyzing their responses. You may have also heard it called controlled experiments or split testing.  It is used  to discover what small differences influence the customers’ behavior and get them to do more of what the company wants them to do.

For example, if a business is looking to increase the amount of social media shares that they are getting, they may create two versions of the content they want to see shared.   The first version would be shared with one group and the second would be shared with another group. They would then analyze the data from each group and see which content was shared more.

This is also useful when trying to attract visitors, creating landing pages or email campaigns, converting visits to sales, getting people to subscribe to emails, and more.  It can be used for just about anything that can be slightly altered in a way that may impact the way that your visitors react to it.

So, why should you use it?  Overall A/B Testing is an underrated tool that produces the kind of data you can’t get anywhere else.  This is direct data based on your company and your products or services.  It is targeted to your own audience and will give you an insight directly into what motivates them to make one choice or another.

Using A/B Testing

It is best to treat A/B Testing as a process.  It doesn’t have to be extremely complicated, but it does require some thought and organization, as well as a significant amount of time both preparing, gathering, and analyzing each part.

Step One: Define What You Want to Know.  Decide exactly what information that you are hoping to gather. What is the purpose of your test?  What are you hoping to accomplish?

You should have only one goal, such as determining if using a different subject line in your email campaign gets more visitors than another one.  You may have many things you want to know, but A/B testing requires doing only one at a time or your results will not be entirely conclusive.

Step Two:  Determine How To Get The Answer.   Once you know what you are trying to figure out, you need to decide what will be different between the two groups. A lot of times the basis of this is included in step one, but this takes it a little further.

Let’s say that you’ve decided you want to know whether using images on your sales page is better than just using text.  You know that you’ll need one page that has images and one that has text.   Note: Nothing else should be changed between the two groups.  With A/B Testing it is critical that only variable be different between what the two groups see and that everything else remain the same, so you know exactly what is causing the difference.

Now you must decide what metric is the most important to you.  It’s very important to know this before you begin or you can get bogged down in all of the different information and trying to decide what’s most important.  For this example, you will probably base it on your conversion rate.  What you decide is what you have to keep in mind later.

Step Three:  Implementing The Change.    This should usually be one of the easiest steps since you should only be making one change.   At this point you make the two different versions and present them to different groups.

If you’re running an email campaign this may be as simple as sending out different versions to different batches of people.   If you are testing variables on your website there are many online tools that can help you with this. Unbounce is an excellent tool that focuses on landing pages. Optimizely is another that gives you more options and allows you to track the results you are looking for. Visual Website Optimizer allows you to create two different versions of your website in order to compare differences. Vertster is a great tool for needs the others may not cover.

Also, before you run the test, make sure that your website is capable of handling the traffic and running the software.  If you aren’t sure, check with your host. If you aren’t capable,  you may want to find a different one so you can get the most out of your site.

Step Four: Running the Test.   This should be the easiest step by far because it requires you to do nothing but sit back and wait.   All you need to be concerned about here is making sure that you run the test long enough, and as many times, as you need to get accurate results.

Generally, you should run it long enough to target 100-200 users, whether that means 24 hours or 24 days for your business. This can also vary based on the data that you are trying to collect.  An email campaign test may only last a week while a test of the conversion rate on a page may last longer.

You also need to try and go for an equal amount of people tested in each group so that the results are truly comprehensive.

Step Five: What To Do With the Data.   When you decide that the test has been run enough times as long as it needs to be, it is time to really look at the data that you received.  Remember the metric you decided in the beginning was the most important, and try to keep yourself from getting influenced by other numbers.

Then you have to accept the data, even if is contrary to what you expected or shows a result that you don’t want to see.  No one wants to know that all of the time and money he spent isn’t the best option for his business. But if that is what the data shows, then you need to adjust accordingly.

Finally, record what you’ve found in detail.  This allows you to come back to it later, to share it with others, and help give yourself a big picture if you run tests on other areas as well.   This step should not be neglected.

A/B Testing can greatly improve the effectiveness of your business, making sure that your marketing tools are optimized for getting the most response from your customers. It is not as difficult as it may seem at first, and the data that you receive will be invaluable.


Nick Robson is an expert in various aspects of SEO. He is also one of the Content Supervisors for He spends an inordinate amount of time reading about the various advancements in the SEO world, just so he is up to date with everything there is to know about it.