A Guide to Split Testing on Shopify

How do you know if your ecommerce store is performing as well as it could?

You may have ideas for how you want to improve your site – but tread carefully! Each change or new functionality has the potential to impact the revenue that a page generates. And that change could be positive or negative.

Unless you have a robust method for testing in place, you won’t be able to measure the impact of your chosen improvements, or to learn how changes affect your customer behaviour.

Good customer experience leads to conversions. But if you want to truly know whether your customers are getting the best experience, be it on mobile or desktop, you’ll need to run tests.

You can use a split test to trial new changes and compare purchasing behaviour on new versions of your webpage with the original.

This allows you to optimise the performance of your store by only making changes that are proven to improve your conversion rate, or other desired metric.

Want to know more about how to put this into practice? Here’s a quick guide on how to do split testing on your Shopify store.

What is split testing and how does it work?

The most common type of split testing is A/B testing. This can be carried out on a webpage, a mobile app or an advert, to test the performance of a variant, or more than one variant, compared to the original version.

Traffic to the page being tested is split into two, with one group viewing the original version of the page – the control – and the other viewing the variant. Visitors are split at random and are unaware that they are viewing one or the other.

The results from each page – metrics such as conversion rate, bounce rate or revenue per user – can then be measured, to determine which version of the page is the winning variant.

Split testing in action

Let’s say you’re looking to increase revenue per user (RPU) on one of your product pages. Below is an example of a test we ran for one of our clients that aimed at encouraging bundle purchases, in order to increase RPU.

On the product page, we tested two variants which promoted the necessity of a spare cover for the product and allowed the customer to quickly and easily add one to their cart. Visitors to the site were shown either the control or one of the two variants when they visited the page.

Control

Variant 1

Variant 2

Compared to the control, we found the conversion rate increased for Variant 1, followed by Variant 2. RPU was also highest for Variant 1, but average order value (AOV) remained higher with the control. During the test period, sales of spare covers increased by 50%.

Thus, the result of the test was that revenue increased when we implemented Variant 1 of this product page, despite individual metrics varying. This gives us valuable data about the impact of making changes to our client’s website. And when such data (and additional split testing) is applied across their Shopify store, it has the potential to create very significant increases in performance and revenue.

You can see more examples of split testing in this case study about our experimentation program with Loop Earplugs (a program that created a €1M annual turnover growth opportunity!).

Swanky’s customer-first approach to split testing: a step by step guide

So, we know what split testing is and how it works. Next, we’re going to explore Swanky’s customer-first approach to this sort of experimentation.

We think of the split testing process as being made up of four parts:

1. Get to know your customers
2. Hypothesise on what might improve performance
3. Prioritise your tests
4. Analyse your results

Let’s explore each stage in a bit more detail.

1. Get to know your customers

At Swanky, our approach is customer-data led, and strategic, meaning that we understand the barriers to success that customers face on the site, as well as the opportunities that present themselves when we listen to customers telling us what they want.

In order to optimise your web pages, you need to know what it is that drives conversion rates. What experience are customers expecting from your site? What might cause them to abandon their purchase mid-way through?

As a general rule, online customers look for speed and convenience, among other factors. But each industry and brand will have a unique customer base, and understanding the needs and expectations of your customers is key to any conversion rate optimisation (CRO) program.

Using the website analytics on your Shopify Plus store or on Google Analytics, you can build a picture of who your customers are, how they’re arriving at your store, and which pages are converting into sales – or not!

Swanky’s CRO experts then carry out further research using exit intent surveys and heat-mapping to gather insight on customers’ behaviour whilst on your site.

2. Hypothesise

Our next step is to hypothesise which changes to your site might have the biggest impact, using the information we have gathered. Our strategists consider what is driving customers to make purchases and we create strategies to test each hypothesis.

For example, if we see that you have lots of users from Europe but they’re dropping off at the payment page, we may recommend that you add more currency options to your site, to reduce barriers to purchases from abroad.

What’s more, our ability to hypothesise improves over time as we have a greater body of data obtained from previous tests, creating a virtuous cycle of testing, learning and continuous improvement.

3. Prioritise your tests

We then commercially prioritise these changes, addressing the opportunities to generate the largest increase in revenue first.

There are probably hundreds of potential tweaks you could make to your store. Each split test takes time and investment though, so it’s important to prioritise elements that will have the greatest commercial impact.

For example, if the majority of your customers are mobile-users, your first priority will be to ensure that your site is fully optimised for mobile, to offer a seamless user experience to these customers. 

4. Analyse your results

We document and record everything, and we tell a story with each test, so we can continually remind ourselves of what, how and why we are testing, and of course the outcomes. It is our diligent process that facilitates learning and growth for ourselves and for our clients.

Since there are multiple metrics involved in each test, results may not be as simple as finding that one variant is a clear winner.

Let’s say that in the variant you have added product suggestions to a product page. Perhaps this will actually have a negative impact on conversion rate but the revenue per user will be significantly higher for those who do convert. The overall outcome may still be that the variant sees a revenue increase overall.

Each test you run will inform further improvements to your site. Even if a variant proves less successful than the control, this will still give you insight into your user behaviour.

Kameleoon: Split testing tool for Shopify

In order to carry out a split test on a website, Swanky uses an A/B testing tool provided by one of our technology partners. Which platform you use will depend on the size of your company and what you’re looking to achieve.

One of our go-to platforms is Kameleoon, an A/B testing platform for mid-sized to enterprise companies with existing optimisation or personalisation programs. Its features allow online retailers to build better customer experiences that boost conversions and turn visitors into loyal shoppers.

We asked Fred de Todaro, Chief Product Officer at Kameleoon, for advice on how to enhance your results by segmenting your audience within tests.

“With features such as baked-in KPIs and segments, you can not only test multiple variants of a webpage, but also segment the audience within the test. This allows us to compare how different users respond to different variants.

For example, we may see that a change in page layout boosts conversion rate among mobile users, but has little effect on desktop users.

This segmentation also facilitates personalisation of the store. It means you can serve each customer with a bespoke experience according to their location, device and so on.

Furthermore, teams who want to scale their experimentation can use predictive targeting.

Kameleoon has machine learning capabilities to predict intent and increase conversions through carefully timed messaging and experiences that respond to user behaviour. Programs can experiment with price sensitivity, the types of pages users visit, the colour or size of a product, and more.”

Split testing with Swanky

Swanky has a tried and tested methodology that facilitates learning and delivers results for our clients, based on data.

Our team is multidisciplinary, which means that they understand more than just split testing. They understand the impact and influence that changes across the wider customer journey have. They can therefore consider these in their hypotheses, but, more importantly, they can feed learnings from CRO into wider business strategy.

We take the time to understand our merchants and their customers, which then informs our CRO strategy. And to Swanky, CRO is more than just on-site, we know that minor changes to marketing strategy or business operations can have an impact on top-line revenue.

We also bring in the expertise of our in-house UX team and developers to ensure that we’re not only considering how to increase conversion rate, but also elevate the brand experience throughout our work.

The CRO agency for you

Our team of experienced CRO experts are all passionate about finding and analysing commercial data to inspire creative and innovative ways to drive revenue for our ecommerce clients. That’s why we were voted as the Conversion Rate Optimisation Agency of the Year at the 2021 Digital Growth Awards!

To learn more about our CRO services and how we could help unlock your ecommerce potential, contact our team of Shopify Plus Experts today.