The Best A/B Testing Tools For Shopify: How Swanky Achieves Top Results

As we covered in our article on how A/B testing works, finding out exactly what makes your unique customers want to click and convert is a fundamental part of any good conversion rate optimisation (CRO) strategy. 

But to ensure that your results are reliable and that your testing projects are worth your investment, it’s also vital to ensure that you (or your CRO agency) are picking the best tools for the job. 

As Shopify Plus Experts and growth consultants, our CRO team at Swanky have spent years looking for the perfect tools to help us boost our clients’ conversion rates. 

Unhappy with the extortionate prices or lack of flexibility offered by a lot of the most widely advertised A/B testing giants, we began searching for the tools which would give our clients the most accurate test results, the most flexibility and, ultimately, the most value for money. 

Of all the tools we’ve reviewed, our favourites by far have been Convert and Dynamic Yield. 

Mixing affordability with the most advanced testing and personalisation features on the market, both of these tools are now our technology partners – with the one we choose to use for each CRO project depending completely on our client’s budget. 

Today, we want to break down exactly why we continually choose Convert and Dynamic Yield over some of the most recognisable names in split testing – and why we think you should too. 

Which A/B testing tools do Swanky recommend?

If you’re searching for a testing tool that has both an impressive feature list and a reasonable price tag, we’d like to introduce you to two fantastic tools; Convert and Dynamic Yield. 

For Shopify stores and small businesses

If you’re a startup or a smaller business looking to boost your conversion rate significantly without breaking the bank, we believe Convert is the testing tool for you. 

Convert is extremely affordable, but comes with a host of features that will put you miles ahead of businesses using other testing tools for a similar price. With Convert, you can start running advanced experiments that will help you make smart, data-driven decisions about every aspect of your store’s layout, copy and structural design. 

So what features does Convert have that other popular, affordable tools like Google Optimize do not? Firstly, Convert won’t restrict you on the number of tests – or variants – you can run at any time. With Convert, you can run an uncapped number of tests on any paid project, and you’ll also be able to run unlimited multi-page experiments with an unlimited number of variations within them. 

Convert also allows you to launch personalisation campaigns, matching specific pages to specific audiences with its advanced geo-targeting features. 

Another bonus of Convert is that they run their testing tool on a super-fast CDN (content delivery network), which we see as a huge advantage considering how much the speed of a tool’s CDN can impact a test’s results. 

We especially love how easy Convert is to install, set up, use and integrate with both Shopify stores and other ecommerce tools (such as Hotjar, HubSpot and Google Analytics). 

In fact, the only real limitation we can see with this testing tool is that you can’t use custom code in your experiments. However, at Swanky, we’ve found a getaround which allows us to use custom code for unique, complex tests that are specific to the clients we work with. 

So while Convert itself is an incredibly advanced and useful tool for an affordable price, when you hire the right CRO agency for the job, you unlock this tool’s full potential for helping you lift your conversions on every page of your Shopify store. 

Let’s take a look at the impressive results Convert has helped us generate for one of our current clients: 

A Convert success story

For contemporary women’s clothing brand Pulse Boutique, we wanted to test whether adding icons to the buttons on their mobile menu would have an effect on their store’s conversion rate. 

Using Convert to run this relatively complex and structurally based test, we were able to ascertain at 99% confidence that switching to an icon-based menu would lead to a staggering 25% increase in RPU (revenue per user) for the Shopify brand. 

After Swanky helped them implement this change to their site, Pulse Boutique’s conversion rate shot up by an impressive 14% in just the first 14 days – with our data highlighting that their customers were adding more products to their cart than ever before. 

This is just one shining example of how Convert can be used to run complex, unique and structural tests to deliver data that helps stores make smarter decisions, sell more and grow. 

For Shopify Plus stores and enterprise businesses

If you’re managing a large or enterprise-sized business and are looking to invest in the most powerful and reliable testing tool on the market, we can’t recommend Dynamic Yield enough.

This powerful personalisation and testing tool is quickly gaining popularity with industry leaders such as Hallmark, Hello Fresh, PacSun and Urban Outfitters. It’s not surprising when you consider its usability, flexibility and the freedom it gives you to optimise every aspect of every business channel for conversions. 

Dynamic Yield is first and foremost a strong personalisation tool, making it easy for you to personalise all aspects of your customers’ journeys using AI-powered predictive targeting. 

You can also customise entire user journeys – rather than single store pages – using Dynamic Yield, and run an unlimited number of experiments simultaneously with an unlimited number of variants. All of this makes Dynamic Yield a fantastic tool for enterprises striving for constant improvement and growth in every area of their business. 

“We are running hundreds of personalization and optimization campaigns at any given time.”

Benjamin Ludigs, CRO at Lamoda

This tool – unlike many of its competitors – also gives you the option to run tests on any channel – including mobile apps, TV apps, loyalty programs and email campaigns – rather than just your website. If you’re a large and growing business, this is a really important feature as it allows you to maximise the lifetime value of your customers not only on your Shopify store but on all of your business channels. 

Another thing which really sets Dynamic Yield apart from its competitors is the ability it gives you to run dynamic tests on your store. Dynamic testing means that as one of your variations in a live test starts to obviously outperform the others, more of your traffic will be automatically driven to that version of the page. The result? More revenue – and more time saved – for you. 

Like Convert, Dynamic Yield also integrates seamlessly with Shopify stores, heatmapping tools and apps like Google Analytics. 

Essentially, Dynamic Yield can deliver all the results that ultra-expensive tools like Optimizely do – and then some – at a fraction of the price. 

Another important advantage of choosing Dynamic Yield over other testing tools on the market is the fact that it uses the Bayesian approach to A/B testing rather than the Frequentist method that most of its competitors use. 

What does this mean? In simple terms, the Frequentist method compares the performance of A and B over one set time period, while tools using the Bayesian method can compare their performance not only during the test’s time frame but also prior to the test (as they have already been tracking your control’s conversions). Tools using the Bayesian method therefore have more confidence in the average performance of the variants versus the control, as they have a larger data and time sample to work from.

To illustrate the difference between the two methods:

The Frequentist method says: We don’t have any prior goal conversion data, so A and B will be tested over the same traffic and time period, meaning we’ll require more time and data to validate our conclusion. We’ll potentially need up to 35,000 sessions to come up with a reliable result. 

The Bayesian method says: As A has converted at 3% for the last 250,000 sessions, we know what its conversion rate is typically like. We therefore only need to test it against B for 25,000 sessions to validate that B’s conversion rate of 3.5% is indeed the winner. 

But never mind the technicalities. The important thing to understand as an ecommerce manager is that the Bayesian method essentially makes the testing process quicker, simpler and more reliable for you. Because Dynamic Yield uses the Bayesian method, it doesn’t have to test for as long as it already knows how the test’s control will perform. 

Overall, we rate Dynamic Yield as the best on the market for larger businesses as it has all of the powerful and impressive features that its main competitors do (plus a couple of extra bonuses), and all for a reasonable price. 

A Dynamic Yield success story

When we began our CRO project with popular surf and adventure apparel company Saltrock, the growing Shopify business were having trouble with their overly compact list-based mobile menu, which was providing a poor UX (user experience) to their mobile shoppers. 

At Swanky, we hypothesised that switching to a button-based menu would allow customers to more easily be able to click on menu items, resulting in more menu use, increased collection page landings and ultimately a higher conversion rate.

So, our design team set to work creating various button-based alternatives, and then used Dynamic Yield to test them against the control (Saltrock’s original list-based mobile menu). 

Thanks to Dynamic Yield’s accuracy, flexibility and incredible ease of use, we were able to confidently determine that the variant which made use of imagery in the button display led to an incredible 12% increase in Saltrock’s conversion rate over the three-week testing period. 

That’s a pretty impressive outcome for such a seemingly small change – and it’s part of why we trust Dynamic Yield to deliver us insights that help us boost our clients’ sales so significantly time and time again. 

Which A/B testing tools have businesses traditionally used?

Despite the power and value of tools like Convert and Dynamic Yield, many businesses still opt for more widely advertised testing tools such as Google Optimize and Optimizely. Let’s take a closer look at why businesses are choosing to rely on them for testing – and how they’re letting these businesses down. 

Smaller businesses

Typically, the go-to testing tool for smaller and medium-sized businesses in the last decade has been Google Optimize. This is largely due to the fact that its most basic version is free and it allows ecommerce owners to get their basic testing projects done quickly and easily without any coding knowledge at all. 

Being a Google product, this tool’s main advantage (besides the fact that it’s free!) is its easy integration with all of your Google Analytics and your Google Ads campaigns, allowing you to have targeted audiences while you’re running tests.

Medium businesses

Optimizely is frequently opted for by large to enterprise-level businesses with a much bigger budget – because this testing tool does not come cheap! 

The tool’s exact pricing will vary depending on your company’s needs, but even getting on Optimizely’s most basic plan will put you back at least $36,000 USD a year. 

So what features do you get for this higher price? Optimizely has much more advanced personalisation options than many tools on the market and allows you to customise your tests a lot more, giving you control over how many people see your experiments and letting you measure your results by segment. 

It’s also been praised for its usability, simple metrics and easy integration with other analytics tools.

But is it really necessary to spend this much money to test on and personalise your store effectively? As we’ve seen, Optimizely isn’t the only testing tool with this array of features – and that neither Optimizely or Google Optimize are without their failings. 

What are the shortcomings of these traditional testing tools?

We know that asking a free tool to be perfect is a bit of a stretch, but with so many small companies relying on Google Optimize for all of their split testing needs, we think it’s worth starting by touching on some of its limitations that are likely to prevent businesses from growing along the line.

In general, Google Optimize is quite a limited tool as it has been created for people who have never tested before. 

This means you’re automatically limited in terms of the number of experiments you can run simultaneously (which is capped at three) as well as the number of goals you can set and the amount of variants you can have per test. This free tool also offers no on-page analytics to help you understand why your variants are performing better in each test. 

Another important setback of Google Optimize compared to Convert is that you can’t run mobile menu tests on it. With sales from mobile devices recently overtaking desktop traffic, savvy ecommerce owners will know that optimising their mobile experience for conversions should be a top priority in terms of their A/B testing strategy. Unfortunately, with Google Optimize, it can’t be. 

Google Optimize also doesn’t offer real-time reporting, which prevents ecommerce managers from actively tracking their variants’ performance day-to-day. This can be particularly frustrating for high-traffic stores looking to make timely decisions based on their experiments. 

Further, while the free tool’s visual editor was created to help ecommerce managers make changes to their existing set-up, it is quite painful to develop custom experiences on. 

Lastly, it’s worth mentioning that the tool only works for business managers operating their stores on Google Chrome. 

So although Google Optimize may be free of charge, making it handy for startups who want to start dabbling in A/B testing without committing too many of their resources to each project, it lacks a lot of the complexity you’ll likely want as a small to medium-sized business looking to invest in a strong CRO strategy. 

Notably, most of Google Optimize’s basic plan’s limitations disappear when you sign up to their enterprise-level plan Google Optimize 360, but as this will set you back at least $1000 USD a month, we’re certain there are better value options out there for smaller businesses. 

On the other hand, Optimizely does offer a lot more important advanced features, but, as we touched on earlier, it’s hugely overpriced for what most of our clients need. In our experience, you can get the same results with a more affordable enterprise-level tool like Dynamic Yield. 

Optimizely also lacks a lot of the flexibility you’d expect from such a highly priced tool. For example, it isn’t well-equipped for testing elements of your webpage’s structure; it only works well for testing your copy. We find this to be a pretty major disadvantage, as we’ve seen huge conversion increases by running tests on structural elements of Shopify stores (e.g. the layout of a mobile menu) using other more flexible testing tools. 

In short, we believe you can get more for your money by choosing a tool that side-steps these issues and gives you greater value for a lower price. 

Time to boost your conversion rate!

When it comes to making a smart CRO investment, the incredible results that we’ve seen while using Dynamic Yield and Convert over more widely advertised testing tools are proof of the importance of thoroughly researching your options. 

Free tools like Google Optimize may sound appealing initially, but they lack a lot of the statistical accuracy, customisation and scope that our preferred tools have got down pat. 

On the other hand, whilst more familiar, advanced tools like Optimizely may seem like attractive choices because of their huge range of features, you’re likely to find all of the same functionality for a much better price when you invest in a tool like Dynamic Yield instead. 

In order to get the most out of your CRO project, make sure you really look into what’s out there – and consult a CRO agency you know you can trust – rather than simply committing to the tool you’ve seen popping up most often on Google. 

This way, you’ll find the testing tool which will give you the most value for your money – and be best suited to your business’ unique needs, budget and goals. Once you find it, we promise you’ll never look back!

As experts in CRO, A/B testing and Shopify Plus, our team at Swanky are proud to have achieved some incredible results for businesses of all shapes and sizes. For more information about the CRO services we offer, please get in touch with our friendly team of ecommerce experts today. We’d love to hear from you!