Case Study - How Tele2 used AB tests with AMP

When it comes to great mobile experiences, speed matters most – for both customer and brand

Goal

After analysing the results, it became clear that this was the most successful A/B test Tele2 ever did
Roderick Eeninkwinkel Senior E-commerce Manager @ Tele2

Set up an A/B test with AMP to measure the impact faster load times would have on their revenue

The Results

  • +56%

    uplift in click-through rate

A full eight weeks of the A/B test showed 56% uplift in click-through rate, guiding more users quickly to the first steps of their purchase. As a result, the e-commerce conversion rate also benefited, with more customers buying from the faster AMP page.

The quicker load times were apparent instantly, with the AMP page loading 66% faster than the non-AMP version. And after just a few weeks of testing it was clear that these faster load times had a positive impact on conversion rates.

 

Approach

Tele2’s E-Commerce Manager and CRO Specialist teamed up to identify the most popular product page to setup their first test. The developers then built an identical version of the product page in AMP. They performed thorough quality assurance to verify that the two pages looked and functioned identically – this is crucial for setting up a fair test that isolates speed as the only variable. 

At this stage, it was also important to ensure that measurement and analytics were set up properly to allow the CRO team to compare conversions from both pages.

The telco’s digital performance agency iProspect set up an A/B test for this page using Google Ads Drafts & Experiments. To test the impact of overall performance they created a 50-50 traffic split of mobile search towards AMP and non-AMP. 
The result was spectacular. We A/B tested this setup in Google Ads with no additional marketing budget. We expected the AMP page to increase our paid search performance but the increase in conversion rate was above everyone’s expectations.
Lenno de Man Senior Online Marketing Manager @ Tele2

Final

This A/B test clearly demonstrated to Tele2 and iProspect just how much speed can influence conversion rates, and confirms that there is a huge opportunity in the telco industry to improve the user experience by speeding up. Now, the marketing and product teams are expanding the experiment to all traffic sources and investing in further speed improvements.