This week, we got A/B testing working on our AMP pages. While we really like AMP, we had more trouble than I expected due to some rough edges and components of the AMP documentation that I think are slightly misleading, so I figured I would write down my experiences here.
This post is slightly technical: I’m not going to go into what AMP is or what A/B testing is: I’m assuming if you’re still here, it’s because you know what these are and you want to get started with testing, but you’re having problems with it.
We use Google Experiments (the Google Analytics A/B testing framework) when running tests on AMP. We’re very familiar with the tool and it offers convenient reporting that we work in regularly and can access via API.
Additionally, the amp-experiment documentation and examples use Google Experiments, so we thought that it would be conducive to quick implementation. Unfortunately, we had to work around a few points of complexity.
Here’s what our amp-experiment element ended up looking like:
- The variants are integers (well, integers cast to strings): Similar to above, we’re passing the variant ID to Google Analytics, and sine GA needs a 0-indexed integer for the variant, we have obliged.
This led to the following CSS code binding the visual treatment to the experiment variants:
Just like in Google’s documentation, we’re putting the experiment name and the variant in the CSS: this is just much less legible than the documentation examples, unfortunately.
This is enough to get variants served. We still have to send GA the data though.
Collecting the Data
AMP supports a number of different methods for sending data, including both
amp-analytics for us (and we assume most other publishers’ AMP pages).
Even within amp-analytics, there are a number of different configurations that we can use. We’ve had success with using the
extraUrlParams element, giving us an
amp-analytics element that looks like this:
Getting this to work with our templating (we use Closure Templates) took a little arm-wrestling, but at the end of this work, we have a fully functioning experiment in Google Analytics.
Google Analytics also gives us a chance to finally put human-readable names on our experiment and variants in its UI: just be careful to order them the same as in your code.
Plug this in and start your test, and in a few hours you should have comprehensible A/B testing data in your GA account!