This week, we got A/B testing working on our AMP pages. While we really like AMP, we had more trouble than I expected due to some rough edges and components of the AMP documentation that I think are slightly misleading, so I figured I would write down my experiences here.

This post is slightly technical: I’m not going to go into what AMP is or what A/B testing is: I’m assuming if you’re still here, it’s because you know what these are and you want to get started with testing, but you’re having problems with it.

Advertisement

We use Google Experiments (the Google Analytics A/B testing framework) when running tests on AMP. We’re very familiar with the tool and it offers convenient reporting that we work in regularly and can access via API.

Additionally, the amp-experiment documentation and examples use Google Experiments, so we thought that it would be conducive to quick implementation. Unfortunately, we had to work around a few points of complexity.

Serving Variants

Here’s what our amp-experiment element ended up looking like:

Some notes:

  • We named the experiment the same as the Google Analytics Experiment ID: Technically, it should be possible to map an experiment name to a random string (like an ID). In practice, the restrictions on running JavaScript on AMP made this more work than we were willing to work around. Note that this is different from most examples of AMP, where experiment names are human-readable.
  • The variants are integers (well, integers cast to strings): Similar to above, we’re passing the variant ID to Google Analytics, and sine GA needs a 0-indexed integer for the variant, we have obliged.

This led to the following CSS code binding the visual treatment to the experiment variants:

Just like in Google’s documentation, we’re putting the experiment name and the variant in the CSS: this is just much less legible than the documentation examples, unfortunately.

Advertisement

Advertisement

This is enough to get variants served. We still have to send GA the data though.

Collecting the Data

AMP supports a number of different methods for sending data, including both amp-analytics and amp-pixel elements. We wanted to pass the experiment version along with the pageview, though, so we need to include it in the pageview event, which goes through amp-analytics for us (and we assume most other publishers’ AMP pages).

Even within amp-analytics, there are a number of different configurations that we can use. We’ve had success with using the extraUrlParams element, giving us an amp-analytics element that looks like this:

the X’s for “xid” and “xvar” refer to the ID of the experiment. For “account”, that’s a Google Analytics account number.

Getting this to work with our templating (we use Closure Templates) took a little arm-wrestling, but at the end of this work, we have a fully functioning experiment in Google Analytics.

Advertisement

Advertisement

Google Analytics also gives us a chance to finally put human-readable names on our experiment and variants in its UI: just be careful to order them the same as in your code.

Plug this in and start your test, and in a few hours you should have comprehensible A/B testing data in your GA account!