Kinja Data Team

it's A/B test time, my dudes

I'm a big believer in publishing negative results. There's a lot of time to be saved by sharing theories that didn't pan out: maybe more than talking about ones that did.

Last week, one of our editors came to me with concerns about how traffic was being tracked.


While there are any number of reasons that all our tracking tags might be wrong together, I was asked to see whether there was any difference between our three main external analytics platforms: Chartbeat, Google Analytics, and Quantcast. I'd been meaning to compare their numbers since I know there are big differences in time-on-site numbers, and this seemed like a great opportunity.

We've written API clients for Chartbeat and GA already, so it was fairly easy to pull the information. (For Chartbeat, to get this data you'd need to use their excellent report builder).

Anyways, as we would hope, there's no real difference in the number of pageviews or the number of uniques (in GA parlance, users) each one tracks.


I calculated the average standard deviation just to quickly get a sense of how variable the numbers were: I saw SD's of about 3% for pageviews and 4% for users/uniques.


The eagle eyed among you might notice that there's a bigger difference on Mondays than other days. Here's a graph of the standard deviation by day, with Mondays marked.


I think this is mostly due to time zone issues: since Sunday night traffic is so low compared to Monday, depending on when you switch days, you get very different traffic numbers. After doing this analysis I looked back and realized we were using local time for GA but UTC for Chartbeat, so at least that bug-fix came out of this analysis.

[photo credit]

Share This Story

Get our newsletter