Kinja Data Team

it's A/B test time, my dudes

As of last week, we have started tracking what percentage of readers complete each post.

What we are doing

We're using a really simple definition here; to complete a post, a user has to scroll down on the screen until it has reached the bottom of the post (before the replies) for half a second. This does not include getting to the replies to the initial post (though as of this week, we're tracking that as well).


Why we are doing it

It's no secret that the web analytics industry has moved to engaged time as the number-du-jour to measure how their sites are performing.

That's great! Engaged time is a massive improvement in measuring how a site is doing compared to pageviews, primarily because engaged time is less prone to mis-alignment than pageviews (pageviews are terrible for this: slideshows, anyone?)

Engaged time is a much better proxy of what the user thinks of a site, but it still can be gamed. And I think we're already seeing the first signs of people adjusting strategies to goose this number: using video for content that better suited for text, putting site navigation at the bottom of the page, etc.


'Post completion' is a simple number we're going to use in conjunction with engaged time. Simply: what percentage of users that come to the page scroll to the end of the post? Our design head mentioned that a certain other site was tracking this number, so we felt like it made sense to at least take a look.


Of course, this number is only a proxy as well: we could juice our numbers by writing more posts with comics (this one has 94% completion) or celebrity pictures (90%) or posts with comics AND celebrity pictures (91%).

But used in conjunction with Engaged Time, post completion gives us a very strong signal into whether or not users are getting what they want out of the pages they are seeing: it helps us control for changes that might make the site more confusing and keep people on the site longer than they want, or rambling posts that users get bored with. Avinash Kaushik was way ahead with looking at task completion as a key number, and our primary task is getting readers to read our stories.


How are we doing it (warning: slightly technical)

Implementing the javascript to fire on post completion took a little bit of thought: we knew we wanted to fire the event when the element at the bottom of the post appeared on-screen (technically, when it was less far from the top than the bottom of the reader's screen). Unfortunately, the only event we we can use is the scroll event, which fires way too frequently and would really impede site performance.


One of our front-end developers, Stephen Kao, suggested that we use underscore's throttle method to only check for the bottom of the post every half second or so, and then jQuery to unbind the event as soon as it is fired so we only fire once per (we were already loading both libraries). Under his guidance I put together this example.


We did this in Google Analytics, if you do too, make sure you mark this a 'non-interaction event' or else you'll see big changes to your bounce rate and time on site numbers!

What is it telling us?

So far, we've been getting interesting information from this event. Our network-wide post completion rate is 55-60%, which is much higher than I expected, with our biggest blogs ranging from 50% to almost 80% completion.


I also had a theory that post completion would be a good predictor of whether a visitor would go to another page (ie: a reader that finishes a post is more likely to stay on our sites). So for our 1,000 most popular articles, I looked at the relationship between exit rate and post completion.


While there's a very clear relationship (posts with higher completion rates have lower exit rates) completion rate is a less strong predictor than time-on-site. Here's that graph, for your reference (time is in seconds, of course).


Combining time and post completion rate does have better predictive power (by both goodness of fit and AIC) than either one alone, though I don't know if that information is particularly actionable.

There have actually been a bunch of other advantages to implementing this event: we're starting to test out different modules at the bottom of posts, so we'll be able to use this as a denominator (ie: how many people see the modules) for evaluating those modules. And we've implemented a second scroll event for the very bottom of the page to get a better sense of how many readers are checking out the entire story, including the discussion.

Share This Story

Get our newsletter