From time-to-time, in an effort to improve the overall site experience, Gawker runs A/B tests on various features of our blogs. As we examine the results of these tests, we will try to publish the results here, to give some perspective on features we are testing and how we think they improve (or don’t improve) the site.
As Jalopnik and Sploid readers have probably already noticed, we’ve made some changes to the appearance and functionality of the sidebar on the left hand side of the site. For more details on the change, check out the product blog.
In addition to looking good, the goal of the new sidebar is to increase recirculation by showcasing our posts in an attractive way. Of course, we don’t want clicks on the sidebar to detract from the overall site experience, so we also tracked whether users stay on the site for as long: we’d like to keep our readers for longer as a sign we’re delivering them content they find interesting.
We tested this change on Jalopnik, and it ended up sparking a lot of internal discussion. We saw a small (5%) but significant (p=1%) increase in session duration with the new sidebar, and we saw an increase in clicks that was right on the edge of being significant (p=7%). Since it’s generally a bad idea to ‘chase significance’ by extending a test beyond its assigned time, I didn’t seriously consider extending the test another day.
We continue to track the clicks on the sidebar since we went live (on the 7th). It appears that the overall click rate continues to increase, most easily seen when we take the 7 day moving average in order to control for weekly ‘seasonal’ variation. Interestingly, while we’ve been paying much closer attention to the ‘Trending’ module (now the ‘Popular from all blogs’ module), it looks like the increase, on a percentage basis, has been much greater on the ‘popular on this blog’ module.
The reasons for this aren’t totally clear to me- we will continue to monitor.