With the launch of the new Kinja Dashboard (if you have a Kinja account, check it out here) comes a bunch of design decisions, and these decisions give us lots of opportunities to A/B test different features.
One question the team debated internally was whether or not to open links in a new browser tab. The argument for opening links in a new tab is that the dashboard is ideally your primary ‘feed’ on Kinja, and so you should keep it open at all times. On the other hand, opening links in the same tab (the traditional way to link) might give a smoother experience. We agreed to test it to figure out what gave the best reader experience.
We agreed that we would use session duration as a measure of how engaged readers were in each set-up. So the variant with the longer average session duration would win. We also tracked pageviews per session and unique pageviews per session in order to estimate how much of the difference was due to re-loading the dashboard in the ‘same tab’ variant.
This was our first test run using Optimizely integrated into Google Analytics. Optimizely is a great A/B testing tool, but it lacks any concept of session length or engaged time. Fortunately, they support an integration with Google Analytics, so we could take advantage of GA’s ability to track session length.
We found that opening links in the ‘same tab’ led to longer sessions by about 9%.
I’m always concerned about Simpson’s Paradox (unseen sub-populations biasing test results), so I took a deeper look into the numbers. One major split in our readers is between mobile, tablet, and desktop, so I wanted to drill down into this data to make sure it wasn’t masking other issues in our results. It turns out it was!
(A quick aside: session duration for readers who visit the dashboard is 4-6 times session duration for our average readers overall. The dashboard is for our most loyal readers and contributors!)
It turns out that the entire difference in session duration is due to readers on their phones. The usage on desktop was not significantly different either way (we did not have enough data on tablets to break them out on their own: previous research indicates we should treat them like desktop readers, though it doesn’t make a meaningful difference to the results).
Readers on phones, however, showed a marked preference for links opening on the same tab. This difference is significant, though there were only a few hundred sessions included in this data set.
This made intuitive sense (I know I dislike it when mobile links open in a new window). One concern, though, was that this difference was due to readers re-loading the dashboard over and over, not because they liked the experience more. I tried to isolate this by looking at the number and percent of pageviews that were unique. If I had seen a big drop in unique pageviews in the ‘same tab’ case, I would be worried that the increase in time was due to page re-loading time, not engagement. I didn’t see a meaningful difference, though, so I believe this number is a fair proxy for engagement.
Since this test was run on a small group of readers and on a brand-new page that readers are adjusting to, we will likely re-run this test in a few months to confirm the results.