Digg Got it 166% Wrong

We already talked about the effect of Digg's January algo change. The summary of that analysis is that the chances of a story getting promoted after getting 25 diggs was cut by 38% after the algo change.

Today we visit the algo change again and, let me cut the story short, the algo change had exactly the opposite of its intended effect. Let's dive in.

Digg's Reasoning

Let's visit Digg's original post again. This quote summarizes the reasoning:

One of the keys to getting a story promoted is diversity in Digging activity.

Basically, Digg was worried that a small group of people were getting their stories promoted too many times. That is, Digg's promotion was not as democratic as it should be. The diversity algo changes were aimed at ensuring that a diverse group of users promotes a story instead of a power user and his/her friends.

The Data

The data is from Social Alerter's database as collected via Digg's API. The data is all popular stories, divided into two groups:

• The Before Data: Data from 20 November 2007 till 20 January 2008. This covers the popular stories before the algo change (which was announced January 23rd).
• The After Data: All popular stories from 28 January 2008 till 29 January 2008.

The gap is meant to flank the algo change to make sure the data is collected before and after the algo's roll out.

The data looked at who submitted the popular stories, and how many popular stories each user submitted. The Before data contained 8673 submissions from 4294 users. The After data contained 5590 submissions from 1810 users.

You can download the full data set. Note that the usernames are encoded so as not to "out" anyone. That's not the point of this analysis.

Analysis

As you can see, the top user in Before and After was the same person, called B1 in the data set. In the Before set, B1 submitted 214 stories, i.e. ~2.5% of all stories. In the After set, B1 submitted ~5% of all stories. This was very surprising and actually holds true throughout the two data sets.

Let's take another metric: the top 10 users. As you can see, seven of the ten top users in the Before set remained in the top 10 in the After set. So much for introducing new diversity.

But here is the bombshell: after the algo change, the top users submitted as many popular stories as they used to. In the Before set, the top ten users submitted ~12.5% of the popular stories. In the After set, the top ten users submitted ~20% of the stories.

Actually, it gets worse: the top 3 users in the Before set submitted ~6% of the popular stories, but submitted ~10.5% of the popular stories in the After set. Also, the first new user that made it into the After rankings, clocked in at rank 21; the first 20 users were already fairly-to-very active Digg users. Pray, do tell me, isn't this the opposite of what Digg wanted to achieve?

Here is another metric: if we compare the first 20 users in the Before and After list, i.e. the league table up to (but excluding) the first new user on the After list, we see another disturbing pattern. For each rank position, the average increase in popular story share is 166%. The first three ranks really broke the records: the first doubled their share of popular stories (i.e. 203% of Before), the second clocked in at 176% of Before, and the third clocked in at 170% of Before.

Conclusions

As you can see, there are many different metrics that point in the same direction: the algo change actually concentrated power into the hands of the top users. This is exactly the opposite of the intention of the algo change.

Coupled with the previous results about the algo change, we have a disturbing conclusion. The top users are able to get their stories promoted better but the chances of promotion are cut. This means that the 'common man', us lowly not power Diggers, are the ones who got hit the worst.

Photo above a crop from brooklyn.