Digg's Algo Change Cut Promotions by 38%

Update below with more info about the data.

Digg changed their popping algo in January, to much outrage, causing Digg's staff to respond. The question is: was the outrage justified? The short answer is a resounding yes.

To answer this, I dug deep into the Social Alerter data. Because SA has been under development for many months, I did have data from before the January 23rd announcement and data from afterwards. With this data in hand we can ask this: for all upcoming stories that reach a given number of diggs, what is the chance that these stories go on to become popular? That is, if an upcoming story has (say) 25 diggs, how likely is it to be become popular? We ask this from both data sets, the pre-change and post-change data and look for a difference. And boy is there a difference.

Results pre announcement:

• 12603 Upcoming stories with 25 diggs that were promoted
• 46465 Total upcoming stories with 25 diggs
• Chance for an upcoming story to be promoted if it has 25 diggs = 27%

Results after announcement:

• 3957 Upcoming stories with 25 diggs that were promoted
• 22800 Total upcoming stories with 25 diggs
• Chance for an upcoming story to be promoted if it has 25 diggs = 17%

See that? That's a 10-percentage point drop in likelihood of promotion. To rephrase, that's about a 38% drop in chances of promotion. I don't know about you, but that's significant in my book.

So yes folks, the outrage was justified - it did have a significant effect. The question is, was the change worth it for Digg? Did the quality of promoted stories go up? Did it increase the number of users? Only Digg the company and the Digg users can answer those. Personally, I didn't notice any change - positive or negative - since the change. What do you think?

Update:

More info about the data sets. They cover stories in the following time ranges:

• Pre-announcement data start: 08 Oct 2007 23:42:26 (that's the time-stamp of the first story in the data set)
• Pre-announcement data end: 20 Jan 2008 00:00:00 (artificially set by me for the analysis)
• Post-announcement data start: 20 Jan 2008 16:02:03 (time stamp, 3 days before the announcement's date because they talk about "changes we’ve been making this week"
• Post-announcement data end: 09 Mar 2008 22:54:39