Twitter - Open Sourced Recommendation Algorithm

[wanderingfool2]

At noon (Pacific) today, Twitter open sourced the recommendation algorithm of Twitter.  This has been hinted at for the last few months, but today it's finally released.  There are portions of the algorithm which remain still closed source but will apparently be added as time goes by.  There is a lot of code and documentation now to dive through

 

Twitter takes its algorithm ‘open-source,’ as Elon Musk promised - The Verge

Quote

Twitter has released the code that chooses which tweets show up on your timeline to GitHub and has put out a blog post explaining the decision. It breaks down what the algorithm looks at when determining which tweets to feature in the For You timeline and how it ranks and filters them.

According to Twitter’s blog post, “the recommendation pipeline is made up of three main stages.” First, it gathers “the best Tweets from different recommendation sources,” then it ranks those tweets with “a machine learning model.” Lastly, it filters out tweets from people you’ve blocked, tweets you’ve already seen, or tweets that are not safe for work, before putting them on your timeline.

GitHub - twitter/the-algorithm: Source code for Twitter's Recommendation Algorithm

 

It will be interesting to see what comes about this, at the very least it adds a bit more transparency in regards to the algorithm.  I'm sure as people pick it apart they will find tid-bits of information and in general be able to make a more neutral algorithm.  Twitter is apparently building towards accepting pull requests from users as well, this sort of reminds me of when Google made Chromium and open-sourced it though.

 

Time will tell to see what happens with the source code though, and whether or not 3rd party input will be accepted into the algorithm.

[OhYou_]

They should rewrite the entire codebase in Rust

[williamcll]

Note the Elon priority as well as US politician Tweets: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b928b479b512ec51ac2c3821f5922/home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeTweetTypePredicates.scala#L225

 

(
	"author_is_elon",
	candidate =>
	candidate
	.getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))),
	(
	"author_is_power_user",
	candidate =>
	candidate
	.getOrElse(AuthorIdFeature, None)
	.exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)),
	(
	"author_is_democrat",
	candidate =>
	candidate
	.getOrElse(AuthorIdFeature, None)
	.exists(candidate.getOrElse(DDGStatsDemocratsFeature, Set.empty[Long]).contains)),
	(
	"author_is_republican",
	candidate =>
	candidate
	.getOrElse(AuthorIdFeature, None)
	.exists(candidate.getOrElse(DDGStatsRepublicansFeature, Set.empty[Long]).contains)),
	)

[LAwLz]

4 hours ago, williamcll said:

Note the Elon priority as well as US politician Tweets: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b928b479b512ec51ac2c3821f5922/home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/decorator/HomeTweetTypePredicates.scala#L225

What does that code do exactly? 

[05032-Mendicant-Bias]

An Ars Tecnica article reports that that piece of code might be collection of metrics:
Twitter posts the code it claims determines which tweets people see, and why | Ars Technica

[Doobeedoo]

The actual algorithm is degeneracy, toxicity, depravity and filth aka people.

Anyway, next maybe FB or better yet YT that be funny.

[LAwLz]

4 hours ago, 05032-Mendicant-Bias said:

An Ars Tecnica article reports that that piece of code might be collection of metrics:
Twitter posts the code it claims determines which tweets people see, and why | Ars Technica

I see.

So it seems like this code is for collecting metrics and not actually a "visibility boost", yet people automatically assume it is because it would confirm their preconceived notions and hopes. Some piece of code mentions Elon? Gotta be evidence for my conspiracy theory I believe in! Who cares what the code actually does, it probably just does what I assume it does!

[wanderingfool2]

12 hours ago, williamcll said:

Note the Elon priority as well as US politician

Based on?  The variable literally has the word stats in the name which implies its used for tracking.

 

Then looking at the code base and where the variable is used you get this fine comment right above that section
https://github.com/twitter/the-algorithm/blob/7f90d0ca342b928b479b512ec51ac2c3821f5922/home-mixer/server/src/main/scala/com/twitter/home_mixer/functional_component/feature_hydrator/RequestQueryFeatureHydrator.scala#L86

Quote

      /**
       * These author ID lists are used purely for metrics collection. We track how often we are
       * serving Tweets from these authors and how often their tweets are being impressed by users.
       * This helps us validate in our A/B experimentation platform that we do not ship changes
       * that negatively impacts one group over others.
       */

 

Congratulations, that code snippet you posted relates to metrics collection.

[Needfuldoer]

Yup. That's code, alright.

[Kisai]

21 hours ago, wanderingfool2 said:

At noon (Pacific) today, Twitter open sourced the recommendation algorithm of Twitter.  This has been hinted at for the last few months, but today it's finally released.  There are portions of the algorithm which remain still closed source but will apparently be added as time goes by.  There is a lot of code and documentation now to dive through

 

Twitter takes its algorithm ‘open-source,’ as Elon Musk promised - The Verge

GitHub - twitter/the-algorithm: Source code for Twitter's Recommendation Algorithm

 

It will be interesting to see what comes about this, at the very least it adds a bit more transparency in regards to the algorithm.  I'm sure as people pick it apart they will find tid-bits of information and in general be able to make a more neutral algorithm.  Twitter is apparently building towards accepting pull requests from users as well, this sort of reminds me of when Google made Chromium and open-sourced it though.

 

Time will tell to see what happens with the source code though, and whether or not 3rd party input will be accepted into the algorithm.

Art Twitter has been linking to this thread:

 

Let's be honest, the algorithm punishes external content, and only promotes what is hosted on it.

 

Quote

 

• Each like gets a 30x boost

• Each retweet a 20x

• Each reply only 1x

 

 

Quote

 

Links hurt, unless you have enough engagement

Generally external links get you marked as spam.

Unless you have enough engagement.

 

 

This one is probably the one that will hurt the most:

 

Quote

 

• Mutes

• Blocks

• Unfollows

• Spam reports

• Abuse reports

 

So that means, that if you are controversial in the slightest, your discovery is getting destroyed.

[Arika]

9 hours ago, LAwLz said:

I see.

So it seems like this code is for collecting metrics and not actually a "visibility boost", yet people automatically assume it is because it would confirm their preconceived notions and hopes. Some piece of code mentions Elon? Gotta be evidence for my conspiracy theory I believe in! Who cares what the code actually does, it probably just does what I assume it does!

Why even make code open source if no one is going to read it....

[Remixt]

On 3/31/2023 at 4:43 PM, OhYou_ said:

They should rewrite the entire codebase in Rust

Not sure if joking, but I un-ironically agree.