Netflix make tool for data collection opensource

[joppetie]

Netflix has made its internal tool for the collection of data opensource.

The tool can be used to collect large amounts of server logs and send them to a central server.

 

The tool, named Suro, can handle different types of data. Suro is used by Netflix to collect the 1.5 million "server-events" a second.

We're talking about server logs, user activity or other operational data. These are collected by Suro so they can be sent to several destinations,

like a Hadoop-cluster or a ElasticSearch-cluster, which makes it possible to see trends like malfunction can be detected.

 

Since today the tool will be opensource, Netflix announces. Suro itself is actually based on an opensource-tool: 

It's an altered and tuned verion on Chukwa, a tool made by the Apache Foundation for data collection. 

According to Netflix, one Sure-instance is able to process 60.000 messages a second.

 

Why Netflix make the tool opensource is onclear. It's happened before that a company has made its tools opensource,

like Facbook did with its php-accelerator and database-software Cassandra.

 

 

 

Source: http://tweakers.net/nieuws/93085/netflix-maakt-tool-voor-verzamelen-data-opensource.html (Be warned, it's in Dutch)

[miyabwah]

I'm going to stop using Netflix, now.

[ionbasa]

I'm going to stop using Netflix, now.

How do you think they compiled that list of "recommended" movies you should watch?

[bob345]

Now I'm glad that i stopped using nsaflix

[EricX2]

I think people are confused on what has happened. They are not open sourcing your data, they are open sourcing the tool that can collect data. So that other people can use the tool to collect OTHER data, not to collect netflix usage data. They just use it currently for that feature, but it could be used for any website to collect any data they want. The main thing is that it scales very high.

 

This doesn't mean I can install Suro on my PC and netflix data will start showing up for people... absolutely nothing close to that.

 

http://gigaom.com/2013/12/09/netflix-open-sources-its-data-traffic-cop-suro/

http://techblog.netflix.com/2013/12/announcing-suro-backbone-of-netflixs.html

 

Also, 'collect data' doesn't necessarily mean collect private data. Usage data can be used to manage servers to make them work better.

Edited December 11, 2013 by EricX2

[bob345]

I think people are confused on what has happened. They are not open sourcing your data, they are open sourcing the tool that can collect data. So that other people can use the tool to collect OTHER data, not to collect netflix usage data. They just use it currently for that feature, but it could be used for any website to collect any data they want. The main thing is that it scales very high.

 

This doesn't mean I can install Suro on my PC and netflix data will start showing up for people... absolutely nothing close to that.

 

http://gigaom.com/2013/12/09/netflix-open-sources-its-data-traffic-cop-suro/

http://techblog.netflix.com/2013/12/announcing-suro-backbone-of-netflixs.html

well, I'm sure you know who now has access to this tool now.