Any info on how the guys are building the new streaming infrastructure

[Walms]

Just finished watching the latest wan show. I can't believe they've put together an alpha streaming service in 3 weeks. Anyone know what they are doing and how they did it so quick?

[tomoki]

I think they did say they're releasing details later on down the road so ...  yeah. As for any guesses, nope. I have no clue about coding for HTML 5. 

[leonfagan71]

It could be interesting, I watched the WAN show a few hours ago, Luke did mention that he would be giving us more information about it in the future, for now... we wait.

[Walms]

Yeah, I'm really keen to see how they did it. I'm a reasonably experienced software developer and if that's a custom infrastructure in 3 weeks I'm just blown away. If I were to quote something like that I'd image the time to market would be close to 9 months.

[GDRRiley]

2 minutes ago, Walms said:

Yeah, I'm really keen to see how they did it. I'm a reasonably experienced software developer and if that's a custom infrastructure in 3 weeks I'm just blown away. If I were to quote something like that I'd image the time to market would be close to 9 months.

it is in beta right now but also remember that it was 3 different people doing it not just 1 person 

[leonfagan71]

Just now, Walms said:

Yeah, I'm really keen to see how they did it. I'm a reasonably experienced software developer and if that's a custom infrastructure in 3 weeks I'm just blown away. If I were to quote something like that I'd image the time to market would be close to 9 months.

It depends. 

 

I am a experienced developer too, I think what they've done is fast, bearing in mind that one of them is still in school...
It is possible though, getting a basic working structure of video streaming and a simple viewer can take about a week and a half. Integrating it into the forums is easier than making the database from scratch so will remove a lot of time from the process. (as well as not having to do the entire design again)

 

The servers must be in some sort of clustered network to handle the load of potentially 1000's concurrent users.

 

Linus mentioned having an RSS feed to be notified on new releases, simple stuff like that will come later.

 

I think the hardest parts will be integrating ads and allowing other content creators add their own videos.

For that, I think that they'd need to have their own website and probably look into making some sort of basic app.

[Walms]

Yeah you're right.
There is a lot I don't know about the requirements and how they're doing. That's kinda why I asked the question. Would be really interesting to know exactly what they've managed to do in such a sort space of time.

[leonfagan71]

5 minutes ago, Walms said:

Yeah you're right.
There is a lot I don't know about the requirements and how they're doing. That's kinda why I asked the question. Would be really interesting to know exactly what they've managed to do in such a sort space of time.

Indeed.
I don't see the complexity other than the video streaming servers cluster or perhaps if they're building the video player from scratch...

[Walms]

5 minutes ago, leonfagan71 said:

Indeed.
I don't see the complexity other than the video streaming servers cluster.

 

Yeah, I think I over estimated the task.

If they are using an off the shelf streaming server, they could just use cloud front and then the job is just a little integration into the forums.

TBH I'd assumed they'd built their own streaming/trans-coding server, but I'm guessing that's a well solved problem.

[leonfagan71]

14 minutes ago, Walms said:

Yeah, I think I over estimated the task.

If they are using an off the shelf streaming server, they could just use cloud front and then the job is just a little integration into the forums.

TBH I'd assumed they'd built their own streaming/trans-coding server, but I'm guessing that's a well solved problem.

Yeah, I think you did  

 

I'd assume they're using an off the shelf program.

The hardest bit is load balancing across multiple servers.

 

I used a product before that essentially did this (Kemp Load Balancer), it would bind an IP address and pointed it to a specific server. 

That's really what's required here (unless they want to sync cache and session data/tokens between servers.)

 

A few scripts to check the current forum user and generate a token for the user and the video.

Done, load a JS player or perhaps a HTML5 video player to stream an MP4 or something.

 

I don't think LinusTechTips have enough time to be building their own software, especially in a cluster environment.

 

[mariushm]

No need for reinvent the wheel by building clusters of servers and using load balancers, especially at the beginning when you have few subscribers.

You could just push the videos to a CDN , something like Amazon Cloudfront, Limelight , Maxcdn ..

 

You get as low as around 6-8 cents per GB of traffic, so if a user pays 3$ a month, even with 30 x 500-800 MB videos in a month (probably typical for 720p content), you'd still only pay about 1..1.5$ a month per user in CDN prices.

Not all subscribers will watch absolutely every video posted, some will watch several times, it all averages out in the end.

 

If you want to allow viewers to download videos, you can basically rent a dedicated server in a large datacenter and let users download the videos from that server instead of CDN. File downloads won't have to be very fast, the minimum download speeds don't have to be at least the bitrate of the video to prevent buffering so the quality of the bandwidth won't matter as much.

 

You can rent servers with 1gbps unmetered internet connection from a company like Leaseweb from as little as $250 a month, here's an example: https://www.leaseweb.com/dedicated-server/configure/24265 or from around $320 in US datacenter : https://www.leaseweb.com/dedicated-server/configure/23763#hardware-container

 

I'd probably just up to memory to 16-32 GB and change the drives to 2-4 TB drives, as each video released would probably use up to around 5-10 GB (considering 4k,1080p, 720p, 360p etc) so disk space would fill up fast. IMHO the majority of the file downloads would be the last 7-14 uploads, and those would simply be cached in memory, so the server would have no problem serving tens to hundreds of separate downloads at same time.

 

[leonfagan71]

26 minutes ago, mariushm said:

No need for reinvent the wheel by building clusters of servers and using load balancers, especially at the beginning when you have few subscribers.

You could just push the videos to a CDN , something like Amazon Cloudfront, Limelight , Maxcdn ..

 

You get as low as around 6-8 cents per GB of traffic, so if a user pays 3$ a month, even with 30 x 500-800 MB videos in a month (probably typical for 720p content), you'd still only pay about 1..1.5$ a month per user in CDN prices.

Not all subscribers will watch absolutely every video posted, some will watch several times, it all averages out in the end.

 

If you want to allow viewers to download videos, you can basically rent a dedicated server in a large datacenter and let users download the videos from that server instead of CDN. File downloads won't have to be very fast, the minimum download speeds don't have to be at least the bitrate of the video to prevent buffering so the quality of the bandwidth won't matter as much.

 

You can rent servers with 1gbps unmetered internet connection from a company like Leaseweb from as little as $250 a month, here's an example: https://www.leaseweb.com/dedicated-server/configure/24265 or from around $320 in US datacenter : https://www.leaseweb.com/dedicated-server/configure/23763#hardware-container

 

I'd probably just up to memory to 16-32 GB and change the drives to 2-4 TB drives, as each video released would probably use up to around 5-10 GB (considering 4k,1080p, 720p, 360p etc) so disk space would fill up fast. IMHO the majority of the file downloads would be the last 7-14 uploads, and those would simply be cached in memory, so the server would have no problem serving tens to hundreds of separate downloads at same time.

 

If it were a smaller channel, you'd be able to start with a NAS and port forwarding...

However, LinusTechTips can sometimes reach a high capacity of simultaneous users, if it were a dedicated server, you have a single point of failure.

Some annoying 13 year old could/would DDOS the server and now everyone wouldn't be able to download the videos...

 

A multi server setup would be a better solution for this, having the ability to add servers when load is high to reduce the bottleneck of one server, meaning users will not have to wait for ages to get a socket to the server because everyone is hitting the one server at once. Any server in a cluster can receive a connection and start communicating to the client.

 

 

I do however agree that the disk space will fill up quickly though.

How content is managed is another thing, perhaps new videos go on faster servers with less storage and the old videos go to older servers with lower specs. So that newer servers can support more viewers whilst older videos don't generally get that many views after the first few months of it being released.

 

I also agree that a download server would be better than using sockets on the streaming server. 

A few dedicated servers would work fine for this. (still setup in a cluster environment)

I know SoYouStart uses OVH's network and I've used SoYouStart before and would recommend.
 

[Clechay]

I've understood that they use one central unit and load balancer for requests for videos because they are delivered by set of independent low-performance servers. I would guess they use simplest OVH Cloud solution since anything else would be an overkill.

Possibility of DDoS is not an issue any more, we have Cloudflare now  

[leonfagan71]

7 hours ago, Clechay said:

I've understood that they use one central unit and load balancer for requests for videos because they are delivered by set of independent low-performance servers. I would guess they use simplest OVH Cloud solution since anything else would be an overkill.

Possibility of DDoS is not an issue any more, we have Cloudflare now  

Sure, I agree with that.

 

The only major thing here is Cloudflare, I'm not sure how much Cloudflare will support all that UDP traffic. It might be a terrible idea to use it, it may be fine.

 

If they're hosting their servers in the OVH infrastructure, OVH have their own DDoS prevention in place that may or may not be sufficient. 

 

 

[Nitroblast]

Luke spoke on his stream about this, and he made it quite clear that every CDN and cloud service they have looked at is far too expensive for them - which is why they're doing it themselves. To my knowledge they're using OVH as the host, and just have a central server that ingests the videos and then pushes them out to the edge servers.

[leonfagan71]

4 minutes ago, Nitroblast said:

Luke spoke on his stream about this, and he made it quite clear that every CDN and cloud service they have looked at is far too expensive for them - which is why they're doing it themselves.

Would be nice if you could quote the poster when replying to this sort of thing...

I certainly never said that they should use a CDN, can be confusing when attempting to follow the few conversations here.

 

Cheers bud

[Nitroblast]

Errr, it was on his stream - so it might be here https://www.twitch.tv/luke_lafr/v/98725324.

[leonfagan71]

Just now, Nitroblast said:

Errr, it was on his stream - so it might be here https://www.twitch.tv/luke_lafr/v/98725324.

Nice of you to quote who you were speaking to...

 

If it were a 1 - 1 conversation, I understand but for a forum of people, it's much harder to work out who you were speaking to without reading every single reply...

I can't even work out who asked for a link to the stream...

[mariushm]

I watched Luke's stream... yeah, not sure I agree renting various servers in various places is such a great idea. 

 

With cheaper dedicated servers you won't get much (if any) DDOS protection, so some bad people could just attack the IPs of your rented servers, then the datacenter would just nullroute the IPs for a day or until they stop the incoming flood... it could turn up as a big pain in the a**.

Also, you have to be careful about some dedicated servers being offered with lots of bandwidth but with some fineprint in the contract saying that "download only" or "streaming" is not allowed.

So yeah, CDN systems can be more expensive, but you get added benefits with that like ddos protection, good transfer speeds for various locations (like try streaming to Hong Kong, Singapore, Japan... then try renting dedicated servers with reasonable bandwidth in those locations)

 

11 formats of video, that's kind of a lot. I assume the 2 codecs are h264 and vp9 (youtube offers h264 and aac in mp4 container, vp9 and aac/ogg/opus in webm container

I don't see what the big deal is about encoding the 11 different files, you could just render once in a lossless format that could take hundreds of GB and then use one or several computers to do encodings in parallel (you guys have 10gbps connections in the rack so shouldn't be a problem). Open source x264 can be very well configured for streaming (so your encoded files don't have sudden bursts of something like 20mbps for a couple of second that would screw up the buffering) , for most compatibility with various playback devices (limit reference frames, levels, basically make sure the maximum memory used for decoding would be relatively small so some older playback devices won't have problems with it)

 

You could basically write a small script or something, which adds the video in a database / cms when you're done rendering the original video in that lossless format,  and the encoding server or servers poll that database and notice the new video to be encoded and each of them start encoding the video into the format they're configured for and when done, they move or upload the encoded file somewhere and update the database to show that specific resolution movie was encoded and placed in the final "distribution folder" or to trigger an error if there's problems.

IF you don't use DASH or some other tricks at playback, you may have additional steps like encoding the audio separately into aac (and others) and muxing the encoded video with audio .. you may want to do 5.1 encoding for high bitrate versions, stereo for 360p to save bandwidth there.. even youtube has two or three ready muxed mp4 files to stream to clients that can only do http or some basic decoding, for everyone else they serve the video and audio separately and they're sync'ed in the player.

 

The distribution script can poll that database and when it detects all 11 formats are ready, it can start to push the files to various servers.  A smart way to do it would be for example to create a torrent file with these 11 files, and send a message to all your streaming servers all over the place with the small torrent file. All streaming servers can download the 11 files and exchange bandwidth between them (you can limit the upload bandwidth so it wouldn't interfere with upload to viewers), so you won't have to upload the 11 videos to 10-20 servers separately through http/https/ftp /whatever

When each server finishes the torrent, a script can be launched to move the files to their streaming locations (if they're different) and to message a central system that the specific server is ready to stream that particular video.

 

You'd want to make the system relatively flexible, for example maybe for WAN shows you may not want to encode 4K versions of the 1-2h show, because it's mostly static background anyway and there's no benefit in that case. May not even want to do 1080p, maybe you want just 720p 60 fps (which should have a bitrate maybe 10-20% higher than 720p 30fps, but a bit below 1080p)

 

So there should be a way when you render a video and want to make various encodings, have a way to say no, don't do 4k or even 1080p, have just 720p the maximum... the script that monitors and determines all video formats desired are encoded and ready to be pushed to various servers should be smart enough to account for that, and not wait forever for the 4k encoding if it's not meant to be encoded.

 

Oh and @Slick and LinusTechTips team ... you should also be careful about licensing codecs ... you're getting into the territory where you may get lawyers from MPEG-LA and others to pay for streaming to users, you may have to pay a fixed sum of money each month plus something like 1-2 cents per view or something like that

See for example http://www.mpegla.com/main/programs/AVC/Documents/avcweb.pdf slide 9/11 , if I read it right it's 2 cents per title (per view if more than 12 minutes ?) and/or  $25k a year if you have more than 100k subscribers.

You'd have to keep logs of all plays and how long each viewer watches, to have reports and pay for royalties..

 

HEVC is even more expensive but hopefully in a couple of years we'll have AV1 from AOM (alliance for open media), which is based on VP9 and other open source codecs which should have no royalties fees. (but you'd still have to support h264 for years for backwards compatibility)

 

 

 

 

[Walms]

7 hours ago, Nitroblast said:

Errr, it was on his stream - so it might be here https://www.twitch.tv/luke_lafr/v/98725324.

Thanks man, that stream was the sort of info I was looking for.

Yeah looks like all they need is a load balancer and something like rysnc to get the files distributed.
They'd need some kind of trans-coding service to suck in the 4k stream and spit out the 11 formats they need.

I'd say one thing they'll need to be very smart with a monitoring solution. To make sure they are delivering a quality stream.

Edited November 27, 2016 by Walms
I should have written more

[Clechay]

8 hours ago, Nitroblast said:

Luke spoke on his stream about this, and he made it quite clear that every CDN and cloud service they have looked at is far too expensive for them - which is why they're doing it themselves. To my knowledge they're using OVH as the host, and just have a central server that ingests the videos and then pushes them out to the edge servers.

I hope they don't push videos from main to all the others but rather to one of them and it pushes videos to the rest.

Pushing media to all servers from core unit would paralyze whole service after every upload 

[Walms]

Would be fun to get involved if they needed help. But from what I understand they're a windows only sorta team. All my expertise is in the Linux world

[Walms]

One thing they might want to look into is an OCS https://en.wikipedia.org/wiki/Online_charging_system.If they build a solution to handle that from the start they can give granular rewards in the future. Even if they don't use all the features of an OSC at the start. It would open up a lot of doors when it comes to turning a profit on this new streaming service. If nothing else it would allow a good way to monitor usage if they needed to get some stats to advertisers. 

But then I might be biased as I work for a company that develops systems like that

[Clechay]

On 11/28/2016 at 0:45 AM, Walms said:

Would be fun to get involved if they needed help. But from what I understand they're a windows only sorta team. All my expertise is in the Linux world

From Luke's speech it's clear he does not want any help or even contact on this subject. He's to cool for forum community to help him </3

[unijab]

1 hour ago, Clechay said:

From Luke's speech it's clear he does not want any help or even contact on this subject. He's to cool for forum community to help him </3

Is that why he said on a twitch stream to contact him on the forum if you were a backend developer?