A Read Only Server for 500 people

[esrootes]

I am trying to build this customized infotainment server from scratch. I need only 2TB total data storage capability. I'm kinda on a budget (about $4500).  The server will have data written to it initially and is expected to serve 500 people all trying to stream 480p movies at the worst case. (assuming using Gigabit Ethernet). So far I have researched a lot on the server configurations and figured that I'll use 16x128GB SAS HDDs (15K rpms) in RAID 0 (as I'm not currently concerned about reliability or up time). I'm kinda a n00b when it comes to server management (and no, I don't want to involve professionals. I wanna do it myself). 
I'm kinda confused regarding the data rates that'll be required for the above requirement to serve so many people. Will setting up a mirror for access help improve the performance ?
I'm also confused about the processing capabilities that this server will require. Like will I really need Xeon processors for this (considering the fact all videos are pre-processed and just needs to be served) or will I be able to get away with an ordinary dual core or quad core ?  
Please advice me in this regards. I'm willing to learn whatever it takes. I'm comfortable with networking but a beginner in storage systems.  

[Tedster]

Welcome to the Forum.

Please, don't raid0 sixteen HDDs. You're saying you don't care about reliability or uptime, but surely you must - why would you be building a server if zero uptime would be okay?

What bitrates would these files be? If they're a reasonable bitrate it seems as though gigabit would be cutting it close.

[leadeater]

Are you talking up to 500 simultaneous streams, that amount is unlikely to run that well off a single server. Not without knowing more about the applications being used etc.

 

Don't get 15K RPM SAS disks, they fail really quickly and are not that much faster than 10K RPM SAS. Don't use RAID 0, just don't. Use RAID 5 or RAID 6 as this will give you the same usable read performance as RAID 0 when talking about that many disks,

 

Other thing I have to question is the cost of 16 SAS disks, 4/5 Samsung 512GB 850 EVOs might be the same cost and will be much faster and more reliable.

 

Also don't be put off by Xeons just because of what they are, the price difference isn't that much. In any case for you you'll need a dual socket system which requires Intel E5-2600 series CPUs anyway.

 

Do you need to buy new or are you looking at used?

[esrootes]

5 minutes ago, Tedster said:

Welcome to the Forum.

Please, don't raid0 sixteen HDDs. You're saying you don't care about reliability or uptime, but surely you must - why would you be building a server if zero uptime would be okay?

What bitrates would these files be? If they're a reasonable bitrate it seems as though gigabit would be cutting it close.

Thank Tedster.
Reliability is not a concern as of now. I mean, it's alright if the server goes down for a while till I get it back up again by swapping the failed disk and rebuilding the data from my backup. (As there is no new data being generated or written to the server that I'll lose in case of a failure). I mean to say that I'm not concerned about 100% uptime. 
The bitrate would be anywhere from 500 to 1000 kbps. The server will contain other documents, non-high quality music files etc. It's a general infotainment system. 
 

[Tedster]

1 minute ago, esrootes said:

Thank Tedster.
Reliability is not a concern as of now. I mean, it's alright if the server goes down for a while till I get it back up again by swapping the failed disk and rebuilding the data from my backup. (As there is no new data being generated or written to the server that I'll lose in case of a failure). I mean to say that I'm not concerned about 100% uptime. 
The bitrate would be anywhere from 500 to 1000 kbps. The server will contain other documents, non-high quality music files etc. It's a general infotainment system. 
 

Alright!

I have to agree with @leadeater that SSDs would probably be better for what you're wanting to do. I however don't think I agree that you'd need a hugely powerful server in terms of the CPU department, but then I don't have much experience.

What would you be using to allow the files to be streamed/downloaded? Would you be using a simple local webserver or something?

 

[esrootes]

7 minutes ago, leadeater said:

Are you talking up to 500 simultaneous streams, that amount is unlikely to run that well off a single server. Not without knowing more about the applications being used etc.

 

Don't get 15K RPM SAS disks, they fail really quickly and are not that much faster than 10K RPM SAS. Don't use RAID 0, just don't. Use RAID 5 or RAID 6 as this will give you the same usable read performance as RAID 0 when talking about that many disks,

 

Other thing I have to question is the cost of 16 SAS disks, 4/5 Samsung 512GB 850 EVOs might be the same cost and will be much faster and more reliable.

 

Also don't be put off by Xeons just because of what they are, the price difference isn't that much. In any case for you you'll need a dual socket system with requires Intel E5-2600 series CPUs anyway.

 

Do you need to buy new or are you looking at used?

Thanks.
500 people simultaneously is the worst case (I'm assuming video streaming is the most intensive task that the server will have to perform). Apart from that, there will be music files and documents that shall be available.

What do you recommend? I was under this impression that multiple small disks are better than larger ones and that's the reason why I thought of using 16 disks. I guess I couldn't be more wrong. And RAID 0 (or 00) because there's no writing to the server - only reading for it. Thanks anyways. 

I'll look into the costs. I prefer using new ones, but in case used one are reliable, I'll consider that. 

[leadeater]

1 minute ago, Tedster said:

Alright!

I have to agree with @leadeater that SSDs would probably be better for what you're wanting to do. I however don't think I agree that you'd need a hugely powerful server in terms of the CPU department, but then I don't have much experience.

What would you be using to allow the files to be streamed/downloaded? Would you be using a simple local webserver or something?

 

The dual socket is more for the number of cores/threads that's needed to serve that many streams. Also depending on how the content is being service the CPU demand will vary wildly.

 

The cost isn't really that high, an Intel E5-2609v4 for example is around $300 USD and two of these would give 16 actual cores.

[Tedster]

Just now, leadeater said:

The dual socket is more for the number of cores/threads that's needed to serve that many streams. Also depending on how the content is being service the CPU demand will vary wildly.

 

The cost isn't really that high, an Intel E5-2609v4 for example is around $300 USD and two of these would give 16 actual cores.

I guessing I'm meaning to ask what's so intensive about serving plenty streams? 

I wonder if I should do a perform a couple of tests myself...

[esrootes]

6 minutes ago, Tedster said:

Alright!

I have to agree with @leadeater that SSDs would probably be better for what you're wanting to do. I however don't think I agree that you'd need a hugely powerful server in terms of the CPU department, but then I don't have much experience.

What would you be using to allow the files to be streamed/downloaded? Would you be using a simple local webserver or something?

 

Yeah, to be more specific, I wanna create a portal kind of thing so that the videos (and all the other information) can be accessed from the browser itself. Maybe use Django or something similar to create a web application to handle it. Idk, I'm still researching the options. 

[leadeater]

Just now, esrootes said:

Thanks.
500 people simultaneously is the worst case (I'm assuming video streaming is the most intensive task that the server will have to perform). Apart from that, there will be music files and documents that shall be available.

What do you recommend? I was under this impression that multiple small disks are better than larger ones and that's the reason why I thought of using 16 disks. I guess I couldn't be more wrong. And RAID 0 (or 00) because there's no writing to the server - only reading for it. Thanks anyways. 

I'll look into the costs. I prefer using new ones, but in case used one are reliable, I'll consider that. 

Used servers that are on ebay are still extremely reliable, they typically only get replaced due to the warranty running out and businesses should only run servers that are under warranty, if the system is in any way important. A 3 year old server will hands down beat anything you could build with new parts in reliability and performance multiple times over, they go really cheap used.

 

Scaling of RAID is rather problematic, you'll find the difference between 8 disks and 16 disks to not be a great deal at all. A single SSD will also out preform 6 10K SAS disks.

[leadeater]

3 minutes ago, esrootes said:

Yeah, to be more specific, I wanna create a portal kind of thing so that the videos (and all the other information) can be accessed from the browser itself. Maybe use Django or something similar to create a web application to handle it. Idk, I'm still researching the options. 

Anything web server protocol based will monster CPU usage, video streaming servers heavily rely on multicast and client driven access. For example flash video players and native HTML5 video players use much less server CPU than other types of access methods. Just something to keep in mind.

[leadeater]

Actually I know the perfect person to bring in to this conversation, not sure if he will join in but worth a shot. @CatBoiler

[esrootes]

7 minutes ago, leadeater said:

Used servers that are on ebay are still extremely reliable, they typically only get replaced due to the warranty running out and businesses should only run servers that are under warranty, if the system is in any way important. A 3 year old server will hands down beat anything you could build with new parts in reliability and performance multiple times over, they go really cheap used.

 

Scaling of RAID is rather problematic, you'll find the difference between 8 disks and 16 disks to not be a great deal at all. A single SSD will also out preform 6 10K SAS disks.

I'll definitely check it out. Thanks.
So a few SSDs are a lot better than several SAS disks. Should I use RAID for the SSDs at all ?
Also thanks a lot. You single-handedly gave me more insights than all my teachers.  

[leadeater]

2 minutes ago, esrootes said:

I'll definitely check it out. Thanks.
So a few SSDs are a lot better than several SAS disks. Should I use RAID for the SSDs at all ?
Also thanks a lot. You single-handedly gave me more insights than all my teachers.  

This is actually my day job, I'm a systems engineer for a university so I deal with storage platforms among many other things.

[leadeater]

17 minutes ago, Tedster said:

I guessing I'm meaning to ask what's so intensive about serving plenty streams? 

I wonder if I should do a perform a couple of tests myself...

Management overhead of each stream, would be good to test it out so you can see. You'll need 2 computers though, one to act as the server and another to run about 16 VMs to really see the effects of multi client access.

[Tedster]

1 minute ago, leadeater said:

Management overhead of each stream, would be good to test it out so you can see. You'll need 2 computers though, one to act as the server and another to run about 16 VMs to really see the effects of multi client access.

I'm setting up a rudimentary test now between two VMs with ridiculous bandwidth between them, just downloading the same file with very slightly different start times with a limited speed a few hundred times over

[leadeater]

9 minutes ago, Tedster said:

I'm setting up a rudimentary test now between two VMs with ridiculous bandwidth between them, just downloading the same file with very slightly different start times with a limited speed a few hundred times over

The more basic and low level the access method the better, a worst case would be something like Plex. The server that I run, VM actually, for that has 4 cores and with just 2 people watching a video with no transcoding it uses around 30%-50% CPU utilization. So 500 people watching Plex would require multiple servers, at least 20. Plex just isn't designed for that but it is a great example of why the applicatiom/access method matters so much.

[Tedster]

1 minute ago, leadeater said:

The more basic and low level the access method the better, a worst case would be something like Plex. The server that I run, VM actually, for that has 4 cores and with just 2 people watching a video with no transcoding it uses around 30%-50% CPU utilization. So 500 people watching Plex would require multiple servers, at least 20. Plex just isn't design for that but it is a great example of why the applicatiom/access method matters so much.

The idea I'm envisioning is just nginx serving the file with nothing wrapping it; I don't see how that would be intensive much

[Tedster]

Damn, I forgot to ratelimit them. I was seeing over 10gbps between those two VPSes...

[leadeater]

4 minutes ago, Tedster said:

The idea I'm envisioning is just nginx serving the file with nothing wrapping it; I don't see how that would be intensive much

That wouldn't no, a single server running ngix and 1Gbps pipe would do it. How you give the link to people would also matter too though. If you served it via a php web page that is nice and pretty with a bunch of images and styling to it that would use more than the actual downloading of the file.

[Tedster]

3 minutes ago, leadeater said:

That wouldn't no, a single server running ngix and 1Gbps pipe would do it. How you give the link to people would also matter too though. If you served it via a web page that is nice and pretty with a bunch of images and styling to it that would use more than the actual downloading of the file.

Welp, I just killed the VM I was creating the test requests from, it seems as though the testing was more intensive than the actual delivering of the files...

And I disagree, if you had a very simple web interface it'd be much less intensive serving that than serving the files, right?

[esrootes]

8 minutes ago, leadeater said:

That wouldn't no, a single server running ngix and 1Gbps pipe would do it. How you give the link to people would also matter too though. If you served it via a php web page that is nice and pretty with a bunch of images and styling to it that would use more than the actual downloading of the file.

Just curious, how much of a problem would it be ? Mostly static elements with a few dynamically loaded images ? Assuming there's plenty of cache.   

[leadeater]

10 minutes ago, Tedster said:

Welp, I just killed the VM I was creating the test requests from, it seems as though the testing was more intensive than the actual delivering of the files...

And I disagree, if you had a very simple web interface it'd be much less intensive serving that than serving the files, right?

Ouch hopefully you can get that back up quickly?

 

Simple webpage yea, but that is kind of the point of my Plex example. The how matters much more than the hardware, because without the how you can't make a correct judgement on the required hardware. A multicast video stream could be done from a Pi for 500+ clients as that's how multicast works, 500 unicast streams couldn't be done from a Pi.

 

500 people browsing a SMB share uses a lot of CPU on the server without anyone actually read or writing files.

[Tedster]

Just now, leadeater said:

Ouch hopefully you can get that back up quickly?

 

500 people browsing a SMB share uses a lot of CPU on the server without anyone actually read or writing.

There's nothing of value on it, but I'll give it a couple hours to see if it comes back on its own. It still seems to be flooding my SSH terminal every few minutes.

 

At my school we can make the SMB server unusable for a few minutes just by having thirty of us try to load listings of the same directory all at once.

[leadeater]

2 minutes ago, Tedster said:

At my school we can make the SMB server unusable for a few minutes just by having thirty of us try to load listings of the same directory all at once.

Wow that must be a pretty crap server, I've found schools have wildly differing server setups. Some are really good and lots are bloody horrid.