Video Request: Server building basics

[john01dav]

Hello,

 

I would like to suggest a video topic for this channel to make (and to simultaneously ask if such a video already exists elsewhere): a video explaining how servers are built. As I understand it, the basic components (motherboard, cpu, ram, psu, gpu (if needed, often unnecessary), etc.) are similar along with similar interfaces to connect them (ie. I am fairy sure that servers also generally use pcie, sata, and nvme). But, I also know of some differences such as specific sizes for rack mount servers (ie. 1U, 2U), with different PSUs for each size. Looking around on Amazon, I am also not seeing any xU PSUs made by brands I recognize, and they also all seem to be sub 500W. How can someone looking to build a server pick a PSU, given this information? Additionally, how are things mounted inside the case? The pictures on amazon almost entirely lack any recognizable drive bays, motherboard backplanes, PSU cages, etc. I would really like to see a video that goes over this information, with the intended audience being someone who already knows a fair bit about building desktop computers. Essentially, I want a server version of those basic "How to build a computer" videos.

 

[NelizMastr]

People generally don't "build" servers. You buy either a barebones server from HPE/Lenovo/Dell or you preconfigure it and get it delivered to your doorstep. There's not much to be saved money wise by going custom. And you get worse support in the process.

[desertcomputer]

5 minutes ago, NelizMastr said:

People generally don't "build" servers. You buy either a barebones server from HPE/Lenovo/Dell or you preconfigure it and get it delivered to your doorstep. There's not much to be saved money wise by going custom. And you get worse support in the process.

^

Also finding actual server parts for sell on website such as amazon is quite hard but there is plenty of second hand though on  ebay etc plus most of these part don't have proper marketing/information and fancy product pictures.

[davrosG5]

For more server/enterprise oriented videos I'd suggest taking a look at Level1Techs.

But yeah, you can technically build a server from any compatible PC parts, the 'server' is really the software OS that goes on it whether that's Linux, Windows or some other UNix variant.

Server grade hardware is generally a bit harder to obtain because it's normally sold to system integrators but you might be able to pick up some stuff from the likes of SuperMicro as they make a lot of server grade hardware. Buying new, assuming you can find a supplier, will be considerably more expensive than the nearest equivalent consumer grade hardware. As EMC says above though you can find a lot of used server hardware on eBay but it'll probably be at least a generation or so behind current consumer stuff as enterprise customers tend to hold onto their hardware for a specific period of time (to achieve ROI on it).

 

If you want an idea of server components go and have a look at the SuperMicro website - they produce everything from the chassis to the power supply, motherboards and add-in cards right up to complete systems. Whether you can find anywhere to sell to you directly is a different question though.

[Syntaxvgm]

I really dont think this is Linus's thing.

Also, you typically get prebuilts or piece together a barebone unit with what components you need, or in larger deployments get some kind of contract with a company that includes your options all built out for you and a service agreement for faulty parts or sometimes the whole RU. 
In large deployments, mostly you'll really only see custom built servers for specialty applications, like GPU based processing, but now that Big Data Deep Learning with AI Pizza Rolls on the Block Chain is a popular thing, you see a lot more prebuilt options for that as well. 

[xriqn]

Technically any old computer can be used as a server so it'd be pointless.

[NelizMastr]

5 minutes ago, xriqn said:

Technically any old computer can be used as a server so it'd be pointless.

A coworker still uses a 286 as a print server, so yeah  

[xriqn]

Just now, NelizMastr said:

A coworker still uses a 286 as a print server, so yeah  

I've ran an old core 2 duo machine as a full on game server before.

[NelizMastr]

3 minutes ago, xriqn said:

I've ran an old core 2 duo machine as a full on game server before.

Same. I ran a minecraft beta server on a mac mini G4 lol.

[xriqn]

Just now, NelizMastr said:

Same. I ran a minecraft beta server on a mac mini G4 lol.

The game server I ran on my core 2 duo machine was a spigot 1.12.2 server with 73 plugins. Exact specs were 4GB DDR2 800, Core 2 Duo E8400, 250GB Samsung Spinpoint HDD and Ubuntu 16.04 (not server variant) x64.

[john01dav]

On 8/21/2018 at 2:49 AM, NelizMastr said:

People generally don't "build" servers. You buy either a barebones server from HPE/Lenovo/Dell or you preconfigure it and get it delivered to your doorstep. There's not much to be saved money wise by going custom. And you get worse support in the process.

Does this mean that there aren't, for example, standardized motherboard screw locations and other specifications to ensure that separately bought parts are compatible? What if I buy a pre-built server and one of the parts fails out of warranty? Does this mean I can't get replacement parts, and/or that they pull that shit laptop manufacturers do where everything is soldered together? If I have a 1U case, is there no way to find a motherboard, PSU, etc. that will fit in it without modification?

 

On 8/21/2018 at 3:52 AM, davrosG5 said:

Server grade hardware is generally a bit harder to obtain because it's normally sold to system integrators but you might be able to pick up some stuff from the likes of SuperMicro as they make a lot of server grade hardware. Buying new, assuming you can find a supplier, will be considerably more expensive than the nearest equivalent consumer grade hardware. As EMC says above though you can find a lot of used server hardware on eBay but it'll probably be at least a generation or so behind current consumer stuff as enterprise customers tend to hold onto their hardware for a specific period of time (to achieve ROI on it).

You mention that server grade hardware is a lot harder to find, but is it even necessary to find it? The only differentiation that Intel seems to have between its server grade and consumer grade hardware is ECC support, more cores, and lower clock speeds. One can get all of these by just buying a ryzen (or threadripper, for more power). Are AMD's ryzen/threadripper CPUs somehow worse than Xeons? What about if their epyc CPUs are taken into account (readily available here)? I know ECC ram is readily available, as PCPartPicker lists a ton of it. A little bit of searching seems to come up with enterprise grade SSDs and HDDs. Combined, that is basically the whole computer.

 

22 hours ago, Syntaxvgm said:

I really dont think this is Linus's thing.

He seems to constantly be building servers for that under-the-stair server closet. Presumably, putting together a (set of) video(s) on that woulden't be too hard.

 

21 hours ago, xriqn said:

Technically any old computer can be used as a server so it'd be pointless.

That's true, and my home server that my ISP keeps throttling is made using consumer parts (but is still distinct due to the complete lack of a GPU), but I imagine colocation facilities would get pissy if I tried to put anything other than a rack mount in a hypothetical rack I may rent in the future.

[NelizMastr]

4 minutes ago, john01dav said:

Does this mean that there aren't, for example, standardized motherboard screw locations and other specifications to ensure that separately bought parts are compatible? What if I buy a pre-built server and one of the parts fails out of warranty? 

You buy them from the manufacturer. The difference between computers and servers is that servers have a supply of replacement parts for at least 5 years (sometimes up to 10 years) and you can buy them directly from the manufacturer. They're non-standard typically so don't bother putting in components that aren't verified as compatible.

[john01dav]

2 minutes ago, NelizMastr said:

You buy them from the manufacturer. The difference between computers and servers is that servers have a supply of replacement parts for at least 5 years (sometimes up to 10 years) and you can buy them directly from the manufacturer. They're non-standard typically so don't bother putting in components that aren't verified as compatible.

If I am interested in more standard server parts, where can I find those? What if I want a server where I can start with an AMD cpu and then, later, switch it for an Intel CPU without replacing everything (ie. only replacing the motherboard and CPU). Also, what are the relevant standards (ie. what is the server equivalent of ATX)? Knowing this information is essential if I want to rackmount consumer grade hardware, which is often sufficient for many server tasks.

[NelizMastr]

3 minutes ago, john01dav said:

If I am interested in more standard server parts, where can I find those? What if I want a server where I can start with an AMD cpu and then, later, switch it for an Intel CPU without replacing everything (ie. only replacing the motherboard and CPU). Also, what are the relevant standards (ie. what is the server equivalent of ATX)? Knowing this information is essential if I want to rackmount consumer grade hardware, which is often sufficient for many server tasks.

If you want consumer grade rackmount stuff, the standard is still ATX. The only issue is that 1U is a tad more difficult and will rely on proprietary PSUs for that particular case (this differs from case to case, pun intended). Look at Norco, Supermicro and iStar USA for ATX compliant rackmount cases. 

 

If you want fully standard ATX, 2U and 4U are the way to go.

[john01dav]

Just now, NelizMastr said:

If you want consumer grade rackmount stuff, the standard is still ATX. The only issue is that 1U is a tad more difficult and will rely on proprietary PSUs for that particular case (this differs from case to case, pun intended). Look at Norco, Supermicro and iStar USA for ATX compliant rackmount cases. 

 

If you want fully standard ATX, 2U and 4U are the way to go.

Will the ATX compliant rackmount cases be marked as such, and, aside from the PSU, still be fully ATX (ie. standard PCIe cards, standard IO shield dimensions, motherboard standoffs)? How does one go about getting a compatible PSU? Will it come with the case? Will any 2/4U case allow a standard ATX power supply, motherboard, io shield, PCIe card, etc.? I am mainly wanting to go for 1U because it looks like all of the colocation providers charge per U, until you get up to renting a full cabinet.

[NelizMastr]

4 minutes ago, john01dav said:

Will the ATX compliant rackmount cases be marked as such, and, aside from the PSU, still be fully ATX (ie. standard PCIe cards, standard IO shield dimensions, motherboard standoffs)? How does one go about getting a compatible PSU? Will it come with the case? Will any 2/4U case allow a standard ATX power supply, motherboard, io shield, PCIe card, etc.? I am mainly wanting to go for 1U because it looks like all of the colocation providers charge per U, until you get up to renting a full cabinet.

Perhaps something like this?

 

https://www.logic-case.com/products/rackmount-chassis/1u/rack-mountable-server-chassis-case-1u-400mm-short-depth-for-atx-mb-sc-13400/

 

You can fit a flexATX PSU and ATX mobo. Expansion cards in 1U cases aren't much of a thing aside from maybe a NIC.

[davrosG5]

3 hours ago, john01dav said:

You mention that server grade hardware is a lot harder to find, but is it even necessary to find it? The only differentiation that Intel seems to have between its server grade and consumer grade hardware is ECC support, more cores, and lower clock speeds. One can get all of these by just buying a ryzen (or threadripper, for more power). Are AMD's ryzen/threadripper CPUs somehow worse than Xeons? What about if their epyc CPUs are taken into account (readily available here)? I know ECC ram is readily available, as PCPartPicker lists a ton of it. A little bit of searching seems to come up with enterprise grade SSDs and HDDs. Combined, that is basically the whole computer.

The AMD equivalent for Intels Xeon is, as you say, the Epyc lineup. The main difference you will find between the higher end Xeon/Epyc cpu's and the Core/Ryzen/Threadripper lines is support for multi-socket operation - so the maximum number of physical Core/Ryzen/Threadripper chips you can have per motherboard is one but with Xeon Scaleable and Epyc its 2 or more depending on the SKU.

One the CPU front, these days the server parts are often optimised more for things like virtualisation which is one reason they tend to also support higher RAM ceilings than the more consumer grade parts. Server parts will also typically offer a greater deal of PCIe connectivity than the consumer parts, for example Intels HEDT parts top out at 44 PCIe lanes, but the Xeon Scaleable allows 48 (and you can have 2 cpu's so your system supports 96 lanes). Threadripper allows 64 PCIe lanes (4 reserved for the chipset or something IIRC) whereas a dual Epyc board will give you 128 lanes.

 

The other oft given justification for the higher price of server gear is that it's built to a higher standard in terms of longevity - servers are generally expected to operate 24/7 for extended periods of time and also in generally hotter conditions than consumer grade stuff. It'll also often be warranted to be able to run under those conditions for a longer period of time than the equivalent consumer hardware.

 

As for storage, enterprise SSD's and HDD's are optimised for typical server workloads so have somewhat different performance characteristics than consumer gear. They may also come with SAS, rather than SATA or NVMe connectors which currently tops out at 12GB/s transfer rate compared with 6 GB/s for SATA. IN addition, they'll typically also be intended for use in much larger sized arrays than consumer NAS type drives for example - consumer NAS drives are typically recommended for anywhere up to 8 - 16 drives max int he array whereas you could have a storage server with 20+ drives (which incidentally is interesting given Seagate provided Linus with what looked like IronWolf Pro drives for his Storinator/Petabyte project IIRC)

 

The server equivalents of things like ATX are the SSI CEB and EEB form-factors. They're similar in size to ATX but have the mounting holes in slightly different locations. You're also more likely to see proprietary shaped motherboards to fit specific chassis - you get some interesting L-shaped mobo's for some types of server for example.

 

Server power supplies often are intended for hot-swap-capable operation - so you'd have dual PSU's in the case, either of which is capable of carrying the full load of the server. If one fails you can pull it and slot in a replacement without having to shut down the server. PSU's for rack servers tend to be long and thin rather than conforming to the more traditional PSU layout you get for consumer or even workstation computers.

 

In summary, whether you'd really see any genuine benefit from actual server grade hardware depends entirely on what it is you want to do with the server and to an extent what case/form factor you use and where it will be located (for example is it going to be a Tower or a Rackmount case?). If you go with rackmount stuff then it's typically much louder than consumer grade hardware because it's expected to operate with more restricted airflow (so the fans need to run at higher rpm's to deliver the necessary cooling).

Otherwise, building a server will be very much like building any other PC, it's just that the parts tend to be more expensive and things don't necessarily screw in/attach in the same places you'd expect in consumer PC.