New server build

[MattBoothDev]

I've just made a purchase order though my employers for a new server (for the company). However, I've never actually built a server before, nor managed one properly, so any advice would be greatly appreciated!

 

The hardware that's coming is:

• 1x Gigabyte Dual Intel Haswell Extreme MD60-SC0
• 2x Intel 8 Core Xeon E5-2620 v4
• 2x Crucial ECC Registered 32GB DDR4 Server RAM 2133 MHz
• 5x Western Digital SE WD4000F9YZ 4TB SATA III
• 1x 650W EVGA SuperNOVA 650 G1
• 1x Logic Case SC-43480B 4U Short Chassis
• 1x 8 Port Highpoint RocketRAID 2720SGL(RR2720) Internal PCI-e 2.0
• 1x 800GB Intel 750 Series AIC SSD, HHHL PCIe 3.0 (x4), NVMe

 

The use-case for this machine is as such:

 

• We will be hosting an installation of Jetbrains YouTrack (Issue tracking. We already have this hosted on an old Mac Pro G6 running Ubuntu, so I will simply migrate it).
• We will be hosting an installation of Jenkins (Currently hosted on Amazon AWS, but will benefit massively from the increased speed of this machine - allowing more build jobs to run concurrently).
• We will be hosting remoteable instances of Windows 7, Windows 8.1 and Windows 10 (used for testing websites and applications). This will need to be fully accessible using VNC or RDP.
• We will be hosting as slew of internal documents and resources that we currently host on a woefully slow QNAP box.
• We may end up hosting our company website on this if/when we have 1Gbps fibre installed at our new location.

How I plan to do this?

• I plan on using the five Western Digital hard drives in RAID 6 using the RocketRAID card to host the NAS resources we currently use. Should be about 12TB available, which is more than we currently need, so should be good for the future.
• The 800GB Intel 750 SSD will host the operating system, Jenkins, YouTrack and Virtual Machines (the Windows boxes)
• I planned on using Ubuntu Server 14.04, as I have the most experience with Ubuntu out of any of the Linux distros
• I was considering partitioning the 800GB SSD into a couple of partitions such as one for the OS, one for YouTrack, one for Jenkins and one for the Virtual Machines, just to make them easier to visualise and manage in the future

My Questions?

Urgh, where to start?!

 

I have managed servers before, usually VPS instances and my own physical home server (basically a NAS box that I built myself using a Fractal Design Node 304 and a Haswell Pentium which mostly uses Plex to server media - didn't need to be beefy for my uses). However, I'd just like to get this right and do things properly, and what I plan on doing may not be the best way to do things.

 

• Is the partitioning thing necessary or stupid?
• Was getting a RAID card a good idea? My logic was that if the board ever dies, at least I cna get another one of a different brand, even socket type, and till use the RAID without any hassle
• Do I need to do anything particular after installing? Can I just fit the hardware, boot Ubuntu, install on the SSD, then install the RAID and configure? I looked at the motherboard manual online and it had some very advanced server features that I've never used before
• Would it be best practice to bridge all the NIC's on the board and issue that bridge MAC address a single IP via the router. 

Is there anything you guys think I should consider. Is there anything I've missed out?

 

Any help or advice is appreicated!

[Mark77]

HGST instead of WD drives, IMHO. 

 

Why put all that on the same machine?  Sounds like a disaster waiting to happen, and you're paying an awful premium for those high-end processors.  I'd up the RAM as well.  Virtualization is great and everything, but its intended for environments with dozens of servers, not just one "motherlode" box. 

 

Not sure about the Highpoint controllers either.  Doesn't seem to be like enterprise-grade kit from what I've read.  I'd be looking more towards LSI as a brand-name.

 

BTW, if this is for a business, I wouldn't even dare to do the integration myself.  Just too many things to get blamed for.  Pick up the Dell, HP, etc. catalogue and order what you need, with the same-day parts replacement service. 

[Jarsky]

That is not a RAID card in your build, it is an HBA - so you would need to do a software configuration really to make any use of it.

You don't need the RocketRAID SAS controller as the board already has the 8 x blue ports on it that can be utilised for RAID with the optional CRA220-8 T-Card which slots into the split PCI-E at the bottom of the board.

 

You'll also need to get some CPU coolers as Xeon's don't ship with them like Desktop CPU's (something a lot of people miss).

 

With utilising it for all those different purposes, you'll probably want to virtualise it - vmWARE ESXi Hypervisor is suitable for that task for a business.

 

As Mark77 pointed out though, when building reasonably relied upon servers for a business - its best to stick with purpose built machines by Dell, HP, Supermicro, etc...as they have business SLA's for part replacements.

[MattBoothDev]

We're a fairly small company. There's ~20-30 employees including myself. There wouldn't be much blaming if things went wrong!

 

The CPU's are fairly low end, I would've thought, at about £350 each (they scale up to £3,500 for the top end).

 

The VM's would not be used very often to be honest. Maybe one or two times a week at a push, for a few hours. Including them would simply be to save space rather than having a physical machine (fairly low spec ones) taking up space. If we could get away with a VM that would be used little and not so often, that was the aim.

 

Damn shame about the RAID card (looks like that exact model is lacking massively): http://www.tomshardware.co.uk/sas-6gb-raid-controller,review-32291-12.html The first LSI card is £400, which is a bit steeper than the £140 for the Highpoint. Any of these fit the bill? https://www.scan.co.uk/shop/computer-hardware/all/hdd-controllers/sata-sas-esata-8-port-6gb-s

 

I'll switch out the WD for HSGT under your recommendations. The price is basically the same. I'd be picking from here: https://www.scan.co.uk/shop/computer-hardware/all/hard-drives-int/sata-500gb-8tb-enterprise

 

The most stress this machine will come under is potentially the Windows VMs, but it's every day use will be a glorified NAS, YouTrack (web application) and build server for Jenkins. The load honestly wouldn't be too much.

[leadeater]

It would be much better to get a used LSI RAID card than use a highpoint so if price is that much of an issue I would consider that. There are tons of OEM HP, IBM Dell etc RAID cards on ebay that are all LSI based and will work.

 

I would also use a supermicro motherboard rather than a Gigabyte for a server and it is better to use 4 smaller sticks of RAM per CPU than to use 1 per CPU, possibly cheaper also. Make sure the RAM is ECC Registered too, there is a difference.

 

As @Jarsky pointed out you'll be wanting to use something like VMware ESXi, Hyper-V or Nutanix. If you use Nutanix you'll be able to use the SSD as a cache for the larger storage pool which works very well, we use Nutanix hardware at work for our development ESXi cluster for development VMs (186 total). The downside to Nutanix is the controller VMs require large amounts of RAM so 64GB may not go that far, the up side is that you'll only need an HBA not a RAID card.

 

Personally I would just use ESXi since this is your first time doing a server build from scratch and managing virtual infrastructure.

[MattBoothDev]

1 hour ago, Jarsky said:

That is not a RAID card in your build, it is an HBA - so you would need to do a software configuration really to make any use of it.

You don't need the RocketRAID SAS controller as the board already has the 8 x blue ports on it that can be utilised for RAID with the optional CRA220-8 T-Card which slots into the split PCI-E at the bottom of the board.

 

You'll also need to get some CPU coolers as Xeon's don't ship with them like Desktop CPU's (something a lot of people miss).

 

With utilising it for all those different purposes, you'll probably want to virtualise it - vmWARE ESXi Hypervisor is suitable for that task for a business.

 

As Mark77 pointed out though, when building reasonably relied upon servers for a business - its best to stick with purpose built machines by Dell, HP, Supermicro, etc...as they have business SLA's for part replacements.

Ah, yeah I missed the CPU coolers on the list. I've chosen two Noctua coolers that are extremely slim and don't go any wider than the socket.

 

Is it not a bad idea to use the motherboard RAID? Theory being that if the board dies it takes the array with it (unless you can source an identical board)?

 

The Highpoint is definitely a RAID controller: https://www.scan.co.uk/products/8-port-highpoint-rocketraid-2720sgl(rr2720)-internal-pci-e-20-x8-to-sas-sata-iii-raid-controller-oem but as pointed out earlier, it seems to pale in comparison to the LSI cards.

 

If I bought a premade server form Dell or HP, I'd still be learning how to manage it on the job, so to speak, rather than building it and installing the software myself. But I'm starting to reconsider  As much as I'd want to build my own server and have all the fun that comes with that... If I can't get it running properly it's an expensive paperweight.

14 minutes ago, leadeater said:

It would be much better to get a used LSI RAID card than use a highpoint so if price is that much of an issue I would consider that. There are tons of OEM HP, IBM Dell etc RAID cards on ebay that are all LSI based and will work.

 

I would also use a supermicro motherboard rather than a Gigabyte for a server and it is better to use 4 smaller sticks of RAM per CPU than to use 1 per CPU, possibly cheaper also. Make sure the RAM is ECC Registered too, there is a difference.

 

As @Jarsky pointed out you'll be wanting to use something like VMware ESXi, Hyper-V or Nutanix. If you use Nutanix you'll be able to use the SSD as a cache for the larger storage pool which works very well, we use Nutanix hardware at work for our development ESXi cluster for development VMs (186 total). The downside to Nutanix is the controller VMs require large amounts of RAM so 64GB may not go that far, the up side is that you'll only need an HBA not a RAID card.

 

Personally I would just use ESXi since this is your first time doing a server build from scratch and managing virtual infrastructure.

 

There are 8 sticks of 8GB RAM on the list (2 x 32GB) with the idea being to use 4 stick per CPU. It's Crucial ECC registered.

 

I'm guessing this is the software you recommend? http://www.vmware.com/uk/products/vsphere-hypervisor/gettingstarted.html

 

 

 

EDIT:

 

Cheers for the input guys. I appreciate the time you're taking to reply

[leadeater]

31 minutes ago, MattBoothDev said:

I'm guessing this is the software you recommend? http://www.vmware.com/uk/products/vsphere-hypervisor/gettingstarted.html

Yep that's it. You can install it on a USB drive or SD card which is the preferred method anyway.

[MattBoothDev]

So, the idea would be to install VMware ESXi in order to install the VM's for Windows, but also install Ubuntu Server as a VM as well and manage them as though I would a VPS? Perhaps have three instances of Ubuntu Server, one for NAS and one for the YouTrack and one Jetbrains installs?

 

Presumably I'd do this with a premade server from Dell or HP also?

[leadeater]

5 hours ago, MattBoothDev said:

So, the idea would be to install VMware ESXi in order to install the VM's for Windows, but also install Ubuntu Server as a VM as well and manage them as though I would a VPS? Perhaps have three instances of Ubuntu Server, one for NAS and one for the YouTrack and one Jetbrains installs?

 

Presumably I'd do this with a premade server from Dell or HP also?

Yep that's the general idea. If you buy from HP/Dell etc you can pick the option for it to come pre-installed with an SD card/USB drive with ESXi on it.

 

Also you could reuse that QNAP NAS you mentioned with Veeam to back up the VMs to it. There is a free version of Veeam that will do what you need.

[MattBoothDev]

Cheers!

 

Going to get some heads together and give this some analysis this afternoon and check over the pros and cons of handling this ourselves over buying a prebuilt machine. I've swapped out the RocketRAID by Highpoint for one of these: https://www.scan.co.uk/products/8-port-lsi-megaraid-sas-9271-8i-sgl-6gb-s-sataplussas-pcie-30-1gb-ddriii which is more costly but based on benchmarks and what you guys have said seems to be the better option.

 

I'm probably going to have to go with the Gigabyte board, though, as I've ordered the new Broadwell chips, and that board is the only one available that offers the ability to update the BIOS without a CPU installed. We don't have any LGA2011-v3 chips available (or any at all) that would be supported in order to perform the BIOS update. Pretty sure from my experience most boards won't even POST without a CPU installed.\

 

Any recommendations on this list of which drives to use? https://www.scan.co.uk/shop/computer-hardware/all/hard-drives-int/sata-500gb-8tb-enterprise Someone mentioned HSGT, which looks similar to the WD ones.

 

These HW recommendations are just an advisory right now while we have a discussion with the team about what to do. Just in case we opt for the build ourselves.

[leadeater]

15 minutes ago, MattBoothDev said:

Cheers!

 

Going to get some heads together and give this some analysis this afternoon and check over the pros and cons of handling this ourselves over buying a prebuilt machine. I've swapped out the RocketRAID by Highpoint for one of these: https://www.scan.co.uk/products/8-port-lsi-megaraid-sas-9271-8i-sgl-6gb-s-sataplussas-pcie-30-1gb-ddriii which is more costly but based on benchmarks and what you guys have said seems to be the better option.

 

I'm probably going to have to go with the Gigabyte board, though, as I've ordered the new Broadwell chips, and that board is the only one available that offers the ability to update the BIOS without a CPU installed. We don't have any LGA2011-v3 chips available (or any at all) that would be supported in order to perform the BIOS update. Pretty sure from my experience most boards won't even POST without a CPU installed.\

 

Any recommendations on this list of which drives to use? https://www.scan.co.uk/shop/computer-hardware/all/hard-drives-int/sata-500gb-8tb-enterprise Someone mentioned HSGT, which looks similar to the WD ones.

 

These HW recommendations are just an advisory right now while we have a discussion with the team about what to do. Just in case we opt for the build ourselves.

HGST is owned by Western Digital fyi.

 

Also you can update the bios of a Supermicro motherboard using IPMI without a CPU installed, http://www.supermicro.com/solutions/SMS_SUM.cfm, however go with what you think is best and most comfortable with. I think you may also have to buy an IPMI license to do it, HP/IBM/Dell allow you to do it without a license and only make you pay for remote control and other advanced features.

 

If you are going to use ESXi it is very important that you buy 512 byte native sector disks, 512e and 4Kn do work but are not officially supported.