datacenter "Completed" Home Datacenter

[EricMarsi]

Hey Everyone!

 

Wanted to start a thread in regards to my home datacenter project where I will post updates on what equipment I am playing around with. Completed is in quotes as you and I both know that nothing is ever 100% complete and is always changing. Just a quick rundown of what I have in my "datacenter" as of 11/29/17 (Top to Bottom):

 

g

• 1x Cisco 2960s POE+ Switch
  • 4x1GB uplink to the core switch
• 1x Cisco 3650 POE Layer 3 Core Switch
• 1x Cisco Meraki MX64 (Wanting to swap this out for an ASA-5506X)
• 1x Motorola SB6120 Cable Modem (Wanting to swap this out for an Arris SB8200)
• 1x Cisco 2504 Wireless LAN Controller
  • Controls Cisco 1702, 2702 and one 1602 Access Points
• 1x Cisco 2901 Router
  • Used as a basic firewall between my internal perimeter network and my internal network (Skype for Business, Exchange Transport, and Reverse Proxies Live Here
• 1x Cisco 2921 Router
  • Used for a Cisco CUCM Deployment. I care more about Skype for Business but it gives me the chance to learn a migration path from one phone system to another.
• 2x Cisco 2960s Switches Stacked for Servers
  • Each Server's Network team is split between the switches so that if either of the switches fail, the servers stay online.
• HV4
  • HP Gen8 Microserver
  • 16gb RAM
  • Xeon 1220L V2
  • LSI 9260-8i Raid Card
  • 1x 250GB SSD for OS
  • Used as a Hyper-V Host for the following VM's
    • File Server
    • System Center VMM Library
    • System Center SCCM Storage
• HV3
  • Dell R210 II
  • 32GB RAM
  • Xeon 1265
  • LSI 9211-4i
  • 2x 120GB SSDs for OS and Virtual Machines
  • 2x 4TB WD Red Drives in RAID 1 for Storage
  • Used as a Hyper-V Host for the following VM's:
    • Veeam Backup and Replication
• HV2
  • HP DL160 Gen8
  • 2x Xeon E5-2670 (8 Cores, 16 threads Each)
  • 192GB Ram (8GB DIMMS)
  • HP P420i Raid Card
  • 2x 72GB 15K SAS Drives in RAID 1 for OS
  • 6x 300GB 15K SAS Drives in RAID 5 for VM Storage
  • Used as a Hyper-V Host for the following VM's:
    • 2nd Domain Controller
    • SCCM Server
    • SCVMM Server
    • System Center SQL Always on AG SQL Server 2
    • Skype for Business Back end SQL Always on SG SQL Server 2
    • 2x Skype for Business Front-End Servers (3 and 4)
    • Many More Skype for Business Servers
    • Many Many More VM's
• HV1
  • HP DL380 Gen8
  • 2x Xeon E5-2670 (8 Cores, 16 threads Each)
  • 192GB Ram (8GB DIMMS)
  • HP P420i Raid Card
  • HP P420 Raid Card
  • 2x 180GB Corsair SSD's in RAID 1 for OS
  • 4x 300GB 15K SAS Drives in RAID 5 for VM Storage
  • 2x random test drives
  • Used as a Hyper-V Host for the following VM's:
    • Primary Domain Controller
    • KMS Server
    • System Center SQL Always on AG SQL Server 1
    • Skype for Business Back end SQL Always on SG SQL Server 1
    • 2x Skype for Business Front-End Servers (1 and 2)
    • Many More Skype for Business Servers
    • Many Many More VM's

Here's a look at the different subnets that I have and some other configuration information that I document (What I consider Confidential information has been redacted) Note that this is just a small part of what actually runs on my network :):

Click to enlarge:

 

 

Here is a video on the current status:

 

 

Please let me know what you think and give some suggestions!

[Essence_of_Darkness]

Very nice home datacenter.

How much did servers cost and is cisco network gear hard to configure ?

[mpsparrow]

I would love to have a setup like that one day, although where I live electricity and power reliability is a huge issue.

[EricMarsi]

23 minutes ago, Essence_of_Darkness said:

Very nice home datacenter.

How much did servers cost and is cisco network gear hard to configure ?

Honestly Ive been getting this stuff over the years so its hard to say "This" was the cost but I believe one of them 1.5 years ago was 1600 total (Used).

 

As for configuring Cisco Equipment, I think it is very easy. Its my preferred vendor. I like to plan out the ports and have base configs so configuration is simple and usually only takes me 5-10 minutes per device. Here is an example of how I plan the ports:

 

 

Thank you for the compliment

 

22 minutes ago, mpsparrow said:

I would love to have a setup like that one day, although where I live electricity and power reliability is a huge issue.

I forgot to mention I do have two UPS'es, one for server hardware and one smaller one for everything else. That ensures that power is stable coming into the rack. I never had one before this upgrade that I just did this past weekend so I too did suffer from the unstable power issue. Now if I could only get one of those large EATON UPS'es like @LinusSebastian  but unfortunately, being independent and in school full time still (only 20) is the drawback from being able to afford a 17K UPS.

[Essence_of_Darkness]

1 minute ago, EricMarsi said:

Honestly Ive been getting this stuff over the years so its hard to say "This" was the cost but I believe one of them 1.5 years ago was 1600 total (Used).

Thank you for the compliment

 

NP m8, this is pure gold especially for the age you've done. I am a little 15 year old twat  that has most of his servers running on X99 mobo's and i7 5820k's

[EricMarsi]

3 minutes ago, Essence_of_Darkness said:

NP m8, this is pure gold especially for the age you've done. I am a little 15 year old twat  that has most of his servers running on X99 mobo's and i7 5820k's

More than I had at 15 I had whatever my laptop was at the time as my Hyper-V Host :P. Towards the end of highschool, My Cisco Networking instructor gave me some servers and other things; DL360 G5's to be exact

[Essence_of_Darkness]

1 minute ago, EricMarsi said:

More than I had at 15 I had whatever my laptop was at the time as my Hyper-V Host :P. Towards the end of highschool, My Cisco Networking instructor gave me some servers and other things; DL360 G5's to be exact

My dad actually brought most of server gear from Germany , I've bought mobos, cpu's, memory and RAID cards for cheap, while he brought server cabinet, server cases, psu's and hard drives.

It was heaven for me to build all of it and configure it.

 

[EricMarsi]

9 minutes ago, Essence_of_Darkness said:

My dad actually brought most of server gear from Germany , I've bought mobos, cpu's, memory and RAID cards for cheap, while he brought server cabinet, server cases, psu's and hard drives.

It was heaven for me to build all of it and configure it.

 

Very Nice; we all start somewhere. I used to like building servers, but its much easier to buy used HP Servers (No a dell server fan). I updated the original post with some more information.

[Essence_of_Darkness]

1 hour ago, EricMarsi said:

Very Nice; we all start somewhere. I used to like building servers, but its much easier to buy used HP Servers (No a dell server fan). I updated the original post with some more information.

Pure freaking gold, I've seen a lot of good configured networks but this one is top notch IMO. Plz don't laugh at my servers LOL

We got most of these parts in Germany on black friday sale.

 

My and Dad's Media Server (Plex Media Server all hosting through FreeNAS) : 

PCPartPicker part list / Price breakdown by merchant

CPU: Intel - Core i7-5820K 3.3GHz 6-Core Processor  ($175.00) 
CPU Cooler: Noctua - NH-L9x65 33.8 CFM CPU Cooler  ($49.95 @ Amazon) 
Motherboard: ASRock - X99 Extreme4 ATX LGA2011-3 Motherboard  ($97.00) 
Memory: Crucial - 8GB (1 x 8GB) DDR4-2133 Memory  ($41.00) 
Memory: Crucial - 8GB (1 x 8GB) DDR4-2133 Memory  ($41.00) 
Memory: Crucial - 8GB (1 x 8GB) DDR4-2133 Memory  ($41.00) 
Memory: Crucial - 8GB (1 x 8GB) DDR4-2133 Memory  ($41.00) 
Storage: Samsung - 850 EVO-Series 250GB 2.5" Solid State Drive  ($57.00) 
Storage: Samsung - 850 EVO-Series 250GB 2.5" Solid State Drive  ($57.00) 
Storage: Western Digital - Red 6TB 3.5" 5400RPM Internal Hard Drive  ($120.00) 
Storage: Western Digital - Red 6TB 3.5" 5400RPM Internal Hard Drive  ($120.00) 
Storage: Seagate - FireCuda 2TB 3.5" 7200RPM Hybrid Internal Hard Drive  ($60.00) 
Storage: Seagate - FireCuda 2TB 3.5" 7200RPM Hybrid Internal Hard Drive  ($60.00) 
Video Card: Gigabyte - GeForce GT 1030 2GB Silent Low Profile Video Card  ($40.00) 
Power Supply: EVGA - SuperNOVA G2 650W 80+ Gold Certified Fully-Modular ATX Power Supply  ($99.99 @ Amazon) 
Other: LSI MegaRAID SAS 9271-8I 8-port 6Gb/s SATA+SAS PCI-Express 3.0 Low Profile RAID Controller Card, Single  ($148.00) 
Other: 3U 26IN  General Purpose Server Chassis  ($145.36) 
Total: $1393.30
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2017-11-29 13:19 EST-0500

 

My Storage Server (FreeNAS)

PCPartPicker part list / Price breakdown by merchant

CPU: Intel - Xeon E5-2620 V4 2.1GHz 8-Core Processor  ($235.00) 
CPU Cooler: Noctua - NH-L9x65 33.8 CFM CPU Cooler  ($49.95 @ Amazon) 
Motherboard: Asus - X99-DELUXE II ATX LGA2011-3 Motherboard  ($175.00) 
Memory: Kingston - ValueRAM 8GB (1 x 8GB) DDR4-2133 Memory  ($59.00) 
Memory: Kingston - ValueRAM 8GB (1 x 8GB) DDR4-2133 Memory  ($59.00) 
Memory: Kingston - ValueRAM 8GB (1 x 8GB) DDR4-2133 Memory  ($59.00) 
Memory: Kingston - ValueRAM 8GB (1 x 8GB) DDR4-2133 Memory  ($59.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Storage: Western Digital - Red 8TB 3.5" 5400RPM Internal Hard Drive  ($185.00) 
Storage: Western Digital - Red 8TB 3.5" 5400RPM Internal Hard Drive  ($185.00) 
Storage: Western Digital - BLACK SERIES 2TB 3.5" 7200RPM Internal Hard Drive  ($60.00) 
Storage: Western Digital - BLACK SERIES 2TB 3.5" 7200RPM Internal Hard Drive  ($60.00) 
Video Card: Gigabyte - GeForce GT 1030 2GB Silent Low Profile Video Card  ($69.99 @ Amazon) 
Power Supply: EVGA - SuperNOVA G2 650W 80+ Gold Certified Fully-Modular ATX Power Supply  ($99.99 @ Amazon) 
Other: NORCO 4U Rack Mount 10-Bays SATA/SAS Server Chassis RPC-450TH  ($165.00) 

Other: LSI MegaRAID SAS 9271-8I 8-port 6Gb/s SATA+SAS PCI-Express 3.0 Low Profile RAID Controller Card, Single  ($148.00) 
Total: $1888.93
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2017-11-29 13:29 EST-0500

 

My and dad's VM Server (VMWare ESXi)

PCPartPicker part list / Price breakdown by merchant

CPU: Intel - Xeon E5-2620 V4 2.1GHz 8-Core Processor  ($235.00) 
CPU: Intel - Xeon E5-2620 V4 2.1GHz 8-Core Processor  ($235.00) 
CPU Cooler: Noctua - NH-L9x65 33.8 CFM CPU Cooler  ($49.95 @ Amazon) 
CPU Cooler: Noctua - NH-L9x65 33.8 CFM CPU Cooler  ($49.95 @ Amazon) 
Motherboard: Asus - Z10PE-D16 WS SSI EEB Dual-CPU LGA2011-3 Motherboard  ($250.00) 
Memory: Crucial - 64GB (4 x 16GB) Registered DDR4-2133 Memory  ($750.00) 
Memory: Crucial - 64GB (4 x 16GB) Registered DDR4-2133 Memory  ($750.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Video Card: Gigabyte - GeForce GT 1030 2GB Silent Low Profile Video Card  ($69.99 @ Amazon) 
Power Supply: EVGA - SuperNOVA G2 850W 80+ Gold Certified Fully-Modular ATX Power Supply  ($119.00) 
Other: LSI MegaRAID SAS 9271-8I 8-port 6Gb/s SATA+SAS PCI-Express 3.0 Low Profile RAID Controller Card, Single  ($148.00) 
Other: NORCO RPC-450FH 4U Server Rackmount Chassis with 5 Hot Swappable Drive Bays  ($95.00) 
Total: $3301.89
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2017-11-29 13:36 EST-0500

 

Encoding Server (Windows 10)

PCPartPicker part list / Price breakdown by merchant

CPU: Intel - Core i7-6950X 3.0GHz 10-Core Processor  ($550.00) 
CPU Cooler: Noctua - NH-L12 37.8 CFM CPU Cooler  ($69.99 @ Newegg Marketplace) 
Motherboard: Asus - X99-DELUXE II ATX LGA2011-3 Motherboard  ($175.00) 
Memory: Crucial - Ballistix Sport LT 32GB (2 x 16GB) DDR4-2400 Memory  ($210.00) 
Storage: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive  ($110.00) 
Video Card: Gigabyte - GeForce GTX 1070 8GB Mini ITX OC Video Card  ($310.00) 
Power Supply: EVGA - SuperNOVA G2 750W 80+ Gold Certified Fully-Modular ATX Power Supply  ($77.98 @ Newegg) 
Other: NORCO 2U Rack Mount Server Chassis - Black RPC-270  ($85.00) 
Total: $1587.97
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2017-11-29 13:42 EST-0500

[Mikensan]

Very cool setup, though I'm curious about the heavy interest in skype for business? I honestly had thought it was entirely hosted on microsoft servers, didn't know there was anything to host on premises.

 

Definitely a ton of ram for your hypervisors, just make sure you don't over allocate CPUs per vm. Majority can get away with just 2 cpus. If you're not noticing any slowness or choppiness then nothing to worry about. Do the hypervisors have any shared storage (vsan / silo)? Beyond the router/firewall are you doing any sort of network monitoring / security (OSSEC / SELKS / PRTG / Nagios / etc..)? Are you using a NFR license for your veeam (my gripe is the single host limitation)?

 

A nice alternative to veeam (albeit simpler) is vSphere Data Protection.

[EricMarsi]

28 minutes ago, Mikensan said:

Very cool setup, though I'm curious about the heavy interest in skype for business? I honestly had thought it was entirely hosted on microsoft servers, didn't know there was anything to host on premises.

 

Definitely a ton of ram for your hypervisors, just make sure you don't over allocate CPUs per vm. Majority can get away with just 2 cpus. If you're not noticing any slowness or choppiness then nothing to worry about. Do the hypervisors have any shared storage (vsan / silo)? Beyond the router/firewall are you doing any sort of network monitoring / security (OSSEC / SELKS / PRTG / Nagios / etc..)? Are you using a NFR license for your veeam (my gripe is the single host limitation)?

 

A nice alternative to veeam (albeit simpler) is vSphere Data Protection.

Thanks for the interest in everything! 

 

Currently there are a few flavors of Skype for Business, Fully on Prem, Fully in Office 365, Hybrid between on prem and Office 365, and Could Connector edition (lets you connect your on prem PSTN to the cloud). Most of the cloud offerings are going away soon and Microsoft Teams is replacing Skype for business online as a separate more modern service. In a way the interested in unified communications is tied to both Skype for Business and Microsoft Teams As for my interest in it, it has just been one of my passions for the past 1.5 years. Ive already passed my 70-334 exam and am currently studying for my 70-333. For 20 years old that's not bad lol. Skype for Business alone is 16 virtual machines since I like making everything HIGHLY available but makes things much more complicated.

 

I try not to over allocate them, but I have had no slow downs other than running out of Disk IOPS. Looking into building/buying a SAN and building a Hyper-V cluster, but that wouldn't happen till next year. Storage currently is all independent per host. For monitoring, I am using PRTG and System Center Operations Manager. and Yes, I have a "special" NFR license since I am a student.

 

I prefer Quest Rapid Recovery (Formerly Dell AppAssure) but it is yet another expense that I just cannot afford. I have licensing for Data protection manager, but I personally do not like it as a backup solution.

[Mikensan]

You will certainly hit the ground running for a majority of IT jobs out there depending on what you want to do (assuming it's in the IT field).

 

I honestly did not know they offered certifications for Skype but then again why wouldn't they. Outside of the specifics for skype, learning the importance and understanding of high availability is very valuable and translates between technologies fairly well. Big kudos on that lesson, amazing what the availability of cheap enterprise equipment that's actually useful has done for those wanting to learn. When I was 20 the enterprise equipment available was still expensive and PIII based which meant you weren't virtualizing anything. 

 

Disk IOPS, bane of all our existence lol. Well even with local storage you could run some sort of replicated SAN giving you that "hyper converged" solution. VMware has vSAN, I feel like microsoft if not another vendor would have a solution. Unfortunately I'm a vmware guy so I don't know what the answer is but I'm sure there is one. Something to potentially play with anyway. @leadeater mentioned a hyperconverged OS once, nutanix I think it was, that would do this under their free/community license (I think). 

 

I honestly hadn't heard of Connectwise before, will have to take a gander into it. I'm liking the idea of a special veeam license, because I really like the software. I overlooked that you're running Hyper-V so I guess VDP wouldn't work for you, but I wouldn't call it a backup either. More of a disaster recovery solution. Personally my data is centralized on a NAS that just gets rsync'd. At work it's Backup Exec but we're looking at commvault. I hadn't heard of AppAssure / Rapid Recovery, just Avamar. I wonder if EMC will keep quest / absorb it, or what will happen. EMC is getting too big.

[leadeater]

4 hours ago, Mikensan said:

At work it's Backup Exec but we're looking at commvault.

If you're mainly backing up VMs and don't have complex SQL clusters go with Veeam, Commvault user btw .

 

I do like Commvault a lot but Veeam is a lot simpler to drive and Veeam + ReFS is super awesome. 

[leadeater]

18 hours ago, EricMarsi said:

EATON UPS'es like @LinusSebastian  but unfortunately, being independent and in school full time still (only 20) is the drawback from being able to afford a 17K UPS.

An Eaton 9130 will do a fine job for you, two is better because dual PSU etc. The 9130 range has all the same power protection features as the one LMG/LTT has and also supports extended battery modules, you might be able to find one on ebay. Only ever buy new official Eaton batteries for it and do so immediately after buying a used one unless you 100% know the condition and history of the UPS, wouldn't be surprised if they only ship without batteries anyway.

 

FYI you tagged a faker , you want @LinusTech.

 

17 hours ago, Essence_of_Darkness said:

Pure freaking gold, I've seen a lot of good configured networks but this one is top notch IMO.

I'll be needlessly critical and say it would have been better if @EricMarsi used a separate VLAN and subnet for iLO/iDRAC/IPMI .

 

14 hours ago, Mikensan said:

Very cool setup, though I'm curious about the heavy interest in skype for business? I honestly had thought it was entirely hosted on microsoft servers, didn't know there was anything to host on premises.

SFB can be an on-premises install, SaaS O365 or hybrid (I believe), it's being replaced with Teams btw. We have a SFB deployment across 3 cities with their own pool each that can fail/move between each datacenter if required so phones still work, so long as that campus still has network access to one of the other datacenters and still has power of course.

 

14 hours ago, EricMarsi said:

but I have had no slow downs other than running out of Disk IOPS

Storage Spaces Direct will allow you to continue to use Hyper-V in a hyper-converged scale out deployment that will solve your IOP problem, just put 2 small SSDs in each host so you can do tiering and write-back caching. You can also use Nutanix but I think the free license only works with AHV (essentially KVM) where the paid versions support Hyper-V and ESXi as the hypervisor.

[TonyStark]

Amazing. Looks like I'm not the only only with enterprise infrastructure in their house ! Way to go man !!! 

 

I see that you have a patch panel on top. I'd do some cable management there and get a rack swing-arm that encloses the cables inside.

 

So, if you have LAN sockets all over the house like electrical sockets, just tag them and map them accordingly to the patch panel. it will help you later on to troubleshoot network issues with great ease and help you to upgrade with the minimum required effort.