Home server array help ?

[GangGangKiwi]

I'm currently building my home server array I have three r610 servers all of them run dual xeon 5650s, two with 48gb ram and one with 96gb or ram. All the servers are running ESXI 6.7.

I'm going to use the top one for Plex and my vault cold storage. 

Second one for game servers. 

Bottom one is running my 5tb ssd freenas array and my streaming PC.

 

Server VM layout 

Top: Plex & freenas vault 

Middle: 7DTD server & modded Minecraft server

Bottom: SSD freenas array & streaming PC

 

I have a couple questions on what would be a good way to optimize data flow and manage resources VM. 

 

As far as networking equipment I have a TP-Link omada AX3600 AP & a trendnet 20 port gigabit web smart switch and for router I run pfsense. All my Ethernet cable is CAT7 rated cable.

 

Questions: 

-What would be a could way to limit the public facing VMs from my private VMs?

-What would be a good way to optimize my streaming PC data flow.

-What would be a good amount of RAM for the middle gaming server?

-How should I connect my network together and how many VLANS should I use.

- should I use multiple ip skemes since atleast three VMs will be mainly Public facing.

 

 

[Needfuldoer]

Those old Nehalem servers chug power, I'd build one nice chassis out of the available parts. (They're not as hungry as their Core 2 predecessors, but I bet you're not going to idle much lower than 200 watts before drives.) I think the stock RAID controllers for it top out at 2 TB per slot, too.

 

You'll want a single slot GPU if you're going to do any Plex transcoding. (Remember this requires a Plex Pass subscription now, too. Jellyfin, an Emby fork, doesn't have this limitation.)

 

Don't worry about VLANs when you've only got a couple devices.

 

Any services you run and want available to the wide open Internet should be exposed by port forwarding on your router.

[LIGISTX]

10 minutes ago, Needfuldoer said:

You'll want a single slot GPU if you're going to do any Plex transcoding

Not necessarily. You can do a fair bit of transcode just on CPU, just depends on the amount of transcodes needed at once, and the bitrate. My old homelab was a i3 6100 and I gave my Plex VM 2 threads and it would do a single blue ray 1080p to 3mbps 720, or multiple already transcoded 1080p’s down to 720. My current homelab with 8 threads does a single blue ray 1080p down to 720 with a 7-8x transcode speed…

 

@GangGangKiwivlans arw great, but you will need a router/firewall that supports vlans, and a hypervisor that does (most do). I run proxmox now and love it, but I ran ESXi 6.5 previously and it is fine. I think proxmox is much nicer, but both will work fine. Also…. Proxmox is free, which is a major plus. I’d recommend looking into it before you get too far down the ESXi path. 

[GangGangKiwi]

32 minutes ago, Needfuldoer said:

I think the stock RAID controllers for it top out at 2 TB per slot, too.

To answer the issue with the raid cards Ive upgrade the top server and bottom one with H700 raid cards to support larger capacity drives. The other thing you said about power is something I've thot about all three servers are running 710 watt 60plus platinum power supplies, also as power is very cheap where I live it's not an issue running these systems all the time maybe see a $10 bump in my electric bill a month running these over the last few months.

19 minutes ago, LIGISTX said:

GangGangKiwivlans arw great, but you will need a router/firewall that supports vlans, and a hypervisor that does (most do

I run a 25watt nivida gpu for the transcoding since I do have alot of 4k content on my system. Also I was hoping I could run VLANS with pfsense since I do have multiple ports on my pfsense box I built. Though was having mass issues getting it setup. 

[GangGangKiwi]

Also I definitely need a KVM soon I think I may be able to source one from work that is old. Also I should probably invest in a rack to house all of this equipment soon. 

[GangGangKiwi]

3 minutes ago, GangGangKiwi said:

Also I definitely need a KVM soon I think I may be able to source one from work that is old. Also I should probably invest in a rack to house all of this equipment soon. 

Is there a good place to find reasonably priced replacement batteries for a rack UPS ?

[Needfuldoer]

1 hour ago, GangGangKiwi said:

Is there a good place to find reasonably priced replacement batteries for a rack UPS ?

They're typically just some off the shelf SLA batteries in an array. If you can get the old pack apart, rebuilding it with new batteries will be less expensive than sourcing even a third party rebuilt battery.

 

1 hour ago, LIGISTX said:

Not necessarily. You can do a fair bit of transcode just on CPU, just depends on the amount of transcodes needed at once, and the bitrate. My old homelab was a i3 6100 and I gave my Plex VM 2 threads and it would do a single blue ray 1080p to 3mbps 720, or multiple already transcoded 1080p’s down to 720.

 

Your i3 was substantially faster than their Xeons (per thread), and it supported Quick Sync.

[LIGISTX]

1 hour ago, GangGangKiwi said:

To answer the issue with the raid cards Ive upgrade the top server and bottom one with H700 raid cards to support larger capacity drives. The other thing you said about power is something I've thot about all three servers are running 710 watt 60plus platinum power supplies, also as power is very cheap where I live it's not an issue running these systems all the time maybe see a $10 bump in my electric bill a month running these over the last few months.

I run a 25watt nivida gpu for the transcoding since I do have alot of 4k content on my system. Also I was hoping I could run VLANS with pfsense since I do have multiple ports on my pfsense box I built. Though was having mass issues getting it setup. 

Why not go software RAID..? So much better IMO. Truenas is hard to beat cuz ZFS is fantastic. 
 

Don’t use multiple ports on your pfsense machine. Use a single port, and pipe all VLANs over that. I have ~7 vlans all going out over a single LAN port (I have a quad NIC, literally only use 2 of the ports, one for WAN one for LAN and VLANs). 

[LIGISTX]

35 minutes ago, Needfuldoer said:

Your i3 was substantially faster than their Xeons (per thread), and it supported Quick Sync.

True, but a buddy of mine had a dual socket 1366 based xeon with low power xeons and that supported “enough” transcode power as well. But I suppose it just depends on use case. 

[GangGangKiwi]

20 hours ago, LIGISTX said:

Why not go software RAID..? So much better IMO. Truenas is hard to beat cuz ZFS is fantastic

I use a hardware card to initiate the disks on the system you kinda have to use a raid card in these types of serves as there is not data ports it's all sas connections on the inside. But I use truenas(freenas) for both my SSD array and cold storage vault array.

20 hours ago, LIGISTX said:

Don’t use multiple ports on your pfsense machine. Use a single port

I'll try that next just need to get some sfp to Ethernet adapter connectors first before I can try that out fully. Currently has to rotate back to my old router for now. 

[LIGISTX]

5 minutes ago, GangGangKiwi said:

I use a hardware card to initiate the disks on the system you kinda have to use a raid card in these types of serves as there is not data ports it's all sas connections on the inside. But I use truenas(freenas) for both my SSD array and cold storage vault array

All of my drives use SAS to SATA breaker cables…. From an LSI RAID controller flashed to IT mode.

 

Figure out if you can flash your card into IT mode, then you can use ZFS. Basically you just use the raid card as a way to plug things into. This is pretty common. 

[LIGISTX]

@GangGangKiwiguess you can’t flash that to IT mode…

 

https://forums.servethehome.com/index.php?threads/dell-h700-cross-flashed-to-it-mode.12130/

 

May actually be worth finding a “lower” tier card that can be flashed, as I’m pretty sure plenty do support large drives. I am not sure which support what size of drive, but that is certainly a normal thing homelab folks do. My card and my buddies are both in IT mode, and mine supports 10TB seeing as I have a 10TB plugged in. I don’t know if there is even a limitation being in IT mode? It passes it through as a block device to the OS (this may be incorrect verbiage here) so I would think as long as ZFS can see it it can control it. 
 

Either way, my H310 in IT mode I can personally say supports 10TB drives in Freenas. And used H310’s are pretty dang cheap. 

[GangGangKiwi]

I'll have to look into that. Well this afternoon is a a sad day ESXI just shat the bed sooo guess I'm wiping my middle and bottom servers and starting over tonight using proxmox. Probably should've used proxmox to begin with just thot since it's Dell servers and ESXI is a subset of fell maybe it would work better. I was wrong luckily all the SSD arrays data isnt worth anything just games and a some stream footage I backed up already.