Questions about VM servers

[Donut417]

In a previous post I asked about Proxmox. I have since been watching all kinds of content from Craft_Computing, Techno Jim, Level 1 Techs and etc. I think I finally for the most part figured out what Id like to do. I know I want to run Plex, PiHole, Guacomole, my database for Bitwarden, possibly a private Discord server, and maybe some gaming servers (only on when I need them) possibly a VPN VM. I have a few questions. 

 

1.  Full VM vs a Container? How do I know what to choose? I seen Proxmox supports both VM's and containers. 
2.  How important is IPMI? Ive been looking for at Supermicro and ASROCK Rack boards. Many have multiple Ethernet  as well as a dedicated IPMI port. 
3. What do I need to look for in a CPU? How many cores / threads? 
4. Storage. How much do I need for Proxmox and the VM's potentially? Plex data will be stored on spinning rust. I do have a 500 Gig Samsung 850 EVO SATA SSD. 
5.  GPU. Most of the boards I have been looking at have on board Video via VGA. Lucky for me my TV still has the port. However for Plex I was going to get a Nvidia quadro. What GPU should I be looking at for up to 3 1080p transcodes or 2 1080p transcodes and 1 direct stream of 4K? 
6.  Ram. How much is a good start? Is ECC worth it? Craft_Computing made some statement that it might make the server more stable. 
7. Im going to take it if I want to tunnel in to my network from the outside Ill need a VPN server running? Also possibly some DDNS type of service as well. 
8. For PiHole how difficult is it going to be? My folks also use the this network. Am I going to be getting a lot of support requests due to websites not working due to Ad Blocking? LOL. I should only have to put the URL for the website in the white list to fix that issue? 

Im trying to get an idea so I can start really looking at hardware. 

[Electronics Wizardy]

1. I normally use a container if you can. Lots of things keep you from using containers, like kernel extntions, specific oses needed, and osme program support.

 

2. Are you on site? If your are, it doesn't really matter for home use, but if you wan to be off cie, you don't need much.

 

3. Really depedns on your exact uses, but you can probably run all of that from a older quadcore, from like haswell.

 

4. Really depends on the size of them. Id say norally 30-50gb per windows vm and 10-20gb per linux vm. A linux containers is like 1gb. Then add the amount of data those vms are store. 500gb is probalby fine to start with. Also snapshots are nice but need more space.

 

5. For only a few 1080p transcodes, id just use the cpu, even a basic cpu can transcode a few 1080p streams without issues.

 

6. Id probalby start at 32gb here. Really depends on how big you want to go. Id go ecc if the hardware supports it, nice to have, keeps data intact.

 

7. Yea if you want a l3 tunnel, thats what a vpn is. Wireguad is nice, fast and secure.

 

8. Pretty painless. Basically one line install, and set as a dns server. Id probably leave it off for other users as it causes weird issues.

[Donut417]

2 minutes ago, Electronics Wizardy said:

d probably leave it off for other users as it causes weird issues.

Well, Ill have to figure out how to change DNS servers on my Fire Stick. As well as hope that I can set per WiFI network IP settings on my MacBook as well as my iPhone. Which is why I was going to run it over the whole network. Plus Ive heard blocking ads can reduce data usage. 

 

5 minutes ago, Electronics Wizardy said:

Really depedns on your exact uses, but you can probably run all of that from a older quadcore, from like haswell.

I currently using an i5 3570K as my Plex server. While it works, it might not run all the services I want. Especially if I ever get a 4K TV/High res content. Im not sure what the requirements are for a 6 man Killing Floor 2 server. I now CS1.6 shouldn't require much, but not sure on Killing Floor. I dont really see a benefit upgrading from Ivybridge to Haswell. I found an Asrock Rack board that supported AM4 and 2 and 3rd gen Ryzen. Im currently running a 2600X I believe in my gaming rig. Might be an excuse to upgrade it. 

 

11 minutes ago, Electronics Wizardy said:

Are you on site? If your are, it doesn't really matter for home use, but if you wan to be off cie, you don't need much.

Im on site. But IF I ever decided to maybe run cables else where and put my server somewhere else, Ill need to invest in another monitor as I dont have any spares. The current server sits at my desk right now. But you never know what the future may hold. 

[Electronics Wizardy]

Just now, Donut417 said:

Well, Ill have to figure out how to change DNS servers on my Fire Stick. As well as hope that I can set per WiFI network IP settings on my MacBook as well as my iPhone. Which is why I was going to run it over the whole network. Plus Ive heard blocking ads can reduce data usage. 

 

DCHP tells it the DNS to use, so just change it in there.

 

1 minute ago, Donut417 said:

I currently using an i5 3570K as my Plex server. While it works, it might not run all the services I want. Especially if I ever get a 4K TV/High res content. Im not sure what the requirements are for a 6 man Killing Floor 2 server. I now CS1.6 shouldn't require much, but not sure on Killing Floor. I dont really see a benefit upgrading from Ivybridge to Haswell. I found an Asrock Rack board that supported AM4 and 2 and 3rd gen Ryzen. Im currently running a 2600X I believe in my gaming rig. Might be an excuse to upgrade it. 

I have a asrock rack x470 with a 3900x and it handles everything well, good board if you got the money. But im guessing that 3570k can do this if you set it up right

 

2 minutes ago, Donut417 said:

Im on site. But IF I ever decided to maybe run cables else where and put my server somewhere else, Ill need to invest in another monitor as I dont have any spares. The current server sits at my desk right now. But you never know what the future may hold. 

Normally you don't need impi for display out, and if you leave it running, remote power on doesn't really matter. But migth as well get it if your buying new parts.

[Windows7ge]

1. In an LXC container the OS instance runs off the host kernel and has direct access to hardware devices like disks for example. In compute applications an LXC container will have less overhead than full virtualization. Better perfomance. Outside of this I can't say there's an overwhelming reason to use one over the other.
2. That's a matter of personal desire. I haven't had much need to use it on my PROXMOX server but it's nice to have when performing a reboot, entering the BIOS, if something does go wrong you don't have to grab a keyboard and monitor and plug it into the server. Just an overall convenient tool.
3. That depends on the workload. If you plan to off-load PLEX to a GPU I can't see the rest of your needs requiring a very heavy CPU. Personally an 8C/16T is plenty of power for most anything you might like to do at home. I'd recommend either a Xeon or EPYC so you get good VT-d/IOMMU support.
4. If you run PROXMOX off it's own boot drive it doesn't use that much space. A 120GB drive is more than enough for the OS and a few dozen .ISO files, LXC images. Personally I'd setup PROXMOX on a mirror for redundancy but that's your call. For spinning-rust you just have to tell PROXMOX what storage pool you want to use when creating your CT/VM.
5. Gonna skip this one. Not my field of expertise.
6. Most of what you're looking to run probably won't use very much. You can over-provision RAM (use more than 50% of what the system has available) but I'd probably start with 32GB based on everything you're thinking of doing. I do recommend ECC with ZFS since it like every other file system assumes what's in memory is correct. If a bit flips and ZFS takes that from memory it could cause corruption/data loss. Depending on what type of system you build ECC doesn't have to be much more expensive than non-ECC.
7. Port Forwarding services can enable this but if you want full access to all your devices behind your router then yes. I have a tutorial on how you can do that if you're interested. You can use No-IP for Dynamic DNS but if your Public IPv4 or v6 is persistent you can use the raw IP too. Whatever you prefer.
8. PiHole is something I've been meaning to get on myself so I can't comment here either.

Are you looking for new | used | old hardware? Modern. last gen? Single/Dual socket? Tower/rack-mount? Budget? :3

[Donut417]

1 minute ago, Electronics Wizardy said:

DCHP tells it the DNS to use, so just change it in there.

No what you were saying is to leave my parents off of PiHole. The only way to do that, and the easiest way to do that would be to statically assign the DNS info to all my devices. Unless we have a family meeting and I just tell them to give me the URL's of the sites that are broken. 

 

6 minutes ago, Windows7ge said:

re you looking for new | used | old hardware? Modern. last gen? Single/Dual socket? Tower/rack-mount? Budget? :3

Not sure. Id be willing to get used if the price was right. I do have a 500 Gig SATA SSD. I have a Noctua NH-D14 if newer brackets are available. I haven't decided if Im going to stick with my Fractal Design R4. The PSU needs to be replaced, the fan has been failing as I think dust got in to the bearing. I currently have my Plex videos on my NAS (8TB). But Im looking to move my Plex storage to this server. My NAS has no redundancy, which Id like to change and store files there. One issue I have with my current Plex setup is that I have to physically download the Plex installer from their site every time there is an update. This is because I choose to use Network storage. The only reason I use Network storage is because my original server was on a Laptop. 

 

I do know its going to be in a Tower, either the R4 or something smaller? I need it to be quiet as possible as its in my bedroom. I do see myself doing much more than what I listed above, so I guess single vs dual will depend on price. Id also like to be somewhat power efficient. Dont want to draw 1000 wats from the wall. For the record I have been looking on Facebook Market Place to see if I find anything. So far not much. With the Covid going on Im wary about meeting up with people. 

 

Budget wise Im not sure. Prices are inflated. Im kinda trying to figure out what all this will cost right now. Id like to say under $1K USD if I can. I might use my Tax Return money, so its going to be a bit before I get that. 

 

 

10 minutes ago, Windows7ge said:

Public IPv4 or v6 is persistent you can use the raw IP too. Whatever you prefer.

I technically have a dynamic IP, I haven't paid attention to how much its changes. 

[Electronics Wizardy]

5 minutes ago, Donut417 said:

No what you were saying is to leave my parents off of PiHole. The only way to do that, and the easiest way to do that would be to statically assign the DNS info to all my devices. Unless we have a family meeting and I just tell them to give me the URL's of the sites that are broken. 

 

You can also setup groups in pihole, so they don't have to deal with pihole. Or make a seperate subnet for them.

 

6 minutes ago, Donut417 said:

Budget wise Im not sure. Prices are inflated. Im kinda trying to figure out what all this will cost right now. Id like to say under $1K USD if I can. I might use my Tax Return money, so its going to be a bit before I get that. 

 

 

22 minutes ago, Windows7ge said:

That budget is probalby enough for this board https://www.newegg.com/asrock-rack-x470d4u-amd-ryzen-2nd-generation-series-processors/p/N82E16813140023 and a 3600 + 32gb ddr4 ecc udimms. Then Id get a seperate boot ssd(like 16gb is fine here, makes it easier to deal with). And you can just move your hdds over.

[Donut417]

2 minutes ago, Electronics Wizardy said:

That budget is probalby enough for this board https://www.newegg.com/asrock-rack-x470d4u-amd-ryzen-2nd-generation-series-processors/p/N82E16813140023 and a 3600 + 32gb ddr4 ecc udimms. Then Id get a seperate boot ssd(like 16gb is fine here, makes it easier to deal with). And you can just move your hdds over.

Thats the board I was looking at. So what your saying is a small SSD for the Proxmox install and use my 500 Gig one for the running of the VM's? 

 

Now Ill have to make a decision on the spinning rust. I have a 8TB disk in my NAS that has some data on it as well as all my Plex videos. I think its formatted in EXT4. So I can pop that in OR should I buy a new disk for my Plex data. Either way I plan on buying some disks for the NAS at some point to add redundancy and actually maybe backup some of my machines properly. 

[Windows7ge]

3 minutes ago, Donut417 said:

Id be willing to get used if the price was right.

Right now you can pick up a Intel Xeon E5-2698v3 16C/32T CPUs on eBay for about $200~250. (This time I checked and verified they aren't bid prices.) This would give you basically infinite growing room. Only bringing up a 16C because of the price right now.

 

12 minutes ago, Donut417 said:

The PSU needs to be replaced, the fan has been failing as I think dust got in to the bearing.

I know you didn't ask but in case you're wondering a redundant server PSU isn't a necessity. They're VERY expensive and are honestly a rather niche need in a homelab. Despite that I do have one in one of my servers.

 

6 minutes ago, Donut417 said:

I do know its going to be in a Tower, either the R4 or something smaller? I need it to be quiet as possible as its in my bedroom.

And just like that retired rack-mount servers are off the table.

 

So standard hardware then. An ATX motherboard will offer you the most expansion options. The Xeon I mentioned is LGA2011-v3 "old" but very capable. I've worked with both ASRock Rack and Supermicro motherboards. I'd say both have their pros & cons but for the application either should do. Keeping it under 1K depends on weather you need the whole server for 1K or just the essential components (CPU/MoBo/RAM/PSU).

[Electronics Wizardy]

6 minutes ago, Donut417 said:

Thats the board I was looking at. So what your saying is a small SSD for the Proxmox install and use my 500 Gig one for the running of the VM's? 

 

Now Ill have to make a decision on the spinning rust. I have a 8TB disk in my NAS that has some data on it as well as all my Plex videos. I think its formatted in EXT4. So I can pop that in OR should I buy a new disk for my Plex data. Either way I plan on buying some disks for the NAS at some point to add redundancy and actually maybe backup some of my machines properly. 

I like to keep the boot drive seperate. You really don't need much. I use old laptop hdds as boot drive without issues.

 

Yea you can just pop in your current nas drive, I don't see a reason to change it for now. Having multiple disks in zfs would be a nice upgrade though

[Donut417]

1 minute ago, Windows7ge said:

o standard hardware then. An ATX motherboard will offer you the most expansion options. The Xeon I mentioned is LGA2011-v3 "old" but very capable. I've worked with both ASRock Rack and Supermicro motherboards. I'd say both have their pros & cons but for the application either should do. Keeping it under 1K depends on weather you need the whole server for 1K or just the essential components (CPU/MoBo/RAM/PSU).

Im probably just going to use my R4 case. I have the SSD like I said. I dont need a lot of hard disks, 1 higher capacity drive will be enough for Plex as its a DVR for the most part. If my Noctua CPU cooler (NH-D14) can be converted to work Ill probably continue to use that. 

 

Just now, Electronics Wizardy said:

Yea you can just pop in your current nas drive, I don't see a reason to change it for now. Having multiple disks in zfs would be a nice upgrade though

I dont need a lot of "Storage" Plex is only used for its DVR capabilities because Im frankly too lazy to rip my DVD's. Plus I dont really see the point in ripping DVD's when they are pretty much dead at this point. I dont own a Bluray drive so I can rip the few Blurays I have. Plus with Streaming services I can find stuff to watch. I mainly use Plex to share the OTA antenna in my room to the whole house. Plus to record my shows that are on OTA TV so I dont have to waste some of my 1.2 TB of data I get monthly with Comcast. My dad likes to channels surf a bit and my mom sometimes likes to have the TV on for the noise. As such I dont care if the Plex drive has redundancy. Ill still be using my QNAP NAS for mass storage for the time being and eventually I get 4 drives in it so I can have redundancy.  

[Windows7ge]

4 minutes ago, Donut417 said:

Im probably just going to use my R4 case. I have the SSD like I said. I dont need a lot of hard disks, 1 higher capacity drive will be enough for Plex as its a DVR for the most part. If my Noctua CPU cooler (NH-D14) can be converted to work Ill probably continue to use that. 

You might like the Supermicro X10SRL-F it's essentially a single socket variant of the dual socket board in one of my servers and would offer a lot more future expansion than going AM4.

 

I should ask. How extensive is your knowledge of ZFS. If you plan to upgrade the pool as you go you can't expand a singular disk in the traditional sense...(or rather you can but you really don't want to)

[Donut417]

1 minute ago, Windows7ge said:

should ask. How extensive is your knowledge of ZFS. If you plan to upgrade the pool as you go you can't expand a singular disk in the traditional sense...(or rather you can but you really don't want to)

0 knowledge. Ive never used ZFS. 

[metaleggman]

1 hour ago, Donut417 said:

For PiHole how difficult is it going to be? My folks also use the this network. Am I going to be getting a lot of support requests due to websites not working due to Ad Blocking? LOL. I should only have to put the URL for the website in the white list to fix that issue? 

Not sure if this has been tackled yet, but if you use the stock blacklists, you most definitely will.  They're very strict, to the point of at times blocking things like Facebook's GIF search functionality, or letting Windows update.  I'd suggest an automatic white list like this.  Since there's a lot of CDNs that are blocked by default, this helps allow ones that aren't ad related.

[Donut417]

4 minutes ago, Windows7ge said:

 Supermicro X10SRL-F 

I think Id be able to use my current cooler as well. Because I recalled a Socket 2011 version. As Amazon sent me that by mistake at first. Im pretty sure the only difference was the bracket. 

 

Keep in mind Im also trying to be somewhat power efficient. My room and mom's room are are the same circuit. 15 amps. We both use Window Air conditioners during the summer. So I do have a bit of limits to what I can do. How will socket 2011 chips compare in power usage to a Ryzen CPU? Don't need my mom bitching about the power bill or that I burned the house down. We already had the garage burned down, dont need that happening to the house. 

[Donut417]

5 minutes ago, metaleggman said:

Not sure if this has been tackled yet, but if you use the stock blacklists, you most definitely will.  They're very strict, to the point of at times blocking things like Facebook's GIF search functionality, or letting Windows update.  I'd suggest an automatic white list like this.  Since there's a lot of CDNs that are blocked by default, this helps allow ones that aren't ad related.

Thanks for the heads up. If it screws with Windows update that would be kinda a blessing. I keep my Windows 10 install not current for a reason. See too much bad stuff being released. I basically wait till Microsoft forces me to upgrade before I upgrade. 

[Windows7ge]

2 minutes ago, Donut417 said:

0 knowledge. Ive never used ZFS. 

I recommend doing A LOT of reading. Seriously, ZFS is not a budget friendly file system. What I mean by that is single disk addition to existing pools isn't a function. Supposedly it's in the works but I haven't heard anything and who knows if the PROXMOX devs will ever implement it in an update.

 

How it works is when you first form your pool be it 1,2,3,4,5 whatever drives regardless of RAID level (RAID1/5/6/"7") this forms a singular vdev. Data is stripped across vdevs. If you create a pool of one disk with plans to add another down the road when you go to add it it will create a RAID0 across the disks as two independent vdevs.

 

For this reason each vdev requires it's own redundancy. A mirror, RAID5, or 6. Then to expand the pool you need to append another mirror, RAID5, or 6.

 

Now it is possible to mix & match different RAID levels across vdevs but I don't recommend it.

 

So to start off your spinning rust for bulk storage of your CT's & VM's you're going to need two drives at least if you ever plan on expanding that pool otherwise losing everything is one one disk failure away (the disk that's alone in it's own vdev).

 

5 minutes ago, Donut417 said:

I think Id be able to use my current cooler as well. Because I recalled a Socket 2011 version. As Amazon sent me that by mistake at first. Im pretty sure the only difference was the bracket. 

 

Keep in mind Im also trying to be somewhat power efficient. My room and mom's room are are the same circuit. 15 amps. We both use Window Air conditioners during the summer. So I do have a bit of limits to what I can do. How will socket 2011 chips compare in power usage to a Ryzen CPU? Don't need my mom bitching about the power bill or that I burned the house down. We already had the garage burned down, dont need that happening to the house. 

It uses the Narrow-ILM mounting bracket as oppose to the traditional Square-ILM.

 

Idle power consumption isn't bad really. Part of the function of Xeons compared to desktop counter parts is power efficiency as that is a major data-centre concern. For a highly parallel workload performance per watt will be better. My own server running two of those Xeons & a whole wack-ton of RAM, 20 SSD's, HBA's, fans, everything idles around 200W from the wall.

 

Cut out most of what you wouldn't have compared to me and it should idle somewhere around 100W or less.

[metaleggman]

39 minutes ago, Donut417 said:

Thanks for the heads up. If it screws with Windows update that would be kinda a blessing. I keep my Windows 10 install not current for a reason. See too much bad stuff being released. I basically wait till Microsoft forces me to upgrade before I upgrade. 

Haha, that's fair.  I feel like the only way to not get fucked over by Windows updates is either to only go when you're forced to, or be on the dog food Insider builds like I am.

[Donut417]

18 hours ago, Windows7ge said:

I recommend doing A LOT of reading. Seriously, ZFS is not a budget friendly file system.

Thanks for the heads up. Hopefully in the next week Ill have time. I was just thinking that Ill likely need storage for more than Plex data anyway. Probably will want to keep ISO's on that storage as well. Do you have any recommended sites to do reading? 

 

18 hours ago, Windows7ge said:

Cut out most of what you wouldn't have compared to me and it should idle somewhere around 100W or less.

My current setup sips 129 watts at idle. LOL. 

 

 

[Windows7ge]

3 hours ago, Donut417 said:

Thanks for the heads up. Hopefully in the next week Ill have time. I was just thinking that Ill likely need storage for more than Plex data anyway. Probably will want to keep ISO's on that storage as well. Do you have any recommended sites to do reading? 

If you use a large enough boot disk you can keep all your .ISO files on there. .ISO files not related to VM's yeah they can go elsewhere.

 

Something you might find really cool is you can mount a CIFS/NFS share as a storage point for Disk images, .ISO files, VM/CT backups, Container templates, etc. So if you have another server on the network you can point PROXMOX to it to save some information there.

 

As for reading material I'm afraid I'm not the best person to ask. I spent years using a very "learn as you go" mentality. As I wanted to understand certain functions/features I Googled it and read from whatever the first so many results were. For ZFS commands though I do keep finding myself going back to docs.oracle.com. They have a very nice collection of commands for various tasks you may be trying to perform.

 

The three big things about ZFS that you want to keep in mind:

1. Don't make a pool consisting of one disk if you want to expand it later. Each vdev needs it's own redundancy. Single disk addition isn't a thing.
2. A SLOG (ZIL device) will NOT accelerate write operations for 99% of use cases (only works for synchronous write operations). VM's can benefit but I've never validated that adding one on PROXMOX makes any difference.
   1. If you do use one make sure it's a server grade SSD. Consumer grade drives won't guarantee the data is safe after transit.
3. Data in ARC memory is ejectable. If you see your RAM being saturated check what's in ARC. If most of it is there everything is fine. ZFS will remove data from ARC as the system requires more for other processes. ARC is a RAM read cache. Good for SMB/NFS etc.

3 hours ago, Donut417 said:

My current setup sips 129 watts at idle. LOL. 

You should be in/around the same ball-park. Modern Xeon based systems have a much lower idle power consumption that Xeons of older generations like the ever popular X5650 era. This with the Supermicro board would give you much higher room to grow without a platform swap for only a few more dollars.

 

I don't have anything against AM4, hell I'm team purple (red+blue) but your plans and thoughts of future plans warrant something with more wiggle room. Long term you won't find yourself out of PCI_e slots or realize a max of 128GB of RAM isn't enough. (I have hit both of these issues myself with PROXMOX).

 

Up to you though I'll back whatever decision you make. If you need help with PROXMOX/ZFS feel free to hollar.

 

[Donut417]

On 2/18/2021 at 9:45 PM, Windows7ge said:

Up to you though I'll back whatever decision you make.

Still trying to figure out my disk situation. I do have an 8TB that is currently in my NAS. It was a shucked drive from WD Easystore. So Im not sure If Im going to use that and get another 8TB disk, or just buy some different disks. 

 

Also do I need Enterprise grade SSD's to install Proxmox on and the VM's, or will consumer work? I dont think Im going to do a SLOG. 

 

 

Third, I was on eBay and found a Supermicro X10SRH-CLNF4 + 64GB ECC REG DDR4 + Xeon E5-1660v3 CPU with 2U Heatsink (Might need to replace due to noise). All for $559 USD + $20 shipping. Seller has a 99% positive rating. 

 

OH and for storage, if on board SATA fine, or should I consider an HBA? 

[Windows7ge]

21 minutes ago, Donut417 said:

So Im not sure If Im going to use that and get another 8TB disk, or just buy some different disks.

Would the plan be RAID1? I'd use the disk you have shucked and just buy the other. Just make sure you buy one of matching RPM/capacity. Cache size may not matter much.

 

Oh, this is another important thing about ZFS. Avoid SMR (Shingle Magnetic Recording) drives like they're the plague. ZFS absolutely hates them. Make sure disks you buy are CMR.

 

25 minutes ago, Donut417 said:

Also do I need Enterprise grade SSD's to install Proxmox on and the VM's, or will consumer work? I dont think Im going to do a SLOG.

Consumer will do the job but keep the server on a UPS. I setup my PROXMOX install on a mirror of 128GB Intel 545 drives. For the storage of your VMs I'd recommend a RAID10 pool on spinning rust. With RAID10 you get significantly higher IOPS compared to parity RAID at the cost of 50% of the usable storage.

 

If you ever find yourself short on RAM you can look into a L2ARC. When RAM is busy this acts as a slower read cache to RAM but faster than spinning rust. Recommend using a PCI_e SSD. There are server grade SSDs that would be appropriate for the application.

 

38 minutes ago, Donut417 said:

Third, I was on eBay and found a Supermicro X10SRH-CLNF4 + 64GB ECC REG DDR4 + Xeon E5-1660v3 CPU with 2U Heatsink (Might need to replace due to noise). All for $559 USD + $20 shipping. Seller has a 99% positive rating. 

 

OH and for storage, if on board SATA fine, or should I consider an HBA? 

That setup could work. At least the socket would give you growing potential and it has more PCI_e slots.

 

I would avoid the ports controlled by the Broadcom 3008 SW controller. Does it support JBOD? Possibly. I don't know. ZFS is great for not caring about what controllers your disks are connected to but it does like direct access to the drives which RAID controllers can (but don't always) interfere with. I would take full advantage of those two SFF-8643 ports though. All you need are some breakout cables.

 

If down the road you need more ports I can recommend both the LSI 9207-8i & LSI 9201-16i.

 

There are SFF-8643 variants if you need 16xSASII/SATAIII ports in a 2U form factor.

[Donut417]

7 hours ago, Windows7ge said:

For the storage of your VMs I'd recommend a RAID10 pool on spinning rust. With RAID10 you get significantly higher IOPS compared to parity RAID at the cost of 50% of the usable storage.

Planning on putting those on SSDs. Spinning Rust is literally for videos and maybe ISOs. 

 

7 hours ago, Windows7ge said:

but keep the server on a UPS

That also presents an issue. My UPSs USB is connected to my NAS. So I would need a way to figure out how to do a shut down signal for both. My NAS has an option to rely it over the network. But I never got it to work.

[Windows7ge]

2 hours ago, Donut417 said:

Planning on putting those on SSDs. Spinning Rust is literally for videos and maybe ISOs. 

However you want to do it. I'm just sharing my own setup basically. I would like to swap out the mechanical VM pool for all Solid State but 16TB worth of server grade SSDs would be a little costly.

 

2 hours ago, Donut417 said:

That also presents an issue. My UPSs USB is connected to my NAS. So I would need a way to figure out how to do a shut down signal for both. My NAS has an option to rely it over the network. But I never got it to work.

This is an issue I have not tackled either. I haven't noticed or researched any function in the PROXMOX UI that let's you set that up. It isn't plug'n'play I can tell you that but a UPS without control is better than no UPS at all.