Official Project Personal Datacentre Build for 2020/2021...
Okay - there's a new option for if I can't get the LGA2011-0 Xeons (E5-2600 v2's) on the table. It's cheap, scalable, and works with a lot of parts that I already own (including a pair of LGA1567 Xeons that I got for under 20 USD). The only downsides are that I can't use 3.5-inch HDDs with it, and that it's from HP/HPE. @Ithanul summoning you for your opinion on this machine - while it appears to be much better than the last option I saw from them (especially for under 300 USD barebones), still may have other issues to contend with. Yes, it's a server - I can handle the noise. This is meant to act as a replacement for my current workstation (in addition to other machines/roles). Currently a part of Project Datacentre Evolution...
This is the final plan - no longer an if/conditional. And found a way to get the 3.5 inch HDDs working with the server. Project New Client is still active dead, as I wait for a chance to get my hands on the new Framework 16 and has even had purchases put toward it recently.
PLAN 0 :: BETA
HPE ProLiant DL580 G7
OS :: VMware ESXi 6.5u3 Enterprise Plus
CPU :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
RAM :: 256GB (64x4GB) PC3-10600R DDR3-1333 ECC
PCIe :: 1x HP 512843-001/591196-001 System I/O board +
1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board
GPU :: 1x nVIDIA GeForce GTX Titan Xp +
1x AMD FirePro S9300 x2 (2x "AMD Radeon Fury X's")
SFX :: 1x Creative Sound Blaster Audigy Rx
NIC :: 1x HPE NC524SFP (489892-B21) +
2x Silicom PE310G4SPI9L-XR-CX3's
STR :: 1x HP Smart Array P410i Controller (integrated) +
1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) +
4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
1x Western Digital WD Blue 3D NAND 500GB SSD +
1x Intel 320 Series SSDSA2CW600G3 600GB SSD +
1x Seagate Video ST500VT003 500GB HDD
STR :: 1x LSI SAS 9201-16e HBA SAS card +
1x Mini-SAS SFF-8088 cable +
1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) +
4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
I/O :: 1x Inateck KU8212 (USB 3.2) +
1x Logitech K845 (Cherry MX Blue) +
1x Dell MS819 Wired Mouse
1x Sonnet Allegro USB3-PRO-4P10-E (USB 3.X) +
1x LG WH16NS40 BD-RE ODD
PRP :: 1x Samsung ViewFinity S70A UHD 32" (S32A700)
1x Sony Optiarc BluRay drive
PSU :: 4x HP 1200W PSUs (441830-001/438203-001)
The details for the ProLiant DL380 Gen9 will appear here once data migration is complete.
The software configuration details I posted last time have changed significantly. Most of these services will be sitting behind a VPN, for security purposes:
VMware vCSA v6.7
Managing and monitoring ESXi 6.5
Windows Server 2019
Active Directory, AD CS, Azure AD Connect, DnsTube, MariaDB, hMailServer, MailStore Server, SoftEther VPN, ManageEngine Endpoint Central, EaseUS Backup Center, etc.
Artix OpenRC (Xfce)
Samba, Docker+Compose {PortainerBE, NGINX Proxy Manager, Wazuh XDR, ClamAV, OnlyOfficeEE, MeshCentral, IT-tools, Nextcloud, OSMR, etc.}
Project:ArcZ,0 (Coming Soon)
Samba, PodMan+Compose {PortainerBE, NGINX Proxy Manager, Wazuh XDR, ClamAV, OnlyOfficeEE, MeshCentral, IT-tools}
Project:ArcZ,1 (Coming Soon)
Samba, PodMan+Compose {PortainerBE, OSRM, Nextcloud}
FreePBX Distro
Dedicated PBX instance (CentOS-based)
macOS Monterey 12.7.X
SyncThing, PleX Media Server (Movie/Show/Music streaming), Tautulli, Radarr+Sonarr, iTunes Sync, AirMessage, Time Machine server, etc.
Windows 10 Enterprise LTSC (2021)
Moonlight Gamestreaming, DaVinci Resolve, Console Emulators, etc.
On the other hand, there are certain tools that most VMs will possibly be equipped with. Some of those tools include:
- UnGoogled-Chromium + KeePassXC
- MeshCentral agent
- MangeEngine Agent
- Wazuh agent
VMware Horizon (VDI) will have to wait for a future phase -- assuming that Horizon is still an option. The current state of self-hosted VDI is Windows-centric, with second-hand support for Linux and no proper support for macOS.
Resource allocations for each VM can be found here. Details on networking and storage appliances used can be found here. Previous product links and details can be found here. Many of the changes intended for the Gen9 are trickling down to the G7.
A small amount of thin provisioning/overbooking (RAM only) won't hurt. macOS and Linux would have gotten Radeon Pro/FirePro cards initially (for best compatibility/stability) but market forces originally prevented this. Windows 10 gets the Audigy Rx and a Titan Xp. The macOS and Linux VMs get whatever audio the FirePro S9300 x2 can provide. The whole purpose of Nextcloud is to phase out the use of Google Drive/Photos, iCloud, Box.com, and other externally-hosted cloud services (Mega can stay though). The reason for why I'm doing this.
Other Project Mirrors:
Edited by TopHatProductions115
Kicked dev/testing and FAH to dedicated Linux VM, No longer pursuing Incus, added MailStore Server...
- MultiGamerClub, WereCat and n4vn33t
-
3
-
-
@Windows7ge In due time, I will switch over to the 4XXX Xeons. Starting with a pair of 2XXX Xeons will allow me to keep the initial cost(s) low until I can afford to purchase more cores...
-
If you know you'll have a use for the CPUs afterwords then that's an option I suppose.
-
On a side note, just caught wind of this:
Kinda needed to see that, seeing that I am now considering ditching the idea of getting two Vega 64's if I can just grab a single SR-IOV card instead and share it between MacOS and Linux. Not like I'll be using them for extreme gaming tasks, so I should be able to save some money by doing this. Current AIB options include:Would also free up some PCI-e slots for other devices that I could use - maybe leaving room for a better capture card in the future?
- https://www.manualshelf.com/manual/hp/hp-proliant-dl580-g7-server/implementing-sr-iov-on-hp-proliant-servers-with-vmware-vsphere-51/page-5.html
- http://h17007.www1.hpe.com/docs/iss/proliant_uefi/UEFI_TM_030617/s_enable_SR-IOV.html
- https://www.juniper.net/documentation/en_US/vmx/topics/task/configuration/vmx-sriov-enabling-vmware.html
-
SR-IOV is something that got brought up to me on the level1techs forum (yes I'm cheating on the linustechtips forum) as a way to divide my multiport 10Gbit NIC between my host and VFIO VM I created and passed though a GPU to and got Looking Glass working on.
I have not yet explored SR-IOV for NICs but the issue is being on the same card they belong to the same IOMMU group of which when you pass a device though to a VM the system really doesn't like splitting IOMMU group devices. That's where SR-IOV would come in. For now I've just created a network bridge in Linux and attached the VM to the bridge. Let's me pass data between the virtual and physical network at 10Gbit. I may move forward with SR-IOV I may not because setup like this if I make more VMs on my desktop I can have them all share the 10Gbit NIC simultaneously.
Depending on your use case SR-IOV is definitely a route you can go. Do as much research as you can that it'll work on the hardware and OS you plan to use. I don't know about vmware/ESXi.
-
-
-
@Windows7ge From what I'm seeing online, consumer Vega doesn't have it. Which leaves server/enterprise cards that are 2-3 years old - and still going for over 1k USD used.
-
It's actually because of VFIO & Looking Glass that I'm able to move away from Windows and just use it in a high performance VM for the few applications I still need it for.
If you don't need the VM's for anything super graphically stressful or GPU acceleration, etc then yeah go for the FirePros if you can find them for a half decent price. That extra benefit here is as you mentioned with the DL580 G7 is that it's scalable. Need to host 20 more VMs? Pop in 1 or 2 more FirePros & spin them up.
What you could do is use the FirePros to consolidate the VMs that need some graphics power but not excessive amount then just have one up to date Vega GPU to give to a VM that needs a lot of power.
-
Just eliminated one of the FirePro options, due to relatively low performance-to-cost. The s7150 X2 was barely better than a Fermi mid-range GPU, but sellers were demanding upwards of ~1k USD for it. Probably solely for the VRAM size (2x8GB GDDR5). While the FirePro that I left on the list comes with only (2x4GB HBM), the memory bandwidth is much higher. Also consumes less power, and is way cheaper (~300 USD average). On a lighter note, I've picked up a pair of Intel Xeon E7-8870's - so, here's to hoping that I can go full S8 in the years to come :3
-
I don't even know where you can find 8 socket boards but each individual CPU can handle up to 4TB of RAM so if you need a lot of that
.
-
Just got my second E7-2870 in the mail today. Also ordered a 3rd E7-8870, in anticipation of the upcoming purchase later this year (DL580 G7). Might pick up some more RAM while I'm at it, to keep the DIMM slots filled
Still deciding on whether to switch every Windows machine I have to Enterprise LTSB/LTSC...
-
Nice. I'm currently in limbo waiting on NEMIX to return my request for RAM replacement. They sent me the wrong stuff. I'll post a Status Update about it tomorrow.
Eh, depending on what you're doing with them I don't know how worth it that'd be. That'd be some costly software that could have been spent on more hardware.
-
Okay - just ordered my 4th E7-8770 for 15 USD (Best Offer is nice), and am waiting for one on another 24GB of RAM. If that goes through, the only thing(s) I'll need to get are the server itself and the PCIe risers. May consider getting the new sound card, to start transitioning my audio setup over (before starting with ESXi). Once I get that figured out, I'll be able to use it as a raw replacement for the T7500 until I get the GPU(s)...
-
-
@Windows7ge At prices like this, and with Best Offer available?
I remember 16GB sticks coming in at ~26 USD per unit when I last bought them.
-
-
@Windows7ge I'll considering switching to 8GB sticks when I get my next job. For now, I have no choice but to stick to 4GB sticks, so that I can still afford purchases scheduled for later this year. There is no guarantee that I'll be able to continue with this streak of serendipity that I've been riding on the last few days, so I have to play it safe...
-
-
-
-
-
My mouse is the only component of my computer that I haven't changed since I first built it back somewhere around 2008~2009. It's worn but still works perfectly.
-
-
It would be a good idea to backup the VM's and ESXi's config. Alternatively a disk-shelf and a RAID controller with external ports should let you extend the storage if you don't have enough internal bays.
.
Still deciding on whether to switch every Windows machine I have to Enterprise LTSB/LTSC...
