First home lab advice

[techfan84]

Budget (including currency): 5-10k USD

Country: Australia 

Games, programs or workloads that it will be used for: Machine learning, modelling, data science, personal cloud storage, some gaming

 

Hi All,

 

I am building my first home lab/server and would greatly appreciate the advice/insights of this community. I have been researching this for a while but keep finding conflicting information regarding parts and am feeling a bit lost.

I have a preferred budget of around 5000 USD. Obviously if the costs can be reduced that would be preferable but not sure how feasible that is given my requirements. As you will see below, my currently specified build is significantly higher than this. If I have to outlay this and it is actually cost effective to do so then I can proceed with this, but I'd really like some advice as to whether I have made any mistakes or errors in the pricing and part selection. I live in Australia so some parts are header to obtain online or as second hand. 

 

Here is an overall summary of my currently specified build: 

 

 

Build and requirements summary:

I currently have a router from my ISP with a home broadband internet plan. I can connect to this router wirelessly throughout my home from eg. my M1 Macbook Pro (which I do most of my coding and work from), my iPhone, etc. I want to deploy infrastructure in two rooms: a work room and a server room. Both rooms each have a single ethernet connection socket in the wall that connects to the router. 

In the server room I want to connect the ethernet port to a network switch. I will then store two main servers in this room (at first...) along with a UPS. The first server is a Computation Server and the second server is a Storage Server. 

 

The Computation Server is for running a range of projects eg. Machine Learning modelling, data science projects, some gaming etc. These will be run by a Proxmox hypervisor with various virtual machines being operated. Most projects will be in linux but the gaming for example will require a Windows virtual machine. 

The Storage Server will run ZFS with one zpool consisting of multiple RAIDZ2 vdevs each containing 5x 18TB HDDs for a total of around 50TB of usable storage each. To upgrade the storage I will need to create a new vdev each time and add five more 18TB HDDs to boost the storage in increments of ~50TB. To accomodate for these expansions I want a RAID card/HBA that accommodates ~25 drives (either directly or by using a SAS expander). A case that incorporates enough hot swappable HDD bays would be ideal. 

 

I want a simple monitor and keyboard in the server room to debug and setup the servers. This can use a basic KVM switch so I can do this for all machines, even those without IPMI support. In the work room I want to connect a mini computer connected to a keyboard and dual monitor that gives me another access point to my servers. I think I want to connect this directly to the ethernet port in this room and then use remote desktop software to interact with my virtual machines over the network. I would like this link to be low latency and high resolution to enable some gaming as well as general interaction with the VMs. I mostly work and code from my Macbook and would like to primarily connect to the servers via wifi from there. The Computation Server will have access to the Storage Server over the network via the network switch using ZFS datasets and SMB or NFS shares. My server motherboards should support IPMI for remote setup/control.

 

I would like to be able to remotely access my home lab from other locations eg. whilst travelling. This would be through a VPN (using a mini computer with OpenVPN) and multi factor authentication (eg. Google Authenticator) for safety. I would like seperate VLANs for my home network web usage, my critical home server hardware, etc for added safety.  I would like to expose specific services on my server to the public/easy access for myself via my domain name and Dynamic DNS.

 

Considering these requirements and my available storage space, I would like to build this in a rack system. A balance between performance, future upgradeability, cost, and power consumption (to a lesser extent, see below) is important. 

 

Regarding power consumption and associated costs. I have a significant solar array at home and a Tesla powerwall. I expect this to significantly reduce electricity costs as I am currently not using much of the generated power. Here are the key specs of the power system:

 

The peak power output of my solar array is 6kW. The yearly total generated is roughly 11.5MWh which gives an average of 31kWh per day. I have a Tesla Powerwall battery with a capacity of 13.5kWh. The yearly usage is about 4MWh which is 11 kWh per day. This is currently 99% provided by the panels and battery with 1% coming from the grid. I have no other renewable energy sources. 

According to my calculations, the main area it fails is if you use >900W permanently and the issue is during the night since the Powerwall can't last all night during winter at that power draw. So we should treat 900W as the max allowed continuous power consumption. If I do require more than this I can use activate systems to adaptively control the resource consumption of the VMs to balance the load. This can change through the day and year. 

 

Summary:

 

As you can see this proposed build is significantly more expensive than my preferred budget. I would be very grateful to get any feedback/advice on this. Do you see major problems with these component choices or do you suggest alternatives? Can you see ways that I could get similar performance/capabilities but with a significantly reduced cost (it is quite terrible currently!). Or is this pricing just about right for this level of capability? 

Thanks for your time and insights! 

 

 

 

 

 

 

 

 

[jaslion]

1 hour ago, techfan84 said:

The Computation Server is for running a range of projects eg. Machine Learning modelling, data science projects, some gaming etc. These will be run by a Proxmox hypervisor with various virtual machines being operated. Most projects will be in linux but the gaming for example will require a

Just get a regular desktop with a used 3090?  Zen and Zen + epyc's suck at single core tasks and tasks that leap over from ccx's. Like for decent performance amd RECOMMENDS turning off cores and ccx's. For your tasks this cpu will be straight up bad.

 

Its also miserable for gaming. Basically a i7 4770 sigle core but it stutters like mad.

 

1 hour ago, techfan84 said:

Storage Server will run ZFS with one zpool consisting of multiple RAIDZ2 vdevs each containing 5x 18TB HDDs for a total of around 50TB of usable storage each. To upgrade the storage I will need to create a new vdev each time and add five more 18TB HDDs to boost the storage in increments of ~50TB. To accomodate for these expansions I want a RAID card/HBA that accommodates ~25 drives (either directly or by using a SAS expander). A case that incorporates enough hot swappable HDD bays would be ideal. 

 

Do you NEED this? What is the timeline you have for filling up 50tb? What are your data cleaning policies? This will have a massive impact on the build.

 

Because w.simply put if ya say 5 years from now then well you may as well juat replace the system and swap to something new then later on.

 

2 hours ago, techfan84 said:

I would like to be able to remotely access my home lab from other locations eg. whilst travelling. This would be through a VPN (using a mini computer with OpenVPN) and multi factor authentication (eg. Google Authenticator) for safety. I would like seperate VLANs for my home network web usage, my critical home server hardware, etc for added safety.  I would like to expose specific services on my server to the public/easy access for myself via my domain name and Dynamic DNS.

Just run that on a vm.

 

Do not EVER expose data of yours to the web without having some GOOD security active.

2 hours ago, techfan84 said:

Most projects will be in linux but the gaming for example will require a Windows virtual machine. 

Anti cheat, drm,... will flag you and prevent you from playing or outright ban you. This is NOT something you can get around. Vms are AUTOMATICALLY banned. You need a bare metal system.

 

2 hours ago, techfan84 said:

This can use a basic KVM switch so I can do this for all machines, even those without IPMI support. In the work room I want to connect a mini computer connected to a keyboard and dual monitor that gives me another access point to my servers. I think I want to connect this directly to the ethernet port in this room and then use remote desktop software to interact with my virtual machines over the network. I would like this link to be low latency and high resolution to enable some gaming as well as general interaction with the VMs.

Since gaming wont happen due to drm, anit cheat,... why not just use a basic laptop?

Like when I work in a server room and remote in I don't wanna use a desktop that means I have to run back and forth to do my work between the server and the desktop.

 

Even small datacenters just have some diagnostic laptops you take with. Only times you use a desktop is when the server is removed from the system and needs big maintenance or a big fix there. Which often is still just done with a laptop.

 

2 hours ago, techfan84 said:

Do you see major problems with these component choices or do you suggest alternatives? Can you see ways that I could get similar

Yes actually.

 

The computation server can just be an am4 or am5 system. A 3950x is only 20% slower than the epyc but has NONE of the CRIPPLING issues zen+ epycs have. Also its cheap now and on a much much much cheaper platform.

 

Then slap a 3090 in there or something instead of a 10gb 3080 and 64gb of ram or so. Since you wont be gaming on it anyway and I dont really see any clear cut tasks? Just wants that you want to do? So first expiriment and then build a system for that purpose.

 

Hence the desktop hardware keeps you flexible.

 

You don't need ipmi. At all really. too btw

 

As for the nas just get a 5600g based rig and call it good. Add a lsi raid card in it mode and done. 16gb of ram will do add 32gb if ya want.

 

The 5600g is faster than the xeon and the xeon system is just overpriced and overkill for what it needs to be.

 

 

Oh and skip the expensive kvm rack monitor. Just have a table next to the rack with a screen and keyboard + mouse and kvm. You only have 2 computers anyway.

 

 

 

All in all I just see a lot of expensive hardware here when from all you are mentioning it can be done on far less costly and better stuff.

 

The compute server also feels a bit aimless with the tasks it needs to do? What are you currentlt actively doing?

 

Most development is done on the machine of the user including running renders and such. Handoff to a sever is only for MASSIVE projects and that goes to a server farm. No a single device.

 

So to me this sounds more like you want:

 

A desktop to game and work on

A nas

A potential rendering server

[techfan84]

On 3/23/2024 at 12:18 PM, jaslion said:

Just get a regular desktop with a used 3090?  Zen and Zen + epyc's suck at single core tasks and tasks that leap over from ccx's. Like for decent performance amd RECOMMENDS turning off cores and ccx's. For your tasks this cpu will be straight up bad.

 

Its also miserable for gaming. Basically a i7 4770 sigle core but it stutters like mad.

 

Thank you for your critical assessment, it is appreciated. I have made a number of changes to the build based on your advice which I am now happier with. 

Part of the challenge is that I am trying to build a very general purpose setup and I have a wide range of interests/projects I want to work on with it. I want it to cater to concrete applications that I'm already bottlenecked by running just on my laptop, and also have scope for future projects that I am not yet aware of and would like to try out easily.

A few examples of my current applications and requirements: 
- I do work with Stable Diffusion and I want more vram to run more render batches to more quickly converge on outputs I am happy with. This is particularly handy for video to video workflows. I will go with your suggested 3090 since I have now found good deals on this. 
- I do various CFD simulations and scientific image processing projects. These are CPU intensive and being able to run different realisations in parallel (eg. Monte Carlo simulations) would be very useful and still leave resources for other projects at the same time in other VMs. I don't necessarily need the fastest of CPUs so I am still in favour of the EPYC over the Ryzen, particularly as the ecosystem allows more PCIE lanes. 

Your points about gaming seem very good, I was not aware of the anti ban issues with VMs. I don't have much time for gaming these days so it is not such a big issue and mostly would be playing single player games. If it doesn't work in a VM, I will take your advice and move to a gaming laptop. Likewise for the ML work, if that performs very poorly I will have to switch to a dedicated GPU Rig but would rather avoid it as I like the flexibility of this general build and VMs. 
 

 

On 3/23/2024 at 12:18 PM, jaslion said:

Do you NEED this? What is the timeline you have for filling up 50tb? What are your data cleaning policies? This will have a massive impact on the build.

 

Because w.simply put if ya say 5 years from now then well you may as well juat replace the system and swap to something new then later on.

 

I already have ~20TB of files (photography and project data) stored randomly on external HDDs which deeply concerns me. I want a safe place for them and for the outputs that are worth keeping. As I get less bottlenecked with compute, I can generate more so I do expect this to grow quickly. I could see myself hitting 50TB in 1 to 2 years. What I choose to retain will depend on the quality of data which I will only know as I go.

 

On 3/23/2024 at 12:18 PM, jaslion said:

Yes actually.

 

The computation server can just be an am4 or am5 system. A 3950x is only 20% slower than the epyc but has NONE of the CRIPPLING issues zen+ epycs have. Also its cheap now and on a much much much cheaper platform.

 

Then slap a 3090 in there or something instead of a 10gb 3080 and 64gb of ram or so. Since you wont be gaming on it anyway and I dont really see any clear cut tasks? Just wants that you want to do? So first expiriment and then build a system for that purpose.

 

Hence the desktop hardware keeps you flexible.

 

You don't need ipmi. At all really. too btw

 

As for the nas just get a 5600g based rig and call it good. Add a lsi raid card in it mode and done. 16gb of ram will do add 32gb if ya want.

 

The 5600g is faster than the xeon and the xeon system is just overpriced and overkill for what it needs to be.

 

 

Oh and skip the expensive kvm rack monitor. Just have a table next to the rack with a screen and keyboard + mouse and kvm. You only have 2 computers anyway.

 

 

 

All in all I just see a lot of expensive hardware here when from all you are mentioning it can be done on far less costly and better stuff.

 

The compute server also feels a bit aimless with the tasks it needs to do? What are you currentlt actively doing?

 

Most development is done on the machine of the user including running renders and such. Handoff to a sever is only for MASSIVE projects and that goes to a server farm. No a single device.

 

So to me this sounds more like you want:

 

A desktop to game and work on

A nas

A potential rendering server

I have updated the plan to consolidate my storage server into the compute server under a VM. This has significantly reduced costs. I will skip the KVM as you suggest. I will think more on just how much performance makes sense for a local system and how much I should get done on the cloud/server farm. This decision can significantly reduce costs, but I haven't come to a conclusion on that yet. 

 

Thanks again for your advice and suggestions! 

 

 

 

 

 

[jaslion]

7 hours ago, techfan84 said:

If it doesn't work in a VM, I will take your advice and move to a gaming laptop. Likewise for the ML work, if that performs very poorly I will have to switch to a dedicated GPU Rig but would rather avoid it as I like the flexibility of this general build and VMs. 
 

The fix here is to just not get a first or second gen epyc. Those suck. 3rd gen and up are totally fine. Can even opt for a lower core count one as the leap in single core between zen and zen 2 is MASSIVE.

 

With the money saved from skipping all the unnecesary cluter you can EASILY get a 3rd gen epyc or better.

 

Gaming still has to be bare metal so just make a gaming desktop or something.

 

Laptops are very much so more costly and do keep in mind that the mobile gpu's are multiple tiers lower compared to their similarly named desktop counterpart

[techfan84]

We are thinking along similar lines here! I am currently looking at an EPYC 9334 with a Supermicro H13SSL-NT motherboard. Swapping them in actually brings the cost closer to the previous estimate, but I am much happier with the build. Unfortunately for that price point it is with a 9334 QS version which despite reassurance from the seller, I am quite skeptical about and do not have the experience to know how much of a gamble it will be. So I may have to drop back to a third gen Epyc or consider alternative options still unless I can find some better deals for these and other parts. 

 

Interesting point regarding the laptops, I hadn't realised they actually have lower tier parts yet with the same name! In that case I will think more about a gaming rig rather than a laptop. 

[jaslion]

1 hour ago, techfan84 said:

We are thinking along similar lines here! I am currently looking at an EPYC 9334 with a Supermicro H13SSL-NT motherboard. Swapping them in actually brings the cost closer to the previous estimate, but I am much happier with the build. Unfortunately for that price point it is with a 9334 QS version which despite reassurance from the seller, I am quite skeptical about and do not have the experience to know how much of a gamble it will be. So I may have to drop back to a third gen Epyc or consider alternative options still unless I can find some better deals for these and other parts. 

 

Interesting point regarding the laptops, I hadn't realised they actually have lower tier parts yet with the same name! In that case I will think more about a gaming rig rather than a laptop. 

How much are the 3090's you're lookin at?

 

What gamez would ya be playin? You can get a pretty darn good system at a lot of budget tiers depending on the needs.

 

How much is the qs sample?

 

Is it on ebay?

 

I am ALWAYS wary about qs stuff as they shouldnt even be available technically

[techfan84]

The 3090s I've been looking at are second hand around ~1kUSD. I am tempted by a 4090 just for the speed boost in ML work but those are a bit too expensive I think, and perhaps I can upgrade later. 

 

I haven't been able to play games for a while due to my laptop specs so I have a long list now! Some of the ones that I have heard are harder to run include: Hogwarts Legacy, Resident Evil 4, and Plague Tale: Requiem. Not sure how important it is for me to run at 4k 120 fps or at lower specs. I need to check out some monitors running comparisons to decide that I guess.

 

The qs samples are on Ebay and are around ~650 USD. This is maybe too good to be true and I should expect big problems are likely... Would be great if I could find a 4th Gen at a better price or at least have a way to test the qs early for a chance of returning to seller as opposed to it just dying months down the line with less chance of refund! 

[igormp]

3 hours ago, techfan84 said:

The 3090s I've been looking at are second hand around ~1kUSD. I am tempted by a 4090 just for the speed boost in ML work but those are a bit too expensive I think, and perhaps I can upgrade later. 

Unless the 4090 is priced at 20% more the 3090, it's usually not worth it. In some places you can even get two 3090s for almost the same price of a 4090 (that's what I did).

[techfan84]

I am very open to this option, just need to find a good price. Will keep searching!