Home Server Power Savings - a recap

[Aragorn-]

I posted some months ago about making my server setup somewhat more efficient:

 

https://linustechtips.com/topic/1462253-unicorn-motherboard-with-3-16x-slots

 

I've finally completed the works, so thought i'd post some figures.

 

At the start, the "server" was pulling 242w. (Measured at the wall with a plug in meter)

 

This was split into two chunks, 140w for the server itself and a further 102w from the MD1200 drive shelf. 

 

The Server consisted of a 2950x Threadripper, Gigabyte X399 board, 32gb ram, LSI 9200-8e HBA, NVS310 (boot GPU), RTX2070 (Jellyfin/Transcode GPU), Firepro W5000 (spare GPU) and two m.2 SSD's (boot and VM storage). The Drive shelf contains 6 spinning disks and an SSD, and connects to the server via an external SAS cable.

 

Not a great deal could be done about the MD1200, however i discovered it had 4 old 3TB drives still installed. Removing those dropped it to 70w (with 6x 6TB drives + 1 2TB SSD)

 

I also removed the spare W5000 GPU (it had been installed while testing out some Windows GPU passthru stuff) and added a ConnectX3 10GBE NIC, this dropped the server from 140w to 130w.

 

So just tidying up some loose ends dropped total consumption from 242w down to 200w. Not much, but something.

 

Then i started looking at new hardware. After much back and forth, i settled on the following: i5-12400, Asus Z690-P, 32GB DDR5 (basic 4800 spec). I retained the LSI HBA, the two m.2 drives, the Connect-X NIC and the 2070. System was now booting from the iGPU, disposing of the need for the NVS310. This swap required a lot of faff, i wanted to simply transplant the install across, which worked, except the linux install was in "BIOS" mode, and the Intel iGPU will not operate in BIOS mode, requiring UEFI. After many hours fighting with it, i managed to shrink a partition, add a small EFI partition, and install the UEFI version of GRUB and get it all booting in UEFI mode with the iGPU active.  Total server draw in this configuration, was 66w. Literally half of the old setup!

 

Lastly, i swapped the 2070 out for a 6600xt, having read the 6600's have a really low idle power consumption. Sure enough, Server is now idling at 45w

 

 

Hopefully i can sell the Threadripper parts and cover most of the cost of the swaps, meaning its all fairly cost neutral, just ongoing savings on electricity.

 

Total system power (server+MD1200) is now sitting at 115w. Less than half of where it was to begin with. In terms of costs, this means its gone from about 5.8kwh a day, costing ~ £1.75, down to about 2.8kwh, or £0.82 per day.

 

Next Steps?

 

The MD1200 seems fairly power hungry (its drawing power power than the whole rest of the server at this point!) and probably isnt really necessary. It only has 7 drives currently installed. if we assume a drive uses around 8w (we can see removing 4 drives saved 30w total) then we can guess the drives themselves must be consuming around 56w of that 70. So ditching the MD1200, and moving the drives inside the server chassis would potentially save a further 15w, potentially a little more depending on the relative efficiency rating of the PSU's etc. Info online suggests the MD1200 can draw 25w just sitting idling empty, so potentially the savings are a little higher. Likely somewhere in that 15-25w range though. However there are ofcourse costs here, a new server chassis with appropriate drive bays is not cheap, and potentially you'd still need a backplane in the chassis which will consume some power. The MD1200 could be sold on, but i'll have to look carefully at the costs here to see if its worth the effort.

 

A better approach might be to combine ditching the MD1200, with replacing the 6x smaller 6tb drives with a smaller number of much larger ones. a pair of 18tb disks would give me the same overall capacity. But this is likely to be very costly. Big drives are expensive, and used smaller drives dont sell for that much. Again, some numbers need to be worked, and this is a project for the future.

 

Instead, i'm going to look at the cheap "low hanging fruit". My router is x86 based, running on some ivy bridge era hardware and uses 25w or so. I've bought a Dell Wyse 5070 unit to replace it, which apparently can go down to 6-7w. So thats the next project.

 

Hope that was an interesting read, or maybe just a boring wall of text?

Cheers!

[WildDagwood]

Obviously gaining a decent amount of single thread performance (on-top of the energy/efficiency gain), but losing a lot of cores/threads in the process.

 

Purely curiosity, but what all do you use the server for? Mainly just media?

 

Guess a follow up question would be if you've noticed any differences in performance on any workloads you do.

[Aragorn-]

yeah, some vaugely googled cinebench figures suggests the 2950x would manage about 1.1k single thread and 18k MT, the 12400 on the other hand managing 1.6k single and 12k MT. The massive 50% boost in ST means that even the MT performance isnt that bad despite dropping from 16 or 6 cores.

 

However the 2950x was always seriously overkill. I had initially intended to run a "2 gamers 1 cpu" style arrangement, with my personal desktop running as a VM inside the server, hence the W5000 GPU i had popped in there for testing some time ago. I had planned to pin 6 or 8 CPU cores permanently to that VM to ensure maximum performance for that client VM. I had it all working, but I decided after a while not to bother migrating over to it. Mostly because i realised that from time to time i like to tinker with the server, and having my main PC also offline at the same time would be really annoying. Also the required long (expensive!) displayport cables that i was going to require put me off.

 

The server runs a few different VM's and services. local fileserver being its core role, but there is a VM running Jellyfin, another VM hosting NZBGet and associated parts, a VM running home assistant, a docker running the unifi controller, and a VM for some CCTV NVR software that isnt quite configured yet but is coming soon. Theres also an EmonCMS instance running on there for my local energy logging system.

 

I suspect the 12400 will have sped things up across the board, especially as the various VM's only have access to a handful of cores. For example unrar and parity checking in NZBGet is now likely faster with more CPU behind it. I havent actually measured ofcourse, but the system was never heavily loaded across all the cores, so the single thread gains will be the biggest impact.

 

A 12100 will probably have done the job. But i wanted a bit of overhead, and because i needed the iGPU, the cost difference between a 12400 and 12100 wasnt that big, most of the cheap 12100 supply seems to be the F sku without GPU.

[E-waste]

Hey, since you care so much about energy use, start running a firewall or dns filter like adgaurd or pi-hole.

 

If you block certain telemetry domains, you'll probably be saving a lot of power that would start up a hard disk on some far away server, going through over two handfuls of network hops before it gets there.

 

Also I recommend thinking about e-waste, (I know you'll try selling the parts) and also what amount of energy went into the new parts available now.

 

Think about this:

 

If only 10% of the current amount of people upgraded their equipment to save electricity, think about how much less chip shipment, manufacturing energy, overall fuel use, human slave-labor needs, and plastic & plastic packaging would be generated instead.  If you truly care about your local energy use, think about all of these factors, that it took to get the new hardware from mineral mine to shipped in plastic to the post office.

 

It may save YOU energy locally, but you just drove up or sustained the current chip demand for that model, so did you really help to save electricity from a global perspective?

 

I'm not trying to shit on your efforts here, I think they are great, I just want you to consider the impacts of always buying new hardware (yes, even every 12 years) and the impact that has on fuel in excavation, and shipping, the coal burned to support the chip manufacture process, the pollution from the world's 16 diesel ships and planes, and the plastic used in the new products, and its accompanying packaging.

 

And I want you to ask yourself, does dropping server consumption from 242 to 102w (or lets say 80-90) make up for EVERYTHING that went into making that new hardware?

 

What do you think?

[Blue4130]

9 hours ago, Timpster said:

Hey, since you care so much about energy use, start running a firewall or dns filter like adgaurd or pi-hole.

 

If you block certain telemetry domains, you'll probably be saving a lot of power that would start up a hard disk on some far away server, going through over two handfuls of network hops before it gets there.

 

Also I recommend thinking about e-waste, (I know you'll try selling the parts) and also what amount of energy went into the new parts available now.

 

Think about this:

 

If only 10% of the current amount of people upgraded their equipment to save electricity, think about how much less chip shipment, manufacturing energy, overall fuel use, human slave-labor needs, and plastic & plastic packaging would be generated instead.  If you truly care about your local energy use, think about all of these factors, that it took to get the new hardware from mineral mine to shipped in plastic to the post office.

 

It may save YOU energy locally, but you just drove up or sustained the current chip demand for that model, so did you really help to save electricity from a global perspective?

 

I'm not trying to shit on your efforts here, I think they are great, I just want you to consider the impacts of always buying new hardware (yes, even every 12 years) and the impact that has on fuel in excavation, and shipping, the coal burned to support the chip manufacture process, the pollution from the world's 16 diesel ships and planes, and the plastic used in the new products, and its accompanying packaging.

 

And I want you to ask yourself, does dropping server consumption from 242 to 102w (or lets say 80-90) make up for EVERYTHING that went into making that new hardware?

 

What do you think?

Some people don't care about the environment and just want to save electricity to lower their energy bill each month. 

[Aragorn-]

It's a fair point to ponder. The very difficult part is that there is a crossover point, where the reduced ongoing emissions exceed the cost of manufacturing. Unfortunately it's very difficult to figure out where that point is, as the exact manufacturing input into a given product is not typically known.

 

You see this a lot with electric vehicles. The antis will claim an ev is more polluting to manufacture than an ice. The pro-ev folks will point out that for a given mileage and electricity grid the ev comes out ahead after X years. They'll then start arguing over the X as each side pulls out different numbers or makes different assumptions to better shore up their arguement.

 

At the end of the day, new hardware is always required at some point. In this particular situation, the new parts were all purchased used from eBay with the exception of the mainboard, as I needed something specific which wasnt available used (ddr5 and at least 3 16x slots). And ofcourse the old parts will be sold on for someone else to use. I hate ewaste and am a regular user of eBay for disposing of old tech, I'd much rather it goes to someone else who can make use of it than it going in the bin (plus I get money back!)

 

At a personal level, decisions are made on the balance of smaller local decisions (IE the electricity bill) as well as larger considerations. Some folks won't consider these larger things at all. Others focus on them to the extreme. I do have an eye on those bigger issues, but cutting our personal use is one of the biggest things we can do (and that includes reducing electricity use, but also other things like improving household insulation, which requires spending money and has a longer payback period), as well as not needlessly consuming junk.

[Needfuldoer]

I'd build everything into one chassis instead of using a drive shelf. That's just another power supply and external SAS controllers you have to keep fed.

[Aragorn-]

18 hours ago, Needfuldoer said:

I'd build everything into one chassis instead of using a drive shelf. That's just another power supply and external SAS controllers you have to keep fed.

Yeah, it's finding a decent hot swappable chassis that doesn't cost the earth. I can always sell the md1200 on to recoup some of the cost but such chassis aren't cheap.

 

I went 4u to allow standard ATX PSU as well as 120mm fans in the fan wall to keep the noise sensible. A 4u hot swap chassis is somewhat rare and seems to cost £300+ and ofcourse they tend to have loads of drive bays which probably helps making them expensive.

 

I guess I'll keep my eyes on eBay.

[Blue4130]

32 minutes ago, Aragorn- said:

Yeah, it's finding a decent hot swappable chassis that doesn't cost the earth. I can always sell the md1200 on to recoup some of the cost but such chassis aren't cheap.

 

I went 4u to allow standard ATX PSU as well as 120mm fans in the fan wall to keep the noise sensible. A 4u hot swap chassis is somewhat rare and seems to cost £300+ and ofcourse they tend to have loads of drive bays which probably helps making them expensive.

 

I guess I'll keep my eyes on eBay.

It's the backplane that drives the cost up normally.