close-to-done! Screw HDD noise! The all-SSD rackmount NAS.

[RollinLower]

Alright! So here i'll be laying out the plan for my new NAS machine.

 

Right now i have the following setup:

- Supermicro SuperChassis 846BE16-R920B
- redundant 920W SQ PSUs
- Supermicro X11SSL-F
- Intel Xeon E3 1230V5
- 64GB DDR4 ECC
- LSI 9201-16I SAS/SATA controller
- LR-Link 10G SFP+ NIC

- about 240TB in HDD capacity

 

Which is awesome for what it is, and because of some Noctua fans it's also very quiet. That is, untill the HDDs start spinning up.

This rack sits right next to my Desk, and the HDD's, while very spacious, can get intensely audible when under heavy I/O. This, is ofcourse completely unacceptable.

 

So here is what i aim to do: Get rid of the old setup completely, and start again from scratch. The complete plan is not set in stone just yet, but here is a draft:

 

- 2U, (supermicro?) 25 bay 2,5" chassis

- 25x 2TB SATA SSDs

- atleast 128GB RAM, and enough CPU horsepower to have this not be the bottleneck

- as silent as possible, maybe watercooled like my other server?

- dual 10G connectivity is a big plus.

 

As stated though, this is a very rough draft, and it all starts with the case. I'll be researching what case or secondhand server i can get that would fit the bill for now. But as always: Expect updates soon!

[Somerandomtechyboi]

10 minutes ago, RollinLower said:

This rack sits right next to my Desk, and the HDD's, while very spacious, can get intensely audible when under heavy I/O. This, is ofcourse completely unacceptable.

You dont just move the thing?

[Shimejii]

The other benefit of this is that you can actually move it without worry as well! 

[RollinLower]

34 minutes ago, Somerandomtechyboi said:

You dont just move the thing?

i live in a pretty small apartment, so moving it is out of the question.

also, i like having all my machines close by!

[Somerandomtechyboi]

47 minutes ago, RollinLower said:

i live in a pretty small apartment, so moving it is out of the question.

also, i like having all my machines close by!

How about adding some sound dampening around the hdds?

[RollinLower]

6 minutes ago, Somerandomtechyboi said:

How about adding some sound dampening around the hdds?

how would you even do that in a 24 bay hotswap chassis?

[Somerandomtechyboi]

10 minutes ago, RollinLower said:

how would you even do that in a 24 bay hotswap chassis?

I meant it more like sound dampening the outside of the chassis where the hdds are stored

[Blue4130]

4 hours ago, RollinLower said:

 

- about 240TB in HDD capacity

25x 2TB SATA SSDs-

The data hoarder in me cries at the reduced size 240tb to 50? Ouch. 

[RollinLower]

Finally getting this back on track!

 

Lots around this project has changed though. pretty much all specs are different, except for the 2U chassis. 

 

first off, i got a stellar deal on some hardware which i could not refuse.

- 2x Xeon Silver 4116 (12C 24T each)

- 512GB RAM in 8x 64GB DIMMs (so potentially up to 1TB in this machine!)

- SuperMicro X11DPI-N

- Intel X520 10G SFP+ NIC

a raid controller which came with the chassis and ofcourse 26 x 2TB Samsung PM863a disks!

 

 

I gotta say, it already looks quite stellar in the rack!

 

I'm also debating the watercooling aspect of this build. As much as i love watercooling my servers, i think this box would be just as fine with just some noctua's and better CPU coolers. This would also greatly help with maintaining this machine.

Also, because i now have to deal with the absolute unit of a X11DPI-N, i don;t think i can fit a radiator and pump somewhere in there.

[RollinLower]

Fans arrived!

 

The tiny 40mm fan is for the raid controller, since this thing a solutely needs some form of direct airflow to prevent it overheating. I just screwed it into the heatsink with some cut to size radiator screws I had laying around.

 

Next up, 3x uln 80mm nocs, uln so they spin at a max of 1500rpm, which should make this a very quiet box indeed.

I had to cut away part of the original fan shrouds, but I guess it's nothing too catastrophic. It definitely does not deserve the beauty price tho.

 

And there she is!

Now you probably saw this coming from a mile away: the CPUs overheat.

I expected as much, and better CPU cooling is on the way. In the meantime I've plopped in some spare fans to get everything installed.

 

 

I've opted to try out treunas scale for this box. Pretty curious how that goes!

 

More updates should be coming soon!

[RollinLower]

little preview of the truenas setup as it is booted now.

 

i would say that is probably enough power for a home nas, yes? 

(geheugen = RAM in dutch)

 

[RollinLower]

alright, so i have most of the initial truenas setup done now. 

I opted to just plop all 26 drives into one big raidZ3. mostly because even the 30-something TB of usable storage i get from this is much more than i plan on using for the forseeable future. Also, the Z3 means i get 3 drive fails before actual data-loss! 

Setup of truenas core has been super easy aswell. After selecting all the SSDs and assigning them to the same pool in raodZ3, i enabled TRIM here:

And i created a user with SMB share rights, started the SMB service and ofcourse an SMB share dataset and presto!

Just like that, 36TB of usable space added to my main system. 

 

Copying a file from my PC to this NAS got me the following result:

Which is quick, just not the 1GB+ the redundant 10G link should be giving me. I wonder what the bottleneck is here. The NAS should have plenty CPU power to handle a 10G link and my main PC aswell, which is running a Threadripper 3960X. The NICs in both systems should also be sufficient, an Intel X720 in my PC and an Intel X520 in the NAS. Linked together through a Ubiquiti USW aggregation switch and multimode fibers.

 

Also, i love how the main dashboard looks on truenas. You get insights on most of the stuff you'd need on the front page. Things like CPU usage and temps, drive health, network usage and RAM.

Also notice how the system is currently idle most of the time, and CPU temps are in the high 40's. This is fine for now as i won't be deploying any containers or VMs on this box yet, but i got more fans on the way that should fix this problem. When CPU temps are under control, the real fun starts!

 

Anyway, expect more updates soon as this build nears the finishing line.

[RollinLower]

holy crap te VM capabilities of TrueNAS scale are something else.

 

i'll be formulating a better post later but i just spun up 5 new Ubuntu 22.04 VM's in less than 5 minutes. This is awesome!

[RollinLower]

I finally got the fans for the CPU coolers in, and i never actually thought about mounting them before now. And since the CPU coolers are just passive heatsinks with thin fins, screwing them in like i did with the HBA cooler is not an option.

 

In the end i came to this 300IQ solution: 

 

rubber bands and paperclips!

The temps are great now. idle they sit at 30C, which is about 8C over ambient.

But the noise is a little distracting. You cannot hear it most of the time, but when they spin up you can get a little whoosh just barely in the background. 

I think i might put some of those low noise adaptors in between that Noctua ships with the fans. 

 

I also added a GPU in here i can use for one of the VMs i plan on using in this box. It's nothing special, just a Nvidia Quadro NVS 510. It will handle the most basic of CUDA, and that's what i need it for.

 

[RollinLower]

Now this is getting quite excessive...

 

I just have this in here for shits n giggles at this point. There is no way i am ever going to deploy enough VM's on this box to utilise over a TB of RAM.

 

That being said, it does look rather nice!

@Windows7getagging you for hardware porn

 

 

 

[Windows7ge]

Someday. Maybe. For right now though I'm good with 512GB.

 

I want to build a distributed cluster using something like Cockpit for a simple NMS. So I probably won't buy denser RAM than what I know each server needs.

 

Doesn't mean I'm not jelly though. Maybe perform the experiment I want to perform:

 

Make a VM and give it a ton of RAM.

Make a ramdisk

Create either an iSCSI or SMB network share.

Attach the disk to another VM.

Speed test. :3

 

You could litterally use RAM through the hypervisors virtual switch (100Gbit) to store an entire VM in RAM. xD

[RollinLower]

Some updates to this machine:

I had a pretty fantastic deal on two EPYC 7552 chips, with 48 cores and 96 threads these are beasts, and would make for a fine upgrade to this box. 

Going from 2nd gen Intel Scalable to EPYC means a socket change though, and that also means a complete platform upgrade.

 

i found the following board on ebay for cheap:

 

it's a Gigabyte MZ32-AR0. I chose it because it was super cheap, but also because it has enough DIMM slots to fit all of the DIMMs i have in this system.

 

Becaue the formfactor is pretty standard SSI-CEB, it fits the Supermicro chassis like a glove.

 

This EPYC chip is a lot more power-hungry compared to the Xeons i was running before though. And the dinky Noctua air cooling setup i was running is not going to be able to cool the 200Watts of power this thing consumes. For reference: The Xeons where 85Watt parts.

Normally i would build my own waterloop in this situation, but since alphacool was running a promotion on their 2U watercooling sets i bought one of those instead. All in one watercooling ready to go! (foreshadowing...)

 

 

At this point i was pretty ecstatic. All i had to do was shorten the tubes on the watercooling setup but other than that everything went super smooth!

I did however find out about 2 things around this point:

 

1. In my previous setup, i also had two normal SATA connectors on the motherboard to connect the last two SSDs in the back hotswap bays.

2. The watercooling actually doesn;t fit in a 2U system.....

 

 

This is kinda baffling to me. this was advertised and sold as a 2U server watercooling all-in-one. And yet, it will not fit a bog-standard 2U server.

I did manage to get the lid closed over top of the fittings, but that leaves the top cover in a bulge. For now that;s fine, but i guess i have to go look for a different pump/block combo or standalone pump and standalone block.

 

Now that everything was together, i could finally power her up. 

The first thing i noticed is that TrueNAS booted up like nothing even changed. It picked up the new hardware without missing a beat, and even my pool was still functioning normally, albeit with 2 drives missing. (1 spare in a Z1 and the global hot-spare)

 

That was pretty short lived through, because the next thing i noticed was that the NIC was not showing up.

I use a 100G Mellanox ConectX-5 in this system, and it was plugged into a PCIe gen 4 x16 slot. I tried everything i could find on google to make it wirk, but it just wouldn't even pop up in the BIOS. The NIC went completely unrecognised by the system. Then i stumbled upon a 4 year old forum post by someone on the servethehome forums where they explained Mellanox cards don;t like different PCIe gens that their own spec, so i swapped the nic to the only compatible Gen3 slot on my motherboard:

 

Which leaves it with a lot less cooling, but the tiny noctua fan attached to it and a very fortunate location of some punch-outs in the case for rail mounts makes it that the card stays within temps! And in the new/old slot, it was picked up right away without any further issues. Even my SMB share to my windows machine just popped back to life!

 

One final thing i noticed is that the two HBA's where getting too hot in this setup, since they have very limited airflow in that position. I solved this by zip-tying another 80mm noctua to the back of the case, drawing hot air away from the HBA area specifically.

 

 

And that is the system back online!
I did order a adapter cable from miniSAS to 4x regular SATA, which i'll use to connect the final 2 drives. And i might have to also see about RAM temperatures. With all this RAM so close together, the DIMMs are also getting a little toasty, eventough they are in the direct path of airflow from the radiator.

[MG2R]

Well that’s some false advertisement from alphacool if I ever saw one. 

 

other than the Fitment, how are the thermals and noise under load? 
 

other notes:

1. How are you paying the power for this thing? From the Dutch in your screenshot I’m deducing your probably paying about 35+ cents per kWh. 
2. That hardline gaming rig is absolutely one of the sexiest machines on the planet

[RollinLower]

10 hours ago, MG2R said:

Well that’s some false advertisement from alphacool if I ever saw one. 

 

other than the Fitment, how are the thermals and noise under load? 
 

The CPU thermals are great, 40C to 55C under load. But i did have to add extra fans to the RAM to keep all of it cool. Especially after spinning up some VMs on this box and RAM utilization goes up.

I ended up using 2 60mm noctuas which cover the RAM completely and i added some zip-ties to keep them aligned within the free space between the RAM sticks. 

 

 

I will probably have something 3D printed though to better funnel the air through the DIMMs. maybe just a shroud to properly funnel the air from the radiator over the RAM aswell?

10 hours ago, MG2R said:

other notes:

1. How are you paying the power for this thing? From the Dutch in your screenshot I’m deducing your probably paying about 35+ cents per kWh. 
2. That hardline gaming rig is absolutely one of the sexiest machines on the planet

I pay about ~25c a kWh, meaning my homelab is currently drawing about 200 euro a month worth of electricity. Quite a bit, but to me the hobby is worth it. My SO also isn't complaining because i'm paying the powerbills anyway.

 

As for the gaming rig, thanks! i built it after my previous pc burned down. That's quite a story in of itself, you can check out the Phoenix link in my signature for that buildlog, eventough the machine has seen some changes since then.

 

[MG2R]

Man I can’t get myself to spend that much on a hobby but I’m happy you do and post about it. That said, I am in the process of acquiring a new homelab. Waiting for parts rn. The main boards arrived but the rest is still in transit. It’s all about low power clustering. 35 c/kWh. Ever watt counts.
 

Those pictures of your burned down gaming rig are absolute horror. My condolences