Asgard, the 60TB FreeNAS Plex Server

Thirdgen89GTA · March 5, 2019

I always intend to sign up for LTT and chat with the peeps here as I lost my Futuremark forum a while ago and I've got this huge Technological hole in my digital social life now.

But I figured why not now.

About a year ago I upgraded my FreeNAS to a new chassis because the old Antec 900 case just could not contain it anymore. So one of the things I needed was a new home.

I am thinking I will move to the Ryzen 3 platform when it finally comes out as it will be an excellent upgrade over the aging 4790k intel based setup that it has now.

The goals for this were a nearly silent FreeNAS server and I met and exceeded those goals. The entire server at full load doesn't exceed 44db, while its not whisper quiet its quieter than my fridge, ceiling fan, pet fountain....etc.

HP 19U Rack for home.

NORCO RPC-4224 24 3.5" bay Hot-Swappable 4U Server Chassis
Corsair RM750x Modular power supply
Noctua NH-U9S 92mm Silent CPU HSF
Intel i7-4790k Socket 1150 CPU
Gigabyte GA-Z97X-SLI Motherboard
32GB Crucial Ballistics Sport DDR3-1866mhz ram
6x right angle SFF-8087 to Straight boot cables
LSI SAS 9211-8i Host Bus Adapter Card
LSI SAS 9201-16i Host Bus Adapter Card
3x Noctua NF-S12A 120mm PWM fans
2x Noctua AF8 80mm PWM fans
6x WD Red 8TB drives (32TB RaidZ1 Array)
6x WD Green 4TB drives (16TB RaidZ2 Array) WDIDLE'd

6x various 256GB SSDs in RaidZ2 (Plex VM, Plex Database, Photos, Personal files, software installers....)
2x Sandisk X400 128GB SSD (FreeNAS OS Boot Drive)

some of the pics from the next post are out of date a bit, so might show a slightly different setup. But all in all, this thing is Rock solid and easily serves up massive amounts of Plex for me both in and out of the house. I average about 50Gb per month of outbound Plex traffic. A ton of that is me just streaming music from home. I purchase all my music on CD and rip lossless. Then use PlexAmp to stream it while on the go.

Energy consumption is excellent as well with it idling in the 60w range and running about 100w with 1 1080p transcode going. Fully loaded it seems to hover around the 170w range.

Thirdgen89GTA · March 5, 2019

On to the pictures of the build!

So, the first thing I did after receiving the chassis is build a sacrificial NAS using old WD Red 3TB drives that I didn't care if they blew up.

I wanted to test all of the Backplanes and ports to make sure it was all working.

Some Parts have arrived.

I got it up and running with an old AMD CPU/Mobo, and tested each backplane 2 at a time.

Taking a look inside one of the Hot-Swap bays and its hard drive caddies.

After I confirmed everything was working it became time to strip the chassis back down and migrate all the Good stuff to the new chassis.

But first the Cats needed to inspect it.

The Cats didn't approve of the noisy stock fans. So out they went, and in went the Noctua silent PWM fans. Starting to add components here.

The internal 2 bay 2.5" HD tray.

The Noctua NH-U9S 92mm HSF, which is so very quiet. And the CPU temps are staying right around 37°C idle.

Starting to install the components.

Cable Managing EVERYTHING from the get go.

Adding in the SAS cards and SAS cables. They are too long, so they make the biggest mess in the case. But I did a decent job with them I think. They are really rigid and difficult to work with. The right angle boots really make the drive cage side super clean and easy.

Running power to the backplanes.

And finally, to aid in keeping the drives cool I installed them staggered for now to increase airflow around them. Pretty blue LEDs!

And here we are, 54TB of formatted storage.

I named the server Asgard, because Thor: Ragnarok. Inside of Asgard live Odin, and Frejya.

[william@Asgard ~]$ zpool list

	NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT

	Freyja        18.1T   824K  18.1T         -     0%     0%  1.00x  ONLINE  /mnt

	Odin          36.2T  17.4T  18.8T         -    11%    48%  1.00x  ONLINE  /mnt

	freenas-boot   118G   840M   117G         -      -     0%  1.00x  ONLINE  -

	jails          230G  63.2G   167G         -    37%    27%  1.00x  ONLINE  /mnt

	[william@Asgard ~]$ [/CODE]

How does 54TB of formatted storage sound? The shear quietness of this server astounds me. It is literally WHISPER quiet. It doesn't even register on the Decible meter on my iPhone over ambient background noise. So awesome.

It sounds like SILENCE! This thing is so quiet I can hear my Pet Fountain and refrigerator over it. its just SOOOO quiet. Was well worth the money spent on the Noctua Fans and Heat Sink combo.

Thirdgen89GTA · March 5, 2019

And the pic of it finally in its home.

Thirdgen89GTA · March 5, 2019

Someone asked me what it was like to use, aka, how responsive was it and that sort of thing.

So I put together a short video just skipping around to the various things on it.

https://youtu.be/7EoTVjJ1_us

Thirdgen89GTA · September 21, 2019

Minor update, I recently ran out of space and added a 9x 4TB disks in RaidZ3. Technically I have almost 86TB of disks in there. But with parity and formatting, I only have about 51TB. I've got about 30TB free right now with the new vDev in place.

NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Frigga        32.5T  17.3T  15.2T        -         -     0%    53%  1.00x  ONLINE  /mnt
Loki          1.38T   705G   711G        -         -    22%    49%  1.00x  ONLINE  /mnt
Odin          43.5T  16.2T  27.3T        -         -     0%    37%  1.00x  ONLINE  /mnt
freenas-boot   118G  1.53G   116G        -         -      -     1%  1.00x  ONLINE  -

The additional disks caused airflow issues, the fans just weren't enough. I swapped out the Noctua S12 120mm fans for the Noctua IPPC 3000RPM fans. Unfortunately this introduced significant noise. I thought that I wouldn't hear them much as the SPL Meter app on my phone reported the average SPL as 45-50dB, and the new Noctua's were 43dB. That wasn't the case. Different frequency, so they are very audible.

However, they did drop the HDD temps enough and bring them back into line with what I had before, averaging 32C° to 38° at load.

So I changed the install order of the disks to stagger the slots. Now, no two 3.5" disks were stacked on top, or next to one another. This dropped the temps significantly, some of the disks are now sub-30C° at idle.

I did however make a note when I was in there, I noticed a 6pin header on the Noctua 120mm fan wall, next to the Molex power connector. Getting excited and needing a better look I ripped the fan wall out to look at the circuit board.

The 120mm fan wall supports PWM signaling for each fan! I'm going to combine the PWM Control pins into a single wire, and run that + a tach signal from one of the fans back to a single PWM header. Probably SysFan3 since its closest, then use the Mobo to control the fan speed and bring the fan speeds down.

FreeNAS doesn't understand how to control the Fan headers since its not an IPMI equipped board. Even though SMARTCTL can read the HD temps, FreeNAS doesn't know how to talk to the motherboard and tell the fans to spin faster. I'll have to rely on the BIOS settings, or just set them statically to a setting that is both functional, and pleasing to the ear.

You can also see that the upper 2 pins each have two traces that route to the 6 pin connector.

And how they all collect at the 6 pin junction.

The left side is the PWM Control signal Pins, and the right side is the TACH signal pins. Since they are all the exact same fans, I only need to get the connector and wire up 4 of the pins.

I will wire up 1 TACH pin to the PWM Fan header on my Mobo, and split the Mobo's PWM Control pin, to the 3 PWM pins on the fan bus. Combined with the MOLEX power connector this will safely power all 3 fans, and allow the motherboard to control all 3 fan's speed with PWM!