Video Editing Server

[Cinewill]

Looking for advice on a NAS for a small (but growing) video editing team.

 

Current Setup:

HPE MicroServer Gen10 running unRaid

32TB raw storage

2 editors editing from it over a 10gig network

Nightly off-site backup

UPS

 

We're quickly running out of capacity (currently generating around 10TB/yr but looking at increasing to +20TB the coming year).

 

Budget is a bit tight (around $4000) so my plan is to build our own TrueNAS system with around 120TB useable capacity.

 

My current parts list:

Motherboard: ASRock X570D4U-2L2T (Server MOBO)

CPU: AMD Ryzen 7 5700X

RAM: 64GB 3200 ECC (potentially looking at increasing to 128GB)

PSU: SeaSonic PRIME TX-750

Storage: 8x Seagate Exos X20 20 TB

Cache SSD: 2x Samsung 980 Pro 2 TB

Case: SilverStone RM21-308 Rackmount Case (Already own)

Cooling: Noctua NH-L9x65 and Noctua fans all the way (I already have the fans and want it to be as quiet as possible)

 

Any thoughts? Considering that:

I'm expecting to use this as our primary video editing server over a 2-3 year lifespan, at which point we'll probably have grown to 4-5 editors.

Yes, I'm willing to sacrifice enterprise grade reliability for a lower cost. The server will of course be properly backed up, so a failure would of course be bad but not business threatening.

The server will live in the same room as the editors for now, but in a rack a few meters away, so it can't be too loud, but some noise is fine.

The server won't be used for anything else other than storage.

I'm quite technical and have managed several (mostly unRaid) servers over the years. I'm not as experienced with TrueNAS (have tinkered with it a bit) but I'm sure I'll manage this quite simple deployment.

 

Thanks in advance

[skimmilk5]

I would recommending bumping up your memory now, rather than later. ZFS uses memory quite heavily, so the general recommendation is 1 gigabyte of memory per 1 terabyte of magnetic storage.

 

TrueNAS doesn't use cache drives like Unraid does. While you may use cache drives in Unraid as a write cache (data gets written to the cache drive, then the mover moves the data off of the cache drives to the magnetic storage array later), TrueNAS doesn't use this, instead using system memory as its write cache. Instead, your SSDs should be used as metadata drives, which you can select while setting up your vdevs. The metadata drive doesn't need to be as big, but what it stores is important: the location and information (the metadata) of as many files as it can on the magnetic storage array. This means that read seek times are a lot faster, since the array doesn't have to hunt around for the data; it can just look up the location of the data on the metadata drive, then directly ask that drive for that data.

 

Your metadata drive doesn't need to be super big, but speed does help. If it were me, I'd drop down to one metadata drive (either 1 TB or 2 TB, I'm sure 1 TB would suffice), then use the extra to increase the amount of memory in your system.

[skimmilk5]

One other recommendation: it appears that you're intending on using the on-board SATA controller, which is fine to start off with, but I'd also consider adding a HBA later and splitting the drives between controllers for bandwidth reasons. For sequential operations, you'll probably saturate the link speed between the storage controller and the processor, but if you have two storage controllers, you can alleviate some of that load. You may also be able to come off of the onboard SATA controller for storage drives, instead just using HBAs. I've always had bad luck using on-board SATA controllers in storage servers. For random operations, the drives themselves will be the limiting factor, but your workload is likely going to be a mix of random and sequential operations, like editing and ingesting, respectively.

 

You also need to spec out a boot drive. Unlike Unraid, TrueNAS is much happier living on a proper drive. A 120GB or 240GB SSD is plenty large (TrueNAS is pretty small, and on a server board, I'd recommend actually running it off of a 32GB SATA DOM), but it's another cost to consider.

 

Essentially, I'd opt to have my boot drive(s) on my on-board SATA storage controller, my metadata drive(s) on my on-board NVMe, and my storage drives on a dedicated HBA or pair of HBAs. Don't get RAID cards!

[Cinewill]

1 hour ago, skimmilk5 said:

Essentially, I'd opt to have my boot drive(s) on my on-board SATA storage controller, my metadata drive(s) on my on-board NVMe, and my storage drives on a dedicated HBA or pair of HBAs. Don't get RAID cards!

Thanks for a lot of great advice!

 

I've usually opted for on-board SATA since HBAs feels like a bit of a jungle sometimes and are usually hard to find (in Sweden).

 

Got any recommendations? I know LSI is reputable but all the models, versions and modes is confusing. 

[skimmilk5]

Sure!

 

HBAs can look confusing on the surface, but they're actually pretty simple devices. HBAs simply turn a PCIe slot into a bunch of SATA or SAS ports that you can plug drives into. They have a controller on the card, but this controller's only purpose is to interface the drives to the rest of the system invisibly. Unlike a RAID card, a HBA does not touch the data on the drive at all, so it'll appear to the system just like if you plugged the drive into the motherboard. In fact, the on-board SATA ports are driven using a HBA!

 

There are two families of HBAs: SATA HBA cards and SAS HBA cards. SATA HBA cards are compatible only with SATA drives, while SAS HBA cards can normally be plugged into a SATA drive or a SAS drive. (Please note that this doesn't always hold true, and you should check the datasheet of a SAS HBA to make sure it can accept SATA drives before buying it!)

 

SATA HBAs can have either a bunch of single SATA ports on them (the ones that look like the ones on your motherboard), or they can be outfitted with a MiniSAS HD connector. A MiniSAS HD connector, despite its name, can accept up to four drives on it using a MiniSAS HD to 4x SATA breakout cable. Some backplanes also have a MiniSAS HD connector on them, so you can just use a MiniSAS HD to MiniSAS HD cable for tider cable management. You might also see the MiniSAS HD connector referred to as SFF-8643.

 

SAS HBAs typically have MiniSAS (also referred to as SFF-8087, not to be confused with MiniSAS HD) or MiniSAS HD connectors on them. Both can be connected to either breakout cables to go directly into drives or standalone cables (MiniSAS to MiniSAS, or MiniSAS HD to MiniSAS HD) to go into a backplane. Like mentioned above, they can typically accept a SATA or a SAS drive.

 

Taking a look at your SilverStone chassis, the RM21-308 appears to have compatibility for either SATA drives using the standard SATA connector or SAS using a SFF-8087 (MiniSAS) connector. What this means is that you can use two MiniSAS HD to MiniSAS cables to connect to your HBA; just two cables can service eight drives.

 

The LSI SAS 9300-8i has two MiniSAS HD ports on it, and they can be had for quite cheaply nowadays. LSI's naming scheme has four numbers (the model), followed by a number (the number of drives supported in total) and a letter (i for internal ports, e for external ports). Regardless of the HBA you choose, make sure that it has MiniSAS HD ports on it, not MiniSAS ports. On the datasheet, you can see that it supports up to eight SATA or SAS drives with its two MiniSAS HD ports. To install it, you'll install the HBA into your motherboard, then plug in the MiniSAS HD end of your cable to the HBA. Plug the other MiniSAS end of the cable into your chassis backplane. Repeat for the other cable.

 

For finding used HBAs, eBay is your friend; these cards are commonly liquidated from enterprises, especially from large datacenters who are removing their fleets of SAS drives and replacing them with PCIe storage at great expense.

 

Hopefully this helps!

[Electronics Wizardy]

7 hours ago, skimmilk5 said:

Your metadata drive doesn't need to be super big, but speed does help. If it were me, I'd drop down to one metadata drive (either 1 TB or 2 TB, I'm sure 1 TB would suffice), then use the extra to increase the amount of memory in your system.

I would keep the mirror here as if that metadata ssd fails all the data is lost.

 

If your using a log drive, you want a cache too(and probbly different drives). But. log drive likely won't make any difference here as your not doing sync writes with samba.

 

 

7 hours ago, skimmilk5 said:

One other recommendation: it appears that you're intending on using the on-board SATA controller, which is fine to start off with, but I'd also consider adding a HBA later and splitting the drives between controllers for bandwidth reasons. For sequential operations, you'll probably saturate the link speed between the storage controller and the processor, but if you have two storage controllers, you can alleviate some of that load. You may also be able to come off of the onboard SATA controller for storage drives, instead just using HBAs. I've always had bad luck using on-board SATA controllers in storage servers. For random operations, the drives themselves will be the limiting factor, but your workload is likely going to be a mix of random and sequential operations, like editing and ingesting, respectively.

 

I don't see the HBA helping with bandwidth. HDDs are pretty slow, so less than 2gB/s peak, and the onboard sata can handle that here. I'd only get a hba if you need more drives.

 

8 hours ago, Cinewill said:

I'm expecting to use this as our primary video editing server over a 2-3 year lifespan, at which point we'll probably have grown to 4-5 editors.

 

What bitrate videos are you working with?

 

I'd be darn tempted to have a ssd pool for current projects and use the hdds for older projects. Multiple editing with higher bitrate footage is gonna slow down a lot with HDDs

 

I don't see a boot drive either here.