New Epyc based server build. But what OS?

[Droz_]

Budget (including currency): Modest (USD)

Country: US

Use case: File server, Plex server, Resilio Sync node, light virtualization

Other details: Fractal Node 804, ASRock Rack ROMED6U-2L2T, AMD Epyc 8 or 16 core CPU, Corsair HX1000

My current setup at home is two Shuttles and a Netgear ReadyNAS 428. 

Shuttle One (Windows 10)

Shuttle SZ270R8
i5-7500
32GB RAM

500GB NVMe SSD

2TB NVMe SSD
Nvidia Quadro P1000
Trendnet 10GigE Card
This handles Plex, Resilio Sync and PiHole (via VMWare Player)

Shuttle Two (Currently testing Truenas Scale)

Shuttle SH370R8
i7-8800k
32GB Ram
500GB NVMe SSD

2TB NVMe SSD
Four 8TB HGST drives in Z1 
Trendnet 10GigE card
Ableconn PEXM2-130 Dual NVMe M.2 card
 

 

ReadyNAS 428

10x HGST 10TB Drives in RAID6 

______________________________________________________________________________________



OK So I'm looking to consolidate things into one server. 

Current setup has Shuttle One using the ReadyNAS as source for Plex. Resilio Sync is using local storage for shared folders.

Things work well. But....I am starting to work with 4k60fps gopro video and ingest on the shuttle is fine. But getting raw and edited video to/from the NAS is painfully slow at 1Gig. Yes the NAS has four 1Gig ports, it will not do proper aggregation across all four ports. I have tested this with a 10Gig card in the shuttle and a 10Gig switch port and best I got was 1Gig (thanks Netgear). 

I was using Shuttle Two as my desktop up until recently. I decided to clean it up and get some minor upgrades for it to try out Truenas. 

Its now been a month since I've been running Truenas Scale. I am seriously disappointed and frustrated with it. The UI for setting up apps is horrible. The WebUI time out is ridiculous, especially since there's no way to change the timeout period.
I tried plain Debian and Ubuntu for a bit. Was using Webmin to make management faster/easier. I've used Webmin in the past but seems that little things like not having UI elements for BTRFS when making a new file system isn't a thing. I'd test with VMWare if I could. But I don't have a RAID card that VMWare support and I'm not about to go out and buy one at this point. I really want to start building the new server. I still need to buy the CPU, RAM and drives. 

 

Server I'm building 

 

Based on the ASRock Rack ROMED6U-2L2T Its AMD Epyc based, six DIMM slots, four PCIe Gen 4 x16 slots, three SlimSAS ports (each of which can support two NVMe U.2/M.2 drives), three Mini-SAS ports and two SATA ports. It also has six 4 pin PWM fan power ports, two 10GigE ports, two 1Gig ports and IPMI. I'll use either the nvidia P1000 card or my single slot GTX 1070 card (bought this on EBay Australia a few years ago) for plex/handbrake transcoding. I'll also be getting a Highpoint card with up to eight NVMe M.2 slots. Eventually one or two more of these in the future. This will all be in a Fractal Node 804 case. 

I already have the board, case, power supply, a 4TB NVMe SSD and an 8TB NVMe SSD. Committed at this point HAHA

My OS options....

1. Windows 10/11
Right now this is the easiest way to go. But it means I'd need to get a raid card as I don't trust Windows Storage Spaces. Big benefit here is that Resilio Sync requires a license for running on Window server. I can keep using VMWare Player or Hyper-V for VM's I'm comfortable with both.

2. Windows Server
Makes more sense for a file server. Especially for compatibility. Apparently my ReadyNAS is unable to connect via SMB to Truenas for a direct backup target. (I have to use Iperius backup jobs on Shuttle One to get files back and forth). Since I will be using the ReadyNAS as a backup target once I have the new server built, this is not ideal. Hyper-V or VMWare player work well on it. Resilio Sync requires a license though

3. Truenas
REALLY F**KING skeptical of Truenas. (I mean I'm sure it works fine JUST AS A NAS. All the other things it can do, infuriating to get setup and working. It should not be that hard to do).The UI leaves a lot to be desired. Docker Apps from charts is a mess. I have never been able to get Plex, Pihole or any other support App from the charts to work. The UI makes a lot of assumptions, mainly that you know a moderate amount about Kubernetes/Docker and how they work. If you don't then you're SOL. And yes, I've looked online for help and most of the videos and articles out there are not for the most recent version of Truenas Scale. The few that are either aren't in depth enough or for some ridiculous reason, the menu options are completely different than what shows up for me

Benefit here is that I won't need to get a RAID card. But, ZFS seems to hate having a Zvol on one drive (e.g. a single NVMe SSD)

Downside is ZFS will use AAAALLLLL the RAM it can. Memory is f**king expensive for this board and I am broke mofo

Also it seems that Truenas' implementation of KVM is pretty light

4. VMWare
The free version of ESXi is more than enough for me. I would need to get a couple RAID cards since VMWare has no support for software raid. Not a big deal but kind of defeats the purpose of the board. I could technically have everything I need run on ESXi, Windows server for file storage and just pass through the RAID card with the mass storage drives. Second raid card for RAID1 boot drives (already have a couple 256GB 2.5 inch SSD's) and some messing around with storage if need be. 

5. Linux

I think I'm going to give Ubuntu another try and use Cockpit for a mangement WebUI. In another life, linux was my bread and butter, but I got into Enterprise IT with Windows, VMWare and Linux. Windows was the majority OS in out infrastructure and the linux side had its own team to manage it. CLI is not my strength anymore. Plus some dyslexia or some kind of motor/control issue I have makes it harder for me to navigate the filesystem by CLI. Also scripting I find really frustrating now days. 


Other options that I looked at

Unraid - LOL no
Proxmox - I'm getting more interested in this. But everything I looked up on this says not to use it for virtualization and native fileserver


Anyone have input?
 

 

Thanks

[Indian pc builder]

list of choices in order of preference

1. Truenas

2.linux

3.VMware

4.windows server

5. windows 10/11

6.Unraid

7.spend years learning how to create your own custom OS and then spend 100000000000000000000000000000000 on hiring someone to do it for you because you are out of money to pay for coding classes

[dbx10]

TrueNAS really isn't that hard, just whip up a Linux server VM and run whatever doesn't run as a plugin there.

EDIT: Scale is not ready for prod yet, I'd stick with Core and virtualize anything you need on Linux, especially if you wanna run Docker.

EDIT 2 electric boogaloo: Windows server is hot garbage and you will lose data and your mind using it.

[Droz_]

7 hours ago, dbx10 said:

TrueNAS really isn't that hard, just whip up a Linux server VM and run whatever doesn't run as a plugin there.

EDIT: Scale is not ready for prod yet, I'd stick with Core and virtualize anything you need on Linux, especially if you wanna run Docker.

EDIT 2 electric boogaloo: Windows server is hot garbage and you will lose data and your mind using it.

@dbx10

I've never had a major failure of a Windows server that resulted in data loss/corruption. Yes I've had motherboards fail, ram fail, CPU fail, raid cards fail, etc. Even with a bad raid card, swapping in a new one the array rebuilt itself and was fine. I've been working with Windows for work in both small and large scale for a very long time. Windows is not a bad OS in any way. 

As for Truenas Core, I feel that its just as unpolished as Scale. If I'm going to just have the same issue configuring an App that I want to run in Core as I do in Scale. I might as well continue using one of my Shuttles as an application host.

Also I'll have onsite backups as well as dumping to Backblaze or S3 for offsite as well. I'm fine if I lose a little non-personal data. 

My personal data I already have multiple backups of

[dbx10]

3 hours ago, Droz_ said:

@dbx10

I've never had a major failure of a Windows server that resulted in data loss/corruption. Yes I've had motherboards fail, ram fail, CPU fail, raid cards fail, etc. Even with a bad raid card, swapping in a new one the array rebuilt itself and was fine. I've been working with Windows for work in both small and large scale for a very long time. Windows is not a bad OS in any way. 

As for Truenas Core, I feel that its just as unpolished as Scale. If I'm going to just have the same issue configuring an App that I want to run in Core as I do in Scale. I might as well continue using one of my Shuttles as an application host.

Also I'll have onsite backups as well as dumping to Backblaze or S3 for offsite as well. I'm fine if I lose a little non-personal data. 

My personal data I already have multiple backups of

Many points:

• Hardware RAID is dead in the enterprise space, and for very good reasons. Even multiple thousand dollar RAID cards fail, and sometimes introduce data corruption into your array, which it is oblivious to. RAID was fine in the 1990s and early 2000s, when it was all we had.
• OpenZFS and hopefully soon BTRFS offer better availability and redundancy out of the box. Your ZFS pool remains accessible in case of a drive failure, and the software regulates data integrity. ZFS doesn't trust a drive as far as it can throw it.
• Windows server is only reliable within their ecosystem of services and validated partners. It CAN work, but there's also a reason why the use of Windows is at an all time low in the server space. It's just not as reliable or as flexible, and certainly not as stable.
• TrueNAS Core's "stable" release train goes through 8 rounds of validation before releasing, it's enterprise grade and rock solid. As for the plugins and the apps, virtualize them if you're not comfortable with BSD's jail system. I'm currently running an Ubuntu server box on bhyve just for my docker containers.

Sure, you can still run a RAID card and an obsolete redundant array, but ultimately there is no cost to running more modern and proven solutions. So why not?

Proxmox, TrueNAS Core, ZFS on the Linux distro of your choice, you can even distro hop and move your ZFS pool every time, should you want to.

I'd also like to know specifically what you mean by "unpolished". Genuinely asking, it's not a snarky rhetorical question.

[Bdavis]

As others have said TrueNas Scale is barely out of beta and needs some time to mature.

 

TrueNAS core is rock solid, proven and reliable. As with any OS there is a learning curve. Lawrence Systems on YouTube by far has the best tutorials for it.

 

Windows Sucks

 

Linux can be made to do whatever you want, but it's much harder to use than TrueNAS IMO.

 

Hardware raid is dead.

 

TrueNas using all your ram is good. It uses any extra ram as a cache to speed things up.

 

[Droz_]

On 5/31/2022 at 11:30 AM, dbx10 said:

Many points:

• Hardware RAID is dead in the enterprise space, and for very good reasons. Even multiple thousand dollar RAID cards fail, and sometimes introduce data corruption into your array, which it is oblivious to. RAID was fine in the 1990s and early 2000s, when it was all we had.
• OpenZFS and hopefully soon BTRFS offer better availability and redundancy out of the box. Your ZFS pool remains accessible in case of a drive failure, and the software regulates data integrity. ZFS doesn't trust a drive as far as it can throw it.
• Windows server is only reliable within their ecosystem of services and validated partners. It CAN work, but there's also a reason why the use of Windows is at an all time low in the server space. It's just not as reliable or as flexible, and certainly not as stable.
• TrueNAS Core's "stable" release train goes through 8 rounds of validation before releasing, it's enterprise grade and rock solid. As for the plugins and the apps, virtualize them if you're not comfortable with BSD's jail system. I'm currently running an Ubuntu server box on bhyve just for my docker containers.

Sure, you can still run a RAID card and an obsolete redundant array, but ultimately there is no cost to running more modern and proven solutions. So why not?

Proxmox, TrueNAS Core, ZFS on the Linux distro of your choice, you can even distro hop and move your ZFS pool every time, should you want to.

I'd also like to know specifically what you mean by "unpolished". Genuinely asking, it's not a snarky rhetorical question.

I would love BTRFS on Truenas. Wishful thinking.

I won't need nor want ZFS caching everything in can in RAM

I disagree with hardware raid being dead, also the reliability of windows is great in my experience

[Droz_]

22 hours ago, Bdavis said:

As others have said TrueNas Scale is barely out of beta and needs some time to mature.

 

TrueNAS core is rock solid, proven and reliable. As with any OS there is a learning curve. Lawrence Systems on YouTube by far has the best tutorials for it.

 

Windows Sucks

 

Linux can be made to do whatever you want, but it's much harder to use than TrueNAS IMO.

 

Hardware raid is dead.

 

TrueNas using all your ram is good. It uses any extra ram as a cache to speed things up.

 

Using Core over Scale doesn't rectify the issues I've been having with the UI elements and learning curve to really know the ins and outs of truenas

Lawrence does have some great videos. But he skips lots of little steps and is a bit biased in his thought process/workflows in things. 

Don't need nor want ZFS using all my ram. Much of my data set is large files that are static. My personal data is under a terabyte and don't need things cached. 

Was recently having a hell of a time getting BTRFS running on Ubuntu. Kept going in circles trying to mount the volume, kept getting bad filesystem or mount point errors. Even asking a friend who does  things like this everyday, got him stumped. Ended up installing Windows 10 on my test server because I needed the drive space and working network shares for a quick project I have. 

[dbx10]

1 hour ago, Droz_ said:

I won't need nor want ZFS caching everything in can in RAM

This is not actually an argument against ZFS, because that's inconsequential to other programs you might be running. ZFS uses all available RAM for caching but dynamically frees any amount you need for other programs. It's designed that way, and your memory is never actually locked to the filesystem.

 

1 hour ago, Droz_ said:

I disagree with hardware raid being dead, also the reliability of windows is great in my experience

Hardware RAID is one memory chip or ROM failure away from an entire array becoming corrupt and inaccessible. Modern filesystems don't implicitly trust the hardware they run on, because hardware fails. Checksums don't lie, and that's why copy-on-write filesystems are inherently better.

 

If this was a conversation about movies, then we haven't watched the same movie you and I.