New Server Build Have questions

[Bmoney]

Uses Plex Media Server, file server, Gamer Server, Personal Cloud, email, Video capture, and video encoding. 
Case: Supermicro 24 Bay chassis with a SAS846-7EL1 backplane

Motherboard: Asus Zenith Extreme (not the Alpha)

Processor: TR 2950X

Ram: ECC 128GB

Video Card: Vega 64

Dual 10G network card (Bonded into a Mikrotek switch)

SAS HBA: LSI 9300-8i (Both cables to the backplane)

Capture Card: El Gato HD 60

other : Asus NVMe Raid card for ZFS caching and torrenting

Power Supply: Cosair HXi 1200W

OS: Ubuntu 18

File system: ZFS with 3 vdevs of 8 drives in a pool. With record size matching data sets. ARC and Caching on the NVMe raid drives. 

Questions:

1) Should my vdevs be in raidz1 or 2? Yes 2 is safer but with 3 vdevs is it worth the lost in space? 
2) I was looking at the EPYC platform for the add server features (IMPI, More RAM, onboard SAS connectors, DUAL sockets) But Epyc performace doesn't seem to scale well in $/performance. With the caching NVMe the Ram shouldn't be a problem. The cost benefit didn't seem to be worth or am I wrong? 2950x only $650 and nearest perfomance Epyc is $899.
3)ZFS set up suggestions or am I right. Any tuning suggestion or ready Wendall level duh things I didn't mention?

[LIGISTX]

I am not really capable of commenting on the specs for things such as video capture/transcode or game server, but I do know for a simple file server and plex server, that is EXTREMELY overpowered. Obviously, 4k capture and transcode would drastically change my previous statement, but the setup in my sig runs 5 Ubuntu server VM's and FreeNas under ESXi, along with a Windows LTSC VM for Veeam backup, and it does it with no sweat at all and its a core i3... lol

 

Anyways, what I can say is the vdev setup. How large are your drives..? What are your redundancy goals? I would highly advice against RAID 5 (Z1) in todays world. Z2 is much more failure tolerant as well as being MUCH safer. Also, why not just run two larger vdevs and not lose as much space to redundancy? You could run 2x12 vdevs with RAID Z2...

 

Also, L2ARC is what will be on your SSD's, ARC is RAM. Just for clarification. And to that point, I don't know if your setup needs L2ARC. It may be a larger benefit depending on the setup to make a separate vdev for the SSD. Write to that as your fast capture storage, and have that offload to the spinning rust every hour or something with a cron job. That said, I could be totally wrong and maybe your plan to use a nvme drive for L2ARC will work out for your needs. I really don't know enough about the needs of 4k capture and how it will play with ZFS. From my experience, I can saturate my gigabit LAN with the setup in my sig, and my FreeNAS VM is running on 16GB of RAM total... My Plex has 2 threads of my i3 and 1.5 GB of RAM, and I can transcode multiple 1080p streams down to 720p at once, and can juuuuust barely do a 4k to 1080 transcode on the fly. It does it fine, but its not stoked about building up a bit of a buffer in the beginning of the transcode, and it never uses more than ~1.2 GB of RAM. And that is running Ubuntu server 18.04.

 

Anyways, thats my two cents. Hopefully that is helpful.

[Windows7ge]

1) This is really dependant on how easily you can recover the data should you lose a vdev. If you have proper backups then you can opt for raidz1. It's still some fault tolerance. Just keep a spare drive around to resilver with immediately.

 

2) Going with EPYC would really only be for the features specific to the server platform. TR can already be used for hosting many VMs, device pass-though, expansion cards can be used for most things you'd get on a SP3 board. The single core performance will be better and you can overclock it if you need to so I think TR would be fine for this broad scope of tasks. Only thing you're really missing is IPMI but hopefully you'll never miss what you never had .

 

3) I'm not certain as to what benefits come from 3x8 disk vdevs as oppose to one large 24 disk raidz2. I've been told ZFS does a great job at handling large arrays of drives but I'd have to do some research of my own to understand why 3x8 would be better. Normally adding vdevs is a means of expanding the storage of an existing pool. I'm not certain if it has any other intended purpose. I'm thinking if you have all the disks available creating one large vdev would be better but perhaps this would hurt fault tolerance in a way I just don't understand.

[LIGISTX]

2 minutes ago, Windows7ge said:

3) I'm not certain as to what benefits come from 3x8 disk vdevs as oppose to one large 24 disk raidz2. I've been told ZFS does a great job at handling large arrays of drives but I'd have to do some research of my own to understand why 3x8 would be better. Normally adding vdevs is a means of expanding the storage of an existing pool. I'm not certain if it has any other intended purpose. I'm thinking if you have all the disks available creating one large vdev would be better but perhaps this would hurt fault tolerance in a way I just don't understand.

Having less disks used to result in higher performance. I can't say that isn't still the case, but.... its mostly a non issue from what I understand these days.

 

The only data integrity difference would be simple statistics. More drives in a vdev = larger chance of failure and larger chance of failure during a resilver which is the largest issue and why Z1 is more or less a horrible idea. But.... that said, as you stated, proper backups help this dramatically.

 

First, second, and third rule of RAID, RAID IS NOT A BACKUP. lol. But I agree with everything you have said, at least with what limited knowledge I have. I definetly know enough to be dangerous, but the folks on freenas forum for instance would know a LOT more about the intricacies of ZFS. OP may want to check that out, they are an abundance of knowledge that I frequent for such things.

[Bmoney]

4 minutes ago, Windows7ge said:

1) This is really dependant on how easily you can recover the data should you lose a vdev. If you have proper backups then you can opt for raidz1. It's still some fault tolerance. Just keep a spare drive around to resilver with immediately.

 

2) Going with EPYC would really only be for the features specific to the server platform. TR can already be used for hosting many VMs, device pass-though, expansion cards can be used for most things you'd get on a SP3 board. The single core performance will be better and you can overclock it if you need to so I think TR would be fine for this broad scope of tasks. Only thing you're really missing is IPMI but hopefully you'll never miss what you never had .

 

3) I'm not certain as to what benefits come from 3x8 disk vdevs as oppose to one large 24 disk raidz2. I've been told ZFS does a great job at handling large arrays of drives but I'd have to do some research of my own to understand why 3x8 would be better. Normally adding vdevs is a means of expanding the storage of an existing pool. I'm not certain if it has any other intended purpose. I'm thinking if you have all the disks available creating one large vdev would be better but perhaps this would hurt fault tolerance in a way I just don't understand.

1) They will be 8-10TB so re-silvering with take a bit. 

2) Yes and a lot web based management aviable but nice to know If on vacation I don't have to worry about pushing buttings. 

3) I was told the 3 vdevs help when resilvering and basically so I don't have to drop nearly 3K on drives in one buy. 

[Bmoney]

8 minutes ago, LIGISTX said:

I am not really capable of commenting on the specs for things such as video capture/transcode or game server, but I do know for a simple file server and plex server, that is EXTREMELY overpowered. Obviously, 4k capture and transcode would drastically change my previous statement, but the setup in my sig runs 5 Ubuntu server VM's and FreeNas under ESXi, along with a Windows LTSC VM for Veeam backup, and it does it with no sweat at all and its a core i3... lol

 

Anyways, what I can say is the vdev setup. How large are your drives..? What are your redundancy goals? I would highly advice against RAID 5 (Z1) in todays world. Z2 is much more failure tolerant as well as being MUCH safer. Also, why not just run two larger vdevs and not lose as much space to redundancy? You could run 2x12 vdevs with RAID Z2...

 

Also, L2ARC is what will be on your SSD's, ARC is RAM. Just for clarification. And to that point, I don't know if your setup needs L2ARC. It may be a larger benefit depending on the setup to make a separate vdev for the SSD. Write to that as your fast capture storage, and have that offload to the spinning rust every hour or something with a cron job. That said, I could be totally wrong and maybe your plan to use a nvme drive for L2ARC will work out for your needs. I really don't know enough about the needs of 4k capture and how it will play with ZFS. From my experience, I can saturate my gigabit LAN with the setup in my sig, and my FreeNAS VM is running on 16GB of RAM total... My Plex has 2 threads of my i3 and 1.5 GB of RAM, and I can transcode multiple 1080p streams down to 720p at once, and can juuuuust barely do a 4k to 1080 transcode on the fly. It does it fine, but its not stoked about building up a bit of a buffer in the beginning of the transcode, and it never uses more than ~1.2 GB of RAM. And that is running Ubuntu server 18.04.

 

Anyways, thats my two cents. Hopefully that is helpful.

I have 4k HDR videos I like playblack and by dual xeon x5430s just are hanging on 1 stream all the time now. 

 

[Bmoney]

Oh I really would like to be able to saturated the dual 10G connection as much as possible. Speed junking here so I know I am not going to get 5G/s writes but want to get as close as possible. 

[Bmoney]

 

LIGISTX

 

"Write to that as your fast capture storage, and have that offload to the spinning rust every hour or something with a cron job"

Do you have a link or can explain this more?

 

[LIGISTX]

7 minutes ago, Bmoney said:

I have 4k HDR videos I like playblack and by dual xeon x5430s just are hanging on 1 stream all the time now. 

 

Those 5430’s are pretty garbage by today’s standards. My i3 can transcode a 4k HDR to 1080p... and only half of it at that since my Plex VM only gets 2 threads of my 4 total, lol. 

 

But that’s just for testing purposes. I normally stream native content to my chrome cast through Plex, no transcode needed. My server doesn’t even try for that because it’s not transcoding... 

[LIGISTX]

4 minutes ago, Bmoney said:

 

LIGISTX

 

"Write to that as your fast capture storage, and have that offload to the spinning rust every hour or something with a cron job"

Do you have a link or can explain this more?

 

I don’t have a link, but I would say go check out freenas forum and post there. They will understand ZFS much better than I do.

 

All I am saying here is you can set up an SSD as the target for your high speed capture needs, and have it offload to the 12x2 or whatever vdev artsy hourly. Or daily. Whatever the case may be. But really, check out freenas forum. They know WAY more than I do. 

[Windows7ge]

7 minutes ago, LIGISTX said:

The only data integrity difference would be simple statistics. More drives in a vdev = larger chance of failure and larger chance of failure during a resilver which is the largest issue and why Z1 is more or less a horrible idea. But.... that said, as you stated, proper backups help this dramatically.

Then I would definitely agree with your proposal of 2x12 raidz2. I think it'd be the best mix of everything.

 

Trust me, I am a strong proponent for the use of raidz2 over raidz1 but depending on how replaceable his data is and if usable storage is more important then to each their own raidz1 is still an option.

 

11 minutes ago, LIGISTX said:

First, second, and third rule of RAID, RAID IS NOT A BACKUP.

I second this. The reason I referred to RAID as "Fault Tolerance" and not "Backup". ZFS is a wonderful file system but things outside ZFS's control can still go wrong. I'm a strong proponent for the 3-2-1 backup rule. I lost all my data once. Never again.

 

12 minutes ago, Bmoney said:

1) They will be 8-10TB so re-silvering with take a bit. 

2) Yes and a lot web based management aviable but nice to know If on vacation I don't have to worry about pushing buttings. 

3) I was told the 3 vdevs help when resilvering and basically so I don't have to drop nearly 3K on drives in one buy. 

1) The reason raidz2 is preferred but the risk only really goes up as the disks age. Especially if you purchased all identical drives from the same place at the same time. 5, 7, or even 10 years from now when you're resilvering the array you'll have no fault tolerance if a disk fails due to age.

 

2) From remote locations you'd still have to remote into a VM or a native machine on the network to access the IPMI. It's a great tool which unfortunately does not have good 3rd party options if you want an add-in-card or external option.

 

3) Possibly. If the unique parity data is special to each vdev then fewer disks per vdev should allow resilvering to "go faster". I can't say that definitively though. Perhaps @LIGISTX can though?

 

If you only want to buy 8 disks at a time then yes, definitely, appending vdevs like this will allow you to expand the existing pools storage capacity. Unfortunately I believe (though can't say this definitely) adding vdevs progressively won't increase the read/write performance of the pool. Take that with a grain of salt though.

[Bmoney]

3 hours ago, LIGISTX said:

I don’t have a link, but I would say go check out freenas forum and post there. They will understand ZFS much better than I do.

 

All I am saying here is you can set up an SSD as the target for your high speed capture needs, and have it offload to the 12x2 or whatever vdev artsy hourly. Or daily. Whatever the case may be. But really, check out freenas forum. They know WAY more than I do. 

Help me I been looking. What should I be searching for on the forum.

[LIGISTX]

13 hours ago, Windows7ge said:

 

3) Possibly. If the unique parity data is special to each vdev then fewer disks per vdev should allow resilvering to "go faster". I can't say that definitively though. Perhaps @LIGISTX can though?

 

I’m not sure either. I don’t know enough to know I’d that is true or not. 

[LIGISTX]

10 hours ago, Bmoney said:

Help me I been looking. What should I be searching for on the forum.

I’d just post over there. Explain the use case, and see what answers you get.