40Gbps NAS/Switch what OS should I use and is my CPU enough.

[aurelienVIE]

Hello,

 

I am building a new NAS and I was wondering was OS should I use and if my CPU will be enough.

My plan is to use it also as a 40/10Gbps switch to save money.

Here is what I plan to use.

-1* X5 1650V4 (I have it from an upgrade I did on my PC 40$)

-1* 16Gb DDR4 ECC REG (19$)

-1* Mellanox connectx-3 pro (45$)

-1* 2.5Gbps pcie1x card (9$)

-2* Crucial P3 NVME PCIE 3.0 (2*59$)

-1* 1.92To SanDisk ECO G2 SSD (80$)

-6* 6/8TB HDD (depending on what I find for 100$/each)

-2* 16G Intel OPTANE for the OS (2*5$)

 

I was planning to use Openmediavault with snapRAID and mergerfs to create a cash write/work cash like this.

2 NVME Raid 0 + SSD parity

4 HHD data + 2 HDD parity

All new files would be written to the NVME pool and on schedule old file would be transfer to the HDD Pool.

I need this NAS to store document and do photo/Video editing from 2 station that would be connected to it with 40Gbps DAC.

The internet connection would come through the 2.5Gbps card.

 

Would this plan work do you recommend a different OS?

Is the CPU powerful enough for the 40Gbps?

Is the RAM enough too? The motherboard can have up to 128GB.

 

Thank you for your advice and sorry if I made any mistake English is not my main language.

[LIGISTX]

1 hour ago, aurelienVIE said:

Hello,

 

I am building a new NAS and I was wondering was OS should I use and if my CPU will be enough.

My plan is to use it also as a 40/10Gbps switch to save money.

Here is what I plan to use.

-1* X5 1650V4 (I have it from an upgrade I did on my PC 40$)

-1* 16Gb DDR4 ECC REG (19$)

-1* Mellanox connectx-3 pro (45$)

-1* 2.5Gbps pcie1x card (9$)

-2* Crucial P3 NVME PCIE 3.0 (2*59$)

-1* 1.92To SanDisk ECO G2 SSD (80$)

-6* 6/8TB HDD (depending on what I find for 100$/each)

-2* 16G Intel OPTANE for the OS (2*5$)

 

I was planning to use Openmediavault with snapRAID and mergerfs to create a cash write/work cash like this.

2 NVME Raid 0 + SSD parity

4 HHD data + 2 HDD parity

All new files would be written to the NVME pool and on schedule old file would be transfer to the HDD Pool.

I need this NAS to store document and do photo/Video editing from 2 station that would be connected to it with 40Gbps DAC.

The internet connection would come through the 2.5Gbps card.

 

Would this plan work do you recommend a different OS?

Is the CPU powerful enough for the 40Gbps?

Is the RAM enough too? The motherboard can have up to 128GB.

 

Thank you for your advice and sorry if I made any mistake English is not my main language.

I really don’t think your going to see success with 40 gigabit networking. 10 will be a stretch, but 40…… I don’t think that is going to be possible.

 

40 gigabit, to put into perspective, is 1/3 as fast as the single channel of DDR4 RAM you have, and is faster then most NVMe drives. When you start trying to move data around that quickly, you very, very quickly run into issues you didn’t even know existed, like, does whatever OS kernel your using even have the ability to move data through the network stack, or even just from drives to NIC that fast.

 

Obviously, 40gigabit is possible, but it will take a good chunk of money, and a lot of time to get it actually going anywhere close to that fast. And…. Likely the only file system that will have much success at it would be ZFS, in which case you will probably want at least the maximum that mobo can support.

 

In closing; I’d shoot for consistent spikes up to 10gigabit, and averages somewhere below that. I don’t think it’s worth your time to try and shoot any higher unless you have the funds and the time to do so. 
 

My .02 anyways. 

[aurelienVIE]

Thanks for your reply. 

So if I want to get the best speed it would be best to populate all the ram to work on quad channels. 

Will it bring more speed if I aim for 4*32GB or it will be the same as 4*16GB?

 

If I plan for a 10Gbps will I have more speed if I use 10Gbps NIC or it will be the same if I use the 40Gbps NIC I have? 

[Nick7]

I'm sorry... but you talk about 40Gbit/s, which is a LOT. Rarely who needs such speeds. Nice to have a brag about it, yes. Needing it? Well.. different story.

 

Feeding 40Gbit/s from drives, any drives (including NVMe), is a lot.

[LIGISTX]

5 hours ago, aurelienVIE said:

Thanks for your reply. 

So if I want to get the best speed it would be best to populate all the ram to work on quad channels. 

Will it bring more speed if I aim for 4*32GB or it will be the same as 4*16GB?

 

If I plan for a 10Gbps will I have more speed if I use 10Gbps NIC or it will be the same if I use the 40Gbps NIC I have? 

If you already have the NIC’s, that’s fine. Just don’t expect to get anywhere near 40 through them. 

If you are serious about getting these sorts of speeds, you will need as much RAM as you can get, and honestly, you won’t get the speeds you are after. I’d go at least 64GB of RAM, and it’ll go as fast as it can go. Maybe post your requirements and what the use case is on the truenas forum and ask for advice - but be warned, they will also tell you it’s not going to happen. They may at least have some better advice on hardware choices tho seeing as your looking for performance that is orders of magnitude above what most of us here can actually provide help with. 
 

I just went 10 gigabit myself, and I get no where close to 10 gigabit speeds. Some decent info and reading about it can be found here: https://www.truenas.com/community/threads/10-gigabit-not-as-fast-as-expected.107896/

[aurelienVIE]

Hello thanks for your warning. I just finish assembling a test bench for it and after dealing with some pcie bifurcation problem I was able to test it. 

I manage a speed of 8.5 Gbps (875 Mo/s) in windows explore without any tweaking. It is the limit my local drive can read so I will need a faster way. Do you have recommendation for a software to test the speed?

In crystal mark I manage a 17Gbps peak in upload.

And I get 17Gbps on Iperf in client-server direction and 8Gbps on Server-Client direction

[LIGISTX]

2 hours ago, aurelienVIE said:

Hello thanks for your warning. I just finish assembling a test bench for it and after dealing with some pcie bifurcation problem I was able to test it. 

I manage a speed of 8.5 Gbps (875 Mo/s) in windows explore without any tweaking. It is the limit my local drive can read so I will need a faster way. Do you have recommendation for a software to test the speed?

In crystal mark I manage a 17Gbps peak in upload.

And I get 17Gbps on Iperf in client-server direction and 8Gbps on Server-Client direction

Set up a ramdisk and use that to test with.