100gbe NAS-4-cheap

[daslolo]

I have two mellanox(es?) and want to play NAS4cheap.

 

I think that pcie lane will be the bottleneck on i5-8400 (my previous cpu) so i'm exploring server grade equipment.

 

e5 is by far the cheapest high lane count cpu.

 

Would a 2650L (oh yeah i want quiet) on a x99 board with 4x nvme handle a mellanox connectX-4 100gbe? by "handle" i mean the bottleneck is not the pcie dmi or cpu.

 

I know a 1920x can handle that but... 40W idle and not *as* cheap

[mariushm]

100gbps is 12.5 GB/s  ... that's 16 pci-e 3.0 lanes (16 x 0.98 GB/s =15.68 GB/s). 

 

The problem will be the pci-e lanes for those nvme drives will be coming from chipset, which is connected to cpu through a pci-e 4.0 link (well dmi or whatever Intel calls it, but it's pci-e links)  

So you'll have ONE nvme connected to CPU (maybe) at pci-e 3.0 x4  (4 GB/s) and the others will be all chocked by the pci-e 3.0 x4 connection between chipset and cpu (or whatever connection it is)

 

There's boards like Gigabyte B550 Aorus Master for example, which split the 16 pci-e lanes from CPU into 8 lanes going to video card slot and  8 lanes split into two M.2 slots so along with the third m.2 slot that uses the dedicated 4 links from cpu, you'd have  3  nvme pci-e 4.0 x4  M.2 slots.  You can plug a cheap 10$ video card in the bottom pci-e slot that gets 2-4 pci-e lanes from the chipset.

 

So you could plug your 100g melanox card into the pci-e x16 slot with only 8 lanes if you use the 2 m.2 slots  so if the card runs in pci-e 3.0 mode you'd have 8 lanes x 0.9 GB/s = ~7.5 GB/s or 60 gbps which is still decent.

 

You would have to check the x99 board to see where the pci-e lanes are coming from. 

The boards which will work with a 1920x will have 48 pci-e 3.0 lanes, some will support bifurcation so you get adapter boards that put 2-4 nvme drives into a pci-e slot ... but not sure that the cpu can sustain 100 gbps throughput either way. 

 

 

[manikyath]

you're not gonna saturate 100G with any consumer hardware, that's the sort of speeds where disk I/O starts to eat a LOT of cpu resources. (see the new new new new new whonnock videos from LTT..)

[daslolo]

8 hours ago, manikyath said:

you're not gonna saturate 100G with any consumer hardware, that's the sort of speeds where disk I/O starts to eat a LOT of cpu resources. (see the new new new new new whonnock videos from LTT..)

The linux whonnock is the newest I found and it doesn't talk about consumer hardware, another one maybe?

I'll test on this desktop, it's a z790 master where all m.2 are gen4+, one gen5 is cpu and one is pch.

pcu to pch: 3.4 GB/s sustained

cpu to cpu and pch to pch drops to 2 GB/s

And cpu usage is going to be an issue

now pch to pch is either bottlenecked by the gen4 m.2 or pch->dmi->pch roundtrip, I wish there was a bus graph in task manager

Anyways, this quick and super non scientific test shows that yep, consumer hardware ain't gonna cut it and perhaps an old 2650Lv4 will do much better.

10 hours ago, mariushm said:

The problem will be the pci-e lanes for those nvme drives will be coming from chipset, which is connected to cpu through a pci-e 4.0 link (well dmi or whatever Intel calls it, but it's pci-e links)  

Looking at the x99 block diagram I can see that 16 GT/s is the maximum. But it also looks like 2x pcie 16 (3.0 I guess?) go straight to the CPU, no DMI bottle neck here. If that's true then the mellanox goes on one port, a Asus quad nvme goes to the other port and maybe it'll work.

 

10 hours ago, mariushm said:

You would have to check the x99 board to see where the pci-e lanes are coming from. 

The boards which will work with a 1920x will have 48 pci-e 3.0 lanes, some will support bifurcation so you get adapter boards that put 2-4 nvme drives into a pci-e slot ...

Not sure this bios has a bifurcation option.

10 hours ago, mariushm said:

... but not sure that the cpu can sustain 100 gbps throughput either way. 

I realize that, at these speed, the bottleneck moves up the chain.

I wonder if servers have a transfer mode that bypasses the cpu or if the cpu is just a fast switch.

The 2560L has a bus rate of 2x9.6GT/s so that's about 7.5GB/s both ways at the same time, the 1920x and 8400 are twice that and the 13600K is 4x that (8x 2560L) and yet the 1920x can be used in a file server but not the 13600k so maybe "bus rate" isn't the right number to look at, it could be a combination of pcie lanes and bus rate.

[mariushm]

I'm not aware of a 2560L ... there's a Xeon E5-2650L but that one is with DDR3 controller, it does only DDR3 ... Intel Xeon E5-2650L - CM8062107185309

The Gigabyte board above seems to be DDR4 only.

[daslolo]

7 minutes ago, mariushm said:

I'm not aware of a 2560L ... there's a Xeon E5-2650L but that one is with DDR3 controller, it does only DDR3 ... Intel Xeon E5-2650L - CM8062107185309

The Gigabyte board above seems to be DDR4 only.

typo. the 2650K and that gaming mobo are compatible