2400G Pcie Lanes for NAS

[FloRolf]

Hello everyone,

 

This is yet another thread about my NAS build because I'm kinda desperate, lol.

 

I have the following components already:
R5 2400G
Gigabyte B450 Aorus M
16gb G.Skill Aegis 3000mhz
Dual 10gbe SFP+ NIC (PCIe 8x)

 

And will (want to?) add these now/later:
6x4TB WD Red or Seagate Ironwolf (in Raidz2) 
1x 250gb Samsung 970evo nvme (As Cache for the RaidZ2) 
2x500gb Samsung 850evo SATA (For Super fast storage in Raid0) 

Now when we add things up, 
I need 8 SATA ports and 12 pcie (4 nvme 8 for 10gbe) lanes. 

 

The problem is, the board only has 6 SATA ports but according to the manual, 2 of them will be disabled when I plug in the nvme drive. 
That means out of the box the motherboard supports 4x SATA + 1x nvme + 8x pcie with the CPU I have. 

 

That again means I still need 4 more SATA3 ports which I could connect only via PCIe BUT I don't have any pcie lanes left, plus the boards second x16 slot only runs pcie 2.0 x 4 (=16gbit=just 2 full SATA 6gbit) and the 1x slot is also not enough, theoretically. 

 

 

So what do I do? 
Do I need a new CPU? 
Do I need a new board? 
Both? 
Any other way to connect a bunch of drives and set them up as I want? 

 

 

My thoughts:
I'm pretty sure I have to get a new CPU, maybe a R5 1400 and at least some X370 board with full dual pcie 3.0 x8 slots. 

 

 

@manikyath may be able to help me out again?

 

 

Cheers guys! 

 

 

Edit:

Oh yeah and please keep in mind I'll be running FreeNAS so I don't need a GPU, only for the for initial 5minutes of installing the OS (hence the 2400G). I could get away with installing a dedicated gpu for setup though and replace it by the NIC after. 

[Streetguru]

5 minutes ago, FloRolf said:

 

You should have gotten a used server of some kind if you needed that many PCI-e lanes and don't want to be bottle necked by chipsets or PLX

[NelizMastr]

The 2400G only has 8 PCIe lanes, where other Ryzen chips without GPUs have 16. There's another 4 coming from the chipset and that's it. 

So there's nothing you can do in this case, unless you give up at least your NVMe.

[Olaf6541]

11 minutes ago, FloRolf said:

The problem is, the board only has 6 SATA ports but according to the manual, 2 of them will be disabled when I plug in the nvme drive. 

Are you sure? My board only disables sata ports when I put a sata sdd in my m.2 slot.

 

Then you can do this:

put the 10gb card in the 16x slot;

the nvme in the m.2 slot;

6 sata drives on the sata ports (assuming the nvme drive won't take any sata ports);

get pcie to sata controllers for the 2 sata evo drives and put in the 4x slot?

[Zagna]

11 minutes ago, Olaf6541 said:

Are you sure? My board only disables sata ports when I put a sata sdd in my m.2 slot.

With AM4, the 4x PCIe block meant for storage from the CPU/APU operates as either 2x SATA or NVMe. So with NVMe, the 2 ASATA ports on AM4 boards, get disabled.

[Olaf6541]

6 minutes ago, Zagna said:

With AM4, the 4x PCIe block meant for storage from the CPU/APU operates as either 2x SATA or NVMe. So with NVMe, the 2 ASATA ports on AM4 boards, get disabled.

Yeah I just looked up the manual of this board, and 8 sata devices, a 4 lane drive and an 8 lane card is just too much for AM4 and a cpu that only has 8 lanes...

Maybe change from 6x4tb raidz2 to 4x5tb raidz1?

[FloRolf]

38 minutes ago, Streetguru said:

You should have gotten a used server of some kind if you needed that many PCI-e lanes and don't want to be bottle necked by chipsets or PLX

Well, the idea was that I already have the 2400G and I was hoping for less power consumption due to the ryzen being much newer than some old Xeon Server. 

 

31 minutes ago, NelizMastr said:

The 2400G only has 8 PCIe lanes, where other Ryzen chips without GPUs have 16. There's another 4 coming from the chipset and that's it. 

So there's nothing you can do in this case, unless you give up at least your NVMe.

Okay, yeah I see. 

But if I got even just the R3 1200 or R5 1400 I'd get the 16 lanes? 

I could split those into 

x8 for NIC

x4 for Nvme

x4 for SATA Controller 

And as they are all 3.0 the bandwidth should be enough? 

 

20 minutes ago, Zagna said:

With AM4, the 4x PCIe block meant for storage from the CPU/APU operates as either 2x SATA or NVMe. So with NVMe, the 2 ASATA ports on AM4 boards, get disabled.

Correct

 

15 minutes ago, Olaf6541 said:

Yeah I just looked up the manual of this board, and 8 sata devices, a 4 lane drive and an 8 lane card is just too much for AM4 and a cpu that only has 8 lanes...

Yeah I think so. This is just too much. 

 

So I guess to start this project, I might just get a different, x370/x470 board now and hook it up to the 2400G and 6sata hdds and when I want to get the other 2 SSDs and the nvme ssd I'll just replace the cpu. 

This is due to being still able to return the board but not the cpu. 

[NelizMastr]

33 minutes ago, FloRolf said:

Well, the idea was that I already have the 2400G and I was hoping for less power consumption due to the ryzen being much newer than some old Xeon Server. 

 

Okay, yeah I see. 

But if I got even just the R3 1200 or R5 1400 I'd get the 16 lanes? 

Yes, but you wouldn't have a video output, so you're losing some lanes there since a video card will be needed. You have an x1 slot available, so you could possibly use that for something stupid like this :

 

[FloRolf]

1 hour ago, NelizMastr said:

Yes, but you wouldn't have a video output, so you're losing some lanes there since a video card will be needed. You have an x1 slot available, so you could possibly use that for something stupid like this :

 

~Snippety McSnippet~

Lmao, I didn't know this existed. 

 

But iirc you only need a video output for the I initial setup. After that you can run it headless and do everything through the browser. 

[NelizMastr]

4 minutes ago, FloRolf said:

Lmao, I didn't know this existed. 

 

But iirc you only need a video output for the I initial setup. After that you can run it headless and do everything through the browser. 

If your board is okay with that, yep. Some boards will hang at the VGA test.

[Olaf6541]

1 hour ago, NelizMastr said:

Yes, but you wouldn't have a video output, so you're losing some lanes there since a video card will be needed. You have an x1 slot available, so you could possibly use that for something stupid like this :

 

But the x1 slot is wired to the chipset so that will cost pcie capacity from the chipset (just like adding a m.2 drive uses up sata ports) so you would need a 8port hba connected to the cpu 16x slot, hook up all drives there and hope the throughput will be high enough, then hook up the nvme with m.2 and 10gb card with pcie 2.0 4x to the chipset.

 

Running headless is fine, until something happens and have to disconnect drives to inspect the problem...

[Salacious B Crum]

What exactly are you doing with 10gbe are you going peer to peer on 2 other rigs?

[FloRolf]

1 hour ago, NelizMastr said:

If your board is okay with that, yep. Some boards will hang at the VGA test.

I mean, doesn't every boars with video outputs allow for you to use Vega graphics? 

1 hour ago, Olaf6541 said:

But the x1 slot is wired to the chipset so that will cost pcie capacity from the chipset (just like adding a m.2 drive uses up sata ports) so you would need a 8port hba connected to the cpu 16x slot, hook up all drives there and hope the throughput will be high enough, then hook up the nvme with m.2 and 10gb card with pcie 2.0 4x to the chipset.

 

Running headless is fine, until something happens and have to disconnect drives to inspect the problem...

You are right, it might be a problem at very rare times. I think I'd just swap the nic for a gpu when there's a problem. 

29 minutes ago, mrbilky said:

What exactly are you doing with 10gbe are you going peer to peer on 2 other rigs?

Correct. I don't want to blow the 500€ for a 10gbe RJ45 switch and 3x100€ for the NiCs. There are some cheap SFP+ cards with two ports. 

[NelizMastr]

3 minutes ago, FloRolf said:

I mean, doesn't every boars with video outputs allow for you to use Vega graphics? 

I thought you wanted to run headless with a 16-lane Ryzen CPU, sorry  

[FloRolf]

17 minutes ago, NelizMastr said:

I thought you wanted to run headless with a 16-lane Ryzen CPU, sorry  

Oh okay, I thought you mean some x370 boards will not support the Vega 11. I'll take a look in the freenas forums again to see if you can run/boot it headless. 

If so, I'll just get an 1st Gen ryzen in a couple months when I get the nvme/sdds. Afaik you can add the nvme cache later on to the raidz array. 

[Salacious B Crum]

4 hours ago, FloRolf said:

Correct. I don't want to blow the 500€ for a 10gbe RJ45 switch and 3x100€ for the NiCs. There are some cheap SFP+ cards with two ports. 

Ok yeah I've have the same 10gbe setup on my NAS to my 2 Ryzen rigs works great

[FloRolf]

Just now, mrbilky said:

Ok yeah I've have the same 10gbe setup on my NAS to my 2 Ryzen rigs works great

Oh cool, very nice. 

What kinda nics do you use? I've heard for freenas only Intel and chelsio really works. 

[Salacious B Crum]

37 minutes ago, FloRolf said:

Oh cool, very nice. 

What kinda nics do you use? I've heard for freenas only Intel and chelsio really works. 

 HP Mellanox ConnectX-2 10 GbE PCI-e G2 Dual SFP+ Ported Ethernet HCA NIC

 HP 10GB Single Port Mellanox Connectx-2 PCI-E 10GBe NIC

 

My NAS is running unRAID as the OS

[AngryBeaver]

So if you aren't used a video card for this build and using the one on the cpu... then

 

So you have

4 sata

1 M.2

then if you add https://www.newegg.com/Product/Product.aspx?Item=9SIA1UH7NS6294&cm_re=8_port_sata_III_pci_card-_-16-124-087-_-Product

thats 8 more sata for 1 pci-e lane

 

Just noticed you want to add a 8x 10gbe card.

https://www.amazon.com/Port-Express-Gigabit-Ethernet-Network/dp/B00LPRS36K
Is a 4x pci-e 10gbe card

Unless I am missing something this SHOULD work.

 

[FloRolf]

23 minutes ago, AngryBeaver said:

So if you aren't used a video card for this build and using the one on the cpu... then

 

So you have

4 sata

1 M.2

then if you add https://www.newegg.com/Product/Product.aspx?Item=9SIA1UH7NS6294&cm_re=8_port_sata_III_pci_card-_-16-124-087-_-Product

thats 8 more sata for 1 pci-e lane

 

Just noticed you want to add a 8x 10gbe card.

https://www.amazon.com/Port-Express-Gigabit-Ethernet-Network/dp/B00LPRS36K
Is a 4x pci-e 10gbe card

Unless I am missing something this SHOULD work.

 

Thanks for the reply

You are sadly missing on one thing and that is i will be using a dual 10gbe card, which needs an 8x slot

 

However, how does a 8x SATA controller work with pcie 1x?

With pcie3.0 that is just 8gbit of bandwidth for 8x SATA3, which needs 6x8=48gbit bandwidth.

Theres 40gbit missing here, lol. 

[AngryBeaver]

4 minutes ago, FloRolf said:

Thanks for the reply

You are sadly missing on one thing and that is i will be using a dual 10gbe card, which needs an 8x slot

 

However, how does a 8x SATA controller work with pcie 1x?

With pcie3.0 that is just 8gbit of bandwidth for 8x SATA3, which needs 6x8=48gbit bandwidth.

Theres 40gbit missing here, lol. 

Because if you are using HDD like it looks like you have in your list above.. then you will never need that much throughput. Lets assume these drives are hitting 160MB read speeds times 8. That is only 1,280 MB. pci-e 3.0 does about 985 MBs per lane each direction(1970MBs total). So it would be a bottleneck if you use all slots and are running everything capped out in one direction... but that is very seldom the case even if they were in a raid setup. Now if you decide to move to SSD's on here that changes drastically, but I was figuring you would only use 4-6 of those ports anyways, using all would run you out of case space anyways would be my guess. So running 4-6 7200 drives on that card would be more than fine on 1 pci-e lane.

 

Also Pci-E isn't calculated on Gbits. It is GT which is GigaTransfers. Don't get me wrong they are close in value, but Gb equiv to GT is less. Just remember 8GT/s (985 MB/s) which is the per lane speed. You can convert that to Gb which would be 7880Mbs or 7.8Gb. When you say sata 6gb that is the theoretical maximum that sata can move over it's connection... a 7200 rpm drive will never hit that neither will a 2.5 ssd. M.2 and u.2 are the only things that can really saturate that connection atm. So you would need a HDD or an SSD that could read/write faster than 750MBs (the fastest atm is the samsung 960 pro 2.5, it does 560MBs read/write).

 

Anyways my point is your should be fine. Here is a dual port spf card for 10gbe
https://www.ebay.com/p/Solarflare-SFN5122F-Dual-Port-10gbe-Pci-e-Server-Card/1150826696?iid=173181555878
 

So lets says you are using both full out that is 20gbe maximum theoretical or 2.5GBs (2500MBs)  so 8 x 985 = 7880MBs or 7.88GBps  or 63 Gbps. So that is more than enough to cover your 10Gbe connection need.

[FloRolf]

12 hours ago, AngryBeaver said:

Because if you are using HDD like it looks like you have in your list above.. then you will never need that much throughput. Lets assume these drives are hitting 160MB read speeds times 8. That is only 1,280 MB. pci-e 3.0 does about 985 MBs per lane each direction(1970MBs total). So it would be a bottleneck if you use all slots and are running everything capped out in one direction... but that is very seldom the case even if they were in a raid setup. Now if you decide to move to SSD's on here that changes drastically, but I was figuring you would only use 4-6 of those ports anyways, using all would run you out of case space anyways would be my guess. So running 4-6 7200 drives on that card would be more than fine on 1 pci-e lane.

 

Also Pci-E isn't calculated on Gbits. It is GT which is GigaTransfers. Don't get me wrong they are close in value, but Gb equiv to GT is less. Just remember 8GT/s (985 MB/s) which is the per lane speed. You can convert that to Gb which would be 7880Mbs or 7.8Gb. When you say sata 6gb that is the theoretical maximum that sata can move over it's connection... a 7200 rpm drive will never hit that neither will a 2.5 ssd. M.2 and u.2 are the only things that can really saturate that connection atm. So you would need a HDD or an SSD that could read/write faster than 750MBs (the fastest atm is the samsung 960 pro 2.5, it does 560MBs read/write).

 

Anyways my point is your should be fine. Here is a dual port spf card for 10gbe
https://www.ebay.com/p/Solarflare-SFN5122F-Dual-Port-10gbe-Pci-e-Server-Card/1150826696?iid=173181555878
 

So lets says you are using both full out that is 20gbe maximum theoretical or 2.5GBs (2500MBs)  so 8 x 985 = 7880MBs or 7.88GBps  or 63 Gbps. So that is more than enough to cover your 10Gbe connection need.

I'm not sure if all that is correct but I think you are right. If I ran just all the hdds over the pcie 1x slot then it would probably be enough because a raidz2 array for 6 drives gives for around 400mb/s speeds, I think in both directions (roughly). 

 

Here's another think though, I just found out there are X370 boards (Asus Prime X370 Pro) with 8x SATA ports. I had a quick look in its manual but didn't find anything about SATA ports being disabled when using the M.2 slot. That would result in the correct amount of ports and pcie lanes even with a 2400G?!

Can anyone confirm this? 

@mrbilky @NelizMastr