Issue with . . . Mainboard, SAS controller or videocard. With unraid machine

[Caennanu]

Gday all,

 

As stated in some other topics, i have repurposed my old MSI X370 Gaming pro Carbon and R7 1700 to be my unraid server.

And the road to what i want has hiccups, as is expected. 

 

Recently i couldn't get a 2nd graphics card to work, or atleast not on a pci-e x1 slot. Fixed this by disabling CSM.

Now however, i'm trying to add yet another one (jeeej virtualization?) but i'm running into issues here as well.

When i put in a 2nd GT710, my SAS controller (who's just a passthrough) starts to malfunction.

Unraid will start having read / write errors in my parity, heck it even disabled a parity disk and i had to rebuild it entirely . . . (luckily no dataloss).

 

Population of the PCI slots on the mainboard (uneven numbers are x1 slots):

1. GT710 (boot gpu)
2. GT1050Ti
3. 1GB Nic
4. SAS controller
5. GT710 (if it wouldn't mess up the controller)
6. disabled, due to using both m.2. slots

Now, i'm thinking with the 3rd videocard, i'm exceeding some bandwidth limit somewhere, or i'm missing some setting or whatever . . .

When i remove the card, it works fine. When i swap it with the card in slot 1, it works fine.

 

Anyone have a clue what could cause the SAS controller to have issues?

Any help appreciated (MSI support staff is . . . mehh)

 

p.s. i'm not using riser cables (i do have some laying around tho)

 

[thewill102]

I think the issue is, that your CPU doesnt have enough PCI-Lanes left to controll a thrid graphics card. I found that the R7 1700 supports up to 20 PCI-Lanes, in a configuration of 1x16+1x4. Im not exactly sure if you can split them p in ore, since you already have 4 running PCI devices.Try for example leaving out the 1050ti and putting the second GT710 in the 5th slot and see where that brings you.

 

I hope this might help you.

[Caennanu]

16 minutes ago, thewill102 said:

I think the issue is, that your CPU doesnt have enough PCI-Lanes left to controll a thrid graphics card. I found that the R7 1700 supports up to 20 PCI-Lanes, in a configuration of 1x16+1x4. Im not exactly sure if you can split them p in ore, since you already have 4 running PCI devices.Try for example leaving out the 1050ti and putting the second GT710 in the 5th slot and see where that brings you.

 

I hope this might help you.

i'm geussing the same. but there's also no way to actually control what slot gets how many lanes. which is kind of annoying.

The SAS controller can only use 4, but the slot is running at x8.

The 1050ti only needs 4 (for my purpose) but running at x8.

The other pci slots can only use 1, so thats 3x x1, + 2x x8 = 19, and then im geussing 4 to the chipset. which totals to 23.

But the SAS controller can only use 4, so we take 4 off again and we're back at 19.

Either way, i can't match lane utilization with applied maths . . .

 

Ill take the 1050ti out sometime soon, see what happens.

 

Edit:

So just found out, ryzen has 24 lanes. 16 for pci-e, 4 for chipset, 4 for m.2.

 

So . . . 1, 8, 1, 8, 1, makes me exceed by 3. unless somehow i can make slot 4 function at x4.

That does not explain however, that when exceeding it by 2 (as in not installing the 2nd GT710), everything works fine?

[Pixel5]

1 hour ago, Caennanu said:

i'm geussing the same. but there's also no way to actually control what slot gets how many lanes. which is kind of annoying.

The SAS controller can only use 4, but the slot is running at x8.

The 1050ti only needs 4 (for my purpose) but running at x8.

The other pci slots can only use 1, so thats 3x x1, + 2x x8 = 19, and then im geussing 4 to the chipset. which totals to 23.

But the SAS controller can only use 4, so we take 4 off again and we're back at 19.

Either way, i can't match lane utilization with applied maths . . .

 

Ill take the 1050ti out sometime soon, see what happens.

 

Edit:

So just found out, ryzen has 24 lanes. 16 for pci-e, 4 for chipset, 4 for m.2.

 

So . . . 1, 8, 1, 8, 1, makes me exceed by 3. unless somehow i can make slot 4 function at x4.

That does not explain however, that when exceeding it by 2 (as in not installing the 2nd GT710), everything works fine?

you will need to check the details for your exact motherboard to find out what is going on here.

probably some of the 1x slots share bandwidth with another slot to make it work at all.

 

the big question from my side would be why do you need all these GPU´s maybe theres a much better solution for this.

[Caennanu]

5 minutes ago, Pixel5 said:

you will need to check the details for your exact motherboard to find out what is going on here.

probably some of the 1x slots share bandwidth with another slot to make it work at all.

tried that, can't find the information i need unfortunately.

 

5 minutes ago, Pixel5 said:

you will need to check the details for your exact motherboard to find out what is going on here.

probably some of the 1x slots share bandwidth with another slot to make it work at all.

 

the big question from my side would be why do you need all these GPU´s maybe theres a much better solution for this.

For unraid, i need atleast a primary GPU so that it won't take an other gpu i want to use for the following purposes.

 

1. 1050Ti is to be used for shinobi CCTV
2. GT710 primary output Unraid
3. GT710 GPU for VM
4. SAS controller - 6 Sata ports is not enough. 4x 4TB primary array, and when unraid supports a 2nd array, 2x 2tb purples for CCTV. 2x SSD for VM's
5. 2x M.2, Cache disks, for Lan-Cache bundle primarily, but also plex encoding location and the likes

[Pixel5]

5 minutes ago, Caennanu said:

tried that, can't find the information i need unfortunately.

 

For unraid, i need atleast a primary GPU so that it won't take an other gpu i want to use for the following purposes.

 

1. 1050Ti is to be used for shinobi CCTV
2. GT710 primary output Unraid
3. GT710 GPU for VM
4. SAS controller - can never have enough Sata ports?
5. 2x M.2, Cache disks, for Lan-Cache bundle primarily, but also plex encoding location and the likes

i just checked your motherboard and now its clear how they do it.

 

your 2nd m.2 slot and all but two PCI-E Slots run at PCI-E 2.0 instead of 3.0 so they can take a few 3.0 lanes with a PCI-E switch and split them into twice as many 2.0 lanes.

That is also why slot 6 is disabled when using the 2nd M.2 slot, they ran out of 2.0 lanes to use so you have to decide.

 

that also means whatever bandwidth you think you have on these slots is actually only half in reality.

 

check if you really need another GPU for that VM, i dont have any GPU installed as i just run VM´s with the software emulated display adapter that is the default.

[Caennanu]

1 minute ago, Pixel5 said:

check if you really need another GPU for that VM, i dont have any GPU installed as i just run VM´s with the software emulated display adapter that is the default.

I do, unfortunately. this is a VM i have assigned 4/8 cores/threads to, and without gpu it maxes cores constantly. with the 1050 it drops to 20%. This frees up so much cycles for other purposes, i kind of have too.

[Caennanu]

Maybe its time to wait for a nice threadripper deal or something . . .  maybe even an epyc

[Caennanu]

@Pixel5 had you found a diagram of the mainboard btw? if so, could you share.

Or is it this one? because i'm not seeing amount of pci-e lanes here, or which slots go to which 'ports'

Spoiler

 

And when looking at this one.  . . . i should have sufficient lanes. As i'm not using PCI_E6

 

Spoiler

Because this would be my utilization then . . . which should in theory leave me 1 lane?

 

Spoiler

 

[Pixel5]

there is no diagram available but the specs of the motherboard were listing exactly what you found here.

 

what i suspect is that there is a limitation of the PCI-E switch in the chip set because the chipset itself is only connected via a 4x link all the lanes coming off the chipset will share this bandwidth.

 

Also i hope you dont really use the 1G NIC for anything that requires speed because a 20. 1x link will give you lees then 500mb/s throughput so you are cutting the already slow 1G LAN speed in half at best.

[Pixel5]

19 hours ago, Caennanu said:

Maybe its time to wait for a nice threadripper deal or something . . .  maybe even an epyc

it would probably be easier to switch to a CPU that has an iGPU so you can remove one GPU from the rig.

 

For that Intel is usually the better choice but then again AMD just announced 5000 series APU´s so maybe thats another good option once they become available.

[Caennanu]

1 hour ago, Pixel5 said:

there is no diagram available but the specs of the motherboard were listing exactly what you found here.

 

what i suspect is that there is a limitation of the PCI-E switch in the chip set because the chipset itself is only connected via a 4x link all the lanes coming off the chipset will share this bandwidth.

 

Also i hope you dont really use the 1G NIC for anything that requires speed because a 20. 1x link will give you lees then 500mb/s throughput so you are cutting the already slow 1G LAN speed in half at best.

It's configured as active-backup. Just in case my 9 camera's start recording at 5mp simultaniously

 

1 hour ago, Pixel5 said:

it would probably be easier to switch to a CPU that has an iGPU so you can remove one GPU from the rig.

 

For that Intel is usually the better choice but then again AMD just announced 5000 series APU´s so maybe thats another good option once they become available.

i hear you. waiting prices to see if thats going to be a viable option.

besides, Vega IGPU graphics are much better than a GT710. Its primarily a price / availability concern.