Should I connect m2 nvme and optane ssd drives via pcie expansion cards instead of chipsets builtin m2 slots which could potentially cause bottleneck?

tesseramous · October 22, 2018

I am planning a new x299 i9 or x399 threadripper build. For storage I will be using a 480gb optane 905p ssd which comes with a m2 adapter cable (not u2). I will also be using a 2tb 970 evo nvme ssd. The optane will be the boot drive and contain the pagefile. The 970 will be a 'data' drive.

First off, do m2 slots always use chipset lanes or do they sometimes use cpu lanes on certain motherboards or in certain configurations? I thought they were chipset lanes but sometimes the drive also has 'pcie' somewhere in the name and this confuses me.

I'm trying to decide whether to connect each of the SSDs via the onboard m2 in the motherboard chipset or connect them via a pcie m2 expansion card or pcie form factor drives. I will have 44 or 60 cpu lanes to spare on this beast platform and pcie cards would give each drive a dedicated 4 lanes to the CPU, instead of sharing the chipset. I know the DMI/UMI from the chipset to the CPU is only 4 lanes and it doesn't make sense to have multiple 4 lane drives a long with my 10 gig ethernet and everything else on the chipset sharing those 4 DMI/UMI lanes as a potential bottleneck.

Im also concerned that since Optane is supposed to have this ultra low latency of 10us, the chipset might add extra latency versus connecting directly to the cpu, and partially defeat the purpose of optane. So lets say I was to connect one drive to pcie expansion and one drive to the chipset (since there is technically enough bandwidth on the chipset for one), I would probably connect the optane to pcie expansion and the 970 to the chipset.

What do you guys think? Should I connect one or both of the drives via pcie expansion cards to solve potential bottlenecks and latency issues, or should I just utilize the m2 slots built into the motherboard? Thanks.

P.S.

I thought the 480gb optane 905p was only a U2 drive but I found just a couple sites that claim to sell it with an M2 adapter cable, like this one:
https://www.bhphotovideo.com/c/product/1410514-REG/intel_ssdpe21d480gam3_optane_ssd_905p_series.html

"Intel's 480GB Optane 905P U.2 Internal SSD includes an M.2 adapter cable" ...

Is this a typo, or does it really connect to M2?

Mira Yurizaki · October 22, 2018

Any latency introduced by the chipset is going to be inappreciable. The use case you're looking at isn't doesn't require that high performance. Also keep in mind that some PCIe slots on the motherboard may not actually wire up to the CPU.

In any case, if you have the available slots and the funds to support it, go for whatever makes you sleep better at night. But I don't expect any appreciable performance degradation going through the chipset.

Edited October 22, 2018 by M.Yurizaki

mariushm · October 22, 2018

Threadripper has 64 pci-e lanes but 4 of them go to chipset, so there's 60 lanes available to be routed to wherever motherboard manufacturer wants.

In most motherboards for TR,at least 2 m.2 connectors are wired directly to cpu, some have more...and some have fineprint like 3rd m.2 connector shares pci-e lanes with u2 connectors so if one is used the other gets disabled.

Intel boards.. usually they have stuff wired to chipset ... they advertise 44 pci-e lanes on some processors or something like that but only 16 come from cpu and the other are from chipset (and you get pretty much same speed between cpu an chipset, equivalent of pci-e x4)

You should ask yourself, do you really do stuff with those drives that you need such low latency? Wouldn't you be able to solve this problem by installing more ram and caching things in memory if latency is a problem?

tesseramous · October 22, 2018

If threadripper already connects the m2 slots directly to the cpu then that takes care of everything. Thanks!

Latency was mianly a concern in the case of using a pagefile. Right now RAM is still kind of expensive, especially the 3200cl14 kits. I plan to buy more in the future when it's cheaper or on sale. However, it still maxes out at 128GB.

mariushm · October 22, 2018

You may want to use ECC memory with Threadripper, in which case the maximum is 2933 I think

You may also want to consider timings (latencies) ... you may not gain significant performance once you go over something like 3000 Mhz but you may gain more by tweaking timings ... and also keep in mind the more sticks you have the more difficult it will be to "drive" all those sticks, so you may find out you can't keep 8 sticks running at 3200 Mhz

Mira Yurizaki · October 22, 2018

1 hour ago, mariushm said:

Intel boards.. usually they have stuff wired to chipset ... they advertise 44 pci-e lanes on some processors or something like that but only 16 come from cpu and the other are from chipset (and you get pretty much same speed between cpu an chipset, equivalent of pci-e x4)

The 44 some PCIe lanes were probably for the HEDT processors, though it's 40 for peripherals and lower end models have 28.