This SSD is Faster Than Your RAM

[AlexTheGreatish]

The Apex Storage X21 is an absolutely wild AIC that allows you to connect twenty one SSDs to a single PCIe slot... and it is very fast.

 

Thanks again to Sabrent for sending us all those SSDs! Check them out at https://sabrent.com/

Check out the Apex Storage X21 NVMe AIC: https://lmg.gg/OFxJ9

Buy a Sabrent Rocket 4 Plus 8TB NVMe M.2 SSD: https://geni.us/O10iXs

Buy a Gigabyte AORUS Gen4 AIC Adaptor: https://geni.us/SRGL9j

Purchases made through some store links may provide some compensation to Linus Media Group.

 

 

 

[olderpybasted]

Long time casual viewer, and I literally signed up for the forums just to mention this...

In the video you guys mention that you couldn't find anything to actually stress this storage... I may actually may have something I've recently put together that could. I have few scripts I put together for my work that can convert a bog-standard Windows 10/11 ISO into a large, PXE or USB bootable Windows PE .WIM/RamDisk for modern UEFI/8GB+ systems.

Basically these ram disks 'cram' the entire install.esd/install.wim source for a windows install into the "X:" RamDisk when Windows PE is built (boot.wim), meaning that windows setup dumps the install image out on disk as fast as possible - from RAM.

Right now I use solid-block, maximum-compressed install.esd to ensure things fit under 6-8gbs -- but it could just as easily be tweaked to support a fully-uncompressed install.wim as the source (provided its booted up on a machine with 32gbs+ of ram).

I haven't really put it up on github or anything yet but probably will once I de-spaghettifi the scripts and make it an easier to understand process. However I could send it your way if you'd like to try and build one out, DM me for details.

[Ottoman420]

Funny enough i have a 600gb WD Velociraptro 10,000rpm still works it was an RMA for the original raptor with the window screeen

[Kilrah]

2 hours ago, olderpybasted said:

Basically these ram disks 'cram' the entire install.esd/install.wim source for a windows install into the "X:" RamDisk when Windows PE is built (boot.wim), meaning that windows setup dumps the install image out on disk as fast as possible - from RAM.

That's not a sustained load of tens of TBs like you'd want for testing this though.

[olderpybasted]

57 minutes ago, Kilrah said:

That's not a sustained load of tens of TBs like you'd want for testing this though.

Yeah, not with the default setup where it just does behaves like a typical Windows Install disk and the ramdisk disappears when the machine reboots to go on to stage 2 of the install from the HDD/SSD. I'd imagine that would be over in less than a few seconds (which would still be cool to see).

The startup script in boot.wim/Windows PE (System32\startnet.cmd) could be set up to just loop a sustained, sequential copy task of a large image or some other huge 'filler' set of files from ram, until the disk is full.

You'd have to pop open a copy of boot.wim, cram it full of files until its maybe 4gbs less than the amount of ram in the test system, edit the startup script to loop a robocopy from X: and then import it into WDS on Windows Server.  You'll probably waiting a while for a payload of 60-120+gbs to be TFTP'd into ram during the PXE boot.

EDIT: You could also just use a tool like EasyBCD to point to your modified boot.wim/ramdisk and have it added as an additional boot entry on an existing OS install.

I've never really monitored the transfer speed in this environment, but I would think that the MS/native RamDisk in WinPE would be faster than some of the other 3rd party RamDisk tools on a live running system. Since PE is a pretty minimal environment -- you don't have all the overhead from defender/av etc running as well.

[13en]

Is there any chance the LTT team was able to measure any practical power usage numbers? I'm extremely interested in the power overhead on that pci switch chip, seemed like it got a little hot a little fast...

With that we could do the math but if you measured the power draw with all those 8TB saberents idling and under load those would be super interesting numbers to me... That might just be me tho, my job involves sourcing / validating ssds to run in a hardware raid product and the denser drives coming out have had some surprisingly challenging power profiles to support. (Obviously this product as a whole is pretty interesting to me in that context)

Edit: Tried to get a little less lazy but I cant find any power spec on the PM400** product pages or that spec sheet we get some clips of in the vid, any ideas would be appreciated. 

[power666]

Those PCIe switches are pricy and hence the high cost of the carrier card.  One of the things on the switch spec sheet for the chip but isn't on the card is a 100 Mbit Ethernet out-of-band management support.   Mention in the video as a use-case is leveraging these chips for compositable infrastructure but like all enterprise grade infrastructure you want management functionalities to be separated from production data paths.  So why would some one want out-of-band management on a particular card like this?  First thing is encryption support as it would provide a means to pass keys around that would not touch the host system or production path ways.  While it would be painfully slow, it does provide an online means of extracting data in the event of a host system failure.  Lastly and seen in the video, is that out-of-band management can pass health information.  For a carrier card crammed like this, being able to run an instance of monitoring software on the card itself to record drive temperatures for a monitoring platform would be great.   

 

One thing not fully explored here but it is a power of the PCIe switch chip is that while link between the CPU and card is 16 lane PCIe 4.0, the drives could be simple 4 lane PCIe 3.0 units and performance would not be significantly impacted.  Similar situation if a 100 lane PCIe 5.0 switch was used with PCIe 4.0 based NVMe drives.  In fact, with a 100 lane PCIe 5.0 switch chip and 21 drives at PCIe 3.0 speeds, the link between the card and the host system will still be a bottleneck vs. the aggregate drive bandwidth.

 

 

[RandomLegoBrick]

Are there any pcie 5 switches that support a full x16 link?

[mariushm]

44 minutes ago, RandomLegoBrick said:

Are there any pcie 5 switches that support a full x16 link?

Yeah, they make gen5 switches with up to 100 lanes : https://ww1.microchip.com/downloads/en/DeviceDoc/00003776.pdf

 

But the pci-e 4.0 chip alone with 100 lanes is around $700 in volume, a pci-e 5.0 would be even more expensive... pcb is another 25-50$ at last, the m.2 connectors and dc-dc converters another 100-200$ and with a markup of around 30-50% you're looking at 1499$ for a card, or something like that. And not so sure you can do pci-e 5.0 like they stacking two boards.

 

The pci-e 3.0 chips with 80 lanes cost around $400 a piece if you buy $25: https://www.digikey.com/en/products/detail/microchip-technology/PM8565B-FEI/7724471

Would be a perfectly reasonable chip to make a card with 16 M.2 connectors on it, for example...

 

17 hours ago, 13en said:

Is there any chance the LTT team was able to measure any practical power usage numbers? I'm extremely interested in the power overhead on that pci switch chip, seemed like it got a little hot a little fast...

Judging by the big heatsink and temperature, I'd guess it dissipates at least 15-20 watts when using that many SSDs.

[Uttamattamakin]

Here is a suggestion for how to stress something like this.  

IF you have TWO of these carrier cards in the same system.  Write data to one until it is full (which would take a long time.) 

Then copy the data from one carrier card to the other.  The limit then would be how fast the CPU and bus can do it.  Might have an excuse reason to build a threadripper pro or Epyc based build just to do such a thing with no possible bottle necks. 

[Spotty]

The test bench was using an AM5 system? Don't all AM5 CPUs have integrated graphics? Was there any reason not to use the integrated graphics on the CPU to avoid needing the dedicated GPU in the x1 PCIe slot? Or was it just for the meme of putting the GPU in the x1 slot?

[swimtome]

10 minutes ago, Spotty said:

The test bench was using an AM5 system? Don't all AM5 CPUs have integrated graphics? Was there any reason not to use the integrated graphics on the CPU to avoid needing the dedicated GPU in the x1 PCIe slot? Or was it just for the meme of putting the GPU in the x1 slot?

It didn't boot without a GPU, so not sure what exactly was going on there...

[dogwitch]

7 hours ago, swimtome said:

It didn't boot without a GPU, so not sure what exactly was going on there...

Could be pci lane issue.

[daddysmackdown]

How did it work?  The thing can use up to 100 pcie lanes and they use 7950x and it only has 44 lanes.  They were able to see all 20 drives.  I thought each drive need at least 4 lanes

[Kilrah]

That's the whole point of the chip, it's a PCIe switch, similar to an Ethernet switch. Splits the 16 lanes it gets into 100. 

[mariushm]

1 hour ago, Kilrah said:

That's the whole point of the chip, it's a PCIe switch, similar to an Ethernet switch. Splits the 16 lanes it gets into 100. 

I think it's more like 100 lanes in total, and up to 16 can be outgoing and you're left with 84 lanes that can be split into x1, x4 and x8 connections and possibly x16 groups of lanes.

They're making  21 x M.2 with 4 lanes = 84 lanes.

 

But yes, it's like a network switch with 21 x 1gbps ethernet ports , and one outgoing 10 gbps port -  but it's 21 x 4 lanes, and one x16 outgoing (to system/ motherboard)