They Told Me NOT to Do This...

[SeanLMG]

We've built a LOT of crazy servers, but when Supermicro asked us to not rebuild one of the storage servers for the Petabyte of flash project, well, we had to do just the opposite!

 

 

Check out the Kioxia CD6 Drives: https://lmg.gg/rkDFz

Check out AMD Epyc 32 Core Processors: https://lmg.gg/5vQAv

Check out AMD Epyc 64 Processors: https://lmg.gg/pN6lh

Check out the Supermicro Storage 1124US-TNRP: https://lmg.gg/6Qqcu

Check out the Supermicro GPU 4124GO-NART: https://lmg.gg/bgJiM

Check out Samsung DDR4 ECC Memory: https://lmg.gg/GAN0z

Check out the NVIDIA MSN3700-VS2F Switch: https://lmg.gg/0SKZE

Check out the NVIDIA ConnectX-6 200GbE Cards: https://lmg.gg/rfSEo

Buy some Infinite Cables Fibre Cables: https://lmg.gg/HGrgD

Buy some Infinite Cables Custom Cables: https://lmg.gg/K4JK8

Buy some FS.com 100GbE Transceivers: https://lmg.gg/PCMdK

 

Purchases made through some store links may provide some compensation to Linus Media Group.
 

[Muffxd]

Did they say whether this was with TrueNAS scale or core? I know it probably eats up into the video time but more of the software side of things would be cool!

[Results45]

On the next update of the A100 server.........you guys gonna do a re-run of Crysis or Doom with Swiftshader or a heavy load of real-time path tracing at 10, 25, 50, or even 100 samples per pixel?

[karanssh]

off topic, what smart watch is linus wearing?

 

[RollinLower]

can we please see some F@H numbers on that A100 beast?

[Distinctly Average]

Always like seeing people new to enterprise kit have fun. Reminds me of the first time I installed a mainframe, or installing a huge storage array. Seeing how different things are in enterprise world compared to desktop kit is always a joy.

[Lurick]

59 minutes ago, Muffxd said:

Did they say whether this was with TrueNAS scale or core? I know it probably eats up into the video time but more of the software side of things would be cool!

TrueNAS Scale, you can see it right around the 17:00 mark for a moment before they go to the drives.

[kokosnh]

can someone familiar with truenas answer the the Mitch H comment on the youtube.

"I'm a storage architect at 45Drives myself. Has anyone else realized yet that they weren't actually even benchmarking the ZFS array they built? LOL.... They were running the fio tests from the root directory (/root).. I know this because they're logged in as root user and there is a tilda in the shell meaning they are in their home directory ~ and the fio tests do not specify disks or mount points - which means fio was running from whichever directory you ran it in...... They mention they disable the ARC for everything but metadata on their NVMe array, but they definitely didn't do that on the boot pool - so when they ran their randomread tests, what they were actually benchmarking was the ARC performance (RAM) lol and of course its going to be amazing (20GiB/s) ----- but when you see when they finally benchmark the write perfromance it is actually only what you'd get out of a single NVMe device (hint hint lol because it was a single NVMe) and they kinda hand waved it away.... but that is exactly what was happening. They were hitting their intel boot NVMe drive the whole time."

 

[Lurick]

24 minutes ago, kokosnh said:

can someone familiar with truenas answer the the Mitch H comment on the youtube.

"I'm a storage architect at 45Drives myself. Has anyone else realized yet that they weren't actually even benchmarking the ZFS array they built? LOL.... They were running the fio tests from the root directory (/root).. I know this because they're logged in as root user and there is a tilda in the shell meaning they are in their home directory ~ and the fio tests do not specify disks or mount points - which means fio was running from whichever directory you ran it in...... They mention they disable the ARC for everything but metadata on their NVMe array, but they definitely didn't do that on the boot pool - so when they ran their randomread tests, what they were actually benchmarking was the ARC performance (RAM) lol and of course its going to be amazing (20GiB/s) ----- but when you see when they finally benchmark the write perfromance it is actually only what you'd get out of a single NVMe device (hint hint lol because it was a single NVMe) and they kinda hand waved it away.... but that is exactly what was happening. They were hitting their intel boot NVMe drive the whole time."

 

Can confirm, if they didn't cd to the directory the pool was on then they ran it from the nvme boot drive instead. Need to triple check but looks correct indeed that they failed to run the benchmark correctly.

 

Edit:

Watched the part again and they indeed did it from the root directory as root and not properly. Unless I'm really missing something obvious

[Fermion]

Can you guys showing what exact FIO commands was used for testing and what was tried to simulate.  Thank you.

 

 

[Lurick]

1 minute ago, Fermion said:

Can you guys showing what exact FIO commands was used for testing and what was tried to simulate.  Thank you.

 

 

See above Just edited it in for one of them.

[der_gin]

Can you guys tell me how to set Arc Metadata only for an pool?

Thanks

[fulminemizzega]

If you are still able to perform new tests, can you also try using io_uring instead of libaio? It should help lower the CPU load. These are old slides I found on this https://www.flashmemorysummit.com/Proceedings2019/08-07-Wednesday/20190807_SOFT-202-1_Verma.pdf

 

Use a kernel as new as possible (newer than what truenas scale uses, I think you could use opensuse tumbleweed or arch to get quickly a test system), this thing is very much in development, see Jens Axboe's twitter profile https://twitter.com/axboe

 

[dogwitch]

7 hours ago, RollinLower said:

can we please see some F@H numbers on that A100 beast?

boots it up.... and power goes out for the whole building...

[DANK_AS_gay]

8 hours ago, Results45 said:

re-run of Crysis or Doom with Swiftshader

PLEASE PLEASE PLEASE PLEASE

I have been wanting to see this on newer hardware for AGES

[alpha754293]

21 hours ago, Lurick said:

Can confirm, if they didn't cd to the directory the pool was on then they ran it from the nvme boot drive instead. Need to triple check but looks correct indeed that they failed to run the benchmark correctly.

 

Edit:

Watched the part again and they indeed did it from the root directory as root and not properly. Unless I'm really missing something obvious

Does TrueNAS Scale do the same thing that TrueNAS Core does where it doesn't show you the path of your current, working directory in the shell? Or at least the last folder of your pwd in your shell (like pretty much all other Linux distro usually does, out of the box)?

[Eric The Tech Guru]

2 hours ago, alpha754293 said:

Does TrueNAS Scale do the same thing that TrueNAS Core does where it doesn't show you the path of your current, working directory in the shell? Or at least the last folder of your pwd in your shell (like pretty much all other Linux distro usually does, out of the box)?

Given we see the tilde (~) inside the brackets there, I'm assuming that's where we'd see the current directory which appears to be the root user's home directory in the video.

[SolarPwr303]

They might not ever sell him another server now...

[tkitch]

3 hours ago, SolarPwr303 said:

They might not ever sell him another server now...

he's not getting to keep these, they're all going back

 

[alpha754293]

21 hours ago, Eric The Tech Guru said:

Given we see the tilde (~) inside the brackets there, I'm assuming that's where we'd see the current directory which appears to be the root user's home directory in the video.

Possibly.

 

It's too bad that I don't have TrueNAS Scale deployed to test it out myself. I know that for my TrueNAS Core system, regardless of what your current pwd is, the default zsh will only show just the hostname and that's it.

[der_gin]

On 4/2/2022 at 11:41 PM, der_gin said:

Can you guys tell me how to set Arc Metadata only for an pool?

Thanks

No one?