Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

They Told Me NOT to Do This...

SeanLMG
 Share

We've built a LOT of crazy servers, but when Supermicro asked us to not rebuild one of the storage servers for the Petabyte of flash project, well, we had to do just the opposite!

 

 

Check out the Kioxia CD6 Drives: https://lmg.gg/rkDFz

Check out AMD Epyc 32 Core Processors: https://lmg.gg/5vQAv

Check out AMD Epyc 64 Processors: https://lmg.gg/pN6lh

Check out the Supermicro Storage 1124US-TNRP: https://lmg.gg/6Qqcu

Check out the Supermicro GPU 4124GO-NART: https://lmg.gg/bgJiM

Check out Samsung DDR4 ECC Memory: https://lmg.gg/GAN0z

Check out the NVIDIA MSN3700-VS2F Switch: https://lmg.gg/0SKZE

Check out the NVIDIA ConnectX-6 200GbE Cards: https://lmg.gg/rfSEo

Buy some Infinite Cables Fibre Cables: https://lmg.gg/HGrgD

Buy some Infinite Cables Custom Cables: https://lmg.gg/K4JK8

Buy some FS.com 100GbE Transceivers: https://lmg.gg/PCMdK

 

Purchases made through some store links may provide some compensation to Linus Media Group.
 

Link to comment
Share on other sites

Link to post
Share on other sites

Did they say whether this was with TrueNAS scale or core? I know it probably eats up into the video time but more of the software side of things would be cool!

Link to comment
Share on other sites

Link to post
Share on other sites

off topic, what smart watch is linus wearing?

 

Link to comment
Share on other sites

Link to post
Share on other sites

Always like seeing people new to enterprise kit have fun. Reminds me of the first time I installed a mainframe, or installing a huge storage array. Seeing how different things are in enterprise world compared to desktop kit is always a joy.

Link to comment
Share on other sites

Link to post
Share on other sites

59 minutes ago, Muffxd said:

Did they say whether this was with TrueNAS scale or core? I know it probably eats up into the video time but more of the software side of things would be cool!

TrueNAS Scale, you can see it right around the 17:00 mark for a moment before they go to the drives.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites

Link to post
Share on other sites

can someone familiar with truenas answer the the Mitch H comment on the youtube.

"I'm a storage architect at 45Drives myself. Has anyone else realized yet that they weren't actually even benchmarking the ZFS array they built? LOL.... They were running the fio tests from the root directory (/root).. I know this because they're logged in as root user and there is a tilda in the shell meaning they are in their home directory ~ and the fio tests do not specify disks or mount points - which means fio was running from whichever directory you ran it in...... They mention they disable the ARC for everything but metadata on their NVMe array, but they definitely didn't do that on the boot pool - so when they ran their randomread tests, what they were actually benchmarking was the ARC performance (RAM) lol and of course its going to be amazing (20GiB/s) ----- but when you see when they finally benchmark the write perfromance it is actually only what you'd get out of a single NVMe device (hint hint lol because it was a single NVMe) and they kinda hand waved it away.... but that is exactly what was happening. They were hitting their intel boot NVMe drive the whole time."

 
Link to comment
Share on other sites

Link to post
Share on other sites

24 minutes ago, kokosnh said:

can someone familiar with truenas answer the the Mitch H comment on the youtube.

"I'm a storage architect at 45Drives myself. Has anyone else realized yet that they weren't actually even benchmarking the ZFS array they built? LOL.... They were running the fio tests from the root directory (/root).. I know this because they're logged in as root user and there is a tilda in the shell meaning they are in their home directory ~ and the fio tests do not specify disks or mount points - which means fio was running from whichever directory you ran it in...... They mention they disable the ARC for everything but metadata on their NVMe array, but they definitely didn't do that on the boot pool - so when they ran their randomread tests, what they were actually benchmarking was the ARC performance (RAM) lol and of course its going to be amazing (20GiB/s) ----- but when you see when they finally benchmark the write perfromance it is actually only what you'd get out of a single NVMe device (hint hint lol because it was a single NVMe) and they kinda hand waved it away.... but that is exactly what was happening. They were hitting their intel boot NVMe drive the whole time."

 

Can confirm, if they didn't cd to the directory the pool was on then they ran it from the nvme boot drive instead. Need to triple check but looks correct indeed that they failed to run the benchmark correctly.

 

Edit:

Watched the part again and they indeed did it from the root directory as root and not properly. Unless I'm really missing something obvious

image.png.165f07d4fdbc7671c974998100ba8b33.png

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites

Link to post
Share on other sites

Can you guys showing what exact FIO commands was used for testing and what was tried to simulate.  Thank you.

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Fermion said:

Can you guys showing what exact FIO commands was used for testing and what was tried to simulate.  Thank you.

 

 

See above 🙂 Just edited it in for one of them.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites

Link to post
Share on other sites

Can you guys tell me how to set Arc Metadata only for an pool?

Thanks 👍🏻

Link to comment
Share on other sites

Link to post
Share on other sites

If you are still able to perform new tests, can you also try using io_uring instead of libaio? It should help lower the CPU load. These are old slides I found on this https://www.flashmemorysummit.com/Proceedings2019/08-07-Wednesday/20190807_SOFT-202-1_Verma.pdf

 

Use a kernel as new as possible (newer than what truenas scale uses, I think you could use opensuse tumbleweed or arch to get quickly a test system), this thing is very much in development, see Jens Axboe's twitter profile https://twitter.com/axboe

 

Link to comment
Share on other sites

Link to post
Share on other sites

7 hours ago, RollinLower said:

can we please see some F@H numbers on that A100 beast?

boots it up.... and power goes out for the whole building...

MSI x399 sli plus  | AMD theardripper 2990wx all core 3ghz lock |Thermaltake flo ring 360 | EVGA 2080, Zotac 2080 |Gskill Ripjaws 128GB 3000 MHz | Corsair RM1200i |100tb | Asus tuff gaming mid tower

Link to comment
Share on other sites

Link to post
Share on other sites

8 hours ago, Results45 said:

re-run of Crysis or Doom with Swiftshader

PLEASE PLEASE PLEASE PLEASE

I have been wanting to see this on newer hardware for AGES

Link to comment
Share on other sites

Link to post
Share on other sites

21 hours ago, Lurick said:

Can confirm, if they didn't cd to the directory the pool was on then they ran it from the nvme boot drive instead. Need to triple check but looks correct indeed that they failed to run the benchmark correctly.

 

Edit:

Watched the part again and they indeed did it from the root directory as root and not properly. Unless I'm really missing something obvious

image.png.165f07d4fdbc7671c974998100ba8b33.png

Does TrueNAS Scale do the same thing that TrueNAS Core does where it doesn't show you the path of your current, working directory in the shell? Or at least the last folder of your pwd in your shell (like pretty much all other Linux distro usually does, out of the box)?

IB >>> ETH

Link to comment
Share on other sites

Link to post
Share on other sites

2 hours ago, alpha754293 said:

Does TrueNAS Scale do the same thing that TrueNAS Core does where it doesn't show you the path of your current, working directory in the shell? Or at least the last folder of your pwd in your shell (like pretty much all other Linux distro usually does, out of the box)?

Given we see the tilde (~) inside the brackets there, I'm assuming that's where we'd see the current directory which appears to be the root user's home directory in the video.

Link to comment
Share on other sites

Link to post
Share on other sites

They might not ever sell him another server now...

Link to comment
Share on other sites

Link to post
Share on other sites

3 hours ago, SolarPwr303 said:

They might not ever sell him another server now...

he's not getting to keep these, they're all going back

 

Link to comment
Share on other sites

Link to post
Share on other sites

21 hours ago, Eric The Tech Guru said:

Given we see the tilde (~) inside the brackets there, I'm assuming that's where we'd see the current directory which appears to be the root user's home directory in the video.

Possibly.

 

It's too bad that I don't have TrueNAS Scale deployed to test it out myself. I know that for my TrueNAS Core system, regardless of what your current pwd is, the default zsh will only show just the hostname and that's it.

IB >>> ETH

Link to comment
Share on other sites

Link to post
Share on other sites

On 4/2/2022 at 11:41 PM, der_gin said:

Can you guys tell me how to set Arc Metadata only for an pool?

Thanks 👍🏻

No one? 

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share


×