Jump to content

They Told Me NOT to Do This...

SeanLMG
By SeanLMG
in LTT Releases
 Share
Posted

We've built a LOT of crazy servers, but when Supermicro asked us to not rebuild one of the storage servers for the Petabyte of flash project, well, we had to do just the opposite!

 

 

Check out the Kioxia CD6 Drives: https://lmg.gg/rkDFz

Check out AMD Epyc 32 Core Processors: https://lmg.gg/5vQAv

Check out AMD Epyc 64 Processors: https://lmg.gg/pN6lh

Check out the Supermicro Storage 1124US-TNRP: https://lmg.gg/6Qqcu

Check out the Supermicro GPU 4124GO-NART: https://lmg.gg/bgJiM

Check out Samsung DDR4 ECC Memory: https://lmg.gg/GAN0z

Check out the NVIDIA MSN3700-VS2F Switch: https://lmg.gg/0SKZE

Check out the NVIDIA ConnectX-6 200GbE Cards: https://lmg.gg/rfSEo

Buy some Infinite Cables Fibre Cables: https://lmg.gg/HGrgD

Buy some Infinite Cables Custom Cables: https://lmg.gg/K4JK8

Buy some FS.com 100GbE Transceivers: https://lmg.gg/PCMdK

 

Purchases made through some store links may provide some compensation to Linus Media Group.
 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Did they say whether this was with TrueNAS scale or core? I know it probably eats up into the video time but more of the software side of things would be cool!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

off topic, what smart watch is linus wearing?

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Always like seeing people new to enterprise kit have fun. Reminds me of the first time I installed a mainframe, or installing a huge storage array. Seeing how different things are in enterprise world compared to desktop kit is always a joy.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
59 minutes ago, Muffxd said:

Did they say whether this was with TrueNAS scale or core? I know it probably eats up into the video time but more of the software side of things would be cool!

TrueNAS Scale, you can see it right around the 17:00 mark for a moment before they go to the drives.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

can someone familiar with truenas answer the the Mitch H comment on the youtube.

"I'm a storage architect at 45Drives myself. Has anyone else realized yet that they weren't actually even benchmarking the ZFS array they built? LOL.... They were running the fio tests from the root directory (/root).. I know this because they're logged in as root user and there is a tilda in the shell meaning they are in their home directory ~ and the fio tests do not specify disks or mount points - which means fio was running from whichever directory you ran it in...... They mention they disable the ARC for everything but metadata on their NVMe array, but they definitely didn't do that on the boot pool - so when they ran their randomread tests, what they were actually benchmarking was the ARC performance (RAM) lol and of course its going to be amazing (20GiB/s) ----- but when you see when they finally benchmark the write perfromance it is actually only what you'd get out of a single NVMe device (hint hint lol because it was a single NVMe) and they kinda hand waved it away.... but that is exactly what was happening. They were hitting their intel boot NVMe drive the whole time."

 
   
 
 
 
Spoiler
CPU : Intel 14gen i7-14700K
COOLER :  Thermalright Peerless Assassin 120 White + thermaltake toughfan 12 white + Thermal Grizzly - CPU Contact Frame Intel 13./14. +  Coollaboratory Liquid Ultra
GPU : MSI RTX 2070 Armor @GPU 2050MHz Mem 8200MHz -> USB C 10Gb/s cable 2m -> Unitek 4x USB HUB 10 Gb/s (Y-HB08003)
MOBO : MSI MEG Z690 UNIFY
RAM :  Corsair VENGEANCE DDR5 RAM 64 GB (2 x 32 GB) 6400 MHz CL32 (CMK64GX5M2B6400C32)
SSD : Intel Optane 905P 960GB U.2 (OS) + 2 x WD SN850X 4TB + 2 x PNY CS3140 2TB + ASM2824 PCIe switch -> 4 x Plextor M8PeG 1TB + flexiDOCK MB014SP-B -> Crucial MX500 2TB + GoodRam Iridium PRO 960GB + Samsung 850 Pro 512GB
HDD : WD White 18TB WD180EDFZ + SATA port multiplier adp6st0-j05 (JMB575) ->  WD Gold 8TB WD8002FRYZ + WD Gold 4TB WD4002FYYZ + WD Red PRO 4TB WD4001FFSX + WD Green 2TB WD20EARS
EXTERNAL
HDD/SSD :  		XT-XINTE LM906 (JMS583) -> Plextor M8PeG 1TB + WD My Passport slim 1TB + LaCie Porsche Design Mobile Drive 1TB USB-C + Zalman ZM-VE350 -> Goodram IRDM PRO 240GB
PSU :  Super Flower leadex platinum 750 W biały -> Bitfenix alchemy extensions białe/białe + AsiaHorse 16AWG White 
UPS :  CyberPower CP1500EPFCLCD -> Brennenstuhl primera-line 8 -> Brennenstuhl primera-line 10
LCD :  LG 32UD59-B + LG flatron IPS236 -> Silverstone SST-ARM11BC
CASE :  Fractal R5 Biały + Lian Li BZ-H06A srebrny + 6 x Thermaltake toughfan 14 white + Thermalright TL-B8W
SPEAKERS :  Aune S6 Pro -> Topping PA3-B -> Polk S20e black -> Monoprice stand 16250
HEADPHONES :  TOSLINK 2m -> Aune S6 Pro -> 2 x Monoprice Premier 1.8m 16AWG 3-pin XLR -> Monoprice Monolith THX AAA 887 -> 4-pin XLR na 2 x 3.5mm 16 cores OCC 2m Cable -> HiFiMAN Edition XS -> sheepskin pads + 4-pin XLR na 2 x 2.5mm ABLET silver 2m  Cable -> Monoprice Monolith M1060 + Brainwavz HM100 -> Brainwavz sheepskin oval pads + Wooden double Ɪ Stand + Audio-Technica ATH-MSR7BK -> sheepskin pads + Multibrackets MB1893 + Sennheiser Momentum 3 +  Philips Fidelio X2HR/00 + JBL J88 White
MIC :  Tonor TC30 -> Mozos SB38
KEYBOARD : Corsair STRAFE RGB Cherry MX Silent (EU) + Glorious PC Gaming Race Stealth Slim - Full Size Black + PQI MyLockey
MOUSE :  Logitech MX ERGO + 2 x Logitech MX Performance + Logitech G Pro wireless + Logitech G Pro Gaming -> Hotline Games 2.0 Plus + Corsair MM500 3xl + Corsair MM300 Extended + Razer goliathus control
CONTROLLERS :  Microsoft xbox series x controller pc (1VA-00002) -> brainwavz audio Controller Holder UGC2 + Microsoft xbox 360 wireless black + Ravcore Javelin
NET :  Intel x520-DA2 -> 2 x FTLX8571D3BCV-IT + 2 x ASUS ZenWiFi Pro XT12
NAS :  Qnap TS-932X-2G -> Noctua NF-P14s redux 1200 PWM -> Kingston 16GB 2400Mhz CL14 (HX424S14IB/16) -> 9 x Crucial MX500 2TB ->  2 x FTLX8571D3BCV-IT -> 2 x Digitus (DK-HD2533-05/3)
Link to comment
Share on other sites
Link to post
Share on other sites
Posted
24 minutes ago, kokosnh said:

can someone familiar with truenas answer the the Mitch H comment on the youtube.

"I'm a storage architect at 45Drives myself. Has anyone else realized yet that they weren't actually even benchmarking the ZFS array they built? LOL.... They were running the fio tests from the root directory (/root).. I know this because they're logged in as root user and there is a tilda in the shell meaning they are in their home directory ~ and the fio tests do not specify disks or mount points - which means fio was running from whichever directory you ran it in...... They mention they disable the ARC for everything but metadata on their NVMe array, but they definitely didn't do that on the boot pool - so when they ran their randomread tests, what they were actually benchmarking was the ARC performance (RAM) lol and of course its going to be amazing (20GiB/s) ----- but when you see when they finally benchmark the write perfromance it is actually only what you'd get out of a single NVMe device (hint hint lol because it was a single NVMe) and they kinda hand waved it away.... but that is exactly what was happening. They were hitting their intel boot NVMe drive the whole time."

 

Can confirm, if they didn't cd to the directory the pool was on then they ran it from the nvme boot drive instead. Need to triple check but looks correct indeed that they failed to run the benchmark correctly.

 

Edit:

Watched the part again and they indeed did it from the root directory as root and not properly. Unless I'm really missing something obvious

image.png.165f07d4fdbc7671c974998100ba8b33.png

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Can you guys showing what exact FIO commands was used for testing and what was tried to simulate.  Thank you.

 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, Fermion said:

Can you guys showing what exact FIO commands was used for testing and what was tried to simulate.  Thank you.

 

 

See above 🙂 Just edited it in for one of them.

Current Network Layout:

Current Build Log/PC:

Prior Build Log/PC:

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Can you guys tell me how to set Arc Metadata only for an pool?

Thanks 👍🏻

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

If you are still able to perform new tests, can you also try using io_uring instead of libaio? It should help lower the CPU load. These are old slides I found on this https://www.flashmemorysummit.com/Proceedings2019/08-07-Wednesday/20190807_SOFT-202-1_Verma.pdf

 

Use a kernel as new as possible (newer than what truenas scale uses, I think you could use opensuse tumbleweed or arch to get quickly a test system), this thing is very much in development, see Jens Axboe's twitter profile https://twitter.com/axboe

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
7 hours ago, RollinLower said:

can we please see some F@H numbers on that A100 beast?

boots it up.... and power goes out for the whole building...

MSI x399 sli plus  | AMD theardripper 2990wx all core 3ghz lock |Thermaltake flo ring 360 | EVGA 2080, Zotac 2080 |Gskill Ripjaws 128GB 3000 MHz | Corsair RM1200i |150tb | Asus tuff gaming mid tower| 10gb NIC

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
8 hours ago, Results45 said:

re-run of Crysis or Doom with Swiftshader

PLEASE PLEASE PLEASE PLEASE

I have been wanting to see this on newer hardware for AGES

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
21 hours ago, Lurick said:

Can confirm, if they didn't cd to the directory the pool was on then they ran it from the nvme boot drive instead. Need to triple check but looks correct indeed that they failed to run the benchmark correctly.

 

Edit:

Watched the part again and they indeed did it from the root directory as root and not properly. Unless I'm really missing something obvious

image.png.165f07d4fdbc7671c974998100ba8b33.png

Does TrueNAS Scale do the same thing that TrueNAS Core does where it doesn't show you the path of your current, working directory in the shell? Or at least the last folder of your pwd in your shell (like pretty much all other Linux distro usually does, out of the box)?

IB >>> ETH

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 hours ago, alpha754293 said:

Does TrueNAS Scale do the same thing that TrueNAS Core does where it doesn't show you the path of your current, working directory in the shell? Or at least the last folder of your pwd in your shell (like pretty much all other Linux distro usually does, out of the box)?

Given we see the tilde (~) inside the brackets there, I'm assuming that's where we'd see the current directory which appears to be the root user's home directory in the video.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
3 hours ago, SolarPwr303 said:

They might not ever sell him another server now...

he's not getting to keep these, they're all going back

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
21 hours ago, Eric The Tech Guru said:

Given we see the tilde (~) inside the brackets there, I'm assuming that's where we'd see the current directory which appears to be the root user's home directory in the video.

Possibly.

 

It's too bad that I don't have TrueNAS Scale deployed to test it out myself. I know that for my TrueNAS Core system, regardless of what your current pwd is, the default zsh will only show just the hostname and that's it.

IB >>> ETH

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
On 4/2/2022 at 11:41 PM, der_gin said:

Can you guys tell me how to set Arc Metadata only for an pool?

Thanks 👍🏻

No one? 

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing LTT Releases

×