WHAT did I just BUY on Facebook Marketplace??

[nhand42]

7 hours ago, Mister_Magister said:

He doesn't know what he's talking about, iscsi is literally tcp/ip same as nfs, literally first result in google, and no iscsi is not SAN it's still NAS

You are confidently incorrect. iSCSI is a block-level protocol. NFS is a file-level protocol. They are completely unrelated. iSCSI is most definitely a SAN protocol and has nothing to do with NAS. I have worked in this space for two decades and I was impressed by how little LTT got wrong in this video. You on the other hand need to read more.

[leadeater]

3 hours ago, themrsbusta said:

Years ago a server like this would take an entire room, now is a rack. How many years to companies have servers with similar size to a Raspberry Pi and performance/storage of a today's rack?

That's why I love tech.

A single PCIe 5.0 enterprise SSD is probably faster than all those disks 

 

The video is showing 10+ year old technology, things move far in 10 years. But things are slowing down improvement wise though so who knows, could be long time or could be 5 years.

[fili0926]

I've been working with NetApp arrays for 14 years.  I've got 4 NetApp arrays in my home lab.  

 

One correction to what Linus said about the battery in the controller, it is not for a RAID card, but rather NVRAM.  It functions similarly to as how Linus explained it.

Linus questioned the multipath capability once he thought it was an "old school RAID card" but the NVRAM is divided into 4 banks.  Two are for the local controller which take turns flushing writes to disk once full, the other two are mirrored from the partner controller and so both controllers have a mirror of the most recent writes from it's partner controller, this is how the HA and multipath are able to function.

 

NetApp uses proprietary RAID levels:

-RAID4 - 1 parity disk - used for root aggregates or cache raid groups, but not typically used for user data aggregates

-RAID-DP - 2 parity disks - supports raid groups 3-28 for SAS disks, 3-20 for NL-SAS/SATA

-RAID-TEC - 3 parity disks - recommended for drives 6TB+, required for 10TB+, supports raid groups 4-28 disks.

 

All 3 of these RAID levels use dedicated parity disks rather than striped parity that you see in RAID5 or RAID6.

 

I also have a 500TB TrueNAS box in my home lab that is using 2x 4246 NetApp Shelves with 16TB WD RED Pro disks.  If you choose to use those 4243 as JBODs you can swap the IOM3 modules for IOM6 to convert from 3G SAS to 6G SAS.  You can also drop from 4 PSUs to 2PSUs when using SATA disks, only the 15k SAS disks require all 4 PSUs.  Although the PSUs will ramp up fan speed unless you have the blanks to fill the empty PSU slots (airflow).

 

Let me know if you need any documentation, firmware, or licensing for that array.  I'm happy to give advice on how to configure it, assuming you don't have someone at NetApp reach out to you.

[fili0926]

1 hour ago, leadeater said:

A single PCIe 5.0 enterprise SSD is probably faster than all those disks 

 

The video is showing 10+ year old technology, things move far in 10 years. But things are slowing down improvement wise though so who knows, could be long time or could be 5 years.

For throughput you might be right about a PCIe 5 being faster, but in the enterprise space you rarely worry about throughput.  IOPs is a much more important thing to optimize for which having 96x15k disks would provide for a TON of IOPs.  A single SSD would not be nearly as fast when dealing with mixed IO requests.  A moderate size SSD pool (12-24) would blow this out of the water though.

 

[leadeater]

2 hours ago, fili0926 said:

For throughput you might be right about a PCIe 5 being faster, but in the enterprise space you rarely worry about throughput.  IOPs is a much more important thing to optimize for which having 96x15k disks would provide for a TON of IOPs.  A single SSD would not be nearly as fast when dealing with mixed IO requests.  A moderate size SSD pool (12-24) would blow this out of the water though.

Yes and NVMe destroys any HDD array always, even single device. Trust me I know, we have a few million dollars of NetApp (A400 and FAS8300, multiple of them in different datacenters).

 

Large queue IOPs is literally worthless of databases for example. Even if you deployed out enough HDDs to breach sustained IOPs of a good Kioxia the per I/O request latency is still far worse. If you are lucky your reads will be in the controller head NVMe cache, if you are lucky. Flash Cache is by no means a replacement for an actual flash array, even a tiny one.

 

Edit:

Full random 4K IOPs on a 15k RPM HDD is less than 200, even with 100% perfect scaling (impossible) 96 is not close to the IOPs of modern NVMe. That's why the storage controllers have read and write caches that are way faster to get achieved IOPs much higher than the disks alone could actually do.

 

P.S. I've been in the storage realm since SCSI-2, I do not miss the ever changing cables and connector standard back then

[Dada216]

5 hours ago, nhand42 said:

Worth mentioning that enterprise is slowly moving away from SAN towards object storage. I don't expect SAN to disappear anytime soon, in fact it'll probably continue to increase in absolute terms, but object storage is growing even quicker and replacing traditional models dominated by SAN.

 

SAN will continue to innovate and improve, and there will be niches where SAN dominates forever, but it's no longer the pinnacle of storage like it was 10 years ago. Many enterprise architects treat it as dinosaur technology, and rightly so.

yes and no.

lots of applications do use object storage as a backend, but you still need an OS with storage attached to it in order to expose it as object storage, also you will never host a database on object storage and many more very important applications. in some areas it's all object storage now, i.e. the static files of a webapp

[Dada216]

1 hour ago, leadeater said:

Large queue IOPs is literally worthless of databases for example. Even if you deployed out enough HDDs to breach sustained IOPs of a good Kioxia the per I/O request latency is still far worse. If you are lucky your reads will be in the controller head NVMe cache, if you are lucky. Flash Cache is by no means a replacement for an actual flash array, even a tiny one.

we've seen some interesting results with mongo/elastic in our testing, plus modern linux kernels with io_uring seem to promise even more efficiency in squeezing those multiple queue on nvme, so you can shard the db like crazy and make those pretty penny you spend on the disk actually count.

Finally, after years of stagnation in the storage tech, we are seeing amazing improvements, they should apply to traditional SQL dbms as well, but we haven't tested those yet.

 

nvme-over-fiber is getting me excited, I want to see those fabric switches in pain.

[dogwitch]

5 minutes ago, Dada216 said:

we've seen some interesting results with mongo/elastic in our testing, plus modern linux kernels with io_uring seem to promise even more efficiency in squeezing those multiple queue on nvme, so you can shard the db like crazy and make those pretty penny you spend on the disk actually count.

Finally, after years of stagnation in the storage tech, we are seeing amazing improvements, they should apply to traditional SQL dbms as well, but we haven't tested those yet.

 

nvme-over-fiber is getting me excited, I want to see those fabric switches in pain.

i forget where i read or seen it.

but i remember they  needed clock timing  to a insane point. due to speed of light and medium its in.

[Dada216]

14 minutes ago, dogwitch said:

but i remember they  needed clock timing  to a insane point.

there's so much involved in the equation of squeezing every bit out of nvme vs (i)scsi.

yes, you need a cpu that can put up with it, and depending on the workload and how it implements the syscall to the kernel that might be a very important factor, a java application uses IO very differently than a C(++) application and, as I mentioned, io_uring is gonna change everything in this regard.

on a desktop/single server it will also heavily depend on the chipset you are using, but in the "enterprise stack" you will rarely use an onboard controller for the IO intensive workloads, the onboard storage is usually just the boot storage, you will use an HBA card of some sort so ultimately speaking what is really doing the work on the disks is the controller you are attaching to via your HBA, in our space these controllers are tailored to the disks they are sold with, usually including custom disks firmware, so all the hardware combinations that are required to make it work like clockwork are handled by the vendor when they design the product.

[themrsbusta]

4 hours ago, fili0926 said:

For throughput you might be right about a PCIe 5 being faster, but in the enterprise space you rarely worry about throughput.  IOPs is a much more important thing to optimize for which having 96x15k disks would provide for a TON of IOPs.  A single SSD would not be nearly as fast when dealing with mixed IO requests.  A moderate size SSD pool (12-24) would blow this out of the water though.

 

All I imagine is a board with the size of a credit card, cpu with performance of a 96c Epyc and tons of microsd express slots, for enterprise level SLC/MLC microsd express cards...

Maybe sodimm slots in the back

[Dada216]

4 hours ago, fili0926 said:

I've been working with NetApp arrays for 14 years.  I've got 4 NetApp arrays in my home lab.  

 

One correction to what Linus said about the battery in the controller, it is not for a RAID card, but rather NVRAM.  It functions similarly to as how Linus explained it.

Linus questioned the multipath capability once he thought it was an "old school RAID card" but the NVRAM is divided into 4 banks.  Two are for the local controller which take turns flushing writes to disk once full, the other two are mirrored from the partner controller and so both controllers have a mirror of the most recent writes from it's partner controller, this is how the HA and multipath are able to function.

 

NetApp uses proprietary RAID levels:

-RAID4 - 1 parity disk - used for root aggregates or cache raid groups, but not typically used for user data aggregates

-RAID-DP - 2 parity disks - supports raid groups 3-28 for SAS disks, 3-20 for NL-SAS/SATA

-RAID-TEC - 3 parity disks - recommended for drives 6TB+, required for 10TB+, supports raid groups 4-28 disks.

 

All 3 of these RAID levels use dedicated parity disks rather than striped parity that you see in RAID5 or RAID6.

 

I also have a 500TB TrueNAS box in my home lab that is using 2x 4246 NetApp Shelves with 16TB WD RED Pro disks.  If you choose to use those 4243 as JBODs you can swap the IOM3 modules for IOM6 to convert from 3G SAS to 6G SAS.  You can also drop from 4 PSUs to 2PSUs when using SATA disks, only the 15k SAS disks require all 4 PSUs.  Although the PSUs will ramp up fan speed unless you have the blanks to fill the empty PSU slots (airflow).

 

Let me know if you need any documentation, firmware, or licensing for that array.  I'm happy to give advice on how to configure it, assuming you don't have someone at NetApp reach out to you.

I was thinking about how they should use / how I would design the LTT storage, from what I gathered from their videos their heaviest workload is their editors workstations accessing those huge video files, and when a project is done that goes to long term storage, I heard linus say they publish 7 videos per week, so 7 projects at a time, plus the rest of the stuff (I imagine they have some servers for IdP, accounting stuff, file shares for the rest of the crew, etc.), I imagine they don't have that much of a dbms workloads (or maybe they will once the lab is up and cranking?)

 

since the workstations have 10Gb and accessing those file would translate to sustained reads on the drives I was wondering:

1) how to split the array in at least 7 performance tier segments

2) how to create a workflow for the editors to create a project on each segment and have it somehow available on the workstations

 

I can imagine there is a workflow before the editors actually fire up the editing software and start accessing the files, then there's actual shooting and ingesting the files, then there's the editors part, then there's uploading and finally the project can be archived.

 

I would make a ceph backend on the controller, use the array they bought just as JBOD directly attached via SAS to the controller server with ceph, create an application to handle the workflow that ultimately does api calls to ceph to create the various segments on the storage and configure the fileserver (probably even the mounts on the clients once a project is created).

I would have a separate virtualization host that host the application and actually mounts the ceph stuff (probably block storage to the fileservers) and exposes it as a networked filesystem (SMB it's allright)

I would have a fiber optic fabric between the controller and the virtualization host with the application and the fileservers (probably a 1:1 between the fileservers and the segments on the ceph cluster) and lacp with a bunch of 25/50/100GB eth to the main switches that ultimately connect to the workstations.

 

another completely separated storage backend for the archived projects, ultimately putting those on tapes, the application should move it between storage backends once they archive the projects

 

Edit: the fact that I would enjoy building it as this doesn't mean it's the best way to do it or even that it's the best way for them to do it.

[leadeater]

55 minutes ago, Dada216 said:

we've seen some interesting results with mongo/elastic in our testing, plus modern linux kernels with io_uring seem to promise even more efficiency in squeezing those multiple queue on nvme, so you can shard the db like crazy and make those pretty penny you spend on the disk actually count.

Finally, after years of stagnation in the storage tech, we are seeing amazing improvements, they should apply to traditional SQL dbms as well, but we haven't tested those yet.

 

nvme-over-fiber is getting me excited, I want to see those fabric switches in pain.

Yea my comment was just in reference to piling up high queue depth behind an HDD array and saying that is good end system performance versus sub-millisecond you get from SSD and more so NVMe.

 

If your per I/O request latency actually matters then you'll want to be hitting cache or anything other than HDDs.

[Indian pc builder]

Who they buy it from?

[leadeater]

26 minutes ago, Indian pc builder said:

Who they buy it from?

I think the company logo/name is near the end on the wall.

[leadeater]

39 minutes ago, Dada216 said:

Edit: the fact that I would enjoy building it as this doesn't mean it's the best way to do it or even that it's the best way for them to do it.

It's not but it would be fun 

 

Thing is if you have a NetApp then just use the NetApp heh, they are actually really good. I have draws full of NetApp official and formatted SSDs too, I can be persuaded to sell them to LTT for the right price 

[Lurick]

7 minutes ago, leadeater said:

It's not but it would be fun 

 

Thing is if you have a NetApp then just use the NetApp heh, they are actually really good. I have draws full of NetApp official and formatted SSDs too, I can be persuaded to sell them to LTT for the right price 

I heard you got some of that good storage

Y'all need any switches to go with that?

[leadeater]

3 minutes ago, Lurick said:

I heard you got some of that good storage

Y'all need any switches to go with that?

100/400 with all the good features?

 

I know what you have, don't hide the good stuff from me.

[Lurick]

41 minutes ago, leadeater said:

100/400 with all the good features?

 

I know what you have, don't hide the good stuff from me.

You know it!

Can even do 400g MACSEC on all ports if you need some secure good stuff

[Lurick]

Back on topic:

Happy to see them at least trying to touch on enterprise SAN stuff, sure it's not super relevant with NVMe over Fabric and other new stuff HOWEVER for cheap block storage it's still nice!

[Dethman]

I use those exact Shelves on my TrueNAS Scale installation. Love them it works awesome!

 

Only down side is that to run SATA disks you need to have to Interposers installed. 

 

Also most the resellers on eBay dont sell the shelf with the caddies, they sell them separate so they can get a little more cash from them. 

[Theurgy]

OK so boy do I have opinions on this video.

So firstly, I am new to the forums, however not new to LTT.

I'll validate my opinions here by stating that I am currently a Systems Administrator at a pretty well known Canadian Not-For-Profit company, and in my previous role actually worked at NetApp as one of the internal Systems Administrators. Meaning I used to manage and maintain the equipment that NetApp developers and QA testers used to develop, test and validate their systems on.

I have probably installed and setup hundreds of these NetApp systems in the years I worked there, so to say I know these systems inside and out is an understatement.

 

With that I would like to clarify a few things mentioned in this discussion. iSCSI is still considered SAN.

SAN vs NAS, the difference is not in how the files are delivered but how those access to the storage is presented to the operating system client.

Meaning, with NAS, you use a protocol such as CIFS and NFS and the underlying filesystem is not in any way managed by the OS. The protocol handles file access, file locking, permissions etc.

With SAN, the storage presented to the OS is from a block level, meaning once presented, the OS has to determine how the data is written to those blocks. Much like if you have Direct Attached Storage (DAS). 

 

NetApp, has a LOT more complexity at the base level as well, and while watching the video, hearing Linus ask the question if he could install Ubuntu on the controllers and this made me laugh so hard. I get it, coming from the world of consumer technology, you would think anything is possible and etc, however these systems are very specialized. In this case these NetApp FAS system (file attached storage) have their own specialized OS called OnTap OS (which is Unix based at its core), it has it down file system called WAFL, its own RAID level called RAID-DP, and as such everything is pretty well locked down.

 

That said, to say that the system he just purchased is a SAN, isn't entirely accurate. This is a FAS system, essentially a storage appliance (hence the name NetApp.. or Network Appliance). This system can handle NAS protocols such as NFS and CIFS, as well as SAN via iSCSI, FC, and FCoE.

The hardware cost on here is pretty cheap, mainly because the true cost here is the support and licensing. 

I would like to extend the LTT team my knowledge should they need more details, however I look forward to seeing more about this setup in the future.

[Nystemy]

19 hours ago, Josh732 said:

That sounds like it was a Fiber Channel switch, rather than iSCSI. A switch meant for iSCSI would still function as a normal Ethernet/IP switch, whereas Fiber (or Fibre) Channel is a different layer 2 (from the OSI model perspective) protocol, so you'd have a difficult time (I'm hesitant to call anything "impossible") running IP traffic over it.

Yes. It is a EMC² DS-32B2 switch. Quite slow by modern standards, but bought it a really long time ago, so it isn't really even worth the effort to try to convert it to Ethernet.

[Theurgy]

So on the topic of disk swapping on these shelves, it is important that you pick disks that fall within their support scope and have the features that the controller will need. 

I'm not saying that swapping for some modern SSD's or similar won't work.. just that swapping them in might cause the controller to freak out. The controller has the ability to standardize the firmware being run across these disks via the IO Modules in the back of the shelves (these IOMs have to be wired in a loop for this purpose.. it uses something called ACP for all that to happen).

Further, a single shelf shouldn't have a mix of disks, so like one of those DS4243's should have the same disks across the board throughout the shelf. 

Speaking of those shelves, the naming convention (DS4243) breaks down as such: DS= Disk Shelf, 4 = 4U, 24 = 24 disk shelf unit and 3 means IOM3 which supports 3Gb SAS shelf to shelf connectivity. 

 

I'm sure you could find decent deals on shelves like a DS2246 with 2.5" SAS drives or even some SSDs.

Heck I think my work has stacks of them in our storage waiting to be e-wasted.

 

 

[Theurgy]

19 hours ago, themrsbusta said:

Years ago a server like this would take an entire room, now is a rack. How many years to companies have servers with similar size to a Raspberry Pi and performance/storage of a today's rack?

That's why I love tech.

This is such a good statement, real-world here.. when we upgraded our NetApp about 3 years ago, which was a FAS8040 with a mix of DS2246 shelves with SAS disks and DS4246 shelves with SATA disks, over to an All Flash system (AFF220), sizing wise, we went from what was 38U of rack space down to 2U of rack space, and we even doubled the overall capacity. It's amazing how quick the storage space evolved.