This is stupid, but I love it (SPONSORED)

[JonoT]

 

Thanks to Kioxia for sponsoring today's video! Check out their Enterprise Data Center SSD storage drives at https://lmg.gg/iEYI8

 

Over the last few years, we've tried to deploy this Gigabyte server chassis as a video editing server NUMEROUS times. Unfortunately, we had issue after issue with the intel drives we were trying to use, but this time, with the help of our sponsor KIOXIA, we're deploying it as my new ultimate home NAS with their CD6 enterprise NVMe SSDs.

[emosun]

I always wonder how much usage they actually get out of "home _____ pc"

like the next one is going to be "dual rtx 3080 guest bathroom media server pc"

[NegroRotary]

what is the server chassis you are using?

[Shreyas1]

Put them all in raid 0 lol

[Upgrade Plans]

The quick dismissal of 2 parity drives by Linus kinda rubbed me the wrong way. I crunched the numbers that I found in Kioxia's datasheets, and I do now agree that it is totally fine to have only one parity drive with this setup. But in general, 2 parity drives is very often not overkill for real world data-center installations.

With the sizes of today's hard-drives, compared to their speeds and lifetimes, a failure during recovery is a very real possibility. If you have a failure during recovery, this means data-loss!

Now this does mean calculating (or looking up) the MTTF, MTBF, and MTTR, and also discussing the risk of data-loss your business decides to allow (1%, 0.1%, 0.0001%, etc.). This calculation should be what determines the number of parity drives, not an "I mean one parity drive is probably fine".

I would accept an "I mean two parity drives are probably fine" by the way, because data-loss in that case works out to have negligible odds with any storage solution for sale today.

Otherwise, great video.

[Nystemy]

I have long been curious to the pros and cons of flash storage, especially in long term storage applications.

 

I will give a quick heads up here in the beginning that I will get technical in this little write up.

 

Due to the way that a Flash cell stores data, it is inherently not persistent long term. As I explained here: 

 

This means for an example if we put a USB thumb drive or SSD into a time capsule for even 20 years, it is likely fairly corrupted or even empty by the time it gets uncovered. _{(There is also a difference depending on the exact manufacturing technology that went into making the flash cells. For an example, an EEPROM is just FALSH, but built to a different spec where it suddenly costs a dollar for every few MB instead of a dollar for ever couple of GB.)}

 

But in a storage server like this, we won't keep these drives off. So the story becomes different compare to the raw data retention time of a flash cell, and it is here I get a bit curious.

 

The flash array could for an example also contain sufficient additional information to be error correcting. This means that if we were to read a given area and detect some corruption, then we can fix said corruption. But is this a thing that drive manufacturers do? _{(if they don't, then bit rotting is a large potential issue to contend with.)}

 

Secondly, if there is some error correction, it would logically only apply locally within the given area of the array that we interacted with, unless the drive also is periodically scanning itself and searching for errors within the array _{(as in once every other year)}. And is this something drive manufacturers do? _{(if they don't, then the error correction only exists for the areas we use sufficiently often. Ie data that never gets looked at for a long time is potentially going to get corrupted.)}

 

Now, some might ask, "but isn't HDDs also subject to sporadic errors over time?" and yes they are. But magnetic media has historically proven itself to have little degradation over time as far as the data is concerned. Not to mention that the loss of magnetic data is a bit more random _{(though, most commonly the drive itself dies due to mechanical reasons or corrupted firmware)}, unlike Flash media that has a more well defined data retention time. _{(but even hard drives do need some degree of error correction as well.)}

 

If there is no inherent error correction in the drives, then we would technically need to have some error correction in our file system instead to implement the two features stated above.

 

The reason I make this post is due to both my own experiences and the experiences I have heard from others over the years.

I have seen SSDs in particular develop issues over time, file corruptions, or that the OS itself suddenly can't boot for inexplicable reasons _{(if the SSD is used for the OS that is)}. As if there is no mechanism there to correct any data corruption that is inevitable for flash media.

 

Though, flash media tends to have a wear leveling mechanisms, and this is the semi-big saving grace to this issue. But it only really applies if one writes relatively often to the drive. _{(as in write a decent percentage (as in 20-40%) of the drive's total capacity every other year. (especially as the drive gets older and gets more forgetful))} Then for a storage server with 10's of TB of FLASH, can one keep up for wear leveling to fix the data corruption? _{(I should however add that wear leveling wouldn't inherently fix any errors, just move the data from place "A" over to "B" before "A" has leaked sufficiently to be corrupted.)}

 

But the core of the issue is the data retention time. It isn't a particularly easily defined metric. And most flash manufacturers just says "about 10 years". And to be fair, 10 years is decent enough, but is this for a brand new drive, or one that has gotten up to its total bytes written (TBW)? And at what operating temperature? _{(If I might take a cynical view, then it is 10 years for a brand new drive, at 25 C (standard room temperature), and that the data retention times has halved when the TBW is reached. But this is a shot in the dark.)} But in any regard, a lot of SSDs starts showing file corruption after 4-6 years of fairly non intense service, and that isn't a particularly long time as far as long term storage is concerned. A simple block based CRC and a yearly check should solve nearly all flash related file corruption. _{(also, as covered in the prior post I linked to above, data retention times roughly halves with every 10 degree C temperature increase. The typical case is a good 10-25 C higher than ambient internally. Then the flash chip itself is likely another 0-20 C warmer than that depending on the workload.)}

 

Some might now say, "but isn't RAID error correcting?" and the answer here is most often "no". Since the RAID system can't really figure what data is incorrect, just that the data is incorrect. However, RAID can figure out what the missing non existing data should have been if a drive is completely lost, but if the drive is still there but giving conflicting data, then it can't know what side of the conflict is correct. _{(RAID 6 might be able to figure out the correct side of a conflict, since it has a 2 bit parity checksum.)}

 

So to end this book.

Is flash a good option for longer term storage?

For storage acceleration, caching, or as a front end data base, Flash storage is wonderful. But for primary storage? _{(now one should have backups regardless, but preferably one shouldn't have to rely on the backups.)}

[NateR36]

They're using NVMe drives, but I didn't see them set the ashift on the zpool to 12 or 13?

 

Aren't they losing a lot of performance from that?

[Dandellion]

It's worth mentioning here that you should probably never RAIDZ-1 an SSD array like this. When using raidz you limit yourself to the IOPS of a single drive, when IOPS is more often than not why you used SSDs in the first place (i.e for a hot database load or similar)

[just_dave]

You blew the chance to name the computer LI-NAS

[Luscious]

Overkill is an understatement when you are rocking U.2 drives for a home server, I started with a 2 drive Buffalo NAS some years ago that doesn't even have redundancy just to stream my DVD rips and it served me well. When the time came to upgrade to Blu Ray though I needed something with a little more performance and additional flexibility. That's when I really began to take a serious look at not just NAS options but alternatives. Keep in mind I'm just a single user only doing movie streaming to one device, no nightly backups from other PC's and no on-the-fly compression or transcoding.

 

My solution was a relatively simple one using four of the sata ports on my mobo to attach four 8TB spinners and creating a virtual drive using StableBit DrivePool software. I've got it configured so that if one drive goes down the data can still be accessible. It kind of mimics RAID in that it will split the data into chunks across the drives so that you can enjoy fast read speeds as well. Overall I'm very happy with the setup and the four 8TB spinners are plenty to store my growing collection of roughly 200 Blu Rays and 500 DVD's. And this is the same box I use to do my Steam game streaming to a XPS laptop over wifi.

 

Of course if I did have multiple users to deal with I would probably avoid any kind of networked streaming altogether. A small HTPC directly attached to a TV offers more than one benefit. Firstly, being direct attached, you avoid using any network bandwidth completely - that's a huge deal when somebody else is trying to download a huge update, game remotely or stream from the web. A single 16TB SSD can store a lot of Blu Rays and will also find a home in a small NUC barely noticeable next to a large TV. Clone the same drive in every NUC and you have backups in place if any single drive/NUC dies.

[microsoftenator]

You don't need ZFS for quick snapshots of arbitrary data (LVM's thin pool snapshots work just as well, IME) and using ZFS comes with *some* downsides - eg. in practice your vdevs have to be identically-sized drives, so expanding the array means buying N new drives (where N is the number of drives in the existing vdevs)

 

My home PC is performing triple duty as a Linux desktop, Windows gaming PC and file server (all VMs on a barebones Linux host) and I've not felt the need to run ZFS yet. I personally prefer doing storage management for the host through LVM as that doesn't require "tainting" the kernel with out-of-tree drivers as is the case for ZFS.

 

Also BTRFS RAID 1 is very stable these days and has the "advantage" for home users of being able to expand one drive at a time (although you probably need to balance the array after adding drives if N != mirrors)

[GeorgeMKane]

9 hours ago, emosun said:

I always wonder how much usage they actually get out of "home _____ pc"

like the next one is going to be "dual rtx 3080 guest bathroom media server pc"

"dual rtx 3090 basement bunker lockdown pc"

[Freakwise]

10 hours ago, NegroRotary said:

what is the server chassis you are using?

Gigabyte oem chassis, it's a barebone server

[GodAtum]

I'd like to setup that remote PC storage system. I guess i would need to convert my PCs to thin clients and network boot? Any guidance while I wait for that video?

[CerealExperimentsLain]

14 hours ago, Luscious said:

Of course if I did have multiple users to deal with I would probably avoid any kind of networked streaming altogether. A small HTPC directly attached to a TV offers more than one benefit. Firstly, being direct attached, you avoid using any network bandwidth completely - that's a huge deal when somebody else is trying to download a huge update, game remotely or stream from the web. A single 16TB SSD can store a lot of Blu Rays and will also find a home in a small NUC barely noticeable next to a large TV. Clone the same drive in every NUC and you have backups in place if any single drive/NUC dies.

Erm... The maximum possible bitrate on a remultiplexed UHD BluRay would be 144mbps...  And this is a 10 000mbps network... That's 1.4% of the total network bandwidth. (And that's assuming that the path to both machines from their sources even goes along the same links, with how it's switched, and what machines are involved, one PC accessing the NAS and another accessing the internet for updates may never be using the same links).  I think that the network will be fine providing data from the NAS and let another machine run Windows Update or a Steam update without issue.

 

I'm running Remuxes across a 1gbps network powered only by a pfSense box and four unmanaged 1gig switches and let me tell you; Yeah, you can run updates just fine.  Pretty sure a 10 gig network will do fine.

 

And, on a side note, as someone running 199TB of storage across two UnRAID machines, I just want to say that I find it absolutely adorable that you think a 16TB drive plugged into a TV would be 'enough' storage.  Seriously, I have a single television series that uses 1.2TB of storage.  You're adorable.

[BaneWilliams2]

I'm going to be super whiny here.

 

The background track for a majority of the video has this really annoying right channel click when played at 2x speed. For a moment I actually thought my headphones had some kind of problem, but thankfully figured out pretty quickly that it was the video, and therefore the background track.

 

I understand that only 8% of watchers watch at 2x speed, and that it's likely that only autistic folks like myself who would even find it an issue to begin with, but if it's zero work on your end would you consider not using that particular track in videos for extended lengths of time?

[driftar]

I just stumbled across that video and I really like the idea of a real home server, with performing hardware. The Kioxia drives are nice, not that expensive but somewhat, but still affordable. But with the server chassis, I'm not sure. I'd love to get the bill of material of that home server. I'm building a house currently, and this could be a good start (installed with Unraid, for backing up my PCs, smart home with home-assistant, and so on).

 

Cheers,

Karl

[mytime34]

Are the "array" ssd's still running strong? or did you have to replace them?