Considering 2x 18 or 20 TB HDD (replace 8+10TB) : 1 - is 3rd-party seller ok? ; 2 - 18TB <$/TB but 20TB more space/bay...

PianoPlayer88Key · November 5, 2022

Hey ... I'm considering buying a couple 18TB or 20TB hard drives, with which I plan to replace an 8TB and 10TB with each. (I'd like 26+TB drives so I can replace 8+8+10TB, but the only one that exists is host-managed SMR afaik.)

The drives I'm considering:

18TB Toshiba MG09ACA18TE - $299.99 (Newegg 3rd party - Platium Micro)
18TB WD (Ultrastar DC H550 0F38459) WUH721818ALE6L4 - $319.99 (B&H, or Newegg 3rd party - Platinum Micro)
18TB WD (Red Pro) WD181KFGX) - $349.99 (Newegg)
20TB WD (Ultrastar DC H560 0F38785) WUH722020BLE6L4 - $379.99 (Newegg)
20TB WD (Ultrastar DC H560 0F38755) WUH722020ALE6L4 - $387.99 (Newegg)

(No Seagates are under consideration, because I & family / friends have had reliability issues with them in the past, etc. I had considered an Exos or Ironwolf Pro but my brother, who's better with tech than me, said avoid Seagate HDDs, although I do have a 2TB Seagate SATA SSD in my laptop and it's been fine.)

The best deal appears to be the 18TB Toshiba, but would I need to be worried about being denied warranty coverage if I buy from a 3rd-party seller, or getting a used / refurb drive? (Also what is Platinum Micro's reputation like?)

Or, I might consider the 18TB or 20TB Ultrastar drives. The 18TB is a bit cheaper per TB ($19.15 vs $20.47 with 7.75% tax), but would it be worth it to go for the 20TB drives? (I don't run a datacenter but I still consider some value in using fewer drives for the same / more capacity.)

Or should I wait until closer to Black Friday, or are the 18+TB drives with 5+ year warranties not likely to be discounted much more? (I did miss a sale in mid/late October where prices on some models were somewhat lower than they are now, for example a 20TB WD Red Pro was available for $359.99, or $284.99 for 18TB.)

(I was going to ask in Discord, but thought I'd ask here instead. Even though replies will be a lot slower (hours or days vs seconds or minutes), there's a better chance that I will see them here.)

Also I'm needing to do some decluttering, get rid of some duplicate files / folders (there's some things that are probably duplicated like 5 or 10 times and scattered across multiple hard drives, some things possibly being 8 or 12 folders deep, some files having different names but same contents, etc, so idk if WinMerge would help much) ... I'm guessing the Programs, Apps and Websites subforum is a better place to ask about that? (I was able to free up about a TB and a half or so just by finding some duplicate video files, comparing them by playing in VLC then deleting the ones with obscure names cause TestDisk / PhotoRec didn't preserve their normal names, but that was the easy part.)

Also how long do you think until SSDs overtake HDDs in TB/$ and TB/drive, assuming the same level in the product stack? (As in, high-end (TLC/DRAM/NVMe) SSD vs high-end (enterprise/CMR/etc) HDD, and the SSDs being able to handle 24/7 writes without dying better than the HDDs, among other things)

heimdali · November 5, 2022

Good selection, what about HGST? And it seemed to me that 16TB is currently more cost effective than the lager ones. Wait until black friday unless you find a really good offer. Those prices seem to change every day.

Perhaps use a file system that supports deduplication.

PianoPlayer88Key · November 5, 2022

33 minutes ago, heimdali said:

Good selection, what about HGST? And it seemed to me that 16TB is currently more cost effective than the lager ones. Wait until black friday unless you find a really good offer. Those prices seem to change every day.

Perhaps use a file system that supports deduplication.

HGST, afaik, was bought by WD. (I heard some of their technology, maybe involving 3.5" disks, went to Toshiba.) The model numbers of the WD Ultrastar drives use very close to the same pattern as some HGST drives had used, and in fact Ultrastar was an HGST brand I believe.

I had been maybe considering 16TB, but I'm planning to replace 10TB and 8TB drives, and last I checked, 10 + 8 > 16. Also what would you consider a really good offer? (For me, I'm thinking 22TB under $360, 20TB under $300 or 18TB under $250 or something like that.) I'd really like 26TB or larger since I want to replace 8+8+10, but those aren't really available yet. (There is one from WD, but it's host-managed SMR among other issues.)

One issue I've been having with the 8 and 10TB drives (HGST Deskstar NAS), and therefore considering replacing them, is that once in a while they'll go to 100% access time in WIndows task manager, and be completely inaccessible. I'm not 100% sure it's the drives though, as often at the same time my motherboard's built-in wifi drops out (to the point where WIndows says no network connections detected or something like that). Sometimes my keyboard/mouse have also quit responding requiring me to use the reset button to restart the PC, and at least once when I'd come back to the PC, it was on a black screen with the GPU pinned at 100% fan speed. Restarting the PC usually cured those issues, until it cropped up again a few weeks or a month or so later.

System is an AMD 5950X, ASRock B550 Taichi, Arctic Liquid Freezer II 360 (I'm aware that some have issues but haven't had trouble with mine yet and don't want to take it apart unless I absolutely have to and am making a significant platform upgrade, for example socket SP5 or DDR6/7 (although this one was a budget-constrained stop-gap incremental upgrade from a 4790K, went 5950X in case it would be more than several years before I could replace it)), Silicon Power 1TB P34A80 boot drive with Windows 10 Pro, 128GB Team Expert DDR4-3600 CL18 running at 3466, EVGA SC GTX 1060 3GB, Corsair RM850 PSU, Fractal Design Define R5 case, plus a 250GB Crucial CT250MX200SSD6 with a Linux install. Current HDDs are 2x 14TB Toshiba MG07ACA14TE, 12TB Toshiba MG07ACA12TE, 2x 10TB HGST HDN721010ALE604, 2x 8TB HGST HDN728080ALE604 and 8TB Toshiba HDWG180XZSTA. Monitor is an ASUS VG289Q, keyboard is Logitech K270, mouse is Logitech G604.

As for deduplication ... I had thought about ZFS deduplication, but the suggestion on some sites for 5GB RAM per TB of raw storage gives me paranoia. Also I feel like dedup would basically just get rid of the actual duplicate data but still leave some kind of pointers / shortcuts intact in the file system where the actual duplicate files had been, but in my case, I want the actual duplicate files / folders gone. (Maybe I'm too much of a packrat, I still have some documents / spreadsheets on the PC dating from when my family first had a '286 PC in Q1'1989 when I was a kid, and some games from the late 80s / early 90s that if I could, I might want to play some of them again.)
I'd like to find some utility that goes through my storage, visually lays them out for me then lets me decide what to keep / merge vs what to delete. Maybe something like WinMerge, but looks across the entire drive for patterns of similar files in the same folders, and also allows for files having different names and dates, but the same size and contents. (Bonus points if it could also find near-matches, for example the same audio, photo, video files encoded with different settings, for example 1.4Mbps wave vs 48kbps mp3, or 16 megapixel Raw / BMP vs 640x480 75% jpg, or 4K 100mbps H.264 vs 640x480 2mbps MJPEG or whatever.)

heimdali · November 6, 2022

3 hours ago, PianoPlayer88Key said:

HGST, afaik, was bought by WD. (I heard some of their technology, maybe involving 3.5" disks, went to Toshiba.) The model numbers of the WD Ultrastar drives use very close to the same pattern as some HGST drives had used, and in fact Ultrastar was an HGST brand I believe.

Have they given up the brand name completely? The HGST Ultrastars were always good and it would seem stupid to give up the brand.

3 hours ago, PianoPlayer88Key said:

I had been maybe considering 16TB, but I'm planning to replace 10TB and 8TB drives, and last I checked, 10 + 8 > 16.

Does it matter? Are you running them in RAID0?

3 hours ago, PianoPlayer88Key said:

Also what would you consider a really good offer?

Hard to say, the prices you're getting are already very good. They're way more expensive here --- so expensive that I would pull the old 3TB reds from my backup server and add them to my server instead of buying more new disks at these prices. Before inflation, I was lucky to get 2x8 gold for EUR 255 each because I happened to hit the low price which went up from there. Those are still more expensive here even today. 12TB gold are about EUR 380 and the 16TB are 470--570, so go figure.

So for all I know, your prices could be double tomorrow or go down or go somewhere in between. You might just have happened to hit that very day where the prices are low. Or prices go down on black friday, who knows. If I could get 16TB or maybe be 12TB for under 300, I might just buy even though I still have enough free space. These days, what you don't buy today will be more expensive tomorrow.

I don't know what you're doing, but there is no way to store data on a single disk, so disks always come in pairs at least. I'm not willing to spend like EUR 1000 or more on just 16TB. I can fit 14 disks in my server, so I could use smaller ones if I have to, or, if prices get even worse, I could use like half of my backup server as archive instead of for backups.

3 hours ago, PianoPlayer88Key said:

One issue I've been having with the 8 and 10TB drives (HGST Deskstar NAS), and therefore considering replacing them, is that once in a while they'll go to 100% access time in WIndows task manager, and be completely inaccessible.

How would that have anything to do with the disks themselves unless they need to wake up from saving energy? Besides, NAS drives tend to be designed for saving energy and to be somewhat slow, at least that goes for the 3 and 4TB reds. I quit buying those for that reason (their failure rate isn't great, either), so if you're looking at NAS disks, you have to ask yourself if they're the right choice. And don't buy 5400rpm ones. Even with the gold ones I disabled the energy management because it had awful latencies when accessing files until I finally found out they go to sleep and crank up their parking count up like crazy. Question is what makes them fail sooner, keeping them running or let them go to park all the time.

3 hours ago, PianoPlayer88Key said:

As for deduplication ... I had thought about ZFS deduplication, but the suggestion on some sites for 5GB RAM per TB of raw storage gives me paranoia.

ZFS has been designed for arrays with 40 disks or more and the idea of protecting the data was paramount while performance was not. I'm not too fond of it at all, especially not with only a few disks. I'd go with btrfs unless you want RAID5 or 56. If you keep multiple generations of backups, I guess deduplication might be particularly great, but I haven't experimented with that yet.

And why is everyone so scared of RAM? Just get a server with 128GB or more, or upgrade it. It's not like it makes so much difference anymore.

3 hours ago, PianoPlayer88Key said:

Also I feel like dedup would basically just get rid of the actual duplicate data but still leave some kind of pointers / shortcuts intact in the file system where the actual duplicate files had been,

How would you access a file that has been deduplicated without a pointer to it where you need it? Transitively spoken, deduplicating files doesn't mean disappearing files.

3 hours ago, PianoPlayer88Key said:

but in my case, I want the actual duplicate files / folders gone.

You can still delete the pointers. The advantage is that you don't need to. If you want to actually delete them, just write a little program that goes through your file system and finds all the duplicates, if there isn't already one.

3 hours ago, PianoPlayer88Key said:

(Maybe I'm too much of a packrat, I still have some documents / spreadsheets on the PC dating from when my family first had a '286 PC in Q1'1989 when I was a kid, and some games from the late 80s / early 90s that if I could, I might want to play some of them again.)

I don't have that because I didn't have the disk space back then to keep it and had to start deleting too often. If you keep hoarding like that, it'll become expensive and unwieldy. Maybe find out how well compressing your data would work and use a file system that compresses it on the fly.

3 hours ago, PianoPlayer88Key said:

I'd like to find some utility that goes through my storage, visually lays them out for me then lets me decide what to keep / merge vs what to delete.

Visually lays it out like how? There was, or still is, some program the name of which I forgot that would make a colourful display of many squares/rectangles with the fields the larger the larger the file it represented is. It helps you identify large files, or, if applied to directories, those directories that take up a lot of space. Then those may be the ones worthwhile to take a closer look at. It's only of very limited help, though. IIRC it was some KDE program.

3 hours ago, PianoPlayer88Key said:

Maybe something like WinMerge, but looks across the entire drive for patterns of similar files in the same folders, and also allows for files having different names and dates, but the same size and contents. (Bonus points if it could also find near-matches, for example the same audio, photo, video files encoded with different settings, for example 1.4Mbps wave vs 48kbps mp3, or 16 megapixel Raw / BMP vs 640x480 75% jpg, or 4K 100mbps H.264 vs 640x480 2mbps MJPEG or whatever.)

That doesn't seem useful. If I were to write a program to find duplicates, I'd probably go through a file systen and dump some information into a sql database, like file date and size of each file. Then make three queryies, one returning the files that have the same date and size, one that returns the files having the same size and one that returns the files having the same date. Then create a checksum like a sha256 or whatever suits best for the files remaining in the deduplicated results of each query and dump the checksums and then list the files the checksums of which are identical. If that's too slow, maybe dump the 512bytes of each file into the database when getting date and size and use that to compare the first 512 bytes to reduce the number of files to actually check further.

That's pretty trivial ...

PianoPlayer88Key · November 6, 2022

11 hours ago, heimdali said:

Have they given up the brand name completely? The HGST Ultrastars were always good and it would seem stupid to give up the brand.

I think they're not using the HGST name anymore, but they're using the Ultrastar name and a very similar model number pattern.

11 hours ago, heimdali said:

Does it matter? Are you running them in RAID0?

I'm not sure that it would matter.
No, I'm not running them in any kind of RAID. I had briefly considered RAID1 in case I bought a Seagate drive and another brand, but I would only be running motherboard or Windows RAID. I've heard concerns about the former having issues sometimes, and I'm not sure if the latter would be recognized on the occasions I boot Linux.

Also, in that capacity range, I right now actually have a total of 6 drives - 3x 8TB HGST Deskstar NAS, 1 8TB Toshiba N300, and 2x 10TB HGST Deskstar NAS.
I currently have them set up so that 2 of the 8TB HGST drives, and 1 of the 10TB is used for data storage for whatever I put on it. (Between an 8TB and 10TB, they house the vast majority of my older stuff.) The remaining drives (1 8TB HGST, the 8TB Toshiba & the other 10TB HGST) are cold-storage (normally unplugged) backups that admittedly are quite overdue for updating as I haven't done that in at least a year or more. (I had basically copied the entire contents of the main data drives to the backup drives, but going forward I need to find a better way to keep backups updated. I've thought of some details as to what I want to be able to do but don't want to clutter this post with that.)

11 hours ago, heimdali said:

Hard to say, the prices you're getting are already very good. They're way more expensive here --- so expensive that I would pull the old 3TB reds from my backup server and add them to my server instead of buying more new disks at these prices. Before inflation, I was lucky to get 2x8 gold for EUR 255 each because I happened to hit the low price which went up from there. Those are still more expensive here even today. 12TB gold are about EUR 380 and the 16TB are 470--570, so go figure.

So for all I know, your prices could be double tomorrow or go down or go somewhere in between. You might just have happened to hit that very day where the prices are low. Or prices go down on black friday, who knows. If I could get 16TB or maybe be 12TB for under 300, I might just buy even though I still have enough free space. These days, what you don't buy today will be more expensive tomorrow.

I don't know what you're doing, but there is no way to store data on a single disk, so disks always come in pairs at least. I'm not willing to spend like EUR 1000 or more on just 16TB. I can fit 14 disks in my server, so I could use smaller ones if I have to, or, if prices get even worse, I could use like half of my backup server as archive instead of for backups.

Ahh, EUR vs USD, yeah there would be a difference there. (I'm east of San Diego, CA, about an hour and 25 minutes from Micro Center in Tustin, but when I've been there they haven't always had the lowest prices on HDDs, or the models that I would want in stock.

It briefly popped into my mind to maybe buy the drives now, but don't open them yet, and if prices drop for BF, see if they'll price match after the fact (some may, some may not), or if different models go on sale, return them and buy the others. (But, there might be a restocking fee to consider which would probably negate any potential savings, so that's probably not a good idea.)
Also I'm not running any kind of server, it's just my home PC. While I do have quite a bit of stuff, I don't think I qualify for r/datahoarders yet.

(I'm guessing you're referring to RAID or backups when you talk about no way to store data on a single disk so they come in pairs, or do you mean something else?) Also 16 TB here can be had for about $300, or considerably less if you're willing to risk not having a warranty because of buying form a 3rd-party seller, or would I have a warranty still and shouldn't be worried about that?
If the $1000 on 16TB includes backups, I'd prefer if possible to spend a lot less on the backup media than on the main storage media. (That was possible in the early/mid 1990s - see the highlighted examples in the pic in the spoiler, comparing the cost of hard drives vs tape media and drives.)

Spoiler

11 hours ago, heimdali said:

How would that have anything to do with the disks themselves unless they need to wake up from saving energy? Besides, NAS drives tend to be designed for saving energy and to be somewhat slow, at least that goes for the 3 and 4TB reds. I quit buying those for that reason (their failure rate isn't great, either), so if you're looking at NAS disks, you have to ask yourself if they're the right choice. And don't buy 5400rpm ones. Even with the gold ones I disabled the energy management because it had awful latencies when accessing files until I finally found out they go to sleep and crank up their parking count up like crazy. Question is what makes them fail sooner, keeping them running or let them go to park all the time.

It's not due to waking up from saving energy, because 1 - I don't think those drives have that function (at least not like the green drives, and I don't have Windows power plan set to turn off the hard disks at all), and 2 - when that issue happens, it's still persisting several minutes later as I"m trying to go through other stuff preparing the PC to restart, and if I go away and come back later they're still stuck at 100% usage with the drives inaccessible to Windows. (In "This PC" where it shows the drives, the drive letter will be there but the bar showing free space, and the capacity, will not be there.) When I restart (which sometimes requires just using the reset button on the case especially when my software restart attempt hangs for several minutes), it's fine until it happens again.
Also that issue seems to be only happening with the 8 & 10 TB HGST drives; my 12 & 14 TB Toshiba drives are fine, and my 8 TB Toshiba might be okay as well but I haven't verified that last one.

The last 5400rpm "green" drive I bought was a WD WD20EZRX several years ago, and I no longer have that drive. Since about 2015, I've only bought 7200rpm drives that are rated for NAS or "enterprise" grade drives, and going forward I plan to only buy drives with at least a 5 year warranty and preferably a 300 or 550+ TB/year workload rating. (No, not because I would push them that hard, but I feel like they would be more reliable than basic consumer-grade desktop drives like WD Blue, Seagate Barracuda, Toshiba P300.)

11 hours ago, heimdali said:

ZFS has been designed for arrays with 40 disks or more and the idea of protecting the data was paramount while performance was not. I'm not too fond of it at all, especially not with only a few disks. I'd go with btrfs unless you want RAID5 or 56. If you keep multiple generations of backups, I guess deduplication might be particularly great, but I haven't experimented with that yet.

And why is everyone so scared of RAM? Just get a server with 128GB or more, or upgrade it. It's not like it makes so much difference anymore.

Ahh, yeah I wasn't sure that ZFS would make sense in my case when I might only have 8 disks. (I have had 10 or 12 or so plugged in at a time, but that was on a motherboard with 10 SATA ports (ASRock Z97 Extreme6), and I had another PCIe to SATA 2 / IDE card plugged in as well. Side note: learned the hard way to not plug drives larger than 2TB into that card. Was able to get some stuff back and clone to other disks, etc, but that's partly why I have a bunch of files with gibberish names that were identical to some other files I also had, if you saw the part earlier where I had to use TestDisk / PhotoRec to recover some stuff.)

I've thought about building a NAS for backup purposes, but details are outside the scope of this post. (I had considered ZFS / TrueNAS for that, but have also been considering UnRAID so that I can add disks one at a time as necessary.)

I could sometimes use more than 128GB RAM I've had my pagefile on my laptop with 64GB RAM reach upwards of 308GB (which with the 64GB RAM used adds up to 372GB committed), and right now my total committed on my desktop with 128GB RAM is about 131GB (interestingly it says only 68GB RAM is in use) and I've seen it quite a bit higher.
(Also I forgot to mention earlier that sometimes only 2 of my 4 RAM sticks are detected, showing 64GB RAM instead of all 4 sticks at 128GB, which usually means I have to restart. There was one time I thought for sure it booted with all 128GB detected, but later came back to the PC and it was only showing 64GB, but I can't see how that would happen without the system crashing or blue screening, and I have it set to not automatically restart after a stop error.)

11 hours ago, heimdali said:

How would you access a file that has been deduplicated without a pointer to it where you need it? Transitively spoken, deduplicating files doesn't mean disappearing files.

You can still delete the pointers. The advantage is that you don't need to. If you want to actually delete them, just write a little program that goes through your file system and finds all the duplicates, if there isn't already one.

I kinda want to get rid of the file system clutter, and not have 5 or 10 copies of the same thing scattered who knows where. Yes, 2 or 3 copies is good for backup purposes, but they of course shouldn't be on the same drive. (I'd need to find a good solution for off-site backup though... there's no possible way I could do the daily full backup (which to me means every sector of every disk, for example using the linux "dd" command to clone disks which is how i've done a few backups & it's the only reliable way I've gotten a cloned Windows install to boot) that some people advocate, when I'd be backing up multiple 10s of TB (and besides there's no way I could afford that storage), especially on a site like BackBlaze when my internet is limited to 10 Mbps upload and a 1TB/month cap.

11 hours ago, heimdali said:

I don't have that because I didn't have the disk space back then to keep it and had to start deleting too often. If you keep hoarding like that, it'll become expensive and unwieldy. Maybe find out how well compressing your data would work and use a file system that compresses it on the fly.

Actually my older data isn't taking up a ton of space, compared to some of the newer stuff. About 6 and a half years ago I bought a camera that records 4K at 100 Mbps, which basically ends up being about 4 GB for every 5 minutes 55 seconds of footage. I've recorded some stuff with that camera, but haven't recorded nearly as much as I want to. (Sometimes I've recorded services at church camps, and that works out to around half a TB or so for each 3 or 4 day weekend event; and even just recording 2 hours will chew up about 80 GB. Do that twice a week, and it adds up; even more so if I also record other things that I've wanted to record but have been holding off for various reasons.)

11 hours ago, heimdali said:

Visually lays it out like how? There was, or still is, some program the name of which I forgot that would make a colourful display of many squares/rectangles with the fields the larger the larger the file it represented is. It helps you identify large files, or, if applied to directories, those directories that take up a lot of space. Then those may be the ones worthwhile to take a closer look at. It's only of very limited help, though. IIRC it was some KDE program.

I think you're talking about WinDirStat, and I've run that on this PC, it's given me a very general overview of things on here.

Spoiler

11 hours ago, heimdali said:

That doesn't seem useful. If I were to write a program to find duplicates, I'd probably go through a file systen and dump some information into a sql database, like file date and size of each file. Then make three queryies, one returning the files that have the same date and size, one that returns the files having the same size and one that returns the files having the same date. Then create a checksum like a sha256 or whatever suits best for the files remaining in the deduplicated results of each query and dump the checksums and then list the files the checksums of which are identical. If that's too slow, maybe dump the 512bytes of each file into the database when getting date and size and use that to compare the first 512 bytes to reduce the number of files to actually check further.

That's pretty trivial ...

Ahh, I hadn't thought of an sql database and the last time I did any coding was like 25 or 30 years ago as a kid, and that was really simple stuff in basic and pascal.
I generally like the idea though, of querying files with same date, and same size, then when matches are found there, start comparing checksums, or first portions of files to see what's identical. (I would probably use a fair bit more than 512 bytes for the initial compare though, then if those match, compare the entire files. It's possible, for example, I might have audio files that are the same size, and the sound is identical except where I might have applied a fade to silence at the very end of a copy, and I'm guessing the duplicate detector would need to parse the entire file to find it.)

Also I'm pretty sure there are programs out there that will find duplicate or near-duplicate files ... but what I was looking for is something that will find multiple duplicate folders.

One bad habit I had gotten to, when copying stuff from an SD card either from my phone when I had one that used it, or from my camera or my audio recorder (Zoom H2/H2n or others), was just dumping the entire contents of the card onto the PC (cause I was too lazy to only choose the portions that I had recorded new things), then leaving stuff on the card cause I still wanted it accessible in-device, then again copying the entire thing another time later to another folder on the PC. (There were some occasions where I only copied the new stuff and deleted the old stuff off the card, but not always.) So, I've got a bunch of duplicate folders scattered all over the place.

Also I have duplicates of older "My Documents" folders, maybe even a few old Windows installs, etc, although on the Windows installs, I'm thinking I should get rid of the ones for which I no longer have the hardware that it was being used on. Problem is, I can't just delete some of them, as back in the day (late 1990s into mid 2000s or so) there were sometimes user / data files that somehow got saved in the Windows folder, or in the program files folder, instead of in the user area (like users\username\(etc), or in my case on a different drive entirely). (Also as for the program files, sometimes I've wanted to be able to run some older programs again, but I'd either need to find installers or find some way to get them to work, and I'd at least want to know which ones they were, and preferably if possible preserve the settings that I had used in the past, but that's outside the scope of this post, and I have too many other things going on to really take any time to pursue that for now.)

heimdali · November 6, 2022

5 hours ago, PianoPlayer88Key said:

I think they're not using the HGST name anymore, but they're using the Ultrastar name and a very similar model number pattern.

Hmm, I wonder why they won't ...

5 hours ago, PianoPlayer88Key said:

I'm not sure that it would matter.
No, I'm not running them in any kind of RAID. I had briefly considered RAID1 in case I bought a Seagate drive and another brand, but I would only be running motherboard or Windows RAID. I've heard concerns about the former having issues sometimes, and I'm not sure if the latter would be recognized on the occasions I boot Linux.

Yes, that's called fake raid. When you use that and your mainboard fails, you will have to find one that can read the disks, so using fake raid isn't an option.

It's also not an option to store data on only a single disk. When the disk fails, you may loose your data, and it's a big hassle.

5 hours ago, PianoPlayer88Key said:

Also, in that capacity range, I right now actually have a total of 6 drives - 3x 8TB HGST Deskstar NAS, 1 8TB Toshiba N300, and 2x 10TB HGST Deskstar NAS.
I currently have them set up so that 2 of the 8TB HGST drives, and 1 of the 10TB is used for data storage for whatever I put on it. (Between an 8TB and 10TB, they house the vast majority of my older stuff.) The remaining drives (1 8TB HGST, the 8TB Toshiba & the other 10TB HGST) are cold-storage (normally unplugged) backups that admittedly are quite overdue for updating as I haven't done that in at least a year or more. (I had basically copied the entire contents of the main data drives to the backup drives, but going forward I need to find a better way to keep backups updated. I've thought of some details as to what I want to be able to do but don't want to clutter this post with that.)

You really need to use redundancy for your active disks. I also recommend it for backups, but when one of the backups fails, you can make a new one and it's not too much hassle since you have to make backups anyway if you have at least the minimum of 2 generations of backups. Hard disks are anything but cheap If you have the disks, there's no reason not to use redundancy for backups as well.

5 hours ago, PianoPlayer88Key said:

Ahh, EUR vs USD, yeah there would be a difference there. (I'm east of San Diego, CA, about an hour and 25 minutes from Micro Center in Tustin, but when I've been there they haven't always had the lowest prices on HDDs, or the models that I would want in stock.

I have to order everthing online; there's no place around that would have anything I would use on stock. Also, they removed all parking, so the only thing you can still get is groceries from the supermarkets that have parking. At the same time, they're complaining that the cities are empty ... What do they expect ... It's funny, a while ago cities used walls to prevent ppl from going there; nowadays they have all parking removed and nobody even wants to go there anymore.

Besides, I remember times when there were interesting stores in town with stuff I might actually buy. Those started disappearing when they started to force customers to walk for miles by closing the roads and removing parking, and since decades, there's no reason to go there anymore. If I couldn't order online, it would be like in the stone age ...

5 hours ago, PianoPlayer88Key said:

It briefly popped into my mind to maybe buy the drives now, but don't open them yet, and if prices drop for BF, see if they'll price match after the fact (some may, some may not), or if different models go on sale, return them and buy the others. (But, there might be a restocking fee to consider which would probably negate any potential savings, so that's probably not a good idea.)

I have no idea ... it didn't occur to me that anyone wouldn't order them online but actually go to all the lengths of going to a store. ... I'm really surprised that you would go to a store for some hard disks. Your overall situation must be way better than it's here that you can afford to waste the fuel on that.

5 hours ago, PianoPlayer88Key said:

Also I'm not running any kind of server, it's just my home PC.

I might be a good idea to do that.

5 hours ago, PianoPlayer88Key said:

While I do have quite a bit of stuff, I don't think I qualify for r/datahoarders yet.

You have way more capacity then I do and kept your data since the 80ies, so of course you do qualify.

5 hours ago, PianoPlayer88Key said:

(I'm guessing you're referring to RAID or backups when you talk about no way to store data on a single disk so they come in pairs, or do you mean something else?)

It is out of the question to store data on only a single disk without redundancy. Backups are not a substitute for redundancy, and redundancy is not a substitute for backups. Redundancy requires to buy disks at least in pairs; how else would you have redundancy?

5 hours ago, PianoPlayer88Key said:

Also 16 TB here can be had for about $300, or considerably less if you're willing to risk not having a warranty because of buying form a 3rd-party seller, or would I have a warranty still and shouldn't be worried about that?

Using a warranty usually would require that you return your disk to someone. There is no way that I would give my data out of hands. Other than that, it doesn't matter here, there are legal requirements so you do have a warranty unless you buy from a private seller who has taken precautions. I don't know how it's in California.

5 hours ago, PianoPlayer88Key said:

If the $1000 on 16TB includes backups, I'd prefer if possible to spend a lot less on the backup media than on the main storage media. (That was possible in the early/mid 1990s - see the highlighted examples in the pic in the spoiler, comparing the cost of hard drives vs tape media and drives.)

What I meant is that if I were to buy 16TB hard drives, I'd have to buy at least two of them and that would cost me over EUR 1000. Discs always come at least in pairs.

5 hours ago, PianoPlayer88Key said:

Reveal hidden contents

It's not due to waking up from saving energy, because 1 - I don't think those drives have that function (at least not like the green drives, and I don't have Windows power plan set to turn off the hard disks at all)

You can always check with hdparm. Especially for the so-called green ones I'd expect them to go to sleep all the time because the idea of the so-called green ones is saving energy.

5 hours ago, PianoPlayer88Key said:

, and 2 - when that issue happens, it's still persisting several minutes later as I"m trying to go through other stuff preparing the PC to restart, and if I go away and come back later they're still stuck at 100% usage with the drives inaccessible to Windows. (In "This PC" where it shows the drives, the drive letter will be there but the bar showing free space, and the capacity, will not be there.) When I restart (which sometimes requires just using the reset button on the case especially when my software restart attempt hangs for several minutes), it's fine until it happens again.

Perhaps you have multiple issues, and it's possible that you need to update the firmware of your disks.

5 hours ago, PianoPlayer88Key said:

Also that issue seems to be only happening with the 8 & 10 TB HGST drives; my 12 & 14 TB Toshiba drives are fine, and my 8 TB Toshiba might be okay as well but I haven't verified that last one.

So is there a firmware update for those, or is there anything else special about them? Maybe broken cables?

5 hours ago, PianoPlayer88Key said:

The last 5400rpm "green" drive I bought was a WD WD20EZRX several years ago, and I no longer have that drive. Since about 2015, I've only bought 7200rpm drives that are rated for NAS or "enterprise" grade drives, and going forward I plan to only buy drives with at least a 5 year warranty and preferably a 300 or 550+ TB/year workload rating. (No, not because I would push them that hard, but I feel like they would be more reliable than basic consumer-grade desktop drives like WD Blue, Seagate Barracuda, Toshiba P300.)

That sounds like a good plan

5 hours ago, PianoPlayer88Key said:

Ahh, yeah I wasn't sure that ZFS would make sense in my case when I might only have 8 disks.

It depends ... Unfortunately, ZFS isn't compatible with Linux only for stupid licensing issues (you can get it to work, but that isn't a good option), so you'd have to go with FreeBSD, but the NFS implementation of FreeBSD sucks and makes that unusable for a file server. Then you may not be too happy with the performance of ZFS, unless perhaps you start tweaking it and using SSD cache. Tweaking it entails the risk of messing things up and/or loosing data --- or maybe not but it's a hassle. Also, the SSD is prone to wearing out when you use it for cache. And/or you're going to need lots of RAM, but you can't exacly cache in RAM because when the power goes out, the contents of the RAM are lost. So ZFS has good features but I'm not happy with it.

And I have to kinda correct my last post. Btrfs doesn't really support deduplication in that you can deduplicate it only afterwards and not while writing. I thought it could do that. That makes the possibility of deduplication pretty useless for backups because it requires that all the data fits on the volume, and if it fits anyway, there's no need to deduplicate it. For non-backups, I wouldn't trust it enough not to mess up my data. I was really disappointed.

But then, there's something that's called VDO. Apparently you create a VDO volume and put any file system on it and VDO can deduplicate it. I need to learn more about it, it would be great if I could use that. See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_storage/deploying-vdo_deduplicating-and-compressing-storage#doc-wrapper for more.

5 hours ago, PianoPlayer88Key said:

(I have had 10 or 12 or so plugged in at a time, but that was on a motherboard with 10 SATA ports (ASRock Z97 Extreme6),

How well did that work? From what I've been seeing, I have a theory that these consumer mainboards aren't able to handle a bunch of disks connected to them very well. It's like the board is overwhelmed with all the data going through it when data is being written to or read from all the disks in the RAID at the same time. It just slows everything down. Some people claimed that newer boards would handle better, but I doubt it. The data still needs to get through somehow.

5 hours ago, PianoPlayer88Key said:

and I had another PCIe to SATA 2 / IDE card plugged in as well. Side note: learned the hard way to not plug drives larger than 2TB into that card. Was able to get some stuff back and clone to other disks, etc, but that's partly why I have a bunch of files with gibberish names that were identical to some other files I also had, if you saw the part earlier where I had to use TestDisk / PhotoRec to recover some stuff.)

Older cards were limited to 2TB. That was a hardware limitation. In their time, disks weren't larger than that, so the cards didn't need to be able to handle them.

5 hours ago, PianoPlayer88Key said:

I've thought about building a NAS for backup purposes, but details are outside the scope of this post. (I had considered ZFS / TrueNAS for that, but have also been considering UnRAID so that I can add disks one at a time as necessary.)

I could sometimes use more than 128GB RAM I've had my pagefile on my laptop with 64GB RAM reach upwards of 308GB (which with the 64GB RAM used adds up to 372GB committed), and right now my total committed on my desktop with 128GB RAM is about 131GB (interestingly it says only 68GB RAM is in use) and I've seen it quite a bit higher.

The overcommitment doesn't really matter until the RAM isn't actually used.

5 hours ago, PianoPlayer88Key said:

(Also I forgot to mention earlier that sometimes only 2 of my 4 RAM sticks are detected, showing 64GB RAM instead of all 4 sticks at 128GB, which usually means I have to restart. There was one time I thought for sure it booted with all 128GB detected, but later came back to the PC and it was only showing 64GB, but I can't see how that would happen without the system crashing or blue screening, and I have it set to not automatically restart after a stop error.)

It seems like you need hardware that's more reliable.

5 hours ago, PianoPlayer88Key said:

I kinda want to get rid of the file system clutter, and not have 5 or 10 copies of the same thing scattered who knows where. Yes, 2 or 3 copies is good for backup purposes, but they of course shouldn't be on the same drive.

Perl is your friend ...

5 hours ago, PianoPlayer88Key said:

(I'd need to find a good solution for off-site backup though... there's no possible way I could do the daily full backup (which to me means every sector of every disk, for example using the linux "dd" command to clone disks which is how i've done a few backups & it's the only reliable way I've gotten a cloned Windows install to boot) that some people advocate, when I'd be backing up multiple 10s of TB (and besides there's no way I could afford that storage), especially on a site like BackBlaze when my internet is limited to 10 Mbps upload and a 1TB/month cap.

You can always use rsync. Some file systems support exporting themselves in some efficient way, too. I haven't tried that, maybe it's a good option. Rsync is more general and easy to use.

5 hours ago, PianoPlayer88Key said:

Actually my older data isn't taking up a ton of space, compared to some of the newer stuff. About 6 and a half years ago I bought a camera that records 4K at 100 Mbps, which basically ends up being about 4 GB for every 5 minutes 55 seconds of footage. I've recorded some stuff with that camera, but haven't recorded nearly as much as I want to. (Sometimes I've recorded services at church camps, and that works out to around half a TB or so for each 3 or 4 day weekend event; and even just recording 2 hours will chew up about 80 GB. Do that twice a week, and it adds up; even more so if I also record other things that I've wanted to record but have been holding off for various reasons.)

And you still think you're not hoarding data? Stuffing more disks into your computer is not the way to go. It takes no more than a disk failing or a little virus hitting you, or, if you don't have an UPS, a power surge and part of or all of your data is gone. Do you have at least ECC RAM in your computer?

5 hours ago, PianoPlayer88Key said:

I think you're talking about WinDirStat, and I've run that on this PC, it's given me a very general overview of things on here.

No, but it looked like the bar on the bottom. It's not too useful.

5 hours ago, PianoPlayer88Key said:

Ahh, I hadn't thought of an sql database and the last time I did any coding was like 25 or 30 years ago as a kid, and that was really simple stuff in basic and pascal.
I generally like the idea though, of querying files with same date, and same size, then when matches are found there, start comparing checksums, or first portions of files to see what's identical.

Like I said, Perl is your friend

5 hours ago, PianoPlayer88Key said:

(I would probably use a fair bit more than 512 bytes for the initial compare though, then if those match, compare the entire files. It's possible, for example, I might have audio files that are the same size, and the sound is identical except where I might have applied a fade to silence at the very end of a copy, and I'm guessing the duplicate detector would need to parse the entire file to find it.)

Why more than 512? I wouldn't use that much but it kinda makes sense to use sector size as orientation. You'd have to experiment with 1024 and 4096 or maybe even stripe size to see what's faster. You also need to consider that the database may be using the same disks you're reading from and you probably don't want to wait days on the results. If the first 512 bytes of two files are identical, you would compare them no further than the first byte that differs.

You could make it so that your result would say to what percentage files are identical. But how do you measure that? How identical are two files when all their bytes are the same but they're in a different order? You wanna sort them before comparing?

There's no reason to assume that files having a different size would be identical because if they were identical, they would have the same size. Files with same size may be identical or not. Files with the same date are suspicious. Files that have bytes that differ and are at the same position of the file are not identical.

5 hours ago, PianoPlayer88Key said:

Also I'm pretty sure there are programs out there that will find duplicate or near-duplicate files ... but what I was looking for is something that will find multiple duplicate folders.

I guess the programs that deduplicate btrfs would have to be like that.

Nothing prevents you from making a check sum of all the check sums of all the files in a directory and from comparing those check sums of check sums.

5 hours ago, PianoPlayer88Key said:

One bad habit I had gotten to, when copying stuff from an SD card either from my phone when I had one that used it, or from my camera or my audio recorder (Zoom H2/H2n or others), was just dumping the entire contents of the card onto the PC (cause I was too lazy to only choose the portions that I had recorded new things), then leaving stuff on the card cause I still wanted it accessible in-device, then again copying the entire thing another time later to another folder on the PC. (There were some occasions where I only copied the new stuff and deleted the old stuff off the card, but not always.) So, I've got a bunch of duplicate folders scattered all over the place.

That's a bad habit. Rsync is your friend.

5 hours ago, PianoPlayer88Key said:

Also I have duplicates of older "My Documents" folders, maybe even a few old Windows installs, etc, although on the Windows installs, I'm thinking I should get rid of the ones for which I no longer have the hardware that it was being used on. Problem is, I can't just delete some of them, as back in the day (late 1990s into mid 2000s or so) there were sometimes user / data files that somehow got saved in the Windows folder, or in the program files folder, instead of in the user area (like users\username\(etc), or in my case on a different drive entirely). (Also as for the program files, sometimes I've wanted to be able to run some older programs again, but I'd either need to find installers or find some way to get them to work, and I'd at least want to know which ones they were, and preferably if possible preserve the settings that I had used in the past, but that's outside the scope of this post, and I have too many other things going on to really take any time to pursue that for now.)

That's why you need a file system that deduplicates. When you have it all deduplicated automatically, sorting it all out may even be worse than leaving it as it is.

PianoPlayer88Key · November 12, 2022

Okay so this is super annoying ... I've been editing this (in the drafts section) and at least a few times now I've gotten most of the way through, then made a mistake and hit Ctrl+Z to undo ... and it didn't just undo that last little mistake I did, it undid almost EVERYTHING I'd been working on.

Also some of my replies to various points were getting pretty long. I left them in spoilers in case you want to see more details, but I put some condensed comments outside the spoilers. (There may be a couple other things in spoilers that are pointed out, those were originally going to be in spoilers anyway even if I didn't put the other things in spoilers.)

For now, re: the original post topic ... I think I'll get a couple 18TB WD Ultrastar HC550 (0F38459 / WUH721818ALE6L4) drives, they're available from B&H for $320 each. (I was also considering 20TB WD Ultrastar HC560 (0F38785 / WUH722020BLE6L4) for $380 each from Newegg, but that's a pretty big jump in price for just an extra 2 TB, so I think I'll go for the 18TB drives.) I just need to order soon, as they do Sabbath there (they're Jewish) and their checkout is closed from Friday sunset to Saturday sunset NYC time.

Update: Ordered 2x 18TB 0F38459 from B&H earlier today (a few hours before sunset in NYC), hopefully they'll be here next Wednesday (16th).

On 11/6/2022 at 3:28 PM, heimdali said:

Yes, that's called fake raid. When you use that and your mainboard fails, you will have to find one that can read the disks, so using fake raid isn't an option.

Ahh. (And I'm guessing real raid would be with a hardware card like an LSI 9211-8i or similar? I've thought in the past about getting one, or a 9200-16e or, if I found a good deal on one, a Highpoint Rocket 750 so I could add more drives, but then my problems would be not enough SATA connectors on my Corsair RM850 PSU or 3.5" bays in my Fractal Define R5 case.)

On 11/6/2022 at 3:28 PM, heimdali said:

It's also not an option to store data on only a single disk. When the disk fails, you may loose your data, and it's a big hassle.

Yeah... I plan to buy two disks of the same size, one will be for the data and the other will be for the backup. But, I'm a bit wary of "fake raid" so I guess I won't be running RAID1, which, among other things, eliminates Seagate drives from my shortlist. I'd want some kind of setup that is platform, OS, cotnroller, etc. agnostic, so no matter what I plug it into, I'd be able to read it, unless of course the specific drive itself had died.

On 11/6/2022 at 3:28 PM, heimdali said:

You really need to use redundancy for your active disks. I also recommend it for backups, but when one of the backups fails, you can make a new one and it's not too much hassle since you have to make backups anyway if you have at least the minimum of 2 generations of backups. Hard disks are anything but cheap If you have the disks, there's no reason not to use redundancy for backups as well.

Ah, would redundancy be RAID1, or something else? (Also my system as-is only supports 8 3.5" drives, which I'd prefer to all be available for storage.)
"HDDs aren't cheap"... true I remember when tape was a tiny fraction of the cost per capacity compared to HDDs, for example in 1994. I was wanting to see a backup solution today that has a similar price ratio. (See the spoiler for more details, but you would be able to get a 16TB tape for ~$21, or a 128TB tape for ~$63, and the tape drive would be about 1/2 to 2/3 the cost of a similar-size HDD.)

Spoiler

Hmm ... "redunancy for active disks" ... wouldn't that require RAID1, or is there another way? (And while my case+mobo+PSU does support 8 3.5" devices, I'd prefer them to be all available for storage and not have to lose some to redunancy, etc, and even then I'd probably be running out of bays/ports. Also what about some of the pitfalls of, say, fake raid....)

"Hard disks are anything but cheap" Tell me about it Actually I was looking at the cost of backup media vs HDDs in an old magazine ad I have from Q1 1994, and... A 250MB HDD was $215, 215MB tape drive was $145, and 215MB tape media was $20. A 2.1GB HDD (SCSI, though), was $1850, a 2.3GB tape drive was $925, and 2GB tape media was $59.
Today, there's a 16TB Toshiba MG08 HDD for $230 from a 3rd-party Amazon seller.
Calculating similar ratios, etc, if tape backup was available today like that, I'd be able to get a 16TB tape drive for $155.12 and a 16TB tape media for $21.40. Or, there would be a 134.4 TB SAS (that's the successor to SCSI, right?) HDD for $1970.07, a 147.2 TB tape drive for $989.53, and 128 TB tape media for $63.12. (It would be nice to be able to have backup media for which I could fit several HDDs worth onto a single backup.)
Yes, I know tape is very slow for random access (I imagine it basically making a HDD look like an NVMe SSD), but it would primarily be for full backups and restores. (And one idea I've had would be to have some kind of secondary SATA SSD or flash drive or something that just stores the data of where things are on the tape, and maybe short previews of things, for example the first couple pages of text files, several second short clips at 24 kbps 11kHz mono for audio or 240p 15fps q=28 or so for video, and you'd use that to find what you're looking for, queue up everything you wanted to restore, THEN hit "go" and it would arrange the queue sequentially then go through and pull everything off the tape. On the other end of the slow vs fast spectrum ... I wonder if we'll ever see HDD duplicators that can duplicate HDDs as fast as commercial optical disk stamping machines can duplicate CDs/DVDs/Blu-Ray...)

On 11/6/2022 at 3:28 PM, heimdali said:

I have to order everthing online; there's no place around that would have anything I would use on stock. Also, they removed all parking, so the only thing you can still get is groceries from the supermarkets that have parking. At the same time, they're complaining that the cities are empty ... What do they expect ... It's funny, a while ago cities used walls to prevent ppl from going there; nowadays they have all parking removed and nobody even wants to go there anymore.

Ahh. Well my local Fry's went out, and now pretty much all we have is Best Buy and WalMart. If I want an actual computer store I have to go to Micro Center about an hour and a half away, and I usually only go there if I'm planning to get multiple things, or if I'm going to be in the area for something else anyway. (At least it's on the side of Los Angeles area closer to me, so I don't have to deal with as much traffic as if I had to completely go through L.A.)

On 11/6/2022 at 3:28 PM, heimdali said:

Besides, I remember times when there were interesting stores in town with stuff I might actually buy. Those started disappearing when they started to force customers to walk for miles by closing the roads and removing parking, and since decades, there's no reason to go there anymore. If I couldn't order online, it would be like in the stone age ...

I remember those times too, although in my case I think it was all the online ordering and things that eventually shut them down. But, there were times when my brother said you could stand on some street corners in the area where most of those stores were, start throwing stones, and probably hit like 8 or 10 computer stores.

In the spoiler are some links to photo albums of old magazine ads from various stores we had in the San Diego, CA area, from 1990 to 2007. (I have quite a few more magazines still in my paper-and-ink collection, but am missing a few; I remember having some as old as 1987.)

Spoiler

The cat on the laptop pic at the end of some albums is just a placeholder, or to remind myself that I've reached the end of the album.

ComputorEdge - 1990-12-07
ComputorEdge - 1993-03-19
ComputorEdge - 1994-01-07 (this one has the HDD vs tape ad near the end.)
ComputorEdge - 1995-01-13
ComputorEdge - 1995-11-24
ComputorEdge - 1997-02-07
ComputorEdge - 1998-01-16
ComputorEdge - 1999-01-01
ComputorEdge - 2000-01-07
ComputorEdge - 2001-03-09
ComputorEdge - 2002-03-08
ComputorEdge - 2003-03-14
ComputorEdge - 2004-08-13
ComputorEdge - 2005-09-02
ComputorEdge - 2006-09-15
ComputorEdge - 2007-09-21

ComputorEdge - assorted) (doesn't quite fit into the other categories, etc)
Articles - ComputorEdge
Articles - Computer Resource (This was a competing magazine for a while. Also there was a "Computer Link" that was an insert in the local San Diego Union Tribune paper for a while, but idk if I still have any of those.)

Also I'll include an album of purchase invoices we've saved. (I removed personal info before posting.) This includes the first PC my dad ever bought, a 286-10, 640k RAM, EGA graphics, 40MB HDD, etc, for around $1800 in January 1989. I cut my computing teeth on that PC at home, although maybe a year or two (not before Q3-1986 cause that's when I started kindergarten / school) before that my parents would take me to a home education center where they had Apple computers, not sure what model they would have been though.

On 11/6/2022 at 3:28 PM, heimdali said:

I have no idea ... it didn't occur to me that anyone wouldn't order them online but actually go to all the lengths of going to a store. ... I'm really surprised that you would go to a store for some hard disks. Your overall situation must be way better than it's here that you can afford to waste the fuel on that.

Ah. As I said / hinted earlier, I typically order online, but might go to a store in certain situations. (I used to only buy at stores though, cause I was afraid of damage in shipping, but I guess me driving home with the HDDs isn't that big of a difference or might even be worse considering i'm not exactly a slow driver.)

On 11/6/2022 at 3:28 PM, heimdali said:

I might be a good idea to do that.

I've thought about it, but right now I don't have any physical place to put a server, never mind any kind of rack cabinet or anything like that.

(Also when I was considering building a NAS / backup server, one of my criteria was having the entire cost of the setup, not counting the storage, be less than the cost of a single HDD.)

On 11/6/2022 at 3:28 PM, heimdali said:

You have way more capacity then I do and kept your data since the 80ies, so of course you do qualify.

I thought I had to have never deleted anything, AND have more storage drives / capacity than a mere mortal's peasant dual-socket 11-PCIE-slot-filled-with-HBAs motherboard could support in order to be a "data hoarder"

Spoiler

I thought to qualify for "data hoarder" I had to have never deleted anything, ever (I have deleted some things), and I had to have more capacity than any system available for purchase (either complete system, or parts) could support.
(For example, let's say I got one of those dual-socket Supermicro motherboards with 11 PCIe slots, and put a Highpoint Rocket 750 in each slot. Each of those cards supports 40 drives, times 11 is 440 drives, times 22 TB per drive (I'm ignoring the WD 26TB host-managed SMR drive) would be 9.68 PB on that system. That doesn't account for on-motherboard ports, or using bifurcation / splitters to, for example, plug 8 PCIe 2.0 x8 cards into a single PCIe 4.0 x16 slot, or whatever would be the eqiuvalent bandwidth. A data hoarder, I thought, would have more than that )

On 11/6/2022 at 3:28 PM, heimdali said:

It is out of the question to store data on only a single disk without redundancy. Backups are not a substitute for redundancy, and redundancy is not a substitute for backups. Redundancy requires to buy disks at least in pairs; how else would you have redundancy?

Ah, I do plan to buy two disks, but idk how I'd set up real-time redunancy if I won't be running somehting like RAID1. (What I've done so far is have one disk of each set of 2 be my data, then the 2nd has stuff manually copied to it then it's unplugged for a while until I need to update or restore, and I'm way behind on updating.)

On 11/6/2022 at 3:28 PM, heimdali said:

Using a warranty usually would require that you return your disk to someone. There is no way that I would give my data out of hands. Other than that, it doesn't matter here, there are legal requirements so you do have a warranty unless you buy from a private seller who has taken precautions. I don't know how it's in California.

True ... but how else would I deal with a failed drive if I wanted to replace it without having to buy another one. Yes, there's the issue of the data on it, but that's what backups are for. (Backups, however, won't save you the cost of a new HDD when you need to replace one that died prematurely, afaik.)
I'm pretty sure warranties don't apply buying used parts from private sellers. Also I've heard that they may not apply when buying refurb parts from non-authorised sellers (in those cases the sellers are supposed to provide the warranty, same if you bought a complete system with a drive inside it from a retailer).

On 11/6/2022 at 3:28 PM, heimdali said:

What I meant is that if I were to buy 16TB hard drives, I'd have to buy at least two of them and that would cost me over EUR 1000. Discs always come at least in pairs.

Ahh, one for data, one for backup basically. "Discs always come at least in pairs" ... maybe it translates to English a bit different han whatever your native language is, but to me, what I imagine is that it's not possible to buy one disk at a time. In reality though, at least here, you CAN buy one disk at a time if you want to, but it's much more advisable to buy two.

On 11/6/2022 at 3:28 PM, heimdali said:

You can always check with hdparm. Especially for the so-called green ones I'd expect them to go to sleep all the time because the idea of the so-called green ones is saving energy.

Ahh... but ...

C:\Windows\system32>hdparm
'hdparm' is not recognized as an internal or external command,
operable program or batch file.

C:\Windows\system32>

Apparently hdparm isn't on here, and idk if it's available. (Also I don't think the drives I'm running now, or ones I plan to get, are the "green" ones, afaik.)

On 11/6/2022 at 3:28 PM, heimdali said:

Perhaps you have multiple issues, and it's possible that you need to update the firmware of your disks.

I wonder if other things could possible be causing issues. I just hope I don't have to rip my entire system apart, I really don't want to do that until I upgrade to a new socket / RAM generation and I might skip DDR5 entirely or at minimum go to something that supports RDIMMs and 128 PCIe lanes if/when I could afford it, which won't be for a while.

Spoiler

Maybe; although idk if firmware updates are available or not, haven't checked (and I've never updated firmware on a HDD anyway that I can remember).

I'm not 100% sure if it's the disks, cause other things also act up around the same time. (I hope it's not something related to the CPU not quite properly being seated in the socket, or some short between the motherboard and case or something, but I wonder if it could be .... I'd think there'd be a lot more issues, and a lot more frequent, if that was the case though. And I really don't want to rip my system apart until I'm doing a significant platform upgrade, like SP5, or even a DDR6 or DDR7 version of that. (Going from AM4 to AM5 or LGA 1700 is too small of a bump for me.) While I'm sure some people can take apart and rebuild an entire system in maybe a half hour or so, it takes me probably a full day, and don't even get me started on configuring my OS, software, etc. the way I want it, or the way I had it before.)

On 11/6/2022 at 3:28 PM, heimdali said:

So is there a firmware update for those, or is there anything else special about them? Maybe broken cables?

I'm not sure about firmware updates. As for broken cables, I believe the ones I'm using are fine, although I also have some other SATA cables that have been slightly modified with the aid of my cat's teeth, one had its retention clip broken off, etc.

Spoiler

As for broken cables, I believe the cables I'm using now are fine, although I do have a couple of other cables that have teeth marks from my cat, and maybe even one or two SATA connectors on the end chewed up pretty good. (I don't think I'm using one of those right now, although it's possible I could be using one that originally had one of the spring clip retention mechanisms that had broken off, I don't remember though and don't feel like pulling my tower out of the desk cubbyhole just yet. I'll need to pull it out to swap the drives when I get them, but I'll be doing that then.)

On 11/6/2022 at 3:28 PM, heimdali said:

It depends ... Unfortunately, ZFS isn't compatible with Linux only for stupid licensing issues (you can get it to work, but that isn't a good option), so you'd have to go with FreeBSD, but the NFS implementation of FreeBSD sucks and makes that unusable for a file server. Then you may not be too happy with the performance of ZFS, unless perhaps you start tweaking it and using SSD cache. Tweaking it entails the risk of messing things up and/or loosing data --- or maybe not but it's a hassle. Also, the SSD is prone to wearing out when you use it for cache. And/or you're going to need lots of RAM, but you can't exacly cache in RAM because when the power goes out, the contents of the RAM are lost. So ZFS has good features but I'm not happy with it.

And I have to kinda correct my last post. Btrfs doesn't really support deduplication in that you can deduplicate it only afterwards and not while writing. I thought it could do that. That makes the possibility of deduplication pretty useless for backups because it requires that all the data fits on the volume, and if it fits anyway, there's no need to deduplicate it. For non-backups, I wouldn't trust it enough not to mess up my data. I was really disappointed.

But then, there's something that's called VDO. Apparently you create a VDO volume and put any file system on it and VDO can deduplicate it. I need to learn more about it, it would be great if I could use that. See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_storage/deploying-vdo_deduplicating-and-compressing-storage#doc-wrapper for more.

I've never used ZFS. Had considered it for NAS/backup server, but would lean more toward UNRAID or similar. (I like how it lets you add drives one at a time, and if > parity fails you only lose the data on the extra failed drives, not the entire array. Single-drive performance is good enough, if I need more I'll use an SSD.) Also I'm not familiar with BTRFS or VDO either.

Dedup/backup ... A while ago I was wanting to back up a bunch of SSDs to a 12TB HDD. If I didn't use compression I could just "dd" to a disk image and they'd still all fit for now, but... (see in the spoiler).

Spoiler

Ahh, I have no experience with ZFS. I had been thinking of using it for a NAS / backup server, but have been leaning more toward something like UNRAID if I was to spin one up. (Something that appeals to me with UNRAID is that you can add disks one at a time, and if you lose more disks than you have parity / redundancy, you only lose the data on those disks, not the entire array. I don't need the extra performance - single-drive performance is good enough for me, if I need more performance for something I'll just use an SSD.)

Also I'm not familiiar at all with BTRFS, or with VDO.

Deduplication....backups ... reminds me of something I was trying to do a while ago, but for now have hit the pause button.
I bought a 12TB Toshiba MG07ACA12TE drive specifically for the purpose of backing up the SSDs I have. All of my current SSDs (although I'm thinking of buying a couple more maybe) will fit on those 12TB drives without compression.
Anyway, I could "dd" (in Linux) the SSDs to image files on the HDD, and if I do that (with no compression like gzip or whatever), I can easily open the image files and see the file / folder structure within.
BUT ... when I tried to "dd" AND gzip the "dd" image (or something, it's been several months or a year and I forget now how I did it), then try to open the image, it insists on trying to decompress the ENTIRE thing to RAM or whatever. I was doing my initial test on a 240GB SSD (which was about half full; the uncompressed "dd" image took up a full 240 GB whereas the compressed image (tar.gz, or .gz) is about 61 GB), but I also have a couple 2 TB SSDs I'd want to back up as well, preferably compressed, and I don't have 2+ TB of RAM (or want to take the time) to decompress the image file before I can access its contents.

On 11/6/2022 at 3:28 PM, heimdali said:

How well did that work? From what I've been seeing, I have a theory that these consumer mainboards aren't able to handle a bunch of disks connected to them very well. It's like the board is overwhelmed with all the data going through it when data is being written to or read from all the disks in the RAID at the same time. It just slows everything down. Some people claimed that newer boards would handle better, but I doubt it. The data still needs to get through somehow.

Haha Not well A couple years ago or so I was DBANning probably 14 HDDs simultaneously, and ... the total bandwidth was 1 GB/s, and a couple drives that on their own were capable of >200 MB/s transfer were throttling to ~40-50MB/s transfer. (No SSDs were involved but I've heard DBAN isn't advised for them anyway.)

Spoiler

HAH .... not well ... The board I was using at the time was an ASRock Z97 Extreme6.
There was a time I had possibly 14 drives plugged in, as I was DBANning a bunch of older drives in preparation for donating them to a local recycler / charity (which gives refurbished PCs, etc, to less fortunate people, for example). The largest drives I had plugged in were 5TB HGST Deskstar NAS drives, and the smallest were probably some double-digit GB PATA drives.
Anyway, the 5TB and 4TB drives were capable on their own of sustaining around 200+ MB/s sequential transfer, but when I was running DBAN on them, some were throttling to around 40-50 MB/s or so. I noticed the total bandwidth running DBAN on all 14 drives was 1 GB/s, which happens to be the bandwidth of the DMI (if I remember the term) between the CPU and the motherboard expansion / ports, not counting the primary graphics. (I think it was either PCIe 3.0 x2 or 2.0 x4, I forget, which is I think 1 GB/s.)
As for how I had 14 drives connected ... 10 of them were plugged into the motherboard's built-in SATA ports, then 2 more SATA drives (< 2 TB) were plugged into a ByteCC / JMicron BT-PESAPA card, and 2 PATA drives were plugged into that card's PATA port. (I also had a Promise Ultra100 TX2 controller, with an extra 2 IDE ports for 4 IDE drives, but it's PCI and that board didn't have any legacy PCI slots.)
I imagine my ASRock B550 Taichi might have a bit more DMI (or whatever it's called) bandwidth, maybe PCIe 4.0 x4 or 3.0 x8 but I forget, so it might do better with HDDs connected. (One factor that made me choose it was the factor that I can have all 8 SATA ports and both M.2 slots populated simultaneously without having ports disabled. I think it might cut one of the PCIe slots bandwidth in half if I have everything populated, but idk if I'll use that slot unless I plug in at least two SAS/SATA HBAs.)

On 11/6/2022 at 3:28 PM, heimdali said:

Older cards were limited to 2TB. That was a hardware limitation. In their time, disks weren't larger than that, so the cards didn't need to be able to handle them.

Ahh, I guess the BT-PESAPA card I had (actually still do, but only use it rarely for PATA) was limited to tthat, it's from around 2005 or so I think so maybe 2TB drives didn't exist then. (I also for a while had a Rosewill external 3.5" enclosure that I learned when buying a 3TB drive that could only format ~700 MB that it had the same limit.) Also several years ago I wanted the ability to pull individual drives out of a system to use with a 3.5" dock somewhere else, but I'm pretty sure USB flash drives and external SSDs would be better for that now.

Spoiler

Ahh, yeah I imagine the card I mentioned above definitely had that limit. Maybe it was due to its age, I think I've seen a copyright date of around 2003 or 2005 on my LGA1150 PC when booting it up. (I've wondered if it was a limitation of SATA II (3Gb/s) though... but I think I've heard of people plugging in >2TB drives into SATA II ports on other systems and it was fine, but idk.
Interesting thing though, the BT-PESAPA's boot spalsh text doesn't pop up on my AM4 board, and I can't see devices plugged into it from WIndows, but I can see them from Linux. Also in Windows on the LGA1150 board, the drives were visible but not SMART data; I had to boot Linux to see the SMART data.

I also, for a while, had an external Rosewill 3.5" HDD dock, forget the model number, but I learned the hard way it didn't support drives > 2 TB. (Not as "hard" as I learned to not plug a 5TB or 8TB drive into the BT-PESAPA, though.) I had bought a 3TB drive, then went to format it and it was only seeing something like 700 GB. I ended up having to return the drive and buy a 2TB drive instead. (At the time, I didn't have a working desktop PC at all, just my dad's Core 2 Duo laptop which we still have but he upgraded in 2018 to an 8th-gen Core i7 laptop, jumping from WIndows XP to 10.)

Also I mentioned earlier about liking UNRAID in that drives could be added one at a time. I've also wanted the ability to pull a single drive with things on it and take it to a friend's house for whatever, but I imagine that a USB drive or external SSD would these days be much better suited for the purpose. (Back when I was wanting to do that, though, you'd pay a couple hundred bucks for just a few tens of GB or single-digit GB or something like that, whereas a TB of HDD space might be around $100 or so but I forget. I could be off on the prices, but it was around 10-15 years ago or so I think.)

On 11/6/2022 at 3:28 PM, heimdali said:

The overcommitment doesn't really matter until the RAM isn't actually used.

Ah ... but I'm often (as in a few times a minute or more) getting lag spikes just using my PC even when I'm only using <10% CPU and ~50% RAM. (I could understand when they're maxed out, but not when resources are still availble...)

Spoiler

Ahh. Also something I haven't figured out is why my PC often gets lag spikes (sometimes small and barely noticeable like the one I just had where I typed 2 or 3 words before it appeared on screen, but sometimes also pretty big where half the system is unresponsive for a minute or two or more), even though I might only be using 5-10% of my CPU, and my RAM is only about half used. (I would have thought I shouldn't be getting input / processing lag spikes or whatever until my CPU and RAM were both maxed out.)

Also just now I was editing a portion of the draft for this post, and... I highlighted a selection of text, Ctrl+X'ed it, clicked the spot where I wanted to paste, Ctrl+V'ed, then clicked another spot to click and drag to highlight and delete some text .... but there was a delay from when I hit Ctrl+V until it actually pasted, and it pasted the text in the other spot where I was going to try to delete.

Side note, it's super annoying when websites either load elements that move things around on the page AFTER it has visually started to appear, or when, on a login page, it jumps your cursor to the username entry field. I've lost count of how many times I go to click one thing and I click something else instead cause it moved, or, I'm in the process of logging in and start typing my password in clear text in the username box.
I remember the days in the MS-DOS era when, while the PC was booting up, there was some kind of keyboard buffer or something so we could start entering commands and things before it had even finished booting. (Why a similar ability doesn't seem to carry forward to modern times escapes me.)

On 11/6/2022 at 3:28 PM, heimdali said:

It seems like you need hardware that's more reliable.

Maybe. I thought the stuff I had was supposed to be at least fairly decent, at least if I remember right reading reviews on it, there weren't a lot of complaints about DOA, failures, etc that I could remember.

I mentioned earlier I'm not considering Seagate HDDs - that's because of possible reliability issues I've heard from some people, including my brother and others. I wonder if the newer IronWolf Pro or Exos drives might be okay though, but I would only consider one if I was running it in a RAID1 array with a different brand drive, like a WD Ultrastar/Gold or a Toshiba MG-series drive. (As for that, see the comments earlier about "fake raid".)

On 11/6/2022 at 3:28 PM, heimdali said:

Perl is your friend ...

Ahh, I'm not at all familiar with perl, or pretty much any other scripting / programming language for that matter. (What little experience I had with basic, pascal, html, batch files, etc, 20-30 years ago or so has pretty much fallen by the wayside.)

On 11/6/2022 at 3:28 PM, heimdali said:

You can always use rsync. Some file systems support exporting themselves in some efficient way, too. I haven't tried that, maybe it's a good option. Rsync is more general and easy to use.

Ah. I actually used rsync recently, but for a different task.

Spoiler

Basically I have a bunch of folders & files in a section of a VM on my laptop, and I was wanting to go through them to decide what to keep vs delete.
I used rsync to clone the entire directory structure (but not the files) to another spot in the VM, then copied that clone to have 3 cloned directory structures. I named the parent trees with various names, like "00 processing", "01 maybe save" and "02 prob del".
Then I went through (won't go into detail how but it took several days or a week or more to work with about 20-30 GB or several tens or few hundreds of thousands of files & folders, IIRC) them and was moving stuff I thought I wanted to keep into "save" and stuff I was pretty sure I wanted to delete into "del". (Main reasons I didn't delete right away was I wanted to see how much disk space I was freeing up relative to how much I was keeping, and just in case I made a mistake with the initial processing and wanted to keep something I had segregated for deletion, before I actually deleted it.)

There were several spots where I noticed some duplicate folder structures. In most situations though, I didn't bother with running WinDirStat although I might have done it once; I just did a right-click+properties and compared number of files and bytes taken up, and in some cases I could just weed out duplicates that way. There were a few situations where there were several duplicates in a row, but then I noticed that one of the duplicates was considerably larger and had more files, and some of the smaller duplicates were missing those files, so I just kept the one larger duplicate for that one.)

There's still some files I want to do some work with paring down, in this case text files where I had kept chat logs of steam conversations quite a few years ago. (I hear notepad++ might be able to do something I want, which is find text (for example so and so is online/offline, you lost your steam connection, so and so is playing xyz game, etc) then mark the entire line, then go through and delete them, but it's across a bunch of files in quite a few different folders and I think N++ only works with one file or folder at a time. (Also there's some other text I want to delete, but in that case I think looking for particular words might not work, as sometimes I might have used different words to mean the same thing, or the same word could mean something completely different in a different context. Take swear filters for example, and look up s'thorpe problem (I left off part of the word, btw just because it starts with "s" it is not referring to certain brown stuff, just in case a swear filter here doesn't understand context) for an idea of what I'm talking about in regards to taking things out of context, or whatever. (Basically that's the name of an actual place I think in England, but people back in the 1990s that lived there were unable to register for accounts on AOL because it thought it was something else, among other things.) Also don't try typing the Latin phrase that starts with "sigma" and ends with "laude" in the LTT discord. (Also I won't say the phrase, but another one comes to mind that in an innocent context could refer to a swimming cat, or something like that. I wonder how long until we actually have swear filters that can properly understand context, etc...)

On 11/6/2022 at 3:28 PM, heimdali said:

And you still think you're not hoarding data? Stuffing more disks into your computer is not the way to go. It takes no more than a disk failing or a little virus hitting you, or, if you don't have an UPS, a power surge and part of or all of your data is gone. Do you have at least ECC RAM in your computer?

Hah, I talked about "not hoarding data" earlier ... "Stuffing more disks", I can't really do that, 8 is my max right now.

Spoiler

Heh well I've run out of drive bays and SATA ports, so stuffing more disks isn't an option at least right now. (I had thought of some idea of a small external case for extra HDDs with its own PSU, that I would connect the HDDs to a 9200-16e card with SAS/SATA breakout cables, but that hasn't happened.)
At this point though, I'd be replacing smaller disks with larger ones, preferably 3 or 4 small drives (or better yet all 8 drives that would fit in the case / connect to the motherboard) with one larger drive that holds the data on all the small ones, and costs the same as one of the small ones had cost when I had bought it a few years prior or whenever. (I had typed up more about my desire to replace a bunch of smaller drives with a single bigger one for the cost like I outlined, but I can't remember if I typed it somewhere else which tells me this post is getting out of hand lol, or if it was in one of my previous edits on this draft and I lost it when my Ctrl+Z completely messed things up.)
I don't have a working UPS now. We've had one in the past, but I think it wasn't working like 15 years ago or so, if I remember right. (Actually we still have it but it's buried in a corner somewhere with a bunch of stuff in the way so I can't get to it to see what make and model it is.)

Also I'm using 4x32GB Team Expert DDR4-3600 CL18 in my current AM4 system, but running at 3466 cause it's not quite stable at 3533 I think, and it won't boot at 3600. Last I checked, it wasn't ECC, and if I was going to get ECC RAM the same capacity, it would be quite at least twice as expensive (like $1K I think vs the ~$500-550 or so I paid for it), and it would be slower, maxing out at DDR4-3200 CL22. (But then I guess RAM speed doesn't matter if I'm running out of the stuff anyway and hitting pagefile...) I think my B550 Taichi + 5950X might support ECC but I'm not 100% sure. (pcpartpicker says it doesn't but I know that's not always accurate, I've heard other sources say AM4 in some situations does support ECC.)
If I had my way, ECC would be ubiquitous / universal, with even the lowest-end consumer / portable systems requiring it. (Also I'd require support for higher density ECC RDIMMs and LRDIMMs in all but the lowest-end Intel Hx10 / AMD Ax20 chipsets, that way you could put a TB of RAM in an AM4 or LGA1700 board, or maybe 2+ TB in a 4-slot DDR5 board like AM5 or the DDR5 version of LGA1700.)

As for "stuffing more disks into my computer"... if there was a way to actually safely mount all of them AND hook them up... (Also if I was using a low profile CPU cooler, like something that's not taller than the VRM heatsinks, then I'd have room for several more HDDs...)

This is how I might have wanted to connect extra drives if I used a SAS HBA with external ports, like tha 9200-16e...

My idea, if my previous Rosewill Thor X2 had its physical hard drive capacity maxed out, and I was using no GPU and a low-profile CPU cooler.

But, maybe it's better to have a dual-chamber case and put all the HDDs on the back side, (but idk of any that exists that's not rackmount), something like the mockup in the lower half of this pic...

Sometimes I've had to run drives outside my case as well...

I've wanted a compact NAS case with dimensions like this, that would support an ATX motherboard and like 10 HDDs...

On 11/6/2022 at 3:28 PM, heimdali said:

Why more than 512? I wouldn't use that much but it kinda makes sense to use sector size as orientation. You'd have to experiment with 1024 and 4096 or maybe even stripe size to see what's faster. You also need to consider that the database may be using the same disks you're reading from and you probably don't want to wait days on the results. If the first 512 bytes of two files are identical, you would compare them no further than the first byte that differs.

You could make it so that your result would say to what percentage files are identical. But how do you measure that? How identical are two files when all their bytes are the same but they're in a different order? You wanna sort them before comparing?

There's no reason to assume that files having a different size would be identical because if they were identical, they would have the same size. Files with same size may be identical or not. Files with the same date are suspicious. Files that have bytes that differ and are at the same position of the file are not identical.

...

Okay now I see the "compare no further than the first byte that differs"... that's a good idea probably. (It's possible that in some cases I might want to compare some of those further, but I'd probably deal with those situations if / when they arise.)

...

Also, some media files might have been en-/trans-coded with different settings, but are ultimately the same thing. In some cases I might have a bunch of different bitrates / codecs of the same thing, and I need to pare some of those down and only keep the max quality / uncompressed one (a "master" so to speak), and maybe a couple lesser bitrate ones probably.

Spoiler

Well I think it's possible that the first 512 or 4096 bytes or whatever might be identical, but somewhere later in the file might not be.
Also I don't really need this to be super fast, as in, finish the entire job in like a half hour or an hour or so. I'd be totally fine if it takes several hours or even a full day or two or three to process. I can just use my laptop in the meantime while my desktop is churning through the data.

As for where to put the database... I do have about 650 GB of space free on my NVMe boot SSD, maybe the database could hopefully fit in a few hundred GB of that, leaving the HDDs free for doing the main operation of going through their data, etc.

(moved out of spoiler, see above)

As for files of different sizes being identical - of course they wouldn't be "identical" ... but they could have been taken from the same audio, photo or video source, and just be a different resolution or bit rate of the same thing. For example, one might be an uncompressed CD-quality wave file, while another might be a 16kbps 11kHz mono mp3 file; or, one might be a high-resolution raw image from a camera, or a BMP, while another is sized down to 640x480 and either lower-quality jpg, or gif, or lower bit depth or something; or, there could be a 4K 100Mbps H.264 file from my Panasonic FZ1000 camera, and a 720p mjpeg or something else of the same video that I had done some transcoding tests, and so on.

On 11/6/2022 at 3:28 PM, heimdali said:

I guess the programs that deduplicate btrfs would have to be like that.

Nothing prevents you from making a check sum of all the check sums of all the files in a directory and from comparing those check sums of check sums.

Ahh, checksums of checksums... but in my case I also have a bunch of partially duplicate folders, some maybe only 2 or 3 levels deep, some well into the double digits.

Spoiler

checksums of checksums ... maybe I should look more into that. Thing is which directories get the checksums checked. (Try saying that 5 times fast lol)
And there's the issue of the structure, for example, E:\a001\b001\c001\d001 might have a similar structure to H:\a003\b003\c003\d003\e003\f003\g003\h003\i003\j003\k003\l003\m003 and so on, and yes I'm pretty sure I have some stuff drillled that deep. (too bad there's multiple reasons why I can't just boot Linux, do cd /, then use a terminal command to recursively list everything then save & upload the resulting txt )

quick note, speaking of backups... the idea of something like "dd if=// ... " comes to mind, but if I'm backing up the ENTIRE filesystem including connected and network devices, problem is what would I back it up TO, if I'm already specifying to back up not just everything directly connected to that PC, but also everything on the network... (Also in my book, if I'm not including every single bit / sector including boot sectors, landing zones or whatever, normally inaccessible areas, firmware, etc, it's not a complete backup.)

(I'll admit that the thought of just deleting everything has come to my mind more than zero times, but I don't really want to exercise a nuclear option as there are some things I want or need to save. )

Also I'm maybe considering getting a couple 2TB or 4TB SSDs (one SATA 2.5", one NVMe M.2) for my laptop, to replace a couple 1TB SSDs so I have more space. (My other 2 bays are already occupied with 2 TB SSDs.) I'm just a bit hesitant to pay that much, for example, for a 4TB; I'd prefer it be closer to what I paid for the 1TB a few years ago but prices aren't coming down as quickly as I'd like. (I miss the days in the 1990s or so when HDD capacity would jump by >2x for the same price in about a year or less.)

Spoiler

Also ... I've been thinking a bit recently about getting a couple more SSDs for my laptop, either 2TB or 4TB to replace a couple 1TB SSDs I have now, but I'm not sure if I'll do that yet or not. (We might be getting kinda close on the 2TB drives being close in price to what I paid for a 1TB, but we're not there yet on 4TB drives being the same cost, afaik.)
I paid about $105 for a 1TB 2.5" WD Blue 3D in August 2020 (my laptop's current Windows 10 boot drive), and $228 for a 1TB NVMe Samsung 970 Evo in November 2018, those are the drives I'm looking at replacing. (No they're not failing, I just want more space, and all 4 SSD slots are filled - the other 2 slots are occupied by a 2TB 2.5" Seagate Barracuda 120 SSD and a 2TB NVMe Silicon Power P34A80.)

Looking at pcpartpicker prices.... I see a 2TB NVMe Mushkin Pilot-E for $138, a 2TB NVMe Silicon Power P34A80 for $145 (I already have one, plus a 1TB that's my Windows 10 boot drive in my desktop; Linux in the desktop is served by a 250GB SATA M.2-2260 Crucial MX200 which had previously been my Windows boot drive in my laptop but I "dd"-ed it to the 1TB WD Blue but now I can't update Windows past 1909 cause it thinks I don't have enough disk space for some reason even though it has ~600+ GB free, and I REALLY don't want to reinstall, hence why I'm still booting from SATA and not NVMe in the laptop cause I couldn't figure out how to migrate an existing install that way)), a 2TB 2.5" WD Blue 3D for $168, and a 2TB 2.5" Crucial MX500 for $172. As for 4TB drives, I see a 4TB 2.5" Samsung 870 QVO (probably the only QLC drive I would dare to maybe consider; also I won't consider any DRAMless drives) for $330, a 4TB 2.5" Crucial MX500 for $345, a 4TB 2.5" WD Blue 3D for $348, or a 4TB NVMe Team MP34 or ADATA SX8100 for $350. An 8TB SSD would be nice, but the cheapest is > $700 or so and it's QLC.

I've also wanted to compare the speed of some old HDDs vs current ones and SSDs -- but not in terms of MB/s transfer rates, I'm thinking more like "How long does it take to fill the entire disk if you're constantly writing to it".
I wanted to buy a couple old <80MB PATA HDDs on ebay, but those are crazy expensive, so maybe someone has some laying around and I should make another thread asking if someone has a way to do that. (I'm thinking that some of the older/smaller drives, like 40MB or 20MB, or if anyone has 10MB or 5MB MFM drives laying around and a working system capable of interfacing with them, could fill the entire drive in maybe half a minute to a few minutes or so, not the day plus that a modern 20+TB HDD would take, or the half hour or hour or longer that I guess a DRAMless 8TB+ QLC SSD might take to fill...)

Spoiler

Also another thing I've thought about experimenting with was checking transfer rates - not in MB/s, but in time to write & read the entire drive, of some older drives, but maybe that's better suited for another thread. Anyway in a nutshell, I had thought of getting a few 40MB or 80MB or so IDE drives on ebay, but when I looked at them, those things are crazy expensive per MB, compared to the $/MB on modern HDDs. Basically though, I think those old drives were "faster" in writing / reading the entire drive - there's an article several years ago on Tom's Hardware (something about 15 years, capacities outran performance or something like that) where you could basically read / write to an entire 40 MB IDE drive in like 2 or 3 minutes. But, it takes probably a full day to read/write to an 8TB or 10TB drive, and I imagine an 18TB or 20TB drive isn't any faster.

Maybe instead of me buying the drives, maybe someone else already has a system with drives like that, or better yet, something that supports MFM drives and has some of the early 5 or 10 MB drives. I wonder how quickly those could be written or read... I hear the ST-506 MFM interface supported up to 5 Mb/s, not sure if that translates to 500 KB/s (10-bit, overhead, etc) or 625 KB/s (straight-up 8-bit), and I realize that they may not have hit that full speed, but if they did, a 5 MB drive could theoretically be written or read in like 8 or 10 seconds. How long until we have modern hard drives or SSDs that are even faster and are NOT expensive... closest concept I can think of (which afaik doesn't exist for HDDs yet) is the comparison to the optical disk stamping I talked about earlier.

heimdali · November 12, 2022

> Okay so this is super annoying ... I've been editing this (in the
> drafts section) and at least a few times now I've gotten most of the
> way through, then made a mistake and hit Ctrl+Z to undo ... and it
> didn't just undo that last little mistake I did, it undid almost
> EVERYTHING I'd been working on. 

Yeah I know how that goes.  The forum needs good emacs integration.
Mail-mode is your friend.  I think I already said quite a while ago
that it needs a mailing list ...

> Also some of my replies to various points were getting pretty long.
> I left them in spoilers in case you want to see more details, but I

Spoilers?

> put some condensed comments outside the spoilers.  (There may be a
> couple other things in spoilers that are pointed out, those were
> originally going to be in spoilers anyway even if I didn't put the
> other things in spoilers.)

Huh?

There's no way I could edit this on the web site.  I've copied it into
emacs so I can edit it in mail-mode, and I guess it'll best to paste
it later as code.

In case it becomes difficult to reply to, let me know and I'll edit
the post and paste it as not-code.

> For now, re: the original post topic ... I think I'll get a couple
> 18TB WD Ultrastar HC550 (0F38459 / WUH721818ALE6L4) drives, they're
> available from B&H for $320 each.

Ah that's what I thought last night when I was looking out for early
black friday offers.

> (I was also considering 20TB WD Ultrastar HC560 (0F38785 /
> WUH722020BLE6L4) for $380 each from Newegg, but that's a pretty big
> jump in price for just an extra 2 TB, so I think I'll go for the
> 18TB drives.)  I just need to order soon, as they do Sabbath there
> (they're Jewish) and their checkout is closed from Friday sunset to
> Saturday sunset NYC time.

Hm, I found a seller that apparently has newer releases of the same
drive, and the newer ones cost like 150 more.  So it's all relative.

> Update: Ordered 2x 18TB 0F38459 from B&H earlier today (a few hours
> before sunset in NYC), hopefully they'll be here next Wednesday
> (16th).

nice :)

> > Yes, that's called fake raid.  When you use that and your
> > mainboard fails, you will have to find one that can read the
> > disks, so using fake raid isn't an option.
>
> Ahh.  (And I'm guessing real raid would be with a hardware card like
> an LSI 9211-8i or similar?

Something like that, I guess.  Hardware RAID should have it's own
cache and a backup battery.

> I've thought in the past about getting one, or a 9200-16e or, if I
> found a good deal on one, a Highpoint Rocket 750 so I could add more
> drives, but then my problems would be not enough SATA connectors on
> my Corsair RM850 PSU or 3.5" bays in my Fractal Define R5 case.)

That's one of the reasons why you need a case with drive bays and a
backplane.  I have one of these wonderful Chenbro cases with 16 bays.

> > It's also not an option to store data on only a single disk.  When
> > the disk fails, you may loose your data, and it's a big hassle.
>
> Yeah... I plan to buy two disks of the same size, one will be for
> the data and the other will be for the backup.

Why don't you use RAID1 with these disks and do something else for
backups?  When you don't, you'll have the big hassle when a disk
fails, and especially new disks can fail.  If they run for 3--4
months, they should be fine; after that, they fail not long after the
warranty has expired and if not, they're likely to last until they're
being replaced by disks with more capacity, with some exceptions.

> But, I'm a bit wary of "fake raid" so I guess I won't be running
> RAID1, which, among other things, eliminates Seagate drives from my
> shortlist.

Do you think Seagate disks would fail even sooner then they usually do
when used in RAID arrays?

> I'd want some kind of setup that is platform, OS, cotnroller,
> etc. agnostic, so no matter what I plug it into, I'd be able to read
> it, unless of course the specific drive itself had died.

There's no such thing.  The very disks themselves do have either a
SATA or a SAS interface (unless you use fiber, but I don't know about
that and it's still an interface) and without that, you can't access
them.  They come with a controller and firmware without which you
can't access them.  The list goes on from there ...

What you can do is use hardware that is easy to replace, or you keep
spares.  Like I have two RAID hardware cards in my backup server
because each can only handle 2x4 disks.  These controllers have been
somewhat abundant and are still readily available on ebay starting at
$30 for the basic card.  They are also somewhat compatible to other
models of the same line, so if I need to replace one, chances are that
even a different model of controller could access the disks.  Then,
it's somehwat unlikely that both would fail at the same time, so if I
had to, I could use the controller that still works.  It's brilliant
because I have a spare without actually needing one ;)

If the backplane fails, I can get mini-SAS to SATA cables and connect
the disks without the backplane.

Thing is you need to use decent hardware and not mess around with
crappy stuff, like stuffing a bunch of hard disks into a case that
doesn't even have bays with trays and no backplane.  You can do it
with like 4 disks, but beyond that, it becomes a mess of cables, tends
to be difficult to cool and just sucks.

Decent hardware makes live so much easier, and live is way too short
for bad hardware.  Seeing your pictures, you need to get a rack and
some nice 19" cases ...

> > You really need to use redundancy for your active disks.  I also
> > recommend it for backups, but when one of the backups fails, you
> > can make a new one and it's not too much hassle since you have to
> > make backups anyway if you have at least the minimum of 2
> > generations of backups.
>
> Ah, would redundancy be RAID1, or something else?  (Also my system
> as-is only supports 8 3.5" drives, which I'd prefer to all be
> available for storage.)

Yes, there are lots of RAID levels that provide redundancy.  ZFS is
doing it's own variants in software.

> > Hard disks are anything but cheap  If you have the disks,
> > there's no reason not to use redundancy for backups as well.
>
> "HDDs aren't cheap"... true  I remember when tape was a tiny
> fraction of the cost per capacity compared to HDDs, for example in
> 1994.  I was wanting to see a backup solution today that has a
> similar price ratio.  (See the spoiler for more details, but you
> would be able to get a 16TB tape for ~$21, or a 128TB tape for ~$63,
> and the tape drive would be about 1/2 to 2/3 the cost of a
> similar-size HDD.)

How did you get tape drives that cheap?  Tapes were basically done
when the capacity of DAT drives couldn't keep up, and those were
expensive and were wearing out all the time.  The tapes weren't too
bad, though.  Anything better than a DAT drive cost a fortune.

I do wish we could just use tapes for backups still.


> Hmm ... "redunancy for active disks" ... wouldn't that require
> RAID1, or is there another way?

Sure, you can use different RAID levels.

> (And while my case+mobo+PSU does support 8 3.5" devices, I'd prefer
> them to be all available for storage and not have to lose some to
> redunancy, etc, and even then I'd probably be running out of
> bays/ports.

The disadvantages of loosing your data, of the hassle and of the
unwieldyness of having to deal with so many individual disks all the
time instead of being able to use a larger volume outweigh the cost
of, at the minimum, using a single disk for parity.

Look out for a decent 19" case with 16 bays (or more).

> Also what about some of the pitfalls of, say, fake raid....)

I've never used it.  What kind of reliability and performance do you
expect?  There's reasons why RAID controllers have backup batteries,
and what does fake raid have?


> "Hard disks are anything but cheap" Tell me about it  Actually I
> was looking at the cost of backup media vs HDDs in an old magazine
> ad I have from Q1 1994, and...  A 250MB HDD was $215, 215MB tape
> drive was $145, and 215MB tape media was $20.  A 2.1GB HDD (SCSI,
> though), was $1850, a 2.3GB tape drive was $925, and 2GB tape media
> was $59.

Yeah it's never been cheap.

> Today, there's a 16TB Toshiba MG08 HDD for $230 from a 3rd-party
> Amazon seller.

Are those any good?

> Calculating similar ratios, etc, if tape backup was available today
> like that, I'd be able to get a 16TB tape drive for $155.12 and a
> 16TB tape media for $21.40.  Or, there would be a 134.4 TB SAS
> (that's the successor to SCSI, right?) HDD for $1970.07, a 147.2 TB
> tape drive for $989.53, and 128 TB tape media for $63.12.  (It would
> be nice to be able to have backup media for which I could fit
> several HDDs worth onto a single backup.)

yeah

> Yes, I know tape is very slow for random access (I imagine it
> basically making a HDD look like an NVMe SSD), but it would

It takes a while to seek, but that's ok for backups.

> primarily be for full backups and restores.  (And one idea I've had
> would be to have some kind of secondary SATA SSD or flash drive or
> something that just stores the data of where things are on the tape,
> and maybe short previews of things, for example the first couple
> pages of text files, several second short clips at 24 kbps 11kHz
> mono for audio or 240p 15fps q=28 or so for video, and you'd use
> that to find what you're looking for, queue up everything you wanted
> to restore, THEN hit "go" and it would arrange the queue
> sequentially then go through and pull everything off the tape.

You still have to seek to where the file is, and that means you must
read the index on the tape or whatever the backup software used to
figure where to start reading some file.  The backup software had to
keep some information as to which tape to use and where the file might
start on it.  Tar works fine, it was made for that kind of thing :)

> On the other end of the slow vs fast spectrum ... I wonder if we'll
> ever see HDD duplicators that can duplicate HDDs as fast as
> commercial optical disk stamping machines can duplicate
> CDs/DVDs/Blu-Ray...)

That might not even be possible if we had replicators like they have
in Star Trek.

>> I have to order everthing online [...]
>
> Ahh.  Well my local Fry's went out, and now pretty much all we have
> is Best Buy and WalMart.  If I want an actual computer store I have
> to go to Micro Center about an hour and a half away, and I usually
> only go there if I'm planning to get multiple things, or if I'm
> going to be in the area for something else anyway.  (At least it's
> on the side of Los Angeles area closer to me, so I don't have to
> deal with as much traffic as if I had to completely go through L.A.)

It's so much easier to order online.  And the attendends in stores
tend to be complete idiots and I'm better off ordering online anyway
because they have no clue what they're doing.  I've had one standing
right in front of a wall display full of headsets/-phones I was
looking at, and when I asked for a headset I could use with a phone,
he said they don't have headsets.  I went home and ordered something
online instead.

> > Besides, I remember times when there were interesting stores in
> > town [...]
>
> I remember those times too, although in my case I think it was all
> the online ordering and things that eventually shut them down.

Oh they closed well before there was internet.

> But, there were times when my brother said you could stand on some
> street corners in the area where most of those stores were, start
> throwing stones, and probably hit like 8 or 10 computer stores.

There were never that many around.

> In the spoiler are some links to photo albums of old magazine ads
> from various stores we had in the San Diego, CA area, from 1990 to
> 2007.  (I have quite a few more magazines still in my paper-and-ink
> collection, but am missing a few; I remember having some as old as
> 1987.)

Cool, maybe scan them and make them available to museums or so so they
can be preserved for generations to come :)

IIRC, here you would find ads like that only in computer magazines,
and only a couple pages of them.

> The cat on the laptop pic at the end of some albums is just a
> placeholder, or to remind myself that I've reached the end of the
> album. [...]

After all so much reading, it's not surprising that it's tired :)

> Also I'll include an album of purchase invoices we've saved.  (I
> removed personal info before posting.)  This includes the first PC
> my dad ever bought, a 286-10, 640k RAM, EGA graphics, 40MB HDD, etc,
> for around $1800 in January 1989.

I looked at it and I thought "wow, what a luxury!" :)  If you wanted
to spend money like that here, you had to figure out where you could
buy something like that, and the store had to order it for you.  A
couple years later, you could find something in larger cities, but not
long after that, they worked harder on removing parking and the stores
closed.

> I cut my computing teeth on that PC at home, although maybe a year
> or two (not before Q3-1986 cause that's when I started kindergarten
> / school) before that my parents would take me to a home education
> center where they had Apple computers, not sure what model they
> would have been though.

That's amazing :)  Back then, there were only a few computers around
here, and they didn't really want anyone to use them because they were
expensive.

The thing is that back then, everyone kept saying you have to learn
about computers, especially when you want to get a job.  Nobody cared
and nobody learned except for a handful of people, so the computers
took over and the jobs were removed.

> > I have no idea ... it didn't occur to me that anyone wouldn't
> > order them online [...]
>
> Ah.  As I said / hinted earlier, I typically order online, but might
> go to a store in certain situations. [...]

Dunno, for a time, in some cases, it seemed advantageous to buy
something locally because you could go back and have them fix it if
necessary.  It became exceedingly difficult to do that, and it has
never been a real advantage to do that.

> (I used to only buy at stores though, cause I was afraid of damage
> in shipping, but I guess me driving home with the HDDs isn't that
> big of a difference or might even be worse considering i'm not
> exactly a slow driver.)

Yes, and if it happens, what's more annoying: Going back to the shop
and take the risk of them telling you you broke it yourself or
returning it?  Here we have the right to return things that we bought
without seeing the seller in person, like everything we ordered
through the phone or online, within 14 days after it arrived, and we
don't need any reason at all.  So if your hard disk arrives broken,
just return it and take the chance that they won't plug it in to see
if works :P It's probably more trouble telling them that it's
broken. And you can always say it worked before you returned it
... They might be unhappy with returned media, but if anything fails,
you can always withdraw the money through your bank account and put
them into the position that they would have to actually sue you, and
that's a pretty bad position for them.  But I never had to do that,
yet it's a last resort.

I'm picturing it:  You drive an hour to the store and drive around
another half hour to find a parking spot.  You walk a mile or two and
finally buy two disks.  You walk all the way back and drive all the
way back home and the day off you had is over.

After about a week, one of the drives fails.  You take another day off
and drive for an hour to the store and drive around to find a parking
spot.  After 3/4 hours you give up and drive back home (yeah I
actually had to do that and I basically quit going to movie theaters
mainly because of it even when there still were some worthwhile going
to, long time ago) and watch some movies for the rest of the day.

So one day, you finally make it to the store and they are willing to
return the disk for you and they'll call you when they got the
warranty case resolved in six weeks or so, and when that happens, you
can come back and pick up the disk. Maybe even that works out for you
somehow.

Three weeks later one of the disks fails again --- perhaps
not very likely, but you could have bought Seagate.  There you are,
more days to waste with driving back and forth, with trying to find
parking and more miles of walking.

Where's your buyer protection you could have through paypal buying
online when you're buying in a store?  Is the store going to pay you
for your time and for all the driving and are they going to somehow
give you the days you had to take off of work back?  With fuel prices
as insane as they are nowadays, anywhere you drive is an emergency you
haven't been able to avoid.

You still consider buying in stores?


> I've thought about it, but right now I don't have any physical place
> to put a server, never mind any kind of rack cabinet or anything
> like that.

You have lots of pictures with many tower cases and piles of disks
laying around all in a mess.  That takes at least as much space as a
rack, probably much less space when you get a 42U rack.  The server
goes into the rack on top of the UPS, and the disks go into the server
and you end up with both more space and more disk space and your data
is more secure.

> (Also when I was considering building a NAS / backup server, one of
> my criteria was having the entire cost of the setup, not counting
> the storage, be less than the cost of a single HDD.)

Well, you just spent like $600 on two disks you might not even need.
You may be able to find both a sever and a rack for less than that.
With a server with a ton of RAM in it you can experiment with
deduplication with ZFS, or you can use btrfs which apparently doesn't
need as much RAM for deduplication as ZFS.  You can get the server
before getting a rack and you might not need so many disks when your
data is deduplicated.

The more disks and data you gather, the more unwieldy and expensive it
gets.  Deduplication doesn't cost you.

It's not like there aren't options:
https://www.youtube.com/watch?v=F1xX3V_n0kw

> > You have way more capacity then I do and kept your data since the
> > 80ies, so of course you do qualify.
> [...]
>
> I thought to qualify for "data hoarder" I had to have never deleted
> anything, ever (I have deleted some things), and I had to have more
> capacity than any system available for purchase (either complete
> system, or parts) could support.

Just talk to HP or Dell and the like.  They'll provide you with
something that can store more data than you could ever gather during
your whole life.

> (For example, let's say I got one of those dual-socket Supermicro
> motherboards with 11 PCIe slots, and put a Highpoint Rocket 750 in
> each slot.

I doubt that would work.  I have an old RAID controller here which I
have retired that would support up to 256 drives through using SAS
extenders.

> Each of those cards supports 40 drives, times 11 is 440

only 40 :)

> drives, times 22 TB per drive (I'm ignoring the WD 26TB host-managed
> SMR drive)

host-managed?

> would be 9.68 PB on that system.  That doesn't account for
> on-motherboard ports, or using bifurcation / splitters to, for
> example, plug 8 PCIe 2.0 x8 cards into a single PCIe 4.0 x16 slot,
> or whatever would be the eqiuvalent bandwidth.  A data hoarder, I
> thought, would have more than that )

Nah, it doesn't really matter much data you're hoarding.  You're
already hoarding a lot of data and a lot of disks.  What are you /not/
hoarding, i. e. how much do you delete?

It doesn't matter anyway.  What matters is that you find a good
solution for you.

> Ah, I do plan to buy two disks, but idk how I'd set up real-time
> redunancy if I won't be running somehting like RAID1.  (What I've
> done so far is have one disk of each set of 2 be my data, then the
> 2nd has stuff manually copied to it then it's unplugged for a while
> until I need to update or restore, and I'm way behind on updating.)

Use a different RAID level maybe?  When you have disks in sets of two
(or another even number) of the same size, you're fundamentally well
prepared to use some kind of RAID.

Perhaps take a look at a basic overview like this one:
https://www.geeksforgeeks.org/raid-redundant-arrays-of-independent-disks/

That would be classic RAID like with hardware RAID controllers or
software RAID with mdadm.  And there are some more RAID levels and
variations.

> >    Using a warranty usually would require that you return your
> >    disk to someone.  There is no way that I would give my data out
> >    of hands.
>
> True ... but how else would I deal with a failed drive if I wanted
> to replace it without having to buy another one.

I don't know, I considered it one time but it seemed to be too much of
a hassle to tell the seller I'd crush the drive with a fork lift or
something before sending it in if they would want me to send it back.

To get on the safer side, you could run the disk for 3--4 months
without using it and assume it might not fail before the warranty has
expired.  But nowadays, the warranty is 5 years.

Or you could encrpty the data.  It's a bit unwieldy and may cost
performance, but at least you can make use of the warranty even if you
need to return the disk.

> Yes, there's the issue of the data on it, but that's what backups
> are for.

Backups won't prevent someone from reading your data from a disk you
returned under warranty.

> (Backups, however, won't save you the cost of a new HDD when you
> need to replace one that died prematurely, afaik.)

right

> I'm pretty sure warranties don't apply buying used parts from
> private sellers.

That depends.  When you have a switch from HP with a lifetime
warranty, and lots of them have one, they'll send you an
as-good-as-new replacement even when the switch is like 15 years old
and you bought it on ebay for $20, at least when you're a company that
returns it.  Fans don't last forever ...

I don't see how it should matter who you bought something from.  Who
the seller is doesn't alter the product in any way.

> Also I've heard that they may not apply when buying refurb parts
> from non-authorised sellers (in those cases the sellers are supposed
> to provide the warranty, same if you bought a complete system with a
> drive inside it from a retailer).

Claiming you could refurbish a hard disk is bullshit.  There's no way
that a seller who got a hard disk could somehow improve its
condidtion.  Such a claim is fraud and I'm surprised that ebay, for
example, tolerates that.

What's the seller gona do to it?  Change out the bearings and other
parts that might have experienced some wear?  Replace some capacitors
that may be on the controller board?  That would cost way more than a
new disk.

All they could do is blow off the dust and delete the data.  Deleting
the data makes the disk worse through usage.  It's been a long time
that you could actually format a hard disk.  Perhaps you still can
when you have the right equipment, but I'd like to see the seller who
has that.  Back then, you could also drive the reading arms all the
way over the platters until they hit the limiter and cause the disk to
make bad noises.

Maybe that's what they do when they're refurbishing them, drive the
arms all over into the limiters through the use of some special
equipment and when the disk makes funny noises it's still good to go
for sale.

And other parts?  What's there to refurbish?  You can pick the good
parts from the hardware you have and use that to put something
together that works, but I wouldn't call that refurbishing,
either. The parts remain unchanged and you're still selling a heap of
old parts that still work.

The broken parts that remain are the ones you can refurbish buy
recycling them to create new parts.

As to warranty, there are legal requirements here.  Of course, when
your hard disk fails after 4.99 years, the seller may not be around
anymore.  If you buy solar panels with a warranty of 25 years and have
them installed on your roof and one fails after 22 years, whoever who
installed them may not be around anymore; neither might be the
manufacturer.

Just check what warranty you get and what the chances are that you can
make use of it before you buy.


> > What I meant is that if I were to buy 16TB hard drives, I'd have
> > to buy at least two of them and that would cost me over EUR 1000.
> > Discs always come at least in pairs.
>
> Ahh, one for data, one for backup basically.

No, it's for redundancy.  Backups are extra.  (Fortunately, I found
chaper ones, but still ...)

> "Discs always come at least in pairs" ... maybe it translates to
> English a bit different han whatever your native language is, but to
> me, what I imagine is that it's not possible to buy one disk at a
> time.  In reality though, at least here, you CAN buy one disk at a
> time if you want to, but it's much more advisable to buy two.

Well, yes, in theory, I can buy a single disk.  But what would I do
with it?  It's not that I couldn't buy one, it's simply not an option
to buy only one because I won't have redundancy, and without that, I
can't store data on it and it's useless other than for backups maybe.

So you might think sure, only $300 for 16TB is a good price and not
expensive.  Actually, it costs $600, and how much data you can store
depends on the RAID level.  With only two disks, that's only 16TB, so
16TB actually cost $600.  What else did you think?

> > You can always check with hdparm. [...]
>
> Apparently hdparm isn't on here, and idk if it's available.

You said something about having Linux somewhere.

> [...]
>
> I wonder if other things could possible be causing issues.

like multiple issues maybe

> I just hope I don't have to rip my entire system apart, [...]

Perhaps you have too many disks in it and it's overwhelmed because of
that.

> Maybe; although idk if firmware updates are available or not,
> haven't checked (and I've never updated firmware on a HDD anyway
> that I can remember).

I have done it once with disks that would disappear after a while. I
had to turn off the computer to get the disk back.  So I had to keep
checking every now and then if had gone away because sooner or later,
I wanted the RAID to rebuild.

Almost 10 years later there finally was a firmware update for these
disks.  One of them had already failed and the other one was usless
because it would disappear all the time.  The firmware actually fixed
it and I was able to finally use the disk.

> I'm not 100% sure if it's the disks, cause other things also act up
> around the same time.

like when you have multiple issues ...

> [...]
> While I'm sure some people can take apart and rebuild an entire
> system in maybe a half hour or so, it takes me probably a full day,

It gets so much easier when using 19" cases like you wouldn't believe.

> and don't even get me started on configuring my OS, software,
> etc. the way I want it, or the way I had it before.)

That was never really possible with Windows.  Just use Linux, then you
have package management amongst other things.

> So is there a firmware update for those, or is there anything else
> special about them?  Maybe broken cables?

Maybe; you could find out.

> [...]
> SATA cables that have been slightly modified with the aid of my
> cat's teeth, one had its retention clip broken off, etc.

Are you hoarding broken cables?  Give them to recycling so you don't
accidentially use broken cables.


> > It depends ...  Unfortunately, ZFS isn't compatible with Linux
> > only for stupid licensing issues (you can get it to work, but that
> > isn't a good option)

I have to correct myself again, you just have to know how to.

> [...]
> I've never used ZFS.  Had considered it for NAS/backup server, but
> would lean more toward UNRAID or similar.  (I like how it lets you
> [...]

Maybe that's a good option for you then.  I guess you can always try
it out and see how you like it.  But how compatible is that with other
stuff, what if you want to switch to something else and can't read
your data anymore?  For all you know, they might call in your license
and then you're screwed.


> Dedup/backup ... A while ago I was wanting to back up a bunch of
> SSDs to a 12TB HDD.  If I didn't use compression I could just "dd"
> to a disk image and they'd still all fit for now, but... (see in the
> spoiler).

What prevents you from compressing the disk image created with dd?

> Also I'm not familiiar at all with BTRFS, or with VDO.

That means you need to learn more.

> [...]
> but I also have a couple 2 TB SSDs I'd want to back up as well,
> preferably compressed, and I don't have 2+ TB of RAM (or want to
> take the time) to decompress the image file before I can access its
> contents.

That seems like a rather unwieldy way to make a backup.  Why don't you
just copy the files you want to back up?  It doesn't seem like you'd
still need a functional disk image.  You're probably wasting 60GB of
space with that.

Maybe use this: https://clonezilla.org/


> > How well did that work?
> [...]
> HAH  .... not well 
> [...]
> I imagine my ASRock B550 Taichi might have a bit more DMI (or
> whatever it's called) bandwidth, maybe PCIe 4.0 x4 or 3.0 x8 but I
> forget, so it might do better with HDDs connected.  (One factor that
> made me choose it was the factor that I can have all 8 SATA ports
> and both M.2 slots populated simultaneously without having ports
> disabled.  I think it might cut one of the PCIe slots bandwidth in
> half if I have everything populated, but idk if I'll use that slot
> unless I plug in at least two SAS/SATA HBAs.)

Imagine what software RAID might do.  That's the beauty of hardware
RAID.  You can have a single RAID card to drive 8 (or even more) disks
without all the overhead and without all the copies of the data going
to each disk going all through the mainboard.  Not having that may be
fine for a file server that doesn't have much else to do, but for the
maschine you're sitting at and are using for what you want to use it
for, it can be a nasty and quite noticable burdon.

> > Older cards were limited to 2TB.
>

> Ahh, I guess the BT-PESAPA card I had (actually still do, but only
> use it rarely for PATA) was limited to tthat, it's from around 2005
> or so I think so maybe 2TB drives didn't exist then.

yeah

> Also several years ago I wanted the ability to pull individual
> drives out of a system to use with a 3.5" dock somewhere else, but
> I'm pretty sure USB flash drives and external SSDs would be better
> for that now. 

You can do that with SAS drives.  I don't recommend it and I still
rather turn the machine off when possible, but replacing a failed disk
by pulling the failed one out and plugging the new one in without any
further ado is really nice and not something I'd worry about (because
when the disk has failed, no data gets lost).  Being able to do that
is another beauty of hardware RAID.  Not being able to do that can be
expensive.  SATA really sucks, why couldn't they make it hotpluggable.

> [...] (I've wondered if it was a limitation of SATA II (3Gb/s)
> though... but I think I've heard of people plugging in >2TB drives
> into SATA II ports on other systems and it was fine, but idk.

IIRC it had to do with the BIOSs (and sometimes the hardware) not able
to handle so many sectors.

> Interesting thing though, the BT-PESAPA's boot spalsh text doesn't
> pop up on my AM4 board,

That's because EFI sucks.  There are tons of issues with older cards
in newer boards.

> and I can't see devices plugged into it from WIndows, but I can see
> them from Linux.

perhaps there's no module that would support the card

> [...]
> Also I mentioned earlier about liking UNRAID in that drives could be
> added one at a time.

... which probably means that it's not compatible to anything else

> I've also wanted the ability to pull a single drive with things on
> it and take it to a friend's house for whatever, but I imagine that
> a USB drive or external SSD would these days be much better suited
> for the purpose.

indeed

Even with SAS drives, what to you expect to happen when you pull a
disk while it's been written to or read from?

> [...]
> > The overcommitment doesn't really matter until the RAM isn't actually used.
> [...]
> Ahh.  Also something I haven't figured out is why my PC often gets
> lag spikes (sometimes small and barely noticeable like the one I
> just had where I typed 2 or 3 words before it appeared on screen,
> but sometimes also pretty big where half the system is unresponsive
> for a minute or two or more), even though I might only be using
> 5-10% of my CPU, and my RAM is only about half used.  (I would have
> thought I shouldn't be getting input / processing lag spikes or
> whatever until my CPU and RAM were both maxed out.)

That's like long interruptions ... try it with only two disks in a
RAID1 and see what happens, instead of having 10 single disks plugged
in.

> Also just now I was editing a portion of the draft for this post,
> and... I highlighted a selection of text, Ctrl+X'ed it, clicked the
> spot where I wanted to paste, Ctrl+V'ed, then clicked another spot
> to click and drag to highlight and delete some text .... but there
> was a delay from when I hit Ctrl+V until it actually pasted, and it
> pasted the text in the other spot where I was going to try to
> delete.

So that's extreme ...  Maybe you're having multiple issues ...

> Side note, it's super annoying when websites either load elements
> that move things around on the page AFTER it has visually started to
> appear,

That could be the web browser figuring out that it needs to display
stuff differently when more of the page gets loaded.

> or when, on a login page, it jumps your cursor to the
> username entry field.

Isn't that where it's supposed to go?

> I've lost count of how many times I go to click one thing and I
> click something else instead cause it moved, or, I'm in the process
> of logging in and start typing my password in clear text in the
> username box.

Just wait until it stabilizes.  Or get faster internet :)

> I remember the days in the MS-DOS era when, while the PC was booting
> up, there was some kind of keyboard buffer or something so we could
> start entering commands and things before it had even finished
> booting.  (Why a similar ability doesn't seem to carry forward to
> modern times escapes me.)

That's still there, isn't it?  The keyboard does that, it least it
should.  Maybe you need a better keyboard?

> > It seems like you need hardware that's more reliable.
>
> Maybe.  I thought the stuff I had was supposed to be at least fairly
> decent, at least if I remember right reading reviews on it, there
> weren't a lot of complaints about DOA, failures, etc that I could
> remember.

I guess it's standard consumer hardware.  Since I started buying used
workstations off ebay with Xeons in them, I don't have weird issues
anymore.  Sure I don't get the fastest and greatest, but it's more
than fast enough and costs a fraction of the money I'd have to spend
buying new.

It might be fun to buy some new parts and put them together, but
besides cost, what holds me back is that I don't know if I can get
something that stable that way.  It's still comsumer hardware, unless
I were to buy a new workstation from Dell or HP.

And that means 'sufficiently stable'. These workstations just work.
It's amazing.  I have a $2700 CPU in my workstation I payed 150 for
when I upgraded the CPU because I wanted a few more cores.  Now I have
2x14 cores.  I guess you get what you pay for.  It redefines what
'sufficiently stable' means.

> I mentioned earlier I'm not considering Seagate HDDs - that's
> because of possible reliability issues I've heard from some people,
> including my brother and others.

Yeah I've seen Seagates consistantly having very high failure rates
over the decades.

> I wonder if the newer IronWolf Pro or Exos drives might be okay
> though, but I would only consider one if I was running it in a RAID1
> array with a different brand drive, like a WD Ultrastar/Gold or a
> Toshiba MG-series drive.  (As for that, see the comments earlier
> about "fake raid".)

Why would you buy Seagate?  I wouldn't want to be forced to encrypt my
disks so I can make use of the warranty.  Even then, what do I do
while I'm waiting on the replacement?  Run the RAID in degraded mode
for the 6 weeks or however long it takes before I get it replaced?

> [...]
> "Stuffing more disks", I can't really do that, 8 is my max right
> now.

A nice Dell R720 can fit that nicely.  I'd go for a 12xLFF, though,
that gives you more to add more disks if you need to.  And you need
something to boot from.

> At this point though, I'd be replacing smaller disks with larger
> ones,

So yes, get your storage server up and running and move the data to
it.  You could go the easy route and simply make a hardware RAID5 and
put btrfs on it.  I was told you can use bees which would do the
deduplication in the background and keep deduplicating stuff while
you're writing.

Then you end up with a bunch of small disks which you can use for
backups.  That's what I do with my old disks.

> I don't have a working UPS now.

Ugh.  Get a nice 19" UPS from APC and put it at the bottom of your
rack because it's heavy.  You can buy them used, it's not like they'd
go bad except for the batteries that need to be replaced every now and
then.  The good thing with APC is that you can get new batteries
easily.

> Also I'm using 4x32GB Team Expert DDR4-3600 CL18 in my current AM4
> system, but running at 3466 cause it's not quite stable at 3533 I
> think, [...]

I'm glad I have my workstations.  They work.

> As for "stuffing more disks into my computer"... if there was a way
> to actually safely mount all of them AND hook them up...

That's what a decent case is for.  When I see all these funny tower
cases and what not that have all these cooling issues, I always wonder
'ok and where are the drive bays?'.  People put useless lights into
their computers but they don't even have drive bays, not to mention
backplanes.  How would that be useful?

> (Also if I was using a low profile CPU cooler, like something that's
> not taller than the VRM heatsinks, then I'd have room for several
> more HDDs...)

There you go:

https://www.ebay.de/itm/144404111197?hash=item219f27df5d:g:9O8AAOSwrZhhZ1kA&amdata=enc%3AAQAHAAAA4Gd1O5vyvyoOW8IY7xiInNzYaIhHzWGYsB9uG%2FBy9No%2FRFycFaB2vLBVCAkTzk2mARHqw4byVGoz5WrfKjZPmHrNC9KznX%2FedCAOPPgLbQPWF3QeO6e3c%2Bo1VzQz%2FE6VlwfrKtqMOxqQspmFyh0WHfKfPIJ5OdVnB0fSgQBpawv5IeTrnCY28NPGZt71JcIcewXylV%2FF%2Bjt0ZOx%2FGA2vHzdOrLINkpbiquIBBKGXfGwIhTV8ScfySa8R401vi6LVUf%2FNisXknEewUWze14IyZvI8w2%2FuebQ5uE%2BoCxJ3f0c2%7Ctkp%3ABk9SR6SEz7yNYQ

https://www.ebay.de/itm/284981250850?_trkparms=amclksrc%3DITM%26aid%3D1110006%26algo%3DHOMESPLICE.SIM%26ao%3D1%26asc%3D20201210111451%26meid%3D28e63db09b324ae4a36854e8ccbea38d%26pid%3D101196%26rk%3D4%26rkt%3D11%26sd%3D144404111197%26itm%3D284981250850%26pmt%3D1%26noa%3D0%26pg%3D2047675%26algv%3DSimplAMLv5PairwiseWebWithBBEV2bAndUBSourceDemotionWithUltimatelyBoughtOfCoviewV1%26brand%3DSupermicro&_trksid=p2047675.c101196.m2219&amdata=cksum%3A28498125085028e63db09b324ae4a36854e8ccbea38d%7Cenc%3AAQAHAAABMIep8YrnpoI5Ip1Fr26ya5Zdwg2V78Yl01uYLsdPZcFTvxKxDts2M2LiSIpUM3yyPCg4I%252FI%252F%252BBr%252B3lbC%252BxYqInYGBmzIxc4EMw3YF0EDxTKyEHLT%252F%252BSMG2X2Go1miFYKtC00MzMQyW9gDfbMtyClMOXZ8xJKGtjoLzUP%252FdiLuNWupvrVbgXCTDewZ4VueuJxRUMMZnXCy%252FDutw%252FxnITm48hlHbLvuMw3l6iBYxzde1cCgdhPRpOr3GLW%252BVqnkBvK2bQ6zIKPiX2RUJokpE%252Fbl83Sl9REJhrS0BYpSBleJScnEzI6pT11MsM153kyy4ma2Rj7Q7cJZh%252BscrwME71FUisxmmg8JNI%252Bwc7NlH8p7l8kzUTYUOCx8vRWWzR%252BuBe7MuZzk1rBbjZsLaRuwoLBA58%253D%7Campid%3APL_CLK%7Cclp%3A2047675&epid=149985814

What's the problem?

> This is how I might have wanted to connect extra drives if I used a
> SAS HBA with external ports, like tha 9200-16e...

I'm not sure which picture that is.  Some look as if you're trying to
create a fire hazard.

> [...]
> But, maybe it's better to have a dual-chamber case

It's better to get a decent case.  Chenbro really makes great stuff.

> Sometimes I've had to run drives outside my case as well...

Ugh.  For testing, ok, but other than that, it's like stiching buttons
to your cheeks.

> [...]
> > Why more than 512?  I wouldn't use that much but it kinda makes
> > sense to use sector size as orientation.
> [...]

> Okay now I see the "compare no further than the first byte that
> differs"... that's a good idea probably. 

Yes, there's no point in comparing any further.

> [...]
> Also, some media files might have been en-/trans-coded with
> different settings, but are ultimately the same thing.

But the files are different.

I was planning to make some backups today.  Maybe I can experiment
with deduplication with btrfs.

> In some cases I might have a bunch of different bitrates / codecs of
> the same thing, and I need to pare some of those down and only keep
> the max quality / uncompressed one (a "master" so to speak), and
> maybe a couple lesser bitrate ones probably.

That'll be a lot of work.

> Well I think it's possible that the first 512 or 4096 bytes or
> whatever might be identical, but somewhere later in the file might
> not be.

Yes, it would only serve to be able to say which files are different.

> Also I don't really need this to be super fast, as in, finish the
> entire job in like a half hour or an hour or so.

But I don't like inefficiency.

> I'd be totally fine if it takes several hours or even a full day or
> two or three to process.  I can just use my laptop in the meantime
> while my desktop is churning through the data.

Is your computer so slow that it would be overwhelmed by gathering data
about a few files?

> As for where to put the database... I do have about 650 GB of space
> free on my NVMe boot SSD, maybe the database could hopefully fit in
> a few hundred GB of that, leaving the HDDs free for doing the main
> operation of going through their data, etc.

How many files do you have?  Don't you have mariadb running on your
server?

> As for files of different sizes being identical - of course they
> wouldn't be "identical" ... but they could have been taken from the
> same audio, photo or video source, and just be a different
> resolution or bit rate of the same thing.

Yes, but why would you keep so many dupes?

> [...]
> Nothing prevents you from making a check sum of all the check sums
> of all the files in a directory and from comparing those check sums
> of check sums.
>
> Ahh, checksums of checksums... but in my case I also have a bunch of
> partially duplicate folders, some maybe only 2 or 3 levels deep,
> some well into the double digits. 

That won't matter because you'd be comparing directories and not
directory hierarchies.  If you were to comare the heirarchies, what
are the chances that there would be dupes of folder hierarchies?

> [...]
> And there's the issue of the structure, for example,
> E:\a001\b001\c001\d001 might have a similar structure to
> H:\a003\b003\c003\d003\e003\f003\g003\h003\i003\j003\k003\l003\m003
> and so on,

How's that simialar?

> [...]  (too bad there's multiple reasons why I can't just boot
> Linux, do cd /, then use a terminal command to recursively list
> everything then save & upload the resulting txt )

Why not?  Don't you have an USB stick you can boot from?

> quick note, speaking of backups... the idea of something like "dd
> if=// ... " comes to mind,

No, dd is not for backups.

> but if I'm backing up the ENTIRE filesystem including connected and
> network devices,

Such devices have their own filesystems.

> problem is what would I back it up TO, if I'm already specifying to
> back up not just everything directly connected to that PC, but also
> everything on the network...

Why would you do that?

> (Also in my book, if I'm not including every single bit / sector
> including boot sectors, landing zones or whatever, normally
> inaccessible areas, firmware, etc, it's not a complete backup.)

Maybe you want to use clonezilla to make a clone.  That would be a
rather unwieldy way of making backups, and you'd be cloning a lot
every time you don't need to clone.

> (I'll admit that the thought of just deleting everything has come to
> my mind more than zero times, but I don't really want to exercise a
> nuclear option as there are some things I want or need to save.  )

You can always get your storage server up and running and copy the
data to it before it's lost, and use deduplication so that the
required storage space doesn't get out of bounds.  Once you have that,
you can make a backup and then start to go through all the data and
delete the stuff you don't need to keep.  Just make a snapshot before
you start deleting, and you may not even need your backup.

> Also I'm maybe considering getting a couple 2TB or 4TB SSDs (one
> SATA 2.5", one NVMe M.2) for my laptop, to replace a couple 1TB SSDs
> so I have more space.

Why don't you get your storage server up and running and store the
data there?  Do these laptops even support redundant disks so it's
possible to store data on them?

> [...] I'm just a bit hesitant to pay that much, for example, for a
> 4TB

Yeah get your storage server up and running.  Not doing that holds you
back and makes everything difficult and totally unreliable, makes you
throw endless amounts of money at buying more storage all the time and
all you accomplish is that you make the mess worse and uselessly store
endless amounts of duplicate data.  And you still have no way to
prevent data loss, like due to disk failures and power surges, and no
way to make backups.

> [...]
> Also ... I've been thinking a bit recently about getting a couple
> more SSDs for my laptop,

How many can you fit into it?

> either 2TB or 4TB to replace a couple 1TB

Why don't you keep the data on your storage server?  There's no need
to have that much storage capacity in a laptop.  Wireguard works fine.

> Looking at pcpartpicker prices.... [...]

Stop buying any more storage before you got your storage server up and running.

> I've also wanted to compare the speed of some old HDDs vs current
> ones and SSDs -- but not in terms of MB/s transfer rates, I'm
> thinking more like "How long does it take to fill the entire disk if
> you're constantly writing to it".

Why?  Put your old disks into your backup server and make backups on
them, then you'll know how fast you can write to them.


> I wanted to buy a couple old <80MB PATA HDDs on ebay,

Stop buying any more storage before you got your storage server up and
running.

PATA?  Wasn't that the predecessor of IDE?  I may have had a 386DX16
that had one.  60MB was kinda huge ...

> (I'm thinking that some of the older/smaller drives, like 40MB or
> 20MB, or if anyone has 10MB or 5MB MFM drives laying around and a
> working system capable of interfacing with them, could fill the
> entire drive in maybe half a minute to a few minutes or so,

You probably mean hours and days, not minutes.

> [...]
> Maybe instead of me buying the drives, maybe someone else already
> has a system with drives like that, or better yet, something that
> supports MFM drives and has some of the early 5 or 10 MB drives.

Oh those are very rare ...

> I wonder how quickly those could be written or read...

You could probably dig up some old benchmarks or something.  You're
gona like this, for example:


1 5 2 8 - 1 5    MICROPOLIS
NO MORE PRODUCED                                      Native³  Translation
						      ÄÄÄÄÄÄÅÄÄÄÄÄÂÄÄÄÄÄÂÄÄÄÄÄ
Form                 5.25"/FH              Cylinders    2100³     ³     ³
Capacity form/unform  1346/ 1535 MB        Heads          15³     ³     ³
Seek time í / track  14.5/ 4.0 ms          Sector/track   84³     ³     ³
Controller           SCSI2 SINGLE-ENDED    Precompensation
Cache/Buffer           256 KB DUAL-PORTED  Landing Zone
Data transfer rate    2.916 MB/S int       Bytes/Sector      512
		      4.800 MB/S ext SYNC
Recording method     RLL                            operating  ³ non-operating
						  ÄÄÄÄÄÄÄÄÄÄÄÄÄÅÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
Supply voltage     5/12 V       Temperature øC         5ö50    ³    -40ö65
Power: sleep              W     Humidity     %        10ö90    ³     10ö90
       standby       25.0 W     Altitude    km    -0.061ö 3.048³ -0.305ö15.240
       idle               W     Shock        g         2       ³     20
       seek          30.0 W     Rotation   RPM      3600
       read/write         W     Acoustic   dBA        43
       spin-up            W     ECC        Bit   48
				MTBF         h     150000
				Warranty Month        60
Lift/Lock/Park     YES          Certificates


That was actually a really good drive.  IIRC it was more like 2MB/s,
and the power consumption was hilarious :)  (You realize that 5.25" is
really 5.25" and not half that like everyone thinks?)  Wow that was
one heavy disk ...  I liked it a lot, it made such a trustworthy heavy
tock tock tock sound when it was doing something that you always knew
when your computer was working.  I mean /working/, you know ...

Oh, look at that: I just noticed it had a 5 year warranty.  Not
surprising, it really was a tank.

> I hear the ST-506 MFM interface supported up to 5 Mb/s, not sure if

Does that seem realistic?

PianoPlayer88Key · November 28, 2022

Okay, this time I just decided to categorize the different reply sections, and reply to the categories here.

I've numbered and labeled the sections (sequentially as they appear in the previous post), and I'll go ahead and put my previous very-long draft in a spoiler at the bottom of the post.

If you want to find a particular section from its number (so you can see the previous "thread"), you can open the spoiler ("Reveal hidden contents") at the very bottom of the post, then search for "► ### - ", where ► is typed on WIndows with Alt+16 (numpad,not over alpha keys), and ### is a 3-digit number from 001 to 112.

Those numbers are referenced in each section below. Some are not exactly used, although they do appear in the draft. Some of those are marked with → (Alt+26) instead of ►.

Also it took me a good while to get this done, have had (and still do) a lot of other things going on as well. (I may have to put some of my data decluttering back on the back shelf, although I'll still be doing some general disk-to-disk migrations as that does need to be done.)

Anyway, once I hit submit on this, next thing I plan to do is prepare my PC for shutdown, install the 18TB drives and start doing some data shuffling / migration, probably booting to the 250GB SSD that has Linux installed on it.

----------------------------------------------------------------------------------------------

► Backup / Redundancy -- 044 ; 076 ; 099

Are you saying something about THREE disks for each one, or what? (Also are you referring to the 3-2-1 backup strategy, or something else?)

For me, I don't really need live-online redundancy (as in immediate seamless failover to a spare disk), but some kind of backup I definitely should have.

Elsewhere I mentioned having poor internet (10Mbps and 1TB/month cap) which makes doing online backups pretty much impossible for me. (I could back up text files and things like that, but that's nothing in the grand scheme of things.) If I was doing online backup though, I'd prefer something priced no higher than what it would cost me to buy that much space on a HDD over time, considering the warranty. (A 16TB WD Red Pro has a 5-year warranty, if it's purchased for $269.36, that's (26936/16/60) ~28.0583¢/TB/year, or if you double that (for 2 disks for redundancy) it's ~56.1167¢/TB/year, much cheaper than Amazon or Google's cloud storage, and probably cheaper than Backblaze too.)

Backing up everything including connected devices was a hypothetical idea - at least for now, although it probably wouldn't be a bad idea to sometime figure out how to back up everything, INCLUDING the backups, and cloud backup, etc.

Some time ago I heard of people talking about doing daily full backups and hourly incremental/differential backups, but with my mutliple 10s of TBs (actually now probably >100TB) capacity, that doesn't seem possible.

Also I've thought of having a backup server be mostly for the occasional full backups, and for incremental backups, have another drive connected to the system that backup files get put on as I'm working on. Then when it gets full I'd either move it to the backup server and get another local backup drive, or connect it to the server and dump its contents, then resume use as a local backup.

► Cables -- 054

I may have a few broken cables laying around still, but I'll want to get rid of them once I've verified they're broken. Also maybe I should make an effort to not follow in the footsteps of my dad on some things... yesterday my mom was complaining that he pulled something out of the trash, cause he was probably going to try to fix it. He used to be really good at that when he was younger, and HIS dad was even better at fixing and even making things, but my dad's showing his age now, he's 76 and not as able to do things as he used to.

► Case -- 008 ; 081 ; 082

Most Chenbro cases I've seen were rackmount, I think.
For the most part I don't like the RGB (although it has proven useful for "is it powered on or not" diagnostics), tempered glass, etc, I prefer something more similar to my Define R5. I did take the front door off, though, broke the door on a Xion XON-303 case I had about 10 or 14 years ago and didn't want to break this one. I probably could technically put it back on as the spot where my Blu-Ray burner would normally go is occupied by my 360mm AIO, and I don't have an extra SATA port for it anyway on this motherboard.

Quite a few years ago I saw cases with drive bays taking the entire height, and some with even an extra partial stack or two behind the main stack. I can't find any cases like that sold new anymore. There's an Antec 900 that's on sale for $118 from Amazon (not 3rd-party, but only says 1 left in stock), which can be configured with 9x 5.25" bays, which in turn can probably house 3 each 5-in-3 drive cages (5x 3.5" in 3x 5.25") for a total of 15 drives, but I'm not planning to get one right now.

Those ebay examples are from Germany, I'm in the USA; also the cheapest is ~25-35% more than my budget for the ENTIRE system not counting storage, if I was using those cases in a backup solution.

► data ; hoarding -- 037 ; 105

I probably delete less data than I should.

Also for things like family / friends documents, photos, audio, video, etc, I'd like to find a good long term storage / backup solution that my great grandkids (I have no children yet) can access even after I'm gone, if they want to.

As I mention in the "server" section, it would be nice to have a central location for everything, so no matter what device I'm using or where I am, I could access it. (I do still want to be safe though.)

► Disks -- 009 ; 043 ; 075 ; 076

If possible, I prefer replacing several smaller disks with a single disk equaling their combined capacity, for the price I paid for one of the smaller disks previously.

(As with anything though, if I have to replace something because of infant mortality (dying before warranty runs out, or otherwise prematurely), I'll try to get what I need at the time.)

"Refurbish HDD" - Yeah I don't think so, unless it's the manufacturer themselves doing it (or at least someone like DriveSavers or similar but I doubt they're in that business). Otherwise, I suppose someone could "call" it a "refurbish" when they DBAN the disk and, well idk for sure if it's possible to wipe / reset SMART data, but I guess so if you have the right tools?

Seagate: Yeah, hearing of consistently high failures, and my brother not liking them, are a couple reasons why I'm wary of them. The only way I would consider using it if it was in a RAID1 array with another disk from a different brand entirely - for example a Seagate Exos combined with a Toshiba MG or WD Ultrastar.

On the other hand, I've heard of people with really old Seagate drives that were still working 30 or even 35 or almost 40 years later. A couple examples on YouTube are here and here, of the ST-506 - the actual drive, not just some other drive that uses that interface, as well as an ST-412 video.

Sometime I'd like to find out exactly how this 8.4GB PATA IBM DTTA-350840 bit the dust. I could maybe open it up myself (since I don't care about any data that may have been on it, but from the scraping sound the heads are making as they sweep across the platter I doubt it's there anymore anyway), but I wouldn't know what to look for to determine how it kicked the bucket.

► Duplicates -- 093 ; 094 ; 095

I don't want to keep "so many" duplicates, just the originals from the media (camera, audio recorder, etc), and a couple others either for editing, or final renders / versions.

Some time ago I had a few extra duplicates laying around, where I was, for example, testing a lot of combinations of mp3 encoding. (I've wanted to also do a similar experiment with video encoding but still need to learn some of the ins and outs of that.)

A few times I've recently come across some duplicate folder hierarchies in different places. (When I was moving stuff around on a few disks the other day to prepare them for copying to the 18TB (wanted to resolve any possible "folder with same name already exists" conflicts), I came across a few duplicate folders which I deleted.)

Some folders may start off at diffeerent levels or different places, but when you drill down farther within the rabbit hole they have the same things in them. (I've also been learning that it would be nice to be able to look in .zip files or similar and find duplicates there.)

► hardware, consumer / workstation / server -- 074 ; 080

If I had the $ I would have liked to build an Epyc system, but I also generally prefer the aesthetics of some consumer grade stuff (simple designs like Fractal, etc, not RGB/TG/etc or the over the top designs like some other "gamer" cases I've seen in t he past.)

For now I'm limited to consumer hardware for my platform. A while ago I thought about building an LGA771 or 1366 based NAS, or maybe even 2011, but was having trouble finding the type of non-rackmount case I wanted...and electricity here is not cheap - several years ago it was upwards of ~40-67¢/kWh or so, and I thought I heard talk of >$1/kWh.

I also thought of LGA1151 v1, as I do still have the i3-6100 I pulled out of my laptop (Clevo P750DM-G) when I upgraded it to an i7-6700K. I was having trouble finding cases then as well, although I did see a couple motherboards that looked promising a few years ago, like an MSI C236A Workstation for around $90 or so, and a couple others, maybe an ASRock Z270 Taichi (but doesn't support ECC) or an ASUS P10S WS or whatever it's called, or maybe ASRock C236 WSI, IIRC.

► hot-plug, SAS / SATA -- 062 ; 067

Actually, SATA does include hot-plug ability in the spec, at least for internal SATA I think. (idk about eSATA though, when I had a board that supported it and an enclosure with that, I don't think I could do it.)

On both my recent motherboards (current B550 Taichi and previous Z97 Extreme6), there's an option to enable or disable SATA hot plug in the BIOS/UEFI. On the B550 Taichi I think it's just one setting to enable it globally, while the Z97 Extreme6 has it selectable for each port. (When I was booting off a SATA SSD, I would set it so the port my boot drive had hot plug disabled, but the rest had it enabled.)

I don't think the M.2 or PCIe slots support hot plug, although I remember a video Linus did a while ago where he was demonstrating a server motherboard that did support PCIe hot plug.

On my B550 Taichi desktop, clicking the "safely remove hardware" menu pulls up several HDDs, as well as an M.2 SATA SSD (although I don't think that supports hotplug, idk why it's in the list).

Also I would want to be careful to not eject a disk while it's being written or read. I try to make sure my system is set to disable write caching, as I don't want the dialog box to disappear (or a command / terminal to say it's done copying) until it's ACTUALLY done copying and has written everything to disk. (In the past, I've ejected disks or memory cards after it said it was done, then the system popped up and said oops some data got corrupted.)

Interesting thing though, I've sometimes seen my RAM usage spike when doing a large copy (and an initial transfer rate of much faster than the capabilities of the disk I'm copying to), even though I thought I had write caching turned off.

► image / dd / clonezilla -- 057 ; 097 ; 100

I did compress the dd image in at least one test. Problem, though, was it would take forever to try to open it in 7Zip, and my RAM usage would ramp up then it would crash when I ran out of RAM. (I'm thinking it was trying to decompress the ENTIRE image to RAM, which on my smallest SSD would be 240GB, my largest current one would be 2TB, and I now have a 4TB WD SN850X and Samsung 870 Evo on the way, thanks, Black Friday.)

I would REALLY like to find a way to compress the images so I can fit more backup images on a single disk, but in a way so I can easily open the image in something like 7-Zip and browse, or extract individual things as needed.

For now, though, I'm thinking I might just dd the drives to image files on the 12TB Toshiba MG07 drive, and as for the two upcoming 4TB SSDs, I might back those up onto the 8TB Toshiba N300 that I already have, once I've dumped its contents onto one of the 18TB drives. (I won't be using "dd" for that though, just a file manager in Linux probably, since I'll be combining two HDDs into one - twice, basically.)

Also, if I'm backing up an OS drive, I want it to be bootable. I think I've tried Clonezilla in the past, and I vaguely remember it worked once a long time ago, but I've tried it since then and it would fail to boot. Using dd, though, does work. (I should maybe test again, but I think a while ago I tested backing up a 240GB SSD to an image file on a HDD, then restoring that image to a 500GB or 1TB SSD that I have laying around as a spare, and the laptop I was using the 240GB SSD in was able to boot off the other SSD just fine I think.)

DD is the only tool I know of that backs up every single sector (although idk if it also catches firmware, or overprovisioned area / spare sector area though, so maybe I'm not really getting complete backups with it.)

You are correct though about dd not being the right tool ... for incremental or differential backups. (I have kinda learned what the difference is, but am forgetting for now so I should probably re-educate myself.)

► keyboard -- 073

For now I'm using a Logitech K270 on my desktop and K360 on my laptop. For some reason on the desktop, I periodically get some kind of lag spikes where I type or key something, and it doensn't appear on the screen until a second later, or sometimes several seconds. (Then it will bunch up and all appear at once.)

Yes, both keyboards are wireless, and maybe wired might have better communication. (I do have the wireless USB dongles sitting near the keyboard for the desktop, via a USB extension cable.) I quit using wired for now when my cat chewed the cords, and I also like the convenience of being able to move around somewhat, although with all the clutter around right now I can't do a lot of moving around.

► laptop -- 102 ; 104 ; 105

I believe my laptop (Clevo P750DM-G) technically does support RAID0 and RAID1, not sure if it supports RAID5 or RAID10 though.

It does have 4 storage slots - two M.2 slots which support both SATA and NVMe, and two 2.5" bays.

I might be moving to Utah sometime (a possible training / job opportunity not involving tech / PC things), although for the first few months or so I might not have my desktop PC, just my laptop, and I won't have any kind of server setup at my parents' house to access data on my PC there, so I'll need to take some stuff with me on the laptop, and decide what I need to have access to while I don't have my main PC. Once I'm more settled in I'd make a trip back to CA to get my desktop PC, some furniture, a couple pianos I have, and other stuff.

► Linux -- 046

I have Linux on a secondary SSD in the desktop, and some VMs in my laptop, but I mostly boot Windows 10 Pro as my main OS on both. (I would consider switching to Linux though, especially if a Windows 12 continues in the direction 11 seems to go in things like wanting online accounts, etc, after 10's support ends. My brother, I believe is still running 7, although he did help my sister and her kids build a PC a weekend or two ago and that one has 11 on it.)

I wonder if there's a Windows tool (maybe something built into powershell) that's similar to hdparm.... I do have CrystalDiskInfo, and I think GSmartControl is available for WIndows as well and I do have it installed.

► magazine ads, museum -- 025

I'd like to find a way to scan the entire magazines sometime (without having to do it page by page, just feed the entire magazine into some kind of scanner, or send it to someone), and archive them that way. I have a couple boxes of the magazines, and would like to get rid most of the physical copies and just keep digital versions. (A while ago I was doing something similar with several hundred CD-Rs, DVD-Rs, etc that I have, but that project is on hold cause of other things that keep coming up, such is life around here.)

I'm missing quite a few of that ComputorEdge magazine though. If I remember a few years ago, the original publisher (Jack Dunning I think, he has auto hotkey stuff on his computoredge.com site last I remember) might still be around, although I haven't reached out to him to see if there's a chance hey may still have the original files from which the magazines were published - maybe there's a way to convert them to searchable PDF or something...

Also I heard that an author of a few "For Dummies" books had spent time writing for ComputorEdge in the '90s or so.

► performance (or lack of) -- 048 ; 090 ; 091 ; 111 ; 112

I don't think too many disks is overwhelming my system, I think it may be something else causing the issues cause other things ar e happening too periodically, often simultaneously. (Some things are more common, like my WiFi dropping out and a few HDDs being inaccessible, while others have only happened once or twice like the keyboard/mouse not responding or the GPU going to black screen and 100% fan.)

I too don't like inefficiency, but I do want a complete job done. (If I'm parsing through data on an entire HDD, I expect it to take at least as long to do that as it takes to sequentially read the entire drive's capacity.)

My 5950X isn't THAT slow, but WinDirStat a week or two ago said I had upwards of 5.77 million items on this PC, and that's with not all the drives plugged in or mounted.

I have a feeling that older disks were "faster" in terms of how long it takes to fill the entire disks. (I've seen a couple clips on youtube where the write speed in a benchmark was such that it would only take a few minutes to fill up an entire drive, for example I calculated ~99 seconds to fill a 40MB Seagate ST-251, and there was a Connor IDE 20MB drive that could do about 500 KB/s or so.)

It took me about 11 minutes to fill another 8.4GB IBM hard drive that's still working (same model as one mentioned elsewhere that has the click of death, but a month older).

Also there's a Tom's Hardware article on "15 Years Of Hard Drive History: Capacities Outran Performance" from November 2006 - the page I linked was "Time Required To Write A Full Platter". The largest drive they tested at the time was a 750GB SATA drive, and it took 52 minutes to read 200GB, or ~27% of the capacity. Compare that to the 37 seconds it took to read 26 MB of a 40 MB IDE drive, or 65% of the capacity. (Extrapolating, I'm guessing it would have taken about 3 hours 17 minutes to read the entire 750 GB drive, vs about 57 seconds for the entire 40 MB drive.)

Re: the Micropolis SCSI drive - Ahh I think I've seen some kind of site similar to that with old drive benchmarks and things, I think stason.org or something like that.
If that 1346MB drive transferred data at 2 MB/s, then the drive could theoretically fill in 11 minutes 13 seconds.
I looked on that site for some smaller drives, and saw several interesting ones, including the 5MB Fujitsu M2231AS and Seagate ST406 (I didn't see the ST506 on the site), the 10MB IBM WD 12, Seagate ST412 and Tandon TM252, which have an internal transfer rate of 0.625 MB/s. Then there's the 21MB Seagate ST325A/X, rated for 1.5MB/s in AT mode and 1.75MB/s in XT mode, and the 42MB Quantum Prodrive ELS 42 AT at 2.5 MB/s. Even if their actual transfer rates are slower, like in your 2MB/s vs 2.916MB/s example, it looks like they would theoretically transfer the entire capacity in about half a minute or less

Okay, the ST-506 may not reach 500 Kb/s in the real world, but I did see a few youtube videos (01, 02, 03, 04) that included benchmark results from some older MFM and IDE hard drives. Some of them might have only transferred data at like 300-500 KB/s, but with small capacities like 20 or 40 MB it would still transfer the entire capacity in about a minute or two.

► RAID -- 007 ; 009 ; 012 ; 014 ; 015 ; 034 ; 107

I've heard things about software raid being more preferred now vs hardware raid. (For example, about ZFS preferring RAID cards be in IT mode, not RAID mode) I've seen a few 9211-8i around $30-40, occasionally 9200-16e ~$25 or so, and rarely the HIghpoint Rocket 750 about as cheap per port (although still more for the card cause more ports)

I've heard of a few RAID levels, and Raid-Z and Unraid, but don't know all the possible levels.
I don't need higher performance than single-drive (if I do I'll use an SSD for those situations).
If disks fail, I want to be able to replace them and rebuild from parity / mirror (I've heard stories of RAID rebuilds taking days or weeks on some drives, I hope to avoid that - hopefully it doesn't take much longer than the time to sequentially write the drive's entire capacity). If too many disks fail, I don't want to lose the entire array, just the extra disks that failed (like Unraid).

I've heard Extenders limit you to single-port bandwidth, so I don't like to use them. With only a few HDDs they might still "fit" within the bandwidth of a single port, but with a lot of SSDs they'll be bottlenecked really quick.

Also I prefer being able to access disks individually, although true, it can be nice to be able to group everything together sometimes. I think Linux might actually do a better job of doing that, at least logically or with the file / folder / mount structure.

UNRAID - Yeah that could be an issue, being unable to read the data if I switch to something else. Also I've heard horror stories of RAID controllers going rogue and committing fornication with the hard drives, spewing Scheiße all over the place, which helps to further my preference of working with the drives individually, not spanning things across disks. (The only time I would do that is if I need to store a file that's larger than the capacity of a single disk, and it's been probably 10 or 15 years since I've done anything with floppies. We still have a few stacks of 5.25" floppies, would be nice to see if there's still some readable data on them, but I've heard they weren't known for longevity...)

► Server / Rack -- 031 ; 051 ;060 ; 078 ; 079

But where would I even PUT a rack cabinet or storage server or anything like that?

My current PC setup is in the first pic. In the 3rd pic, you may just be able to make out a part of a PC monitor sitting on the piano's music desk - that's where I've previously had my PC setup. I have the stand for that Dell U2414H, sometime I should see if I have or lack any necessary other hardware to attach it. I've never mounted it on the stand, have always used it sitting on the piano like that.

I don't have any experience with 19" racks, cases, etc. I do have an audio cabinet that I got from my grandpa when he passed away that happens to be 19" wide internally, but it's only about 15" deep or so, with the back end being open. There's no way to secure a 30" or so deep case in there. (Also right now the cabinet is inaccessible, as there is other furniture and stuff in front of it. It's not visible, but it's toward the back left of the 2nd pic.)

Also if I ran a backup server, that's literally ALL it would do - no plex, no transcoding, probably even no zip/unzip/etc. JUST storing and transferring files.

Although, it would be nice to be able to have some central place where I could store everything and access it from any device, both in the home and, if I could do it safely and if it weren't for only having 10Mbps upload and a 1TB/month cap, from outside as well. (Also another project I've wanted to do is converting a bunch of old audio and video tapes, vinyl records, reel-to-reel, photos, etc, and that might be easier done on a server / rack with multiple devices ingesting simultaneously, but for now that project is postponed for probably a few years or more, until I have a place I can set the stuff up, AND can afford the equipment, for example something like a Nakamichi Dragon cassette deck, among other things.)

► shopping ; online vs store -- 022 ; 023 ; 024 ; 030

I remember going to a store a few years ago and seeing this.

That was Fry's in San Diego, in Q1 2016, I think right before things started really going downhill. One of the last times I was there before they shut down, a song came on over the P.A. that I thought was pretty appropriate: "Highway to Hell" by AC/DC.

In San Diego there used to be a few dozen computer stores or more in the 1990s, but almost all are gone now (most were gone by the mid/late 2000s).

I buy online a lot now (well okay, not a lot, sometimes my Amazon, Newegg or B&H order history has entire years where I don't place a single order), but some things I might buy in a store if it's available and not too inconvenient, or I'm saving enough $ to make it worth it. (I bought my 5950X at Micro Center, for example. it was pretty easy to park and walk in but it wasn't Black Friday or GPU launch day.)

► Storage -- 005 ; 006 ; 009 ; 017 ; 091 ; 092 ; 110 ;

I got the drives a couple days later than I'd hoped, but they're here. Been busy with other things, so as of now I have yet to install them, although I've been forced to restart my PC a couple times.)

I've never used fiber interface for storage, thought that was for network?

I have a few Toshiba MG07s (12 and 14TB), and so far they've been fine.

I heard that when WD got HGST, in the process Toshiba may have gotten some of the 3.5" equipment / technology, but I don't remember for sure.

This is a short video of scrolling through a little of what WinDirStat found, just scratching the surface. (I expanded a couple levels of folders, but didn't drill down very far.)
https://www.youtube.com/watch?v=8w82ux0xm3o
I noticed quite a few folder structures that looked like they could be duplicate hierarchies while looking through that.

WinDirStat says 5.14 million files.

I've seen MFM / RLL drives listed on ebay sometimes, but don't have a system to put them in or the knowledge to work with them. I've seen posts on other forums geared more toward those things, but don't have accounts there (and have been experiencing "online account overload" anyway), although I have also seen some of them on some retro tech youtube channels.

► warranty -- 039 ; 040 ; 042

I'm aware of pitfalls of sending disks with data back, would be nice to not have to do that. (Also they don't preserve / backup the data, they send a blank disk back usually.)

Also for now I generally don't use encryption to make it easier for me or trusted friends to read the data on other systems, although I do have a couple VMs encrypted.

If something happened to me or to my next-in-line trusted friends/family, then if someone else needed to access the data, I want to be able to provide a way to do that, while still being safe.

I just prefer to save $ on replacement disks if it happens within a warranty period. Yes, I'd need to spend the $640 to buy two 18TB hard drives up front (one for storing the data, one to back up the data), but if one of those fails after, say, 1 or 4 years (assuming a 5-year warranty), I don't want to have to pay another $320 or whatever to get a third replacement disk. (I'd restore from my backup.)

A backup will let me have my data right away, and not have to wait for warranty replacement disk. (I think that would then become the new backup.)

I thought buying from the wrong (or a not-authorized) seller would affect whether I even HAVE a warranty from the manufacturer? (Especially if buying a refurb product, but I've heard horror stories of the same thing when someone bought something they thought was new.) There's a seller on Newegg (EOL Tech I think, and others) I've seen selling hard drives produced like 10 or 15 or more years ago as new I think, I doubt they still have an active manufacturer's warranty.

► web browser, sites -- 070 ; 071 ; 072 ; 073

I remember a decade or two ago, sites would actually behave properly when loading, I think, for the most part.

Also I wonder if there's a more sinister reason to move things around on the page after it's loading -- to try to get people to click on more ads. I've lost count of how many times I went to click or tap something I wanted to open, and the page moved, or an ad spawned right there and I clicked it instead of what I wanted.

I grew up in the era when it would take a few minutes to boot up, and you could key in commands at the command prompt before it had finished booting, and queue up.

I want faster internet, but it's not available here. Yeah I might be able to get 1 Gbps down, but would be limited to 35 Mbps up I think. (I don't think fiber is available here at least not yet; also I'd really prefer internet that's at least as fast as my SSDs or RAM, or better yet fast enough so everyone in the house, both family and visitors, could likewise have full-speed access that wouldn't bottleneck their RAM or SSDs, but instead would BE bottlenecked by their RAM or SSDs.)

► misc -- 001 ; 002 ; 003 ; 011 ; 013 ; 036 ; 083 ; 085 ; 092 ; 108

001, 003 - emacs/mail: Ahh, I haven't done mailing lists in a while but I remember the > for quotes. Also I was able, with much effort, to remove the code from the box and prepare it for a reply. (It involved spreadsheets, multiple drafts, etc.)

002 - spolers: They are the "Reveal Hidden Contents" boxes. I'm using them to try to shorten the visual length of the post.

011 - Backplanes: Most backplanes I've seen are crazy expensive, or in expensive server cases. (I've had the idea of "making" my own, by somehow attaching some right-angle SATA data & power cables to a board and putting that in place, but idk yet how I'd pull it off.)

013 - Tape Backup: In 1994, tape was MUCH cheaper, like a tiny fraction of the cost per MB compared to HDDs for the tape, and about half or 2/3 for the drive.
Last I looked, tape was only moderately cheaper (maybe 20-30% but I don't remember), and the drives were several thousand bucks or more.

036 - "host managed": I'm not super sure how it works, but I think the OS / software / system has to be involved with working with where the SMR data is laid out, so is generally not for general consumer use.

055 - ZFS: I've heard ZFS can be made to work with Linux, but it's beyond my level of expertise.

083 - 9200-16e: The picture is blelow the caption. Also what would create a fire hazard?

085 - testing, stitching buttons to: WHICH cheeks, the ones on my face or.....?

Also a while ago I was briefly testing this...

Spoiler

092 - mariadb on server: I'm not sure what that is (and I'm not running a server).

096 - USB stick, boot: I have a couple USB sticks, one with Windows 10 installer and another with several Linux ISOs. I've been thinking of updating the BIOS on my B550 Taichi (it's running P1.30 now). I considered buying another flash drive (but didn't like the high $/GB and slow speeds compared to higher capacity drives, and low enduirance or lack of specs), but my Windows 10 flash drive is FAT32, I suppose I could put the BIOS file on there.

108 - PATA / IDE and predecessor: IDE was renamed to PATA I believe. MFM / RLL, or ST-506 / ST-412 is what predates IDE.

-------------------------------------------------------------------------------------------------

A previous draft, with the sections numbered and labeled, is in the spoiler.

Spoiler

On 11/12/2022 at 7:05 AM, heimdali said:

> Okay so this is super annoying ... I've been editing this (in the
> drafts section) and at least a few times now I've gotten most of the
> way through, then made a mistake and hit Ctrl+Z to undo ... and it
> didn't just undo that last little mistake I did, it undid almost
> EVERYTHING I'd been working on.

Yeah I know how that goes. The forum needs good emacs integration.
Mail-mode is your friend. I think I already said quite a while ago
that it needs a mailing list ...

► 001 - emacs / mailing list

Ahh, it's been a while since I've participated on a mailing list, but I remember the ">" characters for quote replies. (I've also seen other forums that allow you to nest multiple levels of quotes ... I was manually doing that in my draft for my previous reply, although I took those out of the final post.)

On 11/12/2022 at 7:05 AM, heimdali said:

> Also some of my replies to various points were getting pretty long.
> I left them in spoilers in case you want to see more details, but I

Spoilers?

► 002 - spoilers

Spoilers are those boxes that say "reveal hidden contents". People use them for various purposes, I'm using them so that longer blocks of text are initially visually compressed, if you want to read them you click/tap "show hidden contents". (The post would have been even longer if I had left them readily visible.)

On 11/12/2022 at 7:05 AM, heimdali said:

> put some condensed comments outside the spoilers. (There may be a
> couple other things in spoilers that are pointed out, those were
> originally going to be in spoilers anyway even if I didn't put the
> other things in spoilers.)

Huh?

There's no way I could edit this on the web site. I've copied it into
emacs so I can edit it in mail-mode, and I guess it'll best to paste
it later as code.

In case it becomes difficult to reply to, let me know and I'll edit
the post and paste it as not-code.

► 003 - emacs / mail mode

It took a bit of effort, but I was able to take it out of the code box and prepare / format it to reply.

On 11/12/2022 at 7:05 AM, heimdali said:

> For now, re: the original post topic ... I think I'll get a couple
> 18TB WD Ultrastar HC550 (0F38459 / WUH721818ALE6L4) drives, they're
> available from B&H for $320 each.

Ah that's what I thought last night when I was looking out for early
black friday offers.

→ 004 - black friday, 18TB $320 ea

On 11/12/2022 at 7:05 AM, heimdali said:

> (I was also considering 20TB WD Ultrastar HC560 (0F38785 /
> WUH722020BLE6L4) for $380 each from Newegg, but that's a pretty big
> jump in price for just an extra 2 TB, so I think I'll go for the
> 18TB drives.) I just need to order soon, as they do Sabbath there
> (they're Jewish) and their checkout is closed from Friday sunset to
> Saturday sunset NYC time.

Hm, I found a seller that apparently has newer releases of the same
drive, and the newer ones cost like 150 more. So it's all relative.

► 005 - newer release, same drive
I think I've seen a couple things like that, at least on the 20TB drives, there's 2 versions of the same 20TB HC560 or whatever it is, and one site said on the older one's page that there was a newer one, and linked to the other one.

On 11/12/2022 at 7:05 AM, heimdali said:

> Update: Ordered 2x 18TB 0F38459 from B&H earlier today (a few hours
> before sunset in NYC), hopefully they'll be here next Wednesday
> (16th).

nice

► 006 - nice, after ordering 2x 18TB

Turns out I wasn't early enough for it to ship out before Friday sundown, but hopefully it'll still be here by Wednesday. (Earlier when I was looking on Newegg on the 10th, it was saying estimated delivery Nov 11, then when I refreshed the page later (I think Friday morning or so) it said estimated delivery Nov 14.)

Update: Came this afternoon (Wed, 16th), now gotta get my system ready to install them and reboot to Linux so I can start doing some data transfer. (Windows might not see some of the files, and there's at least one ext4 partition on one of the drives.)

On 11/12/2022 at 7:05 AM, heimdali said:

> > Yes, that's called fake raid. When you use that and your
> > mainboard fails, you will have to find one that can read the
> > disks, so using fake raid isn't an option.
>
> Ahh. (And I'm guessing real raid would be with a hardware card like
> an LSI 9211-8i or similar?

Something like that, I guess. Hardware RAID should have it's own
cache and a backup battery.

► 007 - hardware raid

Ahh, I've heard that HW raid is on its way out though, and raid is more done in software or something, but idk.

On 11/12/2022 at 7:05 AM, heimdali said:

> I've thought in the past about getting one, or a 9200-16e or, if I
> found a good deal on one, a Highpoint Rocket 750 so I could add more
> drives, but then my problems would be not enough SATA connectors on
> my Corsair RM850 PSU or 3.5" bays in my Fractal Define R5 case.)

That's one of the reasons why you need a case with drive bays and a
backplane. I have one of these wonderful Chenbro cases with 16 bays.

► 008 - case, bays, backplane.
Chenbro, I've seen a couple, but those seem to mostly be rackmount.

On 11/12/2022 at 7:05 AM, heimdali said:

> > It's also not an option to store data on only a single disk. When
> > the disk fails, you may loose your data, and it's a big hassle.
>
> Yeah... I plan to buy two disks of the same size, one will be for
> the data and the other will be for the backup.

Why don't you use RAID1 with these disks and do something else for
backups? When you don't, you'll have the big hassle when a disk
fails, and especially new disks can fail. If they run for 3--4
months, they should be fine; after that, they fail not long after the
warranty has expired and if not, they're likely to last until they're
being replaced by disks with more capacity, with some exceptions.

► 009 - RAID1, extra backups, last until replaced with more capacity, etc.

"replaced by disks with more capacity" ... I like to be able to replace several smaller disks with a single larger disk that's equal or larger than the smaller disks' combined capacities, for about the same price that I paid for one of the smaller disks a few years earlier.

On 11/12/2022 at 7:05 AM, heimdali said:

> But, I'm a bit wary of "fake raid" so I guess I won't be running
> RAID1, which, among other things, eliminates Seagate drives from my
> shortlist.

Do you think Seagate disks would fail even sooner then they usually do
when used in RAID arrays?

► 010 - seagate failure, RAID, etc

I just don't trust them really, if I was going to try one, I'd want a non-Seagate fallback in case the Seagate failed.

On 11/12/2022 at 7:05 AM, heimdali said:

> I'd want some kind of setup that is platform, OS, cotnroller,
> etc. agnostic, so no matter what I plug it into, I'd be able to read
> it, unless of course the specific drive itself had died.

There's no such thing. The very disks themselves do have either a
SATA or a SAS interface (unless you use fiber, but I don't know about
that and it's still an interface) and without that, you can't access
them. They come with a controller and firmware without which you
can't access them. The list goes on from there ...

What you can do is use hardware that is easy to replace, or you keep
spares. Like I have two RAID hardware cards in my backup server
because each can only handle 2x4 disks. These controllers have been
somewhat abundant and are still readily available on ebay starting at
$30 for the basic card. They are also somewhat compatible to other
models of the same line, so if I need to replace one, chances are that
even a different model of controller could access the disks. Then,
it's somehwat unlikely that both would fail at the same time, so if I
had to, I could use the controller that still works. It's brilliant
because I have a spare without actually needing one

If the backplane fails, I can get mini-SAS to SATA cables and connect
the disks without the backplane.

Thing is you need to use decent hardware and not mess around with
crappy stuff, like stuffing a bunch of hard disks into a case that
doesn't even have bays with trays and no backplane. You can do it
with like 4 disks, but beyond that, it becomes a mess of cables, tends
to be difficult to cool and just sucks.

Decent hardware makes live so much easier, and live is way too short
for bad hardware. Seeing your pictures, you need to get a rack and
some nice 19" cases ...

► 011 - server, rack, etc

fiber interface - I've heard of it but never actually used or seen it.

$30 RAID cards ... I've seen those, for example the LSI 9211-8i, although when I was looking into things like that, I was seeing people recommend making sure they're in IT mode to pass the individual disks to the OS, like for FreeNAS/TrueNAS or similar. I've also occasionally seen the 9200-16e fairly cheap as well which supports 4x4 disks (but via external ports, not internal), and once in a while I see the Highpoint Rocket 750 not as cheap, but still cheaper per disk it supports or at least similar, and that supports 10x4 disks. (But then there's no standard tower / microATX / miniITX case that can mount that many 3.5" drives, that I'm aware of.)

Backplanes from what I can tell are crazy expensive. IF I was going to spend a lot on "decent" hardware, I'd need to spread the cost over a much longer period of time - for example, the case, motherboard, etc. would last through at least 3 or maybe 4 PSU swaps (because the Seasonic Prime PSUs with 12 year warranties died of old age). Also I'd need to wait a while before buying (since I absolutely will NOT use credit / borrow money to pay for it, would be paying cash / debit / similar) - for example, getting a DDR2 or DDR3 based setup around the time DDR5 is getting old and DDR6 is coming out. (But then there's the issue with power consumption with older hardware, but if I could keep the CPU+GPU combined under about 35-50 watts....)

On 11/12/2022 at 7:05 AM, heimdali said:

> > You really need to use redundancy for your active disks. I also
> > recommend it for backups, but when one of the backups fails, you
> > can make a new one and it's not too much hassle since you have to
> > make backups anyway if you have at least the minimum of 2
> > generations of backups.
>
> Ah, would redundancy be RAID1, or something else? (Also my system
> as-is only supports 8 3.5" drives, which I'd prefer to all be
> available for storage.)

Yes, there are lots of RAID levels that provide redundancy. ZFS is
doing it's own variants in software.

► 012 - RAID, ZFS, redundancy

Ahh, I've figured Z1 is like RAID5, Z2 is like RAID6, and there's Z3 which is 3-disk parity but I think RAID7 is something else if it's a thing at all. (I've also heard of RAID50, 60, 10, and a couple others but am not very familiar with them. (I'm aware of a Wikipedia article but I'm sure there's some they left off there as well.)

On 11/12/2022 at 7:05 AM, heimdali said:

> > Hard disks are anything but cheap If you have the disks,
> > there's no reason not to use redundancy for backups as well.
>
> "HDDs aren't cheap"... true I remember when tape was a tiny
> fraction of the cost per capacity compared to HDDs, for example in
> 1994. I was wanting to see a backup solution today that has a
> similar price ratio. (See the spoiler for more details, but you
> would be able to get a 16TB tape for ~$21, or a 128TB tape for ~$63,
> and the tape drive would be about 1/2 to 2/3 the cost of a
> similar-size HDD.)

How did you get tape drives that cheap? Tapes were basically done
when the capacity of DAT drives couldn't keep up, and those were
expensive and were wearing out all the time. The tapes weren't too
bad, though. Anything better than a DAT drive cost a fortune.

I do wish we could just use tapes for backups still.

► 013 - cheap tapes

That was in January 1994, when tapes were much cheaper than HDDs for the same capacity. A 250MB HDD was $215, a 250MB tape drive was $145, and media was $20. A 2.1GB HDD was $1850, 2.3GB tape drive was $925, and 2GB media was $59.
I haven't looked recently, but last I checked, tape was only slightly discounted per TB compared to HDDs (maybe 20-35% or so, idk), it would take 2 or 3 tapes to back up a single large HDD, and the tape drives were several thousand dollars or more.

On 11/12/2022 at 7:05 AM, heimdali said:

> Hmm ... "redunancy for active disks" ... wouldn't that require
> RAID1, or is there another way?

Sure, you can use different RAID levels.

► 014 - RAID levels

Yeah, I'm aware of RAID5, 6, or Z1, Z2, Z3, and I know there's others but haven't looked into them much.

On 11/12/2022 at 7:05 AM, heimdali said:

> (And while my case+mobo+PSU does support 8 3.5" devices, I'd prefer
> them to be all available for storage and not have to lose some to
> redunancy, etc, and even then I'd probably be running out of
> bays/ports.

The disadvantages of loosing your data, of the hassle and of the
unwieldyness of having to deal with so many individual disks all the
time instead of being able to use a larger volume outweigh the cost
of, at the minimum, using a single disk for parity.

Look out for a decent 19" case with 16 bays (or more).

> Also what about some of the pitfalls of, say, fake raid....)

I've never used it. What kind of reliability and performance do you
expect? There's reasons why RAID controllers have backup batteries,
and what does fake raid have?

► 015 - fake raid, reliability/performance
For performance, all I expect is single-HDD performance. (If I want more, I'll use an SSD for those situations.)
For reliability, I want something that if a disk fails I can replace it and rebuild from parity or mirror or something, and if more disks fail than I have for parity, I don't lose the entire array/pool, I only lose the data on the extra disks, like UNRAID.

On 11/12/2022 at 7:05 AM, heimdali said:

> "Hard disks are anything but cheap" Tell me about it Actually I
> was looking at the cost of backup media vs HDDs in an old magazine
> ad I have from Q1 1994, and... A 250MB HDD was $215, 215MB tape
> drive was $145, and 215MB tape media was $20. A 2.1GB HDD (SCSI,
> though), was $1850, a 2.3GB tape drive was $925, and 2GB tape media
> was $59.

Yeah it's never been cheap.

→ 016 - cheap HDDs

On 11/12/2022 at 7:05 AM, heimdali said:

> Today, there's a 16TB Toshiba MG08 HDD for $230 from a 3rd-party
> Amazon seller.

Are those any good?

► 017 - Toshiba 16TB MG08, any good?
I have a couple 14TB and a 12TB MG07, they've been fine so far. (I don't remember where I heard it a while ago, but I heard that when HGST was sold to WD, some of the technology / equipment, like for 3.5" HDDs I think, went to Toshiba, but I don't remember for sure how it all got sorted out.)

On 11/12/2022 at 7:05 AM, heimdali said:

> Calculating similar ratios, etc, if tape backup was available today
> like that, I'd be able to get a 16TB tape drive for $155.12 and a
> 16TB tape media for $21.40. Or, there would be a 134.4 TB SAS
> (that's the successor to SCSI, right?) HDD for $1970.07, a 147.2 TB
> tape drive for $989.53, and 128 TB tape media for $63.12. (It would
> be nice to be able to have backup media for which I could fit
> several HDDs worth onto a single backup.)

yeah

→ 018 - "yeah", to backup media that fits several HDDs

On 11/12/2022 at 7:05 AM, heimdali said:

> Yes, I know tape is very slow for random access (I imagine it
> basically making a HDD look like an NVMe SSD), but it would

It takes a while to seek, but that's ok for backups.

→ 019 - tape takes a while to seek, ok for backups

On 11/12/2022 at 7:05 AM, heimdali said:

> primarily be for full backups and restores. (And one idea I've had
> would be to have some kind of secondary SATA SSD or flash drive or
> something that just stores the data of where things are on the tape,
> and maybe short previews of things, for example the first couple
> pages of text files, several second short clips at 24 kbps 11kHz
> mono for audio or 240p 15fps q=28 or so for video, and you'd use
> that to find what you're looking for, queue up everything you wanted
> to restore, THEN hit "go" and it would arrange the queue
> sequentially then go through and pull everything off the tape.

You still have to seek to where the file is, and that means you must
read the index on the tape or whatever the backup software used to
figure where to start reading some file. The backup software had to
keep some information as to which tape to use and where the file might
start on it. Tar works fine, it was made for that kind of thing

→ 020 - tape, seek, tar

On 11/12/2022 at 7:05 AM, heimdali said:

> On the other end of the slow vs fast spectrum ... I wonder if we'll
> ever see HDD duplicators that can duplicate HDDs as fast as
> commercial optical disk stamping machines can duplicate
> CDs/DVDs/Blu-Ray...)

That might not even be possible if we had replicators like they have
in Star Trek.

→ 021 - stamping CDs, vs HDDs, star trek replicators

On 11/12/2022 at 7:05 AM, heimdali said:

>> I have to order everthing online [...]
>
> Ahh. Well my local Fry's went out, and now pretty much all we have
> is Best Buy and WalMart. If I want an actual computer store I have
> to go to Micro Center about an hour and a half away, and I usually
> only go there if I'm planning to get multiple things, or if I'm
> going to be in the area for something else anyway. (At least it's
> on the side of Los Angeles area closer to me, so I don't have to
> deal with as much traffic as if I had to completely go through L.A.)

It's so much easier to order online. And the attendends in stores
tend to be complete idiots and I'm better off ordering online anyway
because they have no clue what they're doing. I've had one standing
right in front of a wall display full of headsets/-phones I was
looking at, and when I asked for a headset I could use with a phone,
he said they don't have headsets. I went home and ordered something
online instead.

► 022 - online vs store

I've been to a store where... well just take look at the caption in the pic, and what's below it...

maybe paste pic of Fry's, ultimate gaming, 210, HD 5450, 8400 GS

On 11/12/2022 at 7:05 AM, heimdali said:

> > Besides, I remember times when there were interesting stores in
> > town [...]
>
> I remember those times too, although in my case I think it was all
> the online ordering and things that eventually shut them down.

Oh they closed well before there was internet.

► 023 - stores closed before internet

QUIte a few stores here were still around when the internet was getting started in the mid/late 1990s, but I noticed they were going away in the early/mid 2010s. Several years ago there were only 1 or 2 local ones left that I was aware of, and at least one of them closed down, idk about the other one. There were also a couple that had changed from retail to consulting or something like that.

On 11/12/2022 at 7:05 AM, heimdali said:

> But, there were times when my brother said you could stand on some
> street corners in the area where most of those stores were, start
> throwing stones, and probably hit like 8 or 10 computer stores.

There were never that many around.

► 024 - never that many stores around
In San Diego there were probably a few dozen or more.

On 11/12/2022 at 7:05 AM, heimdali said:

> In the spoiler are some links to photo albums of old magazine ads
> from various stores we had in the San Diego, CA area, from 1990 to
> 2007. (I have quite a few more magazines still in my paper-and-ink
> collection, but am missing a few; I remember having some as old as
> 1987.)

Cool, maybe scan them and make them available to museums or so so they
can be preserved for generations to come

IIRC, here you would find ads like that only in computer magazines,
and only a couple pages of them.

► 025 - scan magazine ads, museum, etc
I've actually wanted to find a way to do that sometime, but I don't have the proper equipment to quickly scan entire magazines or stacks of magazines at a time, I'd have to do it one page at a time and it's too big a project for me. (I'd like to get rid of my paper copies except for a few, to clear up the space they take up.)
Also while I have a couple file boxes of them, I don't have all of the magazines from that particular series. I think the original publisher might still be around (magazine was Computor Edge, published from 1983 to 2007 or so, started as Byte Buyer until about 1988, he is Jack Dunning I think), he may have a blog or something on a different topic (I think something like auto hotkey but I forget)... I haven't tried to contact him but I almost wonder if he might still have the original files from which the print copies were made, and maybe there's a way to put the entire collection online or something... Also I've heard that at least one or two authors of "For Dummies" books had done regular articles for ComputorEdge back in the day.

On 11/12/2022 at 7:05 AM, heimdali said:

> The cat on the laptop pic at the end of some albums is just a
> placeholder, or to remind myself that I've reached the end of the
> album. [...]

After all so much reading, it's not surprising that it's tired

→ 026 - cat after all reading tired

On 11/12/2022 at 7:05 AM, heimdali said:

> Also I'll include an album of purchase invoices we've saved. (I
> removed personal info before posting.) This includes the first PC
> my dad ever bought, a 286-10, 640k RAM, EGA graphics, 40MB HDD, etc,
> for around $1800 in January 1989.

I looked at it and I thought "wow, what a luxury!" If you wanted
to spend money like that here, you had to figure out where you could
buy something like that, and the store had to order it for you. A
couple years later, you could find something in larger cities, but not
long after that, they worked harder on removing parking and the stores
closed.

→ 027 - luxury, sort order, larger cities, closed, etc

On 11/12/2022 at 7:05 AM, heimdali said:

> I cut my computing teeth on that PC at home, although maybe a year
> or two (not before Q3-1986 cause that's when I started kindergarten
> / school) before that my parents would take me to a home education
> center where they had Apple computers, not sure what model they
> would have been though.

That's amazing Back then, there were only a few computers around
here, and they didn't really want anyone to use them because they were
expensive.

The thing is that back then, everyone kept saying you have to learn
about computers, especially when you want to get a job. Nobody cared
and nobody learned except for a handful of people, so the computers
took over and the jobs were removed.

→ 028 - few computers here, expensive, learn, job, etc

On 11/12/2022 at 7:05 AM, heimdali said:

> > I have no idea ... it didn't occur to me that anyone wouldn't
> > order them online [...]
>
> Ah. As I said / hinted earlier, I typically order online, but might
> go to a store in certain situations. [...]

Dunno, for a time, in some cases, it seemed advantageous to buy
something locally because you could go back and have them fix it if
necessary. It became exceedingly difficult to do that, and it has
never been a real advantage to do that.

→ 029 - advantage buy local, fix, difficult

On 11/12/2022 at 7:05 AM, heimdali said:

> (I used to only buy at stores though, cause I was afraid of damage
> in shipping, but I guess me driving home with the HDDs isn't that
> big of a difference or might even be worse considering i'm not
> exactly a slow driver.)

Yes, and if it happens, what's more annoying: Going back to the shop
and take the risk of them telling you you broke it yourself or
returning it? Here we have the right to return things that we bought
without seeing the seller in person, like everything we ordered
through the phone or online, within 14 days after it arrived, and we
don't need any reason at all. So if your hard disk arrives broken,
just return it and take the chance that they won't plug it in to see
if works It's probably more trouble telling them that it's
broken. And you can always say it worked before you returned it
... They might be unhappy with returned media, but if anything fails,
you can always withdraw the money through your bank account and put
them into the position that they would have to actually sue you, and
that's a pretty bad position for them. But I never had to do that,
yet it's a last resort.

I'm picturing it: You drive an hour to the store and drive around
another half hour to find a parking spot. You walk a mile or two and
finally buy two disks. You walk all the way back and drive all the
way back home and the day off you had is over.

After about a week, one of the drives fails. You take another day off
and drive for an hour to the store and drive around to find a parking
spot. After 3/4 hours you give up and drive back home (yeah I
actually had to do that and I basically quit going to movie theaters
mainly because of it even when there still were some worthwhile going
to, long time ago) and watch some movies for the rest of the day.

So one day, you finally make it to the store and they are willing to
return the disk for you and they'll call you when they got the
warranty case resolved in six weeks or so, and when that happens, you
can come back and pick up the disk. Maybe even that works out for you
somehow.

Three weeks later one of the disks fails again --- perhaps
not very likely, but you could have bought Seagate. There you are,
more days to waste with driving back and forth, with trying to find
parking and more miles of walking.

Where's your buyer protection you could have through paypal buying
online when you're buying in a store? Is the store going to pay you
for your time and for all the driving and are they going to somehow
give you the days you had to take off of work back? With fuel prices
as insane as they are nowadays, anywhere you drive is an emergency you
haven't been able to avoid.

You still consider buying in stores?

► 030 - ends in "You still consider buying in stores?"
Not as much, but for some things maybe. I wouldn't drive an hour and a half just to buy something cheap for which it costs me more in gas than the item of course but I did buy an AMD CPU at Micro Center cause the cost of the CPU there AND the gas would have been less than the cost of buying it on Newegg or Amazon, at the time. (It's cheaper now of course, but I've already had the CPU for almost a year.) Also it only took maybe a total of a minute or two to find a parking spot and walk into the store, they have a decent size parking lot right by the store. (It wasn't Black Friday or GPU launch day when I went though, I've heard it can get crowded then.)

On 11/12/2022 at 7:05 AM, heimdali said:

> I've thought about it, but right now I don't have any physical place
> to put a server, never mind any kind of rack cabinet or anything
> like that.

You have lots of pictures with many tower cases and piles of disks
laying around all in a mess. That takes at least as much space as a
rack, probably much less space when you get a 42U rack. The server
goes into the rack on top of the UPS, and the disks go into the server
and you end up with both more space and more disk space and your data
is more secure.

► 031 - tower case, piles of disks, rack, etc

but where would I even PUT a rack?
(show pics of cluttered house)

On 11/12/2022 at 7:05 AM, heimdali said:

> (Also when I was considering building a NAS / backup server, one of
> my criteria was having the entire cost of the setup, not counting
> the storage, be less than the cost of a single HDD.)

Well, you just spent like $600 on two disks you might not even need.
You may be able to find both a sever and a rack for less than that.
With a server with a ton of RAM in it you can experiment with
deduplication with ZFS, or you can use btrfs which apparently doesn't
need as much RAM for deduplication as ZFS. You can get the server
before getting a rack and you might not need so many disks when your
data is deduplicated.

The more disks and data you gather, the more unwieldy and expensive it
gets. Deduplication doesn't cost you.

It's not like there aren't options:

→ 032 - server, zfs, youtube link

On 11/12/2022 at 7:05 AM, heimdali said:

> > You have way more capacity then I do and kept your data since the
> > 80ies, so of course you do qualify.
> [...]
>
> I thought to qualify for "data hoarder" I had to have never deleted
> anything, ever (I have deleted some things), and I had to have more
> capacity than any system available for purchase (either complete
> system, or parts) could support.

Just talk to HP or Dell and the like. They'll provide you with
something that can store more data than you could ever gather during
your whole life.

→ 033 - store more data, entire life

On 11/12/2022 at 7:05 AM, heimdali said:

> (For example, let's say I got one of those dual-socket Supermicro
> motherboards with 11 PCIe slots, and put a Highpoint Rocket 750 in
> each slot.

I doubt that would work. I have an old RAID controller here which I
have retired that would support up to 256 drives through using SAS
extenders.

► 034 - RAID, support 256, SAS extenders
Yeah but I"ve heard that extenders / splitters / whatever onl still limit you to the bandwidth on the original ports, hence I don't like to use them if I don't have to. (For example if you used them to plug 256 drives into what ended up being 2 ports (4 drives each), you'd still be limited to the bandwidth of 8 ports. Maybe with 16 or 32 drives it wouldn't be a big deal cause they individually fall short of saturating a port's bandwidth, but with 256 of them that would be an issue, I think.

On 11/12/2022 at 7:05 AM, heimdali said:

> Each of those cards supports 40 drives, times 11 is 440

only 40

→ 035 - "only 40" drives on a card

On 11/12/2022 at 7:05 AM, heimdali said:

> drives, times 22 TB per drive (I'm ignoring the WD 26TB host-managed
> SMR drive)

host-managed?

► 036 - "host managed"

I'm not sure how it works, but I think the OS / software has to be involved with dealing with where the SMR data is, whereas with "drive managed" the drive itself takes care of it. I've heard that only enterprise / server level software / OS's are able to deal with host managed stuff, not consumer level like Linux / Ubuntu, Windows, Mac OS, etc.

On 11/12/2022 at 7:05 AM, heimdali said:

> would be 9.68 PB on that system. That doesn't account for
> on-motherboard ports, or using bifurcation / splitters to, for
> example, plug 8 PCIe 2.0 x8 cards into a single PCIe 4.0 x16 slot,
> or whatever would be the eqiuvalent bandwidth. A data hoarder, I
> thought, would have more than that )

Nah, it doesn't really matter much data you're hoarding. You're
already hoarding a lot of data and a lot of disks. What are you /not/
hoarding, i. e. how much do you delete?

It doesn't matter anyway. What matters is that you find a good
solution for you.

► 037 - hoarding data, delete, solution

I probably delete less data than I should. I hope to find a good solution. Also for some things like family/friends documents, photos, audio, videos, etc, I'd like to someday find a good long-term archive solution that would still be usable several decades from now or longer. When I've passed away I'd want people who are still around to be able to access those things without having to go through any special fancy rituals.

On 11/12/2022 at 7:05 AM, heimdali said:

> Ah, I do plan to buy two disks, but idk how I'd set up real-time
> redunancy if I won't be running somehting like RAID1. (What I've
> done so far is have one disk of each set of 2 be my data, then the
> 2nd has stuff manually copied to it then it's unplugged for a while
> until I need to update or restore, and I'm way behind on updating.)

Use a different RAID level maybe? When you have disks in sets of two
(or another even number) of the same size, you're fundamentally well
prepared to use some kind of RAID.

Perhaps take a look at a basic overview like this one:
https://www.geeksforgeeks.org/raid-redundant-arrays-of-independent-disks/

That would be classic RAID like with hardware RAID controllers or
software RAID with mdadm. And there are some more RAID levels and
variations.

→ 038 - RAID levels, hw/sw, etc

On 11/12/2022 at 7:05 AM, heimdali said:

> >    Using a warranty usually would require that you return your
> >    disk to someone. There is no way that I would give my data out
> >    of hands.
>
> True ... but how else would I deal with a failed drive if I wanted
> to replace it without having to buy another one.

I don't know, I considered it one time but it seemed to be too much of
a hassle to tell the seller I'd crush the drive with a fork lift or
something before sending it in if they would want me to send it back.

To get on the safer side, you could run the disk for 3--4 months
without using it and assume it might not fail before the warranty has
expired. But nowadays, the warranty is 5 years.

Or you could encrpty the data. It's a bit unwieldy and may cost
performance, but at least you can make use of the warranty even if you
need to return the disk.

► 039 - warranty, failed disk

Ahh, yeah I'm aware of the pitfalls of data on a disk being sent back, it would be nice to not have to deal with that. Also in most cases I haven't been encrypting data, except for one of the Linux VMs on my laptop. (I want to be able to access the data without having to deal with that, or if something happened to me I'd want friends and family to be able to access it, even if something simultaneously happened to everyone that might have the credentials to bypass encryption.)

On 11/12/2022 at 7:05 AM, heimdali said:

> Yes, there's the issue of the data on it, but that's what backups
> are for.

Backups won't prevent someone from reading your data from a disk you
returned under warranty.

► 040 - warranty, backups not prevent others reading data

Well that aspect of it is true, but it would still let ME have the data right away and not have to wait for the replacement disk to come back. (Also they would just send a blank disk back anyway.)

On 11/12/2022 at 7:05 AM, heimdali said:

> (Backups, however, won't save you the cost of a new HDD when you
> need to replace one that died prematurely, afaik.)

right

→ 041 - backup save cost new HDD "right"

On 11/12/2022 at 7:05 AM, heimdali said:

> I'm pretty sure warranties don't apply buying used parts from
> private sellers.

That depends. When you have a switch from HP with a lifetime
warranty, and lots of them have one, they'll send you an
as-good-as-new replacement even when the switch is like 15 years old
and you bought it on ebay for $20, at least when you're a company that
returns it. Fans don't last forever ...

I don't see how it should matter who you bought something from. Who
the seller is doesn't alter the product in any way.

► 042 - switch HP life warranty, replacement, seller doesn't alter product

The seller may not alter the product, but I thought buying something from the wrong seller could affect the warranty? Like, "We're sorry, but you didn't buy this from an authorized retailer, so the warranty is void", or "The product you purchased was either refurbished, or intended to be part of another system, or shucked from an external enclosure, so contact the seller for warranty."

On 11/12/2022 at 7:05 AM, heimdali said:

> Also I've heard that they may not apply when buying refurb parts
> from non-authorised sellers (in those cases the sellers are supposed
> to provide the warranty, same if you bought a complete system with a
> drive inside it from a retailer).

Claiming you could refurbish a hard disk is bullshit. There's no way
that a seller who got a hard disk could somehow improve its
condidtion. Such a claim is fraud and I'm surprised that ebay, for
example, tolerates that.

What's the seller gona do to it? Change out the bearings and other
parts that might have experienced some wear? Replace some capacitors
that may be on the controller board? That would cost way more than a
new disk.

All they could do is blow off the dust and delete the data. Deleting
the data makes the disk worse through usage. It's been a long time
that you could actually format a hard disk. Perhaps you still can
when you have the right equipment, but I'd like to see the seller who
has that. Back then, you could also drive the reading arms all the
way over the platters until they hit the limiter and cause the disk to
make bad noises.

Maybe that's what they do when they're refurbishing them, drive the
arms all over into the limiters through the use of some special
equipment and when the disk makes funny noises it's still good to go
for sale.

And other parts? What's there to refurbish? You can pick the good
parts from the hardware you have and use that to put something
together that works, but I wouldn't call that refurbishing,
either. The parts remain unchanged and you're still selling a heap of
old parts that still work.

The broken parts that remain are the ones you can refurbish buy
recycling them to create new parts.

As to warranty, there are legal requirements here. Of course, when
your hard disk fails after 4.99 years, the seller may not be around
anymore. If you buy solar panels with a warranty of 25 years and have
them installed on your roof and one fails after 22 years, whoever who
installed them may not be around anymore; neither might be the
manufacturer.

Just check what warranty you get and what the chances are that you can
make use of it before you buy.

► 043 - refurbish HDD

yeah, to me, "refurbish" an HDD... whaaat? at most that would involve wiping the disk, MAYBE erasing some smart data like power on hours, etc (and there would be problems with that too), basically only "software" "refurbish". Only the manufacturer themselves or MAYBE someone like DriveSavers or another data recovery place capable of opening up a drive in a clean room / flow bench could replace platters, heads, motors, etc, and I imagine they'd rather sell a new drive.

On 11/12/2022 at 7:05 AM, heimdali said:

> > What I meant is that if I were to buy 16TB hard drives, I'd have
> > to buy at least two of them and that would cost me over EUR 1000.
> > Discs always come at least in pairs.
>
> Ahh, one for data, one for backup basically.

No, it's for redundancy. Backups are extra. (Fortunately, I found
chaper ones, but still ...)

► 044 - backup, redundancy.

Are you hinting / saying something about THREE disks for each disk of data or something? One for the data, one for redunancy, one for backup? Or something else?
I thought the 3 disks was for something like .. 3 disks, one with your data, one as a local backup preferably on a different media, one as an off-site backup. (Or basically the 3-2-1 backup strategy.)

On 11/12/2022 at 7:05 AM, heimdali said:

> "Discs always come at least in pairs" ... maybe it translates to
> English a bit different han whatever your native language is, but to
> me, what I imagine is that it's not possible to buy one disk at a
> time. In reality though, at least here, you CAN buy one disk at a
> time if you want to, but it's much more advisable to buy two.

Well, yes, in theory, I can buy a single disk. But what would I do
with it? It's not that I couldn't buy one, it's simply not an option
to buy only one because I won't have redundancy, and without that, I
can't store data on it and it's useless other than for backups maybe.

So you might think sure, only $300 for 16TB is a good price and not
expensive. Actually, it costs $600, and how much data you can store
depends on the RAID level. With only two disks, that's only 16TB, so
16TB actually cost $600. What else did you think?

→ 045 - theory buy single disk, but no redundancy, actually 2x cost per disk because redundancy, etc

On 11/12/2022 at 7:05 AM, heimdali said:

> > You can always check with hdparm. [...]
>
> Apparently hdparm isn't on here, and idk if it's available.

You said something about having Linux somewhere.

► 046 - hdparm, linux

I have Linux on a secondary SSD in my desktop, and in VMs on my laptop, but I mostly boot WIndows 10 Pro for now as my main OS. (In the back of my mind I've been considering switching to Linux, and I may be forced to eventually once WIndows 10's support ends if Windows 11, or 12 if it becomes a thing, doesn't allow me to create just local accounts, among other times. My brother is still running Windows 7, I think I've heard him say things like he'd switch to Linux rather than upgrade to 10 or 11.)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
>
> I wonder if other things could possible be causing issues.

like multiple issues maybe

→ 047 - other things, multiple issues

On 11/12/2022 at 7:05 AM, heimdali said:

> I just hope I don't have to rip my entire system apart, [...]

Perhaps you have too many disks in it and it's overwhelmed because of
that.

► 048 - too many disks, overwhelmed
I don't think it's how many disks I have. I'm suspecting other things but idk what.

On 11/12/2022 at 7:05 AM, heimdali said:

> Maybe; although idk if firmware updates are available or not,
> haven't checked (and I've never updated firmware on a HDD anyway
> that I can remember).

I have done it once with disks that would disappear after a while. I
had to turn off the computer to get the disk back. So I had to keep
checking every now and then if had gone away because sooner or later,
I wanted the RAID to rebuild.

Almost 10 years later there finally was a firmware update for these
disks. One of them had already failed and the other one was usless
because it would disappear all the time. The firmware actually fixed
it and I was able to finally use the disk.

→ 049 - firmware updates, 10 years, etc

On 11/12/2022 at 7:05 AM, heimdali said:

> I'm not 100% sure if it's the disks, cause other things also act up
> around the same time.

like when you have multiple issues ...

→ 050 - multiple issues

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> While I'm sure some people can take apart and rebuild an entire
> system in maybe a half hour or so, it takes me probably a full day,

It gets so much easier when using 19" cases like you wouldn't believe.

► 051 - rebuild, 19" cases, etc

Ahh I haven't had experience with those.
I technically have an audio cabinet that just happens to be 19" wide internally, but not with rails, and it's only about 13-16" deep or so, but the back end is open. No way would I put a 30" deep case in it though.

On 11/12/2022 at 7:05 AM, heimdali said:

> and don't even get me started on configuring my OS, software,
> etc. the way I want it, or the way I had it before.)

That was never really possible with Windows. Just use Linux, then you
have package management amongst other things.

→ 052 - windows, configure, linux, package management

On 11/12/2022 at 7:05 AM, heimdali said:

> So is there a firmware update for those, or is there anything else
> special about them? Maybe broken cables?

Maybe; you could find out.

→ 053 - firmware update, broken cables

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> SATA cables that have been slightly modified with the aid of my
> cat's teeth, one had its retention clip broken off, etc.

Are you hoarding broken cables? Give them to recycling so you don't
accidentially use broken cables.

► 054 - broken cables
I may still have a few laying around, but I'll want to get rid of them once I've tested to be sure they don't work. (By "test" I mean plug it in and see if it recognizes a drive.)

On 11/12/2022 at 7:05 AM, heimdali said:

> > It depends ... Unfortunately, ZFS isn't compatible with Linux
> > only for stupid licensing issues (you can get it to work, but that
> > isn't a good option)

I have to correct myself again, you just have to know how to.

► 055 - ZFS, linux, compatible

I've heard it can be made to work, but ... is probably beyond my level of expertise.

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> I've never used ZFS. Had considered it for NAS/backup server, but
> would lean more toward UNRAID or similar. (I like how it lets you
> [...]

Maybe that's a good option for you then. I guess you can always try
it out and see how you like it. But how compatible is that with other
stuff, what if you want to switch to something else and can't read
your data anymore? For all you know, they might call in your license
and then you're screwed.

► 056 - UNRAID, license

yeah that could be an issue, hadn't thought of that. (I wonder if there's a way to make ZFS behave the same way, or if there's something else that's license-free, that lets me add disks one at a time, and if more than parity fails I only lose that extra data, not everything.

On 11/12/2022 at 7:05 AM, heimdali said:

> Dedup/backup ... A while ago I was wanting to back up a bunch of
> SSDs to a 12TB HDD. If I didn't use compression I could just "dd"
> to a disk image and they'd still all fit for now, but... (see in the
> spoiler).

What prevents you from compressing the disk image created with dd?

► 057 - compess, image, dd

I did compress it in at least on test I did. Problem is, it would take forever (like maybe a few dozen minutes or an hour or more) to open the image in 7-Zip, and when I tried a while ago, it was basically trying to open the entire image in RAM, and failed cause I only had 32GB RAM in that system and was trying to decompress an image of a 240GB SSD.

On 11/12/2022 at 7:05 AM, heimdali said:

> Also I'm not familiiar at all with BTRFS, or with VDO.

That means you need to learn more.

→ 058 - learn more, btrfs, vdo

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> but I also have a couple 2 TB SSDs I'd want to back up as well,
> preferably compressed, and I don't have 2+ TB of RAM (or want to
> take the time) to decompress the image file before I can access its
> contents.

That seems like a rather unwieldy way to make a backup. Why don't you
just copy the files you want to back up? It doesn't seem like you'd
still need a functional disk image. You're probably wasting 60GB of
space with that.

Maybe use this: https://clonezilla.org/

► 059 - backup SSDs to HDD, clonezilla

Well I was wanting to have disk images that, at least for my boot drive, would be bootable, and from what I remember I"ve had trouble doing that with Clonezilla. (Maybe I'm doing something wrong though, idk.)
As for saving space, that's why I was trying to compress the "dd" images with gzip. (Maybe that's where I'm going wrong, maybe gzip isn't the right thing to use...)

On 11/12/2022 at 7:05 AM, heimdali said:

> > How well did that work?
> [...]
> HAH .... not well
> [...]
> I imagine my ASRock B550 Taichi might have a bit more DMI (or
> whatever it's called) bandwidth, maybe PCIe 4.0 x4 or 3.0 x8 but I
> forget, so it might do better with HDDs connected. (One factor that
> made me choose it was the factor that I can have all 8 SATA ports
> and both M.2 slots populated simultaneously without having ports
> disabled. I think it might cut one of the PCIe slots bandwidth in
> half if I have everything populated, but idk if I'll use that slot
> unless I plug in at least two SAS/SATA HBAs.)

Imagine what software RAID might do. That's the beauty of hardware
RAID. You can have a single RAID card to drive 8 (or even more) disks
without all the overhead and without all the copies of the data going
to each disk going all through the mainboard. Not having that may be
fine for a file server that doesn't have much else to do, but for the
maschine you're sitting at and are using for what you want to use it
for, it can be a nasty and quite noticable burdon.

► 060 - sw/hw raid, copies through mainboard, etc

"file server that doesn't have much else to do" yeah basically if I was setting up a backup server that's ALL it would do - no Plex, no transcoding, maybe not even zip/unzip or anything like that, JUST storing and serving files.

On 11/12/2022 at 7:05 AM, heimdali said:

> > Older cards were limited to 2TB.
>

> Ahh, I guess the BT-PESAPA card I had (actually still do, but only
> use it rarely for PATA) was limited to tthat, it's from around 2005
> or so I think so maybe 2TB drives didn't exist then.

yeah

→ 061 - "yeah" BT-PESAPA, 2TB

On 11/12/2022 at 7:05 AM, heimdali said:

> Also several years ago I wanted the ability to pull individual
> drives out of a system to use with a 3.5" dock somewhere else, but
> I'm pretty sure USB flash drives and external SSDs would be better
> for that now.

You can do that with SAS drives. I don't recommend it and I still
rather turn the machine off when possible, but replacing a failed disk
by pulling the failed one out and plugging the new one in without any
further ado is really nice and not something I'd worry about (because
when the disk has failed, no data gets lost). Being able to do that
is another beauty of hardware RAID. Not being able to do that can be
expensive. SATA really sucks, why couldn't they make it hotpluggable.

► 062 - SAS, SATA, hot plug

Actually one of the specs of SATA is that it IS hot-pluggable, at least for internal SATA. (I don't think I was able to hot-plug eSATA though but idk.)
On my B550 Taichi, there's an option in BIOS to enable SATA hot plug, if I remember right it's just one setting to globally enable / disable it, so I have it enabled and can just unplug and plug disks in as I wish without shutting down the system. Also if I click the "eject devices button" in the tray, there's several HDDs listed there, plus an M.2 SATA SSD but I don't think M.2 supports hot plug on my system.
Linus did a video a while ago where he was showing a server motherboard that supported PCIe hot plug. Mine doens't support that, although it would be nice to have in a future system. (Also if it supported hot plug on RAM, and if I had dual sockets, CPUs, that might be interesting.)
On my Z97 Extreme6, each of the 10 ports had its own setting to enable / disable hot plug, so I enabled them on 9 of them, but left SATA0's hot plug disabled cause that's where my boot drive was plugged in.

On 11/12/2022 at 7:05 AM, heimdali said:

> [...] (I've wondered if it was a limitation of SATA II (3Gb/s)
> though... but I think I've heard of people plugging in >2TB drives
> into SATA II ports on other systems and it was fine, but idk.

IIRC it had to do with the BIOSs (and sometimes the hardware) not able
to handle so many sectors.

→ 063 - BIOS, hardware, sectors, limits

On 11/12/2022 at 7:05 AM, heimdali said:

> Interesting thing though, the BT-PESAPA's boot spalsh text doesn't
> pop up on my AM4 board,

That's because EFI sucks. There are tons of issues with older cards
in newer boards.

► 064 - EFI sucks, older cards, newer boards

I wonder what would be a good solution then. I dream, sometimes, of being able to plug in an MFM hard drive and 5.25" or 8" floppy drive into the same system that would also have a U.2 (or is U.3 the newest server version or something) and M.2, among other things. (I've heard of "industrial" motherboards that had ISA and PCI Express slots, maybe with LGA 1150 or 1151 or something like that.)

On 11/12/2022 at 7:05 AM, heimdali said:

> and I can't see devices plugged into it from WIndows, but I can see
> them from Linux.

perhaps there's no module that would support the card

→ 065 - windows, linux, see devices, no module support

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> Also I mentioned earlier about liking UNRAID in that drives could be
> added one at a time.

... which probably means that it's not compatible to anything else

→ 066 - unraid, add one at a time, not compatible anything else

On 11/12/2022 at 7:05 AM, heimdali said:

> I've also wanted the ability to pull a single drive with things on
> it and take it to a friend's house for whatever, but I imagine that
> a USB drive or external SSD would these days be much better suited
> for the purpose.

indeed

Even with SAS drives, what to you expect to happen when you pull a
disk while it's been written to or read from?

► 067 - pull disk while reading/writing
Well I would try to make sure it's not reading or writing to the disk when I pull it. For example, make sure no program is accessing the disk, or if I'm copying something, make sure it's finished copying. Also I try to turn off write caching, as I don't want the progress dialog to disappear until it's ACTUALLY done. I've had issues before where I had caching enabled, then when the dialog hit 100% and disappeared and I pulled the drive it popped up and said "oops we weren't done some of your data didn't properly copy" but if I turned off caching I didn't have that issue.
(One thing though, at least on my other PC, I've sometimes seen my RAM usage spike up when doing a large copy, idk why, maybe there was some caching enabled there or something, idk; also at first it was copying at a rate that was several times faster than the maximum rated sequential transfer rate of the media. Or could that be how dramless SSDs or USB flash drives work, or whatever uses HMB - host memory buffer? I don't think I have any DRAMless SSDs that I know of, I try to avoid those.)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> > The overcommitment doesn't really matter until the RAM isn't actually used.
> [...]
> Ahh. Also something I haven't figured out is why my PC often gets
> lag spikes (sometimes small and barely noticeable like the one I
> just had where I typed 2 or 3 words before it appeared on screen,
> but sometimes also pretty big where half the system is unresponsive
> for a minute or two or more), even though I might only be using
> 5-10% of my CPU, and my RAM is only about half used. (I would have
> thought I shouldn't be getting input / processing lag spikes or
> whatever until my CPU and RAM were both maxed out.)

That's like long interruptions ... try it with only two disks in a
RAID1 and see what happens, instead of having 10 single disks plugged
in.

→ 068 - long interruptions, try 2 disks in RAID1 instead of 10 singles

On 11/12/2022 at 7:05 AM, heimdali said:

> Also just now I was editing a portion of the draft for this post,
> and... I highlighted a selection of text, Ctrl+X'ed it, clicked the
> spot where I wanted to paste, Ctrl+V'ed, then clicked another spot
> to click and drag to highlight and delete some text .... but there
> was a delay from when I hit Ctrl+V until it actually pasted, and it
> pasted the text in the other spot where I was going to try to
> delete.

So that's extreme ... Maybe you're having multiple issues ...

→ 069 - extreme, multiple issues

On 11/12/2022 at 7:05 AM, heimdali said:

> Side note, it's super annoying when websites either load elements
> that move things around on the page AFTER it has visually started to
> appear,

That could be the web browser figuring out that it needs to display
stuff differently when more of the page gets loaded.

► 070 - web browser displaying stuff differently when page continues loading
I think I remember a decade or two ago it didn't do things like that. (Sometimes I wonder if there's a more sinister reason behind it, to "prey" on people that have habits like me, remember how things used to be, and we go to click on one thing expecting it to be what we wanted, but they somehow move things around and we end up clicking on an ad (which to me is anything where they hope to get anything of value, financial or otherwise, as a result) - I've lost count of how many times that happened to me.)

On 11/12/2022 at 7:05 AM, heimdali said:

> or when, on a login page, it jumps your cursor to the
> username entry field.

Isn't that where it's supposed to go?

► 071 - cursor, username, supposed to go

Well, maybe, but not if I've ALREADY clicked / tabbed there, typed, then tabbed to the password field. And I suppose someone (I imagine my brother would be this way) could make the argument that the site should not force / suggest you to do something, the user should have total control over how they interact, what they do, etc, and I'd personally agree at least on the user having total control over how they use their PC, websites, etc.

On 11/12/2022 at 7:05 AM, heimdali said:

> I've lost count of how many times I go to click one thing and I
> click something else instead cause it moved, or, I'm in the process
> of logging in and start typing my password in clear text in the
> username box.

Just wait until it stabilizes. Or get faster internet

► 072 - wait stabilize, faster internet
Yeah, but I grew up in the era when it took minutes to boot up, AND you could start doing things before it finished booting, at least on the PC. As for faster internet... I want it, but it's not available here. Yeah, I can get 1 Gbps down, but only 35 Mbps up, and I want a minimum of 1 Gbps up. (I'd REALLY like internet that's at least as fast as my storage, or as fast as all the PCIe / RAM / VRAM bandwidth combined of all devices in the house that might ever be connected to it.)

On 11/12/2022 at 7:05 AM, heimdali said:

> I remember the days in the MS-DOS era when, while the PC was booting
> up, there was some kind of keyboard buffer or something so we could
> start entering commands and things before it had even finished
> booting. (Why a similar ability doesn't seem to carry forward to
> modern times escapes me.)

That's still there, isn't it? The keyboard does that, it least it
should. Maybe you need a better keyboard?

► 073 - keyboard buffer, better keyboard

Well for now I'm using a Logitech K270 on my desktop and K360 on my laptop. I'd like something better sometime, but I've had trouble finding anything that's actually silent. My existing keyboard is almost blowing out my eardrums from the sound of the keypresses, but maybe I'm typing too hard cause sometimes my fingers are a bit sore.

On 11/12/2022 at 7:05 AM, heimdali said:

> > It seems like you need hardware that's more reliable.
>
> Maybe. I thought the stuff I had was supposed to be at least fairly
> decent, at least if I remember right reading reviews on it, there
> weren't a lot of complaints about DOA, failures, etc that I could
> remember.

I guess it's standard consumer hardware. Since I started buying used
workstations off ebay with Xeons in them, I don't have weird issues
anymore. Sure I don't get the fastest and greatest, but it's more
than fast enough and costs a fraction of the money I'd have to spend
buying new.

It might be fun to buy some new parts and put them together, but
besides cost, what holds me back is that I don't know if I can get
something that stable that way. It's still comsumer hardware, unless
I were to buy a new workstation from Dell or HP.

And that means 'sufficiently stable'. These workstations just work.
It's amazing. I have a $2700 CPU in my workstation I payed 150 for
when I upgraded the CPU because I wanted a few more cores. Now I have
2x14 cores. I guess you get what you pay for. It redefines what
'sufficiently stable' means.

► 074 - consumer hardware vs workstations

Ah... if I had the $ I'd like to get an Epyc system sometime... but I also happen to generally prefer the aesthetics of consumer cases and things, but not all the fancy stuff like glass windows, RGB, etc. I'm satisfied with the aesthetics of my Fractal Design Define R5, for example, although I've taken the front door off it cause I hate those things. (I had a previous case with a door that I broke several years ago.)

On 11/12/2022 at 7:05 AM, heimdali said:

> I mentioned earlier I'm not considering Seagate HDDs - that's
> because of possible reliability issues I've heard from some people,
> including my brother and others.

Yeah I've seen Seagates consistantly having very high failure rates
over the decades.

► 075 - Seagates, high failure

Yeah, that's a reason why I'm wary of them, and would only use one if I was using it in a RAID1 array with another brand of disk.

On the other hand, I've read stories on various retro computing forums, at least several years ago, of people with drives like the ST-225, that were still working totally fine 30 years later.

On 11/12/2022 at 7:05 AM, heimdali said:

> I wonder if the newer IronWolf Pro or Exos drives might be okay
> though, but I would only consider one if I was running it in a RAID1
> array with a different brand drive, like a WD Ultrastar/Gold or a
> Toshiba MG-series drive. (As for that, see the comments earlier
> about "fake raid".)

Why would you buy Seagate? I wouldn't want to be forced to encrypt my
disks so I can make use of the warranty. Even then, what do I do
while I'm waiting on the replacement? Run the RAID in degraded mode
for the 6 weeks or however long it takes before I get it replaced?

► 076 - seagate, warranty, encrypt, replacement

I'd be wary of Seagate, and wouldn't use it as my only / primary drive.

idk where to put this comment but ... I basically want to minimize my cost in case things go wrong.
One thing I was thinking of... if I was using an online backup / storage that charged by how much data I was using, my maximum budget would be something like ... take the hard drive that is the cheapest per TB, accounting for warranty length. For example, if the cheapest is a $256 16TB drive with a 5-year warranty, then my maximum budget would be (250/16/5) $3.20/TB/year. (Or is my assumption wrong that big companies like google, amazon, backblaze, etc) can buy drives in bulk, like several thousand at a time, for a tiny fraction of the cost per drive or per TB of what it would cost me to buy them one at a time?)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> "Stuffing more disks", I can't really do that, 8 is my max right
> now.

A nice Dell R720 can fit that nicely. I'd go for a 12xLFF, though,
that gives you more to add more disks if you need to. And you need
something to boot from.

→ 077 - Dell R720, 12xLFF, etc

On 11/12/2022 at 7:05 AM, heimdali said:

> At this point though, I'd be replacing smaller disks with larger
> ones,

So yes, get your storage server up and running and move the data to
it. You could go the easy route and simply make a hardware RAID5 and
put btrfs on it. I was told you can use bees which would do the
deduplication in the background and keep deduplicating stuff while
you're writing.

Then you end up with a bunch of small disks which you can use for
backups. That's what I do with my old disks.

► 078 - storage server, dedup

Where would I put a storage server though

(insert pics of cluttered rooms)

(((

From

F:\Pics\from Pixel 6a\DCIM ; Camera\2022-11

PXL_20221113_045530581 = living room from entry hall, but not wide

PXL_20221113_045544802 = living room from entry hall, but not wide

PXL_20221113_045731497 = in living room between couch and stuff, looking toward computer, wide angle

PXL_20221113_045803502 = living room from entry hall, wide angle

PXL_20221113_045954563 = living room from pantry, piano on right, wide angle

PXL_20221113_050328828 = my bedroom, from door, wide angle

PXL_20221113_050412105 = in my bedroom, from near bed, toward closet, wide angle

PXL_20221113_050923618 = guest room & work room, from doorway by hall, wide angle

PXL_20221113_050951387 = spare bedroom, from doorway, wide angle

)))

also as I said, I don't know if dedup would be what I exactly want. Yes it would free up space, but my system and I would still think there's logically multiple copies of things there, and THAT is what I want to clean up, get rid of that extra clutter. Problem is which copy in which place to keep, and what to--- or, maybe I just need to completely overhaul my organization system, it's a hodge-podge of methods that hearken back all the way to when we first had a PC in 1989.... I'd like to find a method that I can set it up that way, then keep using it in almost perpetually, even when I'm 80 in 2061, or if I live that long, 90 in 2071 or 100 in 2081. (Then when I've passed away, if my kids / grandkids, or if I never have any, nephews/niece (only 1 so far) want to do things differently they can.)

On 11/12/2022 at 7:05 AM, heimdali said:

> I don't have a working UPS now.

Ugh. Get a nice 19" UPS from APC and put it at the bottom of your
rack because it's heavy. You can buy them used, it's not like they'd
go bad except for the batteries that need to be replaced every now and
then. The good thing with APC is that you can get new batteries
easily.

► 079 - 19" UPS, APC, rack, etc

I just have no place for a rack.

On 11/12/2022 at 7:05 AM, heimdali said:

> Also I'm using 4x32GB Team Expert DDR4-3600 CL18 in my current AM4
> system, but running at 3466 cause it's not quite stable at 3533 I
> think, [...]

I'm glad I have my workstations. They work.

► 080 - glad have workstations

Ahh. I'm for now limited to consumer hardware. A while ago I was thinking of getting LGA 771 or 1366 Xeons and building a NAS / backup server with them, but was having trouble finding standard (not rack) cases that would support all the drives, and then I got to thinking of the power consumption, and the factor of electricity being upwards of 35-67+¢/kWh here, and that was several years ago so who knows it could be flrting with or even past $1/kWh in some cases...

On 11/12/2022 at 7:05 AM, heimdali said:

> As for "stuffing more disks into my computer"... if there was a way
> to actually safely mount all of them AND hook them up...

That's what a decent case is for. When I see all these funny tower
cases and what not that have all these cooling issues, I always wonder
'ok and where are the drive bays?'. People put useless lights into
their computers but they don't even have drive bays, not to mention
backplanes. How would that be useful?

► 081 - decent case, cooling, bays, lights, etc

Haha yeah, I don't really like the fancy lights and stuff. (Although a while ago when I was having power issues with my system, my B550's default RGB lights did come in handy as a "power on" status indicator, although that problem was fixed when I replaced my PSU, from a Corsair AX760 that I'd carried over from the previous system, to an RM850, although I was a bit disappinted to be downgrading from 80+ Platinum to just 80+ peasant Gold.)

I remember several years ago seeing cases with drive bays taking the entire height (and sometimes even ones with a partial second stack), now it's rare to find even a full tower case that has more than 2 3.5" bays, even if it could fit an SSI EEB motherboard. (And contrary to what some people think, if it won't fit SSI EEB, it's not "EATX", or, 12" x 10.7" is not "EATX", but 12" x 13" is.)

On 11/12/2022 at 7:05 AM, heimdali said:

> (Also if I was using a low profile CPU cooler, like something that's
> not taller than the VRM heatsinks, then I'd have room for several
> more HDDs...)

There you go:

https://www.ebay.de/itm/144404111197?hash=item219f27df5d:g:9O8AAOSwrZhhZ1kA&amdata=enc%3AAQAHAAAA4Gd1O5vyvyoOW8IY7xiInNzYaIhHzWGYsB9uG%2FBy9No%2FRFycFaB2vLBVCAkTzk2mARHqw4byVGoz5WrfKjZPmHrNC9KznX%2FedCAOPPgLbQPWF3QeO6e3c%2Bo1VzQz%2FE6VlwfrKtqMOxqQspmFyh0WHfKfPIJ5OdVnB0fSgQBpawv5IeTrnCY28NPGZt71JcIcewXylV%2FF%2Bjt0ZOx%2FGA2vHzdOrLINkpbiquIBBKGXfGwIhTV8ScfySa8R401vi6LVUf%2FNisXknEewUWze14IyZvI8w2%2FuebQ5uE%2BoCxJ3f0c2|tkp%3ABk9SR6SEz7yNYQ

https://www.ebay.de/itm/284981250850?_trkparms=amclksrc%3DITM%26aid%3D1110006%26algo%3DHOMESPLICE.SIM%26ao%3D1%26asc%3D20201210111451%26meid%3D28e63db09b324ae4a36854e8ccbea38d%26pid%3D101196%26rk%3D4%26rkt%3D11%26sd%3D144404111197%26itm%3D284981250850%26pmt%3D1%26noa%3D0%26pg%3D2047675%26algv%3DSimplAMLv5PairwiseWebWithBBEV2bAndUBSourceDemotionWithUltimatelyBoughtOfCoviewV1%26brand%3DSupermicro&_trksid=p2047675.c101196.m2219&amdata=cksum%3A28498125085028e63db09b324ae4a36854e8ccbea38d|enc%3AAQAHAAABMIep8YrnpoI5Ip1Fr26ya5Zdwg2V78Yl01uYLsdPZcFTvxKxDts2M2LiSIpUM3yyPCg4I%2FI%2F%2BBr%2B3lbC%2BxYqInYGBmzIxc4EMw3YF0EDxTKyEHLT%2F%2BSMG2X2Go1miFYKtC00MzMQyW9gDfbMtyClMOXZ8xJKGtjoLzUP%2FdiLuNWupvrVbgXCTDewZ4VueuJxRUMMZnXCy%2FDutw%2FxnITm48hlHbLvuMw3l6iBYxzde1cCgdhPRpOr3GLW%2BVqnkBvK2bQ6zIKPiX2RUJokpE%2Fbl83Sl9REJhrS0BYpSBleJScnEzI6pT11MsM153kyy4ma2Rj7Q7cJZh%2BscrwME71FUisxmmg8JNI%2Bwc7NlH8p7l8kzUTYUOCx8vRWWzR%2BuBe7MuZzk1rBbjZsLaRuwoLBA58%3D|ampid%3APL_CLK|clp%3A2047675&epid=149985814

What's the problem?

► 082 - case, lots of bays, ebay.de, examples

Well those are from Germany, I'm in the USA. Also the cheaper one is about 25-35% or so more expensive than my entire budget for the system would be, not counting storage, if I was using it as a backup server.

On 11/12/2022 at 7:05 AM, heimdali said:

> This is how I might have wanted to connect extra drives if I used a
> SAS HBA with external ports, like tha 9200-16e...

I'm not sure which picture that is. Some look as if you're trying to
create a fire hazard.

► 083 - which pic, 9200-16e

The picture is below the caption, not above.
And what would be creating a fire hazard? The way I'd have the cables set up, or something else? I don't want to have wasted space in my case that could be taken up with hard drives. (If it could be as densely packed as a laptop or tablet or phone, that might be nice. I hate the full tower cases mentioned earlier that only have 2 3.5" bays, that's such a huge waste of space.)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> But, maybe it's better to have a dual-chamber case

It's better to get a decent case. Chenbro really makes great stuff.

► 084 - decent case, chenbro
Hmm... I guess I haven't found any decent non-rack / non-server cases though...

On 11/12/2022 at 7:05 AM, heimdali said:

> Sometimes I've had to run drives outside my case as well...

Ugh. For testing, ok, but other than that, it's like stiching buttons
to your cheeks.

► 085 - testing, stiching buttons,

Yeah, I was testing stuff that way, doing DBAN, etc, but I wouldn't be comfortable running it that way all the time.

Also another situation I was testing was...
(insert, in spoiler, pic of laptop booting from IDE drive, powered from desktop PSU)

And stitching buttons to WHICH cheeks? The ones on my face, or...

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> > Why more than 512? I wouldn't use that much but it kinda makes
> > sense to use sector size as orientation.
> [...]

> Okay now I see the "compare no further than the first byte that
> differs"... that's a good idea probably.

Yes, there's no point in comparing any further.

→ 086 - comparing after first byte differs

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> Also, some media files might have been en-/trans-coded with
> different settings, but are ultimately the same thing.

But the files are different.

I was planning to make some backups today. Maybe I can experiment
with deduplication with btrfs.

→ 087 - experiment deduplication btrfs

On 11/12/2022 at 7:05 AM, heimdali said:

> In some cases I might have a bunch of different bitrates / codecs of
> the same thing, and I need to pare some of those down and only keep
> the max quality / uncompressed one (a "master" so to speak), and
> maybe a couple lesser bitrate ones probably.

That'll be a lot of work.

→ 088 - (may not reply) "lot of work" different bitrates/codecs of same songs

On 11/12/2022 at 7:05 AM, heimdali said:

> Well I think it's possible that the first 512 or 4096 bytes or
> whatever might be identical, but somewhere later in the file might
> not be.

Yes, it would only serve to be able to say which files are different.

→ 089 - (may not reply) 512, 4096 bytes, later different, only serve say which different

On 11/12/2022 at 7:05 AM, heimdali said:

> Also I don't really need this to be super fast, as in, finish the
> entire job in like a half hour or an hour or so.

But I don't like inefficiency.

► 090 - don't like inefficiency / don't need super fast

True, but I also want to make sure I do the job completely. (I do hope it's not a lot slower than about an hour and a half per TB or so.)

On 11/12/2022 at 7:05 AM, heimdali said:

> I'd be totally fine if it takes several hours or even a full day or
> two or three to process. I can just use my laptop in the meantime
> while my desktop is churning through the data.

Is your computer so slow that it would be overwhelmed by gathering data
about a few files?

► 091 - "so slow, overwhelmed, data few files"

No, the 5950X isn't that slow, it's just having so many files. WinDirStat says there's about 5.77 million "items" on this PC as of a week or two ago, and that's with a "backup" hard drive logically unplugged/unmounted. (By that I mean it's still physically plugged in, but I unassigned its drive letter in Disk Management.)

This is a short video of scrolling through a little of what WinDirStat found, just scratching the surface...

(upload "F:\Pics\from Pixel 6a\DCIM ; Camera\2022-11\PXL_20221118_082644231.mp4" to youtube, then link)

I noticed quite a few folder structures that looked like they could be duplicate hierarchies while looking through that. 002

On 11/12/2022 at 7:05 AM, heimdali said:

> As for where to put the database... I do have about 650 GB of space
> free on my NVMe boot SSD, maybe the database could hopefully fit in
> a few hundred GB of that, leaving the HDDs free for doing the main
> operation of going through their data, etc.

How many files do you have? Don't you have mariadb running on your
server?

► 092 - how many files, mariadb on server
About 5.14 million files, and I'm not sure what mariadb is and I'm not running a server for now.

On 11/12/2022 at 7:05 AM, heimdali said:

> As for files of different sizes being identical - of course they
> wouldn't be "identical" ... but they could have been taken from the
> same audio, photo or video source, and just be a different
> resolution or bit rate of the same thing.

Yes, but why would you keep so many dupes?

► 093 - why keep so many dupes

I don't want to keep so many, just a couple, basically the original straight from the camera / recording device, maybe one that was transcoded for editing, and one or two "final" copies for devices, etc. There's a few I might still have laying around someplace, where I was experimenting with mp3 encoding, and basically had every possible bitrate, sample rate, and a couple other things of one song, while I was testing the lame encoder or something like that, to learn what would be good settings to encode mp3 with. I remember several years ago I generally found I got better quality results by using about 2 or 3 bins higher bitrate for a given lowpass filter setting than was automatically suggested.

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> Nothing prevents you from making a check sum of all the check sums
> of all the files in a directory and from comparing those check sums
> of check sums.
>
> Ahh, checksums of checksums... but in my case I also have a bunch of
> partially duplicate folders, some maybe only 2 or 3 levels deep,
> some well into the double digits.

That won't matter because you'd be comparing directories and not
directory hierarchies. If you were to comare the heirarchies, what
are the chances that there would be dupes of folder hierarchies?

► 094 - dupes of folder hierarchies

I already know of quite a few, problem is finding them again. I was working on finding some files earlier, and found the same file in a couple different locations, and I found there were some partial dupes of folder hierarchies. (But finding them will be a chore, cause I deleted some of the duplicate files and folders, but not all of them and I forget the names of what I had left, also either file explorer crashed or the system needed a reboot so i can't easily find it at the moment.)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> And there's the issue of the structure, for example,
> E:\a001\b001\c001\d001 might have a similar structure to
> H:\a003\b003\c003\d003\e003\f003\g003\h003\i003\j003\k003\l003\m003
> and so on,

How's that simialar?

► 095 - folder structure, how's that similar?

Similar once you go down farther in levels. The top level of one duplicate of a particular hierarchy might by 3 levels down, while the top level of another duplicate might be 8 levels down from the root folder, and often even on a different drive. (Maybe even on a different filesystem, but that's less likely.)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...] (too bad there's multiple reasons why I can't just boot
> Linux, do cd /, then use a terminal command to recursively list
> everything then save & upload the resulting txt )

Why not? Don't you have an USB stick you can boot from?

► 096 - have usb stick can boot?

I do have a couple USB sticks, one with a Windows 10 installer and another with multiple Linux ISOs on it, as well as a dead USB stick somewhere and one or two others laying around, one of which i've misplaced so idk where it is. (Side note, I'd like to see about maybe updating the BIOS on my B550 Taichi motherboard, but I don't have a spare blank USB stick to put the bios file on, unless there's a way to do it without USB....)

On 11/12/2022 at 7:05 AM, heimdali said:

> quick note, speaking of backups... the idea of something like "dd
> if=// ... " comes to mind,

No, dd is not for backups.

► 097 - dd not for backups
Ahh, it's what I'm currently using, as it seems to be the only way I know of to do a full backup of every single sector, and the only one that if I back up an OS drive, I can then plug the clone in and boot from it. If I plug the original AND clone in simultaneously, I get a signature collision error, which I have to do something to reassign the signature to get the extra drive to come online. Also a while ago (this one's complicated so I'm gonna leave a lot of stuff out) I'd had an OS clone or two... well one time I had my SSD plugged in, and the clone, hit F11 to pull up the boot manager to boot from the SSD... and it booted from the CLONE, which was a hard disk, instead.

As for differential or incremental backups, you're right, dd is not the tool for that.

Also to me, a full backup wouldn't just restore the files, and also not just the programs, settings, operating systems, it would restore the actual partitions, free space, etc, even to the point of being able to use a tool like TestDisk to recover deleted partitions, if those would have been recoverable from the original media at the time it was backed up.

TestDisk is how I restored some partitions from drives the time or two that I accidentally plugged a 4+TB drive into an older controller that maxed out at 2TB support. I first "dd"-ed the mangled drive to another one of equal or greater size, then I restored to a third drive. (I think I did notice an issue though, is that the "dd" didn't properly get everything - I was able to restore from the original drive, but trying to restore from the dd'ed clone was corrupted I think, so maybe I need a better tool than "dd"; also it might not be the best tool if the drive is actually failing.)
Yes, there was at least in one case a couple years or more between when the drive got borked, to when I did the restore, but I made sure to not use the drive at all (left it unplugged) until I was ready to attempt the restore.

On 11/12/2022 at 7:05 AM, heimdali said:

> but if I'm backing up the ENTIRE filesystem including connected and
> network devices,

Such devices have their own filesystems.

→ 098 - (may not reply) devices, own filesystems, backup entire, connected + network

On 11/12/2022 at 7:05 AM, heimdali said:

> problem is what would I back it up TO, if I'm already specifying to
> back up not just everything directly connected to that PC, but also
> everything on the network...

Why would you do that?

► 099 - why ... back up everything on pc AND network

Well for right now I wouldn't do it, it's just a hypothetical idea, but basically back up all PCs, phones, etc in the house, AND back up the backups as well but maybe that last one is getting into some kind of circular / loop concept or something (I forget the proper term for it)

On 11/12/2022 at 7:05 AM, heimdali said:

> (Also in my book, if I'm not including every single bit / sector
> including boot sectors, landing zones or whatever, normally
> inaccessible areas, firmware, etc, it's not a complete backup.)

Maybe you want to use clonezilla to make a clone. That would be a
rather unwieldy way of making backups, and you'd be cloning a lot
every time you don't need to clone.

► 100 - clonezilla, unweildy, cloning a lot....
Well I'd only do it once in a while, basically for complete backups. For differential or incremental backups I'd just need to back up whatever changed, and I'd need to learn how those work and how to implement them.

A while ago I had thought of maybe having one large hard disk onto which I'd put those diff / incr backups, then when it gets full, pull it and get another one to back up onto the new one. (And/or, dump the entire contents of the full drive to a cold-storage backup server / NAS, or something like that.)
I've also heard that some people at least a number of years ago were advocating doing full backups daily, and incremental or differential backups hourly .. but when I have multiple tens of terabytes of total capacity, doing full daily backups would be rather unweildy, both from the time it would take to read from HDDs, and the cost of all that storage.

On 11/12/2022 at 7:05 AM, heimdali said:

> (I'll admit that the thought of just deleting everything has come to
> my mind more than zero times, but I don't really want to exercise a
> nuclear option as there are some things I want or need to save. )

You can always get your storage server up and running and copy the
data to it before it's lost, and use deduplication so that the
required storage space doesn't get out of bounds. Once you have that,
you can make a backup and then start to go through all the data and
delete the stuff you don't need to keep. Just make a snapshot before
you start deleting, and you may not even need your backup.

► 101 - storage server up & running, copy data, dedup, backup, go through data & delete, snapshot, etc
That actually might be a good idea, thing is right now I don't have a place to put a storage server.

On 11/12/2022 at 7:05 AM, heimdali said:

> Also I'm maybe considering getting a couple 2TB or 4TB SSDs (one
> SATA 2.5", one NVMe M.2) for my laptop, to replace a couple 1TB SSDs
> so I have more space.

Why don't you get your storage server up and running and store the
data there? Do these laptops even support redundant disks so it's
possible to store data on them?

► 102 - storage server up running, laptops support redundant
I've thought of that, cause it actually in some cases would be nice to have a centralized place to have stuff. Several times now I've looked for something on the desktop and realized oh, that's on the laptop, or the same thing the other way around.

From what I understand my laptop supports RAID0 and RAID1, but I'm not using that. (It may also support RAID5 or RAID10 due to having 4 disks, but I don't think it would work when mixing NVMe with SATA.)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...] I'm just a bit hesitant to pay that much, for example, for a
> 4TB

Yeah get your storage server up and running. Not doing that holds you
back and makes everything difficult and totally unreliable, makes you
throw endless amounts of money at buying more storage all the time and
all you accomplish is that you make the mess worse and uselessly store
endless amounts of duplicate data. And you still have no way to
prevent data loss, like due to disk failures and power surges, and no
way to make backups.

→ 103 - (may not reply) get server running, buying storage, endless duplicates, data loss, no backups, etc

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> Also ... I've been thinking a bit recently about getting a couple
> more SSDs for my laptop,

How many can you fit into it?

► 104 - how many SSDs can fit into laptop

My laptop, a Clevo P750DM-G, has two M.2 slots (supports NVMe or SATA) and two 2.5" SATA bays for a total of 4 disks.

On 11/12/2022 at 7:05 AM, heimdali said:

> either 2TB or 4TB to replace a couple 1TB

Why don't you keep the data on your storage server? There's no need
to have that much storage capacity in a laptop. Wireguard works fine.

► 105 - data on storage server instead of "that much" in laptop

That might be okay when I'm at home, but when I'm out somewhere for a weekend or longer, I want some things accessible locally.

Sometimes, for example, I've been in places where there's absolutely no internet connection, so there's no way I could access stuff from home.
And even if I did have a connection, my home internet has a bandwidth cap of 1 TB / month, and a max upload speed of about 10 Mbps, making it unusable for hosting a remote storage server. (FIber isn't available here.)
I'm considering moving to near Provo, Utah sometime for a possible training / job opportunity (not involving PC / tech), and I'm thinking for the first few months or so I might have a more temporary living situation, so I'd probably only bring my laptop with me. (Right now I'm near San Diego, CA.) Once I'm more settled into more permanent housing (whether in Utah or somewhere else), THEN I'd want to set up a storage server, and go get my desktop PC and other furniture (and a couple acoustic upright pianos I have) from California.

(I'm not sure at what point it would be called "permanent" housing... my concept / experience might be a bit skewed, cause so far I've lived my entire life in the same house.)

On 11/12/2022 at 7:05 AM, heimdali said:

> Looking at pcpartpicker prices.... [...]

Stop buying any more storage before you got your storage server up and running.

► 106 - don't buy storage before server up

If only I had a place to put it.... I may end up having to move to my own place THEN set one up.

On 11/12/2022 at 7:05 AM, heimdali said:

> I've also wanted to compare the speed of some old HDDs vs current
> ones and SSDs -- but not in terms of MB/s transfer rates, I'm
> thinking more like "How long does it take to fill the entire disk if
> you're constantly writing to it".

Why? Put your old disks into your backup server and make backups on
them, then you'll know how fast you can write to them.

► 107 - how long to fill disk, put in backup server, know how fast to write to them

Why? well I have a feeling that older disks were "faster" in that way than modern ones and I wanted to find out if it's true.

On 11/12/2022 at 7:05 AM, heimdali said:

> I wanted to buy a couple old <80MB PATA HDDs on ebay,

Stop buying any more storage before you got your storage server up and
running.

PATA? Wasn't that the predecessor of IDE? I may have had a 386DX16
that had one. 60MB was kinda huge ...

► 108 - PATA / predecessor of IDE
PATA was what IDE was renamed to. (Also I've heard that SATA is "technically", in a way, IDE, which means Integrated Drive Electronics if I remember.

I think you're thinking of MFM / RLL, or ST-506 / ST-412 interface. With those, there were two ribbon cables, one for control, one for data, and the controller was a separate expansion card. When IDE came along (later called Parallel Advanced Technology Attachment, if I remember right), then later SATA, the controllers are those boards that are attached to the drive, I believe.

The old drives had boards too, but I think they were more dumb. Those were the days in the 1980s when you had to manually configure sectors, interleave, cylinders, heads, write precompensation, landing zone, etc, and do low-level formatting before you did the actual filesystem formatting. I never had personal experience working with them, as I was born in 1981, but I did use a computer with one of those drives in it.

On 11/12/2022 at 7:05 AM, heimdali said:

> (I'm thinking that some of the older/smaller drives, like 40MB or
> 20MB, or if anyone has 10MB or 5MB MFM drives laying around and a
> working system capable of interfacing with them, could fill the
> entire drive in maybe half a minute to a few minutes or so,

You probably mean hours and days, not minutes.

► 109 - meant hours/days not minutes, fill small drive in a few minutes

I did mean minutes.
It took me about 11 or so minutes to fill an 8.4GB hard drive from 1998 a few years ago.
Also there's a Tom's Hardware article on "15 Years Of Hard Drive History: Capacities Outran Performance" from November 2006 - the page I linked was "Time Required To Write A Full Platter". The largest drive they tested at the time was a 750GB SATA drive, and it took 52 minutes to read 200GB, or ~27% of the capacity. Compare that to the 37 seconds it took to read 26 MB of a 40 MB IDE drive, or 65% of the capacity. (Extrapolating, I'm guessing it would have taken about 3 hours 17 minutes to read the entire 750 GB drive, vs about 57 seconds for the entire 40 MB drive.)

On 11/12/2022 at 7:05 AM, heimdali said:

> [...]
> Maybe instead of me buying the drives, maybe someone else already
> has a system with drives like that, or better yet, something that
> supports MFM drives and has some of the early 5 or 10 MB drives.

Oh those are very rare ...

► 110 - old MFM, very rare

I've seen them for sale on ebay sometimes, but I don't have a system to put them in or the expertise to format / test them. I've seen people on other forums geared toward more retro tech mention having them, but I don't have user accounts on some of those places (and part of my clutter includes already having too many user accounts lol.) I have seen some retro tech youtubers with those though, maybe I should ask there sometime.

On 11/12/2022 at 7:05 AM, heimdali said:

> I wonder how quickly those could be written or read...

You could probably dig up some old benchmarks or something. You're
gona like this, for example:

1 5 2 8 - 1 5    MICROPOLIS
NO MORE PRODUCED                                      Native³ Translation
                              ÄÄÄÄÄÄÅÄÄÄÄÄÂÄÄÄÄÄÂÄÄÄÄÄ
Form                 5.25"/FH              Cylinders    2100³     ³     ³
Capacity form/unform 1346/ 1535 MB        Heads          15³     ³     ³
Seek time í / track 14.5/ 4.0 ms          Sector/track   84³     ³     ³
Controller           SCSI2 SINGLE-ENDED    Precompensation
Cache/Buffer           256 KB DUAL-PORTED Landing Zone
Data transfer rate    2.916 MB/S int       Bytes/Sector      512
              4.800 MB/S ext SYNC
Recording method     RLL                            operating ³ non-operating
                          ÄÄÄÄÄÄÄÄÄÄÄÄÄÅÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
Supply voltage     5/12 V       Temperature øC         5ö50    ³    -40ö65
Power: sleep              W     Humidity     %        10ö90    ³     10ö90
       standby       25.0 W     Altitude    km    -0.061ö 3.048³ -0.305ö15.240
       idle               W     Shock        g         2       ³     20
       seek          30.0 W     Rotation   RPM      3600
       read/write         W     Acoustic   dBA        43
       spin-up            W     ECC        Bit   48
               MTBF         h     150000
               Warranty Month        60
Lift/Lock/Park     YES          Certificates

That was actually a really good drive. IIRC it was more like 2MB/s,
and the power consumption was hilarious (You realize that 5.25" is
really 5.25" and not half that like everyone thinks?) Wow that was
one heavy disk ... I liked it a lot, it made such a trustworthy heavy
tock tock tock sound when it was doing something that you always knew
when your computer was working. I mean /working/, you know ...

Oh, look at that: I just noticed it had a 5 year warranty. Not
surprising, it really was a tank.

► 111 - Micropolis SCSI 1.5GB, 2MB/s 5.25" drive, etc

Ahh I think I've seen some kind of site similar to that with old drive benchmarks and things, I think stason.org or something like that.

If that 1346MB drive transferred data at 2 MB/s, then the drive could theoretically fill in 11 minutes 13 seconds.

I looked on that site for some smaller drives, and saw several interesting ones, including the 5MB Fujitsu M2231AS and Seagate ST406 (I didn't see the ST506 on the site), the 10MB IBM WD 12, Seagate ST412 and Tandon TM252, which have an internal transfer rate of 0.625 MB/s. Then there's the 21MB Seagate ST325A/X, rated for 1.5MB/s in AT mode and 1.75MB/s in XT mode, and the 42MB Quantum Prodrive ELS 42 AT at 2.5 MB/s. Even if their actual transfer rates are slower, like in your 2MB/s vs 2.916MB/s example, it looks like they would theoretically transfer the entire capacity in about half a minute or less

On 11/12/2022 at 7:05 AM, heimdali said:

> I hear the ST-506 MFM interface supported up to 5 Mb/s, not sure if

Does that seem realistic?

► 112 - ST-506 MFM, 5Mb/s, realistic?

Well, it may not quite reach that high in the real world, but I did see a few youtube videos (01, 02, 03, 04) that included benchmark results from some older MFM and IDE hard drives. Some of them might have only transferred data at like 300-500 KB/s, but with small capacities like 20 or 40 MB it would still transfer the entire capacity in about a minute or two.

heimdali · November 28, 2022

> Are you saying something about THREE disks for each one, or what?
> (Also are you referring to the 3-2-1 backup strategy, or something
> else?)

I don't remember what this refers to.

> For me, I don't really need live-online redundancy (as in immediate
> seamless failover to a spare disk), but some kind of backup I
> definitely should have.

You should have both, of course.  You can also use spare disks in a
RAID if you want to.

> Elsewhere I mentioned having poor internet (10Mbps and 1TB/month
> cap) which makes doing online backups pretty much impossible for me.

First you need somewhere to make backups to, and then it can take a
long time.  With only 1TB/month you'd have to prefill the backup at
home and then transfer the differences with rsync.

> (I could back up text files and things like that, but that's nothing
> in the grand scheme of things.)  If I was doing online backup
> though, I'd prefer something priced no higher than what it would
> cost me to buy that much space on a HDD over time, considering the
> warranty.  (A 16TB WD Red Pro has a 5-year warranty, if it's
> purchased for $269.36, that's (26936/16/60) ~28.0583¢/TB/year, or if
> you double that (for 2 disks for redundancy) it's ~56.1167¢/TB/year,
> much cheaper than Amazon or Google's cloud storage, and probably
> cheaper than Backblaze too.)

Huh?  Why would you ever give your data out of hands??  You'd at least
have to encrypt everything with GPG before uploading, and that makes
it even more unwieldy to backup over the internet.

> Backing up everything including connected devices was a hypothetical
> idea

Having backups isn't a hypothetical idea but a requirement.

> Some time ago I heard of people talking about doing daily full
> backups and hourly incremental/differential backups, but with my
> mutliple 10s of TBs (actually now probably >100TB) capacity, that
> doesn't seem possible. 

That's no problem.

> Most Chenbro cases I've seen were rackmount, I think.

19" is the way to go.

> Those ebay examples are from Germany, I'm in the USA;

Yes, they are examples.  You'll have to look out for useful cases ---
or buy new, but that's expensive.

> Also for things like family / friends documents, photos, audio,
> video, etc, I'd like to find a good long term storage / backup
> solution that my great grandkids (I have no children yet) can access
> even after I'm gone, if they want to.

Apparently there are special long term storage DVDs available you
could use.

> As I mention in the "server" section, it would be nice to have a
> central location for everything, so no matter what device I'm using
> or where I am, I could access it.  (I do still want to be safe
> though.)

You need a decent storage server, and that starts with a 19" case.

> If possible, I prefer replacing several smaller disks with a single
> disk equaling their combined capacity, for the price I paid for one
> of the smaller disks previously.

Just move the data to your server.

> I don't want to keep "so many" duplicates, just the originals from
> the media (camera, audio recorder, etc), and a couple others either
> for editing, or final renders / versions.

Then move them to your server and ignore or delete the rest.

> For now I'm limited to consumer hardware for my platform.

Use decent hardware or you're not getting anywhere.  Stop copying your
data until you have ECC RAM.

> A while ago I thought about building an LGA771 or 1366 based NAS, or
> maybe even 2011, but was having trouble finding the type of
> non-rackmount case I wanted...and electricity here is not cheap -
> several years ago it was upwards of ~40-67¢/kWh or so, and I thought
> I heard talk of >$1/kWh.

You need a 19" case.  When electricity is that expensive, you can run
your server only when you're using it, so you need to distinguish
between active data and archived data.

> Actually, SATA does include hot-plug ability in the spec, at least
> for internal SATA I think.  (idk about eSATA though, when I had a
> board that supported it and an enclosure with that, I don't think I
> could do it.)

Yes, but have you ever seen it in practise?  eSATA requires special
plugs that make sure that the wires are dis-/connected in the right
order, and who uses those.

> Also I would want to be careful to not eject a disk

Huh?  Why would you eject disks?  Disks are plugged into the server
and the only disks you need in a client is the ones it boots from.

> I did compress the dd image in at least one test.

Use clonezilla ...

> I would REALLY like to find a way to compress the images so I can
> fit more backup images on a single disk, but in a way so I can
> easily open the image in something like 7-Zip and browse, or extract
> individual things as needed.

Why don't you use a file system that supports compression?  And don't
keep useless disk images around.

> For now, though, I'm thinking I might just dd the drives to image
> files on the 12TB Toshiba MG07 drive,

What for?  There's not much point, if any, in keeping disk images
around.

> Also, if I'm backing up an OS drive, I want it to be bootable.

That's what clonezilla is for and the only reason to keep disk images.

> would consider switching to Linux though, especially if a Windows 12
> continues in the direction 11 seems to go in things like wanting
> online accounts, etc, after 10's support ends.

Requiring online accounts is not an option.


> I'd like to find a way to scan the entire magazines sometime

Maybe there are scanners that can do that.

> I'm missing quite a few of that ComputorEdge magazine though.

Why would you need outdated information?

> I don't think too many disks is overwhelming my system, I think it
> may be something else causing the issues cause other things ar e
> happening too periodically, often simultaneously.  (Some things are
> more common, like my WiFi dropping out and a few HDDs being
> inaccessible, while others have only happened once or twice like the
> keyboard/mouse not responding or the GPU going to black screen and
> 100% fan.)

That sounds as if everything is extremely crappy.

> I too don't like inefficiency, but I do want a complete job done.
> (If I'm parsing through data on an entire HDD, I expect it to take
> at least as long to do that as it takes to sequentially read the
> entire drive's capacity.)

Why would you do that?


> I have a feeling that older disks were "faster" in terms of how long
> it takes to fill the entire disks.

That's only a feeling.  Just consider how long it takes to write a
40MB file now and how long it took when entire disks were 40MB
small.  Writing 40MB you don't even notice.

> I've heard things about software raid being more preferred now vs
> hardware raid.

That depends.  There is no good solution to boot a computer without
hardware RAID when you have one with an EFI BIOS because EFI is
flawed.

> (For example, about ZFS preferring RAID cards be in IT mode, not
> RAID mode)

It's better to use the actual disks with ZFS instead of hardware RAID,
so that is an exception.  It's kinda the only exception since btrfs
still doesn't do RAID56, and AFAIK mdadm isn't better than hardware
RAID.

> I've heard of a few RAID levels, and Raid-Z and Unraid, but don't
> know all the possible levels.

Then you have more homework to do.

> I don't need higher performance than single-drive (if I do I'll use
> an SSD for those situations).

SSDs are too expensive to use for everthing, especially for backups.
Storing data on only a single disk is out of the question, and it's
excrutiatingly slow.

> If disks fail, I want to be able to replace them and rebuild from
> parity / mirror

That's what RAID is for.

> (I've heard stories of RAID rebuilds taking days or weeks on some
> drives,

That depends on the size of the disks.

> If too many disks fail, I don't want to lose the entire array, just
> the extra disks that failed (like Unraid).

You can use appropriate RAID levels for that.

> I've heard Extenders limit you to single-port bandwidth, so I don't
> like to use them.

IIRC they are multplexing the signals ...

> With only a few HDDs they might still "fit" within the bandwidth of
> a single port, but with a lot of SSDs they'll be bottlenecked really
> quick.

Then don't connect SSDs to them.

> Also I prefer being able to access disks individually

After you did that, you'll have to rebuild the RAID.  And what would
be the point?  It's not something you would do.


> UNRAID - Yeah that could be an issue, being unable to read the data
> if I switch to something else.  Also I've heard horror stories of
> RAID controllers going rogue and committing fornication with the
> hard drives, spewing Scheiße all over the place,

There is reasons for using UPSs.  Any board and/or SATA controller or
whatever can fail and damage your data.  That's one of the reasons for
having backups.  And without using ECC RAM you may already be damaging
your data, only you don't know about it.

> which helps to further my preference of working with the drives
> individually,

That is not an option.

> But where would I even PUT a rack cabinet or storage server or
> anything like that?

Clean out all the trash and you have plenty of room.

> I don't have any experience with 19" racks, cases, etc.

You can learn.

> Also if I ran a backup server, that's literally ALL it would do - no
> plex, no transcoding, probably even no zip/unzip/etc.  JUST storing
> and transferring files.

That only depends on how you set it up.

> That was Fry's in San Diego, in Q1 2016, I think right before things
> started really going downhill.

The stores here were gone well before 2000.

> I've never used fiber interface for storage, thought that was for
> network?

Apparently that was used before there was SAS.

> If something happened to me or to my next-in-line trusted
> friends/family, then if someone else needed to access the data, I
> want to be able to provide a way to do that, while still being safe.

You can simply write down the passphrase.


> I remember a decade or two ago, sites would actually behave properly
> when loading, I think, for the most part.

Web sites weren't as overly complicated as they are now.

> Also I wonder if there's a more sinister reason to move things
> around on the page after it's loading -- to try to get people to
> click on more ads.  I've lost count of how many times I went to
> click or tap something I wanted to open, and the page moved, or an
> ad spawned right there and I clicked it instead of what I wanted.

Huh?  Don't you use an adblocker?  Without one, web browsing is
impossible and I wouldn't do it at all.

> 001, 003 - emacs/mail: Ahh, I haven't done mailing lists in a while
> but I remember the > for quotes.  Also I was able, with much effort,
> to remove the code from the box and prepare it for a reply.  (It
> involved spreadsheets, multiple drafts, etc.)

Why didn't you just copy it?

> 011 - Backplanes: Most backplanes I've seen are crazy expensive, or
> in expensive server cases.

That's why you buy them used.  And a case is basically a lifetime
investment which you can use for a long time.  It takes a while before
things become so incompatible that you'd need a new one.


> 013 - Tape Backup: In 1994, tape was MUCH cheaper, like a tiny
> fraction of the cost per MB compared to HDDs for the tape, and about
> half or 2/3 for the drive.

Tape drives were very expensive.

> 055 - ZFS: I've heard ZFS can be made to work with Linux, but it's
> beyond my level of expertise.

With Debian, you just install it through the package management and
then you have ZFS.

> 085 - testing, stitching buttons to: WHICH cheeks, the ones on my
> face or.....? 

yes, yours

> 092 - mariadb on server: I'm not sure what that is (and I'm not
> running a server).

https://en.wikipedia.org/wiki/MariaDB

You really need a server ...

> 096 - USB stick, boot: I have a couple USB sticks, one with Windows
> 10 installer and another with several Linux ISOs.  I've been
> thinking of updating the BIOS on my B550 Taichi (it's running P1.30
> now).

When you have a Dell server or a rackable workstation, you can update
the BIOS and other firmware through idrac with no hassle like USB
sticks, and that's really nice.

> Spoilers are those boxes that say "reveal hidden contents".  People
> use them for various purposes, I'm using them so that longer blocks
> of text are initially visually compressed, if you want to read them
> you click/tap "show hidden contents".  (The post would have been
> even longer if I had left them readily visible.)

It's not really useful in this case.

> It took a bit of effort, but I was able to take it out of the code
> box and prepare / format it to reply.

easy with emacs

> Update: Came this afternoon (Wed, 16th), now gotta get my system
> ready to install them and reboot to Linux so I can start doing some
> data transfer.  (Windows might not see some of the files, and
> there's at least one ext4 partition on one of the drives.)

You need to get your server set up first.

The rest is pretty much unreadable ...

DANK_AS_gay · November 28, 2022

HOLY F#$#% SH!T THESE POSTS ARE LONG!

Sign In

Considering 2x 18 or 20 TB HDD (replace 8+10TB) : 1 - is 3rd-party seller ok? ; 2 - 18TB <$/TB but 20TB more space/bay...

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Topics

Latest From Linus Tech Tips:

De-Google Your Life - Part 1

Latest From Tech Quickie:

Why Can’t You Buy a “Dumb TV?”

Latest From TechLinked:

Microsoft will see EVERYTHING….

Latest From GameLinked:

Can CoD SAVE Game Pass?

Latest From ShortCircuit:

I have been blessed - Moondrop x Crinacle DUSK IEMs

Latest From Mac Address:

Why did you buy an Apple Vision Pro?

Latest From Channel Super Fun:

I Swapped the CEO's Assistant For a Day!