Backblaze: SSDs might be as unreliable as disk drives

[Quackers101]

I'm kind of worried about NVMe's and some SSDs, with mostly quality control and temps, also getting what you actually buy for the money.

If they can't then DON'T SELL IT AS ONE. ugh.

[BachChain]

This is news? Long-term reliability was never the selling point of SSDs, especially in write-heavy enterprise situations.

[Quackers101]

1 minute ago, BachChain said:

Long-term reliability was never the selling point of SSDs, especially in write-heavy enterprise situations.

outside of write heavy applications?

[Kisai]

1 hour ago, BachChain said:

This is news? Long-term reliability was never the selling point of SSDs, especially in write-heavy enterprise situations.

The selling point of SSD's is read speed.

 

In most cases the the bigger the capacity, the higher the reliability and faster the speed of the drive. The problem is that when you use up more than 80% of the drive, the speed and reliability falls off a cliff. This is because it has to be more aggressive.

 

If backblaze was only storing things things and then never re-using them, the reliability of the SSD's should be 100% short of a thermal failure. However that's likely not the case.

 

[Dabombinable]

4 hours ago, jagdtigger said:

Look when i have a wd green from 2011 that even survived running torrents and being in a raid vs 3 hdd from 2017 which died with something like 30k hours on it used in its intended use-case. Thats way more than just bad luck.

(I even have a 200 GB WD somewhere that still works with <10 bad sectors.... [and even those are old AF, the drive was one or two years old i think])

/EDIT

Oh and did i mention for not much more i could get wd dc-hc drives instead of crappy ironwolfs? Yeah seagate can go bust for all i care.....

Were those HDD by chance in slots directly above and below each other, and in non rubber mounts? A drive that I shucked back in 2015, and has only ever run non spaced for short periods (always with rubber mounts) - btw I can raise you a faulty WD Green 2TB (still have it) with fewer hours, a 2TB WD Black (that I returned), and a 2013 WD Blue that has had even more use:

[leadeater]

This is backblaze, they will be using consumer bottom barrel SSDs not ones designed for their needs because to them bottom dollar wins out for lowest service cost.

 

Use actually good SSDs with greater than 1 DWPD and problem solved.

 

News story here is "Tech company cheaps out on SSDs and complains"

[Dabombinable]

5 minutes ago, leadeater said:

This is backblaze, they will be using consumer bottom barrel SSDs not ones designed for their needs because to them bottom dollar wins out for lowest service cost.

 

Use actually good SSDs with greater than 1 DWPD and problem solved.

 

News story here is "Tech company cheaps out on SSDs and complains"

I wonder if they're using Crucial P1 as some of their SSD. With the glitches mine has had several times, that'd explain some of their failures. Mine doesn't like data being moved around too much (and then there is the low endurance).

Intel SSD if they are being used would also be strictly enforce their right endurance limits - so they'd "fail" while they are still actually good to use.

[LAwLz]

1 hour ago, Dabombinable said:

I wonder if they're using Crucial P1 as some of their SSD. With the glitches mine has had several times, that'd explain some of their failures. Mine doesn't like data being moved around too much (and then there is the low endurance).

Intel SSD if they are being used would also be strictly enforce their right endurance limits - so they'd "fail" while they are still actually good to use.

Here are the stats for the SSDs. Please keep in mind that "DELLBOSS VD" is a PCIe to M.2 expansion card so we don't know the model of 176 of those drives. They got 215 SSDs just labeled "Seagate SSD" as well...

 

Seems like most if not all of their drives are consumer models, and in some cases OEM drives.

Edit: Actually, it seems like at least one of the drives (the Micron ending in TDU) is an enterprise drive. It seems like it's from the "Micron 5300 boot" lineup. It's made to be a boot drive and has about 50% lower endurance than the regular "5300 pro" model. The 5300 Max has about 5 times the endurance. So even though it's an SSD made for data centers, it is the bottom of the barrel that is only meant to be used as a boot drive. 

[Dabombinable]

28 minutes ago, LAwLz said:

Here are the stats for the SSDs. Please keep in mind that "DELLBOSS VD" is a PCIe to M.2 expansion card so we don't know the model of 176 of those drives. They got 215 SSDs just labeled "Seagate SSD" as well...

 

Seems like most if not all of their drives are consumer models, and in some cases OEM drives.

Edit: Actually, it seems like at least one of the drives (the Micron ending in TDU) is an enterprise drive. It seems like it's from the "Micron 5300 boot" lineup. It's made to be a boot drive and has about 50% lower endurance than the regular "5300 pro" model. The 5300 Max has about 5 times the endurance. So even though it's an SSD made for data centers, it is the bottom of the barrel that is only meant to be used as a boot drive. 

Yeah just looking at the chart, they're less accurate than WCCFTech.

[Helpful Tech Witch]

10 hours ago, jagdtigger said:

Look when i have a wd green from 2011 that even survived running torrents and being in a raid vs 3 hdd from 2017 which died with something like 30k hours on it used in its intended use-case. Thats way more than just bad luck.

(I even have a 200 GB WD somewhere that still works with <10 bad sectors.... [and even those are old AF, the drive was one or two years old i think])

/EDIT

Oh and did i mention for not much more i could get wd dc-hc drives instead of crappy ironwolfs? Yeah seagate can go bust for all i care.....

I have a seagate drive with almost 100k hours on it, 80gb from 20 years ago and its fine, the drive runs as expected

[BondiBlue]

10 hours ago, jagdtigger said:

(I even have a 200 GB WD somewhere that still works with <10 bad sectors.... [and even those are old AF, the drive was one or two years old i think])

Lol, what? I've got a Toshiba drive around here somewhere that still worked last time I used it, and it had more than 6,000 bad sectors and counting. 10 is nothing. 

[Arika]

5 hours ago, leadeater said:

This is backblaze, they will be using consumer bottom barrel SSDs not ones designed for their needs because to them bottom dollar wins out for lowest service cost.

which is exactly why i put no weight onto anything Backblaze says. Barracudas in a data center? yeah of course they are going to have massive failure rates

 

[DrMacintosh]

I think looking at SSDs in terms of reliability in the long term is not the right perspective to take. 
 

I’ve got a 2TB WD Green HDD that I ripped out of a 4th Gen Apple Time Capsule. That drive is 10 years old and has been used as a bulk storage drive in 2 PCs and currently serves as a NAS drive on my home network. 
 

The point is HDDs are usually very reliable if you don’t abuse them. Small 2.5” HDDs that are often found in laptops though? Those die all the time. Why? Because people throw, hit, drop, and otherwise abuse their laptops all the time. If you swap that laptop HDD with an SSD you’ll find that drive failure rates suddenly go way down. That’s really where SSDs are more reliable. 

[WolframaticAlpha]

17 hours ago, porina said:

Kingston A400

 

[Fetzie]

Backblaze can call me when they start putting drives designed for datacentre use in their datacentres.

[atxcyclist]

As far as SSDs go, one cheap Team Group SSD in my office died, but none of the others in computers I've built have. I have a SanDisk SSD that I used in my workstation, it had a 70 TBW durability rating and I retired it (and moved to a bigger one) after it got to 115 TBW, no failure and the diskinfo was still alright otherwise. I've really never seated SSD reliability, when it's my money I buy decent stuff and it lasts.

. 

 

In contrast, the last what I consider to be a good long-lasting spinning drive was a 10gb Western Digital I put into service in 1999. That computer ran nearly continuously from when I built it and after I sold it, to roughly 2010 when I got the machine back. 11 years, 24 hours a day, always spun-up because it was a media repository. So 95k hours? That's actual reliability.

 

Nearly every spinning drive I've had after that lasted 3-4 years before there were problems with it. At one point, I had piles of my and friends 80-250gb drives from mostly Western Digital that had failed prematurely. Most were unrecoverable, and just had the magnets pulled out of them and destroyed. I have one active mirrored array at my office, but outside of that I keep everything on SSDs and only back-up to spinning drives, but they're externals and don't get run all the time.

 

[mr moose]

15 hours ago, LAwLz said:

No, Seagate drives aren't bad. You have been unlucky if that's your experience. 

Seagate has roughly the same failure rate as all other consumer HDD makers like WD, Toshiba and Samsung. 

Seagate got a bad rep because they had one or two models that were bad like a decade ago. 

Also worth noting that 4 years is the point when critical failures start to become more common regardless of drive brand. 

 

The typical bathtub curve.

[porina]

@WolframaticAlpha Is the A400 that bad?  The one I have, I think I'll have to wipe and include as part of a system sales later. The only requirements I had at the time I got it was it is a SSD and cheap. Boot drive for a cruncher, back in the days when people ran software 24/7 without expecting profit.

[porina]

5 minutes ago, valdyrgramr said:

Not the most reliable, and superior options in that price range.

Can't remember exactly what was on offer around that time, but I didn't have problems with their older V300 series which was my go to in cheap systems before that. I think WD had taken over Sandisk by then, so the old Sandisks weren't so common, and WD Green was more expensive. Crucial BX series was also more expensive, and I don't recall any other branded drives that were on the cheap end at the time. 

[wanderingfool2]

20 hours ago, LAwLz said:

Read the full report now.

Do you actually have the full report link, with the numbers?  [A link to them]  If so I think you were reading a different report than I was

 

I've looked through their site and while there is some correlation, it appears the Q1/Q2 2021 stats that were used were not included in numbers given out (yet were used in their conclusion).  Don't get me wrong, I don't think this is a fully fair test, but it's better to have more of the facts (as you'll see below, as I address some of your talking points)

 

20 hours ago, LAwLz said:

For those wondering, yes, their sample size for SSDs are way smaller than the HDDs.

Sample size in report was 1666 "SSD"s and 1297 HDD's so no.  (As can be seen in their original table, which was even posted in the OP's thread)

 

20 hours ago, LAwLz said:

They have had 7 SSD failures and 551 HDD failures, and somehow they think that's enough data to extrapolate what the numbers will look like several years into the future. They only had a handful of SSD models, one of which they only had 49 drives of and one of them died so they put the AFR of that at model at 2.82%. 

"7 SSD failures" is not true, they experienced 17 failures.  You are looking at the older data.  They also stated their compared the HDD's failures over the same period of time, which totaled to 25 HDD failures.  Ultimately they had 1394 Seagate drives, 40 micron drives and 176 Dell drives (based on where you were pulling your numbers from) plus some more which I couldnt find references to...and even then Seagate made up the "7" drive failures during that time.  [Which again, there are clearly 10 more drive failures happening somewhere, but I don't think that data was released].

 

The "551" number you grabbed is over a period of 7 years, if you did read what they said they specifically said it wasn't a good comparison to do things like that...which is why they later went on to utilize only the first 14.3 months of data they had of the HDD's...which lines up with the SSD numbers.

 

20 hours ago, LAwLz said:

And before anyone asks, yes, they still use consumer grade parts in their server environment. That SSD they only has 49 of? That's a Seagate BarraCuda SSD which is no longer sold, but I found an old listing for it that seems to indicate it was a $80 drive that was "perfect for laptops and desktop PCs".

Yes they are consumer drives, but I think that is sort of their point behind a SSD vs HDD debate, as it faces consumer level stuff [Although I do agree with the caveats that they were doing shucking before which is like the worst case scenario for hard drives].  Some of the server grade stuff costs a fortune (and yes, there are benefits to that)..but for some smaller businesses it can make sense running consumer stuff instead of enterprise stuff and in this case it gives us a good look into at least some of the brands.

 

20 hours ago, LAwLz said:

For crying out loud they even list "DELLBOSS VD" as one of the "drives" they use. The problem is that "DELLBOSS VD" is not even an SSD. It's a PCIe to M.2 expansion card. So we have no idea which SSDs they even put in that.

Given it's dell, likely a WD or their own branded Dell M.2 drive [which in effect would be WD] (i.e. non-descript only the logo and maybe if you look hard enough a model number).  An example of why they might just write DellBoss VD https://www.dell.com/en-ca/shop/dell-m2-sata-class-20-2280-solid-state-drive-256gb/apd/aa615517/storage-drives-media?gacd=9683780-3008-5761040-266662033-0&dgc=st&gclid=EAIaIQobChMIjcjy6-2t8wIV3h-tBh2KKgg9EAQYBCABEgIkYfD_BwE&gclsrc=aw.ds&nclid=BjAcUkJSdx_XE7ZD6on1G_ZrSwISboZQHshkQQJ0A6ICfXU8q-i3Lj37zCkrWQru

 

Looking at the picture, no real markings (which I've experienced before with dell hardware)...but given it's coming from what appears to be M.2 raid solution to PCI I would suspect that the SSD's aren't exactly bottom of the barrel drives.

 

 

Again, not really denying that there is likely a lot not being said in the article, it's just that it's not as stretched as you are making it out to be (in terms of sample sizes and scaling)

[LAwLz]

43 minutes ago, valdyrgramr said:

Well, is the point not to put them through hell to test the reliability?

Not sure how to explain it, but the problem is that they are not actually testing what consumers will see. Running an SSD through a datacenter workload for 1 year is not the same as let's say running a consumer SSD through a typical workload for 3 years. It's more akin to testing the reliability of a Toyota Prius by driving it through the Sahara desert. The test doesn't give you any information at all about how well it will work as a family car in a big city.

 

Why is it not the same? Because in a DC you can have a very different workload characteristics. For example in a standard home PC you will probably have plenty of short bursts of mostly reads but also some writes. As a result, that's what most consumers SSDs are aimed at. They got things like an SLC cache, and are programmed to shuffle data around in the cells in a particular way and so on. They are also made to operate in a particular environment, like a case with fairly good airflow. The temperatures inside DC servers could potentially be much higher than inside a home PC. Not to mention that DC SSDs often have significantly higher write endurance. You might think that high write endurance means they are more reliable, but then thing is that at a certain point it doesn't matter how much write endurance the SSD got.

If I'm only going to write 20TB of data to a drive in its lifespan then it doesn't matter if it has 300TB of endurance or 3,000TB of write endurance.

 

 

Don't judge a fish by its ability to climb a tree.

 

 

58 minutes ago, valdyrgramr said:

You can't use averages because each user is unique. 

I think you are overestimating how "unique" peoples' workloads really are.

I bet 90% of the average computer users will fit within the "mostly reads, writes are small and bursty, and reads are either individual files or sequential reads".

 

 

1 hour ago, valdyrgramr said:

As for the RMA thing, that's flawed too.  It doesn't realistically tell you drive failure rates and is purely assumption based.  Why?  That information is only telling you that many people did the RMA process, but that doesn't mean the drive actually died and it ignores the people who had a dead drive then bought new ones.  You can send a drive you think in is dead, but that doesn't mean it was dead that just means you filled out the paperwork and sent it.  It could be a bad SATA port or something else.

Yes but since those things (incorrect RMAs or people not RMA'ing when they should have) apply to all drives that were RMA'd equally it should even itself out.

Even if you can find some potential flaws in the statistics, I still think it is way more representative of what consumers can expect than someone with a handful of consumer SSDs putting it in a datacenter and then report on how quickly they die.

[Kisai]

23 minutes ago, wanderingfool2 said:

 

 

Given it's dell, likely a WD or their own branded Dell M.2 drive [which in effect would be WD] (i.e. non-descript only the logo and maybe if you look hard enough a model number). 

Dell NVMe drives are always Toshiba, SK Hynix, or Samsung P981 (basically the OEM-only version of the EVO drives.) I never paid attention to the SATA SSD's in the cheaper laptops. Suffice it to say that if they were using SATA SSD's for cheapness, if they were QLC, DRAM-less drives they likely slowed down after being used for 6months. 

 

So the reliability metrics are completely different from mechanical drives, which don't slow down over time, and don't wear out from writing.

 

edit: come to think of it, there is a way to get a list of what Dell is using by getting a list of the firmwares.

source dell.com:

Micron 2200S, 1100

Toshiba KBG20ZMS128G/256G/512G

Toshiba KSG60ZSE256G/KSG60ZSE512G

Toshiba KSG50ZMV256/512G

Toshiba KXG50ZNV256G/512G/1T02

Intel SSDPEMKF256G8/SSDPEMKF512G8/SSDPEMKF010T8

Intel SSDPEKKF256G8/SSDPEKKF512G8/SSDKEKKF010T8

SK Kynix SC311 , SC401, BC501, PC601, SC313, PC401

WD SN730, SN520

ADATA SU810NS38 128/256G SATA

LiteOn CA3-8D256/512 CV8

Sandisk A400

 

Of course that list will be incomplete since some SSD's never have firmware updates.

 

[jagdtigger]

12 hours ago, Dabombinable said:

Were those HDD by chance in slots directly above and below each other, and in non rubber mounts?

The HDDs are on their side in a vertical position, as far as i can remember they "died"* in a random order..... Most of their time was spent in a synology ds416, last year i moved them into a nas i built and has this backplane:
https://icybox.de/product.php?id=333

 

 

*One proper toast, the remaining 2 got kicked out by ZFS for too many errors (and started piling up bad sectors).

[LAwLz]

20 minutes ago, wanderingfool2 said:

Do you actually have the full report link, with the numbers?  [A link to them]  If so I think you were reading a different report than I was

Here is the report I looked at.

https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-2021/

 

 

 

28 minutes ago, wanderingfool2 said:

I've looked through their site and while there is some correlation, it appears the Q1/Q2 2021 stats that were used were not included in numbers given out (yet were used in their conclusion).  Don't get me wrong, I don't think this is a fully fair test, but it's better to have more of the facts (as you'll see below, as I address some of your talking points)

 

Sample size in report was 1666 "SSD"s and 1297 HDD's so no.  (As can be seen in their original table, which was even posted in the OP's thread)

 

"7 SSD failures" is not true, they experienced 17 failures.  You are looking at the older data.  They also stated their compared the HDD's failures over the same period of time, which totaled to 25 HDD failures.  Ultimately they had 1394 Seagate drives, 40 micron drives and 176 Dell drives (based on where you were pulling your numbers from) plus some more which I couldnt find references to...and even then Seagate made up the "7" drive failures during that time.  [Which again, there are clearly 10 more drive failures happening somewhere, but I don't think that data was released].

 

The "551" number you grabbed is over a period of 7 years, if you did read what they said they specifically said it wasn't a good comparison to do things like that...which is why they later went on to utilize only the first 14.3 months of data they had of the HDD's...which lines up with the SSD numbers.

Yep, you're right. I didn't like that they were hiding a bunch of numbers in the SSD vs HDD article so I went looking for the full report, not realizing they were measuring different things.

I still think their data is full of holes though and should not be taken seriously.

 

 

30 minutes ago, wanderingfool2 said:

Yes they are consumer drives, but I think that is sort of their point behind a SSD vs HDD debate, as it faces consumer level stuff [Although I do agree with the caveats that they were doing shucking before which is like the worst case scenario for hard drives].  Some of the server grade stuff costs a fortune (and yes, there are benefits to that)..but for some smaller businesses it can make sense running consumer stuff instead of enterprise stuff and in this case it gives us a good look into at least some of the brands.

Well like I said, this article is only really relevant to people who build data centers in an improper way. I'd argue that if you are going to build a DC, then you should be able to afford building it properly. If you can't afford that, then you probably don't need a DC to begin with and you're better off using XaaS.

 

32 minutes ago, wanderingfool2 said:

Again, not really denying that there is likely a lot not being said in the article, it's just that it's not as stretched as you are making it out to be (in terms of sample sizes and scaling)

Yeah you're right. I'll edit my post.

[leadeater]

36 minutes ago, wanderingfool2 said:

Yes they are consumer drives, but I think that is sort of their point behind a SSD vs HDD debate, as it faces consumer level stuff

What consumer is going to have the amount of active power on time and also write workload as Backblaze, something that is far closer to zero than it is toward everyone, far closer.

 

That's like testing engine reliability for typical sedans by running them at 6000 RPM all day every day then saying the failure rate is high. Not too sure how many people drive around 24/7 and also in 1st gear.