Backblaze releases some more flawed HDD failure statistics for Q3

[porina]

1 hour ago, TheRandomness said:

When will they learn you can't rely on a consumer drive in a datacenter...

They already have. They also learnt you can't rely on more expensive grades and they will also fail. No one makes a 100% reliable disk, and until then, this is how they decide to optimise their costs.

[Oshino Shinobu]

1 minute ago, Jovidah said:

You haven't even mentioned one of the biggest problems. The drives were mounted in all kinds of different storage solutions. So they weren't even given the same treatment. The major improvement the Seagate drives have suddenly seen is quite telling - you'd expect the larger disks to be less reliable, not more.

Not even going to bother with the rest of the flaws. It's garbage. If a student turned this in, they'd fail.

I mention it at the end of the second paragraph. 

 

"They ommit things such as the age of the drives, the proximity of the drives, the temperatures they were running at, the case/rack revision they were mounted in and so on"

 

That was one of the biggest issues in their older testing. Backblaze had two or three different mounting systems, some of which didn't have any vibration dampening at all. 

[Curufinwe_wins]

5 minutes ago, Jovidah said:

You haven't even mentioned one of the biggest problems. The drives were mounted in all kinds of different storage solutions. So they weren't even given the same treatment. The major improvement the Seagate drives have suddenly seen is quite telling - you'd expect the larger disks to be less reliable, not more.

Not even going to bother with the rest of the flaws. It's garbage. If a student turned this in, they'd fail.

Sorry 0/10 please try again. This isn't how hard drives work.

 

This is literally tantamount to claiming a 500 GB 850 Evo is less reliable than a 250 GB 840 Evo (actually a great comparison since parallelism in nand is identical). The obvious statement is "that's bullshit" because the 840 Evo had a controller that often shit the bed. Which guess what, that is also what matters the most for each NEW generation of hard drives (better designed armatures/controller/etc not the disks themselves being more or less reliable).

[Jovidah]

5 minutes ago, Curufinwe_wins said:

Sorry 0/10 please try again. This isn't how hard drives work.

 

This is literally tantamount to claiming a 500 GB 850 Evo is less reliable than a 250 GB 840 Evo (actually a great comparison since parallelism in nand is identical). The obvious statement is "that's bullshit" because the 840 Evo had a controller that often shit the bed. Which guess what, that is also what probably matters the most for each NEW generation of hard drives (better designed armatures/controller/etc not the disks themselves being more or less reliable).

Actually it's not bullshit. It shows in the hardware.fr numbers. The larger disks have higher RMA rates. The larger disks often have more heads / platters. Therefore, more parts that can die. Admittedly, there can be some difference between generations, but when the age-difference isn't too big it tends to hold up.

Now whether they have a higher failure rate per GB is a different matter... and an interesting one at that.

 

It's a bit of a different matter in SSD territory, given how these are still a relatively new and unmature product (compared to HDDs). You still see wild differences between products and RMA rates there.

 

 

[Dabombinable]

4 minutes ago, Curufinwe_wins said:

Sorry 0/10 please try again. This isn't how hard drives work.

 

This is literally tantamount to claiming a 500 GB 850 Evo is less reliable than a 250 GB 840 Evo (actually a great comparison since parallelism in nand is identical). The obvious statement is "that's bullshit" because the 840 Evo had a controller that often shit the bed. Which guess what, that is also what matters the most for each NEW generation of hard drives (better designed armatures/controller/etc not the disks themselves being more or less reliable).

Fun fact: I've had 1 HDD (Samsung Spinpoint 250GB) fail electrically due to it using ESM chips not Samsung like my others of the same model, and another fail mechanically since it got dropped. I switched the control boards around, and I got 1 working HDD out of it, and one that I could simply toss. And all of my other dead HDD (Maxtor, Quantum, Western Digital and Seagate only need the control board replaced-mechanically they are still good)

[Curufinwe_wins]

18 minutes ago, Jovidah said:

Actually it's not bullshit. It shows in the hardware.fr numbers. The larger disks have higher RMA rates. The larger disks often have more heads / platters. Therefore, more parts that can die. Admittedly, there can be some difference between generations, but when the age-difference isn't too big it tends to hold up.

Now whether they have a higher failure rate per GB is a different matter... and an interesting one at that.

 

 

No they don't. http://www.hardware.fr/articles/947-6/disques-durs.html

 

I challenge you to sell me that within the same hardware lineup (which does not mean same generation ofc) there is any statistical correlation between drive size and failure rate.

 

5/6 To:

• 1,70% WD Red WD60EFRX
• 0,00% WD Red WD50EFRX
• 0,47% WD Green WD60EZRX

4 To:

• 0,79% Seagate NAS HDD ST4000VN000
• 1,66% WD Red WD40EFRX
• 1,70% WD Green WD40EZRX

3 To:

• 1,55% Seagate NAS HDD ST3000VN000
• N/A for WD RED
• 0,93% WD Green WD30EZRX

2 To:

• 0,40% Seagate NAS HDD ST2000VN000
• 0,97% WD Red WD20EFRX
• 1,38% WD Green WD20EZRX

http://www.seagate.com/www-content/product-content/nas-fam/nas-hdd/en-us/docs/100724684.pdf 

 

The 5/6/8 TB drives all use significantly improved generational design which is why you see something like this:

5 / 6 To :

• 1,94% Toshiba PX3010E-1HQ0
• 1,70% WD Red WD60EFRX
• 0,47% WD Green WD60EZRX
• 0,00% WD Red WD50EFRX

 

 

 

[Jovidah]

11 minutes ago, Curufinwe_wins said:

No they don't. http://www.hardware.fr/articles/947-6/disques-durs.html

 

I challenge you to sell me that within the same hardware lineup (which does not mean same generation ofc) there is any statistical correlation between drive size and failure rate.

 

5/6 To:

• 1,70% WD Red WD60EFRX
• 0,00% WD Red WD50EFRX

4 To:

• 0,79% Seagate NAS HDD ST4000VN000
• 1,66% WD Red WD40EFRX

3 To:

• 1,55% Seagate NAS HDD ST3000VN000
• N/A for WD

2 To:

• 0,40% Seagate NAS HDD ST2000VN000
• 0,97% WD Red WD20EFRX

http://www.seagate.com/www-content/product-content/nas-fam/nas-hdd/en-us/docs/100724684.pdf 

 

The 5/6/8 TB drives all use significantly improved generational design which is why you see something like this:

5 / 6 To :

• 1,94% Toshiba PX3010E-1HQ0
• 1,70% WD Red WD60EFRX
• 0,47% WD Green WD60EZRX
• 0,00% WD Red WD50EFRX

 

The point is that seagate/WD/HGST/Toshiba RARELY if ever update their lower capacity drives for consumer use (within recent times), and generally speaking you go straight up in generational improvements by increasing capacity.

 

 

So in the numbers you showed:

 

Seagate NAS:

2 TB: 0,4% failure

3 TB: 1,55% failure

4 TB: 0,79% failure

 

WD Red:

2 TB: 0,79% failure

4 TB: 1,55% failure

6 TB: 1,70% failure

(the 5 is mentioned in Italics because it has a low sales volume, so it's statistically useless).

 

If you exclude the 3 TB as an outlier - it is a known 'lemon' after all, it adds up pretty damn well.

Admittedly, all things being equal it might be feasible for the larger ones to be on the same level. But 8 TB as more reliable than the 2 and 4 TB? Inconceivable.

 

Feel free to click back to older versions. It's a pretty consistent pattern. Within the same line you see RMA rates rising - slightly - as capacities increase. Even the 3 TB 'lemon' doesn't always show up in the data. If anything the 3TB versions tend to be 'weird'. They often perform worse than both the 2 and 4 TB (not just for Seagate, also for WD). 

[Curufinwe_wins]

27 minutes ago, Jovidah said:

snip

I have 3 data points, if I exclude one of my 3 data points I can make a line. CON-FUCKING-GRATULATIONS. Statistical significance is a thing.

 

Also see the edit that included the WD-Green lineup.

 

1,38/1,93/1,70/0,47 Yea tell me how statistically significant that relationship is as well. 

 

Also almost ever non-SMR 8TB drive is helium sealed, and yea that does make a huge difference (literally the HGST drive used for "consumers" and sold under the WD brand as well is the EXACT same physical drive as the enterprise one.)

 

Oh random thing, but based on the way the smaller (and older) WD-green drives are built (which is NOT used for the larger ones) you 100% would expect them to fail MUCH more rapidly under the type of workloads NAS/Server/Blackblaze puts on them. Which is why Blackblaze has a 8% annual failure rate for the 2TB model. This is not a big deal for consumer desktop workloads though.

 

 

 

EDIT: Oh, WD actually fixed the IntelliPark stupidity about two years ago, but since the blackblaze 2TB drives are generally older than that that improvement doesn't show up.

 

[SpaceGhostC2C]

2 hours ago, TheRandomness said:

When will they learn you can't rely on a consumer drive in a datacenter...

Why not? As long as all data has some form of redundancy or another, constantly replacing cheap drives may or may not be cheaper than infrequently replacing expensive drives. As long as no catastrophic failure takes place, It's all the same to the end user, so whatever brings the lowest cost to them is fine.

[79wjd]

@Oshino Shinobu It's meaningless, and I have no issue recommending or buying a Seagate drive. Although with that said, if two drives are the same price/capacity and both offer the performs/noise levels/etc... that I need, then I'd still get the WD drive (not a 6tb though) based on these results. If a drive has a lower failure rate in an enterprise environment, then I'd be willing to take the "risk" that it would also hold up better in its intended workflow (of course, that may very likely not be the case, but I'd still a "non-risk" that I'd be willing to take). So while the results are meaningless, they're not entirely meaningless (I wouldn't use it to bash/criticize Seagate or even to promote WD, but for my own personal use, I'd find the data somewhat useful....if that makes sense).

 

Now, of course there are other issues with the data which would make all of the above irrelevant (like a lack of information regarding the age of the drives and/or the environment they were in).

[Sauron]

Over half of their drives are seagate st4000dm000... how can you make a fair statistic when the sample sizes differ so heavily?

[SpaceGhostC2C]

1 minute ago, Sauron said:

Over half of their drives are seagate st4000dm000... how can you make a fair statistic when the sample sizes differ so heavily?

In fact, the difference in sample sizes doesn't matter at all. All that matters is for each of the samples to be sufficiently large. If all are at least of size N* (where N* depends on the statistic you intend to compute), one being N* and the other 1000xN* is irrelevant (you'll just have a different confidence interval for them - something the don't report).

[Jovidah]

4 minutes ago, Curufinwe_wins said:

I have 3 data points, if I exclude one of my 3 data points I can make a line. CON-FUCKING-GRATULATIONS. Statistical significance is a thing.

Actually... yes, it's pretty valid. It's only one out of 6 by the way. And as I said I based my statement not just on that one one comparison.

But even if you don't exclude it, it will still be statistically significant (and give a decent correlation). In fact, in this dataset, given the large numbers, pretty much any difference, regardless of how tiny or meaningless, is likely to be statistically significant.  

 

And as I said... the 3 TBs are (and I don't know exactly why) a returning issue, falling behind the 2 and 4's in a lot of tests. The Seagate 3 TB is a known outlier. 

If you're serious about arguing this point, you should throw all the disks in there from the last few years, taking all datapoints into account. But to be honest I'm too lazy to throw all that crap into SPSS just to calculate what I can see with my own eyes; generally, within the same type / line of product, larger capacity means higher RMA rate.

4 minutes ago, Curufinwe_wins said:

 

Also see the edit that included the WD-Green lineup.

 

1,38/1,93/1,70/0,47 Yea tell me how statistically significant that relationship is as well. 

Are you aware of how statistical significance is calculated? Either you use the raw numbers per drive - which we don't have; and it'll always be significant even if there was no actual difference (due to sample size). Or you use just the 4-5 numbers... and it'll never be significant (because a sample so low never will be). 

 

Besides. Arguing p is pussies. R² / eta² is where it's at.

4 minutes ago, Curufinwe_wins said:

 

Also almost ever non-SMR 8TB drive is helium sealed, and yea that does make a huge difference (literally the HGST drive used for "consumers" and sold under the WD brand as well is the EXACT same physical drive as the enterprise one.)

 

Oh random thing, but based on the way the smaller (and older) WD-green drives are built (which is NOT used for the larger ones) you 100% would expect them to fail MUCH more rapidly under the type of workloads NAS/Server/Blackblaze puts on them. Which is why Blackblaze has a 8% annual failure rate for the 2TB model. This is not a big deal for consumer desktop workloads though.

 

 

 

 

So your point is that ' totally different drives can be more reliable' . Yes... I already admitted that much. Still doesn't mean an 8 GB version would suddenly be several times more reliable than the 4 TB versions. 

[Jovidah]

2 minutes ago, SpaceGhostC2C said:

In fact, the difference in sample sizes doesn't matter at all. All that matters is for each of the samples to be sufficiently large. If all are at least of size N* (where N* depends on the statistic you intend to compute), one being N* and the other 1000xN* is irrelevant (you'll just have a different confidence interval for them - something the don't report).

Yes and no. Even when disregarding the inequality of the conditions, there's still the problem that a lot of the samples are far too small (given the small incidence rate of a failure). A lot of the drives are also tested far too short to really give an honest picture of how they perform.

Deriving AFR from a bunch of drives averaging just 90 days of drive times (in the case of the WD60EFRX) - even if it's in line with industry standards - is just a complete load of bollocks. And that's just one example. 

 

Admittedly, the hardware.fr numbers have their problems too. But at least there you have some degree of randomization - at least within product categories.

 

[Jovidah]

41 minutes ago, SpaceGhostC2C said:

Why not? As long as all data has some form of redundancy or another, constantly replacing cheap drives may or may not be cheaper than infrequently replacing expensive drives. As long as no catastrophic failure takes place, It's all the same to the end user, so whatever brings the lowest cost to them is fine.

Actually you're right from an economic point of view. I'm pretty sure it was their calculation as well; it's cheaper to just have some guy in there full-time swapping all the drives than to buy more expensive drives (as the cost premium is pretty big). As long as you have your redundancy worked out properly this can make decent sense. But you have to factor in above-average numbers of dead drives in your planning.

It also makes the data derived from such non-intended usecases pretty pointless to generalize. 

 

[SpaceGhostC2C]

8 minutes ago, Jovidah said:

Yes and no.

 

You mention a lot of things I didn't talk about. There is no ambiguity in what I said, though: you either have a large enough sample to say something meaningful, or you don't. And if you have it, differences in sample size don't matter, it's just a matter of computing the relevant "sigma squared" and assess the reliability of your statistics.

 

Whether the sample mean they are measuring is a relevant measure of HDD reliability is something i didn't enter in.

[Sauron]

31 minutes ago, SpaceGhostC2C said:

In fact, the difference in sample sizes doesn't matter at all. All that matters is for each of the samples to be sufficiently large. If all are at least of size N* (where N* depends on the statistic you intend to compute), one being N* and the other 1000xN* is irrelevant (you'll just have a different confidence interval for them - something the don't report).

Sure, but they can't use a 34k unit sample size and then a 200 unit one and call the comparison fair... especially since the statistics have double decimal figures. They should all have at least 10000 units for it to work as you said.

[Coaxialgamer]

1 hour ago, Curufinwe_wins said:

Sorry 0/10 please try again. This isn't how hard drives work.

 

This is literally tantamount to claiming a 500 GB 850 Evo is less reliable than a 250 GB 840 Evo (actually a great comparison since parallelism in nand is identical). The obvious statement is "that's bullshit" because the 840 Evo had a controller that often shit the bed. Which guess what, that is also what matters the most for each NEW generation of hard drives (better designed armatures/controller/etc not the disks themselves being more or less reliable).

i wouldn't necessarily compare it to the EVO series. 

Failure rates on hard drives largely depend on the number of platters and much less on controllers.

[SurvivorNVL]

Still wouldn't own Seagate if you paid me.  Nor would I own an HDD beyond cold-storage.  SSD train all the way.  Corsair, Intel, Kingston, and Samsung.

[jagdtigger]

13 minutes ago, Jovidah said:

It also makes the data derived from such non-intended usecases pretty pointless to generalize. 

Not necessarily. We do not know how they were treated, but we do know its the worst case scenario for a consumer HDD. If a specific model has a unusually higher failure rate than the comparable products from other companies then it could be useful to avoid bad surprises...

[Jovidah]

10 minutes ago, SpaceGhostC2C said:

You mention a lot of things I didn't talk about. There is no ambiguity in what I said, though: you either have a large enough sample to say something meaningful, or you don't. And if you have it, differences in sample size don't matter, it's just a matter of computing the relevant "sigma squared" and assess the reliability of your statistics.

 

Whether the sample mean they are measuring is a relevant measure of HDD reliability is something i didn't enter in.

Again yes and no.

 

I agree that, pure statistics-wise there is indeed no problem to have wildly different group sizes, as long as they hall hit a certain minimum level. So that's a yes.

But in this case some of the groups are simply too small (too few drives / too few drive hours) to give a representative measurement, given the low incidence of drive failures. IIRC that's an internal validity issue.

 

If all the numbers were multiplied by a 100, and all the drives had at least a few years under their belt I wouldn't have much of a problem with it. Right now there's a lot of capitalization on chance going on.

 

 

[1823alex]

3 hours ago, TheRandomness said:

When will they learn you can't rely on a consumer drive in a datacenter...

You do realize the point of them using consumer drives in a datacenter is to test for failure rates and such on consumer drives, in order to inform the consumer.

[Jovidah]

3 minutes ago, jagdtigger said:

Not necessarily. We do not know how they were treated, but we do know its the worst case scenario for a consumer HDD. If a specific model has a unusually higher failure rate than the comparable products from other companies then it could be useful to avoid bad surprises...

Several articles already mentioned how the different drives were given different treatment (for example mounted in different cases). So even the numbers within the table cannot be compared to eachother.

The center theme of the last 2 pages was "these numbers are completely unreliable and utter nonsense". Utter nonsense is rather lacking in predictive value.

 

But imagining for a moment that the numbers are reliable and valid. Then there's the problem that their test constitutes a test beyond what any of these drives were designed for, or will be used for. It's like taking a regular consumer Sedan, using them to try Paris-Dakar. How useful are those results when you buy a car to buy groceries, or just drive on highways? Unless you're planning to actually take it on a long drive through endless deserts and other off-road terrain, it's pretty useless. 

Predictive validity is nada. You might as well throw them off a building, see which one bounces up the highest, and rank them as such. 

[Jovidah]

12 minutes ago, 1823alex said:

You do realize the point of them using consumer drives in a datacenter is to test for failure rates and such on consumer drives, in order to inform the consumer.

Err no. The whole point of using consumer drives was because they were cheap and they were actually available (there was a drive shortage at the time).

 

The main purpose of the publication is in my opinion nothing but free publicity. In which they were quite succesful.

If they truly wanted to inform the consumer public, they wouldn't have made this garbage. All it does is misinform. This... 'thing' contains almost every mistake you could possibly commit in a research design. My statistics teacher would have flogged them. 

 

For all intends and purposes, although it's not without its own flaws, the hardware.fr numbers are far more useful.

[AshleyAshes]

I get the rational behind you guys arguing that larger HDDs must have statistically meaningfully larger failure rates, but when your base metric for that is like 1TB-4TB drives being 'more reliable', doesn't this mean that puny HDDs from the 90's that were only hundreds of megabytes or maybe a few gigabytes must have had a negative failure rate?   As drives get larger, the technology also advances the construction quality, and this is why we've seen overall HDD failure rates to stay within a nominal statistical range.