Do hard drives lose data with time?

[MoDz]

Hi. I have external hard drives which I use to backup my data. I'm trying to find out the optimal time between full backups, but in order to make a good decision, I need to know if the modern hard drives lose data (even 1 bit) with time.

Thanks.

[Akolyte]

I can't say for sure, but I think they do over a long period of time.  Just plug your backup drive in every once and a while and browse it, if your worried. 

[alpenwasser]

They don't so much "lose" data as data is corrupted by several factors, the ones

that come to mind being random bit flips (cosmic radiation, environmental radiation

etc.) and physical degradation of the drive itself (there might be others).

Unless you store the drive in really adverse conditions (high humidity, vibrations,

extreme cold or heat, or worst of all, conditions which vary quickly and often),

the second one isn't really much of a concern (unless the drive is old, old drives

are much more likely to fail), but random bit flips due to outside influences can

and do happen.

 

You don't necessarily need to back up your data more often to reduce the risk of

this happening (you can never be 100% secured against it of course), but you should

do periodic integrity checks of your data.

Personally I recommend ZFS for that, this is exactly what it's built for. I'm sure

there are alternatives, but I'll leave that to others since I'm not informed well

enough to really know anything useful about them.

However: As long as you do not have your entire system hardened against

random data corruption, starting to worry about your backup drive is rather

pointless. Your original data is just as susceptible to corruption (if not more,

depending on circumstances) as your backup drive. You can store your backup

drive in a protective enclosure, even  protecting it against radiation etc., but

you are unlikely to do that with your original drive in your machine. Also, even

if both the original drive and the backup drive were integrity-checked periodically,

you would still need ECC memory to protect the data while it's going through your

system, since data can get corrupted during that process as well (that's actually

more likely than it being corrupted while it's on your disk from what I've read,

though I don't have much personal experience with that, unless the disk starts

physically failing).

 

Bottom line: Corruption happens, protection is possible. But if you want to start

protecting your data against corruption, you need to harden the entire data

pipeline, not just the backup drive.

EDIT:

The frequency of this can vary wildly. You may have a random bit flip (or several)

tomorrow, or you may go 20 years without one. This depends on many factors (radiation

varies depending on where you are, for example if you live higher up over sea level

you are more susceptible, and this is leaving out other factors), not least of which

is some simple luck. But it is a very real problem, or else ECC memory would not

have been invented, and neither would ZFS have been engineered the way it is.

[SteepGnomeKing]

No, Unless you are talking like 15+ years, than yes.

 

They do not lose any data within a couple of years.

 

8+ years they will be perfectly fine.

[wpirobotbuilder]

Can't tell, but had an HDD without use for 10 years, and data was there...

The question is, did you check all of the data to make sure it was pristine?

 

It might be a single random bit flip which makes a text document incorrect, or it might contain information about an important project you're working on. Either way, you are at risk of data loss.

 

Also, everything alpenwasser said. Sad to say that not all solutions that come out of the storage industry protect against data loss from silent corruption. ZFS is admirable in this regard.

[MoDz]

Thank you all for the answers.

 

They don't so much "lose" data as data is corrupted by several factors, the ones
that come to mind being random bit flips (cosmic radiation, environmental radiation
etc.) and physical degradation of the drive itself (there might be others).

Unless you store the drive in really adverse conditions (high humidity, vibrations,
extreme cold or heat, or worst of all, conditions which vary quickly and often),
the second one isn't really much of a concern (unless the drive is old, old drives
are much more likely to fail), but random bit flips due to outside influences can
and do happen.
 
You don't necessarily need to back up your data more often to reduce the risk of
this happening (you can never be 100% secured against it of course), but you should
do periodic integrity checks of your data.

Personally I recommend ZFS for that, this is exactly what it's built for. I'm sure
there are alternatives, but I'll leave that to others since I'm not informed well
enough to really know anything useful about them.

However: As long as you do not have your entire system hardened against
random data corruption, starting to worry about your backup drive is rather
pointless. Your original data is just as susceptible to corruption (if not more,
depending on circumstances) as your backup drive. You can store your backup
drive in a protective enclosure, even  protecting it against radiation etc., but
you are unlikely to do that with your original drive in your machine. Also, even
if both the original drive and the backup drive were integrity-checked periodically,
you would still need ECC memory to protect the data while it's going through your
system, since data can get corrupted during that process as well (that's actually
more likely than it being corrupted while it's on your disk from what I've read,
though I don't have much personal experience with that, unless the disk starts
physically failing).
 
Bottom line: Corruption happens, protection is possible. But if you want to start
protecting your data against corruption, you need to harden the entire data
pipeline, not just the backup drive.


EDIT:
The frequency of this can vary wildly. You may have a random bit flip (or several)
tomorrow, or you may go 20 years without one. This depends on many factors (radiation
varies depending on where you are, for example if you live higher up over sea level
you are more susceptible, and this is leaving out other factors), not least of which
is some simple luck. But it is a very real problem, or else ECC memory would not
have been invented, and neither would ZFS have been engineered the way it is.

 

Yes, ZFS is an option, but it's not the most convenient one. That's why I use RAR archives to backup my data. WinRAR saves 256-bit checksums of all files within an archive, so I can check their integrity whenever I want. Just in case, I have the archives' checksums as well. Before backing up, I check the integrity of the files on my internal drive (I have their checksums, too), so I won't backup corrupt files. If I find something that is corrupt:

 

1. On my internal drive - I'll recover the corrupt files using my backup drives.

2. On some of my backup drives - I'll use the 50-percent recovery record of the archives (it can fix a lot of data corruption). If that fails, I'll use the other full backup (I store at least 2 full backups on a single drive). If that fails, the second full backup has a recovery record, too. If that fails, I have another backup drive utilizing the same scheme - at least 2 full backups + 50% recovery record. If that fails, I have my most important data on a USB flash drive (again, using recovery records). If everything fails, I'm screwed. But what are the odds? And I'm considering adding one more drive to backup to...

 

That should be pretty reliable, shouldn't it?

[alpenwasser]

Yes, ZFS is an option, but it's not the most convenient one.

Well, ZFS is very convenient. However, getting a system with it up and running might

be a bit of a trick depending on your know-how. But once it's up and running it's a

breeze to work with, at least the routine stuff I use.

 

That's why I use RAR archives to backup my data. WinRAR saves 256-bit checksums of all files within an archive, so I can check their integrity whenever I want. Just in case, I have the archives' checksums as well. Before backing up, I check the integrity of the files on my internal drive (I have their checksums, too), so I won't backup corrupt files. If I find something that is corrupt:

1. On my internal drive - I'll recover the corrupt files using my backup drives.

2. On some of my backup drives - I'll use the 50-percent recovery record of the archives (it can fix a lot of data corruption). If that fails, I'll use the other full backup (I store at least 2 full backups on a single drive). If that fails, the second full backup has a recovery record, too. If that fails, I have another backup drive utilizing the same scheme - at least 2 full backups + 50% recovery record. If that fails, I have my most important data on a USB flash drive (again, using recovery records). If everything fails, I'm screwed. But what are the odds? And I'm considering adding one more drive to backup to...

Hm, OK, I haven't worked with WinRAR in a very long time, let alone its advanced

features, so just to be sure I have one primary question: It sounds like your plan

does protect against corruption of the WinRAR checksums themselves via the

recovery record, is that correct?

In ZFS there are mechanisms to prevent failure in such a case. Basically, you

have redundancy (in any reasonable setup at least), and if the checksum doesn't

match the data for whatever reason (one of them being corruption of the checksum),

then ZFS will attempt to restore the data from its redundancy records.

I'm also assuming you are protected against corruption of the originals' checksums?

 

That should be pretty reliable, shouldn't it?

Yes, this actually looks to be a pretty proper backup procedure, certainly more

thorough than most I've seen. Provided you're actually protected against checksum

corruption then you have plenty of redundancy to restore from and this will do as

OK of a job as can be done within the limits of reason and you don't need to worry

about HDD's losing data, since if this all works as planned you could recover from

that.

[MoDz]

Well, ZFS is very convenient. However, getting a system with it up and running might

be a bit of a trick depending on your know-how. But once it's up and running it's a

breeze to work with, at least the routine stuff I use.

 

Hm, OK, I haven't worked with WinRAR in a very long time, let alone its advanced

features, so just to be sure I have one primary question: It sounds like your plan

does protect against corruption of the WinRAR checksums themselves via the

recovery record, is that correct?

In ZFS there are mechanisms to prevent failure in such a case. Basically, you

have redundancy (in any reasonable setup at least), and if the checksum doesn't

match the data for whatever reason (one of them being corruption of the checksum),

then ZFS will attempt to restore the data from its redundancy records.

I'm also assuming you are protected against corruption of the originals' checksums?

 

Yes, this actually looks to be a pretty proper backup procedure, certainly more

thorough than most I've seen. Provided you're actually protected against checksum

corruption then you have plenty of redundancy to restore from and this will do as

OK of a job as can be done within the limits of reason and you don't need to worry

about HDD's losing data, since if this all works as planned you could recover from

that.

 

I'm not very Linux-savvy, so I'll need some time to get used to Linux and ZFS. I may do this in the future, but for now WinRAR works decently.

 

Actually I don't know whether the recovery record protects the checksums. Thank you for mentioned that. I just asked the developers of WinRAR. I'll wait for their answer, but even if it doesn't, from now on as soon as I create and test the archives, I'll export the checksums (WinRAR has such option) and store them on all drives I own.

[sperkowsky]

there not suposed to but with computer anything can happen

[alpenwasser]

I'm not very Linux-savvy, so I'll need some time to get used to Linux and ZFS. I may do this in the future, but for now WinRAR works decently.

Actually I don't know whether the recovery record protects the checksums. Thank you for mentioned that. I just asked the developers of WinRAR. I'll wait for their answer, but even if it doesn't, from now on as soon as I create and test the archives, I'll export the checksums (WinRAR has such option) and store them on all drives I own.

Linux + ZFS (or BSD + ZFS, or Solaris + ZFS) is not the only solution to your

problem, and if you export your checksums to all your drives and implement the

procedure you've described above you will have a pretty decent protection against

data corruption IMO. You could always export the checksums to other media

as well (flashdrives, optical media, w/e) for additional protection.

And in any case, ZFS is only really able to play out that integrity card if you're

running it on an ECC-memory system, otherwise you actually run the risk of corrupting

your data while doing a data integrity check on your FS (data is loaded into memory,

corrupted in memory, ZFS detects the discrepancy, assumes the memory data is correct

and then "fixes" your data on the disk).

At least that's what I've read. People seem to disagree a bit though on how severe

the ramifications would be. Some say it might kill all your data, others seem to

think it would only corrupt that specific file, nobody really seems to know for

certain, and if they do, they haven't told the internet yet (or I've been too dumb

to find it :rolleyes: ).

That's actually one thing I'd like to fix about my server. I did not plan on using

ZFS when I built it since it wasn't production-ready on Linux yet, so at some point

I'd like to switch my data to an ECC-enabled system.