Bit rot... a Reality?

[0nRay]

One of my friends introduced me to a new concept the other day and being a research junkie I immediately went on the interwebs to find out more but came up pretty dry. Is bit rot something I should worry about? I have lots of FLAC music files and 1080P movies on my PC. I'm worried that bit rot may degrade my files over time.

 

Q1. I did once have a very (about 6 years) old word doc get corrupted and I lost some very important info that I was logging in that document, could that have be an example of bit rot?

 

Q2. Besides my media and documents can this phenomenon ruin programs or my Windows 7 OS?

 

Q3. Does Windows 7 have systems in place to monitor bit rot and refresh data as needed or is there a program out there that would do that.

 

Q4. My friend tells me he refreshes his data every three years on his archived hard drives that sit unused. Is he overkilling it?

 

Please put your answers under the relevant question.

 

Thanks all!

[AlexStriz]

What the hell is your friend high on?

 

I have never head of "bit rot".. ever. 

 

edit: Is this actually a thing? A quick google search came up with some relative results.. 

[EmeraldFlame]

One of my friends introduced me to a new concept the other day and being a research junkie I immediately went on the interwebs to find out more but came up pretty dry. Is bit rot something I should worry about? I have lots of FLAC music files and 1080P movies on my PC. I'm worried that bit rot may degrade my files over time.

 

Q1. I did once have a very (about 6 years) old word doc get corrupted and I lost some very important info that I was logging in that document, could that have be an example of bit rot?

 

Q2. Besides my media and documents can this phenomenon ruin programs or my Windows 7 OS?

 

Q3. Does Windows 7 have systems in place to monitor bit rot and refresh data as needed or is there a program out there that would do that.

 

Q4. My friend tells me he refreshes his data every three years on his archived hard drives that sit unused. Is he overkilling it?

 

Please put your answers under the relevant question.

 

Thanks all!

 

In some cases what you call 'bit rot' could happen. It will never happen on a hard drive though.

A1. If it was on a hard drive, no. Hundreds of different things can cause file corruption. A sector on your drive could have went bad. Your power may have flickered while it was attempting to save the file. Heck, if you went by your tower and bumped it while it was on, that could even theoretically kill the file.

A2. No, it is not something you have to worry about.

A3. No, because in reality it is not an issue, and even if it was, there is no program that could fix it.

A4. Yes, your friend is mis-informed and has absolutely no clue what he is talking about.

 

The phenomenon you call 'bit rot' is something that can affect flash based memory like SSD's or USB flash drives though. Flash based memory essentially stores a small charge in its cells where an empty cell would read a 0 and a cell with voltage reads a 1. Over time if flash based memory is not used, this voltage will slowly leak and dissipate until the entire drive reads 0. For this to happen you would have to not use the memory device for more than 10 years. The simple act of plugging it into a power source will allow it to refresh its stored voltages essentially. Hard drives physically store data with magnetic fields, short of damage to the drive or exposure to extreme magnetic fields it will keep its data unaffected infinitely. 

[0nRay]

In some cases what you call 'bit rot' could happen. It will never happen on a hard drive though.

A1. If it was on a hard drive, no. Hundreds of different things can cause file corruption. A sector on your drive could have went bad. Your power may have flickered while it was attempting to save the file. Heck, if you went by your tower and bumped it while it was on, that could even theoretically kill the file.

A2. No, it is not something you have to worry about.

A3. No, because in reality it is not an issue, and even if it was, there is no program that could fix it.

A4. Yes, your friend is mis-informed and has absolutely no clue what he is talking about.

 

The phenomenon you call 'bit rot' is something that can affect flash based memory like SSD's or USB flash drives though. Flash based memory essentially stores a small charge in its cells where an empty cell would read a 0 and a cell with voltage reads a 1. Over time if flash based memory is not used, this voltage will slowly leak and dissipate until the entire drive reads 0. For this to happen you would have to not use the memory device for more than 10 years. The simple act of plugging it into a power source will allow it to refresh its stored voltages essentially. Hard drives physically store data with magnetic fields, short of damage to the drive or exposure to extreme magnetic fields it will keep its data unaffected infinitely. 

 

Oh man thanks for the info, I was starting to worry because I haven't reimaged my hard drive in three years but that being said it still runs like the day I built it. Learn something new all the time here.

 

Cheers

[alpenwasser]

Q1. I did once have a very (about 6 years) old word doc get corrupted and I lost some very important info that I was logging in that document, could that have be an example of bit rot?

Maybe, maybe not, impossible to tell. See Emerald's response.

 

Q2. Besides my media and documents can this phenomenon ruin programs or my Windows 7 OS?

Anything that is stored anywhere can be affected by bit rot. Though bit rot is by far not the only thing which can corrupt data.

 

Q3. Does Windows 7 have systems in place to monitor bit rot and refresh data as needed or is there a program out there that would do that.

I'm not up to date on Windows, so can't answer that. I've never read anything about Windows having protection mechanisms against data corruption in place though, at least not on NTFS, but just because I haven't heard of it doesn't mean it doesn't exist.

 

Q4. My friend tells me he refreshes his data every three years on his archived hard drives that sit unused. Is he overkilling it?

Just refreshing the data on the disks is not much good in and of itself though, you'd need to have a mechanism in place to ensure that the data hasn't been corrupted (checksums), and to make sure the source data hasn't been corrupted (it's no good if the data on the backup drives is still good but the data on the source drives has been corrupted and you copy the corrupted data onto the backup drives).

Personally, for any external drive I use for backups, this is what I do:

1. Create checksums for the files from the original data source (HDD); the kind of hash algorithm you use isn't that important, I usually use SHA1 (like git), but you can use MD5 or another SHA variant, w/e.

2. Store those checksums. I store them on the original HDD, the backup HDD and I print them out on paper (only the ones of really important files though, because printing out hundreds of pages of checksums for several million files is a bit expensive and time-consuming). The reason for printing them out is that the checksum files themselves can of course also become corrupted.

3. Copy files to backup drive, verify the copies against the checksum files.

4. The important files I verify against the checksums every month, the not-so-important files every six months or so.

For protecting the original data, I use ZFS on my server, and do about one scrub per month (and yes, it has ECC memory and all that).

 

What the hell is your friend high on?

 

I have never head of "bit rot".. ever. 

 

edit: Is this actually a thing? A quick google search came up with some relative results..

Yes, this is a thing, it's just not a thing regular users normally are informed about or care about because it's not frequent enough, and even when it does happen, it's practially impossible to tell whether the data was corrupted by bit rot or by some other mechanism most of the time.

One of the main reason for developing ZFS is actually to counteract the effects of bit rot (if you set it up correctly). Mind you, it's not just about that, as there are mechanisms in place in ZFS which are intended to prevent and repair all kinds of corruption, but bit rot is among them.

Now, whether or not this is something you actually need to worry about is another question. For example, if you have lots of compressed data on your drive:

Compressed archives are easily corrupted, because compressed files have little redundancy. The adaptive nature of the compression scheme means that the compression tables are implicitly spread all over the archive. If you lose a few blocks, the dynamic construction of the compression tables becomes unsynchronized, and there is little chance that you could recover later in the archive.

source

 

This actually happened to this guy (he also describes some countermeasures in that post).

 

In some cases what you call 'bit rot' could happen. It will never happen on a hard drive though.

I have not been able to find any solid empirical data on this, but I have never come across any credible information indicating that HDDs are not affected by this issue and many (enterprise) storage people saying it is, especially when doing research on ZFS. How probable it is (especially compared with other storage media such as flash, tapes, optical disks etc.) is another question though.

 

A2. No, it is not something you have to worry about.

Depends on how probable it is and how probable something needs to be for you to start being worried. I have so far in 18 months of using ZFS not once had any sort of corruption (of course ZFS can't tell if the corruption was due to bit rot or something else, but still).

 

A3. No, because in reality it is not an issue, and even if it was, there is no program that could fix it.

It's what ZFS was made for (among other things). Not that ZFS makes you immune to data corruption, but this is actually one of the express purposes for which it was designed.

[EmeraldFlame]

I have not been able to find any solid empirical data on this, but I have never come across any credible information indicating that HDDs are not affected by this issue and many (enterprise) storage people saying it is, especially when doing research on ZFS. How probable it is (especially compared with other storage media such as flash, tapes, optical disks etc.) is another question though.

 

I went to college for computer science, and while I was at school I got to meet a bunch of other computer science majors. While I have gone into programming, I have a close friend who went into data recovery and works for a company that handles a bunch of the governments data recovery efforts. I had actually consulted him on this one. A healthy hard drive will never experience 'bit rot', it will never just randomly change information on the hard drive without some sort of input. However there are so many things that can cause data to be corrupted that it can often seem like it happened for no reason, but there always is one behind it whether it be power fluctuations, vibration, magnetic fields, mechanical failure, power failures, or simply errors in the data written.

[alpenwasser]

I went to college for computer science, and while I was at school I got to meet a bunch of other computer science majors. While I have gone into programming, I have a close friend who went into data recovery and works for a company that handles a bunch of the governments data recovery efforts. I had actually consulted him on this one. A healthy hard drive will never experience 'bit rot', it will never just randomly change information on the hard drive without some sort of input. However there are so many things that can cause data to be corrupted that it can often seem like it happened for no reason, but there always is one behind it whether it be power fluctuations, vibration, magnetic fields, mechanical failure, power failures, or simply errors in the data written.

Ah, nice. In that case the point seems to be that we didn't classify what we

actually mean by "bit rot". From what you're saying I gather you understand it

to be "spontaneous corruption of data without external influence"?

I interpreted your earlier statement to be e bit be using a bit more loose

definition going more towards "data corruption on HDDs doesn't happen", which

was a tad cotrary to what I'd researched.

But yeah, classifying bit rot as "spointaneous [...]" and listing all the other

possibilities for data corruption separately I can agree with. My point was

mainly that data corruption on HDDs is indeed a thing, and measures can be taken

to mititage its effects.