Power outage without a UPS

[79wjd]

Due to reasons a UPS may not be practical for me to have. So I was wondering, what kind of data integrity issues would I likely face (and what would I likely face in an extreme case) if there was a power outage (when the system is in the middle of a scrub -- since I assume that's the most likely time for damage to occur?)? 

 

I'm running FreeNAS on a Xeon E3 with 32gb ECC memory. 

 

My server is mainly used for media storage (so nothing super critical) and streaming via Plex. I run (or will run) scrubs and SMART tests weekly (that sums up the majority of the non-streaming usage AFAIK). 

[Cvdasfg]

 

Due to reasons a UPS may not be practical for me to have. So I was wondering, what kind of data integrity issues would I likely face (and what would I likely face in an extreme case) if there was a power outage (when the system is in the middle of a scrub -- since I assume that's the most likely time for damage to occur?)? 

 

I'm running FreeNAS on a Xeon E3 with 32gb ECC memory. 

 

My server is mainly used for media storage (so nothing super critical) and streaming via Plex. I run (or will run) scrubs and SMART tests weekly (that sums up the majority of the non-streaming usage AFAIK). 

 

If your running freeNAS you really should have a ups.

 

You will lose everything if a power outage catches you during a scrub ( assuming you have what 99% of freenas has) 

 

If its not that important then your probably fine.. if its mission critical/only copies I'd just suck it up and buy a ups. They have some that will run that server for an hour for about $200

[Altecice]

Do you have any kind of dedicated RAID controller with some battery backed cache? 

[Oshino Shinobu]

I suppose part of it will be luck. In the worst case scenario, you get a power surge (from a storm or something) that kills everything. If we're just talking powercuts, then you may get some lost data, or data corruption with files that are being used at the time. If you run nightly backups (at least for important data), then you shouldn't ever lose much data as you can just restore from the night before. 

[bobhays]

If your running freeNAS you really should have a ups.

 

You will lose everything if a power outage catches you during a scrub ( assuming you have what 99% of freenas has) 

 

If its not that important then your probably fine.. if its mission critical/only copies I'd just suck it up and buy a ups. They have some that will run that server for an hour for about $200

What happens if you're not home or sleeping while the power goes out. The ups will stay up for an hour but after that is it the same result?

[W-L]

 

Due to reasons a UPS may not be practical for me to have. So I was wondering, what kind of data integrity issues would I likely face (and what would I likely face in an extreme case) if there was a power outage (when the system is in the middle of a scrub -- since I assume that's the most likely time for damage to occur?)? 

I'm running FreeNAS on a Xeon E3 with 32gb ECC memory. 

My server is mainly used for media storage (so nothing super critical) and streaming via Plex. I run (or will run) scrubs and SMART tests weekly (that sums up the majority of the non-streaming usage AFAIK). 

 

 

If you have a raid card make sure it's one with a battery backup where it will remember the cache. There isn't really another option compared to a UPS.

[79wjd]

If your running freeNAS you really should have a ups.

 

You will lose everything if a power outage catches you during a scrub ( assuming you have what 99% of freenas has) 

 

If its not that important then your probably fine.. if its mission critical/only copies I'd just suck it up and buy a ups. They have some that will run that server for an hour for about $200

What if I don't do any scrubs? Bitrot isn't all that critical considering it's just media (where I likely wouldn't notice bitrot anyway). It's not so much a money issue -- I can buy a UPS capable of powering down the system for $65. 

 

Do you have any kind of dedicated RAID controller with some battery backed cache? 

nope.

 

I suppose part of it will be luck. In the worst case scenario, you get a power surge (from a storm or something) that kills everything. If we're just talking powercuts, then you may get some lost data, or data corruption with files that are being used at the time. If you run nightly backups (at least for important data), then you shouldn't ever lose much data as you can just restore from the night before. 

I'm ignoring surges. 

 

 

What happens if you're not home or sleeping while the power goes out. The ups will stay up for an hour but after that is it the same result?

A UPS can interface with the system and signal the system to shutdown.

@bobhays

[Cvdasfg]

What happens if you're not home or sleeping while the power goes out. The ups will stay up for an hour but after that is it the same result?

nope. If you set your freenas server up correctly it will detect when only the ups is running and will turn itself off at a safe time Causing no damage

[Cvdasfg]

 

What if I don't do any scrubs? Bitrot isn't all that critical considering it's just media (where I likely wouldn't notice bitrot anyway). It's not so much a money issue -- I can buy a UPS capable of powering down the system for $65. 

 

nope.

 

I'm ignoring surges. 

 

 

A UPS can interface with the system and signal the system to shutdown.

@bobhays

 

If its not a money issue then why wouldnt you?

[79wjd]

If its not a money issue then why wouldnt you?

Space constraints where the server has to be located and the potential (albeit abysmally small) fire hazard. 

[W-L]

Space constraints where the server has to be located and the potential (albeit abysmally small) fire hazard. 

 

Can you not fit a 1U UPS or say something like this, they come in different wattages to meet certain needs:

 

http://www.apc.com/shop/us/en/products/APC-Smart-UPS-750VA-USB-Serial-RM-1U-120V/P-SUA750RM1U

[79wjd]

Can you not fit a 1U UPS or say something like this, they come in different wattages to meet certain needs:

 

http://www.apc.com/shop/us/en/products/APC-Smart-UPS-750VA-USB-Serial-RM-1U-120V/P-SUA750RM1U

I have my server on top of a wooden cabinet with a footprint more or less just big enough for the server itself. So there isn't really room, and a UPS on top of a 9' tall wooden cabinet scares me a bit as well (due to the additional weight and fire hazard -- the cabinet isn't the sturdiest thing in the world).

[W-L]

I have my server on top of a wooden cabinet with a footprint more or less just big enough for the server itself. So there isn't really room, and a UPS on top of a 9' tall wooden cabinet scares me a bit as well (due to the additional weight and fire hazard -- the cabinet isn't the sturdiest thing in the world).

 

Well a UPS is fairly seal and contained as long as it has airflow it shouldn't be an issue, you can also wall mount it if need be to take the weight off of it and get space. Just note those 1U can get pretty loud.

[79wjd]

Well a UPS is fairly seal and contained as long as it has airflow it shouldn't be an issue, you can also wall mount it if need be to take the weight off of it and get space. Just note those 1U can get pretty loud.

I was looking at an APC BE750G (plugstrip style). 

 

Wall mounting isn't really an option either. As for airflow, it should get decent airflow where it is as there are air vents in the room, but the risk of a fire hazard is still a bit concerning (regardless of the abysmally small risk).

[W-L]

I was looking at an APC BE750G (plugstrip style). 

Wall mounting isn't really an option either. As for airflow, it should get decent airflow where it is as there are air vents in the room, but the risk of a fire hazard is still a bit concerning (regardless of the abysmally small risk).

 

If you can squeeze it in yeah that's a good option while not being overly expensive.

[bobhays]

 

 

 

 

A UPS can interface with the system and signal the system to shutdown.

@bobhays

 

 

 

nope. If you set your freenas server up correctly it will detect when only the ups is running and will turn itself off at a safe time Causing no damage

 

That's pretty cool and extremely useful. Now UPS seems like a much better option then I previously thought.

[alpenwasser]

You will lose everything if a power outage catches you during a scrub ( assuming you have what 99% of freenas has)

Quick question: I have googled around for various combinations of "ZFS", "FreeNAS", "Power loss", "Power outage" etc. and can't find much on this, but I'd be curious if you have any links with more info? What makes this sounds a bit weird is that a scrub is mostly read operations as far as I'm aware. Unless there is actually corruption detected, there doesn't really seem to be any reason to write to the pool, so I'm not sure how that would corrupt things unless you have a power outage right the moment when the pool is having corrected data written to it. But as said, can't really find much info, so I'd be curious to know more on this.

The only thread I've found so far is this one, but it's not specifically about scrubbing:

https://forums.freenas.org/index.php?threads/dealing-with-power-outages.11362

It mentions the whole copy-on-write thing and how the file system should still be consistent after a power loss (although if you haven't had all blocks from a file written yet, it will obviously be incomplete), which is what I'd expect during normal operation.

Also found this, but not related to scrubbing either, and seems to be patched by now:

https://github.com/zfsonlinux/zfs/issues/1390

(Still an interesting read though)

Then there's this guy, also not related to scrubbing:

http://blog.darrenscott.com/tag/freenas/

Seems his OS drive got corrupted, but the zpool was fine.

Anyway, as said, I'd be very curious to get more info on this.

Regardless of that though, I'd agree that a UPS is a good idea, even though I'm at the moment not running one for financial reasons. If I had the money laying around, I'd definitely get one ASAP.

I've never had a power outage, only two hard resets of my server, which isn't quite the same thing. No ill effects though. Reboot, unlock encrypted HDDs, import ZFS pool. Don't think I was scrubbing at the time though.

@djdwosk97 : If you run ZFS without scrubbing you're basically just back at a regular file system with some fancy pool management. Up to you. Note however that in that URE corruption thread there were fatal errors which I could have prevented by doing diligent scrubbing. In the end, the decision is yours though. I run some single HDDs with ZFS without redundancy or scrubbing so that I can do ZFS send/receive on them from my main pool, and use ZFS's convenient interface, because I quite like that. But obviously there's not much protection of my data (well, corruption will probably be detected because it's still checksumming, but I won't be able to fix it because I have no redundancy; but I can just restore from my primary in those cases).

As a side note, these two links might be of interest:

http://docs.oracle.com/cd/E23823_01/html/819-5461/gbbwa.html

http://docs.oracle.com/cd/E23823_01/html/819-5461/gbbwl.html

[79wjd]

@djdwosk97 : If you run ZFS without scrubbing you're basically just back at a regular file system with some fancy pool management. Up to you. Note however that in that URE corruption thread there were fatal errors which I could have prevented by doing diligent scrubbing. In the end, the decision is yours though. I run some single HDDs with ZFS without redundancy or scrubbing so that I can do ZFS send/receive on them from my main pool, and use ZFS's convenient interface, because I quite like that. But obviously there's not much protection of my data (well, corruption will probably be detected because it's still checksumming, but I won't be able to fix it because I have no redundancy; but I can just restore from my primary in those cases).

As a side note, these two links might be of interest:

http://docs.oracle.com/cd/E23823_01/html/819-5461/gbbwa.html

http://docs.oracle.com/cd/E23823_01/html/819-5461/gbbwl.html

But since I have no redundancy, would scrubbing even accomplish anything? 

[alpenwasser]

But since I have no redundancy, would scrubbing even accomplish anything?

Oh, you're planning on running it all striped or as standalone drives, no RAIDZ<n> or mirrors?

[79wjd]

Oh, you're planning on running it all striped or as standalone drives, no RAIDZ<n> or mirrors?

Well, I need 12tb of storage (and that's going to probably grow to 16tb in the next couple of months) and I have 12tb of storage. So RaidZ2 isn't really affordable, and I don't think RaidZ1 is worthwhile with 4tb drives. 

 

I'd love to get five 6tb WD Reds and put them in RaidZ2, but I don't see myself willing/able to drop $1400 on drives. 

[alpenwasser]

Well, I need 12tb of storage (and that's going to probably grow to 16tb in the next couple of months) and I have 12tb of storage. So RaidZ2 isn't really affordable, and I don't think RaidZ1 is worthwhile with 4tb drives. 

 

I'd love to get five 6tb WD Reds and put them in RaidZ2, but I don't see myself willing/able to drop $1400 on drives.

Aha, so you're going to just run 3 4 TB drives as JBODs? Yes, in that case a scrub wouldn't do you that much good. It would detect corruption if something had gone wrong if you still have checksumming enabled, but it wouldn't be able to fix anything unless you had copies>1, in which case you'd also lose storage space and would again be back where you started on that front. But that corruption, if/when it occurs, would also be detected during regular reading, since ZFS always checksums each block as it reads it (and tries to fix it if it detects corruption, which it of course couldn't do in your case).

[79wjd]

Aha, so you're going to just run 3 4 TB drives as JBODs? Yes, in that case a scrub wouldn't do you that much good. It would detect corruption if something had gone wrong if you still have checksumming enabled, but it wouldn't be able to fix anything unless you had copies>1, in which case you'd also lose storage space and would again be back where you started on that front. But that corruption, if/when it occurs, would also be detected during regular reading, since ZFS always checksums each block as it reads it (and tries to fix it if it detects corruption, which it of course couldn't do in your case).

Maybe I should just start a youtube channel and request WD to send me a bunch of drives for reasons  

 

@Captain_WD Sounds like a good plan, no? 

[alpenwasser]

Maybe I should just start a youtube channel and request WD to send me a bunch of drives for reasons  

 

@Captain_WD Sounds like a good plan, no?

I'll take some of that.

Speaking from my own experience: I needed to replace two drives in one of my pools last summer, and used that opportunity to re-arrange my pool layout. So I moved everything I could off to some spare drives I had laying around. But it wasn't enough, so I ended up deleting a bunch of my media files. It was a nice opportunity to clean house a bit. I did make a list of things to re-download once I was able though. But that might not be suitable for everyone.

[79wjd]

I'll take some of that.

Speaking from my own experience: I needed to replace two drives in one of my pools last summer, and used that opportunity to re-arrange my pool layout. So I moved everything I could off to some spare drives I had laying around. But it wasn't enough, so I ended up deleting a bunch of my media files. It was a nice opportunity to clean house a bit. I did make a list of things to re-download once I was able though. But that might not be suitable for everyone.

Well, two of my three 4tb drives are going to be filled with bluray rips since I don't have the time/patience to encode all of them. So a 4tb drive only holds like 100~ movies. And the third 4tb drive is already full of tv shows. 

 

Regardless, I'm going to be moving 1tb of dvd rips off my 4tb drive and onto the 1tb blue that's in my desktop now (that I'm going to put in my server) -- so I need to delete like 250gb worth of crap there. Plus one of my 4tb drives is at 90% capacity with TV shows, so I either have to find a decent way to separate them into two drives or delete about 300GB worth of shows.

[Captain_WD]

~snip~

 

As much as I'd love to help with this, I'm not part of that department. You can contact our support and they can tell you how to contact the appropriate people for this and see if the guys there can do anything about it. Here's a link to our live support: http://products.wdc.com/support/kb.ashx?id=Hw9X70

 

Cheers!

 

Captain_WD.