Non ECC with ZFS

[79wjd]

28 minutes ago, AshleyAshes said:

But you're silly parroting the false dooms day scenario that ZFS without ECC will suddenly destroy everything during a scrub when, actual ZFS devs dismiss it.  There's no risk in not using ECC than there is on any other file system, again, as stated by a ZFS dev.  But this only -ever- comes up during discussions of ZFS.  No one is warning everyone in the 'Storage Devices' section that NTFS without ECC memory could result in every bit of data lost!  

 

(Let's not pretend that people do not keep important data on their boring consumer PCs or laptops that don't have ECC.  Heck, their MOST important data is usually there.)

 

It's still about the 'Evil Scrub' that ZFS devs have dismissed.  There their reasons to use ECC?  Sure.  ECC exists to prevent uncommon errors and if you can add it to a server, sure why not.  But if the home storage solutions running ZFS doesn't have ZFS, there's negligible additional risk.  Honestly, concurrent hard drive failures or the PSU blowing up and taking out half the components hooked up to it is far, far, far, faaaaaaaaaaaaaaaar more likely.

 

This is just needless fear mongering.

It's still a non-super likely scenario. You have to have memory that fails in the right way and that doesn't cause the system to crash. And it's not likely to trash your entire pool, but it has the potential to corrupt data. I should have worded my previous post better, not all checksums will fail, only some checksums will fail (those that are dealing with the bad memory).

 

So because restaurants will give me food that has fallen on the floor, I should do the same? What kind of fucked up logic is that? Also, there is a different expectation that one has with their personal laptop/desktop and their backup server/NAS. 

 

Also a very interesting read: http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf

[AshleyAshes]

20 minutes ago, djdwosk97 said:

It's still a non-super likely scenario. You have to have memory that fails in the right way and that doesn't cause the system to crash. And it's not likely to trash your entire pool, but it has the potential to corrupt data. I should have worded my previous post better, not all checksums will fail, only some checksums will fail (those that are dealing with the bad memory).

 

So because restaurants will give me food that has fallen on the floor, I should do the same? What kind of fucked up logic is that? Also, there is a different expectation that one has with their personal laptop/desktop and their backup server/NAS.

You're still just fear mongering and focusing on an extremely unlikely scenario and speaking about it as if it is a serious risk to consider.

 

Frankly, it's unreasonable to talk about this at such lengths while stressing 'Basically, everything else that could happen to your data, from hardware failures to user errors, is far more likely than anything that I'm talking about'.  This is like only talking about the risk of being struck by a meteor when discussing personal safety and playing it up like it's a serious risk that must be prepared for.

 

The potential issue is not 'non-super likely' it's 'You're literally more likely to be struck by lightning'.  Even your wording of 'Non-super likely' is an effort to coax 'But it could maybe possibly happen!'.  You're specifically choosing wording to avoid admitting how unlikely it is.

[79wjd]

1 minute ago, AshleyAshes said:

You're still just fear mongering and focusing on an extremely unlikely scenario and speaking about it as if it is a serious risk to consider.

 

Frankly, it's unreasonable to talk about this at such lengths while stressing 'Basically, everything else that could happen to your data, from hardware failures to user errors, is far more likely than anything that I'm talking about'.  This is like only talking about the risk of being struck by a meteor when discussing personal safety and playing it up like it's a serious risk that must be prepared for.

 

The potential issue is not 'non-super likely' it's 'You're literally more likely to be struck by lightning'.  Even your wording of 'Non-super likely' is an effort to coax 'But it could maybe possibly happen!'.  You're specifically choosing wording to avoid admitting how unlikely it is.

It's an easily preventable risk that is often relatively cheap to implement. If I'm building a 12TB RaidZ2 server, then I'm going to be spending $750 on drives. Then another $250 for a Pentium based non-ECC build with a cheap H110 board and non-Intel NIC or $350 for an i5 build. If you use ECC, then $350 for the Pentium build and $470 for a 1220v5. You're talking about an extra $100 when you're spending $1000-$1100 -- and in the process you're getting a server grade motherboard, more RAM slots, dual Intel NICs, etc... Now, lets take this build into account. He's looking at a $600 non-ecc build or a $700 ecc build. Still not a massive difference in price, and there are still plenty of other benefits to going the ECC route. 

 

And you're right, there are other risks that are more likely to occur....but again, the cost to implement any defense would be much higher. 

[adaq]

Okey, so i have been changing my mind a little bit. Im thinking of those components:

 

Atom Rangley C2358 2-Core 7W, 4*Gbit mini-ITX motherboard

DDR3 soDIMM ECC Kingston

Cooler Master Elite 110

WD Red SOHO NAS (x3)

I will test with the PSU i have and replace it if it is crap with this one: http://www.ebay.com/itm/picoPSU-150-XT-DC-DC-150-Watt-/400396065779?hash=item5d39771bf3:m:mj4G4THDPBTsh7J4L4AbMdg

 

What do you guys think? Is it enough power in that CPU and will everything work together?

[techmattr]

3 hours ago, adaq said:

Okey, so i have been changing my mind a little bit. Im thinking of those components:

 

Atom Rangley C2358 2-Core 7W, 4*Gbit mini-ITX motherboard

DDR3 soDIMM ECC Kingston

Cooler Master Elite 110

WD Red SOHO NAS (x3)

I will test with the PSU i have and replace it if it is crap with this one: http://www.ebay.com/itm/picoPSU-150-XT-DC-DC-150-Watt-/400396065779?hash=item5d39771bf3:m:mj4G4THDPBTsh7J4L4AbMdg

 

What do you guys think? Is it enough power in that CPU and will everything work together?

 

FreeNAS recommends not using Kingston RAM.

 

On 1/26/2017 at 5:15 PM, knightslugger said:

I run FreeNAS without ECC memory as well, and so far so good. no errors or corrections. All my scrubs are good.

How do you know? That's the point of ECC memory. If some of your data was corrupted without ECC you wouldn't know until you go to open that file. So saying there have been no errors isn't correct unless you've checked every file manually.

[adaq]

14 minutes ago, techmattr said:

 

FreeNAS recommends not using Kingston RAM.

 

For what reason? Whats different with Kingston compared with other brands?

[knightslugger]

22 minutes ago, techmattr said:

 

FreeNAS recommends not using Kingston RAM.

 

How do you know? That's the point of ECC memory. If some of your data was corrupted without ECC you wouldn't know until you go to open that file. So saying there have been no errors isn't correct unless you've checked every file manually.

Does a scrub not compare the data found on the mirror drive?

[techmattr]

21 minutes ago, adaq said:

For what reason? Whats different with Kingston compared with other brands?

Just due to the numbers of issues reported with Kingston RAM by users.

[techmattr]

14 minutes ago, knightslugger said:

Does a scrub not compare the data found on the mirror drive?

If there is an error while writing the data then you have corrupt data that the system thinks is not corrupt. All the checksums in the world won't detect that. 

[dalekphalm]

11 hours ago, knightslugger said:

Does a scrub not compare the data found on the mirror drive?

ECC Memory prevents a "bit flip". When you go to write data, there is a (incredibly super duper very unlikely small) chance that while in System Memory, a bit will flip (change from 1 to 0, or vice versa). Because this happens in RAM, before being committed to the disk, the server will then write the corrupted data (the file with a flipped bit) to the drive, and then will create parity (or write the duplicate, depending on type of ZFS pool created) based on the already-corrupted file. Basically, the file gets corrupted before it is ever written to the drive.

 

But let's be clear. As @AshleyAshes mentioned, this is incredibly unlikely. I personally have ECC RAM in my FreeNAS/ZFS server, but that's because I got the server for free as a decommissioned unit at work, and it came with ECC RAM.

 

If you are on a tight budget, I would not make ECC RAM a "must have". If it's the difference between choosing ECC vs choosing something else better (better CPU, more RAM, more HDD's, bigger HDD's, whatever), then the ECC should be the lower priority.

 

I would only buy ECC RAM if you've already accounted for EVERY OTHER feature you want to have, performance level you want to have, and you've got an extra $100 lying around that you can use in the build.

[techmattr]

10 hours ago, dalekphalm said:

ECC Memory prevents a "bit flip". When you go to write data, there is a (incredibly super duper very unlikely small) chance that while in System Memory, a bit will flip (change from 1 to 0, or vice versa). Because this happens in RAM, before being committed to the disk, the server will then write the corrupted data (the file with a flipped bit) to the drive, and then will create parity (or write the duplicate, depending on type of ZFS pool created) based on the already-corrupted file. Basically, the file gets corrupted before it is ever written to the drive.

 

But let's be clear. As @AshleyAshes mentioned, this is incredibly unlikely. I personally have ECC RAM in my FreeNAS/ZFS server, but that's because I got the server for free as a decommissioned unit at work, and it came with ECC RAM.

 

If you are on a tight budget, I would not make ECC RAM a "must have". If it's the difference between choosing ECC vs choosing something else better (better CPU, more RAM, more HDD's, bigger HDD's, whatever), then the ECC should be the lower priority.

 

I would only buy ECC RAM if you've already accounted for EVERY OTHER feature you want to have, performance level you want to have, and you've got an extra $100 lying around that you can use in the build.

It's not rare. It's actually incredibly common. Just that most people wouldn't know. Especially if you have RAM with a bad IC, damaged in shipping or installation.... or a host of other different issues. If it boots to the OS and is mainly stable then you'd never know it was producing erroneous writes until you go to open the file it corrupted. ECC RAM isn't that much more than non-ECC RAM and in many cases it's the same price. There is no point in not having it. If your data is important then it is absolutely a must have. 

Obviously it isn't important for machine that's just used for gaming or something like that but we're in the "Servers and NAS" forum.

[79wjd]

4 minutes ago, techmattr said:

ECC RAM isn't that much more than non-ECC RAM and in many cases it's the same price.

ECC itself isn't much more expensive, but by the time you get a server-grade/ecc-compatible motherboard and ECC ram, it's about an extra $100-$150 (but you also get more than just ECC support).

[techmattr]

7 minutes ago, djdwosk97 said:

ECC itself isn't much more expensive, but by the time you get a server-grade/ecc-compatible motherboard and ECC ram, it's about an extra $100-$150 (but you also get more than just ECC support).

If comparing to a current gen non-ECC value build then yeah it'll get more expensive. But any mid-range to high end desktop build is already talking $135-$180 motherboards which is the same territory as single socket server boards.

 

I just rebuilt my primary FreeNAS with a Core i3 6100T and a Supermicro X11SSM-F which cost $350 for both and then another $200 for 32GB of Micron DDR4-2400 ECC. So $550 for the primary build components is about what I'd expect to spend on mid-range gaming machine.

[79wjd]

1 minute ago, techmattr said:

If comparing to a current gen non-ECC value build then yeah it'll get more expensive. But any mid-range to high end desktop build is already talking $135-$180 motherboards which is the same territory as single socket server boards.

 

I just rebuilt my primary FreeNAS with a Core i3 6100T and a Supermicro X11SSM-F which cost $350 for both and then another $200 for 32GB of Micron DDR4-2400 ECC. So $550 for the primary build components is about what I'd expect to spend on mid-range gaming machine.

Except you shouldn't be comparing the cost to a mid end gaming system, but rather more of a budget office system -- i.e. a $50~ H110 board vs. a $150~ C232/C236 board. 

[techmattr]

2 minutes ago, djdwosk97 said:

Except you shouldn't be comparing the cost to a mid end gaming system, but rather more of a budget office system -- i.e. a $50~ H110 board vs. a $150~ C232/C236 board. 

Why? Budget office boards are meant to basically be dumb terminals with no functionality. In order to use it in some server capacity you'll need more PCIe add-ons increasing cost.

[79wjd]

2 minutes ago, techmattr said:

Why? Budget office boards are meant to basically be dumb terminals with no functionality. In order to use it in some server capacity you'll need more PCIe add-ons increasing cost.

For 98% of home NAS's an H110 board is MORE than enough. 

 

And no, office boards are not meant to be dumb terminals. Not in any way. 

[techmattr]

5 minutes ago, djdwosk97 said:

For 98% of home NAS' an H110 board is MORE than enough. 

Yeah I see this suggested all the time. Then those people come back wondering why their NAS OS doesn't work or has terrible performance. Why can't I get more than 30MB/s on my new NAS!?!?! Any of the boards in this range have 4 or less SATA ports and either Realtek or Intel I219 NICs that aren't supported by any NAS OS or virtualization.

[79wjd]

1 minute ago, techmattr said:

Yeah I see this suggested all the time. Then those people come back wondering why their NAS OS doesn't work or has terrible performance. Why can't I get more than 30MB/s on my new NAS!?!?! Any of the boards in this range have 4 of less SATA ports and either Realtek or Intel I219 NICs that aren't supported by any NAS OS or virtualization.

It's a good thing there are cheap sata controller cards like the SYBA SI-PEX40064 that can add an extra four ports. Not to mention that the $150~ server motherboard will probably only have six sata ports -- which isn't a huge difference anyway. 

 

The average H110 board has a 3.0x16 and two 2.0x1 slots. Even the 2.0x1 slots are more than enough to handle an additional Intel NIC or a sata controller card. Most people also don't need more bandwidth so the lack of an Intel NIC isn't really an issue either. Most of those people who ask that are just confused as to why they don't get the speeds -- not that they actually need the speeds. 

[techmattr]

If you're going to use something like a SYBA SATA card then you shouldn't be building a NAS. Use Amazon Cloud Drive or CrashPlan. You can go back and forth all day bickering about how cheap and crappy you can build a server for but bottom line is if you care about your data enough to build a server to store it then you should be doing it correctly. Not with parts that more likely to cause you to lose data than store it. 

 

If you need absolute bottom of the barrel pricing on a NAS server then you're much better off building a Supermicro X8 1366 server. You can get CPU + Mobo + RAM builds on eBay for under $200 and you'll have compatible parts supported by NAS OS's that easily saturate GbE and will be much more reliable than cutting corners with current gen budgeting. 

[79wjd]

8 minutes ago, techmattr said:

If you're going to use something like a SYBA SATA card then you shouldn't be building a NAS. Use Amazon Cloud Drive or CrashPlan. You can go back and forth all day bickering about how cheap and crappy you can build a server for but bottom line is if you care about your data enough to build a server to store it then you should be doing it correctly. Not with parts that more likely to cause you to lose data than store it. 

 

If you need absolute bottom of the barrel pricing on a NAS server then you're much better off building a Supermicro X8 1366 server. You can get CPU + Mobo + RAM builds on eBay for under $200 and you'll have compatible parts supported by NAS OS's that easily saturate GbE and will be much more reliable than cutting corners with current gen budgeting. 

There's nothing wrong with using a sata controller card to get extra SATA ports and nothing about an H110 board or sata controller is going to cause you to lost data. I ran my NAS on an H81 board and that sata controller for almost a year and never once had any issues whatsoever, and there are plenty of people who have had similar setups with no issues either. 

[techmattr]

1 minute ago, djdwosk97 said:

There's nothing wrong with using a sata controller card to get extra SATA ports and nothing about an H110 board or sata controller is going to cause you to lost data. 

Using one of those SATA cards is almost a sure fire way to lose data and using a motherboard with it's built in unsupported NIC is definitely putting your data at risk. Plus it's using inferior parts with non-ECC RAM. It's a recipe for failure. There is absolutely 0 reason to use such a build when proper solutions are available at the same price point.

[AshleyAshes]

9 minutes ago, techmattr said:

If you're going to use something like a SYBA SATA card then you shouldn't be building a NAS. Use Amazon Cloud Drive or CrashPlan. You can go back and forth all day bickering about how cheap and crappy you can build a server for but bottom line is if you care about your data enough to build a server to store it then you should be doing it correctly. Not with parts that more likely to cause you to lose data than store it. 

You just declared that a Syba SATA controller if you 'care about your data's while making no effort to substantiate what possible threats to SATA integrity there could be.  If you can't back up claims, why make claims at all?

[AshleyAshes]

2 minutes ago, techmattr said:

Using one of those SATA cards is almost a sure fire way to lose data

Explain how it is 'sure fire'.  If the controller is known to corrupt data, you should have no issue finding reports and documentation to support your claim.

[79wjd]

9 minutes ago, techmattr said:

Using one of those SATA cards is almost a sure fire way to lose data and using a motherboard with it's built in unsupported NIC is definitely putting your data at risk. Plus it's using inferior parts with non-ECC RAM. It's a recipe for failure. There is absolutely 0 reason to use such a build when proper solutions are available at the same price point.

Wait wait wait wait wait.....explain to me how an unsupported NIC is "definitely putting your data at risk". 

 

And, no, they aren't the same price. There is about a $100-$150 difference to step up to ECC/server-grade hardware. You gain other features as well (IPMI, Intel NIC, potentially more memory slots, more sata ports, lower power draw, better quality, etc...), but it's definitely not the same price. 

[techmattr]

2 minutes ago, AshleyAshes said:

Explain how it is 'sure fire'.  If the controller is known to corrupt data, you should have no issue finding reports and documentation to support your claim.

Read their own FAQ. Their FAQ is littered with disconnect issues that would 100% cause data corruption. http://www.sybausa.com/faq.php?type=1 

 

Visit any forum dedicated to storage and read the hundreds of posts from people who lose data because they used SATA multiplication. Port multipliers with software RAID (which almost every NAS OS uses) are going to fail. It's not a case of if it's a case of when. Those cards are single channel cards that use port multiplication. Any SATA multiplier is going to disconnect under heavy load. When you scrub your data you're going to under as heavy a load as possible.