New to FreeNAS: Help Avoiding Mistakes

[Mr Technician]

I am building a freenas server in the near future (hopefully) on some old server grade hardware I have and am still new to ZFS. I've done research but there are a number of different perspectives and "best practices" out there. First, what are some general things to avoid or to do during setup? I'd like to avoid the stupid mistakes. Second, what raid level and drives would make the most sense for a 6-8 tb usable array? 3tb and 4tb wd red and seagate ironwolf seem to have the best price/gb. Third, I'm planning to use some older drives (possibly in hotswap) to backup the most critical data (irreplaceable documents and family pictures), and how is this best done? I've heard the backup process can be run from the terminal which I am comfortable with. If it is relevant I have an Intel Xeon W3530 @2.80Ghz and 12gb ddr3 ECC RAM.

[Cyanara]

The best RAID level comes down to what you want out of a RAID (uptime/capacity/sequential speed) and how many drives you have.

However, I have to ask, in this case is it just about having fun learning something new? Because from a technical stand point the best setup would be to buy a 6-8TB hard drive and back up to a separate drive in another enclosure or computer. Every RAID configuration adds complications and risk that should really be weighed up against the benefits, and it's hard to see the benefits of a RAID system that's half the capacity a single hard drive can do today.

This article is a bit old, but the principle is still there: https://www.pugetsystems.com/labs/articles/Why-RAID-is-usually-a-Terrible-Idea-29/

 

Still, if fixing a degraded array is something that you'd be interested to do, and you have separate physical backups that can be made available immediately to your family in the meantime, then you may as well learn how to do it.

[jde3]

Be cautious with the plugins you use. A NAS system (and storage in general) is supposed to be boring. Its supposed to store your files, not be a VPS, Media Server, VPN etc. This is one of the reasons I like NAS4Free as opposed to FreeNAS for plain old simple NAS systems but FreeNAS has it's place for sure. If you do want to do the above said bells and whistles, what you want instead of FreeNAS might be FreeBSD itself. You might take a look at CBSD if you want a web frontend.

 

About ZFS:

ZFS can't extend a raid 5 (what it calls a raidz1, it isn't really raid 5) at least yet.. the feature, called reflow, is coming maybe in FreeBSD 12 tho don't quote me.. A lot of people prefer to use ZFS on mirrors due to this (or raid5+0 config) ... you can use Raidz1 etc no problem and I do myself but understand it has this limitation so growing a pool will need either replacing all the existing disks with larger disks and resilvering one at a time or adding the exact same number and converting it to a raid50. - So all this is just an effect if you want to make your existing array larger in the future.

 

Some people will say ZFS needs ECC ram, in an enterprise environment everything needs ECC. But at home ZFS does not need it any more than any other filesystem. (this is just a byproduct of Sun's old marketing for ZFS and its intended use in enterprise but you'll see this suggestion out there on the web sometimes, if you want the best protection then you need ECC but.. every file system is that way. ZFS isn't special here.)

 

ZFS's command line tools are great. Learn to use them. They were written with Unix and Sun philosophy and thus are very concise and clear with minimal feedback. The plus side of this is they are efficient quick and scriptable the downside is they don't hold your hand. You'll get no confirmation from doing crazy stuff so make sure you have it right. (it does have a pretend flag showing you what it would do -n I believe off the top of my head)

 

ZFS Snapshots are a little special, they take nearly no resources to create and sending them to other systems is superior to other implementations as well as ZFS can only send the blocks that changed without having to first calculate what changed. A nice feature.. you can snapshot a lot too even up to the minute if you want. You can do auto snapshot retention from cron. ZFS Tools is one such way to do automatic rotating incremental snapshots. https://github.com/bdrewery/zfstools in the example on that page it snapshots every 15 minutes and keeps them for a year rotating them out.

 

Spoiler

 

Kinda a cool feature for the advanced user. ZFS on BSD also can use boot environments so it can roll back updates or boot entire different OS's installed to the same pool. You really don't even have to re-partition if you want to install a different OS now just stick the new OS into a different dataset and change the boot environment.

 

An example of that where all the OS's share the same /home might look like this:

pool: tank

tank/root/freebsd

tank/root/trueos

tank/root/gentoo

tank/root/opensuse

tank/home

 

(Linux might be a little picky with its initramdisk but I've seen examples of this out there)

 

 

ZVOLS: ZFS lets you create block devices called ZVOLS that can be use to mount VM images to, Swap devices or exported to iSCSI.

Yes, you can put NTFS and Windows booting on a ZVOL from either iSCSI or a VM and have all the features of ZFS on Windows with it still running on its (rather pathetic) native filesystem.

 

Databases on ZFS require special tuning for optimal performance due to the way databases store information. You don't have to do this but it is the optimal way. Some of the reasons for that is the write block size and to make sure ZFS dosen't cache what the database has already cached. Look for a guide out there to help you with this for your particular database. If you intend to run mysql or postgres in production that is something to keep in mind.

 

Dedup, avoid it. It doesn't do what most people think it does and it uses an extreme amount of ram. Some work is being put into making this not suck but... currently it kinda does.

 

Spoiler

reference

 

 

LZ4 compression is great pretty much always because it does not compress uncompressable data such as media. If you know for sure that data is never going to be compressed then you can shut LZ4 off but for the most part leave it on. FreeBSD also compresses it's arc cache in ram now. (I do no believe this feature is available on Linux yet, however Linux has the ability to encrypt entire pools and datasets.. sometimes it takes a little time to get one new development feature into other OSs. BSD also has zstd for better compression and the expense of speed if you want it.)

 

Edit: Oh yea the last thing. ZFS is forked between "OpenZFS" (what is found in Linux, BSD's, Illumos and Mac OS) and "Oracle ZFS" (what is found in Solaris post Solaris 11) - Oracle has lots of guides on ZFS but they may not always apply to OpenZFS (most do). FreeBSD's manual page is a very good resource that applies to OpenZFS. https://www.freebsd.org/doc/handbook/zfs.html

 

 

Spoiler

A entertaining and informative talk about that fork is here, if you want some background as to why.

 

 

[Mr Technician]

4 hours ago, Cyanara said:

The best RAID level comes down to what you want out of a RAID (uptime/capacity/sequential speed) and how many drives you have.

However, I have to ask, in this case is it just about having fun learning something new? Because from a technical stand point the best setup would be to buy a 6-8TB hard drive and back up to a separate drive in another enclosure or computer. Every RAID configuration adds complications and risk that should really be weighed up against the benefits, and it's hard to see the benefits of a RAID system that's half the capacity a single hard drive can do today.

This article is a bit old, but the principle is still there: https://www.pugetsystems.com/labs/articles/Why-RAID-is-usually-a-Terrible-Idea-29/

 

Still, if fixing a degraded array is something that you'd be interested to do, and you have separate physical backups that can be made available immediately to your family in the meantime, then you may as well learn how to do it.

High performance will be a benefit as I will likely be working with uncompressed image sequences and may also switch to 10 gigabit networking in the future so additional speed could be utilized. This is less to learn something new as it is to establish a proper storage server and OS backup system, so it's not just an experiment. I also like raid due to the automatic backup from hardware failure (ideally with an offsite backup too).

 

@jde3 I intend to use this mainly as a NAS so other functions would be secondary (if they are implemented at all), and I am aware of the inability of ZFS to expand an array, hence the large capacity I mentioned (at least for a home server, I know it's not much in the server world). I've heard people say ECC is unnecessary and I've also heard people say not having it will destroy your data, but I have a system with ECC anyway. Snapshots look nice, I'll have to look into implementing them. So it would be easy to clone a portion of the file system to a separate hotswap drive from the terminal?

[jde3]

16 minutes ago, ThatFlashCat said:

High performance will be a benefit as I will likely be working with uncompressed image sequences and may also switch to 10 gigabit networking in the future so additional speed could be utilized. This is less to learn something new as it is to establish a proper storage server and OS backup system, so it's not just an experiment. I also like raid due to the automatic backup from hardware failure (ideally with an offsite backup too).

 

@jde3 I intend to use this mainly as a NAS so other functions would be secondary (if they are implemented at all), and I am aware of the inability of ZFS to expand an array, hence the large capacity I mentioned (at least for a home server, I know it's not much in the server world). I've heard people say ECC is unnecessary and I've also heard people say not having it will destroy your data, but I have a system with ECC anyway. Snapshots look nice, I'll have to look into implementing them. So it would be easy to clone a portion of the file system to a separate hotswap drive from the terminal?

NAS4Free might be a good choice in that case. I had a NAS4Free system running on a opteron pedistal server for.. hell 6 years without a hitch. very boring, never died, always reliable. Turned it off not because it broke but just because it was so old. 8 250gb disks man.

 

Matt Ahrens (one of the original designers at Sun) has said there is nothing special about ZFS and ECC than any other file system, pretty much putting the topic to rest. Yes, corrupted ram in ZFS ruins your data.. yes it ruins your data on every other file system too.

 

Yeah, ZFS send and recieve can do that from one pool to another, it's often used remotely but that can be done on the same system (being both disks are zfs)

 

Not sure exactly what you want but something like maybe .. zfs send -v pool@snapshot | zfs receive otherpool/snapshot

[Mr Technician]

Why NAS4Free over FreeNAS in your opinion? Both use ZFS and presumably the data is equally safe on both, so what advantage is there in NAS4Free? @jde3

[jde3]

@ThatFlashCat I edited above, I do that. There isn't that much difference in the platforms to be fair. NAS4Free tends to follow FreeBSD more closely, FreeNAS is made by the people who make TrueOS so it sometimes takes new things from TrueOS.

 

In the above if both flie systems are not zfs then your best bet would be to use rsync but zfs send can be used to sent to a raw file for import later.

[Mr Technician]

20 hours ago, jde3 said:

@ThatFlashCat I edited above, I do that. There isn't that much difference in the platforms to be fair. NAS4Free tends to follow FreeBSD more closely, FreeNAS is made by the people who make TrueOS so it sometimes takes new things from TrueOS.

 

In the above if both flie systems are not zfs then your best bet would be to use rsync but zfs send can be used to sent to a raw file for import later.

NAS4Free does look nice, I assume that using ZFS for both means the data would be equally safe with either using the same settings? I've heard that raid Z1 is no longer worth using with large drives, so is this the case? I also noticed that NAS4Free allows for UPS monitoring, and is a UPS that important? I understand that data being written could be lost but is existing written data at risk? Yes, this post is entirely questions.

[jde3]

Ya, it's more of an industry NAS, it runs from CF flash etc. Less fun, less home user'ey, still reliable.

 

imo raidz1 is ok. It has it's use case for what it is. With ZFS data in flight is never at risk.. it uses a transactional model like a database so it will roll back anything that isn't completely written to disk and won't give you a partial write.

[Mr Technician]

6 minutes ago, jde3 said:

Ya, it's more of an industry NAS, it runs from CF flash etc. Less fun, less home user'ey, still reliable.

 

imo raidz1 is ok. It has it's use case for what it is. With ZFS data in flight is never at risk.. it uses a transactional model like a database so it will roll back anything that isn't completely written to disk and won't give you a partial write.

Of course I could always restore a completely failed array from a separate offsite backup, correct? Ok, so a power loss isn't a huge deal without a UPS which is a beneficial cost savings. The power here is generally still reliable but incidents still occur (once a fault light socket managed to short circuit and trip the main breaker). The point being, it is likely that there will be a power loss during operation from time to time.

[jde3]

The power reliability on my one home rig was about as good as you can expect in Somalia (it was in the US but the reliability was utter garbage due to the incompetent local power co and government) No problems there.. but it is harder on the hardware I believe. Like once a month, it was unbelievable.

 

Ppl get confused by this sometimes. A backup is having your data in two places at once. A NAS can be a backup but Raid is not a backup. (This is because raid dosen't protect from accidental or intentional destruction)

[Mr Technician]

Just now, jde3 said:

The power reliability on my one home rig was about as good as you can expect in Somalia (it was in the US but the reliability was utter garbage due to the incompetent local power co and government) No problems there.. but it is harder on the hardware I believe.

 

Ppl get confused by this sometimes. A backup is having your data in two places at once. A NAS can be a backup but Raid is not a backup. (This is because raid dosen't protect from accidental or intentional destruction)

By backup I refer to data stored offsite or at least separate from the main array. I realize that the protection from raid comes from its ability to rebuild even if a drive is lost. It's extra insurance but not a true backup separate from any damage caused to the main array.

[jde3]

Oh ya... so there is zfs send and receive but that won't work with encryption, if you need this for offsite... yet.. (I like duplicity if you need encrypted remote incremental)

 

I actually looked at the git source on Linux and.. it has feature - com.datto:encryption

So ZFS encryption is in the Linux development tree now.

 

that means that soon you'll be able to send an encrypted backup across the wire.

[Mr Technician]

What about creating a backup by copying directly to a physically inserted drive instead of using the network? @jde3

[jde3]

It's ok if it's not stored in the system.. ala put the drive (or tape) in the system, write the backup to it then put the drive in the safe. (or rotate 2 or 3 of them)

 

But trouble comes when it's attached and you get lazy and it's never removed.. its better than nothing but its not great.

 

I've worked storage for 20 some odd years and this has kinda become a long brain picking thread but its alright... hopefully other people read this.

[Mr Technician]

Right, I think I'll try NAS4Free and experiment a bit with it. Yeah, hopefully someone else reads this.