ZFS (FreeNAS) Speeds and Caching

[Karasuni]

Hey all,

 

I've spent a large portion over the past couple of weeks reading blogs, watching videos and listening to podcasts on ZFS (and some UnRAID) and all the small details of each configuration, down to a-shifts and sector size optimizations depending on the types of usage. Even with all these sources I still find it very difficult to decide on the configuration I want to go to. For example in a system with 4 x 12 TB drives I could go 2 x 2 mirrored or 1 x 4-way RAIDZ1 with 1 parity.

 

Currently I have a few questions where I constantly find conflicting information.

- How do mirror vdevs read/write performance compare to RAIDZ1/RAIDZ2 configurations?

- Would you enable compression (lz4, lzbj?) for all vdevs?

- Is there a significant impact of 7200 rpm vs 5400 rpm drives in these configurations?

- Is there a significant impact of an SSD drive for caching (SATA2 / AHCI / SATA3 / M.2 / Optane) ?

- Is it possible to cache metadata and file indexes in ZFS / FreeNAS on SSD to speed up directory index time?

 

Especially mirror vs raidz performance is a conflicting debate. According to this blogpost an experienced user and one of the ZFS developers claim mirrored vdevs have a superior performance. Other sources claim nearly double the performance in raidz1 compared to mirrored.

 

Any answers and guidance would be very helpful.

[Jisagi]

1. You can mroe or less compare zfs performance to normal performance if you comapre mirror<->raid1, raidz1<->raid5 and raidz2<->raid6. It might be a bit slower on zfs because it's a enterprise grade filesystem, but not by too much. The "problem" with each raid is the same on zfs as with any normal raid. Good writeup: https://calomel.org/zfs_raid_speed_capacity.html
2. Yes, lz4 is pretty fast. You can disable it, if you know there are (close to) no compressible files in a specific dataset, but i wouldn't disable it for the whole pool

3. Tbh, I wouldn't buy 5400rpm drives anyway. More rpm isn't always more/better performance depending on the workload but any decent hdd has 7200rpm and i'd go with that compared to 5400rpm

4. Caching really depends on the usage, are VMs used, what kind of data is written or read from a pool, how much system memory do you have, .... There is no rule of thumb for all cases (sadly). A few things to keep in mind though. There are 2 types of caches, L2ARC and SLOG. The first one is a read cache, the other one a write cache. Normally you dont need either. Adding a read cache to a system takes up normal sytem memory and a write cache has to eventually write the data to disk anyway. Both have their usecases but both can, if not used correctly, slow down a system by a lot. Something to read for this: https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/

5. Don't quote me on this one, but that is basically a read cache. Usually you use system memory as your ARC to cache files and the metadata for it is the memory as well. That'S also the reason why L2ARC needs system memory to store it'S metadata.

[Windows7ge]

1. With a RAID10 configuration you get significantly more IOPS. This is best when running a large number of virtual machines. From my own experience the raidz1/2 performance doesn't cause a huge read/write throughput penalty on ZFS while giving you more usable storage but that depends on your use case.
2. If the data you have is compressable it's usually not a huge performance penalty to use lz4 to save some space. It's worth it IMO.
3. With spinning platters a higher RPM would be directly proportional to the overall read/write performance of the pool regardless of RAID level.
4. Using an L2ARC/ZIL or SLOG device is only worthwhile for tasks such as hosting Virtual Machines or Databases. The reason being is these functions only accelerate Synchronous reads/writes. Functions such as SMB are almost if not entirely Asynchronous. This means the cache will only be read or written from/to at the speed at which the pool can be read from/written to. ZFS already uses RAM as a read cache as well (ARC) which will be much faster than even a current M.2 SSD. If this is to be a File Server/Media Server there's no point in a cache drive.
5. Can't answer this one. Don't know enough about what metadata is.

[Karasuni]

1 hour ago, Jisagi said:

1. You can mroe or less compare zfs performance to normal performance if you comapre mirror<->raid1, raidz1<->raid5 and raidz2<->raid6. It might be a bit slower on zfs because it's a enterprise grade filesystem, but not by too much. The "problem" with each raid is the same on zfs as with any normal raid. Good writeup: https://calomel.org/zfs_raid_speed_capacity.html
2. Yes, lz4 is pretty fast. You can disable it, if you know there are (close to) no compressible files in a specific dataset, but i wouldn't disable it for the whole pool

3. Tbh, I wouldn't buy 5400rpm drives anyway. More rpm isn't always more/better performance depending on the workload but any decent hdd has 7200rpm and i'd go with that compared to 5400rpm

4. Caching really depends on the usage, are VMs used, what kind of data is written or read from a pool, how much system memory do you have, .... There is no rule of thumb for all cases (sadly). A few things to keep in mind though. There are 2 types of caches, L2ARC and SLOG. The first one is a read cache, the other one a write cache. Normally you dont need either. Adding a read cache to a system takes up normal sytem memory and a write cache has to eventually write the data to disk anyway. Both have their usecases but both can, if not used correctly, slow down a system by a lot. Something to read for this: https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/

5. Don't quote me on this one, but that is basically a read cache. Usually you use system memory as your ARC to cache files and the metadata for it is the memory as well. That'S also the reason why L2ARC needs system memory to store it'S metadata.

Thanks for replying

 

1. Yeah, that's the same source I was using. I don't know how old it is or how representative it is for random I/O. I still can't tell if mirror is faster than raidz1 or not.

2. For some volumes I'll enable some encryption so that's probably when I'm disabling it. Movies and music I couldn't tell.

3. I currently own some 5400rpm drives that I'll use in the pool. I'm curious if 7200rpm would actually go up to 33% faster or other considerations

4. The blogpost sounds like adding an SSD is a cost-efficient way to expand the system RAM:

Quote

 You can then add a Level 2 Adaptive Replacement Cache (L2ARC) to extend the ARC to a dedicated disk (or disks) to dramatically improve read speeds, effectively giving the user all-flash performance.

Setting standard configuration for the ZIL it could probably reside on the same SSD and provide some extra protection.

 

[Karasuni]

For point 5 I think I was referring to an in-development Metadata Allocation Classes. I might be confused as to it's actual impact but it sounded like it would greatly improve seek time.

[Jisagi]

To expand a bit on 2 & 4:

2. Movies, series and music is already compressed and therefore not worth using any compression for. As @Windows7ge already said though, the penalty for using lz4 on a more or less recent cpu is negligible, so no harm using it on a dataset with this kind of data.

4. Adding an ssd as either l2arc or slog will never have the same result as adding more system memory. Not "not a bit" but "not at all". From what you let us know so far, i don't see any usecase for either one which would mean you might get even less performance by adding one or both to the system. Every read or write is different and a private vault for stuff like videos, photos and music which is accessed by a few people only (usually) doesn't need either one.