ZFS orientated followup questions

[For Science!]

Sorry, this post isn't finished, but I need to go now and so I'm posting it prematurely with just two questions (albeit long ones). I will post additional questions later)

 

Dear All,

 

Following an earlier post, I've been doing some reading around ZFS and have some further questions. I hope these aren't too "out there" in terms of understanding. Please assume if something doesn't make sense then it is because I may be using the incorrect term or concept, which I would also appreciate any corrections for.

 

Here it goes:

 

1. I understand that ZFS is rather RAM heavy, or at least benefits from having free RAM due to its use as ARC cache. Does this mean that a system running the ZFS filesystem is inherently not good for being a workstation itself?

- At the worst case, for example if a RAM intesive job (using, say, 100 GB of 120 GB of system memory), will the system grind to a halt, or will it be able to maintain some degree of disk i/o nonetheless

 

2. After doing all this reading, I am under the impresssion that for me, I would benefit from just installing FreeNAS/BSD to see what works and what is a waste. Does this kind of make sense?

 

I have the following hardware already from other builds for experimentation:

Ryzen 7 1700, NH-D15, 1080Ti

Crosshair VI Hero - 8 SATA ports, 1 NVMe slot, ECC-Support

non-ECC memory, 64 GB (16 x 4 GB)

2x 500 GB SATA (Samsung 850 EVO)

4x 8000 GB SATA HDD (Seagate STA8000AS0002)

2x 500 GB M.2 NVMe SSD (Samsung 970 EVO)

+ 2 other functioning computers with a spare PCIe slot

 

I intend on buying a very basic 10G Switch: MikroTik CRS305-1G-4S+IN

and some cabling to go with it: 10GTEK SFP Transceiver Module (for example)

as well as some 10G PCIe NICs to go with it: ASUS 10G Network Card (dunno, any recommendations? Location: Germany)

I don't have a server room as such and so will be confined to "Towers" and therefore non-rack orientated solutions

 

I have a relatively defined set of workloads that I can test out and so here are the iterations that I think make sense:

The only catch here is that it is rare that I would run the same job over and over again when it comes to my real workload, and so I probably need to clear the ZFS caches before the each run.

 

Establish baseline - With just the 4x SATA HDD in RAIDZ2 

See if having L2ARC helps - With 4x SATA HDD in RAIDZ2 + 1x 500 GB SATA SSD as L2ARC

See if having a larger L2ARC helps- With 4x SATA HDD in RAIDZ2 + 2x 500 GB SATA SSD as L2ARC

See if having a faster L2ARC helps- With 4x SATA HDD in RAIDZ2 + 1x 500 GB NVMe SSD as L2ARC

See if having a ZIL/SLOG helps - With 4x SATA HDD in RAIDZ2 + 1x 500 GB NVMe SSD as L2ARC + 1x 500 GB SATA SSD as L2ARC

See if having a faster ZIL/SLOG helps - With 4x SATA HDD in RAIDZ2 + 1x 500 GB SATA SSD as L2ARC + 1x 500 GB NVMe SSD as ZIL/SLOG

 

 

[unijab]

Imo raidz2 with only 4 drives doesn't make sense. 

With 4 drives, should do 2 vdev mirrors.

[For Science!]

5 minutes ago, unijab said:

Imo raidz2 with only 4 drives doesn't make sense. 

With 4 drives, should do 2 vdev mirrors.

I get what you're saying, but if were to set this up for real with more drives I guess i would opt for z2. Not entirely sure on that one yet, needs more reading.

[leadeater]

1 hour ago, For Science! said:

1. I understand that ZFS is rather RAM heavy, or at least benefits from having free RAM due to its use as ARC cache. Does this mean that a system running the ZFS filesystem is inherently not good for being a workstation itself?

- At the worst case, for example if a RAM intesive job (using, say, 100 GB of 120 GB of system memory), will the system grind to a halt, or will it be able to maintain some degree of disk i/o nonetheless

You can run it under a workstation scenario, for that you would limit ZFS arc size to what you feel is appropriate. ZFS itself can run on extremely small amount of ram in which case you'll be getting performance as close to that of the underlying hardware backing the pool. When it comes to workstation usage there are other options that may make more sense, I'd use ZFS if I was intending on using many many drives and managing a large pool of storage otherwise I'd go with a more simple option.

 

http://fibrevillage.com/storage/169-zfs-arc-on-linux-how-to-set-and-monitor-on-linux

 

1 hour ago, For Science! said:

After doing all this reading, I am under the impresssion that for me, I would benefit from just installing FreeNAS/BSD to see what works and what is a waste. Does this kind of make sense?

There isn't a reason to not just do it directly on the OS you would use for the workstation.

[brwainer]

When Sun was first developing ZFS, it was promoted for use on workstations as much as on servers. AFAIK on modern systems the ARC will shrink automatically if other applications require RAM, but you can also set it manually to whatever you are comfortable with, like 2GB. ZFS doesn’t require a large amount of RAM unless you enable deduplication, in which case it needs to keep the block hash table in memory for optimal writes. Deduplication is where the recommendation of 1GB RAM per 1TB storage comes from, and in that case I would say the system shouldn’t be used for other tasks (unless you had ram above and beyond the recommendation). So no deduplication means no heavy RAM requirement/usage.

 

Use ZFS on whatever OS you would otherwise use anyway. ZoL (ZFS on Linux) has come a long way - the recently released 0.8.0 closed the major feature gaps compared to BSD ZFS, like TRIM support.

 

In terms of L2ARC and ZIL, for a workload that won’t read a particular bit more than once in a while extra ARC is not useful. ARC doesn’t do any guesses as to what you might read in the future, in only caches the things that were just read that might need to be read again. L2ARC just adds a second tier - things that get kicked out of ARC in memory are stored in L2ARC so you have a greater chance of having a cache hit. SLOG is more likely to be able to help you, but it only accelerates Sync writes, not Async writes. For Sync writes, ZFS will first write them to the ZIL so that it can quickly turn around and tell the application that the data is written. Without an SLOG device, the ZIL is spread out across a small bit of each disk in the whole array. For Async writes, ZFS just keeps it in memory until it can write it out to disk.

[AbydosOne]

On 7/30/2019 at 4:20 AM, unijab said:

Imo raidz2 with only 4 drives doesn't make sense. 

With 4 drives, should do 2 vdev mirrors.

From a performance standpoint, yes. However, you can lose any two drives from Z2, versus one from each mirror in a two mirror vdev setup.

 

My four drives are in Z2, and I can't say I've noticed a difference from when I had a two mirror vdev array.

[gamebrigada]

5 hours ago, AbydosOne said:

From a performance standpoint, yes. However, you can lose any two drives from Z2, versus one from each mirror in a two mirror vdev setup.

 

Mine four drives are in Z2, and I can't say I've noticed a difference from when I had a two mirror vdev array.

Noticeable and functional difference are different things. 

 

With RaidZ2 you'll have double the read speed of a single drive, and no difference in write speed over a single drive. 

With Raid10(2 mirror vdevs) you'll have quadruple the read speed, and double the write speed over a single drive.