Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
For Science!

ZFS orientated followup questions

Recommended Posts

Posted · Original PosterOP

Sorry, this post isn't finished, but I need to go now and so I'm posting it prematurely with just two questions (albeit long ones). I will post additional questions later)

 

Dear All,

 

Following an earlier post, I've been doing some reading around ZFS and have some further questions. I hope these aren't too "out there" in terms of understanding. Please assume if something doesn't make sense then it is because I may be using the incorrect term or concept, which I would also appreciate any corrections for.

 

Here it goes:

 

1. I understand that ZFS is rather RAM heavy, or at least benefits from having free RAM due to its use as ARC cache. Does this mean that a system running the ZFS filesystem is inherently not good for being a workstation itself?

- At the worst case, for example if a RAM intesive job (using, say, 100 GB of 120 GB of system memory), will the system grind to a halt, or will it be able to maintain some degree of disk i/o nonetheless

 

2. After doing all this reading, I am under the impresssion that for me, I would benefit from just installing FreeNAS/BSD to see what works and what is a waste. Does this kind of make sense?

 

I have the following hardware already from other builds for experimentation:

Ryzen 7 1700, NH-D15, 1080Ti

Crosshair VI Hero - 8 SATA ports, 1 NVMe slot, ECC-Support

non-ECC memory, 64 GB (16 x 4 GB)

2x 500 GB SATA (Samsung 850 EVO)

4x 8000 GB SATA HDD (Seagate STA8000AS0002)

2x 500 GB M.2 NVMe SSD (Samsung 970 EVO)

+ 2 other functioning computers with a spare PCIe slot

 

I intend on buying a very basic 10G Switch: MikroTik CRS305-1G-4S+IN

and some cabling to go with it: 10GTEK SFP Transceiver Module (for example)

as well as some 10G PCIe NICs to go with it: ASUS 10G Network Card (dunno, any recommendations? Location: Germany)

I don't have a server room as such and so will be confined to "Towers" and therefore non-rack orientated solutions

 

I have a relatively defined set of workloads that I can test out and so here are the iterations that I think make sense:

The only catch here is that it is rare that I would run the same job over and over again when it comes to my real workload, and so I probably need to clear the ZFS caches before the each run.

 

Establish baseline - With just the 4x SATA HDD in RAIDZ2 

See if having L2ARC helps - With 4x SATA HDD in RAIDZ2 + 1x 500 GB SATA SSD as L2ARC

See if having a larger L2ARC helps- With 4x SATA HDD in RAIDZ2 + 2x 500 GB SATA SSD as L2ARC

See if having a faster L2ARC helps- With 4x SATA HDD in RAIDZ2 + 1x 500 GB NVMe SSD as L2ARC

See if having a ZIL/SLOG helps - With 4x SATA HDD in RAIDZ2 + 1x 500 GB NVMe SSD as L2ARC + 1x 500 GB SATA SSD as L2ARC

See if having a faster ZIL/SLOG helps - With 4x SATA HDD in RAIDZ2 + 1x 500 GB SATA SSD as L2ARC + 1x 500 GB NVMe SSD as ZIL/SLOG

 

 

Link to post
Share on other sites

Imo raidz2 with only 4 drives doesn't make sense. 

With 4 drives, should do 2 vdev mirrors.


Can Anybody Link A Virtual Machine while I go download some RAM?

 

Link to post
Share on other sites
Posted · Original PosterOP
5 minutes ago, unijab said:

Imo raidz2 with only 4 drives doesn't make sense. 

With 4 drives, should do 2 vdev mirrors.

I get what you're saying, but if were to set this up for real with more drives I guess i would opt for z2. Not entirely sure on that one yet, needs more reading.

Link to post
Share on other sites
1 hour ago, For Science! said:

1. I understand that ZFS is rather RAM heavy, or at least benefits from having free RAM due to its use as ARC cache. Does this mean that a system running the ZFS filesystem is inherently not good for being a workstation itself?

- At the worst case, for example if a RAM intesive job (using, say, 100 GB of 120 GB of system memory), will the system grind to a halt, or will it be able to maintain some degree of disk i/o nonetheless

You can run it under a workstation scenario, for that you would limit ZFS arc size to what you feel is appropriate. ZFS itself can run on extremely small amount of ram in which case you'll be getting performance as close to that of the underlying hardware backing the pool. When it comes to workstation usage there are other options that may make more sense, I'd use ZFS if I was intending on using many many drives and managing a large pool of storage otherwise I'd go with a more simple option.

 

http://fibrevillage.com/storage/169-zfs-arc-on-linux-how-to-set-and-monitor-on-linux

 

1 hour ago, For Science! said:

After doing all this reading, I am under the impresssion that for me, I would benefit from just installing FreeNAS/BSD to see what works and what is a waste. Does this kind of make sense?

There isn't a reason to not just do it directly on the OS you would use for the workstation.

Link to post
Share on other sites

When Sun was first developing ZFS, it was promoted for use on workstations as much as on servers. AFAIK on modern systems the ARC will shrink automatically if other applications require RAM, but you can also set it manually to whatever you are comfortable with, like 2GB. ZFS doesn’t require a large amount of RAM unless you enable deduplication, in which case it needs to keep the block hash table in memory for optimal writes. Deduplication is where the recommendation of 1GB RAM per 1TB storage comes from, and in that case I would say the system shouldn’t be used for other tasks (unless you had ram above and beyond the recommendation). So no deduplication means no heavy RAM requirement/usage.

 

Use ZFS on whatever OS you would otherwise use anyway. ZoL (ZFS on Linux) has come a long way - the recently released 0.8.0 closed the major feature gaps compared to BSD ZFS, like TRIM support.

 

In terms of L2ARC and ZIL, for a workload that won’t read a particular bit more than once in a while extra ARC is not useful. ARC doesn’t do any guesses as to what you might read in the future, in only caches the things that were just read that might need to be read again. L2ARC just adds a second tier - things that get kicked out of ARC in memory are stored in L2ARC so you have a greater chance of having a cache hit. SLOG is more likely to be able to help you, but it only accelerates Sync writes, not Async writes. For Sync writes, ZFS will first write them to the ZIL so that it can quickly turn around and tell the application that the data is written. Without an SLOG device, the ZIL is spread out across a small bit of each disk in the whole array. For Async writes, ZFS just keeps it in memory until it can write it out to disk.


Looking to buy GTX690, other multi-GPU cards, or single-slot graphics cards: 

 

Link to post
Share on other sites
On 7/30/2019 at 4:20 AM, unijab said:

Imo raidz2 with only 4 drives doesn't make sense. 

With 4 drives, should do 2 vdev mirrors.

From a performance standpoint, yes. However, you can lose any two drives from Z2, versus one from each mirror in a two mirror vdev setup.

 

My four drives are in Z2, and I can't say I've noticed a difference from when I had a two mirror vdev array.


Main System (Byarlant): Ryzen 5 1600X | Asus B350-F Strix | Corsair H80i V2 | 16GB G.Skill DDR4 3200MHz CAS-14 | Reference Nvidia GTX 770 2GB | Samsung 960 PRO 512GB / Samsung 970 EVO 500GB / Seagate 7200RPM 3TB | Corsair CX650M | Mellanox ConnectX-2 10G NIC | Anidees AI-07BW Case | Dell U3415W Monitor | Microsoft Modern Keyboard

 

FreeNAS Server (Veda): Core i3-4170 | Supermicro X10SLL-F | Corsair H60 | 32GB Micron DDR3L ECC 1600MHz | 4x 10TB WD Whites / 1x Samsung PM961 128GB SSD / 1x Kingston 16GB SSD | Corsair CX430M | Mellanox ConnectX-2 10G NIC | LSI 9207-8i LBA | Fractal Design Node 804 Case (side panels swapped to show off drives)

 

Media Center (Jesta): Core i7-2600 | Asus H77M-PRO | Stock Cooler | 8GB No-name DDR3 | EVGA GTX 750Ti SC | Sandisk UltraII SSD 64GB / Seagate 1.5TB HDD | Corsair CX450M | Hauppauge ImpactVCB-PCIe | Syba USB3.1 Gen 2 Card | LG UH12NS30 BD-ROM | Silverstone Sugo SG-11 Case

 

Laptop (Rozen-Zulu): Sony VAIO VPCF13WFX | Core i7-740QM | 8GB Patriot DDR3 | GT 425M | Kingston 120GB SSD | Blu-ray Drive | (Still my favorite typing keyboard)


Tablet (ReGZ): Asus T102HA (BIOS clock doesn't tick, loses time when sleep/off) (I kill tablets with disturbing regularity)

Tablet (Unicorn): Surface Pro 2 (battery will reset total capacity to current charge, leading Windows to think it's always 100% charged until it dies)

Tablet (Loto): Dell Venue 8 Pro (screen discoloration issues, wouldn't update to Windows 10)

Tablet: iPad 2 16GB (WiFi died, basically useless after that)

 

Testbed/Old Desktop (Kshatriya): Xeon X5470 @ 4.0GHz | ZALMAN CNPS9500 | Gigabyte EP45-UD3L | 8GB Nanya DDR2 400MHz | XFX HD6870 DD | OCZ Vertex 3 Max-IOPS 120GB | Corsair CX430M (?) | HooToo USB 3.0 PCIe Card | NZXT H230 Case

 

Camera: Sony A7II (w/ Meike Grip) | Sony SEL24240 (better than most reviews say) | Sony SEL2870 (kit lens) | Sony SEL55F18F (meh, don't buy) | Sony SEL28F20 (on loan from a friend; really sharp) | PNY Elite Perfomance SDXC cards

Link to post
Share on other sites
5 hours ago, AbydosOne said:

From a performance standpoint, yes. However, you can lose any two drives from Z2, versus one from each mirror in a two mirror vdev setup.

 

Mine four drives are in Z2, and I can't say I've noticed a difference from when I had a two mirror vdev array.

Noticeable and functional difference are different things. 

 

With RaidZ2 you'll have double the read speed of a single drive, and no difference in write speed over a single drive. 

With Raid10(2 mirror vdevs) you'll have quadruple the read speed, and double the write speed over a single drive. 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×