ZFS Memory requirements

[dalekphalm]

9 hours ago, AbsoluteFool said:

Doesen't pfsense still advertise that 8GB EEC ram is recomended if you do ZFS install still? That router would pratically become a beast amongs <>< in the ocean  

If you're using FreeNAS for pfsense in particular, perhaps?

 

But as a router OS, would you even need that much RAM? Seems overkill to me - or is pfsense simply that inefficient?

[jde3]

All of this is true. RAM affects ZFS's ARC performance, that's it. If you don't have enough then it may evict objects from cache that you would ordinarily want in there slowing the disk down. The ARC is magic, it's a fantastic cache system and makes ZFS work so well so you do want it to run right but a lack of cache won't affect ZFS's integrity.

 

One thing I've noticed is that if your using ZFS on Linux, Linux's VM for whatever reason seems to fight with ZFS's and if the system gets under RAM pressure the system might get really slow and start page swapping even though it seems the ARC could release objects instead. This is just an observation and my guess at what is going on. I don't have anything backing this up. If you run into this problem tuning vm.swappniess to a low number may help. Linux seems to be too happy to preemptively push things to swap and ZFS's ARC will just go oh.. I have free ram? ok. I'll just use that.

[namAehT]

I had only ever heard the "1GB per 1TB" advice, never knew that the ZFS devs think that's bullshit. The FreeNAS Guide doesn't even recommend it as profusely anymore, simply saying:

Quote

An old, somewhat overstated guideline is 1 GB of RAM per terabyte of disk capacity.

 

BTW, as per this line in the ZFS on Linux GitHub, the ARC Metadata Limit (à la Dedupe Table) is actually allowed to use 75% of the ARC Limit (all memory), rather than the 12.5% the ZFS dev suggested (don't know when that changed). On FreeBSD (or at least FreeNAS) it is 25% of the ARC Limit.

So if you want the max ram used by the dedupe table, it's simply [Max Pool Size in TB]/[Block Size]*320 Bytes. Gonna attempt to dedupe my Media/Seeding dataset.

[PianoPlayer88Key]

So what if someone like me wanted to build a NAS on an extreme budget, strictly for backup purposes?

 

For example, let's say the budget for the entire build, not including the storage, is around $160-240, or cheaper than a single 6 or 8TB HDD.  As for storage, the preference is to find something that's a fraction of the cost per TB compared to a shingled 5400rpm HDD.

 

The NAS would be used ONLY for backups, it wouldn't be a media / streaming / rendering server, etc.  (If I could spend more, I might do those things though.  One board I'd consider if my budget wasn't so limited would be the Supermicro X8DTH-6F, but the cheapest one on eBay right now ($164.99) would exhaust the entire budget.)

 

Speaking of ECC RAM, etc ... if you're on an extreme budget, which would be preferred?  Going for something like

• 32GB (4x8GB) DDR2-667 Kingston (on eBay for $25.25) and a compatible board (like the X7DCL-i + Xeon 5130 + a 9200-16e or 9211-8i), or,
• 12GB (3x4GB) DDR3-1333 Micron (eBay, $24.99), or, ...
• hmm, no DDR4 around $25, cheapest is a 4GB DDR4-2133 Hynix for $40 and there's 1 left.

If I used HDDs in the NAS, the smallest I'd use would be 4 or 6 TB, so support for drives larger than 2TB would be required.  I'd be open to other storage media, although they'd need to be at least several TB each, ruling out BD-R.  I have quite a lot of drives to back up, and a fair bit of data on them (or at least some of them).  

[leadeater]

16 minutes ago, PianoPlayer88Key said:

For example, let's say the budget for the entire build, not including the storage, is around $160-240, or cheaper than a single 6 or 8TB HDD.  As for storage, the preference is to find something that's a fraction of the cost per TB compared to a shingled 5400rpm HDD.

Ebay and used server parts are always king for that kind of thing, when you can get L5630's for $6 each it's not too hard to get a nice system so long as you don't want a nice multi bay rackmount setup.

[jde3]

15 hours ago, namAehT said:

I had only ever heard the "1GB per 1TB" advice, never knew that the ZFS devs think that's bullshit. The FreeNAS Guide doesn't even recommend it as profusely anymore, simply saying:

 

BTW, as per this line in the ZFS on Linux GitHub, the ARC Metadata Limit (à la Dedupe Table) is actually allowed to use 75% of the ARC Limit (all memory), rather than the 12.5% the ZFS dev suggested (don't know when that changed). On FreeBSD (or at least FreeNAS) it is 25% of the ARC Limit.

So if you want the max ram used by the dedupe table, it's simply [Max Pool Size in TB]/[Block Size]*320 Bytes. Gonna attempt to dedupe my Media/Seeding dataset.

I wouldn't, Dedup is kind of busted. They are rewriting how it works from a hash table to a log format. Hard links are better for now, (you just got to be aware of them)

[dalekphalm]

13 hours ago, PianoPlayer88Key said:

So what if someone like me wanted to build a NAS on an extreme budget, strictly for backup purposes?

 

For example, let's say the budget for the entire build, not including the storage, is around $160-240, or cheaper than a single 6 or 8TB HDD.  As for storage, the preference is to find something that's a fraction of the cost per TB compared to a shingled 5400rpm HDD.

 

The NAS would be used ONLY for backups, it wouldn't be a media / streaming / rendering server, etc.  (If I could spend more, I might do those things though.  One board I'd consider if my budget wasn't so limited would be the Supermicro X8DTH-6F, but the cheapest one on eBay right now ($164.99) would exhaust the entire budget.)

 

Speaking of ECC RAM, etc ... if you're on an extreme budget, which would be preferred?  Going for something like

• 32GB (4x8GB) DDR2-667 Kingston (on eBay for $25.25) and a compatible board (like the X7DCL-i + Xeon 5130 + a 9200-16e or 9211-8i), or,
• 12GB (3x4GB) DDR3-1333 Micron (eBay, $24.99), or, ...
• hmm, no DDR4 around $25, cheapest is a 4GB DDR4-2133 Hynix for $40 and there's 1 left.

If I used HDDs in the NAS, the smallest I'd use would be 4 or 6 TB, so support for drives larger than 2TB would be required.  I'd be open to other storage media, although they'd need to be at least several TB each, ruling out BD-R.  I have quite a lot of drives to back up, and a fair bit of data on them (or at least some of them).  

To be honest, if you are just using this as a backup server, I would limit yourself to 8GB RAM max - even 4GB would be totally fine - if a backup server is a bit slower, then that's not really much of a concern realistically.

 

There's really no need to exceed 8GB of RAM if you aren't doing any other things with the server. I have FreeNAS in a VM limited to 8GB of RAM - it runs my file server (backups and media - but no streaming, a Windows VM runs my Plex Server). The performance is pretty good.

 

The biggest cost is likely going to end up being HDD's, because you can get started with a cheap used Desktop or Workstation - for example, a used Dell Optiplex tower.

 

You could check to see if you can find some cheap servers on eBay too, but Servers typically are a lot louder, so do keep that in mind.

 

If you're going to buy used HDD's, I would check your local used market first (Kijiji/Craigslist/Facebook Marketplace/Whatever). Check them out in person before buying. If possible, ask the seller to have the HDD plugged in and running when you get there so you can pull up CrystalDiskInfo and check SMART Stats before leaving.

 

Buying used HDD's on eBay is typically fine, since the buyer protection will afford you a replacement or refund if you end up getting a DOA.

[namAehT]

On 7/21/2018 at 2:20 AM, jde3 said:

I wouldn't, Dedup is kind of busted. They are rewriting how it works from a hash table to a log format. Hard links are better for now, (you just got to be aware of them)

Rather than just enabling it on my dataset, I created a test one and copied everything to it. It was all fine until there was about 14TB (10TB unique), then my write performance dropped to ~20MB/s. I deleted the dataset an everything went back to normal.

I didn't want to go the hard link route because my library is constantly changing (Radarr & Sonarr) and when I eventually stop seeding I don't want my files to vanish.
 

On 7/21/2018 at 1:44 AM, PianoPlayer88Key said:

If I used HDDs in the NAS, the smallest I'd use would be 4 or 6 TB, so support for drives larger than 2TB would be required.  I'd be open to other storage media, although they'd need to be at least several TB each, ruling out BD-R.  I have quite a lot of drives to back up, and a fair bit of data on them (or at least some of them).  

 

I've bought refurb drives from GoHardDrive before and have had no problems.

[dalekphalm]

40 minutes ago, namAehT said:

Rather than just enabling it on my dataset, I created a test one and copied everything to it. It was all fine until there was about 14TB (10TB unique), then my write performance dropped to ~20MB/s. I deleted the dataset an everything went back to normal.

I didn't want to go the hard link route because my library is constantly changing (Radarr & Sonarr) and when I eventually stop seeding I don't want my files to vanish.
 

I've bought refurb drives from GoHardDrive before and have had no problems.

Just out of curiosity, how much RAM is installed in your FreeNAS system when you were doing the test dedupe?

[newgeneral10]

I still think you need to focus on the ZFS tunables instead of RAM size, i.e. enabling prefetch for stream boxes, setting appropriate compression and recordsize on your datasets.

 

On 7/20/2018 at 10:32 AM, namAehT said:

BTW, as per this line in the ZFS on Linux GitHub, the ARC Metadata Limit (à la Dedupe Table) is actually allowed to use 75% of the ARC Limit (all memory), rather than the 12.5% the ZFS dev suggested (don't know when that changed). On FreeBSD (or at least FreeNAS) it is 25% of the ARC Limit.

According to this page, arc limits are configurable as well.

[namAehT]

On 8/1/2018 at 9:13 AM, dalekphalm said:

Just out of curiosity, how much RAM is installed in your FreeNAS system when you were doing the test dedupe?

48GB, but dedupe was (in therory) using something like 2GB (Duplicate entries * size in core + unique entries * size in core) so RAM size wasn't a bottle neck.

My guess is that it takes time to hash the block, look through the dedupe table for matching entries and if it finds a match, verify the block is truely unique. Moving a 50GB file means hashing and checking 50,000 blocks against the dedupe table, even if the whole thing is in RAM it can only process so fast.

 

EDIT: Post to back this up.

[dalekphalm]

1 hour ago, namAehT said:

48GB, but dedupe was (in therory) using something like 2GB (Duplicate entries * size in core + unique entries * size in core) so RAM size wasn't a bottle neck.

My guess is that it takes time to hash the block, look through the dedupe table for matching entries and if it finds a match, verify the block is truely unique. Moving a 50GB file means hashing and checking 50,000 blocks against the dedupe table, even if the whole thing is in RAM it can only process so fast.

 

EDIT: Post to back this up.

Makes sense - I was just checking to make sure you didn't have insufficient RAM installed - because Dedupe is one of the few instances in FreeNAS where more RAM actually matters (to a point, as you noted).

 

I think you've just reached the limitations of speed when using Dedupe. It's always been a performance hog.

 

Personally I would only use dedupe on an archive style backup server that doesn't see a lot of read/writes.

[namAehT]

11 minutes ago, dalekphalm said:

Personally I would only use dedupe on an archive style backup server that doesn't see a lot of read/writes.

2

That's the same conclusion I reached, I'll probably use dedupe when I get around to making an offsite backup server just to cut down on costs and increase capacity.

[Mikensan]

I tested dedupe on a zvol shared over iscsi to ESXi for virtual machines. I didn't get very high numbers, maybe 3x. I have 32gb and when I checked I had consumed about 6GB for dedupe tables. Roughly 1TB worth of data.. Speeds suffered quite a bit. Was getting almost 440mbyte/s and after dedupe it dropped to 80-100... Switched to good old LZ4 and got 2.2x which I'm perfectly happy with.

[namAehT]

Just now, Mikensan said:

Switched to good old LZ4 and got 2.2x which I'm perfectly happy with.

 

The biggest problem for me is that all my data (Movies & TV) is already compressed, so I'd get about 1.02 if I enabled it.

[Mikensan]

58 minutes ago, namAehT said:

The biggest problem for me is that all my data (Movies & TV) is already compressed, so I'd get about 1.02 if I enabled it.

Yea honestly I don't enable compression or even attime on my movie/seed datasets and just eat the cost of storage. I just trim the fat every 30 days, pretending there won't be a day I need more storage lol.

 

Speaking of media - did you change the default block size or anything for your media datasets?

 

Was mostly writing to also state the shortcomings of dedupe, although I read there's plans in the works to improve upon it.

[namAehT]

Just now, Mikensan said:

Yea honestly I don't enable compression or even attime on my movie/seed datasets and just eat the cost of storage. I just trim the fat every 30 days, pretending there won't be a day I need more storage lol.

 

Speaking of media - did you change the default block size or anything for your media datasets?

 

Was mostly writing to also state the shortcomings of dedupe, although I read there's plans in the works to improve upon it.

I just keep adding storage and have become a frequent user of /r/DataHoarder. My recently expanded 38TB pool is sitting at the 50% mark and I'm already planning on buy more 8TB Easystores or MyBooks when they go on sale. It's become an addiction...

 

I didn't change the default blocksize since there really isn't a point to it. ZFS automagically decides what the best blocksize is for a given file as you're writing it, so if you're writing a multi-gig file it will write 1MB blocks, but if you're writing a bunch of 1k text files it will use small blocks. At least that's my understanding of it.

Since (nearly) everything on this dataset is over 1MB, I'm assuming that all the blocks on it are 1MB for any approximations.