Strange FreeNAS behavior

[Mikensan]

Specs:

FreeNAS1:

E3-1220v3 / 32GB / 5x 4TB RAIDZ1

DD test: 400 read/write

==volume==

zil/log: 1x SSD

compression (default lz4, NFS dataset: off)

atime: (default on, NFS dataset: off)

sync: (default on, NFS dataset: standard)

 

FreeNAS2:

E3-1225v3 / 12GB / 3x 500gb Stripe (SSD)

**12GB It has 1x 4GB and 1x 8GB...

==Volume==

atime: no

sync: standard (temporarily set to off for test)

Compression: no

DD test: 1.2gB/s write, 1.6gB/s read

 

 

Virtual Host:

Xeon X5670 x2 / 96GB 

 

All using intel gigabit cards of one model or another.

 

Issue:

Since the SSD volume got some pretty awesome throughput (imo), I *should* be able to saturate a gigabit or whatever the line can handle. This was extremely long winded initially, so I just summed up the results...

 

Tested the below protocols:

iSCSI: 100mB/s (Virtual host to FreeNAS2 using 7gB vhd)

NFS: 10-20mB/s (sync: Standard/Off = no difference) (Virtual host to FreeNAS2 using 7gB vhd)

rsync: 10-20mB/s (FreeNAS1 to FreeNAS2 using 10gB movie)

SMB: 100mB/s (Workstation to FreeNAS2 using 10gB movie)

 

Thoughts

Since memory being an issue is beaten to death on the FreeNAS forums, I'm lead to believe only having 12gb (or the mismatch pair) is my issue. Since when pushing data over NFS/rsync does get up to 20mB/s at time it doesn't appear to be something in my network only connecting at 100mbit (also the fact that iSCSI/SMB nearly hit 1gbit). NFS on my FreeNAS1 box has no issues (the volume has a SSD zlog). I'm baffled.

[code_glitch]

by mB/s I am assuming megaByte/s? If so, youre 100mB/s speed is pretty much what I'd expect to see from gigabit LAN without jumbo frames. If thats in megabits/s then something is very wrong and you should probably test to make sure your network is indeed running gigabit.

 

Since most example of performance are for well configured, recommended setups you might find some worst-case (read: youre abusing freenas and everything freenas stands for). I have actually have 2 of these boxes setup, one with me and one back home across the country (because off-site, in house and all that). Details collapsed in spoiler bit below.

 

Spoiler

 

Physical hardware in both cases is similar: FM1/FM2 socket athlon X4 performance level cpu, 8GB DDR3 on some cheap board in a crummy case. The OS on the metal is lightly customized proxmox which run a number of VMs (FreeNAS, torrent box, media server, ssh access, virtual linux desktop for long jobs I cant be asked to run on my machine (transcoding/sorting out single disk rips... things you can do slowly with extra cycles), virtual windows box (solidworks breaks under modern wine and extra-crappy apps that dont run in wine either I may need for whatever reason).

 

The FreeNAS VMs in both cases are given ~2GiB of ram. All these VMs have their boot disks on an ssd, the hdds are almost exclusively allocated to freenas..

Machine 1 runs RaidZ1 on 2x 2TB WD blue drives and 2x 2TB WD Green drives (it serves a 100m ethernet network so just needs to be cheap). Over SMB, this generally reads at ~250mbit/s, writes are a bit worse at around 150mbit/s. I havent bothered with NFS on this box since I dont use it often.

 

Machine 2 has 6x 3TB WD green drives in Raidz2. Over SMB I can write at ~400mbit/s, reads are in excess of 700mbit/s. On NFS I generally see about 20% faster speeds in everything and I can generally saturate my 1gbit LAN while reading. Note that with the torrent boxes and indexing and other users these arent the theoretical peak performance but more of a day-day case.

 

FWIW: Each machine has something like 16GiB of L2ARC on the boot SSD which helps even things out a lot. On my hardware I found jumbo frames led to some extra flakiness I didnt care for so I'm not too fussed about only getting about 70% of the theoretical maximum speed. I run a lot of stuff of it (all the steam games and to be completely honest, once L2ARC has indexed what you use most, it rarely gets bottlenecked anyway).

Edit: Each FreeNAS VM is given 2 cores and guaranteed its ram so it doesnt ever get swapped out. My CPU cores are over-allocated but looking at the performance stats, CPU usage is >NEVER< over about 80% on the metal on either machine, although the freenas VMs to tend to reach 100% load if you count steal + io time.

 

 

[unijab]

OS on hypervisor?

[leadeater]

ESXi? I know that needs a little bit of tweaking to get good NFS performance.

 

Quote

1) tyou can use MTU JUMBO FRAME. it will increase the throughput.  2)  could you please check what is the  NFS TcpipHeapSize value configured in your host. ( im not sure in 5.5 but 5.1 default value is 0) you can check this configuration under advanced settings- NFS option. (Net.TcpIpHeapSize is the size of the memory (in MB) which is allocated up front by the VMkernel to TCP/IP heap. Net.TcpIpHeapMax is the maximum amount of memory which can be consumed by TCP/IP as heap). when you want to change this value you should need to consider other parameters.  you can change these parameters as recommended Net/TcpipHeapSize=32, Net/TcpipHeapMax=128, NFS/HeartbeatMaxFailures=10, NFS/HeartbeatFrequency=12, NFS/HeartbeatTimeout=5, NFS/MaxVolumes=256, for fallback plan, before make changes, please note down the default values.

https://communities.vmware.com/thread/462853

 

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-nfs-bestpractices-white-paper-en.pdf

[Mikensan]

17 hours ago, code_glitch said:

by mB/s I am assuming megaByte/s? If so, youre 100mB/s speed is pretty much what I'd expect to see from gigabit LAN without jumbo frames. If thats in megabits/s then something is very wrong and you should probably test to make sure your network is indeed running gigabit.

 

Since most example of performance are for well configured, recommended setups you might find some worst-case (read: youre abusing freenas and everything freenas stands for). I have actually have 2 of these boxes setup, one with me and one back home across the country (because off-site, in house and all that). Details collapsed in spoiler bit below.

 

  Reveal hidden contents

 

Physical hardware in both cases is similar: FM1/FM2 socket athlon X4 performance level cpu, 8GB DDR3 on some cheap board in a crummy case. The OS on the metal is lightly customized proxmox which run a number of VMs (FreeNAS, torrent box, media server, ssh access, virtual linux desktop for long jobs I cant be asked to run on my machine (transcoding/sorting out single disk rips... things you can do slowly with extra cycles), virtual windows box (solidworks breaks under modern wine and extra-crappy apps that dont run in wine either I may need for whatever reason).

 

The FreeNAS VMs in both cases are given ~2GiB of ram. All these VMs have their boot disks on an ssd, the hdds are almost exclusively allocated to freenas..

Machine 1 runs RaidZ1 on 2x 2TB WD blue drives and 2x 2TB WD Green drives (it serves a 100m ethernet network so just needs to be cheap). Over SMB, this generally reads at ~250mbit/s, writes are a bit worse at around 150mbit/s. I havent bothered with NFS on this box since I dont use it often.

 

Machine 2 has 6x 3TB WD green drives in Raidz2. Over SMB I can write at ~400mbit/s, reads are in excess of 700mbit/s. On NFS I generally see about 20% faster speeds in everything and I can generally saturate my 1gbit LAN while reading. Note that with the torrent boxes and indexing and other users these arent the theoretical peak performance but more of a day-day case.

 

FWIW: Each machine has something like 16GiB of L2ARC on the boot SSD which helps even things out a lot. On my hardware I found jumbo frames led to some extra flakiness I didnt care for so I'm not too fussed about only getting about 70% of the theoretical maximum speed. I run a lot of stuff of it (all the steam games and to be completely honest, once L2ARC has indexed what you use most, it rarely gets bottlenecked anyway).

Edit: Each FreeNAS VM is given 2 cores and guaranteed its ram so it doesnt ever get swapped out. My CPU cores are over-allocated but looking at the performance stats, CPU usage is >NEVER< over about 80% on the metal on either machine, although the freenas VMs to tend to reach 100% load if you count steal + io time.

 

 

by mB/s I am assuming megaByte/s? If so, youre 100mB/s speed is pretty much what I'd expect to see from gigabit LAN without jumbo frames. If thats in megabits/s then something is very wrong and you should probably test to make sure your network is indeed running gigabit.

Yup bytes, hence why I'm concerned about it only going 10-20mbyte/s via NFS/rsync. 

read: youre abusing freenas and everything freenas stands for

Are you saying I'm abusing freenas and everything it stands for?

 

15 hours ago, unijab said:

OS on hypervisor?

Nope, all 3 separate machines. 

 

15 hours ago, leadeater said:

ESXi? I know that needs a little bit of tweaking to get good NFS performance.

 

Yup ESXi managed via vcenter.

https://communities.vmware.com/thread/462853

Oddly enough the first thing I did when setting up my ESXi server was using the dell image, much better drivers. In regards to heap size the host's configuration appears to work well for my other FreeNAS box using NFS (in the PDF it mentions to adjust the heap once you break 8 NFS datastores).

 

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-nfs-bestpractices-white-paper-en.pdf

NFS traffic is already on a dedicated VLAN, no trunking or shared lines. Only two NFS mounts. No NIC teaming to configure.

 

 

When using NFS shares from my FreeNAS1 box on the ESXI host, it works flawlessly. Only two real differences between the NASs is 12gb vs 32gb and the FreeNAS2 box using a SSD volume (which I assumed it to be better). Also since rsync is have the same crap-tastic write speeds as NFS I presume it isn't entirely NFS' fault.

 

I'm going to try and connect these SSDs to my other (FreeNAS1) box and see if there's an issue with FreeNAS and all-SSD arrays. I believe I have at least 3 SATA ports left, not sure about power though ~_~, need splitters I believer.

[leadeater]

1 hour ago, Mikensan said:

When using NFS shares from my FreeNAS1 box on the ESXI host, it works flawlessly. Only two real differences between the NASs is 12gb vs 32gb and the FreeNAS2 box using a SSD volume (which I assumed it to be better). Also since rsync is have the same crap-tastic write speeds as NFS I presume it isn't entirely NFS' fault.

You using rsync over ssh or NFS? I've seen many complaints about rsync bad performance anyway so kinda lumped this in as a protocol performance issue since iSCSI, SMB and DD showed perfectly fine performance.

 

Looking back over your config the issue is likely NFS mounted with sync writes option? Which is ESXi default. ZFS sucks performance wise for sync writes because it disables almost everything that helps performance wise (buffers/delayed write etc). The standard way to improve this performance is to add an SSD SLOG which you have done for FreeNAS1 server and will be why that one is performing fine but the all SSD FreeNAS2 server is not.

 

I know in your post you mentioned that you tried disabling sync but was that on the FreeNAS side of things or on the ESXi mount options?

 

https://forums.freenas.org/index.php?threads/sync-writes-or-why-is-my-esxi-nfs-so-slow-and-why-is-iscsi-faster.12506/

[Mikensan]

1 hour ago, leadeater said:

You using rsync over ssh or NFS? I've seen many complaints about rsync bad performance anyway so kinda lumped this in as a protocol performance issue since iSCSI, SMB and DD showed perfectly fine performance.

 

Looking back over your config the issue is likely NFS mounted with sync writes option? Which is ESXi default. ZFS sucks performance wise for sync writes because it disables almost everything that helps performance wise (buffers/delayed write etc). The standard way to improve this performance is to add an SSD SLOG which you have done for FreeNAS1 server and will be why that one is performing fine but the all SSD FreeNAS2 server is not.

 

I know in your post you mentioned that you tried disabling sync but was that on the FreeNAS side of things or on the ESXi mount options?

 

https://forums.freenas.org/index.php?threads/sync-writes-or-why-is-my-esxi-nfs-so-slow-and-why-is-iscsi-faster.12506/

Used SSH for rsync (logged into FreeNAS1 and ran rsync -Pv /mnt/volume/dataset/file username@hostIP:/mnt/volume/dataset), and that makes me feel a little better that it's not know for speed. Given the way it works I always thought there would be a lot of overhead, but I've never used it in "real world" scenarios.

 

I tried disabling sync writes on the FreeNAS side.  FreeNAS has 3 modes, Off/Standard/Always - standard will only perform sync writes when requested but as I understand NFS is sync writes 100% of the time. I also created a new 2 disk SSD stripe with the third disk used as a SLOG to no avail :-(. 

 

As I write this I'm mounting the NFS share a linux box to see if it's just vmware that's not playing nice. 

 

Also the only reason I'm using NFS is because iSCSI via FreeNAS on vmware = space destroyer. It infinitely grows and never reclaims space. A quick example is I move a 10gb mail server to the iSCSI target, datastore reports 10.0gb flat. iSCSI extent shows 10.5gb - ok so there's overhead possibly because of block sizes not being perfect? Then I delete the mail server (poof gone) and move my web server, which is 5gb to the iSCSI share. Datastore reports 5gb used, iSCSI extent reports 16gb used. My 1TB datastore had ate up 6TB on my NAS. (on my SSD iscsi the 300gb datastore had hit somewhere around 800gb). This to me seems like a FreeNAS issue and just doesn't feel worth it to fight when NFS is supposedly an acceptable solution (as such with my FreeNAS1 box).

[code_glitch]

Well youve beaten me to the punch by a literally a minute...

3 hours ago, Mikensan said:

read: youre abusing freenas and everything freenas stands for

Are you saying I'm abusing freenas and everything it stands for?

 

Apologies I should probably have been a bit clearer. My use case is abusing FreeNAS and what it stands for (data integrity and reliability).

 

I'll confess to not having touched esxi in a very long time but my hypothesis would be something on the esxi side being screwy. If you're pushing good speeds over iSCSI and SMB then really your NFS speeds should be equal to or faster than those. Rsync is generally just not the fastest so I wouldn't put too much stock in that since as far as I can tell its more obsessed with security & minimal bandwidth than outright speed.

 

What version of NFS is esxi running, and what write speed does it get over NFS? Looking at the number of articles out there on optimizing NFS I'd say that esxi + NFS is the weirdness in your setup to be honest, and your idea of testing on another linux box should put that to rest pretty soon.

 

From your OP though, if you want to put your suspicions of ram being an issue to bed then you could boot some live cd of some basic distro and run something like pmbw.

 

[Mikensan]

Seems like an issue with FreeNAS. I installed CentOS and setup a NFS share, getting ~800-900mb/s and all I did was install zfs and import the pool. I'm not sure what FreeNAS is doing differently. Maybe I'll just keep it this way ~_~

[newgeneral10]

On 8/29/2017 at 1:47 PM, Mikensan said:

Used SSH for rsync (logged into FreeNAS1 and ran rsync -Pv /mnt/volume/dataset/file username@hostIP:/mnt/volume/dataset), and that makes me feel a little better that it's not know for speed. Given the way it works I always thought there would be a lot of overhead, but I've never used it in "real world" scenarios.

 

I tried disabling sync writes on the FreeNAS side.  FreeNAS has 3 modes, Off/Standard/Always - standard will only perform sync writes when requested but as I understand NFS is sync writes 100% of the time. I also created a new 2 disk SSD stripe with the third disk used as a SLOG to no avail :-(. 

 

As I write this I'm mounting the NFS share a linux box to see if it's just vmware that's not playing nice. 

 

Also the only reason I'm using NFS is because iSCSI via FreeNAS on vmware = space destroyer. It infinitely grows and never reclaims space. A quick example is I move a 10gb mail server to the iSCSI target, datastore reports 10.0gb flat. iSCSI extent shows 10.5gb - ok so there's overhead possibly because of block sizes not being perfect? Then I delete the mail server (poof gone) and move my web server, which is 5gb to the iSCSI share. Datastore reports 5gb used, iSCSI extent reports 16gb used. My 1TB datastore had ate up 6TB on my NAS. (on my SSD iscsi the 300gb datastore had hit somewhere around 800gb). This to me seems like a FreeNAS issue and just doesn't feel worth it to fight when NFS is supposedly an acceptable solution (as such with my FreeNAS1 box).

Do you have snapshots set up on your FreeNas?  As a Copy on Write filesystem, moving data around is creating several duplicate copies and could therefore eat up your space.  If you have a crap ton of ram you can set up deduplication to help with that

[Mikensan]

5 hours ago, newgeneral10 said:

Do you have snapshots set up on your FreeNas?  As a Copy on Write filesystem, moving data around is creating several duplicate copies and could therefore eat up your space.  If you have a crap ton of ram you can set up deduplication to help with that

I was using FreeNAS for snapshotting and neither box can take more than 32gb of ram :'( My speed tests were done using VMs without snapshots, snapshots cleared from the ZFS volumes, and a movie file for the rsync and linux > zfs:NFS mount testing. I've given up for now, since linux seems to be happily chugging along with ZFS, though I do miss the easyness of FreeNAS.

[newgeneral10]

On 9/6/2017 at 9:32 PM, Mikensan said:

I was using FreeNAS for snapshotting and neither box can take more than 32gb of ram :'( My speed tests were done using VMs without snapshots, snapshots cleared from the ZFS volumes, and a movie file for the rsync and linux > zfs:NFS mount testing. I've given up for now, since linux seems to be happily chugging along with ZFS, though I do miss the easyness of FreeNAS.

I am still fairly new to FreeNas and while I too love how easy it is to use on a regular basis, there is always things like this that trip me up...

[Mikensan]

2 hours ago, newgeneral10 said:

I am still fairly new to FreeNas and while I too love how easy it is to use on a regular basis, there is always things like this that trip me up...

Very true, just the random things here and there that just throw me for a loop. This weekend I'm going to switch from CentOS to OMV+ZFS and see how that goes. While I do enjoy delving into CLI for certain tasks, the simpler things I'd like to do via GUI. 

[paddy-stone]

Agree, actually I have memory problems and is harder for me to remember CLI now. So most things I have to find a GUI for it, that's what annoys me about proxmox, there doesn't appear to be a way of setting up hard drives on it that doesn't involve CLI, so ended up going with ESXi instead, or trying.. still working some thinsg out.

But have learned loads from freenas, have been running it for around 2 years+ now, but had a very simplistic set up until a month or 2 ago, found some freenas bugs too along the way, but once I knew they were bugs I just reset back to defaults, loaded on my last good backed up version I had of the system and did it again and it worked, everything was done entirely the same too. Anyway, doing that and finding ways around certain problems got me to know a bit more about freenas. I am just glad that I already knew to have a good system backup before making any changes... that saved my ass

 

[Mikensan]

So turns out that the issue is my Samsung Evo SSDs, they're not good SLOG devices nor can they handle sync writes. I'm not sure why my earlier testing of disabling sync writes didn't make a difference (maybe I goofed somewhere) but these little fellas aint meant for sync writes.