FreeNAS RaidZ 6x4TB - 500MB/s+ Write But 25MB/s Read?!

[Inrix]

Im accessing a FreeNAS network share via a p2p 10Gbe Adaptor. I can write to the array at over 500MB/s but when reading I only get 25MB/s...

Do you have any idea of what could be causing it to be so slow? I have tried without Raid, just one spanning volume and have had the same issue. Compression dedupe etc is off...

[leadeater]

What's the RAID level and how many disks?

 

Do some tests local to the FreeNAS server with something like dd or bonnie++.

[Inrix]

Just now, leadeater said:

What's the RAID level and how many disks?

 

Do some tests local to the FreeNAS server with something like dd.

Z-1 and 6x4TB. I just tested with a single disk volume and got the normal 114mb/s write that you would expect... So its something to do with having more than one drive in the array I think

[leadeater]

4 minutes ago, Inrix said:

Z-1 and 6x4TB. I just tested with a single disk volume and got the normal 114mb/s write that you would expect... So its something to do with having more than one drive in the array I think

Weird, I'd still do a local test just in case but you should be getting something like this

 

Quote

 5x 4TB, raidz1 (raid5),       15.0 TB,  w=469MB/s , rw=79MB/s  , r=598MB/s

https://calomel.org/zfs_raid_speed_capacity.html

 

Maybe it's a bad disk in the array or something like that.

[Inrix]

Just now, leadeater said:

Weird, I'd still do a local test just in case but you should be getting something like this

 

https://calomel.org/zfs_raid_speed_capacity.html

 

Maybe it's a bad disk in the array or something like that.

Im about to do a dd test from the array to the SSD the os is hosted on, But none of the disks in the array should be bad they are all brand new minus two which I was just using and were working fine on my windows system

[Inrix]

Just did a DD test... Same read results. Any suggestions on how I can test to try and figure out whats causing this?

[Inrix]

5 minutes ago, leadeater said:

-Snip-

Re ^^

[leadeater]

4 minutes ago, Inrix said:

Just did a DD test... Same read results. Any suggestions on how I can test to try and figure out whats causing this?

Not without breaking the array and making single disk volumes to check all of them then create a new pool and add the disks in one by one and test each time etc.

 

Other option is getting some raw physical disk stats up like queue and I/O latency so see if one disk is dragging the array down.

[Inrix]

Just now, leadeater said:

Not without breaking the array and making single disk volumes to check all of them then create a new pool and add the disks in one by one and test each time etc.

 

Other option is getting some raw physical disk stats up like queue and I/O latency so see if one disk is dragging the array down.

Well its a brand new array so thankfully I can do that. Gonna suck though  Im gonna test over FTP to make sure it isnt SMB then see if there is one disk with issues... Wish me luck ;-;

 

[Windows7ge]

I wonder if checking Reporting > Disk under the FreeNAS WebUI and looking at the disk activity/file transfer speeds would point out any one disk dragging down the array or if they all show ~ identical activity.

 

I'm doubting it's a controller issue, I expect you didn't configure a hardware raid before setting up the software raid correct?

 

I've had bad SATA cables cause drives to malfunction. Checking disk reporting would show a performance irregularly on the affected disk if it doesn't drag down all the others identically.

 

I forget if there's a way to check for drive errors on FreeNAS.

 

Those are my only ideas ATM. Try different SATA ports/cables but that's pretty basic and probably not the issue.

[Inrix]

31 minutes ago, leadeater said:

Not without breaking the array and making single disk volumes to check all of them then create a new pool and add the disks in one by one and test each time etc.

 

Other option is getting some raw physical disk stats up like queue and I/O latency so see if one disk is dragging the array down.

Ok just tested each drive as its own share. They all read fine, this is looking to be more a FreeNAS issue now

[Inrix]

12 minutes ago, Windows7ge said:

-Snip-

Im using a SAS card configured to passthrough SATA so the controller shouldnt have any issues... Its beginning to look like FreeNAS just slows down when I add a 5th drive to the array

[Windows7ge]

16 minutes ago, Inrix said:

Im using a SAS card configured to passthrough SATA so the controller shouldnt have any issues... Its beginning to look like FreeNAS just slows down when I add a 5th drive to the array

Software wise freeNAS shouldn't care. Odd numbers shouldn't impact performance. Hardware wise could be different as if there's a very slow read buffer for some reason. Check the servers processes. Is anything using excessive resources? If you've used up your physical ram and it's gone into using disk as ram you'll have severely reduced performance. Though this is doubtfully the case.

[Inrix]

Just did a fresh install and now im getting this error 

Unable to GPT format the disk "da0": gpart: geom 'da0': File exists

At this point im seriously considering trying a different OS. FreeNAS seems to just be buggy af

[leadeater]

1 hour ago, Inrix said:

Just did a fresh install and now im getting this error 

Unable to GPT format the disk "da0": gpart: geom 'da0': File exists

At this point im seriously considering trying a different OS. FreeNAS seems to just be buggy af

Need to run a clean on the disks to erase the GPT/MBR info, I actually find that easier to do on Windows (diskpart clean)

[Inrix]

Just now, leadeater said:

Need to run a clean on the disks to erase the GPT/MBR info, I actually find that easier to do on Windows (diskpart clean)

Just spent 2 hours trying to clean in cli before giving up and wiping the disks with a windows hdd I had laying around. Recreated the entire thing and tested on both adapters. Now I was able to get 100mb/s over the 1gb link but only 24mb/s read over the 10gig p2p. So I think it might be a issue with the 10gig link... *sigh* Im gonna look into the networking stuff and try figure out whats causing it to be upset, its on a different subnet and everything

[leadeater]

2 minutes ago, Inrix said:

Just spent 2 hours trying to clean in cli before giving up and wiping the disks with a windows hdd I had laying around. Recreated the entire thing and tested on both adapters. Now I was able to get 100mb/s over the 1gb link but only 24mb/s read over the 10gig p2p. So I think it might be a issue with the 10gig link... *sigh* Im gonna look into the networking stuff and try figure out whats causing it to be upset, its on a different subnet and everything

And local dd/bonnie++ gives equally good results?

 

On the 10Gb links bump up the send/receive buffers and enable jumbo frames (or disable if it's on already to test that).

[Inrix]

2 minutes ago, leadeater said:

And local dd/bonnie++ gives equally good results?

 

On the 10Gb links bump up the send/receive buffers and enable jumbo frames (or disable if it's on already to test that).

Nevermind  It was just windows caching the transfer from before..... So FreeNAS is still having issues with the array itself. What do you suggest I do now, because I know the drives are fine, I think the controller card/cables might have a issue but im not entirely sure. TBH I think im just going to turn everything off and leave this until tomorrow, I have been trying to get this to work for the past 9 hours and am really over fucking around with it.

 

If you have any good ideas do shout them at me

[Inrix]

5 minutes ago, leadeater said:

--

Do you think it could be ram related? The system only has 8GB atm. I have another 8GB of ECC memory that I could put in it but its not the same make so I dunno if it would be worth testing with it or if it would just cause more errors

[leadeater]

5 minutes ago, Inrix said:

Do you think it could be ram related? The system only has 8GB atm. I have another 8GB of ECC memory that I could put in it but its not the same make so I dunno if it would be worth testing with it or if it would just cause more errors

Shouldn't be ram, not for read anyway. Could add the ram and find out.

 

Able to plug the disks in to the motherboard instead of the SAS controller to rule that out.

 

Although if the disks work fine individually I'm kind of more leaning towards a software issue of some kind. Try a different version of FreeNAS.

[Inrix]

3 minutes ago, leadeater said:

Shouldn't be ram, not for read anyway. Could add the ram and find out.

 

Able to plug the disks in to the motherboard instead of the SAS controller to rule that out.

 

Although if the disks work fine individually I'm kind of more leaning towards a software issue of some kind. Try a different version of FreeNAS.

Im currently running 

11.1-U4   

What do you suggest I try rollback to?

 

http://iso.cdn.freenas.org/11/

[leadeater]

2 minutes ago, Inrix said:

Im currently running 

11.1-U4   

What do you suggest I try rollback to?

 

http://iso.cdn.freenas.org/11/

No idea, I'm not a FreeNAS user so I couldn't say which version is known good myself. You could also use a Linux Live CD an install the ZFS package and import the pool and retest, fully agree with the screw this do it tomorrow though.

[Inrix]

4 minutes ago, leadeater said:

No idea, I'm not a FreeNAS user so I couldn't say which version is known good myself. You could also use a Linux Live CD an install the ZFS package and import the pool and retest, fully agree with the screw this do it tomorrow though.

If I was to use something other than FreeNAS what would you recommend?

[leadeater]

2 minutes ago, Inrix said:

If I was to use something other than FreeNAS what would you recommend?

Depends what you need from the server, either CentOS Minimal or Ubuntu Server

[Inrix]

Just now, leadeater said:

Depends what you need from the server, either CentOS Minimal or Ubuntu Server

I just want a redundant setup for a NAS with Raid5 type capabilities. But I didnt want to go with a minimal linux server because I really want a WebGui with stats etc for easy management.