Not getting the expected performance out of my FreeNAS server and I don't know why.

[Vitalius]

Hi guys,

Let me preface this with the server specs:

AMD Opteron 6212 Interlagos 2.6GHz 8-core CPU
SUPERMICRO MBD-H8SGL-F-O ATX Server Motherboard 

Wintec Server Series 64GB (4 x 16GB) ECC Registered DDR3 1600

Four 4TB Western Digital RE 7200RPM Hard Drives in (what I assume to be) RAID 10

Kingston HyperX 3K 120GB SSD for L2ARC/ZIL (NOT INSTALLED YET)
Two Supermicro AOC-USAS2-L8E 8-Port SAS 6Gb/s Internal Raid Adapters (two 4TB drives attached to each)

 

So, imo, this thing should be pretty darn fast in terms of read/write speeds. Relative to normal storage systems.

Here's the thing, I'm getting 20-25MB/s copying from the RAID 10 to the same RAID 10 (moving files from one volume to another on the RAID). 

I don't understand why it's so slow. I feel I would get faster speeds on a single drive by itself. I know it's in RAID 1, but RAID 0 should help that. Note the "what I assume to be" RAID 10. The GUI isn't clear how it goes with setting up RAID volumes. It was a 2x2 square, so I assume each column represents a mirror and each period represents a stripe.

I mean what else could it be?



Any help is appreciated. 

Thanks,
Vitalius

P.S. I noted the 120GB SSD for the L2ARC and ZIL because I'm not sure how it will affect performance. Hopefully a lot. 

[Vitalius]

Any way you can test 1 drive at a time?

I didn't consider doing that because of the work I went through to get it all setup. It wasn't much to be honest, but for some reason I didn't consider it originally. Give me an hour, tops.

[wpirobotbuilder]

What sort of data are you copying? Large files, lots of small ones?

 

Also, since you're copying from one volume to another, data might not be hitting the ARC or the L2ARC at all (I have no experience with FreeNAS). If your data isn't in the ARC, then it can be much slower.

 

Do you have any other settings turned on? Compression, deduplication?

 

I can't imagine it would be your CPU.

[Vitalius]

What sort of data are you copying? Large files, lots of small ones?

 

Also, since you're copying from one volume to another, data might not be hitting the ARC or the L2ARC at all (I have no experience with FreeNAS). If your data isn't in the ARC, then it can be much slower.

 

Do you have any other settings turned on? Compression, deduplication?

 

I can't imagine it would be your CPU.

Mix. Some as small as a few KB (Excel documents), some as large as 10GB (Thunderbird profiles, as Thunderbird saves your emails in a single large file). 

Different volumes, same RAID array (so the same disks). That is true. I have no L2ARC yet. I'm going to stress test the HDDs then rebuild the setup with the L2ARC and ZIL added. 

Nope. I had scrubs turned on but that's on a scheduled basis. I have Snapshots turned on, but again, scheduled basis. And there were none running while I did this.

CPU is registering less than 10% utilization even while copying, so it's not that. The RAM is staying at around 50GB (of 64) used, so it's not that (FreeNAS uses 80% of your RAM regardless of how much it 'needs')

[f22luke]

I would focus your attention on the controllers.  Are they in JBOD mode what's their firmware version so on....    

[Vitalius]

I would focus your attention on the controllers.  Are they in JBOD mode what's their firmware version so on....    

They come in IT mode. Which basically makes the controller a bunch of extra SATA ports. Not JBOD mode.

From the FreeNAS GUI, they appear as 4 drives, and that's how they are set up. They would appear as two 8 TB drives if it were in JBOD mode.

[f22luke]

They come in IT mode. Which basically makes the controller a bunch of extra SATA ports. Not JBOD mode.

From the FreeNAS GUI, they appear as 4 drives, and that's how they are set up. They would appear as two 8 TB drives if it were in JBOD mode.

JBOD = just a bunch of disks 

 

so yes you are in JBOD mode which is good.

 

I think you have it set to just use RAID1  try changing the "mirror" to "RAIDZ1" or "RAIDZ2" 

 

http://doc.freenas.org/index.php/Hardware_Requirements#RAID_Overview

[Vitalius]

JBOD = just a bunch of disks 

 

so yes you are in JBOD mode which is good.

 

I think you have it set to just use RAID1  try changing the "mirror" to "RAIDZ1" or "RAIDZ2" 

 

http://doc.freenas.org/index.php/Hardware_Requirements#RAID_Overview

 

double-parity ZFS software solution that is similar to RAID-6

We aren't going for Parity RAID. We are going for RAID 10. 

[f22luke]

We aren't going for Parity RAID. We are going for RAID 10. 

Ah yea I just automatically think of ZFS when I'm working with freeNAS.

 

The way you should interpret the picture in your original post is that there are two logical volumes each containing to drives but those drives are not in any form of RAID.  The RAID is done on the two volumes and in this case they are just mirrors of each other.  So given this you should expect lower than typical performance as all it is doing is introducing overhead with no striping.

[Vitalius]

Ah yea I just automatically think of ZFS when I'm working with freeNAS.

 

The way you should interpret the picture in your original post is that there are two logical volumes each containing to drives but those drives are not in any form of RAID.  The RAID is done on the two volumes and in this case they are just mirrors of each other.  So given this you should expect lower than typical performance as all it is doing is introducing overhead with no striping.

Then how would I create a RAID 10 using this? 

I'd assume I would make two RAID 1's then stripe them? Or two RAID 0's then Mirror them?

[f22luke]

Then how would I create a RAID 10 using this? 

I'd assume I would make two RAID 1's then stripe them? Or two RAID 0's then Mirror them?

Yes 

 

I would think about maybe doing a hardware RAID1 and the set freeNAS to stripe.

 

Just stick with software RAID so it will have automatic healing.

[f22luke]

Then how would I create a RAID 10 using this? 

I'd assume I would make two RAID 1's then stripe them? Or two RAID 0's then Mirror them?

Is there no option to set it to software RAID10?

[f22luke]

Then how would I create a RAID 10 using this? 

I'd assume I would make two RAID 1's then stripe them? Or two RAID 0's then Mirror them?

I found this

 

"to extend a ZFS mirror, add the same number of drives. The resulting striped mirror is a RAID 10. For example, if you have 10 drives, you could start by creating a mirror of two drives, extending this mirror by creating another mirror of two drives, and repeating three more times until all 10 drives have been added."

 

The GUI is terrible and I can see why there is so much confusion here, in fact they are complaining about it over on the freeNAS forums :http://forums.freenas.org/threads/need-guide-to-set-up-raid-10-in-freenas-8-3.10652/

 

Edit: Here is a guide I found :http://www.stokecs.co.uk/techdocs/Create%20a%20RAID%2010%20Array%20Using%20FreeNAS.doc

[Vitalius]

I found this

 

"to extend a ZFS mirror, add the same number of drives. The resulting striped mirror is a RAID 10. For example, if you have 10 drives, you could start by creating a mirror of two drives, extending this mirror by creating another mirror of two drives, and repeating three more times until all 10 drives have been added."

 

The GUI is terrible and I can see why there is so much confusion here, in fact they are complaining about it over on the freeNAS forums :http://forums.freenas.org/threads/need-guide-to-set-up-raid-10-in-freenas-8-3.10652/

Thanks for the research. This should help a lot. So it's "Make a RAID 1, Then Stripe that RAID 1 to make RAID 10". Simple enough. The GUI for this part is terrible indeed. The rest of it is very intuitive, but there's no explanation for how this works aside from what you've found.

[f22luke]

Thanks for the research. This should help a lot. So it's "Make a RAID 1, Then Stripe that RAID 1 to make RAID 10". Simple enough. The GUI for this part is terrible indeed. The rest of it is very intuitive, but there's no explanation for how this works aside from what you've found.

Just to be clear, it's make 2 raid 1 arrays and then format them for software raid again and then stripe them to make RAID10.  Let us know what the numbers look like once you get it all sorted out; I am curious by nature   .

[Eric1024]

Just go into the terminal and create the pool from there. Then you will know exactly how it is set up. If performance issues persist, PM me. I've been working with a research team doing ZFS benchmarks so I know a good deal about its inner workings. I'm more of a ZFS on Linux guys and not FreeNAS, but they're more or less the same filesystem.

[alpenwasser]

Just go into the terminal and create the pool from there. Then you will know exactly how it is set up. If performance issues persist, PM me. I've been working with a research team doing ZFS benchmarks so I know a good deal about its inner workings. I'm more of a ZFS on Linux guys and not FreeNAS, but they're more or less the same filesystem.

^This. The terminal interface for ZFS is very straightforward, and that way you

will know exactly what is actually being done.

[Vitalius]

Just go into the terminal and create the pool from there. Then you will know exactly how it is set up. If performance issues persist, PM me. I've been working with a research team doing ZFS benchmarks so I know a good deal about its inner workings. I'm more of a ZFS on Linux guys and not FreeNAS, but they're more or less the same filesystem.

^This. The terminal interface for ZFS is very straightforward, and that way you

will know exactly what is actually being done.

I understand that's the best way, and that it's fairly easy. 

However, others are going to have to reproduce what I'm about to do. Less CLI = better because these people don't know anything about Linux or using a CLI in this way.

The GUI would be de facto preferred if it weren't terrible for this specific setup and was well explained. 

But, I guess I can do it in the terminal with this one, then figure the GUI out later. 

Edit:

This was posted on the FreeNAS forums by a moderator,

 

You should get far more than 25MB/sec from a RAID10 setup (which is what you have).

 

I wouldn't try to add the L2ARC or ZIL because you'll have this hybrid solution that will sometimes mask and sometimes not mask the problem. Figure out the problem with your pool performance, THEN look at adding an L2ARC or ZIL. I'd have expected that worst case you'd see 70MB/sec, but probably something in the range of 150-200MB/sec.

So apparently that's how you set up a RAID 10 in the GUI. Maybe it just doesn't work (a bug?). We'll see if the Shell side fixes it.

[Eric1024]

We'll see if the Shell side fixes it.

Keep us posted.

[Vitalius]

Keep us posted.

Will do. Thanks for checking back. 

The Hard Drives are at 10% of the test remaining (each test takes 9-10 hours). 2 have already passed. The other 2 look like they aren't going to be having issues either (SMART data says so anyway).

After reading the FreeNAS guide again, it mentions that Samba is single threaded. Me having an Opteron isn't going to help it, and I was copying the files between the server, through a switch, to my computer, and back on a Windows machine using a CIFS share. 

So the network/Samba use could have been the limiting factor. I'm going to try using the terminal to copy files and see if that changes things after I try copying from my computer (to test if it was just RAID 10 not being set up correctly). According to the moderator on the FreeNAS forums, that is the correct way to setup RAID 10 using the GUI (the screenshot I posted). 

I'm still going to try the terminal first and see if that fixes things. However, it is heavily suggested that you don't use the terminal to do things regarding pool creation because it's "behind the GUI's back" meaning that if you do things with the GUI to mess with pools you made with the terminal, bad things can happen. 

I'll rebuild the system today, but I'm not going to be doing the testing today. I have 15 minutes until I get off, and I will be getting overtime this weekend when I move everyone from our old file server to this new one, so I have to be careful about getting too much. 

Expect an update 17 hours from this posting. 

[wpirobotbuilder]

After reading the FreeNAS guide again, it mentions that Samba is single threaded. Me having an Opteron isn't going to help it, and I was copying the files between the server, through a switch, to my computer, and back on a Windows machine using a CIFS share. 

So the network/Samba use could have been the limiting factor. I'm going to try using the terminal to copy files and see if that changes things after I try copying from my computer (to test if it was just RAID 10 not being set up correctly). According to the moderator on the FreeNAS forums, that is the correct way to setup RAID 10 using the GUI (the screenshot I posted). 

Is iSCSI an option? Or is this for serving common data to everyone?

[Vitalius]

Is iSCSI an option? Or is this for serving common data to everyone?

Common data to everyone. 

It's going to house a bunch of PDF, excel, and other similar files that ~30-50 people need access to throughout the day. Odds are most of them will access the server for a file 5-10 times an hour (these are rough guesstimations).

[Vitalius]

Hmm, well that took FOREVER. 

I moved 60GB of data to the newly created drive over the network. That took about 30 minutes. Then I took that data and moved it from the parent pool to the dataset within it. That took 8 hours. Just to discover all the files. 

... And now I'm getting off in 10 minutes. So I can't do anything else today. Update postponed until tomorrow. Sorry.

[Vitalius]

Alright, I have 60GB of dummy data on the system. 

When copying it from one dataset to another within the same volume using a Windows CIF Share, it starts copying at 20 MB/s roughly, then slowly moves up to a maximum of 26MB/s then slowly falls back down to as low as 10MB/s. That's copying the entire 60GB block, which consists of many small files and a few large ones (some as big as 5GB). 

When copying it from one dataset to another using the Shell command cp, it can copy a 5.3GB file in less than 1 minute. 60 seconds in a minute, and 5,300MB means around 90 MB/s, but that's for one large file. 

That's using the GUI to make the RAID 10^. I'm going to now try making the RAID 10 with the terminal and go from there. wpirobotbuilder Eric1024

Edit: Also, I tried copying the files from a server in the same room connected to the same switch as the FreeNAS setup. It copied at around 45MB/s, which is a noticeable improvement. Strange. Very strange. 

[Eric1024]

Alright, I have 60GB of dummy data on the system. 

When copying it from one dataset to another within the same volume using a Windows CIF Share, it starts copying at 20 MB/s roughly, then slowly moves up to a maximum of 26MB/s then slowly falls back down to as low as 10MB/s. That's copying the entire 60GB block, which consists of many small files and a few large ones (some as big as 5GB). 

When copying it from one dataset to another using the Shell command cp, it can copy a 5.3GB file in less than 1 minute. 60 seconds in a minute, and 5,300MB means around 90 MB/s, but that's for one large file. 

That's using the GUI to make the RAID 10^. I'm going to now try making the RAID 10 with the terminal and go from there. wpirobotbuilder Eric1024

Edit: Also, I tried copying the files from a server in the same room connected to the same switch as the FreeNAS setup. It copied at around 45MB/s, which is a noticeable improvement. Strange. Very strange. 

Sounds like this issue isn't necessarily an issue with ZFS. Might be an issue with your network FS as well. 

 

If copy performance from one dataset to another (I'm assuming these are from the same array?) is giving you 90MB/s of throughput, then you're actually getting combined 90MB/s read and 90MB/s write performance out of the same array, which is 180MB/s of throughput, which is about what I would expect. There's the large file/small file issue still but we can work that our later. Right now we need to figure out where the bottleneck actually is.