FreeNAS read/write performance

[79wjd]

What kind of performance should I be expecting in terms of read/write speeds to/from/and within my FreeNAS server? I feel like my read/write speeds are quite a bit lower than what they should be. 

 

The most recent test I did was with a 200MB~ file and I got the following results when copying from one place to the other: 
computer -> server : 5MB/s
server -> computer: 14MB/s
server drive 1 -> server drive 2: 3MB/s

I've ran smart tests on all the drives and everything looks good, and the only other thing running is a Plex media server plugin (nothing is streaming). All the reporting graphs are basically at idle -- the cpu usage is at 0%, all the drives are sitting around 0% (except when I try writing/reading during the file test). (Other graphs in reporting section: http://imgur.com/a/s8Qx2 ). Since the performance with transfers within the server is pretty poor as well my guess is that something else is going on, but I don't know what. 

Hardware: 
Pentium G3250 
MSI H81m-p33
8gb non-ECC ram 
3x 4TB WD Reds (not in any RAID config) 
2x 1TB (WD Green, Seagate barracuda in Raid 1)

[MangoIce]

I dunno anything but ifeel you should have at least an i3

[79wjd]

I dunno anything but ifeel you should have at least an i3

The Pentium is enough to handle three 40GB (6MB/s) transcoded streams over Plex -- I'll occasionally run into some issues with three streams (two streams is never an issue), but that's probably caused by the same thing that's limiting my read/write speeds. 

[manikyath]

whats the network setup between the two devices you used for testing?

 

I dunno anything but ifeel you should have at least an i3

not at all.

[79wjd]

whats the network setup between the two devices you used for testing?

 

not at all.

Well, there is also an issue with internal transfers, so if the network is causing an issue it's a secondary one and irrelevant to the main issue. 

 

But my router is an AC68u, my freeNAS is hardwired while the other two computers I used for testing are connected via AC wireless (one is an Asus PCE-56u and the other is whatever AC card the 2013 rMBP uses -- probably something intel).

 

EDIT: What would happen if I were to shutdown the system and disconnect the two drives in Raid1 before turning it back on? How would I go about re-adding them at a later point? -- would I just have to plug them back in and then re-add the volumes in the volume manager section? 

[EmeraldFlame]

What Raid/ZFS level are you using? Raid 5/6 and it's ZFS equivalents are going to see lower write performance because of parity calculation. 

[79wjd]

What Raid/ZFS level are you using? Raid 5/6 and it's ZFS equivalents are going to see lower write performance because of parity calculation. 

The two 1tb drives are in Raid 1, and the three 4tb drives are independent. 

[EmeraldFlame]

The two 1tb drives are in Raid 1, and the three 4tb drives are independent. 

Oh... well in that case there really isn't any overhead. It sounds like this has been running for a while, so I am assuming that the raids have fully initialized? Has it always been this slow, or is this a new machine?

May want to run a stress test to make sure the CPU isn't throttling for some odd reason. Running a memtest probably wouldn't be a bad idea either. If you have ram throwing errors and having to recover all the time that will definitely slow things down.

 

[79wjd]

Oh... well in that case there really isn't any overhead. It sounds like this has been running for a while, so I am assuming that the raids have fully initialized? Has it always been this slow, or is this a new machine?

May want to run a stress test to make sure the CPU isn't throttling for some odd reason. Running a memtest probably wouldn't be a bad idea either. If you have ram throwing errors and having to recover all the time that will definitely slow things down.

 

Yeah, it's been running for about 6 months now (although I recently went through and fixed a bunch of things). 

 

As it turns out I was just being an idiot. My transfer speeds are more like 60MB/s when transferring from my server to my computer, my computer back to the server, and between two drives on the server. When I switch to wireless the transfer speeds drop off to 5MB/s~, which still seems abnormally low to me -- but I guess it's just the unfortunate nature of wifi. 

 

My only question now is can 8GB of RAM cause random issues when transcoding? Sometimes I have no issues with three 6MB/s transcoded plex streams while other times I don't. And would stepping up to 16gb of RAM improve the arc hit rate -- and does that matter (I assume that hit rates refers to what's been cached in memory).

[Liquid_Static]

Yes. I was having similar issues transcoding with my Haswell Xeon until moving up to 16gb of RAM.

[79wjd]

Yes. I was having similar issues transcoding with my Haswell Xeon until moving up to 16gb of RAM.

Interesting.

[EmeraldFlame]

When I switch to wireless the transfer speeds drop off to 5MB/s~, which still seems abnormally low to me -- but I guess it's just the unfortunate nature of wifi. 

That is pretty low, but it all depends on your wireless network. It sounds like you need a serious upgrade to your access point. Do you now what standards your router is supporting? A/B/G/N/AC? Is it 2.4 or 5GHz? What kind of signal strength are you getting? Same thing can be said for the wireless card in your computer, what is it supporting?

The AC wifi standard is really quite awesome to have and you may want to think about upgrading to it if the wireless file transfers are important to you.

[79wjd]

That is pretty low, but it all depends on your wireless network. It sounds like you need a serious upgrade to your access point. Do you now what standards your router is supporting? A/B/G/N/AC? Is it 2.4 or 5GHz? What kind of signal strength are you getting? Same thing can be said for the wireless card in your computer, what is it supporting?

The AC wifi standard is really quite awesome to have and you may want to think about upgrading to it if the wireless file transfers are important to you.

Both are basically top end (one step down) AC options (Asus RT-AC68u and Asus PCE56) connected via 5ghz. If I had to take a guess it's because of multiple cameras that are setup on my guest network (hosted on the same router) that are causing the problem. 

 

Wireless transfers aren't such a big deal, so it really doesn't matter that much. 

[EmeraldFlame]

Both are basically top end (one step down) AC options (Asus RT-AC68u and Asus PCE56) connected via 5ghz. If I had to take a guess it's because of multiple cameras that are setup on my guest network (hosted on the same router) that are causing the problem. 

 

Wireless transfers aren't such a big deal, so it really doesn't matter that much. 

Wireless security cameras I'm assuming? If that's the case they can definitely eat up some bandwidth. Try connecting your computer to the 2.4GHz band instead and see if speeds improve. If the cameras are all on 5GHz and eating up the bandwidth and time there, then switching to 2.4 should give you a clearer channel with less traffic to deal with.

 

My only question now is can 8GB of RAM cause random issues when transcoding? Sometimes I have no issues with three 6MB/s transcoded plex streams while other times I don't. And would stepping up to 16gb of RAM improve the arc hit rate -- and does that matter (I assume that hit rates refers to what's been cached in memory).

Forgot to answer this too, but someone else said it too, yes it definitely can. It depends on the bitrate of the source mainly. A higher quality source is gonna take up more space in RAM for it to down-convert. I would try and watch your RAM usage while the encodes are going just to see, but my guess is a bump to 16 would be beneficial if you are running that many plex streams.

[79wjd]

Wireless security cameras I'm assuming? If that's the case they can definitely eat up some bandwidth. Try connecting your computer to the 2.4GHz band instead and see if speeds improve. If the cameras are all on 5GHz and eating up the bandwidth and time there, then switching to 2.4 should give you a clearer channel with less traffic to deal with.

 

Forgot to answer this too, but someone else said it too, yes it definitely can. It depends on the bitrate of the source mainly. A higher quality source is gonna take up more space in RAM for it to down-convert. I would try and watch your RAM usage while the encodes are going just to see, but my guess is a bump to 16 would be beneficial if you are running that many plex streams.

I threw all the cameras on a 2.4ghz guest network while basically only I use the 5ghz network (iirc my mac, my desktop, and my Android TV are the only things connected to the 5ghz network -- all support AC wireless). 

 

I'm referring to 6-10MB/s streams as that's where I sometimes run into issues. I can't really watch the memory management since memory doesn't get freed when it's not in use anyway. 

 

I'll probably upgrade to 16gb of RAM, I just have to decide whether I'm going to spend $30 on another 8gb stick or $300 and upgrade to a board/16gb ECC memory. 

[EmeraldFlame]

I threw all the cameras on a 2.4ghz guest network while basically only I use the 5ghz network (iirc my mac, my desktop, and my Android TV are the only things connected to the 5ghz network -- all support AC wireless). 

 

I'm referring to 6-10MB/s streams as that's where I sometimes run into issues. I can't really watch the memory management since memory doesn't get freed when it's not in use anyway. 

 

I'll probably upgrade to 16gb of RAM, I just have to decide whether I'm going to spend $30 on another 8gb stick or $300 and upgrade to a board/16gb ECC memory. 

Now that is really odd, is there any other high-bandwidth devices on the 5GHz? If not, I would say you have some sort of issue somewhere. Your devices on the 2.4GHz won't cause interference or slowness on 5GHz, unless you just have poor signal strength you should be getting significant;y better speeds than that.

Also are you using the ZFS file system on any of the drives. I know earlier you said you were basically RAID 1 and JBOD, if that's the case you really don't need ECC, it's only required on FreeNAS for the ZFS file system.

[79wjd]

Now that is really odd, is there any other high-bandwidth devices on the 5GHz? If not, I would say you have some sort of issue somewhere. Your devices on the 2.4GHz won't cause interference or slowness on 5GHz, unless you just have poor signal strength you should be getting significant;y better speeds than that.

Also are you using the ZFS file system on any of the drives. I know earlier you said you were basically RAID 1 and JBOD, if that's the case you really don't need ECC, it's only required on FreeNAS for the ZFS file system.

I can't imagine what could be causing interference -- I'm getting about 20MB/s right next to the router vs. 8MB/s 30~ feet and a couple walls away. 

 

I'm using ZFS. 

[EmeraldFlame]

I can't imagine what could be causing interference.

 

I'm using ZFS. 

Hard to tell, I'm assuming you had good signal strength when you did your tests right?

With ZFS, I would definitely recommend going the ECC route. When ZFS uses ram kind of like a cache for recent read/write operations. If it ever encounters an error, it's default is to recover from RAM, so you want that to be bullet-proof so it doesn't cause errors in data.

[79wjd]

Hard to tell, I'm assuming you had good signal strength when you did your tests right?

With ZFS, I would definitely recommend going the ECC route. When ZFS uses ram kind of like a cache for recent read/write operations. If it ever encounters an error, it's default is to recover from RAM, so you want that to be bullet-proof so it doesn't cause errors in data.

I don't think I ever tested the connection other than seeing if I was able to get my full up/down speeds (50mbps). But the signal strength is good. Right next to the router I get about 20MB/s. 

 

Yeah, I know why I should go with ECC, it's just a question of whether or not I want to spend $300 to do it, and the answer is that I really don't. I'm mainly using my NAS for plex, so the worst that happens is something goes wrong and corrupts an episode of a tv show/a movie....is it worth spending $300 to avoid that? Maybe...maybe not. 

[79wjd]

Hard to tell, I'm assuming you had good signal strength when you did your tests right?

With ZFS, I would definitely recommend going the ECC route. When ZFS uses ram kind of like a cache for recent read/write operations. If it ever encounters an error, it's default is to recover from RAM, so you want that to be bullet-proof so it doesn't cause errors in data.

Wait, I was wondering something....So from my understanding ECC memory is important because it stops data in memory from getting corrupted and written back to storage. Well, most things I do on the server will be read only (aside from when I add new media to the server), so is there a a way to stop the server from storing what's in memory (and if it's possible, would that make ECC just as good as non-ECC)?