New FreeNAS Build (Slow Throughput & Low Transfer Speeds)

[NATYPE]

Hello all!

 

Just finished building my first FreeNAS box and I've run into quite a few problems. I've tried a few different versions of FreeNAS (Though I'm back on 9.3 now) and I'm getting consistently awful throughput. I'm sitting at around 11-13 Megabytes per second when both reading and writing from the server (I'll list the specs below so you can see what I'm working with). Now, I ran i test using iperf/jperf from my desktop to the server and it's giving me 13 MBps as my result. I'm fairly new to this, but I believe that it's only testing my highest possible throughput through the network right? Not actually testing the drive pool's speed? Either way, that seems oddly low. Over wireless with my laptop it's less than half of that.

 

Connection test with jperf to my server: http://puu.sh/fhBWb/f3e842b14c.png

 

FreeNAS Box Specs:

 

OS: FreeNAS 9.3 (Stable Release)

CPU: AMD Phenom X6 1090T

MOBO: ASUS Sabertooth 990FX Rev.1

RAM: 8Gb Mushkin Silverline 1333Mhz (Yes, I know I need more of it )

NIC: Realtek 8111E (Onboard NIC) 

PSU: EVGA SuperNOVA 650G1

HDD's: 5x3TB WD Red's in RaidZ

 

I just have a CIFS share setup which is how I'm accessing the pool through Windows. Also, and this could be important too, but I have the server upstairs next to my desktop. Both are connected to a WD Livewire (Powerline Kit) which is connected to my router. I've never had any issues with it before, especially with speed. I also connected the server directly to the router and performance was actually a bit worse, though it improved over wireless which I thought was interesting.

 

Any help would be greatly appreciated! I'd really like to get this up and running before the weekend it over!

 

If there's anything I need to give you all to help with the troubleshooting process please let me know! I'll do my best to get it!

 

 

Update 1: Still having issues with throughput to the server. Even when I connected my laptop and the FreeNAS Box directly to my router, the connection speed was still awful. With iperf/jperf I was only getting about 5 Megabytes per second. The router (Linksys EA6900) has gigabit ports on the back and I used Cat6 cable to connect everything up.

 

Update 2: Okay! So it looks like it has to be something with the way my local network is configured. I just found an old D-Link gigabit switch and connected my desktop and the FreeNAS server to it. Then connected that to my powerline kit. Much, much better! 57-58 Megabytes per second now!

 

jperf test: http://puu.sh/fi27R/fe28bd3817.png

 

So now I can at least use it with my desktop at much more reasonable speeds. 

 

I'm still very curious as to where the problem lies on my local network though. It seems very odd.

 

Update 3: Looks like everything is working great with the server now! I'm consistently reading and writing to the pool at over 100 MB/s! 

 

Seems like all the issues were being caused by my router. I honestly still don't know why, but I do know it has nothing to do with my FreeNAS Box anymore. I'll continue looking into it since my current setup isn't ideal, though I'm very excited to finally have everything working!

 

Thank you all for your help today! 

[Profquast]

Are you sure it's not 100mbit ethernet? 100mbit is 100/8=12,5MBps, so could well be possible. Also, '300N' is most of the time more like 30N.

[NATYPE]

Are you sure it's not 100mbit ethernet? 100mbit is 100/8=12,5MBps, so could well be possible. Also, '300N' is most of the time more like 30N.

 

For the speed, are you referring to the LAN on the Motherboard? If so, the product page says it's gigabit.

 

If you're talking about the powerline kit, here's the product page for it: http://www.newegg.com/Product/Product.aspx?Item=9SIA0AJ0UV7861

 

And, I apologize, I'm not sure what you mean when you say "300N". 

[Godlygamer23]

I tried FreeNAS 9.3, but it bricked my damned flash drive's memory block.

[NATYPE]

I tried FreeNAS 9.3, but it bricked my damned flash drive's memory block.

 

Really? How did that happen? If you don't mind my asking of course.

[MrBucket101]

@Nathn 

 

Are you using megabit or gigabit ethernet? Your NAS has a gigabit NIC, but does your router, or that powerline thing?

 

RAID 5 suffers from slow write speeds to begin with, so that probably isn't helping you very much.

 

You should probably move up to Raid 6. Chances of a bad sector on another disk when rebuilding are pretty high at that size.

[Kyuubixchidori]

Are you sure it's not 100mbit ethernet? 100mbit is 100/8=12,5MBps, so could well be possible. Also, '300N' is most of the time more like 30N.

second this. I set up a nas4free server, and the switchs in my house are 100mb, need to upgrade them. its running at 11-12.5mb right now

[NATYPE]

@Nathn 

 

Are you using megabit or gigabit ethernet? Your NAS has a gigabit NIC, but does your router, or that powerline thing?

 

RAID 5 suffers from slow write speeds to begin with, so that probably isn't helping you very much.

 

You should probably move up to Raid 6. Chances of a bad sector on another disk when rebuilding are pretty high at that size.

 

I would assume I'm full gigabit, though the powerline kit is a bit old so I'll look into it a bit more. All the cabling I'm using is Cat6/5e, the NIC's in my desktop and NAS are both gigabit capable, and the router is a Linksys EA6900 which I'm sure has gigabit ports on it.

 

I really do like the total capacity I'm getting with RaidZ though, and I understand that the writes will suffer. But I could always switch to RaidZ2 right? Isn't that essentially RAID 6?

[NATYPE]

second this. I set up a nas4free server, and the switchs in my house are 100mb, need to upgrade them. its running at 11-12.5mb right now

 

Okay! Good to know. I'll hook up my NAS and my laptop directly to the router so I can test speeds without the powerline kit.

[Godlygamer23]

Really? How did that happen? If you don't mind my asking of course.

No clue. The installation "worked" but every time I booted into FreeNAS, I would get an error. I reinstalled it numerous times. I even went back to FreeNAS 9.2. Same issue. I bought another flash drive and put FreeNAS 9.2 on it again. It totally worked. After that whole fiasco, I decided to copy a large enough file that would match FreeNAS. The transfer got to the half way point and froze.

[NATYPE]

No clue. The installation "worked" but every time I booted into FreeNAS, I would get an error. I reinstalled it numerous times. I even went back to FreeNAS 9.2. Same issue. I bought another flash drive and put FreeNAS 9.2 on it again. It totally worked. After that whole fiasco, I decided to copy a large enough file that would match FreeNAS. The transfer got to the half way point and froze.

 

Wow, that's really interesting. I'm assuming it was the flash drive at fault though, right? Had you installed FreeNAS from it before, successfully?

[flibberdipper]

Are you sure your network can handle it?

[Godlygamer23]

Wow, that's really interesting. I'm assuming it was the flash drive at fault though, right? Had you installed FreeNAS from it before, successfully?

Yup. It ran totally fine. The flash drive was the issue, but FreeNAS ran totally fine on that flash drive for a month. It was when I decided to update to 9.3 that I experienced those issues.

[MrBucket101]

I would assume I'm full gigabit, though the powerline kit is a bit old so I'll look into it a bit more. All the cabling I'm using is Cat6/5e, the NIC's in my desktop and NAS are both gigabit capable, and the router is a Linksys EA6900 which I'm sure has gigabit ports on it.

 

I really do like the total capacity I'm getting with RaidZ though, and I understand that the writes will suffer. But I could always switch to RaidZ2 right? Isn't that essentially RAID 6?

Yeah everything is gig-e except for that powerline kit. I have Cat5e all over my house cause I'm cheap, but it's capable of carrying gig-e speeds. So it's not your cables

 

Also, RaidZ2 = Raid6, so yeah. You can go read up on it it's just a problem with RAID 5 and high capacity drives. When rebuilding an array, after a drive has failed, a bad sector on another drive in the set could mean a failed rebuild. Then you've lost 1 drive and all your data. The physical number of sectors at 12TB+ just makes the probability of a bad sector located elsewhere pretty high. Lot of people run RAID 5 at high capacities, I just like to mention the risk they're taking

[NATYPE]

Are you sure your network can handle it?

 

I assume so! But it's really looking like my powerline kit is going to be what's bottlenecking the entire setup. I'm testing it now!

[NATYPE]

Yeah everything is gig-e except for that powerline kit. I have Cat5e all over my house cause I'm cheap, but it's capable of carrying gig-e speeds. So it's not your cables

 

Also, RaidZ2 = Raid6, so yeah. You can go read up on it it's just a problem with RAID 5 and high capacity drives. When rebuilding an array, after a drive has failed, a bad sector on another drive in the set could mean a failed rebuild. Then you've lost 1 drive and all your data. The physical number of sectors at 12TB+ just makes the probability of a bad sector located elsewhere pretty high. Lot of people run RAID 5 at high capacities, I just like to mention the risk they're taking

 

Yeah, I understand. I read up on all of that a bit before building this. I would definitely considering switching to RaidZ2 but performance is pretty important to me. Any idea on what kind of performance hits I might see?

 

I read a few horror stories on r/DataHoarder about failed rebuilds with a RAID 5 array due to another drive dying from the stress of the rebuild. I can't even imagine the frustration!

[NATYPE]

UPDATE!

 

Still having issues with throughput to the server. Even when I connected my laptop and the FreeNAS Box directly to my router, the connection speed was still awful. With iperf/jperf I was only getting about 5 Megabytes per second. The router (Linksys EA6900) has gigabit ports on the back and I used Cat6 cable to connect everything up.

 

What should I do now?

[MrBucket101]

Yeah, I understand. I read up on all of that a bit before building this. I would definitely considering switching to RaidZ2 but performance is pretty important to me. Any idea on what kind of performance hits I might see?

 

I read a few horror stories on r/DataHoarder about failed rebuilds with a RAID 5 array due to another drive dying from the stress of the rebuild. I can't even imagine the frustration!

RAID 6, compared to RAID 5, has no performance differences. The only difference between the 2, is that you sacrifice a second drive for parity, instead of just 1. The 2nd drive gives you room for up to 2 drives to fail without data loss. RAID 6 still has the same penalty to writes that RAID 5 has.

 

 

 

UPDATE!

 

Still having issues with throughput to the server. Even when I connected my laptop and the FreeNAS Box directly to my router, the connection speed was still awful. With iperf/jperf I was only getting about 5 Megabytes per second. The router (Linksys EA6900) has gigabit ports on the back and I used Cat6 cable to connect everything up.

 

What should I do now?

 

I'm not sure if this is entirely relevant, but doesn't RaidZ need 1GB of ECC memory per TB of storage? I don't beleive that would impact your speed at all. But maybe???

 

What kind of laptop do you have? Does it have gig-e?

One of my buddies made this mistake a while ago. He had a 12 disk raid 6 setup with 2TB drives. He was using new 2TB drives with 4k sector sizes. He found an old 2TB drive laying around his house and tossed it in his array and expanded. After the expansion he was getting HORRENDOUS speeds. Turns out the drive he put in the array had 512byte sector sizes. He removed the drive and replaced it with a 4k sector count and his problem went away.

 

With you using all new WD Reds, I don't beleive that this affects you in anyway..but as it is, I'm kinda out of ideas

[NATYPE]

UPDATE 2!

 

Okay! So it looks like it has to be something with the way my local network is configured. I just found an old D-Link gigabit switch and connected my desktop and the FreeNAS server to it. Then connected that to my powerline kit. Much, much better! 57-58 Megabytes per second now!

 

jperf test: http://puu.sh/fi27R/fe28bd3817.png

 

So now I can at least use it with my desktop at much more reasonable speeds.

 

I'm still very curious as to where the problem lies on my local network though. It seems very odd.

[NATYPE]

RAID 6, compared to RAID 5, has no performance differences. The only difference between the 2, is that you sacrifice a second drive for parity, instead of just 1. The 2nd drive gives you room for up to 2 drives to fail without data loss. RAID 6 still has the same penalty to writes that RAID 5 has.

 

 

I'm not sure if this is entirely relevant, but doesn't RaidZ need 1GB of ECC memory per TB of storage? I don't beleive that would impact your speed at all. But maybe???

 

What kind of laptop do you have? Does it have gig-e?

One of my buddies made this mistake a while ago. He had a 12 disk raid 6 setup with 2TB drives. He was using new 2TB drives with 4k sector sizes. He found an old 2TB drive laying around his house and tossed it in his array and expanded. After the expansion he was getting HORRENDOUS speeds. Turns out the drive he put in the array had 512byte sector sizes. He removed the drive and replaced it with a 4k sector count and his problem went away.

 

With you using all new WD Reds, I don't beleive that this affects you in anyway..but as it is, I'm kinda out of ideas

 

Okay, interesting! I was always under the impressions that RAID 6 performed a bit worse than RAID 5 due to the extra parity drive. If that's not the case then I see no reason not to switch to RaidZ2 as soon as I get everything figured out.

 

I believe the ideal setup is 1Gb of RAM for each TB of storage? And if it's ECC even better. I've heard of it impacting performance but only when you're under the recommended amount. I could be wrong of course.

 

My laptop does have a gigabit LAN port, I double checked though since you brought it up. All good there!

 

Based on what I just tested, and since all of the drives are brand new I don't think I'm running into an issues with sector sizes. However, that's very interesting and I'm happy you brought it up!

[MrBucket101]

Okay, interesting! I was always under the impressions that RAID 6 performed a bit worse than RAID 5 due to the extra parity drive. If that's not the case then I see no reason not to switch to RaidZ2 as soon as I get everything figured out.

 

I believe the ideal setup is 1Gb of RAM for each TB of storage? And if it's ECC even better. I've heard of it impacting performance but only when you're under the recommended amount. I could be wrong of course.

 

My laptop does have a gigabit LAN port, I double checked though since you brought it up. All good there!

 

Based on what I just tested, and since all of the drives are brand new I don't think I'm running into an issues with sector sizes. However, that's very interesting and I'm happy you brought it up!

 

Some routers let you configure at which speed the ports will run at, maybe that's it.

 

57-58 MB/s is definitely an improvement, but I would have liked to see those numbers in the 80-90MB/s range. With you running Software RAID, the CPU obviously needs to do a lot more calculations before storing the data, but with 6 cores I would have thought it to have performed a lot better.

With my server, I'm able to get the maximum speed from gig-e, ~120MB/s read/write...But I'm running hwRD and I also built the array with maximum speed in mind. Have a look in my sig if you want some more info.

[NATYPE]

Some routers let you configure at which speed the ports will run at, maybe that's it.

 

57-58 MB/s is definitely an improvement, but I would have liked to see those numbers in the 80-90MB/s range. With you running Software RAID, the CPU obviously needs to do a lot more calculations before storing the data, but with 6 cores I would have thought it to have performed a lot better.

With my server, I'm able to get the maximum speed from gig-e, ~120MB/s read/write...But I'm running hwRD and I also built the array with maximum speed in mind. Have a look in my sig if you want some more info.

 

I'll definitely look into the settings on my router next since it seems like that's where a lot of problems are coming from.

 

Also, as far as speed go, my real-world performance seems a lot better! Transferring media back and forth from an SSD I'm seeing closer to 70-80 MB/s with it peaking just above 100 MB/s from time to time. Are your numbers from iperf or what you're actually seeing?

[MrBucket101]

I'll definitely look into the settings on my router next since it seems like that's where a lot of problems are coming from.

 

Also, as far as speed go, my real-world performance seems a lot better! Transferring media back and forth from an SSD I'm seeing closer to 70-80 MB/s with it peaking just above 100 MB/s from time to time. Are your numbers from iperf or what you're actually seeing?

what I'm actually seeing

never heard of iperf tbh

[NATYPE]

what I'm actually seeing

never heard of iperf tbh

 

Okay, thank you! That's good to know.

[NATYPE]

UPDATE 3!

 

Looks like everything is working great with the server now! I'm consistently reading and writing to the pool at over 100 MB/s!

 

Seems like all the issues were being caused by my router. I honestly still don't know why, but I do know it has nothing to do with my FreeNAS Box anymore. I'll continue looking into it since my current setup isn't ideal, though I'm very excited to finally have everything working!

 

Thank you all for your help today!