10gb Speed drops

[tommy-b-10]

Hi guys!

 

I finally got my 10gbe cards up and running and am running into an issue.

 

Setup:

One 10gb card in server, one in pc. Directly connected using sfp+ fiber.

Both are on a seperate subnet to the router to allow direct file transfers over the 10gb.

Server is running raid 5 array with 4 2tb drives, on a dedicated raid controller.

Transfer is going onto a ssd on my pc, the file is a 30GB movie.

 

Issue:

I attempted to transfer a file from server to pc, and for the first 30 seconds or so it was looking good with my speeds averaging around 160mb/s (proving its faster than gigabit), however, after 30 seconds, the speed dropped to around 30mb/s and stays that way for the rest of the transfer. Whats going on here?

 

 

 

Thanks!!

 

[leadeater]

First off check the link has a stable full 10Gb connection using iperf or a similar tool. The test shown is coping a file from the server to the desktop correct (initiated from the desktop)? Although I would expect better from a RAID 5 array it may be something like read ahead cache and once that is exhausted the speed drops to the true performance of the array, do a disk benchmark with multiple tools to confirm this. HD Tune and Crystal Disk mark would be good ones to use.

 

Also what is the OS on the server, Windows?

 

Rather than repeat a bunch of trouble shooting steps here's a link to another similar forum topic about 10Gb performance issues:

http://linustechtips.com/main/topic/505757-10gbe-network-performance-help/

[tommy-b-10]

First off check the link has a stable full 10Gb connection using iperf or a similar tool. The test shown is coping a file from the server to the desktop correct (initiated from the desktop)? Although I would expect better from a RAID 5 array it may be something like read ahead cache and once that is exhausted the speed drops to the true performance of the array, do a disk benchmark with multiple tools to confirm this. HD Tune and Crystal Disk mark would be good ones to use.

 

Also what is the OS on the server, Windows?

 

Rather than repeat a bunch of trouble shooting steps here's a link to another similar forum topic about 10Gb performance issues:

http://linustechtips.com/main/topic/505757-10gbe-network-performance-help/

 

Thanks!

 

Im not able to get iperf working on the different subnets, do you have any advice? Even binding to the correct ip using -B dosent work.

 

I did change a few of the NIC's settings as per the topic you linked and now I manage to get around 170 reads and 180 writes.

 

Running windows server 2012 R2 as server and windows 10 as pc

[leadeater]

Thanks!

 

Im not able to get iperf working on the different subnets, do you have any advice? Even binding to the correct ip using -B dosent work.

 

I did change a few of the NIC's settings as per the topic you linked and now I manage to get around 170 reads and 180 writes.

 

Running windows server 2012 R2 as server and windows 10 as pc

 

For iperf just use the server setting on the server and then on the client just specify the IP address of the server, don't need to do anything special as the OS networking layer will sort out what NIC to use and where to send the data etc.

 

Those new speeds you mentioned sound about perfect for a 4 disk RAID 5 though so issue may now be sorted, could be higher but would have to compare with local disk throughput test.

 

Edit: Turn off the Windows firewall when doing the iperf testing, likely the port is blocked

[tommy-b-10]

For iperf just use the server setting on the server and then on the client just specify the IP address of the server, don't need to do anything special as the OS networking layer will sort out what NIC to use and where to send the data etc.

Those new speeds you mentioned sound about perfect for a 4 disk RAID 5 though so issue may now be sorted, could be higher but would have to compare with local disk throughput test.

Edit: Turn off the Windows firewall when doing the iperf testing, likely the port is blocked

Cheers, I will try turning off the firewall. The issue was occurring if I specified the server ip via the 10gb subnet, if I used the gigabit up it was fine (but not 10gb speeds ) so will try firewall.

Doing a file transfer from of to server gives me 180mb/s but using crystal disk mark only gives a write speed of 15mb/s?? Seems strange to me haha.

[tommy-b-10]

For iperf just use the server setting on the server and then on the client just specify the IP address of the server, don't need to do anything special as the OS networking layer will sort out what NIC to use and where to send the data etc.

 

Those new speeds you mentioned sound about perfect for a 4 disk RAID 5 though so issue may now be sorted, could be higher but would have to compare with local disk throughput test.

 

Edit: Turn off the Windows firewall when doing the iperf testing, likely the port is blocked

 

Turns out iperfs issue was that the 10gb connection was showing up as a public connection and the firewall was blocking it So I managed to get that working.

 

Here is a screenshot of the results: