Jump to content

10GbE network performance help

Strikermed
By Strikermed
in Networking
 Share
Posted

Hello all,

 

I wanted to log my successes and failures with setting up a direct connect 10GbE network between a "server" PC and my Workstation PC.

 

First I want to start with my original equipment.

 

Workstation PC:

intel i7 3930K (six Core, hyper threaded = 12 logical cores)

Asus Sabertooth x79 rev2 motherboard

intel x540t1 10GbE network interface card

GTX 580

32GB DDR 3 Memory Corsair

Samgung 840Pro 1TB SSD

120GB ssd

1TB Western Digital Black

2x 2TB Western Digital Black in RAID 0

Corsair Obsidian 750D case

Corsair 1000Watt power supply

 

"Server" PC

AMD Phenom 2 X4 Quad Core (no hyper threading, so 4 logical cores) [Replaced with intel i7 3930k]

Asus M5A99FX PRO r2 [Replaced with P9X79 PRO]

32GB DDR3 Memory

Scandisk Extreme Pro 240GB SSD

intel 540x-t1 10GbE Network interface card

LSI MegaRAID 9266 8i RAID controller

xfx 750 watt powersupply

Roswill 12 bay 4u server case

2x 2TB Western Digital Black

 

Now to the story.

 

I started with the typical install of all the parts.  I have my existing workstation with Windows 10 (upgraded from windows 7), and I installed Server 2012 R2 Essentials on the "Server" PC.

 

My first challenge was configuring the two cards to communicate with each other in a way that I could easily understand.  They auto detect each other and connect automatically, but I wanted to be able to map drives and everything without any trouble, so I continued to edit the IPV4 configuration so that each would have a static IP (different from my on board 1GbE domain and DNS).  I chose 192.168.2.1 for my "Server" PC and I chose 192.168.2.2 for my Workstation.  I let the subnet auto populate, and I chose the "Server" computer to be the DNS, so I entered 192.168.2.1 into both the "Server" PC and the Workstation.

 

Now that that is talking to each other in an easy way to remember, I can start testing, and configuring the intel x540 t1 NIC in both the server and PC.  First issue I ran into was communicating and getting iperf to even register the two.  After hours of fiddling, I found that the firewall built into Server 2012 R2 Essentials waspreventing me from getting any throughput.  I disabled the firewall and could get a connection. 

 

Now that I know I'm getting throughput It's time to test the connection speed with a file transfer.  I utilized SoftPerfect's Ram Drive to create 10GB ram disks on both my "Server" PC and my workstation to do direct transfers via shared drives.  This way my hardware wouldn't be the bottleneck and I could truly test the 10GbE interface speeds.

 

I found I was getting sub par speeds (200MB/s with 95% CPU utilization).  After a long time fiddling, I got some advice via this forum, and Cinevate's blog, that says to disable the Security policies for encryption.  Me being a newbie to Server 2012, I didn't realize how sophisticated and out of my league this OS is.  I continued testing.  I adjusted the intel x540 t1 settings on both the "Server" PC and the Workstation: Turned on Jumbo packets to their largest setting, maxed the transmit and receive buffers, and set my RSS Cueues to match the number of logical cores (4 on the "Server" PC and 8 being the only option without going over the logical 12 core count on my Workstation)  I was able to break the 200MB/s barrier, but ran into a 500MB/s Cap.  I had 50% CPU usage on my "Server" PC and less than 15% on my Workstation.

 

With a lot of trial and error and literally writing down the results I got with different configurations, I came to a conclusion that my bottleneck was actually my CPU in my "Server"PC.  With only 4 cores I'm limiting the number of streams of data I can send down the pipe, preventing me from reaching maximum transfer speeds.

 

My solution was to find a better processor and motherboard.  I was fortunate to track down the exact same chipset motherboard and six core processor that my Workstation has.  I installed the new hardware, and it was off to the masses!  I locked in and configured the hardware, and presto!  I reached 900MB/s transfers instantly!  I did some more tweaking and I could get 1.1 GB/s with no problem.

 

Now I have been putting quotations around "Server" PC this entire article due to the fact that I wanted something more user friendly than a Server OS.  So I installed a copy of Windows 10 on my "Server" PC.  I ran into a couple hicups that were Microsoft issues with SSD's and high utilization, which I eventually just swapped out my SSD with another and for the most part has solved my problem, but the NIC configuration has changed, and the results haven't changed.  Everything works flawlessly. 

 

Right now I'm in the process of ensuring stability with Windows 10 as it often times gives me high utilization with my OS SSD, which eventually calms.  I'm suspecting Windows updates, since I noticed it was downloading automatically.  I'm also testing RAID operations. 

 

I had a RAID 0 in the Workstation which I removed, and I had a RAID 0 set up on my LSI Card.  I ran into some trouble which I hope this will solve for many.  I needed to use the diskpart command to clean the drives to remove any data relating to a RAID on them.  You can google search how to use the commands, it's very easy, just be careful to select the proper disk when using it.  I was able to build a RAID 0 with 4 WD 2TB Black and I got 600-750MB transfers between the RAID0 on my "Server" PC and a RAM Drive on my workstation.  From here I'm going to test how image editing and Video editing work with a h264, CDNG, and RAW workflow with Adobe Products. 

 

My End Goal:

My end goal is to utilize the server case for 8 Bays of RAID 6 storage for Redundancy and large backup utilizing WD RE or Red 4TB drives, and install another LSI Mega RAID 9266 4i card to create a RAID0 with my 4x 2TB WD Black drives for a working platform.

 

 

Now I did leave out a lot of my troubleshooting, but this is just a general guide to what limited me in my pursuit for true 10GbE performance.  Server 2012 R2 had security issues that blocked or created too much overhead for my file transfers, and my CPU was my bottleneck in the end.  So keep that in mind when you are looking to build something capable of 10GbE transfers.  Consider your Logical Core count.  You could probably get away with a quad core hyperthreaded processor and get 1GB/s Transfers, but if you have a Dual 10GbE NIC you will need twice that to handle the through put, mainly the Receive side scaling (RSS) lanes. 

 

I'm relieved with my setup, but it came at a cost.

intel x540-t1 NIC came in at $300 each

I got lucky with the mother board and CPU getting those for $400, but I could probably find a cheaper or adequate option for that price.

The RAID controller was $500 plus drives (Which I can verify is the best option, especially if you're hardware needs to be changed.  I went from one motherboard to another without loosing my RAID0 and worked perfectly in the new/revised setup)

 

 

 

 

 

 

ORIGINAL Problem

 

Hello,

 

I am direct connecting windows server 2012 R2 Essentials with a Windows 10 PC. I'm having trouble with transfer speeds. I'm finding my cap at about 200MB/s speeds (2.0 roughly gigabits). enabling jumbo frames and all other options that Linus and the cinevate blog covers, I'm only getting 300MB/s speeds (2.8 roughly gigabits). I'm getting pour performance and I've spent 2 days trying to discover the issue. If anyone has any input, I would truly appreciate it.

 

I'm utilizing 2 intel x540T1 NIC.  To give you an idea of my network configuration.  My Server NIC is 192.168.2.1 and my Windows 10 PC is 192.168.2.2.  My 1GbE network is on the default gateway 192.168.1.1.  I'm pretty sure I have this portion of it configured correctly, as I can transfer files back and forth between RAM drives, and I'm able to get a reading with ntttcp.  ntttcp gave me a throughput of 1128MB/s and when I turned on Jumbo frames it gave me a reading of 1048MB/s.  So the speed is there, I'm just wondering what the bottleneck is, or if I need to do some more configuration.

 

Thanks in advance.

Windows Server 2012 R2 Essentials:
AMD Phenom x4 955
Asus M5a99FX Pro R2.0
32GB RAM
Intel x540T1 in the main PCI-Express 16x slot (Blue)(removed the video card and remote access)
LSI MegaRAID 9266-8i in the second PCI-Express 16x Slot (Blue)
Configured 20GB of RAM into a RAM Drive to test transfers
Note: I get 80% CPU utilization when I get 300MB/s Transfers

Windows 10 PC
Intel i7 3930 six core
Sabertooth x79 motherboard
32GB RAM
GTX 580 in first PCI Express 16x slot
Intel x540T1 in the second PCI-Express 16x slot
Configured 20GB of RAM into a RAM Drive to test transfers
NOTE: I get 4% CPU utilization when I get 300MB/s Transfers

If anyone can detect my bottleneck, or have an info on configuration, I'm all ears as I have tried and tried with this set up. I'm hoping to utilize my server as a mass storage to work off of for video content. I would really like to saturate the RAID that I have in mind for the future, but first i need to prove that I can get atleast 750 MB/s or even 1GB/s speeds.

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

-snip-

 

Interesting that you get full speed with the synthetic network test but can't coping to and from a ram disk. The high cpu sounds like TCP offload etc is turned off, this will have a huge impact on performance. Also make sure the nic drivers are up to date and all the Intel utilities are installed for it.

 

I have almost the exact same setup, two Intel X540-T1 directly connected between my server and desktop. However I use ESXi and Intel SR-IOV to spawn virtual nics and passthrough  a virtual nic to my storage VM. I get full transfer speeds with SMB3 and iSCSI. I'm also coping too and from arrays of SSDs.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Lead eater, thanks for the quick reply.

A few follow ups: I have the most recent drivers installed. As for the Intel utilities, which utilities are you talking about? This may be something i haven't discovered yet.

Also I'll double check offloading, but I'm pretty sure it's enabled.

My follow up questions are how do I verify which SMB protocol I am currently utilizing, and iSCSI? I had an option when installing the driver in my server to install some kind of iSCSI utility I believe. Forgive me, but I'm very new to networking beyond your simple plug and play 1GbE. I'm absorbing as much as I can as fast as I can.

I guess what I'm asking is, what is the simplest way to utilize the full throughput of my NIC's

Thanks!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

-snip-

 

Likely best to ignore the iSCSI stuff, you probably not going to use it. As for the utility I just checked my Win 10 desktop and realised that the utility is no longer needed, don;t have it installed either. Was required in old OS for setting up nic teaming etc, can't do that on single port cards anyway.

 

Good place to start would be to post the output of the following powershell commands, will link to source at bottom.

 

 

5.1. Verifying if you’re using SMB Multichannel

 

You can use the following steps to verify you are using SMB Multichannel.

Step 1: Verify network adapter configuration

Use the following PowerShell cmdlets to verify you have multiple NICs and/or to verify the RSS and RDMA capabilities of the NICs. Run on both the SMB server and the SMB client.

Get-NetAdapter

Get-NetAdapterRSS

Get-NetAdapterRDMA

Get-NetAdapterHardwareInfo

 

Step 2: Verify SMB configuration

Use the following PowerShell cmdlets to make sure SMB Multichannel is enabled, confirm the NICs are being properly recognized by SMB and that their RSS and RDMA capabilities are being properly identified.

On the SMB client, run the following PowerShell cmdlets:

Get-SmbClientConfiguration | Select EnableMultichannel

Get-SmbClientNetworkInterface

On the SMB server, run the following PowerShell cmdlets:

Get-SmbServerConfiguration | Select EnableMultichannel

Get-SmbServerNetworkInterface

 

Step 3: Verify the SMB connection

On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right version of SMB and that SMB Multichannel is working:

Get-SmbConnection

Get-SmbMultichannelConnection

Get-SmbMultichannelConnection -IncludeNotSelected

 

Source: http://blogs.technet.com/b/josebda/archive/2012/05/13/the-basics-of-smb-multichannel-a-feature-of-windows-server-2012-and-smb-3-0.aspx

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

My output for reference on desktop:

 

 

PS C:\Windows\system32> Get-NetAdapter

 
Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Ethernet                  Intel® 82579V Gigabit Network Conn...      11 Up           E0-3F-49-6F-52-2E         1 Gbps
Ethernet 2                Intel® Ethernet Converged Network ...       4 Up           A0-36-9F-52-E7-AA        10 Gbps
Wi-Fi                     Broadcom 802.11ac Network Adapter             6 Disconnected 54-27-1E-6A-B7-11          0 bps
Bluetooth Network Conn... Bluetooth Device (Personal Area Netw...       3 Disconnected 24-0A-64-EC-5A-6C         3 Mbps
 
 
PS C:\Windows\system32> Get-NetAdapterRSS
 
 
Name                                            : Ethernet
InterfaceDescription                            : Intel® 82579V Gigabit Network Connection
Enabled                                         : True
NumberOfReceiveQueues                           : 1
Profile                                         : Closest
BaseProcessor: [Group:Number]                   : 0:0
MaxProcessor: [Group:Number]                    : 0:10
MaxProcessors                                   : 4
RssProcessorArray: [Group:Number/NUMA Distance] : 0:0/0  0:2/0  0:4/0  0:6/0  0:8/0  0:10/0
IndirectionTable: [Group:Number]                :
 
Name                                            : Ethernet 2
InterfaceDescription                            : Intel® Ethernet Converged Network Adapter X540-T1
Enabled                                         : True
NumberOfReceiveQueues                           : 6
Profile                                         : NUMAStatic
BaseProcessor: [Group:Number]                   : 0:0
MaxProcessor: [Group:Number]                    : 0:10
MaxProcessors                                   : 6
RssProcessorArray: [Group:Number/NUMA Distance] : 0:0/0  0:2/0  0:4/0  0:6/0  0:8/0  0:10/0
IndirectionTable: [Group:Number]                : 0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
 
 
 
PS C:\Windows\system32> Get-NetAdapterRDMA
PS C:\Windows\system32> Get-NetAdapterHardwareInfo
 
Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
----                           ------- --- ------ -------- ---- -------- ------------- ------------- -------
Ethernet                             0   0     25        0                     Unknown
Ethernet 2                           0   2      0        0                    5.0 GT/s             8 1.1
Wi-Fi                                0  13      0        0                    2.5 GT/s             1 1.1
 
PS C:\Windows\system32> Get-SmbClientConfiguration | Select EnableMultichannel
 
EnableMultichannel
------------------
              True
 
 
PS C:\Windows\system32> Get-SmbClientNetworkInterface
 
Interface Index RSS Capable RDMA Capable Speed   IpAddresses                                     Friendly Name
--------------- ----------- ------------ -----   -----------                                     -------------
4               True        False        10 Gbps {fe80::15a4:b76b:2ffd:7e4b, 10.1.1.2, 10.1.1.5} Ethernet 2
6               False       False        0  bps  {fe80::cd7d:37c2:ab04:e6}                       Wi-Fi
8               False       False        0  bps  {fe80::5dee:bd31:5a2:f9df}                      Local Area Connecti...
10              False       False        0  bps  {fe80::5efe:192.168.10.18}                      isatap.{FAC9AF8C-C1...
7               False       False        0  bps  {fe80::5efe:10.1.1.2, fe80::5efe:10.1.1.5}      isatap.{31F11FE0-5D...
11              False       False        1 Gbps  {fe80::69ae:75d:8272:af8f, 192.168.10.18}       Ethernet
3               False       False        0  bps  {fe80::d86a:2f46:2adb:5943}                     Bluetooth Network C...
 
 
PS C:\Windows\system32> Get-SmbConnection
 
ServerName ShareName UserName            Credential     Dialect NumOpens
---------- --------- --------            ----------     ------- --------
10.1.1.1   d$        <Removed>\<Removed> <Removed>\<Removed> 3.0.2   1
 
 
PS C:\Windows\system32> Get-SmbMultichannelConnection
 
Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ -------------------
10.1.1.1    True     10.1.1.2  10.1.1.1  4                      14                     False              False
 
 
PS C:\Windows\system32> Get-SmbMultichannelConnection -IncludeNotSelected
 
Server Name Selected Client IP     Server IP     Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- ---------     ---------     ---------------------- ---------------------- ------------------ -------------------
10.1.1.1    True     10.1.1.2      10.1.1.1      4                      14                     False              False
10.1.1.1    False    192.168.10.18 192.168.10.17 11                     12                     False              False
10.1.1.1    False    10.1.1.5      10.1.1.1      4                      14                     False              False
10.1.1.1    False    192.168.10.18 172.16.6.1    11                     32                     False              False
10.1.1.1    False    192.168.10.18 172.16.5.1    11                     28                     False              False
10.1.1.1    False    192.168.10.18 10.1.2.1      11                     14                     False              False
10.1.1.1    False    192.168.10.18 10.1.1.1      11                     14                     False              False
 
Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Great!  I'll try these commands and post them in the next day or so.  I unfortunately may not have much time to work on this tonight :(.

 

A quick question on running the commands, as I'm pretty new to comand prompt as well.  Do I need to run these commands as an Administrator, and is powershell very similar to Command prompt?

 

Thanks!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Great!  I'll try these commands and post them in the next day or so.  I unfortunately may not have much time to work on this tonight :(.

 

A quick question on running the commands, as I'm pretty new to comand prompt as well.  Do I need to run these commands as an Administrator, and is powershell very similar to Command prompt?

 

Thanks!

 

Yea some require admin so just run powershell as administrator to avoid problems. Yes powershell is similar to command prompt, way more advanced but cmd commands also work in powershell.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Ok here are my results.  For the copy session I used 3 large video files equaling about 18GB.  I included a note on the transfer speeds.  It's very consistent with, but different depending on which direction you are going.  I used RAM Disks to conduct the transfer.  Server to PC I was getting 300MB/s average.  PC to Server I was getting 300MB/s until the last file which was 8GB (not the largest file) where it shot up to 500MB/s.  These are average maxes rounding to the nearest 10.  I ran the Final 3 in each direction, because I'm not sure if it accounts whether you are copying one direction or the other, so forgive me if I over did it there.

 

Let me know what you think of the results. 

 

 

PC Results:

Windows PowerShell
Copyright © 2015 Microsoft Corporation. All rights reserved.

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Local Area Connection     Intel® 82579V Gigabit Network Conn...       9 Up           74-D0-2B-2B-D0-6A         1 Gbps
Ethernet                  Intel® Ethernet Converged Network ...      11 Up           A0-36-9F-75-44-82        10 Gbps


PS C:\WINDOWS\system32> Get-NetAdapterRSS


Name                                            : Local Area Connection
InterfaceDescription                            : Intel® 82579V Gigabit Network Connection
Enabled                                         : True
NumberOfReceiveQueues                           : 2
Profile                                         : NUMAStatic
BaseProcessor: [Group:Number]                   : 0:0
MaxProcessor: [Group:Number]                    : 0:10
MaxProcessors                                   : 4
RssProcessorArray: [Group:Number/NUMA Distance] : 0:0/0  0:2/0  0:4/0  0:6/0  0:8/0  0:10/0
IndirectionTable: [Group:Number]                :

Name                                            : Ethernet
InterfaceDescription                            : Intel® Ethernet Converged Network Adapter X540-T1
Enabled                                         : True
NumberOfReceiveQueues                           : 6
Profile                                         : NUMAStatic
BaseProcessor: [Group:Number]                   : 0:0
MaxProcessor: [Group:Number]                    : 0:10
MaxProcessors                                   : 6
RssProcessorArray: [Group:Number/NUMA Distance] : 0:0/0  0:2/0  0:4/0  0:6/0  0:8/0  0:10/0
IndirectionTable: [Group:Number]                : 0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2
                                                  0:4    0:6    0:8    0:10    0:0    0:2    0:4    0:6
                                                  0:8    0:10    0:0    0:2    0:4    0:6    0:8    0:10
                                                  0:0    0:2    0:4    0:6    0:8    0:10    0:0    0:2



PS C:\WINDOWS\system32> Get-NetAdapterRDMA
PS C:\WINDOWS\system32> Get-NetAdapterHardwareInfo

Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
----                           ------- --- ------ -------- ---- -------- ------------- ------------- -------
Local Area Connection                0   0     25        0                     Unknown
Ethernet                             0   2      0        0                    5.0 GT/s             8 1.1


PS C:\WINDOWS\system32> Get-SmbClientConfiguration


ConnectionCountPerRssNetworkInterface : 4
DirectoryCacheEntriesMax              : 16
DirectoryCacheEntrySizeMax            : 65536
DirectoryCacheLifetime                : 10
DormantFileLimit                      : 1023
EnableBandwidthThrottling             : True
EnableByteRangeLockingOnReadOnlyFiles : True
EnableInsecureGuestLogons             : True
EnableLargeMtu                        : True
EnableLoadBalanceScaleOut             : True
EnableMultiChannel                    : True
EnableSecuritySignature               : True
ExtendedSessionTimeout                : 1000
FileInfoCacheEntriesMax               : 64
FileInfoCacheLifetime                 : 10
FileNotFoundCacheEntriesMax           : 128
FileNotFoundCacheLifetime             : 5
KeepConn                              : 600
MaxCmds                               : 50
MaximumConnectionCountPerServer       : 32
OplocksDisabled                       : False
RequireSecuritySignature              : False
SessionTimeout                        : 60
UseOpportunisticLocking               : True
WindowSizeThreshold                   : 8



PS C:\WINDOWS\system32> Get-SmbClientConfiguration | Select EnableMultichannel

EnableMultichannel
------------------
              True


PS C:\WINDOWS\system32> Get-SmbClientNetworkInterface

Interface Index RSS Capable RDMA Capable Speed    IpAddresses                                Friendly Name
--------------- ----------- ------------ -----    -----------                                -------------
11              True        False        10 Gbps  {fe80::51f2:e0de:deb2:2a07, 192.168.2.2}   Ethernet
5               False       False        100 Kbps {fe80::ffff:ffff:fffe}                     Teredo Tunneling Pseudo...
3               False       False        100 Kbps {fe80::5efe:192.168.2.2}                   isatap.{AF39D4D2-6F46-4...
9               False       False        1 Gbps   {fe80::dd4f:4718:d882:9096, 192.168.1.106} Local Area Connection
6               False       False        100 Kbps {fe80::5efe:192.168.1.106}                 isatap.{9180C8C3-8ED8-4...


PS C:\WINDOWS\system32> Get-SmbServerConfiguration | Select EnableMultichannel

EnableMultichannel
------------------
              True


PS C:\WINDOWS\system32> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed   IpAddress
---------- --------------- ----------- ------------ -----   ---------
*          11              True        False        10 Gbps 192.168.2.2
*          11              True        False        10 Gbps fe80::51f2:e0de:deb2:2a07
*          9               False       False        1 Gbps  192.168.1.106
*          9               False       False        1 Gbps  fe80::dd4f:4718:d882:9096


(Windows 10 PC to Server 2012 Essentials - Same 3 video files. Under 300MB/s then peaked at 500MB/s on last video file)
Windows PowerShell
Copyright © 2015 Microsoft Corporation. All rights reserved.

PS C:\WINDOWS\system32> Get-SmbConnection

ServerName     ShareName        UserName     Credential   Dialect NumOpens
----------     ---------        --------     ----------   ------- --------
192.168.1.40   Media            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Music            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Net My Documents Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Photo            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Video            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.2.1    V                Eric-PC\Eric Eric-PC\Eric 3.0.2   4
PerryWinServer D                Eric-PC\Eric Eric-PC\Eric 3.0.2   1

PS C:\WINDOWS\system32> Get-SmbMultichannelConnection

Server Name    Selected Client IP   Server IP   Client Interface Index Server Interface Index Client RSS Capable Client
                                                                                                                  RDMA
                                                                                                                 Capabl
                                                                                                                 e
-----------    -------- ---------   ---------   ---------------------- ---------------------- ------------------ ------
PerryWinServer True     192.168.2.2 192.168.2.1 11                     18                     True               False
192.168.2.1    True     192.168.2.2 192.168.2.1 11                     18                     True               False


PS C:\WINDOWS\system32> Get-SmbMultichannelConnection -IncludeNotSelected

Server Name    Selected Client IP     Server IP    Client Interface Index Server Interface Index Client RSS Capable Cli
                                                                                                                    ent
                                                                                                                     RD
                                                                                                                    MA
                                                                                                                    Cap
                                                                                                                    abl
                                                                                                                    e
-----------    -------- ---------     ---------    ---------------------- ---------------------- ------------------ ---
PerryWinServer True     192.168.2.2   192.168.2.1  11                     18                     True               Fal
PerryWinServer False    192.168.1.106 192.168.1.60 9                      13                     False              Fal
PerryWinServer False    192.168.1.106 192.168.2.1  9                      18                     False              Fal
192.168.2.1    True     192.168.2.2   192.168.2.1  11                     18                     True               Fal
192.168.2.1    False    192.168.1.106 192.168.1.60 9                      13                     False              Fal
192.168.2.1    False    192.168.1.106 192.168.2.1  9                      18                     False              Fal


(Server 2012 Essentials to Windows 10 PC - 3 Video Files - 300MB/s Max)
PS C:\WINDOWS\system32> Get-SmbConnection

ServerName     ShareName        UserName     Credential   Dialect NumOpens
----------     ---------        --------     ----------   ------- --------
192.168.1.40   Media            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Music            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Net My Documents Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Photo            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.1.40   Video            Eric-PC\Eric Eric-PC\Eric 1.5     1
192.168.2.1    V                Eric-PC\Eric Eric-PC\Eric 3.0.2   4
PerryWinServer D                Eric-PC\Eric Eric-PC\Eric 3.0.2   1

Windows PowerShell
Copyright © 2015 Microsoft Corporation. All rights reserved.

PS C:\WINDOWS\system32> Get-SmbMultichannelConnection

Server Name    Selected Client IP   Server IP   Client Interface Index Server Interface Index Client RSS Capable Client
                                                                                                                  RDMA
                                                                                                                 Capabl
                                                                                                                 e
-----------    -------- ---------   ---------   ---------------------- ---------------------- ------------------ ------
PerryWinServer True     192.168.2.2 192.168.2.1 11                     18                     True               False
192.168.2.1    True     192.168.2.2 192.168.2.1 11                     18                     True               False


PS C:\WINDOWS\system32> Get-SmbMultichannelConnection -IncludeNotSelected

Server Name    Selected Client IP     Server IP    Client Interface Index Server Interface Index Client RSS Capable Cli
                                                                                                                    ent
                                                                                                                     RD
                                                                                                                    MA
                                                                                                                    Cap
                                                                                                                    abl
                                                                                                                    e
-----------    -------- ---------     ---------    ---------------------- ---------------------- ------------------ ---
PerryWinServer True     192.168.2.2   192.168.2.1  11                     18                     True               Fal
PerryWinServer False    192.168.1.106 192.168.1.60 9                      13                     False              Fal
PerryWinServer False    192.168.1.106 192.168.2.1  9                      18                     False              Fal
192.168.2.1    True     192.168.2.2   192.168.2.1  11                     18                     True               Fal
192.168.2.1    False    192.168.1.106 192.168.1.60 9                      13                     False              Fal
192.168.2.1    False    192.168.1.106 192.168.2.1  9                      18                     False              Fal


PS C:\WINDOWS\system32>
 

 

 

 

And here are my Server 2012 R2 Essentials Results:

 

Windows PowerShell
Copyright © 2014 Microsoft Corporation. All rights reserved.

PS C:\Users\Eric> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Ethernet 2                Realtek PCIe GBE Family Controller           13 Up           30-5A-3A-07-C0-45         1 Gbps
Ethernet 4                Intel® Ethernet Converged Networ...#3      18 Up           A0-36-9F-75-42-54        10 Gbps


PS C:\Users\Eric> Get-NetAdapterRSS


Name                                            : Ethernet 4
InterfaceDescription                            : Intel® Ethernet Converged Network Adapter X540-T1 #3
Enabled                                         : True
NumberOfReceiveQueues                           : 4
Profile                                         : NUMAStatic
BaseProcessor: [Group:Number]                   : :0
MaxProcessor: [Group:Number]                    : :
MaxProcessors                                   : 16
RssProcessorArray: [Group:Number/NUMA Distance] :
IndirectionTable: [Group:Number]                :



PS C:\Users\Eric> Get-NetAdapterRDMA
PS C:\Users\Eric> Get-NetAdapterRDMA
PS C:\Users\Eric> Get-NetAdapterHardwareInfo

Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
----                           ------- --- ------ -------- ---- -------- ------------- ------------- -------
Ethernet 2                           0   9      0        0                    2.5 GT/s             1 1.1
Ethernet 4                           0   1      0        0                    5.0 GT/s             8 1.1


PS C:\Users\Eric> Get-SmbClientConfiguration


ConnectionCountPerRssNetworkInterface : 4
DirectoryCacheEntriesMax              : 16
DirectoryCacheEntrySizeMax            : 65536
DirectoryCacheLifetime                : 10
EnableBandwidthThrottling             : True
EnableByteRangeLockingOnReadOnlyFiles : True
EnableLargeMtu                        : True
EnableMultiChannel                    : True
DormantFileLimit                      : 1023
EnableSecuritySignature               : True
ExtendedSessionTimeout                : 1000
FileInfoCacheEntriesMax               : 64
FileInfoCacheLifetime                 : 10
FileNotFoundCacheEntriesMax           : 128
FileNotFoundCacheLifetime             : 5
KeepConn                              : 600
MaxCmds                               : 50
MaximumConnectionCountPerServer       : 32
OplocksDisabled                       : False
RequireSecuritySignature              : False
SessionTimeout                        : 60
UseOpportunisticLocking               : True
WindowSizeThreshold                   : 1



PS C:\Users\Eric> Get-SmbClientConfiguration | Select EnableMultichannel

                                                                                                     EnableMultichannel
                                                                                                     ------------------
                                                                                                                   True


PS C:\Users\Eric> Get-SmbClientNetworkInterface

Interface Index     RSS Capable         RDMA Capable        Speed               IpAddresses         Friendly Name
---------------     -----------         ------------        -----               -----------         -------------
18                  True                False               10 Gbps             {fe80::386a:a062... Ethernet 4
16                  False               False               0  bps              {fe80::5efe:192.... isatap.{B9E70B2E...
13                  False               False               0  bps              {fe80::58fb:4e2:... Ethernet 2
14                  False               False               0  bps              {fe80::5efe:192.... isatap.{6C6C1BAE...


PS C:\Users\Eric> Get-SmbServerConfiguration


AnnounceServer                  : False
AsynchronousCredits             : 64
AutoShareServer                 : True
AutoShareWorkstation            : True
CachedOpenLimit                 : 5
AnnounceComment                 :
EnableDownlevelTimewarp         : False
EnableLeasing                   : True
EnableMultiChannel              : True
EnableStrictNameChecking        : True
AutoDisconnectTimeout           : 0
DurableHandleV2TimeoutInSeconds : 30
EnableAuthenticateUserSharing   : False
EnableForcedLogoff              : True
EnableOplocks                   : True
EnableSecuritySignature         : True
ServerHidden                    : True
IrpStackSize                    : 15
KeepAliveTime                   : 2
MaxChannelPerSession            : 32
MaxMpxCount                     : 50
MaxSessionPerConnection         : 16384
MaxThreadsPerQueue              : 20
MaxWorkItems                    : 1
NullSessionPipes                :
NullSessionShares               :
OplockBreakWait                 : 35
PendingClientTimeoutInSeconds   : 120
RequireSecuritySignature        : True
EnableSMB1Protocol              : True
EnableSMB2Protocol              : True
Smb2CreditsMax                  : 2048
Smb2CreditsMin                  : 128
SmbServerNameHardeningLevel     : 0
TreatHostAsStableStorage        : False
ValidateAliasNotCircular        : True
ValidateShareScope              : True
ValidateShareScopeNotAliased    : True
ValidateTargetName              : True
EncryptData                     : False
RejectUnencryptedAccess         : True



PS C:\Users\Eric> Get-SmbServerConfiguration | Select EnableMultichannel

                                                                                                     EnableMultichannel
                                                                                                     ------------------
                                                                                                                   True

Windows PowerShell
Copyright © 2014 Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> Get-SmbServerNetworkInterface

Scope Name          Interface Index     RSS Capable         RDMA Capable        Speed               IpAddress
----------          ---------------     -----------         ------------        -----               ---------
*                   18                  True                False               10 Gbps             192.168.2.1
*                   18                  True                False               10 Gbps             fe80::386a:a062:...
*                   13                  False               False               1 Gbps              192.168.1.60
*                   13                  False               False               1 Gbps              fe80::58fb:4e2:b...




(Windows 10 to the Server 2012 Essentials - 500MB/s Max)
PS C:\Windows\system32> Get-SmbConnection

ServerName          ShareName           UserName            Credential          Dialect             NumOpens
----------          ---------           --------            ----------          -------             --------
192.168.2.2         Desktop RAM Disk    PERRYHOMESERVER\... PERRYHOMESERVER\... 3.02                1
PERRYWINSERVER      Shared Folders      PERRYHOMESERVER\... PERRYHOMESERVER.... 3.02                0
PERRYWINSERVER      sysvol              PERRYHOMESERVER\... PERRYHOMESERVER.... 3.02                0

Windows PowerShell
Copyright © 2014 Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> Get-SmbMultichannelConnection

Server Name    Selected       Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                            Interface      Interface      Capable        Capable
                                                            Index          Index
-----------    --------       ---------      ---------      -------------- -------------- -------------- --------------
192.168.2.2    True           192.168.2.1    192.168.2.2    18             11             True           False


PS C:\Windows\system32> Get-SmbMultichannelConnection -IncludeNotSelected

Server Name    Selected       Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                            Interface      Interface      Capable        Capable
                                                            Index          Index
-----------    --------       ---------      ---------      -------------- -------------- -------------- --------------
192.168.2.2    True           192.168.2.1    192.168.2.2    18             11             True           False
192.168.2.2    False          192.168.1.60   192.168.1.106  13             9              False          False
192.168.2.2    False          127.0.0.1      192.168.2.2    1              11             False          False
192.168.2.2    False          192.168.2.1    192.168.1.106  18             9              False          False
192.168.2.2    False          192.168.1.60   192.168.2.2    13             11             False          False
192.168.2.2    False          127.0.0.1      192.168.1.106  1              9              False          False


(Server 2012 Essentials to Windows 10 - 300MB/s Max)
PS C:\Windows\system32> Get-SmbConnection

ServerName          ShareName           UserName            Credential          Dialect             NumOpens
----------          ---------           --------            ----------          -------             --------
192.168.2.2         Desktop RAM Disk    PERRYHOMESERVER\... PERRYHOMESERVER\... 3.02                1
PERRYWINSERVER      Shared Folders      PERRYHOMESERVER\... PERRYHOMESERVER.... 3.02                0


PS C:\Windows\system32> Get-SmbMultichannelConnection

Server Name    Selected       Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                            Interface      Interface      Capable        Capable
                                                            Index          Index
-----------    --------       ---------      ---------      -------------- -------------- -------------- --------------
192.168.2.2    True           192.168.2.1    192.168.2.2    18             11             True           False


PS C:\Windows\system32> Get-SmbMultichannelConnection -IncludeNotSelected

Server Name    Selected       Client IP      Server IP      Client         Server         Client RSS     Client RDMA
                                                            Interface      Interface      Capable        Capable
                                                            Index          Index
-----------    --------       ---------      ---------      -------------- -------------- -------------- --------------
192.168.2.2    True           192.168.2.1    192.168.2.2    18             11             True           False
192.168.2.2    False          192.168.1.60   192.168.1.106  13             9              False          False
192.168.2.2    False          127.0.0.1      192.168.2.2    1              11             False          False
192.168.2.2    False          192.168.2.1    192.168.1.106  18             9              False          False
192.168.2.2    False          192.168.1.60   192.168.2.2    13             11             False          False
192.168.2.2    False          127.0.0.1      192.168.1.106  1              9              False          False


 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Also, when running these copies, I ran at 40% CPU usage on the server when I was transferring at 300MB/s and I hit 90% when I was at 500MB/s.  My PC maxed at 19% CPU usage during the 500MB/s transfer.  On the flip side, copying from the server to the PC I'm getting flat 300MB/s transfers with a steady 40% load.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Here is my theory from running tests and consider RSS QUESUES. When I send data to the server I can get speeds in the 500MB/s range. This is coming from a 6core multithreaded cpu.

Now if I copy from the server. The 4 core cpu is sending data out, but on less possible RSS QUESUES due to the 4 core limitation.

If I compare cores to data rates, you can take what ever amount of physical cores you have and subtract one, then multiply that number by 100MB/s and that is your capable output. At least that is what I'm seeing with these two machines.

Does this have any logic to it?

Well I have trouble believing it, but I'm looking at the pattern right in front of me with days of troubleshooting and tweaking. The reason I have trouble believing it is bc I can use a RAM disk and get 2GB/s crystal mark benchmark tests on the server with 4 cores and nearly 3GB/s tests on the PC with 6 cores.

Anyway, in my effort to get 500MB/s or greater transfers especially coming from the server side of things, I would like to get info ASAP, as I may return this motherboard for something that will

Support and Intel product with equal or

Greater cores than my PC.

Thanks for all the help. If you can disprove my logic or help me improve these speeds id be greatful. If they can't improve, I might be looking at next week as a return date for this board.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

And here are my Server 2012 R2 Essentials Results:

 

I think your issue may be the limitations of Essentials. I have not confirmed this will do some digging now but there are a few potential issues I spotted in the outputs.

 

 

Get-NetAdapterRSS

 

NumberOfReceiveQueues                           : 4  (Should be 16)

 

Get-SmbClientConfiguration

 

EnableLoadBalanceScaleOut             : True (your missing)

WindowSizeThreshold                   : 1 (mine is 8)

 

Download a copy of Windows Server 2012 R2 Standard from the Microsoft website and see if you get better results. There is a free trial.

 

 

Edit: Having a too small windows size could easily limit your achievable speed.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Also just to make sure I was still able to get 1GB/s+ transfer between my computers I installed starwind ram disk and it worked correctly.

 

Edit: 40%~ cpu on both computers in both directions. Workstation is 6 cores 12 threads and server VM is 4 cores.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

-snip-

 

I can also confirm that you are using SMB3 and also multi channel so from that side of things you should be able to hit full speeds. Definitely looks like it is a networking issue, where it is I'm not totally sure.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Well it's good to know that I'm utilizing SMB3 and multi channel.

Does anyone know if there is a relation between how many cores you have and the nominal speed of the Intel 10GbE NIC?

What I am seeing is 95% load on my AMD quad core when I reach 500MB/s transfers, making me think that my bottleneck is my server processor.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

I changed the Adapter settings following a guideline from a blog I was reading about a similar setup.  I changed the RSS Queues to 4 to match the number of processors.  I'll change it back to 16 and see if i get better results.

 

Is there anything I can do about Enabling Load Balance Scale Out?  I suppose that may need to be done with Windows 2012 R2 Standard which I also have a copy of.  I'll try that after seeing if the above doesn't work.

 

As for Windows Size Threshold, is there anything I can do with that as well?  If not, I'll see if it improves with an install of Windows 2012 R2 Standard.

 

I suppose that will be my next step in testing.  It still gets me that I have 95% utilization on the server when I copy at 500MB/s from my workstation, and I get half that when I copy from my server to my workstation with 300MB/s.  That's still a mystery to me.

 

I'll report back with my findings Tonight or tomorrow.  Thanks for the next steps of troubleshooting ideas.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Try

 

Set-SmbClientConfiguration -WindowSizeThreshold <value>

 

I don't know what unit the command wants but usually it will accept something like '8MB'. Also it might be the CPU, 10Gb is rather stressful on cpu resources so if it is a lower end AMD then it may just be that.

 

My desktop is an i7-4930k and server E5-2620.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Here is tonight's update. I played around with the queues again and found no change, but also sparatic good changes. I'll explain.

I got a 3ft cat6a cable in the mail today, hooked that up with little to no influence on performance. On the client Windows 10 PC I changed the link speed from auto to 10Gb. This made sparatic changes on the server to client transfers. My method is transfer 18GB worth of video files (3 files) to the server. I got the 500MB/s I've been getting with 95% cup utilization on the server. I copy those back to the PC (mind you both of these are RAN disks) and I get 500MB/s now. I did the same process again to test it wasn't a fluke, and I saw 300MB/s server to PC with only 45% cup usage. I did this a couple more times and saw 50/50 results...

In the end I've seen the best performance with max queues on both and allowing max cores on the server (or the driver defaults). From what I've read the drivers auto negotiate based on your system's specs, and it seems to have given me best results.

I'm still working toward that ever in the distance 10Gb, but I'm in fear that my old amd quad core isn't going to cut it.

I'll try those settings above, and I'm going to attempt server 2012 r2 standard, and also try FreeNAS before I return this motherboard and seek out something beefier. I may have the opportunity to get another 3930 used from my work, so we will see...

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Here is tonight's update. I played around with the queues again and found no change, but also sparatic good changes. I'll explain.

I got a 3ft cat6a cable in the mail today, hooked that up with little to no influence on performance. On the client Windows 10 PC I changed the link speed from auto to 10Gb. This made sparatic changes on the server to client transfers. My method is transfer 18GB worth of video files (3 files) to the server. I got the 500MB/s I've been getting with 95% cup utilization on the server. I copy those back to the PC (mind you both of these are RAN disks) and I get 500MB/s now. I did the same process again to test it wasn't a fluke, and I saw 300MB/s server to PC with only 45% cup usage. I did this a couple more times and saw 50/50 results...

In the end I've seen the best performance with max queues on both and allowing max cores on the server (or the driver defaults). From what I've read the drivers auto negotiate based on your system's specs, and it seems to have given me best results.

I'm still working toward that ever in the distance 10Gb, but I'm in fear that my old amd quad core isn't going to cut it.

I'll try those settings above, and I'm going to attempt server 2012 r2 standard, and also try FreeNAS before I return this motherboard and seek out something beefier. I may have the opportunity to get another 3930 used from my work, so we will see...

 

Yea it's not often you actually have to mess around with the NIC driver settings, even more so when directly connecting between two of the same cards. Only thing I do change is the queue sizes to the max possible settings.

 

The speed difference could be in the windows delayed write and caching the file transfer in memory, yea i know its already a ram disk. But the delayed write is more efficient than the ram disk due to going through less storage API's etc. One way you can check this is open resource monitor and check the disk tab, you'll see activity after the copy finishes and lots of it.

 

Is it possible just to borrow another faster computer to do testing just to confirm it is hardware/cpu related issue?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

So I discovered via another forum that the windows server had privileged settings that enabled authenticity signing and encryption which prevented me from getting higher speeds (due to CPU usage).  I've disabled those settings and I'm getting between 500MB/s and 750MB/s.  I'm going to revert my drivers and play with settings again to see if I can't get them at peak performance, now that I've figured out a bottleneck in that area.

 

My question for you is, would it be more ideal in my situation to just set my server hardware (which is consumer hardware) on a Windows 10 install, and eliminate my privilege and future issues I may have?  I could essentially disable services and reduce the CPU usage and just run the basic needs for me to use it as a NAS.  Do you think that would save me a lot of headaches since I'm very new to server OS and it takes a lot of digging to eliminate problems like this.

 

Thanks!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

So I discovered via another forum that the windows server had privileged settings that enabled authenticity signing and encryption which prevented me from getting higher speeds (due to CPU usage).  I've disabled those settings and I'm getting between 500MB/s and 750MB/s.  I'm going to revert my drivers and play with settings again to see if I can't get them at peak performance, now that I've figured out a bottleneck in that area.

 

My question for you is, would it be more ideal in my situation to just set my server hardware (which is consumer hardware) on a Windows 10 install, and eliminate my privilege and future issues I may have?  I could essentially disable services and reduce the CPU usage and just run the basic needs for me to use it as a NAS.  Do you think that would save me a lot of headaches since I'm very new to server OS and it takes a lot of digging to eliminate problems like this.

 

Thanks!

 

Nice find. By any chance is it these setting?

 

 

PS C:\Users\Eric> Get-SmbServerConfiguration

 
RequireSecuritySignature        : True
EnableSecuritySignature         : True

 

On my server these values are false. I did spot it before but didn't expect it to make too much of an impact so didn't say anything.

 

Technically speaking the server OS should be less resource demanding than Windows 10 but that may not always be true in all cases as you have just found.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Yes, i had to follow these instructions:

If you are using a Windows 2012 or newer server, and it is configured as a domain controller, it may be defaulted to encrypting all traffic. This will slow down 10Gbe connections!

Disable these two settings in the Default Domain Controller Group Policy on the server.

Domain Member: Digitally encrypt or sign secure channel data (always)
Microsoft network server: Digitally sign communications (always)

References:
https://technet.microsoft.com/en-us/library/cc736800(v=ws.10).aspx
https://technet.microsoft.com/en-us/library/cc786681(v=ws.10).aspx

- See more at: http://www.cinevate.com/blog/confessions-of-a-10-gbe-network-newbie-part-5-smb3/#sthash.qV8I6Ygn.dpuf

If you are using a Windows 2012 or newer server, and it is configured as a domain controller, it may be defaulted to encrypting all traffic. This will slow down 10Gbe connections!

Disable these two settings in the Default Domain Controller Group Policy on the server.

Domain Member: Digitally encrypt or sign secure channel data (always)
Microsoft network server: Digitally sign communications (always)

References:
https://technet.microsoft.com/en-us/library/cc736800(v=ws.10).aspx
https://technet.microsoft.com/en-us/library/cc786681(v=ws.10).aspx

- See more at: http://www.cinevate.com/blog/confessions-of-a-10-gbe-network-newbie-part-5-smb3/#sthash.qV8I6Ygn.dpuf

https://technet.microsoft.com/en-us/library/cc736800(v=ws.10).aspx

and

https://technet.microsoft.com/en-us/library/cc786681(v=ws.10).aspx

 

You telling me to run those power shell commands brought light to it.  That's why I was seeing so much overhead on my CPU maxing it out.

 

Now that I've taken care of that, I'm trying to get these drivers tuned up.  I've ran through all the settings:

 

1. Turned on Jumbo frames and set them to 9014

2. Changed RSS Processor to 4 on the server, it isn't available as an option on the windows 10 machine.

 

3. Changed RSS Queues on the server to 4 and Windows 10 to 8 (since I have 12 logical cores no option for 12 though, I read that you should do equal or under) (at this point I got almost 1GB/s copies to the server and only 200-300MB/s with spikes in the 500MB/s range to the PC.  Prior to this change I got 500-700MB/s going to the Server and 500MB/s going to the PC)

 

4. I tried Changing RSS Ques on the PC to 4 and left the Server at 4, I got 500-600MB/s to the server and I got on average 490MB/s going to the PC.

 

5. I changed Recieve buffer on both the PC and the Server to 4,096.  I got 300-600MB/s with dips when copying to the server.  I got 490MB/s consistent copying to the PC.

 

6. Changing the Transmit Buffer to 16384 on both the server and PC I saw 500-700MB/s copying to the server and 460-500MB/s copying to the PC.

 

 

I'm wondering if there is a correlation with how many RSS Queues you have and how much data can go down the pipe.  It looks as though my PC with more cores and logical cores via hyperthreading can send more down the pipe than my server with 4 physical cores and no hyperthreading.  Although, I can receive that large amount of data with the quad core processor and copy it, I just can't send that much out, thus leaving me with lesser results in sending to the PC.

 

 

 I made these changes on a Default setup of the drivers to see how it would effect each adjustment and so I could try and locate a setting to change in the end.

These are the settings I'm going to lock in for a while and then test some new hardware (my work was nice enough to loan me a mother board and the same processor I'm using in my Windows 10 PC.)  If my theory is correct, I should be able to get 1GB/s transfers when I hook up the new hard ware.

 

I had these changes and I got this performance: Server to PC = Flat 500MB/s performance PC to Server got fluctuating 400-600MB/s Performance.

Jumbo Frames - 9014

RSS Processor - 4 (server option only, set for the 4 physical cores I have in AMD Phenom II X4)

RSS Queues - Server 4 & PC 8

Performance Read Buffer 4096 abd Transfer Buffer 16384 on both PC and Server

Receive side scaling and interrupt were enabled by default

 

Here are the settings I'm going to lock in.  I got 550-750MB/s going to the server and 500MB/s transfer to the PC

Jumbo Frame - 9014

RSS Processor - 4 (server option only, set for the 4 physical cores I have in AMD Phenom II X4)

RSS Queues 4 on both the Server and PC

Performance Read Buffer 4096 and Transfer Buffer 16384 on both PC and Server

Rceive side scaling and interrupt were enabled by default.

 

I'll play around some more with this to find the best settings, but I think I'm capped at 500 coming from the server to my client.  Equipment will tell if I can get higher, or it may be an operating system issue...  Can you think of anything else with Server 2012 that could be slowing transfer to clients from the Server?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Sounds like you have everything tuned as best you can. Encrypted data sessions will have a big impact on CPU load as you have seen, great to hear you have that solved.

 

The advice you have seen about the RSS queues is correct, equal to amount of cores is always best if you can. Also as a test try setting the jumbo packet size to 4088 and then no jumbo frames. The high payloads might actually be hurting your speeds in a cpu constrained situation, unlikely but worth a try.

 

Only big test left would be to try the server end with a better cpu.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

I'll try both those suggestions with Jumbo Frames.

 

As for trying new hardware.  In the Windows Server 2012 R environment, can I just install a new motherboard and processor and not reinstall the OS? Will that work for a test, or would it be best to reinstall and go through changing permissions again?

 

Attempting today with the new hardware.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

I'll try both those suggestions with Jumbo Frames.

 

As for trying new hardware.  In the Windows Server 2012 R environment, can I just install a new motherboard and processor and not reinstall the OS? Will that work for a test, or would it be best to reinstall and go through changing permissions again?

 

Attempting today with the new hardware.

 

Should be able to just use the new hardware, it'll install drivers etc and want to reboot. Also Windows may want to active again.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Awesome, well I did exactly that.  So my AMD processor without any hyperthreading was my bottleneck with only get 750MB/s transfers.  I tried a hex core intel processor with hyperthreading allow me to get more RSS Queues, and I got 1GB/s transfers.  So that answers the question of whether hardware is a bottleneck for the x540 T1 NIC or not.

 

Now I'm trying different Operating Systems.  The Server 2012 R2 operating system doesn't support my 1GbE on board connection and a number of other devices that I'm unaware of, so I'm trying Windows 10 as a more user friendly and familiar platform.  I want to see if I can get the same speeds so I can utilize the onboard features, and opt out of the Server/Domain features I don't need in my home network.

 

Here's my issue starting from the install.  Everything installs correctly from what I can tell, but I started seeing ridiculous lag when setting up Windows 10 (havn't seen it on my upgrade from win 7 to 10).  I eventually get logged into Windows 10 on the "server" case, and I find that I'm still getting lag.  I opened task manager, and found 100% utilization of my disk (my OS hard drive to be exact).  I've tried everything I can find on the internet, disable super fetch, prefetch, disable alerts of all kinds, installed drivers for the motherboard chipset and SATA controllers, updated BIOS, checked firmware on my SSD, and a handful of other things short of registry edits.

 

I'm going to attempt a restore, and if that doesn't work another fresh install.  Beyond that I don't know what to do.  I have one other SSD to try if all else fails, but this seems like a big Microsoft mess up.

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Networking

×