Jump to content

Help with a 10gbe network setup

iller643
By iller643
in Networking
 Share
Posted

hey everyone, first time poster here and a newbie when it comes to this stuff but im hoping some more experienced peeps can shed some light on my issue.

 

i have been setting up a 10gbe connection in my home lab. I have:

 

HP Z440

Xeon E5-1650 v3 @ 3.5200GHz

32GB Ram

1x Samsung SSD 860 EVO 1TB

Mellanox ConnectX-2 MT26448 10Gbe NIC

 

Dell Poweredge T410

2x Xeon X5560 @ 2.80Ghz (total of 8 cores, 16 logical)

64GB Ram

Dell PERC H700

4x Samsung SSD 860 EVO 4TB in Raid 5 (AJA system test gives me 1163 Write , 1082 Read)

 

when i do a local host iperf3 test on just the Z440 i get:

 

Connecting to host localhost, port 5201
[  4] local ::1 port 52228 connected to ::1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.14 GBytes  9.81 Gbits/sec
[  4]   1.00-2.00   sec  1.90 GBytes  16.3 Gbits/sec
[  4]   2.00-3.00   sec  2.11 GBytes  18.1 Gbits/sec
[  4]   3.00-4.00   sec  2.10 GBytes  18.1 Gbits/sec
[  4]   4.00-5.00   sec  2.11 GBytes  18.1 Gbits/sec
[  4]   5.00-6.00   sec  2.11 GBytes  18.1 Gbits/sec
[  4]   6.00-7.00   sec  2.11 GBytes  18.1 Gbits/sec
[  4]   7.00-8.00   sec  2.12 GBytes  18.2 Gbits/sec
[  4]   8.00-9.00   sec  1.07 GBytes  9.22 Gbits/sec
[  4]   9.00-10.00  sec  1.32 GBytes  11.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  18.1 GBytes  15.5 Gbits/sec                  sender
[  4]   0.00-10.00  sec  18.1 GBytes  15.5 Gbits/sec                  receiver

 

 

but when i do the same thing on the Dell T410 i get:

 

Connecting to host localhost, port 5201
[  4] local ::1 port 49712 connected to ::1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   193 MBytes  1.61 Gbits/sec
[  4]   1.01-2.00   sec   245 MBytes  2.07 Gbits/sec
[  4]   2.00-3.00   sec   258 MBytes  2.16 Gbits/sec
[  4]   3.00-4.01   sec   208 MBytes  1.73 Gbits/sec
[  4]   4.01-5.01   sec   210 MBytes  1.76 Gbits/sec
[  4]   5.01-6.00   sec   270 MBytes  2.28 Gbits/sec
[  4]   6.00-7.00   sec   208 MBytes  1.73 Gbits/sec
[  4]   7.00-8.00   sec   249 MBytes  2.09 Gbits/sec
[  4]   8.00-9.00   sec   266 MBytes  2.23 Gbits/sec
[  4]   9.00-10.00  sec   270 MBytes  2.26 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec                  receiver

 

when i do an iperf from the Z440 to Dell T410 i get:

 

[  4] local 192.168.15.106 port 53566 connected to 192.168.15.253 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   489 MBytes  4.10 Gbits/sec
[  4]   1.00-2.00   sec   591 MBytes  4.96 Gbits/sec
[  4]   2.00-3.00   sec   576 MBytes  4.83 Gbits/sec
[  4]   3.00-4.00   sec   616 MBytes  5.17 Gbits/sec
[  4]   4.00-5.00   sec   600 MBytes  5.04 Gbits/sec
[  4]   5.00-6.00   sec   585 MBytes  4.90 Gbits/sec
[  4]   6.00-7.00   sec   607 MBytes  5.09 Gbits/sec
[  4]   7.00-8.00   sec   593 MBytes  4.97 Gbits/sec
[  4]   8.00-9.00   sec   601 MBytes  5.04 Gbits/sec
[  4]   9.00-10.00  sec   597 MBytes  5.01 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  5.72 GBytes  4.91 Gbits/sec                  sender
[  4]   0.00-10.00  sec  5.72 GBytes  4.91 Gbits/sec                  receiver

 

 

 

i have reset the Dells network adapter, changed the Jumbo frames buffer etc but i cant seem to get the Dell numbers to improve - the NIC is plugged into Slot 5 (which is a x8 lane)

 

transferring a single large file i only reach 250mb speeds. my question i guess is - is the Dell just too old for this type of connection or am i missing some setting?

 

thanks for any advice you can share

Z440 to Dell T410.PNG

Link to comment
https://linustechtips.com/topic/1246174-help-with-a-10gbe-network-setup/
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Hi Electronics Wizardy,

 

i have 9014 jumbo packets running, when using the -p 10 i get:

 

Connecting to host localhost, port 5201
[  4] local ::1 port 49989 connected to ::1 port 5201
[  6] local ::1 port 49990 connected to ::1 port 5201
[  8] local ::1 port 49991 connected to ::1 port 5201
[ 10] local ::1 port 49992 connected to ::1 port 5201
[ 12] local ::1 port 49993 connected to ::1 port 5201
[ 14] local ::1 port 49994 connected to ::1 port 5201
[ 16] local ::1 port 49995 connected to ::1 port 5201
[ 18] local ::1 port 49996 connected to ::1 port 5201
[ 20] local ::1 port 49997 connected to ::1 port 5201
[ 22] local ::1 port 49998 connected to ::1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[  6]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[  8]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[ 10]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[ 12]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[ 14]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[ 16]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[ 18]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[ 20]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[ 22]   0.00-1.04   sec  20.0 MBytes   161 Mbits/sec
[SUM]   0.00-1.04   sec   200 MBytes  1.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[  6]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[  8]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[ 10]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[ 12]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[ 14]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[ 16]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[ 18]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[ 20]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[ 22]   1.04-2.05   sec  18.8 MBytes   156 Mbits/sec
[SUM]   1.04-2.05   sec   188 MBytes  1.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[  6]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[  8]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[ 10]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[ 12]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[ 14]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[ 16]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[ 18]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[ 20]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[ 22]   2.05-3.00   sec  18.8 MBytes   165 Mbits/sec
[SUM]   2.05-3.00   sec   188 MBytes  1.65 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[  6]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[  8]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[ 10]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[ 12]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[ 14]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[ 16]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[ 18]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[ 20]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[ 22]   3.00-4.03   sec  21.2 MBytes   173 Mbits/sec
[SUM]   3.00-4.03   sec   212 MBytes  1.73 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[  6]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[  8]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[ 10]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[ 12]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[ 14]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[ 16]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[ 18]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[ 20]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[ 22]   4.03-5.03   sec  18.8 MBytes   158 Mbits/sec
[SUM]   4.03-5.03   sec   188 MBytes  1.58 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[  6]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[  8]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[ 10]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[ 12]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[ 14]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[ 16]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[ 18]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[ 20]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[ 22]   5.03-6.00   sec  18.8 MBytes   161 Mbits/sec
[SUM]   5.03-6.00   sec   188 MBytes  1.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[  6]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[  8]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[ 10]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[ 12]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[ 14]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[ 16]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[ 18]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[ 20]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[ 22]   6.00-7.03   sec  20.0 MBytes   163 Mbits/sec
[SUM]   6.00-7.03   sec   200 MBytes  1.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[  6]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[  8]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[ 10]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[ 12]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[ 14]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[ 16]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[ 18]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[ 20]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[ 22]   7.03-8.04   sec  20.0 MBytes   166 Mbits/sec
[SUM]   7.03-8.04   sec   200 MBytes  1.66 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[  6]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[  8]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[ 10]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[ 12]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[ 14]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[ 16]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[ 18]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[ 20]   8.04-9.02   sec  26.0 MBytes   222 Mbits/sec
[ 22]   8.04-9.02   sec  25.9 MBytes   221 Mbits/sec
[SUM]   8.04-9.02   sec   260 MBytes  2.22 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[  6]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[  8]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[ 10]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[ 12]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[ 14]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[ 16]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[ 18]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[ 20]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[ 22]   9.02-10.01  sec  26.2 MBytes   224 Mbits/sec
[SUM]   9.02-10.01  sec   262 MBytes  2.24 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[  4]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[  6]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[  6]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[  8]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[  8]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[ 10]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[ 10]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[ 12]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[ 12]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[ 14]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[ 14]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[ 16]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[ 16]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[ 18]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[ 18]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[ 20]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[ 20]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  receiver
[ 22]   0.00-10.01  sec   208 MBytes   175 Mbits/sec                  sender
[ 22]   0.00-10.01  sec   208 MBytes   174 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  2.04 GBytes  1.75 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  2.03 GBytes  1.75 Gbits/sec                  receiver

Link to post
Share on other sites
Posted
  • Author

not sure if it helps with trouble shooting but these are my adapter settings on the Dell:

 

10gbe                     Transmit Control Blocks        16                             NumTcb          {16}
10gbe                     Large Send Offload (LSO)       Enabled                        LargeSendOff... {1}
10gbe                     Receive Completion Method      Adaptive                       RecvCompleti... {1}
10gbe                     Jumbo Packet                   9014                           *JumboPacket    {9014}
10gbe                     Receive Buffers                4096                           *ReceiveBuffers {4096}
10gbe                     Send Buffers                   4096                           *TransmitBuf... {4096}
10gbe                     NetworkDirect Functionality    Enabled                        *NetworkDirect  {1}
10gbe                     Receive Side Scaling           Enabled                        *RSS            {1}
10gbe                     Preferred NUMA node            Default Settings               *NumaNodeId     {65535}
10gbe                     Maximum Number of RSS Queues   12                             *NumRSSQueues   {12}
10gbe                     RSS load balancing Profile     ClosestProcessor               *RSSProfile     {1}
10gbe                     Number of Polls on Receive     10000                          ThreadPoll      {10000}
10gbe                     Rx Interrupt Moderation Type   Adaptive                       RxIntModeration {2}
10gbe                     Flow Control                   Rx & Tx Enabled                *FlowControl    {3}
10gbe                     Interrupt Moderation           Enabled                        *InterruptMo... {1}
10gbe                     Rx Interrupt Moderation Pro... Moderate                       RxIntModerat... {1}
10gbe                     Tx Interrupt Moderation Pro... Moderate                       TxIntModerat... {1}
10gbe                     IPV4 Checksum Offload          Rx & Tx Enabled                *IPChecksumO... {3}
10gbe                     TCP/UDP Checksum Offload (I... Rx & Tx Enabled                *TCPUDPCheck... {3}
10gbe                     TCP/UDP Checksum Offload (I... Rx & Tx Enabled                *TCPUDPCheck... {3}
10gbe                     Large Send Offload Version ... Enabled                        *LsoV1IPv4      {1}
10gbe                     Large Send Offload V2 (IPv4)   Enabled                        *LsoV2IPv4      {1}
10gbe                     Large Send Offload V2 (IPv6)   Enabled                        *LsoV2IPv6      {1}
10gbe                     Priority & Vlan Tag            Priority & VLAN Enabled        *PriorityVLA... {3}
10gbe                     Virtual Machine Queues         Enabled                        *VMQ            {1}
10gbe                     VMQ VLAN Filtering             Enabled                        *VMQVlanFilt... {1}
10gbe                     R/RoCE Max Frame Size          Auto                           RoceMaxFrame... {0}
10gbe                     VLAN ID                        0                              VlanID          {0}

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Networking

×