4x10Gb NIC performance

[michael_p]

Hi,

 

I'm new to this forum and I have an issue.

I have point to point network, 4 clients connected to one server directly (no switch):

• 1 server - i9-9900 with 4 port 10Gb NIC (intel x710),  windows10
• 4 clients - xeon with 1 port 10Gb NIC, windows10

When I run iPerf I get the following results:

1. 1 to 1 - 10Gb (doesn't meter which one)
2. 2 to 1 - 10Gb x 2 (20Gb total)
3. 3 to 1 - 8Gb x 3 (24Gb total)
4. 4 to 1 - 6Gb x 4 (24Gb total)

The server is maxed out at ~25Gb/s

I tried to change the buffers, number of connection and nothing helps.

 

Please help,

Michael

[Falconevo]

20 minutes ago, michael_p said:

Hi,

 

I'm new to this forum and I have an issue.

I have point to point network, 4 clients connected to one server directly (no switch):

• 1 server - i9-9900 with 4 port 10Gb NIC (intel x710),  windows10
• 4 clients - xeon with 1 port 10Gb NIC, windows10 

When I run iPerf I get the following results:

1. 1 to 1 - 10Gb (doesn't meter which one) 
2. 2 to 1 - 10Gb x 2 (20Gb total) 
3. 3 to 1 - 8Gb x 3 (24Gb total)
4. 4 to 1 - 6Gb x 4 (24Gb total)

The server is maxed out at ~25Gb/s 

I tried to change the buffers, number of connection and nothing helps.

 

Please help,

Michael

Are you sure you aren't maxing out the storage you are using?   This is assuming you aren't using a ram disk or null to perform throughput testing.

If you are performing testing that is direct to storage, then you are likely limited by IO constraints.  Assuming you are using Windows and using somthing like NTtccp for testing, here are some additional options you will need enabling for the network config/driver;

 

Latest available Intel drivers

RSS (Recieve Side Scaling)
RSS Queues - Max the driver is capable of using

RSS Processors - Max threads on the system (logical processors)

MTU 9000 (Jumbo Frames) **this also NEEDS to be enabled on all your switching equipment**

Transmit Buffers - Max the driver is capable of using

Recieve Buffers - Max the driver is capable of using

 

If you can provide more details on how you are performing testing and what software you are using that would also be helpful.

[mariushm]

Is the data going out the network card or just flowing between ports?

 

If it has to get out the card, it could be an issue of pci-e bandwidth ... if the slot is pci-e 2.0 x8, then you have a max 8 x 500 MB = 4 GB/s or 32 gbps theoretical ... with overhead and stuff, you're looking at maybe 28 gbps maximum

Same if the slot is pci-e 3.0 x4 (slot may be x16 physically but only x4 electrically) ... a single 3.0 lane is ~ 970 MB/s so a x4 will be less than 4 GB/s.

 

Also keep in mind if the slot is created by PCH/chipset you have a DMI 3.0 connection to the cpu which has a maximum bandwidth of around 4 GB/s (at least it was that much on previous generations, not sure about i9 9900 )

[michael_p]

Hi,

 

Thank you for replay.

• I double check that I'm using full PCIe 3 x 8 minimum
• All card/ports set to jumbo frame 9000
• All RSS are maxed out
• All buffers are maxed out

I perform the test using -

• iPerf its network speed test that doesn't copy any thing to storage, only receiving data and discards it
• My own software that basically does the same as iPerf

 

Is there something to do with windows?

 

Thanks

[tankhunter8192]

which version of windows do you use?

some older versions have an explorer with limited transfer speed(around 3-6GB/s) [like Linus showed in the "100GBit/s NICs"]