10Gig Networking in ESXI VMs

[DrDrummer]

Hello there,

 

Iam from Germany and i will try my best to Write an understandable text here. 

 

We currently try to virtualize our Company Servers with VMware vSphere. I run into a Problem were my virtual machines dont use the bandwith of our 10Gig NICs. All VMs Show up a 10Gbit/s Connection But just transfer Data with a Maximum of Around 110MB/s. Same Thing from VM to VM our from VM to the Network.

 

Is there a Configuration i have to take?

We use Mellanox Connecxt-3 and Intel X540-T2 Cards.

The Servers are Windows Server 2016, 2012 and 2012R2.


I Hope you could understand my Problem and have an Idea to help me out.

 

Greets from Germany
Niklas

[Senzelian]

What kind of files are you trying to transfer and about what kind of storage mediums are we talking about?
Maybe there's a bottleneck somewhere because the VMs would eat into the available storage resources. Just a thought... :s 

Btw, Yaaaay for Germanaay!  

[DrDrummer]

I try to Transfers 4GB .iso and .rar files for testing.

 

The VM-Host is installed on a SATA-SSD RAID1 and Uses a RAID5 with 6 SAS Drives for Storage.

 

The Client for Testing is a Windows 10 PC with an SATA-SSD.

 

[leadeater]

Use iperf to test network bandwidth between the the VMs and between a VM to another client or physical server. Most often it's not actually network bandwidth that is the issue.

[Senzelian]

30 minutes ago, DrDrummer said:

I try to Transfers 4GB .iso and .rar files for testing.

 

The VM-Host is installed on a SATA-SSD RAID1 and Uses a RAID5 with 6 SAS Drives for Storage.

 

The Client for Testing is a Windows 10 PC with an SATA-SSD.

 

It's weird that your network speeds caps out at around 110MB/s. That is exactly what I'd expect from a 1Gbit network.

I think leadeater is correct and testing your network bandwith with iperf should help us to identify whether it's a network issue or not.

[DrDrummer]

Thanks for your replys so far. This is the iperf result between two of the VMs

 

[leadeater]

1 hour ago, DrDrummer said:

Thanks for your replys so far. This is the iperf result between two of the VMs

Which VM NIC type is being used? E1000 or VMXNET3?

[DrDrummer]

3 hours ago, leadeater said:

Which VM NIC type is being used? E1000 or VMXNET3?

VMXNET3 for all of the virtual Adapters

[BloodKnight7]

1) (you surely must have this but Ill put it anyways) make sure you are running latest virtual hardware version supported by your vmware version and latest vmware tools installed.

2)  You may need to disable LRO in a VMXNET3 network adapter. On Windows, the LRO technology is also referred to as Receive Side Coalescing (RSC). For more information search Microsoft help for how to Enable or Disable LRO on a VMXNET3 Adapter on a Windows Virtual Machine. https://kb.vmware.com/s/article/2129176

 

Let me know how it goes, Ive seen this before, mainly when dealing with SQL Server VMs.

[DrDrummer]

i just get about 40MBits/s after Disabling LRO

Edit: I added a second virtual NIC for Testing (VMXNET3 too) and now we are talking about 1.08 to 1.39 Gbits per second. Maximum here was 162Mbytes/s.
So we are over the 1gbit but dont nearly reach the 10Gbit mark.

 

Edit2: just connected two Clients via 10Gbit to Test the Overall Performance without VM-Magic.
After Set die Jumbo Packts to the Max Value i geht respectable 4.27Gbit/s with iperf. Transfer my testfiles Ende up with a very throttled speed. This Happens with Evers file i tested so far.
You can check this on the following pictures. 

 

Both Clients with SSD Storage and 10Gbit Network Cards (Intel 540-T2 and ASUS XG-C100C

 

Niklas

[DrDrummer]

After reconfigure all virtual Network Connections from Scratch, it works (sort of..).
Gettin 500MB/s between two SSD-Clients and about 350MB/s to the RAID-Server.

 

Disable RSX and configure Jumbo Packets helps alot.

Thanks for your Support!
 

[vano411]

From my experience, the VM layers limitation is 4Gbps. We couldn't get past it no matter what we tried. So we just gave the VM direct access to a 10Gbps network card using PCI passthrough and it started flying at full speed.

[DrDrummer]

If i understand this correctly, i would need a dedicated NIC for each VM then?

[vano411]

Yes, that's correct.

 

Our VM's couldn't even communicate with each other at 10Gbps (also due to the 4Gbps limit in the hypervisor) which was mandatory for our use case. We had to install two 2-port NICs on the machine, dedicate one to each VM and connect them to each other. Funny setup but it worked like a charm.

[DrDrummer]

That would be an Option but i think the actual Stats are ok for now. We dont have to transfer Lots of files in our Daily Business.

[leadeater]

4 hours ago, vano411 said:

Yes, that's correct.

 

Our VM's couldn't even communicate with each other at 10Gbps (also due to the 4Gbps limit in the hypervisor) which was mandatory for our use case. We had to install two 2-port NICs on the machine, dedicate one to each VM and connect them to each other. Funny setup but it worked like a charm.

Our VMs can all do 10Gbps happily, ESXi. NIC settings do need optimizing though.

[Jarsky]

You might want to try enabling "Receive Side Scaling" which will balance the CPU usage across the cores. 

You need to do this in vSphere on the VMXNET3 adapter, as well as in the guest OS on the adapter properties. 

[BloodKnight7]

Just a "control" question... if you have 2VMs in the same host and assigned to the same virtual switch,  if you do an iperf3 test...whats the speed you are getting? Is it superior to 10Gbps? 

 

Oh! and another thing! you did configue jumbo frames on your physical switches and virtual switches besides your host OS, correct? 

[DrDrummer]

I dont get this.. today, the maximum transferrate between the two machines is up to 180MB/s. The Iperf3 Test result is just 1.24Gbit/s

 

Correct. Did the Jumbo Frame Configuration in the VMs and to the host itsself.

[BloodKnight7]

1) what about in your network switch? was the jumbo frames configuration applied there as well? 

 

2) the machines you used for testing, are they in separate hosts? are they in the same host? would you care to draw a simple diagram of your infrastructure and post it? would be easier to troubleshoot