UNRAID VM to VM 10gigabit transfer

[Strikermed]

Hey Guys, 

 

I have another question over UNRAID and 10gigabit speeds.  This isn't necessarily a networking question, but more of a KVM question regarding transfers between 2 VM's on the same UNRAID server.  

 

I have 2 VM's running on UNRAID, one is freenas, and the other is windows 10.  I have mapped a network drive on windows to freenas via the Virtual connection created in KVM (which shows 10gigabit).  The initial transfer gets 200-245 MB/s.  I know Freenas is capable of doing at least 400 MB/s, so the fact that it's below boggles me a little bit.  I know I'm transfering to SSD, so I can reach that 500MB/s mark at minimum.  

 

So my next step is to do some driver tuning like I do windows to windows with intel NICs. 

 

On FreeNAS the only tuning I was able to do is change MTU to 9000

 

On Windows 10 I changed the MTU to 9000, changed RSS Queues to 16 since I have 16 logical processors, and I changed Transmit and receive buffers to their max.

 

Normally this would make leaps in progress for reaching 10gigabit speeds, but it actually did worse.  Now I still get 200-245MB/s, but then about half way through a file transfer it comes to a crashing halt.  It also appears that all those mapped drives become disconnected at that time for a few minutes.  Only whenI cancel the transfer does it recover.

 

What's worse is I can't get the connections to be stable after changing all the settings back.

 

Any thoughts?

 

Thanks!

 

 

[leadeater]

Check CPU, Ram and disk utilization during transfers on client and server side. You might have an issue where data isn't flushing from ram fast enough so after a while you hit max and the transfer stalls.

 

Also check unRAID host CPU usage as well.

[Strikermed]

8 hours ago, leadeater said:

Check CPU, Ram and disk utilization during transfers on client and server side. You might have an issue where data isn't flushing from ram fast enough so after a while you hit max and the transfer stalls.

 

Also check unRAID host CPU usage as well.

Will do.  It's funny though.  I have 16 cores and gave it 16GB of memory.  It seems strange that at 3GB file would fill the RAM, with it only being utilized at less than half a idle.

 

I'll diagnose more with your suggestion.  Thanks.

 

 

 

I have discovered this since I troubleshooted.  I removed the extra Virtual Etherenet interface.  I assumed that creating a separate interface, it that it would in fact help me keep traffic separate.  I also assumed that what ever traffic went on my regular LAN would be routed through my router and essentially make a round trip back over the 1GbE connections, making it inefficient.  When I actually tested it with a real word file transfer utilizing my LAN, I realized that's not the case.  It actually routes locally, making the 10Gigabit connection between the 2 VM.  This allowed me to get over 500MB/s transfers.

 

 

I still don't understand why my VM's with an extra virtual Ethernet acted the way it did.  They were both on the same subnet and could ping each other, but when I attempted to transfer files, it would crap out.  More testing and configuring will need to be done, but it just didn't make sense to me.  It's like plugging in another cable, or even direct connecting them, but it just belly flops under any stress.

[dalekphalm]

Slightly off topic, but I wanted to know why you chose to use unRAID as your base hypervisor, and then you virtualized FreeNAS anyway.

 

Why not simply use FreeNAS as the hypervisor and run the Windows 10 VM from there?

 

Or are you utilizing other features of unRAID that you prefer over FreeNAS?

[Strikermed]

On 12/8/2017 at 2:02 PM, dalekphalm said:

Slightly off topic, but I wanted to know why you chose to use unRAID as your base hypervisor, and then you virtualized FreeNAS anyway.

 

Why not simply use FreeNAS as the hypervisor and run the Windows 10 VM from there?

 

Or are you utilizing other features of unRAID that you prefer over FreeNAS?

That's what I originally intended to do.  I have the Asus Z10PE-D16 WS motherboard which has 2 CPU sockets.  I'm using 2 Xeon E5-2683 processors with 128GB ECC memory.  When I attempted to install FreeNAS whether 1 socket or both were occupied, I couldn't get it past this screen shown the video below.  

 

Knowing I wanted to utilize the FreeNAS features (as a storage medium), and not necessarily all the added features like lockers and virtualization I found another solution through UNRAID.  UNRAID made it pretty easy for an advanced user like me to set up and get FreeNAS running in a virtual machine with direct access to the drives.  It took some trouble shooting, but I got it locked in using FreeNAS 9.1.  Using a 10GbE card I can direct connect with my PC getting 500-600MB/s transfers so I can work on video projects directly from the server.  

 

The last reason I like it was just by chance... When I was setting up my VM I wanted to transfer an installation of Windows Server 2012 R2 Essentials (which currently sits dormant due to limited time to work on it and learn) on an SSD.  I also wanted to install certain VMs directly to SSD, so I have a total of 5 SSD in my system.  3 for the UNRAID array, and 2 for installations.  It just worked out well for me, and I didn't have to deal with any of the insecurity of FreeNAS not installing, and potentially not updating properly in the future.  There always seems to be bugs going from one version to another, especially with hardware.

 

[Strikermed]

Ugh!!!  I'm having the same issue again!  

 

I have 2 connections.  One is a VLAN and the other is to my general network.  All this is setup through UNRAID, and it seems that I can't get speeds over 145MB/s.

 

At one point this week I had 500-600MB/s, but now I'm back down to the wimpy slow speed.  I'm not sure what I did or can change, since the system becomes so unstable if I touch anything on the windows VM.  Anything beyond the default driver settings with the exception of the VLAN priority settings causes my connection to drop, and my VM to lock up.

 

Any help on VM to VM file transfers reaching 1Gigabit speeds would be appreciated.

 

 

I will mention, testing on an actual Windows 10 PC with a 10GbE NIC in it will transfer from that same FreeNAS with a dedicated 10GbE card at 800-950MB/s, so I'm not hitting any kind of hard drive bottleneck or anything.