Helping Linus with Microsoft SMB 3.0 multipath/channel

[.:MARK:.]

How are you connecting from the client to the server?

 

Are you typing an IP address or hostname to connect to the share?

if you're using the hostname, have you setup multiple dns entries for the server?

 

Isn't that unnecessary? I thought the point of multipath is that it will query the server and find all relevant interface IPs to create TCP sessions with.

[MountainPro]

In my testing the server is only being used for one client. It is not in a cluster, and no they are not domain members.

Linus I understand you are using windows, I have been able to bind my NICs by running ubuntu on the server and then opensource smb software, which works without issues. I don't know if you would ever want to look into it but that is just another option. I hope you are able to get it to work! good luck keep us posted! 

[unijab]

Isn't that unnecessary? I thought the point of multipath is that it will query the server and find all relevant interface IPs to create TCP sessions with.

Multipath is something different from multichannel. But I know what you mean.

 

But the examples I have seen on youtube were using host names to connect to shares in a domain environment.

Making host records for each server interface IP is still a good idea.

[.:MARK:.]

Multipath is something different from multichannel. But I know what you mean.

 

But the examples I have seen on youtube were using host names to connect to shares in a domain environment.

Making host records for each server interface IP is still a good idea.

 

Yeah multichannel, you know what I mean, but I don't know if Linus is running an Active Directory domain controller.

[unijab]

Which is why i havent suggested connecting to a domain yet.

 

Only asked about dns records and how he was connecting.

[krapplejaxx]

I tuned SMB2 using a guide from Dennis Wood. He also has one on SMB3, and it looks like it's worth a shot: http://www.cinevate.com/blog/confessions-of-a-10-gbe-network-newbie-part-5-smb3/

[unijab]

In theory multichannel should work without rdma

Also rss is enabled by default on them

 

Linus:

I did a test today with 2 virtual machines. 

Server 2012 R2

2 virtual nics each vm

created a default nic team (in server manager)

test copied a 1.87GB file

 

I could see by using netstat that it created 4 connections, im going to add another thread to each VM and see if more smb connections are created....

[unijab]

continued from above....

 

removed nic team..

 

Have 3 cpu threads and 3 virtual nics for each VM now.

 

Removing the team allowed 4 connections per NIC.

On file copy: Viewing the task manager you can clearly see the traffic spread across each of the NICs.

 

@LinusTech

 

[xurtis]

Hi Guys & Linus,

 

Thought I could help out after spending a day fiddling around with various services on WS2008 all day, specifically routing, so I've been running into some similar style of issues, so I thought I could help out. If you do take the time to be patient and read through this Linus we may not have a solution by the end, but we could well have ruled out a few possibilities.

 

Why does Windows take so long to install?

 

Anyway, 2 Windows 8.1 w/ Update VM installs later (yes, I could have just cloned them but shut up), time for some testing.

 

For reference, here is the set up of the two machines, both running in VirtualBox under Ubuntu.

 

Windows 8.1 x86 was used for both machines as I did not have the ISO files for x86_64 8.1 or 2012 at hand, and Australian internet is bad enough here that I wasn't going to wait overnight for the download.

 

Network

VirtualBox Internal Network: 10.0.2.0/24

Gateway: 10.0.2.1

 

Machine 1 - Client

OS: Windows 8.1 w/ Update x86

CPU: (2 Cores) i5-4690 @ 3.5GHz

Memory: 2GB

Network 1: Intel Pro/1000 MT Desktop - Connected to NAT / static to 10.0.2.4

Network 2: Intel Pro/1000 MT Desktop - Connected to NAT / static to 10.0.2.5

 

Machine 2 - Server

OS: Windows 8.1 w/ Update x86

CPU: (4 Cores) i5-4690 @ 3.5GHz

Memory: 4GB

Network 1: Intel Pro/1000 MT Desktop - Connected to NAT / static to 10.0.2.6

Network 2: Intel Pro/1000 MT Desktop - Connected to NAT / static to 10.0.2.7

Network 3: Intel Pro/1000 MT Desktop - Connected to NAT / static to 10.0.2.8

Network 4: Intel Pro/1000 MT Desktop - Connected to NAT / static to 10.0.2.9

 

In order to PING test, the following Firewall Inbound rules were enabled on both machines:

• File and Printer Sharing (Echo Request - ICMPv4-In) Private, Public
• File and Printer Sharing (Echo Request - ICMPv6-In) Private, Public​

A folder was created on the Server machine under the C:\ as follows:

Folder: C:\SHARE

Shared via: Properties > Sharing > Advanced Sharing...

Checked "Share this folder"

Pressed OK for default settings

 

The following Inbound rule was enabled on the Server machine:

• File and Printer Sharing (SMB-In) Private, Public

The share was then mapped to the Z: drive on the Client machine with \\10.0.2.6\share\

 

A large file was then created with the following small batch file, cancelling half way:

@Echo off

echo test.txt >> test.txt

large.bat

 

It's a times like these that I miss /dev/random. The final file was at least 5GB in size, test.txt was initially just ABCDEFGHIJKLMNOP1234567890.

 

This file was then copied from the Server machine to the Client. The following netstat of each shows the connections after the file was transferred:

 

The mapped drive was then disconnected, leaving the following connections:

 

Finally, I navigated to \\10.0.2.6\share\ in explorer without mapping the share, and copied the file again, producing the following connections:

 

And, to show that all NICs are being used, the graphs from task manager:

 

Unlike @unijab above, however, I was only ever getting a single connection per NIC on the server, with two per NIC on the client. This was probably determined by the number of cores dedicated to the server (4).

 

As can be seen, the connections for multichannel are being created and used, both for a mapped drive and for a direct connection. The speeds I experienced were, however, sub-optimal, sticking around 27MB/s, although this is likely to simply be due to the virtual machine environment. Nothing fancy had to be set up on either machine, except for the the firewall.

 

My recommendation would be to repeat this process, making sure to note the following:

• Verify that each interface is assigned a separate IP address, note which addresses are assigned to which interface.
• Before testing copy, verify that you can PING between hosts for every combination of IP addresses, just to be sure.
• Verify that the appropriate firewall rules are active, if you are using domain bound computers, ensure that the rules appropriate for Domain are enabled with  appropriate Remote Address access (All, for IP addresses on different subnets or Local Subnet if the IPs are on the same subnet). If you are not using domain bound computers, ensure that the appropriate rules for Private, Public are enabled with the same Remote Address access specified.
• File and Printer Sharing (Echo Request - ICMPv4-In) Private, Public / Domain
• File and Printer Sharing (Echo Request - ICMPv6-In) Private, Public​  / Domain
• File and Printer Sharing (SMB-In) Private, Public / Domain
• Try copying files from both a mapped drive and directly from the share and make sure to not the network connections produced by netstat as a result. It should not be hard to see what connections are used in each as they remain open for a few minutes after the transfer finishes.

 

All this after spending, literally, my entire day routing and networking physical and virtual machines with windows server and CentOS...

 

Anyway, I really hope this helps. I can't wait to see you guys up and running in the new office.

[.:MARK:.]

-snip-

 

 

I agree that this is the way to do it, as I have suggested briefly. Do you think that if this were tested on an ESXi environment with both VMs having 4 cores and VMXNET3 NICs, we could get 40Gb/s plus throughput?

[xurtis]

I think that, even with a hypervisor, you wouldn't be able to get full 40Gb/s, you might get close if your machine had a physical core fore each network interface in all of the VMs plus, so maybe 8 or 12 Core machine.

 

I am wondering if the firewall could be causing Linus more issues than one might first anticipate. By default on Windows, it appears that all ICMP traffic is blocked. Multichannel might use PING during the handshake to determine which IPs can be used to access the share. It may provide a list of known IPs  for the server to the client, and the client may PING, or something similar using ICMP traffic, each of those IPs to determine which ones it has access to before initiating a connection with them.

 

Microsoft does seem to think ICMP traffic is important to File and Printer Sharing, grouping under the same heading as all F&P services in the firewall rules. I have also noticed that, for non domain members, rules don't seem to get activated automatically such as when they might do in a domain.

 

I am also curios about the server Linus is using in this case. if it's the extreme NAS setup, I'd like to know what processor he's using along with how many PCIe channels IT has versus the amount of channels used by the NIC plus all the RAID cards. If the RAID cards are using most of the lanes, the NIC might not be left with a full 8x connection to the processor.

[911delorean]

http://www.intel.com/support/network/sb/cs-009747.htm

http://www.intel.com/support/network/sb/CS-032486.htm

The links are to some Intel support pages that might help.

http://blogs.technet.com/b/josebda/archive/2012/05/13/the-basics-of-smb-multichannel-a-feature-of-windows-server-2012-and-smb-3-0.aspx

And this other link was just a blog I found while searching, I didn't read it thoroughly though.

Sorry if this post is a bit noobish i'm kinda new on the forum, posting wise and networking isn't my strongest suit. Anyway, who knows maybe it could be something to do with the NIC or the drivers itself? Have you tried linking up individual LAN ports? Like the ones that are built into the motherboard if there are any multiple ones and see if it connects that way? Or even doing a direct connection with Crossover cables instead?

[dajinn]

@LinusTech

@Slick

 

Did you guys ever make any progress on this? I am having a similar issue where using teaming, whether it be in "switch independent" or "LACP" mode(with LAG/LACP properly config'd at the switch)doesn't result in aggregate multichannel throughput from SMB shares. It only seems to work if I un-team the NICs or just team only the client NICs.