Networking ProTips for @LinusTech (NIC Teaming, Link Aggregation, Load Balancing, and SMB Multi Channel -- Oh My!)
TL;DR
Forget about NIC Teaming / Link Aggregation (LACP). Configure each physical interface with it's own IP address (on the same subnet) and allow SMB Multi Channel to do it's multiple (TCP) connection thing.
I enjoyed your rant about networking -- sorry to hear SMB multi channel isn't working for you. Here is some background context to help explain why (I think) you're not able to exceed one link throughput for your file transfer test.
Disclaimer: I don't have first hand experience with SMB 3.0 multi channel, but I have a strong background in enterprise networking including working on the test team for a large NAS product that featured support for LACP as well as IP-based failover mechanisms. My suggestions below are based on how I guess SMB 3.0 works.
Load Balancing strategies at layer 2 (Ethernet) have historically been very simplistic in the way they work, and they all relied on a large number of clients (different MAC addresses, IP addresses, or TCP ports) connecting to the server so they had some distribution that could be deterministically spread across the available interfaces. This works ok in the client-server case where there are many clients connecting to one server but doesn't help the case of one fast client connecting to one fast server (the conversation will always take the same path through the trunk). Moreover, the server can't control (by itself) which interfaces will be used for incoming traffic (only which interfaces it uses to send out traffic). For both sides of the 'trunk' to load-balance, the switch must be involved (hence LACP).
A brief read-through of the SMB 3.0 multi channel feature seems to indicate that by using multiple TCP connections (to separate IPs, each on their own interface), the protocol can achieve higher aggregate throughput than a single link. The documentation claims to support lots of variations on this theme, including support for 'NIC Teaming' aka Link Aggregation (with or without LACP), however - it's unclear if the multi channel part of the system (layers 3/4) can actually make use of all those aggregated links (and I strongly suspect it can't). Usually when you configure 'Link Aggregation' on the host, you specify 2 or more physical links to form a single logical interface (aka Team aka Aggregation). You then assign one IP address to this group. The problem (I'm guessing) is that the SMB part of the stack sees the 1 IP address and interprets that as 1 link and doesn't attempt to do it's multi-channel thing.
So, to summarize:
- Layer 2 link aggregation only provides a benefit in 1-to-many topologies
- LACP requires switch support (smart switch) to distribute load from many clients to a multi-homed server
- SMB 3.0 multi channel is a different thing than layer 2 stuff, and works by making multiple TCP connections to different IP addresses on different NICs
- by configuring layer 2 aggregations and only presenting those logical interfaces to higher layers, you're effectively reducing the number of interfaces that SMB thinks it has
So, my suggestion would be to abandon all the layer 2 stuff (teaming, LACP, aggregations) and configure each physical interface with it's own IP address on the subnet (on both server and client). This is the simplest config that SMB multi channel claims to support and it has the added benefit of not requiring any switch-side support.
Good Luck.
