SMB3.0 Multichannel VS Etherchannel

[Windows7ge]

I have in the works the plan to aggregate two 10Gbit interfaces to create a single 20Gbit link but now I'm a little bit confused on how I bond the two. I'm using FreeNAS so someone I met on the forum suggested SMB3.0 with Multichannel. At the same time Etherchannel also appears to be an option as far as I understand in the FreeNAS documentation. I have quite the lack of understanding of both so I'm asking the question of what are the pros and cons of both? 

 

Also as a side note. It has been mentioned in my own studies that it is potentially important that the NIC(s) support RSS or RDMA. How important is this and where can I find if the NIC supports it?

[brwainer]

what model is the NICs, and what are they going to be attached to? If its a switch, what model is the switch? These will change what options you have or should take.

[Levisallanon]

You will ofcourse need a managed switch. From which vendor is the switch you are connecting too?
And you would need to find which model your NIC is as @brwainer says to see if it supports etherchannel.

[brwainer]

7 minutes ago, Levisallanon said:

You will ofcourse need a managed switch. From which vendor is the switch you are connecting too?
And you would need to find which model your NIC is as @brwainer says to see if it supports etherchannel.

You don’t need a managed switch for SMB3 multichannel.

[Levisallanon]

1 minute ago, brwainer said:

You don’t need a managed switch for SMB3 multichannel.

I don't know for multichannel, but for etherchannel you do.

[Windows7ge]

7 hours ago, brwainer said:

what model is the NICs, and what are they going to be attached to? If its a switch, what model is the switch? These will change what options you have or should take.

The current NICs I'm looking at are the SEDNA - PCIE 8X Dual 10 Gigabit Ethernet Server Adapter - INTEL 82599 Chipset though I'm probably going to look for a cheaper alternative.

The switch I'm looking at is the Ubiquiti Networks US-16-XG-US

[Windows7ge]

6 hours ago, Levisallanon said:

You will ofcourse need a managed switch. From which vendor is the switch you are connecting too?
And you would need to find which model your NIC is as @brwainer says to see if it supports etherchannel.

The plan is to connect to a Ubiquiti Networks US-16-XG-US

 

As far as I understand SMB is mostly plug'n'play with the exception of setting it up on the server side but Etherchannel requires more configuring.

 

Another point is that I've been told a managed switch isn't necessary for SMB and that windows will auto-configure it so with FreeNAS on one end and a Windows 10 client on the other it seems like my better option but I want to know about etherchannel and I have to verify hardware support for whichever protocol I use.

 

I've also been told that SMB prefers RDMA over RSS if it can be had.

[brwainer]

7 hours ago, Levisallanon said:

I don't know for multichannel, but for etherchannel you do.

SMB multichannel looks at the multiple interfaces on the computer on one end or the other, and then opens multiple connections between the computers using the available adapters. it will work just having multiple connections to the same dumb switch.

46 minutes ago, Windows7ge said:

The current NICs I'm looking at are the SEDNA - PCIE 8X Dual 10 Gigabit Ethernet Server Adapter - INTEL 82599 Chipset though I'm probably going to look for a cheaper alternative.

The Intel 82599 supports RSS but not RDMA. It is a somewhat older chipset, first released in 2008. If you want options that support RDMA but on a budget, you'll have to look at the used market. I don't think the new Aquantia chips support it either, but I'm not sure about that.

36 minutes ago, Windows7ge said:

The plan is to connect to a Ubiquiti Networks US-16-XG-US

That switch will work for what you are trying to do.

38 minutes ago, Windows7ge said:

I've also been told that SMB prefers RDMA over RSS if it can be had.

RDMA is almost always better, but I'm not aware of any card that has RDMA and not RSS. It is possible taht you can't use both at the same time, but every chipset that supports RDMA should be able to be used in RSS mode also.

[Windows7ge]

7 minutes ago, brwainer said:

The Intel 82599 supports RSS but not RDMA. It is a somewhat older chipset, first released in 2008. If you want options that support RDMA but on a budget, you'll have to look at the used market. I don't think the new Aquantia chips support it either, but I'm not sure about that.

Is RDMA really that enterprise-y? If SMB can run fine with RSS then I won't go though the hassle of finding a NIC that supports RDMA. I haven't seen any documentation or forums saying that it has to be RDMA just that it's preferred. I have to buy two of these cards so I'd like to keep the cost around or under $200/per. I spotted some dual port NIC's for about 1/2 the cost so I'm probably going to find something else but I'll make sure it supports RSS. To use SMB Multichannel is there any other required hardware support? I'm told SMB is "best buddies" with windows so it should auto-configure and FreeNAS has an SMB service that I'll have to configure but other than that...

 

Saying that RDMA is always better. Why is that? Is it newer? More up to date? Less buggy? Better features? Better at maxing out the aggregated link? What makes it superior if it can be had?

 

As far as I've been shown for an enthusiast home setup SMB seems like the better option. Less configuring, requires less expensive hardware.

[brwainer]

1 hour ago, Windows7ge said:

Is RDMA really that enterprise-y? If SMB can run fine with RSS then I won't go though the hassle of finding a NIC that supports RDMA. I haven't seen any documentation or forums saying that it has to be RDMA just that it's preferred. I have to buy two of these cards so I'd like to keep the cost around or under $200/per. I spotted some dual port NIC's for about 1/2 the cost so I'm probably going to find something else but I'll make sure it supports RSS. To use SMB Multichannel is there any other required hardware support? I'm told SMB is "best buddies" with windows so it should auto-configure and FreeNAS has an SMB service that I'll have to configure but other than that...

 

Saying that RDMA is always better. Why is that? Is it newer? More up to date? Less buggy? Better features? Better at maxing out the aggregated link? What makes it superior if it can be had?

 

As far as I've been shown for an enthusiast home setup SMB seems like the better option. Less configuring, requires less expensive hardware.

RDMA (remote direct memory access) lets the NIC put the incoming traffic directly into RAM, and then the OS can handle the data from there at its leisure. Without it, the CPU actually has to be involved in transferring the data from the NIC's incoming buffer to RAM. I'm not sure whether RDMA does something similar for outgoing traffic, I assume it does. RDMA drastically reduces the CPU load associated with data transfers. Unless you hit a CPU limit (traditionally single threaded, RSS allows multiple CPUs to be involved in receiving data) RDMA doesn't increase your speeds.

[Windows7ge]

19 minutes ago, brwainer said:

RDMA (remote direct memory access) lets the NIC put the incoming traffic directly into RAM, and then the OS can handle the data from there at its leisure. Without it, the CPU actually has to be involved in transferring the data from the NIC's incoming buffer to RAM. I'm not sure whether RDMA does something similar for outgoing traffic, I assume it does. RDMA drastically reduces the CPU load associated with data transfers. Unless you hit a CPU limit (traditionally single threaded, RSS allows multiple CPUs to be involved in receiving data) RDMA doesn't increase your speeds.

So RDMA is preferred because it reduces the amount of processing the CPU has to do to move the files between systems. If this is a multi threaded workload then I'm not concerned (server is running dual Xeon E5 2670's, desktop is a 5960X). This isn't a professional operation that I'm doing where CPU efficiency is important. So long as it doesn't bottleneck the link I'm happy.

 

If RSS primary function is to allow multiple cores/threads to have a hand in the file transfer is it even a serious requirement to form the link? Without it I'm to theorize that I'd still be able to run the aggregated config but I'd see both on client and server side a single thread being used, probably maxed out at 100% trying to push 20Gbit (something a single core probably cannot do) so RSS sounds vital at least at these speeds.

[brwainer]

34 minutes ago, Windows7ge said:

If RSS primary function is to allow multiple cores/threads to have a hand in the file transfer is it even a serious requirement to form the link? Without it I'm to theorize that I'd still be able to run the aggregated config but I'd see both on client and server side a single thread being used, probably maxed out at 100% trying to push 20Gbit (something a single core probably cannot do) so RSS sounds vital at least at these speeds.

RSS isn't a requirement for anythiong, but the NIC you were looking at supports it, and I believe it gets used automatically (I am probably wrong on that). Also, having RSS or not has nothing to do with whether you are using aggregated connections.

[leadeater]

RDMA is not that important unless you need extremely low latency. RDMA is still only on the most expensive NICs and you need it on both ends.

 

SMB 3 Multichannel only works for SMB traffic, it also requires the most up to date version of SAMBA and you need to enable it and it's still an experimental feature.

 

NIC Teaming/Aggregation is only effective at increasing bandwidth for multiple devices, not really a singular point to point connection speed increase.

[Windows7ge]

4 hours ago, brwainer said:

RSS isn't a requirement for anythiong, but the NIC you were looking at supports it, and I believe it gets used automatically (I am probably wrong on that). Also, having RSS or not has nothing to do with whether you are using aggregated connections.

It just allows more than one core or thread to handle the session as to divide up the workload across the CPU. Still sounds like a useful feature to have when using faster than average connections.

[Windows7ge]

2 hours ago, leadeater said:

SMB 3 Multichannel only works for SMB traffic, it also requires the most up to date version of SAMBA and you need to enable it and it's still an experimental feature.

I updated FreeNAS from 9.3.1 to 11.0-U3 a few months ago. Hopefully that updated SAMBA with it.

 

2 hours ago, leadeater said:

NIC Teaming/Aggregation is only effective at increasing bandwidth for multiple devices, not really a singular point to point connection speed increase.

This is something I'm already aware of when it comes to LACP but I'm not sure if Etherchannel falls under the same principal or if it will bond multiple links like SMB3 multichannel.

 

Also I'm thinking Etherchannel is Cisco proprietary and like hell I'm spending the amount of money necessary for a 10G Cisco switch with at least 8 ports.

[leadeater]

1 minute ago, Windows7ge said:

This is something I'm already aware of when it comes to LACP but I'm not sure if Etherchannel falls under the same principal or if it will bond multiple links like SMB3 multichannel.

It does, Etherchannel is just another name for NIC Teaming/Link Aggregation etc. LACP is actually a protocol for automatic configuration of ports in an Aggregation group.

[Windows7ge]

38 minutes ago, leadeater said:

It does, Etherchannel is just another name for NIC Teaming/Link Aggregation etc. LACP is actually a protocol for automatic configuration of ports in an Aggregation group.

That explains why every google search kept taking me to Link Aggregation which isn't what I wanted and Link Aggregation Control Protocol. I need to stop using acronyms when I can't fully recall what they represent. Hopefully I got my point across that's what's important.

 

So SMB3 is experimental? I've been researching it and apparently its been considered experimental for quite some time now. I'm curious to know what issues I'll have but if everything works how everyone says it should then all I have to do is setup the interface addresses set the SMB maximum protocol to smb3.0_11 and plug everything in then it should set up everything else itself.

[leadeater]

1 minute ago, Windows7ge said:

So SMB3 is experimental?

Only under *NIX operating systems that use the SAMBA package to create SMB shares, under Windows it's completely stable.

 

You can use it under *NIX operating systems and it does work but there might be bugs, it's just currently not considered stable.

[Windows7ge]

5 minutes ago, leadeater said:

Only under *NIX operating systems that use the SAMBA package to create SMB shares, under Windows it's completely stable.

 

You can use it under *NIX operating systems and it does work but there might be bugs, it's just currently not considered stable.

My research says that SMB was designed for Windows OS but versions of it were created for Linux and MacOS so they could all use the same protocol to communicate for file sharing over a network. I'll have to look up what people's experiences have been using it on freeNAS.

[brwainer]

2 hours ago, Windows7ge said:

My research says that SMB was designed for Windows OS but versions of it were created for Linux and MacOS so they could all use the same protocol to communicate for file sharing over a network. I'll have to look up what people's experiences have been using it on freeNAS.

as a general technology, SAMBA is stable and reliable, its the more advance features of SMB3 that aren't quite reliable yet, including Multichannel.

[Windows7ge]

59 minutes ago, brwainer said:

as a general technology, SAMBA is stable and reliable, its the more advance features of SMB3 that aren't quite reliable yet, including Multichannel.

Searching online the past day has shown very inconclusive results. It looks like if I want answers I'll have to test it myself. I don't have the hardware to test 10Gbit bonding but I should be able to test 1Gbit bonding using the built in quad Gbit NIC and the built in dual NIC on another motherboard and using a dumb 1G switch. If I can get that to work then in theory I should just have to apply the same config to the 10G network