Boost Link Aggregation Write Performance

[Mulawin Galang]

Hi everyone!

 

I just setup our first server in our production company. I'm quite satisfied with the performance since everyone (our editors, animators) are working seamlessly. I just have one concern though.

 

I am running the Windows Server on a Asrock c2750d4i with 16GB of RAM with an additional 4 Gigabit NICS. Read performance is great for all our laptops and workstations but write performance is a bit slow when clients simultaneously write to the server. I am able to saturate all 6 NICS or 6GB/s to around 90%. Each client has a maximum read speed of an average of 115mb/s. But when 2 or more clients write to the server, even if their hard drives are separated on the server, they tend to divide the 115mb/s speed instead of each having 115mb/s write speed individually.

 

Is there a way I can increase the write speed to my server?

Here's my setup:

What I did was instead of the hard drives running on raid, each client PC has an assigned hard drive that is mapped to their laptop/workstation. The setup goes like this:

 

Server:
120GB SSD - OS/Media Cache
1 2TB RED Drive - Client 1
1 2TB RED Drive - Client 2
1 2TB RED Drive - Client 3

1 2TB RED Drive - Client 4

1 2TB RED Drive - Client 5

1 2TB RED Drive - Client 6

4 x 4TB RED Drive - Archives / Old Projects

All client drives are backed by a software called Bvckup to an extenal ESATA 5-bay JBOD which has 5 x 4TB drives in JBOD in real time (monitors any changes and backs them up real-time and archives deleted files just in case files were deleted accidentaly)

 

Hope you can help me! Thanks everyone!

[brwainer]

I would first check your link aggregation is working on both ends, the server and the switch it connects to. It sounds like all of your writes are going to a single gigabit port, and that might indicate an issue with the aggregation on the switch side of the link. Alternatively, check what type of aggregation you have selected if using the Server 2012 R2 default aggregation (may also happen on other OSes too), because some of the modes aren't true aggregation, and are more of using all available links to send, causing data received to still only come in on one port.

[Codymm03]

Are the workstations wired over ethernet

[d33g33]

I could be wrong but, no matter what you do with the 1gb NICs they will only be able to handle a maximum of 1gbit throughput even though you have 4 of them. You essentially have 4 redundant links. The only way to increase throughput would be to step up to 10gbit.

 

You may be able to sorta get around it if you set up 2 IP's and team 2 NICs together. If half the office is directed through 1x IP and the other half the other you may be able to create 2x 1gbit pipes. Though I don't fully know if that is correct.

[unijab]

nevermind

[brwainer]

I could be wrong but, no matter what you do with the 1gb NICs they will only be able to handle a maximum of 1gbit throughput even though you have 4 of them. You essentially have 4 redundant links. The only way to increase throughput would be to step up to 10gbit.

 

You may be able to sorta get around it if you set up 2 IP's and team 2 NICs together. If half the office is directed through 1x IP and the other half the other you may be able to create 2x 1gbit pipes. Though I don't fully know if that is correct.

Proper link aggregation, aka teaming, if set up on both ends of the multiple parallel links, will combine the links into one big virtual link. If there is 4+ links between a server and a switch, and 4+ clients connected to that same switch or others in the network without bottlenecks, then the switch will take traffic from the 4 clients and send it down the next available link to the server. The server will do the same, data being sent to any of the 4 clients will be sent down whatever the next available link is. But this all relies on it being properly set up as true aggregation.

[Mulawin Galang]

I would first check your link aggregation is working on both ends, the server and the switch it connects to. It sounds like all of your writes are going to a single gigabit port, and that might indicate an issue with the aggregation on the switch side of the link. Alternatively, check what type of aggregation you have selected if using the Server 2012 R2 default aggregation (may also happen on other OSes too), because some of the modes aren't true aggregation, and are more of using all available links to send, causing data received to still only come in on one port.

Everything's connected on a Single 24-port Gigabit switch with LAG, specifically the Netgear  JGS524Ev2. All 6 Gigabit LAN Ports of my server (2 onboard, 4 pci) are in a LAG Group. On Windows Server it's running on Static Teaming, Address Hash giving me a 6GB/s Link. Did I miss anything on my setup?

Thanks!

[Mulawin Galang]

Are the workstations wired over ethernet

Yes! All workstation are wired via Gigabit Ethernet on a Single 24-port Netgear  JGS524Ev2 Switch.

[Mulawin Galang]

I could be wrong but, no matter what you do with the 1gb NICs they will only be able to handle a maximum of 1gbit throughput even though you have 4 of them. You essentially have 4 redundant links. The only way to increase throughput would be to step up to 10gbit.

 

You may be able to sorta get around it if you set up 2 IP's and team 2 NICs together. If half the office is directed through 1x IP and the other half the other you may be able to create 2x 1gbit pipes. Though I don't fully know if that is correct.

Hmm, 10GBs is just way too expensive for us at the moment. We're quite happy with the overall performance for every client. It's just the  write speed that's making me concerned because if one of the clients ingests the footage to the server, write speed of other clients slow down.

[unijab]

if you dont mind doing some testing...

 

remove the lag and the teaming...

 

setup the network load balancer over the 6 NICs with equal priority for each IP

[d33g33]

Proper link aggregation, aka teaming, if set up on both ends of the multiple parallel links, will combine the links into one big virtual link. If there is 4+ links between a server and a switch, and 4+ clients connected to that same switch or others in the network without bottlenecks, then the switch will take traffic from the 4 clients and send it down the next available link to the server. The server will do the same, data being sent to any of the 4 clients will be sent down whatever the next available link is. But this all relies on it being properly set up as true aggregation.

Fair enough, I dont know enough about it. I tried to team 2 gbit NICs through to my NAS to ideally get a 2gbit pipe. When I asked about it the answer I got was it will only ever go 1gbit from client to client.

[brwainer]

Everything's connected on a Single 24-port Gigabit switch with LAG, specifically the Netgear JGS524Ev2. All 6 Gigabit LAN Ports of my server (2 onboard, 4 pci) are in a LAG Group. On Windows Server it's running on Static Teaming, Address Hash giving me a 6GB/s Link. Did I miss anything on my setup?

Thanks!

Sounds right so far, but try the other Hash types available. I did research into them a while back but haven't looked at it recently. I haven't needed to do aggregation on my server yet, but this is making me curious to try it out again and duplicate your results.

[brwainer]

Fair enough, I dont know enough about it. I tried to team 2 gbit NICs through to my NAS to ideally get a 2gbit pipe. When I asked about it the answer I got was it will only ever go 1gbit from client to client.

Sounds right, each client has an individual limit of 1Gbps up and 1Gbps down on their connection to the network. The point of aggregating 2 links on the NAS is to allow two clients to both get 1Gb/s up or down at the same time.