iSCSI Link Aggregation

[Helly]

So i have gotten a SAN (Promise VessRAID 1830i) from work which supports 12 drives and doesn't make all that much noise. I've gotten the thing up and running now and have connected it to my 10Gbit switch with 4 cables. The SAN itself supports 1Gbit per port. My server which is connected to the same switch supports 10Gbit. So i hoped, it could use the SAN with 4Gbit. I have my doubts that it even works that way but that's why i'm here.

I copied over a 34GB file and the explorer window said it was copying at 700+MB/s. At first i thought yay it's working, but when it "finished" i noticed the network was copying at around 110MB/s and kept going after the copying window was long gone. The lights on the switch only blink for port 1 so i'm guessing its not working.

 

I did some googling and ran into MPIO. But that looks like it works by having multiple connections between the server and SAN. Not what i can do here, don't have more connections available really. Then there's also the fact that the SAN itself seems to be bundling the connections itself. Because when i turn on the link aggregation (master port 1 and 2,3,4 as slaves), all ports show up as having the same IP.

 

So the question here is, is what i want even possible? If so, then how? This is my first experience with iSCSI and getting it to even connect was hard enough, and i doubt i can do it again if i reset it :P. I honestly barely know what i'm doing, so please help XD.

[leadeater]

17 minutes ago, Helly said:

I did some googling and ran into MPIO. But that looks like it works by having multiple connections between the server and SAN. Not what i can do here, don't have more connections available really.

MPIO is what you want and is the only way which works correctly and will give you 4Gbps bandwidth read and write. On the SAN give each port their own IP, on the server install and enable MPIO then connect to the iSCSI target and enable multi-path. After this you need to add 3 more connections to the SAN but you need to specify initiator IP and target IP pairs to there is 4 unique pair paths to the SAN, one to many endpoint to destination (fan out). Make sure the multipath mode is Active-Active Round Robin.

 

You only need the 4 ports going to the switch on the SAN, server can use the single 10Gb just fine.

[Jarsky]

I think you mean NAS, a SAN is a block storage network, so generally at least 2 units with some form of switching. 

 

Anyway, that's not how LAG/Port-bonding works. You can bond them together so you have a single IP interface, however TCP will only transmit over 1 path at a time, so for a single transfer you are limited to 1Gbit, however you have 4 paths so you could have 4 different transfers all hitting it at 100MB/s at the same time. 

[unijab]

1 minute ago, Jarsky said:

, a SAN is a block storage network,

Which is exactly what iscsi is.

[leadeater]

3 minutes ago, Jarsky said:

I think you mean NAS, a SAN is a block storage network, so generally at least 2 units with some form of switching. 

 

Anyway, that's not how LAG/Port-bonding works. You can bond them together so you have a single IP interface, however TCP will only transmit over 1 path at a time, so for a single transfer you are limited to 1Gbit, however you have 4 paths so you could have 4 different transfers all hitting it at 100MB/s at the same time. 

Protocol being used is iSCSI though so it's SANish. Either way LACP doesn't do what many people expect and iSCSI specifically should not use LACP/bonding when you have the option to use MPIO.

Edited December 26, 2019 by leadeater

[Helly]

4 hours ago, leadeater said:

MPIO is what you want and is the only way which works correctly and will give you 4Gbps bandwidth read and write. On the SAN give each port their own IP, on the server install and enable MPIO then connect to the iSCSI target and enable multi-path. After this you need to add 3 more connections to the SAN but you need to specify initiator IP and target IP pairs to there is 4 unique pair paths to the SAN, one to many endpoint to destination (fan out). Make sure the multipath mode is Active-Active Round Robin.

 

You only need the 4 ports going to the switch on the SAN, server can use the single 10Gb just fine.

I think i set it up right. Followed some more specific guides for MPIO. For multiple connections i assumed i needed to use MCS. But i'm still getting unexpected results. Namely a max of around 170MB/s. All the ports are blinking so it's an improvement. But i think it could be better. Below is a screenshot of the settings. Anything i might have missed?

Or is the 170 just going to be the max for me?

 

[leadeater]

Looks like you've got it setup correctly, 170MB/s might be the maximum you can get from that SAN with those disks. Seems a bit low though. Maybe as a test reconfigure the array to RAID 0. 

[Helly]

15 hours ago, leadeater said:

Looks like you've got it setup correctly, 170MB/s might be the maximum you can get from that SAN with those disks. Seems a bit low though. Maybe as a test reconfigure the array to RAID 0. 

I thought about that. But it took 3 days to initialize the raid6. So i guess ill take the loss and keep it running like this. Thanks for the help.

[leadeater]

19 hours ago, Helly said:

I thought about that. But it took 3 days to initialize the raid6. So i guess ill take the loss and keep it running like this. Thanks for the help.

I had a thought, maybe there is a battery cache unit in the SAN and the battery is faulty. The performance you're getting is typical of a system with non functioning write-back cache which is what happens when the battery fails. Would be worth opening it up and looking to see if there is a battery.

[Helly]

On 12/28/2019 at 11:52 AM, leadeater said:

I had a thought, maybe there is a battery cache unit in the SAN and the battery is faulty. The performance you're getting is typical of a system with non functioning write-back cache which is what happens when the battery fails. Would be worth opening it up and looking to see if there is a battery.

Nope, there's no battery in it. The web interface says there's no battery detected and i've been told there never was one. So unfortunately that's not the cause. Unless no battery gives the same result. Never had that problem with my other raid cards though. Never had a battery for them either. Would it be different for each brand of card/system?

[leadeater]

20 minutes ago, Helly said:

Unless no battery gives the same result.

If there is a connector for a battery and there isn't one then yes. Higher end RAID cards have battery connectors and write-back cache so when you're using RAID 5/6 you get around 800MB/s write performance, without the battery/write-back disabled 80-200 MB/s write.