iSCSI connections

[JStan]

Currently I've got a NAS datastore (running Truenas) connected to my VM machine over 4 x 1 GB connections.  I'm upgrading my servers (both sides) which will include 10Gb network cards.

Do I still need 4 direct connections?

 

Any thoughts would be really appreciated.  Thanks.

[Eigenvektor]

I assume you mean 4x 1 Gb (not GB) connections.

 

What kind of drives do you have in both systems? That will determine whether 4x 1 GbE was ever necessary or whether you'd still see any performance benefit for going with more than one 10 GbE connection.

 

10 GbE has a theoretical maximum throughput of 1.25 GB/s, so you can saturate it with a PCIe 3.0 NVMe (or faster). Though the remaining question is if your use case is limited at that speed and benefits from an increase.

[JStan]

Thanks

5 hours ago, Eigenvektor said:

I assume you mean 4x 1 Gb (not GB) connections.

Currently it is 4x 1Gb

I'm buying a second hand Supermicro AS-2023US-TR4 I'm planning to drop in a 10Gb (4 port) card.  So I am trying to work out how many of these need to be dedicated to connections to the VM machine. 

 

5 hours ago, Eigenvektor said:

What kind of drives do you have in both systems?

4 NVMe drives and the rest will be a mixture of SAS and SATA

 

I assume with a round-robin setting it will use as much bandwidth as it needs.  So I can have fewer connections (2?) and add more if I'm hitting saturation.  Sorry a naive question - is this a question of calculating a theoretical throughput (and so assigning X connections) or should I be measuring this (somehow)?

 

Thanks!

[Eigenvektor]

7 hours ago, JStan said:

Sorry a naive question - is this a question of calculating a theoretical throughput (and so assigning X connections) or should I be measuring this (somehow)?

A calculation based on theoretical numbers is potentially "good enough" already. A benchmark would give you more realistic results, but it also means you'll need a good understanding of your performance profile. Is disk access predominantly sequential, random or some mix of the two? Is access primarily limited by throughput or latency? Is it limited by the disk(s), the network card, the CPU, memory, …? So it might be somewhat difficult to set up a realistic scenario that gives usable results.

 

Since you have a working system, monitoring current hardware usage is likely the best way to get an understanding of how the hardware is used and where the limitations are. That would help determine whether bonding multiple network cards is worthwhile, or your simply increasing performance in an area that isn't actually the limiting factor. For example if compression is involved, you might also be limited by CPU performance. Or maybe RAM performance, etc.

 

Based on theoretical numbers, I would expect 3x 10 GbE to be all you need, provided you're bound by disk write speed, rather than something else. A PCIe 3.0 NVMe should give you around 3 GB/s here. 10 GbE should top out around 1.25 GB/s, if we're being pessimistic we could make that an even 1 GB/s (e.g. TCP/IP protocol overhead).

[Jarsky]

If the 1gbE connections are in a LAG group (aggregate/bonded) then i would keep them in place; and use those for your iSCSi Connector and use the 10gbE aggregated for your Production interface (VM networks, accessing TrueNAS,etc...)...or vice versa depending on the work load and if you need to prioritise faster storage or networks. It's quite common to split these when using Software iSCSI adapters. Or just remove the 1gbE; and run the 10gbE independant...1 for iSCSi traffic, 1 for network. 

 

Edit: Sorry just saw its a 4 port card...I would create 2 LAG's and just get rid of the 1gbE all together