NAS & Link Aggregation

[asheenlevrai]

Hi

 

I have a question about how data transfer works...

Let's imagine a home NAS with 2 HDDs in RAID1 and a GbE connection.

 

A) Each HDD is capable of ~150MBps (read/write)

1) -> Does RAID1 mean the theoretical max for reads is going to be around 300MBps (both drives can be accessed simultaneously)

2) -> Does RAID1 mean the theoretical max for writes is going to be below 150MBps (around 75MBps?) since both drives need to be written to simultaneously? Or will it stay around 150MBps?

 

B) 1GbE = 125MBps so this will not be a bottleneck, right?

 

Now enters link aggregation (IEEE 802.3ad). Let's say 4x 1GbE NICs. IIUC this means up to 4 concurrent clients could access the NAS at 1GbE, no one can go above 1GbE since the connections are not "pooled".

 

Now let's imagine 4 clients connect simultaneously to the NAS and start a large transfer of data (they read data from the NAS). If the answer to 1-A above is 300MBps, could all 4 clients actually get 125MBps? Or do these 300MBps need to be shared between the 4 clients?

 

Now let's say we remove the bottleneck from the disk array by replacing these HDDs with sATA SSDs:

C) Each SSD is capable of 500MBps (read/write)

3) -> RAID1, so 1000MBps reads?

 

Now, with link aggregation, could all 4 clients reach 125MBps reads? I guess so, right?

 

I imagine I probably made several mistakes in my reasoning.

Please help and correct me.

 

Thank you very much in advance for sharing your knowledge.

Best,

-a-

[Lurick]

1GbE is 125MB/s

RAID1 means you get no benefit to read or writes and no penalty either (iirc) but just a second copy on the second drive.

 

[Electronics Wizardy]

9 minutes ago, asheenlevrai said:

1) -> Does RAID1 mean the theoretical max for reads is going to be around 300MBps (both drives can be accessed simultaneously)

 

Yup, but I very rarely see those speeds. Also depends on the implemtation.

 

9 minutes ago, asheenlevrai said:

2) -> Does RAID1 mean the theoretical max for writes is going to be below 150MBps (around 75MBps?) since both drives need to be written to simultaneously? Or will it stay around 150MBps?

Raid 1 would have the 

 

10 minutes ago, asheenlevrai said:

 

Now enters link aggregation (IEEE 802.3ad). Let's say 4x 1GbE NICs, all connected to a 10GbE router. IIUC this means up to 4 concurrent clients could access the NAS at 1GbE, no one can go above 1GbE since the connections are not "pooled".

 

Or you can pool all the connections using something like SMBv3 multichannel. THen no network config is needed.

 

10 minutes ago, asheenlevrai said:

Now, with link aggregation, could all 4 clients reach 250MBps reads? I guess so, right?

 

You probably want smbv3 multichannel. It will work much better here, and it won't need any network configuration.

[kokosnh]

If you have already 10Gbps Switch, then just buy a NAS with 10Gbps port, or a adin 10Gbps NIC for it. 
iscsi mpio and SMB multichannel is generaly use, to avoid spending money on faster network hardware. If you already have that network infrastructure, just use it.  Diagnosing leter the performance issues will be easier.

[asheenlevrai]

14 hours ago, Lurick said:

1GbE is 125MB/s

 

 

indeed!  I'll fix that in the OP. Thanks

[asheenlevrai]

14 hours ago, Electronics Wizardy said:

Raid 1 would have the 

 

Seems like part of your reply was cropped

14 hours ago, Electronics Wizardy said:

 

Or you can pool all the connections using something like SMBv3 multichannel. THen no network config is needed.

 

You probably want smbv3 multichannel. It will work much better here, and it won't need any network configuration.

I need to look into smbv3 then. I never heard about it and didn't know such a thing was possible (I was always told link aggregation couldn't break the limit of a single connection). I also need to check if my NAS is compatible with smbv3.

 

Thanks a lot

[asheenlevrai]

9 hours ago, kokosnh said:

If you have already 10Gbps Switch, then just buy a NAS with 10Gbps port, or a adin 10Gbps NIC for it. 
iscsi mpio and SMB multichannel is generaly use, to avoid spending money on faster network hardware. If you already have that network infrastructure, just use it.  Diagnosing leter the performance issues will be easier.

You are right. This was a mistake of mine to include a 10GbE switch in that example. Let's say none of the network is compatible with 10GbE.

I'll change that in the OP.

[Electronics Wizardy]

2 minutes ago, asheenlevrai said:

I need to look into smbv3 then. I never heard about it and didn't know such a thing was possible (I was always told link aggregation couldn't break the limit of a single connection). I also need to check if my NAS is compatible with smbv3.

 

Thanks a lot

what nas do you have?

 

What switch and network gear are you using?

 

Id try to use 10gbe as its just simpler than multiple links for home use.

[asheenlevrai]

1 minute ago, Electronics Wizardy said:

what nas do you have?

 

What switch and network gear are you using?

 

Id try to use 10gbe as its just simpler than multiple links for home use.

Mostly Synologys. Most of them do not support AIC.

Network is not compatible with 10GbE AFAIK and I cannot modify it.

Thanks

[Electronics Wizardy]

1 minute ago, asheenlevrai said:

Mostly Synologys. Most of them do not support AIC.

Network is not compatible with 10GbE AFAIK and I cannot modify it.

Thanks

Then I know there are some hacks to get smbv3 multichannel working on them.

 

But you may be cpu limited on those before hitting network/disk limits on some models.

 

 

[asheenlevrai]

10 minutes ago, Electronics Wizardy said:

Then I know there are some hacks to get smbv3 multichannel working on them.

 

But you may be cpu limited on those before hitting network/disk limits on some models.

 

 

Thanks

 

Now back to my original question, If we ignore the network limitation, would the 4 clients accessing the array simultaneously need to share the max throughput of the disk array? Or can the disk provide the same transfer speed to all clients in parallel.

In the OP I mentioned : "Now let's imagine 4 clients connect simultaneously to the NAS and start a large transfer of data (they read data from the NAS). If the answer to 1-A above is 300MBps [limit of the disk array], could all 4 clients actually get 125MBps [limit of the network connection]? Or do these 300MBps need to be shared between the 4 clients?"[so 75MBps per client if all 4 access the NAS at the same time].

 

How can I evaluate how the CPU would be limiting here? Is there any metric that could let me know about this?

[Electronics Wizardy]

1 minute ago, asheenlevrai said:

 

In the OP I mentioned : "Now let's imagine 4 clients connect simultaneously to the NAS and start a large transfer of data (they read data from the NAS). If the answer to 1-A above is 300MBps [limit of the disk array], could all 4 clients actually get 125MBps [limit of the network connection]? Or do these 300MBps need to be shared between the 4 clients?"[so 75MBps per client if all 4 access the NAS at the same time].

Yea, the clients are limited to the max speed of the disk array.

 

Depending on the workload, this can be a bit faster or slower than a single client using the disk, but you normally won't see much faster than 300mB/s in this example.

 

Also HDDs will slow down as they are filled as the diameter of the disks go down.

[asheenlevrai]

1 hour ago, Electronics Wizardy said:

Yea, the clients are limited to the max speed of the disk array.

 

Depending on the workload, this can be a bit faster or slower than a single client using the disk, but you normally won't see much faster than 300mB/s in this example.

But is it 300MBps for each client or 300MBps shared among all the concurrent clients? Parallel vs sequential handling of the requests.

 

In the case it is shared among clients, would it be different if using SSDs? Can SSDs handle multiple concurrent requests in parallel better than HDDs or they are just faster at sequentially handling requests?

[Electronics Wizardy]

15 minutes ago, asheenlevrai said:

But is it 300MBps for each client or 300MBps shared among all the concurrent clients? Parallel vs sequential handling of the requests.

 

In the case it is shared among clients, would it be different if using SSDs? Can SSDs handle multiple concurrent requests in parallel better than HDDs or they are just faster at sequentially handling requests?

The number of client doesn't affect the speed of the disks, the queue depth does. More tasks at once = higher queue depth, so more clients normally would have a higher queue depth. HDDs don't get that much faster with a high queue depth, while ssds do(but your gonna be network limited with ssds here anyways).

 

But this also really depends on your use case. If performance is an issue, what models of nas are you using? You want the higerperformacne ones, and you want 10gbe on them.