Connecting 24 drives to 1 motherboard

[leadeater]

I agree with @scottyseng, go with the SAS expander model. That does mean you can't use the reverse cable and SATA ports on the motherboard though but you would only have to buy 1 SAS HBA card.

[scottyseng]

On ‎2016‎年‎5‎月‎18‎日 at 9:57 AM, jowdemanne said:

I'm putting in a dual CPU board in there even though I'm only going to put one Xeon in there.

Ah, be careful with that, usually dual CPU motherboards will not start without both CPU sockets filled.

 

Also, really consider if you need that much CPU power. I have a 14 core Xeon in my NAS, and even pummeling the storage I have (10 drives), I can't remotely get the Xeon to go above 2% CPU usage usually. Storage doesn't take much CPU (Though might need more if you're doing a software RAID).

 

Ah, the other major thing to note is that even if your NAS is 12Gb/s capable, your 1Gb/s Ethernet port on the motherboard / your PC's motherboard, will limit it to 125MB/s (1Gb/s = 125MB/s). My server is capable of 550MB/s, but it's always limited to 125MB/s on my PC due to the Ethernet bottleneck.

[jowdemanne]

6 minutes ago, scottyseng said:

Ah, be careful with that, usually dual CPU motherboards will not start without both CPU sockets filled.

 

Also, really consider if you need that much CPU power. I have a 14 core Xeon in my NAS, and even pummeling the storage I have (10 drives), I can't remotely get the Xeon to go above 2% CPU usage usually. Storage doesn't take much CPU (Though might need more if you're doing a software RAID).

 

Ah, the other major thing to note is that even if your NAS is 12Gb/s capable, your 1Gb/s Ethernet port on the motherboard / your PC's motherboard, will limit it to 125MB/s (1Gb/s = 125MB/s). My server is capable of 550MB/s, but it's always limited to 125MB/s on my PC due to the Ethernet bottleneck.

The plans changed from when I wrote this post :-). I'm going to put in 2 Xeon E5 2670's because they are so cheap at the moment. And the reason for this is: 1. very cheap now and 2: I work in IT and would like to have a lot of possibilities later, like doing some image recognition on my camera footage, webserver, mailserver and much much more :-).

 

@scottyseng I didn't know the 12G was for each lane haha, I thought it was for every drive. So that means that if I buy a motherboard with 8port SAS connectors, that I can have 2 mini SAS connectors going to the backplane (2 of the 6) and have 12 GBps for each drive? I thought I needed to be split between the drives :-). Also, I'm seeing about getting a motherboard with at least 2 Gb ethernet ports and aggregate them or even 2 ports of 10 Gb, so I won't have that problem :-).

 

As it stands now I think I'm going to go with the one without the expander, just the 6 mini SAS connections. This way I'll buy a motherboard with an 8port 12G SAS controller on it so I can already use 8 drives. Later I can just add 2 HBA's or even use one of the reverse breakout cables to connect 4 more drives.

[leadeater]

4 minutes ago, jowdemanne said:

As it stands now I think I'm going to go with the one without the expander, just the 6 mini SAS connections. This way I'll buy a motherboard with an 8port 12G SAS controller on it so I can already use 8 drives. Later I can just add 2 HBA's or even use one of the reverse breakout cables to connect 4 more drives.

If you did this then you can still use the SAS expander model and the 8 disks as is, unless you actually mean a board with 8 SATA ports?

[jowdemanne]

@leadeater I'm thinking the chassis with expander will be more expensive right? I'm having a hard time choosing because I can't find prices anywhere, so I'm just searching for all the possibilities and I'm going to try and ask for some prices all at once :-)

[leadeater]

1 minute ago, jowdemanne said:

@leadeater I'm thinking the chassis with expander will be more expensive right? I'm having a hard time choosing because I can't find prices anywhere, so I'm just searching for all the possibilities and I'm going to try and ask for some prices all at once :-)

Yea it will be more, hopefully not much more though.

[jowdemanne]

7 minutes ago, leadeater said:

Yea it will be more, hopefully not much more though.

But it means I just have to have one 4port connector on my motherboard right? So if I have 12G SAS, that means I'll get 12G * 4 in total = 48 Gbps, with 24 drives attached so 2 Gbps, which means 250 MBps per drive?

[leadeater]

2 minutes ago, jowdemanne said:

But it means I just have to have one 4port connector on my motherboard right? So if I have 12G SAS, that means I'll get 12G * 4 in total = 48 Gbps, with 24 drives attached so 2 Gbps, which means 250 MBps per drive?

Depends on the SAS expander, some require 2 inputs but even with only 1 populated that's 12 disks as a starting point.

[jowdemanne]

1 minute ago, leadeater said:

Depends on the SAS expander, some require 2 inputs but even with only 1 populated that's 12 disks as a starting point.

Those that I found have only one cause I've found a guide saying that a dual expander backplane is useless if you're going to use SATA drives and no SAS drives (something with multipaths). So the fact remains if I have a chassis with expander built in the backplane and only one mini SAS connector I'm only going to get 250 MB/s second to all drives at once (of course this means I'd have to access them all at the same time and yada yada :-P).

[leadeater]

4 minutes ago, jowdemanne said:

Those that I found have only one cause I've found a guide saying that a dual expander backplane is useless if you're going to use SATA drives and no SAS drives (something with multipaths). So the fact remains if I have a chassis with expander built in the backplane and only one mini SAS connector I'm only going to get 250 MB/s second to all drives at once (of course this means I'd have to access them all at the same time and yada yada :-P).

That is correct, SAS disks are dual port so with the expander and one SAS connection you can have either 12 SATA disks or 24 SAS disks single path, should have explained that. Speed doesn't change.

 

If you plug a SATA disk in to a secondary path port with one SAS connection on the expander it just won't show up since its not actually connected.

[jowdemanne]

1 minute ago, leadeater said:

That is correct, SAS disks are dual port so with the expander and one SAS connection you can have either 12 SATA disks or 24 SAS disks single path, should have explained that. Speed doesn't change.

 

If you plug a SATA disk in to a secondary path port with one SAS connection on the expander it just won't show up since its not actually connected.

Thanks! I find it very weird I never learned anything about SAS in school haha :-P, seems like a much used technology with a lot of potential and application value :-). 

Thank you very much for the awesome feedback, info and advice! I've sent my email and will update when I've made a decision :-D

[scottyseng]

11 hours ago, jowdemanne said:

Thanks! I find it very weird I never learned anything about SAS in school haha :-P, seems like a much used technology with a lot of potential and application value :-). 

Thank you very much for the awesome feedback, info and advice! I've sent my email and will update when I've made a decision :-D

Yeah, SAS is pretty much enterprise only, where schools usually focus on consumer ports. In my Computer Organization Class at college, we covered Sata, but there was not mention at all of SAS.

 

If it helps, my SuperMicro 24 bay single expander backplane will accept either one or two SAS inputs, but if you use two SAS inputs, you get the full bandwidth of the two SAS ports on the HBA / RAID card connected to it. It also has a SAS output port for connecting to an external chassis (Though with that many drives, you'd probably be better off getting another HBA...with hard drives you'd still be safe though).

 

Good luck with the project though. SAS is very fun, but can be very expensive...the WD Re SAS drive I own is like $50-100 more than the regular WD Re Sata drive).

[jowdemanne]

10 hours ago, scottyseng said:

Yeah, SAS is pretty much enterprise only, where schools usually focus on consumer ports. In my Computer Organization Class at college, we covered Sata, but there was not mention at all of SAS.

 

If it helps, my SuperMicro 24 bay single expander backplane will accept either one or two SAS inputs, but if you use two SAS inputs, you get the full bandwidth of the two SAS ports on the HBA / RAID card connected to it. It also has a SAS output port for connecting to an external chassis (Though with that many drives, you'd probably be better off getting another HBA...with hard drives you'd still be safe though).

 

Good luck with the project though. SAS is very fun, but can be very expensive...the WD Re SAS drive I own is like $50-100 more than the regular WD Re Sata drive).

Do you mean that if I connected 2 SAS from my board to the backplane (let' say 12G SAS) I'd have the bandwidth of 2*4*12Gbps?

[scottyseng]

3 hours ago, jowdemanne said:

Do you mean that if I connected 2 SAS from my board to the backplane (let' say 12G SAS) I'd have the bandwidth of 2*4*12Gbps?

Yeah, if the motherboard has true SAS ports (Some do, but not cheap). The Sata ports to SAS via reverse breakout wouldn't work with a SAS expander (Sata source still don't connect a SAS device).

[jowdemanne]

Could a moderator mark this as solved? A lot of useful information was put here, maybe change the title to 'Info over SAS HBA and expanders' or something like that? 

[Lurick]

16 minutes ago, jowdemanne said:

Could a moderator mark this as solved? A lot of useful information was put here, maybe change the title to 'Info over SAS HBA and expanders' or something like that? 

There should be a check mark between the quote arrow and the Share This button for you to pick a "best" solution.

[jowdemanne]

So to recap (and so I can make this the solved post if people want to know more about it):

- A SAS expander is like a router that splits up a SAS connection in more, but keeping the same bandwith of course

- Every SFF-8087 connector has 4 lanes, each having the bandwidth associated with that generation of SAS

- You could buy an enclosure with an expander inside of it, meaning you'd only need 1 SFF-8087 connector to connect to the whole aray (keep in mind your bandwidth is for example 12 Gbps * 4 / 24 = 250 MBps per drive)

- Having an expander still means you need an HBA or native SAS ports in your case

- SAS backplane can take SATA drives 

- You can transform 4 SATA ports (for example on your motherboard) into one SFF-8087 connection with a reverse breakout cable, no SAS drives can be used then (of course)

- If you buy an HBA or ebay it is possible you need to 'flash it to IT mode'

Thanks to @scottyseng @leadeater @Lurick @Jarsky @.:MARK:. for the great advice and info!