Connecting 24 drives to 1 motherboard

[jowdemanne]

Hello,

I come here to ask what I find an easy question. If i would buy a case like this one, how can I connect all those drives to my motherboard? 

The way I found now is buying this or buying 3 of these. Is that the only solution?

 

This is all for a home server that I'm planning to build. It has 24 bays, way to much for now but I want scalability, that's also why I'm putting in a dual CPU board in there even though I'm only going to put one Xeon in there.

Can anyone please help me better than Google can? :-)

[BelgiumCastGijs]

Some server motherboards have that many sata connectors, but they are expensive, I'd go with a raid card. Sata ones tend to be less expensive but are less stable as well. There is another type of connector (can't remember the name), it is much more stable and can handle up to 4 sata connectors on every plug, they tend to be a bit more expensive though. But if you need 24 sata ports, will be expensive no matter how you put it.

[Lurick]

Depending on if you want Software or Hardware RAID you can go with three of these:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816118142

and then these:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816151078

 

I have that setup with a Linux box and software raid 10, I use one card per 8 drives and 2 cables per card.

 

With 24 drives in RAID 10 you'll get 10Gbps to 12Gbps transfer speeds

[Windows7ge]

If you're dead set on using one of these RAID controllers go for the 36 port one. If you buy three of the other ones you'll have to configure the 3 arrays in software RAID and that'll be a potential nightmare on top of one more point of failure. 

[jowdemanne]

Sorry, I should've specified I'm gonna put FreeNAS on this box and ZFS2 will be the format, probably with 5 drives at a time in one vdev. So I need a card with JBOD capabilities as ZFS can't handle RAID cards

[Jarsky]

You could get a board like this:
 http://www.asrockrack.com/general/productdetail.asp?Model=E3C224D4I-14S

And get a SAS expander card:

http://www.amazon.com/Intel-RAID-Expander-Card-RES2SV240/dp/B0042NLUVE

 

Then connect the onboard LSI sas to the expander and have 24 ports.

This guy confirms that the LSI 2308 chip works with the LSI based RES2SV240 expander: https://www.reddit.com/r/homelab/comments/28nc2q/supermicro_x10sl7f_sas2_sata_connectors_to_sas/

[jowdemanne]

58 minutes ago, Jarsky said:

You could get a board like this:
 http://www.asrockrack.com/general/productdetail.asp?Model=E3C224D4I-14S

And get a SAS expander card:

http://www.amazon.com/Intel-RAID-Expander-Card-RES2SV240/dp/B0042NLUVE

 

Then connect the onboard LSI sas to the expander and have 24 ports.

This guy confirms that the LSI 2308 chip works with the LSI based RES2SV240 expander: https://www.reddit.com/r/homelab/comments/28nc2q/supermicro_x10sl7f_sas2_sata_connectors_to_sas/

I'd like to get the concept too, like if I would choose a board with only SATA ports,what would I need to be able to connect those drives?

I had a kind of case in mind, the one I linked in the first post. I think it has fan out cables already in it? So I get 6 mini SAS connectors that I would need to connect to my motherboard. Do I only need an SAS expander board for that? Sometimes I'm struggling with the terminology too, like 'ports' and such. I think 1 mini SAS can connect to 4 drives right? So when an expander card has 2 connectors I can connect 8 drives to it? And would it be a problem if I bought like 3 cards with 2 connectors using FreeNAS?

[Lurick]

30 minutes ago, jowdemanne said:

I'd like to get the concept too, like if I would choose a board with only SATA ports,what would I need to be able to connect those drives?

I had a kind of case in mind, the one I linked in the first post. I think it has fan out cables already in it? So I get 6 mini SAS connectors that I would need to connect to my motherboard. Do I only need an SAS expander board for that? Sometimes I'm struggling with the terminology too, like 'ports' and such. I think 1 mini SAS can connect to 4 drives right? So when an expander card has 2 connectors I can connect 8 drives to it? And would it be a problem if I bought like 3 cards with 2 connectors using FreeNAS?

The limit for a SAS cable is 4 to 6 drives. The first one you linked would be able to run 24 drives off the 4 internal ports with 6 drives per SAS port. If you went with multiple cards you linked and did a software raid through FreeNAS or something you would get more bandwidth compared to a single card with all cards hooked up to that.

[Jarsky]

When you build your own machine, youre unlikely to find any board with 24+ ports (ideally you'd want 26 - to connect SSD's).

SAS connectors can break-out to 4 SATA connections per SAS port - but SAS has the awesome ability to be daisy chained.

 

So you can use a board with SAS onboard, or a SAS HBA card + a SAS expander card like the Intel one above (it has SAS ports on the back for connecting to a RAID/HBA card, and then 6 expansion ports which can support 4 SATA connections each - giving you 24 ports)

 

I've never seen a SAS breakout to 6 SATA drives - I believe the card above is 4 internal SFF8087 connectors providing 16 SATA channels, and 2 external  SFF8088 connectors for a further 8 SATA channels.

 

You can just 3 x 2 port, or 2 x 4 port HBA's if you're going to be using Software to RAID them, like FreeNAS ZFS, Linux MDADM, unRAID, or Windows Storage Spaces. Because these use JBOD configurations, they don't have to be on the same controller chip like a RAID.

 

As for if you chose a board that only had SATA connectors - you would be best to get a RAID Card + Expander if you were going to go the hardware route - or just a bunch of HBA cards if you were to go the software - such as 3 x LSI 9211-8i's or of course you can look at cheaper HBA's.

 

 

[leadeater]

43 minutes ago, Jarsky said:

SAS connectors can break-out to 4 SATA connections per SAS port - but SAS has the awesome ability to be daisy chained.

 

So you can use a board with SAS onboard, or a SAS HBA card + a SAS expander card like the Intel one above (it has SAS ports on the back for connecting to a RAID/HBA card, and then 6 expansion ports which can support 4 SATA connections each - giving you 24 ports)

@jowdemanne

 

To further expand on this information you will want to be using SAS HBA's for that many disks, you won't find a native SATA solution that supports that many disks.

 

The SAS standard is 4 lanes per port which is why SAS to SATA break outs are 4 SATA ports. A SAS expander allows you expand a single SAS port in to multiple ports which then allows you to convert these extra SAS ports to 4 SATA ports.

 

For 24 disks I would suggest three LSI 9211-8i (IBM M1015/IBM M1115) IT mode as these cards are so cheap buying expanders is a little pointless and will cost the same anyway.

[jowdemanne]

So i could just search ebay for like 1 of those cards and I could already use it for 8 drives (I am planning to expand, having only 5 drives in right now and upgrading with vdevs of 5 drives as I go bigger). 

I've searched a bit on ebay and found those cards are like 60 bucks, so that's way cheaper than I previously thought!

So to recap, I'm going for ZFS so I'm gonna need 3 HBA's (LSI 9211-8i / IMB M1015 / IMB M1115) and I can just connect all those mini SAS connectors to the connectors the case has in the front.

Thanks everybody for the info! I think I have what I need now.

[Guest]

46 minutes ago, jowdemanne said:

So i could just search ebay for like 1 of those cards and I could already use it for 8 drives (I am planning to expand, having only 5 drives in right now and upgrading with vdevs of 5 drives as I go bigger). 

I've searched a bit on ebay and found those cards are like 60 bucks, so that's way cheaper than I previously thought!

So to recap, I'm going for ZFS so I'm gonna need 3 HBA's (LSI 9211-8i / IMB M1015 / IMB M1115) and I can just connect all those mini SAS connectors to the connectors the case has in the front.

Thanks everybody for the info! I think I have what I need now.

Keep in mind you may want to flash it to IT mode. Removes the RAID functionality for better compatibility. Certain LSI cards want you to initialize the disk in them even if it's purely being used as a HBA and not into a RAID array.

[.:MARK:.]

I have that exact case that you want. It has 6 SFF-8087 connectors on the backplanes, so you will need 6 of those cables for that.

I use an HP SAS Expander that allows for a link aggregate of SFF-8087 cables to my RAID card, LSI 9260-8i with BBU.

So my parts were:

That case

HP SAS Expander

LSI 9260-8i

8xSFF-8087 cables

 

The SAS Expander looks like this:

Spoiler

Be sure to get the one with the green PCB like above, the yellow one is not firmware upgrade-able to faster SAS afaik.

[leadeater]

4 minutes ago, .:MARK:. said:

I have that exact case that you want. It has 6 SFF-8087 connectors on the backplanes, so you will need 6 of those cables for that.

I use an HP SAS Expander that allows for a link aggregate of SFF-8087 cables to my RAID card, LSI 9260-8i with BBU.

So my parts were:

That case

HP SAS Expander

LSI 9260-8i

8xSFF-8087 cables

 

The SAS Expander looks like this:

  Reveal hidden contents

Be sure to get the one with the green PCB like above, the yellow one is not firmware upgrade-able to faster SAS afaik.

Currently those SAS expanders on ebay cost more than 2 LSI 9211-8i and the LSI 9260-8i costs a significant amount compared to a single one of those, in this case since ZFS is going to be used rather than hardware RAID the cheaper and more supported option is the 3 HBA cards.

 

If hardware RAID was the intended use then yea I would definitely agree that card is a great pick (with BBU).

[.:MARK:.]

8 minutes ago, leadeater said:

Currently those SAS expanders on ebay cost more than 2 LSI 9211-8i and the LSI 9260-8i costs a significant amount compared to a single one of those, in this case since ZFS is going to be used rather than hardware RAID the cheaper and more supported option is the 3 HBA cards.

 

If hardware RAID was the intended use then yea I would definitely agree that card is a great pick (with BBU).

Was recommending the HP SAS Expander, and just giving the rest of my spec for a further understanding of my use case.

Since the Intel one wouldn't work in that case, you would lose bays.

Also it really depends where you are, here in the UK a second hand HP SAS Expander is cheaper than 1 second hand 9211-8i !

 

So maybe a 9211-8i with a HP SAS Expander would work well?

[jowdemanne]

@.:MARK:. how does a SAS Expander work actually? Still can't wrap my mind around it.

Also, do the 2 solutions have different throughput?

[leadeater]

3 hours ago, jowdemanne said:

@.:MARK:. how does a SAS Expander work actually? Still can't wrap my mind around it.

Also, do the 2 solutions have different throughput?

SAS expander has a SAS expander IC on it, very basically it works like ethernet switching. The SAS protocol supports switching where SATA does not.

 

https://en.wikipedia.org/wiki/Serial_Attached_SCSI#SAS_expanders

http://www.avagotech.com/products/server-storage/sas-expanders/

[Jarsky]

5 hours ago, jowdemanne said:

@.:MARK:. how does a SAS Expander work actually? Still can't wrap my mind around it.

Also, do the 2 solutions have different throughput?

Imagine a USB Hub.

You plug the hub into a single USB Port, and then the hub might have 10 USB ports on it.

 

Also imagine a TCP/IP network switch. 

You plug an 8 port switch into your modem/router, and then all the devices connected to the switch now have access to the internet.

 

USB and TCP/IP are both switching protocols which is why you can "daisy chain" them. The SAS design falls into this switching protocol.

Basically each SAS device you connect has a "SAS address" similar to networking where each Physical device has a MAC address - this is how the controller is able to locate the device on an expander.

 

[jowdemanne]

So theoretically I'd have more throughput with 3 HBA cards right? Otherwise everything would go through the expander to the drives?

[leadeater]

4 hours ago, jowdemanne said:

So theoretically I'd have more throughput with 3 HBA cards right? Otherwise everything would go through the expander to the drives?

Really isn't an issue unless you are using SSD's only. Each SAS port is 4 lanes so the actual bandwidth per port for a 6G SAS card is 4 x 6G = 24G, that is a hell of a lot of bandwidth then double that for a 2 port card. You'll be saturating the PCI-E x4 bus before hitting the limit of the card.

 

But yes in theory 3 HBA's is more bandwidth, the better reason to use them is redundancy if you design your disk pools correctly. You should be able to lose an HBA without the array going offline.

[jowdemanne]

After all this I have only 2 more questions actually. Would it be possible to connect the mini SAS connector to 4 sata ports on my motherboard? With the knowledge I'd only use sata drives in the enclosure of course. That way if I have a board with 8 sata ports I could buy 2 cable that transform 4 sata to 1 mini SAS and hook up the backplane for 8 drives?

Also, what would I be better off with? With supermicro you have the choice of having a SAS backplane with our without expander inside of it. Would it be fast enough to have one with the expander and 24 drives? The way I see it that I have 12 Gbps with 24 drives, give me only 62.5 MBps per drive, and that's too low for me.

[leadeater]

22 minutes ago, jowdemanne said:

After all this I have only one more question actually. Would it be possible to connect the mini SAS connector to 4 sata ports on my motherboard? With the knowledge I'd only use sata drives in the enclosure of course. That way if I have a board with 8 sata ports I could buy 2 cable that transform 4 sata to 1 mini SAS and hook up the backplane for 8 drives?

I think the cable to do that is called a mini SAS to SATA reverse. @Jarsky or @scottyseng would likely know, I only use SAS to SATA forwards or native SAS disks.

 

Quote

To make it simple, if the host-controller side is a SAS connector (SFF-8470) and the target side is SATA drives then you must always use a SAS to SATA Forward Breakout Cable. If The Motherboard/host-Controller side are SATA connectors and the backplane is a SAS connector then you must always use a SAS to SATA Reverse Breakout Cable.

http://serverfault.com/questions/410512/are-there-different-kinds-of-mini-sas-to-4x-sata-reverse-breakout-cables

[jowdemanne]

@leadeater thanks for the name! I knew there was something like it but couldn't come up with a good google search term :-)

So it is possible with the right cables, which means I could actually buy a case with 6 internal mini SAS ports, put 8 drives in them and connect them via the SAS backplane back to 8 SATA ports on the motherboard. Would save me a lot of money :-)

[leadeater]

2 minutes ago, jowdemanne said:

@leadeater thanks for the name! I knew there was something like it but couldn't come up with a good google search term :-)

So it is possible with the right cables, which means I could actually buy a case with 6 internal mini SAS ports, put 8 drives in them and connect them via the SAS backplane back to 8 SATA ports on the motherboard. Would save me a lot of money :-)

Yes but it would only get you started with the first 8 disks of course, but that's fine you can buy SAS HBAs at a later date when you have exceeded the capacity of the current setup and need more. Does bring down the starting cost significantly.

 

Unless anyone else can come up with a good reason not to do this?

[scottyseng]

27 minutes ago, jowdemanne said:

After all this I have only 2 more questions actually. Would it be possible to connect the mini SAS connector to 4 sata ports on my motherboard? With the knowledge I'd only use sata drives in the enclosure of course. That way if I have a board with 8 sata ports I could buy 2 cable that transform 4 sata to 1 mini SAS and hook up the backplane for 8 drives?

Also, what would I be better off with? With supermicro you have the choice of having a SAS backplane with our without expander inside of it. Would it be fast enough to have one with the expander and 24 drives? The way I see it that I have 12 Gbps with 24 drives, give me only 62.5 MBps per drive, and that's too low for me.

Yeah, it's possible with a SAS reverse cable (And you know that you only can use SATA drives / no SAS expanders).

 

I personally would vouch for a HBA card with a SuperMicro Expander inside of it (It lets you expand even more drives to the system if you get another SAS backplane). It does cost more though. I just like the future expandability of it all.

 

Ah, you know one SAS connector has four lanes right? 12Gbps per lane is quite a bit (Roughly 1.5 GBps) on one lane x 4 = 6GB/s. The average HBA card has two SAS ports, so 12GB/s total (Though I think the card itself has a hard limit due to PCIe lanes if I remember right). That's 500MB/s per drive if you have 12 drives. This is also if you were crazy enough to run 24 drives in RAID0 (I see no other RAID where you'd access all 24 drives at the same time). You'd probably be looking at SSDs too. If you have hard drives, it's even harder to bottleneck SAS.