Bulk Storage Using Disk Shelves?

[Kered124]

Hello-

 

I am wondering what it would take to re-purpose an old disk shelf to use as storage for Freenas. I'd like to use 4TB SAS drives, as they are readily available on Ebay along with the disk shelves and provide a really solid value for the money. I'm wondering how these disk shelves work, though.

 

Questions I have include:

-How are they connected?

-What is actually inside them?

-How loud are they (and could they potentially have fans replaced etc to make them more suitable for home use)

-Do they have disk controllers built in that could be flashed into IT mode/ HBA? 

-Could it potentially be added to my existing server (Freenas on an i5-4440) 

 

I know I have skipped some details-  I'm just trying to figure out if this is possible, I can provide additional details as needed.

 

Thanks!

[dalekphalm]

41 minutes ago, Kered124 said:

Hello-

 

I am wondering what it would take to re-purpose an old disk shelf to use as storage for Freenas. I'd like to use 4TB SAS drives, as they are readily available on Ebay along with the disk shelves and provide a really solid value for the money. I'm wondering how these disk shelves work, though.

 

Questions I have include:

-How are they connected?

-What is actually inside them?

-How loud are they (and could they potentially have fans replaced etc to make them more suitable for home use)

-Do they have disk controllers built in that could be flashed into IT mode/ HBA? 

-Could it potentially be added to my existing server (Freenas on an i5-4440) 

 

I know I have skipped some details-  I'm just trying to figure out if this is possible, I can provide additional details as needed.

 

Thanks!

I have a question, first: What disk shelf? Be specific with the make and model.

 

Now, onto your questions:

1. Most Disk Shelves use External SAS (typically SFF-8088) - you'd have a RAID Card (or HBA) with SFF-8088 connectors on the outside of the PCIe bracket, and another set of connectors on the back of the disk shelf. You'd use an SFF-8088 cable to connect them. One SFF-8088 connector is actually multi-channel (4 channels - each channel is used by a HDD and has a given bandwidth which is defined by the SAS standard used throughout the system).

2. A Powersupply (or two), a Controller Unit, and a backplane - there is also usually a SAS Expander (SAS Expanders allow you to use more disks than available channels). More advanced units may have additional components inside, but this is the basic setup.

3. They are usually VERY VERY VERY VERY VERY LOUD!!! I have a Dell MD1200 12-bay disk shelf, and it's even louder than my rackmount servers (which are very loud). Whether you can replace the fans depends on the specific units. I used to have an old RackableSystems SE3016 16-bay disk shelf, and I performed a Noctua fan swap-out, and that worked well enough. The Dell MD1200 is pickier though, as it actually monitors the fans, so messing with the fans can cause alerts to be triggered. It can usually be done but is often way too involved. Also, the PSU fans are typically the loudest, and it's dangerous to replace PSU fans - only those who know what they are doing should try.

4. Disk shelves don't have RAID Controllers - they do have Controller Units, but they aren't the same as a RAID Controller. Disk Shelves will work with either RAID Cards (In IR or IT mode), or with HBA's out of the box.

5. Yes, Disk shelves can be used with FreeNAS. My Dell MD1200 is being used in this way.

 

Disk Shelves can be very useful - but they're also on the more expensive side for good ones (The MD1200 goes for $250-$300 on the low end on eBay, and that's for one that has no HDD caddys, so add another $50-$100 for HDD caddys). I got mine for free, so yeah. But if I was building from scratch, I'd look at getting a server or case that could hold my drives instead of using an external disk shelf.

[Kered124]

Thank you for the information; I have started looking at servers that can hold the amount of drives I want. I am glad I asked before purchasing. I had been looking at the NetApp DS4246 disk shelf. Any recommendations on a replacement RAID card for the server I am looking at? It seems the P420 isn't well supported by Freenas. The server is an HP Proliant DL380E G8.

 

Thanks again for all of the help!

[dalekphalm]

43 minutes ago, Kered124 said:

Thank you for the information; I have started looking at servers that can hold the amount of drives I want. I am glad I asked before purchasing. I had been looking at the NetApp DS4246 disk shelf. Any recommendations on a replacement RAID card for the server I am looking at? It seems the P420 isn't well supported by Freenas. The server is an HP Proliant DL380E G8.

 

Thanks again for all of the help!

If you're using FreeNAS, you ideally don't want a RAID Card at all - an HBA is a far better choice. Though if you can get a RAID Card flashed with IT Mode firmware, those operate as if they were HBA's.

 

Check the FreeNAS Hardware recommendations:

https://www.ixsystems.com/blog/a-complete-guide-to-freenas-hardware-design-part-ii-hardware-specifics/

 

Popular recommendations include the LSI 9207-8i (though you'd want the LSI 9207-8e instead - i for Internal SAS connectors, e for External SAS connectors). Other good choices include the 9211-8e and the IBM m1015 (flashed to IT).

 

If you get an HBA, no flashing is required. If you get a RAID Card, I'd recommend getting one pre-flashed to IT Mode firmware.

[Kered124]

So I just purchased the server I was looking at. It comes with the P420 Raid card that supports HBA mode, but isn't well supported by FreeNas as far as I can tell. Does anyone have experience with this? I'm also wondering what a good drop in replacement card would be for my use case.I would like to keep the same form factor instead of moving to a PCI card if possible- as I have been reading about weird PCI compatibility issues with the DL380E and non-HP expansion cards.

 

EDIT:

To be clear- I have now Dumped the idea of using a seperate disk shelf, and have purchased an HP server with 12 3.5" drive bays to suite my needs.

[dalekphalm]

Just now, Kered124 said:

So I just purchased the server I was looking at. It comes with the P420 Raid card that supports HBA mode, but isn't well supported by FreeNas as far as I can tell. Does anyone have experience with this? I'm also wondering what a good drop in replacement card would be for my use case.I would like to keep the same form factor instead of moving to a PCI card if possible- as I have been reading about weird PCI compatibility issues with the DL380E and non-HP expansion cards.

If the P420 is not flashed to IT Mode, it will not work in the way you want. After some quick googling, it appears that some have gotten that to work - I have no idea what is involved in the process. You should research that carefully and ensure you understand the entire process of flashing the card to IT Mode (What people seem to be calling "HBA mode").

 

Another thing to note is that the P420 has internal SAS connectors. This means you'll need an SFF-8087 to SFF-8088 cable, and you'll have to pop off an empty PCI bracket and feed the cable from the inside of your server, out the back, to the NetApp disk shelf.

 

This can work, but is definitely not ideal.

 

I listed several cards above that can easily be found on eBay that are drop in ready. The LSI 9207-8e seems to be the most abundant:

https://www.ebay.ca/sch/i.html?_odkw=lsi+9211-8e&_osacat=0&_from=R40&_trksid=m570.l1313&_nkw=lsi+9207-8e&_sacat=0

[Kered124]

2 minutes ago, dalekphalm said:

Another thing to note is that the P420 has internal SAS connectors. This means you'll need an SFF-8087 to SFF-8088 cable, and you'll have to pop off an empty PCI bracket and feed the cable from the inside of your server, out the back, to the NetApp disk shelf.

It sounds like I may have made my edit a little too late; my apologies. 

 

At this point I believe I am looking for an internal SAS controller for the 12 drive bays my server has that supports IT mode. (Please correct me if I am wrong)

 

Id like it to fit wherever the P420 currently resides in the server, and be compatible with the HP hardware so the server doesn't throw a fit and mess with my fan speeds.

 

Thanks again!

[dalekphalm]

3 minutes ago, Kered124 said:

It sounds like I may have made my edit a little too late; my apologies. 

 

At this point I believe I am looking for an internal SAS controller for the 12 drive bays my server has that supports IT mode. (Please correct me if I am wrong)

 

Id like it to fit wherever the P420 currently resides in the server, and be compatible with the HP hardware so the server doesn't throw a fit and mess with my fan speeds.

 

Thanks again!

Ah yes, I didn't see the edit. So, I'd just try the P420 when it arrives - test it out. Load into the P420's RAID BIOS and see if there's an option to just switch on IT/HBA mode. Plug in individual disks without configuring them, and see if FreeNAS will pick them up. Likely not, but sometimes it works.

 

If it doesn't work, I'd look at either a LSI 9207-8i or a LSI 9211-8i. Both should be plentiful on eBay, and both should just drop right in. The 9207-8i is the preferred model, in my opinion.

 

It should just slot right into the PCIe slot that the P420 lives in, and the cables connected to the P420 should connect up to the 9207.

[Kered124]

1 minute ago, dalekphalm said:

It should just slot right into the PCIe slot that the P420 lives in, and the cables connected to the P420 should connect up to the 9207

I guess I have some homework to do- I assumed that the P420 would be similar to the PERC 5/i in my poweredge 2950 and be a weird form factor and off to the side.

 

One rising concern I am having is related to hard drives- it seems some people are having issues with the server not liking drives that aren't HP branded... Any insight?

 

Thanks for the information!

 

 

[dalekphalm]

1 hour ago, Kered124 said:

I guess I have some homework to do- I assumed that the P420 would be similar to the PERC 5/i in my poweredge 2950 and be a weird form factor and off to the side.

Note: I've never used nor even seen the P420 in action. Some RAID Controllers are indeed in unusual form factors. Modules, or even integrated directly into the motherboard.

 

I'm frankly assuming that the P420 is a standard PCIe form factor. Though, even if it's not, you should still be able to drop the LSI 9207-8i into any of the free PCIe 8x or 16x slots in the system. If you do this, you can simply disconnect the cables from the P420 and switch them over to the LSI card. You should also be able to remove the P420, assuming that it's not a fully integrated model.

1 hour ago, Kered124 said:

One rising concern I am having is related to hard drives- it seems some people are having issues with the server not liking drives that aren't HP branded... Any insight?

Once you install a new HBA, that issue won't matter anymore. Though yes, some HP (and other Server OEM) branded RAID Cards will often check for matching OEM firmware on the HDD. This isn't as common anymore.

 

The risk should be minimal.

1 hour ago, Kered124 said:

Thanks for the information!

You're welcome!

[Kered124]

On 7/26/2019 at 5:20 PM, dalekphalm said:

I'm frankly assuming that the P420 is a standard PCIe form factor. Though, even if it's not, you should still be able to drop the LSI 9207-8i into any of the free PCIe 8x or 16x slots in the system. If you do this, you can simply disconnect the cables from the P420 and switch them over to the LSI card. You should also be able to remove the P420, assuming that it's not a fully integrated model.

So, here is an odd question- During bootup, my current server gives me an opportunity to enter my RAID config and modify the array and such- will the server act differently if I get rid of its existing RAID controller and replace it with the one you recommended? I guess I'm wondering if I am given that boot option because of the server's BIOS, or because the RAID card gives me that option once the computer starts to boot from that device.

[dalekphalm]

2 hours ago, Kered124 said:

So, here is an odd question- During bootup, my current server gives me an opportunity to enter my RAID config and modify the array and such- will the server act differently if I get rid of its existing RAID controller and replace it with the one you recommended? I guess I'm wondering if I am given that boot option because of the server's BIOS, or because the RAID card gives me that option once the computer starts to boot from that device.

That option is caused by the RAID cards internal BIOS/configuration utility. 

 

An HBA will not have that option because HBA’s have no RAID. A RAID card will though. 

 

So if you replace the P420 with a different RAID card, you’ll still have that option (though loading into it may look different depending on what manufacturer made the RAID card). 

[Kered124]

2 hours ago, dalekphalm said:

That option is caused by the RAID cards internal BIOS/configuration utility. 

 

An HBA will not have that option because HBA’s have no RAID. A RAID card will though. 

 

So if you replace the P420 with a different RAID card, you’ll still have that option (though loading into it may look different depending on what manufacturer made the RAID card). 

This perfectly answers my question- Thank you!

[dalekphalm]

22 minutes ago, Kered124 said:

This perfectly answers my question- Thank you!

My pleasure!

[Kered124]

On 7/26/2019 at 5:20 PM, dalekphalm said:

Once you install a new HBA, that issue won't matter anymore. Though yes, some HP (and other Server OEM) branded RAID Cards will often check for matching OEM firmware on the HDD. This isn't as common anymore

So my server arrived yesterday- still waiting on my drives. I thew in a few spare 1Tb hard drives to play around and try to get the P420 into IT mode, and noticed one of my disks didn't have a backlight or activity lights. Reading through the RAID controller I noticed it mentioned it couldn't verify that one of my drives was HP branded, and because of this the status LEDS wouldn't work. I'm wondering if my status LEDS will work at all if I change the RAID card out for the one you recommended... I know they are just lights, but I do like having them haha!

[dalekphalm]

4 hours ago, Kered124 said:

So my server arrived yesterday- still waiting on my drives. I thew in a few spare 1Tb hard drives to play around and try to get the P420 into IT mode, and noticed one of my disks didn't have a backlight or activity lights. Reading through the RAID controller I noticed it mentioned it couldn't verify that one of my drives was HP branded, and because of this the status LEDS wouldn't work. I'm wondering if my status LEDS will work at all if I change the RAID card out for the one you recommended... I know they are just lights, but I do like having them haha!

The status lights themselves? I'm not sure if that's controlled by the backplane or the RAID Card - obviously the RAID Card issues commands relating to the status lights but it might be the backplane itself checking for HP Firmware.

 

@leadeater, any ideas on whether an HBA will allow the status lights to function?

[leadeater]

2 hours ago, dalekphalm said:

@leadeater, any ideas on whether an HBA will allow the status lights to function?

Normally yes, sometimes they don't if you disk swap and use the HP caddy with a non HP disk, either beacuse the light tubes don't line up or the caddy has a bit more smarts (Gen8 and newer). Sometimes vendors are dicks about it and won't allow it if it's not their disk.

[Kered124]

22 hours ago, leadeater said:

Normally yes, sometimes they don't if you disk swap and use the HP caddy with a non HP disk, either beacuse the light tubes don't line up or the caddy has a bit more smarts (Gen8 and newer). Sometimes vendors are dicks about it and won't allow it if it's not their disk.

So my drives got here about an hour ago and I have put them all into caddies and started building an array just to test things out a little. I am having tons of trouble getting the P420 into IT mode because of the paywall HP has in place for their SPP software.. I have two disks reporting that they aren't genuine HP drives and saying their lights won't be controlled.. I am wondering if swapping out to this controller: https://ebay.to/2SZhgGS Would help resolve the non-genuine disk "issue" or will make my situation worse. The P420 is also reporting that it is running at 85-88 degrees Celsius at idle, which seems absurd. Any input?

 

Thanks so much for all of the help so far!

[dalekphalm]

17 minutes ago, Kered124 said:

So my drives got here about an hour ago and I have put them all into caddies and started building an array just to test things out a little. I am having tons of trouble getting the P420 into IT mode because of the paywall HP has in place for their SPP software.. I have two disks reporting that they aren't genuine HP drives and saying their lights won't be controlled.. I am wondering if swapping out to this controller: https://ebay.to/2SZhgGS Would help resolve the non-genuine disk "issue" or will make my situation worse. The P420 is also reporting that it is running at 85-88 degrees Celsius at idle, which seems absurd. Any input?

 

Thanks so much for all of the help so far!

Those temps are certainly high - some server equipment does run hot (because lack of active cooling - they assume you're gonna stick it inside a temp controlled Server Room), but even still that's very high.

 

In terms of whether the HP H220 will do the trick? Well it will certainly work with FreeNAS and ZFS (as well as Storage Spaces) - it may or may not resolve the light issues, since it depends on whether the HP H220 checks OEM drive firmware or not. I don't know if it does.

[Kered124]

Hmm... Since the H220 is an HP branded controller, I'm assuming it will also be looking for HP drives, but I don't know. I took the top of the server chassis off and moved the SAS cables out of the way- they had been blocking a lot of the airflow. The server is much quieter now, but is still ramping up and down. I'm very tempted to pull the heatsink off of the P420 and stick some thermal compound under it, as for what I can tell it has none. 

[Kered124]

7 minutes ago, dalekphalm said:

In terms of whether the HP H220 will do the trick? Well it will certainly work with FreeNAS and ZFS (as well as Storage Spaces) - it may or may not resolve the light issues, since it depends on whether the HP H220 checks OEM drive firmware or not. I don't know if it does.

I think those words pretty well talked me into ordering the H220. Is the one I linked above the correct controller? I believe it is, but I'd really like someone else to double check so I know I'm not missing anything haha. If it looks right Ill order that today and get it on the way.

[dalekphalm]

Just now, Kered124 said:

Hmm... Since the H220 is an HP branded controller, I'm assuming it will also be looking for HP drives, but I don't know.

Yeah I wasn't sure about that either - it could certainly be a potential issue. But since it's already been flashed to IT mode, it's possible whoever flashed it used a non-OEM branded firmware (For example, used the official LSI firmware, rather than the HP firmware).

 

To totally bypass this, you could just get an LSI branded HBA, like the 9207 - the actual hardware is essentially identical to the HP H220.

Just now, Kered124 said:

I took the top of the server chassis off and moved the SAS cables out of the way- they had been blocking a lot of the airflow. The server is much quieter now, but is still ramping up and down. I'm very tempted to pull the heatsink off of the P420 and stick some thermal compound under it, as for what I can tell it has none. 

Wouldn't be a bad thing to do - though I think replacing the P420 might end up being the better course of action if you cannot figure out flashing it to IT mode.

1 minute ago, Kered124 said:

I think those words pretty well talked me into ordering the H220. Is the one I linked above the correct controller? I believe it is, but I'd really like someone else to double check so I know I'm not missing anything haha. If it looks right Ill order that today and get it on the way.

The one you linked is pre-flashed and ready to go for ZFS, etc. Make sure to pay via PayPal and you'll get the full eBay/PayPal buyers protection. This means that if for some reason it isn't properly flashed, you can exchange it for a different one, or simply get a return.

[Kered124]

25 minutes ago, dalekphalm said:

The one you linked is pre-flashed and ready to go for ZFS, etc. Make sure to pay via PayPal and you'll get the full eBay/PayPal buyers protection. This means that if for some reason it isn't properly flashed, you can exchange it for a different one, or simply get a return.

It has been ordered! I am hoping that it will fix a few of these issues, minimally the temperature Issue. I would really still like to flash the P420 into IT mode if I can for other projects- I have another server that would definitely benefit from this controller if I could get it working at a later date.

 

After it finishes building this array, I think I'll pull the P420 and put some arctic silver between the controller and the heatsink. My only hesitation with buying an LSI branded controller compared to the H220 is that I have heard the gen 8 servers are very picky about PCI devices and tend to run the fans at 100% if it isn't an HP branded device... 

 

I am sort of regretting buying this HP box after having such positive experiences with my poweredge servers...  Hopefully I can find a way to get ahold of the Service Pack for Proliant software and get the firmware updated on everything+ force IT mode on for the P420. All of those things are done with the SPP software image from what I have read/watched 

Edited August 2, 2019 by Kered124
Spelling

[leadeater]

17 hours ago, Kered124 said:

Hopefully I can find a way to get ahold of the Service Pack for Proliant software and get the firmware updated on everything+ force IT mode on for the P420. All of those things are done with the SPP software image from what I have read/watched 

Give me the link and I'll get you a copy via PM by way of attached file, or if there is a file size limit I'll just give you a OneDrive link to it. You can however just boot in to Intelligent Provisioning and update the firmware through that, iLO can pull firmware direct from HPE.

[Kered124]

1 hour ago, leadeater said:

You can however just boot in to Intelligent Provisioning and update the firmware through that, iLO can pull firmware direct from HPE.

I tried looking through things in the ILO interface, but couldn't find a way to get the command line I need for enabling IT mode on the controller.. I did find a spot where they will let me check the versions of current firmwares, but no options to update them that I noticed. 

 

On a happier note- I'm currently building a RAID 60 with my drives just to stress them a little before my new controller gets here next week on Friday. I'm worried about the drives because they arrived looking a little rough, thrown everywhere in the box.. It looks like they came loose from their tray during shipping and had a rough ride after that. Two disks of 12 are showing up as non-genuine HP disks even though they have the exact same firmware as the other 10 disks that are properly recognized.

 

Thank you so much for the help; I really appreciate it.