linux JBOD controller ideas...

[techno10]

So I just returned a highpoint rocket 640L controller as it failed. It seemed to initially work "ok" but running 4 240gig SSD's for vmware workstation images it finally just gave up (would boot but as soon as you started writing/reading it would drop disks). It only lasted around 3 days. I'm not necessarily looking for speed though that's great (I'm using SSD for power and reliability).

 

Also I'm thinking of sticking with linux mdadm as I can rebuild and move disks across systems and not be controller dependent. 

 

So the reason I'm even posting is this because it seems like Marvell has cornered the market for sub $100 controllers.. Syba, Highpoint, Startech, Vantec and everyone else seem to use similar Marvell chipsets.. I'm pretty sure (looking at all of the reviews across multiple sites) that these chipset exhibit all the same characteristics and eventually just die under "heavy" usage.

 

I guess 1 option is to buy a 10 port SATA motherboard as I've had fairly good luck with them even tho they sometimes use Marvell chipsets. Right now I have 2 arrays, 1x4 2 TB red SATA drives and 1x4 240g SSD's.. so I want 9 total ports in 1 host. The current motherboard has 6 ports so I need another 4 or so.

 

Looking for suggestions. I have 2 4 lane PCI-express slots to use.

 

thanks,

Mike

[alpenwasser]

- snip -

For around 100 USD you can get an LSI 9211-8i on eBay new in box (or at least you could when I last

checked). It's not the newest or fastest card, but it is from a higher price bracket (still retails

for around 290 USD where I live) than the cheap Marvell stuff.

I've bought three of them on ebay, and flashed them to the IT firmware (tutorial see here: http://linustechtips.com/main/topic/104425-flashing-a-lsi-9211-8i-raid-card-to-it-mode-for-zfssoftware-raid/ )

for using them with software RAID (well, ZFS).

Been using them in my server for about a year now, and they're working well so far (*knocks on wood*).

They probably won't be setting any world records, and I haven't tested them with SSDs, but it's

what comes to mind off the top of my head. There are newer and faster cards on the market now,

though I'm not sure if you can get them for similar price/performance.

Product Link: http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9211-8i#specifications

[techno10]

I actually worked for LSI before they got bought by Avago and gutted. So I'm not sure how I'd feel like buying their stuff at this point. Also not sure how their support is after that. Bunch of people got laid off, etc..

 

I did check ebay. A couple of used and a couple from China/Hong Kong (which I'm guessing might be knock-offs).

 

But it looks like it might be one of the few options to actually use...

 

thanks!

Mike

[techno10]

real quick question.. what does the IT firmware get you? I know ZFS doesn't like to run on raid but can't you configure the controller with normally firmware to just be JBOD?

 

thanks,

Mike

[alpenwasser]

I actually worked for LSI before they got bought by Avago and gutted. So I'm not sure how I'd feel like buying their stuff at this point. Also not sure how their support is after that. Bunch of people got laid off, etc..

Ah yes, the glorious procedure of murder and execution mergers and acquisitions.

Alas, I can't really answer that moral dilemma for you.

 

I did check ebay. A couple of used and a couple from China/Hong Kong (which I'm guessing might be knock-offs).

 

I bought two from Taiwan, one from Hong Kong, as best as I can tell they

are genuine. They all had the same retail packaging despite coming from

different sources, and the accessories one would expect. Plus, the genuine

LSI firmware runs on all of them, so if they are fakes, they're pretty

good ones.

My general experience with ebay sellers from Hong Kong and region has

been pretty good to be honest. I think as long as you stick to sellers

with a good rating (high percentage of satisfied customers, preferably

high total numbers of successful transactions as well, and ideally they've

been around for a while), the chances of being duped aren't that high.

Not zero, but low enough that I'm usually not really worried about it.

Shipping times from Taiwan were much faster though, about one week to

Switzerland. Hong Kong can sometimes take a few weeks. For me that's

usually not an issue, but YMMV.

Still, I'm going to knock on some wood now, for my next purchase, just

to be on the safe side.

 

real quick question.. what does the IT firmware get you? I know ZFS doesn't like to run on raid but can't you configure the controller with normally firmware to just be JBOD?

I have not personally tested this, so take my word with a grain of salt.

But as best as I have been able to find out without first-hand experience

of JBOD with IR firmware, when doing that, the drives will be presented

to the OS as logical drives instead of actual hardware, and some stuff may

not work properly.

I haven't actually been able to find that much on the subject, since it seems

that at some point somebody said "ZFS => IT MODE OR GTFO!" and we all just

followed that like the sheep that we are.

 

Most important difference between JBOD and HBA is that JBODs are still logical drives and as such do not support all scsi commands that regular disks do. For instance temperature reporting is not available for logical drives. There are of course RAID cards that do support scsi command pass-through to individual disks making up a logical drive.

source: https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/edvq1GVT77o

I have read some other stuff on it over the years, but it's been in bits

and pieces, so I'm sorry I don't have any other links, let alone a true

and proper definite answer. I have also read on occasion from people using

IR firmware with ZFS and haven't heard about any horrible consequences

yet, but until I hear otherwise, I'm going to go with IT mode for ZFS (or

sofware RAID in general I suppose).

[techno10]

The only thing I can add about fakes.. A short story is years ago (when working for Agere before merging with LSI) we purchased a quad port intel network card. It looked great and I assumed it was normal. I couldn't get the card to work under Linux. Open a case with intel. They told me the card was a fake... couldn't believe it. They said that it would work under windows (and it did) but some limiting factor would not allow the fake card to work under Linux. Ever since then I've always been a little more cautious on where and what I buy.. spent like 8+ hours total time debugging that.. just not worth the time to same some money.

 

That said I just purchased a used card off of ebay from the states... hope it is a legit card.

 

I'll check out the various versions of firmware when I get the card.. hoping to just get it up and running at this point. I might do some testing as to whether to run both my arrays off of the same controller OR only use 4 ports on the controller and 4 sata ports on the motherboard.. Thinking 8 lane PCI express for all 8 drives might be better?

 

thanks for all the great info!

Mike

 

 

Ah yes, the glorious procedure of murder and execution mergers and acquisitions.
Alas, I can't really answer that moral dilemma for you.
 
 
I bought two from Taiwan, one from Hong Kong, as best as I can tell they
are genuine. They all had the same retail packaging despite coming from
different sources, and the accessories one would expect. Plus, the genuine
LSI firmware runs on all of them, so if they are fakes, they're pretty
good ones.

My general experience with ebay sellers from Hong Kong and region has
been pretty good to be honest. I think as long as you stick to sellers
with a good rating (high percentage of satisfied customers, preferably
high total numbers of successful transactions as well, and ideally they've
been around for a while), the chances of being duped aren't that high.
Not zero, but low enough that I'm usually not really worried about it.

Shipping times from Taiwan were much faster though, about one week to
Switzerland. Hong Kong can sometimes take a few weeks. For me that's
usually not an issue, but YMMV.

Still, I'm going to knock on some wood now, for my next purchase, just
to be on the safe side.

 

I have not personally tested this, so take my word with a grain of salt.
But as best as I have been able to find out without first-hand experience
of JBOD with IR firmware, when doing that, the drives will be presented
to the OS as logical drives instead of actual hardware, and some stuff may
not work properly.

I haven't actually been able to find that much on the subject, since it seems
that at some point somebody said "ZFS => IT MODE OR GTFO!" and we all just
followed that like the sheep that we are.
 

source: https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/edvq1GVT77o

I have read some other stuff on it over the years, but it's been in bits
and pieces, so I'm sorry I don't have any other links, let alone a true
and proper definite answer. I have also read on occasion from people using
IR firmware with ZFS and haven't heard about any horrible consequences
yet, but until I hear otherwise, I'm going to go with IT mode for ZFS (or
sofware RAID in general I suppose).

[alpenwasser]

I'll check out the various versions of firmware when I get the card.. hoping to just get it up and running at this point. I might do some testing as to whether to run both my arrays off of the same controller OR only use 4 ports on the controller and 4 sata ports on the motherboard.. Thinking 8 lane PCI express for all 8 drives might be better?

I seem to very vaguely recall reading about the 9211-8i topping out at ~2 GB/s with

a full complement of SSDs, so if you think your SSDs can do more, might be worth a shot

to split them across controllers. If you have redundancy in your array, depending on its

layout, it also might give you a higher resilience against failure (say, RAID10 across two

controllers with each drive in each mirror being connected to a different controller would

allow you to march on in case of a controller's failure, just as an example).

[techno10]

Oh yeah not sure why I didn't think of splitting arrays across controllers (thanks!).. Though if the speed difference between my 2 different controllers is too much it might cause a problem. I'll have to see.. I might just leave my 4 spinning drives alone on the motherboard controller for now as it's been working (and just to clarify I don't mean mixing my SSDs with my spinning drives.. I just meant only use 4 channels on the Avago controller for the SSDs and leave the spinning drives on the motherboard controller.. chances are the WD RED's wouldn't get much more speed anyways as I think they run at 5400 RPM (non-pro)). But maybe if I get bored I can move them to the avago controller and test. Also since I'm doing raid 5 with only 4 disks if I lose a controller my array is going offline anyways

 

So ultimately checking on the 2 firmware revisions... the non-IT firmware (think it's called IR or something?) is definitely for LSI/Avago's Raid setup.. Meaning I think you have to define each disk as a raid set before it'll appear in Linux. In doing this it will wipe the drives clean. I kind of want the data and also want to stick with the cheaper raid 5 software solution so I'm going to go with the IT firmware... IT firmware makes the system work without raid.. It can present the disks just fine without any raid definitions (I'm working nights the next 2 nights so I will try this on Saturday for sure).

 

If I remember the 9211-8i raid's implementation is still a software raid based solution anyways.. it's not true "hardware" raid anyways. For that you need to shell out 400+ bucks new and for 400 bucks you typically only get 4 connections, not 8. Don't quote me on that though

 

 

I seem to very vaguely recall reading about the 9211-8i topping out at ~2 GB/s with
a full complement of SSDs, so if you think your SSDs can do more, might be worth a shot
to split them across controllers. If you have redundancy in your array, depending on its
layout, it also might give you a higher resilience against failure (say, RAID10 across two
controllers with each drive in each mirror being connected to a different controller would
allow you to march on in case of a controller's failure, just as an example).

[techno10]

So, I finally got my 9211 card up and running. Bought 2 rosewill SFF-8087 cables from newegg.. The 4 ssd array is fast.. around 500 MB/s (that's a big MB/s ). My 4 disk raid 5 setup of WD Red's is slow... and during any weekly rebuild/check it becomes pretty much useless. I believe the max throughput is around 140 MB/S which isn't that bad for non-ssd but it just seems soooo slow (anyone have any ideas?) at around 6 TB the check/rebuild takes around 6-7 hours (which again I think it's NOT that bad but just seems slow).

 

a couple of notes in case anyone tries something similar:

1) I nearly died trying to get UEFI shell on 2 different motherboards to work.. This has to be the most frustrating, undocumented piece of crap ever.. I'm sure the intentions were good but it just never seemed to really take off just yet (I see it trying to become more popular). What I had to finally do is create a sub-directory on my USB drive called /EFI/boot and put the stupid shell files in there.. almost every single post said to place the files in the root directory on the USB drive but this just never, ever worked... I seriously wanted to go kick puppies because this didn't work for the longest time.

 

2) I'm not sure why but LSI/Avago software flat out sucks. The DOS and Linux updates just don't work.. would have been like 100 times easier if they did. I got the stupid PAL error for DOS and Linux said there were no available controllers found (though I didn't have the module loaded, see next note). That said I just tried it and I think the firmware updates WOULD have work if I had the stupid module loaded.....!!!! (wow, could have avoided #1 most likely as Linux seems to work):

 

[root@media1 Installer_P20_for_Linux]# ./sas2flash -listall

LSI Corporation SAS2 Flash Utility

Version 20.00.00.00 (2014.09.18)

Copyright © 2008-2014 LSI Corporation. All rights reserved

 

        Adapter Selected is a LSI SAS: SAS2008(B2)

 

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr

----------------------------------------------------------------------------

 

0  SAS2008(B2)     20.00.04.00    14.01.00.08    07.39.00.00     00:01:00:00

 

        Finished Processing Commands Successfully.

        Exiting SAS2Flash.

 

 

3) Fedora 22 now has the kernel modules in a separate RPM... so my card was being detected but no kernel modules were loaded and it's not built into the kernel by default.. this was kind of my fault but it's kind of stupid from a Linux perspective (i get why they want to do this but not sure I agree).

 

4) use "depmod -a" and "modprobe" for linux module detection.. I guess insmod isn't the same anymore

 

 

So I was my own worst enemy regarding trying to use uefi and not just digging into why I couldn't see the card in Linux initially.. let this be a lesson for others then

 

Mike

[braneopbru]

I feel your pain. Took me a while to get it figured out. 

 

At one point I thought I had killed the card; managed to erase the firmware on the card from the DOS utility but couldn't start the flash utility.

 

In the end I successfully used the EFI shell on a supermicro motherboard. I was able to get it to go by just putting the files on the root of a flash drive. If you want to speed up your boot times, you can erase the card, then reflash the 2118it.bin file, but skip the the mptsas2.rom file and it won't flash the boot rom to the card. should give you a good 5 second boost on boot up.

 

As far as the card not showing up in linux - I tried a couple of different flavours and never noticed anything weird. Linux just sees all the drives present. Actually, even in the BIOS I see all the drives plugged into the card and can pick any of those drives to boot to.

 

My card is just a standard LSI branded 9211-8i that was in IR mode.

[alpenwasser]

- snip -

Yup, sounds similar to my own martyrdom experiences. EFI shell was a bloody
PITA, but in the end it was the only thing which I could get to work. Thanks for the
info, I'll add a link to your post from my tutorial thread.

In the end I successfully used the EFI shell on a supermicro motherboard. I was able to get it to go by just putting the files on the root of a flash drive. If you want to speed up your boot times, you can erase the card, then reflash the 2118it.bin file, but skip the the mptsas2.rom file and it won't flash the boot rom to the card. should give you a good 5 second boost on boot up.

Interesting, wasn't aware of that. Thanks for the tip!