[Freenas] HBA firmware/driver mismatch. What now?

[dfsdfgfkjsefoiqzemnd]

I was messing around in the Freenas menus (setting up mail etc) when I noticed a blinking warning light.  I clicked it to find a "Firmware version 19 does not match driver version 16 for /dev/mps0" message.

It looks like the LSI 9211-8i card was flashed using firmware version 19, but FreeNAS only has version 16 of the driver. 

 

The support section of LSI's site only lists a version 20 driver for FreeBSD.  They say these are compatible with FreeNAS. 

I came across a link to version 16 firmware, but it doesn't look like that's the IT version.

 

What should I do?  Install the version 20 driver, install different firmware (I absolutely HATE flashing firmware because I usually brick it) or just ignore it?  Any other suggestions? 

I haven't noticed any issues really, but am reading about horror stories of pools disappearing etc.  Better to be safe than sorry.

 

If I do anything, do I need to back all my data up first or won't it affect the RAIDZ2 array that is connected to the card?  Logic dictates that a software RAID shouldn't be affected by a hardware flash.

 

Sorry for the noob questions, but there's lots of conflicting info on the net so I'd rather ask to make sure.  I really don't want to mess up my brand-new card, break the RAIDZ2 array or lose all my data later on.

[alpenwasser]

According to this thread: https://bugs.freenas.org/issues/6678

It seems there can indeed be serious issues with this.

So yeah, you could either manually upgrade your BSD drivers or reflash the card to a lower

firmware version. You can actually still download the P16 Firmware from LSI's site it seems.

I just swapped out the P20 with P16 in the URL, and it seems to work:

http://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus%20Adapters%20Common%20Files/SAS_SATA_6G_P16/9211_8i_Package_P16_IR_IT_Firmware_BIOS_for_MSDOS_Windows.zip

That's the IR/IT firmware package for Windows, if you you need something else, just try

it out starting with the P20 links on LSI's support page.

You can also find old versions with the search tool on LSI's site, then select 'archived'

versions of the results, but P16 somehow doesn't show up for me with that.

[MrBucket101]

@Captain Chaos In theory, you shouldn't need to backup your data (However, what you decide to do is your own choice). Software and hardware RAID store the RAID configs on the drives themselves. With software RAID, doesn't matter what host or device you are using, as long as you can see all the devices, you can reconstruct the RAID, usually fairly effortlessly.

 

Hardware RAID is almost the same, but you need to be using a RAID card from a compatible family. So if you have a LSI 92xx series card, any other 92xx series card, shouldn't have any difficulty recreating the array.

 

Anyways, as far as your data is concerned, I wouldn't worry about it. Installing drivers, and or flashing firmware is harmless. The cards have a backup flash, where the new firmware is flashed. Then on reboot, the card swaps out the flash and then installs the update itself. Helps prevent any problems. I've flashed the firmware on my card, live, 3x now. Nothing really changes till reboot.

 

Unfortunately, I'm not at all well versed in FreeBSD, or FreeNAS. But on ubuntu, installing/updating new drivers, usually causes the kernel to recompile itself to include the new driver.

 

I've also found there is a thin line between "Officially Supported" and it'll work just fine. Technically the Megaraid driver I have installed on my system is only supposed to be compatible with Ubuntu 11.04. I've used the same driver since 12.04, and I'm now running 14.04. I just keep upgrading and it keeps working, so yeah. Each upgrade I have to make a couple minor tweaks, but for the most part it just works. 

 

Anyways, onto the LSI support website, they should have an archive section, where you can find the older versions. Or at least they used to, I remember finding it a while back. I have to use a deprecated version of the LSI MegaRaid windows client since I'm using such an old driver. 


EDIT: disregard anything I said that conflicts with what @alpenwasser said. He's WAY more informed on FreeNAS than I am.

[dfsdfgfkjsefoiqzemnd]

@alpenwasser : So driver version 20 should work with firmware version 19? 

If so, I think I'll do that tomorrow (still copying data from the old WD EX4 to the new NAS, 6 hours to go but it's 1AM here). 

 

I need to shut it down to replace the side fan, label the hard drives and move it to its regular spot anyway.

[alpenwasser]

EDIT: disregard anything I said that conflicts with what @alpenwasser said. He's WAY more informed on FreeNAS than I am.

Funnily enough I've never actually run FreeNAS myself, never had a machine to try it out

on and my server runs Arch. I have played around with FreeBSD itself though at least, plus

since ZFS did the whole OpenZFS umbrella project thing it's reasonably similar across the

different platforms.

 

@alpenwasser : So driver version 20 should work with firmware version 19? 

If so, I think I'll do that tomorrow (still copying data from the old WD EX4 to the new NAS, 6 hours to go but it's 1AM here). 

 

I need to shut it down to replace the side fan, label the hard drives and move it to its regular spot anyway.

Honestly, I'm not sure, you'll need to try it out. If you take the download link to driver

version P20 and substitute the 20 for 19, you get the download link for P19 though, you could

try out that:

http://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus%20Adapters%20Common%20Files/SAS_SATA_6G_P19/Free_BSD_Driver_P19.zip

(That's the P19 FreeBSD driver)

It seems the old files stick around on LSI's site, so just pick whichever link for P20 you

need and substitute the P20 in the URL for the version you want. So far I haven't hit a dead

link yet.

[dfsdfgfkjsefoiqzemnd]

Right, this is WAY beyond my skill level and it appears that Google isn't my friend today either.

The P19 driver is not an option, it's missing one of the sub-directories (9.3.0). There are loads of other sub-directories (9.2.0, 9.1.0 etc) with files that have the same name, but I don't want to risk getting an incompatible one.

I did extract the needed file from the P20 driver.

According to the instructions, I now need to copy the file to FreeNAS' boot/kernel folder ... but how am I supposed to do that? I see no command line in the freenas menu and the FreeNAS USB stick flat-out refuses to mount under Windows as well as Linux.

Please don't tell me I need to attach a monitor to the NAS to get to the terminal, because I already know that the motherboard doesn't output any video signal whatsoever.

I could try to rip a 770 out of my main PC and use that as a video port, but that's a fair bit of work.

[MrBucket101]

Right, this is WAY beyond my skill level and it appears that Google isn't my friend today either.

According to the instructions, I now need to copy the file to FreeNAS' boot/kernel folder ... but how am I supposed to do that? I see no command line in the freenas menu and the FreeNAS USB stick flat-out refuses to mount under Windows as well as Linux.

does FreeNAS have ssh support?

 

Being your OS of choice, you'll need to hit the terminal every now and then. Better get used to to it now . Took me around 6mo to get proficient with my machine. But I forced myself to learn, not so bad anymore.

 

I'll browse around the LSI website, there should be an archive section.

 

EDIT: 

 

Here is the link to the FreeBSD P16 driver

http://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus%20Adapters%20Common%20Files/SAS_SATA_6G_P16/Free_BSD_Driver_P16.zip

 

EDIT #2: Is this the firmware you mentioned in the first post?

 

EDIT #3:

 

http://www.lsi.com/downloads/Public/Host%20Bus%20Adapters/Host%20Bus%20Adapters%20Common%20Files/SAS_SATA_6G_P16/9211_8i_Package_P16_IR_IT_Firmware_BIOS_for_MSDOS_Windows.zip

 

This should be the the P16 IT firmware that matches the P16 driver

 

Did I just win the internets?

[dfsdfgfkjsefoiqzemnd]

I have the correct firmware already, but I just don't find any way to apply it. I must put stuff in places and I don't have the faintest idea where these places are or how to put it there. I just seem to have no access whatsoever to the FreeNAS OS' folder structure via my PC or via the NAS itself.

I haven't tried SSH yet, looked into setting it up but it's another one of those guides for people to whom "command line" is their native language. It seems easier in Linux than having to go through the whole cygwin setup, so perhaps I'll try it that way. Unfortunately I'm working today so I'm stuck in Windows.

EDIt : I'm in using the netbook. Still not a clue on how to proceed now. I might have this cracked by 2026.

I usually find any solution to any problem within a couple of hours. Not this time though and it's really getting on my nerves.

I already looked into sending it somewhere to get it flashed, but no success there either.

[alpenwasser]

I have the correct firmware already, but I just don't find any way to apply it. I must put stuff in places and I don't have the faintest idea where these places are or how to put it there. I just seem to have no access whatsoever to the FreeNAS OS' folder structure via my PC or via the NAS itself.

Wait, are you trying to flash the P16 firmware onto your card so that it matches with

the P16 driver from FreeBSD, or are you trying to upgrade the FreeBSD driver to P19 so

that it matches with the card's firmware?

If you've already managed to flash the card once, why not just flash it again, but with the

different firmware version (P16 I reckon)? Since you have that part figured out (I think),

I'd guess that that'd be easier than trying to fiddle around with FreeBSD's drivers.

[dfsdfgfkjsefoiqzemnd]

@alpenwasser : I bought the card pre-flashed.

I did look into the P19 driver link you supplied, but that didn't have the correct folder structure so I decided not to take any risks and flash the P16 firmware. I have over $1000 worth of bricked cellphones here due to firmware flashes, so I'm really not looking forward to that.

The problem is that the whole command line and ssh thing is way beyond my skillset. I literally have no idea what I'm doing or what these commands even mean. I have very basic MS-DOS knowledge from when I was 5years old and playing games on my dad's 286 and I know how to update Linux via the command line, but that's about where it stops for me.

Currently looking at the firmware @MrBucket101 linked at the bottom of his post, but I assume I'd have to take the card out of the NAS and put it in the main PC then. It looks like I just have to boot the PC and run the .exe within the sas2flash_win_x64_rel folder.

[alpenwasser]

@alpenwasser : I bought the card pre-flashed.

Ah, sorry, had missed that.

I did look into the P19 driver link you supplied, but that didn't have the correct folder structure so I decided not to take any risks and flash the P16 firmware. I have over $1000 worth of bricked cellphones here due to firmware flashes, so I'm really not looking forward to that.

The problem is that the whole command line and ssh thing is way beyond my skillset. I literally have no idea what I'm doing or what these commands even mean.

Currently looking at the firmware @MrBucket101 linked at the bottom of his post, but I assume I'd have to take the card out of the NAS and put it in the main PC then. It looks like I just have to boot the PC and run the .exe within the sas2flash_win_x64_rel folder.

I did a tutorial on flashing the card controller here: http://linustechtips.com/main/topic/104425-flashing-a-lsi-9211-8i-raid-card-to-it-mode-for-zfssoftware-raid/

I recommend looking at the very last post in the thread, I haven't gotten around to

integrating that into the main tutorial, but it seems that if you have a Windows PC

available, that's the easiest way to do it.

And hey, look on the bright side: You're learning new stuff, even if it is annoying!

[dfsdfgfkjsefoiqzemnd]

I downloaded that software and knowing that I'd need Windows, I assumed that I had to put the LSI card in the PC itself.

I shut down the NAS, shut down the PC, took out one of the graphics cards (no free PCI-E slots), took the LSI card out of the NAS and put it inside the PC, booted the PC, launched the program ... and then see a box where I'm supposed to be entering the server's IP. Oh ffs !

So I shut down the PC again, took out the LSI card and put it back in the NAS, waited for the NAS to boot, launched the program again ... and while the NAS works just fine, the program doesn't find any server on either the 192.168.0.1 (my router's local IP) or 192.168.0.190 (the fixed local IP address I assigned to the NAS).
"Servers could not be found because sever may be down or server not installed in the host system"
Does that mean I need to run Windows on the NAS just to flash it? That would be the same as putting the card inside the PC itself.

I guess it's time to tag @Cheatdeath hoping for a more thorough explanation.

EDIT : So it looks like the LSI card should be in the PC after all and you then have to point the storage manager to the PC's own IP.  Seriously?

Currently inside the storage manager. With some luck I'll be able to flash it.

EDIT 2 : Nope, I can't flash it.  I went through the menus as Cheatdeath described, selected the 2118it.bin file, confirmed and immediately got "The firmware flash image attached NVDATA is invalid". :angry: Downloaded the thing 3 times to make sure it wasn't a corrupt download, but to no avail.

Perhaps I need to wipe the card first because I'm downgrading.  That's not possible with MSM though, so it looks like this whole option is off the table. 

[Cheatdeath]

I downloaded that software and knowing that I'd need Windows, I assumed that I had to put the LSI card in the PC itself.

I shut down the NAS, shut down the PC, took out one of the graphics cards (no free PCI-E slots), took the LSI card out of the NAS and put it inside the PC, booted the PC, launched the program ... and then see a box where I'm supposed to be entering the server's IP. Oh ffs !

So I shut down the PC again, took out the LSI card and put it back in the NAS, waited for the NAS to boot, launched the program again ... and while the NAS works just fine, the program doesn't find any server on either the 192.168.0.1 (my router's local IP) or 192.168.0.190 (the fixed local IP address I assigned to the NAS).

"Servers could not be found because sever may be down or server not installed in the host system"

Does that mean I need to run Windows on the NAS just to flash it? That would be the same as putting the card inside the PC itself.

I guess it's time to tag @Cheatdeath hoping for a more thorough explanation.

EDIT : So it looks like the LSI card should be in the PC after all and you then have to point the storage manager to the PC's own IP.  Seriously?

Currently inside the storage manager. With some luck I'll be able to flash it.

EDIT 2 : Nope, I can't flash it.  I went through the menus as Cheatdeath described, selected the 2118it.bin file, confirmed and immediately got "The firmware flash image attached NVDATA is invalid". :angry: Downloaded the thing 3 times to make sure it wasn't a corrupt download, but to no avail.

Perhaps I need to wipe the card first because I'm downgrading.  That's not possible with MSM though, so it looks like this whole option is off the table. 

 

First off the only thing I know for sure about flashing these cards is its a pain in the ass. I want to say I read somewhere that you cant go backwards on firmware/bios updates using MSM. I do not remember anything about checking IP or being asked for an IP when I used MSM. Have you tried the DOS method or EFI Shell?

[dfsdfgfkjsefoiqzemnd]

First off the only thing I know for sure about flashing these cards is its a pain in the ass. I want to say I read somewhere that you cant go backwards on updates using MSM. I do not remember anything about checking IP or being asked for an IP when I used MSM. Have you tried the DOS method or EFI Shell?

 

It's a pain in the ass allright.  I just launched it in Windows 7 and the first thing I was greeted with was a windows with a box where I was supposed to enter the IP. 

Oh well, I guess it doesn't matter now, this method won't work for me. 

 

Perhaps I should just get it over with and buy an unflashed card instead.  I think I'm going to do just that.  This whole downgrade nonsense is beyond stupid.

I spent 10 hours on it already and got nowhere.  If I had spent that time at work, I would have already made enough money to buy one that actually works without causing Freenas to throw warnings all over the place.

[alpenwasser]

First off the only thing I know for sure about flashing these cards is its a pain in the ass. I want to say I read somewhere that you cant go backwards on firmware/bios updates using MSM. I do not remember anything about checking IP or being asked for an IP when I used MSM. Have you tried the DOS method or EFI Shell?

 

 

It's a pain in the ass allright.  I just launched it in Windows 7 and the first thing I was greeted with was a windows with a box where I was supposed to enter the IP. 

Oh well, I guess it doesn't matter now, this method won't work for me. 

 

Perhaps I should just get it over with and buy an unflashed card instead.  I think I'm going to do just that.  This whole downgrade nonsense is beyond stupid.

I spent 10 hours on it already and got nowhere.  If I had spent that time at work, I would have already made enough money to buy one that actually works without causing Freenas to throw warnings all over the place.

Oh yes, pain in the ass indeed. It took me a lot of research and trial and error until

I had found a method that worked for me, and as is easily noticeable, it doesn't actually

work for everyone, there's often stuff which somehow gets in the way. Very frustrating

indeed. :angry:

I don't really have time today, but I might be able to look into how to use a newer

driver in FreeBSD tomorrow. If that works, no need to flash the card to a lower firmware.

Have you tried the other flashing method via EFI shell?

[dfsdfgfkjsefoiqzemnd]

I haven't.  Looked into it, but just haven't started it because it's a gazillion-step procedure including all kinds of things I just don't understand. EDIT : nvm, I had it confused with SSH. EFI turned out to be the solution, as explained in my next post. 

The furthest I got in any of the suggested procedures was to get access to the NAS via SSH, but I have no idea how to get the firmware or driver files to where they're supposed to be because the USB stick just can't be accessed apparently.

 

Flashing a card seems easy enough, downgrading it is another matter.

I don't have time for this anymore the next few days, I use my PC for work (working from home) and am already way too far behind because of this whole thing.

[MrBucket101]

It's a pain in the ass allright.  I just launched it in Windows 7 and the first thing I was greeted with was a windows with a box where I was supposed to enter the IP. 

Oh well, I guess it doesn't matter now, this method won't work for me. 

 

Perhaps I should just get it over with and buy an unflashed card instead.  I think I'm going to do just that.  This whole downgrade nonsense is beyond stupid.

I spent 10 hours on it already and got nowhere.  If I had spent that time at work, I would have already made enough money to buy one that actually works without causing Freenas to throw warnings all over the place.

I've never had any luck AT ALL, flashing my card with the MSM.

 

I always have to flash my card using the command line, using MegaCLI

give this a go? https://tinkertry.com/how-to-flash-the-firmware-of-your-lsi-raid-controller-using-megacli-on-a-dos-boot-usb-drive

 

EDIT: if you need to run some commands on the files, via ssh, heres what I do

 

copy the files onto a NAS share, from there you can access them via the command line.

 

find / -iname "<filename>"  

(may need root access)

will search the entire root directory for your filename, then spit out the path of anything that matches, then cd into that directory, and now you can act on the files.

[dfsdfgfkjsefoiqzemnd]

I came across this tutorial, which seems a lot easier than all the SSH stuff. I have the P16 firmware package and all the files mentioned in the tutorial are in there, so it looks like there's no real difference between downgrading from 19 to 16 or from 8 7.

According to the instructions I should start the server in DOS, but I know the server board doesn't output video so I'll be substituting the PC as server.

Disconnecting all the SSDs, taking out the bottom graphics card yet again to make room for the HBA, it's going to be fun. I really should build me a cheapo PC for this kind of stuff, I'm already getting tired of taking half the PC apart.

All the relevant files (sas2flsh for DOS, 2118it.bin and mptsas2.rom) are in the root of the DOS stick, so I should be good to go. Either I fix it or I brick it. Fingers crossed.

 

-----

 

EDIT : Currently on the netbook.  The PC says it has succesfully finished flashing the card to P16 firmware and to the new BIOS, so it should work without warnings now. 

Following the tutorial earlier in this post, I immediately ran into the "failed to initialize PAL" error, but luckily I came across this guide when I Ixquicked (or googled if you wish) the error. 

I downloaded the efi version of the sas2flash and put it on the DOS stick, rebooted the PC to EFI shell, identified the stick as fs0 and followed the guide form there on.

 

One tricky bit though : We use azerty over here, but DOS and these shells all use qwerty only.  I can type the A, Z, Q and W correctly without giving it much thought, but I'm too used to having the M key next to the L, not next to the N.  Also all the symbols are in different positions so I usually have to mash buttons until I find what I'm looking for.

 

Time to shut down the PC and put it back together again.  I'll edit again if it all works as it should

-----

EDIT 2 : Eureka ! The card is running firmware version 16 and the error is gone.

The RAIDZ2 array with all the data is intact, so I can finally shut down the other NASes and move this one to its place (the kitchen, where the EX4 used to be)

What an ordeal, but what a relief too.

Thanks for the help, guys and/or girls

[MrBucket101]

-snip-

I always loved/hated the feeling of upgrading. For me, backing up 20TB just isn't an option. I mean I built the damn server to store all the data, and now I have to spend another grand on HDD's just to back it all up, "nah it'll be fine".

 

You go through with the upgrade and get a sinking feeling in your gut, and then BOOM! goes the dynamite, and it works. It's such a rush haha.

 

Even knowing ahead of time, it should save your data, it's still one helluva ride.

[dfsdfgfkjsefoiqzemnd]

Oh for f***s sake !

I let the NAS update yesterday, and now I'm getting this warning :

So FreeNAS updated the driver, and now my firmware is out of date. Now I can flash the stupid thing again !

[MrBucket101]

Oh for f***s sake !

I let the NAS update yesterday, and now I'm getting this warning :

So FreeNAS updated the driver, and now my firmware is out of date. Now I can flash the stupid thing again !

 

Do you just need to flash it to the newest version?

[dfsdfgfkjsefoiqzemnd]

Do you just need to flash it to the newest version?

 

Yeah, luckily this time it's a newer version, so it shouldn't be that difficult. 

Still, I need to shut down the NAS, remove it from its corner so I can open it, take the card out, shut down the PC, remove one of the graphics cards (so I have a free PCIe x16 slot), do the flash, then rebuild everything. 

 

I hope this doesn't happen too often, otherwise I may look into ditching Freenas in favor of Linux. 

 

Come to think of it, that wouldn't be such a bad idea really.  I could add another couple of SSDs then and host my own site and mail server then.

[MrBucket101]

Yeah, luckily this time it's a newer version, so it shouldn't be that difficult. 

Still, I need to shut down the NAS, remove it from its corner so I can open it, take the card out, shut down the PC, remove one of the graphics cards (so I have a free PCIe x16 slot), do the flash, then rebuild everything. 

 

I hope this doesn't happen too often, otherwise I may look into ditching Freenas in favor of Linux. 

 

Come to think of it, that wouldn't be such a bad idea really.  I could add another couple of SSDs then and host my own site and mail server then.

That's why I use Ubuntu

 

Lots of support, and you can do SO MUCH WITH IT

 

There is ZFS support in ubuntu, so I don't even think you'll need to destroy your array, just reimport it or something along those lines

FreeNAS isn't meant to do anything crazy, but it is good at what it can do.