Bad HBA card? FreeNAS errors

[Thirdgen89GTA]

Thats what I'm thinking it is.

 

2 days ago my FreeNAS server rebooted unexpectedly. After reviewing the logs I found these messages:

 

Now, I've had CAM errors occasionally before with a bad SATA cable in an older build.  But it was very odd to see these resets to see, and affecting 3 different drives.

 

My Server uses a NORCO 4224 case with 6 backplanes.

 

da11, and 13 are on the 4th backplane.

da 14 is on another backplane.

 

Each backplane uses its own SFF-8087 cable, so the drives don't even share a cable.

 

The only common component between them is the SAS 9211-8i HBA card.

Aug  8 00:00:00 Asgard syslog-ng[2491]: Configuration reload request received, reloading configuration;
Aug  8 00:00:00 Asgard syslog-ng[2491]: Configuration reload finished;
Aug  8 04:27:27 Asgard  (da11:mps1:0:9:0): READ(16). CDB: 88 00 00 00 00 01 2b 2f c1 a8 00 00 00 40 00 00 length 32768 SMID 409 Aborting command 0xfffffe00011138d0
Aug  8 04:27:27 Asgard mps1: Sending reset from mpssas_send_abort for target ID 9
Aug  8 04:27:27 Asgard  (da14:mps1:0:26:0): WRITE(10). CDB: 2a 00 0b 02 b7 00 00 00 08 00 length 4096 SMID 194 Aborting command 0xfffffe0001101ea0
Aug  8 04:27:27 Asgard mps1: Sending reset from mpssas_send_abort for target ID 26
Aug  8 04:27:27 Asgard  (da13:mps1:0:25:0): WRITE(10). CDB: 2a 00 0b 02 b7 00 00 00 10 00 length 8192 SMID 353 Aborting command 0xfffffe000110ef50
Aug  8 04:27:27 Asgard mps1: Sending reset from mpssas_send_abort for target ID 25

Server Reboots here.

Aug  8 04:35:49 Asgard syslog-ng[2661]: syslog-ng starting up; version='3.20.1'
Aug  8 04:35:49 Asgard Copyright (c) 1992-2018 The FreeBSD Project.
Aug  8 04:35:49 Asgard Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Aug  8 04:35:49 Asgard  The Regents of the University of California. All rights reserved.
Aug  8 04:35:49 Asgard FreeBSD is a registered trademark of The FreeBSD Foundation.
Aug  8 04:35:49 Asgard FreeBSD 11.2-STABLE #0 r325575+6aad246318c(HEAD): Mon Jun 24 17:25:47 UTC 2019
Aug  8 04:35:49 Asgard root@nemesis:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
Aug  8 04:35:49 Asgard FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
Aug  8 04:35:49 Asgard VT(efifb): resolution 1280x1024
Aug  8 04:35:49 Asgard CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz (4000.08-MHz K8-class CPU)

MPS1 is my 9211-8i card.

mps1: <Avago Technologies (LSI) SAS2008> port 0xd000-0xd0ff mem 0xf7dc0000-0xf7dc3fff,0xf7d80000-0xf7dbffff irq 17 at device 0.0 on pci2
mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd

A day later and I come home and DHCP is entirely non-functional on my VLAN. Nothing could get an IP.  I spent about 20 minutes trouble shooting the router and testing with various things.  But then I noticed that all of my WiFi stuff was working fine.  WiFi is on a separate VLAN, and DHCP was working there.  So I started disconnecting network cables.  As soon as I disconnected the cable leading to my NAS DHCP started working again.

 

When I got to my FreeNAS server, I found it hard locked.  Reset button wouldn't work.  Had to hard power cycle the server.

 

When the server rebooted, I found more reset's in the log.

Aug  8 11:38:30 Asgard mps1: IOC Fault 0x40002622, Resetting
Aug  8 11:38:30 Asgard mps1: Reinitializing controller,
Aug  8 11:38:30 Asgard mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
Aug  8 11:38:30 Asgard mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Aug  8 11:38:30 Asgard mps1: mps_reinit finished sc 0xfffffe00010ce000 post 4 free 3
Aug  8 11:38:31 Asgard mps1: SAS Address for SATA device = dc271f4dc5b18f80
Aug  8 11:38:31 Asgard mps1: SAS Address from SATA device = dc271f4dc5b18f80
Aug  8 11:38:31 Asgard mps1: SAS Address for SATA device = 78928e54100f94a5
Aug  8 11:38:31 Asgard mps1: SAS Address from SATA device = 78928e54100f94a5
Aug  8 11:38:31 Asgard mps1: SAS Address for SATA device = dd3e2c4dc5b18f80
Aug  8 11:38:31 Asgard mps1: SAS Address from SATA device = dd3e2c4dc5b18f80
Aug  8 11:38:31 Asgard mps1: SAS Address for SATA device = 78928e54100f95a2
Aug  8 11:38:31 Asgard mps1: SAS Address from SATA device = 78928e54100f95a2
Aug  8 11:41:14 Asgard mps1: IOC Fault 0x40002622, Resetting
Aug  8 11:41:14 Asgard mps1: Reinitializing controller,
Aug  8 11:41:14 Asgard mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
Aug  8 11:41:14 Asgard mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Aug  8 11:41:14 Asgard mps1: mps_reinit finished sc 0xfffffe00010ce000 post 4 free 3
Aug  8 11:41:15 Asgard mps1: SAS Address for SATA device = dc271f4dc5b18f80
Aug  8 11:41:15 Asgard mps1: SAS Address from SATA device = dc271f4dc5b18f80
Aug  8 11:41:15 Asgard mps1: SAS Address for SATA device = 78928e54100f94a5
Aug  8 11:41:15 Asgard mps1: SAS Address from SATA device = 78928e54100f94a5
Aug  8 11:41:16 Asgard mps1: SAS Address for SATA device = dd3e2c4dc5b18f80
Aug  8 11:41:16 Asgard mps1: SAS Address from SATA device = dd3e2c4dc5b18f80
Aug  8 11:41:16 Asgard mps1: SAS Address for SATA device = 78928e54100f95a2
Aug  8 11:41:16 Asgard mps1: SAS Address from SATA device = 78928e54100f95a2

This is where I did the hard reset.  You can see the time jump.

Aug  8 16:59:32 Asgard syslog-ng[2504]: syslog-ng starting up; version='3.20.1'
Aug  8 16:59:32 Asgard Copyright (c) 1992-2018 The FreeBSD Project.
Aug  8 16:59:32 Asgard Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Aug  8 16:59:32 Asgard  The Regents of the University of California. All rights reserved.

 

 

I moved all of the drives over to the other backplanes which are connected to 9201-16i HBA card hoping it would help.  Because the issue caused one of my drives to accumulate 100+ UDMA CRC errors.  Which annoys me because I know its not the drive but now those errors are permanently there.

 

I checked a bit ago again, and the server is hard locked once more.

 

When I go home from work today I'll shut the server down and remove the 9211-8i card then see what kind of issues pop-up.  Super frustrating.  A 9211-8i isn't expensive, I just don't want to spend it.  Grrr.

[Windows7ge]

Definitely try it in another system and see if it keeps showing errors. Is it getting adequate airflow? Those get pretty hot and the airflow from the fan wall in the 4224 really isn't enough. You really need a fan sitting on the LSI controllers.

 

Also the 9211-8i is a RAID controller. Have you flashed it to IT mode? If not you really shouldn't be using it with FreeNAS. ZFS likes direct access to the drives and RAID controllers screw with that. If you have to replace it consider the LSI 9207-8i. It's an HBA.

[Thirdgen89GTA]

9 hours ago, Windows7ge said:

Definitely try it in another system and see if it keeps showing errors. Is it getting adequate airflow? Those get pretty hot and the airflow from the fan wall in the 4224 really isn't enough. You really need a fan sitting on the LSI controllers.

 

Also the 9211-8i is a RAID controller. Have you flashed it to IT mode? If not you really shouldn't be using it with FreeNAS. ZFS likes direct access to the drives and RAID controllers screw with that. If you have to replace it consider the LSI 9207-8i. It's an HBA.

I replaced all of the OEM fans on it as soon as I got it with Noctua 120mm fans.  My 4224 came with the 120mm x 3 fan wall.. So that's fine. Drive temps typically idle in the 34-36c range.  My SSD array usually idles in the 31-32c range because its just a 2.5" drive in a 3.5" bay, so more air flow over the top.

 

Even during peak loads the temps overall in the case are pretty good. Loads like maxing out Plex till it buffers, which is around 9 simultaneous 1080p h264 transcodes.  Even at 100% cpu loads during heavy transcoding tests with Plex it stays under 70c.  That said, I was eying some of Noctua's 3000 series fans recently.  But if I have to buy a new HBA, I won't be buying the fans.

 

Ever since I moved all of the drives to the other HBA, the 9201-16i, its been stable.

 

However, on the last scrub, it reported Checksum errors on one of the drives, ended up resilvering 64k of data.  I think thats because it froze twice in the middle of a scrub with the 9211 controller acting up and I had to hard reset it.  So I assume there was some data that wasn't flushed to disk based on all the ICRC errors.

 

I took the offending drive offline out of the pool and am doing a long SMART test on it now.  That won't finish till sometime tomorrow.  18hrs.

 

And yes, its flashed to IT mode, was the first thing I did after taking it out of the box.  Its the happy good firmware version.

 

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01) 
Copyright (c) 2008-2013 LSI Corporation. All rights reserved 

	Adapter Selected is a LSI SAS: SAS2008(B2)   

	Controller Number              : 1
	Controller                     : SAS2008(B2)   
	PCI Address                    : 00:02:00:00
	SAS Address                    : 500605b-0-0341-cb50
	NVDATA Version (Default)       : 14.01.00.08
	NVDATA Version (Persistent)    : 14.01.00.08
	Firmware Product ID            : 0x2213 (IT)
	Firmware Version               : 20.00.07.00
	NVDATA Vendor                  : LSI
	NVDATA Product ID              : SAS9211-8i
	BIOS Version                   : 07.39.02.00
	UEFI BSD Version               : N/A
	FCODE Version                  : N/A
	Board Name                     : SAS9211-8i
	Board Assembly                 : H3-25250-02B
	Board Tracer Number            : SP10600615

	Finished Processing Commands Successfully.
	Exiting SAS2Flash.

 

And the offending drive.  The SMART log was taking during the middle of the Scrub, so basically 100% activity for almost 16 hours.  I'm not surprised it was at 41c.  I'll be sending the drive into WD for warranty, had one drive fail on me last year and WD sent back a brand new Red with the 256MB cache.  Hoping to get the same this time around.

 

The good news is no relocated or unrecoverable sectors. So, pending the SMART long test results, the drive is most likely fine.  Still going to send it in though.

 

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   132   132   054    Pre-fail  Offline      -       112
  3 Spin_Up_Time            0x0007   166   166   024    Pre-fail  Always       -       345 (Average 449)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       33
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       16472
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       33
 22 Unknown_Attribute       0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   076   076   000    Old_age   Always       -       29600
193 Load_Cycle_Count        0x0012   076   076   000    Old_age   Always       -       29600
194 Temperature_Celsius     0x0002   146   146   000    Old_age   Always       -       41 (Min/Max 24/44)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       102

SMART Error Log Version: 1
ATA Error Count: 102 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 102 occurred at disk power-on lifetime: 108 hours (4 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 10 a0 78 4a d6 40 08   4d+09:48:49.407  READ FPDMA QUEUED
  60 18 b0 90 26 da 40 08   4d+09:48:49.405  READ FPDMA QUEUED
  60 18 a8 e8 b7 d7 40 08   4d+09:48:49.402  READ FPDMA QUEUED
  60 18 98 d0 d9 d3 40 08   4d+09:48:49.396  READ FPDMA QUEUED
  60 18 90 80 0d d1 40 08   4d+09:48:49.394  READ FPDMA QUEUED

Error 101 occurred at disk power-on lifetime: 107 hours (4 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 f8 d0 51 8d 40 08   4d+09:43:15.474  READ FPDMA QUEUED
  61 00 30 40 69 5c 40 08   4d+09:43:15.473  WRITE FPDMA QUEUED
  61 00 28 40 68 5c 40 08   4d+09:43:15.472  WRITE FPDMA QUEUED
  61 00 20 40 67 5c 40 08   4d+09:43:15.472  WRITE FPDMA QUEUED
  61 00 18 40 66 5c 40 08   4d+09:43:15.472  WRITE FPDMA QUEUED

Error 100 occurred at disk power-on lifetime: 107 hours (4 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 c8 28 02 7f 40 08   4d+09:40:31.756  READ FPDMA QUEUED
  60 00 d8 68 05 7f 40 08   4d+09:40:31.736  READ FPDMA QUEUED
  60 c0 d0 28 03 7f 40 08   4d+09:40:31.736  READ FPDMA QUEUED
  ef 02 00 00 00 00 40 08   4d+09:40:31.734  SET FEATURES [Enable write cache]
  ef aa 00 00 00 00 40 08   4d+09:40:31.732  SET FEATURES [Enable read look-ahead]

Error 99 occurred at disk power-on lifetime: 107 hours (4 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 40 e0 38 0f 6d 40 08   4d+09:36:40.613  READ FPDMA QUEUED
  61 80 d8 a0 4a b5 40 08   4d+09:36:40.613  WRITE FPDMA QUEUED
  61 40 d0 60 4a b5 40 08   4d+09:36:40.612  WRITE FPDMA QUEUED
  60 40 c8 f8 0e 6d 40 08   4d+09:36:40.612  READ FPDMA QUEUED
  60 40 c0 38 0d 6d 40 08   4d+09:36:40.610  READ FPDMA QUEUED

Error 98 occurred at disk power-on lifetime: 107 hours (4 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 c0 60 79 67 40 08   4d+09:35:34.864  READ FPDMA QUEUED
  60 00 00 60 7a 67 40 08   4d+09:35:34.864  READ FPDMA QUEUED
  61 00 f8 a0 86 4e 40 08   4d+09:35:34.852  WRITE FPDMA QUEUED
  61 00 f0 a0 85 4e 40 08   4d+09:35:34.852  WRITE FPDMA QUEUED
  61 00 e8 a0 84 4e 40 08   4d+09:35:34.851  WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     16410         -
# 2  Short offline       Completed without error       00%     16338         -
# 3  Short offline       Completed without error       00%     16290         -
# 4  Short offline       Completed without error       00%     16242         -
# 5  Short offline       Completed without error       00%     16170         -
# 6  Short offline       Completed without error       00%     16122         -
# 7  Short offline       Completed without error       00%     16074         -
# 8  Short offline       Completed without error       00%     16002         -
# 9  Short offline       Completed without error       00%     15954         -
#10  Short offline       Completed without error       00%     15906         -
#11  Short offline       Completed without error       00%     15834         -
#12  Short offline       Completed without error       00%     15786         -
#13  Short offline       Completed without error       00%     15738         -
#14  Short offline       Completed without error       00%     15668         -
#15  Short offline       Completed without error       00%     15620         -
#16  Short offline       Completed without error       00%     15572         -
#17  Extended offline    Completed without error       00%     15518         -
#18  Short offline       Completed without error       00%     15500         -
#19  Short offline       Completed without error       00%     15452         -
#20  Short offline       Completed without error       00%     15404         -
#21  Short offline       Completed without error       00%     15332         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Lots of interface errors showed up during the times when the HBA crashed the server out.

 

There haven't been any more ICRC errors since I moved them all to the 9201 controller.  So after the SMART test completes I'll likely order a new 9211.  Getting off of it made all the instability go away.

[Windows7ge]

Have you tried putting the 9211 in another system yet? See if errors from it persist?

[Thirdgen89GTA]

No other systems to put it into.  Nothing else has SFF-8087 backplanes to connect it to.  And I'm not going to order break-out cables.  Don't feel like disassembling the gaming PC to put the case, the gaming PC is a mini-itx anyways with only once PCI-E slot.

 

I ordered a new 9211.  The system has been performing absolutely fine for about 2 years now with no changes to the backplanes or cables.  Then suddenly becomes completely unstable until I take all the drives off it.   I can always return the new 9211 if I don't use it.  This way I'll have it sooner rather than later.

 

The drives are working fine on the other controller.  After SMART completes on the drive showing check-sum errors I'll look at the log and if it looks fine I"ll run another scrub.  Still will likely send the drive in for RMA anyways. 

 

Calling it a bad card, can't see two backplanes, and two SFF cables going bad at the exact same time.

[Thirdgen89GTA]

So, after a long Smart test, no errors.

 

Was 100% stable as soon as all drives were off the 9211.

 

Thought about ordering another 9201-16i, but cheapest I found was $130 with shipping.  The reason being I could use SFF-8087 to 8088 adapter to give me two external SAS ports for expansion later if I so chose.

 

But reality set in, if I really do need more than 24 bays, I'll spend the money then.

[Thirdgen89GTA]

Finally, no checksum errors found during the scrub.  It was all the 9211.

 

New one is in the mail.