LSI MegaRaid controller with 10TB drives?

[bfairs]

I have an LSI MegaRaid SAS 9271-4i controller currently running 4x4TB WD SE drives in RAID10.  I'm wanting to upgrade to 4x10TB Seagate Enterprise ST10000NM0016 (Petabyte project drives I believe), and I'm wondering if I'll have compatibility issues with my RAID card.  Broadcom has a compatibility report, but it hasn't been updated since Nov 26, 2016, and doesn't list any 10TB drives.  I know Linus used LSI controllers, but I doubt the same model I have, so does anyone know if I'll have issues? I don't really want to buy the drives and find out I need to upgrade my raid card. Thanks!

[leadeater]

It'll work fine, modern cards with the newer firmware and partitioning support very large disks. The theoretical limit is something really big. Only the really old stuff had problems with disks larger than 2TB, some cards got firmware updates to support it.

 

Also why are you using RAID 10? If the array is for file storage RAID 6 is actually higher performance for that I/O pattern and you'll get much more usable space.

[bfairs]

3 minutes ago, leadeater said:

It'll work fine, modern cards with the newer firmware and partitioning support very large disks. The theoretical limit is something really big. Only the really old stuff had problems with disks larger than 2TB, some cards got firmware updates to support it.

 

Also why are you using RAID 10? If the array is for file storage RAID 6 is actually higher performance for the I/O pattern and you'll get much more usable space.

That's good to know, we were pretty certain it would work, but wanted to test the waters before investing a couple grand.

 

The drives are used in a storage server that acts as an archive, but files are also actively edited from across the network.  For the most part the files are emails, spreadsheets, AutoCAD and pictures.  As to why RAID10 over RAID6: when initially setting it up a couple of years ago, I read that 10 is faster and safer than 6.  With 4 drives, they also have the same capacity.  Is the opposite true though?

[leadeater]

18 minutes ago, bfairs said:

As to why RAID10 over RAID6: when initially setting it up a couple of years ago, I read that 10 is faster and safer than 6.  With 4 drives, they also have the same capacity.  Is the opposite true though?

For sequential reading and writing RAID 6 is faster because there are more active disk spindles that can be used for I/O operations, RAID 10 you lose half of them. Modern RAID cards such as yours have much faster processors on them so can do parity calculations very quickly and with write-back cache the overall achievable performance is very good, can be much higher than RAID 10.

 

RAID 10 is still recommended for high random write I/O tasks were latency consistency is important such as database servers. However this is becoming less true, for quite a few years now all enterprise disk systems pooled disks in a RAID 6 like manor.

 

As for redundancy and reliability of RAID 6 vs RAID 10 in practical real world terms both are effectively as safe as each other. RAID 10 can become less safe than RAID 6 if there is a very large number of disks in the array because the chances of a 2 disk failure in the same mirror happening increases, this usually happens after a disk has failed and you put in a new one and start the rebuild.

 

The most common time for disks to fail is during a rebuild as disks get stressed much more than usual for a long period of time and if a disk fails during the rebuild and it's a RAID 10 array the whole array is dead, if it's the partner disk in the mirror pair being rebuilt but that is the most likely disk to fail. RAID 5/6 have their own downside too for array rebuilds as every disk in the array gets stressed during a rebuild and if all the disks are old you can get a cascading disk failure and lose the array.

 

Basically the most risky time for a RAID array is a rebuild after a failed disk has been replaced, rather ironic if you think about why RAID exists in the first place.

 

For some actual performance differences of RAID 10 vs RAID 6 for the type of use case you have @scottyseng should be able to give you those, he converted over to RAID 6 a while ago.

[TheCherryKing]

High capacity drives should work fine with a RAID SAS 2 or SAS 3 RAID controller. 

[scottyseng]

5 hours ago, bfairs said:

-snip-

The RAID card should be able to easily handle 10TB drives.

 

I would recommend RAID6 as well, I used to run RAID10 but ah, I realized the speed for random IO wasn't really important for my use (just data storage of files). Also, only RAID5 and 6 are expandable for LSI RAID cards. With the other RAID levels, you either have to make a whole new array, or overwrite the existing one. I've abused this feature to go from four to eight to ten drives in my RAID6 array.

 

If I remember right, I had 4 4TB WD Reds pushing 350 MB/s sequential in RAID10. Around 400MB/s in RAID6 (though random IO went down a little). Indeed you can see a boost in speed even with the parity calculations as all of the drives are being used in RAID6.

 

I would say your 4 x 10TB drives should be able to net you around 450MB/s+ in RAID6. My WD Reds only push out 140MB/s per drive. I think the denser drives usually push 180-200MB/s per drive.

[bfairs]

On 15/09/2017 at 8:13 PM, scottyseng said:

*snip

Thanks for the input. With the new drives I'll need to rewrite the array anyway so it's as good a time as any to switch to RAID6.  So with my 4 channel card, in the future I could expand this array with a SAS expander card? And by doing so I wouldn't have to make a new array or anything?

 

FYI this whole situation came up because one of the drives is about to die (or did die?); the RAID card started blaring at us and the drive health was at 9%. We didn't have a cold backup so it was either trying to track down a matching drive, or use the opportunity to just upgrade the whole array. We went with the latter (drives are in hand, going to do the swap tomorrow).

[scottyseng]

Ah, yeah, sounds like a good time to switch over. I actually had to buy drives to put into RAID0 temporarily to store my data while the RAID10 array changed over to a RAID6 one. 

 

Yes, you can expand it out with a SAS expander card, or alternatively buy a used SAS 2 (SAS 6Gb/s) SuperMicro server (They have SAS expander backplanes built in). Just be weary to get a SAS 2 backplane, they tend to sell the older SAS1 (3Gb/s) on ebay. 

 

Yep, no need to make a new array. The RAID card senses the drives. The SAS expander is literally just a pass through device. I went from direct wire (SAS to 4 SATA) in my PC. Then I built my server, plugged the two ports of my LSI RAID card into the server expander backplane, and moved my drives into the front bays. No issues. Now I have a second server that I feed SAS wire from the first backplane into the second. I can run 48 drives now (Though I currently only have 16). 

 

Hmm, surprised your RAID card didn't kick it already. My LSI RAID card is very kick happy on the drives. Like I bought refurbished server hard drives (never again), and the instant one of them had read errors, my card kicked them from the array. RAID6 saved me because I had two of these drives go out at one point and paranoia got to the better of me and I ordered a new server drive to hold the array while the other two were being RMA'd. 

 

Yeah, for me, it's getting harder to get WD Re SAS 4TB drives. They made a new line of WD Gold drives, but they're SATA only at the moment. You can't mix SAS and SATA drives in the same RAID array. 

[bfairs]

8 minutes ago, scottyseng said:

 

That's all good to know.  We're just going all SATA drives still (SAS to 4 SATA), but it seems like SAS 2 is OK with SATA? I'm not fully versed in all of the protocols.  It's more of a "it's great if it will work in the future" if we need it, but not even something we were thinking of doing in the first place.

[leadeater]

36 minutes ago, scottyseng said:

Yeah, for me, it's getting harder to get WD Re SAS 4TB drives. They made a new line of WD Gold drives, but they're SATA only at the moment. You can't mix SAS and SATA drives in the same RAID array. 

Switch to HGST, still WD technically, but that is where all the SAS offerings are now.

[scottyseng]

17 minutes ago, bfairs said:

That's all good to know.  We're just going all SATA drives still (SAS to 4 SATA), but it seems like SAS 2 is OK with SATA? I'm not fully versed in all of the protocols.  It's more of a "it's great if it will work in the future" if we need it, but not even something we were thinking of doing in the first place.

Yep, SAS2 (6Gb/s) works fine with SATA3 (6Gb/s). Yeah, it's very confusing with the terminology. 

[bfairs]

 

On 15/09/2017 at 2:36 PM, leadeater said:

 

 

On 15/09/2017 at 8:13 PM, scottyseng said:

 

RAID is built and all is well! There were some minor hiccups, mostly user error getting the data from the old RAID to the new. I did a clone to a storage drive, then cloned to the new array. However I didn't realize the sector size was 4KB which meant I could only use 16TB of my new 18.1TB array. I did manage to expand the sector size (only went to 16KB) and can fully utilize the whole array, as well as a good bit of future expansion.

 

I did a quick benchmark before and after (as well as with a single 10TB drive). Overall an improvement over the old 4TB WB Se drives.

 

Old 4x4TB WD Se RAID10

Spoiler

-----------------------------------------------------------------------
CrystalDiskMark 5.2.2 x64 (C) 2007-2017 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   739.435 MB/s
  Sequential Write (Q= 32,T= 1) :   391.728 MB/s
  Random Read 4KiB (Q= 32,T= 1) :     5.300 MB/s [  1293.9 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     9.461 MB/s [  2309.8 IOPS]
         Sequential Read (T= 1) :   434.380 MB/s
        Sequential Write (T= 1) :   368.229 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     2.253 MB/s [   550.0 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     5.092 MB/s [  1243.2 IOPS]

  Test : 1024 MiB [L: 34.8% (2593.2/7450.9 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/09/21 10:09:03
    OS : Windows Server 2012 R2  [6.3 Build 9600] (x64)
  

2x10TB Seagate RAID1

Spoiler

-----------------------------------------------------------------------
CrystalDiskMark 5.2.2 x64 (C) 2007-2017 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   879.845 MB/s
  Sequential Write (Q= 32,T= 1) :   187.976 MB/s
  Random Read 4KiB (Q= 32,T= 1) :    11.433 MB/s [  2791.3 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     1.033 MB/s [   252.2 IOPS]
         Sequential Read (T= 1) :   541.348 MB/s
        Sequential Write (T= 1) :    74.441 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     5.802 MB/s [  1416.5 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     0.416 MB/s [   101.6 IOPS]

  Test : 1024 MiB [F: 34.8% (2593.0/7450.9 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/09/21 10:14:55
    OS : Windows Server 2012 R2  [6.3 Build 9600] (x64)
  

4x10TB Seagate RAID6

Spoiler

-----------------------------------------------------------------------
CrystalDiskMark 5.2.2 x64 (C) 2007-2017 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   738.994 MB/s
  Sequential Write (Q= 32,T= 1) :   568.160 MB/s
  Random Read 4KiB (Q= 32,T= 1) :    18.030 MB/s [  4401.9 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     6.428 MB/s [  1569.3 IOPS]
         Sequential Read (T= 1) :   675.317 MB/s
        Sequential Write (T= 1) :   562.801 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     3.888 MB/s [   949.2 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     3.298 MB/s [   805.2 IOPS]

  Test : 1024 MiB [L: 0.0% (0.5/18626.9 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/09/21 11:23:58
    OS : Windows Server 2012 R2  [6.3 Build 9600] (x64)
  

 

[TheCherryKing]

8 minutes ago, bfairs said:

 

 

RAID is built and all is well! There were some minor hiccups, mostly user error getting the data from the old RAID to the new. I did a clone to a storage drive, then cloned to the new array. However I didn't realize the sector size was 4KB which meant I could only use 16TB of my new 18.1TB array. I did manage to expand the sector size (only went to 16KB) and can fully utilize the whole array, as well as a good bit of future expansion.

 

I did a quick benchmark before and after (as well as with a single 10TB drive). Overall an improvement over the old 4TB WB Se drives.

 

Old 4x4TB WD Se RAID10

  Hide contents

-----------------------------------------------------------------------
CrystalDiskMark 5.2.2 x64 (C) 2007-2017 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   739.435 MB/s
  Sequential Write (Q= 32,T= 1) :   391.728 MB/s
  Random Read 4KiB (Q= 32,T= 1) :     5.300 MB/s [  1293.9 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     9.461 MB/s [  2309.8 IOPS]
         Sequential Read (T= 1) :   434.380 MB/s
        Sequential Write (T= 1) :   368.229 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     2.253 MB/s [   550.0 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     5.092 MB/s [  1243.2 IOPS]

  Test : 1024 MiB [L: 34.8% (2593.2/7450.9 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/09/21 10:09:03
    OS : Windows Server 2012 R2  [6.3 Build 9600] (x64)
  

2x10TB Seagate RAID1

  Hide contents

-----------------------------------------------------------------------
CrystalDiskMark 5.2.2 x64 (C) 2007-2017 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   879.845 MB/s
  Sequential Write (Q= 32,T= 1) :   187.976 MB/s
  Random Read 4KiB (Q= 32,T= 1) :    11.433 MB/s [  2791.3 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     1.033 MB/s [   252.2 IOPS]
         Sequential Read (T= 1) :   541.348 MB/s
        Sequential Write (T= 1) :    74.441 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     5.802 MB/s [  1416.5 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     0.416 MB/s [   101.6 IOPS]

  Test : 1024 MiB [F: 34.8% (2593.0/7450.9 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/09/21 10:14:55
    OS : Windows Server 2012 R2  [6.3 Build 9600] (x64)
  

4x10TB Seagate RAID6

  Reveal hidden contents

-----------------------------------------------------------------------
CrystalDiskMark 5.2.2 x64 (C) 2007-2017 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   738.994 MB/s
  Sequential Write (Q= 32,T= 1) :   568.160 MB/s
  Random Read 4KiB (Q= 32,T= 1) :    18.030 MB/s [  4401.9 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     6.428 MB/s [  1569.3 IOPS]
         Sequential Read (T= 1) :   675.317 MB/s
        Sequential Write (T= 1) :   562.801 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     3.888 MB/s [   949.2 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     3.298 MB/s [   805.2 IOPS]

  Test : 1024 MiB [L: 0.0% (0.5/18626.9 GiB)] (x5)  [Interval=5 sec]
  Date : 2017/09/21 11:23:58
    OS : Windows Server 2012 R2  [6.3 Build 9600] (x64)
  

 

How did you get sequential read speeds of 879 MB/s from a hard disk drive?!?!?

[leadeater]

4 minutes ago, TheCherryKing said:

How did you get sequential read speeds of 879 MB/s from a hard disk drive?!?!?

RAID card cache 

[GDRRiley]

2 hours ago, leadeater said:

RAID card cache 

just enough space to super charger it. 

[bfairs]

On 22/09/2017 at 5:05 PM, TheCherryKing said:

How did you get sequential read speeds of 879 MB/s from a hard disk drive?!?!?

 

On 22/09/2017 at 5:08 PM, leadeater said:

RAID card cache 

 

On 22/09/2017 at 7:08 PM, GDRRiley said:

just enough space to super charger it. 

Yeah, we have a second RAID card that was originally going to be a cold spare, but decided to just use it for some backup drives. Cards have 1GB cache, but only the main RAID6 storage array card has a battery backup (well, capacitors).  Whole server is on a pretty robust UPS too.  Didn't realize that cache would be so effective! What should I have expected? 250MB/s?

[leadeater]

2 hours ago, bfairs said:

Yeah, we have a second RAID card that was originally going to be a cold spare, but decided to just use it for some backup drives. Cards have 1GB cache, but only the main RAID6 storage array card has a battery backup (well, capacitors).  Whole server is on a pretty robust UPS too.  Didn't realize that cache would be so effective! What should I have expected? 250MB/s?

The two disks in a mirror without cache wouldn't have gone over 200MB/s.

[scottyseng]

11 hours ago, bfairs said:

Yeah, we have a second RAID card that was originally going to be a cold spare, but decided to just use it for some backup drives. Cards have 1GB cache, but only the main RAID6 storage array card has a battery backup (well, capacitors).  Whole server is on a pretty robust UPS too.  Didn't realize that cache would be so effective! What should I have expected? 250MB/s?

Yeah, I would've expected 200MB/s or lower for the mirror. Around 350MB/s for the RAID6 of four drives.

 

If you want to see how it performs without the cache, just set the test file size in Crystal Disk Mark to something like 2GB or 4GB instead of the default 1GB.

 

Happy it all works though.

 

Yeah, it took me eight drives to get over 1GB/s: