Raid Card

[LordMastodon]

So, I'm working on a file server for a school, and while we're mostly stuck with having to buy from Dell, I'm trying to convince them to custom build by showing them the price tag difference (which the Dell system so far costs something like 233% what the system I've put together does). We're trying to set up 4 960 GB SSDs in a RAID 10 array, but we don't want to use software RAID, first of all because it pretty much nullifies the performance boost you get with RAID 10, and because it's risky, so we're going with hardware RAID. I'd like to know if there are any cards that support RAID 10 that you can recommend, because I haven't really built a system with hardware RAID ever and I don't want to screw it up. Thanks! 

[ShadowCaptain]

9 minutes ago, LordMastodon said:

SNIP

Not necessarily true, with modern CPUs the benefit of dedicated raid cards for performance is pretty negligible especially if you chose to use a low end raid card

 

Software raid could be considered safer, since if your raid card fails, you need to find the exact same RAID card for it to work again, but with software raid you can transplant it much easier

 

 

[LordMastodon]

Just now, ShadowCaptain said:

Not necessarily true, with modern CPUs the benefit of dedicated raid cards for performance is pretty negligible especially if you chose to use a low end raid card

 

Software raid could be considered safer, since if your raid card fails, you need to find the exact same RAID card for it to work again, but with software raid you can transplant it much easier

 

 

2

 

Hmm. Well at any rate the higher-ups have decided to go hardware RAID, I can't really argue the small details with them. Although we really don't have to go with low-end RAID card.

[ShadowCaptain]

2 minutes ago, LordMastodon said:

Hmm. Well at any rate the higher-ups have decided to go hardware RAID, I can't really argue the small details with them. Although we really don't have to go with low-end RAID card.

Why dont they just buy a dell server?

 

they probably want to go dell for software/engineer support, so buying a random RAID card might not be the best idea either, though there are good ones on the market it might clash with dell support if the RAID hardware is ever an issue since dell warranty might not cover that card being in there if an engineer is called out

[LordMastodon]

Just now, ShadowCaptain said:

Why dont they just buy a dell server?

 

they probably want to go dell for software/engineer support, so buying a random RAID card might not be the best idea either, though there are good ones on the market it might clash with dell support if the RAID hardware is ever an issue since dell warranty might not cover that card being in there if an engineer is called out

 

We are planning to go with a Dell server, but I've noticed that with Dell we're paying more than double for a warranty, case, motherboard, OS and builder, which is ridiculous. Not to mention the fact that an 800 GB SSD costs $600 from Dell and a 960 GB SSD costs $220 from PCPartPicker. The Dell one would come with a RAID card built in, which preconfigured RAID 10.

 

I was working on a custom parts list because I wanted to see the price difference, but I think that now this actually viable option, mainly because of the absolutely ludicrous price difference. So I because the higher-ups decided that they wanted hardware RAID no matter what direction we went, I want a RAID card suggestion just to make sure I get the actual final price for the custom built so that I can present it to the higher-ups and see what they think.

[ShadowCaptain]

6 minutes ago, LordMastodon said:

We are planning to go with a Dell server, but I've noticed that with Dell we're paying more than double for a warranty, case, motherboard, OS and builder, which is ridiculous. Not to mention the fact that an 800 GB SSD costs $600 from Dell and a 960 GB SSD costs $220 from PCPartPicker. The Dell one would come with a RAID card built in, which preconfigured RAID 10.

 

I was working on a custom parts list because I wanted to see the price difference, but I think that now this actually viable option, mainly because of the absolutely ludicrous price difference. So I because the higher-ups decided that they wanted hardware RAID no matter what direction we went, I want a RAID card suggestion just to make sure I get the actual final price for the custom built so that I can present it to the higher-ups and see what they think.

a good one like from LSI could be around $200-$500

[leadeater]

1 minute ago, LordMastodon said:

We are planning to go with a Dell server, but I've noticed that with Dell we're paying more than double for a warranty, case, motherboard, OS and builder, which is ridiculous. Not to mention the fact that an 800 GB SSD costs $600 from Dell and a 960 GB SSD costs $220 from PCPartPicker. The Dell one would come with a RAID card built in, which preconfigured RAID 10.

 

I was working on a custom parts list because I wanted to see the price difference, but I think that now this actually viable option, mainly because of the absolutely ludicrous price difference. So I because the higher-ups decided that they wanted hardware RAID no matter what direction we went, I want a RAID card suggestion just to make sure I get the actual final price for the custom built so that I can present it to the higher-ups and see what they think.

There is a big difference between SSD product lines, make sure you are comparing the same. If you look at the Samsung enterprise SSDs which are OEM for Dell/HP etc along with Intel enterprise SSDs you'll see a much smaller price difference. Also bear in mind even Samsung 850 Pro SSDs are not meant to be put in large RAID arrays to run multiple VMs off, also they are missing a critical server feature which is power loss protection.

 

The best cost break I can recommend is buy the base server from Dell and then supply your own SSDs, you can buy Dell 2.5" disk sleds on ebay, I do this for my IBM x3500 M4 at home.

 

If you want to go cheap and reliable go with Hyper-V as the virtual host and use Windows Storage Spaces to create a redundant storage pool.

 

P.S. Don't buy a Windows license now if you can help it, wait till September after Microsoft Ignite as that is when Server 2016 releases and there is some key/awesome improvements to Storage Spaces and ReFS.

[leadeater]

As for a good RAID card I can recommend an LSI 9361-8i with Fast Path feature upgrade

[LordMastodon]

14 minutes ago, leadeater said:

There is a big difference between SSD product lines, make sure you are comparing the same. If you look at the Samsung enterprise SSDs which are OEM for Dell/HP etc along with Intel enterprise SSDs you'll see a much smaller price difference. Also bear in mind even Samsung 850 Pro SSDs are not meant to be put in large RAID arrays to run multiple VMs off, also they are missing a critical server feature which is power loss protection.

 

The best cost break I can recommend is buy the base server from Dell and then supply your own SSDs, you can buy Dell 2.5" disk sleds on ebay, I do this for my IBM x3500 M4 at home.

 

If you want to go cheap and reliable go with Hyper-V as the virtual host and use Windows Storage Spaces to create a redundant storage pool.

 

P.S. Don't buy a Windows license now if you can help it, wait till September after Microsoft Ignite as that is when Server 2016 releases and there is some key/awesome improvements to Storage Spaces and ReFS.

 

 

Buying the base server from Dell is still ridiculously expensive, and I want to go as cost-effectively as possible. We're really not intending to put the SSDs in a large RAID array or even run a single VM. We're going to have 4 drives in RAID 10, and that's really about it. What do you mean when you say power loss protection? We've got two redundant power supplies and a UPS, so I don't think it should be a huge problem...

 

11 minutes ago, leadeater said:

As for a good RAID card I can recommend an LSI 9361-8i with Fast Path feature upgrade

Alright, thanks, I'll look into that one. Does it support RAID 10?

[leadeater]

24 minutes ago, LordMastodon said:

Buying the base server from Dell is still ridiculously expensive, and I want to go as cost-effectively as possible. We're really not intending to put the SSDs in a large RAID array or even run a single VM. We're going to have 4 drives in RAID 10, and that's really about it. What do you mean when you say power loss protection? We've got two redundant power supplies and a UPS, so I don't think it should be a huge problem...

 

Alright, thanks, I'll look into that one. Does it support RAID 10?

Yep supports every RAID configuration you can dream of.

 

Enterprise SSDs have inbuilt power protection which among protecting you from faults with the UPS, improper shutdowns and careless mistakes when pulling power cords reports this fact to RAID controllers and SAS/SATA controllers which effects how they get treated. SSDs without power loss protection get their cache disabled when put in to RAID arrays which for something like a 850 Pro turns it from a 550 MB/s monster into a 30MB/s-100MB/s useless pile of crap. You can sometimes, not always, force the cache back on but this introduces a risk that if there is an issue with the RAID controller or configuration you could corrupt the array.

 

The other benefit enterprise SSDs have is that they have high internal flash over provisioning and better garbage collection, remember SSDs on a hardware RAID card lose TRIM support so their performance drops significantly and the wear is much higher. Consumer SSDs often heavily rely on TRIM to perform well and not wear out in 3-5 years let alone sustain server load.

 

For server use holding important data only server rated SSDs should be used, also many SSDs explicitly state that their warranty is void if put in a server.

 

I have tried running an array of 6 Samsung 850 Pros on both an IBM M5110 RAID card and an LSI 9361-8i with Fast Path and multiple times the array got screwed up and I had to take all the SSDs out, put them in my desktop and run a full clean and erase on them to bring them back to life. Consumer SSDs and server RAID cards don't play nice, plenty do like the Intel SSD since they use LSI controllers but don't count on being able to throw any SSD behind a hardware RAID card.

 

Also by the sounds of it you don't actually need SSDs, would 10K SAS not be enough? Is this a high performance SQL server that would actually be able to make use of this disk performance?

 

Edit: For SSDs anything more than a 2 disk RAID 1 mirror I would put in the category of a large array as all the extra considerations need to be made beyond that point whether it's 3 SSDs or 24 SSDs.

Edited August 3, 2016 by leadeater

[LordMastodon]

Just now, leadeater said:

Yep supports every RAID configuration you can dream of.

 

Enterprise SSDs have inbuilt power protection which among protecting you from faults with the UPS, improper shutdowns and careless mistakes when pulling power cords reports this fact to RAID controllers and SAS/SATA controllers which effects how they get treated. SSDs without power loss protection get their cache disabled when put in to RAID arrays which for something like a 850 Pro turns it from a 550 MB/s monster into a 30MB/s-100MB/s useless pile of crap. You can sometimes, not always, force the cache back on but this introduces a risk that if there is an issue with the RAID controller or configuration you could corrupt the array.

 

The other benefit enterprise SSDs have is that they have high internal flash over provisioning and better garbage collection, remember SSDs on a hardware RAID card lose TRIM support so their performance drops significantly and the wear is much higher. Consumer SSDs often heavily rely on TRIM to perform well and not wear out in 3-5 years let alone sustain server load.

 

For server use holding important data only server rated SSDs should be used, also many SSDs explicitly state that their warranty is void if put in a server.

 

I have tried running an array of 6 Samsung 850 Pros on both an IBM M5110 RAID card and an LSI 9361-8i with Fast Path and multiple times the array got screwed up and I had to take all the SSDs out, put them in my desktop and run a full clean and erase on them to bring them back to life. Consumer SSDs and server RAID cards don't play nice, plenty do like the Intel SSD since they use LSI controllers but don't count on being able to throw any SSD behind a hardware RAID card.

 

Also by the sounds of it you don't actually need SSDs, would 10K SAS not be enough? Is this a high performance SQL server that would actually be able to make use of this disk performance?

7

This is a file server, but again, the higher ups have mandated the use of SSDs (I'm just as confused as you are). You know a whole lot more about this subject than I do; can you recommend any server grade SSDs with power loss protection? I seem to remember Linus used a whole bunch of Kingston KC310s for Whonnock, but the RAID cards there failed (from what I remember), so I don't know if those are the best option.

[leadeater]

1 hour ago, LordMastodon said:

This is a file server, but again, the higher ups have mandated the use of SSDs (I'm just as confused as you are). You know a whole lot more about this subject than I do; can you recommend any server grade SSDs with power loss protection? I seem to remember Linus used a whole bunch of Kingston KC310s for Whonnock, but the RAID cards there failed (from what I remember), so I don't know if those are the best option.

The RAID card dying was just bad luck, anything can fail and what he was using was very well respected hardware. Linus also later suspected that the motherboard had a part to play in the issue also.

 

The Kingston KC310s Linus uses are good and have power loss protection, other options I'll list below (just go with what ever is cheaper your not going to be pushing these hard):

Samsung PM863 (Most $$ I think)

Intel DC S3710

Intel DC S3700

Intel DC S3610

Sandisk Cloudspeed Eco Gen II (Least $$ I think)

 

I would probably recommend going with the Kingston KC310 as it uses a Sandforce controller which should pair nicely with an LSI RAID card, Sandforce is owned by LSI. LSI is the largest player in the hardware RAID market and almost all OEM Dell/Lenovo/Intel are actually LSI, HP now being the exception as they switched to PMC Adaptec recently.

[LordMastodon]

Just now, leadeater said:

The RAID card dying was just bad luck, anything can fail and what he was using was very well respected hardware. Linus also later suspected that the motherboard had a part to play in the issue also.

 

The Kingston KC310s Linus uses are good and have power loss protection, other options I'll list below (just go with what ever is cheaper your not going to be pushing these hard):

Samsung PM863 (Most $$ I think)

Intel DC S3710

Intel DC S3700

Intel DC S3610

Sandisk Cloudspeed Eco Gen II (Least $$ I think)

 

I would probably recommend going with the Kingston KC310 as it uses a Sandforce controller which should pair nicely with an LSI RAID card, Sandforce is owned by LSI. LSI is the largest player in the hardware RAID market and almost all OEM Dell/Lenovo/Intel are actually LSI, HP now being the exception as they switched to PMC Adaptec recently.

 

Alright, I'll probably go with the Sandisk then. I couldn't find the Kingston KC310 on Kingston's website though, maybe they were discontinued? Also, isn't that RAID controller SAS? I'm pretty sure that the Sandisk one is SATA, which from what I understand only really results in a loss of speed, but that might not even matter with SSDs because Google said that it just causes the drive to spin at higher RPM. I don't think it really matters at all anyway, because I watched Newegg's video on that RAID card you recommended and it said that the thing comes with adapters, so it's not a huge deal.

[scottyseng]

2 minutes ago, LordMastodon said:

-snip-

One SAS port breaks out to four SATA ports. You have to buy a SAS to SATA breakout cable. Be careful as the 12Gb/s SAS port is different from the 6Gb/s and 3Gb/s SAS port. There's not really a performance loss converting from SAS to SATA.

[LordMastodon]

Just now, scottyseng said:

One SAS port breaks out to four SATA ports. You have to buy a SAS to SATA breakout cable. Be careful as the 12Gb/s SAS port is different from the 6Gb/s and 3Gb/s SAS port.

 

From what I saw in that video, the RAID card comes with a SAS to SATA 12 GB/s breakout cable, but I could be wrong.

[leadeater]

2 minutes ago, LordMastodon said:

Alright, I'll probably go with the Sandisk then. I couldn't find the Kingston KC310 on Kingston's website though, maybe they were discontinued? Also, isn't that RAID controller SAS? I'm pretty sure that the Sandisk one is SATA, which from what I understand only really results in a loss of speed, but that might not even matter with SSDs because Google said that it just causes the drive to spin at higher RPM. I don't think it really matters at all anyway, because I watched Newegg's video on that RAID card you recommended and it said that the thing comes with adapters, so it's not a huge deal.

Yea SAS is cross compatible with SATA, but you can't go the other way. For 12Gb SAS you need SFF-8643 to 4x SATA forward cables to connect SATA disks to them directly or you can use a multi disk drive bay that has either a SAS connector or multiple SATA connectors on it for ease of access and hot-swap.

[scottyseng]

Just now, LordMastodon said:

From what I saw in that video, the RAID card comes with a SAS to SATA 12 GB/s breakout cable, but I could be wrong.

Yeah, I would double check to make sure. They sell just the card itself and in a combo kit. Also, make sure to get the Cachevault battery for the RAID card.

[leadeater]

3 minutes ago, LordMastodon said:

From what I saw in that video, the RAID card comes with a SAS to SATA 12 GB/s breakout cable, but I could be wrong.

From memory when I got mine yes it came with the breakout cables.

 

2 minutes ago, scottyseng said:

Yeah, I would double check to make sure. They sell just the card itself and in a combo kit. Also, make sure to get the Cachevault battery for the RAID card.

Yea I got the pack/combo kit

Edited August 3, 2016 by leadeater

[LordMastodon]

Just now, leadeater said:

Yea SAS is cross compatible with SATA, but you can't go the other way. For 12Gb SAS you need SFF-8643 to 4x SATA forward cables to connect SATA disks to them directly or you can use a multi disk drive bay that has either a SAS connector or multiple SATA connectors on it for ease of access and hot-swap.

3

Yeah, Paul (from Paul's Hardware) apparently made a video on that exact card in 2014 and in it he said that it does come with exactly what you said, as in the SFF-8643 12 GB/s port to 4x SATA. That video can be found here. 

 

It would seem as though the Kingston KC310 is the cheapest, so I'll probably just go with that one.

[LordMastodon]

6 minutes ago, scottyseng said:

Yeah, I would double check to make sure. They sell just the card itself and in a combo kit. Also, make sure to get the Cachevault battery for the RAID card.

 

What's the Cachevault battery?

[scottyseng]

3 hours ago, LordMastodon said:

We're trying to set up 4 960 GB SSDs in a RAID 10 array

Out of curiosity, why'd they pick RAID10 over RAID6? I'd imagine losing half of the space of a SSD array is pretty brutal on the wallet.

 

I switched from RAID10 to RAID6 on my home NAS (eight 4TB WD Re in RAID6 and six 4TB WD Reds in RAID6) and got a pretty notable gain in sequential file speeds. I did lose some in the random IO, but I don't run any databases off of my server so sequential speed matters more to me.

[leadeater]

5 minutes ago, LordMastodon said:

Yeah, Paul (from Paul's Hardware) apparently made a video on that exact card in 2014 and in it he said that it does come with exactly what you said, as in the SFF-8643 12 GB/s port to 4x SATA. That video can be found here. 

 

It would seem as though the Kingston KC310 is the cheapest, so I'll probably just go with that one.

Btw give the card good airflow, preferably directly in the air stream of a fan as these cards run really hot and are designed to be in server cases with very high full case airflow, something not typically found in desktop configurations. Shouldn't be a problem if you are going to be putting this in a server case though, just use good fans. 

[scottyseng]

4 minutes ago, LordMastodon said:

What's the Cachevault battery?

http://www.newegg.com/Product/Product.aspx?Item=N82E16816118232&cm_re=cachevault_9361-_-16-118-232-_-Product

 

This thing. I think it might come with the card in the combo kit, but I can't remember correctly.

[leadeater]

6 minutes ago, LordMastodon said:

What's the Cachevault battery?

Yet another power loss prevention component that protects the onboard cache of the RAID card, servers have tons of batteries hidden in all sorts of places 

[LordMastodon]

Just now, scottyseng said:

Out of curiosity, why'd they pick RAID10 over RAID6? I'd imagine losing half of the space of a SSD array is pretty brutal on the wallet.

 

I switched from RAID10 to RAID6 on my home NAS (eight 4TB WD Re in RAID6 and six 4TB WD Reds in RAID6) and got a pretty notable gain in sequential file speeds. I did lose some in the random IO, but I don't run any databases off of my server so sequential speed matters more to me.

 

 

 

If I understand RAID correctly, RAID 5/6 has only one redundant drive (drive that could fail and nobody would care), so if two drives fail before you're able to fix the array, you're pretty much screwed. For RAID 10, two drives could fail (as long as they're on different channels) and you'd still be fine. And if you look at it statistically, after one drive has failed, there's a 1/3 chance the next drive to fail will be in the same channel as the one that just failed, so yeah. Performance isn't a huge problem as long as they're SSDs, and redundancy is super duper important because most of this stuff is mission critical.