Should I use hardware raid if the server already comes with the controller or is software raid is better?

[EugeneLevinson]

Hello! First time posting on the forum!

My dad and I are looking at building a NAS. We originally were looking just o build a PC with some NAS drives, and run FreeNAS/TrueNAS on it. Our budget is around £600 including the drives. Aiming to get around 4TB of useful storage at the start, and expand later if needed. I was able to come up with a build for this fairly easily, however, this is when used server hardware caught my attention. Specifically, I am looking at sites where you can configure servers from refurbished hardware. There are many great choices that would work for us. It also does give me the ability to use that server for some of my programming projects.

 

The only issue we have is that all these servers come with a raid controller. From what I was able to gather online, this means I can't run FreeNAS on top of it, and you can't disable the raid controller since the drives are connected through it. I've also read that Non-raid, also won't work since it doesn't fully expose the drives like if they were connected directly through SATA. And that brings me here, there are a couple of ways we can approach this problem, but I don't have enough understanding to make a decision, so I have a couple of questions:

 

Could I just run RAID-0 and have each drive as a separate volume? I've seen this a lot online but haven't seen a breakdown on how well this works. Will FreeNAS/TrueNAS work well if I run RAID-0 on the hardware controller?

Some controllers specify that they are "software raid enabled" like the PERC S110. Does that mean I can run FreeNAS on top of it or does it mean something else?

 

Could I just run hardware raid? I am trying to figure out the difference between FreeNAS and hardware raid in our specific use case. From what I understand as long as the processing power is not very low, software raid should be a little better than hardware raid, but we are not very concerned with minor performance differences. Hardware raid should be easier to set up than software raid. It's my understanding that FreeNAS will handle the whole "network storage" part but from what I can see it's also not that hard to do it yourself on Linux. So I don't see any major disadvantages of using hardware raid instead, but just woudnering if I am missing something important.

 

Do you guys have any suggestions/advice?

 

Thank you in advance!

[LionSpeck]

1 hour ago, EugeneLevinson said:

Hello! First time posting on the forum!

My dad and I are looking at building a NAS. We originally were looking just o build a PC with some NAS drives, and run FreeNAS/TrueNAS on it. Our budget is around £600 including the drives. Aiming to get around 4TB of useful storage at the start, and expand later if needed. I was able to come up with a build for this fairly easily, however, this is when used server hardware caught my attention. Specifically, I am looking at sites where you can configure servers from refurbished hardware. There are many great choices that would work for us. It also does give me the ability to use that server for some of my programming projects.

 

The only issue we have is that all these servers come with a raid controller. From what I was able to gather online, this means I can't run FreeNAS on top of it, and you can't disable the raid controller since the drives are connected through it. I've also read that Non-raid, also won't work since it doesn't fully expose the drives like if they were connected directly through SATA. And that brings me here, there are a couple of ways we can approach this problem, but I don't have enough understanding to make a decision, so I have a couple of questions

Hey there, welcome to the forums!
Great project you have in mind; one thing on the refurbished enterprise stuff: yes, they can be great deals, but it's mostly old hardware, so be aware that they may be inefficient, have atrociously slow POST times, and may require specific hardware for compatibility (i.e. HP / Dell custom part numbers). Source: happy owner of a ProLiant

Indeed, most servers come with a RAID controller; technically you can run anything you want on it, but, as you pointed out yourself, it is a terrible idea to run TrueNAS on it. The simple reason why everywhere they tell you to forget about hardware RAID is that you're not likely to find the same replacement for the RAID card when it breaks, therefore all your data would become inaccessible. Why do they seem to break so frequently? Because people use software RAID (mostly ZFS) on top of these cards, which poorly handle the monumental amount of IOPS that ZFS issues. In short: although technically you could use hardware RAID, you probably shouldn't; the worst possible thing would be to use it with ZFS as well. By far the best solution is to use ZFS on JBOD (Just a Bunch Of Drives - means bare drives, without any hardware RAID) / bare drives (an HBA).

Some RAID cards allow you to choose a JBOD mode, but this shouldn't be trusted too well (research needs to be done on the specific card model on forums), as depending on the card, some do not give the OS bare direct access to the blocks and the SMART data, both of which are required for a good ZFS implementation. Some others allow you to flash them to HBA mode / IT mode (Initiator Target mode). This is what you basically want, as it gives that bare direct access aforementioned, aka if the card dies, you replace it with any other HBA and everything is accessible and readable by ZFS. The alternative is to buy directly an LSI HBA, which are cards that don't need to be flashed. They tend to be more expensive, because of this very reason.

To answer your questions:
 

1 hour ago, EugeneLevinson said:

Could I just run RAID-0 and have each drive as a separate volume?

NO, this is a recipe for disaster. That RAID0 is not direct access to the disk, plus the data is coded (though not encrypted) in proprietary ways by the card on the disk. This means when the card dies, unless you replace it with an exact same model, that data is gone. And yes, the card will fail, since ZFS will fry its poor processor with humongous amounts of IOPS. (Note that this isn't an issue if the card is in HBA mode, since there's no calculations / strange trickeries that it has to do, just issue the SAS commands to the disks.)

1 hour ago, EugeneLevinson said:

Will FreeNAS/TrueNAS work well if I run RAID-0 on the hardware controller?

It will work, but you shouldn't do it.

 

1 hour ago, EugeneLevinson said:

Some controllers specify that they are "software raid enabled" like the PERC S110. Does that mean I can run FreeNAS on top of it or does it mean something else?

I've done a little research, and it looks like this particular card is an Intel chip with software RAID capabilities. It could be a very great scenario, since it means it's basically an HBA (with packaged RAID software that you obviously won't use). If this were the case (and further research on TrueNAS forums / Reddit needs to be done to confirm this) then yes, you could use ZFS directly on this with no problems.

 

1 hour ago, EugeneLevinson said:

Could I just run hardware raid?

Yes, you could. Everyone absolutely despises this, but I think it comes more out of stigma and bad usage more than anything else. I would not recommend this solution either, but I'll explain why it exists and what are its characteristics.

In the enterprise industry, hardware RAID is pretty much the norm (except for some more recent cloud / storage enterprises). It is so because

- it's far easier to maintain,

- it requires no software layers (allows for hypervisors like ESXi, which don't have any software RAID, but really want redundancy),

- it offsets parity processing on an external processor instead of the CPU,

- allows for additional write cache (instead of eating up RAM), and

- this cache can be battery backed-up for days in case of power loss, which is something you just plain can't do with any software RAID (unless you run a UPS, which is way more expensive and will keep your system up only for minutes).

The main reason why you don't want a hardware RAID though, is that you're not an enterprise; you don't have a contract with Dell / HP for a technician to come and replace the RAID card / whole server if it fails. So, if it does, you either find an exact replacement for that card, and have your data back; or you say goodbye to said data.

Depending on how much you'd pay to keep your data safe, I think it's worth paying <50$ for an HBA, and not have to worry about any of that proprietary bullshit.

Also, hardware RAID is mostly self-managed, so you don't have access to any setting, SMART data or any of that. If the card thinks it's time to replace a drive, it will scream until you do as it wishes, regardless of whether that drive has actually failed.

 

1 hour ago, EugeneLevinson said:

It's my understanding that FreeNAS will handle the whole "network storage" part but from what I can see it's also not that hard to do it yourself on Linux.

This is correct; either way, regardless of using ZFS or hardware RAID, it's a very easy thing to do.

 

***

My personal advice: run either Linux with ZFS or TrueNAS (FreeBSD with ZFS) and forget about hardware RAID. Read the TrueNAS hardware recommendation guide, which tells actually a lot of useful information regarding all kinds of hardware for any NAS. Everything that's said there applies to both FreeBSD and Linux, with ZFS. There is really, really important stuff there.

Then do some specific research on that PERC S110 (I will do some as well in the meantime) to make sure that it gives the system bare direct access to the drives, or that it can be flashed to a firmware that allows that. This is crucial.

If this is not the case, read this list of top picks for HBAs and this complete list of HBAs and controllers to find cards on the market that either are born HBAs, or that are hardware RAID cards with LSI controllers that can be flashed into becoming HBAs. I have one of such cards myself, flashed into HBA mode.

If there's anything that isn't clear, I'll do my best to clarify. Cheers!

[LionSpeck]

Ok, so all I could find on this PERC S110 is these two pages:
https://www.truenas.com/community/threads/perc-s110-raid-controller.57030/

Here there is only supposed compatibility, but nothing to hold onto, really.

 

https://www.dell.com/community/PowerEdge-HDD-SCSI-RAID/R620-with-4-drive-backplane-no-PERC-SAS-SATA/td-p/5182272

Here it seems like the card allows for direct pass-through of the disks, but it's not explicitly stated.

 

Other than this, even on official Dell sources, I couldn't find unequivocal information regarding disk pass-through. If I were you, I would consider in my budget an HBA / flashable RAID card regardless; then, when time comes and you have your server, you can try to set up disk pass-through. If it's not a thing, then buy the HBA and hook up the backplane to that.

[wseaton]

PERC S110 is utter garbage and one of the most unreliable storage controllers I've ever worked with. Yes, it's basically a diet Intel RST on a daughter board with a PCI interface glued on. Even in JBOD mode I wouldn't trust it. I've had 50% failure rates of that controller.

 

The only PERC controllers worth running are 700s and higher. 330 is meh/ok. The problem is they only have value as full hardware mode controllers, and this is pretty much a non-sequitur when it comes the Linus NAS.

 

Next, I don't get this fascination with old servers. There is utterly no benefit to doing this with a simple software NAS box. For 1Gb use I would just get a Synology and be done with it. For tinkering with FreeNAS / TruNAS just a get a refurbished i7 / i9 tower and stuff it full of drives and use the onboard SATA controllers in AHCI mode. Done.

[LionSpeck]

48 minutes ago, wseaton said:

PERC S110 is utter garbage and one of the most unreliable storage controllers I've ever worked with. Yes, it's basically a diet Intel RST on a daughter board with a PCI interface glued on. Even in JBOD mode I wouldn't trust it. I've had 50% failure rates of that controller.

Yikes, that was kind of my fear. Technically usable, but a ticking bomb that will have to be replaced shortly. Oh, Dell
 

48 minutes ago, wseaton said:

Next, I don't get this fascination with old servers. There is utterly no benefit to doing this with a simple software NAS box. For 1Gb use I would just get a Synology and be done with it. For tinkering with FreeNAS / TruNAS just a get a refurbished i7 / i9 tower and stuff it full of drives and use the onboard SATA controllers in AHCI mode. Done.

Well, yes and no, imo. I agree with the "no fascination for old servers"; I needed more than just a NAS box and got a great deal with my ProLiant, but I'm paying over time in electricity for the low upfront cost.

I personally loathe NAS boxes: they're outrageously expensive for what they offer, are proprietary and live in their own dreamland where you depend on the mercy of Synology updates, security, compatibility, features and support. Is it a good solution for "the parents who just need their family photos backed up"? Sure. More than that, and literally anything would be better. Again, imo.

I also agree with the refurbished tower, though I think an i9 is way too overkill, Jesus H Christ. An i5 should be plenty, even exaggerated if it's just a NAS with an SMB / NFS share. Only things you're giving up with consumer hardware is ECC support (which isn't required but recommended); in exchange, you're getting silence of operation, energy efficiency, better form factor, other comforts here and there.

[Needfuldoer]

50 minutes ago, wseaton said:

The only PERC controllers worth running are 700s and higher. 330 is meh/ok. The problem is they only have value as full hardware mode controllers, and this is pretty much a non-sequitur when it comes the Linus NAS.

They’re not bad for TrueNAS/ZFS if you can cross-flash them to IT mode. 
 

https://fohdeesha.com/docs/perc.html