Transplanting drives into NAS

[invaderSnarf]

tl;dr

Would I be able to just drop the old drives from my main rig into the NAS and still have access to all of the data on them, or would I need to move that data to my other harddrives and reformate the old drives before installing them in the NAS?

 

I want to consolidate my storage down to a NAS instead of having it float between 4 seperate computers in my house (2 desktops and 2 laptops). I have found a 12bay (was thinking of doing RAID2 so have 10 total drives of storage with 2 capable of failing without worry) NAS that I believe will work for my needs. My primary rig is in a Lian Li o11 Dynamic XL and is where I have the most storage, just shy of 32tb. A majority of the space is made from 2 12tb drives and it would be what I would want to fill out the NAS with eventually. Primary use of the NAS would be to store my raw recording footage as well as work as an archive of my finished videos that have been uploaded already so I can free up space on my primary machine. I am not looking to edit footage directly off the NAS as I only have a 1gb switch, so it would just be for storage and archival purposes. 

[heimdali]

You found a NAS device that actually does RAID2?

 

If you can move the disks over to it depends on what the NAS device supports.  Don't you have any redundancy, and what are you doing for backups?

 

[invaderSnarf]

3 minutes ago, heimdali said:

You found a NAS device that actually does RAID2?

Sorry, I think I used the wrong nomenclature. I am planning on having 2 parity drives out of the total 12, not actual RAID2

 

Right now I am doing the dumb thing and have no redundancy or backups, but I am also not very concerned with a majority of the content on the drives. The items I am concerned about I have copied across multiple computers as my "backup". I was planing on doing full backups of everything once I build the NAS. 

[heimdali]

Then the NAS device would probably create some kind of RAID array with the disks and thereby probably overwrite the data that's on them.

 

[invaderSnarf]

2 minutes ago, heimdali said:

Then the NAS device would probably create some kind of RAID array with the disks and thereby probably overwrite the data that's on them.

 

Got it. I'll be sure to backup the drives before I install them. Looking at the NAS, it looks like it has a H710P raid controller. Would you suggest running RAID 5+0 or RAID 6+0? I'm planning on filling the bays with 12tb drives and aiming for over 100tb of usable storage. 

[heimdali]

44 minutes ago, invaderSnarf said:

Got it. I'll be sure to backup the drives before I install them. Looking at the NAS, it looks like it has a H710P raid controller. Would you suggest running RAID 5+0 or RAID 6+0? I'm planning on filling the bays with 12tb drives and aiming for over 100tb of usable storage. 

Wow that's like a $5000 budget, and that for only a small part of the data.

 

With 12x12TB disks in RAID50 or RAID60, that'll give you either 60 or 48TB of usable storage space, not considering that with 12TB disks, each gives you only what, like 10TB or so? because manufacturers are cheating with the units, and I leave you to do the math.

 

Without the math, I'd consider using 4x16TB in a RAID1 instead because it gives you about the same amount of storage than the RAID50/60 for way less money, and you do the math   The more disks you have, the more will (can) fail, the more power you'll have to pay for, the more cooling you need and the louder it may get, and the more unwieldy the whole thing becomes.

 

If you ask 4 people what to use, you'll probably get 32 different answers and then some.  I'm not fond at all of NAS devices because I think they're way overpriced.  You can get like a Dell R720 with 8 or 12 LFF bays for way less than what the NAS costs, and you can do anything you like with it instead of being stuck with a NAS device.  But then, do you want hardware RAID or would you rather use software and ZFS or btrfs or maybe xfs or ext4?  Do you want snapshots, like for incremental backups?  It all has its advantages and disadvantages.

 

And have you considered how compression and deduplication might figure in?

 

Maybe delete the data you don't care about before starting to plan

 

[LIGISTX]

1 hour ago, invaderSnarf said:

Got it. I'll be sure to backup the drives before I install them. Looking at the NAS, it looks like it has a H710P raid controller. Would you suggest running RAID 5+0 or RAID 6+0? I'm planning on filling the bays with 12tb drives and aiming for over 100tb of usable storage. 

Are you building or buying your nas? Do you have an old PC laying around you could repurpose as one? 
 

An unraid or truenas machine requires very little cpu power, and harddrives can be easily added with a HBA for about 50 bucks used on eBay, so amount of SATA ports on the mobo is a non-issue. 
 

But, regardless, I wouldn’t use 50 or 60, that is pointless for a home NAS unless you have a specific reason you need the addition speed and IOPS, but you will almost certainly be limited by your LAN first anyways. Gigabit networking is only 125 megs a second. A single harddrive can easily saturate gigabit, none the less multiple in an array. I would go RAID 6, or if you go truenas and use ZFS, that would be called RAID Z2. Single drive parity is just not statistically enough anymore. The chance of data loss due to a second drive failing as your rebuilding your first drive is actually a relatively not low number… thus why you want a second parity drive. I have had a second drive fall out of my array while doing a rebuild on a new drive, if I only had single drive residency I would have been SOL. Also, the larger the drive, the longer the rebuild takes, exposing you for a longer period of time. 

[invaderSnarf]

1 hour ago, heimdali said:

You can get like a Dell R720 with 8 or 12 LFF bays for way less than what the NAS costs, and you can do anything you like with it instead of being stuck with a NAS device. 

I am looking at a renewed Dell R720xd for under $500 that is prebuilt just without the drives. For the harddrives I was going to put the two drives have now and possibly slot in one or two more and then purchase the additional drives to fill it out as I go. 

13 minutes ago, LIGISTX said:

A single harddrive can easily saturate gigabit, none the less multiple in an array. I would go RAID 6, or if you go truenas and use ZFS, that would be called RAID Z2. Single drive parity is just not statistically enough anymore. The chance of data loss due to a second drive failing as your rebuilding your first drive is actually a relatively not low number… thus why you want a second parity drive.

This is also what I was planning on doing once I set things up. From what I've absorbed through LTT, Truenas made the most sense for what I was looking to acheive. Apologies if I was getting the nomenclature wrong, this is the first time I've actually put some time into building a NAS for myself. 

 

Would mixing the two 12tb drives that I have now with 16tb drives later cause issues? Would I be better off saving for longer and getting all of the drives to populate the server at once rather than build it and slowly implement more and more drives as I need?

[heimdali]

25 minutes ago, invaderSnarf said:

I am looking at a renewed Dell R720xd for under $500 that is prebuilt just without the drives. For the harddrives I was going to put the two drives have now and possibly slot in one or two more and then purchase the additional drives to fill it out as I go.

That isn't a NAS device.  It says "Currently unavailable.".

 

When you want to add more drives later, you may want to resize the storage volume.

25 minutes ago, invaderSnarf said:

 This is also what I was planning on doing once I set things up. From what I've absorbed through LTT, Truenas made the most sense for what I was looking to acheive.

That's using ZFS, so you don't want hardware RAID.

25 minutes ago, invaderSnarf said:

Apologies if I was getting the nomenclature wrong, this is the first time I've actually put some time into building a NAS for myself. 

 

Would mixing the two 12tb drives that I have now with 16tb drives later cause issues? Would I be better off saving for longer and getting all of the drives to populate the server at once rather than build it and slowly implement more and more drives as I need?

probably not

[heimdali]

55 minutes ago, LIGISTX said:

and harddrives can be easily added with a HBA for about 50 bucks

How do you get these controllers to work with the backplane the server comes with so that you can leave the hardware RAID controller unused?

55 minutes ago, LIGISTX said:

Single drive parity is just not statistically enough anymore. The chance of data loss due to a second drive failing as your rebuilding your first drive is actually a relatively not low number… thus why you want a second parity drive.

What kind of statistic says such things?  Do you have good evidence that rebuilding a RAID actually causes disk failures?

[LIGISTX]

15 hours ago, invaderSnarf said:

Apologies if I was getting the nomenclature wrong, this is the first time I've actually put some time into building a NAS for myself. 

No problem! Just trying to help you learn the terms so you can do more productive google searching   

 

15 hours ago, invaderSnarf said:

Would mixing the two 12tb drives that I have now with 16tb drives later cause issues? Would I be better off saving for longer and getting all of the drives to populate the server at once rather than build it and slowly implement more and more drives as I need?

You can’t add drives to a ZFS vdev after the fact… for instance, if you build out a let’s just say 10tb X 10 drive Z2 vdev, that’s 100 TB, with 2 drives lost to parity, so 80 TB usable. You can never add any more than 10 drives to this vdev tho, so the only 2 ways to add more space later to your NAS is:

 

1) swap out each drive 1 by one with larger drives, let’s say you upgrade to 16TB. Once all drives have been upgraded, you will then have the new 160 TB (-32 TB so 128 TH usable) of space. 
 

2) add a second vdev. But… remember, each vdev has its own redundancy requirements. So you will need to buy extra parity drives for this new vdev. So if you want to add another 80TB on top of the original 80, you need to buy 10 drives, 2 will be lost to redundancy assuming you want this vdev to also have 2 drive redundancy. 
 

So… depending on your plans, maybe? But the general rule of thumb is “buy the drives you need up front, because ZFS is not easy to most later” as defined by my previous examples. This is why a lot of home uses like unraid, it’s not RAID… so you can add random sizes drives in, and you can add them whenever you want. 
 

14 hours ago, heimdali said:

How do you get these controllers to work with the backplane the server comes with so that you can leave the hardware RAID controller unused?

Depending on the RAID card, you flash it with an HBA firmware. For LSI cards this is called IT mode. Then they act as SAS HBA’s. SAS from HBA to backplane. That said, I am not sure what cars or backplane the Dell server OP is planning to buy is. So I can’t confirm this will work. But this is the typical solution, works great for Supermicro chassis, and works great in my ATX case using SAS to 4x SATA break out cables (and a SAS expander as well). 
 

14 hours ago, heimdali said:

16 hours ago, LIGISTX said:

 

What kind of statistic says such things?  Do you have good evidence that rebuilding a RAID actually causes disk failures?

There are many folks who have built fault tolerance, MTBF, and age statistics which help show what your chance of failure is. I know there used to be a post with a website linked to one of these calculators on the truenas forums years and years ago. But no, a rebuild doesn’t “cause” a failure (although a rebuild like a scrub is when the drives are actually under extended load, so it is more likely to fail then just sitting there idle, but not by much), but the real issue is just random chance. It can take 4-5 days to rebuild a 12-16 TB disc. Within that 4-5 days, there is a non zero chance another drive will randomly die. 
 

Quick search found this, but this isn’t what I was looking for and is not “very helpful”. I remember seeing a site some created that helps show the chance of a single or multi drive failure, and the chance of a second drive failing during a resilver of a new drive. Again…. It has happened to me. So the chance is clearly >0. 

[heimdali]

6 hours ago, LIGISTX said:

Depending on the RAID card, you flash it with an HBA firmware.

Provided you can find a firmware to flash, and then the card is bricked ...  From what I've been reading, it seemed anything but easy and not so likely to have success doing that.  And who guarantees that this firmware doesn't cause isses during operation somewhere down the road.  So I decided against making such experiments.  It's not something I'd recommend.

 

There's reasons why server manufacturers tell you to use which firmware version for which controller with which firmware version of wich hard disks.  Sure flashing some RAID card with some firmware from an unverifyable source may work.  Or it might not.  Do you have the source code this firmware was created from?

6 hours ago, LIGISTX said:

For LSI cards this is called IT mode. Then they act as SAS HBA’s. SAS from HBA to backplane. That said, I am not sure what cars or backplane the Dell server OP is planning to buy is. So I can’t confirm this will work. But this is the typical solution, works great for Supermicro chassis, and works great in my ATX case using SAS to 4x SATA break out cables (and a SAS expander as well).

I'd like to know in general if that works.  Most servers have integrated RAID controllers and backplanes desiged to work with those.  Chances are I can't even plug a cable into the backplane because it's all integrated.  Even if I can plug a cable in, will that backplane work with the arbitrary HBA I plug it into?  How do I know that'll will work before buying all the hardware?

 

I'm not too fond of Supermicro.  Will it work with a Dell R720 or R730 when I plug some HBA that does JBOD into a PCI slot of the server?

 

It's all goood and nice in theory, but in practise, it isn't so easy.

 

As to drives failing during rebuilds, that can of course happen.  And if it does happen, that easily creates the myth that rebuilding a RAID makes it particularly likely that other drives fail because it's so inconvenient when it happens that the experience sticks.  How often does that actually happen and how many arrays are being rebuild without other disks failing, i. e. what is the percentage of surviving volumes vs. volumes lost during rebuilds?

 

[LIGISTX]

1 hour ago, heimdali said:

Provided you can find a firmware to flash, and then the card is bricked ...  From what I've been reading, it seemed anything but easy and not so likely to have success doing that.  And who guarantees that this firmware doesn't cause isses during operation somewhere down the road.  So I decided against making such experiments.  It's not something I'd recommend.

 

There's reasons why server manufacturers tell you to use which firmware version for which controller with which firmware version of wich hard disks.  Sure flashing some RAID card with some firmware from an unverifyable source may work.  Or it might not.  Do you have the source code this firmware was created from?

The firmware is verified…. This is standard practice, and it is not difficult to do. Pretty much any Supermicro box running a ZFS array is using this. If you are not familiar with this, go on truenas’s forums and look for “IT firmware”, or heck, just google “LSI IT firmware”, there is a plethora of information. 
 

This is the official documentation from Broadcom (they own LSI). 

https://www.broadcom.com/support/knowledgebase/1211161501344/flashing-firmware-and-bios-on-lsi-sas-hbas

 

1 hour ago, heimdali said:

I'd like to know in general if that works.  Most servers have integrated RAID controllers and backplanes desiged to work with those.  Chances are I can't even plug a cable into the backplane because it's all integrated.  Even if I can plug a cable in, will that backplane work with the arbitrary HBA I plug it into?  How do I know that'll will work before buying all the hardware?

If it’s a SAS backplane, I don’t know why it wouldn’t work. But I only have personally have experience with Supermicro’s backplanes, and no backplanes at all. My personal homelab is in a standard ATX case and I use SAS to SATA breakout cables. A buddy used a Supermicro 4U, and I have 2 SAS cables going from his IT firmware flashed HBA to the backplane, all 24 drive work, and have worked for years. Again, this is extremely common practice…

 

1 hour ago, heimdali said:

As to drives failing during rebuilds, that can of course happen.  And if it does happen, that easily creates the myth that rebuilding a RAID makes it particularly likely that other drives fail because it's so inconvenient when it happens that the experience sticks.  How often does that actually happen and how many arrays are being rebuild without other disks failing, i. e. what is the percentage of surviving volumes vs. volumes lost during rebuilds?

Yes, it’s of course not common. The odds of losing a second drive in the ~week timespan it takes to get a new drive delivered and resilvered is low, but again, it is not 0. But this is why Z2 is a good idea, because while the chance is low, that extra parity disc may save you a lot of headache and frustration. 

[heimdali]

24 minutes ago, LIGISTX said:

The firmware is verified….

Verified by whom?

24 minutes ago, LIGISTX said:

This is standard practice, and it is not difficult to do.

You mean updating the firmware?  Yes, when you use the firmware you got from the manufacturer.

24 minutes ago, LIGISTX said:

Pretty much any Supermicro box running a ZFS array is using this.

I don't have anything from Supermicro.

24 minutes ago, LIGISTX said:

If you are not familiar with this, go on truenas’s forums and look for “IT firmware”, or heck, just google “LSI IT firmware”, there is a plethora of information.

I did that a couple years ago.  You could get some modifed firmware from someone who modified it to make some LSI controllers support JBODs, and there were multiple versions of the firmware so you couldn't tell which one to use, accompanied with a big fat warnings that the risk that it wouldn't work is very high.  There was some agreement that it better to buy a controller that already has been flashed, and those were few and expensive.

24 minutes ago, LIGISTX said:

This is the official documentation from Broadcom (they own LSI). 

https://www.broadcom.com/support/knowledgebase/1211161501344/flashing-firmware-and-bios-on-lsi-sas-hbas

I'm not sure what this page is supposed to tell me.  It must be out of context.  It mentions the so-called it firmware and doesn't say where to get it or for which controllers it is other than "LSI SAS HBAs".  Which ones?  It mentions Supermicro and I could guess that is limited for a particular mainboard made by Supermicro.

 

So for which controllers is this firmware, who provides it and where do you get it?

24 minutes ago, LIGISTX said:

If it’s a SAS backplane, I don’t know why it wouldn’t work.

SAS backplanes have firmware too, so I have to assume that they don't necessarily work with arbitrary SAS controllers even if I can plug SAS cables in.

24 minutes ago, LIGISTX said:

But I only have personally have experience with Supermicro’s backplanes, and no backplanes at all. My personal homelab is in a standard ATX case and I use SAS to SATA breakout cables. A buddy used a Supermicro 4U, and I have 2 SAS cables going from his IT firmware flashed HBA to the backplane, all 24 drive work, and have worked for years. Again, this is extremely common practice…

Only 2 SAS cables for 24 drives?  Is that a special controller?

 

Maybe it's common practise, maybe not.  Supermicro servers and chassis are not common and rather expensive here, with few exceptions, so they aren't an option that would come to mind easily.  So it's questionable if it isn't better to get a more common server, and (unfortunately), I haven't seen one yet with a hardware RAID controller that would support JBOD.  That also makes it questionable if it's a good idea to use ZFS for storage because ZFS goes better with JBODs than with hardware RAID volumes.

 

It's just not as easy as recommending FreeNAS and Unraid and pretending it would be childsplay to just plug in an HBA and all problems magically solve themselves.  The OP is planning on 12 disks with a $5000 budget, and that deserves more attention to detail than assumptions and childsplay.  Such details include that you may find that you can't plug your $50 HBA into the backplane of the server you bought or that it's not compatible.  The question about adding drives later doesn't just apply to ZFS.  You can also do it with the hardware RAID the server to be used is likely to come with, and then the next question is what to do about the file system on the volume when the volume is enlarged, or what when a second volume is created from the additonal disks ...  That means you have to go back to the question if and how you want to use the hardware RAID.

24 minutes ago, LIGISTX said:

Yes, it’s of course not common. The odds of losing a second drive in the ~week timespan it takes to get a new drive delivered and resilvered is low, but again, it is not 0. But this is why Z2 is a good idea, because while the chance is low, that extra parity disc may save you a lot of headache and frustration. 

It would be interesting to know what the rate is.  And what if another disk fails during the rebuild ...  Is Z2 better than RAID6?

 

[LIGISTX]

4 hours ago, heimdali said:

Verified by whom?

6 hours ago, LIGISTX said:

Dell is the most common... https://downloads.dell.com/FOLDER01181650M/33/SASHBA_Firmware_6GBPS-SAS-HBA_07.03.06.00_A10_ZPE.exe

 

Lots of people use Dell H310's, which is why this is the most common. Again, this is not random firmware written by someone in their basement, this is official firmware... I don't think you have actually done any research into this... I would advise looking into this before proclaiming it is not a supported use case.

 

4 hours ago, heimdali said:

You mean updating the firmware?  Yes, when you use the firmware you got from the manufacturer.

6 hours ago, LIGISTX said:

Again, sas2flash is supported, and above is official Dell firmware.

 

4 hours ago, heimdali said:

I'm not sure what this page is supposed to tell me.  It must be out of context.  It mentions the so-called it firmware and doesn't say where to get it or for which controllers it is other than "LSI SAS HBAs".  Which ones?  It mentions Supermicro and I could guess that is limited for a particular mainboard made by Supermicro.

 

So for which controllers is this firmware, who provides it and where do you get it?

Again, this is really more intended for enterprise users and use cases, I believe LSI used to have a link with all firmware available via FTP (its been years since I have needed it... so its been a while), but I don't know if Broadcom still has that page active. But realistically, all the info you could possibly need is here: https://www.truenas.com/community/resources/detailed-newcomers-guide-to-crossflashing-lsi-9211-9300-9305-9311-9400-94xx-hba-and-variants.54/

 

4 hours ago, heimdali said:

SAS backplanes have firmware too, so I have to assume that they don't necessarily work with arbitrary SAS controllers even if I can plug SAS cables in.

The only SAS backplanes I personally have experience with are supermicro, and as far as I know they are just SAS expanders... They just break out SAS to SATA across the entire backplane.... while I am sure there is firmware involved, all it is doing is breaking out the signal and passing data between drives and the SAS card... 

 

4 hours ago, heimdali said:

Only 2 SAS cables for 24 drives?  Is that a special controller?

No.... not special. You can address *lots* of drives over SAS, again, I would recommend research here: https://www.sasexpanders.com/faq/

 

4 hours ago, heimdali said:

t's just not as easy as recommending FreeNAS and Unraid and pretending it would be childsplay to just plug in an HBA and all problems magically solve themselves.  The OP is planning on 12 disks with a $5000 budget, and that deserves more attention to detail than assumptions and childsplay.  Such details include that you may find that you can't plug your $50 HBA into the backplane of the server you bought or that it's not compatible.  The question about adding drives later doesn't just apply to ZFS.  You can also do it with the hardware RAID the server to be used is likely to come with, and then the next question is what to do about the file system on the volume when the volume is enlarged, or what when a second volume is created from the additonal disks ...  That means you have to go back to the question if and how you want to use the hardware RAID.

Again, this is not childs play.... this is a VERY well used use case. Most of the builds on the truenas forum run on such setups, as does servethehome etc. This isn't just a random idea I came up with....... I recommend doing research yourself before saying ideas are bad. ZFS offers some fantastic features compared to hardware RAID, but there are pluses and minuses with every setup - there is no "1 solution to fit every need", but many folks have decided ZFS is the choice for them, including many enterprises. 

 

I admit, I am not certain if the backplane the OP is referencing will work - but that is for OP to do research on. I guarantee OP isn't the first person trying to use it for this reason, and some research will give them the answers they need. A quick google by me shows many truenas.com results and reddit/homelab results, I am sure the OP will be able to find whatever info they want in order to satisfy their curiosity around ZFS and if it will work ok or not. 

 

Again, this is quite possibly one of the most common homelab infrastructures beyond simple synology or qnap devices. Myself and 3 friends all run varying flavors of ZFS (2 Truenas Core, 1 truenas scale, 1 sell rolled archlinux) and we all use HBA's which were flashed to IT mode (I did the flashing on 3 of them myself)... It is not hard, it is not "childs play" or unsupported. The instructions can be a little confusing, but if you would rather not flash it yourself, just buy one pre-flashed from ebay - they cost the same. 

 

https://www.ebay.com/itm/143333701511?hash=item215f5ab787:g:IsIAAOSwhQJdMq63&amdata=enc%3AAQAHAAAA4M4klSWHnBd18XVHz2lNPCjwFmX%2BvDVzriWYhq9nMNlnZO6WRFZlqqq700h2Z186DJgNQ2aKYBpLORrovUA9FW6XZ1%2FidryIgXUzpagjD6SBL3A40aSC3Te9on%2Bz1COghWdoHgyhaStbZKAPffu%2FjRQwZEtL2h4lyhd130JMkMFqYU3z0VCAO4ylPW6bI8KiON26FuFqzDRQLlGR5EFQHrMr56axUcYslUBGvnGVm5TfnITcwT7tP0oBTRKnH4%2BBDVzpTU%2Bb84zD8klziUEXOZpve%2FVYQBNS3A8xh3NQZnjS|tkp%3ABFBMzNXG441h

 

P20 is the newest as far as I know, but again, research is your friend. 

 

I recommend giving all of the links I posted a thorough read through

[Blue4130]

4 hours ago, heimdali said:

Verified by whom?

Broadcomm, the people who make the card.

4 hours ago, heimdali said:

You mean updating the firmware?  Yes, when you use the firmware you got from the manufacturer.

Yes, the firmware is once again from braodcomm.

4 hours ago, heimdali said:

I'm not sure what this page is supposed to tell me.  It must be out of context.  It mentions the so-called it firmware and doesn't say where to get it or for which controllers it is other than "LSI SAS HBAs".  Which ones?  It mentions Supermicro and I could guess that is limited for a particular mainboard made by Supermicro.

https://www.broadcom.com/support/download-search

Here is their download page. Search for the card in question and voila, firmware. From the manufacturer, on the manufacturer's page. For the card you choose...

 

You seem to be deadset in making this seem very difficult and scary and not a supported thing. It's far from it. It's industry standard. It is a well documented aspect of their products.

4 hours ago, heimdali said:

Maybe it's common practise, maybe not.  Supermicro servers and chassis are not common and rather expensive here, with few exceptions, so they aren't an option that would come to mind easily.  So it's questionable if it isn't better to get a more common server, and (unfortunately), I haven't seen one yet with a hardware RAID controller that would support JBOD.  That also makes it questionable if it's a good idea to use ZFS for storage because ZFS goes better with JBODs than with hardware RAID volumes

I am not sure where you are located, but pretty much the world over, supermicro is an extremely popular server company. They are VERY common. As for ZFS on a RAID card, you are correct. Don't use it if you are using a Raid card in Raid mode. It must be in HBA mode, which is why Broadcomm has verified firmware to change it.

[LIGISTX]

8 minutes ago, Blue4130 said:

You seem to be deadset in making this seem very difficult and scary and not a supported thing. It's far from it. It's industry standard. It is a well documented aspect of their products.

Very much this.....

 

And also this............

8 minutes ago, Blue4130 said:

I am not sure where you are located, but pretty much the world over, supermicro is an extremely popular server company. They are VERY common. As for ZFS on a RAID card, you are correct. Don't use it if you are using a Raid card in Raid mode. It must be in HBA mode, which is why Broadcomm has verified firmware to change it.

 

[heimdali]

9 hours ago, LIGISTX said:

Dell is the most common... https://downloads.dell.com/FOLDER01181650M/33/SASHBA_Firmware_6GBPS-SAS-HBA_07.03.06.00_A10_ZPE.exe

I have no use for a program that requires Windows.  When I search for H310 firmware, I'm getting this: https://www.dell.com/support/home/de-de/drivers/driversdetails?driverid=r09pj

 

It doesn't say anything about JBOD.  The specs for the controller don't say anything about JBOD, either: https://i.dell.com/sites/doccontent/shared-content/data-sheets/Documents/dell-perc-h310-spec-sheet.pdf

 

Have you ever used controller cards from Dell?  Their cards have always been troublesome and I'd only get one if I had Dell hardware to use it in.  Otherwise it might not work, I've had that a couple times.

 

9 hours ago, LIGISTX said:

Lots of people use Dell H310's, which is why this is the most common. Again, this is not random firmware written by someone in their basement, this is official firmware...

I don't see that.  You're posting a link to an unusable file I know nothing about.

9 hours ago, LIGISTX said:

I don't think you have actually done any research into this... I would advise looking into this before proclaiming it is not a supported use case.

I have.  Just now I have again, and I have found that the H310 doesn't support JBOD.  The latest firmware release is from 2016 and is unusable, and it says nothing about JBOD.

 

So where does Dell the controller does JBOD?  Since I don't have Dell hardware I could use it in, I'd have to get something else.

9 hours ago, LIGISTX said:

Again, sas2flash is supported, and above is official Dell firmware.

 

Is it?

 

dnf search sas2flash
Last metadata expiration check: 2:49:02 ago on Sun 13 Nov 2022 07:15:45 AM CET.
No matches found.

 

Broadcom has a guide (https://docs.broadcom.com/doc/12353205) but it doesn't say how to get this program.  And the kernel version they're referring to is ancient.  Now what?

 

The H310 isn't exactly the most common card here when I look on ebay.  It would be a bad choice for me.

 

9 hours ago, LIGISTX said:

Again, this is really more intended for enterprise users

Like enterprises would mess around like that, yeah, sure.

9 hours ago, LIGISTX said:

and use cases, I believe LSI used to have a link with all firmware available via FTP (its been years since I have needed it... so its been a while), but I don't know if Broadcom still has that page active. But realistically, all the info you could possibly need is here: https://www.truenas.com/community/resources/detailed-newcomers-guide-to-crossflashing-lsi-9211-9300-9305-9311-9400-94xx-hba-and-variants.54/

The guide is a bit dated and it proves my point.  This isn't something I would want to do, even if there's an official firmware version, not at home and not at work.

 

And I wonder why don't they make these controllers so that they support JBOD and RAID to begin with.  Why not give their customers give that option?

9 hours ago, LIGISTX said:

 

The only SAS backplanes I personally have experience with are supermicro, and as far as I know they are just SAS expanders... They just break out SAS to SATA across the entire backplane.... while I am sure there is firmware involved, all it is doing is breaking out the signal and passing data between drives and the SAS card... 

So you can't use SAS drives in Supermicro backplanes?

9 hours ago, LIGISTX said:

No.... not special. You can address *lots* of drives over SAS, again, I would recommend research here: https://www.sasexpanders.com/faq/

The only backplanes I have come across aren't SAS expanders.

9 hours ago, LIGISTX said:

Again, this is not childs play.... this is a VERY well used use case. Most of the builds on the truenas forum run on such setups, as does servethehome etc. This isn't just a random idea I came up with....... I recommend doing research yourself before saying ideas are bad.

I'm not saying it's a bad idea.  I'm saying it needs to be thought through more thoroughly.

9 hours ago, LIGISTX said:

ZFS offers some fantastic features compared to hardware RAID, but there are pluses and minuses with every setup - there is no "1 solution to fit every need", but many folks have decided ZFS is the choice for them, including many enterprises. 

One of the biggest disadvantages of file systems using software RAID is that they work best with JBOD.  It's a big disadvantage because the hardware you get usually doesn't support JBOD but comes with hardware RAID.  Changing that hardware to JBOD isn't easy.

9 hours ago, LIGISTX said:

I admit, I am not certain if the backplane the OP is referencing will work - but that is for OP to do research on.

What you can research is limited.  Do you have an email address so I can ask Dell?  I don't, and even if I had one, I probably won't get an answer.

9 hours ago, LIGISTX said:

I guarantee OP isn't the first person trying to use it for this reason, and some research will give them the answers they need. A quick google by me shows many truenas.com results and reddit/homelab results, I am sure the OP will be able to find whatever info they want in order to satisfy their curiosity around ZFS and if it will work ok or not. 

You're very optimistic

9 hours ago, LIGISTX said:

Again, this is quite possibly one of the most common homelab infrastructures beyond simple synology or qnap devices. Myself and 3 friends all run varying flavors of ZFS (2 Truenas Core, 1 truenas scale, 1 sell rolled archlinux) and we all use HBA's which were flashed to IT mode (I did the flashing on 3 of them myself)... It is not hard, it is not "childs play" or unsupported. The instructions can be a little confusing, but if you would rather not flash it yourself, just buy one pre-flashed from ebay - they cost the same. 

You keep trying to make it sound like it's childs play, which it isn't.  Assume I bought a nice server, like a Dell R720 with 12 LFF bays, and I wanted to use it for storage with ZFS.  Now what?  The server comes with hardware RAID.

9 hours ago, LIGISTX said:

https://www.ebay.com/itm/143333701511?hash=item215f5ab787:g:IsIAAOSwhQJdMq63&amdata=enc%3AAQAHAAAA4M4klSWHnBd18XVHz2lNPCjwFmX%2BvDVzriWYhq9nMNlnZO6WRFZlqqq700h2Z186DJgNQ2aKYBpLORrovUA9FW6XZ1%2FidryIgXUzpagjD6SBL3A40aSC3Te9on%2Bz1COghWdoHgyhaStbZKAPffu%2FjRQwZEtL2h4lyhd130JMkMFqYU3z0VCAO4ylPW6bI8KiON26FuFqzDRQLlGR5EFQHrMr56axUcYslUBGvnGVm5TfnITcwT7tP0oBTRKnH4%2BBDVzpTU%2Bb84zD8klziUEXOZpve%2FVYQBNS3A8xh3NQZnjS|tkp%3ABFBMzNXG441h

 

P20 is the newest as far as I know, but again, research is your friend. 

 

I recommend giving all of the links I posted a thorough read through

I researched this problem a couple years ago and the best option that was available was a different one.  I still don't like this option and it works fine, and I still don't have a better one without spending quite some money.

 

ZFS is not always the best solution.  If you had bought a server like the abovementioned one, which is readily available for a reasonable price and ready to go once you install some disks, you're probably best off by just using the hardware RAID it comes with and put btrfs, xfs or ext4 on the RAID volume.  To use ZFS, you have to go ways out of your way and may end up not getting that server to work with a JBOD.

[heimdali]

10 hours ago, Blue4130 said:

Broadcomm, the people who make the card.

Yes, the firmware is once again from braodcomm.

https://www.broadcom.com/support/download-search

Here is their download page. Search for the card in question and voila, firmware. From the manufacturer, on the manufacturer's page. For the card you choose...

Cool, thanks!  You mean, for example, this: https://docs.broadcom.com/docs/9500_8i_Pkg_P25_MIXED_FW_BIOS_UEFI.zip

They have a README that seems to mention IT here: https://docs.broadcom.com/docs/README_9500_8i_Pkg_P25_MIXED_FW_BIOS_UEFI.txt

 

Is that what I would use?

10 hours ago, Blue4130 said:

You seem to be deadset in making this seem very difficult and scary and not a supported thing. It's far from it. It's industry standard. It is a well documented aspect of their products.

Are you saying if I were to buy a Dell R720, I could flash the firmware of the RAID controller to support JBOD and it would work?

10 hours ago, Blue4130 said:

I am not sure where you are located, but pretty much the world over, supermicro is an extremely popular server company.

They are very well known, yes.  They've been around like forever.  That doesn't mean they're common.

10 hours ago, Blue4130 said:

They are VERY common.

Check German ebay.  You can get stuff from Supermicro, but servers from HP and Dell are way more common and less costly here.  You can also get spare parts more easily for servers that are more common.  Perhaps Supermicro is very common in the US, but not so much in Europe.  Otherwise I'd be seeing way more hardware from Supermicro.

 

[Blue4130]

11 minutes ago, heimdali said:

 

Are you saying if I were to buy a Dell R720, I could flash the firmware of the RAID controller to support JBOD and it would work?

 

Yes. Just look at the number of people running freenas on Dell r720's on the other servers in their lineup. JBOD works perfectly fine.

11 minutes ago, heimdali said:

They are very well known, yes.  They've been around like forever.  That doesn't mean they're common.

Check German ebay.  You can get stuff from Supermicro, but servers from HP and Dell are way more common and less costly here.  You can also get spare parts more easily for servers that are more common.  Perhaps Supermicro is very common in the US, but not so much in Europe.  Otherwise I'd be seeing way more hardware from Supermicro.

 

I am not in North America. No idea what the availability is there, where I am, they are very common. As for spare parts, they use pretty standard stuff. Much moreso than dell or HPE. At least with SuperMicro I can use a standard ATX powersupply, not some proprietary HPE or Dell or Lenovo thing.

 

36 minutes ago, heimdali said:

I have no use for a program that requires Windows.  When I search for H310 firmware, I'm getting this: https://www.dell.com/support/home/de-de/drivers/driversdetails?driverid=r09pj

 

It doesn't say anything about JBOD.  The specs for the controller don't say anything about JBOD, either: https://i.dell.com/sites/doccontent/shared-content/data-sheets/Documents/dell-perc-h310-spec-sheet.pdf

 

Have you ever used controller cards from Dell?  Their cards have always been troublesome and I'd only get one if I had Dell hardware to use it in.  Otherwise it might not work, I've had that a couple times.

 

I don't see that.  You're posting a link to an unusable file I know nothing about.

I have.  Just now I have again, and I have found that the H310 doesn't support JBOD.  The latest firmware release is from 2016 and is unusable, and it says nothing about JBOD.

So you want to know a dirty secret (that really isn't a secret)? The Dell h310 is just a LSI SAS 2008. It is perfectly happy to work as a HBA for JBOD use.

 

https://www.servethehome.com/lsi-sas-2008-raid-controller-hba-information/

36 minutes ago, heimdali said:

The H310 isn't exactly the most common card here when I look on ebay.  It would be a bad choice for me.

It is when you look at it's other name that I posted above. Obviously, it has been replaced with newer versions, but the 2008 is a workhorse. It's legacy lives on.

36 minutes ago, heimdali said:

 

Like enterprises would mess around like that, yeah, sure.

The guide is a bit dated and it proves my point.  This isn't something I would want to do, even if there's an official firmware version, not at home and not at work.

That's on you then. It is a factory feature, if you choose not to use it, well nobody can help you with that.

 

36 minutes ago, heimdali said:

And I wonder why don't they make these controllers so that they support JBOD and RAID to begin with.  Why not give their customers give that option?

They do give that option. Flash the card using the supplier provided software.

 

 

 

36 minutes ago, heimdali said:

So you can't use SAS drives in Supermicro backplanes?

Yes, you can use SAS in supermicro backplanes.

https://www.supermicro.org.cn/en/products/accessories/type

This has a list of 79 different SAS backplanes.

 

 

36 minutes ago, heimdali said:

The only backplanes I have come across aren't SAS expanders.

It's quite common to have the backplane be an expander. That way you don't need to populate the machine with more PCI-e cards than necessary. After all, many modern HBA cards can address 1024 drives.

 

 

36 minutes ago, heimdali said:

I'm not saying it's a bad idea.  I'm saying it needs to be thought through more thoroughly.

The guy you are aguing against here ( @LIGISTX ) has proven his chops in this area. Pretty sure he has thought it more than you give him credit for.

 

 

36 minutes ago, heimdali said:

One of the biggest disadvantages of file systems using software RAID is that they work best with JBOD.  It's a big disadvantage because the hardware you get usually doesn't support JBOD but comes with hardware RAID.  Changing that hardware to JBOD isn't easy.

It's not a disadvantage if you flash the card THAT SUPPORTS jbod using tools provided by the cards manufacturer and is fully supported and approved by them. Changing it is easy. You seem to be running A flavor of Linux, It should be a walk in the park for you.

 

 

36 minutes ago, heimdali said:

You keep trying to make it sound like it's childs play, which it isn't.  Assume I bought a nice server, like a Dell R720 with 12 LFF bays, and I wanted to use it for storage with ZFS.  Now what?  The server comes with hardware RAID.

No, it comes with a controller card that is in RAID mode. If you don't like that it is in RAID mode, don't buy that particular server loadout. Get one with the card already in IT mode.

 

[LIGISTX]

6 hours ago, heimdali said:

 

Have you ever used controller cards from Dell?  Their cards have always been troublesome and I'd only get one if I had Dell hardware to use it in.  Otherwise it might not work, I've had that a couple times.

Again, I don’t think you are actually doing any research here and are just continually stating it as a bad idea since you don’t want to actually do the research or try it out. I have a Dell h310, as does a friend of mine. Mine has ran in a HPE system, and is now in a Supermicro system. My buddies is in my old HPE system. They work perfectly as intended… because they are just LSI 2008 which is abundantly available. LSI just allowed OEM’s to use their chips on their card. This is the same as GPU’s. Asus and gigabyte don’t make GPU die’s, they use nvidia chips on their boards, but at the end of the day, they are all nvidia GeForce GPU’s. Same with Intel NIC’s. You can get a quad port NIC from Dell, but as long as it’s an Intel nic they used, it will work just like any other Intel based NIC.

 

Yes, some companies with some products do lock things down to only work in their ecosystems… H310’s are not one of those. Again, just do a google search for h310 IT mode, and there will be so many more guides than you even know what to do with.
 

I don’t know what else to tell you. I can only lead the horse to water, I can’t force it to drink. 

[heimdali]

3 hours ago, Blue4130 said:

Yes. Just look at the number of people running freenas on Dell r720's on the other servers in their lineup. JBOD works perfectly fine.

That should be interesting.  I guess you suggest I should look at the TrueNAS forum to find these people?  It would make the R720 a good option because it gives you all possibilities.

3 hours ago, Blue4130 said:

I am not in North America. No idea what the availability is there, where I am, they are very common. As for spare parts, they use pretty standard stuff. Much moreso than dell or HPE. At least with SuperMicro I can use a standard ATX powersupply, not some proprietary HPE or Dell or Lenovo thing.

Being able to use standard PSUs can have it's advantages.  It becomes a bit difficult when you want a dual PSU, though.

3 hours ago, Blue4130 said:

 

So you want to know a dirty secret (that really isn't a secret)? The Dell h310 is just a LSI SAS 2008. It is perfectly happy to work as a HBA for JBOD use.

Yes, I know.

3 hours ago, Blue4130 said:

https://www.servethehome.com/lsi-sas-2008-raid-controller-hba-information/

It is when you look at it's other name that I posted above. Obviously, it has been replaced with newer versions, but the 2008 is a workhorse. It's legacy lives on.

At some point, hardware can get too old to find the software needed to use it.

3 hours ago, Blue4130 said:

That's on you then. It is a factory feature, if you choose not to use it, well nobody can help you with that.

Have you looked at the guide?  It makes it seem like it's a long, difficult and dangerous endeavour.  A factory feature would be 'follow the instructions relevant for the operating system you're using, download the right version of the firmware from the web pages of the manufacturer and perform update'.  Usually, that doesn't require an external guide.

3 hours ago, Blue4130 said:

They do give that option. Flash the card using the supplier provided software.

I mean without modifying the firmware, of course.  The cards can do JBOD, so why don't they support that along with the RAID levels they support without questionable guides and firmware that seems phony.

3 hours ago, Blue4130 said:

Yes, you can use SAS in supermicro backplanes.

https://www.supermicro.org.cn/en/products/accessories/type

This has a list of 79 different SAS backplanes.

Yeah, they have a lot of backplanes.  Some are SATA only.

3 hours ago, Blue4130 said:

It's quite common to have the backplane be an expander.

Common where?  With backplanes Supermicro makes or others as well?

3 hours ago, Blue4130 said:

The guy you are aguing against here ( @LIGISTX ) has proven his chops in this area. Pretty sure he has thought it more than you give him credit for.

Maybe, maybe not.  If he has thought it all through, that doesn't show.  According to him, it's all childsplay, and I'm merely saying it's not.

3 hours ago, Blue4130 said:

It's not a disadvantage if you flash the card THAT SUPPORTS jbod using tools provided by the cards manufacturer and is fully supported and approved by them. Changing it is easy. You seem to be running A flavor of Linux, It should be a walk in the park for you.

That requires that the card you have even can support JBOD.  When I look at the R720, they come with H710 cards, and when I search for "H710P jbod", it doesn't seem to be easy at all to get it to do JBOD.  If that's an option supported by the manufacturer just like any other firmware update, then why does it appear to be so difficult to do?  Can you guarantee that when I buy an R720, that I can get it easily to support JBOD?  What if not?  And what about HP servers?

3 hours ago, Blue4130 said:

No, it comes with a controller card that is in RAID mode. If you don't like that it is in RAID mode, don't buy that particular server loadout. Get one with the card already in IT mode.

Sure, show me one that does.

[heimdali]

4 hours ago, LIGISTX said:

Again, I don’t think you are actually doing any research here and are just continually stating it as a bad idea since you don’t want to actually do the research or try it out.

You probably never used controller cards from Dell in non-Dell hardware.  I tried it a couple times with different cards and they did not work.  I don't need to research that any further.

 

I could make more experiments with Dell cards, but are you gona pay for it?

[LIGISTX]

25 minutes ago, heimdali said:

You probably never used controller cards from Dell in non-Dell hardware.  I tried it a couple times with different cards and they did not work.  I don't need to research that any further.

Maybe your not reading my posts..? I have a Dell H310 in my system, as do a few of my friends. None of us have any other Dell hardware.

[Blue4130]

53 minutes ago, heimdali said:

You probably never used controller cards from Dell in non-Dell hardware.  I tried it a couple times with different cards and they did not work.  I don't need to research that any further.

 

I could make more experiments with Dell cards, but are you gona pay for it?

I have given many useful links to back up my statements. You give nothing but "what if's" and doubt. It's not on me to convince you.