FreeNAS setup

[martward]

Hi everyone,

 

I am a complete beginner in the field of NAS systems, and I want to build one next summer for my dad's company.

They don't have very much storage, so I would like to build them a a NAS with 2TB effective storage to start with, and have the option to expand this later on.

 

I have done some research, and the things I found were:

- WD Red's are good for NAS systems

- I should use multiple vdevs instead of one vdev with raidz2 (raid 5).

 

So I was planning on getting 4x1TB HDD's and put them in 2 vdevs.

I still have a couple of questions about this though:

 

- Should I put the 2 drives in each vdev in raid 1 or raid 0? I know the difference, but I do not know whether the vdevs are in any form of raid, and if so, in which type.

- Do I get any speed improvement over the normal disk speed (like raid 0/5/6) if I put the disks within the vdevs in raid 1?

- Is it possible to add a vdev of a different size (like a vdev of 2x2TB) to the pool and would I lose more than the normal redundancy capacity if I do?

- Is 8GB of ram enough for a NAS with less than 10TB of storage? I read somewhere that it is enough for up to 24TB and somewhere else that you need 8+n GB where n is the number of TB in you NAS (so for a 4TB NAS that would be 12GB)

 

I tried to google this, but I found a lot of contradictory information and answers to questions which were like the ones I have but don't answer my questions completely.

 

Thanks

[79wjd]

Hi everyone,

 

I am a complete beginner in the field of NAS systems, and I want to build one next summer for my dad's company.

They don't have very much storage, so I would like to build them a a NAS with 2TB effective storage to start with, and have the option to expand this later on.

 

I have done some research, and the things I found were:

- WD Red's are good for NAS systems

- I should use multiple vdevs instead of one vdev with raidz2 (raid 5).

 

So I was planning on getting 4x1TB HDD's and put them in 2 vdevs.

I still have a couple of questions about this though:

 

- Should I put the 2 drives in each vdev in raid 1 or raid 0? I know the difference, but I do not know whether the vdevs are in any form of raid, and if so, in which type.

- Do I get any speed improvement over the normal disk speed (like raid 0/5/6) if I put the disks within the vdevs in raid 1?

- Is it possible to add a vdev of a different size (like a vdev of 2x2TB) to the pool and would I lose more than the normal redundancy capacity if I do?

- Is 8GB of ram enough for a NAS with less than 10TB of storage? I read somewhere that it is enough for up to 24TB and somewhere else that you need 8+n GB where n is the number of TB in you NAS (so for a 4TB NAS that would be 12GB)

 

I tried to google this, but I found a lot of contradictory information and answers to questions which were like the ones I have but don't answer my questions completely.

 

Thanks

I'd say (depending on the data), either go with three 1tb drives in Raid 5 or two arrays of two 1tb drives in Raid 0 (so, two logical drives of 2tb each in Raid 1) (the latter being for super critical information that you're extremely paranoid about losing). 

 

Only Raid 0 will offer a performance increase (with a reliability decrease).

 

8GB of ram is plenty. You can ignore most of the FreeNAS forums as most of the people who post their are elitists assholes who don't seem to understand that FreeNAS was originally (and still is) developed as an easy to run NAS OS that could (can) run on old hardware.

[martward]

I'd say (depending on the data), either go with three 1tb drives in Raid 5 or two arrays of two 1tb drives in Raid 0 (so, two logical drives of 2tb each in Raid 1) (the latter being for super critical information that you're extremely paranoid about losing). 

 

Only Raid 0 will offer a performance increase (with a reliability decrease).

 

8GB of ram is plenty. You can ignore most of the FreeNAS forums as most of the people who post their are elitists assholes who don't seem to understand that FreeNAS was originally (and still is) developed as an easy to run NAS OS that could (can) run on old hardware.

I would really like to avoid raid 5 since it is (apparantly) less safe and more difficult to expand. So the thing is that adding extra drives to a raid is impossible with zfs, but adding a new vdev to the pool is possible. So did you mean that I put the vdevs in raid (if that is even possible) or that I basically create one vdev with a raid 10 config? Because I can't add new drives to a raid 10 config right?

[79wjd]

I would really like to avoid raid 5 since it is (apparantly) less safe and more difficult to expand. So the thing is that adding extra drives to a raid is impossible with zfs, but adding a new vdev to the pool is possible. So did you mean that I put the vdevs in raid (if that is even possible) or that I basically create one vdev with a raid 10 config? Because I can't add new drives to a raid 10 config right?

Create one vdev with a Raid 10 config, but Raid 5 is still pretty safe. Raid 5 works by including parity information on each drive such that if any one drive fails the remaining drives have the information (parity) to rebuild the data on the dead drive. Whereas Raid 1 just duplicates the data. So, Raid 5 is redundant, and is what's used most often in large data pools as Raid 1 doesn't make sense unless the data is super critical. As for being harder to expand, well, ZFS doesn't allow you to expand arrays, so Raid 5 wouldn't be any more difficult than Raid 1. 

 

So like I said, it really depends on what the data is. If it's super critical then Raid 1 is fine (although, I would go with two 2tb drives in Raid 1 rather than four 1tb drives in Raid 10), if it's not than Raid 5 would be fine. 

[alpenwasser]

Hi everyone,

 

I am a complete beginner in the field of NAS systems, and I want to build one next summer for my dad's company.

They don't have very much storage, so I would like to build them a a NAS with 2TB effective storage to start with, and have the option to expand this later on.

 

Just a question: Have you looked at Synology's or QNAP's offerings? They're really quite good,

and tend to be easier to set up and maintain than a DIY setup (which might be relevant for company

stuff where data being unavailable can mean losing money).

Not that I'm trying to dissuade you from building your own NAS, just thought I'd mention the

possibility of alternatives since you say you're a beginner, for enabling you to make a well-informed

decision. Nothing wrong with building your own of course, if that's what you truly wish to do.

 

I have done some research, and the things I found were:

- WD Red's are good for NAS systems

- I should use multiple vdevs instead of one vdev with raidz2 (raid 5).

 

Yes, WD Reds are a good choice for this task.

There are advantages and downsides to having multiple vs. single vdevs. The advantage of doing

multiple vdevs is that you'll end up with a faster setup, since the vdevs will essentially be

run in something like RAID0 (so, two RAIDZ2 vdevs would be similar to RAID60).

The advantage of having it all in a single vdev is that you can have any two drives fail,

and the vdev will still work (in the case of RAIDZ2). If you span across multiple vdevs and start

having multiple drives fail, at some point they must distribute across the vdevs, if you get too

many in one vdev, the vdev will fail. This is for example why I run my personal data pool of four

drives in RAIDZ2. I could do two mirror vdevs as well, and would have the same amount of storage

available, but the zpool could only survive two drive failures simultaneously as long as those

drives are not in the same mirror, whereas in a RAIDZ2 setup, any of the four drives can fail.

But of course, two mirror vdevs would be faster.

 

So I was planning on getting 4x1TB HDD's and put them in 2 vdevs.

I still have a couple of questions about this though:

 

- Should I put the 2 drives in each vdev in raid 1 or raid 0? I know the difference, but I do not know whether the vdevs are in any form of raid, and if so, in which type.

You can't put drives inside a vdev in RAID0 mode, RAID0 is basically reserved for the zpool

level (so, doing multiple vdevs will always result in those vdevs doing basically a RAID0,

except cache devices and such).

 

- Do I get any speed improvement over the normal disk speed (like raid 0/5/6) if I put the disks within the vdevs in raid 1?

If you do mirrors, your write speed will be bound by the slowest drive in the mirror. Read speed

will indeed increase as ZFS reads from all drives in the mirrors, not just one.

A RAIDZ* setup will only be as fast as a single disk as far as I have been able to determine

(i.e. reading around on the web and doing my own tests). If you want RAIDZ* speed improvements,

you'll need to span your zpool across multiple RAIDZ devices like in RAID50 or RAID60. But for

that to make sense you'd need more than 4 drives.

 

- Is it possible to add a vdev of a different size (like a vdev of 2x2TB) to the pool and would I lose more than the normal redundancy capacity if I do?

You can combine vdevs made up of different drives and numbers of drives pretty much as you

want in ZFS. Redundancy is always managed on the vdev level in ZFS, not on the zpool level

(so you never want a vdev to fail because then you'd lose your entire pool).

So, if you have one vdev consisting of a single drive, and another vdev in RAIDZ2, if that

single drive from the first vdev fails, the redundancy in your second vdev is useless and

you'll lose your pool. Always make sure you ahve your vdevs well-designed, but you can combine

vdevs of different sorts (for example, I used to run a vdev of three 3 TB drives and a vdev

of four 2 TB drives, both in RAIDZ1, without issues).

If you start doing weird combinations, you might suffer some speed degradation (especially

if the vdev you had in your pool first is very full and you then add another vdev to gain

more space, because the data won't be redustributed, so your first vdev will stay full

and the pool performance will be impaired somewhat).

When you mix drives of different sizes inside a vdev, you are of course restricted by

the smallest drive (so, a 1 TB drive and a 2 TB drive in a mirror config would yield you

1 TB of usable space, for example).

 

- Is 8GB of ram enough for a NAS with less than 10TB of storage? I read somewhere that it is enough for up to 24TB and somewhere else that you need 8+n GB where n is the number of TB in you NAS (so for a 4TB NAS that would be 12GB)

Ignore the "1 GB of RAM for 1 TB of storage" rule, it's usually irrelevant for home

users but somehow keeps popping up. ZFS loves RAM, and will use it to increase performance,

but I used to run a 17 TB pool with ZFS on Linux on 4 GB of RAM and it was perfectly

fine. ZFS doesn't break when it doesn't have enough RAM, it just becomes very slow.

FreeNAS itself might have some issues though, so I'd stick with at least 8 GB, but

unless you start noticing horrible performance, I wouldn't just go beyond that with

your amount of storage for now.

I do heavily recommend ECC RAM though.

 

I'd say (depending on the data), either go with three 1tb drives in Raid 5 or two arrays of two 1tb drives in Raid 0 (so, two logical drives of 2tb each in Raid 1) (the latter being for super critical information that you're extremely paranoid about losing).

Yeah, personally I would find a zpool of two vdevs, each vdev being 2 × 1 TB drive in

a mirror config (so, RAID10 basically), to be a pretty good setup for this scenario.

It's a good compromise of speed and reliability IMHO. Or, as said, RAIDZ2, which offers

more safety at the cost of speed.

Also, don't rely on RAIDZ as your backup, make sure to have an actual backup, as

RAID is not one.

[79wjd]

Yeah, personally I would find a zpool of two vdevs, each vdev being 2 × 1 TB drive in

a mirror config (so, RAID10 basically), to be a pretty good setup for this scenario.

It's a good compromise of speed and reliability IMHO. Or, as said, RAIDZ2, which offers

more safety at the cost of speed.

I thought Z2 was Raid 5 equivalent....or is that Z1 and Z2 is Raid 6?

[alpenwasser]

I thought Z2 was Raid 5 equivalent....or is that Z1 and Z2 is Raid 6?

zpool top level ~ RAID0

mirror ~ RAID1 (you can do more than two drives mirroring in ZFS though, unlike RAID1 I think)

RAIDZ1 ~ RAID5 (one drive parity)

RAIDZ2 ~ RAID6 (two drives parity)

RAIDZ3 would be RAID7, if such a thing existed (three drives parity).

zpool with multiple mirrors ~ RAID10

zpool with multiple RAIDZ1 ~ RAID50

zpool with multiple RAIDZ2 ~ RAID60

zpool with multiple RAIDZ3 doesn't really have an equivalent since no RAID7, but you get the idea.

And of course you could also mix vdevs of different sorts in a single zpool, in which case

there is no conventional equivalent either (though how much sense this makes is another

question ).

[79wjd]

zpool top level ~ RAID0

mirror ~ RAID1 (you can do more than two drives mirroring in ZFS though, unlike RAID1 I think)

RAIDZ1 ~ RAID5 (one drive parity)

RAIDZ2 ~ RAID6 (two drives parity)

RAIDZ3 would be RAID7, if such a thing existed (three drives parity).

zpool with multiple mirrors ~ RAID10

zpool with multiple RAIDZ1 ~ RAID50

zpool with multiple RAIDZ2 ~ RAID60

zpool with multiple RAIDZ3 doesn't really have an equivalent since no RAID7, but you get the idea.

And of course you could also mix vdevs of different sorts in a single zpool, in which case

there is no conventional equivalent either (though how much sense this makes is another

question ).

Is there a zpool with multiple stripes? I.e. if I wanted to run my three 4tb Reds in Raid 0 but without striping across multiple drives (i.e. that way they would look like one logical storage drive but would also maintain the ability for one drive to fail without compromising the data on the other two)?

[martward]

zpool top level ~ RAID0

mirror ~ RAID1 (you can do more than two drives mirroring in ZFS though, unlike RAID1 I think)

RAIDZ1 ~ RAID5 (one drive parity)

RAIDZ2 ~ RAID6 (two drives parity)

RAIDZ3 would be RAID7, if such a thing existed (three drives parity).

zpool with multiple mirrors ~ RAID10

zpool with multiple RAIDZ1 ~ RAID50

zpool with multiple RAIDZ2 ~ RAID60

zpool with multiple RAIDZ3 doesn't really have an equivalent since no RAID7, but you get the idea.

And of course you could also mix vdevs of different sorts in a single zpool, in which case

there is no conventional equivalent either (though how much sense this makes is another

question ).

Awesome info, thank you (for both posts ofcourse). I was planning to go with the zpool with multiple mirrors, but I couldn't find (or didn't understand) how the vdevs were connected to each other. 

 

Create one vdev with a Raid 10 config, but Raid 5 is still pretty safe. Raid 5 works by including parity information on each drive such that if any one drive fails the remaining drives have the information (parity) to rebuild the data on the dead drive. Whereas Raid 1 just duplicates the data. So, Raid 5 is redundant, and is what's used most often in large data pools as Raid 1 doesn't make sense unless the data is super critical. As for being harder to expand, well, ZFS doesn't allow you to expand arrays, so Raid 5 wouldn't be any more difficult than Raid 1. 

 

So like I said, it really depends on what the data is. If it's super critical then Raid 1 is fine (although, I would go with two 2tb drives in Raid 1 rather than four 1tb drives in Raid 10), if it's not than Raid 5 would be fine. 

It's not super critical since backups will be made very regularly, but maintenance and expansion are pretty important.

 

The expansion option I wanted to have would probably be like adding another vdev in RAIDZ1 to the pool. So I would add, for instance, a vdev of 2x2TB in raid 1 to the pool. I am not going to get creative too much since the amount of storage and speed needed is not that big. It is a relatively small company with only 4-6 computers attached to the NAS at the same time.

 

Most of all, I like doing projects like this, which is why I want to do it myself . Plus doing it myself greatly increases the price/performance ratio, since I can build a NAS with more RAM, more storage and a better CPU for less money than Qnap and Synology. They have a very ancient windows server with 500GB at the moment, so it will be pretty hard to disappoint them anyway ^^.

 

Thanks everyone!!

[alpenwasser]

Is there a zpool with multiple stripes? I.e. if I wanted to run my three 4tb Reds in Raid 0 but without striping across multiple drives (i.e. that way they would look like one logical storage drive but would also maintain the ability for one drive to fail without compromising the data on the other two)?

 

I'm not quite sure I understand what you mean, but if you do RAID0 you always stripe

across different drives, that's the nature of that RAID mode (well, at least as far

as I have been able to find out).

If you do a RAID0 setup (so, a zpool consisting of single-drive vdevs without reduncany)

you can't lose any drives without losing your pool.

If I've misunderstood your intentions, feel free to clarify.

 

Awesome info, thank you (for both posts ofcourse). I was planning to go with the zpool with multiple mirrors, but I couldn't find (or didn't understand) how the vdevs were connected to each other. 

 

It's not super critical since backups will be made very regularly, but maintenance and expansion are pretty important.

Yeah, the whole vdev thing can be a bit confusing if you're new to ZFS. Just keep in

mind that whatever happens, no vdev must ever fail, or else your pool is dead (because

as mentioned, the pool itself basically does RAID0 across all vdevs), and plan your

pool setup accordingly, then you should be fine.

The expansion option I wanted to have would probably be like adding another vdev in RAIDZ1 to the pool. So I would add, for instance, a vdev of 2x2TB in raid 1 to the pool. I am not going to get creative too much since the amount of storage and speed needed is not that big. It is a relatively small company with only 4-6 computers attached to the NAS at the same time.

Just a small note: RAID1 is not RAIDZ1, as I mentioned in the post above.

RAID1 in a regular RAID setup is called 'mirror' in ZFS (which makes way more sense

with 2 drives than RAIDZ1, which I think needs a minimum of three drives anyway).

RAIDZ1 is similar to what is called RAID5 in a regular RAID environment.

And yes, adding another mirror to your existing pool would work fine.

Most of all, I like doing projects like this, which is why I want to do it myself . Plus doing it myself greatly increases the price/performance ratio, since I can build a NAS with more RAM, more storage and a better CPU for less money than Qnap and Synology. They have a very ancient windows server with 500GB at the moment, so it will be pretty hard to disappoint them anyway ^^.

 

Thanks everyone!!

Hehe, I get that, have felt that itch myself on occasion.

[79wjd]

 

I'm not quite sure I understand what you mean, but if you do RAID0 you always stripe

across different drives, that's the nature of that RAID mode (well, at least as far

as I have been able to find out).

If you do a RAID0 setup (so, a zpool consisting of single-drive vdevs without reduncany)

you can't lose any drives without losing your pool.

If I've misunderstood your intentions, feel free to clarify.

I'm considering setting up my three 4tb drives in such a way as to use all 12tb of storage rather than just 8 if I did Raid 5. But I'd also like to minimize potential losses and not stripe data across the three drives. I.e. I'd like the three drives to be setup so they look like one drive (with total storage of 12tb), but data would only be written to one drive at a time and data would only get written to a second/third drive if there wasn't enough room on the first drive (thus if one drive fails then the other two would remain intact). 

[alpenwasser]

I'm considering setting up my three 4tb drives in such a way as to use all 12tb of storage rather than just 8 if I did Raid 5. But I'd also like to minimize potential losses and not stripe data across the three drives. I.e. I'd like the three drives to be setup so they look like one drive (with total storage of 12tb), but data would only be written to one drive at a time and data would only get written to a second/third drive if there wasn't enough room on the first drive (thus if one drive fails then the other two would remain intact).

Ah, right. As far as I'm aware, ZFS doesn't allow that, no. There might be other storage

software which allows for this, but I've never done it myself.

[martward]

I'm considering setting up my three 4tb drives in such a way as to use all 12tb of storage rather than just 8 if I did Raid 5. But I'd also like to minimize potential losses and not stripe data across the three drives. I.e. I'd like the three drives to be setup so they look like one drive (with total storage of 12tb), but data would only be written to one drive at a time and data would only get written to a second/third drive if there wasn't enough room on the first drive (thus if one drive fails then the other two would remain intact). 

That is basically JBOD right? So you don't have redundancy and you don't have any improvement in speed. I don't know if FreeNAS (if that is what you want to use) waits for one drive to be full before writing to a new drive.

 

 

 

Just a small note: RAID1 is not RAIDZ1, as I mentioned in the post above.

RAID1 in a regular RAID setup is called 'mirror' in ZFS (which makes way more sense

with 2 drives than RAIDZ1, which I think needs a minimum of three drives anyway).

RAIDZ1 is similar to what is called RAID5 in a regular RAID environment.

Yeah I meant RAID1 there, not RAIDZ1. I don't know who came up with those names, but that's seriously confusing. I mean they could've just named it RAIDZ5 and everybody would have understood it...

[alpenwasser]

That is basically JBOD right? So you don't have redundancy and you don't have any improvement in speed. I don't know if FreeNAS (if that is what you want to use) waits for one drive to be full before writing to a new drive.

 

It's JBOD with the addon of having all three drives only visible as one.

Oh, @djdwosk97: unRAID /might/ allow this, not totally sure. I think I read

something about it somewhere at some point.

 

Yeah I meant RAID1 there, not RAIDZ1. I don't know who came up with those names, but that's seriously confusing. I mean they could've just named it RAIDZ5 and everybody would have understood it...

Well it's not like RAID5 is logically named, it's just a hisorical number as far as I

know, the ZFS guys just named the RAIDZ* stuff after the number of parity drives, so

there's some logic behind both approaches. But yeah, it doesn't exactly help to avoid

confusion.