Levent

54 minutes ago, Levent said:

So lets say, my 500G disk vdev had 3 disk failure and are in need of a resilvering, does that mean my I can use other array to recover entirety of the data?

RAIDZ3 can sustain a three disk failure, so in this case you would be fine.

But what I think you're asking: no, each VDEV only contains its own data (which is a subset of the full stored data); if you lose a VDEV, you lose the whole pool.

57 minutes ago, Levent said:

I got multiple capacity vdevs in the same pool.

This is maybe the one real advantage to pooling VDEVs (I used to do it when I was getting into FreeNAS/TrueNAS/ZFS).

1 hour ago, Levent said:

I got multiple capacity vdevs in the same pool.

AFAIK you shouldn't do that, they get RAID0'd so you'd essentially lose all the extra capacity of the first, like RAID0ing a 500GB drive and a 1TB drive, you get 1TB usable and waste the other 500GB.

1 hour ago, Levent said:

So lets say, my 500G disk vdev had 3 disk failure and are in need of a resilvering, does that mean my I can use other array to recover entirety of the data?

No, when you have a pool with multiple vdevs it's still "one volume", you don't get to choose where things go, and all vdevs have to be functional. E.g. if you have a pool with 3 raidz1 vdevs, one vdev loses 2 disks, the entire pool is gone.

Seems you'd rather want to do 2 separate pools. 

3 minutes ago, Kilrah said:

AFAIK you shouldn't do that, you'd lose all the extra capacity of the first.

ZFS allows mismatched VDEVs. The VDEVs themselves should be comprised of the same size drives for that reason, but that restriction doesn't apply to pooling VDEVs.

Older versions of FreeNAS/TrueNAS would let you pool a single drive into any array, which was a one-way operation. I believe they've "removed" that "feature" since...