Sanity check on my ZFS commands before I set up my storage server

[18358414]

I'm about to set up a ZFS storage server (zfsonlinux) and just want some feedback/sanity checking on the commands I'm about to run to make sure I don't dump everything in the garbage.  My configuration is as follows:

 

2x 2TB hard drives (Empty /dev/sda, and /dev/sdb which currently has my data on an EXT4 partition at 68% capacity)

2x 1TB hard drives (Empty /dev/sdc and /dev/sdd which currently has my data on an EXT4 partition at 97% capacity)

 

What I plan to do is create a mirrored ZFS pool with the 2TB drives mirroring each other and the 1TB drives mirroring each other for a total pool size of 3TB.  This is for a backup server which syncs with backblaze weekly so I'm not particularly worried about raw performance. 

 

The commands I intend to run to create this are as follows:
 

#create 3TB zfs

zpool create tank /dev/sda
zpool add tank /dev/sdc
zfs create tank/backup

#copy files to zfs (they don't overlap, so additional directories are not needed)
rsync -avr /files/from/sdb/ /tank/backup/
rsync -avr /files/from/sdd/ /tank/backup/

#create redundancy and hope nothing fails in the meantime
zpool attach tank /dev/sda /dev/sdb
zpool attach tank /dev/sdd /dev/sdc

 

Aside from the period of time where I'm running without redundancy, does this look like it'll do what I'm looking for?  After everything's finished, I plan to run a quick export/import so that I'm using physical device ids instead of /dev/sd*. 

 

Thanks,

[Nick7]

There's several things you are doing wrong.

First - you cannot modify disks/vdevs inside zpool.

What you need to do in create zfs pool with mirrored devices. To achieve this in zfs you create mirror between disk and sparse file, and fail that file - so later you can replace it with real device.

Second, it's always recommended to use partitions.

Third, never use /dev/sdX names in zfs pool! Use by partition uuid.

 

From top of my head, you should do something like:

1) Create 2TB and 1Tb sparse files:

# dd if=/dev/zero of=/tmp/sparse_file1 bs=1 count=0 seek=2T

# dd if=/dev/zero of=/tmp/sparse_file2 bs=1 count=0 seek=1T

 

2) Create partitions on disk. Basically, only one partition needed, so all space goes into it.

3) Find out partition uuids with 'blkid' command (check PARTUUID)

4) Create zpool. Use ashift 12!

# zpool create -o ashift=12 tank mirror /dev/disk/by-partuuid/AAAAAA /tmp/sparse_file1 mirror /dev/disk/by-partuuid/BBBBBB /tmp/sparse_file2

4) Fail (offline) sparse files

# zpool offline tank /tmp/sparse_file1 /tmp/sparse_file2

5) Now you can use 'zfs create' to add stuff, etc.. copy all you need

 

When all is done with copying, you need to recreate mirror

6) On drives you want to add, again, add partition, check PARTUUID

7) Replace sparse files with proper devices

# zpool replace tank /tmp/sparse_file1 /dev/disk/by-partuuid/CCCCCCC

# zpool replace tank /tmp/sparse_file2 /dev/disk/by-partuuid/DDDDDDD

 

And that's it.

[18358414]

4 hours ago, Nick7 said:

First - you cannot modify disks/vdevs inside zpool.

Strange, if I create a test zpool using file devices, this works; Oracle's documentation of zfs shows you can attach a disk to a zpool to create a mirrored ZFS storage pool from a previously nonredundant pool (https://docs.oracle.com/cd/E19253-01/819-5461/gcfhe/index.html).  You can't do it with raidz, but you can with mirror. I believe that's what `zpool attach` does according to the documentation.

 

One of my hard drives is inaccessible at the moment, so I  was thinking of modifying this to create a 2TB ZFS and then later add the 1TB drives to expand capacity instead.  That also works using test files from what I've seen thus far (see also: https://docs.oracle.com/cd/E19253-01/819-5461/gazgw/index.html; however it will complain about me adding a non-redundant disk to a redundant array so '-f' may be needed).  So if I do `zpool create tank /some/file`, `zfs create tank/backup`, then `zpool add tank /some/other/file` then my ZFS tank capacity increases, so I'm not sure why you say it doesn't work.  Isn't that what `zpool add [...]` does?

 

4 hours ago, Nick7 said:

Third, never use /dev/sdX names in zfs pool! Use by partition uuid.

 

10 hours ago, Yuri Fury said:

I plan to run a quick export/import so that I'm using physical device ids instead of /dev/sd*

I don't have copy/paste, and I'm dyslexic.  Punching in the UUIDs by hand is not something that's going to happen (and if I try, there's a very significant risk of screwing it up).  The export/import trick should pick up the UUIDs automatically via `zfs export tank` then `zfs import -d /dev/disk/by-partuuid/ tank`

 

I'll build the GPT partitions, like you said, but I can't see why I wouldn't be able to expand a zfs pool.  Thanks for the tip on using 'ashift', but my physical device sector size is 4096 bytes, so wouldn't it be ashift=12?

[Nick7]

Ah, cool... that actually does work, tested it. I was always working with RAIDZ, so for that it did not work (attaching mirror vdev).

export/import also works for using UUID's, but it's important to do it.

ashift=12 is for 4k disks. Often it's on by default, but on some occasions it can fail - so it's safer to specify it while creating.

 

Good luck, and be careful not to overwrite wrong disks!

[Electronics Wizardy]

6 hours ago, Nick7 said:

What you need to do in create zfs pool with mirrored devices. To achieve this in zfs you create mirror between disk and sparse file, and fail that file - so later you can replace it with real device.

You can totally make single drive into mirrors, ive done it many times before

 

6 hours ago, Nick7 said:

Second, it's always recommended to use partitions.

No, id use the full disk, zfs will create a partition for its self, no reson to

 

2 hours ago, Yuri Fury said:

'll build the GPT partitions, like you said, but I can't see why I wouldn't be able to expand a zfs pool.  Thanks for the tip on using 'ashift', but my physical device sector size is 4096 bytes, so wouldn't it be ashift=12?

ZFS will make GPT partition for you, use the full disks here

 

2 hours ago, Yuri Fury said:

I don't have copy/paste, and I'm dyslexic.  Punching in the UUIDs by hand is not something that's going to happen (and if I try, there's a very significant risk of screwing it up).  The export/import trick should pick up the UUIDs automatically via `zfs export tank` then `zfs import -d /dev/disk/by-partuuid/ tank`

Can't you use copy paste over ssh? But ye thta will work when you import/export.