Zpool Drive Replace problems.

[Bmoney]

I just had to replace 2 drives in my zpool. 
Running 2 raidz2 vdevs. with 24 drives. 1 drive form each vdev. 

As before I pulled the dead drive. Added the new one re-silvering started but it been 3 days now. 

The status won't scan past 227M out of 52.T. The pool is online and the drive have no errors. But I keep running into this scan not progressing. 

[Jarsky]

Considering each of your vdev's is 12 disks at I assume 3TB, if theyre reasonably full it could take some time for it to  complete.

 

Can you run these commands and paste the outputs (maybe pastebin the events)

 

zpool status -v
zpool events -v
zpool get all | grep feature

 

[Bmoney]

Apr 13 2020 18:22:31.533065015 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Leyline"
        pool_guid = 0x300dadf56802a59b
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "GalacticLeyline"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0xd0b39
        history_time = 0x5e94f437
        time = 0x5e94f437 0x1fc5ed37
        eid = 0x14a9

Apr 13 2020 18:22:31.541065023 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Leyline"
        pool_guid = 0x300dadf56802a59b
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "GalacticLeyline"
        history_internal_str = "errors=0"
        history_internal_name = "starting deferred resilver"
        history_txg = 0xd0b39
        history_time = 0x5e94f437
        time = 0x5e94f437 0x203fff3f
        eid = 0x14aa

Apr 13 2020 18:22:31.541065023 sysevent.fs.zfs.resilver_start
        version = 0x0
        class = "sysevent.fs.zfs.resilver_start"
        pool = "Leyline"
        pool_guid = 0x300dadf56802a59b
        pool_state = 0x0
        pool_context = 0x0
        time = 0x5e94f437 0x203fff3f
        eid = 0x14ab

Apr 13 2020 18:22:31.541065023 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Leyline"
        pool_guid = 0x300dadf56802a59b
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "GalacticLeyline"
        history_internal_str = "func=2 mintxg=3 maxtxg=854594"
        history_internal_name = "scan setup"
        history_txg = 0xd0b39
        history_time = 0x5e94f437
        time = 0x5e94f437 0x203fff3f
        eid = 0x14ac

Apr 13 2020 18:22:36.825070802 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Leyline"
        pool_guid = 0x300dadf56802a59b
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "GalacticLeyline"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0xd0b3a
        history_time = 0x5e94f43c
        time = 0x5e94f43c 0x312d94d2
        eid = 0x14ad

[Bmoney]

Leyline  feature@async_destroy          enabled                        local
Leyline  feature@empty_bpobj            enabled                        local
Leyline  feature@lz4_compress           active                         local
Leyline  feature@multi_vdev_crash_dump  enabled                        local
Leyline  feature@spacemap_histogram     active                         local
Leyline  feature@enabled_txg            active                         local
Leyline  feature@hole_birth             active                         local
Leyline  feature@extensible_dataset     active                         local
Leyline  feature@embedded_data          active                         local
Leyline  feature@bookmarks              enabled                        local
Leyline  feature@filesystem_limits      enabled                        local
Leyline  feature@large_blocks           enabled                        local
Leyline  feature@large_dnode            enabled                        local
Leyline  feature@sha512                 enabled                        local
Leyline  feature@skein                  enabled                        local
Leyline  feature@edonr                  enabled                        local
Leyline  feature@userobj_accounting     active                         local
 

[Jarsky]

On 4/14/2020 at 11:24 AM, Bmoney said:

Apr 13 2020 18:22:31.533065015 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Leyline"
        pool_guid = 0x300dadf56802a59b
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "GalacticLeyline"
        history_internal_str = "errors=0"
        history_internal_name = "scan aborted, restarting"
        history_txg = 0xd0b39
        history_time = 0x5e94f437
        time = 0x5e94f437 0x1fc5ed37
        eid = 0x14a9

Apr 13 2020 18:22:31.541065023 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "Leyline"
        pool_guid = 0x300dadf56802a59b
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "GalacticLeyline"
        history_internal_str = "errors=0"
        history_internal_name = "starting deferred resilver"
        history_txg = 0xd0b39
        history_time = 0x5e94f437
        time = 0x5e94f437 0x203fff3f
        eid = 0x14aa

 

It looks like you're stuck in a loop of resilver restarting. It looks like this is related to the resilver_defer feature per this bug: https://github.com/openzfs/zfs/pull/9338

 

It looks like you only pasted a portion of the output for 'zpool get all' as there should be a stack more features unless you have a very old ZFS version possibly. But if resilver_defer is disabled, you could try enabling the resilver_defer feature to see if this resolves it. 

zpool set feature@resilver_defer=enabled zfspool

 

I'm using ZFS 0.8.3-1 on Linux and its enabled by default. 

[Bmoney]

After a few reboots. Got it working. Had to reboot then online both replacement drives.