Personal tools
You are here: Home Issue tracker Resilvering interrupted when a vdev from another pool was disconnected

#12 — Resilvering interrupted when a vdev from another pool was disconnected

State Resolved
Version:
Area Functionality
Issue type Bug
Severity Medium
Submitted by (anonymous)
Submitted on Dec 22, 2009
Responsible Seth Heeren
Target release: 0.6.9
Return to tracker
Last modified on May 22, 2010 by Seth Heeren
Resilvering process was interrupted (1 permanent error) after I unmount and physically disconnected a disk that was part of another pool.
Steps to reproduce:
zpool create mypool /dev/disk/by-id/usb-WD-part1 // disk 1

zpool create tank /dev/disk/by-id/usb-SAMSUNG1-part1 // disk 2
zpool attach tank /dev/disk/by-id/usb-SAMSUNG1-part1 /dev/disk/by-id/usb-SAMSUNG2-part1 // disk3: new mirror attached

/*Resilvering operation began .. While a background resilvering was in progress, I issued these commands:*/

zfs unmount mypool
zfs export mypool // issued by mistake, but everything was fine till here

now i have physically removed (switched power off) external disk 1 (/dev/disk/by-id/usb-WD-part1) and lights on my both samsungs stopped blinking.

After issuing: zfs status, I got:
1 Permanent error on tank. After I rebooted pc, resilvering operation restarted from beginning.
Added by (anonymous) on Dec 22, 2009 02:44 AM
Can someone establish upstream behaviour?

I.e. check what opensolaris does in such a case?
Added by (anonymous) on Dec 22, 2009 07:04 AM
Additional info, that I forgot to add to my bug report:

My environment:
Gentoo 2010,
running gentoo-sources kernel version gentoo-sources-2.6.31-r8
with sys-fs/fuse-2.8.1
and latest (zfs-fuse 0.6.0) manually build (not in portage yet - latest version in portage is 0.5.0)
zpool pool version is 16

Added by (anonymous) on Dec 22, 2009 08:23 PM
i just played with a OSol VM:

Right after install adding a pool

zpool create -O compression=on tank raidz2 c9t{0,1,2,3}d0 leads to

    NAME STATE READ WRITE CKSUM
    rpool ONLINE 0 0 0
      c7d0s0 ONLINE 0 0 0

    NAME STATE READ WRITE CKSUM
    tank ONLINE 0 0 0
      raidz2 ONLINE 0 0 0
        c9t0d0 ONLINE 0 0 0
        c9t1d0 ONLINE 0 0 0
        c9t2d0 ONLINE 0 0 3
        c9t3d0 ONLINE 0 0 0

Then I reboot adding another disk and attach it to the rpool (after format -e and other hairy stuff... I don't grok OSol)

zpool attach rpool c7d0s0 c9t7d0p1 # this starts a lengthy resilver

I then tried to reproduce the problem situation by tinkering with *the other* pool (tank):

zpool offline tank c9t1d0
zpool offline tank c9t2d0
zpool export tank
zpool import tank
zpool offline rpool c9t7d0p1 # pauses the resilver but remembers the state
zpool export tank
zpool online rpoot c9t7d0p1 # no matter what I do, every thing is peaches

So it really seems that if we can reproduce the stopped resilver on zfs-fuse there is a difference iwth upstream.




Added by (anonymous) on Dec 23, 2009 02:19 AM
1. I did this with 3 external devices (2 in a mirrored pool: A and B, 1 alone in another pool: C)
2. I physically disconnected C, while resilvering was in progress on A&B and after I unmounted zfs filesystem on C.

Where can I find a zfs log? What can I do, to give you more info/feedback?

Yesterday I was playing a lot with zfs,
I found other problems: when resilver operation is in progress, my zfs pool became very very sensitive. A device removed from system (that is part of at least one pool) will interrupt resilvering. Also, I put that C device on share (using samba/cifs) and put synchronization in work (my winxp laptop was sending data through samba to my PC share, which was storing that data on device C), after 10 mins, I got more than 300 errors in column WRITE on device C. Devices A&B were also showing some errors on all three columns: READ, WRITE, CHKSUM. Soon resilvering was interrupted again. Command zpool iostat -v free has frozen my terminal, also "zpool unmount samsung" didn't was stuck. After a reboot, my pool was in DEGRADED state.
Added by Seth Heeren on Dec 23, 2009 07:13 AM
(PS the solaris behaviour was my post)

Could you look in /var/log/messages (or syslog) and see whether there is anything infteresting? While doing my testing I get strange behaviour on out-of-memory:

Dec 23 13:02:46 karmic zfs-fuse: put_nvlist: out of memory 5248 > 4096
Dec 23 13:04:04 karmic zfs-fuse: put_nvlist: out of memory 5248 > 4096
Dec 23 13:07:09 karmic zfs-fuse: put_nvlist: out of memory 5248 > 4096
Dec 23 13:08:34 karmic zfs-fuse: last message repeated 205 times

The condition leads to request lockups (zfs-fuse remains responsive though)

It would be useful to know what is in your logs anyway :)
Added by (anonymous) on Dec 24, 2009 03:52 AM
My log from 21.12.2009-23.12.2009: cat /var/log/messages | grep zfs

Dec 21 20:48:42 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 21 20:48:42 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 21 20:48:42 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 21 20:48:42 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 21 20:48:42 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 21 21:25:12 hp zfs-fuse: enabling fuse big_writes
Dec 21 21:25:12 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 21 21:29:57 hp zfs-fuse: enabling fuse big_writes
Dec 21 21:29:57 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 21 22:58:44 hp zfs-fuse: enabling fuse big_writes
Dec 21 22:58:44 hp zfs-fuse: mount options: fsname=wd1000,allow_other,suid,dev,big_writes
Dec 22 06:35:49 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 22 06:35:49 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 22 06:35:49 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 22 06:35:49 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 22 06:35:49 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 22 06:35:53 hp zfs-fuse: enabling fuse big_writes
Dec 22 06:35:53 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 22 19:21:15 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 22 19:21:15 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 22 19:21:15 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 22 19:21:15 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 22 19:21:15 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 22 19:21:19 hp zfs-fuse: enabling fuse big_writes
Dec 22 19:21:19 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 22 19:33:09 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 22 19:33:09 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 22 19:33:09 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 22 19:33:09 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 22 19:33:09 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 22 19:33:13 hp zfs-fuse: enabling fuse big_writes
Dec 22 19:33:13 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 22 19:52:34 hp zfs-fuse: enabling fuse big_writes
Dec 22 19:52:34 hp zfs-fuse: mount options: fsname=wd1000,allow_other,suid,dev,big_writes
Dec 22 22:11:17 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 22 22:11:17 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 22 22:11:17 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 22 22:11:18 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 22 22:11:18 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 22 22:11:23 hp zfs-fuse: enabling fuse big_writes
Dec 22 22:11:23 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 22 22:11:23 hp zfs-fuse: enabling fuse big_writes
Dec 22 22:11:23 hp zfs-fuse: mount options: fsname=wd1000,allow_other,suid,dev,big_writes
Dec 22 22:39:58 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 22 22:39:58 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 22 22:39:58 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 22 22:39:59 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 22 22:39:59 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 22 22:40:04 hp zfs-fuse: enabling fuse big_writes
Dec 22 22:40:04 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 22 22:40:04 hp zfs-fuse: enabling fuse big_writes
Dec 22 22:40:04 hp zfs-fuse: mount options: fsname=wd1000,allow_other,suid,dev,big_writes
Dec 22 23:35:57 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 22 23:35:57 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 22 23:35:57 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 22 23:35:57 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 22 23:35:57 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 22 23:36:01 hp zfs-fuse: enabling fuse big_writes
Dec 22 23:36:01 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 22 23:38:17 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 22 23:38:17 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 22 23:38:17 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 22 23:38:17 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 22 23:38:17 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 22 23:38:21 hp zfs-fuse: enabling fuse big_writes
Dec 22 23:38:21 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 22 23:43:36 hp zfs-fuse: enabling fuse big_writes
Dec 22 23:43:36 hp zfs-fuse: mount options: fsname=wd1000,allow_other,suid,dev,big_writes
Dec 23 06:41:47 hp zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Dec 23 06:41:47 hp zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Dec 23 06:41:47 hp zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Dec 23 06:41:48 hp zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Dec 23 06:41:48 hp zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Dec 23 06:41:53 hp zfs-fuse: enabling fuse big_writes
Dec 23 06:41:53 hp zfs-fuse: mount options: fsname=samsung,allow_other,suid,dev,big_writes
Dec 23 06:41:53 hp zfs-fuse: enabling fuse big_writes
Dec 23 06:41:53 hp zfs-fuse: mount options: fsname=wd1000,allow_other,suid,dev,big_writes
Dec 23 17:17:09 hp zfs-fuse: put_nvlist: error Bad address on xcopyout
Added by Seth Heeren on Jan 22, 2010 03:59 PM
Issue state: unconfirmedopen
I haven't totally forgotten, you know :)

I have been able to reproduce the put_nvlist: error Bad address on xcopyout message
by continuously writing to a pool and doing 'zfs list' in another terminal.

The message is generated in the postprocessing of the IOCTL function zfs_ioc_pool_configs. The userland function was main -> zfs_do_list -> zfs_for_each -> zfs_iter_root -> namespace_reload (lib/libzfs/libzfs_config.c:147)

Hope this helps anyone finding more info, I'd personally be interested :)

My current hunch is that
(a) the namespace_reload thingie is (among other things) there to isolate changes in unrelated pools/fs-es
(b) if it fails, the scrub is restarted as if there was a change in the current namespace config's generation

This is apt guess-work so no warranties...
Added by Seth Heeren on May 22, 2010 01:41 PM
Issue state: openresolved
Target release: None0.6.9
Responsible manager: (UNASSIGNED)sgheeren
with posterior knowledge:

"put_nvlist: error Bad address on xcopyout" points to issue #22 (unstable under load) which happens to coincide with your statement "pool becomes very sensitive" when scrubbing.

I strongly suggest this is no longer an issue with 0.6.9_beta. Please retest accordingly.

Closing for now