Tracker log

An overview of recent activity in the tracker.

Seth Heeren added a new response to »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid «:

Ok, the crash seems the most tangible handle now. By any chance, did that leave a core file? Check ulimit -c (init script?) for more info.

I suppose the core file could be uploaded to the wiki (see notes on zfs-fuse.net/issues). Otherwise, I could set up an ftp location ad hoc

The pool being harmed after a crash corresponds with the fact that disks are not syncing. I don't have to tell you, obviously, that using zfs-fuse on this type of configuration is not recommendable. OTOH the net result is not unlike running with zil_disable==1 (see ZFS Evil Tuning Guide[1])

It will be most interesting to find out what exactly is confusing the scrub. I feel that the same confusion might trigger the crash. This will have to be some essential type of confusion, which I cannot reproduce[2]. That is why I still have (in the back of my head) the thought that perhaps logical volumes are overlapping blocks in some way, or something is wrong with the major/minor numbers of the vdevs as seen by zfs-fuse, to generate a few random long shots.

[1]http://www.solarisinternals.com/[…]/ZFS_Evil_Tuning_Guide
[2]Oh, my tests are on Lucid Ubuntu 10.04.1 LTS, 2.6.32-24-generic-pae i686, 8GB

Added 21 hour(s) ago.

drwatson added a new response to »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid «:

hi, to answer your question: i upgraded after the cheese was grated :-D
after the upgrade to 0.6.9 I created another test pool.
I destroyed the existing zfs_temp01 pool, i dded 20g from /dev/zero and teed it through md5sum. then I md5summed it reading it from the newly created pool. without errors. Then zfs-fuse completely crashed. after a reboot I tried to hash the "zero" file again and now I get this:

root@host:~# zpool status -v zfs_temp02
  pool: zfs_temp02
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: none requested
config:

    NAME STATE READ WRITE CKSUM
    zfs_temp02 ONLINE 0 0 2
      mapper/lv_host_TEMP01_01_crypt ONLINE 0 0 4

errors: Permanent errors have been detected in the following files:

        /zfs_temp02/temp/zero.zero
root@host:~#

Added 1 day(s) ago.

Seth Heeren added a new response to »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid «:

> between 12 and 14hrs I upgraded to 0.6.9 and created another test pool.
then when I tried to write to it, zfs-fuse crashed.

Was that _after_ it became Swiss cheese? In that case it is irrelevant to my diagnostics. However, you will probably want to make sure you get your data safe as quickly as possible.

Added 1 day(s) ago.

drwatson added a new response to »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid «:

thanks for your response. I hope I'm not wasting your time with an error that affects only my installation.

here the syslog:

Jul 28 09:40:09 host zfs-fuse: put_nvlist: out of memory 5440 > 4096
Jul 28 09:52:59 host zfs-fuse: put_nvlist: out of memory 5440 > 4096
Jul 28 09:53:00 host zfs-fuse: put_nvlist: out of memory 1704 > 1696
Jul 28 09:53:29 host zfs-fuse: enabling fuse big_writes
Jul 28 09:53:29 host zfs-fuse: mount options: fsname=zfs_temp01/deleteme,allow_other,suid,dev,big_writes
Jul 28 10:20:39 host zfs-fuse: put_nvlist: out of memory 5440 > 4096
Jul 28 10:21:02 host zfs-fuse: put_nvlist: out of memory 5440 > 4096
Jul 28 10:21:02 host zfs-fuse: put_nvlist: out of memory 1704 > 1696
Jul 28 11:20:29 host zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Jul 28 11:20:29 host zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Jul 28 11:20:29 host zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Jul 28 11:20:29 host zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Jul 28 11:20:29 host zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Jul 28 11:20:41 host zfs-fuse: put_nvlist: out of memory 5376 > 4096
Jul 28 11:20:46 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_A_02_crypt'. Data on pool 'zfs_host_vms01' may be lost if power fails. No further warnings will be given.
Jul 28 11:20:46 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_A_01_crypt'. Data on pool 'zfs_host_vms01' may be lost if power fails. No further warnings will be given.
Jul 28 11:20:47 host zfs-fuse: enabling fuse big_writes
Jul 28 11:20:47 host zfs-fuse: mount options: fsname=zfs_host_vms01,allow_other,suid,dev,big_writes
Jul 28 11:20:49 host zfs-fuse: enabling fuse big_writes
Jul 28 11:20:49 host zfs-fuse: mount options: fsname=zfs_host_vms01/banking,allow_other,suid,dev,big_writes
Jul 28 11:20:51 host zfs-fuse: enabling fuse big_writes
Jul 28 11:20:51 host zfs-fuse: mount options: fsname=zfs_host_vms01/u104base,allow_other,suid,dev,big_writes
Jul 28 11:20:52 host zfs-fuse: enabling fuse big_writes
Jul 28 11:20:52 host zfs-fuse: mount options: fsname=zfs_host_vms01/u904_zfs_testserver,allow_other,suid,dev,big_writes
Jul 28 11:20:55 host zfs-fuse: enabling fuse big_writes
Jul 28 11:20:55 host zfs-fuse: mount options: fsname=zfs_host_vms01/u904_zfs_testserver/datastorage,allow_other,suid,dev,big_writes
Jul 28 11:20:57 host zfs-fuse: enabling fuse big_writes
Jul 28 11:20:57 host zfs-fuse: mount options: fsname=zfs_host_vms01/ubuntu104,allow_other,suid,dev,big_writes
Jul 28 11:20:59 host zfs-fuse: enabling fuse big_writes
Jul 28 11:20:59 host zfs-fuse: mount options: fsname=zfs_temp01,allow_other,suid,dev,big_writes
Jul 28 11:21:00 host zfs-fuse: enabling fuse big_writes
Jul 28 11:21:00 host zfs-fuse: mount options: fsname=zfs_temp01/deleteme,allow_other,suid,dev,big_writes
Jul 28 11:21:09 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_04_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 11:21:09 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_05_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 11:21:09 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_01_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 11:21:09 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_02_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 11:21:09 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_03_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 11:29:06 host zfs-fuse: put_nvlist: out of memory 5440 > 4096
Jul 28 11:55:40 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/myramdisk'. Data on pool 'ramdisk' may be lost if power fails. No further warnings will be given.
Jul 28 11:55:42 host zfs-fuse: enabling fuse big_writes
Jul 28 11:55:42 host zfs-fuse: mount options: fsname=ramdisk,allow_other,suid,dev,big_writes
Jul 28 11:55:45 host zfs-fuse: put_nvlist: out of memory 6316 > 4096
Jul 28 12:03:42 host zfs-fuse: put_nvlist: out of memory 6316 > 4096
Jul 28 12:13:18 host zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Jul 28 12:13:18 host zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Jul 28 12:13:18 host zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Jul 28 12:13:18 host zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Jul 28 12:13:18 host zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Jul 28 12:13:26 host zfs-fuse: put_nvlist: out of memory 5440 > 4096
Jul 28 12:13:27 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_A_02_crypt'. Data on pool 'zfs_host_vms01' may be lost if power fails. No further warnings will be given.
Jul 28 12:13:27 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_A_01_crypt'. Data on pool 'zfs_host_vms01' may be lost if power fails. No further warnings will be given.
Jul 28 12:13:28 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:28 host zfs-fuse: mount options: fsname=zfs_host_vms01,allow_other,suid,dev,big_writes
Jul 28 12:13:29 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:29 host zfs-fuse: mount options: fsname=zfs_host_vms01/banking,allow_other,suid,dev,big_writes
Jul 28 12:13:29 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:29 host zfs-fuse: mount options: fsname=zfs_host_vms01/u104base,allow_other,suid,dev,big_writes
Jul 28 12:13:30 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:30 host zfs-fuse: mount options: fsname=zfs_host_vms01/u904_zfs_testserver,allow_other,suid,dev,big_writes
Jul 28 12:13:31 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:31 host zfs-fuse: mount options: fsname=zfs_host_vms01/u904_zfs_testserver/datastorage,allow_other,suid,dev,big_writes
Jul 28 12:13:32 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:32 host zfs-fuse: mount options: fsname=zfs_host_vms01/ubuntu104,allow_other,suid,dev,big_writes
Jul 28 12:13:32 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:32 host zfs-fuse: mount options: fsname=zfs_temp01,allow_other,suid,dev,big_writes
Jul 28 12:13:34 host zfs-fuse: enabling fuse big_writes
Jul 28 12:13:34 host zfs-fuse: mount options: fsname=zfs_temp01/deleteme,allow_other,suid,dev,big_writes
Jul 28 12:13:56 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_04_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 12:13:56 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_05_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 12:13:56 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_01_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 12:13:56 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_03_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 12:13:56 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_02_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 12:15:33 host zfs-fuse: put_nvlist: out of memory 5440 > 4096
Jul 28 14:25:12 host zfs-fuse: initial max_map_count 65530
Jul 28 14:25:12 host zfs-fuse: ARC caching: maximum ARC size: 100 MiB
Jul 28 14:25:12 host zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Jul 28 14:25:12 host zfs-fuse: ARC setup: max ARC size set to 104857600 bytes
Jul 28 14:25:17 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_A_02_crypt'. Data on pool 'zfs_host_vms01' may be lost if power fails. No further warnings will be given.
Jul 28 14:25:17 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_A_01_crypt'. Data on pool 'zfs_host_vms01' may be lost if power fails. No further warnings will be given.
Jul 28 14:25:53 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_05_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 14:25:53 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_01_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 14:25:53 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_02_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 14:25:53 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_03_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 14:25:53 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_04_crypt'. Data on pool 'zfs_temp01' may be lost if power fails. No further warnings will be given.
Jul 28 14:39:36 host zfs-fuse: WARNING: Failed to flush write cache on device '/dev/mapper/lv_host_TEMP01_01_crypt'. Data on pool 'zfs_temp02' may be lost if power fails. No further warnings will be given.
Jul 28 14:39:36 host zfs-fuse: !created version 23 pool zfs_temp02 using 23
Jul 28 14:46:35 host zfs-fuse: ERROR: buffer modified while frozen!
Jul 28 14:46:54 host zfs-fuse: initial max_map_count 65530
Jul 28 14:46:54 host zfs-fuse: ARC caching: maximum ARC size: 100 MiB
Jul 28 14:46:54 host zfs-fuse: WARNING: /var/run/zfs-fuse.pid already exists; aborting.

between 12 and 14hrs I upgraded to 0.6.9 and created another test pool.
then when I tried to write to it, zfs-fuse crashed.

In my case it doesn't matter whether the pool is on an lvs volume (newly created zfs pools) or on a regular partition (migrated older version zfs pools) (both using luks encryption)

root@host:~# lvs
  LV VG Attr LSize Origin Snap% Move Log Copy% Convert
  lv_chome vg_croot -wi-ao 16.76g
  lv_croot vg_croot -wi-ao 11.18g
  lv_cswap vg_croot -wi-ao 18.62g
  lv_host_A_01 vg_host_A -wi-ao 100.00g
  lv_host_A_02 vg_host_A -wi-ao 100.00g
  lv_host_A_03 vg_host_A -wi-ao 100.00g
  lv_host_A_04 vg_host_A -wi-ao 100.00g
  lv_host_TEMP01_01 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_02 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_03 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_04 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_05 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_06 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_07 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_08 vg_host_TEMP01 -wi-ao 200.00g
  lv_host_TEMP01_09 vg_host_TEMP01 -wi-ao 200.00g




root@host:~# vgdisplay -v
    Finding all volume groups
    Finding volume group "vg_host_TEMP01"
  --- Volume group ---
  VG Name vg_host_TEMP01
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 10
  VG Access read/write
  VG Status resizable
  MAX LV 256
  Cur LV 9
  Open LV 9
  Max PV 256
  Cur PV 1
  Act PV 1
  VG Size 1.82 TiB
  PE Size 4.00 MiB
  Total PE 476931
  Alloc PE / Size 460800 / 1.76 TiB
  Free PE / Size 16131 / 63.01 GiB
  VG UUID DMIKX0-QY7l-LuIE-ymWQ-HJC3-nQbI-Cl5zYJ
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_01
  VG Name vg_host_TEMP01
  LV UUID YZPWdF-rxtQ-d5HQ-1UIk-Govq-OXY2-cxf7dn
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:0
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_02
  VG Name vg_host_TEMP01
  LV UUID zA8Kat-fxNk-Qoer-HuMn-l86P-ipWc-deIfs2
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:1
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_03
  VG Name vg_host_TEMP01
  LV UUID TCWTFC-zNkT-Gn45-jndC-l9ki-tn2D-rXRPnc
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:2
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_04
  VG Name vg_host_TEMP01
  LV UUID 0pHCSc-hLut-3ncs-2Iv2-6ce5-n0mb-Hq0mha
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:3
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_05
  VG Name vg_host_TEMP01
  LV UUID rg9KDY-xT2s-YID2-r7Ai-kqfp-yd3K-WoOW1l
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:4
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_06
  VG Name vg_host_TEMP01
  LV UUID J9Ye1f-6oAB-Rsmv-GA9g-nIp3-pmYq-j0Lg8l
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:5
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_07
  VG Name vg_host_TEMP01
  LV UUID Z3Sojg-WwEF-5myN-F7Y2-DHPM-zjKY-N36uN8
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:6
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_08
  VG Name vg_host_TEMP01
  LV UUID 3BtT42-eOp6-eXQ4-3Q4w-bRIh-2fNM-fL8k5f
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:7
   
  --- Logical volume ---
  LV Name /dev/vg_host_TEMP01/lv_host_TEMP01_09
  VG Name vg_host_TEMP01
  LV UUID 8rGiev-v2bB-MFyc-QbiZ-XpHI-xJm6-2JyYt2
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 200.00 GiB
  Current LE 51200
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:8
   
  --- Physical volumes ---
  PV Name /dev/sdb1
  PV UUID n3pmNT-Vbjs-jO8D-y5Q8-Z0tc-AD7K-asKkIn
  PV Status allocatable
  Total PE / Free PE 476931 / 16131
   
    Finding volume group "vg_croot"
  --- Volume group ---
  VG Name vg_croot
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 4
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 3
  Open LV 3
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size 46.56 GiB
  PE Size 4.00 MiB
  Total PE 11920
  Alloc PE / Size 11920 / 46.56 GiB
  Free PE / Size 0 / 0
  VG UUID h7PszD-E0nb-tyxk-KGBk-XcM9-lad2-W9s2PM
   
  --- Logical volume ---
  LV Name /dev/vg_croot/lv_croot
  VG Name vg_croot
  LV UUID H0rHlD-z6tP-3Rn2-wYk3-dFPt-N7gX-vZJ9o3
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 11.18 GiB
  Current LE 2861
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:14
   
  --- Logical volume ---
  LV Name /dev/vg_croot/lv_cswap
  VG Name vg_croot
  LV UUID p3yC6c-AYp3-DjB8-FMFv-9zsh-uz8P-jR5Ekv
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 18.62 GiB
  Current LE 4768
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:15
   
  --- Logical volume ---
  LV Name /dev/vg_croot/lv_chome
  VG Name vg_croot
  LV UUID wfBFkK-3nfE-lgr3-lavP-k5e7-4Cee-K8wrOx
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 16.76 GiB
  Current LE 4291
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:16
   
  --- Physical volumes ---
  PV Name /dev/mapper/sda5_crypt
  PV UUID LsB8Bh-yUrD-2KUk-Ssbo-d9D8-AOaf-EztObd
  PV Status allocatable
  Total PE / Free PE 11920 / 0
   
    Finding volume group "vg_host_A"
  --- Volume group ---
  VG Name vg_host_A
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 5
  VG Access read/write
  VG Status resizable
  MAX LV 256
  Cur LV 4
  Open LV 4
  Max PV 256
  Cur PV 1
  Act PV 1
  VG Size 405.20 GiB
  PE Size 4.00 MiB
  Total PE 103732
  Alloc PE / Size 102400 / 400.00 GiB
  Free PE / Size 1332 / 5.20 GiB
  VG UUID PXKLly-ILxz-vdXQ-QSeV-Vui2-HVpe-v3Le5D
   
  --- Logical volume ---
  LV Name /dev/vg_host_A/lv_host_A_01
  VG Name vg_host_A
  LV UUID 0oZ7FX-gaey-H7Lu-NrK0-393n-MP34-MdlTfR
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 100.00 GiB
  Current LE 25600
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:9
   
  --- Logical volume ---
  LV Name /dev/vg_host_A/lv_host_A_02
  VG Name vg_host_A
  LV UUID NH03id-al2g-WkTj-Rg1i-yEC7-39sz-oEPC9Z
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 100.00 GiB
  Current LE 25600
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:10
   
  --- Logical volume ---
  LV Name /dev/vg_host_A/lv_host_A_03
  VG Name vg_host_A
  LV UUID 2GM0q2-5LVc-8BrE-zxrE-Jz24-Zohz-Qx1nfB
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 100.00 GiB
  Current LE 25600
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:11
   
  --- Logical volume ---
  LV Name /dev/vg_host_A/lv_host_A_04
  VG Name vg_host_A
  LV UUID UJzVIV-HDED-RCwV-uPyd-z86U-Vzn3-OamA0k
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 100.00 GiB
  Current LE 25600
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 251:12
   
  --- Physical volumes ---
  PV Name /dev/sda6
  PV UUID UkZbxe-dK7q-z8nf-Ca8e-WxMQ-5lS4-eCcLPw
  PV Status allocatable
  Total PE / Free PE 103732 / 1332



haven't done anything else with lvm except creating the volumes.
no snapshots or anything.

Added 1 day(s) ago.

Seth Heeren added a new response to »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid «:

Please post results of 'lvs' and possibly 'vgdisplay -v'
so we can rule out any de-activated lvm snapshot/origins ?

Added 1 day(s) ago.

Seth Heeren added a new response to »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid «:

I'm sorry I cannot reproduce. I only left lvm out of the picture for now
There is no difference for me whether I use 0.6.0 (0.6.0+critical20100301-2 a.k.a. git 3de9c7c4f) or 0.6.9 (testing branch for 0.7.0).

Even leaving the tests running for a long time, and simultaneously rewriting the data file(s) did not (ex)pose any problems... [these extended tests were done on 0.6.0 as mentioned above)

Do you reckon lvm could be misconfigured/misbehaving?

Here is what I did:

0. created two sparse files in /tmp/

    ls /tmp/tank_blk/* -ltr
    -rw-r--r-- 1 root root 68719476736 2010-07-28 16:23 /tmp/tank_blk/za1
    -rw-r--r-- 1 root root 68719476736 2010-07-28 16:23 /tmp/tank_blk/za2

1. create loop devices from them
    losetup -f /tmp/tank_blk/za1
    losetup -f /tmp/tank_blk/za2

2. create luks containers in them
    cryptsetup luksFormat /dev/loop0
    cryptsetup luksFormat /dev/loop1

3. open them

    cryptsetup luksOpen /dev/loop0 crypt1
    cryptsetup luksOpen /dev/loop1 crypt2

4. create a mirrored pool

    zpool create -f luks1 mirror /dev/mapper/crypt[12]

5. write a 1 gb file

    dd if=/dev/zero bs=1M count=1024 of=/luks1/test.zeroes
    
6. stress test using
  
    while true; do sleep 10; zpool scrub luks1; done&
    watch zpool status -v

99. teardown
    
    zpool destroy luks1
    <reboot in the future>

Added 1 day(s) ago.

Seth Heeren added a new response to »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid «:

Thanks for the report. This is very creepy behaviour.

Could you give me some instructions on how to create luks encrypted devices, so I can (try to) reproduce this?

Meanwhile, does 'grep -i zfs /var/log/syslog' contain anything alarmy? For starters, I expect that these cryptvols cannot be sync-ed; expect a fair number of 'WARNING: Failed to flush write cache on device '/tmp/tank_blk/za2'. Data on pool 'tank' may be lost if power fails. No further warnings will be given.' lines for this reason. Anything else would be most interesting.

Issue state: unconfirmed → open. Responsible manager: (UNASSIGNED) → sgheeren. Added 1 day(s) ago.

New issue »zpool scrub scrambles data with 0.6.0-1 on Ubuntu 10.04 Lucid « added by drwatson

Hi all,

thank you very much for your efforts in developing zfs-fuse for linux.

I've just set up a new Ubuntu 10.04 Lucid 64 bit system on a system with 16GB DDR3 RAM and a core i5-650 processor.
I ported my harddisks from my old Ubuntu 9.04 system with zfs-fuse 0.6.0+critical20100301-3 and imported the mirrorred pool to my new system without upgrading the pool.
The pools reside on luks-encrypted devices.
Doing reads and writes works on the old pool as well as the new one.

Until I did a scrub. The more I scrub, the more errors it finds.
My first suspicion was the harddisks. They don't report any actual read/write errors.
Then I created a pool on a brandnew 2tb disk.
I dd-ed from /dev/zero to a large 20G file on that pool, while teeing it through md5. The checksum returned was always identical. Until I did a scrub again on that pool. The more I run scrub, the more checksum errors it finds and trying to md5 the file gives me I/O erros.

root@host:~# zpool status -v zfs_temp01
  pool: zfs_temp01
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: scrub completed after 0h14m with 10 errors on Wed Jul 28 12:30:50 2010
config:

    NAME STATE READ WRITE CKSUM
    zfs_temp01 ONLINE 0 0 11
      mapper/lv_host_TEMP01_01_crypt ONLINE 0 0 7
      mapper/lv_host_TEMP01_02_crypt ONLINE 0 0 11
      mapper/lv_host_TEMP01_03_crypt ONLINE 0 0 22
      mapper/lv_host_TEMP01_04_crypt ONLINE 0 0 9
      mapper/lv_host_TEMP01_05_crypt ONLINE 0 0 11

errors: Permanent errors have been detected in the following files:

        /zfs_temp01/loeschmich/leer.zero

Until now I have done a memcheck. I have tried whether the aesni_intel module used by default produces errors. I have created a encrypted 15GB file in ram, reading, writing, reading, checksumming. no errors. I have tried the same on the same logical volume which I used as a zfs test pool before: no errors.
The only thing that remains is that theres a bug in the zpool scrub function.

Since you guys already published 0.6.9 I don't know if this is of any concern to you. But if this bug is real, it might concern Ubuntu Lucid users. I'd be willing to give you any debug output you might want, but I'd also like the data from my pools a.s.a.p. given that it looks like Swiss cheese now, so I will upgrade to 0.6.9 asap and see if that helps.

Added 1 day(s) ago.

(anonymous) added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

Sorry, yes, I can confirm that "release" does work correctly. I had mangled the snapshot name in my tests, because of a copy-and-paste error, but, after I fixed that, I was able to "release" holds, just not list them using the "zfs holds" command. I think this issue is correct now.

Added 3 day(s) ago.

Seth Heeren added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

I can release a hold just fine using the maint branch. I assume you should be able to do as much. Are you sure you sepcify the tag (correctly)?

Here is what I did on my system:

sehe@lucid:/tmp$ dd of=backing.vdev bs=512M seek=1024 count=0
sehe@lucid:/tmp$ zpool create sparse /tmp/backing.vdev
sehe@lucid:/tmp$ zfs snapshot -r sparse@now
sehe@lucid:/tmp$ zfs hold -r sparse@now
  usage:
      hold [-r] <tag> <snapshot> ...

  For the property list, run: zfs set|get

  For the delegated permission list, run: zfs allow|unallow

sehe@lucid:/tmp$ zfs hold -r mytag sparse@now
sehe@lucid:/tmp$ zfs release -r yourtag sparse@now
  cannot release 'yourtag' from 'sparse@now': no such tag on this dataset
sehe@lucid:/tmp$ zfs release -r mytag sparse@now

Indeed, when you try to use 'holds' you get the expected error:


sehe@lucid:/tmp$ zfs holds
  internal error: /usr/lib/zfs/pyzfs.py not found

Added 5 day(s) ago.

Seth Heeren added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

Ok that seems odd. i will look into it a bit later

For the moment: the following are not supported:


sehe@lucid:~/custom/_ZFS$ grep zfs_do_python src/cmd/zfs/zfs_main.c
static int zfs_do_python(int argc, char **argv);
    { "allow", zfs_do_python, HELP_ALLOW },
    { "unallow", zfs_do_python, HELP_UNALLOW },
    { "holds", zfs_do_python, HELP_HOLDS },
zfs_do_python(int argc, char **argv)

Note that 'zfs userspace' is partly supported by fallback C-version

Added 6 day(s) ago.

Bryan Pendleton added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

When I reported the issue, I hadn't realized that "release" wasn't working either. When you do a "zfs release" it gives command line guidance, but add any more arguments and it gives you a full list of possible commands (which ommits hold, holds, and release). Not sure if the command is just blocked, or if release isn't implemented. No need for "holds", just maybe not nice to allow us to set a hold with "zfs hold" if "zfs release" is unavailable.

Added 6 day(s) ago.

Seth Heeren added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

Erm... I don't know

I don't use holds,

is 'zfs holds' needed to release the hold? I did not take that away from the original report. I read it to mean that 'zfs holds' merely lists the holds.

I'd be happy to change the issue title back if this is in fact a stopper in that way

Issue state: postponed → open. Added 6 day(s) ago.

Bryan Pendleton added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

Perhaps, if this bug is going to continue to exist, then "zfs hold" should be disabled? Apparently, it is possible to set a hold, but not release it. Should I open a separate issue for that?

Added 6 day(s) ago.

Seth Heeren added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

Currently pyzfs has has not been ported to zfs-fuse

This is blocking a few subcommands of zfs; some of the commands get replaced by less fullfeatured C-coded versions.

In general the areas are rolebased accounting and reporting and they are not high-priority. If someone with a little time on their hands want to start porting pyzfs, we would be most grateful.

Perhaps in time we could have derived works like the time-slider nautilus extension!

Issue state: unconfirmed → postponed. Responsible manager: (UNASSIGNED) → sgheeren. Added 6 day(s) ago.

Seth Heeren added a new response to »Broken link to .deb packages on front page«:

Good call. Fixed

Issue state: unconfirmed → resolved. Responsible manager: (UNASSIGNED) → sgheeren. Added 6 day(s) ago.

Seth Heeren added a new response to »Enable scrub to report errors«:

You could use a custom cron with zpool status -x

This will report nothing if there are no alerts/errors

This is WON'T FIX for the moment, as I don't think ZFS has any (other) standardized mechanism for this

You could look at integrating this with something modular like logwatch or munin

Issue state: unconfirmed → rejected. Severity: Medium → Low. Target release: 0.7.0 → None. Responsible manager: (UNASSIGNED) → sgheeren. Added 6 day(s) ago.

Bryan Pendleton added a new response to »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]«:

Not sure about the "proper" fix, but anyone else running into this might like to know that you can get an idea of what holds might be in place by inspecting your "zpool history" output.

Unfortunately, I'm not sure that "zpool release" works right now, in any case.

Added 8 day(s) ago.

New issue »pyzfs not implemented [WAS: "zfs hold" works to add holds, "zfs holds" doesn't work to view them]« added by Bryan Pendleton

I am able to set a hold on a snapshot using "zfs hold tag pool@snapshot", but when I execute "zfs holds" to list existing holds, I'm given the cryptic error "internal error: /usr/lib/zfs/pyzfs.py not found"

Added 8 day(s) ago.

New issue »Broken link to .deb packages on front page« added by Eric Astor

The "Get ZFS for Linux for Ubuntu Jaunty" links (both 32- and 64-bit) on the front page are broken, linking to non-existent pages in Launchpad. In my opinion, these links should really be replaced by a single link to Seth's zfs-fuse PPA (https://launchpad.net/~bugs-sehe/+archive/zfs-fuse/), since this page will otherwise need to be maintained in sync with Seth's PPA, as superseded files in Launchpad are quickly removed.

Added 10 day(s) ago.