#51 — zfs umount -a invalid dataset name

You are having trouble with the kstat-mount.

In my experience (see my invalid issue #46) this mostly happens when the exec calls to mount the kstat fs fail

Could you report

(a) the exact version used (pref-bly by way of:'')
      git log -1
(b) the distro and pref-bly fuse version
      uname -a
      lsb_release -a # if available
      fusermount -V # if available
(c) the contents of /etc/mtab before launching zfs-fuse and after getting the error message
(d) do you see anything special in the logs?
      egrep 'zfs|fuse' /var/log/messages # (or simply attach the log)

You could test that the problem is absent when starting with zfs-fuse --no-kstat-mount

Cheers

Thanks for the prompt response.

0.6.9 is scheduled for today and I'm keen on fixing any issues that might affact a larger user base up front.

> Adding the option --no-kstat-mount helps (the error doesn't appear, neither does the original problem). Does it have any adverse side-effects?
Yes, namely the kstat mount being absent. Apparently, it is failing anyways so... Kstat is like the /proc fs for the linux kernel, giving insight in internal (performance) statistics of the zfs daemon.

Have you got any advice on how to replicate your debian (?) setup? I'd like to reproduce this in my own environment. The kernel name (remotejava) doesn't ring a bell, so if you could spare more details, thanks.

Last idea: have you checked the ulimit settings for the daemon. Specifically you might dump the output of ulimit -a to a file _at the location_ in in the initscript (important!) where you launch the zfs-fuse daemon process. Bash shell is preferred for me, though most shells have a proper ulimit builtin.

I've been able to reproduce some problems on a brand new Sqeeuze box.

Are you able to confirm _any_ mounted zfs filesystems at all? I get 'cannot mount' with 'Make sure the FUSE module is loaded'. That is both when running a debug=2 and a debug=0 build.

You might also try something like this

bash$ for a in {a,b,c,d,e}{,/a,/b,/c,/d,/e}; do zfs create yourpoolname/$a; done

And see if there are errors. Please check /etc/mtab as well, as the error messages may not be accurate.

I'm currently building the vanilla kernel as well.

==================

Actions:
    0 install empty Squeeze base image
    1 sudo apt-get install libaio-dev libattr1-dev libacl1-dev libz-dev libz-dev libfuse-dev libfuse2 scons libssl-dev build-essential git-core
    2 git clone http://git.zfs-fuse.net/official -b testing
    3 cd official/
    4 git checkout ad9248b3b6349b3bc92dc047117341676cce253e
    9 (cd src && sudo scons -j3 debug=2 install)
   10 sudo ~/issue51.sh

(script is attached)
Output:

root@tilaa:/home/sehe# ./issue51.sh
19805
=========== EXPORT/DESTROY done ==============
forced clean relaunch of zfs-fuse
======== building pool 1 named issue511 at /tmp/issue511_blk =================================
removed `/var/lock/zfs/zfs_lock'
zpool create -O mountpoint=/tmp/issue511 issue511 /tmp/issue511_blk/za1
cannot mount 'issue511': Input/output error.
Make sure the FUSE module is loaded.
  pool: issue511
 state: ONLINE
 scrub: none requested
config:

        NAME STATE READ WRITE CKSUM
        issue511 ONLINE 0 0 0
          /tmp/issue511_blk/za1 ONLINE 0 0 0

errors: No known data errors
NAME USED AVAIL REFER MOUNTPOINT
issue511 81K 27.4M 21K /tmp/issue511

SYSTEM DETAILS

root@tilaa:/home/sehe# uname -a
Linux tilaa 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 x86_64 GNU/Linux
root@tilaa:/home/sehe# dpkg --get-selections | xargs
acpi install acpi-support-base install acpid install adduser install apt install apt-utils install aptitude install base-files install base-passwd install bash install binutils install bsdmainutils install bsdutils install build-essential install busybox install bzip2 install ca-certificates install console-tools install coreutils install cpio install cpp install cpp-4.4 install cron install dash install debconf install debconf-i18n install debian-archive-keyring install debianutils install dhcp3-client install dhcp3-common install diffutils install dmidecode install dpkg install dpkg-dev install e2fslibs install e2fsprogs install eject install ethtool install findutils install g++ install g++-4.4 install gcc install gcc-4.4 install gcc-4.4-base install gettext-base install git install git-core install gnupg install gpgv install grep install groff-base install grub-common install grub-pc install gzip install hostname install ifupdown install info install initramfs-tools install initscripts install insserv install install-info install installation-report install iproute install iptables install iputils-ping install klibc-utils install laptop-detect install less install libacl1 install libacl1-dev install libaio-dev install libaio1 install libapr1 install libaprutil1 install libattr1 install libattr1-dev install libblkid1 install libboost-iostreams1.40.0 install libbsd0 install libbz2-1.0 install libc-bin install libc-dev-bin install libc6 install libc6-dev install libcomerr2 install libconsole install libcurl3-gnutls install libcwidget3 install libdb4.5 install libdb4.7 install libdb4.8 install libdevmapper1.02.1 install libdigest-sha1-perl install libdpkg-perl install libedit2 install libept0 install liberror-perl install libexpat1 install libfreetype6 install libfuse-dev install libfuse2 install libgcc1 install libgcrypt11 install libgdbm3 install libgmp3c2 install libgnutls26 install libgomp1 install libgpg-error0 install libgpm2 install libgssapi-krb5-2 install libidn11 install libk5crypto3 install libkeyutils1 install libklibc install libkrb5-3 install libkrb5support0 install libldap-2.4-2 install liblocale-gettext-perl install liblog4cxx10 install liblzma2 install libmpfr1ldbl install libncurses5 install libncursesw5 install libnewt0.52 install libpam-modules install libpam-runtime install libpam0g install libperl5.10 install libpopt0 install libreadline5 install libreadline6 install libruby1.8 install libsasl2-2 install libselinux1 install libselinux1-dev install libsepol1 install libsepol1-dev install libsigc++-2.0-0c2a install libslang2 install libsqlite3-0 install libss2 install libssl-dev install libssl0.9.8 install libstdc++6 install libstdc++6-4.4-dev install libtasn1-3 install libtext-charwidth-perl install libtext-iconv-perl install libtext-wrapi18n-perl install libtimedate-perl install libusb-0.1-4 install libuuid1 install libwrap0 install libxapian15 install linux-image-2.6-amd64 install linux-image-2.6.32-3-amd64 install linux-libc-dev install locales install locate install login install logrotate install lsb-base install lzma install make install man-db install manpages install mawk install mime-support install module-init-tools install mount install nano install ncurses-base install ncurses-bin install net-tools install netbase install netcat-traditional install openssh-blacklist install openssh-client install openssh-server install openssl install os-prober install passwd install patch install perl install perl-base install perl-modules install procps install python install python-minimal install python-support install python2.5 install python2.5-minimal install readline-common install rsyslog install scons install screen install sed install sensible-utils install sudo install sysv-rc install sysvinit install sysvinit-utils install tar install tasksel install tasksel-data install tcl8.4 install traceroute install tzdata install ucf install udev install usbutils install util-linux install vim install vim-common install vim-nox install vim-runtime install vim-tiny install wget install whiptail install xz-utils install zlib1g install zlib1g-dev install

Mmmm this solved it for my teset machine with standard (non-vanilla) kernel:

bash$ apt-get install fuse-utils
bash$ ./issue51.sh
=========== EXPORT/DESTROY done ==============
fusectl on /sys/fs/fuse/connections type fusectl (rw)
forced clean relaunch of zfs-fuse
======== building pool 1 named issue511 at /tmp/issue511_blk =================================
removed `/var/lock/zfs/zfs_lock'
zpool create -O mountpoint=/tmp/issue511 issue511 /tmp/issue511_blk/za1
  pool: issue511
 state: ONLINE
 scrub: none requested
config:

        NAME STATE READ WRITE CKSUM
        issue511 ONLINE 0 0 0
          /tmp/issue511_blk/za1 ONLINE 0 0 0

errors: No known data errors
NAME USED AVAIL REFER MOUNTPOINT
issue511 81K 27.4M 21K /tmp/issue511
bash$ mount | tail -3
fusectl on /sys/fs/fuse/connections type fusectl (rw)
kstat on /zfs-kstat type fuse (rw,nosuid,nodev,allow_other)
issue511 on /tmp/issue511 type fuse (rw,allow_other)


I cannot see how this could be your problem, since 'dpkg --search fusermount' returns fuse-utils (so it must be installed).
Still, you might want to check

bash$ lsmod | grep fuse

for output...

About your earlier response: of course (sorry about that)

Compiling the vanilla kernel is taking a bit longer than expected. I'll keep you posted

Sorry can't reproduce the issue. Here is the steps I took. I had to enable the AMD PCNET32 kernel module because I tested in a Virtualbox guest.
Steps are for creation of the kernel from kernel.org tarball, and the output is immediately subsequent on reboot.
I can demonstrate full ability to use zfs mounts and /zfs-kstat as well.

I'm afraid I'm going to have to ask you to supply more pointers for me to be able to reproduce this issue. I'd suggest sending
    * /etc/apt/sources.list
    * output of lsmod
    * output of dpkg --get-selections

Anything else you can think of (presence/absense of udev/hal, SELinux, console-kit, memory limits and usage, ...?)

Regards,
Seth

------------------------- STEPS & OUTPUTS ------------------------

    1 cd /usr/src/
    2 wget -c http://www.kernel.org/[…]/linux-2.6.34.tar.bz2
    3 tar xjf linux-2.6.34.tar.bz2
    4 ln -s linux-2.6.34 linux
    5 cd linux
    6 cp /root/config.gz .
    7 zcat config.gz > .config
    8a vi .config # enable network module PCNET32
    8b make oldconfig
    9 gunzip config.gz
   10 diff .config config

4c4
< # Thu Jun 3 10:54:41 2010
---
> # Mon May 31 23:07:51 2010
1177c1177
< CONFIG_PCNET32=y
---
> # CONFIG_PCNET32 is not set

   11 make clean
   12 make deb-pkg
   13 cd ..
   14 dpkg -i linux-image-2.6.34_2.6.34-1_i386.deb
   15a echo "GRUB_DISABLE_LINUX_UUID=true" >> /etc/default/grub
   15b update-grub

Reboot

root@debian:~# uname -a
Linux debian 2.6.34 #2 SMP PREEMPT Thu Jun 3 10:55:33 EDT 2010 i686 GNU/Linux

root@debian:~# fusermount -V
fusermount version: 2.8.1
root@debian:~# ./issue51.sh
=========== EXPORT/DESTROY done ==============
fusectl on /sys/fs/fuse/connections type fusectl (rw)
forced clean relaunch of zfs-fuse
======== building pool 1 named issue511 at /tmp/issue511_blk =================================
removed `/var/lib/zfs/zpool.cache'
zpool create -O mountpoint=/tmp/issue511 issue511 /tmp/issue511_blk/za1
  pool: issue511
 state: ONLINE
 scrub: none requested
config:

        NAME STATE READ WRITE CKSUM
        issue511 ONLINE 0 0 0
          /tmp/issue511_blk/za1 ONLINE 0 0 0

errors: No known data errors
NAME USED AVAIL REFER MOUNTPOINT
issue511 81K 27.4M 21K /tmp/issue511

root@debian:~# mount
/dev/hda1 on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
kstat on /zfs-kstat type fuse (rw,nosuid,nodev,allow_other)
issue511 on /tmp/issue511 type fuse (rw,allow_other)

root@debian:~# cat /zfs-kstat/zfs/zfetchstats/misses
22

root@debian:~# for a in {a,b,c,d,e}{,/a,/b,/c,/d,/e}; do zfs create issue511/$a; done
zfs list
root@debian:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
issue511 962K 26.6M 27K /tmp/issue511
issue511/a 132K 26.6M 27K /tmp/issue511/a
issue511/a/a 21K 26.6M 21K /tmp/issue511/a/a
issue511/a/b 21K 26.6M 21K /tmp/issue511/a/b
issue511/a/c 21K 26.6M 21K /tmp/issue511/a/c
issue511/a/d 21K 26.6M 21K /tmp/issue511/a/d
issue511/a/e 21K 26.6M 21K /tmp/issue511/a/e
issue511/b 132K 26.6M 27K /tmp/issue511/b
issue511/b/a 21K 26.6M 21K /tmp/issue511/b/a
issue511/b/b 21K 26.6M 21K /tmp/issue511/b/b
issue511/b/c 21K 26.6M 21K /tmp/issue511/b/c
issue511/b/d 21K 26.6M 21K /tmp/issue511/b/d
issue511/b/e 21K 26.6M 21K /tmp/issue511/b/e
issue511/c 132K 26.6M 27K /tmp/issue511/c
issue511/c/a 21K 26.6M 21K /tmp/issue511/c/a
issue511/c/b 21K 26.6M 21K /tmp/issue511/c/b
issue511/c/c 21K 26.6M 21K /tmp/issue511/c/c
issue511/c/d 21K 26.6M 21K /tmp/issue511/c/d
issue511/c/e 21K 26.6M 21K /tmp/issue511/c/e
issue511/d 132K 26.6M 27K /tmp/issue511/d
issue511/d/a 21K 26.6M 21K /tmp/issue511/d/a
issue511/d/b 21K 26.6M 21K /tmp/issue511/d/b
issue511/d/c 21K 26.6M 21K /tmp/issue511/d/c
issue511/d/d 21K 26.6M 21K /tmp/issue511/d/d
issue511/d/e 21K 26.6M 21K /tmp/issue511/d/e
issue511/e 131K 26.6M 26K /tmp/issue511/e
issue511/e/a 21K 26.6M 21K /tmp/issue511/e/a
issue511/e/b 21K 26.6M 21K /tmp/issue511/e/b
issue511/e/c 21K 26.6M 21K /tmp/issue511/e/c
issue511/e/d 21K 26.6M 21K /tmp/issue511/e/d
issue511/e/e 21K 26.6M 21K /tmp/issue511/e/e

Ok, that reminds me of asking for the init.d/zfs-fuse script (unless it is the one from ./contrib in the source tree?)

I'm not actually using that, and it might contain some subtly funny business (not unlikely: old quirks that used to be necessary back-when).

I'll try later tonight. But I'll probably let the release pass anyway, since it

(a) has not proven simple to reproduce
(b) it is most likely an environmental issue/the init-script.

The init-script as such is usually provided by the respective packagers. If you don't use any existing packaged version, I'm assuming you won't personally mind too much to have to update after the official 0.6.9 release :)

I'll keep this issue up-to-date just in case

I'm going to respond to these points:

A. On the script
B. On the issue (description, a patch that could help for your testing, and proposed workaround if all else fails).

A. Ok: on the script
=====================================================================
Propose to update to ./contrib/zfs-fuse.initd.ubuntu (source tree)

Which has the following changes:
1. mention remote_fs in dependencies
2. remove double slash daemon path
3. provide recommended default DAEMON_OPTS
4. no longer clobber /dev/fd0 (see issue #49 — zpool import slow because of /dev/fd0)
5. no longer sleep (see issue #36 — zfs-fuse fedora init script sometimes attempts ...)
6. issue zfs share -a (due to new nfs sharing capabilities (see http://zfs-fuse.net/releases/0.6.9)

You could reinforce the zpool import -a -f if you need it.
Your DAEMON_OPTS is most likely overridden by /etc/default/zfs-fuse (in line 156). This, incidentally is the recommended location for such changes, so future versions of the init-script can be updated by your package manager of choice without risking losing your options

B. On the issue itself
===================
I cannot get the script to voluntarily give the reported behaviour. However, I _can_ trigger the same behaviour by making /zfs-kstat stick around

     bash$ killall -9 zfs-fuse
     bash$ rm -f /var/run/zfs-fuse.pid
     bash$ /etc/init.d/zfs-fuse start

Problem shows itself. So that is the real bug there. I can do some debugging there. You might try my issue51 branch (which has a patch merged from the issue47 branch) that should allow zfs-fuse to do a graceful shutdown under almost any circumstances. This might alleviate your situation that lets the kstat mount 'hang' after daemon exit.

Obviously, another sane approach would be to add an 'umount -fl /zfs-kstat' after daemon shutdown in your init-script.

Hi back, please check that all processes (zfs-fuse, zfs, zpool) are being run as root.

Note: issue51 branch has just been merged into testing, available as daily snapshot from here:
http://zfs-fuse.net/releases/0.7.0

> I can no longer reproduce this problem with the current release
Ok, sorry to bug you but can you reconfirm with what revision that is (git log -1)?

> and haven't seen it happen for a while, despite having removed my crude umount /dev/fuse fix). The only occasion where 'zfs-fuse stop' tends to fail (due to timeout?) is when scrub is underway, but it's another (yet unreported) issue

closing

grrrrr really closing :)