Personal tools
You are here: Home Issue tracker zpool import doesn't mount anything

#60 — zpool import doesn't mount anything

State Resolved
Version: 0.6.9
Area Functionality
Issue type Bug
Severity Important
Submitted by Christopher May
Submitted on Jun 13, 2010
Responsible Seth Heeren
Target release:
Return to tracker
Last modified on Sep 19, 2010 by Seth Heeren
Hi, I have a zpool that was created in FreeBSD 8 (v14) then the server was reinstalled with Solaris (v15).

Well now I managed to installed 0.6.9 (through Gentoo portage) and I had to wait for the zpool to resilver.


the problem is that whenever I import the pool I get the following msg;

shadow ~ # zfs mount -a
cannot mount 'tank': Input/output error.
Make sure the FUSE module is loaded.

Same as with zpool import tank. I had several mountpoints configured and none of them worked. I can confirm that the fuse module is loaded. Sshfs works fine.

Kernel is 2.6.34-gentoo; I can downgrade easily enough.

I recompiled zfs-fuse with debug enabled, so will happily provide more information.

Also from messages;

Jun 14 01:03:39 shadow zfs-fuse: kstat: fuse_mount error - trying to umount



Cheers,
Maquis196
Added by Seth Heeren on Jun 14, 2010 01:45 AM
Responsible manager: (UNASSIGNED)sgheeren
There is some confusion: you say 'zpool import fails' but if 'zfs mount -a' does anything at all, that isn't entirely true. Could you show the output of 'zpool status -v' before the mount -a step?

from the top of my head:

provide 'fusermount -V'
check ulimit -a <important>from within the init script that launches zfs, preferrably just before the launch</important>

Also you may want to check file permissions on /var/lib/zfs, /etc/zfs, /var/run/zfs-fuse.pid and /var/run/zfs/zpool.cache

Further note this FAQ: http://zfs-fuse.net/documentation/upgrading/pool-discovery
Though it is unlikely, you may need to 'zpool import -d /dev -d /dev/mapper tank'
Added by Christopher May on Jun 14, 2010 02:48 AM
Hi, sorry I shouldn't have bothered with zfs mount -a at all at this point. If I focus on zpool import;

The zpool does import but as I said nothing mounts and although I can set mountpoints to legacy I'm unsure how I can mount then manually. The mountpoints that are needed are created by zpool import though.

shadow ~ # fusermount -V
fusermount version: 2.8.1

'zpool import -d /dev -d /dev/mapper tank' again imports the tank but still;

shadow ~ # zpool export tank
shadow ~ # zpool import -d /dev/ -d /dev/mapper tank
cannot mount 'tank': Input/output error.
Make sure the FUSE module is loaded.

zpool status -v reports no errors (all the resilvering was completed yesterday without any problems)

The zfs launch script is the one provided in the ebuild and no mention of ulimit. I'll give the 0.6.9 source version a try just in case this is something Gentoo specific.

Cheers,
Maq
Added by Christopher May on Jun 14, 2010 03:35 AM
Hi, please disregard my request. I'm not sure the error msg I got was a feature or a bug but if I start zfs-fuse with;

--no-kstat-mount

then an import tank does correctly mount what is required. This was with the compiled from source version so I shall give it a go with the Gentoo specific one now.

Cheers for the good work people.

Maq
Added by Seth Heeren on Jun 14, 2010 03:47 AM
Issue state: unconfirmedopen
Severity: MediumImportant
Target release: None0.6.9
mmm the kstat mount not working _is_ a bug. I've not yet heard any reports that prevented normal zfs filesystem to mount in this situation.

If you want to help investigate the cause, I'd be most thankful

(a) you could add a 'ulimit -a' inside your initscript to see what the limits are
(b) could you 'cat /etc/mtab' before launching zfs-fuse _and_ after doing the pool import that fails to mount

Also, for your info issue #51, notably respone 11:
   http://zfs-fuse.net/issues/51/11

I have a pointers on how to workaround this issue (#51) as well:
   http://zfs-fuse.net/issues/51/14
> You might try my issue51 branch (which has a patch merged from the issue47 branch)
> that should allow zfs-fuse to do a graceful shutdown under almost any circumstances.
> This might alleviate your situation that lets the kstat > mount 'hang' after daemon exit.

> Obviously, another sane approach would be to add an 'umount -fl /zfs-kstat' after daemon shutdown in your init-script.
Added by Christopher May on Jun 14, 2010 11:16 AM
Hi, would be glad to help, will post results later when I get home.

Cheers,
Maq
Added by Christopher May on Jun 15, 2010 12:04 PM
Ulimit -a information;

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31170
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31170
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited



Here is the mtab before;


rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,barrier=1,data=ordered 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
rc-svcdir /lib64/rc/init.d tmpfs rw,nosuid,nodev,noexec,relatime,size=1024k,mode=755 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
udev /dev tmpfs rw,nosuid,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/md4 /var ext4 rw,noatime 0 0
usbfs /proc/bus/usb usbfs rw,noexec,nosuid,devmode=0664,devgid=85 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0
none /sys/fs/fuse/connections fusectl rw 0 0

and after;

rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,barrier=1,data=ordered 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
rc-svcdir /lib64/rc/init.d tmpfs rw,nosuid,nodev,noexec,relatime,size=1024k,mode=755 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
udev /dev tmpfs rw,nosuid,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/md4 /var ext4 rw,noatime 0 0
usbfs /proc/bus/usb usbfs rw,noexec,nosuid,devmode=0664,devgid=85 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,noexec,nosuid,nodev 0 0
none /sys/fs/fuse/connections fusectl rw 0 0


I did see some mention of /zfs-kstat in mtab before I rebooted to make sure I had a plain environment for these tests.

Cheers,
Maq
Added by Seth Heeren on Jun 15, 2010 02:15 PM
Mmm obviously the mount points are missing. This should be because you experienced the error.

Or it could be because you hadn't a pool imported and / or --no-kstat-mount was active.

Most importantly, can you confirm that the mtab snapshots were from a scenario in which you saw the originally reported error ("cannot mount 'tank': Input/output error."...)

Thanks again; if this is annoying to you, rest assured, I am asking because it is hard to diagnose a remote problem like this and it is easy to jump to the wrong conclusion. Nothing worse than the sole tracker maintainer being occupied with wind-mills [1]

[1] http://en.wikipedia.org/wiki/Don_Quixote
Added by Christopher May on Jun 15, 2010 03:21 PM
No thats fine dude, I can confirm the zpool was indeed imported and the --no-kstat-mount wasn't being used. I'll double check this for you now to be certain.

If this problem is something you want fixed I'd happily try more complex procedures (I'm a Unix sysadmin by trade so know my way around Linux :)). Also if it helps I would be happy to provide ssh access for you to look at the the problem remotely.

Cheers,
Maq
Added by Seth Heeren on Jun 15, 2010 03:44 PM
Ok thanks. I have a feeling that you are certain enough, no need to double check the doubly checked.

On the other help, I might take you up on that. At the moment I have test VMs for just about everything (Arch64, Suse64, Lenny32, Squeeze32, Lucid32, Lucid64, building my Mandriva64 today (sweet install!) and OpenSolaris both physical and virtual...) The closest thing I have got installed to gentoo is Sabayon - but I'm not sure how related they are these days. I used to be a gentoo crack, until I succumbed to Debian/Ubuntu (gargggg)!

At first thought, though, I think I would like to ask you to try the mentioned approaches from the other issue. Second thought says it will probably not help since we're not actually seeing any funny mountpoints or stale mountpoints... sigh. I'll let you know a bit later on because I'm doing two other issues and the packaging 'game' at the same time :)
Added by Christopher May on Jun 15, 2010 03:55 PM
Yeah I tried to switch to Debian (for my desktop) but everytime I hit something that didn't work I wasn't sure were anything was and crept back to the comfort of Gentoo (been using it since 02 roughly so know it very well :P).

I have no idea how similar sabayon is but thats the problem with gentoo I guess, what is a "default" install? I do feel sorry for those debugging Gentoo issues since you would need to know so much (think emerge -info is the main one?)

Well when you want to jump on let me know and ill sort you out a login, etc (over PM or email of course lol)

Cheers,
Maq
Added by Seth Heeren on Jun 22, 2010 07:23 AM
Hi there.

I did try to contact you off-list, I hope you received that mail.

Meanwhile, we have uncovered a permissioning issue with the gentoo ebuild, see http://groups.google.com/group/zfs-fuse/msg/57764d892a4d7ac

I hope that helps
Added by (anonymous) on Jun 22, 2010 07:41 AM
Hi, I think our spam filters have done our legs in here! I got the 2nd email from you (it referred to a first I never received). I did reply saying you should send the first one again but I think your spam filter must have got that one as well :p.

I'll take a look at that link and get back to you

Cheers,
Maq
Added by Seth Heeren on Jun 22, 2010 07:55 AM
Mmm ok, my sgheeren<AT>hotmail.com does not accept any mail except from known news groups :)

Apparently my other email address got stuck in your filters. Catch-22 :)

Anyways, I think your problem will have been solved now :)
Added by (anonymous) on Jun 22, 2010 10:42 AM
I admit I noticed the lack of root-ness when trying to get it working. But to be fair; when I was trying to do it all as root from the CL I still got the errors, only the don't mount kstat option works for me.

Although to make this stranger, the other day the script DID work as daemon, even had kstat mounted on /. I could restart it and it was fine and mounting properly, I'm not sure what I did or changed because after I rebooted (before trying to re-create the issue) it went away and has not worked since!

How bizarre lol.

Well we can see how that bug turns out but your welcome to ssh access whenever you want.

Cheers,
Maq
Added by Seth Heeren on Jun 22, 2010 12:23 PM
probably because you did once have the good notion to chown daemon /zfs-kstat or chmod a+rwx /zfs-kstat
/or/ you just adopted the --no-kstat-mount option :)

I suggest you run as root, until the gentoo package gets it's act straight. It should not be impossible, but we (a.t.m.) do not support other than root configs
Added by Seth Heeren on Sep 19, 2010 05:09 PM
Issue state: openresolved
Target release: 0.6.9None
fixed in gentoo portage as of

22 Jun 2010; Samuli Suominen <ssuominen@gentoo.org>
+zfs-fuse-0.6.9-r1.ebuild, +files/zfs-fuse:
Run as "root" instead of "daemon" wrt #242390 by Stefan G. Weichinger.