#1 — zfs-fuse dies with libsolkerncompat_init: Assertion `vnode_cache != NULL` failed.

Until I found your "official" zfs-fuse repo, I tried with zfs-fuse from http://gitorious.org/zfs-fuse, got the same error there. I couldn't test zfs-fuse-0.5.0 though because of a compile error:

[...]
gcc -o lib/libnvpair/build-user/nvpair.o -c -pipe -Wall -Werror -std=c99 -Wno-switch -Wno-unused -Wno-missing-braces -Wno-parentheses -Wno-uninitialized -fno-strict-aliasing -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT -DTEXT_DOMAIN=\"zfs-fuse\" -DLINUX_AIO -ggdb -DDEBUG -Ilib/libnvpair/include -Ilib/libsolcompat/include lib/libnvpair/nvpair.c
lib/libnvpair/nvpair.c:29:25: error: sys/stropts.h: No such file or directory

(cf. https://bugzilla.redhat.com/show_bug.cgi?id=439403)

I had attached the backtrace as a file (zfs-gdb.txt) to this report, is there any other way to createa full backtrace?

My fuse libraries are from "stock" Fedora12 (well, 11.92/rawhide, actually, since v12 is not out yet):

fuse-debuginfo-2.8.1-1.fc12.x86_64
fuse-libs-2.8.1-1.fc12.x86_64
fuse-devel-2.8.1-1.fc12.x86_64
fuse-2.8.1-1.fc12.x86_64

I should've run this though strace(1) in the first place, there I can see:

-----------------------------------------------
mmap(NULL, 4096, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fd57d596000
mmap(0x7fd57d596000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 EACCES (Permission denied)
write(2, "lib/libsolkerncompat/main.c:73: libsolkerncompat_init: Assertion `vnode_cache != NULL` failed.\n"..., 95) = 95
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
tgkill(3323, 3323, SIGABRT) = 0
-----------------------------------------------

EACCES, hm. Fedora comes with SELinux enabled nowadays and (temporarily) disabling does the trick:

# setenforce 0

Now zfs-fuse starts just fine, so it's really SELinux issue - sorry for the noise :(

Thanks,
Christian.


For the sake of the archives: "setenforce 0" just disables the "enforcing of a policy" but now it generates a warning at least:

-----------------------------------------------
SELinux is preventing /usr/bin/zfs-fuse "execmem" access on <Unknown>.

Allowing Access:

One of the following booleans is set incorrectly: allow_execstack, allow_execmem

Fix Command:

Choose one of the following to allow access:
Allow unconfined executables to make their stack executable. This should never,
ever be necessary. Probably indicates a badly coded executable, but could
indicate an attack. This executable should be reported in bugzilla")
# setsebool -P allow_execstack 1
Allow unconfined executables to map a memory region as both executable and
writable, this is dangerous and the executable should be reported in bugzilla")
# setsebool -P allow_execmem 1

Additional Information:

Source Context unconfined_u:system_r:initrc_t:s0
Target Context unconfined_u:system_r:initrc_t:s0
Target Objects None [ process ]
Source zfs-fuse
Source Path /usr/bin/zfs-fuse
Port <Unknown>
Host mallory.housecafe.de
Source RPM Packages zfs-fuse-0.5.0-9.20081221.fc12.1
Target RPM Packages
Policy RPM selinux-policy-3.6.32-27.fc12
Selinux Enabled True
Policy Type targeted
MLS Enabled True
Enforcing Mode Permissive
Plugin Name catchall_boolean
Raw Audit Messages

node=mallory.housecafe.de type=AVC msg=audit(1255851335.821:34609): avc: denied { execmem } for pid=3519 comm="zfs-fuse" scontext=unconfined_u:system_r:initrc_t:s0 tcontext=unconfined_u:system_r:initrc_t:s0 tclass=process

node=mallory.housecafe.de type=SYSCALL msg=audit(1255851335.821:34609): arch=c000003e syscall=9 success=yes exit=47180574871552 a0=2ae91567d000 a1=1000 a2=7 a3=32 items=0 ppid=1 pid=3519 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=2 comm="zfs-fuse" exe="/usr/bin/zfs-fuse" subj=unconfined_u:system_r:initrc_t:s0 key=(null)
-----------------------------------------------

FWIW, regarding the warnings[0], some able person should probably fix those PROT_EXEC occurences:

# grep -rn PROT_EXEC .
./src/lib/libumem/vmem_mmap.c:47:#define ALLOC_PROT PROT_READ | PROT_WRITE | PROT_EXEC
./src/zfs-fuse/zfs_vnops.c:4206: if ((prot & (PROT_READ | PROT_EXEC)) &&

In the meantime, instead of enabling execmem systemwide, the following can be executed to allow it only for the zfs-fuse binary. Works for me:

# ls -LZ /usr/bin/zfs-fuse
-rwxr-xr-x. root root system_u:object_r:bin_t:s0 /usr/bin/zfs-fuse

# chcon -t execmem_exec_t /usr/bin/zfs-fuse
# ls -LZ /usr/bin/zfs-fuse
-rwxr-xr-x. root root system_u:object_r:execmem_exec_t:s0 /usr/bin/zfs-fuse

# /etc/init.d/zfs-fuse start
Cleaning up stale zfs-fuse PID file in /var/run/zfs-fuse.pid
Starting zfs-fuse: [ OK ]
Immunizing zfs-fuse against OOM kills [ OK ]
Mounting zfs partitions: [ OK ]
mallory# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 154G 5.11G 149G 3% ONLINE -


C.

[0] http://people.redhat.com/drepper/selinux-mem.html

Unfortunately my original Fedora testsystem is gone and I had to use another one, with slightly different version numbers:

---------
Linux fedora 2.6.30.9-102.fc11.i686.PAE #1 SMP Fri Dec 4 00:19:26 EST 2009 i686 i686 i386 GNU/Linux
glibc 2.10.1
NPTL 2.10.1
fuse-sshfs-2.2-5.fc11.i586
fuse-devel-2.8.1-1.fc11.i586
fuse-2.8.1-1.fc11.i586
fuse-libs-2.8.1-1.fc11.i586
---------

With SELinux enabled and allow_execstack and allow_execmem set to "off" (with 'allow_execstack=on' alone it's working!), I got the same error with the "official" tree. When using the latest checkout from Emmanuel Anne (e23119283a6007b87c1e75ddb8d06b3f8bf23ca4), zfs-fuse will die with another error message:

---------
# /opt/zfs-fuse_emmanuel/zfs-fuse --no-daemon
lib/libsolkerncompat/thread.c:45: zk_thread_create: Assertion `pthread_create(&tid, &attr, (void *(*)(void *)) func, arg) == 0` failed.
Aborted
---------

strace(1) reveals, again:

---------
mmap2(NULL, 32768, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = -1 EACCES (Permission denied)
write(2, "lib/libsolkerncompat/thread.c:45: zk_thread_create: Assertion `pthread_create(&tid, &attr, (void *(*)(void *)) func, arg) == 0` failed.\n", 136lib/libsolkerncompat/thread.c:45: zk_thread_create: Assertion `pthread_create(&tid, &attr, (void *(*)(void *)) func, arg) == 0` failed.
) = 136
---------

gdb knows:

---------
# gdb /opt/zfs-fuse_emmanuel/zfs-fuse
(gdb) run --no-daemon
Starting program: /opt/zfs-fuse_emmanuel/zfs-fuse --no-daemon
[Thread debugging using libthread_db enabled]
lib/libsolkerncompat/thread.c:45: zk_thread_create: Assertion `pthread_create(&tid, &attr, (void *(*)(void *)) func, arg) == 0` failed.

Program received signal SIGABRT, Aborted.
0x00e32424 in __kernel_vsyscall ()

(gdb) bt full
#0 0x00e32424 in __kernel_vsyscall ()
No symbol table info available.
#1 0x0013b7c1 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
        resultvar = <value optimized out>
        pid = 2609140
        selftid = 2187
#2 0x0013d092 in *__GI_abort () at abort.c:88
        act = {__sigaction_handler = {sa_handler = 0xb39e41 <_dl_lookup_symbol_x+305>, sa_sigaction = 0xb39e41 <_dl_lookup_symbol_x+305>}, sa_mask = {__val = {3221222532,
              134513256, 3221222520, 11863996, 0, 6624200, 1, 0, 1, 0, 0, 0, 0, 11932320, 0, 11932760, 1140812, 6623232, 11863648, 0, 11864088, 3221222592, 3221222520,
              3221222532, 1, 1, 0, 134518048, 6624200, 0, 1, 3391964}}, sa_flags = 98316, sa_restorer = 0x33a000}
        sigs = {__val = {32, 0 <repeats 31 times>}}
#3 0x0810457e in zk_thread_create (func=0x8103f00 <taskq_thread>, arg=0xb7fecdc8) at lib/libsolkerncompat/thread.c:45
        tid = <value optimized out>
        attr = {__size = "\0\0\0\0\0\0\0\0\1\0\0\0\0\20\0\0\0\0\0\0\0\200", '\0' <repeats 13 times>, __align = 0}
        __PRETTY_FUNCTION__ = "zk_thread_create"
#4 0x08102ffa in taskq_thread_create (tq=<value optimized out>) at lib/libsolkerncompat/taskq.c:1253
No locals.
#5 taskq_create_common (tq=<value optimized out>) at lib/libsolkerncompat/taskq.c:1672
        tq = 0xb7fecdc8
        ncpus_online = 1
        bsize = 1
        max_nthreads = <value optimized out>
        __PRETTY_FUNCTION__ = "taskq_create_common"
#6 0x081035ba in system_taskq_init () at lib/libsolkerncompat/taskq.c:857
No locals.
#7 0x0804c2dc in do_init () at zfs-fuse/util.c:92
        __PRETTY_FUNCTION__ = "do_init"
        attr = {__size = "\200\241\31\b\337C#K\364\317'\0\0\0\0\0\0\0\0\0\70\366\377\277\327\254\36\0\5\0\0\0\377\377\377\377", __align = 135897472}
#8 0x0804b523 in main (argc=2, argv=0xbffff724) at zfs-fuse/main.c:387
        ret = <value optimized out>
---------


Thank you for your time,
Christian.

I've recompiled with "scons debug=3" and attached another gdb backtrace.