#48 — Segfault on zfs send -D

Yes I can confirm quick and sudden death on zfs send -D

The problem seams to be that internally the pipe seems to be connected backwards, resulting in a permission error when the checksumming (ddt) thread tries to open it's input fd for reading.

man PIPE(2) tells us that 'pipefd[0] refers to the read end of the pipe. pipefd[1] refers to the write end of the pipe'. Inspection of /proc/???/fd/ shows that to be true:

l-wx------ 1 root root 64 2010-05-28 22:57 12 -> pipe:[1351962]
lr-x------ 1 root root 64 2010-05-28 22:57 11 -> pipe:[1351962]

The amazing thing is, that onnv_gate contains the same ordering of fds, even in hg revision

changeset: 12514:a89ab814a56d
tag: tip
user: Janice Chang <Janice.Chang@Sun.COM>
date: Fri May 28 13:47:12 2010 -0400
summary: 6944258 Investigate alternatives to ZFS_FORCE

Afaict (and google) pipe(2) should not operate backwards on linux vs. solaris?!

I'll ponder this a bit. Meanwhile, if you care to debug this type of thing:

$ gdb --args zfs send -DvR pool@snap
(gdb) break main
Breakpoint 1 at 0x8053758: file cmd/zfs/zfs_main.c, line 4034.
(gdb) run
(gdb) call dup2(creat("/dev/null", 0600), 1)

That will reattach stdout to /dev/null so you can debug it.

Ah

reading the solaris man-page on pipe(2) reveals that

"The files associated with fildes[0] and fildes[1] are
     streams and are both opened for reading and writing"
...
"Since a pipe is bi-directional, there are two separate flows
     of data. Therefore, the size (st_size) returned by a call to
     fstat(2) with argument fildes[0] or fildes[1] is the number
     of bytes available for reading from fildes[0] or fildes[1]
     respectively. Previously, the size (st_size) returned by a
     call to fstat() with argument fildes[1] (the write-end) was
     the number of bytes available for reading from fildes[0]
     (the read-end)."

We'll have to see if have a portability solution for this

shute: man pipe(7)

Portability notes

On some systems (but not Linux), pipes are bidirectional: data can be transmitted in both directions between the pipe ends. According to POSIX.1-2001, pipes only need to be unidirectional. Portable applications should avoid reliance on bidirectional pipe semantics.

I'm working on a solution involving tmpnam and mkfifo...

Keep you posted

Woah. Not so bad. socketpair(2) to the rescue. I'm not very well versed in socket programming, but (on a limb) I did the simplest thing I imagined could work, and, behold!

I pushed it to testing, so please report your findings.

I just successfully sent 15+G of (dedup-ed) data over and everything was fine.

Output:
root@karmic:~# zfs send -DvR desktop@zfssend | pv | zfs receive -nvFd tmppool
sending from @ to desktop@zfssend ]
would receive full stream of desktop@zfssend into tmppool@zfssend
sending from @ to desktop/home@zfssend ]
would receive full stream of desktop/home@zfssend into tmppool/home@zfssend
WARNING: could not send desktop/home/marlies@zfssend: does not exist <=> ]
WARNING: could not send desktop/home/marlies/.wine@zfssend: does not exist
WARNING: could not send desktop/home/marlies/Photos@zfssend: does not exist
WARNING: could not send desktop/home/sehe@zfssend: does not exist
sending from @ to desktop/PersoonlijkeBestanden@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden@zfssend into tmppool/PersoonlijkeBestanden@zfssend
sending from @ to desktop/PersoonlijkeBestanden/Belastingdienst@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/Belastingdienst@zfssend into tmppool/PersoonlijkeBestanden/Belastingdienst@zfssend
sending from @ to desktop/PersoonlijkeBestanden/_oud_en_problematisch@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/_oud_en_problematisch@zfssend into tmppool/PersoonlijkeBestanden/_oud_en_problematisch@zfssend
sending from @ to desktop/PersoonlijkeBestanden/VTLBBerekeningen@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/VTLBBerekeningen@zfssend into tmppool/PersoonlijkeBestanden/VTLBBerekeningen@zfssend
sending from @ to desktop/PersoonlijkeBestanden/KLANTBESTAND@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/KLANTBESTAND@zfssend into tmppool/PersoonlijkeBestanden/KLANTBESTAND@zfssend
sending from @ to desktop/PersoonlijkeBestanden/OutlookMail@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/OutlookMail@zfssend into tmppool/PersoonlijkeBestanden/OutlookMail@zfssend
sending from @ to desktop/PersoonlijkeBestanden/Email@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/Email@zfssend into tmppool/PersoonlijkeBestanden/Email@zfssend
sending from @ to desktop/PersoonlijkeBestanden/BUDGETTEER@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/BUDGETTEER@zfssend into tmppool/PersoonlijkeBestanden/BUDGETTEER@zfssend
sending from @ to desktop/PersoonlijkeBestanden/FORMULIEREN@zfssend <=>]
would receive full stream of desktop/PersoonlijkeBestanden/FORMULIEREN@zfssend into tmppool/PersoonlijkeBestanden/FORMULIEREN@zfssend
sending from @ to desktop/PersoonlijkeBestanden/MyPictures@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/MyPictures@zfssend into tmppool/PersoonlijkeBestanden/MyPictures@zfssend
sending from @ to desktop/PersoonlijkeBestanden/boekhoudingen@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/boekhoudingen@zfssend into tmppool/PersoonlijkeBestanden/boekhoudingen@zfssend
sending from @ to desktop/PersoonlijkeBestanden/Mubi@zfssend <=> ]
would receive full stream of desktop/PersoonlijkeBestanden/Mubi@zfssend into tmppool/PersoonlijkeBestanden/Mubi@zfssend
15.6GB 0:06:20 [41.8MB/s] [

Sweet, closing this bug then!