Announcing: ZFS-fuse 0.7.0
#35 — Datasets go missing, yet still "exist", and space still taken up
| State | Resolved |
|---|---|
| Version: | 0.6.0 |
| Area | Functionality |
| Issue type | Bug |
| Severity | Important |
| Submitted by | Dean Carpenter |
| Submitted on | Apr 02, 2010 |
| Responsible | Seth Heeren |
| Target release: |
—
|
Last modified on
May 22, 2010
by
Seth Heeren
This particular issue has happened twice now, though I don't seem to be able to reproduce it at will. Unfortunately I do not have notes from last time.
In brief, fairly heavy load will trigger an error condition within zfs-fuse that only comes into play at next boot. Until that reboot, the datasets all appear to be OK, data can be copied between them just fine, and seems to verify just fine. After reboot, 6 out of 10 datasets are missing completely, yet the total space is still consistent - the data is THERE, it just can't be seen.
This time, the problem seems to have been triggered by several large copies from one dataset to another.
Before reboot, the only indication of a problem is the status :
root@Bondi:~# zpool status
pool: stuff
state: ONLINE
zpool: cmd/zpool/zpool_main.c:3208: status_callback: Assertion `reason == ZPOOL_STATUS_OK' failed.
Aborted
root@Bondi:~# shutdown -r now
System configuration:
ASRock A780LM motherboard, 4gig ram, X2 240 cpu
Debian Stable/Lenny
zfs-fuse 0.6.0-1 from testing
2x Hitachi HDS72202 2TB drives, with 2x 1TB partitions each
2x IBM HUA721010KLA330 1TB drives, with 1x 1TB partition each
Pool created as follows :
zpool create stuff -m /stuff raidz1 /dev/sdb1 /dev/sdd1 /dev/sde1 raidz1 /dev/sdc1 /dev/sdd2 /dev/sde2
Normal status looks like this :
root@Bondi:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
stuff 5.44T 4.71T 748G 86% ONLINE -
root@Bondi:~# zpool status
pool: stuff
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
stuff ONLINE 0 0 0
raidz1 ONLINE 0 0 0
sdb1 ONLINE 0 0 0
sdd1 ONLINE 0 0 0
sde1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
sdc1 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
errors: No known data errors
10 datasets are created, resulting in a tree like this :
root@Bondi:~# ls -la /stuff/
total 43
drwxr-xr-x 12 root root 12 2010-03-15 23:58 .
drwxr-xr-x 24 root root 4096 2010-03-15 20:04 ..
drwxr-xr-x 17 root root 29 2010-03-02 00:01 Backups
drwxr-xr-x 5 root root 5 2010-03-15 23:59 Downloads
drwxr-xr-x 84 root root 99 2010-03-02 23:06 Images
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Mail
drwxr-xr-x 10 root root 11 2010-03-01 23:56 md5
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Music
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Other
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Software
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Videos
drwxr-xr-x 2 root root 2 2010-03-01 23:54 VirtualBox
After the reboot, the list is like this :
root@Bondi:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
stuff 3.13T 440G 42.6K /stuff
stuff/Backups 176G 440G 176G /stuff/Backups
stuff/Downloads 4.00G 440G 4.00G /stuff/Downloads
stuff/Images 10.9G 440G 10.9G /stuff/Images
stuff/md5 41.9M 440G 41.9M /stuff/md5
As you can see, 6 out of the 10 datasets are totally missing. The directories are still there, they're just empty. Yet the total USED space shown for stuff is at the expected 3.13T - it still sees the space as being taken up, but I can't seem to find it. ZFS still knows about the missing datasets :
root@Bondi:~# zfs create stuff/Music
cannot create 'stuff/Music': dataset already exists
root@Bondi:~#
I have exported and imported the pool to no avail. Last time this happened (same missing directories I think - is that indicative ?) I finally wound up just destroying the pool and creating it fresh from scratch, then restoring the data from the backup system.
How can I get access to the missing datasets again ? Any debugging processes you wish me to try ?
Thanks -
D.
In brief, fairly heavy load will trigger an error condition within zfs-fuse that only comes into play at next boot. Until that reboot, the datasets all appear to be OK, data can be copied between them just fine, and seems to verify just fine. After reboot, 6 out of 10 datasets are missing completely, yet the total space is still consistent - the data is THERE, it just can't be seen.
This time, the problem seems to have been triggered by several large copies from one dataset to another.
Before reboot, the only indication of a problem is the status :
root@Bondi:~# zpool status
pool: stuff
state: ONLINE
zpool: cmd/zpool/zpool_main.c:3208: status_callback: Assertion `reason == ZPOOL_STATUS_OK' failed.
Aborted
root@Bondi:~# shutdown -r now
System configuration:
ASRock A780LM motherboard, 4gig ram, X2 240 cpu
Debian Stable/Lenny
zfs-fuse 0.6.0-1 from testing
2x Hitachi HDS72202 2TB drives, with 2x 1TB partitions each
2x IBM HUA721010KLA330 1TB drives, with 1x 1TB partition each
Pool created as follows :
zpool create stuff -m /stuff raidz1 /dev/sdb1 /dev/sdd1 /dev/sde1 raidz1 /dev/sdc1 /dev/sdd2 /dev/sde2
Normal status looks like this :
root@Bondi:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
stuff 5.44T 4.71T 748G 86% ONLINE -
root@Bondi:~# zpool status
pool: stuff
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
stuff ONLINE 0 0 0
raidz1 ONLINE 0 0 0
sdb1 ONLINE 0 0 0
sdd1 ONLINE 0 0 0
sde1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
sdc1 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
errors: No known data errors
10 datasets are created, resulting in a tree like this :
root@Bondi:~# ls -la /stuff/
total 43
drwxr-xr-x 12 root root 12 2010-03-15 23:58 .
drwxr-xr-x 24 root root 4096 2010-03-15 20:04 ..
drwxr-xr-x 17 root root 29 2010-03-02 00:01 Backups
drwxr-xr-x 5 root root 5 2010-03-15 23:59 Downloads
drwxr-xr-x 84 root root 99 2010-03-02 23:06 Images
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Mail
drwxr-xr-x 10 root root 11 2010-03-01 23:56 md5
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Music
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Other
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Software
drwxr-xr-x 2 root root 2 2010-03-01 23:54 Videos
drwxr-xr-x 2 root root 2 2010-03-01 23:54 VirtualBox
After the reboot, the list is like this :
root@Bondi:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
stuff 3.13T 440G 42.6K /stuff
stuff/Backups 176G 440G 176G /stuff/Backups
stuff/Downloads 4.00G 440G 4.00G /stuff/Downloads
stuff/Images 10.9G 440G 10.9G /stuff/Images
stuff/md5 41.9M 440G 41.9M /stuff/md5
As you can see, 6 out of the 10 datasets are totally missing. The directories are still there, they're just empty. Yet the total USED space shown for stuff is at the expected 3.13T - it still sees the space as being taken up, but I can't seem to find it. ZFS still knows about the missing datasets :
root@Bondi:~# zfs create stuff/Music
cannot create 'stuff/Music': dataset already exists
root@Bondi:~#
I have exported and imported the pool to no avail. Last time this happened (same missing directories I think - is that indicative ?) I finally wound up just destroying the pool and creating it fresh from scratch, then restoring the data from the backup system.
How can I get access to the missing datasets again ? Any debugging processes you wish me to try ?
Thanks -
D.
Added by
Dean Carpenter
on
Apr 04, 2010 09:24 AM
Interesting ... I noticed in another issue report that there was a problem with put_nvlist, and that there was a fix for it. So, I pulled down the source and built it :
root@Bondi:/usr/src# git clone http://git.zfs-fuse.net/official
Initialized empty Git repository in /usr/src/official/.git/
got b79138c93d33261003512c33c0a849668d9a9169
Getting alternates list for http://git.zfs-fuse.net/official
:
:
root@Bondi:/usr/src# cd official
root@Bondi:/usr/src/official# git checkout origin/critical
Note: moving to "origin/critical" which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
git checkout -b <new_branch_name>
HEAD is now at 3de9c7c... issue #16: BUGS edits
root@Bondi:/usr/src/official# cd src
root@Bondi:/usr/src/official/src# scons debug=2
I exported the stuff pool, stopped the stock Debian Lenny/stable zfs, then started the new zfs-fuse and imported the stuff pool. No luck - datasets still missing. Though I did see this in the syslog :
Apr 4 10:13:02 Bondi zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Apr 4 10:13:02 Bondi zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Apr 4 10:13:02 Bondi zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Apr 4 10:13:02 Bondi zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Apr 4 10:13:02 Bondi zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Apr 4 10:13:27 Bondi zfs-fuse: put_nvlist: out of memory 1820 > 1680
Apr 4 10:13:27 Bondi zfs-fuse: put_nvlist: out of memory 1820 > 1680
Apr 4 10:13:27 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:27 Bondi zfs-fuse: mount options: fsname=stuff,allow_other,suid,dev,big_writes
Apr 4 10:13:27 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:27 Bondi zfs-fuse: mount options: fsname=stuff/Backups,allow_other,suid,dev,big_writes
Apr 4 10:13:28 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:28 Bondi zfs-fuse: mount options: fsname=stuff/Downloads,allow_other,suid,dev,big_writes
Apr 4 10:13:28 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:28 Bondi zfs-fuse: mount options: fsname=stuff/Images,allow_other,suid,dev,big_writes
Apr 4 10:13:28 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:28 Bondi zfs-fuse: mount options: fsname=stuff/md5,allow_other,suid,dev,big_writes
Apr 4 10:13:29 Bondi zfs-fuse: put_nvlist: out of memory 1820 > 1680
Out of memory ??
D.
root@Bondi:/usr/src# git clone http://git.zfs-fuse.net/official
Initialized empty Git repository in /usr/src/official/.git/
got b79138c93d33261003512c33c0a849668d9a9169
Getting alternates list for http://git.zfs-fuse.net/official
:
:
root@Bondi:/usr/src# cd official
root@Bondi:/usr/src/official# git checkout origin/critical
Note: moving to "origin/critical" which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
git checkout -b <new_branch_name>
HEAD is now at 3de9c7c... issue #16: BUGS edits
root@Bondi:/usr/src/official# cd src
root@Bondi:/usr/src/official/src# scons debug=2
I exported the stuff pool, stopped the stock Debian Lenny/stable zfs, then started the new zfs-fuse and imported the stuff pool. No luck - datasets still missing. Though I did see this in the syslog :
Apr 4 10:13:02 Bondi zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Apr 4 10:13:02 Bondi zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Apr 4 10:13:02 Bondi zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Apr 4 10:13:02 Bondi zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Apr 4 10:13:02 Bondi zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Apr 4 10:13:27 Bondi zfs-fuse: put_nvlist: out of memory 1820 > 1680
Apr 4 10:13:27 Bondi zfs-fuse: put_nvlist: out of memory 1820 > 1680
Apr 4 10:13:27 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:27 Bondi zfs-fuse: mount options: fsname=stuff,allow_other,suid,dev,big_writes
Apr 4 10:13:27 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:27 Bondi zfs-fuse: mount options: fsname=stuff/Backups,allow_other,suid,dev,big_writes
Apr 4 10:13:28 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:28 Bondi zfs-fuse: mount options: fsname=stuff/Downloads,allow_other,suid,dev,big_writes
Apr 4 10:13:28 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:28 Bondi zfs-fuse: mount options: fsname=stuff/Images,allow_other,suid,dev,big_writes
Apr 4 10:13:28 Bondi zfs-fuse: enabling fuse big_writes
Apr 4 10:13:28 Bondi zfs-fuse: mount options: fsname=stuff/md5,allow_other,suid,dev,big_writes
Apr 4 10:13:29 Bondi zfs-fuse: put_nvlist: out of memory 1820 > 1680
Out of memory ??
D.
Added by
Dean Carpenter
on
Apr 06, 2010 03:35 PM
Another update. Tried the latest rainemu version. Similar process ... I've modified /etc/init.d/zfs-fuse to use the /usr/local/sbin/zfs-fuse binary rather than stock.
root@Bondi:/usr/src# git clone http://rainemu.swishparty.co.uk/git/zfs
root@Bondi:/usr/src# cd zfs/src
root@Bondi:/usr/src/zfs/src# scons
Stop current zfs :
root@Bondi:/usr/src/zfs/src# /etc/init.d/zpool export stuff
root@Bondi:/usr/src/zfs/src# /etc/init.d/zfs-fuse stop
root@Bondi:/usr/src/zfs/src# scons install
root@Bondi:/usr/src/zfs/src# /etc/init.d/zfs-fuse start
Walla. All datasets are there. So this is good. But, there's always a but. The zfs and zpool commands just hang, never to return. I can ctrl-C them, but they don't do anything. Not even zpool status. syslog entries are better ...
Apr 6 15:45:11 Bondi zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Apr 6 15:45:11 Bondi zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Apr 6 15:45:11 Bondi zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Apr 6 15:45:11 Bondi zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Apr 6 15:45:11 Bondi zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Apr 6 15:45:14 Bondi zfs-fuse: put_nvlist: out of memory 9396 > 5240
Apr 6 15:45:16 Bondi zfs-fuse: !imported version 16 pool stuff using 23
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Backups,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Downloads,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Images,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Mail,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Music,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Other,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Software,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Videos,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/VirtualBox,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/md5,allow_other,suid,dev,big_writes
There's still the "put_nvlist: out of memory 9396 > 5240" problem, but it doesn't stop all the mounting. I've verified a few large files in the various datasets, and they all seem to be there OK. I'll do more md5sum checks against the backup system tonight/tomorrow, but I think the data is all there OK.
What next ?
D.
root@Bondi:/usr/src# git clone http://rainemu.swishparty.co.uk/git/zfs
root@Bondi:/usr/src# cd zfs/src
root@Bondi:/usr/src/zfs/src# scons
Stop current zfs :
root@Bondi:/usr/src/zfs/src# /etc/init.d/zpool export stuff
root@Bondi:/usr/src/zfs/src# /etc/init.d/zfs-fuse stop
root@Bondi:/usr/src/zfs/src# scons install
root@Bondi:/usr/src/zfs/src# /etc/init.d/zfs-fuse start
Walla. All datasets are there. So this is good. But, there's always a but. The zfs and zpool commands just hang, never to return. I can ctrl-C them, but they don't do anything. Not even zpool status. syslog entries are better ...
Apr 6 15:45:11 Bondi zfs-fuse: caching mechanisms: ARC 1, block cache 1 page cache 1
Apr 6 15:45:11 Bondi zfs-fuse: ARC caching: maximum ARC size: compiled-in default
Apr 6 15:45:11 Bondi zfs-fuse: FUSE caching: attribute timeout 0.000000, entry timeout 0.000000
Apr 6 15:45:11 Bondi zfs-fuse: ARC setup: min ARC size set to 16777216 bytes
Apr 6 15:45:11 Bondi zfs-fuse: ARC setup: max ARC size set to 134217728 bytes
Apr 6 15:45:14 Bondi zfs-fuse: put_nvlist: out of memory 9396 > 5240
Apr 6 15:45:16 Bondi zfs-fuse: !imported version 16 pool stuff using 23
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Backups,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Downloads,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Images,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Mail,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Music,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Other,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Software,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/Videos,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/VirtualBox,allow_other,suid,dev,big_writes
Apr 6 15:45:18 Bondi zfs-fuse: enabling fuse big_writes
Apr 6 15:45:18 Bondi zfs-fuse: mount options: fsname=stuff/md5,allow_other,suid,dev,big_writes
There's still the "put_nvlist: out of memory 9396 > 5240" problem, but it doesn't stop all the mounting. I've verified a few large files in the various datasets, and they all seem to be there OK. I'll do more md5sum checks against the backup system tonight/tomorrow, but I think the data is all there OK.
What next ?
D.
Added by
Dean Carpenter
on
Apr 08, 2010 03:52 PM
Is there anyone out there ? Is there anything I should try here ?
How can I adjust/control the memory usage by put_nvlist() ?
D.
How can I adjust/control the memory usage by put_nvlist() ?
D.
Added by
Dean Carpenter
on
Apr 20, 2010 03:45 PM
I hate to sound like a pest, but is there any activity here with zfs ? I need to put this box back into use again.
The stock Debian Stable/Lenny zfs-fuse 0.6.0 appears to be vulnerable (at least in my case) to a load issue, since I've seen the same situation happen twice. The rainemu version from 4/6/2010 seems to work in that the datasets mount, but the utilities just hang. I see that there may be a few updates there, so I'll try them shortly.
Is the rainemu build considered stable ? I would really like to use zfs, but so far it doesn't look production ready. I'd like to help find the problem, so I haven't modified the zfs setup yet. If there is any debugging or digging you want, let me know. Otherwise I'm going to wipe the zfs config and rebuild it from scratch, possibly using rainemu/current.
D.
The stock Debian Stable/Lenny zfs-fuse 0.6.0 appears to be vulnerable (at least in my case) to a load issue, since I've seen the same situation happen twice. The rainemu version from 4/6/2010 seems to work in that the datasets mount, but the utilities just hang. I see that there may be a few updates there, so I'll try them shortly.
Is the rainemu build considered stable ? I would really like to use zfs, but so far it doesn't look production ready. I'd like to help find the problem, so I haven't modified the zfs setup yet. If there is any debugging or digging you want, let me know. Otherwise I'm going to wipe the zfs config and rebuild it from scratch, possibly using rainemu/current.
D.
Added by
(anonymous)
on
Apr 21, 2010 10:34 AM
Interesting. I had mentioned earlier that the 4/6/2010 rainemu utilities (zfs, zpool) seemed to be hanging. I haven't messed with the box in about a week, and when I tried it last night, the utils worked fine. Hrm.
So, rebooted with the 4/6/2010 rainemu build in place, and everything seems to have come up fine. Doing some stress tests now - moving large files about between datasets, and so far so good. zpool status wants to upgrade the on-disk format which I have not done yet. I'll let it do a scrub, then upgrade.
D.
So, rebooted with the 4/6/2010 rainemu build in place, and everything seems to have come up fine. Doing some stress tests now - moving large files about between datasets, and so far so good. zpool status wants to upgrade the on-disk format which I have not done yet. I'll let it do a scrub, then upgrade.
D.
Added by
Seth Heeren
on
Apr 21, 2010 12:17 PM
Issue state:
unconfirmed → open
Severity:
Medium → Important
Responsible manager:
(UNASSIGNED) → sgheeren
I'm so sorry. We must have had a reliability issue with the mailserver/website setup, as I literally just received the first message on this bug issue. Note that I'm normally to receive all bug reports/updates. Oops.
Soooo. I had the recollection with the put_nvlist issue as well. However, that fix was merged long time ago and I can't imagine Emmanuel regressing that particular bug himself (seeing he/we fixed that over a longer period of time... :)).
The syslog messages mentioning put_nvlist are debug messages only and can be safely ignored (all it says is that put_nvlist is requesting a larger buffer size. That request obviously succeeds if zfs and the userland tools proceed after that).
On to the real issue: you mention 'Debian Stable/Lenny zfs-fuse 0.6.0' - do you happen to know which revision is packaged? The 0.6.0 tag in git contains known stability issues - see http://zfs-fuse.net/news/fixes-released.
The rainemu build is considered unstable (!). I have pretty good results with a semi-stable revision myself: 0c54eb8d4a
If you didn't upgrade the pool to a bleeding edge version, I suggest you could try to bisect the bug (by repeatedly checking out more recent versions until the bug appears). Newtons approximation works best (pick the middle revision each time to partition the revision space recursively). Well, I hope you get my idea.
It appears something is breaking the dataset iteration code out of it's loop. I strongly suspect you could probably mount the 'hidden' datasets using 'zfs mount' explicitely.
You might try variations on the zfs list command to see whether the problem is 'kernel' (i.e.: daemon) side or not. What happens if you say 'zfs list -Ho name'? The same datasets or more? Less?
Soooo. I had the recollection with the put_nvlist issue as well. However, that fix was merged long time ago and I can't imagine Emmanuel regressing that particular bug himself (seeing he/we fixed that over a longer period of time... :)).
The syslog messages mentioning put_nvlist are debug messages only and can be safely ignored (all it says is that put_nvlist is requesting a larger buffer size. That request obviously succeeds if zfs and the userland tools proceed after that).
On to the real issue: you mention 'Debian Stable/Lenny zfs-fuse 0.6.0' - do you happen to know which revision is packaged? The 0.6.0 tag in git contains known stability issues - see http://zfs-fuse.net/news/fixes-released.
The rainemu build is considered unstable (!). I have pretty good results with a semi-stable revision myself: 0c54eb8d4a
If you didn't upgrade the pool to a bleeding edge version, I suggest you could try to bisect the bug (by repeatedly checking out more recent versions until the bug appears). Newtons approximation works best (pick the middle revision each time to partition the revision space recursively). Well, I hope you get my idea.
It appears something is breaking the dataset iteration code out of it's loop. I strongly suspect you could probably mount the 'hidden' datasets using 'zfs mount' explicitely.
You might try variations on the zfs list command to see whether the problem is 'kernel' (i.e.: daemon) side or not. What happens if you say 'zfs list -Ho name'? The same datasets or more? Less?
Added by
Seth Heeren
on
Apr 21, 2010 12:22 PM
Oh, and I'm most interested in a backtrace of the daemon and the zpool/zfs binary when they are hung (as you mention with rainemu master?)
One way you could do this is
1. be sure to build _and_ install debug=2:
# sudo scons install debug=2
2. launch and reproduce the hang
3. in another terminal, use gdb to produce a backtrace, e.g.
# gdb --pid $(pgrep zfs-fuse)
[...]
gdb> thread apply all bt full
(use 'set pagination on' or 'set logging on' etc. if you wish)
One way you could do this is
1. be sure to build _and_ install debug=2:
# sudo scons install debug=2
2. launch and reproduce the hang
3. in another terminal, use gdb to produce a backtrace, e.g.
# gdb --pid $(pgrep zfs-fuse)
[...]
gdb> thread apply all bt full
(use 'set pagination on' or 'set logging on' etc. if you wish)
Added by
Dean Carpenter
on
Apr 21, 2010 03:29 PM
Seth - thanks for coming back, not that you were gone :)
OK, there didn't seem to be any (obvious) problems with the 4/6/2010 rainemu install in place. I added a bunch of large files, moved some around, copied others and so on. Nothing jumped out, and everything seems to be working OK. I have NOT upgraded the pool - definitely wanted to wait on that, and I'm taking your caveat on the stability of rainemu to heart ...
I just re-enabled the original 0.6.0 version and rebooted. It still has the same problem - missing datasets. Your suspicion about a problem in the loop iteration code is probably correct since I was just now able to mount the missing datasets with explicit "zfs mount stuff/dataset" commands. No log errors, and all the data appears to be normally accessible. I used the 0.6.0 version of zfs.
Interestingly, a "/sbin/zfs mount" shows all the datasets mounted, yet a "/sbin/zfs list" and "/sbin/zfs list -Ho name" only show the ones that mounted at startup, not the manually mounted ones.
Currently, Debian only has 0.6.0 available in thes standard repositories, though it seems that there were at least one or two critical fixes put in place for the testing and unstable releases. I'm betting that the unstable release has the two critical hotfixes you mention in the fixes-released page.
# apt-cache policy zfs-fuse
zfs-fuse:
Installed: 0.6.0-1
Candidate: 0.6.0+critical20100301-1
Version table:
0.6.0+critical20100301-2 0
300 http://http.us.debian.org unstable/main Packages
0.6.0+critical20100301-1 0
400 http://http.us.debian.org testing/main Packages
*** 0.6.0-1 0
100 /var/lib/dpkg/status
I think next step here would be to upgrade to the 0.6.0+critical20100301-2 version and see how it goes. If that still displays the same problems, then I'll start pulling down the more recent git codedrops as you described.
Hrm - last interesting note. With the 0.6.0-1 installed and running right now, the 0.6.0 zfs/zpool binaries work fine, but the rainemu ones hang. Ctrl-C breaks them out. But, as I said before, initially the rainemu ones were hanging, and then after some days they seemed to be working fine. Wonder what was timing out ?
More as I get it ... Thanks Seth.
D.
OK, there didn't seem to be any (obvious) problems with the 4/6/2010 rainemu install in place. I added a bunch of large files, moved some around, copied others and so on. Nothing jumped out, and everything seems to be working OK. I have NOT upgraded the pool - definitely wanted to wait on that, and I'm taking your caveat on the stability of rainemu to heart ...
I just re-enabled the original 0.6.0 version and rebooted. It still has the same problem - missing datasets. Your suspicion about a problem in the loop iteration code is probably correct since I was just now able to mount the missing datasets with explicit "zfs mount stuff/dataset" commands. No log errors, and all the data appears to be normally accessible. I used the 0.6.0 version of zfs.
Interestingly, a "/sbin/zfs mount" shows all the datasets mounted, yet a "/sbin/zfs list" and "/sbin/zfs list -Ho name" only show the ones that mounted at startup, not the manually mounted ones.
Currently, Debian only has 0.6.0 available in thes standard repositories, though it seems that there were at least one or two critical fixes put in place for the testing and unstable releases. I'm betting that the unstable release has the two critical hotfixes you mention in the fixes-released page.
# apt-cache policy zfs-fuse
zfs-fuse:
Installed: 0.6.0-1
Candidate: 0.6.0+critical20100301-1
Version table:
0.6.0+critical20100301-2 0
300 http://http.us.debian.org unstable/main Packages
0.6.0+critical20100301-1 0
400 http://http.us.debian.org testing/main Packages
*** 0.6.0-1 0
100 /var/lib/dpkg/status
I think next step here would be to upgrade to the 0.6.0+critical20100301-2 version and see how it goes. If that still displays the same problems, then I'll start pulling down the more recent git codedrops as you described.
Hrm - last interesting note. With the 0.6.0-1 installed and running right now, the 0.6.0 zfs/zpool binaries work fine, but the rainemu ones hang. Ctrl-C breaks them out. But, as I said before, initially the rainemu ones were hanging, and then after some days they seemed to be working fine. Wonder what was timing out ?
More as I get it ... Thanks Seth.
D.
Added by
Seth Heeren
on
Apr 21, 2010 03:37 PM
go with 0.6.0+critical20100301-?
It's a shame we don't have the packaging people closely-knit to the project. I think this rss may be of interest to you: http://packages.qa.debian.org/z/zfs-fuse/news.rss20.xml
Apparently there might be a ...-2 by now (since 8th of april)
If you have the hangs (that you can Ctrl-C out of) would you be willing to make a back trace of some sort?
PS.: And just a hint, don't mix builds of zfs-fuse and zfs/zpool :) Normally they will _not_ be compatible (using statically linked libzfs that may be silently incompatible ABI)
It's a shame we don't have the packaging people closely-knit to the project. I think this rss may be of interest to you: http://packages.qa.debian.org/z/zfs-fuse/news.rss20.xml
Apparently there might be a ...-2 by now (since 8th of april)
If you have the hangs (that you can Ctrl-C out of) would you be willing to make a back trace of some sort?
PS.: And just a hint, don't mix builds of zfs-fuse and zfs/zpool :) Normally they will _not_ be compatible (using statically linked libzfs that may be silently incompatible ABI)
Added by
Dean Carpenter
on
Apr 21, 2010 04:52 PM
I installed the 0.6.0+critical20100301-2 version from the unstable release. Came right up, all datasets are there. So far so good :)
Interesting ... df -h seems to be a little confused compared to zfs list
# df -h
Filesystem Size Used Avail Use% Mounted on
stuff 400G 43K 400G 1% /stuff
stuff/Backups 576G 176G 400G 31% /stuff/Backups
stuff/Downloads 401G 1.2G 400G 1% /stuff/Downloads
stuff/Images 411G 11G 400G 3% /stuff/Images
stuff/Software 757G 357G 400G 48% /stuff/Software
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
stuff 3.17T 400G 42.6K /stuff
stuff/Backups 176G 400G 176G /stuff/Backups
stuff/Downloads 1.15G 400G 1.15G /stuff/Downloads
stuff/Images 10.9G 400G 10.9G /stuff/Images
stuff/Software 356G 400G 356G /stuff/Software
df -h doesn't seem to see the Size correctly. The Used column is correct. I'll do some more stress tests, see if I can get it to stumble, but so far things look good.
D.
Interesting ... df -h seems to be a little confused compared to zfs list
# df -h
Filesystem Size Used Avail Use% Mounted on
stuff 400G 43K 400G 1% /stuff
stuff/Backups 576G 176G 400G 31% /stuff/Backups
stuff/Downloads 401G 1.2G 400G 1% /stuff/Downloads
stuff/Images 411G 11G 400G 3% /stuff/Images
stuff/Software 757G 357G 400G 48% /stuff/Software
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
stuff 3.17T 400G 42.6K /stuff
stuff/Backups 176G 400G 176G /stuff/Backups
stuff/Downloads 1.15G 400G 1.15G /stuff/Downloads
stuff/Images 10.9G 400G 10.9G /stuff/Images
stuff/Software 356G 400G 356G /stuff/Software
df -h doesn't seem to see the Size correctly. The Used column is correct. I'll do some more stress tests, see if I can get it to stumble, but so far things look good.
D.
Added by
Seth Heeren
on
Apr 22, 2010 02:50 AM
Hi good to hear.
df anomalies are known and explainable. Search the google list for zfs-fuse to find some of the explanations.
As you can see below the thing seems to be that du reports the volume size as Used+Avail. Now Avail is the 'global' pool remaining space (which is only an estimate of course) so the 'size' varies randomly. Note that things get spectacular once you throw in clones and dedup.
I seem to remember that there isn't mucht that can be done. I don't remember whether that was a du limitation, fuse, or zfs-fuse.
FWIW, here's du -h on an OpenSolaris box near me:
Filesystem size used avail capacity Mounted on
rpool/ROOT/opensolaris
98G 5.6G 88G 6% /
/devices 0K 0K 0K 0% /devices
/dev 0K 0K 0K 0% /dev
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 1.7G 404K 1.7G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap1.so.1
93G 5.6G 88G 6% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
swap 1.7G 12K 1.7G 1% /tmp
swap 1.7G 88K 1.7G 1% /var/run
rpool/export 98G 23K 88G 1% /export
rpool/export/home 98G 23K 88G 1% /export/home
rpool/export/home/sehe
98G 13M 88G 1% /export/home/sehe
rpool 98G 78K 88G 1% /rpool
/export/home/sehe 88G 13M 88G 1% /home/sehe
bbs 1.3T 26K 582G 1% /bbs
bbs/backup 1.3T 30K 582G 1% /bbs/backup
bbs/backup/jauntyhome
1.3T 8.9G 582G 2% /bbs/backup/jauntyhome
bbs/backup/kohler 1.3T 23K 582G 1% /bbs/backup/kohler
bbs/backup/kohler/raw
1.3T 22K 582G 1% /bbs/backup/kohler/raw
bbs/backup/laptop 1.3T 22K 582G 1% /bbs/backup/laptop
bbs/backup/mubi_sendfiles
1.3T 45G 582G 8% /bbs/backup/mubi_sendfiles
bbs/backup/sdu 1.3T 2.8G 582G 1% /bbs/backup/sdu
bbs/data 1.3T 24K 582G 1% /bbs/data
bbs/data/freedb 1.3T 3.8G 582G 1% /bbs/data/freedb
bbs/data/muziek_etc 1.3T 723M 582G 1% /bbs/data/muziek_etc
bbs/iscsi 1.3T 21K 582G 1% /bbs/iscsi
bbs/media 1.3T 31K 582G 1% /bbs/media
bbs/media/AUDIO 1.3T 11G 582G 2% /bbs/media/AUDIO
bbs/media/Afbeeldingen
1.3T 12G 582G 3% /bbs/media/Afbeeldingen
bbs/media/Music 1.3T 21G 582G 4% /bbs/media/Music
bbs/media/Ripit 1.3T 291G 582G 34% /bbs/media/Ripit
bbs/media/RipitFlac 1.3T 237G 582G 29% /bbs/media/RipitFlac
bbs/media/RipitMp3 1.3T 102G 582G 15% /bbs/media/RipitMp3
bbs/media/Video 1.3T 19G 582G 4% /bbs/media/Video
On my linux box it is
root@karmic:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ssd-root 9.9G 8.5G 952M 91% /
udev 4.0G 368K 4.0G 1% /dev
none 4.0G 840K 4.0G 1% /dev/shm
none 4.0G 3.7M 4.0G 1% /tmp
none 4.0G 408K 4.0G 1% /var/run
none 4.0G 0 4.0G 0% /var/lock
none 4.0G 0 4.0G 0% /lib/init/rw
/dev/mapper/ssd-home 24G 16G 6.9G 70% /home
none 4.0G 908K 4.0G 1% /var/tmp
desktop/home 12G 31K 12G 1% /MUBI/home
desktop 12G 23K 12G 1% /MUBI/zfs
desktop/home/marlies 13G 1.3G 12G 11% /MUBI/home/marlies
desktop/home/marlies/.wine
12G 180M 12G 2% /MUBI/home/marlies/.wine
desktop/home/marlies/Photos
13G 676M 12G 6% /MUBI/home/marlies/Photos
desktop/home/sehe 12G 179M 12G 2% /MUBI/home/sehe
desktop/PersoonlijkeBestanden
12G 207M 12G 2% /MUBI/zfs/PersoonlijkeBestanden
desktop/PersoonlijkeBestanden/BUDGETTEER
12G 31M 12G 1% /MUBI/zfs/PersoonlijkeBestanden/BUDGETTEER
desktop/PersoonlijkeBestanden/Belastingdienst
12G 35M 12G 1% /MUBI/zfs/PersoonlijkeBestanden/Belastingdienst
desktop/PersoonlijkeBestanden/Email
12G 421M 12G 4% /MUBI/zfs/PersoonlijkeBestanden/Email
desktop/PersoonlijkeBestanden/FORMULIEREN
12G 566K 12G 1% /MUBI/zfs/PersoonlijkeBestanden/FORMULIEREN
desktop/PersoonlijkeBestanden/KLANTBESTAND
12G 150M 12G 2% /MUBI/zfs/PersoonlijkeBestanden/KLANTBESTAND
desktop/PersoonlijkeBestanden/Mubi
14G 2.5G 12G 18% /MUBI/zfs/PersoonlijkeBestanden/Mubi
desktop/PersoonlijkeBestanden/MyPictures
20G 7.9G 12G 41% /MUBI/zfs/PersoonlijkeBestanden/MyPictures
desktop/PersoonlijkeBestanden/OutlookMail
12G 51K 12G 1% /MUBI/zfs/PersoonlijkeBestanden/OutlookMail
desktop/PersoonlijkeBestanden/VTLBBerekeningen
12G 20K 12G 1% /MUBI/zfs/PersoonlijkeBestanden/VTLBBerekeningen
desktop/PersoonlijkeBestanden/boekhoudingen
12G 95M 12G 1% /MUBI/zfs/PersoonlijkeBestanden/boekhoudingen
df anomalies are known and explainable. Search the google list for zfs-fuse to find some of the explanations.
As you can see below the thing seems to be that du reports the volume size as Used+Avail. Now Avail is the 'global' pool remaining space (which is only an estimate of course) so the 'size' varies randomly. Note that things get spectacular once you throw in clones and dedup.
I seem to remember that there isn't mucht that can be done. I don't remember whether that was a du limitation, fuse, or zfs-fuse.
FWIW, here's du -h on an OpenSolaris box near me:
Filesystem size used avail capacity Mounted on
rpool/ROOT/opensolaris
98G 5.6G 88G 6% /
/devices 0K 0K 0K 0% /devices
/dev 0K 0K 0K 0% /dev
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 1.7G 404K 1.7G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
sharefs 0K 0K 0K 0% /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap1.so.1
93G 5.6G 88G 6% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
swap 1.7G 12K 1.7G 1% /tmp
swap 1.7G 88K 1.7G 1% /var/run
rpool/export 98G 23K 88G 1% /export
rpool/export/home 98G 23K 88G 1% /export/home
rpool/export/home/sehe
98G 13M 88G 1% /export/home/sehe
rpool 98G 78K 88G 1% /rpool
/export/home/sehe 88G 13M 88G 1% /home/sehe
bbs 1.3T 26K 582G 1% /bbs
bbs/backup 1.3T 30K 582G 1% /bbs/backup
bbs/backup/jauntyhome
1.3T 8.9G 582G 2% /bbs/backup/jauntyhome
bbs/backup/kohler 1.3T 23K 582G 1% /bbs/backup/kohler
bbs/backup/kohler/raw
1.3T 22K 582G 1% /bbs/backup/kohler/raw
bbs/backup/laptop 1.3T 22K 582G 1% /bbs/backup/laptop
bbs/backup/mubi_sendfiles
1.3T 45G 582G 8% /bbs/backup/mubi_sendfiles
bbs/backup/sdu 1.3T 2.8G 582G 1% /bbs/backup/sdu
bbs/data 1.3T 24K 582G 1% /bbs/data
bbs/data/freedb 1.3T 3.8G 582G 1% /bbs/data/freedb
bbs/data/muziek_etc 1.3T 723M 582G 1% /bbs/data/muziek_etc
bbs/iscsi 1.3T 21K 582G 1% /bbs/iscsi
bbs/media 1.3T 31K 582G 1% /bbs/media
bbs/media/AUDIO 1.3T 11G 582G 2% /bbs/media/AUDIO
bbs/media/Afbeeldingen
1.3T 12G 582G 3% /bbs/media/Afbeeldingen
bbs/media/Music 1.3T 21G 582G 4% /bbs/media/Music
bbs/media/Ripit 1.3T 291G 582G 34% /bbs/media/Ripit
bbs/media/RipitFlac 1.3T 237G 582G 29% /bbs/media/RipitFlac
bbs/media/RipitMp3 1.3T 102G 582G 15% /bbs/media/RipitMp3
bbs/media/Video 1.3T 19G 582G 4% /bbs/media/Video
On my linux box it is
root@karmic:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ssd-root 9.9G 8.5G 952M 91% /
udev 4.0G 368K 4.0G 1% /dev
none 4.0G 840K 4.0G 1% /dev/shm
none 4.0G 3.7M 4.0G 1% /tmp
none 4.0G 408K 4.0G 1% /var/run
none 4.0G 0 4.0G 0% /var/lock
none 4.0G 0 4.0G 0% /lib/init/rw
/dev/mapper/ssd-home 24G 16G 6.9G 70% /home
none 4.0G 908K 4.0G 1% /var/tmp
desktop/home 12G 31K 12G 1% /MUBI/home
desktop 12G 23K 12G 1% /MUBI/zfs
desktop/home/marlies 13G 1.3G 12G 11% /MUBI/home/marlies
desktop/home/marlies/.wine
12G 180M 12G 2% /MUBI/home/marlies/.wine
desktop/home/marlies/Photos
13G 676M 12G 6% /MUBI/home/marlies/Photos
desktop/home/sehe 12G 179M 12G 2% /MUBI/home/sehe
desktop/PersoonlijkeBestanden
12G 207M 12G 2% /MUBI/zfs/PersoonlijkeBestanden
desktop/PersoonlijkeBestanden/BUDGETTEER
12G 31M 12G 1% /MUBI/zfs/PersoonlijkeBestanden/BUDGETTEER
desktop/PersoonlijkeBestanden/Belastingdienst
12G 35M 12G 1% /MUBI/zfs/PersoonlijkeBestanden/Belastingdienst
desktop/PersoonlijkeBestanden/Email
12G 421M 12G 4% /MUBI/zfs/PersoonlijkeBestanden/Email
desktop/PersoonlijkeBestanden/FORMULIEREN
12G 566K 12G 1% /MUBI/zfs/PersoonlijkeBestanden/FORMULIEREN
desktop/PersoonlijkeBestanden/KLANTBESTAND
12G 150M 12G 2% /MUBI/zfs/PersoonlijkeBestanden/KLANTBESTAND
desktop/PersoonlijkeBestanden/Mubi
14G 2.5G 12G 18% /MUBI/zfs/PersoonlijkeBestanden/Mubi
desktop/PersoonlijkeBestanden/MyPictures
20G 7.9G 12G 41% /MUBI/zfs/PersoonlijkeBestanden/MyPictures
desktop/PersoonlijkeBestanden/OutlookMail
12G 51K 12G 1% /MUBI/zfs/PersoonlijkeBestanden/OutlookMail
desktop/PersoonlijkeBestanden/VTLBBerekeningen
12G 20K 12G 1% /MUBI/zfs/PersoonlijkeBestanden/VTLBBerekeningen
desktop/PersoonlijkeBestanden/boekhoudingen
12G 95M 12G 1% /MUBI/zfs/PersoonlijkeBestanden/boekhoudingen
Added by
Seth Heeren
on
May 06, 2010 05:57 AM
closing
Added by
Seth Heeren
on
May 06, 2010 06:55 AM
closing
Added by
Seth Heeren
on
May 22, 2010 05:24 AM
Issue state:
open → resolved
Trying to close - again -
