GNU bug report logs - #77086
Filesystems not unmounted on reboot

Previous Next

To reply to this bug, email your comments to 77086 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ian Eure <ian <at> retrospec.tv>
To: bug-guix <at> gnu.org
Subject: Filesystems not unmounted on reboot
Date: Mon, 17 Mar 2025 14:09:09 -0700

Starting recently (last 2-3 weeks), my Guix System machines have
failed to unmount filesystems when restarted.  ex. inside an Emacs
EXWM X11 session, running `sudo reboot' in a shell causes the 
system
to perform a lengthy fsck on next boot, saying "recovering 
journal."

Others on IRC have noticed this as well.  Some think it’s 
correlated
to running `sudo guix system reconfigure', but I observe it 
whether
I’ve reconfigured or not.

I’m using LUKS1 whole-disk encryption.

Here’s the `guix system describe' output from a laptop (ThinkPad 
X280)
which is exhibiting the problem:

  Generation 46	Mar 17 2025 07:22:11	(current)
    file name: /var/guix/profiles/system-46-link
    canonical file name:
    /gnu/store/zjk9w5jwdrlsc6kr8s4iq3317ys17shf-system
    label: GNU with Linux 6.13.7
    bootloader: grub-efi
    root device: /dev/mapper/cryptroot
    kernel:
    /gnu/store/q8vjcaaykh0p7769xk1rz29di1axry07-linux-6.13.7/bzImage
    channels:
      guix:
        repository URL:
        https://git.savannah.gnu.org/git/guix.git
        branch: master
        commit: 98be320183579b3d09cf4059e86a9781485628b4
      nonguix:
        repository URL: https://gitlab.com/nonguix/nonguix
        branch: master
        commit: fa416ebdf9e4d5c3b9676ded8829c5875bcf4f0e
      atomized:
        repository URL:
        https://codeberg.org/ieure/atomized-guix.git
        branch: main
        commit: bd4ef7fad637b7213e1a96885b61a6ca471eeb0b
    configuration file:
    /gnu/store/dx2hzsqh47iz2pbiwiqsd8w7lpc6f4jy-configuration.scm

-- Ian

Message #8 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 77086 <at> debbugs.gnu.org
Subject: Filesystems not unmounted on reboot
Date: Tue, 18 Mar 2025 06:43:02 +0100

I have also seen this on a netbook where I have never used "guix 
system reconfigure"; I only ever use "guix deploy" from another 
machine to upgrade that system.

This is the version of Guix used:

 guix 9212459
   repository URL: https://codeberg.org/guix/guix-mirror
   branch: master
   commit: 92124591eedf27e988c84f75acd4b4d99ff43122
 rosenthal 5172fac
   repository URL: https://codeberg.org/hako/rosenthal.git
   branch: trunk
   commit: 5172fac369025bc98cbfb925b728b2cae20ac2c5

The kernel is linux-libre-lts.

-- 
Ricardo

Message #13 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> ditigal.xyz>
To: 77086 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Felix Lechner <felix.lechner <at> lease-up.com>, Ian Eure <ian <at> retrospec.tv>
Subject: Re: Filesystems not unmounted on reboot
Date: Fri, 21 Mar 2025 14:43:50 +0100

[Message part 1 (text/plain, inline)]

Hello guys,

so I started looking into this a bit (not promising any results though),
and I can pretty confidently say that there is indeed an issue in the
unmount.

I have created a VM system, tried rebooting a few times and it was fine,
however then I tried reconfiguring, and for that run I got an error upon
reboot. Not only that, I can't boot it anymore :) the filesystem got
corrupted in a way that prevents boots. Welp.

I can't really say at current moment what is causing this, but the
problem is that the root device is busy so it can't be unmounted.
I have a hypothesis given the log I see: that the root filesystem
is being unmounted first rather than last like it should be.
Could the reconfigure throw shepherd off? I am also CCing Ludovic,
I hope he won't mind.

I would like to also point out an e-mail in the
guix-devel sent recently that someone got a timer service running after
reconfigure, but not after reboot, where after reboot the timer module
is not imported by default unless put to service's modules, but after
reconfigure it works, so this leaves me with yet another point for the
impression that shepherd behaves differently on reboot as opposed to
reconfigure reload.

I will try digging more, but I am not that knowledgeable about
shepherd yet, so it will take longer time.

I am attaching both log (starting after reboot command)
and the configuration used for the vm.

[reboot_log (text/plain, attachment)]

[simple.scm (text/plain, attachment)]

[Message part 4 (text/plain, inline)]

Regards,
Rutherther

Message #16 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> ditigal.xyz>
To: 77086 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Felix Lechner <felix.lechner <at> lease-up.com>, Ian Eure <ian <at> retrospec.tv>
Subject: Re: Filesystems not unmounted on reboot
Date: Fri, 21 Mar 2025 17:50:51 +0100

[Message part 1 (text/plain, inline)]

Hello guys,

I am writing with an update, because not everything I wrote in the first
e-mail was true exactly. Please let me know if updates distrub you and I
won't be CCing you again.

"Rutherther" <rutherther <at> ditigal.xyz> writes:

>
> I have created a VM system, tried rebooting a few times and it was fine,
> however then I tried reconfiguring, and for that run I got an error upon
> reboot. Not only that, I can't boot it anymore :) the filesystem got
> corrupted in a way that prevents boots. Welp.

This wasn't the case, I just haven't realized the correct partition to
boot from is /dev/vda2, not /dev/vda1, I copied the image from guix repo
and haven't realized it's wrong (I suppose it haven't used efi before,
but switched to it?). Since the images get the FS replaced by custom
partitions, this error was visible only after reconfigure. After
changing it to /dev/vda2 I am able to boot, but fsck is ran to repair
damage done.

>
> I am attaching both log (starting after reboot command)
> and the configuration used for the vm.
>

Since this was the first reconfigure after the system obtained by guix
system image, the log isn't exactly accurate to what would happen on
subsequent reconfigures.

The previous time the error was that / is busy. Maybe it was because of a
change in the filesystem services from the initial image to custom
config with `file-systems`
While that is definitely an issue as well, it might not be completely
related to the issue reported here by others. (imo there should be a way
for cases where the user is making big changes like changing file
systems to tell reconfigure to not apply it to the running system, to
only apply the bootloader and switch only after boot)

I am attaching log of reboot/halt after subsequent reconfigures where
the issue is manifested a bit differently. It is not / that would be
busy, but /run/user that is busy. I still think it could be because of
wrong order of stopping services, /run/user/0 is unmounted last,
but root file system tries to be unmounted prior to that.

Note that this doesn't happen without a reconfigure, I get this behavior
only after a reconfigure. The reconfigure can happen with no changes to
the config nor guix instance ran. I get this behavior consistently
on every reconfigure ran!

[  202.675565] shepherd[1]: Ignoring error while stopping root-file-system: (system-error "umount" "~S: ~A" ("/run/user" "Device or resource busy") (16))

Regards,
and apologies for two messages when it could've been one if I paid
more attention.
Rutherther

[reboot_log (text/plain, attachment)]

Message #19 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> ditigal.xyz>
To: 77086 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Felix Lechner <felix.lechner <at> lease-up.com>, Ian Eure <ian <at> retrospec.tv>
Subject: Re: Filesystems not unmounted on reboot
Date: Fri, 21 Mar 2025 19:57:31 +0100

Hello,

I was testing on commmit Ian sent to reproduce, and now moved to
newest guix, seems to have been solved (at least the specific error I
was getting),
possibly by shepherd 1.0.3 update.

With newest guix both the first reconfigure of guix system image
and subsequent ones are fine.

Let me know if you're still experiencing this issue after updating
and I might try harder to reproduce if I got a different issue on first
try. I am afraid this will be hard to debug on real hw as you don't
really get the log in /var/log/messages for shutting down the system, I
had to get it through a serial line via stdout of qemu.

Also I was able to reproduce the issue on the older
guix just by running the shepherd services upgrade scm script,
no need for full reconfigure, this shows that something has gone
wrong when shepherd was reloading the services. Do we have some kind of
a test for this in guix / shepherd so it can't happen anymore in the future?

Message #22 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Rutherther via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: 77086 <at> debbugs.gnu.org
Subject: Re: bug#77086: Filesystems not unmounted on reboot
Date: Sat, 22 Mar 2025 00:50:41 -0400

On Fri, Mar 21, 2025 at 05:50:51PM +0100, Rutherther via Bug reports for GNU Guix wrote:
> I am writing with an update, because not everything I wrote in the first
> e-mail was true exactly. Please let me know if updates distrub you and I
> won't be CCing you again.

Please don't hesitate to send these kinds of investigations again! It's
helpful.

Message #28 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Ian Eure <ian <at> retrospec.tv>
To: 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Fri, 28 Mar 2025 08:40:01 -0700

I’m still seeing this problem on at least one machine.

Here’s the system configuration: https://paste.debian.net/1365931/

It depends on my personal channel: 
https://codeberg.org/ieure/atomized-guix

I captured the state of the machine just now, then ran `sync' and 
`sudo reboot'.  It fsck’d on boot.

Script started on 2025-03-28 08:32:12-07:00 [TERM="dumb" 
TTY="/dev/pts/1" COLUMNS="191" LINES="39"]
isot0pe!ieure:~$ w
08:32:13 up 2 days, 19:43,  2 users,  load average: 1.23, 0.34, 
0.12
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
ieure    seat0     Tue13    0.00s  0.00s  0.03s 
/gnu/store/wpdr1pfczc5a5yn7ii80qjgcgdv42jr2-gdm-46.2/libexec/gdm-x-session 
--register-session --run-script 
/gnu/store/mmw3bf5wachcd96i84770yzf
ieure    :0        Tue13   ?xdm?   1:59   0.03s 
/gnu/store/wpdr1pfczc5a5yn7ii80qjgcgdv42jr2-gdm-46.2/libexec/gdm-x-session 
--register-session --run-script 
/gnu/store/mmw3bf5wachcd96i84770yzf
isot0pe!ieure:~$ guix describe
Generation 62	Mar 25 2025 11:59:36	(current)
 guix dbef60e
   repository URL: https://git.savannah.gnu.org/git/guix.git
   branch: master
   commit: dbef60edb356246855ad6749936ee511fc1a9b4b
 atomized f6fba0e
   repository URL: https://codeberg.org/ieure/atomized-guix.git
   branch: main
   commit: f6fba0e2b207b69ab093f2be47b306242c748ab8
 nonguix a96e245
   repository URL: https://gitlab.com/nonguix/nonguix
   branch: master
   commit: a96e2451bda5aaf9b48339edee392c6a3017d730
isot0pe!ieure:~$ guix system describe
Generation 49	Mar 25 2025 12:05:36	(current)
 file name: /var/guix/profiles/system-49-link
 canonical file name: 
 /gnu/store/wkzxc2nbiydwr42v8a0702xl128n65b0-system
 label: GNU with Linux 6.13.7
 bootloader: grub-efi
 root device: /dev/mapper/cryptroot
 kernel: 
 /gnu/store/5mac890q3yhfi6pvxdl8mmpyngmra5s0-linux-6.13.7/bzImage
 channels:
   guix:
     repository URL: https://git.savannah.gnu.org/git/guix.git
     branch: master
     commit: dbef60edb356246855ad6749936ee511fc1a9b4b
   atomized:
     repository URL: https://codeberg.org/ieure/atomized-guix.git
     branch: main
     commit: f6fba0e2b207b69ab093f2be47b306242c748ab8
   nonguix:
     repository URL: https://gitlab.com/nonguix/nonguix
     branch: master
     commit: a96e2451bda5aaf9b48339edee392c6a3017d730
 configuration file: 
 /gnu/store/dhmn4825hvfvkbfbvb7xg6rkhdcj38ii-configuration.scm
isot0pe!ieure:~$ exit

Script done on 2025-03-28 08:32:30-07:00 [COMMAND_EXIT_CODE="0"]

-- Ian

Message #31 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Ian Eure <ian <at> retrospec.tv>
To: Rutherther <rutherther <at> ditigal.xyz>
Cc: Ricardo Wurmus <rekado <at> elephly.net>,
 Ludovic Courtès <ludo <at> gnu.org>,
 Felix Lechner <felix.lechner <at> lease-up.com>, 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Sat, 29 Mar 2025 11:10:15 -0700

Hi all,

Here’s a less dead link to the config for my system exhibiting the 
problem: https://paste.debian.net/1366307/

I set it to never expire this time.

The FS stuff is very pedestrian, it’s more or less what the 
graphical installer generates, since that’s how I installed that 
system.

Thanks,
 -- Ian

Message #34 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Rutherther <rutherther <at> ditigal.xyz>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, Ian Eure <ian <at> retrospec.tv>,
 Felix Lechner <felix.lechner <at> lease-up.com>, 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Tue, 01 Apr 2025 09:25:09 +0200

Hi,

Rutherther <rutherther <at> ditigal.xyz> skribis:

> I was testing on commmit Ian sent to reproduce, and now moved to
> newest guix, seems to have been solved (at least the specific error I
> was getting),
> possibly by shepherd 1.0.3 update.
>
> With newest guix both the first reconfigure of guix system image
> and subsequent ones are fine.

I believe I’m still experiencing it on my laptop (but it’s harmless, I
only briefly see e2fsck saying “recovering journal” at boot time), but
of course, not the slightest clue in /var/log/messages.

> Let me know if you're still experiencing this issue after updating
> and I might try harder to reproduce if I got a different issue on first
> try. I am afraid this will be hard to debug on real hw as you don't
> really get the log in /var/log/messages for shutting down the system, I
> had to get it through a serial line via stdout of qemu.
>
> Also I was able to reproduce the issue on the older
> guix just by running the shepherd services upgrade scm script,
> no need for full reconfigure, this shows that something has gone
> wrong when shepherd was reloading the services. Do we have some kind of
> a test for this in guix / shepherd so it can't happen anymore in the future?

There’s a system test, “root-unmount” in (gnu tests base), but it
succeeds: <https://ci.guix.gnu.org/build/9790935/details>.

Perhaps the problem only shows up with more complex system configs?
My root partition is on a LUKS device, but I think the problem is more
something like EBUSY upon ‘umount’ due to stale processes.

Thanks,
Ludo’.

Message #37 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> ditigal.xyz>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, Ian Eure <ian <at> retrospec.tv>,
 Felix Lechner <felix.lechner <at> lease-up.com>, 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Wed, 02 Apr 2025 17:31:43 +0200

Hello Ludo,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi,
>
> Rutherther <rutherther <at> ditigal.xyz> skribis:
>
>> I was testing on commmit Ian sent to reproduce, and now moved to
>> newest guix, seems to have been solved (at least the specific error I
>> was getting),
>> possibly by shepherd 1.0.3 update.
>>
>> With newest guix both the first reconfigure of guix system image
>> and subsequent ones are fine.
>
> I believe I’m still experiencing it on my laptop (but it’s harmless, I
> only briefly see e2fsck saying “recovering journal” at boot time), but
> of course, not the slightest clue in /var/log/messages.

To make it more clear, what I was able to get in the VM was full disk
recovery that was clearing wrong inodes, every time and I even got disk
corruptions for the files produced during reconfigure (which was the
last thing I did before reboot). So I am not sure if this is the same
thing, do you know, is this one line printed only when there is
something wrong with the journal, or is it printed every time like sort
of a 'welcome' message?

>
>> Let me know if you're still experiencing this issue after updating
>> and I might try harder to reproduce if I got a different issue on first
>> try. I am afraid this will be hard to debug on real hw as you don't
>> really get the log in /var/log/messages for shutting down the system, I
>> had to get it through a serial line via stdout of qemu.
>>
>> Also I was able to reproduce the issue on the older
>> guix just by running the shepherd services upgrade scm script,
>> no need for full reconfigure, this shows that something has gone
>> wrong when shepherd was reloading the services. Do we have some kind of
>> a test for this in guix / shepherd so it can't happen anymore in the future?
>
> There’s a system test, “root-unmount” in (gnu tests base), but it
> succeeds: <https://ci.guix.gnu.org/build/9790935/details>.

Thanks for pointing out this test, good to know about it, although it's
not exactly what I had in mind. I was able to reproduce previously by
running the upgrade-shepherd-services.scm that is ran upon reconfigure,
without it, root fs unmounted cleanly. So the test I had in mind would
be to test that when shepherd is upgraded like this, there aren't
changes to how the services are stopped.

>
> Perhaps the problem only shows up with more complex system configs?
> My root partition is on a LUKS device, but I think the problem is more
> something like EBUSY upon ‘umount’ due to stale processes.

I've tried reproducing with bjoli's config last friday, they were
experiencing it and sent their config to IRC chat. I was unable to
reproduce (though I have to confess I changed the fs to ext4 for my
convenience).
So I am not sure if it is reproducible even given a config (though maybe
something is relevant on the vm vs real machine boot, but I wouldn't
expect it...)

I have yet to try Ian's config, but it's going to take me some time to
get to it as their config is more complicated, mainly the network disk,
which I would like to not skip any disk related stuff (and I think Ian
is missing file-systems requirement on their shepherd autofs service,
but it shouldn't really cause this issue as root filesystem unmount
should be recursive).

Regards,
Rutherther

Message #40 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Rutherther <rutherther <at> ditigal.xyz>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, Ian Eure <ian <at> retrospec.tv>,
 Felix Lechner <felix.lechner <at> lease-up.com>, 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Thu, 03 Apr 2025 11:19:45 +0200

Hello,

Rutherther <rutherther <at> ditigal.xyz> skribis:

>> There’s a system test, “root-unmount” in (gnu tests base), but it
>> succeeds: <https://ci.guix.gnu.org/build/9790935/details>.
>
> Thanks for pointing out this test, good to know about it, although it's
> not exactly what I had in mind. I was able to reproduce previously by
> running the upgrade-shepherd-services.scm that is ran upon reconfigure,
> without it, root fs unmounted cleanly. So the test I had in mind would
> be to test that when shepherd is upgraded like this, there aren't
> changes to how the services are stopped.

I’m not sure I understand: do you have a reproducer in a VM, either
manual or automatic?  If you do, please share; if it’s manual, you can
share the steps you followed.

Thanks in advance! :-)

Ludo’.

Message #43 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> ditigal.xyz>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, Ian Eure <ian <at> retrospec.tv>,
 Felix Lechner <felix.lechner <at> lease-up.com>, 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Thu, 03 Apr 2025 11:37:53 +0200

Hello, yes, I shared a (manual) reproducer earlier in this thread. But as I was saying, it was only for commits from Ian's initial email. I was unable to reproduce with same steps after updating to newest (which among other things has shepherd 1.0.3, but I havent bisected to see what commit has fixed the case I got into). Unfortunately people still seem to experience it, so I must have seen other issue / just one possible cause of this one when there are several. I havent sent the final config, one change to it I shared was just mentioned in text. I can send the final config later. I was able to get error on unmounting every time I run reconfigure or just the upgrade shepherd services script.


On April 3, 2025 11:19:45 AM GMT+02:00, "Ludovic Courtès" <ludo <at> gnu.org> wrote:
>Hello,
>
>Rutherther <rutherther <at> ditigal.xyz> skribis:
>
>>> There’s a system test, “root-unmount” in (gnu tests base), but it
>>> succeeds: <https://ci.guix.gnu.org/build/9790935/details>.
>>
>> Thanks for pointing out this test, good to know about it, although it's
>> not exactly what I had in mind. I was able to reproduce previously by
>> running the upgrade-shepherd-services.scm that is ran upon reconfigure,
>> without it, root fs unmounted cleanly. So the test I had in mind would
>> be to test that when shepherd is upgraded like this, there aren't
>> changes to how the services are stopped.
>
>I’m not sure I understand: do you have a reproducer in a VM, either
>manual or automatic?  If you do, please share; if it’s manual, you can
>share the steps you followed.
>
>Thanks in advance! :-)
>
>Ludo’.

Message #46 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Felix Lechner <felix.lechner <at> lease-up.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, Rutherther <rutherther <at> ditigal.xyz>,
 Ian Eure <ian <at> retrospec.tv>, 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Thu, 03 Apr 2025 06:25:59 -0700

Hi,

On Tue, Apr 01 2025, Ludovic Courtès wrote:

> I believe I’m still experiencing it on my laptop

Unmounting properly is great, but would it also make sense to add a call
to sync(2) after a reconfigure since some people reboot quickly?

Kind regards
Felix

Message #49 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Felix Lechner <felix.lechner <at> lease-up.com>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, Rutherther <rutherther <at> ditigal.xyz>,
 Ian Eure <ian <at> retrospec.tv>, 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Tue, 08 Apr 2025 11:45:01 +0200

Felix Lechner <felix.lechner <at> lease-up.com> skribis:

> On Tue, Apr 01 2025, Ludovic Courtès wrote:
>
>> I believe I’m still experiencing it on my laptop
>
> Unmounting properly is great, but would it also make sense to add a call
> to sync(2) after a reconfigure since some people reboot quickly?

The ‘root-file-system’ service calls ‘sync’ when shutting down.

Ludo’.

Message #52 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> ditigal.xyz>
To: Ludovic Courtès <ludo <at> gnu.org>, Felix Lechner
 <felix.lechner <at> lease-up.com>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, Ian Eure <ian <at> retrospec.tv>,
 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Mon, 21 Apr 2025 10:13:07 +0200

Hi,

recently I switched from power-profiles-daemon to tlp. I am experiencing
this issue from time to time! I don't know the exact trigger nor am able
to 'reproduce' it, it happens just sometimes.
I wasn't experiencing this issue before.

This seems to me that tlp could be (one of the) trigger(s) of this bug, somehow.

I checked tlp's systemd service and it seems to be working the same way
the shepherd one does, so I don't really get why that would happen.

Ludovic Courtès <ludo <at> gnu.org> writes:

> Felix Lechner <felix.lechner <at> lease-up.com> skribis:
>
>> On Tue, Apr 01 2025, Ludovic Courtès wrote:
>>
>>> I believe I’m still experiencing it on my laptop
>>
>> Unmounting properly is great, but would it also make sense to add a call
>> to sync(2) after a reconfigure since some people reboot quickly?
>
> The ‘root-file-system’ service calls ‘sync’ when shutting down.

Why then are people ending up with corrupt store, I don't really get it.
Is sync really doing full sync to the disk? Shepherd doesn't shutdown
the system until the service stops, successfully or not, right? So it
should be called on every shutdown/reboot.

Regards,
Rutherther

Message #55 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: "Ashish SHUKLA" <ashish.is <at> lostca.se>
To: <77086 <at> debbugs.gnu.org>
Subject: Re: Filesystems not unmounted on reboot
Date: Tue, 29 Apr 2025 20:41:46 +0000

[Message part 1 (text/plain, inline)]

Hi,

Filesystems not unmounted on reboot, or rather I get filesystem check for "/" volume after I power on, even though I shut it down (as rebooting was ending up in this situation almost everytime). This only happens if I do "sudo guix system reconfigure" prior to reboot/shutdown. Of note is, I'm running a custom Linux kernel (linux-xanmod in my custom channel):

==========================================================================================
% lsblk
NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1          259:0    0 931,5G  0 disk  
├─nvme0n1p1      259:1    0   512M  0 part  /boot
├─nvme0n1p2      259:2    0     4G  0 part  [SWAP]
├─nvme0n1p3      259:3    0   696G  0 part  
│ └─cryptroot    253:0    0   696G  0 crypt 
│   ├─crypt-root 253:1    0   250G  0 lvm   /gnu/store
│   │                                       /
│   └─crypt-home 253:2    0   446G  0 lvm   /home
└─nvme0n1p4      259:4    0   231G  0 part  

% guix describe
Generación 169	28 abr 2025 22:15:25	(actual)
 guix 1710c09
   URL del repositorio: https://codeberg.org/guix/guix-mirror
   rama: master
   revisión: 1710c0941db517453ac2b88c0e854e8348172603
 nonguix f05b33d
   URL del repositorio: https://gitlab.com/nonguix/nonguix
   /rama: master
   revisión: f05b33d75491a6418e7f0b3575207c5c2b2ef8f1
 abbe 673623e
   URL del repositorio: https://codeberg.org/group/guix-modules
   rama: mainline
   revisión: 673623e0abc60eff50b133e74d15ee89eb9f6087
 abbe-emacs 9394d74
   URL del repositorio: https://git.sr.ht/~abbe/guix-emacs-channel
   rama: mainline
   revisión: 9394d749ea88b72173f08f99210b8b81c9e5e47f
==========================================================================================

In terms of filesystems, I have:

- / ext4 as LVM LV over LUKS2 volume cryptroot
- /home ext4 as LVM LV over LUKS2 volume cryptroot
- /boot vfat a separte EFI partition
- /tmp tmpfs

Unfortunately, nothing in /var/log/messages that could hint what happens at the lat moments before reboot/shutdown, and quite often, I end up with NUL bytes in the log file:

==============================================================================
000592a0  78 77 6e 2d 6f 70 65 6e  72 65 73 6f 6c 76 2d 33  |xwn-openresolv-3|
000592b0  2e 31 33 2e 32 2f 73 62  69 6e 2f 72 65 73 6f 6c  |.13.2/sbin/resol|
000592c0  76 63 6f 6e 66 0a 00 00  00 00 00 00 00 00 00 00  |vconf...........|
000592d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0005a000  32 30 32 35 2d 30 34 2d  32 39 20 31 39 3a 35 30  |2025-04-29 19:50|
0005a010  3a 33 32 20 6c 6f 63 61  6c 68 6f 73 74 20 73 68  |:32 localhost sh|
0005a020  65 70 68 65 72 64 5b 31  5d 3a 20 5b 73 79 73 63  |epherd[1]: [sysc|
0005a030  74 6c 5d 20 6e 65 74 2e  69 70 76 34 2e 74 63 70  |tl] net.ipv4.tcp|
0005a040  5f 77 69 6e 64 6f 77 5f  73 63 61 6c 69 6e 67 20  |_window_scaling |
==============================================================================

It's sometimes broken /gnu/store as well. Sometimes empty/broken initrds, which I was able to salvage by booting previous entries. To minimize damage to /gnu/store, I try to explicit "sync" after "sudo guix system reconfigure", but the problem is still there with log files, at least.

Also, as I use custom mountpoints (i.e. /home, /tmp), do I need to specify file-system dependencies in my custom filesystem mountpoints, i.e. /boot, /home, /tmp, all depending on "/" ? Right now, I only have mapped-devices dependencies (for the LUKS2 volume) in the definitions for / and /home filesystems which reside on a LVM volume backed by LUKS2 volume.

Anyways, if there is anything I can help with to get to the bottom of this, then I'll be happy help. It's quite annoying, almost making me stop using, so any help is appreciated.

Thanks!
-- 
Ashish SHUKLA | GPG: F682 CDCC 39DC 0FEA E116  20B6 C746 CFA9 E74F A4B0

"If I destroy you, what business is it of yours ?" (Dark Forest, Liu Cixin)

[signature.asc (application/pgp-signature, inline)]

Message #60 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "Ashish SHUKLA" via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Cc: Ashish SHUKLA <ashish.is <at> lostca.se>, 77086 <at> debbugs.gnu.org
Subject: Re: bug#77086: Filesystems not unmounted on reboot
Date: Fri, 02 May 2025 18:07:49 +0200

Hello,

"Ashish SHUKLA" via Bug reports for GNU Guix <bug-guix <at> gnu.org> writes:

> Filesystems not unmounted on reboot, or rather I get filesystem check for "/" volume after I power on, even though I shut it down (as rebooting was ending up in this situation almost everytime). This only happens if I do "sudo guix system reconfigure" prior to reboot/shutdown. Of note is, I'm running a custom Linux kernel (linux-xanmod in my custom channel):

[...]

> Unfortunately, nothing in /var/log/messages that could hint what happens at the lat moments before reboot/shutdown, and quite often, I end up with NUL bytes in the log file:

Are there any hints on tty12 (syslog output) while shutting down?

These issues are all quite mysterious and I’m not sure how to best
approach it if we cannot reproduce them in a VM.

Ludo’.

Message #66 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Ashish SHUKLA" <ashish.is <at> lostca.se>
To: Ludovic Courtès <ludo <at> gnu.org>, "Ashish SHUKLA via Bug
 reports for GNU Guix" <bug-guix <at> gnu.org>
Cc: 77086 <at> debbugs.gnu.org
Subject: Re: bug#77086: Filesystems not unmounted on reboot
Date: Sat, 03 May 2025 10:48:35 +0000

[Message part 1 (text/plain, inline)]

El 2025-05-02 18:07, Ludovic Courtès escribió:
> Hello,
>
> "Ashish SHUKLA" via Bug reports for GNU Guix <bug-guix <at> gnu.org> writes:
>
>> Filesystems not unmounted on reboot, or rather I get filesystem check for "/" volume after I power on, even though I shut it down (as rebooting was ending up in this situation almost everytime). This only happens if I do "sudo guix system reconfigure" prior to reboot/shutdown. Of note is, I'm running a custom Linux kernel (linux-xanmod in my custom channel):
>
> [...]
>
>> Unfortunately, nothing in /var/log/messages that could hint what happens at the lat moments before reboot/shutdown, and quite often, I end up with NUL bytes in the log file:
>
> Are there any hints on tty12 (syslog output) while shutting down?
>
> These issues are all quite mysterious and I’m not sure how to best
> approach it if we cannot reproduce them in a VM.
>
> Ludo’.

I started recording tty12 output before shutting down (or rebooting)[0]. So far it's not manifested yet (filesystem check on boot, or 00 00 00 bytes in the log files).

References:
[0] sudo chvt 12 && sleep 20 && sudo reboot

Thanks!
-- 
Ashish SHUKLA | GPG: F682 CDCC 39DC 0FEA E116  20B6 C746 CFA9 E74F A4B0

"If I destroy you, what business is it of yours ?" (Dark Forest, Liu Cixin)

[signature.asc (application/pgp-signature, inline)]

Message #72 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: "Ashish SHUKLA" <ashish.is <at> lostca.se>
To: Ludovic Courtès <ludo <at> gnu.org>, "Ashish SHUKLA via Bug
 reports for GNU Guix" <bug-guix <at> gnu.org>
Cc: 77086 <at> debbugs.gnu.org
Subject: Re: bug#77086: Filesystems not unmounted on reboot
Date: Sat, 03 May 2025 12:01:30 +0000

[Message part 1 (text/plain, inline)]

El 2025-05-03 10:48, Ashish SHUKLA escribió:
> El 2025-05-02 18:07, Ludovic Courtès escribió:
>> Hello,
>>
>> "Ashish SHUKLA" via Bug reports for GNU Guix <bug-guix <at> gnu.org> writes:
>>
>>> Filesystems not unmounted on reboot, or rather I get filesystem check for "/" volume after I power on, even though I shut it down (as rebooting was ending up in this situation almost everytime). This only happens if I do "sudo guix system reconfigure" prior to reboot/shutdown. Of note is, I'm running a custom Linux kernel (linux-xanmod in my custom channel):
>>
>> [...]
>>
>>> Unfortunately, nothing in /var/log/messages that could hint what happens at the lat moments before reboot/shutdown, and quite often, I end up with NUL bytes in the log file:
>>
>> Are there any hints on tty12 (syslog output) while shutting down?
>>
>> These issues are all quite mysterious and I’m not sure how to best
>> approach it if we cannot reproduce them in a VM.
>>
>> Ludo’.
>
> I started recording tty12 output before shutting down (or rebooting)[0]. So far it's not manifested yet (filesystem check on boot, or 00 00 00 bytes in the log files).
>
> References:
> [0] sudo chvt 12 && sleep 20 && sudo reboot
>
> Thanks!

I just rebuilt kernel and problem happened again after guix system reconfigure. I made following not so helpful (at least I can not spot anything) videos (available for at least a week from today):

https://www.lostca.se/~abbe/VID20250503134553.mp4 (chvt 12 && sleep 20 && reboot)
https://www.lostca.se/~abbe/VID20250503134801.mp4 (after reboot, "recovering journal" message, and 00 00 00 bytes in the /var/log/messages indicate it was an unclean shutdown)

Thanks!
-- 
Ashish SHUKLA | GPG: F682 CDCC 39DC 0FEA E116  20B6 C746 CFA9 E74F A4B0

"If I destroy you, what business is it of yours ?" (Dark Forest, Liu Cixin)

[signature.asc (application/pgp-signature, inline)]

Message #78 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "Ashish SHUKLA" <ashish.is <at> lostca.se>
Cc: Ashish SHUKLA via Bug reports for GNU Guix <bug-guix <at> gnu.org>,
 77086 <at> debbugs.gnu.org
Subject: Re: bug#77086: Filesystems not unmounted on reboot
Date: Sat, 03 May 2025 17:59:09 +0200

Hello,

"Ashish SHUKLA" <ashish.is <at> lostca.se> writes:

> I just rebuilt kernel and problem happened again after guix system
> reconfigure. I made following not so helpful (at least I can not spot
> anything) videos (available for at least a week from today):
>
> https://www.lostca.se/~abbe/VID20250503134553.mp4 (chvt 12 && sleep 20 && reboot)
> https://www.lostca.se/~abbe/VID20250503134801.mp4 (after reboot, "recovering journal" message, and 00 00 00 bytes in the /var/log/messages indicate it was an unclean shutdown)

Thanks for taking the time to capture these videos.  Indeed, I cannot
really draw a conclusion either from this video.  I’m surprised that the
last message is terminating ‘term-console’, I would expect more messages
after that but I’m not sure.

Did you try this config in ‘guix system vm --no-graphic’?  Non-trivial
configs are often not reproducible as-is (due to Certbot certificates,
static networking configs, storage devices, etc.) but who knows.

Thanks,
Ludo’.

Message #84 received at 77086 <at> debbugs.gnu.org (full text, mbox):

From: David Pflug <david <at> pflug.io>
To: 77086 <at> debbugs.gnu.org
Subject: Re: Filesystems not unmounted on reboot
Date: Tue, 10 Jun 2025 16:57:53 +0000

I was just bit by this on a fresh install. Once I've got this setup
finalized, I'll try to reproduce on a vm and prune it down to a
minimal test case.

-- 
David Pflug

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.