Subject: bug#78355: guix-ownership inconsistent state

From: Rutherther <[email-masked]>
Date: Thu, 15 May 2025 09:35:52 +0200

Hi Ben,

I am CCing you to get more information about the inconsistent ownership,
if you could help with that.

The most important questions are probably:
0. Are you sure the service actually ran after you reconfigured to root?
  It should definitely run after reboot, not sure if after reconfigure
  as the service already exists, it's just modified
1. Do you think you could've killed the service when it was running
after you reconfigured back to privileged daemon, ie. by rebooting when
it was running?
2. Do you know what folders had wrong owners?
  - Was everything under /var/guix fully owned by 971?
  - Was everything under /etc/guix fully owned by 971?
  - Was everything under store fully owned by 971?

Hi Ludo,

Ludovic Courtès <[email-masked]> writes:

> Hi,
>
> Rutherther <[email-masked]> writes:
>
>> There are reports from users with inconsistencies in ownership, it seems that at
>> least /var/guix is sometimes left with wrong owner, but maybe even parts
>> of the store? I cannot verify that.
>
> Would be nice to get their reports here, otherwise we’re left
> speculating.

I am afraid we will be left speculating either way, that's why I chose
this approach. That is because none of the users I know of took time to
debug it and just fixed it. CCing Ben Sturmfels who
encountered it (see
https://lists.gnu.org/archive/html/help-guix/2025-05/msg00052.html).
For the other user I know of, I don't know their e-mail.

Note that on IRC I recommended the user to chown $USER /gnu/store
($USER just cause it's easiest, any non-root user would be fine) and
herd start guix-ownership and that fixed the issue. So the service
definitely is doing its job. See here https://logs.guix.gnu.org/guix/2025-05-10.log#171215

>
>> The guix-ownership service checks /gnu/store ownership to check if the
>> whole store and all files important for the daemon (/etc/guix,
>> /var/guix) are owned by the appropriate user.
>>
>> If the folder isn't owned by appropriate user, it moves to those steps:
>> 1. Fix permissions in /gnu/store - first under it, then /gnu/store
>> itself as last step
>> 2. Fix /var/guix
>> 3. Fix /etc/guix
>> 4. Fix /var/log/guix
>>
>> So from those laid out steps it should be obvious that if guix-ownership
>> service somehow stops between steps 1 and 2, it will never recover
>> ownerships of /var/guix, /etc/guix and /var/log/guix. /gnu/store should
>> change owner as last.
>
> Well, the fundamental assumption is that ‘guix-ownership’ is not
> interrupted during its work; manual intervention is needed to repair
> things if it is interrupted.

I think it would at least be good if there was a script to do what
guix-ownership does, but force it without the /gnu/store ownership
check, to make it easier for users to recover. Maybe even an optional argument to
guix-ownership, where you could `sudo herd start guix-ownership 1` and
that would force the chown'ing?

>
> I don’t see any way around that but perhaps we should warn about it more
> clearly?

That would definitely be great, I think you can easily oversee that the
service has started. Now I am not sure if one-shot services are started
after change when you reconfigure, if they are, I think it's going to be
a common issue - people reconfigure & reboot! Meaning they will usually
stop the service, or am I mistaken here?

>
>> On the other hand it feels much of a coincidence users would be
>> consistently hitting reboots between those steps. So maybe I am
>> overlooking another thing. I checked the file-system-fold, it goes to
>> /gnu/store as last, so that would mean putting step 1 after 4 should fix
>> that. Still, maybe only /gnu/store itself should be skipped instead of moving
>> the step, and done as last, step 5 to ensure it's fine even if
>> file-system-fold somehow changed the ordering? Not sure how exactly it
>> should behave in that regard.
>
> Doing /gnu/store last is a good idea because it reduces the window
> during which the inconsistent state could go undetected.

I think it completely removes it. Or why do you think not?
If it really doesn't I think it would be good if we came up with an
approach that would remove this window. The best way to achieve no
inconsistence-window would be to just check all of the permissions, but
that's probably an overkill.
For example creating a 'stamp'
somewhere that says it was done, at the end, and checking if that stamp
file matches what we expect.

>
> Feel free to propose a patch; otherwise I’ll give it a try, but not
> before next week.
>
> Thanks,
> Ludo’.

Thanks
Rutherther