GNU bug report logs - #77716
Corrupted store after Guix deploy

Previous Next

To reply to this bug, email your comments to 77716 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tomas Volf <~@wolfsden.cz>
To: bug-guix <at> gnu.org
Subject: Corrupted store after Guix deploy
Date: Thu, 10 Apr 2025 23:32:53 +0200

Hello,

I just finished Guix deploy.  The command has finished successfully,
without any warnings or errors.  However when I checked the store, I can
see that some paths are corrupted:

--8<---------------cut here---------------start------------->8---
$ guix gc --verify=contents
reading the store...
checking path existence...
checking hashes...
path `/gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden' was modified! expected hash `b40d36eb683b0143cb192e37ba18f1bb31a437016e8a2f82a3623942b39c93b8', got `b3ecf9df08d6bd70dc2482ff68d606802aa8d1623b8efd8bcd0485bbe670cd4c'
path `/gnu/store/my2g20c7spa8ixpnr92s302wj97cxl86-nonguix' was modified! expected hash `4e478e5e311d2a1205b01681de5d4fd2aaba7e9367bdf83ac2c74b3a18322bc5', got `9b8c244c6a1592c1a067bb0f1682b087322258c52dd91a95da7f2d7fbb03a7c3'
--8<---------------cut here---------------end--------------->8---

I know there already are some issues in the similar spirit, but
consensus there seems to be that missing/wrong sync is to blame.  So I
am making a new bug, because this one is not affected by (possible) sync
issues.  I did not even reboot (yet), I have verified the store *before*
rebooting.

Indeed, the guix hashes do differ:

Laptop:

--8<---------------cut here---------------start------------->8---
$ guix hash -r /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
1f4kkjrl4fb2lf12z2kf04vs8cdvy4cbldrf375l609vd3mkc3dl
--8<---------------cut here---------------end--------------->8---

Remote:

--8<---------------cut here---------------start------------->8---
$ guix hash -r /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
0k6df3kbp184rn5zv3ivcb8shal00vb6izw24kf71gfn13gzkv5k
--8<---------------cut here---------------end--------------->8---

I have checked the files themselves (sha256sum), and they are the same
on both machines.  However I have noticed couple of differences listed
below:

The permissions on the very top level directory differ (which one are
correct?):

Laptop:

--8<---------------cut here---------------start------------->8---
$ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
drwxr-xr-x   1 root          root              16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
--8<---------------cut here---------------end--------------->8---

Remote:

--8<---------------cut here---------------start------------->8---
$ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
dr-xr-xr-x  1 root root           16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
--8<---------------cut here---------------end--------------->8---



The target of symbolic link in the store item differs.

Laptop:

--8<---------------cut here---------------start------------->8---
$ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
total 4
lrwxrwxrwx 1 root root 61 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d/
--8<---------------cut here---------------end--------------->8---

Remote:

--8<---------------cut here---------------start------------->8---
$ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
total 4
lrwxrwxrwx 1 root root 60 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d
--8<---------------cut here---------------end--------------->8---

Notice that for the Laptop, the target ends in `/', for the Remote it
does not.



Additionally it is interesting that in addition I have exactly on more
corrupted path, and that is another channel:

--8<---------------cut here---------------start------------->8---
$ guix gc --verify=contents
reading the store...
checking path existence...
checking hashes...
path `/gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden' was modified! expected hash `b40d36eb683b0143cb192e37ba18f1bb31a437016e8a2f82a3623942b39c93b8', got `b3ecf9df08d6bd70dc2482ff68d606802aa8d1623b8efd8bcd0485bbe670cd4c'
path `/gnu/store/my2g20c7spa8ixpnr92s302wj97cxl86-nonguix' was modified! expected hash `4e478e5e311d2a1205b01681de5d4fd2aaba7e9367bdf83ac2c74b3a18322bc5', got `9b8c244c6a1592c1a067bb0f1682b087322258c52dd91a95da7f2d7fbb03a7c3'
--8<---------------cut here---------------end--------------->8---



I have copied the store item back using `rsync -a ...', and this is what
diffoscope has to say (the same as above):

--8<---------------cut here---------------start------------->8---
2025-04-10 21:06:56 W: diffoscope.comparators.xml: Vulnerable version of pyexpat detected; disabling comparison of XML documents. Install defusedxml or upgrade your pyexpat.
--- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
+++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
├── /gnu/store/lm25j7769nfzgfmx8gh28zarrlczgh8r-coreutils-9.1/bin/stat {}
│ @@ -1,8 +1,8 @@
│
│    Size: 16        	Blocks: 0          IO Block: 4096   directory
│  Device: 0,25	Links: 1
│ -Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│ +Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│
│  Modify: 1970-01-01 00:00:01.000000000 +0000
│   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share
├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share
│ │   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile
│ ├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile
│ │ │   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site
│ │ ├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site
│ │ │ │   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/3.0
│ │ │ ├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/3.0
│ │ │ │┄ symlink
│ │ │ │ @@ -1 +1 @@
│ │ │ │ +destination: /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d
│ │ │ │ -destination: /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d/
│ │ │ │ ├── /gnu/store/lm25j7769nfzgfmx8gh28zarrlczgh8r-coreutils-9.1/bin/stat {}
│ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │
│ │ │ │ │ +  Size: 60        	Blocks: 8          IO Block: 4096   symbolic link
│ │ │ │ │ -  Size: 61        	Blocks: 8          IO Block: 4096   symbolic link
│ │ │ │ │  Device: 0,25	Links: 1
│ │ │ │ │  Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │ │ │
│ │ │ │ │  Modify: 1970-01-01 00:00:01.000000000 +0000
--8<---------------cut here---------------end--------------->8---

Any idea what might have happen?

I cannot really reproduce it now, not sure it the issue went away with
newer Guix or whether it was random flap.  I just wonder whether the
other end should not checksum the entries before putting them into the
store and reject them if the hash is wrong.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

Message #8 received at 77716 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Tomas Volf <~@wolfsden.cz>
Cc: 77716 <at> debbugs.gnu.org
Subject: Re: bug#77716: Corrupted store after Guix deploy
Date: Mon, 21 Apr 2025 16:47:12 +0200

Hello,

Tomas Volf <~@wolfsden.cz> writes:

> The permissions on the very top level directory differ (which one are
> correct?):
>
> Laptop:
>
> --8<---------------cut here---------------start------------->8---
> $ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
> drwxr-xr-x   1 root          root              16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
> --8<---------------cut here---------------end--------------->8---
>
> Remote:
>
> --8<---------------cut here---------------start------------->8---
> $ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
> dr-xr-xr-x  1 root root           16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
> --8<---------------cut here---------------end--------------->8---
>
>
>
> The target of symbolic link in the store item differs.
>
> Laptop:
>
> --8<---------------cut here---------------start------------->8---
> $ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
> total 4
> lrwxrwxrwx 1 root root 61 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d/
> --8<---------------cut here---------------end--------------->8---
>
> Remote:
>
> --8<---------------cut here---------------start------------->8---
> $ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
> total 4
> lrwxrwxrwx 1 root root 60 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d
> --8<---------------cut here---------------end--------------->8---
>
> Notice that for the Laptop, the target ends in `/', for the Remote it
> does not.

Interesting.

There are several code paths to add a store item to the store.  One is
through (guix store …), use by ‘guix offload’.  One is through ‘guix
substitute’, which restores nars it downloaded.  One is through a build
via guix-daemon (C++ code).

Could you tell how these two differing things were obtained?

On each machine, run ‘guix build --log-file /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden’.  If
there’s a local log file, it was built locally or offloaded; otherwise
it was substituted.

Could you also report the guix-daemon versions that are running on each
machine?

An interesting puzzle.  :-)

Thanks,
Ludo’.

Message #11 received at 77716 <at> debbugs.gnu.org (full text, mbox):

From: Tomas Volf <~@wolfsden.cz>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77716 <at> debbugs.gnu.org
Subject: Re: bug#77716: Corrupted store after Guix deploy
Date: Mon, 21 Apr 2025 20:34:52 +0200

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hello,
>
> Tomas Volf <~@wolfsden.cz> writes:
>
>> The permissions on the very top level directory differ (which one are
>> correct?):
>>
>> Laptop:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
>> drwxr-xr-x   1 root          root              16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
>> --8<---------------cut here---------------end--------------->8---
>>
>> Remote:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
>> dr-xr-xr-x  1 root root           16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
>> --8<---------------cut here---------------end--------------->8---
>>
>>
>>
>> The target of symbolic link in the store item differs.
>>
>> Laptop:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
>> total 4
>> lrwxrwxrwx 1 root root 61 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d/
>> --8<---------------cut here---------------end--------------->8---
>>
>> Remote:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
>> total 4
>> lrwxrwxrwx 1 root root 60 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d
>> --8<---------------cut here---------------end--------------->8---
>>
>> Notice that for the Laptop, the target ends in `/', for the Remote it
>> does not.
>
> Interesting.
>
> There are several code paths to add a store item to the store.  One is
> through (guix store …), use by ‘guix offload’.  One is through ‘guix
> substitute’, which restores nars it downloaded.  One is through a build
> via guix-daemon (C++ code).
>
> Could you tell how these two differing things were obtained?
>
> On each machine, run ‘guix build --log-file /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden’.  If
> there’s a local log file, it was built locally or offloaded; otherwise
> it was substituted.

Definitely, I am pretty sure I know how these were acquired.  It is my
personal Guix channel, and I am not using any substitutes other then the
official ones.  On my local system (the "Laptop" above), I have executed
the following:

--8<---------------cut here---------------start------------->8---
guix pull --no-offload -C wolfsnet/files/channels-base.sex
--8<---------------cut here---------------end--------------->8---

During that, Guix was pulled using this channels file:

--8<---------------cut here---------------start------------->8---
(cons* (channel
        (name 'guix)
        (url "https://git.wolfsden.cz/.git/guix")
        (introduction
         (make-channel-introduction
          "028e445a2028068e3c83996daa281057f19141a0"
          (openpgp-fingerprint
           "B783 49B3 8C14 7D36 2988  68A4 2FBF EE7D B67F C1A9"))))
       (channel
        (name 'wolfsden)
        (url "https://git.wolfsden.cz/.git/wolfsden")
        (introduction
         (make-channel-introduction
          "49d7c72d2c3ae36ae8cf045dc7c00da29801af2c"
          (openpgp-fingerprint
           "B783 49B3 8C14 7D36 2988  68A4 2FBF EE7D B67F C1A9"))))
       (channel
        (name 'nonguix)
        (url "https://gitlab.com/nonguix/nonguix")
        (introduction
         (make-channel-introduction
          "897c1a470da759236cc11798f4e0a5f7d4d59fbc"
          (openpgp-fingerprint
           "2A39 3FFF 68F4 EF7A 3D29  12AF 6F51 20A0 22FB B2D5"))))
       (filter (negate guix-channel?) %default-channels))
--8<---------------cut here---------------end--------------->8---

Since 'wolfsden has no substitutes on official servers, the store item
was built locally (on "Laptop").  After that, I did `guix deploy' onto
"Remote".  Since I am using the channel in few places in the system
definition, it was copied over to "Remote" as part of the deploy.

Additionally I am using the "current" Guix as the (guix) field of the
guix-configuration, but I am pretty sure that is not relevant here, so I
am including it just for completeness (in case I am wrong).

--8<---------------cut here---------------start------------->8---
(define (%current-guix)
  (let ((guix-bin (car (command-line))))
    (unless (string-suffix? "/bin/guix" guix-bin)
      (error "Does not look like guix binary" guix-bin))
    (let* ((guix-profile (string-drop-right guix-bin 9))
           (guix-profile (canonicalize-path guix-profile)))
      (package
        (name "custom-guix")
        (version "1")
        (source #f)
        (build-system trivial-build-system)
        (arguments
         (list
          #:builder
          #~(symlink #$guix-profile #$output)))
        (home-page #f)
        (synopsis #f)
        (description #f)
        (license #f)))))

[..]

(operating-system
        ...
        (services
          (modify-services %base-services
            (guix-service-type
             config => (guix-configuration
                        (inherit config)
                        (channels %channels)
                        (guix (%current-guix)))))))
--8<---------------cut here---------------end--------------->8---

To sum up, to the "broken" system ("Remote") the store item got via
`guix deploy'.

>
> Could you also report the guix-daemon versions that are running on each
> machine?

Sadly I am not sure how to tell now :/.  My laptop got updated and the
VM got shut down.  I will be sure to gather this information next time.
Maybe it would be useful to include this information in the log files
for the builds.  All I can say now is that guix-daemon on both was
"reasonably recent", definitely under 2 months old, probably under
month.

>
> An interesting puzzle.  :-)

I am glad you have found this interesting. :)

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.