Subject: bug#77716: Corrupted store after Guix deploy

From: Tomas Volf <[email-masked]>
Date: Thu, 10 Apr 2025 23:32:53 +0200

Hello,

I just finished Guix deploy.  The command has finished successfully,
without any warnings or errors.  However when I checked the store, I can
see that some paths are corrupted:

--8<---------------cut here---------------start------------->8---
$ guix gc --verify=contents
reading the store...
checking path existence...
checking hashes...
path `/gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden' was modified! expected hash `b40d36eb683b0143cb192e37ba18f1bb31a437016e8a2f82a3623942b39c93b8', got `b3ecf9df08d6bd70dc2482ff68d606802aa8d1623b8efd8bcd0485bbe670cd4c'
path `/gnu/store/my2g20c7spa8ixpnr92s302wj97cxl86-nonguix' was modified! expected hash `4e478e5e311d2a1205b01681de5d4fd2aaba7e9367bdf83ac2c74b3a18322bc5', got `9b8c244c6a1592c1a067bb0f1682b087322258c52dd91a95da7f2d7fbb03a7c3'
--8<---------------cut here---------------end--------------->8---

I know there already are some issues in the similar spirit, but
consensus there seems to be that missing/wrong sync is to blame.  So I
am making a new bug, because this one is not affected by (possible) sync
issues.  I did not even reboot (yet), I have verified the store *before*
rebooting.

Indeed, the guix hashes do differ:

Laptop:

--8<---------------cut here---------------start------------->8---
$ guix hash -r /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
1f4kkjrl4fb2lf12z2kf04vs8cdvy4cbldrf375l609vd3mkc3dl
--8<---------------cut here---------------end--------------->8---

Remote:

--8<---------------cut here---------------start------------->8---
$ guix hash -r /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
0k6df3kbp184rn5zv3ivcb8shal00vb6izw24kf71gfn13gzkv5k
--8<---------------cut here---------------end--------------->8---

I have checked the files themselves (sha256sum), and they are the same
on both machines.  However I have noticed couple of differences listed
below:

The permissions on the very top level directory differ (which one are
correct?):

Laptop:

--8<---------------cut here---------------start------------->8---
$ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
drwxr-xr-x   1 root          root              16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
--8<---------------cut here---------------end--------------->8---

Remote:

--8<---------------cut here---------------start------------->8---
$ ls -al /gnu/store/ | grep 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
dr-xr-xr-x  1 root root           16 Jan  1  1970 1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
--8<---------------cut here---------------end--------------->8---

The target of symbolic link in the store item differs.

Laptop:

--8<---------------cut here---------------start------------->8---
$ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
total 4
lrwxrwxrwx 1 root root 61 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d/
--8<---------------cut here---------------end--------------->8---

Remote:

--8<---------------cut here---------------start------------->8---
$ ls -l /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/
total 4
lrwxrwxrwx 1 root root 60 Jan  1  1970 3.0 -> /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d
--8<---------------cut here---------------end--------------->8---

Notice that for the Laptop, the target ends in `/', for the Remote it
does not.

Additionally it is interesting that in addition I have exactly on more
corrupted path, and that is another channel:

--8<---------------cut here---------------start------------->8---
$ guix gc --verify=contents
reading the store...
checking path existence...
checking hashes...
path `/gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden' was modified! expected hash `b40d36eb683b0143cb192e37ba18f1bb31a437016e8a2f82a3623942b39c93b8', got `b3ecf9df08d6bd70dc2482ff68d606802aa8d1623b8efd8bcd0485bbe670cd4c'
path `/gnu/store/my2g20c7spa8ixpnr92s302wj97cxl86-nonguix' was modified! expected hash `4e478e5e311d2a1205b01681de5d4fd2aaba7e9367bdf83ac2c74b3a18322bc5', got `9b8c244c6a1592c1a067bb0f1682b087322258c52dd91a95da7f2d7fbb03a7c3'
--8<---------------cut here---------------end--------------->8---

I have copied the store item back using `rsync -a ...', and this is what
diffoscope has to say (the same as above):

--8<---------------cut here---------------start------------->8---
2025-04-10 21:06:56 W: diffoscope.comparators.xml: Vulnerable version of pyexpat detected; disabling comparison of XML documents. Install defusedxml or upgrade your pyexpat.
--- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
+++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden
├── /gnu/store/lm25j7769nfzgfmx8gh28zarrlczgh8r-coreutils-9.1/bin/stat {}
│ @@ -1,8 +1,8 @@
│
│    Size: 16        	Blocks: 0          IO Block: 4096   directory
│  Device: 0,25	Links: 1
│ -Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│ +Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│
│  Modify: 1970-01-01 00:00:01.000000000 +0000
│   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share
├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share
│ │   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile
│ ├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile
│ │ │   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site
│ │ ├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site
│ │ │ │   --- /gnu/store/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/3.0
│ │ │ ├── +++ /tmp/xxxx/1bjf1lza4i7fc1jh8qzb1n9xnv3rfi17-wolfsden/share/guile/site/3.0
│ │ │ │┄ symlink
│ │ │ │ @@ -1 +1 @@
│ │ │ │ +destination: /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d
│ │ │ │ -destination: /gnu/store/rmwi813mcsg8d5ilzq0rxwqd8dp558r5-wolfsden-989d91d/
│ │ │ │ ├── /gnu/store/lm25j7769nfzgfmx8gh28zarrlczgh8r-coreutils-9.1/bin/stat {}
│ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │
│ │ │ │ │ +  Size: 60        	Blocks: 8          IO Block: 4096   symbolic link
│ │ │ │ │ -  Size: 61        	Blocks: 8          IO Block: 4096   symbolic link
│ │ │ │ │  Device: 0,25	Links: 1
│ │ │ │ │  Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │ │ │
│ │ │ │ │  Modify: 1970-01-01 00:00:01.000000000 +0000
--8<---------------cut here---------------end--------------->8---

Any idea what might have happen?

I cannot really reproduce it now, not sure it the issue went away with
newer Guix or whether it was random flap.  I just wonder whether the
other end should not checksum the entries before putting them into the
store and reject them if the hash is wrong.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.