GNU bug report logs - #50015
Rust packages are not reproducible

Previous Next

To reply to this bug, email your comments to 50015 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: bug-guix <at> gnu.org
Subject: Rust packages are not reproducible
Date: Wed, 11 Aug 2021 23:15:09 +0200

Hello!

Rust packages, which are essentially empty, are not bit-reproducible:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix challenge rust-rocket-codegen --substitute-urls='https://ci.guix.gnu.org https://bordeaux.guix.gnu.org' 
/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
  no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
  https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
  https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
  differing file:
    /share/cargo/registry/rocket_codegen-0.4.7.crate

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive
ludo <at> ribbon ~/src/guix$ ./pre-inst-env guix challenge rust-rocket-codegen
/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
  no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
  https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
  https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
  differing file:
    /share/cargo/registry/rocket_codegen-0.4.7.crate

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive
$ git log |head -1
commit 973842acbc2d0dc1ab41738a534d4abda6d9efa7
--8<---------------cut here---------------end--------------->8---

The diffoscope output suggests it’s about timestamps on one file in the
.crate archive:

--8<---------------cut here---------------start------------->8---
│ │ │ │   --- /tmp/guix-directory.ii5wmv/share/cargo/registry/rocket_codegen-0.4.7.crate
│ │ │ ├── +++ /tmp/guix-directory.uTTKSw/share/cargo/registry/rocket_codegen-0.4.7.crate
│ │ │ │ ├── rocket_codegen-0.4.7.crate-content
│ │ │ │ │ ├── file list
│ │ │ │ │ │ @@ -1,67 +1,67 @@
│ │ │ │ │ │ --rw-r--r--   0        0        0     1293 2021-07-27 15:22:18.000000 rocket_codegen-0.4.7/Cargo.toml
[…]
│ │ │ │ │ │ +-rw-r--r--   0        0        0     1293 2021-07-27 22:01:49.000000 rocket_codegen-0.4.7/Cargo.toml
--8<---------------cut here---------------end--------------->8---

Does that ring a bell?

Thanks,
Ludo’.

PS: I noticed this via
    <http://data.guix.gnu.org/revision/973842acbc2d0dc1ab41738a534d4abda6d9efa7/package-reproducibility>
    with help from Chris.  Fixing this could noticeably improve our
    stats.  :-)

Message #8 received at 50015 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 50015 <at> debbugs.gnu.org
Subject: Re: bug#50015: Rust packages are not reproducible
Date: Thu, 12 Aug 2021 09:44:08 +0300

[Message part 1 (text/plain, inline)]

On Wed, Aug 11, 2021 at 11:15:09PM +0200, Ludovic Courtès wrote:
> Hello!
> 
> Rust packages, which are essentially empty, are not bit-reproducible:
> 
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix challenge rust-rocket-codegen --substitute-urls='https://ci.guix.gnu.org https://bordeaux.guix.gnu.org' 
> /gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
>   no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
>   https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
>   https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
>   differing file:
>     /share/cargo/registry/rocket_codegen-0.4.7.crate
> 
> 1 store items were analyzed:
>   - 0 (0.0%) were identical
>   - 1 (100.0%) differed
>   - 0 (0.0%) were inconclusive
> ludo <at> ribbon ~/src/guix$ ./pre-inst-env guix challenge rust-rocket-codegen
> /gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7 contents differ:
>   no local build for '/gnu/store/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7'
>   https://ci.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 0j6zf2afvc49jnp7i6z7yvbxm0bmw8yc65hz3lncgvw5lc6z1bc1
>   https://bordeaux.guix.gnu.org/nar/lzip/09hlac18bwl1kcnhdig7z1v2j8ly1czw-rust-rocket-codegen-0.4.7: 015kb637b56mqcsg3f6x1qggm2bybiszji2069gb913wxbj6rs7w
>   differing file:
>     /share/cargo/registry/rocket_codegen-0.4.7.crate
> 
> 1 store items were analyzed:
>   - 0 (0.0%) were identical
>   - 1 (100.0%) differed
>   - 0 (0.0%) were inconclusive
> $ git log |head -1
> commit 973842acbc2d0dc1ab41738a534d4abda6d9efa7
> --8<---------------cut here---------------end--------------->8---
> 
> The diffoscope output suggests it’s about timestamps on one file in the
> .crate archive:
> 
> --8<---------------cut here---------------start------------->8---
> │ │ │ │   --- /tmp/guix-directory.ii5wmv/share/cargo/registry/rocket_codegen-0.4.7.crate
> │ │ │ ├── +++ /tmp/guix-directory.uTTKSw/share/cargo/registry/rocket_codegen-0.4.7.crate
> │ │ │ │ ├── rocket_codegen-0.4.7.crate-content
> │ │ │ │ │ ├── file list
> │ │ │ │ │ │ @@ -1,67 +1,67 @@
> │ │ │ │ │ │ --rw-r--r--   0        0        0     1293 2021-07-27 15:22:18.000000 rocket_codegen-0.4.7/Cargo.toml
> […]
> │ │ │ │ │ │ +-rw-r--r--   0        0        0     1293 2021-07-27 22:01:49.000000 rocket_codegen-0.4.7/Cargo.toml
> --8<---------------cut here---------------end--------------->8---
> 
> Does that ring a bell?
> 
> Thanks,
> Ludo’.
> 
> PS: I noticed this via
>     <http://data.guix.gnu.org/revision/973842acbc2d0dc1ab41738a534d4abda6d9efa7/package-reproducibility>
>     with help from Chris.  Fixing this could noticeably improve our
>     stats.  :-)
> 

I tried patching this a couple of ways, but it looks like the best
option is going to be a 'patch-and-repack phase after 'install. the
.crate file is really a gzip tarball, and I suspect that each time we
run 'cargo <something>' the timestamp gets updated.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 50015 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 50015 <at> debbugs.gnu.org
Subject: Re: bug#50015: Rust packages are not reproducible
Date: Thu, 12 Aug 2021 10:06:56 +0200

Hello!

Efraim Flashner <efraim <at> flashner.co.il> skribis:

> I tried patching this a couple of ways, but it looks like the best
> option is going to be a 'patch-and-repack phase after 'install. the
> .crate file is really a gzip tarball, and I suspect that each time we
> run 'cargo <something>' the timestamp gets updated.

So that ‘Cargo.toml’ file is not something taken from the build tree?
In that case we could reset the timestamp before the tarball is
created.  But otherwise yeah, patch’n’repack.

Ludo’.

Message #14 received at 50015 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 50015 <at> debbugs.gnu.org, Efraim Flashner <efraim <at> flashner.co.il>
Subject: Re: bug#50015: Rust packages are not reproducible
Date: Tue, 03 Oct 2023 23:30:15 -0400

Hello,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hello!
>
> Efraim Flashner <efraim <at> flashner.co.il> skribis:
>
>> I tried patching this a couple of ways, but it looks like the best
>> option is going to be a 'patch-and-repack phase after 'install. the
>> .crate file is really a gzip tarball, and I suspect that each time we
>> run 'cargo <something>' the timestamp gets updated.
>
> So that ‘Cargo.toml’ file is not something taken from the build tree?
> In that case we could reset the timestamp before the tarball is
> created.  But otherwise yeah, patch’n’repack.

A better solution would be to have cargo honor SOURCE_DATE_EPOCH,
perhaps?  They'd probably accept such an improvement upstream.

-- 
Thanks,
Maxim

Message #17 received at 50015 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 50015 <at> debbugs.gnu.org, Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: bug#50015: Rust packages are not reproducible
Date: Wed, 4 Oct 2023 15:06:45 +0300

[Message part 1 (text/plain, inline)]

On Tue, Oct 03, 2023 at 11:30:15PM -0400, Maxim Cournoyer wrote:
> Hello,
> 
> Ludovic Courtès <ludo <at> gnu.org> writes:
> 
> > Hello!
> >
> > Efraim Flashner <efraim <at> flashner.co.il> skribis:
> >
> >> I tried patching this a couple of ways, but it looks like the best
> >> option is going to be a 'patch-and-repack phase after 'install. the
> >> .crate file is really a gzip tarball, and I suspect that each time we
> >> run 'cargo <something>' the timestamp gets updated.
> >
> > So that ‘Cargo.toml’ file is not something taken from the build tree?
> > In that case we could reset the timestamp before the tarball is
> > created.  But otherwise yeah, patch’n’repack.
> 
> A better solution would be to have cargo honor SOURCE_DATE_EPOCH,
> perhaps?  They'd probably accept such an improvement upstream.

That'd be an interesting idea, having 'cargo package' set the timestamp
of all the files to SOURCE_DATE_EPOCH.  I guess I can look into how
feasible that would be and if they'd be likely to accept a change like
that.

I have a local patch which unpacks, resets the timestamp and repacks the
crate. I'll definitely push it to the rust-team branch before the next
merge.

With it I introduced an issue where the 'package phase would repack all
the crates, not just the current one, and ran into our
underscore-to-dash naming convention causing issues with how I'm reusing
the filename to work on the crate. I'll fix that, probably by only
repacking the current crate instead of all crates in the environment.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 50015 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Efraim Flashner <efraim <at> flashner.co.il>
Cc: 50015 <at> debbugs.gnu.org, Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: bug#50015: Rust packages are not reproducible
Date: Wed, 04 Oct 2023 11:05:01 -0400

Hi Efraim,

Efraim Flashner <efraim <at> flashner.co.il> writes:

> On Tue, Oct 03, 2023 at 11:30:15PM -0400, Maxim Cournoyer wrote:
>> Hello,
>> 
>> Ludovic Courtès <ludo <at> gnu.org> writes:
>> 
>> > Hello!
>> >
>> > Efraim Flashner <efraim <at> flashner.co.il> skribis:
>> >
>> >> I tried patching this a couple of ways, but it looks like the best
>> >> option is going to be a 'patch-and-repack phase after 'install. the
>> >> .crate file is really a gzip tarball, and I suspect that each time we
>> >> run 'cargo <something>' the timestamp gets updated.
>> >
>> > So that ‘Cargo.toml’ file is not something taken from the build tree?
>> > In that case we could reset the timestamp before the tarball is
>> > created.  But otherwise yeah, patch’n’repack.
>> 
>> A better solution would be to have cargo honor SOURCE_DATE_EPOCH,
>> perhaps?  They'd probably accept such an improvement upstream.
>
> That'd be an interesting idea, having 'cargo package' set the timestamp
> of all the files to SOURCE_DATE_EPOCH.  I guess I can look into how
> feasible that would be and if they'd be likely to accept a change like
> that.
>
> I have a local patch which unpacks, resets the timestamp and repacks the
> crate. I'll definitely push it to the rust-team branch before the next
> merge.
>
> With it I introduced an issue where the 'package phase would repack all
> the crates, not just the current one, and ran into our
> underscore-to-dash naming convention causing issues with how I'm reusing
> the filename to work on the crate. I'll fix that, probably by only
> repacking the current crate instead of all crates in the environment.

From past interactions with Rust developers (via their web-based chat
system), I could get some change merged rather easily.  If you are
motivated to fix it cleanly I encourage you to pursue that way!

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.