GNU bug report logs - #44112
SBCL is not reproducible

Previous Next

To reply to this bug, email your comments to 44112 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: bug-guix <at> gnu.org
Cc: glv <at> posteo.net, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: SBCL is not reproducible
Date: Wed, 21 Oct 2020 10:12:14 +0200

Dear,

Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.

The output of “guix challenge” is:

--8<---------------cut here---------------start------------->8---
  differing files:
    /lib/sbcl/contrib/sb-rt.fasl
    /lib/sbcl/contrib/uiop.fasl
    /lib/sbcl/contrib/sb-cover.fasl
    /lib/sbcl/contrib/sb-capstone.fasl
    /lib/sbcl/contrib/sb-executable.fasl
    /lib/sbcl/contrib/sb-bsd-sockets.fasl
    /lib/sbcl/contrib/sb-simple-streams.fasl
    /lib/sbcl/contrib/sb-queue.fasl
    /lib/sbcl/contrib/sb-gmp.fasl
    /lib/sbcl/contrib/sb-aclrepl.fasl
    /lib/sbcl/contrib/sb-sprof.fasl
    /lib/sbcl/contrib/sb-grovel.fasl
    /lib/sbcl/contrib/sb-posix.fasl
    /lib/sbcl/contrib/asdf.fasl
    /lib/sbcl/contrib/sb-mpfr.fasl
    /lib/sbcl/contrib/sb-cltl2.fasl
    /lib/sbcl/contrib/sb-concurrency.fasl
    /lib/sbcl/contrib/sb-md5.fasl
    /lib/sbcl/contrib/sb-introspect.fasl
    /lib/sbcl/contrib/sb-rotate-byte.fasl
    /lib/sbcl/sbcl.core
    /bin/sbcl
    /share/sbcl/src/runtime/coreparse.o
    /share/sbcl/src/runtime/sbcl

1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive
--8<---------------cut here---------------end--------------->8---

which is confirmed by the experimental Data Service webpage:

https://data.guix-patches.cbaines.net/revision/4e0498c4d3f5ef03ac5afa4a3e570205c10c28c5/package-derivation-outputs?search_query=sbcl&output_consistency=not-matching&system=x86_64-linux&target=none&after_path=&limit_results=&all_results=on


I do not know if the patches in ’staging’ will fix this.

Note that this issue does not imply that the build system
’asdf-build-system/sbcl’ is or will be not reproducible.  However, this
issues cuts any Diverse Double Compiling attempt.

All the best,
simon

Message #8 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Guillaume Le Vaillant <glv <at> posteo.net>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 44112 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: Re: SBCL is not reproducible
Date: Wed, 21 Oct 2020 14:41:48 +0200

[Message part 1 (text/plain, inline)]

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.
>
> [...]
>
> I do not know if the patches in ’staging’ will fix this.
>
> Note that this issue does not imply that the build system
> ’asdf-build-system/sbcl’ is or will be not reproducible.  However, this
> issues cuts any Diverse Double Compiling attempt.

IIRC, when compiling, SBCL puts the timestamp of the source file in the
compiled file. It's not a problem when just doing basic compilation
because the source files' timestamps have been set to "1970-01-01".

However, some packages generate some source files at build time, usually
containing things like data type sizes fetched from system header in
order to use C libraries with FFI. The timestamp of a generated file
is the current time, therefore the build is not reproducible.

IIRC, SBCL itself is built in 2 stages. First its core is compiled
using another Common Lisp implementation (currently clisp in Guix), then
the complete SBCL is compiled using the core compiled in stage 1. There
is probably also an embedded timestamp issue here (coming for clisp,
from SBCL, or both) causing the reproducibility issue.

The patches currently in the staging branch don't have any effect on
the generation of source files or on the format of the compiled files,
so they will not help with this issue.

Removing this source file timestamp from compiled files would simplify
things. Maybe nothing really depends on it and it would be possible...

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Guillaume Le Vaillant <glv <at> posteo.net>
Cc: 44112 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: Re: SBCL is not reproducible
Date: Wed, 21 Oct 2020 15:37:41 +0200

Hi,

Thank you for the explanations.

On Wed, 21 Oct 2020 at 14:42, Guillaume Le Vaillant <glv <at> posteo.net> wrote:

> However, some packages generate some source files at build time, usually
> containing things like data type sizes fetched from system header in
> order to use C libraries with FFI. The timestamp of a generated file
> is the current time, therefore the build is not reproducible.

This is an issue.  Do you think it is affordable to fix these
timestamps to 1970-01-01?


> IIRC, SBCL itself is built in 2 stages. First its core is compiled
> using another Common Lisp implementation (currently clisp in Guix), then
> the complete SBCL is compiled using the core compiled in stage 1. There
> is probably also an embedded timestamp issue here (coming for clisp,
> from SBCL, or both) causing the reproducibility issue.

Yes.  But I have replaced this "clisp" by ECL or by SBCL itself.
Still unreproducible.

Out-of-scope with this bug report, my aim is to have a fixed point:

 - first compile SBCL with CLISP (using the 2 stages you describe):
produce SBCL-A
 - second recompile SBCL with the previous SBCL-A (again using the 2
stages): produce SBCL-B
 - third recompile SBCL with the previous SBCL-B (again using the 2
stages): produces SBCL-C

The binary SBCL-A is not deterministic probably because of CLISP (and
CLISP should be fixed but that's another story :-)).
However, SBCL-B and SBCL-C must be deterministic.  And ideally
bit-to-bit identical but that's another story. :-)
And they are not; from my experiments at least.

Even, you could do the same procedure replacing CLISP by ECL,
producing SBCL-{A,B,C}-bis.  Then SBCL-C and SBCL-C-bis should be
bit-to-bit identical.


> Removing this source file timestamp from compiled files would simplify
> things. Maybe nothing really depends on it and it would be possible...

Thanks for the explanation.


All the best,
simon

Message #14 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Guillaume Le Vaillant <glv <at> posteo.net>
Cc: 44112 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: Re: bug#44112: SBCL is not reproducible
Date: Wed, 21 Oct 2020 15:43:24 +0200

On Wed, 21 Oct 2020 at 15:39, zimoun <zimon.toutoune <at> gmail.com> wrote:

> > IIRC, SBCL itself is built in 2 stages. First its core is compiled
> > using another Common Lisp implementation (currently clisp in Guix), then
> > the complete SBCL is compiled using the core compiled in stage 1. There
> > is probably also an embedded timestamp issue here (coming for clisp,
> > from SBCL, or both) causing the reproducibility issue.

In addition, GCC is also involved in the party.  And I have also
replaced it by Clang with the same effect.

Cheers,
simon

Message #17 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: zimoun <zimon.toutoune <at> gmail.com>, Guillaume Le Vaillant <glv <at> posteo.net>
Cc: 44112 <at> debbugs.gnu.org
Subject: Re: bug#44112: SBCL is not reproducible
Date: Wed, 21 Oct 2020 16:32:06 +0200

[Message part 1 (text/plain, inline)]

zimoun <zimon.toutoune <at> gmail.com> writes:

> In addition, GCC is also involved in the party.  And I have also
> replaced it by Clang with the same effect.

Where?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: Guillaume Le Vaillant <glv <at> posteo.net>, zimoun <zimon.toutoune <at> gmail.com>
Cc: 44112 <at> debbugs.gnu.org
Subject: Re: SBCL is not reproducible
Date: Wed, 21 Oct 2020 16:32:26 +0200

[Message part 1 (text/plain, inline)]

Guillaume Le Vaillant <glv <at> posteo.net> writes:

> IIRC, SBCL itself is built in 2 stages. First its core is compiled
> using another Common Lisp implementation (currently clisp in Guix), then
> the complete SBCL is compiled using the core compiled in stage 1.

Yes, this is correct.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Pierre Neidhardt <mail <at> ambrevar.xyz>
Cc: Guillaume Le Vaillant <glv <at> posteo.net>, 44112 <at> debbugs.gnu.org
Subject: Re: bug#44112: SBCL is not reproducible
Date: Wed, 21 Oct 2020 17:13:17 +0200

On Wed, 21 Oct 2020 at 16:32, Pierre Neidhardt <mail <at> ambrevar.xyz> wrote:

> > In addition, GCC is also involved in the party.  And I have also
> > replaced it by Clang with the same effect.
>
> Where?

--8<---------------cut here---------------start------------->8---
(define-module (ddc-sbcl)
  #:use-module (guix packages)
  #:use-module (guix utils)
  #:use-module ((guix build utils) #:select (modify-phases))
  #:use-module (srfi srfi-1)
  #:use-module (ice-9 match)

  #:use-module (gnu packages lisp)
  #:use-module (gnu packages commencement)
  #:use-module (gnu packages llvm))

(define  (make-sbcl name string-compil compil cc)
  (package
    (inherit sbcl)
    (name name)
    (arguments
     (substitute-keyword-arguments (package-arguments sbcl)
       ((#:phases phases)
        `(modify-phases ,phases
           (replace 'build
             (lambda* (#:key outputs #:allow-other-keys)
             (setenv "CC" ,cc)
             (invoke "sh" "make.sh" ,string-compil
                     (string-append "--prefix="
                                    (assoc-ref outputs "out"))
                     "--dynamic-space-size=2Gb"
                     "--with-sb-core-compression"
                     "--with-sb-xref-for-internals")))))))
    (native-inputs
     `(,@(alist-delete "clisp" (package-native-inputs sbcl))
       ("compil" ,compil)))))


(define-public sbcl-A
  (package-with-c-toolchain
   (make-sbcl "sbcl-A" "clisp" clisp "gcc")
   `(("gcc" ,gcc-toolchain))))      ;fail with 6

(define-public sbcl-B
  (package-with-c-toolchain
   (make-sbcl "sbcl-B" "ecl" ecl "clang")
   `(("clang" ,clang-toolchain))))



(define-public sbcl-AA
  (package-with-c-toolchain
   (make-sbcl "sbcl-AA" "sbcl" sbcl-A "gcc")
   `(("gcc" ,gcc-toolchain))))

(define-public sbcl-BB
  (package-with-c-toolchain
   (make-sbcl "sbcl-BB" "sbcl" sbcl-B "clang")
   `(("clang" ,clang-toolchain))))


(define-public sbcl-CC
  (package-with-c-toolchain
   (make-sbcl "sbcl-CC" "sbcl" sbcl-A "clang")
   `(("clang" ,clang-toolchain))))

(define-public sbcl-DD
  (package-with-c-toolchain
   (make-sbcl "sbcl-DD" "sbcl" sbcl-B "gcc")
   `(("gcc" ,gcc-toolchain))))
--8<---------------cut here---------------end--------------->8---

Message #26 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Guillaume Le Vaillant <glv <at> posteo.net>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 44112 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: Re: bug#44112: SBCL is not reproducible
Date: Wed, 21 Oct 2020 18:45:43 +0200

[Message part 1 (text/plain, inline)]

zimoun <zimon.toutoune <at> gmail.com> skribis:

>> Removing this source file timestamp from compiled files would simplify
>> things. Maybe nothing really depends on it and it would be possible...
>
>Thanks for the explanation.

A developer of SBCL agrees that the timestamp should be removed [1], but
currently Slime has a function depending on it. I asked if this could
be fixed [2]. We'll see...

[1] https://sourceforge.net/p/sbcl/mailman/message/37133640/
[2] https://github.com/slime/slime/issues/583

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: Guillaume Le Vaillant <glv <at> posteo.net>, zimoun <zimon.toutoune <at> gmail.com>
Cc: 44112 <at> debbugs.gnu.org
Subject: Re: bug#44112: SBCL is not reproducible
Date: Wed, 21 Oct 2020 18:55:40 +0200

[Message part 1 (text/plain, inline)]

Guillaume Le Vaillant <glv <at> posteo.net> writes:

> A developer of SBCL agrees that the timestamp should be removed [1], but
> currently Slime has a function depending on it. I asked if this could
> be fixed [2]. We'll see...
>
> [1] https://sourceforge.net/p/sbcl/mailman/message/37133640/
> [2] https://github.com/slime/slime/issues/583

Thank you so much for asking!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[signature.asc (application/pgp-signature, inline)]

Message #32 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Guillaume Le Vaillant <glv <at> posteo.net>
Cc: 44112 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: Re: bug#44112: SBCL is not reproducible
Date: Thu, 22 Oct 2020 13:19:08 +0200

Hi Guillaume,

Thank you for asking upstream.


On Wed, 21 Oct 2020 at 18:45, Guillaume Le Vaillant <glv <at> posteo.net> wrote:

> A developer of SBCL agrees that the timestamp should be removed [1], but
> currently Slime has a function depending on it. I asked if this could
> be fixed [2]. We'll see...
>
> [1] https://sourceforge.net/p/sbcl/mailman/message/37133640/

Ouch!  Hard to read when not subscribed. :-)

> [2] https://github.com/slime/slime/issues/583

If upstream does not want to patch, do you think it is doable to patch
SLIME and maintain the patch with the Guix package?

Thanks,
simon

Message #35 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Pierre Neidhardt <mail <at> ambrevar.xyz>
To: zimoun <zimon.toutoune <at> gmail.com>, Guillaume Le Vaillant <glv <at> posteo.net>
Cc: 44112 <at> debbugs.gnu.org
Subject: Re: bug#44112: SBCL is not reproducible
Date: Thu, 22 Oct 2020 14:13:49 +0200

[Message part 1 (text/plain, inline)]

zimoun <zimon.toutoune <at> gmail.com> writes:

> If upstream does not want to patch, do you think it is doable to patch
> SLIME and maintain the patch with the Guix package?

Accepted or not, we also need to ping the SLY maintainer to apply the
same change.  (SLY is a fork of SLIME.)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[signature.asc (application/pgp-signature, inline)]

Message #38 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Guillaume Le Vaillant <glv <at> posteo.net>
Cc: 44112 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: Re: bug#44112: SBCL is not reproducible
Date: Tue, 19 Jan 2021 17:48:01 +0100

Hi,

On Wed, 21 Oct 2020 at 14:41, Guillaume Le Vaillant <glv <at> posteo.net> wrote:
> zimoun <zimon.toutoune <at> gmail.com> skribis:
>
>> Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.

[...]

> Removing this source file timestamp from compiled files would simplify
> things. Maybe nothing really depends on it and it would be possible...

Any news from upstream?  Still unreproducible for me.


All the best,
simon

Message #41 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Guillaume Le Vaillant <glv <at> posteo.net>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 44112 <at> debbugs.gnu.org, Pierre Neidhardt <mail <at> ambrevar.xyz>
Subject: Re: bug#44112: SBCL is not reproducible
Date: Sat, 23 Jan 2021 10:19:37 +0100

[Message part 1 (text/plain, inline)]

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Hi,
>
> On Wed, 21 Oct 2020 at 14:41, Guillaume Le Vaillant <glv <at> posteo.net> wrote:
>> zimoun <zimon.toutoune <at> gmail.com> skribis:
>>
>>> Using Guix 58af4c9, the package ’sbcl’ seems not-reproducible.
>
> [...]
>
>> Removing this source file timestamp from compiled files would simplify
>> things. Maybe nothing really depends on it and it would be possible...
>
> Any news from upstream?  Still unreproducible for me.
>
>
> All the best,
> simon

So far Slime still uses the function getting the source file timestamp,
so these timestamps can't be removed from SBCL.
A patch for Slime would be accepted (see [2]), but as far as I know
nobody has worked on it yet.

[2] https://github.com/slime/slime/issues/583

[signature.asc (application/pgp-signature, inline)]

Message #44 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: Hendursaga <hendursaga <at> aol.com>
To: 44112 <at> debbugs.gnu.org
Date: Fri, 25 Nov 2022 12:22:20 -0500

The patch[1] has been closed for some time, is there any other pending patches blocking going forward? 

[1] https://github.com/slime/slime/issues/583

Message #47 received at 44112 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Hendursaga <hendursaga <at> aol.com>, 44112 <at> debbugs.gnu.org
Cc: glv <at> posteo.net
Subject: Re: bug#44112:
Date: Fri, 09 Dec 2022 19:50:37 +0100

Hi,

Thank you for the follow up.

On Fri, 25 Nov 2022 at 12:22, Hendursaga via Bug reports for GNU Guix <bug-guix <at> gnu.org> wrote:
> The patch[1] has been closed for some time, is there any other pending patches blocking going forward? 
>
> [1] https://github.com/slime/slime/issues/583

Indeed, but the package sbcl is not yet reproducible.  The remaining
issue is about PDF.

Using Guix 28d1e51, I get:

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference /gnu/store/0q9z6j5ml4rkfrg57axznasdjfm7x803-sbcl-2.2.11-doc{,-check}
Binary files /gnu/store/0q9z6j5ml4rkfrg57axznasdjfm7x803-sbcl-2.2.11-doc/share/doc/sbcl/sbcl.pdf and /gnu/store/0q9z6j5ml4rkfrg57axznasdjfm7x803-sbcl-2.2.11-doc-check/share/doc/sbcl/sbcl.pdf differ
--8<---------------cut here---------------end--------------->8---

Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.