Subject: [bug#33066] [PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues

From: Nikolai Merinov <[email-masked]>
Date: Sat, 20 Oct 2018 00:44:45 +0500

Hi Danny,

I fixed patches according to your comments. Also I updated rust 1.29.1
to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
reproducibility of each package previously only in private my private
repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
slow machine, but currently I already checked reproducibility with
suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.

Both updated patches attached.

Danny Milosavljevic <[email-masked]> writes:

> Hi Nikolai,
>
> On Tue, 16 Oct 2018 02:32:11 +0500
> Nikolai Merinov <[email-masked]> wrote:
>
>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>
> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>
Fixed.

>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>
> "New file".
Fixed.

>
>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>
> Hmm, I'm not sure about doing this in the same commit.
> Is it also related to reproducibility?

Looks like it should not, but still I started investigation with
non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
related changes at very beginning and never tested wihtout it. I not
sure that it's good idea to remove it now.

>
>> during cargo build. Note: libgit2 still builded as part of cargo build,
>> because cargo tests assume specific libgit2 minor release.
>
> What does this mean?  Does it mean "bundled"?
Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).

>
>> (rust-1.23): inherit native-inputs from previous package.
>
> Ok.
>
>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>> https://github.com/rust-lang/rust/issues/50556 issue.
>
> Please add the reasoning as a comment inside the source code instead.
Added comment for rust-1.25 package.

>
>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>
> Maybe add "[source]".
I repharase this comment and added source URL to patch file itself.

>
>> -             (add-after 'configure 'enable-codegen-tests
>> -               (lambda _
>> -                 (substitute* "config.toml"
>> -                   (("codegen-tests = false") ""))
>> -                 #t))
>
> I think I had reproducibility problems when enabling codegen tests and
> parallel tests.  Is that not the case anymore?
Neither me nor Joe Hillenbrand in
https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got
reproducibility issues with this tests.
>
>>               ;; FIXME: Re-enable this test if it's indeed supposed to work.
>>               ;; See <https://github.com/rust-lang/rust/issues/54178>.
>
> Note to myself: I think the issue comments indicate that the newer gdb output
> is better - so we should create a patch similar to
> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.

>From 253286c455a32007f28eeff965f4b33e07856f58 Mon Sep 17 00:00:00 2001
From: Nikolai Merinov <[email-masked]>
Date: Tue, 16 Oct 2018 02:32:11 +0500
Subject: [PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues
To: [email-masked]

* gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch:
New file.
* gnu/local.mk (dist_patch_DATA): Add new patch file.
* gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
during cargo build. Note: libgit2 still bundled, because cargo
tests assume specific libgit2 minor release.
(rust-1.23): Inherit native-inputs from previous package.
(rust-1.25): Switch back to llvm 3.9.1 as workaround for
https://github.com/rust-lang/rust/issues/50556 issue.
(rust-1.27): Apply changes from
https://github.com/rust-lang-nursery/mdBook/pull/692 to make
generation of "searchindex.js" files reproducible. Disable cargo
test that required llvm 6.
---
 gnu/local.mk                                  |  1 +
 ...ible-builds-by-forcing-window.search.patch | 28 +++++++++++++
 gnu/packages/rust.scm                         | 41 ++++++++-----------
 3 files changed, 47 insertions(+), 23 deletions(-)
 create mode 100644 gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 1fa2eaba3..8695ce472 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1130,6 +1130,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-1.25-accept-more-detailed-gdb-lines.patch \
   %D%/packages/patches/rust-bootstrap-stage0-test.patch		\
   %D%/packages/patches/rust-coresimd-doctest.patch		\
+  %D%/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch \
   %D%/packages/patches/rxvt-unicode-escape-sequences.patch	\
   %D%/packages/patches/scheme48-tests.patch			\
   %D%/packages/patches/scotch-test-threading.patch		\
diff --git a/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch b/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch
new file mode 100644
index 000000000..b4ae3a4b8
--- /dev/null
+++ b/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch
@@ -0,0 +1,28 @@
+From b9ca108fcae2b738ca3f0c88c84ae5dc5a6f843f Mon Sep 17 00:00:00 2001
+From: Tim Ryan <[email-masked]>
+Date: Mon, 14 May 2018 06:22:21 -0400
+Subject: [PATCH] Support reproducible builds by forcing window.search to use
+ stable key ordering. (#692)
+See <https://github.com/rust-lang-nursery/mdBook/pull/692>
+---
+ src/vendor/mdbook/src/renderer/html_handlebars/search.rs | 4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+index d49772f8b..1ee66a511 100644
+--- a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
++++ b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+@@ -205,6 +205,10 @@ fn write_to_js(index: Index, search_config: &Search) -> Result<String> {
+         searchoptions,
+         index,
+     };
++
++    // By converting to serde_json::Value as an intermediary, we use a
++    // BTreeMap internally and can force a stable ordering of map keys.
++    let json_contents = serde_json::to_value(&json_contents)?;
+     let json_contents = serde_json::to_string(&json_contents)?;
+ 
+     Ok(format!("window.search = {};", json_contents))
+-- 
+2.19.0
+
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index 40633854a..190dbcc3a 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -480,7 +480,8 @@ test = { path = \"../libtest\" }
      `(("jemalloc" ,jemalloc-4.5.0)
        ("llvm" ,llvm-3.9.1)
        ("openssl" ,openssl)
-       ("libcurl" ,curl))) ; For "cargo"
+       ("libssh2" ,libssh2) ; For "cargo"
+       ("libcurl" ,curl)))  ; For "cargo"
 
     ;; rustc invokes gcc, so we need to set its search paths accordingly.
     ;; Note: duplicate its value here to cope with circular dependencies among
@@ -649,18 +650,11 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
     (name "rust")
     (version "1.23.0")
     (source (rust-source version "14fb8vhjzsxlbi6yrn1r6fl5dlbdd1m92dn5zj5gmzfwf4w9ar3l"))
+    ;; Use [email-masked] package to build rust 1.23
     (native-inputs
-     `(("bison" ,bison) ; For the tests
-       ("cmake" ,cmake)
-       ("flex" ,flex) ; For the tests
-       ("gdb" ,gdb)   ; For the tests
-       ("git" ,git)
-       ("procps" ,procps) ; For the tests
-       ("python-2" ,python-2)
-       ("rustc-bootstrap" ,rust-bootstrap)
-       ("cargo-bootstrap" ,rust-bootstrap "cargo")
-       ("pkg-config" ,pkg-config) ; For "cargo"
-       ("which" ,which)))
+     (alist-replace "cargo-bootstrap" (list rust-bootstrap "cargo")
+                    (alist-replace "rustc-bootstrap" (list rust-bootstrap)
+                                   (package-native-inputs rust-1.20))))
     (arguments
      (substitute-keyword-arguments (package-arguments rust-1.20)
        ((#:phases phases)
@@ -692,6 +686,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                    (("fn test_loading_cosine") "#[ignore]\nfn test_loading_cosine"))
                  #t)))))))))
 
+;;; Rust 1.25 release support work with llvm 6, but builds with llvm 6 is
+;;; not determenistic due to https://github.com/rust-lang/rust/issues/50556
+;;; Keep using llvm 3.9.1 until builds become determenistic
 (define-public rust-1.25
   (let ((base-rust
          (rust-bootstrapped-package rust-1.24 "1.25.0"
@@ -699,10 +696,6 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
           #:patches '("rust-1.25-accept-more-detailed-gdb-lines.patch"))))
     (package
       (inherit base-rust)
-      (inputs
-       ;; Use LLVM 6.0
-       (alist-replace "llvm" (list llvm)
-                      (package-inputs base-rust)))
       (arguments
        (substitute-keyword-arguments (package-arguments base-rust)
          ((#:phases phases)
@@ -713,14 +706,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                    ;; This test wants to update the crate index.
                    (("fn no_index_update") "#[ignore]\nfn no_index_update"))
                  #t))
-             (add-after 'configure 'enable-codegen-tests
-               (lambda _
-                 (substitute* "config.toml"
-                   (("codegen-tests = false") ""))
-                 #t))
              ;; FIXME: Re-enable this test if it's indeed supposed to work.
              ;; See <https://github.com/rust-lang/rust/issues/54178>.
-             (add-after 'enable-codegen-tests 'disable-nil-enum-test
+             (add-after 'configure 'disable-nil-enum-test
                (lambda _
                  (substitute* "src/test/debuginfo/nil-enum.rs"
                    (("ignore-lldb") "ignore-gdb"))
@@ -795,7 +783,8 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                                     #:patches
                                     '("rust-coresimd-doctest.patch"
                                       "rust-bootstrap-stage0-test.patch"
-                                      "rust-1.25-accept-more-detailed-gdb-lines.patch"))))
+                                      "rust-1.25-accept-more-detailed-gdb-lines.patch"
+                                      "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
     (package
       (inherit base-rust)
       (arguments
@@ -808,4 +797,10 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                  ;; `prefix' directory should exist before `install' call
                  (mkdir-p (assoc-ref outputs "out"))
                  (mkdir-p (assoc-ref outputs "cargo"))
+                 #t))
+             (add-after 'patch-cargo-tests 'disable-thinlto-test
+               (lambda* _
+                 ;; thinlto required llvm 6.0 for work
+                 (substitute* "src/tools/cargo/tests/testsuite/path.rs"
+                   (("fn thin_lto_works") "#[ignore]\nfn thin_lto_works"))
                  #t)))))))))
-- 
2.19.0

>From 9cf7f62414c5e714d7bb0cc2874d91f561e418df Mon Sep 17 00:00:00 2001
From: Nikolai Merinov <[email-masked]>
Date: Tue, 16 Oct 2018 02:37:41 +0500
Subject: [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages
To: [email-masked]

* gnu/packages/rust.scm (rust-1.27): rename "rust" to "rust-1.27"
(rust-1.28): Use llvm-6.0 finally. Disable test that broken on x86_64 machines
without "avx". Enable codegen tests. Enable cargo test for thinlto.
(rust): Package for rust 1.29.2.
---
 gnu/packages/rust.scm | 46 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 45 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index 190dbcc3a..a0e891249 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -776,7 +776,7 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                    (("fn no_index_update") "#[ignore]\nfn no_index_update"))
                  #t)))))))))
 
-(define-public rust
+(define-public rust-1.27
   (let ((base-rust
          (rust-bootstrapped-package rust-1.26 "1.27.2"
                                     "0pg1s37bhx9zqbynxyydq5j6q7kij9vxkcv8maz0m25prm88r0cs"
@@ -804,3 +804,47 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
                  (substitute* "src/tools/cargo/tests/testsuite/path.rs"
                    (("fn thin_lto_works") "#[ignore]\nfn thin_lto_works"))
                  #t)))))))))
+
+(define-public rust-1.28
+  (let ((base-rust
+         (rust-bootstrapped-package rust-1.27 "1.28.0"
+                                    "11k4rn77bca2rikykkk9fmprrgjswd4x4kaq7fia08vgkir82nhx"
+                                    #:patches
+                                    '("rust-coresimd-doctest.patch"
+                                      "rust-bootstrap-stage0-test.patch"
+                                      "rust-1.25-accept-more-detailed-gdb-lines.patch"
+                                      "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+    (package
+      (inherit base-rust)
+      (inputs
+       ;; Use LLVM 6.0
+       (alist-replace "llvm" (list llvm)
+                      (package-inputs base-rust)))
+      (arguments
+       (substitute-keyword-arguments (package-arguments base-rust)
+         ((#:phases phases)
+          `(modify-phases ,phases
+             (add-after 'configure 'enable-codegen-tests
+               ;; codegen tests should pass with llvm 6
+               (lambda* _
+                 (substitute* "config.toml"
+                   (("codegen-tests = false") ""))
+                 #t))
+             (add-after 'patch-tests 'disable-amd64-avx-test
+               ;; This test will fail on x86_64 machines without avx
+               (lambda* _
+                 (substitute* "src/test/run-pass/issue-44056.rs"
+                   (("only-x86_64") "ignore-test"))
+                 #t))
+             ;; thinlto test should pass with llvm 6
+             (delete 'disable-thinlto-test))))))))
+
+(define-public rust
+  (let ((base-rust
+         (rust-bootstrapped-package rust-1.28 "1.29.2"
+                                    "1jb787080z754caa2w3w1amsygs4qlzj9rs1vy64firfmabfg22h"
+                                    #:patches
+                                    '("rust-1.25-accept-more-detailed-gdb-lines.patch"
+                                      "rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch"))))
+    (package
+      (inherit base-rust))))
-- 
2.19.0