Subject: bug#46806: closed (Re: [bug#46806] [PATCH] gnu: ocaml-4.07: Bootstrap.)

From: [email-masked] (GNU bug Tracking System)
Date: Wed, 03 Mar 2021 16:15:03 +0000

Your bug report

#46806: [PATCH] gnu: ocaml-4.07: Bootstrap.

which was filed against the guix-patches package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
46806: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=46806
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: [bug#46806] [PATCH] gnu: ocaml-4.07: Bootstrap.
From: Julien Lepiller <[email-masked]>
To: zimoun <[email-masked]>
Cc: [email-masked]
Date: Wed, 3 Mar 2021 17:14:16 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

Le Wed, 3 Mar 2021 15:05:18 +0100,
zimoun <[email-masked]> a écrit :

> On Wed, 3 Mar 2021 at 14:46, Julien Lepiller <[email-masked]>
> wrote:
> >
> > We'd have to check we actually can build them. The plan was to
> > build menhir with this ocaml4.07, since it's required to generate
> > the parser for ocaml >= 4.08, and improve camlboot to support more
> > recent versions. We would prevent the need for a chain of
> > bootstraps that way.  
> 
> Thanks for the explanations.
> I have tried to replace gcc by clang for camlboot but it fails.
> And also tried other Scheme than Guile, obviously fails. :-)
> 
> 
> > I don't think ocaml has so many dependants that it requires
> > core-updates. Not even sure it needs staging.  
> 
> Oh right, I thought that more OCaml packages were in Guix. :-)
> 
> Cheers,
> simon

Pushed as 0bd58ae7ff0d5c3935b0d660819aca4a0493ef8a, thanks for the
review!

Yeah, we might need some work to support other schemes. There's no C in
camlboot, but we reuse some of OCaml's sources, for the bytecode
interpreter (camlrun) and parser generator (ocamlyacc). So that's a
problem with OCaml.

Subject: [PATCH] gnu: ocaml-4.07: Bootstrap.
From: Julien Lepiller <[email-masked]>
To: [email-masked]
Date: Sat, 27 Feb 2021 00:13:39 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi Guix!

This small series adds camlboot, a project to reimplement a bootstrap
for OCaml, which lets us remove the pre-built boot/ocaml{c,lex} :)

The first patch builds the bootstraped ocamlc and ocamllex. This takes
around 4 hours to build. The second patch rebuilds them using the source
code for ocaml 4.07.1, instead of the modified sources used in
camlboot, and reuse them to run the main Makefile (make world.opt).

As a result, we have identical files for this bootstrap and the
unbootstrapped OCaml (up to output store paths and hash of some files
that get embedded in native files, which differ because of the
different output path).
>From 8fd06caa83e55d27dc5998f7548a7d45c55c6a91 Mon Sep 17 00:00:00 2001
From: Julien Lepiller <[email-masked]>
Date: Sat, 27 Feb 2021 00:07:20 +0100
Subject: [PATCH 1/2] gnu: Add camlboot.

* gnu/packages/ocaml.scm (camlboot): New variable.
---
 gnu/packages/ocaml.scm | 66 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/gnu/packages/ocaml.scm b/gnu/packages/ocaml.scm
index 9e68359a42..01c14a0ba0 100644
--- a/gnu/packages/ocaml.scm
+++ b/gnu/packages/ocaml.scm
@@ -52,6 +52,7 @@
   #:use-module (gnu packages glib)
   #:use-module (gnu packages gnome)
   #:use-module (gnu packages gtk)
+  #:use-module (gnu packages guile)
   #:use-module (gnu packages libevent)
   #:use-module (gnu packages libffi)
   #:use-module (gnu packages llvm)
@@ -105,6 +106,71 @@
                               ".tar.gz"))
           (sha256 (base32 hash))))
 
+(define-public camlboot
+  (let ((commit "506280c6e0813e0e794988151a8e46be55373ebc")
+        (revision "0"))
+    (package
+      (name "camlboot")
+      (version (git-version "0.0.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                       (url "https://github.com/Ekdohibs/camlboot")
+                       (commit commit)
+                       (recursive? #t)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0vimxl4karw9ih3npyc5rhxg85cjh6fqjbq3dzj7j2iymlhwfbkv"))
+                (modules '((guix build utils)))
+                (snippet
+                 `(begin
+                    ;; Remove bootstrap binaries and pre-generated source files,
+                    ;; to ensure we actually bootstrap properly.
+                    (for-each delete-file (find-files "ocaml-src" "^.depend$"))
+                    (delete-file "ocaml-src/boot/ocamlc")
+                    (delete-file "ocaml-src/boot/ocamllex")
+                    ;; Ensure writable
+                    (for-each
+                      (lambda (file)
+                        (chmod file (logior (stat:mode (stat file)) #o200)))
+                      (find-files "." "."))))))
+      (build-system gnu-build-system)
+      (arguments
+       `(#:make-flags (list "_boot/ocamlc"); build target
+         #:tests? #f; no tests
+         #:phases
+         (modify-phases %standard-phases
+           (delete 'configure)
+           (add-before 'build 'no-autocompile
+             (lambda _
+               ;; prevent a guile warning
+               (setenv "GUILE_AUTO_COMPILE" "0")))
+           (replace 'install
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (bin (string-append out "/bin")))
+                 (mkdir-p bin)
+                 (install-file "_boot/ocamlc" bin)
+                 (rename-file "miniml/interp/lex.byte" "ocamllex")
+                 (install-file "ocamllex" bin)))))))
+      (native-inputs
+       `(("guile" ,guile-3.0)))
+      (home-page "https://github.com/Ekdohibs/camlboot")
+      (synopsis "OCaml bootstrap")
+      (description "OCaml is written in OCaml. Its sources contain a pre-compiled
+bytecode version of ocamlc and ocamllex that are used to build the next version
+of the compiler. Camlboot implements a bootstrap for the OCaml compiler and
+provides a bootstrapped equivalent to these files.
+
+It contains a compiler for a small subset of OCaml written in Guile Scheme,
+an interpreter for OCaml written in that subset and a manually-written lexer
+for OCaml.  These elements eliminate the need for the binary bootstrap in
+OCaml and can effectively bootstrap OCaml 4.07.
+
+This package produces a native ocamlc and a bytecode ocamllex.")
+      (license license:expat))))
+
 (define-public ocaml-4.11
   (package
     (name "ocaml")
-- 
2.30.0

>From 8fa9e59c8d90339478d65e269b37cbb3b201232d Mon Sep 17 00:00:00 2001
From: Julien Lepiller <[email-masked]>
Date: Sat, 27 Feb 2021 00:07:30 +0100
Subject: [PATCH 2/2] gnu: ocaml-4.07: Bootstrap.

* gnu/packages/ocaml.scm (ocaml-4.07): Implement bootstrap via camlboot.
---
 gnu/packages/ocaml.scm | 179 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 176 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/ocaml.scm b/gnu/packages/ocaml.scm
index 01c14a0ba0..272ed3a2d6 100644
--- a/gnu/packages/ocaml.scm
+++ b/gnu/packages/ocaml.scm
@@ -255,7 +255,7 @@ functional, imperative and object-oriented styles of programming.")
                (base32
                 "1v3z5ar326f3hzvpfljg4xj8b9lmbrl53fn57yih1bkbx3gr3yzj"))))))
 
-(define-public ocaml-4.07
+(define ocaml-4.07-boot
   (package
     (inherit ocaml-4.09)
     (version "4.07.1")
@@ -267,11 +267,102 @@ functional, imperative and object-oriented styles of programming.")
                     "/ocaml-" version ".tar.xz"))
               (sha256
                (base32
-                "1f07hgj5k45cylj1q3k5mk8yi02cwzx849b1fwnwia8xlcfqpr6z"))))
+                "1f07hgj5k45cylj1q3k5mk8yi02cwzx849b1fwnwia8xlcfqpr6z"))
+              (modules '((guix build utils)))
+              (snippet
+               `(begin
+                  ;; Remove bootstrap binaries and pre-generated source files,
+                  ;; to ensure we actually bootstrap properly.
+                  (for-each delete-file (find-files "." "^.depend$"))
+                  (delete-file "boot/ocamlc")
+                  (delete-file "boot/ocamllex")))))
+    (arguments
+     `(#:tests? #f
+       #:phases
+       (modify-phases %standard-phases
+         (add-before 'configure 'copy-bootstrap
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let ((camlboot (assoc-ref inputs "camlboot")))
+               (copy-file (string-append camlboot "/bin/ocamllex") "boot/ocamllex")
+               (copy-file (string-append camlboot "/bin/ocamlc") "boot/ocamlc")
+               (chmod "boot/ocamllex" #o755)
+               (chmod "boot/ocamlc" #o755))))
+         (replace 'configure
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (mandir (string-append out "/share/man")))
+               (invoke "./configure"
+                       "--prefix" out
+                       "--mandir" mandir))))
+         (replace 'build
+           (lambda* (#:key parallel-build? #:allow-other-keys)
+             (define* (make . args)
+               (apply invoke "make"
+                      (append (if parallel-build?
+                                  `("-j" ,(number->string (parallel-job-count)))
+                                  '())
+                              args)))
+             (define (touch file) (invoke "touch" file))
+             (touch "byterun/.depend")
+             (make "-C" "byterun" "depend")
+             (make "-C" "byterun" "all")
+             (copy-file "byterun/ocamlrun" "boot/ocamlrun")
+             (touch ".depend")
+             (make "ocamlyacc")
+             (copy-file "yacc/ocamlyacc" "boot/ocamlyacc")
+             (touch "stdlib/.depend")
+             (make "-C" "stdlib" "sys.ml")
+             (make "-C" "stdlib" "CAMLDEP=../boot/ocamlc -depend" "depend")
+             (make "-C" "stdlib" "COMPILER="
+                   "CAMLC=../boot/ocamlc -use-prims ../byterun/primitives"
+                   "all")
+             (for-each
+               (lambda (file)
+                 (copy-file file (string-append "boot/" (basename file))))
+               (cons* "stdlib/stdlib.cma" "stdlib/std_exit.cmo" "stdlib/camlheader"
+                      (find-files "stdlib" ".*.cmi$")))
+             (symlink "../byterun/libcamlrun.a" "boot/libcamlrun.a")
+             (touch "tools/.depend")
+             (make "-C" "tools"
+                   "CAMLC=../boot/ocamlc -nostdlib -I ../boot -use-prims ../byterun/primitives -I .."
+                   "make_opcodes" "cvt_emit")
+             (touch "lex/.depend")
+             (make "-C" "lex" "CAMLDEP=../boot/ocamlc -depend" "depend")
+             (make "CAMLDEP=boot/ocamlc -depend" "depend")
+             (make "CAMLC=boot/ocamlc -nostdlib -I boot -use-prims byterun/primitives"
+                   "ocamlc")
+             (make "-C" "lex"
+                   "CAMLC=../boot/ocamlc -strict-sequence -nostdlib -I ../boot -use-prims ../byterun/primitives"
+                   "all")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin")))
+               (mkdir-p bin)
+               (install-file "ocamlc" bin)
+               (install-file "lex/ocamllex" bin)
+               (install-file "byterun/ocamlrun" bin)))))))
+    (native-inputs
+     `(("camlboot" ,camlboot)
+       ("perl" ,perl)
+       ("pkg-config" ,pkg-config)))))
+
+(define-public ocaml-4.07
+  (package
+    (inherit ocaml-4.07-boot)
     (arguments
       (substitute-keyword-arguments (package-arguments ocaml-4.09)
         ((#:phases phases)
          `(modify-phases ,phases
+            (add-before 'configure 'copy-bootstrap
+              (lambda* (#:key inputs #:allow-other-keys)
+                (let ((ocaml (assoc-ref inputs "ocaml")))
+                  (copy-file (string-append ocaml "/bin/ocamllex") "boot/ocamllex")
+                  (copy-file (string-append ocaml "/bin/ocamlc") "boot/ocamlc")
+                  (copy-file (string-append ocaml "/bin/ocamlrun") "boot/ocamlrun")
+                  (chmod "boot/ocamlrun" #o755)
+                  (chmod "boot/ocamllex" #o755)
+                  (chmod "boot/ocamlc" #o755))))
             (replace 'configure
               (lambda* (#:key outputs #:allow-other-keys)
                 (let* ((out (assoc-ref outputs "out"))
@@ -280,7 +371,89 @@ functional, imperative and object-oriented styles of programming.")
                   ;; --prefix=<PREFIX> syntax (with equals sign).
                   (invoke "./configure"
                           "--prefix" out
-                          "--mandir" mandir))))))))))
+                          "--mandir" mandir))))
+            (add-before 'build 'generate-depend
+              (lambda* (#:key parallel-build? #:allow-other-keys)
+                (define* (make . args)
+                  (apply invoke "make"
+                         (append (if parallel-build?
+                                     `("-j" ,(number->string (parallel-job-count)))
+                                     '())
+                                 args)))
+                ;; These files were deleted in a snippet because they are
+                ;; pre-generated
+                (invoke "touch" ".depend" "stdlib/.depend" "byterun/.depend"
+                        "tools/.depend" "lex/.depend" "asmrun/.depend"
+                        "debugger/.depend" "ocamltest/.depend" "ocamldoc/.depend"
+                        "ocamldoc/stdlib_non_prefixed/.depend"
+                        "otherlibs/bigarray/.depend"
+                        "otherlibs/graph/.depend"
+                        "otherlibs/raw_spacetime_lib/.depend"
+                        "otherlibs/str/.depend"
+                        "otherlibs/systhreads/.depend"
+                        "otherlibs/threads/.depend"
+                        "otherlibs/unix/.depend"
+                        "otherlibs/win32unix/.depend")
+                (make "-C" "byterun" "depend")
+                (make "-C" "asmrun" "depend")
+                (make "-C" "byterun" "all")
+                (copy-file "byterun/ocamlrun" "boot/ocamlrun")
+                (make "ocamlyacc")
+                (copy-file "yacc/ocamlyacc" "boot/ocamlyacc")
+                (make "-C" "stdlib" "sys.ml")
+                (make "-C" "stdlib"
+                      "CAMLDEP=../boot/ocamlc -depend"
+                      "depend")
+                (make "-C" "stdlib" "COMPILER="
+                      "CAMLC=../boot/ocamlc -use-prims ../byterun/primitives"
+                      "all")
+                (for-each
+                  (lambda (file)
+                    (copy-file file (string-append "boot/" (basename file))))
+                  (cons* "stdlib/stdlib.cma" "stdlib/std_exit.cmo" "stdlib/camlheader"
+                         (find-files "stdlib" ".*.cmi$")))
+                (symlink "../byterun/libcamlrun.a" "boot/libcamlrun.a")
+                (make "-C" "lex" "CAMLDEP=../boot/ocamlc -depend" "depend")
+                (make "-C" "lex" "all")
+                (substitute* "tools/Makefile"
+                  (("\\$\\(CAMLRUN\\) ./ocamldep") "../boot/ocamlc -depend"))
+                (substitute* '("otherlibs/graph/Makefile"
+                               "otherlibs/systhreads/Makefile"
+                               "otherlibs/threads/Makefile"
+                               "otherlibs/unix/Makefile")
+                  (("\\$\\(CAMLRUN\\) ../../tools/ocamldep")
+                   "../../boot/ocamlc -depend"))
+                (substitute* '("otherlibs/bigarray/Makefile"
+                               "otherlibs/raw_spacetime_lib/Makefile"
+                               "otherlibs/str/Makefile"
+                               "otherlibs/win32unix/Makefile")
+                  (("\\$\\(CAMLRUN\\) \\$\\(ROOTDIR\\)/tools/ocamldep")
+                   "../../boot/ocamlc -depend"))
+                (make "-C" "tools" "depend")
+                (make "-C" "otherlibs/bigarray" "depend")
+                (make "-C" "otherlibs/graph" "depend")
+                (make "-C" "otherlibs/raw_spacetime_lib" "depend")
+                (make "-C" "otherlibs/str" "depend")
+                (make "-C" "otherlibs/systhreads" "depend")
+                (make "-C" "otherlibs/threads" "depend")
+                (make "-C" "otherlibs/unix" "depend")
+                (make "-C" "otherlibs/win32unix" "depend")
+                (make "-C" "debugger" "CAMLDEP=../boot/ocamlc -depend" "depend")
+                (make "-C" "ocamltest" "ocamldep=../boot/ocamlc -depend -slash" "depend")
+                (make "parsing/parser.mli")
+                (substitute* "ocamldoc/Makefile"
+                  (("include Makefile.unprefix")
+                   "include Makefile.unprefix
+depend: $(STDLIB_MLIS) $(STDLIB_DEPS)"))
+                (make "-C" "ocamldoc" "OCAMLDEP=../boot/ocamlc -depend -slash" "depend")
+                (make "-C" "ocamldoc/stdlib_non_prefixed"
+                      "OCAMLDEP=../../boot/ocamlc -depend -slash" "depend")
+                (make "CAMLDEP=boot/ocamlc -depend" "depend")
+                ))))))
+    (native-inputs
+     `(("ocaml" ,ocaml-4.07-boot)
+       ("perl" ,perl)
+       ("pkg-config" ,pkg-config)))))
 
 (define-public ocaml ocaml-4.11)
 
-- 
2.30.0