Subject: Re: bug#36685: ant-bootstrap fails on core-updates (409 dependents)

From: Ricardo Wurmus <[email-masked]>
Date: Wed, 17 Jul 2019 10:09:39 +0200

Ricardo Wurmus <[email-masked]> writes:

> However, this doesn’t seem to help.  Yes, the bootstrap script no longer
> aborts but it gets stuck compiling things.  I can’t get it to tell me
> anything about the compilation progress, but strace shows me that it
> keeps stat’ing for non-existent files like "/tmp/files16bfb86414e_b"
> until the end of the day.
>
> Judging by the name of the file I think this is ant’s
> src/main/org/apache/tools/ant/taskdefs/compilers/DefaultCompilerAdapter.java
> that creates them, which is used by Jikes.java.  We should see if
> there’s a race somewhere or if perhaps “jikes” crashes and thus gets
> executed over and over again.

The problem appears to be in GNU Classpath (classpath-bootstrap).  Its
“native/jni/java-io/java_io_VMFile.c” defines
“Java_java_io_VMFile_exists”, which returns “1” when a file exists.  It
does so by calling “cpio_isFileExists” which is defined in
“native/jni/native-lib/cpio.c”:

--8<---------------cut here---------------start------------->8---
int cpio_isFileExists (const char *filename)
{
  struct stat statbuf;

  if (stat(filename, &statbuf) < 0)
    {
      return errno;
    }

  return 0;
}
--8<---------------cut here---------------end--------------->8---

I have confirmed with printf debugging that there’s a mismatch between
what the Java side thinks and what the C side tells it.  On the C side
the temporary file is determined to not exist yet, but on the Java side
it is said to exist — this is in spite of the fact that the Java side
only converts the C side’s return value.

Here’s an example from the printf output:

    exists? /tmp/files16bfd145b82_1 : 2 -- 0;
    classpath trying to create: /tmp/files16bfd145b82_1 exists? true

The first line is from C, the 2 is ENOENT, and 0 is the return value.
The zero is taken to be a Java boolean, so it maps to false.  The second
line comes from Java.

It hangs in “while (VMFile.exists(file.path))”:

--8<---------------cut here---------------start------------->8---
    File file;
    if (!VMFile.IS_DOS_8_3)
      {
        do
          {
            long now = System.currentTimeMillis();
            if (now > last_tmp)
              {
                // The last temporary file was created more than 1 ms ago.
                last_tmp = now;
                n_created = 0;
              }
            else
              n_created++;

            String name = Long.toHexString(now);
            if (n_created > 0)
              name += '_'+Integer.toHexString(n_created);
            String filename = prefix + name + suffix;
            file = new File(directory, filename);
          }
        while (VMFile.exists(file.path));
      }
    else …
--8<---------------cut here---------------end--------------->8---

I have confirmed that this is the problem by replacing “while
(VMFile.exists(file.path))” with “while (false)”.  The build doesn’t
fully complete then either, but it gets past the compilation of the Ant
source files.  This clears JamVM and Jikes.

I’m attaching my embarrassing printf debugging patches.

Any ideas?

--
Ricardo

diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 403c446a82..0245b299f8 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -116,6 +116,8 @@
                (base32
                 "1qqldrp74pzpy5ly421srqn30qppmm9cvjiqdngk8hf47dv2rc0c"))))
     (build-system gnu-build-system)
+    (native-inputs
+     `(("gcc" ,gcc-5)))
     (home-page "http://jikes.sourceforge.net/")
     (synopsis "Compiler for the Java language")
     (description "Jikes is a compiler that translates Java source files as
@@ -152,13 +154,43 @@ and binary format defined in The Java Virtual Machine Specification.")
              "--disable-gjdoc")
        #:phases
        (modify-phases %standard-phases
+         (add-after 'unpack 'foo
+           (lambda _
+             (substitute* "native/jni/native-lib/cpio.c"
+               (("\\(stat\\(filename, &statbuf\\) < 0\\)")
+                "(stat(filename, &statbuf) != 0)"))
+             (substitute* "native/jni/java-io/java_io_VMFile.c"
+               (("return result == CPNATIVE_OK \\? 1 : 0;")
+                "return ((result == CPNATIVE_OK) ? 1 : 0);")
+               (("result = cpio_isFileExists.*" m)
+                (string-append m "\nfprintf(stderr, \"exists? %s : %d -- %d;\\n\", filename, result, ((result == CPNATIVE_OK) ? 1 : 0));")))
+             (substitute* "java/io/File.java"
+               (("String filename = ") "filename =")
+               (("File file;" m)
+                (string-append m "\nString filename;"))
+
+               ;; (("while \\(VMFile.exists\\(file.path\\)\\)")
+               ;;  "while (false)")
+               (("while \\(VMFile.exists\\(file.path\\)\\)")
+                "while (VMFile.exists(directory.path + \"/\" + filename))")
+               (("// Grab the system" m)
+                (string-append "System.err.println(\"classpath called createTempFile: prefix \" + prefix + \" suffix \" + suffix + \" directory \" + directory.path );\n"
+                               m))
+               (("// Verify that we" m)
+                (string-append "System.err.println(\"classpath after loop with: \" + prefix);\n
+file = new File(directory, filename);"
+                               m))
+               (("file = new File.*" m)
+                (string-append m "\nSystem.err.println(\"classpath trying to create: \" + file.path + \" exists? \" + String.valueOf(VMFile.exists(file.path)));\n")))
+             #t))
          (add-after 'install 'install-data
            (lambda _ (invoke "make" "install-data"))))))
     (native-inputs
      `(("jikes" ,jikes)
        ("fastjar" ,fastjar)
        ("libltdl" ,libltdl)
-       ("pkg-config" ,pkg-config)))
+       ("pkg-config" ,pkg-config)
+       ("gcc" ,gcc-5)))
     (home-page "https://www.gnu.org/software/classpath/")
     (synopsis "Essential libraries for Java")
     (description "GNU Classpath is a project to create core class libraries
@@ -191,6 +223,8 @@ language.")
      `(("classpath" ,classpath-bootstrap)
        ("jikes" ,jikes)
        ("zlib" ,zlib)))
+    (native-inputs
+     `(("gcc-5" ,gcc-5)))
     (home-page "http://jamvm.sourceforge.net/")
     (synopsis "Small Java Virtual Machine")
     (description "JamVM is a Java Virtual Machine conforming to the JVM
@@ -225,7 +259,6 @@ JNI.")
        #:tests? #f ; no "check" target
        #:phases
        (modify-phases %standard-phases
-         (delete 'bootstrap)
          (delete 'configure)
          (replace 'build
            (lambda* (#:key inputs #:allow-other-keys)
@@ -244,9 +277,12 @@ JNI.")
              (setenv "HOME" "/tmp")
              (with-output-to-file "/tmp/.ant.properties"
                (lambda _ (display "")))
+             (with-output-to-file ".ant.properties"
+               (lambda _ (display "")))
 
              ;; Use jikes instead of javac for <javac ...> tags in build.xml
-             (setenv "ANT_OPTS" "-Dbuild.compiler=jikes")
+             (setenv "ANT_OPTS"
+                     "-Dbuild.compiler=jikes -Djava.io.tmpdir=/tmp")
 
              ;; jikes produces lots of warnings, but they are not very
              ;; interesting, so we silence them.
@@ -254,15 +290,49 @@ JNI.")
 
              ;; Without these JamVM options the build may freeze.
              (substitute* "bootstrap.sh"
+               ;;(("-emacs") "-debug")
                (("^\"\\$\\{JAVACMD\\}\" " m)
                 ,@(if (string-prefix? "armhf" (or (%current-system)
                                                   (%current-target-system)))
                       `((string-append m "-Xnocompact "))
                       `((string-append m "-Xnocompact -Xnoinlining ")))))
 
+             (substitute* "src/main/org/apache/tools/ant/version.txt"
+               (("VERSION=.*") (string-append "VERSION=" ,version "\n"))
+               (("DATE=.*") "DATE=reproducible"))
+
+             ;; XXX: Copying the source files appears to be necessary because
+             ;; ant won't find the XML and txt resources in src/main for some
+             ;; reason.  It really shouldn't be needed, so maybe this can be
+             ;; avoided by setting some environment variable.
+             (substitute* "bootstrap.sh"
+               (("cp src/script/antRun bin" m)
+                (string-append m "
+cp -ar {src/main,build/classes}/org/apache/tools/ant/version.txt
+cp -ar {src/main,build/classes}/org/apache/tools/ant/antlib.xml
+")))
+             (substitute* "src/main/org/apache/tools/ant/taskdefs/compilers/DefaultCompilerAdapter.java"
+               (("tmpFile = FILE_UTILS.createTempFile" m)
+                (string-append "attributes.log(\"REKADO createTempFile\", Project.MSG_WARN);"
+                               m))
+               (("out = new.*" m)
+                (string-append m
+                               "\nattributes.log(\"REKADO created\", Project.MSG_WARN);"))
+               (("exe.execute\\(\\)" m)
+                (string-append "attributes.log(\"REKADO executing\", Project.MSG_WARN);"
+                               m)))
+
+             (substitute* "src/main/org/apache/tools/ant/util/FileUtils.java"
+               (("result = File.*" m)
+                (string-append "System.err.println(\"Trying to create \" + prefix);\n"
+                               m
+                               "\nSystem.err.println(\"Created file \" + result);")))
+             
              ;; Disable tests because we are bootstrapping and thus don't have
              ;; any of the dependencies required to build and run the tests.
              (substitute* "build.xml"
+               ;; (("includeantruntime=\"false\"")
+               ;;  (string-append "includeantruntime=\"true\" createMissingPackageInfoClass=\"false\" fork=\"true\""))
                (("depends=\"jars,test-jar\"") "depends=\"jars\""))
              (invoke "bash" "bootstrap.sh"
                      (string-append "-Ddist.dir="