Subject: bug#43525: closed (Re: [bug#43525] [PATCH] gnu: Add antlr4.)

From: [email-masked] (GNU bug Tracking System)
Date: Sun, 20 Sep 2020 12:01:02 +0000

Your bug report

#43525: [PATCH] gnu: Add antlr4.

which was filed against the guix-patches package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
43525: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=43525
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: [bug#43525] [PATCH] gnu: Add antlr4.
From: Julien Lepiller <[email-masked]>
To: Ricardo Wurmus <[email-masked]>
Cc: [email-masked]
Date: Sun, 20 Sep 2020 14:00:09 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Pushed as 1d630fe0f64d16d7428a78ee4bf94e8179ce4a94 to
f9a6e3894b1648e32371e1916dfdf820de496050, thank you!

Subject: [PATCH] gnu: Add antlr4.
From: Julien Lepiller <[email-masked]>
To: [email-masked]
Date: Sun, 20 Sep 2020 03:18:17 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi Guix!

I tried to update groovy to its latest version today, so here's a patch
series that adds ANTLR 4 instead :p (it's a build dependency for the
newest version of groovy).
>From e9fa7e24a17870f3d3fc29742a85b7c699fbffdd Mon Sep 17 00:00:00 2001
From: Julien Lepiller <[email-masked]>
Date: Sun, 20 Sep 2020 03:01:09 +0200
Subject: [PATCH 1/3] gnu: Add java-treelayout.

* gnu/packages/java.scm (java-treelayout): New variable.
---
 gnu/packages/java.scm | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 1ef2ab3584..e05cc6b28c 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -8146,6 +8146,37 @@ import org.antlr.grammar.v2.ANTLRTreePrinter;"))
     (propagated-inputs
      `(("stringtemplate" ,java-stringtemplate-3)))))
 
+(define-public java-treelayout
+  (package
+    (name "java-treelayout")
+    (version "1.0.3")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                     (url "https://github.com/abego/treelayout")
+                     (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "18my8ql9b1y0n0zrvkih7xfhf3dpgfhyfifvkcfhmwcvw3divxak"))))
+    (build-system ant-build-system)
+    (arguments
+     `(#:jar-name (string-append ,name "-" ,version ".jar")
+       #:source-dir "org.abego.treelayout/src/main/java"
+       #:test-dir "org.abego.treelayout/src/test"))
+    (inputs
+     `(("java-junit" ,java-junit)))
+    (native-inputs
+     `(("java-hamcrest-core" ,java-hamcrest-core)))
+    (home-page "http://treelayout.sourceforge.net")
+    (synopsis "Tree Layout Algorithm in Java")
+    (description "TreeLayout creates tree layouts for arbitrary trees.  It is
+not restricted to a specific output or format, but can be used for any kind of
+two dimensional diagram.  Examples are Swing based components, SVG files, etc.
+This is possible because TreeLayout separates the layout of a tree from the
+actual rendering.")
+    (license license:bsd-3)))
+
 (define-public java-commons-cli-1.2
   ;; This is a bootstrap dependency for Maven2.
   (package
-- 
2.28.0

>From 5d2fb32320f99ce733adaac0ee7381ae953f38bb Mon Sep 17 00:00:00 2001
From: Julien Lepiller <[email-masked]>
Date: Sun, 20 Sep 2020 03:06:48 +0200
Subject: [PATCH 2/3] gnu: Add java-antlr4-runtime.

* gnu/packages/java.scm (java-antlr4-runtime): New variable.
* gnu/packages/patches/java-antlr4-Add-standalone-generator.patch: New
file.
* gnu/packages/patches/java-antlr4-fix-code-too-large.java: New file.
* gnu/local.mk (dist_patch_DATA): Add them.
---
 gnu/local.mk                                  |  2 +
 gnu/packages/java.scm                         | 34 +++++++
 ...java-antlr4-Add-standalone-generator.patch | 52 +++++++++++
 .../java-antlr4-fix-code-too-large.java       | 90 +++++++++++++++++++
 4 files changed, 178 insertions(+)
 create mode 100644 gnu/packages/patches/java-antlr4-Add-standalone-generator.patch
 create mode 100644 gnu/packages/patches/java-antlr4-fix-code-too-large.java

diff --git a/gnu/local.mk b/gnu/local.mk
index 1a1a6488f9..dc9f07ba97 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1158,6 +1158,8 @@ dist_patch_DATA =						\
   %D%/packages/patches/jacal-fix-texinfo.patch			\
   %D%/packages/patches/jamvm-2.0.0-disable-branch-patching.patch	\
   %D%/packages/patches/jamvm-arm.patch				\
+  %D%/packages/patches/java-antlr4-Add-standalone-generator.patch	\
+  %D%/packages/patches/java-antlr4-fix-code-too-large.java	\
   %D%/packages/patches/java-apache-ivy-port-to-latest-bouncycastle.patch	\
   %D%/packages/patches/java-commons-collections-fix-java8.patch \
   %D%/packages/patches/java-jeromq-fix-tests.patch		\
diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index e05cc6b28c..b3a0bfcb65 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -8177,6 +8177,40 @@ This is possible because TreeLayout separates the layout of a tree from the
 actual rendering.")
     (license license:bsd-3)))
 
+(define-public java-antlr4-runtime
+  (package
+    (name "java-antlr4-runtime")
+    (version "4.8")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                     (url "https://github.com/antlr/antlr4")
+                     (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1qal3add26qxskm85nk7r758arladn5rcyjinmhlhznmpbbv9j8m"))
+              (patches
+                (search-patches "java-antlr4-Add-standalone-generator.patch"
+                                "java-antlr4-fix-code-too-large.java"))))
+    (build-system ant-build-system)
+    (arguments
+     `(#:jar-name "java-antlr4-runtime.jar"
+       #:source-dir "runtime/Java/src/org"
+       #:tests? #f; tests depend on java-antlr4 itself
+       #:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'copy-resources
+           (lambda _
+             (copy-recursively "runtime/Java/src/main/dot"
+                               "build/classes")
+             #t)))))
+    (home-page "https://antlr.org")
+    (synopsis "ANTLR runtime library")
+    (description "This package contains the runtime library used with generated
+sources by ANTLR.")
+    (license license:bsd-3)))
+
 (define-public java-commons-cli-1.2
   ;; This is a bootstrap dependency for Maven2.
   (package
diff --git a/gnu/packages/patches/java-antlr4-Add-standalone-generator.patch b/gnu/packages/patches/java-antlr4-Add-standalone-generator.patch
new file mode 100644
index 0000000000..75b6dce39f
--- /dev/null
+++ b/gnu/packages/patches/java-antlr4-Add-standalone-generator.patch
@@ -0,0 +1,52 @@
+From 3b43b79da15be994348f13035474925ba592fe1f Mon Sep 17 00:00:00 2001
+From: Julien Lepiller <[email-masked]>
+Date: Fri, 15 Sep 2017 10:06:42 +0200
+Subject: [PATCH] Add standalone template generator
+
+---
+ tool/src/org/antlr/v4/unicode/UnicodeRenderer.java | 33 ++++++++++++++++++++++
+ 1 file changed, 33 insertions(+)
+ create mode 100644 tool/src/org/antlr/v4/unicode/UnicodeRenderer.java
+
+diff --git a/tool/src/org/antlr/v4/unicode/UnicodeRenderer.java b/tool/src/org/antlr/v4/unicode/UnicodeRenderer.java
+new file mode 100644
+index 0000000..9e53213
+--- /dev/null
++++ b/tool/src/org/antlr/v4/unicode/UnicodeRenderer.java
+@@ -0,0 +1,33 @@
++package org.antlr.v4.unicode;
++
++import org.stringtemplate.v4.*;
++import org.stringtemplate.v4.misc.ErrorBuffer;
++import org.antlr.v4.unicode.UnicodeDataTemplateController;
++
++import java.io.File;
++import java.io.FileWriter;
++import java.io.IOException;
++import java.util.Map;
++import java.util.Map.Entry;
++
++public class UnicodeRenderer extends UnicodeDataTemplateController {
++    public static void main(String[] arg)
++        throws IOException {
++        String inputdir =  arg[0];
++        String input =     arg[1];
++        String output =    arg[2];
++
++        FileWriter fileWriter = new FileWriter(new File(output));
++        ErrorBuffer listener = new ErrorBuffer();
++
++        STGroupDir group = new STGroupDir(inputdir);
++        ST st = group.getInstanceOf(input);
++
++        for(Entry<String, Object> entry : getProperties().entrySet())
++        	st.add(entry.getKey(), entry.getValue());
++
++        st.write(new AutoIndentWriter(fileWriter), listener);
++        fileWriter.flush();
++        fileWriter.close();
++    }
++}
+-- 
+2.13.5
+
diff --git a/gnu/packages/patches/java-antlr4-fix-code-too-large.java b/gnu/packages/patches/java-antlr4-fix-code-too-large.java
new file mode 100644
index 0000000000..9e9f92349d
--- /dev/null
+++ b/gnu/packages/patches/java-antlr4-fix-code-too-large.java
@@ -0,0 +1,90 @@
+From d22db2048534bdf3d9615117291f9d86564ff10d Mon Sep 17 00:00:00 2001
+From: Julien Lepiller <[email-masked]>
+Date: Sat, 19 Sep 2020 21:05:48 +0200
+Subject: [PATCH] Separate addPropertyAliases in two methods
+
+The quantity of data used to generate addPropertyAliases creates a big
+method, that is too big for java and results in "error: code too large".
+This is most likely due to added data between the expected version of
+icu and the actual version of icu in Guix.
+---
+ .../org/antlr/v4/tool/templates/unicodedata.st | 17 ++++++++++++++---
+ .../unicode/UnicodeDataTemplateController.java | 18 ++++++++++--------
+ 2 files changed, 24 insertions(+), 11 deletions(-)
+
+diff --git a/tool/resources/org/antlr/v4/tool/templates/unicodedata.st b/tool/resources/org/antlr/v4/tool/templates/unicodedata.st
+index 0f22c73..3573873 100644
+--- a/tool/resources/org/antlr/v4/tool/templates/unicodedata.st
++++ b/tool/resources/org/antlr/v4/tool/templates/unicodedata.st
+@@ -1,4 +1,4 @@
+-unicodedata(propertyCodePointRanges, propertyAliases) ::= <<
++unicodedata(propertyCodePointRanges, propertyAliasesA, propertyAliasesB) ::= <<
+ package org.antlr.v4.unicode;
+ 
+ import java.util.Arrays;
+@@ -15,7 +15,7 @@ import org.antlr.v4.runtime.misc.Interval;
+  */
+ public abstract class UnicodeData {
+        private static final Map\<String, IntervalSet\> propertyCodePointRanges = new HashMap\<\>(<length(propertyCodePointRanges)>);
+-       private static final Map\<String, String\> propertyAliases = new HashMap\<\>(<length(propertyAliases)>);
++       private static final Map\<String, String\> propertyAliases = new HashMap\<\>(<length(propertyAliasesA)> + <length(propertyAliasesB)>);
+ 
+        // Work around Java 64k bytecode method limit by splitting up static
+        // initialization into one method per Unicode property
+@@ -30,9 +30,20 @@ static private void addProperty<i>() {
+        propertyCodePointRanges.put("<k>".toLowerCase(Locale.US), codePointRanges);
+ \}}; separator="\n\n">
+ 
++       // Property aliases
++       static private void addPropertyAliases1() {
++              <propertyAliasesA.keys:{ k | propertyAliases.put("<k>".toLowerCase(Locale.US), "<propertyAliasesA.(k)>".toLowerCase(Locale.US)); }; separator="\n">
++       }
++
++       // Property aliases
++       static private void addPropertyAliases2() {
++              <propertyAliasesB.keys:{ k | propertyAliases.put("<k>".toLowerCase(Locale.US), "<propertyAliasesB.(k)>".toLowerCase(Locale.US)); }; separator="\n">
++       }
++
+        // Property aliases
+        static private void addPropertyAliases() {
+-              <propertyAliases.keys:{ k | propertyAliases.put("<k>".toLowerCase(Locale.US), "<propertyAliases.(k)>".toLowerCase(Locale.US)); }; separator="\n">
++             addPropertyAliases1();
++             addPropertyAliases2();
+        }
+ 
+        // Put it all together
+diff --git a/tool/src/org/antlr/v4/unicode/UnicodeDataTemplateController.java b/tool/src/org/antlr/v4/unicode/UnicodeDataTemplateController.java
+index da244a3..dc591cb 100644
+--- a/tool/src/org/antlr/v4/unicode/UnicodeDataTemplateController.java
++++ b/tool/src/org/antlr/v4/unicode/UnicodeDataTemplateController.java
+@@ -78,17 +78,19 @@ public abstract class UnicodeDataTemplateController {
+ 		addTR35ExtendedPictographicPropertyCodesToCodePointRanges(propertyCodePointRanges);
+ 		addEmojiPresentationPropertyCodesToCodePointRanges(propertyCodePointRanges);
+ 
+-		Map<String, String> propertyAliases = new LinkedHashMap<>();
+-		addUnicodeCategoryCodesToNames(propertyAliases);
+-		addUnicodeBinaryPropertyCodesToNames(propertyAliases);
+-		addUnicodeScriptCodesToNames(propertyAliases);
+-		addUnicodeBlocksToNames(propertyAliases);
+-		addUnicodeIntPropertyCodesToNames(propertyAliases);
+-		propertyAliases.put("EP", "Extended_Pictographic");
++		Map<String, String> propertyAliases1 = new LinkedHashMap<>();
++		Map<String, String> propertyAliases2 = new LinkedHashMap<>();
++		addUnicodeCategoryCodesToNames(propertyAliases1);
++		addUnicodeBinaryPropertyCodesToNames(propertyAliases1);
++		addUnicodeScriptCodesToNames(propertyAliases1);
++		addUnicodeBlocksToNames(propertyAliases2);
++		addUnicodeIntPropertyCodesToNames(propertyAliases2);
++		propertyAliases2.put("EP", "Extended_Pictographic");
+ 
+ 		Map<String, Object> properties = new LinkedHashMap<>();
+ 		properties.put("propertyCodePointRanges", propertyCodePointRanges);
+-		properties.put("propertyAliases", propertyAliases);
++		properties.put("propertyAliasesA", propertyAliases1);
++		properties.put("propertyAliasesB", propertyAliases2);
+ 		return properties;
+ 	}
+ 
+-- 
+2.28.0
+
-- 
2.28.0

>From 7141f332ac488f52de4674a4cb9e6079ebb9ba97 Mon Sep 17 00:00:00 2001
From: Julien Lepiller <[email-masked]>
Date: Sun, 20 Sep 2020 03:09:23 +0200
Subject: [PATCH 3/3] gnu: Add antlr4.

* gnu/packages/java.scm (antlr4): New variable.
---
 gnu/packages/java.scm | 158 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 158 insertions(+)

diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index b3a0bfcb65..cc0ef1a759 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -8211,6 +8211,164 @@ actual rendering.")
 sources by ANTLR.")
     (license license:bsd-3)))
 
+(define-public antlr4
+  (package
+    (inherit java-antlr4-runtime)
+    (name "antlr4")
+    (arguments
+     `(#:jar-name "antlr4.jar"
+       #:source-dir "tool/src"
+       #:test-dir "tool-testsuite/test:runtime-testsuite/test:runtime-testsuite/annotations/src"
+       #:test-include (list "**/Test*.java")
+       #:test-exclude (list
+                        ;; no runnable method
+                        "**/TestOutputReading.java"
+                        ;; no @Test methods
+                        "**/TestParserErrors.java"
+                        "**/TestSemPredEvalParser.java"
+                        "**/TestSets.java"
+                        "**/TestListeners.java"
+                        "**/TestParseTrees.java"
+                        "**/TestParserExec.java"
+                        "**/TestLexerErrors.java"
+                        "**/TestPerformance.java"
+                        "**/TestCompositeParsers.java"
+                        "**/TestLexerExec.java"
+                        "**/TestSemPredEvalLexer.java"
+                        "**/TestLeftRecursion.java"
+                        "**/TestFullContextParsing.java"
+                        "**/TestCompositeLexers.java"
+                        ;; Null pointer exception
+                        "**/TestCompositeGrammars.java"
+                        ;; Wrong assumption on emoji
+                        "**/TestUnicodeData.java")
+       #:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'fix-build.xml
+           (lambda _
+             ;; tests are not in a java subdirectory
+             (substitute* "build.xml"
+               (("\\$\\{test.home\\}/java") "${test.home}"))
+             #t))
+         ;; tests require to have a working antlr4 binary
+         (delete 'check)
+         (add-after 'bin-install 'check
+           (lambda _
+             (invoke "ant" "compile-tests")
+             (invoke "ant" "check" "-Dtest.home=runtime-testsuite/annotations/src")
+             (invoke "ant" "check" "-Dtest.home=runtime-testsuite/test")
+             (invoke "ant" "check" "-Dtest.home=tool-testsuite/test")))
+         (add-before 'check 'remove-unrelated-languages
+           (lambda _
+             ;; There are tests for other languages that ANTLR can generate, but
+             ;; we don't have the infrastructure for that yet.  Let's test Java
+             ;; generation only.
+             (for-each
+               (lambda (language)
+                 (delete-file-recursively
+                   (string-append "runtime-testsuite/test/org/antlr/v4/test/runtime/"
+                                  language)))
+               '("cpp" "csharp" "go" "javascript" "php" "python" "python2"
+                 "python3" "swift"))
+             #t))
+         (add-before 'check 'generate-test-parsers
+           (lambda* (#:key outputs #:allow-other-keys)
+             (define (run-antlr dir filename package)
+               (invoke "antlr4" "-lib" dir "-visitor" "-no-listener"
+                       "-package" package (string-append dir "/" filename)
+                       "-Xlog"))
+             (setenv "PATH" (string-append (getenv "PATH") ":"
+                                           (assoc-ref outputs "out") "/bin"))
+             (run-antlr "runtime-testsuite/test/org/antlr/v4/test/runtime/java/api"
+                        "Java.g4" "org.antlr.v4.test.runtime.java.api")
+             (run-antlr "runtime-testsuite/test/org/antlr/v4/test/runtime/java/api"
+                        "VisitorBasic.g4" "org.antlr.v4.test.runtime.java.api")
+             (run-antlr "runtime-testsuite/test/org/antlr/v4/test/runtime/java/api"
+                        "VisitorCalc.g4" "org.antlr.v4.test.runtime.java.api")
+             #t))
+         (add-before 'check 'remove-graphemes
+           (lambda _
+             ;; When running antlr on grahemes.g4, we get a runtime exception:
+             ;; set is empty. So delete the file that depends on it.
+             (delete-file "runtime-testsuite/test/org/antlr/v4/test/runtime/java/api/perf/TimeLexerSpeed.java")))
+         (add-after 'install 'bin-install
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((jar (string-append (assoc-ref outputs "out") "/share/java"))
+                   (bin (string-append (assoc-ref outputs "out") "/bin")))
+               (mkdir-p bin)
+               (with-output-to-file (string-append bin "/antlr4")
+                 (lambda _
+                   (display
+                     (string-append "#!" (which "sh") "\n"
+                                    "java -cp " jar "/antlr4.jar:"
+                                    (string-join
+                                      (apply
+                                        append
+                                        (map
+                                          (lambda (input)
+                                            (find-files (assoc-ref inputs input)
+                                                  ".*\\.jar"))
+                                          '("antlr3" "java-stringtemplate"
+                                            "java-antlr4-runtime" "java-treelayout"
+                                            "java-jsonp-api" "java-icu4j")))
+                                      ":")
+                                    " org.antlr.v4.Tool $*"))))
+               (chmod (string-append bin "/antlr4") #o755)
+               #t)))
+         (add-before 'build 'copy-resources
+           (lambda _
+             (copy-recursively "tool/resources/" "build/classes")
+             #t))
+         (add-before 'build 'generate-unicode
+           (lambda _
+             ;; First: build the generator
+             (invoke "javac" "-cp" (getenv "CLASSPATH")
+                     "tool/src/org/antlr/v4/unicode/UnicodeRenderer.java"
+                     "tool/src/org/antlr/v4/unicode/UnicodeDataTemplateController.java")
+             ;; Then use it
+             (invoke "java" "-cp" (string-append (getenv "CLASSPATH")
+                                                 ":tool/src:runtime/Java")
+                     "org.antlr.v4.unicode.UnicodeRenderer"
+                     "tool/resources/org/antlr/v4/tool/templates"
+                     "unicodedata"
+                     "tool/src/org/antlr/v4/unicode/UnicodeData.java")
+             (begin
+               ;; It seems there is a bug with our ST4
+               (substitute* "tool/src/org/antlr/v4/unicode/UnicodeData.java"
+                 (("\\\\>") ">"))
+               ;; Remove the additional file
+               (delete-file "tool/src/org/antlr/v4/unicode/UnicodeRenderer.java")
+               #t)))
+         (add-before 'build 'generate-grammar
+           (lambda* (#:key inputs #:allow-other-keys)
+             (with-directory-excursion "tool/src/org/antlr/v4/parse"
+               (for-each (lambda (file)
+                           (format #t "~a\n" file)
+                           (system* "antlr3" file))
+                         '("ANTLRLexer.g" "ANTLRParser.g" "BlockSetTransformer.g"
+                           "GrammarTreeVisitor.g" "ATNBuilder.g"
+                           "ActionSplitter.g" "LeftRecursiveRuleWalker.g")))
+             (with-directory-excursion "tool/src/org/antlr/v4/codegen"
+               (copy-file "../parse/ANTLRParser.tokens" "ANTLRParser.tokens")
+               (format #t "SourceGenTriggers.g\n")
+               (invoke "antlr3" "SourceGenTriggers.g"))
+             #t)))))
+    (inputs
+     `(("antlr3" ,antlr3)
+       ("java-antlr4-runtime" ,java-antlr4-runtime)
+       ("java-icu4j" ,java-icu4j)
+       ("java-jsonp-api" ,java-jsonp-api)
+       ("java-stringtemplate" ,java-stringtemplate)
+       ("java-treelayout" ,java-treelayout)))
+    (native-inputs
+     `(("java-junit" ,java-junit)))
+    (synopsis "Parser and lexer generator in Java")
+    (description "ANTLR (ANother Tool for Language Recognition) is a powerful
+parser generator for reading, processing, executing, or translating structured
+text or binary files.  It's widely used to build languages, tools, and
+frameworks.  From a grammar, ANTLR generates a parser that can build and walk
+parse trees.")))
+
 (define-public java-commons-cli-1.2
   ;; This is a bootstrap dependency for Maven2.
   (package
-- 
2.28.0