GNU bug report logs - #58591
Java packages do not appear to keep a reference to their inputs

Previous Next

To reply to this bug, email your comments to 58591 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: bug-guix <bug-guix <at> gnu.org>
Subject: Java packages do not appear to keep a reference to their inputs
Date: Mon, 17 Oct 2022 17:04:47 -0400

Hello,

I'm not a Java expert, but this appears to me problematic:

--8<---------------cut here---------------start------------->8---
$ guix build java-commons-dbcp
/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0

$ guix gc -R /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
--8<---------------cut here---------------end--------------->8---

Digging a bit more, peeking into the .jar file, which is a ZIP archive:

--8<---------------cut here---------------start------------->8---
$ unzip /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/\
share/java/java-commons-dbcp.jar -d /tmp/java-commons-dbcp.jar

$ grep -rin CLASSPATH /tmp/java-commons-dbcp.jar
$ grep -rin /gnu/store /tmp/java-commons-dbcp.jar
/tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST:3:/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar

$ cat /tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST
JarIndex-Version: 1.0

/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar
org
org/apache
org/apache/commons
org/apache/commons/dbcp2
org/apache/commons/dbcp2/cpdsadapter
org/apache/commons/dbcp2/datasources
org/apache/commons/dbcp2/managed
--8<---------------cut here---------------end--------------->8---

Still, no traces of the other libraries such as 'java-commons-pool'
which should be referenced.

I assume this means grafts doesn't currently work for Java libraries.

-- 
Thanks,
Maxim

Message #8 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to their inputs
Date: Tue, 18 Oct 2022 00:03:27 +0200

[Message part 1 (text/plain, inline)]

You're right, java package don't retain references to there input, that's why we propagate required dependencies (mh… sometimes). I don't know how they could reference dependencies directly.

Le 17 octobre 2022 23:04:47 GMT+02:00, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> a écrit :
>Hello,
>
>I'm not a Java expert, but this appears to me problematic:
>
>--8<---------------cut here---------------start------------->8---
>$ guix build java-commons-dbcp
>/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
>
>$ guix gc -R /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
>/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
>--8<---------------cut here---------------end--------------->8---
>
>Digging a bit more, peeking into the .jar file, which is a ZIP archive:
>
>--8<---------------cut here---------------start------------->8---
>$ unzip /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/\
>share/java/java-commons-dbcp.jar -d /tmp/java-commons-dbcp.jar
>
>$ grep -rin CLASSPATH /tmp/java-commons-dbcp.jar
>$ grep -rin /gnu/store /tmp/java-commons-dbcp.jar
>/tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST:3:/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar
>
>$ cat /tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST
>JarIndex-Version: 1.0
>
>/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar
>org
>org/apache
>org/apache/commons
>org/apache/commons/dbcp2
>org/apache/commons/dbcp2/cpdsadapter
>org/apache/commons/dbcp2/datasources
>org/apache/commons/dbcp2/managed
>--8<---------------cut here---------------end--------------->8---
>
>Still, no traces of the other libraries such as 'java-commons-pool'
>which should be referenced.
>
>I assume this means grafts doesn't currently work for Java libraries.
>
>-- 
>Thanks,
>Maxim
>
>
>

[Message part 2 (text/html, inline)]

Message #11 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Julien Lepiller <julien <at> lepiller.eu>, Maxim Cournoyer
 <maxim.cournoyer <at> gmail.com>, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Mon, 17 Oct 2022 21:43:28 -0400

Julien Lepiller <julien <at> lepiller.eu> writes:

> You're right, java package don't retain references to there input,
> that's why we propagate required dependencies (mh… sometimes). I don't
> know how they could reference dependencies directly.

A better workaround would be to add a phase that installs file(s) in the
output(s) that contain references to the required store items.  They
could simply be text files with one line per reference.  That would at
least protect the dependencies from the garbage collector.

The remaining unsolved problem is, of course, grafting.

       Mark

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>.

Message #14 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Mon, 17 Oct 2022 22:45:54 -0400

Hi Julien,

Julien Lepiller <julien <at> lepiller.eu> writes:

> You're right, java package don't retain references to there input,
> that's why we propagate required dependencies (mh… sometimes). I don't
> know how they could reference dependencies directly.

Could we, along with installing Java classes as directories instead of
.jar archive files [0] at a more specific prefix, define a search path
specification that'd set CLASSPATH?  Currently I don't see anything
setting CLASSPATH outside of the build systems, so even if we propagate
Java things, I don't see how it'd find them in a profile.

[0]  Not exactly sure how that's done yet, but it's mentioned here: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/classpath.html

--
Thanks,
Maxim

Message #17 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>,
 Julien Lepiller <julien <at> lepiller.eu>, Mark H Weaver <mhw <at> netris.org>
Cc: 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 09:01:15 +0200

[Message part 1 (text/plain, inline)]

On 18-10-2022 04:45, Maxim Cournoyer wrote:
> [...] setting CLASSPATH outside of the build systems, so even if we propagate
> Java things, I don't see how it'd find them in a profile.

FWIW, when I used java things in Guix, I manually did 
CLASSPATH=$GUIX_ENVIRONMENT/... or the CLI equivalent (some option 
argument of 'java').

Some more automatisation, e.g. in the form of search paths as you 
propose, would be nice though.

Greetings,
Maxime.

[OpenPGP_0x49E3EE22191725EE.asc (application/pgp-keys, attachment)]

[OpenPGP_signature (application/pgp-signature, attachment)]

Message #20 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> ist.tugraz.at>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>, Julien Lepiller
 <julien <at> lepiller.eu>
Cc: 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 09:36:44 +0200

Am Montag, dem 17.10.2022 um 22:45 -0400 schrieb Maxim Cournoyer:
> Hi Julien,
> 
> Julien Lepiller <julien <at> lepiller.eu> writes:
> 
> > You're right, java package don't retain references to there input,
> > that's why we propagate required dependencies (mh… sometimes). I
> > don't
> > know how they could reference dependencies directly.
> 
> Could we, along with installing Java classes as directories instead
> of .jar archive files [0] at a more specific prefix, define a search
> path specification that'd set CLASSPATH?  Currently I don't see
> anything setting CLASSPATH outside of the build systems, so even if
> we propagate Java things, I don't see how it'd find them in a
> profile.
I'd recommend writing an xml file like 

  <path id="${java-package-name}.classpath">
    <pathelement location="${output-jar}" />
    <pathelement path="${input1.classpath}" />
    ... 
    <pathelement path="${inputn.classpath}" /> 
  </path>

to a well-known location.  Then we could reuse those files in ant-
build-system.

Cheers

Message #23 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Liliana Marie Prikler <liliana.prikler <at> ist.tugraz.at>
Cc: Julien Lepiller <julien <at> lepiller.eu>, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 09:14:03 -0400

Hello,

Liliana Marie Prikler <liliana.prikler <at> ist.tugraz.at> writes:

> Am Montag, dem 17.10.2022 um 22:45 -0400 schrieb Maxim Cournoyer:
>> Hi Julien,
>> 
>> Julien Lepiller <julien <at> lepiller.eu> writes:
>> 
>> > You're right, java package don't retain references to there input,
>> > that's why we propagate required dependencies (mh… sometimes). I
>> > don't
>> > know how they could reference dependencies directly.
>> 
>> Could we, along with installing Java classes as directories instead
>> of .jar archive files [0] at a more specific prefix, define a search
>> path specification that'd set CLASSPATH?  Currently I don't see
>> anything setting CLASSPATH outside of the build systems, so even if
>> we propagate Java things, I don't see how it'd find them in a
>> profile.
> I'd recommend writing an xml file like 
>
>   <path id="${java-package-name}.classpath">
>     <pathelement location="${output-jar}" />
>     <pathelement path="${input1.classpath}" />
>     ... 
>     <pathelement path="${inputn.classpath}" /> 
>   </path>
>
> to a well-known location.  Then we could reuse those files in ant-
> build-system.

A nice read is [0], which mentions the existence of a 'Class-Path' main
attribute that can go in the manifest file.  If using unpacked jars
works the same as .jars (which are just zip files) for Java, then we
could not only have dependency correctly referenced and loaded via
'Class-Path', but also the grafting mechanism would work, since the
paths would appear in clear (not obfuscated due to zip compression).

Our current usage of JarIndex doesn't suite the bill it was intended
for; this is a performance trick to index all the .jars of a .jar pack;
it'll only list its dependencies if they are packed in the same jar,
which is not what we do or want as a distribution.

[0]  https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html

-- 
Thanks,
Maxim

Message #26 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Liliana Marie Prikler <liliana.prikler <at> ist.tugraz.at>
Cc: Julien Lepiller <julien <at> lepiller.eu>, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 09:29:48 -0400

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

[...]

> A nice read is [0], which mentions the existence of a 'Class-Path' main
> attribute that can go in the manifest file.  If using unpacked jars
> works the same as .jars (which are just zip files) for Java, then we
> could not only have dependency correctly referenced and loaded via
> 'Class-Path', but also the grafting mechanism would work, since the
> paths would appear in clear (not obfuscated due to zip compression).

Ugh, Class-Path only accepts relative path, not absolute paths:

   The location of the JAR file or directory represented by this entry
   is contained within the containing directory of the context JAR. Use
   of "../" to navigate to the parent directory is not permitted, except
   for the case when the context JAR is loaded from the file system.

Perhaps we could patch Java so that it's loader is more adapted for our
use case, or extend its manifest with a Guix-specific Guix-Class-Path
section that'd allow for absolute paths.

> [0] https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html

-- 
Thanks,
Maxim

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: Liliana Marie Prikler <liliana.prikler <at> ist.tugraz.at>,
 Julien Lepiller <julien <at> lepiller.eu>, bug-guix <at> gnu.org, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 15:21:20 +0200

[Message part 1 (text/plain, inline)]

Hi Maxim,

Maxim Cournoyer 写道：
> not obfuscated due to zip compression

Groan.  Which package(s) compress .jars?

(I found a few in -checkouts, which is its own potential thing, 
but that aside.)

Kind regards,

T G-R

[signature.asc (application/pgp-signature, inline)]

Message #35 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: liliana.prikler <at> ist.tugraz.at, julien <at> lepiller.eu, bug-guix <at> gnu.org,
 58591 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 16:17:09 +0200

[Message part 1 (text/plain, inline)]

Tobias Geerinckx-Rice via Bug reports for GNU Guix 写道：
> Groan.  Which package(s) compress .jars?

OK, found one: openjdk <at> 16.0.1's /lib/jrt-fs.jar.

Kind regards,

T G-R

[signature.asc (application/pgp-signature, inline)]

Message #41 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: Liliana Marie Prikler <liliana.prikler <at> ist.tugraz.at>,
 Julien Lepiller <julien <at> lepiller.eu>, bug-guix <at> gnu.org, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 10:53:21 -0400

Hi Tobias!

Tobias Geerinckx-Rice <me <at> tobias.gr> writes:

> Hi Maxim,
>
> Maxim Cournoyer 写道：
>> not obfuscated due to zip compression
>
> Groan.  Which package(s) compress .jars?

Oh, aren't they all?  I hadn't realized .jar compression was optional.
I believe our ant-build-system produces compressed jars; in fact it uses
'zip' directly to pack them.

-- 
Thanks,
Maxim

Message #47 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: liliana.prikler <at> ist.tugraz.at, julien <at> lepiller.eu, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 10:56:01 -0400

Hello,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Tobias Geerinckx-Rice <me <at> tobias.gr> writes:

[...]

>> Groan.  Which package(s) compress .jars?
>
> Oh, aren't they all?  I hadn't realized .jar compression was optional.

Actually, reading [0] again, it seems a JAR *is* a zip archive, so
cannot be either compressed or uncompressed.

[0]  https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html

-- 
Thanks,
Maxim

Message #50 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>,
 Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: liliana.prikler <at> ist.tugraz.at, 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to their inputs
Date: Tue, 18 Oct 2022 17:32:32 +0200

[Message part 1 (text/plain, inline)]

Hi, replying to a few emails at once.

The ant-build-system uses zip -0 to produce an uncompressed archive. By default, jar produces a compressed one, so there's a repack phase for that:
 http://git.savannah.nongnu.org/cgit/guix.git/tree/guix/build/ant-build-system.scm#n226

Embedding the classpath in the manifest is possible but would not have the expected effect. That's because a line in the manifest cannot exceed 72 bytes (see "line length" in https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Notes_on_Manifest_and_Signature_Files), so the classpath will look like:

Class-Path: ../../../1234567891011
 1213141516/share/java/foo.jar

Although java would read that fine, the grafter will not see it, nor be able to graft foo in a meaningful manner: java would still use the ungrafted version even if another file references foo.

Le 18 octobre 2022 16:56:01 GMT+02:00, Maxim Cournoyer <maxim.cournoyer <at> gmail.com> a écrit :
>Hello,
>
>Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> Tobias Geerinckx-Rice <me <at> tobias.gr> writes:
>
>[...]
>
>>> Groan.  Which package(s) compress .jars?
>>
>> Oh, aren't they all?  I hadn't realized .jar compression was optional.
>
>Actually, reading [0] again, it seems a JAR *is* a zip archive, so
>cannot be either compressed or uncompressed.
>
>[0]  https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html
>
>-- 
>Thanks,
>Maxim

[Message part 2 (text/html, inline)]

Message #53 received at 58591 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: liliana.prikler <at> ist.tugraz.at, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 58591 <at> debbugs.gnu.org
Subject: Re: bug#58591: Java packages do not appear to keep a reference to
 their inputs
Date: Tue, 18 Oct 2022 19:25:57 -0400

Hello,

Julien Lepiller <julien <at> lepiller.eu> writes:

> Hi, replying to a few emails at once.
>
> The ant-build-system uses zip -0 to produce an uncompressed
> archive. By default, jar produces a compressed one, so there's a
> repack phase for that:
>  http://git.savannah.nongnu.org/cgit/guix.git/tree/guix/build/ant-build-system.scm#n226

Ah, I had missed the -0 == uncompressed part.  Thank you.

> Embedding the classpath in the manifest is possible but would not have
> the expected effect. That's because a line in the manifest cannot
> exceed 72 bytes (see "line length" in
> https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Notes_on_Manifest_and_Signature_Files),
> so the classpath will look like:
>
> Class-Path: ../../../1234567891011
>  1213141516/share/java/foo.jar

Although it looks like the 72 bytes line width limitation may has to do
with binary data:

   Binary data of any form is represented as base64. Continuations are
   required for binary data which causes line length to exceed 72
   bytes. Examples of binary data are digests and signatures.

Worth a try in my opinion (I'm giving it a shot as I write this).

Thanks for the explanations!

Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.