Subject: bug#9088: Java, JARS primary?

From: Stefano Lattarini <[email-masked]>
Date: Wed, 15 May 2013 15:09:26 +0200

On 05/15/2013 01:52 PM, Michael Zucchi wrote:
> On 14/05/13 03:47, Stefano Lattarini wrote:
> 
>> Instead, let's start implementing something *correct*, in line with
>> the Java philosophy, and with a clean API.  We'll think about enhancing
>> it when (and if!) the need arise.
> 
> Seems the way to go.
> 
> On that, here are a few more thoughts on java's specific way of doing
> things ...
> 
> files vs dirs
> --------------
> 
> java projects/build systems tend to have a fairly basic mechanism to
> define the source tree(s) which would be nice to emulate, but may not
> fit well with the portability requirements of automake.
> 
> They simply define the root directory and let the build system do the rest.
> 
> Not necessary but would be nice.  Or put another way it would basically
> suck if it required every source file to be listed explicitly and auto*
> having to be re-run every time one is added.
>
But then, how could make handle the case of generated java files?  I.e.,
if half of your files are generated by (say) an awk+sed incantation,
make has to know it must build them before trying to call the java compiler.
But see below.

> In a GNU makefile I can
> just use $(shell find) (or other fancier stuff), but for automake?
>
The solution is simpler than it seems at first sight: we should just
implement the new feature in a way that can handle the use of GNU make
specific functions, without requiring too much cooperation from Automake
(that means we have to delegate most of the work at make runtime).
So, people only interested in having their Makefile work with GNU make
only will be able to use all the needed tricks with "$(shell find ...)"
or "$(wildcard ...)", while people interested in wider portability we'll
have to bite the bullet and explicitly list all their java source files.
A sort of graceful degradation.

For an example of how this approach could work, take a look at the
EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh').

> source vs data
> --------------
> 
> A source tree can also include non-source data which is copied into the
> jar file verbatim at the same relative path location.  At least this is
> how ant does it and given the way these resources are resolved at
> runtime makes the most sense.
> 
> Again I think this needs to be specified by root directory not file.  It
> could just hang off the _SOURCES or use _RESOURCES.
>
Seems reasonable.

> makefile location
> -----------------
> 
> C projects often have a make file per directory, java has the build
> script at the top-level at least, outside of the source trees.
>
> This is desirable.
>
In some ways, this is desirable also for C projects actually; see the
article "Recursive Make Considered Harmful" from Peter Miller:

    <http://miller.emu.id.au/pmiller/books/rmch/>

And recently, the GNU world has started moving towards making
non-recursive make-based build system a first class citizen.
For example, important GNU packages like Coreutils and Bison has
been converted to a non-recursive build system; and Automake has
been enhanced to make its use in non-recursive setups easier and
more convenient:

   <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13524>

> For the compiled code this is simple enough since the makefile will
> compile every file every time and the compiler will dump the class files
> where it's told.
> 
> For the resource files the Makefile needs to resolve the resource root,
> and strip it out of the names used for the dependencies.  So either it
> needs a (set of) explicit resource_root, resource_root_files variables,
> or it just needs to be defined as the root dir and handled either in the
> makefile fragment or in automake.in.
> 
Doing too much pre-processing in automake.in (that is, at Automake time)
would prevent us from being able to grasp GNU make specific stuff.  So
let's try to keep such pre-processing at a minimum.

> annotation processors
> ---------------------
> 
> I think these can be handled by using _JAVACFLAGS or something similar.
>  They can generate source for new classes at compile time, but this is
> all handled by javac in separate locations so make doesn't need to know
> about it.
> 
Sounds good.

> javadoc, jni, check/test, etc
> ------------------------
> 
> Some built-in support would be good.  javadoc is easy, check/test should
> run junit tests I guess although i'm not too familiar with junit,
>
Or we might leave that to the use from now (they can use the 'check-local'
target), and maybe implement it in a later Automake version, once an usage
pattern has emerged (and most importantly, *if* there actually is any
request in that direction).

> jni will need some thought but should be limited in scope (support for
> javah basically, maybe not enough need to include it in automake so long as
> the dev can depend on the jar).
>
Sorry, I don't know enough on JNI to comment on this.  I understand that
it is a way to include platform-specific code in a Java application, but
I have no idea how it actually work, not how much common it is.

> first cut
> ---------
> 
> I've attached a pretty nasty bit of unportable makefile which attempts
> some of the above to see how it might be done.
>
Comments for that are below.  I understand this is just a preliminary and
rough implementation, so please don't allow my doubts and criticism blow
your morale.

BTW, while writing this first draft, have taken into consideration all
the comments in <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9088>,
as well as the extensive and (AFAICS) *working* code at
<https://github.com/stumbleupon/opentsdb/blob/6059488/Makefile#L207>?
We cannot allow all those earlier thoughts and efforts go wasted ...

>  Even with gnu make stuff
> it still needs some support from automake.in to handle multiple resource
> directories, and it doesn't handle everything quite properly.
> 
> The build steps from this are simple:
> 
> $ rm foo_jar.stamp ; make -f Auto.mak
> javac -cp lib/jjmpeg.jar:lib/test.jar -sourcepath src -d
> build/classes/foo_jar src/z/util/d.java src/z/util/c.java src/z/a.java
> src/z/b.java
> touch foo_jar.stamp
> jar cf foo.jar -C build/classes/foo_jar . \
>         -C ./src ./z/util/util-names.txt \
>         -C ./res ./foo.txt
> $
> 
> Regards,
>  Michael

>   =====   Auto.mak   =====
>
> # default is /usr/share/java/package ?
> jardir = $(datadir)/java/@PACKAGE@
>
Or simply '$(datadir)/java'; then we can pre-define pkgjardir
to be '$(datadir)/java/$(PACKAGE)'.  Something similar is
already done for datadir, includedir, libdir, and libexecdir:
<http://www.gnu.org/software/automake/manual/automake.html#index-pkglibdir-131>

> srcdir=.
Just a reminder: we must also be prepared to handle VPATH builds, that
is, '$(srcdir)' != '.'; no need to think about this right away, but it
should be made to work before the feature is integrated in Automake (so
we want a design that will allow it to work!)

> all: foo.jar
> jar_JARS=foo.jar
> foo_jar_SOURCES = src
NAK.  We should make the use list all the Java files, while (as said
before) being able to support GNU make extensions in the definition
of this variable.  Unless you have strong objections to this idea;
objections which I'd like to hear ASAP.

For more complete rationales about Automake's general opposition
against the use of wildcards by default, see:
<http://www.gnu.org/software/automake/manual/automake.html#Wildcards>

> # multiple resources dirs
> foo_jar_RESOURCES = src res
>
Same rationales as above apply.

> # same as in the way one would add a in-package libtool lib, but
> # this is in the build jar path
> foo_jar_LIBADD = lib/jjmpeg.jar lib/test.jar
>
Sounds good.  But maybe s/LIBADD/JARADD/?

> ## foo.jar
>
> # jar.am template, could handle multi-valued foo_jar_SOURCES
> foo_jar_SOURCES_list=$(shell cd $(srcdir) ; find $(foo_jar_SOURCES) -name '*.java')
>
This will no longer be required once $(foo_jar_SOURCES) is required
to be an explicit list.

> # automake.in generated, the find exclusions are not nearly enough to handle version control
> foo_jar_RESOURCES_list0=$(shell cd $(srcdir)/src; find . -type f -a \! -name '*.java' \! -name '*~')
> foo_jar_RESOURCES_list1=$(shell cd $(srcdir)/res; find . -type f -a \! -name '*.java' \! -name '*~')
>
Ditto, with 's/foo_jar_SOURCES/foo_jar_RESOURCES/'.

> # windows needs ;, of course ...
> # automake.in generates this from LIBADD?
>
I think it should.  And of course, should allow the user

> foo_jar_CLASSPATH=lib/jjmpeg.jar:lib/test.jar
>
> foo_jar.stamp: $(foo_jar_SOURCES_list) $(foo_jar_LIBADD)
> 	@-rm -rf build/classes/foo_jar
>
We don't want to ignore failures here, nor to silence commands.  Please
drop the leading '@-' in the line above.

> 	@-mkdir -p build/classes/foo_jar
>
Ditto.

In addition, "mkdir -p" is unfortunately not multithred-safe on all platforms.
We have code that should work around the issue; to take advantage of that,
just use the $(MKDIR_P) macro instead of "mkdir -p".

> 	javac -cp $(foo_jar_CLASSPATH) -sourcepath src -d build/classes/foo_jar $(filter %.java,$^)
>
We cannot unilaterally squat on a common file/directry name like "build"
here.  No need to fix that right now though; we can easily return on the
issue later.

Also, the '$(filter %.java,$^)' is GNU make specific; it should be juse
substituted by $(foo_jar_SOURCES), no?

> 	touch $@

> foo.jar: foo_jar.stamp $(patsubst %,src/%,$(foo_jar_RESOURCES_list0)) $(patsubst %,res/%,$(foo_jar_RESOURCES_list1))
> 	jar cf $@ -C build/classes/foo_jar . \
> 		$(patsubst %,-C $(srcdir)/src %,$(foo_jar_RESOURCES_list0)) \
> 		$(patsubst %,-C $(srcdir)/res %,$(foo_jar_RESOURCES_list1))
>
This is too much GNU make specific.  Moreover, I have zero knowledge about
how the "Java resources" are expected to work, so I don't know how to
suggest fixes or improvements.  Do you have links to noob-understendable
documentation that explains in enough detail what Java resources are and
how they are suppose to work?

> .PHONY javadoc: javadoc-foo_jar
> javadoc-foo_jar:
> 	exit 1
> 	build javadoc stuff, not done yet
>

Is see that we still lack clean and installation rules.  They are quite
important, and should be introduced ASAP.

Thanks, and best regards,
  Stefano