Re: bug#9088: Java, JARS primary?

[Michael Zucchi]

On 15/05/13 22:39, Stefano Lattarini wrote:
> On 05/15/2013 01:52 PM, Michael Zucchi wrote:
>> On 14/05/13 03:47, Stefano Lattarini wrote:
>>
>>> Instead, let's start implementing something *correct*, in line with
>>> the Java philosophy, and with a clean API.  We'll think about enhancing
>>> it when (and if!) the need arise.

And then most of the objections are from trying the java way.  Oh well.

>> Seems the way to go.
>>
>> On that, here are a few more thoughts on java's specific way of doing
>> things ...
>>
>> files vs dirs
>> --------------
>>
>> java projects/build systems tend to have a fairly basic mechanism to
>> define the source tree(s) which would be nice to emulate, but may not
>> fit well with the portability requirements of automake.
>>
>> They simply define the root directory and let the build system do the rest.
>>
>> Not necessary but would be nice.  Or put another way it would basically
>> suck if it required every source file to be listed explicitly and auto*
>> having to be re-run every time one is added.
>>
> But then, how could make handle the case of generated java files?  I.e.,
> if half of your files are generated by (say) an awk+sed incantation,
> make has to know it must build them before trying to call the java compiler.
> But see below.

Well if that's a requirement, then it just has to be added right?  It's
not impossible, it's only software ...

Either explicitly listed separately or handled automatically, it's quite
easy to tell a file from a directory.

>> In a GNU makefile I can
>> just use $(shell find) (or other fancier stuff), but for automake?
>>
> The solution is simpler than it seems at first sight: we should just

Actually it isn't quite that simple.  I did consider this. More below
about resources.

> For an example of how this approach could work, take a look at the
> EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh').

Although ... one can specify directories to EXTRA_DIST (even if not
recommended).

EXTA_DIST is defined relative to the makefile it is in, but that isn't
good enough.  It works fine for a root makefile since well, its'
generating the source tree and that is by definition relative to the
root makefile.  For jars it needs to be relative to a another location
which is relative to the makefile.

The bit about nobase in the manual has an impractical solution of
listing every directory separately.

Somehow the makefile needs this information, either by listing it or by
it's location, it cannot determine it automatically.  More detail and an
example well below.

>> makefile location
>> -----------------
>>
>> C projects often have a make file per directory, java has the build
>> script at the top-level at least, outside of the source trees.
>>
>> This is desirable.
>>
> In some ways, this is desirable also for C projects actually; see the
> article "Recursive Make Considered Harmful" from Peter Miller:
> 
>     <http://miller.emu.id.au/pmiller/books/rmch/>

Yeah well that's just an opinion.  I don't like it myself because it
makes emacs a total pain to use and I personally consider that pretty
important.

>> For the resource files the Makefile needs to resolve the resource root,
>> and strip it out of the names used for the dependencies.  So either it
>> needs a (set of) explicit resource_root, resource_root_files variables,
>> or it just needs to be defined as the root dir and handled either in the
>> makefile fragment or in automake.in.
>>
> Doing too much pre-processing in automake.in (that is, at Automake time)
> would prevent us from being able to grasp GNU make specific stuff.  So
> let's try to keep such pre-processing at a minimum.

I'm only talking about iterating a list of strings and printing them in
various ways, i'm pretty sure it does that kind of thing already.  It
wont be running find or walking directory trees since that doesn't
provide anything useful at 'make' time.

>> javadoc, jni, check/test, etc
>> ------------------------
>>
>> Some built-in support would be good.  javadoc is easy, check/test should
>> run junit tests I guess although i'm not too familiar with junit,
>>
> Or we might leave that to the use from now (they can use the 'check-local'
> target), and maybe implement it in a later Automake version, once an usage
> pattern has emerged (and most importantly, *if* there actually is any
> request in that direction).

javadoc+test are pretty important in the java world, and of course
documentation is a critical component for any free software.  jni is
just a personal need and I am familiar with how it works.  I also bring
up jni because this is one area where java build tools really don't
stack up at all and automake some non-zero chance of becoming a primary
contender.

I'm afraid based on the state of things that 'waiting for a request'
will be a rather long wait ... the horse has quite bolted on the 'build
tool' front in the last few years for all languages, and one can only
surmise it was due to the complexity/inadequacy of the ones we already
had.  It's hard to argue that auto* wasn't part of the reason many of
these tools exist in the first place (i doubt anyone remember imake).
Pity all the new ones tend to suck in their own ways not least of which
being they all still suffer from the learning curve issue.

>> first cut
>> ---------
>>
>> I've attached a pretty nasty bit of unportable makefile which attempts
>> some of the above to see how it might be done.
>>
> Comments for that are below.  I understand this is just a preliminary and
> rough implementation, so please don't allow my doubts and criticism blow
> your morale.

Not sure why you bothered pointing out the gnu make stuff every time
when i stated that was a shortcoming for inclusion in automake.

> BTW, while writing this first draft, have taken into consideration all
> the comments in <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9088>,
> as well as the extensive and (AFAICS) *working* code at
> <https://github.com/stumbleupon/opentsdb/blob/6059488/Makefile#L207>?
> We cannot allow all those earlier thoughts and efforts go wasted ...

Yeah I looked at both of those, and the java.am stuff.

>> The build steps from this are simple:
>>
>> $ rm foo_jar.stamp ; make -f Auto.mak
>> javac -cp lib/jjmpeg.jar:lib/test.jar -sourcepath src -d
>> build/classes/foo_jar src/z/util/d.java src/z/util/c.java src/z/a.java
>> src/z/b.java
>> touch foo_jar.stamp
>> jar cf foo.jar -C build/classes/foo_jar . \
>>         -C ./src ./z/util/util-names.txt \
>>         -C ./res ./foo.txt
>> $
>>
>> Regards,
>>  Michael
> 
>>   =====   Auto.mak   =====
>>
>> # default is /usr/share/java/package ?
>> jardir = $(datadir)/java/@PACKAGE@
>>
> Or simply '$(datadir)/java'; then we can pre-define pkgjardir
> to be '$(datadir)/java/$(PACKAGE)'.  Something similar is
> already done for datadir, includedir, libdir, and libexecdir:
> <http://www.gnu.org/software/automake/manual/automake.html#index-pkglibdir-131>

I don't really know on this one.  I'm just looking at where java stuff
is installed on my fedora system - although i never use any of that
because it's easier/safer (configuration wise) to just include all the
jars in my own packages particularly when supporting cross-platform
applications.

I guess any jar installed into /usr/share/java would want a version in
it's name.

>> srcdir=.
>>
> Just a reminder: we must also be prepared to handle VPATH builds, that
> is, '$(srcdir)' != '.'; no need to think about this right away, but it
> should be made to work before the feature is integrated in Automake (so
> we want a design that will allow it to work!)

srcdir is just set so that i have a working standalone makefile but can
still follow some automake conventions inside of it to demonstrate the
principle.   I was going to add a comment to that effect but thought it
might've been a little ... obvious.

I thought i stated that I think it should support a vpath build, but I
haven't yet tested that.  So yes, I have already thought about it and
it's something i'm quite well aware of.

>> all: foo.jar
>>
>> jar_JARS=foo.jar
>>
>> foo_jar_SOURCES = src
>>
> NAK.  We should make the use list all the Java files, while (as said
> before) being able to support GNU make extensions in the definition
> of this variable.  Unless you have strong objections to this idea;
> objections which I'd like to hear ASAP.

Well it was just an alternative.  This is the alternative I would like,
and I'm pretty sure every normal java developer would expect.  If they
have to manually list every source file every time a new one is added
and then re-run autostuff, they wouldn't bother ever considering
automake as it would just totally suck.

So, totally sucking is my only real objection.

My initial version listed every file, and used the gnu-make specific
extensions in order to make it not "totally suck".   But then I ran into
problems with how to handle resources and rather than work on solving
those in a way which could fit into an automake.am fragment, I thought
i'd try the approach I had desired from the start.

I have some more thoughts lower, and in general they work either way.

>> # same as in the way one would add a in-package libtool lib, but
>> # this is in the build jar path
>> foo_jar_LIBADD = lib/jjmpeg.jar lib/test.jar
>>
> Sounds good.  But maybe s/LIBADD/JARADD/?

Could be anything.  I followed the way libtool does it.  Previous
comments in the bug seemed to suggest the desirability of sticking with
existing names, so it wasn't a totally arbitrary choice.

> In addition, "mkdir -p" is unfortunately not multithred-safe on all platforms.
> We have code that should work around the issue; to take advantage of that,
> just use the $(MKDIR_P) macro instead of "mkdir -p".

ok no worries.

>>      javac -cp $(foo_jar_CLASSPATH) -sourcepath src -d build/classes/foo_jar 
>> $(filter %.java,$^)
>>
> We cannot unilaterally squat on a common file/directry name like "build"
> here.  No need to fix that right now though; we can easily return on the
> issue later.

This is just what ant uses, and not surprisingly, it could be anything.
 As a developer I would prefer it wasn't hidden but it's kind of here
nor there.  No java tools I know ever hide it, it's a critical path in
the build process and one can just run code from there, ignoring any jar
file.

I think a documented/makefile usable location will be needed for tools
that work on the class files, but I don't have familarity with those.

> Also, the '$(filter %.java,$^)' is GNU make specific; it should be juse
> substituted by $(foo_jar_SOURCES), no?

Yes I know I was using gnu make extensions.

Well, no, $(foo_jar_SOURCES) can't be used, at least not directly.
It doesn't expand VPATH as the dependencies do but it could (obviously!)
just use a similar approach as in the current java.am file.  Although
that also uses the expanded dependencies so it would still need to
filter out .java files.

This particular target was all very simple until i added the LIBADD
dependencies which meant i could no longer just use $^ as i had.

I couldn't think of a way of having $^ just list .java files whilst
still depending on the LIBADD stuff for running the compiler but if that
was possible the problem would vanish.

>>      touch $@
>>
> 
>> foo.jar: foo_jar.stamp $(patsubst %,src/%,$(foo_jar_RESOURCES_list0)) 
>> $(patsubst %,res/%,$(foo_jar_RESOURCES_list1))
>>      jar cf $@ -C build/classes/foo_jar . \
>>              $(patsubst %,-C $(srcdir)/src %,$(foo_jar_RESOURCES_list0)) \
>>              $(patsubst %,-C $(srcdir)/res %,$(foo_jar_RESOURCES_list1))
>>
> This is too much GNU make specific.  Moreover, I have zero knowledge about
> how the "Java resources" are expected to work, so I don't know how to
> suggest fixes or improvements.  Do you have links to noob-understendable
> documentation that explains in enough detail what Java resources are and
> how they are suppose to work?

Yes I know I was using gnu make extensions.

As previously explained they're typically just every file in the source
tree which doesn't end in '.java' but they could be from (mutiple) other
trees.  I never read any documentation about this I just picked it up
osmoticly[sic] from using netbeans.  Unlike other languages, java files
really must end in '.java', so the distinction is easy.

They can be anything that is app-specific but there are also others
which are to support java meta-data such as service plugin hooks.  But
at the end of the day they're just files in a directory tree.  Obviously
this means that a jar may also be purely data with no code whatsoever.

Newbie: It's a zip file of a directory tree of files.  It can contain
anything a zip can.


It is definitely a hard requirement that they be able to be placed into
the source tree in addition to other locations.  And this is why ...

e.g. a source tree:

a/a.java
a/a.txt

jar:

a/a.class
a/a.txt

a.java fragment:

  InputStream a.class.getResourceAsStream("a.txt");

i.e. a class has direct access to these resources relative to it's own
class.  They are just loaded from the zip at runtime.

The problem is this:

a) the jar needs to depend on the resources, they need to be defined
relative to the makefile
b) to include the files in the jar command, they need to be specified as
they appear in the jar file, i.e. relative to the jar root.
c) with no a-priori knowledge of the root directory in which the
resources reside, how can these two independent file lists be created?
And support multiple directory trees?

This problem does not arise as such with the building of the java source
because the compiler is already doing the translation for you, and the
individual .class files are not tracked by make.  However, the problem
does arise with extra class files which might be included/created
somewhere else.

c is not an issue with EXTRA_DIST because the root directory is known
a-priori due to it being where the Makefile is.

Basically say I have this:

foo_jar_RESOURCES=src/a/a.txt src/b/b.txt res/c/c.txt

I need two results from this input (taking into account out of source
builds with the abstract use of $(srcdir) here, regardless of whether
this is the correct automake foo or not):

1) dependencies: $(srcdir)/src/a/a.txt $(srcdir)/src/a/a.txt
$(srcdir)/res/c/c.txt
 - this is just handled by VPATH already and so happens automagically
'for free'.

2) relative paths (and their corresponding root):

root $(srcdir)/src  files a/a.txt b/b.txt
root $(srcdir)/res  files c/c.txt

Actually the end result I need explicitly for jar is this:

 -C $(srcdir)/src a/a.txt \
 -C $(srcdir)/src b/b.txt \
 -C $(srcdir)/res c/c.txt

However this can be in a file rather than a command argument (and
doesn't need the newline escapes then).

Ideally jar could be used directly for this, alternatively a staging
area could be used but this would slow the build down/waste disk space.

Possible solutions (not exhaustive, not suggestions):

a) As included in the makefile so far.  Supply the roots and let the
script handle everything.  It has enough information to do everything
required.  This is how every java coder on the planet will want and
expect it to work.

b) specify each resource root using multiple variables.

Possible non-suggestions:

b.0)
foo_bar_RESOURCES = src res ../android/res
foo_bar_src_RESOURCES = a/a.txt b/b.txt
foo_bar_res_RESOURCES = c/c.txt
foo_bar____android_res_RESOURCES = pics/bob.jpg

(obvious problems with relative paths with this idea)

b.1)
foo_bar_RESOURCES_anything = src
foo_bar_RESOURCES_anything_FILES = a/a.txt b/b.txt
foo_bar_RESOURCES_blah = res
foo_bar_RESOURCES_blah_FILES = c/c.txt
etc.

b.2)
foo_bar_RESOURCESDIRS = src res
foo_bar_RESOURCES = src/a/a.txt src/b/b.txt root/c/c.txt

(use regex/sed to strip either root: it can be safely assumed they are
independent trees so this should be easy)

b.3) Something more implicit like the way nobase_ works, but i don't
know how you'd define the root in that case.

There may be automake.in conventions which make these or other
arrangements more desirable.  But whatever it is this basic information
is required in the end: a root and a list relative to that root.

c) Define everything twice.  The 'jar' name (plus root, somehow), and
the 'dependency name'.

d) force individual makefiles into the 'src' or 'res' roots.  Then
require everything to be copied into a staging area.  Then use another
makefile to create the actual jar.

e) List every source dir and target dir by hand (example given in the
nobase_ part of the manual S7.3 - node 'Alternative').

a) nicest to user.
b) bearable.
c) sucks, messy
d) slow, messy, painful to use.
e) ugh.

Whatever it is the same mechanism should probably be used for multiple
source trees and pre-existing/make-generated class trees.

Actually just using SOURCES alone would work for everything (i.e. no
_RESOURCES stuff, but it still needs to know the relative root), and
just filter out '.java' for the resources/extra classes.  This is
assuming one never wanted a .java source file in such a generated jar,
which I think is a fairly reasonable assumption as these are about code
not source.

> 
>> .PHONY javadoc: javadoc-foo_jar
>> javadoc-foo_jar:
>>      exit 1
>>      build javadoc stuff, not done yet
>>
> 
> Is see that we still lack clean and installation rules.  They are quite
> important, and should be introduced ASAP.

These are just trivialities.  I thought i'd get the hard problems out of
the way before giving you a final solution you wouldn't accept anyway -
i know how this works.  I was worried the fragment presented wasn't
complete enough but now i'm glad I didn't spend more time on it before
feedback.

You seem to have got quite hung up on the gnu make stuff even though i
stated that wasn't a consideration at this point.  Obviously anything it
does can be implemented using sed or grep, and java tools allow
arguments to be placed into files for better scalability.

Questions then arise: what of such basic tools can be used, and what
limitations on filenames must be adhered to should they be used for
argument lists (if this is in hacking just leave it at that).  Even if
every file was listed by hand these capabilities are still going to be
required.

I notice that automake tends to just use `` instead: aren't command line
limits a portability issue anymore, or is it just not likely a problem
with c source?

 !Z