Re: bug#9088: Java, JARS primary?

[Stefano Lattarini]

On 05/16/2013 05:57 AM, Michael Zucchi wrote:
> On 15/05/13 22:39, Stefano Lattarini wrote:
>> On 05/15/2013 01:52 PM, Michael Zucchi wrote:
>>> On 14/05/13 03:47, Stefano Lattarini wrote:
>>>
>>>> Instead, let's start implementing something *correct*, in line with
>>>> the Java philosophy, and with a clean API.  We'll think about enhancing
>>>> it when (and if!) the need arise.
> 
> And then most of the objections are from trying the java way.  Oh well.
> 
>>> Seems the way to go.
>>>
>>> On that, here are a few more thoughts on java's specific way of doing
>>> things ...
>>>
>>> files vs dirs
>>> --------------
>>>
>>> java projects/build systems tend to have a fairly basic mechanism to
>>> define the source tree(s) which would be nice to emulate, but may not
>>> fit well with the portability requirements of automake.
>>>
>>> They simply define the root directory and let the build system do the rest.
>>>
>>> Not necessary but would be nice.  Or put another way it would basically
>>> suck if it required every source file to be listed explicitly and auto*
>>> having to be re-run every time one is added.
>>>
>> But then, how could make handle the case of generated java files?  I.e.,
>> if half of your files are generated by (say) an awk+sed incantation,
>> make has to know it must build them before trying to call the java compiler.
>> But see below.
> 
> Well if that's a requirement, then it just has to be added right?  It's
> not impossible, it's only software ...
>
I failed to parse this.  Can you rephrase?

> Either explicitly listed separately or handled automatically, it's quite
> easy to tell a file from a directory.
>
Ah, OK, so you mean something like the following:

   jar_JARS = foo.jar
   # All .java files in directory 'foo' ...
   foo_jar_SOURCES = foo/
   # ... plus this one that is not expected to be present at automake
   # time, but will be generated by make.
   foo_jar_SOURCES += foo/generated.java

>>> In a GNU makefile I can
>>> just use $(shell find) (or other fancier stuff), but for automake?
>>>
>> The solution is simpler than it seems at first sight: we should just
> 
> Actually it isn't quite that simple.  I did consider this. More below
> about resources.
> 
>> For an example of how this approach could work, take a look at the
>> EXTRA_DIST support (and to the test 't/extra-dist-wildcards-gnu.sh').
> 
> Although ... one can specify directories to EXTRA_DIST (even if not
> recommended).
>
> EXTA_DIST is defined relative to the makefile it is in,
>
Correct.

> but that isn't good enough.  It works fine for a root makefile since well,
> it's generating the source tree and that is by definition relative to the
> root makefile.
>
I'm lost again.  EXTRA_DIST is routinely used in subdir Makefiles ...

> For jars it needs to be relative to a another location
> which is relative to the makefile.
>
Failed to parse, sorry (and my utter ignorance about the Java world
is not helping me out here).

> The bit about nobase in the manual has an impractical solution of
> listing every directory separately.
>
But 'nobase_' is only needed for files that are going to be installed.
In our case, no .java, .class or resource file is going to be installed;
they are going to be used to create the .jar files, and *that* is going
to be installed.

> Somehow the makefile needs this information, either by listing it or by
> it's location, it cannot determine it automatically.  More detail and an
> example well below.
> 
>>> makefile location
>>> -----------------
>>>
>>> C projects often have a make file per directory, java has the build
>>> script at the top-level at least, outside of the source trees.
>>>
>>> This is desirable.
>>>
>> In some ways, this is desirable also for C projects actually; see the
>> article "Recursive Make Considered Harmful" from Peter Miller:
>>
>>     <http://miller.emu.id.au/pmiller/books/rmch/>
> 
> Yeah well that's just an opinion.  I don't like it myself because it
> makes emacs a total pain to use and I personally consider that pretty
> important.
> 
>>> For the resource files the Makefile needs to resolve the resource root,
>>> and strip it out of the names used for the dependencies.  So either it
>>> needs a (set of) explicit resource_root, resource_root_files variables,
>>> or it just needs to be defined as the root dir and handled either in the
>>> makefile fragment or in automake.in.
>>>
>> Doing too much pre-processing in automake.in (that is, at Automake time)
>> would prevent us from being able to grasp GNU make specific stuff.  So
>> let's try to keep such pre-processing at a minimum.
> 
> I'm only talking about iterating a list of strings and printing them in
> various ways, i'm pretty sure it does that kind of thing already.
>
Yes, but the price is reduce flexibility at Automake time.  Which might
be irrelevant or very important, depending on the expected usage patterns
of a feature.

> It wont be running find or walking directory trees since that doesn't
> provide anything useful at 'make' time.
> 
>>> javadoc, jni, check/test, etc
>>> ------------------------
>>>
>>> Some built-in support would be good.  javadoc is easy, check/test should
>>> run junit tests I guess although i'm not too familiar with junit,
>>>
>> Or we might leave that to the use from now (they can use the 'check-local'
>> target), and maybe implement it in a later Automake version, once an usage
>> pattern has emerged (and most importantly, *if* there actually is any
>> request in that direction).
> 
> javadoc+test are pretty important in the java world, and of course
> documentation is a critical component for any free software.  jni is
> just a personal need and I am familiar with how it works.  I also bring
> up jni because this is one area where java build tools really don't
> stack up at all and automake some non-zero chance of becoming a primary
> contender.
>
OK, then we need integration with javado and juinit.  But first let's
try to get the jar stuff right (that seems already non-trivial).

> I'm afraid based on the state of things that 'waiting for a request'
> will be a rather long wait ... the horse has quite bolted on the 'build
> tool' front in the last few years for all languages, and one can only
> surmise it was due to the complexity/inadequacy of the ones we already
> had.  It's hard to argue that auto* wasn't part of the reason many of
> these tools exist in the first place (i doubt anyone remember imake).
> Pity all the new ones tend to suck in their own ways not least of which
> being they all still suffer from the learning curve issue.
>
>>> first cut
>>> ---------
>>>
>>> I've attached a pretty nasty bit of unportable makefile which attempts
>>> some of the above to see how it might be done.
>>>
>> Comments for that are below.  I understand this is just a preliminary and
>> rough implementation, so please don't allow my doubts and criticism blow
>> your morale.
> 
> Not sure why you bothered pointing out the gnu make stuff every time
> when i stated that was a shortcoming for inclusion in automake.
>
Because it is not obvious to me how this usages will be later converted
to be portable.

OTOH, we might decide to simplify our lives and just require GNU make to
be used to run Makefiles implementing Java support (we already do so for
Vala support).  What is the percentage of machines out there that have
'javac' installed and but lack GNU make?

>> BTW, while writing this first draft, have taken into consideration all
>> the comments in <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9088>,
>> as well as the extensive and (AFAICS) *working* code at
>> <https://github.com/stumbleupon/opentsdb/blob/6059488/Makefile#L207>?
>> We cannot allow all those earlier thoughts and efforts go wasted ...
> 
> Yeah I looked at both of those, and the java.am stuff.
> 
>>> The build steps from this are simple:
>>>
>>> $ rm foo_jar.stamp ; make -f Auto.mak
>>> javac -cp lib/jjmpeg.jar:lib/test.jar -sourcepath src -d
>>> build/classes/foo_jar src/z/util/d.java src/z/util/c.java src/z/a.java
>>> src/z/b.java
>>> touch foo_jar.stamp
>>> jar cf foo.jar -C build/classes/foo_jar . \
>>>         -C ./src ./z/util/util-names.txt \
>>>         -C ./res ./foo.txt
>>> $
>>>
>>> Regards,
>>>  Michael
>>
>>>   =====   Auto.mak   =====
>>>
>>> # default is /usr/share/java/package ?
>>> jardir = $(datadir)/java/@PACKAGE@
>>>
>> Or simply '$(datadir)/java'; then we can pre-define pkgjardir
>> to be '$(datadir)/java/$(PACKAGE)'.  Something similar is
>> already done for datadir, includedir, libdir, and libexecdir:
>> <http://www.gnu.org/software/automake/manual/automake.html#index-pkglibdir-131>
> 
> I don't really know on this one.  I'm just looking at where java stuff
> is installed on my fedora system - although i never use any of that
> because it's easier/safer (configuration wise) to just include all the
> jars in my own packages particularly when supporting cross-platform
> applications.
> 
> I guess any jar installed into /usr/share/java would want a version in
> it's name.
>
>>> srcdir=.
>>>
>> Just a reminder: we must also be prepared to handle VPATH builds, that
>> is, '$(srcdir)' != '.'; no need to think about this right away, but it
>> should be made to work before the feature is integrated in Automake (so
>> we want a design that will allow it to work!)
> 
> srcdir is just set so that i have a working standalone makefile but can
> still follow some automake conventions inside of it to demonstrate the
> principle.   I was going to add a comment to that effect but thought it
> might've been a little ... obvious.
>
> I thought i stated that I think it should support a vpath build, but I
> haven't yet tested that.  So yes, I have already thought about it and
> it's something i'm quite well aware of.
>
OK.

>>> all: foo.jar
>>>
>>> jar_JARS=foo.jar
>>>
>>> foo_jar_SOURCES = src
>>>
>> NAK.  We should make the use list all the Java files, while (as said
>> before) being able to support GNU make extensions in the definition
>> of this variable.  Unless you have strong objections to this idea;
>> objections which I'd like to hear ASAP.
> 
> Well it was just an alternative.  This is the alternative I would like,
> and I'm pretty sure every normal java developer would expect.  If they
> have to manually list every source file every time a new one is added
> and then re-run autostuff, they wouldn't bother ever considering
> automake as it would just totally suck.
> 
> So, totally sucking is my only real objection.
> 
> My initial version listed every file, and used the gnu-make specific
> extensions in order to make it not "totally suck".   But then I ran into
> problems with how to handle resources and rather than work on solving
> those in a way which could fit into an automake.am fragment, I thought
> i'd try the approach I had desired from the start.
>
Given my ignorance of Java, I'm fully ready to change my mind on
this; but then I need examples that are much more clear, complete,
self-explanatory and *working*.  I think you should start perusing
the java tests in the 't/' subdirectory of the Automake source tree,
and start writing similar tests that show how you envision your
feature to work in different scenarios.  At this stage, using GNU
make specific constructs is OK; I promise I won't bother you about
them anymore.

> I have some more thoughts lower, and in general they work either way.
> 
>>> # same as in the way one would add a in-package libtool lib, but
>>> # this is in the build jar path
>>> foo_jar_LIBADD = lib/jjmpeg.jar lib/test.jar
>>>
>> Sounds good.  But maybe s/LIBADD/JARADD/?
> 
> Could be anything.  I followed the way libtool does it.  Previous
> comments in the bug seemed to suggest the desirability of sticking with
> existing names, so it wasn't a totally arbitrary choice.
>
And anyway, not a problem worth worrying about right now.  Better to
revisit it later, if needed.  Sorry for brining it up right away and
compounding to the confusion.

>> In addition, "mkdir -p" is unfortunately not multithred-safe on all 
>> platforms.
>> We have code that should work around the issue; to take advantage of that,
>> just use the $(MKDIR_P) macro instead of "mkdir -p".
> 
> ok no worries.
> 
>>>     javac -cp $(foo_jar_CLASSPATH) -sourcepath src -d build/classes/foo_jar 
>>> $(filter %.java,$^)
>>>
>> We cannot unilaterally squat on a common file/directry name like "build"
>> here.  No need to fix that right now though; we can easily return on the
>> issue later.
> 
> This is just what ant uses, and not surprisingly, it could be anything.
>  As a developer I would prefer it wasn't hidden but it's kind of here
> nor there.  No java tools I know ever hide it, it's a critical path in
> the build process and one can just run code from there, ignoring any jar
> file.
>
No need to hide it; and it's pretty easy to make it configurable, as
with:

  AM_JAVA_OUTDIR = build # What ant does.

  # ... later in the rules:
  javac -cp $(foo_jar_CLASSPATH) -sourcepath src -d 
$(AM_JAVA_OUTDIR)/classes/foo_jar ...

> I think a documented/makefile usable location will be needed for tools
> that work on the class files, but I don't have familarity with those.
> 
>> Also, the '$(filter %.java,$^)' is GNU make specific; it should be juse
>> substituted by $(foo_jar_SOURCES), no?
> 
> Yes I know I was using gnu make extensions.
> 
> Well, no, $(foo_jar_SOURCES) can't be used, at least not directly.
> It doesn't expand VPATH as the dependencies do but it could (obviously!)
> just use a similar approach as in the current java.am file.  Although
> that also uses the expanded dependencies so it would still need to
> filter out .java files.
> 
> This particular target was all very simple until i added the LIBADD
> dependencies which meant i could no longer just use $^ as i had.
>
> I couldn't think of a way of having $^ just list .java files whilst
> still depending on the LIBADD stuff for running the compiler but if that
> was possible the problem would vanish.
> 
Consider that $^ is not portable to all non-GNU makes, IIRC (but I'm not
100% sure of this, I'll have to double check), so that we'd likely still
need to rewrite such idioms when we make the transition from your
prototype from the final implementation.  Not a big deal right now, but
something worth keeping in mind.


>>>     touch $@
>>>
>>
>>> foo.jar: foo_jar.stamp $(patsubst %,src/%,$(foo_jar_RESOURCES_list0)) 
>>> $(patsubst %,res/%,$(foo_jar_RESOURCES_list1))
>>>     jar cf $@ -C build/classes/foo_jar . \
>>>             $(patsubst %,-C $(srcdir)/src %,$(foo_jar_RESOURCES_list0)) \
>>>             $(patsubst %,-C $(srcdir)/res %,$(foo_jar_RESOURCES_list1))
>>>
>> This is too much GNU make specific.  Moreover, I have zero knowledge about
>> how the "Java resources" are expected to work, so I don't know how to
>> suggest fixes or improvements.  Do you have links to noob-understendable
>> documentation that explains in enough detail what Java resources are and
>> how they are suppose to work?
> 
> Yes I know I was using gnu make extensions.
> 
> As previously explained they're typically just every file in the source
> tree which doesn't end in '.java' but they could be from (multiple) other
> trees.  I never read any documentation about this I just picked it up
> osmoticly[sic] from using netbeans.  Unlike other languages, java files
> really must end in '.java', so the distinction is easy.
> 
> They can be anything that is app-specific but there are also others
> which are to support java meta-data such as service plugin hooks.  But
> at the end of the day they're just files in a directory tree.  Obviously
> this means that a jar may also be purely data with no code whatsoever.
> 
> Newbie: It's a zip file of a directory tree of files.  It can contain
> anything a zip can.
> 
> It is definitely a hard requirement that they be able to be placed into
> the source tree in addition to other locations.  And this is why ...
> 
> e.g. a source tree:
> 
> a/a.java
> a/a.txt
> 
> jar:
> 
> a/a.class
> a/a.txt
> 
> a.java fragment:
> 
>   InputStream a.class.getResourceAsStream("a.txt");
> 
> i.e. a class has direct access to these resources relative to it's own
> class.  They are just loaded from the zip at runtime.
> 
OK, got it.  Thanks for the dumbed-down explanation.

> The problem is this:
> 
> a) the jar needs to depend on the resources, they need to be defined
> relative to the makefile
>
> b) to include the files in the jar command, they need to be specified as
> they appear in the jar file, i.e. relative to the jar root.
>
> c) with no a-priori knowledge of the root directory in which the
> resources reside, how can these two independent file lists be created?
> And support multiple directory trees?
> 
> This problem does not arise as such with the building of the java source
> because the compiler is already doing the translation for you, and the
> individual .class files are not tracked by make.  However, the problem
> does arise with extra class files which might be included/created
> somewhere else.
> 
> c is not an issue with EXTRA_DIST because the root directory is known
> a-priori due to it being where the Makefile is.
> 
> Basically say I have this:
> 
> foo_jar_RESOURCES=src/a/a.txt src/b/b.txt res/c/c.txt
> 
> I need two results from this input (taking into account out of source
> builds with the abstract use of $(srcdir) here, regardless of whether
> this is the correct automake foo or not):
> 
> 1) dependencies: $(srcdir)/src/a/a.txt $(srcdir)/src/a/a.txt
> $(srcdir)/res/c/c.txt
>  - this is just handled by VPATH already and so happens automagically
> 'for free'.
> 
> 2) relative paths (and their corresponding root):
> 
> root $(srcdir)/src  files a/a.txt b/b.txt
> root $(srcdir)/res  files c/c.txt
>
> Actually the end result I need explicitly for jar is this:
> 
>  -C $(srcdir)/src a/a.txt \
>  -C $(srcdir)/src b/b.txt \
>  -C $(srcdir)/res c/c.txt
> 
But how can the code know, in general, to only strip the first component
of the resources file?  That is, why:

    -C res c/c.txt

and not

    -C src/c c.txt

?

And it gets worse; if you have a resource in a/b/c/d/foo.txt, which one
of these possibilities should be used:

   -C . a/b/c/foo.txt
   -C a b/c/foo.txt
   -C a/b c/foo.txt
   -C a/b/c foo.txt

?

> However this can be in a file rather than a command argument (and
> doesn't need the newline escapes then).
> 
> Ideally jar could be used directly for this, alternatively a staging
> area could be used but this would slow the build down/waste disk space.
> 
> Possible solutions (not exhaustive, not suggestions):
> 
> a) As included in the makefile so far.  Supply the roots and let the
> script handle everything.  It has enough information to do everything
> required.  This is how every java coder on the planet will want and
> expect it to work.
> 
> b) specify each resource root using multiple variables.
> 
> Possible non-suggestions:
> 
> b.0)
> foo_bar_RESOURCES = src res ../android/res
> foo_bar_src_RESOURCES = a/a.txt b/b.txt
> foo_bar_res_RESOURCES = c/c.txt
> foo_bar____android_res_RESOURCES = pics/bob.jpg
> 
> (obvious problems with relative paths with this idea)
>
And doesn't integrate well with the typical Automake naming scheme.
That is, 'foo_bar_src_RESOURCES' would suggest you have a
'foo/bar/src.jar' Jar file to build ...

> b.1)
> foo_bar_RESOURCES_anything = src
> foo_bar_RESOURCES_anything_FILES = a/a.txt b/b.txt
> foo_bar_RESOURCES_blah = res
> foo_bar_RESOURCES_blah_FILES = c/c.txt
> etc.
> 
> b.2)
> foo_bar_RESOURCESDIRS = src res
>
(or foo_bar_RESOURCE_PREFIXES even)

> foo_bar_RESOURCES = src/a/a.txt src/b/b.txt root/c/c.txt
> 
> (use regex/sed to strip either root: it can be safely assumed they are
> independent trees so this should be easy)
>
This sounds the best approach so far.  I'd love to see a preliminary
implementation of that ;-)

> b.3) Something more implicit like the way nobase_ works, but i don't
> know how you'd define the root in that case.
> 
> There may be automake.in conventions which make these or other
> arrangements more desirable.  But whatever it is this basic information
> is required in the end: a root and a list relative to that root.
> 

> c) Define everything twice.  The 'jar' name (plus root, somehow), and
> the 'dependency name'.
> 
> d) force individual makefiles into the 'src' or 'res' roots.  Then
> require everything to be copied into a staging area.  Then use another
> makefile to create the actual jar.
> 
> e) List every source dir and target dir by hand (example given in the
> nobase_ part of the manual S7.3 - node 'Alternative').
> 

> Judgments:
>
> a) nicest to user.
>
But might be problematic to implement, and goes against Automake's
"Uniform Naming Scheme".

> b) bearable.
>
And logical, easy to grasp, and simple for the user to control.

> c) sucks, messy
> d) slow, messy, painful to use.
> e) ugh.
> 
> Whatever it is the same mechanism should probably be used for multiple
> source trees and pre-existing/make-generated class trees.
> 
> Actually just using SOURCES alone would work for everything (i.e. no
> _RESOURCES stuff, but it still needs to know the relative root), and
> just filter out '.java' for the resources/extra classes.
>
I'm OK with this unified approach, as long as it doesn't complicate the
implementation.

> This is
> assuming one never wanted a .java source file in such a generated jar,
> which I think is a fairly reasonable assumption as these are about code
> not source.
>
Agreed.

>>
>>> .PHONY javadoc: javadoc-foo_jar
>>> javadoc-foo_jar:
>>>     exit 1
>>>     build javadoc stuff, not done yet
>>>
>>
>> Is see that we still lack clean and installation rules.  They are quite
>> important, and should be introduced ASAP.
> 
> These are just trivialities.  I thought i'd get the hard problems out of
> the way before giving you a final solution you wouldn't accept anyway -
> i know how this works.  I was worried the fragment presented wasn't
> complete enough but now i'm glad I didn't spend more time on it before
> feedback.
> 
> You seem to have got quite hung up on the gnu make stuff even though i
> stated that wasn't a consideration at this point.
>
Not a show stopper, but we have to avoid painting ourselves in a corner
by using GNU make features that we then are not able to "port" to non-GNU
makes (unlikely, but possible).  Or we can decide up-front to require
GNU make unconditionally for the Java support, and forger about the
portability issues.

>  Obviously anything it
> does can be implemented using sed or grep, and java tools allow
> arguments to be placed into files for better scalability.
> 
> Questions then arise: what of such basic tools can be used,
>
Basically, the POSIX tools (sed, grep, awk, etc), avoiding all GNU
extensions.  But don't overly worry about that, its easy for us to
spot and fix possible unportable uses.  Just don't get upset by the
associated nitpicking ;-)

> and what
> limitations on filenames must be adhered to should they be used for
> argument lists (if this is in hacking just leave it at that).
>
I fear HACKING is quite incomplete in this regard; the best reference
for what can and can't be done portably with POSIX tools in the
Autconf manual, section 11 "Portable Shell Programming":

<http://www.gnu.org/software/autoconf/manual/autoconf.html#Portable-Shell>

(no, you are not supposed to read all of that, don't worry :-)

>  Even if
> every file was listed by hand these capabilities are still going to be
> required.
> 
> I notice that automake tends to just use `` instead:
>
What do you intend with `` exactly?  Are you referring to command
substitutions?  If yes, how are they related to command line limits
issues.

> aren't command line
> limits a portability issue anymore, or is it just not likely a problem
> with c source?
>
Mostly, it's *extremely* difficult to exceed command line length limits
with a list of sources, unless you are going to build a program or
library (or Jar file, in our case) that has thousands of sources ...

>  !Z

Regards,
  Stefano