Re: bug#9088: Java, JARS primary?

[Michael Zucchi]

On 13/05/13 02:28, Stefano Lattarini wrote:
> On 05/12/2013 06:29 AM, Michael Zucchi wrote:

>> Using $? will not suffice as a dependency check, as it's trivially easy
>> to create an example which will compile ok after a change but create a
>> broken jar. e.g. add a new abstract method to an abstract class but
>> forget to fix all sub-classes.
>>
> I don't really follow here, sorry (likely because I know almost nothing
> about Java).  Do you mean that, if you have a bunch of .java files that
> get compiled into a single jar, and you change just one of these files,
> you also need to recompile all the other ones in order not to risk ending
> up with a broken and/or inconsistent jar?  If it is so, that's awful :-(

Well if you were only going on the timestamps of the changed files it
would not work.

I'm not sure it's really that awful - it's done in C because that's how
C is compiled, and compilation is so slow.  If it was made fast enough
the need for all the considerable complexity of creating and managing
dependencies would go away.

For a point of reference with a small library i have, a build of 104
java files with 22KLOC takes under 3s on my workstation.  Building them
"one source file at a time" (see next quoted block) takes about a minute.

For big projects code that doesn't change often can be placed in it's
own jar negating the need to rebuild that part every time downstream
code changes - although one does have to rebuild everything downstream
when that jar does.

Hmm, now i think about it there is a possible solution to the dependency
problem but i'm not sure it's worth it and it has some caveats.  The
main one being that stale classes could still make it to the jar (so you
would have to manually make clean before make install), and it also
needs a dependency tool.

Because of the strict rules on java package names (they must match the
directory they file is in), the import statements could be used to
generate some coarse-but-accurate dependencies.  As anything in the same
package does not need an import statement, one would have to assume
every class in the same package is included (i.e. ostensibly the same
directory, but it may span multiple trees).

Actually it isn't quite that simple - as you can just use fully
qualified names without any import statement, so the dependency tool
would have to scan the whole file.

However, assuming these coarse dependencies could be generated reliably
... the following might work.

Lets say the project is something like

src/z/a.java imports c
src/z/b.java
src/z/util/c.java
src/z/util/d.java

A deps file is created with the cross dependencies for each file.  It is
simply everything in the same package, and then everything within the
project that a given file imports.

the deps file:

src/z/a.java src/z/b.java
src/z/a.java src/z/util/c.java
src/z/util/c.java src/z/util/d.java

i.e.
src/z/*.java, src/z/utill/*.java, then dependencies for any files

the Makefile:

# this is so plain package names can be used
VPATH=src

JAVASRC=z/a.java z/b.java z/util/c.java z/util/d.java

all: z.jar

build.stamp: $(JAVASRC)
        @-mkdir classes
        @-rm buildsrc
        for n in $? ; do grep $$n deps >> buildsrc ; done
        javac -sourcepath src -d classes @buildsrc
        touch build.stamp
        rm buildsrc

# just classes - in reality it needs to deal with other files
z.jar: build.stamp
        jar cf $@ -C classes .

So it takes the modified list of files and expands it to include any
forward requirements.

So for example, touch src/z/util/c.java will rebuild a, c, and d - all
the classes which might possibly have something to with c.

This simple example has some bugs in that it might over-build (e.g.
touch a.java will build a, b as required but also c which is not) but
that is fixable.

Given the caveats, is it worth it?

>> So without compiler support for dependency generation I think the only
>> practical solution will be to build all files every time.  Even the
>> sub-directory holding the classes will probably need to be wiped away
>> otherwise the jar could contain extraneous classes no longer generated
>> from the corresponding source which would probably not be a good thing.

> Couldn't we put the *.class files obtained compiling a foo.java file
> into a (say) 'foo.d' directory, and remove & rebuild only that directory
> whenever foo.java changes?

You would have to then invoke the compiler individually on each file,
which is at least 1-3 orders of magnitude slower than just compiling
every file at the same time.  And it wouldn't have the desired result
anyway, if it can't find any required .class files it finds the
corresponding source and builds them into the target location as well so
then build order becomes an issue.

I can go into more detail if you like ...

One might ask why use make then, but there is more to real projects than
a bunch of java source files that need compiling.

 Michael