Feature request, with implementation, test and rationale.

[Fredrik Öhrström]

When I designed the new makefiles for the OpenJDK project
(http://openjdk.java.net/projects/build-infra/guide.html)
The first and foremost  problem was,
how to efficiently compile Java software from makefiles.

That included:
1  not wanting to spawn a jvm, to check if any java sources had changed.
2  efficient and correct incremental compiles
3  using multiple cores when compiling Java
4  the makefile should be reasonable easy to write

2 and 3 cannot be solved without improvements to javac. These have now
been implemented
in the smart javac wrapper as part of the build-infra project, they
are not yet part of the official OpenJDK.

For 4, I created a neat way of passing named parameters to make
macros. Thus with the utility library
JavaCompilation.gmk
(http://hg.openjdk.java.net/build-infra/jdk8/file/345dbc49f7d0/common/makefiles/JavaCompilation.gmk)
I can write:

$(eval $(call SetupJavaCompiler,GENERATE_NEWBYTECODE, \
     JAVAC:=javac,\
     FLAGS:=-g)

$(eval $(call SetupJavaCompilation,BUILD_JAXP,\
     SETUP:=GENERATE_NEWBYTECODE,\
     SRC:=jaxp/src,\
     BIN:=output/classes,\
     SRCZIP:=output/src.zip),\
     JAR:=output/classes.jar))

all: output/src.zip output/classes.jar

This will setup all the necessary makefile rules to generate src.zip
and classes.jar
and do proper dependency checking and incremental builds.

Now, how about 1: not wanting to spawn a jvm, to check if any java
sources had changed.

You want to find all relevant sources to compile, easily done using find,
for example

SRCS:=$(shell find src -name "*.java")
SRCS:=do all sorts of filtering on the SRCS
_the.batch : $(SRCS)
       rm _the.batch
       javac -d bin $(SRCS)
       touch _the.batch

Simple eh? Unfortunately the command line length limits hits us very quickly
when compiling java! For example the jdk in the OpenJDK project has ~8800
source files, and this is a small Java project! Its not a problem for make which
deals with these number of files fast and efficiently. As you know the
number of Java files
in a project is a magnitude larger compared to if the project was written in C.
If Java source code was more modular, we could of course rely on developers
to split up the sources into smaller jars. But it won't help on some
platforms like cygwin
where the command line length limit is so short that it can barely
handle small java project.

To get around this particular problem I implemented a workaround marcro
called ListPathsSafely that writes the contents of a variable to disk.
(http://hg.openjdk.java.net/build-infra/jdk8/file/345dbc49f7d0/common/makefiles/MakeBase.gmk)
that lies somewhere in the borderlands between genius and insanity....

I would like to get rid of ListPathsSafely, to do so I would like to
suggest adding two new functions to
GNU make: file-write and file-writelns. For example:
   $(file-writelns list.txt,$(SRCS))
would write each word in SRCS on each own line into the file list.txt.
   $(file-write list.txt,$(SRCS))
would write all the sources on a single line with spaces intact.

This is a very simple addition to GNU make, I have supplied
implementation and a test
at the end of this email. I do not believe it would cause any security
holes in make.
It would give a lot of developers, not just Java developers, a very useful tool
to get around severe command line lengths limits on platforms like cygwin.

With these two new functions it would be trivial to write a portable,
efficient makefile for Java
that can deal with large projects.

SRCS:=$(shell find src -name "*.java")
SRCS:=do all sorts of filtering on the SRCS
_the.batch : $(SRCS)
       rm _the.batch
       $(file-writelns _the.batch.tmp,$(SRCS))
       javac -d bin @_the.batch.tmp
       mv _the.batch.tmp _the.batch

If you are interested in more details, I gave a presentation on this
topic on JavaOne:
CON6659 - Building Large Java Projects Faster: Multicore javac and
Makefile Integration
http://www.myexpospace.com/JavaOne2012/SessionFiles/CON6659_PDF_6659_0001.pdf
http://www.myexpospace.com/JavaOne2012/SessionFiles/CON6659_mp4_6659_001.mp4

Fredrik Öhrström

----------------------------------------
In function.c
/*
  $(file-write file-name,content)
  $(file-writeln file-name,content)

  Write the content into the newly opened (truncated) file-name.
  The file-name argument is trimmed from any leading or ending
whitespace.
  file-write writes the content as is, to disk.
  file-writelns will write '\n' after each and every word in content.
*/
static char *
func_file_write (char *o, char **argv, const char *funcname)
{
  const char *file_name = argv[0];
  const char *content = argv[1];
  FILE *f;
  int is_writelns = streq (funcname, "file-writelns");

  if (file_name != NULL && content != NULL)
    {
      const char *end = file_name + strlen (file_name) - 1;
      strip_whitespace (&file_name, &end);
      ((char*)end)[1] = '\0';

      if (file_name != NULL)
        {
          f = fopen(file_name, "w");
          if (f != NULL)
            {
                if (is_writelns) {
                    /* Append a new line after each word found in content. */
                    const char *word_iterator = content;
                    unsigned int len;
                    const char *w = content;
                    while ((w = find_next_token (&word_iterator, &len)) != 0) {
                        fwrite(w, 1, len, f);
                        fwrite("\n", 1, strlen("\n"), f);
                    }
                } else {
                    /* Write content, as is, to disk. */
                    int n = strlen(content);
                    fwrite(content, 1, n, f);
                }
                fclose(f);
            }
        }
    }
  return o;
}

and
  { STRING_SIZE_TUPLE("file-write"),    1,  2,  1,  func_file_write},
  { STRING_SIZE_TUPLE("file-writelns"), 1,  2,  1,  func_file_write},

and a new test tests/scripts/functions/file-write

#
-*-perl-*-

$description = 'Test the $(file-write ...) function.';

$details = '';

$file_to_write = $workpath.'/'.$dir.'/generated';

#### Test a simple generation

run_make_test('
.PHONY: all
define NL


endef

$(file-write '.$file_to_write.',hi$(NL))

all: ; @cat '.$file_to_write.'

','','hi');

#### Test writelns

run_make_test('
.PHONY: all

$(file-writelns '.$file_to_write.',hi)

all: ; @cat '.$file_to_write.'

','','hi');

#### Test multiple lines

run_make_test('
.PHONY: all

$(file-writelns '.$file_to_write.',alfa beta gamma)

all: ; @cat '.$file_to_write.'

','','alfa
beta
gamma');

#### Test that make handles bad input to file-write without crashing.

run_make_test('
.PHONY: all
$(file-write)
$(file-write,hi)
$(file-write,,hi)
$(file-writelns)
$(file-writelns,hi)
$(file-writelns,,hi)
all: ; @echo hi
','','hi');

1;