Re: Automatically Handling Tools that Produce Many Outputs

[Ralf Wildenhues]

Hello Olly,

Thanks for bringing this up.

* Olly Betts wrote on Tue, Nov 27, 2007 at 01:30:31AM CET:
> I've read (and use the recipes from) the automake manual section
> "Handling Tools that Produce Many Outputs".  However, these recipes
> require a lot of boilerplate code which annoyingly obscures the actual
> rule they are protecting - a two line rule is swamped by a dozen lines of
> boilerplate.

Indeed.

> It's just struck me that automake is good at inserting lots of
> boilerplate code into makefiles!

;-)

> Has anyone considered adding some sort of support to automake to allow
> such rules to get rewritten automatically?

AFAIK no.

> I don't have a concrete proposal, but it seems that this would need
> to be controllable rather than automatically adjusting any rule with
> multiple targets, if only to avoid messing with rules which are already
> protected.

Not sure what you mean here.  But surely rewriting rules is bad.

> Two initial ideas - either a new automake option, or special handling
> for a rule which is written with some other token instead of a ":"
> (or perhaps both?).
> 
> GNU make support "::" rules, but ":::" could be used perhaps.  The multiple
> colons are mnemonic for multiple outputs.
> 
> So this in Makefile.am would mean "automatically protect this rule for me":
> 
>     data.c data.h::: data.foo
>             foo data.foo

Yuck.  Let's better not overload `make' syntax.  That would drive us
further away from being able to copy the contents of Makefile.am into
the output file, which is one Automake design item.

> The main difficulty I see is what to name the auxiliary files (witnesses,
> locks, etc) so as avoid clashing with user files.  Perhaps creating a
> subdirectory for them would be best - it would make it easy to ignore
> them in a VCS.

Yuck.  No, that doesn't play well with input/output files that live in
subdirs.


FWIW, here's a set of unreflected thoughts about this:

You need:
- list of outputs; ideally this should be completely dynamic
  (i.e., possibly different at `make' time than at `automake' time;
  but nonempty at `make' time),
- list of inputs; ideally this should also be pretty dynamic,
- stamp file name,
- lock file name, and a temp file name.  I guess both could be derived
  from the stamp file name,
- the actual command to create the files.

Let's try for the last example in the chapter of the manual:

Given this:

ELFILES = one.el two.el three.el ...
ELCFILES = $(ELFILES:=c)

You would supply this in Makefile.am (the command is just to make things
easy to look at in the example; but yes, in the general case I would
assume that):

MULTITARGETS = elc-stamp
elc_stamp_TARGETS = $(ELCFILES)
elc_stamp_SOURCES = $(ELFILES)
elc_stamp_COMMAND = \
for f in : $(elc_stamp_SOURCES); do \
  test "$$f" != : || continue; \
  touch "$${f}c"; \
done

Automake would then add this code:

elc-stamp: $(elc_stamp_SOURCES)
        @rm -f address@hidden
        @touch address@hidden
        $(elc_stamp_COMMAND)
        @mv -f address@hidden $@

$(elc_stamp_TARGETS): elc-stamp
        @stamp=elc-stamp; \
        if test -f $@; then :; else \
          trap 'rm -rf "$$stamp-lck" "$$stamp"' 1 2 13 15; \
          if mkdir "$$stamp-lck" 2>/dev/null; then \
            rm -f "$$stamp"; \
            $(MAKE) $(AM_MAKEFLAGS) "$$stamp"; \
            rmdir "$$stamp-lck"; \
          else \
            while test -d "$$stamp-lck"; do sleep 1; done; \
            test -f "$$stamp"; exit $$?; \
          fi; \
        fi

and also clean the stamp, lock, and temp file appropriately.
I suppose the usual prefixing with dist_, nodist_ could apply
(possibly to both sources and targets, in which case the stamp
would have to be distributed as well).  Not sure which *clean
should clean which files, or how to make that choosable by the
user.

_TARGETS would contain no notion of installability, so I guess
you would need to add them to some foo_DATA as well.

For *_TARGETS with directory components, I guess automake should
add dependencies on the respective .dirstamp files.

Dunno if _TARGETS should better be named differently.
The name sounds a bit too generic; likewise for `MULTITARGETS'.
(Maybe that variable should allow for prefixing?)

This notion deviates a bit from the current Automake convention,
and it unfortunately puts a stress on what should be an implementation
detail: the stamp file name.  And for one new feature, it adds quite a
bit of namespace.


FWIW, if I were you, for now I'd just define this
(and automake would probably have an am__make_many_locked like this):

make_many_locked = \
if test -f $@; then :; else \
  trap 'rm -rf "$$stamp-lck" "$$stamp"' 1 2 13 15; \
  if mkdir "$$stamp-lck" 2>/dev/null; then \
    rm -f "$$stamp"; \
    $(MAKE) $(AM_MAKEFLAGS) "$$stamp"; \
    rmdir "$$stamp-lck"; \
  else \
    while test -d "$$stamp-lck"; do sleep 1; done; \
    test -f "$$stamp"; exit $$?; \
  fi; \
fi

and use it like this:

elc_stamp_command = \
for f in : $(ELFILES); do \
  test "$$f" != : || continue; \
  touch "$${f}c"; \
done
elc-stamp: $(ELFILES)
        @rm -f address@hidden
        @touch address@hidden
        $(elc_stamp_command)
        @mv -f address@hidden $@

$(ELCFILES): elc-stamp
        @stamp=elc-stamp; $(make_many_locked)

# This is for the case where every user should build the files:
EXTRA_DIST += $(ELFILES)
CLEANFILES += elc-stamp elc-stamp-lck elc-stamp-t $(ELCFILES)

# This is for the case where the files are distributed by default:
EXTRA_DIST += $(ELFILES) elc-stamp $(ELCFILES)
CLEANFILES += elc-stamp-lck elc-stamp-t
MAINTAINERCLEANFILES += elc-stamp $(ELCFILES)


I would appreciate if you could test on your specific use case whether
the above is general enough to be useful, and whether the resulting
Makefile.am code looks too ugly for words or not.  Also, I haven't
thought at all about other interaction with current automake.

And no, I haven't decided whether I volunteer.

Cheers,
Ralf