Re: access to m4 macros when configuring in subdirectories

[Ralf Wildenhues]

Hi David,

* David Byron wrote on Tue, May 09, 2006 at 06:45:22PM CEST:
> On Monday, May 08, 2006 Ralf Wildenhues wrote: 
> 
> <quoted out of order>
> 
> > If you're going to merge the Makefile.am's, or include one
> > into the other, be sure to do the configure.ac merge first.
> 
> Your post just got this to sink in for me.  I was trying to keep the
> configure.ac's separate since it felt better to have the autoconf logic
> together with the modules that needed it instead of in one massive
> configure.ac file.  Is there a way to include configure.ac fragments?

Yes, most certainly.  Many configure snippets come in the form of
AC_DEFUN'ed macros.  If your code isn't in this form, you could rewrite
it to be that way (but that can be a very tedious task).  Name the files
foobar.m4, put them in a directory m4 parallel to where they logically
belong.  You can put several macros in one file, or in separate files; I
suggest grouping macros in a file that will be needed together anyway.

Learn about how modern aclocal finds those macros.  For each directory,
you're going to need macros from, tell aclocal about it:
  aclocal -I m4 -I subpackage_1/m4 ...

Put the options in the ACLOCAL_AMFLAGS line in the Makefile.am next to
the configure.ac where it will be needed.

> > Maybe you can show more of your setup, so we can understand
> > package specific constraints.
> 
> I'm starting with a system of homegrown makefiles.  Lots of directories,
> lots of recursive make invocations.  As well, there's a central
> directory where include files get "installed" to.  That's the directory
> in the include path.
> 
> When I add a new directory, I use automake/autoconf.  Each
> automake/autoconf directory is completely independent at the moment.
> And, they follow the same convention of #including header files from the
> central location.  To pull this off I pass --prefix to configure and run
> a make install.  This breaks the dependency information and more stuff
> gets rebuilt than is necessary.
> 
> I can resolve this by adding the paths where the header files live
> before installing them (and taking out the install operation).

Yes, to me this seems a reasonable thing to do.

>  I figured I would give a shot to the full-on non-recursive make at
>  the same time.

You can also go partially non-recursive.  For example, (quoting out of
order as well):

> top/app_1
> top/app_2
> top/related_libs_1
> top/related_libs_1/lib_1_a
> top/related_libs_1/lib_1_b
> top/related_libs_1/lib_1_c
> top/related_libs_2
> top/related_libs_2/lib_2_a
> top/related_libs_2/lib_2_b
> top/related_libs_2/lib_2_c

Depending on how independent (and to some extent also, how large) this
stuff is, I'd go for either
- merge everything into one package (i.e., one configure.ac); or
- two packages: either all *_1 stuff in one, and all *_2 stuff (if that
  is possible and libs and apps are not intertwined); or all apps in
  one, and libs separately; or
- aim at 4 packages: app_1, app_2, *libs_1, *libs_2.

(My notation is: a package is something that has one configure.ac.
A package tree has several configure.ac, possibly connected by
AC_CONFIG_SUBDIRS().)


Inside one package, I'd start going more and more non-recursive if
possible.

But independent of all of that: you can refer to uninstalled headers and
libraries of subpackages in a package tree.  Just make sure the build
order is correct, and add appropriate include paths
  -I$(top_srcdir)/related_libs_1/include

or so to AM_CPPFLAGS (for generated headers you may need build dir
references).  With libraries, relative link paths
  -L../related_libs_1/src

are not to be preferred over
  ../related_libs_1/src/libfoo1.a

because the latter enables Automake to emit dependency information upon
that library.

> > the configure.ac together with their AC_CONFIG_SUBDIRS()
> > links form an acyclic directed graph (tree); each
> > Makefile.am that stands on its own (i.e., is not included by
> > another Makefile.am), should be listed (with name Makefile)
> > in AC_CONFIG_FILES of exactly one configure.ac, otherwise
> > havoc will ensue.
> 
> Makes sense.  If I'm including Makefile.am fragments, I need to make
> sure there's only one AC_CONFIG_FILES([Makefile]).

Yes.

> No shared libraries.  Everything statically linked.  When we release
> something, it's either app_1 or app_2.
> 
> Let's say app_1 links with lib_1_a, lib_2_a, and lib_2_b.  app_2 links
> with lib_1_a and lib_2_c.
> 
> When people work, they typically either work on app_1 or app_2.  Either
> way they get all of related_libs_1 and related_libs_2.
> 
> If I treat all the lib_*_* as totally independent, everything pretty
> much works.  I configure, make, make install in each directory.  But,
> too much stuff gets rebuilt because everything depends on files in the
> install destination and those timestamps change because the install
> recipe makes a new copy every time.

Yes.  Fixing that point only should be your first goal.  And it would
probably be best to fix that while not working on the other ideas: it'll
help you keep the tree working easier.  (For the other stuff, unless you
already know what I'm having in mind, some serious manual reading is
very helpful.)

> As I said above I think I can fix that, but the other benefits of one
> big Makefile are tempting.  Ideally I'd like one big one for app_1 and
> one for app_2.  Note that there's nothing in the top directory at the
> moment...No makefile, Makefile.am, configure.ac...nothing.  It's a
> consequence of the way we're using revision control.  I can't put
> anything there.  (Each of app_1, app_2, related_libs_1, and
> related_libs_2 are clearcase VOBs.  The top level directory isn't in a
> VOB so nothing can go there.)

I can't help you much here, I don't know clearcase.

> I bet as I add more autotools-aware libraries that I'll be duplicating a
> bunch of configure tests.  It would be nice to avoid that as well.

Not sure which kind of duplication you are speaking about here.  There
exist several kinds:
- don't write the same configure.ac tests several times.  As above, try
  to AC_DEFUN macros, and use those macros instead.  (Be sure to use a
  specific name prefix, not "AC_", for your macros; "db_" would be good,
  or something connected with those libraries.)
- as above have aclocal emit aclocal.m4 files which just m4_include the
  macro files you wrote.
- write tests that use cache variables, so that when you share a cache
  with a sub-configure script, the test won't be executed again
  (there are traps here: you also need to make sure the test result
  isn't dependent on the way it is used in that specific configure.ac,
  say, on previous tests)

>  I guess I can share cache files...Not exactly sure what my options
>  are here, especially with the top of the directory tree forbidden.

Are only checked-in files forbidden in the top tree?  What about files
generated by the autotools?  What about files generated at or after
configure time (i.e., in the build tree)?

>  If I
> have to move everything around, I'll consider it, but I'm not sure what
> a good solution is.  I can make a giant VOB with everything in it so the
> top level becomes a valid place to put files.  My reflex is to have
> app_1 and app_2 not know anything about each other (i.e. have totally
> separate Makefile.am's, configure.ac's), but I bet the normal way to
> pull this off is to have one configure.ac and one Makefile.am for both
> and define two targets in Makefile.am.  Is that right?

That really depends on how connected these apps are.

Maybe some pointers to example applications are in order.  On the one
side, you can look at GraphicsMagick, which has consolidated its build
into one huge Makefile.am (with included snippets); on the other extreme
(currently) is the Xorg tree, which has lots of separately built
packages (each having a configure.ac, and 3-5 Makefile.am's, for no
special reason).  And you could look at OpenMPI, which has around 200
Makefile.am files, with some non-recursiveness in it (consolidated from
something like 300 or more Makefile.am files).  I'm sure there are more
good examples, those are just ones I happen to know.

Cheers,
Ralf