Re: Failing to use gnulib bootstrap in libtool

[Bruno Haible]

Hi Gary,

Gary V. Vaughan wrote:
> Is gnulib bootstrap designed for reuse in other projects?
> 
> I'm finding it extremely difficult to understand a lot of the code, ...
> ... a lot of questions about the design choices, below.  In short, it could
> use a complete rewrite.

It is well possible to use gnulib without using 'bootstrap'. As examples,
please see the 'autogen.sh' scripts of some packages:
  - libunistring
    <http://git.savannah.gnu.org/gitweb/?p=libunistring.git;a=blob;f=autogen.sh>
  - gettext
    <http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=autogen.sh>
  - libiconv
    <http://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=blob;f=autogen.sh>

With 'bootstrap', the per-project customization is limited to a single file,
data-driven. Without 'bootstrap', you have more code to write but have complete
freedom regarding the number of gnulib-tool invocations, how you invoke the
autotools (autoreconf or aclocal+autoconf+automake), which version control
system you use, whether you check tools versions or not, etc.

'bootstrap' was designed in the first place to maximize code reuse between
'coreutils', 'bison', and 'tar'.

I think its biggest design mistake is that it is not extensible enough by
code, because for most of the tasks it allows only customization through
variables, not through shell functions. For example, the handling of the
'runtime-po' directory is specific to 'bison' - no other package has this -,
but it is put into the common 'bootstrap', rather than in the bison
specific parameterization.

> 2. source_base, m4_base etc
> ===========================
> 
> Why hardcode another copy of these locations into bootstrap, where they can
> fall out of sync.  Far better to either extract them from or inject them
> into `gnulib-cache.m4'.  Extracting them with `autom4te --language=Autoconf 
> --trace=gl_SOURCE_BASE --trace=gl_M4_BASE
> ... configure.ac' seems like the most robust way of tackling this problem
> to me.

The reason is to avoid mixing inputs and outputs, whereas your proposal
amounts to introducing circular dependencies.

It is *very* important to design things so that every file is either input
(maintained by the developers, manually) or output (generated by some tools),
but not both. If you don't follow this principle,
  - The file will be messed up and require manual correction when the
    developer invoked the tools with wrong parametrizations, or when some
    unintended failure happened.
  - Multiple developers working on one project will get confused about
    changes that one developer made (unintentionally) and another developer
    considers wrong.
  - If the file is committed in a version control system, the developers will
    suffer from occasional conflicts. If it is not committed in the version
    control system, the contents of the version control system is incomplete.
  - You will not be able to evolve the format of that file easily.

The contents of gnulib-cache.m4 is an output of the invocation of gnulib-tool,
and therefore a derivative of the parameters that you pass to gnulib-tool.

> 3. MSGID_BUGS_ADDRESS
> =====================
> 
> > address@hidden
> 
> Wouldn't it be better to extract this from AC_INIT

Indeed it is a good idea to make this variable default to the third argument
of AC_INIT. But some packages don't use AC_INIT in this way.

> 4. bootstrap.conf
> =================
> ...
> I don't like the way gnulib `bootstrap' forces a `gnulib-tool --import'
> invocation on every run, when a `gnulib-tool --update' is quite sufficient
> most of the time.

'gnulib-tool --update' is equivalent to 'gnulib-tool --import ...' where the
options and list of modules are fetched from gnulib-cache.m4. Because of the
principle to not mix inputs and outputs (see above), and because
gnulib-cache.m4 is an output (produced by the previous gnulib-tool execution),
you must ensure that gnulib-cache.m4 is not used by subsequent gnulib-tool
invocations. This means
  1. You must not use 'gnulib-tool --update'.
  2. You even have to remove gnulib-cache.m4 before running
     'gnulib-tool --import ...'. Because when gnulib-cache.m4 is present,
     'gnulib-tool --import ...' will add the specified modules but not throw
     away modules that are listed in gnulib-cache.m4 and not listed on the
     command lines. In other words, if you don't remove gnulib-cache.m4,
     you are never able to de-import modules.

> 6. AC_CONFIG_AUX_DIR detection
> ==============================
> 
> Why another bunch of forks?  Might as well get this data while running sed 
> over
> configure the first time, by adding the following to `extract_package_values':
> 
>     /AC_CONFIG_AUX_DIR(/         {
>         s|^.*AC_CONFIG_AUX_DIR([['"$tab"' ]*\([^])]*\).*$|config_aux_dir=\1|
>         p
>     }

Yes, this does make sense. configure.ac is an input, bootstrap.conf is an input;
they do not need to both specify the same thing.

> 7. buildreqs
> ============
> 
> Why do we have to keep 2 copies of this information in sync?  `bootstrap' can
> easily extract the prequisite versions of Autoconf, Automake, Libtool and
> Gettext from configure.ac for us.

Note that this information is different from the build requirements of a
package (that is, the requirements for someone who uses a tarball). That
information IMO belongs in human-readable file, and that the INSTALL file
should link to it.
  <http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=INSTALL>
  <http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=DEPENDENCIES>

Here it's about the prerequisites of running 'bootstrap'. Logically, this
belongs in comments at the top of 'bootstrap', or at the top of
'bootstrap.conf', or in the output of 'bootstrap --help'. Or in the
README-hacking file.

> It would be nice if gnulib also had a template for README-prereq with 
> instructions
> for Autotools already entered.

Yet another file? Why is coreutils/README-prereq not part of README-hacking?
Both apply to "people working on the checked-out sources".

> 8. gnulib submodule
> ===================
> ...
> And why does GNULIB_SRCDIR contain the location of the referenced gnulib tree
> for the first half of the script, and then get reused to hold the location of
> the submodule itself for the rest of the script?  `$gnulib_path' seems to
> hold the right value already, so there's no need to change the meaning of
> `GNULIB_SRCDIR' partway down.

There are more problems in this area; see
  <http://lists.gnu.org/archive/html/bug-gnulib/2010-03/msg00135.html>

> 9. i18n
> =======
> 
> Libtool doesn't have this.  And my brain hurts enough from trying to make
> sense of the rest of this file, so I haven't looked at any of the po files
> code.  Although I think someone who understands the issues should, since it
> probably needs some love too.

You can put at low priority on your list, because work is underway to have
PO files handled by Automake - at which point the 'bootstrap' script will
likely have nothing to do about them.

> 10. slurp()
> ==========
> 
> I can't figure out what this function's purpose is.  It seems to care about 
> the
> `gnulib.mk' file, and probably has something to do with making lists of files
> that gettext doesn't like and editing away the discrepancies when copying
> imported files out of the temporary tree into their correct location?  If 
> someone
> could explain what it's for (I can follow the code line-by-line, but out of 
> context
> it still makes no sense), or commit some comments that would be a HUGE help.
> ...
> I really don't understand why gnulib is installed to a temporary directory and
> bits of it copied across with name mangling to compensate.  Is this to 
> work-around
> a timestamp issue or something?

I think it has/had two purposes:
  1. It allowed Paul to use symbolic links or hard links to gnulib files,
     at a time when gnulib-tool did not have options for symbolic links.
  2. It prevents some files from being edited. Again, this dated from before
     it become gnulib-tool standard to mark generated files with "DO NOT EDIT!".

> 11. gnulib_tool_options
> =======================
> 
> This is broken quite badly:
> 
>   `--import': Most of the time I'd rather use `--update'

That's a bad idea, see above.

>       but there's no way 
>       to override it without using a gnulib-local patch (and I couldn't even
>       get that to work for `bootstrap' since it's not even in a module).
> 14. Updating
> ============
> 
> There's no obvious way for bootstrap to update itself.

Since 'bootstrap' is not in a gnulib module, you need the command
'gnulib-tool --copy-file build-aux/bootstrap ...' in order to get a copy of
it. This command _does_ consider modifications made in a gnulib-local/ 
directory.
So yes you can have your local diffs stored in a
gnulib-local/build-aux/bootstrap.diff file. But as I said above, it would be
better if 'bootstrap' itself would provide more customization hooks.

>       And in any case, these are already stored in gnulib-cache.m4, so we
>       shouldn't have to keep two references in sync.

You're mixing inputs and outputs again. Forget about fetching the --lib value
from gnulib-cache.m4.

> 12. AUTOHEADER and lack of AUTORECONF
> =====================================
> ...
> And for that matter, why are libtoolize, aclocal and others called directly
> at all?

Recall that the bootstrap script dates back to 2006, before libtool-2.2.2
was released. And recall that in libtool versions 1.x the 'libtoolize'
script was completely unreliable because it copied only the ltmain.sh file
while letting you happily use the libtool.m4 file installed in
/usr/share/aclocal that another version of libtool had installed.

> Autoreconf does a better job here too.  I suggest replacing this entire block
> with:
> 
>     ${AUTORECONF-autoreconf} $autoreconf_flags

You also have to define AUTOPOINT=: for the duration of this command, because
the autopoint program installed by the user may be of an older version, not
compatible with the .m4 files that are in gnulib.

Hope this helps.

          Bruno