Re: Failing to use gnulib bootstrap in libtool

[Gary V. Vaughan]

On 3 Sep 2010, at 17:03, Bruno Haible wrote:
> Hi Gary,

Hallo Bruno,

Thanks for your comments.

> Gary V. Vaughan wrote:
>> Is gnulib bootstrap designed for reuse in other projects?
>> 
>> I'm finding it extremely difficult to understand a lot of the code, ...
>> ... a lot of questions about the design choices, below.  In short, it could
>> use a complete rewrite.
> 
> It is well possible to use gnulib without using 'bootstrap'.

Oh, absolutely.  But, rather than develop our own monolithic bootstrap script
I'd like to see if we can help get gnulib's bootstrap script into good enough
shape that Libtool can use it too - and in the process maybe make it a viable
option for M4 and others.

> As examples,
> please see the 'autogen.sh' scripts of some packages:
>  - libunistring
>    
> <http://git.savannah.gnu.org/gitweb/?p=libunistring.git;a=blob;f=autogen.sh>
>  - gettext
>    <http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=autogen.sh>
>  - libiconv
>    <http://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=blob;f=autogen.sh>

And not forgetting GNU M4!

I'd be more interested in a list of packages that *do* use the existing
script so that I can test the patches I am queuing up prior to submission.

> With 'bootstrap', the per-project customization is limited to a single file,
> data-driven. Without 'bootstrap', you have more code to write but have 
> complete
> freedom regarding the number of gnulib-tool invocations, how you invoke the
> autotools (autoreconf or aclocal+autoconf+automake), which version control
> system you use, whether you check tools versions or not, etc.
> 
> 'bootstrap' was designed in the first place to maximize code reuse between
> 'coreutils', 'bison', and 'tar'.
> 
> I think its biggest design mistake is that it is not extensible enough by
> code, because for most of the tasks it allows only customization through
> variables, not through shell functions. For example, the handling of the
> 'runtime-po' directory is specific to 'bison' - no other package has this -,
> but it is put into the common 'bootstrap', rather than in the bison
> specific parameterization.

Interesting.  I don't have paperwork on file for bison, so I can't offer
patches directly, but I'll certainly test whatever changes I make to ensure
that it's easy to fix that mistake.

The architecture of my current rewrite of gnulib bootstrap, moves all
the top-level code into functions, and then hooks them all together
with a require mechanism.  Then `bootstrap.conf' is sourced, where
the nodes of that "shell-function-require-tree" can be overwritten
with replacement functions to radically alter the behaviour of the
script if necessary... and finally, the top-level function is called
which runs the require tree in whatever state `bootstrap.conf' left it.
Other than the interface between `bootstrap' and `bootstrap.conf'
massively widening in scope, the semantics are unchanged... more about
this in a moment.


>> 2. source_base, m4_base etc
>> ===========================
>> 
>> Why hardcode another copy of these locations into bootstrap, where they can
>> fall out of sync.  Far better to either extract them from or inject them
>> into `gnulib-cache.m4'.  Extracting them with `autom4te --language=Autoconf 
>> --trace=gl_SOURCE_BASE --trace=gl_M4_BASE
>> ... configure.ac' seems like the most robust way of tackling this problem
>> to me.
> 
> The reason is to avoid mixing inputs and outputs, whereas your proposal
> amounts to introducing circular dependencies.

I don't think you can apply that argument to a cache file, which by its nature
is both an input and an output.  There's no reason to worry about circular
dependencies in this case as long as the process still works in the first
instance before there is an existing cache of values calculated in a previous
run.

> It is *very* important to design things so that every file is either input
> (maintained by the developers, manually) or output (generated by some tools),
> but not both. If you don't follow this principle,
>  - The file will be messed up and require manual correction when the
>    developer invoked the tools with wrong parametrizations, or when some
>    unintended failure happened.

Reading a cache file with sensible fall-back options does not have this
problem, and if some other tool messes up the cache, then it can easily
be deleted at the penalty of having to recalculate its contents again from
first principles.  Really, it's just a cache, and a way to short circuit
repeated identical calculations on consecutive runs.

>  - Multiple developers working on one project will get confused about
>    changes that one developer made (unintentionally) and another developer
>    considers wrong.

The project will need a policy to avoid this.  core-utils, gettext and others
have a policy of not checking in gnulib-cache.m4, and re-running the entire
gnulib --import from scratch on every invocation of bootstrap.  M4 and Libtool
(when the work is complete) have a policy of checking in gnulib-cache.m4 so
that developers can enjoy the enormous speed and convenience advantage of
running gnulib --update after the initial import.

I don't think the developers on any of those projects find it confusing, as
long as the policy is clear enough.

>  - If the file is committed in a version control system, the developers will
>    suffer from occasional conflicts. If it is not committed in the version
>    control system, the contents of the version control system is incomplete.

You mean gnulib-cache.m4?  I don't see a problem with committing it, or not-
committing it as long as the developers on each project reach a consensus.

>  - You will not be able to evolve the format of that file easily.

Easily averted by recalculating the values that you wanted to extract from the
cache-file, but which weren't available from the cache (because the cache is
missing, or has changed formats).  In the mean time, all the developers save
a ton of CPU cycles on repeated bootstrap invocations for projects which
choose a policy of checking in the cache.

> The contents of gnulib-cache.m4 is an output of the invocation of gnulib-tool,
> and therefore a derivative of the parameters that you pass to gnulib-tool.

Agreed, and I have no intention of enforcing the M4/Libtool preferred policy
on other gnulib bootstrap users - I'd just like the flexibility to use the
majority of the gnulib bootstrap as is, while overriding the parts that
conflict with the existing project policy.


>> 3. MSGID_BUGS_ADDRESS
>> =====================
>> 
>>> address@hidden
>> 
>> Wouldn't it be better to extract this from AC_INIT
> 
> Indeed it is a good idea to make this variable default to the third argument
> of AC_INIT. But some packages don't use AC_INIT in this way.

I think that's fine.  Either extract it from AC_INIT if available, or let the
project set it directly in `bootstrap.conf' if they'd prefer (hopefully not
putting it in both places at once - D.R.Y.)... or, as a last resort, assume
a gnu package with a bug-$package mailing list address.

>> 4. bootstrap.conf
>> =================
>> ...
>> I don't like the way gnulib `bootstrap' forces a `gnulib-tool --import'
>> invocation on every run, when a `gnulib-tool --update' is quite sufficient
>> most of the time.
> 
> 'gnulib-tool --update' is equivalent to 'gnulib-tool --import ...' where the
> options and list of modules are fetched from gnulib-cache.m4. Because of the
> principle to not mix inputs and outputs (see above), and because
> gnulib-cache.m4 is an output (produced by the previous gnulib-tool execution),
> you must ensure that gnulib-cache.m4 is not used by subsequent gnulib-tool
> invocations. This means
>  1. You must not use 'gnulib-tool --update'.

Huh?  That makes no sense to me!

>  2. You even have to remove gnulib-cache.m4 before running
>     'gnulib-tool --import ...'. Because when gnulib-cache.m4 is present,
>     'gnulib-tool --import ...' will add the specified modules but not throw
>     away modules that are listed in gnulib-cache.m4 and not listed on the
>     command lines. In other words, if you don't remove gnulib-cache.m4,
>     you are never able to de-import modules.

That is definitely a wrinkle.  But, actually, you can save the full --import
line listed in gnulib-cache.m4 before deleting it, and then run the import
by hand one time to redo the whole thing.  Actually, with the current tools,
you only need to remove the modules you want to de-import from the list in
gnulib-cache.m4 and re-run --update, although I admit that is pretty messy.

>> 6. AC_CONFIG_AUX_DIR detection
>> ==============================
>> 
>> Why another bunch of forks?  Might as well get this data while running sed 
>> over
>> configure the first time, by adding the following to 
>> `extract_package_values':
>> 
>>    /AC_CONFIG_AUX_DIR(/         {
>>        s|^.*AC_CONFIG_AUX_DIR([['"$tab"' ]*\([^])]*\).*$|config_aux_dir=\1|
>>        p
>>    }
> 
> Yes, this does make sense. configure.ac is an input, bootstrap.conf is an 
> input;
> they do not need to both specify the same thing.

Better yet, I've written a shell function to extract these values from 
configure.ac
using autom4te traces, which is much less fragile (not subject to use of 
unexpanded
macros in the arguments etc.).


>> 7. buildreqs
>> ============
>> 
>> Why do we have to keep 2 copies of this information in sync?  `bootstrap' can
>> easily extract the prequisite versions of Autoconf, Automake, Libtool and
>> Gettext from configure.ac for us.
> 
> Note that this information is different from the build requirements of a
> package (that is, the requirements for someone who uses a tarball). That
> information IMO belongs in human-readable file, and that the INSTALL file
> should link to it.
>  <http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=INSTALL>
>  <http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=DEPENDENCIES>
> 
> Here it's about the prerequisites of running 'bootstrap'. Logically, this
> belongs in comments at the top of 'bootstrap', or at the top of
> 'bootstrap.conf', or in the output of 'bootstrap --help'. Or in the
> README-hacking file.

Ah, yes.  That makes sense.

I think that in the case of autoconf, automake, libtool and gettext the version
numbers stored in the configure input files are the same.  Libtool already makes
a distinction between the minimum versions of autoconf and automake required to
build libtool from a git checkout (the values listed in configure.ac) and the
lower minimum versions of the same required to build from a tarball, or use the
installed libtool macros in your own projects (these numbers are listed in
libtool.m4).

>> It would be nice if gnulib also had a template for README-prereq with 
>> instructions
>> for Autotools already entered.
> 
> Yet another file? Why is coreutils/README-prereq not part of README-hacking?
> Both apply to "people working on the checked-out sources".

Agreed.  Better to no have the bootstrap script point at that file at all in 
that
case.

>> 8. gnulib submodule
>> ===================
>> ...
>> And why does GNULIB_SRCDIR contain the location of the referenced gnulib tree
>> for the first half of the script, and then get reused to hold the location of
>> the submodule itself for the rest of the script?  `$gnulib_path' seems to
>> hold the right value already, so there's no need to change the meaning of
>> `GNULIB_SRCDIR' partway down.
> 
> There are more problems in this area; see
>  <http://lists.gnu.org/archive/html/bug-gnulib/2010-03/msg00135.html>

Interesting thanks.  I like this patch too, it's a pity the conversation stalled
at that point.

>> 9. i18n
>> =======
>> 
>> Libtool doesn't have this.  And my brain hurts enough from trying to make
>> sense of the rest of this file, so I haven't looked at any of the po files
>> code.  Although I think someone who understands the issues should, since it
>> probably needs some love too.
> 
> You can put at low priority on your list, because work is underway to have
> PO files handled by Automake - at which point the 'bootstrap' script will
> likely have nothing to do about them.

Phew.  One less thing to worry about :)  Thanks.

>> 10. slurp()
>> ==========
>> 
>> I can't figure out what this function's purpose is.  It seems to care about 
>> the
>> `gnulib.mk' file, and probably has something to do with making lists of files
>> that gettext doesn't like and editing away the discrepancies when copying
>> imported files out of the temporary tree into their correct location?  If 
>> someone
>> could explain what it's for (I can follow the code line-by-line, but out of 
>> context
>> it still makes no sense), or commit some comments that would be a HUGE help.
>> ...
>> I really don't understand why gnulib is installed to a temporary directory 
>> and
>> bits of it copied across with name mangling to compensate.  Is this to 
>> work-around
>> a timestamp issue or something?
> 
> I think it has/had two purposes:
>  1. It allowed Paul to use symbolic links or hard links to gnulib files,
>     at a time when gnulib-tool did not have options for symbolic links.
>  2. It prevents some files from being edited. Again, this dated from before
>     it become gnulib-tool standard to mark generated files with "DO NOT EDIT!"

So, with a little work in existing gnulib bootstrap using projects, slurp can 
be removed?


>> 11. gnulib_tool_options
>> =======================
>> 
>> This is broken quite badly:
>> 
>>  `--import': Most of the time I'd rather use `--update'
> 
> That's a bad idea, see above.

I'm not at all convinced that the added convenience and speed is at all 
outweighed by
any of the issues you explained earlier.  But, as I said, I'm not asking to 
change
the default semantics here, only to have the flexibilty to override them without
giving up on using gnulib bootstrap altogether.

>>      but there's no way 
>>      to override it without using a gnulib-local patch (and I couldn't even
>>      get that to work for `bootstrap' since it's not even in a module).
>> 14. Updating
>> ============
>> 
>> There's no obvious way for bootstrap to update itself.
> 
> Since 'bootstrap' is not in a gnulib module, you need the command
> 'gnulib-tool --copy-file build-aux/bootstrap ...' in order to get a copy of
> it. This command _does_ consider modifications made in a gnulib-local/ 
> directory.
> So yes you can have your local diffs stored in a
> gnulib-local/build-aux/bootstrap.diff file.

Ah interesting!  I wasn't aware of that.

> But as I said above, it would be
> better if 'bootstrap' itself would provide more customization hooks.

Agreed.

>> 12. AUTOHEADER and lack of AUTORECONF
>> =====================================
>> ...
>> And for that matter, why are libtoolize, aclocal and others called directly
>> at all?
> 
> Recall that the bootstrap script dates back to 2006, before libtool-2.2.2
> was released. And recall that in libtool versions 1.x the 'libtoolize'
> script was completely unreliable because it copied only the ltmain.sh file
> while letting you happily use the libtool.m4 file installed in
> /usr/share/aclocal that another version of libtool had installed.

Makes sense.  No reason not to update it now though.

>> Autoreconf does a better job here too.  I suggest replacing this entire block
>> with:
>> 
>>    ${AUTORECONF-autoreconf} $autoreconf_flags
> 
> You also have to define AUTOPOINT=: for the duration of this command, because
> the autopoint program installed by the user may be of an older version, not
> compatible with the .m4 files that are in gnulib.

Okay, will do.

Cheers,
-- 
Gary V. Vaughan (address@hidden)

PGP.sig
Description: This is a digitally signed message part