Libtool Roadmap

[Alex Ameen]

Howdy, a few weeks ago I talked about sending out a road-map.
I've gotten that prepared today. This roadmap is informal, opinionated, and I encourage feedback and discussion on it. I have my own biases and opinions about things I like/dislike and want added based on my `libtool' usage over the years. However, I would be doing a disservice to users if I forced them to adopt "my way of using `libtool'" over their own. With that in mind it's important for y'all to share your own use cases and pain points so that I can address them. This might be more accurately called a "giant RFC dump" than a roadmap, because it's hard for me to imagine that this remains unchanged.

The only notable item I have not included here is some organizational changes concerning how bugs/patches are submitted ( currently there's 3 ways to submit bug reports and 2 of them are miserable to process ); but I'll address that in another thread.

Also note that in my discussion of "CI" ( Continuous Integration ) setup, I intend to provide sufficient abstraction that with some tailoring automated `libtool' testing can be accessible to other developers. This will not be a "core" feature of `libtool', it is rather a practical necessity for maintenance. I am not and will not be making commitments on behalf of the `libtool' team to gate releases based on regression for any particular combination of platforms or tools. Rather you can look at this as "I want to improve the existing Hydra infrastructure, make it more accessible, and find ways to support platforms which cannot natively run Nix or Hydra", this may mean creating more abstract job declarations in a format with can be processed by a wider range of CI tools, it may mean VMs will be used for some testing, it may mean that Hydra is dropped altogether ( unlikely ). Time will tell.

These tasks are what I have in mind for the immediate future. Long term I'd like to work more closely with the other Autotools teams to improve integration with the rest of the family, and make `libtool' more extensible, but those are conversations for another day.

---

                           ━━━━━━━━━━━━━━━━━
                            LIBTOOL-ROADMAP

                               Alex Ameen
                           ━━━━━━━━━━━━━━━━━


Table of Contents
─────────────────

1. Roadmap
.. 1. Improve `.la' libraries, or phase them out on certain platforms.
..... 1. Libtool should not use `libfoo.la' when `libfoo.EXT' are requested.
..... 2. Make installation of `.lt' libraries optional.
..... 3. Prevent over-linking of ELF binaries.
..... 4. Security
.. 2. Support `$ORIGIN' in ELF binaries’ `RPATH' and `RUNPATH' entries.
.. 3. Avoid use of wrapper scripts for `noinst_' and `nodist_' binaries.
.. 4. Refactor argument parsing, particularly parsing linker flags.
.. 5. Migrate platform, arch, and tool specific conditionals to `autoconf'.
.. 6. Expand usage of `M4SH' for creating `ltmain.sh' and `libtoolize.in'.
.. 7. Simplify `bootstrap' process.
.. 8. Make the creation of test cases more user friendly
.. 9. Create CI pipeline resources for distribution and tool maintainers
.. 10. Permit installable LTLIBRARIES to conditionally be “convenience libraries”





1 Roadmap
═════════

1.1 Improve `.la' libraries, or phase them out on certain platforms.
────────────────────────────────────────────────────────────────────

1.1.1 Libtool should not use `libfoo.la' when `libfoo.EXT' are requested.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  What I mean is, if I say `-lfoo', of `-l:libfoo.la', then go right
  ahead, do your thing `libtool'. However, when I went out of my way to
  say `-l:libfoo.so' or `/trust/the/user/libfoo.a', it is evil for
  `libtool' to silently decide that it knows best and decide to use
  `libfoo.la' regardless.
  • This behavior leads a large number of package managers and
    distributions to delete `.la' files before distributing binary
    tarballs.
  • This frequently leads to incorrect/unexpected libraries and flags
    being used because they were written as dependencies or inherited
    flags in a `.la' file.


1.1.2 Make installation of `.lt' libraries optional.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • Closing Argument:
    ┌────
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    │     libtool: link: warning: library `/tmp/usr/lib/libstdc++.la' was moved.
    └────
    Should people learn to use `DESTDIR'? Sure, but it’s hard enough to
    defend the usefulness of `.la' when it has a meltdown like this over
    binaries which were safely moved.


1.1.3 Prevent over-linking of ELF binaries.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  Unless a popular patch for `ltmain.sh' was applied to a project,
  `libtool' will over-link unneeded dependencies in ELF binaries. This
  often leads to unexpected/incorrect library load/initialization
  ordering, particularly in cases where libraries have circular
  dependencies.
  • “Libraries shouldn’t have circular dependencies” is not a valid
    excuse.
  • This has a measurable impact on performance in large applications,
    and makes interposing symbols nearly impossible.
  • This behavior severely aggravates C++ initialization ordering
    issues.
  • Makes the use of ELF filtering extensions impractical.


1.1.4 Security
╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  The ability to pass flags to consumers is useful, but this behavior
  requires more visibility, and serious scrutiny concerning security.
  • Expecting users to inspect `.la' files recursively, or always build
    dry runs to audit flags is utterly fanciful.


1.2 Support `$ORIGIN' in ELF binaries’ `RPATH' and `RUNPATH' entries.
─────────────────────────────────────────────────────────────────────


1.3 Avoid use of wrapper scripts for `noinst_' and `nodist_' binaries.
──────────────────────────────────────────────────────────────────────

  “When possible” of course. While these scripts mean well, they are
  difficult to circumvent when a use case requires direct execution of a
  binary. Further, the availability of `$ORIGIN' on ELF platforms makes
  these wrappers unnecessary or counter-intuitive.


1.4 Refactor argument parsing, particularly parsing linker flags.
─────────────────────────────────────────────────────────────────

  • It’s incomprehensible, and inline comments indicate that for several
    years people have been unable to figure out why certain quirks like
    “libraries are in reverse order in this block for some reason”
    exist.
  • When performing any changes to the parser test cases should be
    created to ensure that behavior remains consistent ( unless a latent
    bug is fixed ).


1.5 Migrate platform, arch, and tool specific conditionals to `autoconf'.
─────────────────────────────────────────────────────────────────────────

  For example, if I configure `./configure --host_os='$(build_os)'', or
  if a compiler/linker is known at configure time, much of the runtime
  probing which is currently performed is redundant.
  • Obviously, for a global `libtool' install on a system, this
    optimization would not be performed. Rather this aims to optimize
    `ltmain.sh --> libtool' creation performed in a project tree
    alongside `autoconf'.


1.6 Expand usage of `M4SH' for creating `ltmain.sh' and `libtoolize.in'.
────────────────────────────────────────────────────────────────────────

  In practice I aim to split these scripts into smaller parts which are
  easier to read and reason about in isolation.
  • More modular snippets may be easier to unit test.
  • Use of `M4SH' may also permit users to patch/extend `libtool' more
    easily.
  • This will allow us to phase out `funclib.sh' over time, and align
    idiomatically with the other `autotools'.


1.7 Simplify `bootstrap' process.
─────────────────────────────────

  While the bootstrapping script is really more geared for maintainers
  and contributors working in the repository, as opposed to the source
  tarballs which users download, the existing process’ complexity and
  dependence on external repositories complicates efforts to build
  `libtool' using CI.
  • Use of `git' and reliance on a network connection to vendor external
    repositories during the bootstrap phase complicates conventional
    hashing approaches used to cache CI artifacts.
  • Requiring a network connection for bootstrapping raises security
    concerns which would make automatic testing of patch submissions
    unsafe.
  • Ideally bootstrapping `libtool' should be possible using a minimal
    tool-chain. Likely `coreutils' ( or equivalent ), `awk', `sed',
    `grep', `tar', `cc', `binutils' ( or equivalent ), `make',
    `autoconf', `automake', `m4', `findutils', `tar', and `xz'.
    Dependence on `texinfo', `help2man', etc could be optional.
  • Parts of `gnulib' which we require should be vendored. The existing
    bootstrap process checks out a specific commit of `gnulib' and
    `gl/bootstrap', so vendoring this code is functionally equivalent.


1.8 Make the creation of test cases more user friendly
──────────────────────────────────────────────────────

  While the existing documentation recommends that folks submitting
  issues create a test case which reproduces their problem, the
  complexity of the test suite makes that impractical.
  • The existing test suite is robust, and I have no intention of
    replacing it, however creating an avenue for contributors/users to
    draft simple `sh' scripts or minimal `autotools' projects for
    reproducing issues should be as user friendly as possible.
  • If creating test cases is not convenient, contributors and users
    simply won’t do it, and the burden of authoring these tests will
    fall to the `libtool' team. Often `libtool' team members do not
    readily have access to the effected platform, or lack experience
    with a given platform and tool-set’s quirks. A practical and
    predictable, consequence is that certain platforms and tool-chains
    lack robust testing.


1.9 Create CI pipeline resources for distribution and tool maintainers
──────────────────────────────────────────────────────────────────────

  This would likely be placed in a branch or separate repository. The
  goal here is that a declarative CI pipeline for testing `libtool' be
  drafted such that members of the `libtool' team, or external projects
  which want to integrate with `libtool' may self-host a set of CI jobs
  used to build and test `libtool' with parameterized inputs.
  • For example, if I am a `coreutils' contributor, it may be useful to
    drop in an alpha build of `coreutils' to see if it blows up
    `libtool'.
  • Similarly, if I am a `libtool' contributor, it may be useful to
    automatically test a matrix of `autoconf' and `automake' versions to
    see if we have broken compatibility with a particular release of
    those tools.
  • Currently we often don’t hear about changes which broke a niche
    combination of platform, arch, and tool-set combinations until after
    a release is out, and people submit issues. While it’s not possible
    to expand an infinite matrix of combinations, real time feedback for
    systems other than a team member’s daily driver could catch a large
    number of issues before they are released.


1.10 Permit installable LTLIBRARIES to conditionally be “convenience libraries”
───────────────────────────────────────────────────────────────────────────────

  This aims to support projects which produce different sets of
  libraries when a static vs. shared library build is being run. Largely
  this effects the handling of internal utility libraries with hidden
  symbol visibility.
  • For clarity “convenience” libraries are simply `PIC' `libfoo.a'
    libraries, or equivalent for a given platform.
  • This is particularly applicable to `noinst_PROGRAMS' used at build
    time which depend on installable libraries. A `noinst_PROGRAMS'
    executable would likely prefer linking with a static “convenience”
    library rather than a shared library, since they generally do not
    require a wrapper script.
  • With a plain `Makefile' it is trivial to produce either a
    `libfoo.so', static `libfoo.a', and `PIC' `libfoo.a' during a single
    run. `libtool'’s concept of “convenience” libraries make it
    difficult to replicate this workflow for arbitrary reasons.
  • Ex: Project wants to produce `libfoo.la' and `libbar.la' to make
    their public interfaces available to users. Both `libfoo.la' and
    `libbar.la' depend on `libquux.la', which does not contain “user
    facing” public interfaces. Project may wish to install `libfoo.so'
    and `libbar.so' which link `PIC' `libquux.a' with `hidden' symbol
    visibility when `-shared' is in effect, but may wish to install
    `libfoo.a', `libbar.a', and `libquux.a' when `-static' is in effect.
  • Ex: A project may wish to conditionally produce a `libquux.so', or
    roll `libquux.a' into `libfoo.so' such that `libbar.so' may satisfy
    its undefined references in for `libquux.la' through `libfoo.so'.
  • Ex: When performing unit tests linking `PIC' `libfoo.a' rather than
    `libfoo.so' into a test executable `mytest' allows `mytest' run
    without a wrapper, so that it may be moved to and run in arbitrary
    directories. Frankly if I’m authoring tests, the `libtool' wrapper
    over executables may be more of a headache than a convenience.
  • As niche as these use-cases may sound, there are variety of reasons
    for a project to produce any of `libfoo.so', static `libfoo.a', and
    `PIC' `libfoo.a' during a single build/install.
  • To accomplish this in existing releases of `libtool', project
    authors would be required to define `libquux.la' “twice”, once as a
    “convenience” library, and again as an installable `LTLIBRARY' with
    identical sources and flags. It might also be necessary for users to
    configure and/or build twice to get the desired outputs.
    My preference is “DRY”.