RE: Automake Digest, Vol 216, Issue 1

[FOURNIER Yvan]

Hello,

Regarding edition of .la files when they are incorrect, yes, this is what I do 
where I have write access (i.e. on my laptop), but this is not the case on 
large systems, and I would not be comfortable asking an admin to change a given 
line for a library at the bottom of a large dependency chain for fear of 
breaking something else.

Your solution of writing a compiler wrapper to adjust options as a workaround 
is a good idea. I have a similar wrapper to launch debuggers, but hadn't though 
of using that approach as a libtool workaround. So I'll keep that option in 
mind.

Still, a --ignore-la-files option or something similar for libtool would avoid 
jumping through such hoops...

Best regards,

  Yvan Fournier

________________________________________
De : Automake <automake-bounces+yvan.fournier=edf.fr@gnu.org> de la part de 
automake-request@gnu.org <automake-request@gnu.org>
Envoyé : lundi 3 mai 2021 18:00
À : automake@gnu.org
Objet : Automake Digest, Vol 216, Issue 1

Send Automake mailing list submissions to
        automake@gnu.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.gnu.org/mailman/listinfo/automake
or, via email, send a message with subject or body 'help' to
        automake-request@gnu.org

You can reach the person managing the list at
        automake-owner@gnu.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Automake digest..."


Today's Topics:

   1. Re: Future plans for Autotools (FOURNIER Yvan)
   2. Re: Future plans for Autotools (Thomas Jahns)


----------------------------------------------------------------------

Message: 1
Date: Sun, 2 May 2021 17:49:33 +0000
From: FOURNIER Yvan <yvan.fournier@edf.fr>
To: "automake@gnu.org" <automake@gnu.org>
Subject: Re: Future plans for Autotools
Message-ID: <1619977772999.94614@edf.fr>
Content-Type: text/plain; charset="iso-8859-1"

Hello,


Sorry for reacting a bit late to the January discussion about Autotool's future.


As a longtime user of the GNU Autotools for a large computational dynamics code 
(code_saturne.org), here are a few hopefully constructive remarks about our 
experience so far.


Sorry if my post is a bit long.


Most-liked aspects

-------------------------


- excellent cross-compilation support


- `configure --help` is very practical (CMake's equivalent is much messier, and 
one of the reasons we did not switch)


- "almost" no dependencies on the target system except for a shell and Make. 
This is especially useful on compute clusters which may have a minimal system.


- pretty good documentation


- it is very easy to find the configure options from a previous run at the top 
of the config.log and config.status files, and then copy/past them so as to 
generate an new build in a clean (empty) build directory. This is IMHO a strong 
point compared to CMake, whose caching aspects can lead to complex issues, and 
where regenerating a configuration command is not so easy.


Some Inconsistencies

-----------------------------


While configure/automale/libtool seem to be designed to work together, there 
are a few inconsistencies :


- autoconf can generate multiple configuration files (for example 
public/private header files), but automake assumes only one is used. I don't 
remeber the details, but had to work around this 15 years ago to re-implement 
the generation of a "light" external config.h in m4 and shell instead of using 
built-n features due to this.


- Though one of the initial advantages of the GNU autotools was that a shell 
and Make were sufficient on the target system, GNOME devs started breking that 
with pkg-config, and now it seems resorting to that has become ubiquitous in 
many autotools-based projects, so the dependency list has increased in 
practice. Tools like autopoint also add complexity, as they are available on 
some systems not on others.


- Our code is a mix of Fortran and C, with a bit of C++. Automake still deos 
not support Fortran 90-type module dependencies, so we have to manage manual 
dependencies in one of our Makefile.am's. More modern systems handle Fortran 
(not quite the latest fad) much better.


- When using Gettext, "make update-po" modifies code in the source tree based 
on a command in the build tree. When using separate source and build trees, 
this is somewhat surprising, so having an equivalent command (not based on the 
generated Makefiles) for the source tree would be more consistent as regards 
source/build separation.


Major issue

---------------


Worse, autoconf does not take advantage of libtool's ".am" files to find 
dependencies, while libtool does not allow bypassing these files. This means 
that:


- we need to manually search for dependencies anyways

- but when it encounters .la files with incorrect .la files, libtool adds the 
incorrect dependencies or options, with no workaround


So while .la files were probably a great idea in the early days of the GNU 
project, they are of limited use now (especially since tools not based on 
autotools do not provide them), and are a source of issues.


I have encountered several examples of this :


- .la files broken by Linux packaging: in the case of an MPI library built with 
various dependencies, some of which are loaded dynamically as plugins, and not 
installed by default with the minimal packages, the .la file lists some 
non-installed (and non-needed) libraries as dependencies.


- incorrect options: recently, trying to use the Cray compiler toolchain on an 
Arm64 machine, some low-level dependency related to back-end gcc libraries 
added a "-pthread" option in the link line, while the top-level compiler used a 
different syntax and did not accept that.

Although I have quite a bit of experience working around libtool's quirlks, I 
was simply not able to build my code using that toolchain on that machine. 
Total failure just before the finish line...


-I have had a similar issue generating shared libraries with with CUDA, though 
in that case I could work around this by duplicating and patching the generated 
libtool file, and adding a Makefile.am rule for .cu files. But I have never 
seen another project optionally using CUDA which bothered doing this rather 
than using another build system (ie. if there is a better solution, I did not 
find it).


Libtool has many nice aspects, such as handling of versioning, rpaths, and 
more, but its

insistence that it is always right, even in situations where it produces 
unusable commands can really be a showstopper.


I am seriously contemplating removing libtool (while keeping autoconf/automake) 
due to the blocking "no option to ignore .la files even when incorrect" issue, 
as I have no other workaround. This will probably require quite a bit of extra 
work, but is better than hitting a wall.


Comparison with other tools

-------------------------------------


Frequently installing dependencies with CMake, and even adding a call to CMake 
to detect configuration options in our Autotools scripts to detect a 
CMake-based library's configuration, I can attest that CMake is not always fun 
to work with either when things go wrong or less than perfect configuration 
files are used. So CMake can also be quite frustrating.


I have not looked at Meson much so far. In the short term, the fact that it 
requires ninja, which is not yet ubiquitous on HPC systems, makes things a bit 
less comfortable in our case, but the fact that is uses Python is an advantage 
(Python is so prevalent in scientific software that just about all users and 
developers in our community need to learn at least some rudiments anyways, 
which is not true of m4, Makefiles, or cmake). So it could become a serious 
contender (whereas the other Python-based contenders such as scons and Waf seem 
to have less momentum).


General impression

---------------------------


while we have a large investment (several thousand lines of local .m4, plus 
Makefile.am) in packaging with the GNU autotools, two opposite options which 
would make us happy would be to :


- either better integrate them, so as to remove the 
inconstencies/interoperability issues between packages


- or make them more modular, and better able to work independently. This is 
probably more of a documentation / example code issue than an actual 
development issue.

As an application software developer and maintainer, I fully appreciate the 
hard and painstaking work that goes into the autotools, and  know how difficult 
it can be to build a working and realistic roadmap and balance development and 
support efforts, and thank to autotools team for keeping these tools in a 
usable state.

Just hoping these comments are useful.

Best regards,

  Yvan Fournier







Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à 
l'intention exclusive des destinataires et les informations qui y figurent sont 
strictement confidentielles. Toute utilisation de ce Message non conforme à sa 
destination, toute diffusion ou toute publication totale ou partielle, est 
interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le 
copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si 
vous avez reçu ce Message par erreur, merci de le supprimer de votre système, 
ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support 
que ce soit. Nous vous remercions également d'en avertir immédiatement 
l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie 
électronique arrivent en temps utile, sont sécurisées ou dénuées de toute 
erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the 
addressees. The information contained in this Message is confidential. Any use 
of information contained in this Message not in accord with its purpose, any 
dissemination or disclosure, either whole or partial, is prohibited except 
formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any 
part of it. If you have received this message in error, please delete it and 
all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or 
virus-free.


------------------------------

Message: 2
Date: Mon, 3 May 2021 11:33:28 +0200
From: Thomas Jahns <jahns@dkrz.de>
To: automake@gnu.org
Subject: Re: Future plans for Autotools
Message-ID: <90fba7d1-9219-46a6-da50-e83b3c2b516a@dkrz.de>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

Hello,

some comments with a similar HPC background:

On 2021-05-02 19:49, FOURNIER Yvan wrote:
[...]
> - it is very easy to find the configure options from a previous run at the 
> top of the config.log and config.status files, and then copy/past them so as 
> to generate an new build in a clean (empty) build directory. This is IMHO a 
> strong point compared to CMake, whose caching aspects can lead to complex 
> issues, and where regenerating a configuration command is not so easy.

This is indeed valuable, but could be improved by adding the appropriate quotes.

[...]
> - autoconf can generate multiple configuration files (for example 
> public/private header files), but automake assumes only one is used. I don't 
> remeber the details, but had to work around this 15 years ago to re-implement 
> the generation of a "light" external config.h in m4 and shell instead of 
> using built-n features due to this.

We also found that we could not install config.h and use a Perl script to insert
config.h macros into the installed headers.

> - Our code is a mix of Fortran and C, with a bit of C++. Automake still deos 
> not support Fortran 90-type module dependencies, so we have to manage manual 
> dependencies in one of our Makefile.am's. More modern systems handle Fortran 
> (not quite the latest fad) much better.

At our site, there are a number of scripts in use that generate the Fortran
module/file dependencies on demand. For those not knowing Fortran: source files
are compiled to both, .mod files for use in runs of the compiler and .o files
for the later link step. The .mod files can follow a number of conventions in
respect to suffix and upper/lower case basename and suffix.

[...]
> - .la files broken by Linux packaging: in the case of an MPI library built 
> with various dependencies, some of which are loaded dynamically as plugins, 
> and not installed by default with the minimal packages, the .la file lists 
> some non-installed (and non-needed) libraries as dependencies.

I found that in general it's advisable to look through .la files and edit them
if needed.

> - incorrect options: recently, trying to use the Cray compiler toolchain on 
> an Arm64 machine, some low-level dependency related to back-end gcc libraries 
> added a "-pthread" option in the link line, while the top-level compiler used 
> a different syntax and did not accept that.

We encounter the same problem with the NAG Fortran compiler (equivalent options
are -Wc,-pthread -Wl,-pthread). The simplest solution was a wrapper around to
the compiler to patch up the options, another colleague patched ltmain.sh to
adjust appropriately.

> Although I have quite a bit of experience working around libtool's quirlks, I 
> was simply not able to build my code using that toolchain on that machine. 
> Total failure just before the finish line...
>
>
> -I have had a similar issue generating shared libraries with with CUDA, 
> though in that case I could work around this by duplicating and patching the 
> generated libtool file, and adding a Makefile.am rule for .cu files. But I 
> have never seen another project optionally using CUDA which bothered doing 
> this rather than using another build system (ie. if there is a better 
> solution, I did not find it).

Since CUDA depends on the installed kernel module, it's also giving us headaches
regarding stability of installed libraries like MPI.  There is support for
optional .cu files in our highest level software, I'll have to look up what the
responsible colleague did that might have eluded you. We already depend on a
build environment with the PGI/NVIDIA compiler for OpenACC when GPU support is
needed and that adds everything for cuda below the layer visible to libtool, we
might not be seeing your specific issue.

[...]
> Libtool has many nice aspects, such as handling of versioning, rpaths, and 
> more, but its
>
> insistence that it is always right, even in situations where it produces 
> unusable commands can really be a showstopper.

I can second that, we now use 7 patch sets for ltmain.sh to address issues
regarding compilers not matching the expectations of libtool.

> I have not looked at Meson much so far. In the short term, the fact that it 
> requires ninja, which is not yet ubiquitous on HPC systems, makes things a 
> bit less comfortable in our case, but the fact that is uses Python is an 
> advantage (Python is so prevalent in scientific software that just about all 
> users and developers in our community need to learn at least some rudiments 
> anyways, which is not true of m4, Makefiles, or cmake). So it could become a 
> serious contender (whereas the other Python-based contenders such as scons 
> and Waf seem to have less momentum).

I found scons to be sufficiently complex to be impenetrable to a number of
scientists, who otherwise knew and used Python.

Regards, Thomas
--
Thomas Jahns
HPC-Gruppe
Abteilung Anwendungssoftware

Deutsches Klimarechenzentrum GmbH
Bundesstraße 45a • D-20146 Hamburg • Germany

Phone:  +49 40 460094-151
Fax:    +49 40 460094-270
Email:  Thomas Jahns <jahns@dkrz.de>
URL:    www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5348 bytes
Desc: S/MIME Cryptographic Signature
URL: 
<https://lists.gnu.org/archive/html/automake/attachments/20210503/0d8057a1/attachment.bin>

------------------------------

Subject: Digest Footer

_______________________________________________
Automake mailing list
Automake@gnu.org
https://lists.gnu.org/mailman/listinfo/automake


------------------------------

End of Automake Digest, Vol 216, Issue 1
****************************************



Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à 
l'intention exclusive des destinataires et les informations qui y figurent sont 
strictement confidentielles. Toute utilisation de ce Message non conforme à sa 
destination, toute diffusion ou toute publication totale ou partielle, est 
interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le 
copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si 
vous avez reçu ce Message par erreur, merci de le supprimer de votre système, 
ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support 
que ce soit. Nous vous remercions également d'en avertir immédiatement 
l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie 
électronique arrivent en temps utile, sont sécurisées ou dénuées de toute 
erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the 
addressees. The information contained in this Message is confidential. Any use 
of information contained in this Message not in accord with its purpose, any 
dissemination or disclosure, either whole or partial, is prohibited except 
formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any 
part of it. If you have received this message in error, please delete it and 
all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or 
virus-free.