Re: Future plans for Autotools

[Thomas Jahns]

Hello,

some comments with a similar HPC background:

On 2021-05-02 19:49, FOURNIER Yvan wrote:
[...]
> - it is very easy to find the configure options from a previous run at the top 
> of the config.log and config.status files, and then copy/past them so as to 
> generate an new build in a clean (empty) build directory. This is IMHO a strong 
> point compared to CMake, whose caching aspects can lead to complex issues, and 
> where regenerating a configuration command is not so easy.

This is indeed valuable, but could be improved by adding the appropriate quotes.

[...]
> - autoconf can generate multiple configuration files (for example public/private header 
> files), but automake assumes only one is used. I don't remeber the details, but had to 
> work around this 15 years ago to re-implement the generation of a "light" 
> external config.h in m4 and shell instead of using built-n features due to this.

We also found that we could not install config.h and use a Perl script to insert 
config.h macros into the installed headers.

> - Our code is a mix of Fortran and C, with a bit of C++. Automake still deos 
> not support Fortran 90-type module dependencies, so we have to manage manual 
> dependencies in one of our Makefile.am's. More modern systems handle Fortran 
> (not quite the latest fad) much better.

At our site, there are a number of scripts in use that generate the Fortran 
module/file dependencies on demand. For those not knowing Fortran: source files 
are compiled to both, .mod files for use in runs of the compiler and .o files 
for the later link step. The .mod files can follow a number of conventions in 
respect to suffix and upper/lower case basename and suffix.

[...]
> - .la files broken by Linux packaging: in the case of an MPI library built with 
> various dependencies, some of which are loaded dynamically as plugins, and not 
> installed by default with the minimal packages, the .la file lists some 
> non-installed (and non-needed) libraries as dependencies.

I found that in general it's advisable to look through .la files and edit them 
if needed.

> - incorrect options: recently, trying to use the Cray compiler toolchain on an Arm64 
> machine, some low-level dependency related to back-end gcc libraries added a 
> "-pthread" option in the link line, while the top-level compiler used a 
> different syntax and did not accept that.

We encounter the same problem with the NAG Fortran compiler (equivalent options 
are -Wc,-pthread -Wl,-pthread). The simplest solution was a wrapper around to 
the compiler to patch up the options, another colleague patched ltmain.sh to 
adjust appropriately.

> Although I have quite a bit of experience working around libtool's quirlks, I 
> was simply not able to build my code using that toolchain on that machine. 
> Total failure just before the finish line...
>
>
> -I have had a similar issue generating shared libraries with with CUDA, though 
> in that case I could work around this by duplicating and patching the generated 
> libtool file, and adding a Makefile.am rule for .cu files. But I have never 
> seen another project optionally using CUDA which bothered doing this rather 
> than using another build system (ie. if there is a better solution, I did not 
> find it).

Since CUDA depends on the installed kernel module, it's also giving us headaches 
regarding stability of installed libraries like MPI.  There is support for 
optional .cu files in our highest level software, I'll have to look up what the 
responsible colleague did that might have eluded you. We already depend on a 
build environment with the PGI/NVIDIA compiler for OpenACC when GPU support is 
needed and that adds everything for cuda below the layer visible to libtool, we 
might not be seeing your specific issue.

[...]
> Libtool has many nice aspects, such as handling of versioning, rpaths, and 
> more, but its
>
> insistence that it is always right, even in situations where it produces 
> unusable commands can really be a showstopper.

I can second that, we now use 7 patch sets for ltmain.sh to address issues 
regarding compilers not matching the expectations of libtool.

> I have not looked at Meson much so far. In the short term, the fact that it 
> requires ninja, which is not yet ubiquitous on HPC systems, makes things a bit 
> less comfortable in our case, but the fact that is uses Python is an advantage 
> (Python is so prevalent in scientific software that just about all users and 
> developers in our community need to learn at least some rudiments anyways, 
> which is not true of m4, Makefiles, or cmake). So it could become a serious 
> contender (whereas the other Python-based contenders such as scons and Waf seem 
> to have less momentum).

I found scons to be sufficiently complex to be impenetrable to a number of 
scientists, who otherwise knew and used Python.

Regards, Thomas
--
Thomas Jahns
HPC-Gruppe
Abteilung Anwendungssoftware

Deutsches Klimarechenzentrum GmbH
Bundesstraße 45a • D-20146 Hamburg • Germany

Phone:  +49 40 460094-151
Fax:    +49 40 460094-270
Email:  Thomas Jahns <jahns@dkrz.de>
URL:    www.dkrz.de

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784

smime.p7s
Description: S/MIME Cryptographic Signature