Re: parallelization of ./configure compiler test processes

[Thomas Jahns]

On Mar 30, 2023, at 17:48 , Bob Friesenhahn <bfriesen@simple.dallas.tx.us> 
wrote:
> 
> On Thu, 30 Mar 2023, Thomas Jahns wrote:
> 
>> Hello Danny,
>> 
>> I spent some time thinking about improvements to autoconf configure scripts 
>> (while waiting for builds to proceed). In my view, it is currently still 
>> easier to seek small efficiency gains that, in sum, could still improve 
>> run-time substantially than parallelizing the whole would be, because there 
>> is so much often untapped potential:
>> 
>> * Use bash builtin shell commands to fork and especially exec less. In 
>> modern systems with comparatively fast data paths compared to anything that 
>> affects resource control, like changing memory mappings, dropping caches 
>> etc., syscalls can be a substantial source of slow down.
>> * Use TMPDIR to prevent temporary files from hitting disk (use /dev/shm or 
>> similar instead of /tmp)
>> * In the case of spack I've seen substantial improvements from running the 
>> whole build in /dev/shm and think spack should at least mention the 
>> possibility for systems with substantial amounts of RAM (and guess what kind 
>> of system many sites using spack just happen to have?).
>> * The gcc option -pipe is similarly helpful to get build phases to start as 
>> soon as possible.
> 
> The main problem with the above is that it is focused on one particular 
> shell, and operating system.  While the combination is very popular, Autoconf 
> is still also a portability tool.

I should have clarified that I was particularly thinking of the kind of system 
spack is typically used for: massively parallel server machines running Linux 
attached to (usually) high-throughput, high-latency network filesystems. Sure, 
autoconf itself has much broader appeal, but the context in which Danny 
presumably wants to speed up configure scripts is much narrower. Also I think 
having quicker turnaround in the usual development environment is still very 
much valuable, if seldom used deployment targets still are fully functional. If 
one compares what the original ksh does in terms of temporary files and forked 
processes to how a modern zsh or bash runs the very same script, one can find 
there's also historical precedent for the kind of optimization I'm proposing.

> While continued optimization is important, I suggest that a good strategy is 
> to also spend more time on investigating useful caching features.  It is 
> common that a given test permutation is executed more than once.  By 
> implementing a smart shared cache, it should be possible to identify similar 
> permutations and cache the results so that the next time they are 
> encountered, the cached results will simply be used.  The cache could be at 
> the user and/or whole system level.
> 
> Autoconf already has good support for a local cache so that a re-configure is 
> much faster, and it has support for a system-wide shared cache.
> 
> Speed-up due to caching can be explored by using the --config-cache option, 
> or using a config.cache file.
> 
> One would think that a "compiler" test should be cacheable given the same 
> compiler with similar options.


That's certainly true for an unchanged configure scripts running from the same 
initial state (arguments, environment variables etc.) and can probably be used 
to good effect in e.g. CI/CD setups. For tools like spack, where probably a 
rebuild almost always means that some aspect of the environment has changed too 
(OS update, changed flags, newer compiler version) I'm not so sure test results 
can safely be cached. At least not without keeping track of substantially more 
state than current autoconf caching provides.

Kind regards,
Thomas

smime.p7s
Description: S/MIME cryptographic signature