Hello gcc and libtool lists,
Summary: both Autoconf-generated configure tests as well as some Libtool
construct invoke undefined behavior. Question is how to deal with it,
and whether GCC, as QoI, may want to define behavior in these cases.
1) Autoconf-generated configure tests often fake the prototype of some
function; e.g., AC_CHECK_FUNC([func]) uses
char func();
and tries to link that. Using this is undefined according to C99, if
func has a different actual prototype, and when all system libraries are
LTO'ed, gcc -flto may even detect this kind of inconsistency and could
act accordingly (nasal demons and such).
2) libtool has a feature that makes it extract symbol lists from
objects and turn them into fake declarations and function/object
pointers: fake static preloaded modules.
It currently works by running nm or a similar tool over the object, then
converting the output with a couple of sed script or so
(global_symbol_pipe, global_symbol_to_cdecl, and a couple more) to a
synthesized extra source file that then contains code like this:
extern int func();
extern char variable;
typedef struct {
const char *name;
void *address;
} lt_dlsymlist;
extern const lt_dlsymlist
lt__PROGRAM__LTX_preloaded_symbols[];
const lt_dlsymlist
lt__PROGRAM__LTX_preloaded_symbols[] =
{ { "@PROGRAM@", (void *) 0 },
{"func", (void *)&func},
{"variable", (void *)&variable},
{0, (void *) 0}
};
This is invoking undefined behavior in a couple of respects:
a) Pointers to functions are stored in pointer-to-void variables.
This could be fixed with an incompatible API change to using a union of
object and function pointer, I guess.
b) The symbols 'func' and 'variable' likely have the wrong prototypes,
i.e., elsewhere, they might be declared as
void func(int, double);
double variable[42];
instead. I haven't come across any actual issues with this yet, except
now LTO may rightfully complain about it.
Question is, what can we do about this? We could ensure to never pass
-flto or -fwhopr to the compilation of the libtool symfile object, and
remove it from some or all link tests done in configure. That's ugly.
Would that even be sufficient though? Conversely, would GCC developers
be willing to agree that, when GCC detects such inconsistencies, it
wouldn't take adverse advantage of it (e.g., by turning off LTO in this
case, or similar)?
Other possibilities for Autoconf would be to work toward a new set of
checking macros (or extensions of current one) where the configure.ac
author passes a full prototype for each function to check (Autoconf
could keep a list of known prototypes for often-checked functions).
I'm not sure how to fix the libtool symfile in a C99-conforming way.
Thanks for reading this far.
Cheers,
Ralf