Partial linking with _RELOCATABLES - Proposed enhancement

[Marc Alff]

Hi all

This email describe a proposed enhancement to Automake, to support 
partial linking (also known as "ld -r").

This is the discussion thread; a proposed patch is available on 
address@hidden

All the errors are mine and sorry for the long babble,
but I think it's important to explain why this enhancement can help is 
some cases.

============= Part I : What is partial linking ? =============

Building code typically consist of a chain like this :

*.c, *.cpp --> *.o --> lib*.a, lib*.so or binaries

Partial linking allows to do this :

part1.o, part2.o ... partn.o --> glued.o

A typical "partial link" or "relocatable (-r)"  command is :
ld -r -o glued.o part1.o part2.o ... partn.o

As a result, a build chain can be longer :

*.c, *.cpp --> *.o --> * glued.o --> * super-glued.o --> lib*.a, lib*.so 
or binaries

The difference between "glued.o" and "libglued.a" is that a 
"relocatable" object
contains all the code in one block, with internal link dependencies 
already resolved,
where *.a is just an aggregation of individual objects in an archive.
In some cases, that DOES matter (see below).

=============== Part II : Motivation ===============

1) Organizational

Consider a project that delivers only 1 binary at the end,
but which consist of so many files, components and configuration options 
that it's highly desirable
to organize the code base in a lot or directories, sub directories etc.
For example, the linux kernel :)
In this case, linking a kernel image from a huge list of *.o is very 
impractical;
instead it's better to link small parts together first, and link bigger 
parts from "convenience" relocatables.

The main concern is that, how the source code is organized into directories
with recursive makefiles is an implementation choice,
it should **not** impact what the final deliverable looks like (maybe I 
still want ONE *.a or *.so or binary at the end,
not expose a collection of *.a or *.so to my users).

2) Technical limitations due to volume of code

Sometimes you have no choice : linking all the *.o at once makes the 
command line too big, or flat out kills the linker.
There is also a performance aspect to it in extreme cases
(Historical note: I have first seen this technique used 15 years ago on 
a project
that would take 1 full day to build on a dedicated machine, using ld -r 
was just the only way to get a result at all).

3) Subtle differences between static libraries, dynamic libraries and 
relocatables

Consider a program where main.o contains all the "core" logic, and is 
capable or calling plug-ins.
Various plug-ins plugin-1.o, plugin-2.o, ... all implement the same 
plug-in interface.

The problem is that main.o does not contain any call to (read : does not 
reference any symbol present in) plugin-*.o,
since that code is **independent** of the plug-ins implementation.

3-a) Using static libraries (libplugin-1.a)

Link the binary with main.o -llibplugin-1.a ... nothing happens

To make things work, you have to "register" the plug-ins somewhere, to 
force a reference to them.

In C, this can be done like this :

$ cat all_plugin.c

#ifdef HAVE_PLUGIN_1
#include "plugin-1.h"
#endif

#ifdef HAVE_PLUGIN_2
#include "plugin-2.h"
#endif

(...)

struct plugininfo * plugin_tab [] = {

#ifdef HAVE_PLUGIN_1
& plugin_1_info,
#endif

#ifdef HAVE_PLUGIN_1
& plugin_2_info,
#endif

(...)

} ;

It works if you know at build time what to use, and if the list of available
plug-ins is fairly controlled and known, which might be an invalid 
assumption.

3-b) Using dynamic libraries

This time, plug-ins are loaded dynamically using dlopen().
This works great if the choice of which plug-in to use is done at 
runtime (instead of build time),
but you need a configuration file somewhere to list what dll to dlopen().

3-c) Using relocatables and C++

Each plugin contains a line like this :

static PluginInfo myInfo(...) ;

The C++ class PluginInfo is implemented in main.o, and happen to do 
something like :

PluginInfo::PluginInfo(...)
{
  all_plugin.register(this) ;
}

What happens is magic between __main and _main due to the static myInfo().

The final binary is linked with :
main.o all-plugin.o more-plugin-if-you-like.o

Note that using
- main.o all-plugin.so more-plugin-if-you-like.so
- main.o all-plugin.a more-plugin-if-you-like.a
would **not** produce the same result (at least, last time i tried).

The interesting point is :
- no logic ever in the core (main.o) implementation ever reference 
explicitly plugin-x.o
- there is no configuration file either to maintain with a list of dll 
to dlopen().

From the end-user point of you : I need X : I put X.o in my link 
command; I don't : then I don't.
That's it, KISS. No configuration to mess with.

This is a remote case, but I happen to depend on it heavily for a 
project of mine,
where the "plug-in" stuff happen to be generated code that varies all 
the time.

I think this enhancement has value just because of point 1) alone,
and can help in case of projects with a fairly sizable code base.

============ Part III : Proposed enhancement ============

In short, Makefile.am can be written like this

noinst_RELOCATABLES = glued.o
glued_o_SOURCES = part-1.cpp part-2.cpp

glued.o is a **notation**, not the actual name of a file :

..._RELOCATABLES = glued.o super-glued.obj
glued_o_... = ...
super-glued_obj_... = ...

will work both on platforms that use .o or .obj for objects,
and in both cases the generated makefile will contain names like :
glued.$(OBJEXT) : ...
super-glued.$(OBJEXT) : ...

You can add these if you like :
glued_o_LDR = /my/own/ldr/program
glued_o_LDRFLAGS = -please -make -section blah -start -at -address blah
glued_o_LIBADD = more-part-X.$(OBJEXT) more-part-Y.$(OBJEXT)

The generated Makefile uses LIBTOOL to actually do the linking work,
so it should be portable : $LIBTOOL --mode=link xxx

============ Part IV : Conclusion ============

This enhancement has been implemented already, since I happen to need it 
and depends on it,
and it has been tested on some platforms already.

The point here is to propose it for discussion, and possibly inclusion 
in the "official" Automake code base.

Thanks all for your comments,
and thanks to Ralf Wildenhues for early comments and suggestions that 
encouraged this babble :)

Cheers,
Marc Alff.