Re: non-recursive build question

[Sandy Currier]

On Thursday 29 April 2004 10:43 am, Paul D. Smith wrote:
> %% Noel Yap <address@hidden> writes:
>
>   ny> In the end, though, the OP should be able to do what he wanted
>   ny> (assuming IIUC)
>
> That I don't know about; you could well be correct--I hope Sandy will
> provide a list of explicit requirements that we can use to determine
> that because I, at least, don't have any good feeling for exactly what
> they are.

Ok - I think I finally have a reasonable understanding of the problem and at
least one solution.

From the beginning, there was at least one case of pilot error - I 
personally
thought that if a dependent's target rules fired, that gmake would then
automatically fire the target's rules as well.  As explained and 
demonstrated,
this is not the case.  Whew.

Backgound:

When building shared libraries, it is sometimes the case that one does 
not want
to export (expose) all the symbols in the shared library.  In fact, it is
usually the case that one only wants to make available a limited number 
of the
symbols in a shared library.  One way to do this with gcc and ld is use the
following switch to gcc/g++:

-Wl,--version-script,<foo.exp>

where foo.exp is an ascii file containing the list of symbols to export.  I
believe that the .exp suffix is not a standard suffix.  Here it is done 
with a
script that scans, among other things, the (nm output of the) object 
files that
will comprise the .so and some text files generated when the .o's were 
created.
The .exp file is a prerequisite of the .so and used by the linker.

With a valid .exp and .so for each shared library, one can then 
construct a DAG
such that, since it is a .so and not a .a, downstream targets that require
linking against the shared library need only depend (DAG wise) on the 
.exp.  As
mentioned, this saves large amount of developer time not waiting for 
gratuitous
links of _everything_ that links against the .so.  There are perhaps 
between a
1000 and 2000 unit tests just on the single product at hand, and there are
perhaps about 70 some odd products.  So even though something like
http://www.electric-cloud.com/ claims a 20x speed-up, it is still probably
better to not even try to distribute the 1000's of unnecessary links in the
first place.  Note that much of the time, particularly during the end-game
cycle of a codeline before FCS, last minute bug fixes rarely change the 
.exp's.
And this is the time when schedules are late and fast incremental build and
test cycles are important.

The Requirements:

1) the link of the .so requires the .exp
2) downstream targets that explicitly depend on the .exp actually also 
need the
correct .so to exist and for it to be valid
3) the targets of 2) above cannot depend on the .so and can only depend 
on the
.exp
4) both the .exp and the .so depend on the same list of .o's, namely 
those that
comprise the shared library
5) the link of the .so is not short but is not huge
6) the step of building the .exp is smart - the script knows that if the 
.exp
does not change, it will not update the file
7) vpath must work (target rules must be in terms of $@ etc.)

I now believe that it is 2) that more or less causes the curve ball.
Downstream targets only want to depend on the .exp but also must wait 
until the
.so is completely updated if it is going to be updated.

I do not think that an order dependency will work because something needs to
add the creation of the .so into the DAG of the downstream target (and 
an order
rule will not do it).  Specifying a downstream unittest target for example
needs to add the .so to the DAG so that it gets rebuilt.

So, below is the solution that I tried and seems to work:

foo.so: $(ALL_FOO_OFILES) [plus various other stuff like the .exp generator]
       create_the_exp_file_smartly $(basename $@).exp $(filter-out 
[...],$^)
   create_the_so_file $@ $(filter-out [...],$^)

foo.exp: foo.so

The above uses the foo.so rule to build the foo.exp (smartly - it is only
actually updated if it changes) as a side effect.  The second target 
specifies
no commands - it is there to just add the .so as a dependency.

With the above, I cannot seem to break the build by specifying any 
combination
of targets or any combination of -jN.

Interestingly, when a .c file is touched which causes the .o to be out 
of date
(but the .exp does not change), gmake appears to never even print via 
'-d' the
fact that foo.exp must be rebuilt:

$ touch foo1.c
$ make -d
[...]
   Considering target file `../do_glnx86/productA/src/foo/foo.exp'.
    Looking for an implicit rule for 
`../do_glnx86/productA/src/foo/foo.exp'.
    No implicit rule found for `../do_glnx86/productA/src/foo/foo.exp'.
     Pruning file `../do_glnx86/bin/libfoo.so'.
    Finished prerequisites of target file 
`../do_glnx86/productA/src/foo/foo.exp'.
    Prerequisite `../do_glnx86/bin/libfoo.so' is newer than target 
`../do_glnx86/productA/src/foo/foo.exp'.
   No commands for `../do_glnx86/productA/src/foo/foo.exp' and no 
prerequisites actually changed.
   No need to remake target `../do_glnx86/productA/src/foo/foo.exp'.
[...]

Comments appreciated, and thanks in advance for any review!
-sandy