Re: Request for comments: CONS specification

[H. S. Teoh]

On Fri, May 28, 2004 at 07:10:25PM +0200, Pierre THIERRY wrote:
> > The main barrier to adding this to the current version of Cons is that
> > its current monolithic design makes it very difficult to do [parallel
> > building] without major overhaul of the code.
> 
> I think that if we refactor Cons in clean way, new features will be very
> straightforward to implement. If everything is very modular, inserting a
> new module on top or inbetween should be easy.

Which is the point behind this rewrite, right? :-)

[snip]
> I like one the design goals of SCons, which I see as one of mine, but in
> a very particular point of view: a non-hacker should be able to use
> SCons to build something.
> 
> When I download a tar.gz, and start the compilation, I expect having
> just to do ./configure --prefix=/usr/loca/ && make && make install. I
> should not have to provide special options the obtain a stable build.

I actually have a package in Debian (atom4) that uses Cons to build
itself. The build instructions are dead simple: just type
        cons .

and it does everything. :-) Of course, I did have a more fancy Conscript
which can take options to configure where to build it and stuff, which I
use for building the Debian package. But even that is quite simple, the
entire build is triggered just by:
        cons opt1=val1 opt2=val2 .


[snip]
> And last but not least, this iteration thing should be fine tuned for
> each language. In C/C++, this would be disabled by default, and for TeX,
> we should include tests for Cons to know whether iterating is worth
> (e.g. the ReRun in the aux file).

I agree. We should provide an API that is flexible enough so that every
language specification can do things like this without needing to hack
Cons code. E.g., in the type spec, pass in a coderef that does the
checking.

[snip]
> > Only when you want to do a final shipping build, you want to make sure
> > everything is consistent. So it should be OK to just run TeX once
> > during development, and only run it in "full" mode when you want a
> > final build where everything must be consistent.
> 
> The default prupose of the build tool must remain building consistent
> things. The developper would build a variant, with iteration disabled.

Good idea.


[snip]
> > in the .consign file (or some other startup file)
> 
> I think we'll avoid having a monolithic design even here. A .cons
> directory will be far more flexible.

That's fine with me. We just need a cache somewhere, doesn't matter how
that's implemented.

[snip]
> > The problem now is that Cons wouldn't know whether main.c would be
> > produced or main.cc should be produced, unless the user specified it
> > explicitly.
> 
> There are a lot a ways of solving this ambiguousity: either giving Cons
> what the default language is for the build (either all the build, or for
> a particular subdirectory) or having a way to find out in the Flex or
> Bison files what the target type is.

In the case of Flex/Bison, actually, it might make sense to have separate
language specs for C and C++. This is because you have to actually run
Flex/Bison with different command-line options in order to make it produce
C++ code instead of just C code. (The difficulty with automating this is
that it's not necessarily a straightforward 1-to-1 mapping between
command-line options and language type.)

Unless we can think of a clean way to specify language type variants.

[snip]
> > Another problem is that *many* possible source file types are compiled
> > into .o files, and there may be a large graph of possible sources for
> > each .o file specified by the user. This might slow down Cons, if it
> > has to check through a long list of possibilities each time, plus
> > resolve ambiguities in the graph each time.
> 
> It's not a big deal for modern CPUs to handle such graphs. There's a lot
> of stable algorithms out there to walk and build graphs (use Graph;).

True.

[snip]
> > maybe we can force each file type to only ever produce a fixed set of
> > outputs, so Flex .l files will be assumed to only produce .c unless
> > the user explicitly overrides this.
> 
> We should never make assumption that have no serious justification. If
> Flex can produce C++, Objective-C or anything else, it would be a grave
> bug that Cons, by default, consider it outputs C files only.

Then we'll need a way to specify, in a file type spec, what are the
different possible output file types for a given command, and how to make
it produce one or the other file type.

[snip]
> > Probably we don't need to worry about the .s file? It's probably safe
> > to assume that for the common case, we'll always compile a .c file
> > into .o "directly".
> 
> Cons must be aware that there are intermediate steps. And for each set
> of tools it will know about, it will also know if, by default, they
> /forget/ these steps, or make them implicitly...

In the general case, this is not necessary, so it seems a bit unnecessary.
But now that I think about it, I can see why it might be useful to tell
Cons to, e.g., keep all intermediates for a source file that has some
subtle machine-code level bug. In this case, Cons should be able to figure
out what switches to pass to gcc to make it keep intermediates.

[snip]
> > Good point. I think it might make sense to refine the dependency graph
> > *during* the build process.
> 
> Happily, it will not be needed in most cases.

I'm not sure you can say that. In many of my own projects, I use Perl
scripts to generate C code (or parts of C code). The output .c files need
to be scanned at build time for dependencies, since they could be
generated from different places (either hardcoded in the Perl template or
specified explicitly in the input definition file). I suspect that this
method of generating .c source code isn't as rare as one might think.
There's always the choice of manually specifying the dependencies, of
course, but I'd rather not if it can be automated in Cons.

[snip]
> > The other possible solution is for the .l file type definition to
> > state explicitly that the resulting .c file will always have a
> > dependency on some corresponding .h file
> 
> I would consider it as a bug, again. If our guessing system don't work
> in a particular situation, we must not insert dirty hacks in it to
> bypass the problem.

This isn't the guessing system, this is the dependency tree which *must*
be correct in order for Cons to function properly! If you don't want to do
build-time dependency scanning, I don't see how you can get around it
without hardcoding the dependency in somehow. 


> > The problem is that for many programming languages, deducing
> > class/function dependencies amounts to parsing the source file(s) and
> > building a symbol table --- things which should be done by the
> > compiler, not by Cons.
> 
> Some compiler may provide a interface to do that. I agree that Cons must
> not become a semi-compiler.
[snip]

If it's not as bad as I thought, as Steven says, maybe scanning for
function declarations is OK. I'm a bit wary of this, though, 'cos parsing
C function declarations isn't as trivial as one might think. (Look at some
of the files in /usr/include/, for example.)


T

-- 
The design document is what the program should do. The source code is what
the program actually does. On a good day, they might actually resemble each
other...

signature.asc
Description: Digital signature