Re: Dependencies and other issues with generated files

[David Sveningsson]

On 06/27/2012 10:47 PM, Nick Bowler wrote:
> Hello David,
> 
> On 2012-06-27 18:13 +0200, David Sveningsson wrote:
>> I'm having some problems when generating source files. I have written a
>> tool which generates both a c source-file, a header and dependencies of
>> the source, all at the same time using a single command.
> 
> There is an entire chapter of the Automake manual about exactly this:
> 
>   §28.9 Handling Tools that Produce Many Outputs
>   https://www.gnu.org/software/automake/manual/automake.html#Multiple-Outputs
> 
> The short version is that it is hard, but possible, to do this in make.
> Modifying your tool so that it can be invoked multiple times, producing
> one output file per invocation will be much simpler, although this is
> not always a satisfying option.

I missed that part of the manual. In the end it seems to be easier to
modify the tool rather than adding such complex rules, even it the build
time increases.

> [snip]
>>
>>     SUFFIXES = .foo
> 
> Most of the time, Automake is smart enough to add the required suffixes
> automatically, so you do not need to specify SUFFIXES in this example.

Ok, I did not know this. The manual seemed to indicate it was needed.

>>     app_SOURCES = test.foo main.c
> 
> I'm not sure how well listing the .foo file in _SOURCES is going to
> work, although someone who knows more about automake guts might be able
> to tell us exactly what happens.  You should probably list the .c file
> here, or in nodist_app_SOURCES if the generated source file is not meant
> to be distributed.

The source is meant to be distributed, not the generated file. The tool
will be required to build. Adding it to sources seems to work but maybe
it is wrong and only works by accident.

>>     .foo.c:
>>      ./mytool -f $< -e $(basename $@).h -o $@
>>
>> (main.c includes test.h)
>>
>> The first problem I'm having is that test.h may not have been built when
>> compiling main.c (especially if using parallel build). I guess a
>> workaround would be to add a dependency like "main.c: test.c". Is there
>> a way to handle this dependency in a more automatic way?
> 
> Again, this is covered in depth in the Automake manual.  Regardless, you
> *will* need to add an explicit prerequisite on the header file, because
> automatic dependency tracking does not work for generated headers.

Ok, if it is the only way I guess I have too.

As suggested by Charles Brown BUILT_SOURCES also seems to solve the
issue of generating test.{c,h} before building main.c. It works when the
tool is available but not for my in-tree unit-testing, but that is fine
as I can manually add dependencies for those.

>> Secondly if test.foo is in a subdirectory and I'm doing an out-of-tree
>> build the folder does not exist. Is there a way to automatically create
>> the required folders or rename the file similar to object files (e.g.
>> myapp-main.o)? I got it working using "@test -e $(dir $@) || mkdir -p
>> $(dir $@)", which I guess would be fine but it also seems like a hack to me.
> 
> Instead of the above, you should use something like the following at the
> start of your rule:
> 
>   test x"$(@D)" = x || $(MKDIR_P) "$(@D)"
> 
> $(MKDIR_P) is provided by autoconf and works even when the system mkdir
> does not support -p.
> 
> Unlike $(dir $@), the $(@D) construct is defined in POSIX and works on
> every make implementation I have access to.  However, contrary to POSIX,
> dmake expands $(@D) to the empty string for targets in the current
> working directory (which will cause the mkdir to fail), so we need to
> avoid calling mkdir in that case (hence the test).
> 
> If all the relevant targets are in subdirectories, you can remove the
> test as dmake will produce a useful string in this case.

That is quite useful, I didn't know make provided such variables. I'll
read that section of the manual again.

>> Next I'm having issues when cleaning. I would like "make clean" to
>> remove the generated files. I was expecting at least test.c to be
>> cleaned automatically but none of the generated files was removed.
>> Manually adding to CLEANFILES or clean-local is tiresome and error-prone.
> 
> Automake does not, and cannot, generate clean rules for files it does
> not know about.  So I'm afraid that's your job.  Such rules are usually
> pretty easy to write, for example if you have all your .foo files in a
> variable:
> 
>  FOOFILES = a.foo b.foo c.foo
> 
> and the tool generates .c and .h files from a .foo file, then something
> like
> 
>  CLEANFILES = $(FOOFILES:.foo=.c) $(FOOFILES:.foo=.h)
> 
> is probably sufficient.  For more involved cases, you might add a
> --clean option to your tool which deletes any files that it generates,
> and then call it from clean-local.

But it knows .c is compiled to .o and clean it properly. Does automake
have such filename mappings hardcoded?

Would it be possible to add a custom primary such as

    app_FOOFILES = a.foo b.foo

kind of similar to "python_PYTHON"?

Thanks for the help so far!