RE: new feature idea: ingesting processed rulesets

[Zoltán Turányi]

Hi David,

Thanks for your informative response, comments and references.

I do see your point (and in a way wanted to attack the separation of the DAG 
myself with this topic).
However, the advice to place all make information into a top-level Makefile.am 
is not always practical. The point of subdirectories is isolation, this is why 
we put files into separate directories. The dependencies between such 
directories would be best placed besides them not in some central makefile.

Henrik mentioned inclusion. I find that a more attractive solution as it seems 
to combine the best of both worlds: it has both isolation and results in a 
single DAG. Do you think autotools is possible to instrument to do that? Any 
advice on how to set it up?

Zoltan

-----Original Message-----
From: David A. Wheeler <dwheeler@dwheeler.com> 
Sent: Tuesday, May 23, 2023 8:32 PM
To: Henrik Carlqvist <hc94@poolhem.se>; bug-make <bug-make@gnu.org>
Cc: Zoltán Turányi <zoltan.turanyi@ericsson.com>
Subject: Re: new feature idea: ingesting processed rulesets



> 
> On Tue, 23 May 2023 12:07:51 -0400
> "David A. Wheeler" <dwheeler@dwheeler.com> wrote: 
>> The solution is to *NOT* use recursive make. Have *ONE* process run 
>> the makefile, with the correct data. Now you can enable parallel 
>> jobs, and have it run really quickly, because the make process has the 
>> correct information.
>> Using this approach you can routinely run large make jobs in a 
>> fraction of a second.

> On May 23, 2023, at 1:16 PM, Henrik Carlqvist <hc94@poolhem.se> wrote:
> If you prefer to do it that way, you can have your top Makefile 
> include all sub.mk files in the directory structure that it is able to find.

Yes, absolutely. The point is that you want a single process to have all the 
information, so it can perform the correct actions... multiple *files* is fine. 
As noted in "Recursive Make Considered Harmful" by Peter Miller <"Recursive 
Make Considered Harmful" by Peter Miller, e.g.:
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-01613997b4600749&q=1&e=93bda6e5-ed75-46ba-a224-c1c4e19d671a&u=https%3A%2F%2Faccu.org%2Fjournals%2Foverload%2F14%2F71%2Fmiller_2004%2F%2F>:
"The above [long list of problems] is based on one simple action: the DAG was 
artificially separated into incomplete pieces. This separation resulted in all 
of the problems familiar to recursive make builds.
Did make get it wrong? No. This is a case of the ancient GIGO principle: 
Garbage In, Garbage Out . Incomplete Makefiles are wrong Makefiles.
To avoid these problems, don’t break the DAG into pieces; instead, use one 
Makefile [process] for the entire project...
modern make implementations have include statements. . By including a relevant 
fragment from each module, the total size of the Makefile and its include files 
need be no larger than the total size of the Makefile s in the recursive case."

I find that "whole project" Makefiles are *incredibly* fast. I auto-generate 
dependencies from source in larger projects, but store them in ".d" files that 
depend on their sources; thus, regeneration of dependencies only happens when 
it needs to happen. Modern filesystems (with modern caches) can load all those 
make fragments in a blink of an eye. And they completely eliminate the endless 
problems of recursive make approaches.

Here are some things you might find useful for using make well:
"A Super-Simple Makefile for Medium-Sized C/C++ Projects" 
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-de62c1656fb0d189&q=1&e=93bda6e5-ed75-46ba-a224-c1c4e19d671a&u=https%3A%2F%2Fspin.atomicobject.com%2F2016%2F08%2F26%2Fmakefile-c-projects%2F
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-f93c4da8de9d600e&q=1&e=93bda6e5-ed75-46ba-a224-c1c4e19d671a&u=http%3A%2F%2Fmake.mad-scientist.net%2Fpapers%2Fadvanced-auto-dependency-generation%2F
My make-booster: 
https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-908c634bcb7bf69a&q=1&e=93bda6e5-ed75-46ba-a224-c1c4e19d671a&u=https%3A%2F%2Fgithub.com%2Fdavid-a-wheeler%2Fmake-booster

--- David A. Wheeler