Re: Idea: Add .COMMANDCHANGE and .CACHE

[Britton Kerin]

On Mon, Jun 10, 2019 at 9:20 AM David A. Wheeler <address@hidden> wrote:
>
> Idea: Add .COMMANDCHANGE and .CACHE
>
> Problem:
>
> Paul Smith noted on Sun, 09 Jun 2019 22:11:32 -0400:
> > Most of the requests I see these days that would require a "last state
> > database" wouldn't be helped by md5 checks: mainly they're asking for
> > things like rebuilding targets automatically when compiler options have
> > changed etc. Things like that can be done today with a lot of fancy
> > trickery but people would rather it "just worked".
>
> As noted above, "make"
> detects when *external* files change, but if a command *inside* a Makefile
> changes (including its expanded parameters) then make doesn't notice the 
> change.
> This could be resolved if make could automatically notice a change in
> expanded command. Doing this will require the ability to store state somehow.

The trickery required is not too fancy:

     foo.o: foo.c Makefile
             ...
or

     foo.o: foo.c $(PARANOID_REBUILD)
             ...

Then use when desired:
     (export PARANOID_REBUILD=Makefile && Make)

I do something like one or other (depending on project size) of these
most everywhere.

> In addition, sometimes recipes take a while, and it'd be nice to be able to
> cache old results. If we have to store state anyway,
> we may as well support caching previous results.

This is IMO an entirely new and mostly orthogonal project from Make,
and there is
already ccache (and presumably similar things for other langs).

Britton

>
>
> Proposed solution:
>
> I propose a new special target .COMMANDCHANGE.
> Its dependencies (or all targets if no target is listed) add an implied 
> dependency
> on the expanded commands of the rule.  Thus, if the commands (once expanded)
> change, the rule will be considered obsolete & be rerun.
>
> This requires a way to notice a change.  Instead of a conventional database
> (which is hard to share across instances & can be complicated to implement),
> I propose using a directory specified in MAKESTATE
> (by default ".makestate") for storing shared state.
>
> Every time a non-.PHONY expanded target with .COMMANDCHANGE and a non-empty 
> rule
> is being considered for execution, the
> change state file $(MAKESTATE)/$(FIRST TARGET).sha256 is first examined.
> If the file doesn't exist, or its contents do not match the SHA-256 hash
> of the expanded command for that target, then the target is considered
> out-of-date and the rule is re-run.
> If the set of commands executes successfully, then its SHA-256 hash
> is written to $(MAKESTATE)/$(FIRST TARGET).sha256
> (after creating its directory if necessary).
> (Note: this could be stored in a directory name, instead of file contents,
> so file opening wouldn't be necessary.  However, that would make it
> easy to have a weird inconsistent state where there are multiple entries,
> and I think it's worth avoiding that problem.)
>
> If first_target is an absolute path the state path
> will still be in $(MAKESTATE) by treating the target name as beginning as 
> ".TREEROOT".
> E.g., if the target is /usr/share/mystuff/stuff,
> then the change state file is .makestate//usr/share/mystuff/stuff.sha256.
> (This might require some nonsense on Windows, but it's doable.)
>
> In addition, if an executed rule is the target of .CACHE, has 1+ commands,
> and MAKESTATECACHENUM is defined as a positive integer, then that rule's
> execution results are cached:
>
> * After after executing
>   all of the generated targets are copied (preserving all metadata possible)
>   into $(MAKESTATE)/$(FIRST_TARGET).cache/$(SHA256 of expanded 
> command)/$(TARGET_NAME).
>   It then looks at "$(MAKESTATE)/$(FIRST_TARGET).cache" to see if there are
>   more than MAKESTATECACHENUM entries, and if so, recursively deletes
>   the oldest entries to get it down to that number.
>   My guess is MAKESTATECACHENUM should default to 3.
> * Before executing a rule that's outdated, check for
>    $(MAKESTATE)/$(FIRST_TARGET).cache/$(SHA256 of expanded 
> command)/$(TARGET_NAME).
>    If it exists, copy its contents to the appropriate targets and *delete* 
> targets
>    of the rule if there's no corresponding entry.
>    Note that .TREEROOT still applies.
>
> I suspect .COMMANDCHANGE is a terrible name; better ones welcome.
>
> --- David A. Wheeler
>
>
> _______________________________________________
> Bug-make mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/bug-make