Re: Identify included files

[David Wright]

On Sun 24 May 2020 at 16:28:53 (-0400), Fr. Samuel Springuel wrote:
> > On 23 May, 2020, at 7:34 PM, David Wright <address@hidden> wrote:
> > 
> > If you follow some simple rules in your source layout, constructing
> > such a list might be most straightforward to do in the shell, using
> > grep.
> 
> It’s not quite as straight-forward as you seem to think:

That's why I pasted the sentence above from my previous suggestion.
I didn't think it was worth your generalising your script to cover
every possibility that LP offers, but that's in your hands.

> 1) You haven’t accounted for the possibility of multiple folders with varying 
> levels of hierarchy.  The changed file might be in ../ relative to one file, 
> ../../ relative to another, ../some/dir/levels relative to a third and other 
> possible variations.  It might also be specified /with/an/absolute/path in 
> some files.  All of that complicates the search string for grep.

That's right. You can also flip back and forth between
relative-to-top-level-file's-directory and
relative-to-calling-file's-directory in the same file,
which my grep will know nothing about.

You can make it yet tougher: place different files named foo.ily in
several directories, and then select different versions from different
locations, guided entirely by the directories from which each \include
originates. Again, grep will be blind to such tricks.

With constructions baroque enough to match the music, a shell script
would obviously be more work than it's worth, particularly compared
with DK's method. You'd probably be better off concentrating on
writing a script to deal with automating the DK method and using its
results (as well as keeping track of all this complexity in the
source, if it hasn't already caused insanity).

> 2) You haven’t accounted for -I options when lilypond is called.  Some of the 
> files included might not be in the same directory tree at all.  You’ll need 
> access to the list of -I options with which lilypond is called in order to 
> properly identify all the possible top-of-tree/ folders.  You might be able 
> to extract what -I flags a particular project has that are specific to it by 
> examining some file in the project folders, but what do you do if that 
> information is stored in a personal shell script or alias setup that catches 
> all lilypond calls and makes sure certain -I flags are added (as might be 
> common for someone using openLilyLib or with a personal library of formatting 
> snippets)?  Now your shell script has to go analyze your personal shell 
> script or shell profile.

Oh, come on, that's the easy bit. Whatever options you call LP with,
you can make the same information available to the script when you
use it as they're both at the level of the shell.

As for catching all LP calls, do people actually type their LP
commands, with all its options, directly into the shell each time they
run it, rather than using scripts?

BTW, bear in mind that you may need a fresh run of parse-only.ly
to generate a new .dep file/Makefile ("DEP" to save typos) whenever
you change your set of includes, else they're stale.

> 3) What about the relative-includes option?  If an included file includes 
> other files, where are these files located?  Is the given path relative to 
> the current file or to the top-level file?  Our grep search string is getting 
> even more complicated to account for this.

Already treated under (1).

> Sure, your ideas would probably catch most cases, and it would be possible to 
> write a shell script that accounted for these complications.  But you’re 
> duplicating a whole lot of work that’s already been done.  LilyPond itself 
> already knows how to deal with these factors to find the included files 
> (either because it’s built-in or because you’ve already told it what to do 
> when you configured your system).  Why reinvent this wheel in a shell script 
> if LilyPond can be made to output what it already knows for you?

I agree. there's no magic here. If you want to add complexity in one
place, you have to match it in another. Just so long as you need it.
(We haven't seen your use case.)

> Another issue: How are you keeping track of which files have changed (and 
> thus should seed your script)?  If you recompile frequently after only 
> changing a file or two, it’s fairly easy, but what if the change you’re 
> working on takes several days to (either due to complexity or restricted 
> time) or touches a whole bunch of files?  Are you supposed to manually 
> remember all those changes?  Or maybe you’re planning on querying your 
> version control system (you are using one, right?) for that information?  As 
> a last resort, you might also look at modified times on the pdfs (if they 
> exist) and the source files.

That's a question I might ask you: these are your files, your edits,
your working methods, your problem area.

> Of course, given a proper list of file dependencies, make will do all this 
> work for you.  That’s what it is for.  Pair make with a LilyPond based tool 
> which can determine dependencies automatically and it becomes even simpler 
> because the system is self-updating.

I've yet to see how make helps you at this level, ie within the LP
run. As I wrote earlier, I can see how it helps in constructing a
number of LP's *outputs* (like PDFs), and some may use it in the
way I use a script: to select the LP version, assemble the commandline
options, postprocess the output files, and suchlike.

But all LP needs to do its job is the name of the top-level file
and a search-list of libraries. It can find the \includes for itself
in the libraries given and, having found them, it compiles them all
regardless. Completely unlike, say, a C program.

But it seems to me that your OP had the makings of an A/B problem.
You originally asked for a script that worked in the forward
direction: a list of top-level file's dependencies, for constructing
DEP. Having got LP to perform that with DK's method, it turns out
your goal is something different: to discover all the top-level files
that need to be recompiled when you edit an arbitrary \include file.

That means that, having edited a mid- to low-level source file in your
library, you have to search every DEP to see whether that file is
mentioned. This includes those of the "someone" who uses your library.
And all those DEPs must be kept up-to-date if the search is to be
performed correctly.

The idea behind my later script was: given "some simple rules in your
source layout" and the constituency of your LP sources, the backwards
search for reverse dependencies can be carried out any time any source
file is altered, without the need to keep DEP files up-to-date,
because it greps the active set of files.

> Now, I will grant you, that if LilyPond could not be convinced to output what 
> it knows about file dependencies (and the Frescobaldi project hadn’t written 
> python-ly which has the include_args() command in the ly.slexer module), then 
> I probably would have written a shell script like you describe to get the 
> dependencies information (or, more likely, to just do the recompiling itself 
> and not bothered with make at all).  However, once David K. showed that it 
> was possible for LilyPond to output that information (and even better, 
> without going through all the work necessary to produce a score at the same 
> time) it makes the most logical sense to me to use that rather than start 
> from scratch with a shell script (or start from include_args() in python-ly 
> to write a Python script).

Given a top-level .ly file and DK's code, I don't see how you
would avoid any of the recompilng required when you use DK's
output. The last line of the script says

    #(format #t "~{~a\n~^~}" (ly:source-files))

How do you use source files without recompiling them—that's what 's
perplexed me all along in this thread. All I've tried to do is give
an idea for finding a list of current top-level files that \include an
arbitrary, specific file. Anyone, with assistance from LP, can make
this as hard as they like for themselves.

Cheers,
David.