Re: Identify included files

[David Wright]

On Fri 22 May 2020 at 12:12:17 (+0100), Wols Lists wrote:
> On 22/05/20 02:08, David Wright wrote:
> > On Thu 21 May 2020 at 17:54:38 (+0100), antlists wrote:
> >> On 21/05/2020 16:36, David Wright wrote:
> >>> On Thu 21 May 2020 at 13:57:00 (+0100), antlists wrote:
> 
> >> I have tried accidentally
> >> to do things like "lilypond voiceTrombone.ily" and it blew up quite
> >> spectacularly. I would expect most of mine, and in truth most other
> >> peoples', .ily files to blow up if fed straight to lilypond.
> > 
> > Big deal. I won't ask what's in voiceTrombone.ily.
> 
> A variable declaration. You know - EXACTLY THE SAME as C programmers
> often stick in .h files ...

Sure, but am I supposed to divine what's wrong with it from your
statement that it "blows up".

$ cat scale.ily
scale = \relative { c' d e f g a b c }
$ ~/lilypond-2.21.0-1.linux-64/bin/lilypond scale.ily 
GNU LilyPond 2.21.0
Processing `scale.ily'
Parsing...
scale.ily:1: warning: no \version statement found, please add

\version "2.21.0"

for future compatibility
Success: compilation successfully completed
$ 

> >> And if they do compile, the output probably isn't much use ... :-)
> 
> >> Does bar.c contain a main{} definition?
> > 
> > Why would it?
> 
> Why WOULDN'T it?

Because, in trying to understand the equivalences that were given,

.ily -> .h
.ly -> .c
.pdf -> .o

I'm not deliberately generating corner cases.

> You said that bar.c contained a load of functionality that you wanted in
> foo.c. How am I supposed to know whether bar.c is a program or a library?

Because of the next paragraph.

> >>> Now we can compile foo.c (which #includes bar.h) over and over while
> >>> debugging it, all without ever recompiling bar.c. foo.c knows all it
> >>> needs to know about bar.c because of bar.h's declarations.

Main programs have to know about the libraries they call. That's what
header lines were invented for. Then the header lines were put in a
separate file so that the library source could include the header
lines too (just as a confirmatory check).

> >> Can you now link bar.o into foo? I honestly don't know - I don't know
> >> how a linker will behave if fed two main{} functions ... ?
> > 
> > This isn't too hard. As I wrote, bar.c has functions, and bar.h has
> > declarations for those functions so that compiling foo.c can check
> > the calls of those functions without ever seeing bar.c (ie without
> > recompiling bar.c over and over).
> > 
> >>> I now have a suite.ly that typesets my Suite for flute and piano.
> >>> It \includes nine files: flute{I,II,III}.ily, piano{I,II,III}.ily
> >>> (the notes), and flute-part.ily, piano-part.ily and complete-score.ily,
> >>> (my \score templates). It also \includes {marks,dyns,fingers}.ily that
> >>> contain LP code for my preferred styles of markings, dynamics and 
> >>> fingering.
> >>>
> >>> I have a file, suite.dep, which contains the information that suite.ly
> >>> depends on these files to produce three targets, {flute,piano,suite}.pdf
> >>> The same file might also indicate that I need just flute{I,II,III}.ily,
> >>> flute-part.ily and {marks,dyns,fingers}.ily for just a flute.pdf.
> >>>
> >>> What do I write in a top-level theFlute.ly file to produce this
> >>> flute.pdf and how does it benefit from the suite.sep information?
> >>> How do I use this information profitably, in the way that bar.h did?
> 
> I don't have a clue. Your workflow sounds completely weird and totally
> incomprehensible to me. You know what? I don't give a monkeys - your
> world view is what you choose to see. You see things different to me!

It isn't my workflow. It's just an attempt to split a project into
hypothetical .ly and .ily files in some manner suitable for matching
the equivalences above.

My turn to write a list of equivalences, with flute≡cello, piano≡horn:

flute{I,II,III}.ily        cello.ily
piano{I,II,III}.ily        horn.ily
{marks,dyns,fingers}.ily   figures.ily
flute-part.ily             symphony-cello.ly
piano-part.ily             symphony-horn.ly
complete-score.ily         symphony.ly
theFlute.ly                symphony-cello.ly and cello.ily for cello part
      → flute.pdf                → symphony-cello.pdf
thePiano.ly                symphony-horn.ly and horn.ily for horn part
      → piano.pdf                → symphony-horn.pdf
theSuite.ly                symphony.ly and all the above for full symphony
      → suite.pdf                → symphony.pdf

That's not very different from the LP documentation (in the right hand
column). My *hypothetical* workflow allowed for movement II to become
outrageously popular, such that people would want to buy it separately :)

When theFlute.ly is run, it \includes the appropriate .ily files,
which LP finds without needing any suite.dep file to point out that
theFlute.ly depends on flute-part.ily. Hence my question about the
benefit it's meant to give me.

> >>> ¹ I'm abbreviating with shell's "brace expansion" notation.
> >>>
> >> Now we get to the meat of the matter. You've put bar.c's
> >> *declarations* into bar.h. Does lilypond even have declarations? Also,
> >> it's perfectly normal practice to put *definitions* in .h files -
> >> static variables, macros, inline functions, etc etc.
> >>
> >> I put most of my lilypond variable *definitions* into .ily files. Yup,
> >> .h files *tend* to contain declarations not definitions, but as I say,
> >> I don't think lilypond syntax has  declarations?
> >>
> >> So yes, am I right in thinking you view .ily files as closer to .o
> >> files? With lilypond closer to a linker than a compiler?
> > 
> > No.
> 
> Okay.

As I said, none of this is "my view". However, I'd already accepted
that one might see an equivalence between .o and .pdf files and had
asked what program (like, say, LaTeX) might be suggested as the
equivalent of a link-loader.

> >> At the end of the day, I see it as .h files can contain declarations,
> >> .ily files contain declarations. .h files are included, .ily files are
> >> included. Some languages (like lilypond) are monolithic and don't
> >> really have the concept of libraries. So I find it extremely easy to
> >> view .ly/.ily as being similar to .c/.h. Not the same, because
> >> lilypond is not gcc ... :-)
> >>
> >> There's no one absolute view - you see things different to me, doesn't
> >> mean you (or me) is right.
> > 
> > Look¹, I'm not the one making this analogy. I'm trying to make any
> > sense of it, and why this "dependency file" is being built.
> 
> I didn't make it either! I just happen to totally agree with it.
> > 
> > How do I use .ily files—almost all of them are definitions of
> > layouts, score structures, paper formats, variables' default values,
> > my abbreviations, and scheme code where it's wrapped in LP syntax.
> > They reside in a library directory that also has three subdirectories:
> > a collection of systematically named paper-margin files, a load of
> > boilerplate for exhaustively constructing MIDI file voice combinations,
> > and raw .scm files (like swing.scm).
> > 
> > One or two .ily files consist of source that's been factorised out of
> > the .ly file(s) that calls them. So, for example, there are festal
> > hymns that have words for the different seasons, so most of the
> > seasonal versions can be factorised out into an .ily file and then
> > inserted at the appropriate points in the main .ly file. There's
> > no saving in volume of compilation, only duplication.
> 
> WHICH IS EXACTLY A REASON FOR .h FILES!

Astonishing idea. There's no documented interface between each pair
of files. I had a file like:

notes = {the notes}
words = {Easter words}
\score
words = {Pentecost words}
\score

so I split it into this pair which reside in the same directory:

a.ly:
notes = {the notes}
words = {Easter words}
\book{
\bookOutputSuffix "easter"
\include b.ily
}
words = {Pentecost words}
\book{
\bookOutputSuffix "pentecost"
\include b.ily
}

b.ily:
\score

I see no similarity here to a header file or a library file.

> Removing duplication is a good
> way of removing errors (or making them easier to fix :-)

Agreed. In one of these hymn pairs, the \score is actually a sequence
of three scores: the refrain, and two alternating verse structures.
So it helps prevent accidents.

> > I haven't used any equivalent of .o files in LP and don't know how
> > you would (which is why I'm asking). The nearest is with LaTeX
> > documents that construct PDFs from .tex files and small .pdf files,
> > so LaTeX (and sometimes an enclosing shell script) is acting as
> > the "link-loader". Even here, I don't save a lot of compilation.
> > For example, when I typeset a psalm for the choir, the (Anglican)
> > chants are automatically recompiled by LP so that they can be
> > transposed on the fly, to make the changes in chant work. (I often
> > repoint the psalm to the chants, too.)
> > 
> Well, the way *I* see *ME* using .o files in lilypond is exactly the way
> I use make in lilypond - they are virtual targets that don't have a
> physical existence. The fact that they happen to have the same name as a
> physical file that does exist is neither here nor there. If a .ily has a
> bunch of include statements, I have a virtual target (a .o if you like)
> with the parent name that depends on the sub-names.
> 
> In other words, I am using make to manage my .ily files in EXACTLY THE
> SAME way I would use make to manage my .h files in a C project. It
> sounds to me like you don't work that way.

I manage .ily files more like I used to manage my .py module library
when I had Python projects of a respectable size, where import is
roughly equivalent to \include. But the comparison is extremely
superficial. About the only similarity is that .ily&.py files are
included&imported by disparate programs, which is why they reside in
~/LilyLib&~/PythonLib rather than in ~/scores/&~/bin/.

But again, there were .py modules which were specific to one or two
main programs, and these resided with their caller rather than in
the library. They would be developed and debugged together, and
have no separate existence.

> What on earth are your .dep
> files? What are your .sep files (or was that a typo? :-)

[Yes, a typo.]

I didn't quote the two lines following the list of equivalents,
mainly because I didn't know what it meant:

   "To this I would also add one more file type for storing dependency 
information
    .dly -> .d"

So my brain misremembered dependencies as .dep, probably from years of
generating modules.dep files in the linux kernel.

> Your workflow
> doesn't make sense to me!

Well, for the sort of scores I produce, I have enough idle CPU time
for battling with dependencies and make to be a waste of time.
When I run a score, I generally let LP generate all the versions.
If I'm typing in new material, I make sure that the first format
that's typeset is the one that's producing a copy that matches my
source material. (I put temporary line and page breaks in the
\global so that my PDF output matches the source manuscript.)
LP carries on running any other formats while I'm looking at the
PDF and editing the source file. It's finished by the time I next
want to run the source. (The formats might be two and four staves,
± any accompaniment, ± accompaniment alone, ± soloists, descants,
conductor's copy, separate 𝄞/𝄢 copies for tenors, anything …)

I don't run LP directly, but from a script, and that looks after
filenames, directory locations, library inclusion, MIDI renaming
(for sensible collation), cropping, and so on.

Where things are more complicated, like the transposed Anglican
chants that I mentioned, I use Python, so I can write in LaTeX
\newpagesnippet{../chants/d-hawes-E@f}{}
and a double chant in E by Hawes will be printed in F at the
top of the next page.

> I know I'm weird - I "don't do" visual, and guis, and all that sort of
> stuff. I'm much happier with text. I know that makes me a "unix
> greybeard" - it's only about 10% of the population that feels that way,
> but I REALLY don't understand why most people are much happier with
> pictures.

Ditto.

> I know you say you want to understand what we're thinking, but you're
> coming across as though you believe your POV is the "right" one. From
> *my* POV yours doesn't make sense! Forget C, forget lilypond, look at
> the broader picture, look at make, and as far as I'm concerned, .ily
> files fit in my make worldview exactly the same way as C .h files fit
> in. For me that analogy is perfect.

Not intended. Having suggested one method (a simple, unix way) of
searching for \includes, I was asking for clarification of why this
dependency information was required, and how it would reduce the
amount of compilation (seeing that LP has no equivalent of .pyc
(compiled byte code) or .o (compiled object code) that can be stored).

*My* POV (or my workflow) is irrelevant and, as you can see, bears no
resemblance to the workflow you were apparently attributing to me and
criticising.

Cheers,
David.