Re: Specifying dependencies among package outputs?

[Simon South]

Thank you Brett, Julien and Tobias for your responses. They've been
super helpful and have cleared things up for me quite a bit.

Julien Lepiller <julien@lepiller.eu> writes:
> For tracking runtime dependencies, guix uses another mechanism: when a
> package holds a reference to another package (by embedding the store
> path of that other package), it depends on it, anl guix will pull them
> both.

This was the key piece of information I was missing: Guix packages don't
really deal with "dependencies" per se. Instead a package has inputs and
outputs, plus a build process that transforms one set to the other; and
the references among these items that result naturally from the
transformation process are what Guix uses to determine which packages
should be installed alongside which.

So a packager's focus ought to be on specifying the package's inputs,
outputs and transformation correctly, after which Guix can generally be
relied on to do the right thing automatically with regard to
"dependencies".

This makes sense, and even seems painfully obvious now that I've written
it out (hello, functional programming). But it wasn't obvious before.

This also clears up for me a bit of remaining confusion around the
distinction between "inputs" and "propagated inputs": I wondered why, if
a package's inputs are its dependencies, the "propagated-inputs" field
is needed at all since surely a package's inputs would always be
installed alongside it. The explanation is that a package's inputs are
_not_ its dependencies; they are merely inputs to its build process, and
whether one becomes a "dependency" depends entirely on whether a
reference to it remains in any of the package's outputs. The
"propagated-input" field is used to ensure this association is made,
even when there is no apparent reference (that Guix can find) in the
outputs.

Tobias Geerinckx-Rice <me@tobias.gr> writes:
> If you apply the patch below you'll see (e.g., with ‘guix size’) that
> installing only knot:tools will pull in knot{:out,:lib} without any
> human-made hints to that effect.

Thank you for this! This is amazing, and exactly the sort of thing I had
in mind. (Though I wonder if the "tools" output would better be called
"utils", to match the isc-bind package?)

Are you planning on committing these changes? I think they're great.

> A saving of 13.9 MiB or <10% is not considered impressive.

It's not that bad either, and anyway it seems logical to me for the
package to be split up this way considering each of the outputs serves a
distinct purpose.

>> However, Knot's daemon and utilities have the same dependency on its
>> own libraries, so pulling those into a separate "lib" output would be
>> liable to break everything else.
>
> Why?

Assuming you didn't mean this rhetorically: My assumption was that
without an explicitly stated dependency between the "out" and "lib"
packages, Guix wouldn't know to install the two together. So a user
running "guix install knot" would get only a binary that aborts at
startup with a complaint about missing libraries.

Not being aware of "references" in this context is what had me confused.

> Which other examples or documentation have you read?

I actually did a fair bit of searching in multiple places but because I
was looking for "dependencies" and not "references", none of what I
found seemed very helpful. The manual could certainly do more to
highlight this distinction; in its "package" reference for instance it
says quite plainly regarding the "input" fields, "These fields list
dependencies of the package."[0]

That said, looking at the manual again this morning I quickly found
this[1]:

       The outputs of derivations—i.e., the build results—have a set of
    “references”, as reported by the ‘references’ RPC or the ‘guix gc
    --references’ command (*note Invoking guix gc::).  References are
    the set of run-time dependencies of the build results.  References
    are a subset of the inputs of the derivation; this subset is
    automatically computed by the build daemon by scanning all the files
    in the outputs.

That more-or-less lays it out, but I wouldn't have found it since

- It's in the page on "Derivations", which I'm accustomed to thinking of
  as a low-level detail that can safely be ignored;

- A quick scan of the paragraph gives the impression it has more to do
  with the "guix gc" command than with writing package definitions; and

- The paragraph makes no mention of "dependencies" (nor how they and
  "references" relate to one another), so I wouldn't have landed on it
  with a plain-text search.

To be fair, that paragraph _is_ linked to in the concept index under
"dependencies, run-time", so I perhaps should have found it a bit sooner
than I did.

Even so I think it might be good to add a brief section to the manual
that addresses directly how "dependencies" are managed by Guix, since I
doubt I'll be the last person who arrives with a traditional
package-management mindset and is confused about how Guix handles
things.

> How would you consider Knot more complex or problematic?

Knot seemed to be unique in that its distribution contains the source
code for both its executables and the libraries on which they depend,
all of which are meant to be built together in a single run.

That didn't strike me as unusual at first but searching through
gnu/packages, I found plenty of packages whose associated libraries were
distributed in a completely separate source distribution and thus were
naturally built as a separate package, which made clear the dependency
between them.

Not finding any obvious examples of packages dependent on their own
outputs made me start to think this was perhaps actually very uncommon
and not supported by Guix for this reason---though now I understand I
didn't find anything because stating this kind of dependency is
generally unnecessary, so I was basically searching for something that
doesn't exist.

[0] 
https://guix.gnu.org/manual/en/html_node/package-Reference.html#package-Reference
[1] https://guix.gnu.org/manual/en/html_node/Derivations.html#Derivations

-- 
Simon South
simon@simonsouth.net