Re: Reproducing a Python project environment (using guix inferiors)

[zimoun]

Hi,

On Mon, 30 Nov 2020 at 14:39, Brandon Ellington <branjam4@gmail.com> wrote:

>>> | package of interest | guix commit  | status |
>>> |---------------------+--------------+--------|
>>> | python-matplotlib   | "7e06086522" | bad    |
>>> | python-pandas       | ce2cfcabfc   | bad    |
>>> | python-networkx     | 269f100330   | good   |
>>> | python-numpy        | 4d6ed794dd   | bad    |
>>> | python-scipy        | 02ddafef55   | good   |

Which version of <package of interest> do you need?


>> What do you mean by “bad”?
>
> For <package of interest>, guix has a definition for it at commit
> <guix commit>. When I declare an inferior channel at <guix commit> and
> try to build that channel, I note in the <status> column whether it
> builds ("good") or I got the [3] message following a backtrace ("bad").
> If it were to build, then I would use (first (lookup-inferior-packages
> (<<package-of-interest>>-inferior))) in a manifest for the
> package itself.

Because, I think that “bad” commits 7e06086522 (Jan 2018), ce2cfcabfc
(Feb 2018) and 4d6ed794dd (Jan 2018) simply pre-date the introduction of
«Inferior».  Therefore, they are not reachable by this mechanism.
That’s one of the motivations for the channel “guix-past”. :-)

Basically, the Backtrace should say something as:

--8<---------------cut here---------------start------------->8---
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
error: %guix-register-program: unbound variable
--8<---------------cut here---------------end--------------->8---


>> What did you run?  The command and the file(s)?
> I attached the file I used to define the channels and inferiors. As for
> the command, I just load these definitions piecemeal into the repl.

I see.

>>>     619:8  4 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>>    626:19  3 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>>     155:9  2 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>>    223:20  1 (proc #(#(#(#<directory (build-self) 1842aa0>) "/…" …) …))
>>> In unknown file:
>>>            0 (%resolve-variable (7 . %guix-register-program) #<direc…>)
>>> #+end_example

Ah, that’s the same backtrace message showed above. :-)


> I do not think so. I have three different scenarios I'm working with:
> 1. python-foo does not exist in guix :: So I use guix import to give me
> a package definition and work from there (so far this is okay, but if it
> weren't because of its dependencies I see where we might need to follow
> your hypothetical).
> 2. python-foo does exist in guix, but the desired version is in the past
> :: So I (attempt to) use a guix inferior from a channel where it was
> defined (this is where I have issues, but I assume that if guix had a
> definition for it, then python-bar and python-baz would both be
> available in the entire commit range that it defines python-foo).
> 3.
> python-foo does exist in guix, but I need a newer version :: I inherit
> from the old version, changing the necessary fields (like 1. I see where
> we would need to follow the hypothetical, but I haven't seen any missing
> dependencies errors).
>
> I successfully made the channel https://github.com/branjam4/cna-python
> (with packages from 3. and 1.) when I last worked on this. Hopefully
> having a successful channel build isn't deceiving me, but I thought
> given the successful build that I got lucky with needing to figure out
> dependencies. My primary block is this "unregisted program" error I get
> when using guix inferiors that I'd like to add to my manifest.

What I will do is: a) fix a Guix version b) create a channel containing
all the necessary variants (backporting dependencies if required, i.e.,
copy/pasting old package definition and fix them if they does not build
anymore) to build my short list of <packages of interest>.

This is more or less the strategy used to feed the channel “guix-past“.

And I will not use the inferior mechanism because it adds a lot of
complexity and will not solve your problem since old packages are not
reachable and/or your need to add old dependencies.


>> From my experience, one simple way to start is:
>>
>>   guix time-machine --commit=<old> \
>>        -- build -L <path/to/local/packages> <your-package>
>>
>> where <old> is an old Guix commit providing bunch of dependencies used
>> by <your-package> of interest defined in the folder
>> <path/to/local/packages>.
>> 
>> A channels.scm file with the option ’--channels/-C’ seems even better.
>
> Hmm, my end goal is to make /something/ that will be simple for
> people with minimal assumed experience in guix (my professor, for
> example) to run in a virtual machine with guix and direnv available, so
> they can pull a git repository, then follow code examples from the book
> without focusing on dependency building (there would be the initial cost
> of waiting for things to build the first time, though). Assuming
> time-machine works, is it straightforward to make a profile from it (or
> something else enabling direnv to understand what I want when I go into
> the cna-python directory)? That was something I missed while reading the
> time-machine part of the manual.

The time-machine is orthogonal with the way to distribute, IMHO.  It is
simply a easy CLI to fix the Guix version.

Usually, I do:

  guix describe -f channels > channels.scm

  edit my-manifest.scm # containing <package-of-interest>
  
  [.. hack my custom variants ..]
  [.. backport variant dependencies when I need for these variants ..]
  guix time-machine -C channels.scm
       -- build -L /path/to/variants/ -m my-manifest.scm
  [.. loop hack until it works ..]

and then I create what my audience expects, e.g., Docker image:

  guix time-machine -C channels.scm
       -- pack -f docker -L /path/to/variants/ -m my-manifest.scm

or relocatable tarball (pack -RR) or ‘system docker-image’ or whatever.


The /path/to/variants is a Git repo and I add the files channels.scm and
manifest.scm.  And now, it is easy to rebuild everything in the future.


> Though with recent guix pulls it fails at that same commit, saying that
> syntax has changed. I'm not as concerned about that at the moment
> compared with the guix inferior channels issue though.

The fail is about your channel.  Well, I better understand your initial
questions:

        From what I can gather, inferiors are the recommended way to declare
        packages defined in older revisions of guix.

>From my understanding, the answer is: it depends.

For commits after the complete introduction of Inferiors mechanism
(~July 2018, please read [1]), it is ok.  Be aware that each inferior
has a cost and if your package foo depends on X packages coming from
inferiors, then it means you are running under the hood X time: ‘guix
pull’ then ’guix build <dependency>’ which could require to also build
other dependencies, therefore installing foo could be long.

Explicitly, imagine that python-foo depends on python, python-bar and
python-baz.  Imagine that you use 2 inferiors, one for python-bar and
one for python-baz.  Now imagine that python-bar and python-baz depends
both on the package python.  Then, there is a high probability to build
3 times tiny variants of the package python.  And maybe the 3 packages
python-foo, python-bar and python-baz perfectly works with the same
variant of python.  And it is worse because that applies to all the
implicit dependencies (compilers and so on).

For commits before ~July 2018, the strategy is the one of ‘guix-past’
channel; which I roughly described above.

1: 
https://guix.gnu.org/en/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/


        But how would I know what a "good" guix inferior commit is for
        all of these packages?  Would using the definitions instead of
        asking guix to reproduce five different worlds for one package
        each be an anti-pattern?

Now, these questions and the other ones about ’guix-past’ are answered,
right?



All the best,
simon