Re: Reproducing a Python project environment (using guix inferiors)

[Brandon Ellington]

Thanks for replying simon!

zimoun <zimon.toutoune@gmail.com> writes:

> I am not sure to understand what you are naming “inferior” here.

Sorry for the ambiguity, was not being self-deprecating. I meant
attempts to use certain guix inferiors as described in the guix manual
failed (from Package Management > Inferiors):

> Sometimes you might need to mix packages from the revision of Guix
> you’re currently running with packages available in a different
> revision of Guix.  Guix “inferiors” allow you to achieve that by
> composing different Guix revisions in arbitrary ways.

So answering your questions about what I did:
>> [4]
>> | package of interest | guix commit  | status |
>> |---------------------+--------------+--------|
>> | python-matplotlib   | "7e06086522" | bad    |
>> | python-pandas       | ce2cfcabfc   | bad    |
>> | python-networkx     | 269f100330   | good   |
>> | python-numpy        | 4d6ed794dd   | bad    |
>> | python-scipy        | 02ddafef55   | good   |

> What do you mean by “bad”?

For <package of interest>, guix has a definition for it at commit
<guix commit>. When I declare an inferior channel at <guix commit> and
try to build that channel, I note in the <status> column whether it
builds ("good") or I got the [3] message following a backtrace ("bad").
If it were to build, then I would use (first (lookup-inferior-packages
(<<package-of-interest>>-inferior))) in a manifest for the
package itself.

> What did you run?  The command and the file(s)?
I attached the file I used to define the channels and inferiors. As for
the command, I just load these definitions piecemeal into the repl.

(define-module (cna-manifests guix-inferiors)
 #:use-module (guix inferior)
 #:use-module (guix channels))

(define python-matplotlib-channel
  ;; This is the old revision from which we want to
  ;; extract python-matplotlib.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git";)
         (commit
          "7e06086522"))))

(define-public python-matplotlib-inferior
   ;; An inferior representing the revision for python-matplotlib.
   (inferior-for-channels python-matplotlib-channel))

(define python-pandas-channel
  ;; This is the old revision from which we want to
  ;; extract python-pandas.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git";)
         (commit
          "ce2cfcabfc"))))

(define-public python-pandas-inferior
   ;; An inferior representing the revision for python-pandas.
   (inferior-for-channels python-pandas-channel))

(define python-networkx-channel
  ;; This is the old revision from which we want to
  ;; extract python-networkx.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git";)
         (commit
          "269f100330"))))

(define-public python-networkx-inferior
   ;; An inferior representing the revision for python-networkx.
   (inferior-for-channels python-networkx-channel))

(define python-numpy-channel
  ;; This is the old revision from which we want to
  ;; extract python-numpy.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git";)
         (commit
          "4d6ed794dd"))))

(define-public python-numpy-inferior
   ;; An inferior representing the revision for python-numpy.
   (inferior-for-channels python-numpy-channel))

(define python-scipy-channel
  ;; This is the old revision from which we want to
  ;; extract python-scipy.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git";)
         (commit
          "02ddafef55"))))

(define-public python-scipy-inferior
   ;; An inferior representing the revision for python-scipy.
   (inferior-for-channels python-scipy-channel))

>> [3]
>> #+begin_example scheme
>> In current input:
>>      12:3 12 (_)
>> In guix/store.scm:
>
> [...]
>
>>     619:8  4 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>    626:19  3 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>     155:9  2 (_ #(#(#(#<directory (build-self) 1842aa0>) "/gnu…" …) …))
>>    223:20  1 (proc #(#(#(#<directory (build-self) 1842aa0>) "/…" …) …))
>> In unknown file:
>>            0 (%resolve-variable (7 . %guix-register-program) #<direc…>)
>> #+end_example

>> But how would I know what a "good" guix inferior commit is for all of
>> these packages?
>
> Let’s fix ideas and let’s say you are interested by the package
> python-foo at the version 1.2.3.  This package depends on python-bar at
> version x.y.z available at commit range 12345-67890 and on python-baz at
> version m.n available at commit range ABCDE-FGHIJ.
>
> Then there is no commit where python-bar *and* python-baz are both at
> the correct versions (resp. x.y.z and m.n.).  In other words, the 2
> commit ranges 12345-67890 and ABCDE-FGHIJ do not overlap.
>
> Is it your use case?

I do not think so. I have three different scenarios I'm working with:
1. python-foo does not exist in guix :: So I use guix import to give me
a package definition and work from there (so far this is okay, but if it
weren't because of its dependencies I see where we might need to follow
your hypothetical).
2. python-foo does exist in guix, but the desired version is in the past
:: So I (attempt to) use a guix inferior from a channel where it was
defined (this is where I have issues, but I assume that if guix had a
definition for it, then python-bar and python-baz would both be
available in the entire commit range that it defines python-foo). 3.
python-foo does exist in guix, but I need a newer version :: I inherit
from the old version, changing the necessary fields (like 1. I see where
we would need to follow the hypothetical, but I haven't seen any missing
dependencies errors).

I successfully made the channel https://github.com/branjam4/cna-python
(with packages from 3. and 1.) when I last worked on this. Hopefully
having a successful channel build isn't deceiving me, but I thought
given the successful build that I got lucky with needing to figure out
dependencies. My primary block is this "unregisted program" error I get
when using guix inferiors that I'd like to add to my manifest.

> From my experience, one simple way to start is:
>
>   guix time-machine --commit=<old> \
>        -- build -L <path/to/local/packages> <your-package>
>
> where <old> is an old Guix commit providing bunch of dependencies used
> by <your-package> of interest defined in the folder
> <path/to/local/packages>.
> 
> A channels.scm file with the option ’--channels/-C’ seems even better.

Hmm, my end goal is to make /something/ that will be simple for
people with minimal assumed experience in guix (my professor, for
example) to run in a virtual machine with guix and direnv available, so
they can pull a git repository, then follow code examples from the book
without focusing on dependency building (there would be the initial cost
of waiting for things to build the first time, though). Assuming
time-machine works, is it straightforward to make a profile from it (or
something else enabling direnv to understand what I want when I go into
the cna-python directory)? That was something I missed while reading the
time-machine part of the manual.

>> Believing guix could more faithfully/cleanly reproduce the environment
>> from the book than my former approach using pip, I:
>> + imported/wrote definitions for packages not in guix
>
> Usually, this is done via the option “--load-path” or Custom Channels.
>
> <https://guix.gnu.org/manual/devel/en/guix.html#Creating-a-Channel>

True, I have older generations with my repo as a channel:
#+BEGIN_EXAMPLE bash
  cna-python 35290a1
    repository URL: https://github.com/branjam4/cna-python.git
    branch: master
    commit: 35290a12d09ad528babba6052d37bcd09bd4c9a4
  guix 0e1b095
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 0e1b0958bde5ccc34a4fed9a09cf949d5f9c9519
#+END_EXAMPLE

Though with recent guix pulls it fails at that same commit, saying that
syntax has changed. I'm not as concerned about that at the moment
compared with the guix inferior channels issue though.

>> + wrote inferiors for older versions of packages that were in guix.
>
> You might be interested by this Channel:
>
>   <https://gitlab.inria.fr/guix-hpc/guix-past>
>

I see. First looked at this channel a few days ago, but went straight to
its package definitions and only skimmed the readme (which at the
beginning mentioned using software from ten years ago, throwing me off
from accepting it for my use case). Under the "Hacking" section it says:

> It makes most sense to add software dating back to before 2019, around
> the time where time travel became possible.

The packages I'm trying to reproduce would have been current in mid
2017. Does that mean neither guix inferiors nor guix time-machine will
work reliably for commits in that year?

> BTW, a tool to find the range of commit where one specific version is
> available is the Data Service.  Give a look at:
>
> <https://data.guix.gnu.org/repository/1/branch/master/package/python-scipy/output-history>
>

Thanks, this definitely seems more robust than my git log grepping!

> I have tried to write a script for my personal needs running via “guix
> repl”.  It fetches the JSON file from the Data Service, which give the
> commit range for the required version per package.  Then I have not
> finished it (yet) but the idea was to be able to spot out the commit
> providing all the packages at the required versions, i.e., the
> intersection.  Pieces of the necessary material is in guix/git.scm.
>
> <https://git.savannah.gnu.org/cgit/guix.git/tree/guix/git.scm#n444>

Ah, definitely intriguing. While I was on my grepping expedition I
wondered whether I could use one or two old guix commits instead of
five, so I like this concept.

> Being on a foreign distro does not change.  The main difference between
> between foreign and native is only about services, not packages.

Figured, thank you for confirming.

--Brandon