[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Reproducing a Python project environment (using guix inferiors)
From: |
Brandon Ellington |
Subject: |
Re: Reproducing a Python project environment (using guix inferiors) |
Date: |
Tue, 01 Dec 2020 12:35:36 -0800 |
Thanks again for the reply simon!
zimoun <zimon.toutoune@gmail.com> writes:
> Hi,
>
> On Mon, 30 Nov 2020 at 14:39, Brandon Ellington <branjam4@gmail.com> wrote:
>
>>>> | package of interest | guix commit | status |
>>>> |---------------------+--------------+--------|
>>>> | python-matplotlib | "7e06086522" | bad |
>>>> | python-pandas | ce2cfcabfc | bad |
>>>> | python-networkx | 269f100330 | good |
>>>> | python-numpy | 4d6ed794dd | bad |
>>>> | python-scipy | 02ddafef55 | good |
>
> Which version of <package of interest> do you need?
You can kind of get it from [2] in my initial email, but here is a
cleaner table:
| package of interest | version (rec. minimum) |
|---------------------+------------------------|
| python | 3.6.5 | do not rebuild
|---------------------+------------------------|
| python-matplotlib | 2.0.2 | used to exist in guix
| python-pandas | 0.22.0 |
| python-networkx | 2.1 |
| python-numpy | 1.13.3 |
| python-scipy | 1.0.1 |
|---------------------+------------------------|
| python-pygraphviz | 1.3 | bump from current guix
| python-nltk | 3.2.5 |
|---------------------+------------------------|
| python-wikipedia | 1.4.0 | importing
| python-toposort | 1.5 |
| python-community | 0.10 |
| python-louvain | 0.14 |
|---------------------+------------------------|
| (author's own) | (not applicable) |
> Because, I think that “bad” commits 7e06086522 (Jan 2018), ce2cfcabfc
> (Feb 2018) and 4d6ed794dd (Jan 2018) simply pre-date the introduction of
> «Inferior». Therefore, they are not reachable by this mechanism.
> That’s one of the motivations for the channel “guix-past”. :-)
>
> [...]
>
> For commits before ~July 2018, the strategy is the one of ‘guix-past’
> channel; which I roughly described above.
>
> 1:
> https://guix.gnu.org/en/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/
Ah, lesson learned! I'll direct my attention toward guix-past then :)
> For commits after the complete introduction of Inferiors mechanism
> (~July 2018, please read [1]), it is ok. Be aware that each inferior
> has a cost and if your package foo depends on X packages coming from
> inferiors, then it means you are running under the hood X time: ‘guix
> pull’ then ’guix build <dependency>’ which could require to also build
> other dependencies, therefore installing foo could be long.
Indeed I noticed this. Building the two good channels was not too bad,
but I have not tried building networkx or scipy yet. As for your
hypothetical...
> Explicitly, imagine that python-foo depends on python, python-bar and
> python-baz. Imagine that you use 2 inferiors, one for python-bar and
> one for python-baz. Now imagine that python-bar and python-baz depends
> both on the package python. Then, there is a high probability to build
> 3 times tiny variants of the package python. And maybe the 3 packages
> python-foo, python-bar and python-baz perfectly works with the same
> variant of python. And it is worse because that applies to all the
> implicit dependencies (compilers and so on).
...I can imagine it, and that would be terrible yes. I thought python
wouldn't be a dependency of python packages in part for that reason
though. Or maybe I read this for something else like emacs and
extrapolated to python. Maybe I'm dreaming :P. In any case, the overall
idea makes sense, especially the implicit dependencies thing. For me, if
I went for full reproducibility including the older version of python,
then all these packages would work on 3.6.5, and I'd still have a lot to
learn about the implicit dependencies.
> What I will do is: a) fix a Guix version b) create a channel containing
> all the necessary variants (backporting dependencies if required, i.e.,
> copy/pasting old package definition and fix them if they does not build
> anymore) to build my short list of <packages of interest>.
>
> This is more or less the strategy used to feed the channel “guix-past“.
>
> And I will not use the inferior mechanism because it adds a lot of
> complexity and will not solve your problem since old packages are not
> reachable and/or your need to add old dependencies.
Got it. The last part of the last sentence is surprising (because the
mechanism must have been the right choice to someone/for something) but
everything else adds up.
> The time-machine is orthogonal with the way to distribute, IMHO. It is
> simply a easy CLI to fix the Guix version.
>
> Usually, I do:
>
> guix describe -f channels > channels.scm
>
> edit my-manifest.scm # containing <package-of-interest>
>
> [.. hack my custom variants ..]
> [.. backport variant dependencies when I need for these variants ..]
> guix time-machine -C channels.scm
> -- build -L /path/to/variants/ -m my-manifest.scm
> [.. loop hack until it works ..]
>
> and then I create what my audience expects, e.g., Docker image:
>
> guix time-machine -C channels.scm
> -- pack -f docker -L /path/to/variants/ -m my-manifest.scm
>
> or relocatable tarball (pack -RR) or ‘system docker-image’ or whatever.
>
>
> The /path/to/variants is a Git repo and I add the files channels.scm and
> manifest.scm. And now, it is easy to rebuild everything in the future.
Okay, definitely worth a try!
> Now, these questions and the other ones about ’guix-past’ are answered,
> right?
Yes, I appreciate you helping me out here! Now I can go down the correct
rabbit hole to build my project :)
--Brandon