Re: persistent reproducibility ?

[zimoun]

Hi!

Thank you for commenting and helping me to go forward.

>
> A guix package recipe, as you point, points to an exact revision/version
> of software (either through a version spec, or through a commit hash).
> The package's dependencies are specified as pure scheme variables that
> in turn resolve to package recipes.
>
> These package recipes in turn specify the exact revision of the
> dependencies.
>
> So for `foo` to be fully reproducible at t1 you require the package tree
> that corresponds to `foo`, `bar` & `baz` at t1.  This tree would carry
> all the meta information related to `foo` and it's exact dependencies.

I am not sure to understand, neither to be able to catch this exact
revision of the dependencies.

Let consider the example of the package Gmsh (I know a bit how it is developed):
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/maths.scm?id=v0.12.0-2125-g283f04b#n1097

Here, `gmsh' depends on `fltk', `gfortran', `lapack' etc.

If the commit hash of the Guix source tree is provided and known, then I agree.
It is possible to restore (build) the bit-identical `gmsh' at any time.
(assuming that the upstream is still available, which is another issue
and Ludo pointed out that some fallbacks by the Software Heritage are
in the pipes \o/)

At this point, I agree that on the paper, all the pieces are there and
it "just" lacks some UI glue.
And this point is a BIG step, compared to other solution on the
market. Thanks to Nix thesis and to you Guixers to implement it in
Guile. :-)


Now, as researcher, Alice forks Gmsh to explore a proof-of-concept of
her new algorithm.
Upstream refuses to include, for whatever reason, or Alice refuses to
push for whatever reason. Well this new algorithm will never be in the
Guix source tree.
Moreover, her new algorithm needs a specific patch of `lapack', and
this variant is too much specific to be included in the Guix source
tree.
At t1, Alice publishes all the materials, forked Gmsh commit hash and
its "reproducible hash", also the Guix source tree commit hash,
explained her `lapack' trick from the paper but she forgets to
precisely point the commit hash. Then, it is not possible to
bit-identically reproduce the forked Gmsh of the paper.

Ok, Guix cannot fix automagically what people does not do correctly. For sure!


But, should be somehow possible (and useful?) to add something to prevent that ?


I have in mind something in the flavor of `guix pack' at the source level.

When Alice releases her paper, she could release a binary pack of
everything useful to be able to run and in the same time a source pack
of everything useful to be able to build (over time). A kind of
derivation ? (it is what I called meta-package, hum?)


>
> I guess you may also need the correct version of Guix.  I'm not sure how
> that affects reproducibility, so I won't comment further on that.

I agree that experiments need to be performed.

>
> In any case, if you have `foo`, `bar` & `baz` as part of a guix source
> tree, then the revision of that source tree at t1 should be sufficient
> to reproduce `foo` at t1.

I agree.
And my question is: how to do if one of them is not part of a guix source tree ?

>
> I guess this is where the discussion of channels, and whether to package
> the programs for Guix itself or somehow outside become relevant, but
> again, I don't feel super qualified to comment on this.

I will browse the discussion about channels.

Thank you for your comment.
My feeling is that a feature lacks to Guix to deal with this
situation. Well, I will explore more details and see if I am able to
use your advices.

>
> All the meta-information required is carried by that revision.
>
> I wasn't sure what you meant by manual-metapackage, so I hope my answer
> provides you with the information you were looking for!

Writing by hand a package that provides all the necessary meta-data to
build bit-identical at any time.

If I install Guix, and for example I do "guix package -i foo -p pro".
Then, over months, I do operations as: create profiles, install,
remove, garbage collect, refresh, pull, etc.
Now, month after, I am not sure to be able to bit-identically build
the previous `foo' package.

Maybe I miss a point. Especially about `manifest'.

>
> I won't comment on the latter parts of your email, as it sounds like
> it's more related to fully reproducible environments rather than
> individual packages.  Personally I use manifests, but that might not be
> good enough for your requirements.

Thank you for your advices.
Time to play with keyboards :-)


All the best !

--
simon