Re: Conda environments and reproducibility

[Konrad Hinsen]

Hi Hugo,

> Thanks. Those dependencies indeed do not contain the hashes, so it is
> probably created with "conda env export --no-build".

I can't say, I didn't set up this environment.

> I think such a file without build hashes would probably be what you want
> when you are giving a course, because it would allow students to install
> these exact versions of the packages, but build for their specific
> environment (e.g. Linux / macOS / Windows). It would provide limited

That was exactly our objective. And we knew that in theory,
reproducibility and multi-platform are incompatible. We just hoped that
the conda approach would work long enough for the purposes of our
course. It didn't.

> reproducibility in the future, as you noticed. I guess you'd want three
> sets of environment files for a conda environment for a course:

Sounds good... in theory. In practice, we'd have to explain the reasons
for these three environment files. Which we could do at best at the end
of the course. And even then, it would have been a difficult task, as we
couldn't go into all the details in a course aimed at junior researchers
with little technical background in computing.

> However, nowadays everyone can run linux, either directly, or through WSL
> (windows subsystem for linux), or through containers. And everyone knows
> how to do this, and it is integrated in IDE's and such. So conda isn't
> really necessary anymore.

Indeed.

We are currently working on a follow-up for dealing with reproducibility
at scale (big data, complex code, long computations). We decided to give
up multi-platform, and concentrate on Linux (explaining why). The two
approaches to reproducible environment we plan to cover are

 - Docker containers from reproducible Dockerfiles, based on Debian snapshots
 - Guix

The point is that, once you accept that Docker images are acceptable
only when reproducible, Guix appears as a simplification.

> I agree with you on a philosophical level; ultimately understanding
> everything would be easier with guix. But we aren't there yet, I don't
> understand most of the guix packages I've looked at. That is probably
> because my guile/scheme skills are lacking.

Maybe not. A big part of the complexity of Guix packaging is the need to
patch most software, in order to make its build reproducible and in
order to remove tacit dependencies in the build process on FHS
conventions.

Once Guix becomes the norm in the Linux world, the next step is to
encourage software developers to develop with Guix in mind. Produce
software that doesn't require any patches to compile under Guix.
World dominance is in sight!  ;-)

Cheers,
  Konrad.
-- 
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: konrad DOT hinsen AT cnrs DOT fr
http://dirac.cnrs-orleans.fr/~hinsen/
ORCID: https://orcid.org/0000-0003-0330-9428
Twitter: @khinsen
---------------------------------------------------------------------