guix is the guildhall that we always wanted!

[Andy Wingo]

Hi all!

I've always wondered about the right solution for getting Guile users to
share code with each other.  At one point I thought that we would have a
Guile-specific packaging system like CPAN or NPM, but one with GNU
stow-like characteristics.  We had problems with C extensions though:
how do you build one?  Where do you get the compilers?  Where do you get
the libraries?

Guix (https://guixsd.org/) solves this in a comprehensive way.  From the
four or five bootstrap binaries, Guix can download and build the world
from source, for any of its supported architectures.  The result is a
farm of weirdly-named files in /gnu/store, but the transitive closure of
a store item works on any distribution of that architecture.

I got the feeling that Guix was what the guildhall should be long ago,
but something recent made it clear to me.  For background, consider
https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html,
the binary installation instructions for Guix.  These are directions for
how to install Guix on a "foreign" distro like Fedora.  Basically you
just untar a /gnu/store onto your system, make sure that your /var/guix
has some metadata so that Guix knows what's in the store, and there you
are.

The process of building that weird binary installation tarball was
always a bit ad-hoc though, geared to Guix's installation needs.

But it turns out that we can use the same strategy to distribute
reproducible binaries for any package that Guix includes.  Notably, if
you run:

  guix pack -S /opt/guile-fibers-1.0.0=/ guile-next guile-fibers 
glibc-utf8-locales

Then what you get is a tarball that has Guile, Fibers, and everything
they depend on.  It's safe to extract in / because it only adds files to
/gnu/store, and then it adds a symlink for /opt/guile-fibers-1.0.0.

Crucially, this "guix pack" is really an SDK for Guile and Fibers.
Fibers is a library.  You can now make an SDK for anything!!!!!!!!!!!  I
mean, I know, that's a lot of exclamation marks, but check it: you can
develop locally, say using

  /opt/guile-fibers-1.0.0/bin/guile -L /path/to/my/code

and it just works.  If you package up your code, you can make a Guix
package out of it and make a "guix pack" out of that closure and
distribute it to your users: REGARDLESS OF WHAT OPERATING SYSTEM THEY
RUN, AND WITH FULL SOURCE REPRODUCIBILITY.  It's huge.  It's the missing
piece of docker: not only do you get space-sharing (installing
guile-fibers-1.0.0 and my-app-built-on-guile-fibers-2.0 can share common
libraries as needed, instead of many huge image files), you also have
the keys to rebuild that image in a standard way.

So:

  * local development using packaged libraries: check.
  * distribution of your work to people without $package-manager: check.
  * easy addition of your code to a common NPM-like registry: ?

For this last bit, we have two options.  One is to add your package to
Guix.  It's relatively easy; here's what I added for Fibers:

    (define-public guile-fibers
      (package
        (name "guile-fibers")
        (version "1.0.0")
        (source (origin
                  (method url-fetch)
                  (uri (string-append "https://wingolog.org/pub/fibers/fibers-";
                                      version ".tar.gz"))
                  (sha256
                   (base32
                    "0vjkg72ghgdgphzbjz9ig8al8271rq8974viknb2r1rg4lz92ld0"))))
        (build-system gnu-build-system)
        (native-inputs
         `(("texinfo" ,texinfo)
           ("pkg-config" ,pkg-config)))
        (inputs
         `(("guile" ,guile-next)))
        (synopsis "Lightweight concurrency facility for Guile")
        (description
         "Fibers is a Guile library that implements a a lightweight concurrency
    facility, inspired by systems like Concurrent ML, Go, and Erlang.  A fiber 
is
    like a \"goroutine\" from the Go language: a lightweight thread-like
    abstraction.  Systems built with Fibers can scale up to millions of 
concurrent
    fibers, tens of thousands of concurrent socket connections, and many 
parallel
    cores.  The Fibers library also provides Concurrent ML-like channels for
    communication between fibers.

    Note that Fibers makes use of some Guile 2.1/2.2-specific features and
    is not available for Guile 2.0.")
        (home-page "https://github.com/wingo/fibers";)
        (license license:lgpl3+)))

OK, so this uses gnu-build-system, which requires a tarball; we need to
extend this to have a "guile-build-system" for modules that are just
Scheme, in which we just "guild compile" everything.  That way you can
have a repo on gitlab or whatever and you just specify the URL and the
revision and you are done.  I don't know if we can get around specifying
the sha256 when we specify the git revision; probably not a good idea in
light of the SHA1 breakage.  Anyway, that's a thing.

But while getting your package into guix is easier than you think, it's
not automatic.  I think there's room for a middle ground that's a bit
more decentralized than Guix, but more centralized than people using
GUIX_PACKAGE_PATH to add random directories of Guix package files.

So!  My proposal for this new "guildhall" would be:

    1. a web service

    2. on which users registers projects

    3. a project is a name + a git repository with a /package.scm file

    4. the package.scm contains Guix package definitions for that project

    5. the web service administers a git repository collecting those
       packages
       - without any hydra.gnu.org overhead
       - without any manual checks
       - in a form that you can just check out once and add to GUIX_PACKAGE_PATH

    6. adding a new tag to a project's git repo of the form vX makes a
       release X and updates the guildhall package
       - it could be the web service has to poll git repos
       - or maybe you have to invoke some command to update guildhall

    7. probably we need to steal many ideas from npm.org

You might ask, why is this better than the current guildhall?  It does
exist after all.  I respond in the following way:

    1. adding projects to guildhall currently takes manual intervention

    2. guildhall does not include C extensions, much less Guile itself

    3. guildhall has no story for deploying on systems without guildhall
       or even Guile

    4. guildhall currently misses some compilation phases (AFAIU?)

Open questions would be, what about other targets like macOS or Windows
or whatever?  There I don't know.  I suspect that if Guix becomes
popular enough, someone will make a thing whereby you can cross-compile
to macOS from GNU/Linux at least.  I know you can do that already with
MinGW.  As for native "guildhall" operation on these targets, I don't
know.  But I think that is a cost that people will pay for in time.

I think there's still room and value for pushing Guildhall packages into
Guix proper.  I don't know the right relationship, but the barriers to
Guix contribution are really low so I have a feeling this should just
work out; and that otherwise, the web service can automate a lot of
review on the specific packages in consideration.  Dunno.

But!  Who is going to build the guildhall v2.2? :-)

Andy