Re: Updates for Go

[Katherine Cox-Buday]

Summary: these are good things to talk about. I think we should focus on 
using the current approach to get our Go ecosystem into a supported 
state before we talk about these things.

On 8/19/23 5:31 AM, Attila Lendvai wrote:
> at one point i tried to compile some large projects written in golang in a 
> reproducible way, and making sure that they use the exact same versions of all 
> their dependencies.
>
> in short: there's a philosophical mismatch between how guix and the golang crowd looks at 
> building go apps. guix likes to explicitly represent every dependency in the guix 
> namespace. golang has its own way of gathering the dependencies (and the nixos crowd 
> don't mind "vendoring" the dependencies).

Yeah, there's a lot of unpack in this space for sure. To repeat what I 
think you're saying to ensure I understand it, the way I would 
characterize it is that there are two concerns:

1. The version of module dependencies a module is built with

2. How the toolchain resolves dependencies


(1) is not a new or unique problem to Go. Distros and software have 
always had mis-aligned views on what specific versions a package should 
use. Distros want to maintain fewer packages, and developers want to 
reference specific versions of dependencies. Guix is in a a better 
position in that it's theoretically possible to carry many different 
versions of the same package without conflicts. My understanding is that 
the limiting factor now are the resources that go into doing that (e.g. 
build-farms, cognitive overhead, etc.).

I don't think (2) is actually a problem. A lot of projects like to 
"vendor" their dependencies (i.e. check in the version of their 
dependency in the same repository as the host module), and there's valid 
reasons to do that. Fortunately, Go's vendoring is strongly tied to 
specific versions and the code is checksummed to ensure the vendored 
copy isn't modified. So it follows that all Guix has to do is ignore 
vendored dependencies and refer to the exact same version of the Guix 
package.

That is to say, at least the way I've framed it, I think (1) is the root 
of any issue that exists.

> i've also worked on the golang importer (https://issues.guix.gnu.org/55242 
> which needs a bit more care). once it works reliably enough, then it'd be 
> possible to import the entire transitive closure of the dependencies into an 
> isolated namespace in guix, which would be a halfway solution between the two 
> conflicting philosophies.
Very nice! Our Go importer has improved so much, and further 
improvements such as these will make managing the Go ecosystem even easier.
> for now i've abandoned that endeavour in favor of adding binary packages to my 
> channel, but i made some notes on the way:
>
> https://issues.guix.gnu.org/43872
>
> a go-build-system patch
> http://issues.guix.gnu.org/50227
> discussion with iskarian
> https://logs.guix.gnu.org/guix/2021-08-31.log#024401

I don't think this is an issue specifically with Go. As long as I've 
been involved in software, there has been an issue with N software 
depending on M versions of the same thing.

> https://logs.guix.gnu.org/guix/2021-08-31.log#180940
> <lfam>    iskarian: If you search for my name and "go" or "golang" in the 
> mailing list archives, you should find a lot of discussion
> https://savannah.gnu.org/users/lfam
> <lfam>    Here's a more recent message from me: 
> https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00227.html
> <iskarian>        Ah, I see I've unknowingly repeated you! 
> https://lists.gnu.org/archive/html/guix-patches/2021-08/msg01222.html
> <lfam>    Heh, it's gratifying that someone else came to the same conclusion. 
> It means I wasn't totally in the weeds

The smallest divisible unit of a Go repository that is independently 
distributable is now a Go module. Modules are what are resolvable, 
versioned, and check-summed. As a rule that may have exceptions: Guix 
packages should neither encapsulate anything larger nor smaller than 
that. Some of the messages you're referencing are right around the time 
modules were being reified into the Go ecosystem.

I think there is an opportunity and need for Guix to try and reach a 
consensus on what our primitives and approaches are for the Go ecosystem 
and write them down in the manual. I think our current approach is 
workable, but there's obviously still some confusion and maybe debate to 
be had.

Having said all of that, I think we should focus on using our current 
approach to get everything compiling on a supported version of Go. I 
think our packages and substitutes are probably carrying CVEs that have 
been fixed upstream, and, in my opinion, we need to resolve that ASAP.

WDYT?