Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[Katherine Cox-Buday]

Sarah Morgensen <iskarian@mgsn.dev> writes:

>> As an aside, when I started writing the importer, I didn't know it was a
>> possibility to just use the Go toolchain to help us generate packages. I
>> thought "the Guix way" was to do it all in scheme. It's nice that it's in
>> scheme, but I would love to leverage the Go toolchain instead.
>>
>> IMO, module resolution and graph dependencies in Go are complicated enough
>> that I'd much rather take the effort we put in trying to replicate and keep
>> up with the Go toolchain's behavior, and spend that effort elsewhere.
>>
>> Does anyone have opinions on this?
>
> Hmmm.  Setting aside whether or not we want to use external tools in
> general...
>
> If we use the Go tool, we shift the problem domain into "translating between
> `go' and Guix."
>
> For example: when Go downloads a module, it only saves the module, not the
> whole repository*, so we can't use that to generate the hash.  (* Except it
> does if you used GOPROXY=direct, but that's in the module cache, and only a
> bare repository.)

We could use the ~GOMODCACHE~ environment variable[0] to specify where these 
files are placed.

> Or, the fact that we import the latest available version of packages (unless
> --pin-versions is used), but Go uses exact versions selected with MVS [0].

Won't the importer import the import path you specify? The Go toolchain can 
also specify a ~@latest~ "version" to fetch the latest.

> You might also be interested in taking a look at Gentoo dealing with this
> translation problem [1].
>
> I'm not saying that this would necessarily be a bad tradeoff either, but it's
> definitely a tradeoff.

Agreed. I think the problem space of managing the translation between the 
toolchain and Guix is probably both smaller and less volatile.

I.e., the toolchain operations that dump dependency graph type stuff tend to 
support JSON output, and the download bits have options for where stuff gets 
stored on disk. As you know, the steps to go from an import path to a 
repository on-disk is complicated. If we could keep that complexity 
encapsulated upstream, which also happens to be the reference implementation, 
it could probably make the importer resilient to change and maybe even faster.

> Did you have something particular in mind as far as leveraging the Go tooling?

I haven't though too much about it, but ~GOPROXY=direct~ was my guess too. 
Thinking about this, would a shallow fetch into a bare repository be so bad?

It made me start wondering why our ~git-download~ type /doesn't/ have this 
behavior. Are we unnecessarily pulling down the entire commit history with all 
tags and branches of a large repository?

I don't even know what the philosophy is being doing this or not. Isn't the 
only commitment a Guix package makes for software at a particular 
version/commit that the store contains that immutable version of the source, 
not the complete repository?

Also, Guix supports all kinds of download types, one of which is zip files. The 
rust importer generates packages that fetch ~.tar.gz~'s from https://create.io. 
When I started writing the importer, I specifically kept the packages it 
generated decoupled from Google's module cache, but to make the comparison: the 
Go importer could do the same thing as the Rust importer: download zips from 
https://proxy.golang.org.

If this line of reasoning is wrong, it also looks like the toolchain respects 
standard VCS configuration. I wonder if there's a way to manipulate the fetches 
so that they're not shallow or bare.

TLDR: A shallow fetch into a bare repository might actually be /ideal/! It's 
small, and still directly connected to the package's actual source.

> [0] <https://golang.org/ref/mod#minimal-version-selection>
>
> [1] <https://github.com/golang/go/issues/35922> cmd/go: allow extraction of
> urls used to download dependencies
>
> --
> Sarah

[0] - https://golang.org/ref/mod#environment-variables
[1] - https://golang.org/ref/mod#private-module-repo-auth

-- 
Katherine