Re: [RFC v3 13/32] rust: use vendored-sources

[Daniel P . Berrangé]

On Thu, Sep 09, 2021 at 05:02:01PM +0100, Peter Maydell wrote:
> On Tue, 7 Sept 2021 at 13:32, <marcandre.lureau@redhat.com> wrote:
> >
> > From: Marc-André Lureau <marcandre.lureau@redhat.com>
> >
> > Most likely, QEMU will want tighter control over the sources, rather
> > than relying on crates.io downloading, use a git submodule with all the
> > dependencies. However, cargo --offline was added in 1.36.
> >
> > "cargo vendor" helps gathering and updating the dependencies.
> >
> > Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > ---
> >  configure                 | 8 ++++++++
> >  meson.build               | 7 ++++++-
> >  .cargo/config.toml.in     | 5 +++++
> >  .cargo/meson.build        | 5 +++++
> >  .gitmodules               | 4 ++++
> >  rust/vendored             | 1 +
> >  scripts/archive-source.sh | 2 +-
> >  scripts/cargo_wrapper.py  | 1 +
> >  8 files changed, 31 insertions(+), 2 deletions(-)
> >  create mode 100644 .cargo/config.toml.in
> >  create mode 100644 .cargo/meson.build
> >  create mode 160000 rust/vendored
> 
> So, this is a lot of extra code in a submodule. Historically we've
> found that submodules are a colossal pain, and so I think we should
> think about whether we really want to have all our rust dependencies
> in a submodule forever.
> 
> I am definitely only at the beginner stage with Rust, but I think
> we should have a discussion about what the different alternative
> options are here, and what we want to achieve, so that we know
> why we're doing this and what we're gaining from the pain...
> 
> For instance, could we instead commit Cargo.lock in git and
> use that to nail down specific versions of the dependencies ?
> 
> FWIW, the "why submodules" for the C dependencies we ship
> like that is basically
>  * C doesn't have a package manager, so if we need a dependency that
>    distros don't ship then we need to wrap it up and provide it ourselves
>  * where we ship binary blobs (guest BIOS etc) we want to also ship
>    the source code for those blobs
> I think for Rust dependencies those don't really apply.

Even for our existing non-rust usage of submodules, it is very
borderline questionable whether the benefit outweighs the pain
they are causing us. I frequently wish we would just go "cold
turkey" and drop all our submodules and spin off all the bundled
blobs into separate downloads, despite the disruption it would
cause in the short term.

> Overall, I think that to the extent that we can look like a "normal"
> user of Rust, that's a good plan. Distros may well want to be able
> to do "build against our packaged rust stuff rather than downloading
> from crates.io" but I imagine they have machinery for that already;
> if we act like most other Rust programs we have better chances of
> not breaking that machinery.

Yes, distros do have machinery for this, although it is often
hard to fit in with it when you have a mixed language project.
Their machinery typically assumes pure single language project,
so would work nicer if any QEMU rust pieces were separately
released from the rest of QEMU. Obviously this is easier said
than done since QEMU tends towards a monolothic repo approach
historically.

> We do already effectively do "download code when QEMU is built" --
> the makefile invokes scripts/git-submodule-update which pulls
> down submodule code. (Thanks to Ian for pointing out this framing
> of the question.)
> 
> (I'm not personally a fan of the "download everything from crates.io"
> Rust ecosystem, but it is what it is, and wishing the Rust world
> worked more like a trad Linux-distro-provides-all-your-dependencies
> isn't, alas, going to make it so :-))

Yes, I'm inclined to agree here. For better or worse the battle is
over and "download everything from <repo> on the fly" is the accepted
approach for pretty much all modern languages. The language specific
repo essentially is the OS distro from their POV.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|