bug#46330: Guile-provided GMP allocators interfere with GnuTLS

[Ludovic Courtès]

Fellow Debian hackers identified a bug causing memory corruption in
Nettle data structures used by GnuTLS when GnuTLS is used from Guile:

  https://bugs.debian.org/964284

In a nutshell, Guile installs its own GMP memory allocation routines
(when ‘scm_install_gmp_memory_functions’ is true, which is the default)
so that GMP allocates via libgc.  GnuTLS uses Nettle, which uses GMP, so
Nettle too ends up allocating via libgc; however, since pointers to that
memory are not scanned by libgc, they end up being reclaimed early.

In practice, memory corruption is relatively rare, to the point that we
did not notice it in Guix.  In Debian, it would lead to a failure of the
‘tests/reauth.scm’ test in GnuTLS.  With minor modifications to the
test, as noted in the thread above, I can reproduce it on Guix as well.

The thread above mentions possible workaround, but there’s nothing
satisfactory.

The longer-term solution is to use mini-GMP in Guile (which is also nice
as a way to reduce dependencies).

To be continued…

Ludo’.