Re: RFC: libgit2 is slow/inefficient; switch to git command?

[Philip McGrath]

Hi,

On Tue, Nov 22, 2022, at 10:39 AM, zimoun wrote:
> Hi,
>
> On Mon, 21 Nov 2022 at 21:21, Maxim Cournoyer <maxim.cournoyer@gmail.com> 
> wrote:
>
>> Given that:
>>
>> * the git CLI doesn't suffer from such poor performance;
>> * This kind of performance problem has been known for years in libgit2
>>   [0] with no fix in sight;
>> * other projects such as Cargo support using the git CLI and that
>>   projects are using it for that reason [1];
>
> And I would add the lack of «Support for shallow repositories» [1].
>
> 1: <https://github.com/libgit2/libgit2/issues/3058>
>

>
> PS: For the record, Software Heritage, which ingests *a lot* of Git
> repositories, relies on Dulwhich [2] (pure Python implementation), IIUC.
>
> 2: <https://www.dulwich.io/>

Along those lines, there’s an implementation of clone/checkout in pure Racket 
(for the package manager) that could probably be ported to Guile relatively 
easily. I’d expect libgit2 to be faster for the things that it supports, but 
the Racket implementation does support shallow checkout, so it might pay off if 
that skips a lot of work.

Code: 
https://github.com/racket/racket/blob/master/racket/collects/net/git-checkout.rkt
Docs: https://docs.racket-lang.org/net/git-checkout.html

(More broadly, I haven’t investigated performance issues, but my basic 
inclination would be toward improving libgit2 over running the git executable.)

-Philip