bug#44053: Poor profile generation performance on spinning disks

[Maxim Cournoyer]

Hello Simon,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Sun, 18 Oct 2020 at 05:37, Maxim Cournoyer <maxim.cournoyer@gmail.com> 
> wrote:
>
>> On my home machine, will is still relying on 2 x 1 TB spinning drives in
>> RAID1, rebuilding my user profile, which contains 182 entries, takes on
>> average about 20 minutes, even when there are no packages to be built:
>
> The database build time seems correlated to the number of packages in
> the very profile.  Well, I have not benchmarked to see if it is
> linear, quadratic; or worse or better.

It is.  Small profiles operations are much faster than large profiles.
Attached is a slightly trimmed version of the manifest I currently use
for my user profile, if you'd like to benchmark it on your system.

manifest.scm
Description: Text document

I'm betting that the part to optimize is:

--8<---------------cut here---------------start------------->8---
  ;; Make the symlinks.
  (union-build output inputs
               #:symlink symlink
               #:log-port (%make-void-port "w"))
--8<---------------cut here---------------end--------------->8---

from the 'build-profile' procedure in (guix build profiles).

>
>> --8<---------------cut here---------------start------------->8---
> [...]
>> building XDG MIME database...
>> successfully built 
>> /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
>> successfully built 
>> /gnu/store/cir84qj587i6is4akgqand7ahg9bj938-xdg-mime-database.drv
>> /gnu/store/j0bznlj2ibnhirijhnwpkkxzz4qfk8wb-xdg-mime-database
>>
>> real    1m7.344s
>> user    0m1.331s
>> sys     0m0.053s
>> --8<---------------cut here---------------end--------------->8---
>
> Well, it is all clear to me, but I do not know if we can do better
> because these 2 XDG updates seem relying on the external binaries
> "/bin/update-desktop-database" and "/bin/update-mime-database".  I
> have not timed the function 'xdg-desktop-database' but all the time
> should be spent on these, I bet. :-)

They do, and this is indeed what takes time.  But, this doesn't stop
motivated wizards from implementing our own database generator in Scheme
that would do the same.  Ludovic had done just that for the man-db
database generator.  The custom code in Guix can do it much faster that
it initially could back when it was relying on man-db to do that (that's
also probably the reason why we have issues such as
https://issues.guix.gnu.org/38838; perhaps the generated database is
slightly different -- but that's another topic :-)).

Thanks,

Maxim