Re: File search

[Antoine R. Dumont (@ardumont)]

Hello,

As mentioned last week (on irc), I've improved a bit the implementation
as per the last discussions in the thread. Please, find enclosed the
patch with those changes (hopefully a tad better attached than last
time...).

Here is the rough changelog:

- The local db cache is now versioned. Migration will transparently
  happen for users at each index command calls (if need be).

  Note: Care should be taken by devs to provide the migration script
  step for each db schema bump (examples inside). The rest will happen
  on its own.

- The cli parsing got rewritten to be more flexible (inspired from
  existing code from guix, notably `guix home`).

- We can now choose the indexation method using the
  `--with-method={store|manifests}` flag. The "manifests" method is the
  default, seel the help message for more details).

- Finally, the indexation methods are displayed using a progress bar.

Heads up, I did not yet address the "output" part. Thanks @zimoun for
the clarification btw ;)

> In the package case, the number of packages is known ahead.

@civodul For the index 'store' implementation, ^ I did not find that
information.

So, as a costly implementation detail, I'm folding over all packages
first to know the total number of packages (for the progress bar). And
then another round trip to actually do the insert.

I don't like it at all. Plus, that number seems off to me (21696)
packages in regards to the number of packages indexed (522). So, if you
could have a rapid look to fix or tell me what's wrong, that'd be great.
I'm pretty sure it will hit you immediately (while i still do not find
it ¯\_(ツ)_/¯ ;).

----

Here is a rapid sample of the current command usage:

--8<---------------cut here---------------end--------------->8---
$ guix index --version
Extension local cache database:
- path: /home/tony/.cache/guix/index/db.sqlite
- version: 2

guix index (GNU Guix) 5ccb5837ccfb39af4e3e6399a0124997a187beb1
Copyright (C) 2022 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ guix index --with-method=store --db-path=/tmp/db/db.sqlite
Registering 21696 packages  [########                                 ]

$ guix index search git
...
git-minimal@2.38.1   
/gnu/store/xf734fz3jihgc5x4979ypyaxn8aday1k-git-minimal-2.38.1/bin/git
git@2.38.1           
/gnu/store/wx965ym3c5fwbcdp7i9lvzad3479vv7m-git-2.38.1/libexec/git-core/git
git@2.38.1           
/gnu/store/wx965ym3c5fwbcdp7i9lvzad3479vv7m-git-2.38.1/etc/bash_completion.d/git
git@2.38.1
/gnu/store/wx965ym3c5fwbcdp7i9lvzad3479vv7m-git-2.38.1/bin/git

$ guix index --help
Usage: guix index [OPTIONS...] [search FILE...]
Without argument, indexes (package, file) relationships from the machine.
This allows indexation with 2 methods:

- out of the local manifests. This is the fastest implementation but this
indexes less packages. That'd be typically the use case for user local
indexation.

- out of the local store. This is slower due to implementation details (it
discusses with the store daemon for one). That'd be typically the use case for
building the largest db in one of the build farm node.

With 'search FILE', search for packages installing FILE.

Note: Internal cache is located at ~/.cache/guix/index/db.sqlite by default.
See --db-path for customization.

The valid values for OPTIONS are:

  -h, --help          Display this help and exit
  -V, --version       Display version information and exit
  --db-path=DIR       Change default location of the cache db

  --with-method=METH  Change default indexation method. By default it uses the
                      local "manifests" (faster). It can also uses the local
                      "store" (slower, typically on the farm build ci).
The valid values for ARGS are:

  search FILE     Search for packages installing the FILE (from cache db)

  <EMPTY>         Without any argument, it index packages. This fills in the
                  db cache using whatever indexation method is defined.
Report bugs to: bug-guix@gnu.org.
GNU Guix home page: <https://guix.gnu.org>
General help using Guix and GNU software: <https://guix.gnu.org/en/help/>

--8<---------------cut here---------------end--------------->8---

Hope you'll find it mostly to your taste!

Note: I gather we'll rework the commits at some point (when it's ready)
so I did not bother too much right now.

Cheers,
--
tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------
gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8

add-extension-guix-index.patch
Description: Text Data

Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> "Antoine R. Dumont (@ardumont)" <ardumont@softwareheritage.org> skribis:
>
>> |-----------+-------------+----------+----------|
>> | Iteration | Host System | Time (s) | Packages |
>> |-----------+-------------+----------+----------|
>> | 1st       | Debian      |   121.88 |      284 |
>> |           | Guix System |   413.55 |      749 |
>> |-----------+-------------+----------+----------|
>> | 2nd       | Debian      |      1.3 |      101 |
>> |           | Guix System |    11.54 |      354 |
>> |-----------+-------------+----------+----------|
>
> Ah, that’s a significant difference.
>
> I guess we can keep both methods: the exhaustive one that goes over all
> packages, and the quick one.  Then we can have a switch to select the
> method.
>
> Typically, we may want to use the expensive one on the build farm to
> publish a full database, while on user’s machines we may want to default
> to the cheaper one.
>
>>> Oh, and progress bars too.
>>
>> I'm a bit unsettled on this. Hopefully it was mostly a joke ;)
>
> It wasn’t.  :-)
>
> In the manifest case, we get ‘all-profiles’ is almost instantaneous, so
> we immediately known the number of manifests we’ll be working on.
>
> In the package case, the number of packages is known ahead.
>
> The (guix progress) module provides helpers.
>
> But anyway, that’s more like icing on the cake, we can leave that for
> later.
>
> Thanks,
> Ludo’.

signature.asc
Description: PGP signature