[bug#39258] [PATCH v2 0/3] Xapian for Guix package search

[Arun Isaac]

>> This could be accomplished even with pre-rendering. Xapian provides
>> "slots" to store arbitrary strings with a document. Instead of storing
>> the pre-rendered document as a whole, we could store pre-rendered fields
>> in separate slots. Then, during `guix search` time, we can assemble the
>> result from these pre-rendered fields.
>
> I’m not sure I understand.  The index wouldn’t store pre-rendered
> strings for every possible terminal width, right?

No, it wouldn't. It would store a partially pre-rendered string, that is
without fill-paragraph. We run fill-paragraph at `guix search` time to
complete the rendering.

> I think we need to take the whole user experience into account, not
> just ‘guix search’.  ‘guix pull’ already feels very slow, and it’s a
> fairly common operation.  Conversely, ‘guix search’ takes roughly
> between 0.5 and 2 seconds and is an uncommon operation on a “slow
> path” (in the sense that when you’re searching for software, you’ll
> probably have to spend more than a couple of seconds to find what
> you’re looking for.)

I agree we can't compromise too much on `guix pull` performance.

> To me, adding 20–50 seconds on ‘guix pull’ would be undesirable.  :-/

Maybe I'm missing something here. guix pull takes around 40 minutes on
my machine. In comparison to that, is another 20-50 seconds (roughly 1
minute) a big deal? How much time would it be acceptable to spend on
building the Xapian index?

Also, is it possible to somehow provide substitutes for the Xapian index
so that the user does not have to actually build it locally during `guix
pull` time?

> I’m not sufficiently familiar with Xapian’s query language.  The
> examples I had in mind were:
> It’s not so much about regexps than it is about selecting individual
> fields.

I have totally not tested this, but I imagine that equivalent Xapian
queries might look something like:

>   guix search | recsel -p name -e 'license ~ "LGPL 3"'

guix search license:LGPL3

>   guix search crypto library | \
>     recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis

guix search crypto library AND (NOT ghc) AND (NOT perl) AND (NOT python)
AND (NOT ruby)

> What I meant was that we could use (statprof) to see whether/how Texinfo
> rendering/parsing can be optimized.

Oh, ok. I'll try this if we decide not to pre-render.

signature.asc
Description: PGP signature