[bug#39258] [PATCH v2 0/3] Xapian for Guix package search

[zimoun]

On Mon, 9 Mar 2020 at 11:29, Ludovic Courtès <address@hidden> wrote:

> > Back to the topic: I believe that Xapian is a huge win both for the
> > shell and the future GUI :)
>
> It could be, but we need to consider all the aspects of the story,
> including the maintenance cost and overhead moved to ‘guix pull’.  So
> it’s not so much about “beliefs” at this point, but rather about
> demonstrating what can be done, and I’m glad Arun is exploring that
> space!

I agree.
What is currently tested with Xapian is:
 1- speeding up (or not) using an inverted index
 2- the accuracy using the state-of-art of information retrieval (BM25)

About 1- I do not have a strong opinion; even if I find "guix search"
terribly slow as I mentioned earlier (one year ago ;-)).

About 2- as I mentioned earlier, the 'relevance' function could be
improved. Currently, the score is computed only considering the
package itself and not the other packages (the words they use, their
number etc.). BM25 is the state-of-art using what I tried to explained
some time ago when I showed for example TF-IDF. The question is so
what the best move to improve the accuracy. And the improvement
necessarily uses a global index (of terms, at least). But on the other
hand, the improvement should not pay off because it would add
complexity and burden, more than the improvement itself.

Without testing, we cannot say. Thank you Arun for pushing forward.


All the best,
simon