Re: babelia

[Amirouche Boubekki]

Le sam. 16 nov. 2019 à 11:06, Amirouche Boubekki
<address@hidden> a écrit :
>
> I restarted working on my personal search engine.
>
> It used to be called culturia [0] with too many planned features. At
> some point, I called it asylum [1] and focused on personal knowledge
> base aspects and the last iteration was called gotofish [2]
>
> [0] https://framagit.org/a-guile-mind/culturia
> [1] https://framagit.org/a-guile-mind/culturia.next
> [2] https://git.sr.ht/~amz3/guile-gotofish
>
> I learned much from all this projects. In particular, I learned that
> it will be a long long long project, even if I focus only on "personal
> search engine" line of work.
>
> The last iteration, gotofish, was not too bad even if it has bitrot.
> Based on my research and practical experiment, it seems very clear
> that there is no workaround the use of map-reduce, that might be known
> as n-par-for-each [3].
>
> [3] 
> https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html#index-n_002dpar_002dfor_002deach
>
> I made a prototype similar to that n-par-for-each, except it works
> with guile-fibers, is asynchronous and works with a shared pool of
> threads instead of spawning N threads for each incoming query like
> gotofish does.
>
> Related blog post: https://hyper.dev/blog/on-the-road-to-babelia.html
>
> If you want to help or discuss those matters, do not hesitate to reply
> to this message.

I forgot to add that there is several big-ish tasks that can be
tackled in parallel (see the above blog post). In particular, a parser
for wet or warc files, see https://en.wikipedia.org/wiki/Web_ARChive.
This is the most common format of the output of crawlers e.g.
http://commoncrawl.org/