guile-user
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HELP] a search engine in GNU Guile


From: Amirouche Boubekki
Subject: Re: [HELP] a search engine in GNU Guile
Date: Fri, 23 Sep 2016 07:52:36 +0200
User-agent: Roundcube Webmail/1.1.2

Héllo,


I made some progress regarding culturia. Now there is a web
interface available at hypermove.net [0].

[0] http://hypermove.net/?query=guile+algorithms+-wingolog

Now you can:

- search a bunch of website related to Guile I quickly selected
- use minus char "-" to exclude some keyword from the result set
- index small-ish domains

There is no UI yet to add your website to the index, if you want
your website to be part of the experiment reach me.

I tried two search backends. One based on a graph database which
is more versatile but much slower and another based on inverted
index. The latter is the one that is used currently. It's fast
enough right now.

What will be done next:

- Add wikipedia, wiktionary, stackoverflow and hackernews to the
  database

- Improve how results are displayed

- Remove feeds from results

- Index pdf

- Remove duplicate entries from the results

- Paginate

- Make use of fibers to run the http server

- Better crawling algorithm that support updating the index

At some point there will be a 0.1 release, not sure when.

The code is also available at framagit [1] the web interface
is implemented in a single module [2].

[1] https://framagit.org/a-guile-mind/culturia
[2] https://framagit.org/a-guile-mind/culturia/blob/master/src/web.scm

Happy hacking!

On 2016-08-13 17:25, Amirouche Boubekki wrote:
Héllo,


The goal of Culturia is to create a framework that makes it easy
to tape into Natural Language Understanding algorithms (and NLP)
and provide an interface for common tasks.

Culturia is an intelligence augmentation software.

It's primary interface is a search engine. Another important aspect
of the project is that it wants to be useable offline as such it will
come with infrastructure to dump, load and store dataset for offline use.

The current state of the project can be described as a big ball of mud.
There is a tiny search engine with crawling skills and that's basically
all of it.

The immediate changes that should happen are in order of preference:

- offline stackoverflow (cf. sotoki.scm) and use the generated
  website to create a zim for kiwix [0]. This is great occasion to
  show how great GNU Guile is!
- port whoosh/lucene to guile to improve text search
- offline hackernews, wikidata, wikipedia, wiktionary
- implement BM25f

Culturia is a reference to _Culture and Empire_ by Pieter Hintjens.

It has a sparse documentation is available online [1].
It's hosted on github [2] (This can change, if contributors
don't want to use github).

The TODO list is big, here is some stuff that needs to be done:

- finish GrammarLink bindings
- create sophia [3] bindings
- implement TextRank
- implement PageRank
- create a GUI using sly or html
- explore ways to easily share database among several processus

And many other things! Newbies are accepted obviously!

Send me a mail or use #guile @ irc.freenode.net, I am amz3.


Happy hacking!


[0] http://www.kiwix.org/wiki/Main_Page
[1] https://amirouche.github.io/Culturia/doc/
[2] https://github.com/amirouche/Culturia
[3] http://sophia.systems/

--
Amirouche ~ amz3 ~ http://www.hyperdev.fr



reply via email to

[Prev in Thread] Current Thread [Next in Thread]