[silpa-discuss] Re: Optimization idea

[Santhosh Thottingal]

On Sat, Apr 24, 2010 at 11:05 PM, Vasudev Kamath
<address@hidden> wrote:
> Ok then let me summarize our discussion
> 1. Dictionary files will have a index file containing alphabets and line 
> number
> for the words with that alphabets begin. (Assumption dictionary will have each
> words in seperate lines) Doubt- Will there be a single index file containing
> information on all dictionaries or multiple?

one index file for one language will be clean implimentation. let the
name of the file be [langcode].index

> 2. Format of the index file
>     a=1
>     b=2000

ok.

> 3. How file will be saved? normal file or python pickles

Normal file.

> 4. A generic python program for  generating this index file should be written.
>
> Once the index file is created how it will be used with in the silpa?..
> Currently when train is called if not already read the dictionary file is
> completely read and placed in the dictionary with langauge as key. What about
> the new approach ?

Instead of loading the dictionary load the index file initially.  And
we should not keep the wordlist in memory.
train should take an argument, which is  indexfile name.

But feel free to make your own changes to the approach. Just focus on
the improved results. Need not follow what I say ;)

-Santhosh