[silpa-discuss] Re: Optimization idea

[Vasudev Kamath]

Yeah thats good enough information for me to start. I'll start working on 
indexing script as stand alone then once i finished we can start with changes 
to silpa modules by branching the repository.

Thanks and Regards
Vasudev Kamath

On Saturday 24 Apr 2010 11:13:45 pm Santhosh Thottingal wrote:
> On Sat, Apr 24, 2010 at 11:05 PM, Vasudev Kamath
> 
> <address@hidden> wrote:
> > Ok then let me summarize our discussion
> > 1. Dictionary files will have a index file containing alphabets and line
> > number for the words with that alphabets begin. (Assumption dictionary
> > will have each words in seperate lines) Doubt- Will there be a single
> > index file containing information on all dictionaries or multiple?
> 
> one index file for one language will be clean implimentation. let the
> name of the file be [langcode].index
> 
> > 2. Format of the index file
> >     a=1
> >     b=2000
> 
> ok.
> 
> > 3. How file will be saved? normal file or python pickles
> 
> Normal file.
> 
> > 4. A generic python program for  generating this index file should be
> > written.
> >
> > Once the index file is created how it will be used with in the silpa?..
> > Currently when train is called if not already read the dictionary file is
> > completely read and placed in the dictionary with langauge as key. What
> > about the new approach ?
> 
> Instead of loading the dictionary load the index file initially.  And
> we should not keep the wordlist in memory.
> train should take an argument, which is  indexfile name.
> 
> But feel free to make your own changes to the approach. Just focus on
> the improved results. Need not follow what I say ;)
> 
> -Santhosh
>