Re: [Varnamproject-discuss] Bug 40714 : without rowid

[Kevin Martin]

No diff? I could see a diff (line 34). I have deleted the merge request and have created a new one with the same changes. Please check if you can see the diff

On Sat, Mar 22, 2014 at 1:55 PM, Kevin Martin <address@hidden> wrote:

1) There is an improvement in time taken to learn. with rowid - 8 minutes 13 seconds. without rowid - 7 minutes 9 seconds.

2) No change in query plan. Used 'explain query plan select word_id from patterns_content;'.

Results attached.

On Sat, Mar 22, 2014 at 10:09 AM, Navaneeth K N <address@hidden> wrote:

Hello Kevin,

Looks like there are some issues with the merge request. I don't see any
diffs. Can you update it after looking into the following points?

* Have you verified the time required for learn? Is it improved or it
takes more time? Just use the word corpus and learn one file and see how
long it takes. Do the same after you apply your changes. Make sure you
compile and install after applying your changes.

* Get into sqlite3 command line tool, and look at the query plan
generated for the query we use on patterns_content. Check the plan
without your changes and with your changes. See if that has changed. And
if it has changes, make sure it still uses indexes and executes as
quickly as possible.

Other than this, the change looks good. Thanks for working on it.

On Friday 21 March 2014 11:44 AM, Kevin Martin wrote:
> Merge request sent
>
>
> On Fri, Mar 21, 2014 at 1:59 AM, Kevin Martin
> <address@hidden>wrote:
>
>> I have done the tests on patterns_content and found it to be helpful.
>> Please see[1]. I used the varnam word corpus to populate ml.vst.learnings.
>> The difference in size was very significant. The DB shrinked from about
>> 500mb to 294mb. I have attached the sqlite_analyzer output on the bug
>> tracker. If this is really as good as it looks, I will use 'without rowid'
>> on all tables and submit a pull request.
>>
>> [1] https://savannah.nongnu.org/bugs/index.php?40714
>>
>>
>>
>> On Thu, Mar 20, 2014 at 11:59 PM, Navaneeth K N <address@hidden> wrote:
>>

> Hello Kevin,
>
> I have updated SQLite and made a commit. Pull to get latest changes.
>
> On 3/20/14 9:35 PM, Kevin Martin wrote:
>>>>> I was able to reproduce it once. But not again. I will look into this
> and
>>>>> file a bug if I can figure out why this happens. Meanwhile, how can I
> try
>>>>> out sqlite3 3.8.3 with varnam? Needs it to use the without rowid
> feature.
>>>>> How can I change the embedded sqlite?
>>>>>
>>>>>
>>>>> On Thu, Mar 20, 2014 at 11:47 AM, Navaneeth K N <address@hidden> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> `libvarnam` embeds SQLite. It won't use the system version. Primarily
>>>>> because SQLite does lots of performance improvements to each versions
>>>>> and system packages won't get updates all the time. For example, in
>>>>> Debian, updates are very rare.
>>>>>
>>>>> This makes `libvarnam`'s performance dependent on system's installed
>>>>> SQLite package. To avoid that, libvarnam compiles SQLite statically. In
>>>>> fact, that is the preferred way of using SQLite.
>>>>>
>>>>> On 3/20/14 11:34 AM, Kevin Martin wrote:
>>>>>>>> I'm trying to use sqlite 3.8.2 with libvarnam to see if 'without
> rowid'
>>>>> is
>>>>>>>> helpful. However using without rowid throws me a syntax error when I
> use
>>>>> it
>>>>>>>> with libvarnam. I can create tables without rowid in sqlite 3. I
> noticed
>>>>> a
>>>>>>>> file sqlite3.c in deps/ . libvarnam is not using the libsqlite3-dev
>>>>> package
>>>>>>>> in the system? If sqlite3.c is written for sqlite3 < 3.8.2, without
> rowid
>>>>>>>> will not work.
>>>>>>>>
>>>>>>>> Another curious thing is that I tried building libvarnam after
> removing
>>>>>>>> sqlite3 from my system. The build was successfull, even though
>>>>>>>> transliteration did not work ('kevin' was transliterated to 'kevin'
>>>>> instead
>>>>>>>> of the malayalam pattern). Wouldn't it be nice if we can add a few
> checks
>>>>>>>> to see if sqlite3 is installed?
>>>>>
>>>>> Transliteration should work because `libvarnam` is not dependent on
>>>>> system's SQLite package. Do you have the VST file available in the
>>>>> search path? Can you reproduce the behavior?
>>>>>
>>>>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>
>
>>>
>>>
>>
>

--

Cheers,
Navaneeth