Re: persistent data feature

[Tassilo Horn]

Qiantan Hong <qhong@mit.edu> writes:

>> There should be no reason to cap its number of entries or the number
>> of things it tracks.  By default it would also track sender, subject,
>> and recipients of messages.  I guess that would easily quadruple its
>> size for me.
> I agree the number should not be restricted. I will try to implement
> split bucketing in probably the next^2 revision of resist! then.
>
> There’s still a technical question unanswered. If I’m to use multiple
> files to store one kv store, is it better to let user supply a
> directory, or let user supply a filename and we add suffixes?

I'd use a directory but don't care too much.

>> I also think the Gnus registry is an example of where a DB with a
>> specific schema and queries would make sense instead of some
>> persistent variable.  There's no value in having that huge pile of
>> hash-maps in memory if I just want to, e.g., move the current mail
>> into the folder where its parent already resides.
> I think in your case, even if the db quadruples, it’s still < 20MB in
> textual representation and probably much smaller when resided in the
> heap. It’s still a pretty long way to go before memory-residence
> become problematic.

Sure, but as said I've restricted the registry to 10000 entries.  I
actually have 100000 mails in my spool and maybe the same number of
newsgroup articles.  Then we're around 300 MB and that's a size I'd
start worrying about.

And my main point is that only a tiny fraction of that data will ever be
used but which of it will be used cannot be known beforehand.  A typical
scenario for the Gnus registry is: given a single message, show the
complete thread.  Unless you have one store per thread, that's a query
which will inevitably require loading large parts if not all of the data
into memory.  Real databases are good in evaluating such queries quickly
without excessive memory usage.

Bye,
Tassilo