Re: persistent data feature

[Stefan Monnier]

>>> There will, of course, be large amounts of data, and that will be slow.
>>> (People already do this with a number things, and the results are
>>> predictably sluggish.)
>>
>> To help me understand this discussion, I think it would help me to have
>> examples of such large databases currently implemented as text files,
>> along with an idea of what "large" means in this context (how many MBs)
>> and where the "slow"ness manifests itself.
>
> I can think of two examples of large text databases: elfeed and recent
> org-mode's caching (org-persist).

I think I'm beginning to see a pattern: most of those uses (same for
org-roam) are for cache-like data, i.e. data that can be recreated from
other data.

I suspect the same holds for gnus-registry, but doesn't for ecomplete.

Cache-like data has the property that you don't need to version it, and
users should never need to manipulate it directly: if there's a problem
with it you should be able to just delete it and start over.

For such cases, there's no great benefit to use a "transparent" text
representation (but it's important to provide some way to rebuild the
cache).


        Stefan