Re: persistent data feature

[Qiantan Hong]

> On Dec 11, 2021, at 5:53 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>>> There will, of course, be large amounts of data, and that will be slow.
>>>> (People already do this with a number things, and the results are
>>>> predictably sluggish.)
>>> 
>>> To help me understand this discussion, I think it would help me to have
>>> examples of such large databases currently implemented as text files,
>>> along with an idea of what "large" means in this context (how many MBs)
>>> and where the "slow"ness manifests itself.
>> 
>> I can think of two examples of large text databases: elfeed and recent
>> org-mode's caching (org-persist).
> 
> I think I'm beginning to see a pattern: most of those uses (same for
> org-roam) are for cache-like data, i.e. data that can be recreated from
> other data.
Interesting.
> 
> I suspect the same holds for gnus-registry, but doesn't for ecomplete.
> 
> Cache-like data has the property that you don't need to version it, and
> users should never need to manipulate it directly: if there's a problem
> with it you should be able to just delete it and start over.
> 
> For such cases, there's no great benefit to use a "transparent" text
> representation (but it's important to provide some way to rebuild the
> cache).
I think a good fit to that would be incremental image.
Is it possible to have Emacs save/load part of the heap
(e.g. reachable from a root object) quickly, potentially
by directly mapping the memory in the load case?
Is pdumper useful for that purpose?