Re: New package: resist!

[Alexandre Garreau]

Le dimanĉo, 12-a de decembro 2021, 2-a horo kaj 35:48 CET Michael 
Heerdegen a écrit :
> Alexandre Garreau <galex-713@galex-713.eu> writes:
> 
> 
> 
> 
> > > How does your package handle values whose print syntax can't just be
> > > `read' to recreate the value?
> 
> 
> 
> > that’s an issue for pretty much any persistence package, and it would
> > just as well be while using sqlite, persist, eieio-persistent or
> > anything else.
> 
> Yes, but it's the hard part.  It is what makes saving the Gnus registry
> slow, and it is what makes eieio-persistent look "like an object
> oriented prin1".

No, as said before you’re conflating two separated issues: variable 
persistence, and element-by-element retrieving (from a kv store).  We use 
a kv-store for variable persistence, but that feature is different from the 
later one because you save each value once in its entirety, and then fetch 
it back once in its entirety.  So reading or printing it is actually 
necessary, you cannot make it faster, and it’s not a bottleneck (I/O is 
the bottleneck then), you have to serialize anyway.  When you want to 
fetch individual elements without restoring the whole value in its 
entirety (or perhaps doing just as if it was, in a lazy way, but that’s 
yet again a new feature which has not been discussed yet, and which enters 
in competition with the kernel’s paging/swaping features), you need to 
destructure it manually, you don’t do that automatically, and you want its 
restoration to be coordinated in some determined way… that’s not variable 
persistence.  So the problem you’re talking about, where “read” and 
“write” are slow, is not variable persistence.

I feel like the features we came to discuss are so abstract people keep 
confusing and conflating them together.  They all in some way slightly 
overtake one onto the other, so it makes it understanding overall even 
more difficult…

> > Now we could try to see what would happen concretely: what kind of
> > values are you thinking to?
> 
> The Gnus registry is one example.  Apart from that: most values,
> actually, apart from trivial ones: lists, hash-tables, structs.  Most
> applications will not just need to save integers and strings.

That’s a separate problem, again.  These value are perfectly printable.  
You just claim we can store them so that they’re more efficient to 
(partially) retrieve.  And that’s not a problem solveable by variable 
persistence (since the concept of variable persistence is you restore back 
the entire variable into memory), so it’s not even related to the current 
feature being discussed.

If you want to solve that, you have to use the lower level feature we 
discussed before, a kv storage mechanism, to store your hashtable or 
alist.  A struct is by definition limited so there’s little interest in 
storing it strangely (but you store an array of them you may want a 
relational database, indeed… except until now we only discussed about 
needs of a simple kv storage, which is lower level than a relational 
database).  A *general* list, or an array, may be amenable to a hashtable/
alist with succesive increments as keys, but if you have any more adapted 
idea I’d like to hear it (because otherwise it looks a little overkill).

But you then have to *manually* prepare a store for that, and manually 
split up your sequence into that store.

We cannot apply that to the issue discussed now, variable persistence, 
hence any variable, because that would mean simply recursively make all 
lisp objects which are sequences stored into kv stores (because, as you 
know, a hashtable can contain another hashtable, and two sequences can 
share elements), that’s just overkill, it consists in changing the entire 
elisp runtime, and in the end you just replace every variable with a query 
to a relational database… which is interesting, but has not been discussed 
until now.  You would make one table for each type, and pointers would be 
replaced by foreign keys.  Is that what you want? is that what you’re 
talking about?  That would be the “most efficient way” if we wanted to store 
all emacs ram to disk, and yet be able to retrieve it while being sure to 
do it most efficiently wrt what a relational database can do, with the least 
reading from disk and the least stuff kept into memory.

The ideas in this discussion start getting very weird… now I’m wondering 
if it would be possible, in a perfect world, to at the same time describe 
perfectly the native way of storing variables/(PI/repositionable)pointers/
number in memory so it can directly be dumped to disk and portably be 
decoded at will, and design a relational database format so that it can be 
mmaped to memory and be used to represent variables and complex objects 
directly (or the otherway around: encode, and defragment/fragment/group 
(through action of a special garbage collector) objects into memory so 
that if it’s dumped to disk it gives a most efficient relational database 
data, and yet be effecient to read (like data often used together would be 
near each other)), and use that to represent all data on a computer, for 
every programming language/runtime/software…