Technical problems [was: Re: My view on bootstrappable effort (Re: Checkpoint)]

[Amirouche Boubekki]

On 2018-08-29 10:26, Arne Babenhauserheide wrote:
> Amirouche Boubekki <address@hidden> writes:
>
> > On 2018-08-28 19:06, Amirouche Boubekki wrote:
> > > a) ordered key-value stores are such a primitive
>
> why ordered?

Sometime people call them "sorted". Ordered is required
to have more assumptions about how the data is structured.
Because keys are ordered, you know that "similar" keys
will be "collocated". That what makes _range_ scan possible
(in opposition to full scan). range scan will return a slice
of key that start with a given prefix. By construction (at insertion
time) it might be all the keys that represent a given document.

There is various pattern you can implement with ordered key value
store the one I know and experimented with are for instance
space filling curve. But to build a leaderboard for instance,
you can implement a probabilistic data structure inside the ordered
key value store.

If you remove the oredered, you end up with something like gdbm.
In that situation there is no "extra structure" that you can take
advantage to speed up queries. It always have to be a full scan.
Or single key get.

> > > b) priority and age-based distributed task queue is another

Indeed, it's a distributed peer to peer network (in the sens every node
can do anything). But it's a trusted network. Untrusted peers can not
access the task queue directly.

> How do you prioritize?

That's the question! I will start another topic with that question.

> What’s your source of truth?

The database, that is FoundationDB in the distributed setup.

> Does truth have to be defined centrally or does it center around each 
> individual

Yes truth is defined centrally, but with fault tolerance. Every peer (or 
node)
has a heartbeat to check every minute that every role is fulfilled.
It's fulfilled if a lock is taken in FDB. If it's not there (or expired) 
there is a
race for taking on that role. FDB is atomic and consistent. So only one 
"peer"
will take a given role at a given moment. Until the peer goes down
they will fulfill their duty for the role they have taken.
Like keeping the task queue up-to-date.

In practice, the role is fulfilled by a fiber of a peer.

> (hopefully with eventual consistency within society)?

When it comes to FDB, it's consistent. Every peer see the same version 
of the truth.

> I’m asking because the minimum truth you need for any communication
> system is (spam? item).

Like I said earlier, this is a controlled environment. Some kind of peer 
to peer
architecture for scaling centralized systems.

> For reference:
> https://www.draketo.de/english/freenet/deterministic-load-decentralized-spam-filter

thanks for the link.

> Best wishes,
> Arne
>
> PS: sorry for keeping this short.

Thanks for replying.

--
Amirouche ~ amz3 ~ http://www.hyperdev.fr