Re: [GSoC 23] distributed substitutes, cost of storage

[Maxime Devos]

Op 25-03-2023 om 20:00 schreef Attila Lendvai:
> welcome on board Anand!
>
>
> > In case a user requests for a substitute and there is a missing
> > block in the decoding process, a HTTP request for block would sent
> > to the substitute server and the server will encode the
> > corresponding block in real time and push it back into the
> > network. The block will be searched again and retrieved. > something to consider here: whose responsibility should it be that a 
block, that is missing from a p2p network, is (re-)uploaded there? the 
clients? or the current substitute server?
> my gut instinct says that it's better if the clients do the (re-)upload of the 
> blocks.
>
> in this architecture the substitute server is just another storage mechanism 
> along the other storage backends (although with a different reliability 
> characteristics), and it's the clients that are doing the 
> mirroring/spreading/distribution of the blocks among the various backends. the 
> clients of course will/should keep the current substitute servers at the bottom 
> of their list of backends in their configuration.
>
> this way the load is distributed, and we don't need to add (too much) extra 
> complexity to the substitute server codebase, and the actors are less tightly 
> coupled.
>
> it's another question whether this mirroring should be enabled by default in 
> the clients. probably it shouldn't,

It probably should -- if things aren't mirrored, then it's not p2p; you 
would lose the main performance benefit of p2p systems.

More cynically, some p2p systems (e.g. GNUnet) have mechanisms to 
disincentive freeloaders -- clients that aren't being peers will get 
worse downloading speed.

> and the project infrastructure should be running clients where it is turned on. 
> altruistic third parties could also enable this mirroring feature, and donate 
> their bandwidth/resources.
>
> there's an issue with this, though:
>
> some p2p storage backends will require some form of payment/credentials to use 
> their resources. arguably, all p2p storage networks that will survive into the 
> future will have some mechanism to limit the infinite abuse of their resources. 
> it is to be researched how these payment mechanisms work on the various p2p 
> networks, and whether it is possible that the Guix project pays for the storage 
> globally, and then the random clients will have the necessary credentials to 
> (re-)upload the missing blocks.

GNUnet has a built-in mechanism for mirroring and for avoiding overuse 
of resources.  From what I recall of the documentation and the GNUnet 
papers:

* The more a peer A requests stuff of peer B,
  the more peer B dislikes peer A.
* Likewise, the more peer A fulfills requests of peer B,
  the more peer B likes peer A.
* Requests by liked peers are prioritized.

(If you squint, I suppose this could be considered a form of payment, 
but no literal currencies are involved, so no need for any financing.)

Mirroring:

* When putting a resource on the network, a few copies
  are stored in the network.  (I assume this discreases the dislike of
  the peer that received the copy by the peer that sent the copy, and
  increases the dislike by the peer that sent the copy by the peer that
  receives the copy.)

* The more popular a resource is, the more replicas are stored in
  the network (I don't recall the mechanism, but IIRC this is an
  automatic process).

* Peers set a quotum on how much bytes they are willing to store;
  when exceeded, they throw out old stuff and low-priority stuff.

(The only way to be 100% sure a resource remains on the network, is to 
have a local copy in the local peer, so you can't really reliably ‘save’ 
something on the network, but you can use it as a CDN to spread the load 
and tolerate occasional downtime.)

Greetings,
Maxime

OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key

OpenPGP_signature
Description: OpenPGP digital signature