Re: [GSoC 23] distributed substitutes, cost of storage

[Vijaya Anand]

Hi Attila

Thanks for the welcome!

I agree that the responsibility of re-uploading the blocks back to the network should be with the clients rather than the substitute server. Also I didn't really think about the point about having to pay for the p2p services at some point of time. In this case we will have to pay for the storage of substitutes both on the p2p storage backend as well as for storage in the substitute server am I right? So ideally we will want to eliminate the usage of these substitute servers and shift totally to p2p services and in this case we will have to shift the responsibility of re-uploading the blocks to the clients itself.

Also if we dont keep the re-uploading blocks option as default for the users, won't users usually not choose to enable it? Maybe we can keep it on as default and resource conscious users can choose to turn it off? Please let me know your thoughts on these points and I will change the implementation point of my proposal accordingly. 

Thank you

Vijaya Anand

On Sun, 26 Mar 2023 at 00:30, Attila Lendvai <attila@lendvai.name> wrote:

welcome on board Anand!


> In case a user requests for a substitute and there is a missing
> block in the decoding process, a HTTP request for block would sent
> to the substitute server and the server will encode the
> corresponding block in real time and push it back into the
> network. The block will be searched again and retrieved.


something to consider here: whose responsibility should it be that a block, that is missing from a p2p network, is (re-)uploaded there? the clients? or the current substitute server?

my gut instinct says that it's better if the clients do the (re-)upload of the blocks.

in this architecture the substitute server is just another storage mechanism along the other storage backends (although with a different reliability characteristics), and it's the clients that are doing the mirroring/spreading/distribution of the blocks among the various backends. the clients of course will/should keep the current substitute servers at the bottom of their list of backends in their configuration.

this way the load is distributed, and we don't need to add (too much) extra complexity to the substitute server codebase, and the actors are less tightly coupled.

it's another question whether this mirroring should be enabled by default in the clients. probably it shouldn't, and the project infrastructure should be running clients where it is turned on. altruistic third parties could also enable this mirroring feature, and donate their bandwidth/resources.

there's an issue with this, though:

some p2p storage backends will require some form of payment/credentials to use their resources. arguably, all p2p storage networks that will survive into the future will have some mechanism to limit the infinite abuse of their resources. it is to be researched how these payment mechanisms work on the various p2p networks, and whether it is possible that the Guix project pays for the storage globally, and then the random clients will have the necessary credentials to (re-)upload the missing blocks.

this architecture shouldn't be impossible, because the content is authenticated by its hash, and if the payment/authorization mechanism is based on the hashes of the blocks (probably), then any client could (re-)upload a missing block that was already paid for.

i'll look into this, especially in the context of Swarm.

meta: i think such specific discussions should be kept off-list, but the financing of the storage fees is probably something that should be known about more widely.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
Every lie is a debt to the truth.