[bug#33899] [PATCH 0/5] Distributing substitutes over IPFS

[Ludovic Courtès]

Hi Hector,

Happy new year to you too!  :-)

Hector Sanjuan <address@hidden> skribis:

> 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs
> has a read/write API (on :5001) and a read-only API that we call "gateway"
> and which runs on :8080. The gateway, apart from handling most of the
> read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>"
> or "/ipns/<name>" gracefully, and returns an autogenerated webpage for
> directory-type CIDs. The gateway does not allow to "publish". Therefore I 
> think
> the doc strings should say "IPFS daemon API" rather than "GATEWAY".

Indeed, I’ll change that.

> 2) I'm not proficient enough in schema to grasp the details of the
> "directory" format. If I understand it right, you keep a separate manifest
> object listing the directory structure, the contents and the executable bit
> for each. Thus, when adding a store item you add all the files separately and
> this manifest. And when retrieving a store item you fetch the manifest and
> reconstruct the tree by fetching the contents in it (and applying the
> executable flag). Is this correct? This works, but it can be improved:

That’s correct.

> You can add all the files/folders in a single request. If I'm
> reading it right, now each files is added separately (and gets pinned
> separately). It would probably make sense to add it all in a single request,
> letting IPFS to store the directory structure as "unixfs". You can
> additionally add the sexp file with the dir-structure and executable flags
> as an extra file to the root folder. This would allow to fetch the whole thing
> with a single request too /api/v0/get?arg=<hash>. And to pin a single hash
> recursively (and not each separately). After getting the whole thing, you
> will need to chmod +x things accordingly.

Yes, I’m well aware of “unixfs”.  The problems, as I see it, is that it
stores “too much” in a way (we don’t need to store the mtimes or
permissions; we could ignore them upon reconstruction though), and “not
enough” in another way (the executable bit is lost, IIUC.)

> It will probably need some trial an error to get the multi-part right
> to upload all in a single request. The Go code HTTP Clients doing
> this can be found at:
>
> https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96
>
> As you see, a directory part in the multipart will have the content-type 
> Header
> set to "application/x-directory". The best way to see how "abspath" etc is set
> is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:5001).
>
> Once UnixFSv2 lands, you will be in a position to just drop the sexp file
> altogether.

Yes, that makes sense.  In the meantime, I guess we have to keep using
our own format.

What are the performance implications of adding and retrieving files one
by one like I did?  I understand we’re doing N HTTP requests to the
local IPFS daemon where “ipfs add -r” makes a single request, but this
alone can’t be much of a problem since communication is happening
locally.  Does pinning each file separately somehow incur additional
overhead?

Thanks for your feedback!

Ludo’.