GNU bug report logs - #33899
[PATCH 0/5] Distributing substitutes over IPFS

Previous Next

Full log

Message #29 received at 33899 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Hector Sanjuan <code <at> hector.link>
Cc: "go-ipfs-wg <at> ipfs.io" <go-ipfs-wg <at> ipfs.io>,
 Pierre Neidhardt <mail <at> ambrevar.xyz>,
 "33899 <at> debbugs.gnu.org" <33899 <at> debbugs.gnu.org>
Subject: Re: [PATCH 0/5] Distributing substitutes over IPFS
Date: Mon, 14 Jan 2019 14:17:36 +0100

Hi Hector,

Happy new year to you too!  :-)

Hector Sanjuan <code <at> hector.link> skribis:

> 1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs
> has a read/write API (on :5001) and a read-only API that we call "gateway"
> and which runs on :8080. The gateway, apart from handling most of the
> read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>"
> or "/ipns/<name>" gracefully, and returns an autogenerated webpage for
> directory-type CIDs. The gateway does not allow to "publish". Therefore I think
> the doc strings should say "IPFS daemon API" rather than "GATEWAY".

Indeed, I’ll change that.

> 2) I'm not proficient enough in schema to grasp the details of the
> "directory" format. If I understand it right, you keep a separate manifest
> object listing the directory structure, the contents and the executable bit
> for each. Thus, when adding a store item you add all the files separately and
> this manifest. And when retrieving a store item you fetch the manifest and
> reconstruct the tree by fetching the contents in it (and applying the
> executable flag). Is this correct? This works, but it can be improved:

That’s correct.

> You can add all the files/folders in a single request. If I'm
> reading it right, now each files is added separately (and gets pinned
> separately). It would probably make sense to add it all in a single request,
> letting IPFS to store the directory structure as "unixfs". You can
> additionally add the sexp file with the dir-structure and executable flags
> as an extra file to the root folder. This would allow to fetch the whole thing
> with a single request too /api/v0/get?arg=<hash>. And to pin a single hash
> recursively (and not each separately). After getting the whole thing, you
> will need to chmod +x things accordingly.

Yes, I’m well aware of “unixfs”.  The problems, as I see it, is that it
stores “too much” in a way (we don’t need to store the mtimes or
permissions; we could ignore them upon reconstruction though), and “not
enough” in another way (the executable bit is lost, IIUC.)

> It will probably need some trial an error to get the multi-part right
> to upload all in a single request. The Go code HTTP Clients doing
> this can be found at:
>
> https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96
>
> As you see, a directory part in the multipart will have the content-type Header
> set to "application/x-directory". The best way to see how "abspath" etc is set
> is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:5001).
>
> Once UnixFSv2 lands, you will be in a position to just drop the sexp file
> altogether.

Yes, that makes sense.  In the meantime, I guess we have to keep using
our own format.

What are the performance implications of adding and retrieving files one
by one like I did?  I understand we’re doing N HTTP requests to the
local IPFS daemon where “ipfs add -r” makes a single request, but this
alone can’t be much of a problem since communication is happening
locally.  Does pinning each file separately somehow incur additional
overhead?

Thanks for your feedback!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.