Re: [Gnu-arch-users] RFC: arch protocol, smart server, and tla implementation prototypes

[Aaron Bentley]

Colin Walters wrote:

>On Fri, 2004-01-30 at 17:02, Aaron Bentley wrote:
>
>  
>
>>I disagree: solving the latency problem for archd is nice, but solving
>>it for all protocols is nicer.  
>>    
>>
>
>I don't see how it's possible for dumb-fs protocols.  
>
>  
>
>>And I think we'll I'll get close to that
>>functionality in the backwards-builder anyhow.
>>    
>>
>
>Backwards-building from a cacherev?
>
Well, I mainly want to solve the build backwards from library problem, 
but cacherevs will come as part of that package.

>  But you still need to *find* that
>cacherev, whether you work forwards or backwards, and that in general
>involves traversing a bunch of revisions, with a roundtrip for each.
>  
>
If you can query in parallel (and you usually can) then you can reduce 
the lag for all queries to the same as the lag for one query.

>>Crossing tag boundaries will make the problem harder.  While tla
>>implicitly uses the call stack to build forwards, crossing tag
>>boundaries requires tla to map out several paths, and determining the
>>best one will require a cost assessment based on
>>
>>1. aggregate download size
>>2. download cost for a given archive
>>    
>>
>
>How do you figure that out (efficiently)?  I'm not sure you really can.
>  
>
1. Most of our supported systems provide a way of determining file size.
2. Up for grabs.  Could be based on past performance or user tweaking.

>>The other advantage of allowing high performance with dumb servers is
>>that smart servers may run out of CPU time and I/O bandwidth before they
>>run out of network throughput.
>>    
>>
>
>I doubt that; as long as your server isn't trying to merge changesets
>itself, etc., serving shouldn't be much more complicated than HTTP.
>  
>
Tom's proposal for Delta appeared to propose merging changessets on the 
server.

>>pfs-sftp, pfs-http, pfs-dav etc. can use arch_compose_changesets plus
>>streaming or multiple simultaneous connections to implement
>>arch_archive_delta.
>>    
>>
>
>dumb-fs level streaming might be possible per-protocol (I'm not sure if
>sftp or http support that), but simultaneous connections would not be a
>win.  You still have to do at least one round-trip per connection, and
>possibly more for authentication, etc.  
>  
>
If you need to download 17 files with 1 sec lag, serial download has 17s 
lag, while parallel download has 1s lag.  Theoretically, anyway.  http 
supports ways of reusing a tcp connection, though they're not 
universally supported.

>>I think that archd might well be implemented at a higher level.  It
>>could conceivably use sftp, http, etc as backends.
>>    
>>
>
>Yeah, but why?  The connection between archd and a dumb fs would likely
>be just as slow.
>  
>
It would be a design where the server was used with pipes locally and 
with socket connections remotely.  One implementation of the daemon 
would be part of arch, and just use pfs.  Rather like running an X11 
server on the same machine as the clients.

Aaron