Re: Channel sessions

[Carl Fredrik Hammar]

Hi,

<olafBuddenhagen@gmx.net> writes:

> Hi,
>
> On Wed, Aug 22, 2007 at 05:23:45PM +0200, Carl Fredrik Hammar wrote:
>> <olafBuddenhagen@gmx.net> writes:
>
>> > Channel instances are pretty much of an internal detail IMHO. When a
>> > user thinks of setting up a channel, he doesn't care about
>> > individual client connections; it's the whole entity he thinks
>> > about.
>> >
>> > Also, consider stores: The store is the whole entity, not some
>> > client connection. The channel is the whole entity, not some
>> > individual connection; why should it be anything else? Really, the
>> > static setup is the most visible, most obvious part of the concept;
>> > it doesn't make any sense to use the generic term "channel" for
>> > anything else.
>> >
>> > I *strongly* object to any other use.
>> 
>> Fortunately this terminology does not need to surface to users of
>> channelio, for them there is no need to distinguish the hub from the
>> file it's sits on.  But for our discussion and in the implementation
>> we must distinguish them.
>
> The fact that a pure end user needs less precise terminology, doesn't
> mean it's OK to use the same terms in a different meaning. That would be
> *extremely* confusing.

I didn't suggest that term should have a different meaning for users.
My point was that the term `hub' doesn't need to be exposed to
end-users, because its mostly an internal matter for libchannel and
users of the library, like channelio.

>> Also, with stores things are much easier.  Every client operates on
>> the same store and changes made to it is seen by all clients, i.e.
>> there is only a single store instance.  The amount of client state is
>> also minimal, only an offset is needed.  So I don't think its fair to
>> draw analogies between channels and stores.
>
> One being simpler than the other doesn't mean we are not allowed to draw
> analogies. It's perfectly reasonable and desirable to use equivalent
> terms for eqivalent concepts, regardless of the fact that channels need
> additional concepts.

What you say is obviously true, but what I meant is that stores and
channels aren't equivalent concepts (at least with my terminology),
and that therefore the analogies weren't valid.

I should have elaborated this subject earlier.  As this is the major
difference between libchannel and libstore, more on this bellow.

> Also, I'm not really convinced of stores being inherently simpler. It's
> true that in most cases, the implicit junction at the top of the store
> stack is fine; but I guess one could construct cases where having the
> junction further down in the stack would be desirable. I think the main
> reason why such considerations don't seem relevant for stores (and thus
> the current store framework doesn't handle such things), is that usually
> stores are used as backends for a filesystem, and filesystems expect
> exclusive access to the store anyways -- you just don't have multiple
> store client instances in the normal case.

Stores *are* inherently simpler when it comes to handling multiple
clients.  A store is shared between clients, modifications will be
seen by other clients.  This property is the same for all stores,
which allows the storeio to handle them uniformly.  The only client
state ever needed is an offset, which can be communicated through the
io interface, so it only needs to be maintained by storeio.

This also means that stores handle multiple clients just fine.  There
is simply no need for stores to have a notion of different clients.
(storeio does need it of course.)

The only stores that might violate this property would be stores where
changes are kept private to clients.  But I'm not sure if this is
useful enough to warrant complicating store sessions.  This could
possibly be handled by storeio as a special case.  It wouldn't be a
problem in a store channel-package.

Stores are more complex than channels in other areas, such as dealing
with sparse data and paging.  But this isn't relevant to our
discussion.

> This however could quickly change, if the store concept is applied to
> other use cases -- which would be very desirable IMHO: I believe there
> is a whole class of translators (including "kitten" and "rollover", but
> also gzip and some others) that would be useful to implement using
> stores or some store-like facility.

There is already a (read-only) `gzip' store.  It might be the case
that `rollover' requires access to the offset maintained by storeio,
but this would only require minor changes to libstore.  I'm not
familiar with `kitten', and it didn't turn up in any of the searches I
made.  But I like its name :-), could you give me some pointers to
where it's described?

> In fact, as you pointed out yourself in the beginning, stores can be
> considered a special case of channels; and once the channel framework is
> matured a bit, it will probably be desirable to reimplement stores as a
> channel package...

libchannel would probably need some changes to handle stores as well,
but I think it would be worth while doing it.  I'll explain how stores
would fit into libchannel.  Perhaps it will clear up the difference
between channels and stores.

A store would be an end-point which a client communicates with through
a channel.  Such store channels differ from other channels in that
they implement seeking, mmapping and paging, and that they may be
sparse.

A store would normally act as a front-end junction, since all client
state (an offset) can be communicated through the io interface.  All
store channel would have to do is seek the back-end before any io, or
if the channel io functions are extended, give the offset as an
argument to the single back-end channel or port.

With my terminology, channels remain a useful concept when
discussing stores.  Calling a store a special channel would be wrong
IMHO, since different client receive different streams depending on
their current offset.

> Yeah, I think I understand what you mean by type. I still don't get the
> distinction to "class", though. I'm ignoring "class" for now; hope it
> doesn't cause confusion.

It shouldn't confuse things.  The distinction between type and class
is mostly only relevant in the implementation, where type usually
refers to the name of the class, rather than the data structure
itself.

Since the distinction isn't very useful, it might be a good idea to
separate them and let class mean package.  I think this would be more
appropriate in the end, but for now lets keep things as it is in
libstore.

>> > An audio channel for example has some additional facilities aside
>> > from the generic channel stuff to control bitrates etc., and also
>> > requires specific semantics when merging/splitting instances. So an
>> > audio channel stack will most likely include some audio-specific
>> > hubs (perhaps in addition to some generic ones). A network channel
>> > stack will use network-specifc hubs. What term would you use to
>> > describe such stacks (and hubs) for handling particular kinds of
>> > data?
>> 
>> I guess I would call them channel types that handle a certain stream
>> type.  But it would be useful to handle this more firmly.
>> 
>> Lets call it a `package', a package would simply be a collection of
>> classes that can be used together.
>
> Well, that isn't optimal, as it doesn't catch the fact that all the hub
> types in a package not only have some package-specific semantics, but
> often also have a specific interface. (e.g. specific packeting
> properties, specific control commands etc.)
>
> But maybe it's clear enough. I'll stick to "package" for now.

Agreed, it isn't optimal.  The name also suggest that types from
different packages can be intermixed, which isn't usually the case,
and in I think such types should be put in all packages they make
sense to be in.

Perhaps instead of packages, we should provide a translator for every
package, e.g. `audioio', `storeio' and `netio'.  And provide a
`libchannelio', which would be similar to libtrivfs, but geared
towards channels.  This way junctions and buffering can be handled
in a more implicit manner by the specialized translators.

> BTW, you used the term "elements" above. How about using this instead of
> "hub"? Seems more natural...

I don't think `element' makes sense out of context.  It works when you
say ``stack element'', but ``channel element'' seems to mean channel
rather a than channel hub.

How about `spec'.  It captures the fact that the mainly used when
creating new channels.  To make it a more firm concept channels should
not be able modify it.  And to refer to all channels created from a
spec, perhaps simply saying `layer' would suffice.

Coordination between channels in a layer can be handled in a hook,
which the spec holds a reference to so it can provide it to channels
when they are created.  I think this makes sense, since it's mostly
junctions that need such coordination.

I think this would fit a bit better into your intuitive terminology,
while still keeping my emphasis that channel is primarily a per-client
entity.

>> It would be nice if each such package had its own namespace instead of
>> prefixing the name to classes of the same package, e.g. the audio
>> package would have one `tee' and the network package another, instead
>> of `audio-tee' and `net-tee'.
>
> Well, how would that work in practice? I guess the channel interface
> would require explicit namespace parameters in addition to the actual
> names? And for the channel translators, one probably would need
> subdirectories, like /hurd/audio/tee and /hurd/net/tee?
>
> (Or do you think all channel translators should be set up using only
> channelio with different module parameters?...)
>
> Well, to be honest, I prefer the prefix variant -- not only because it
> doesn't require special handling...

The part in parenthesis is what I thought.  Namespace would be
specified by argument.  One translator per type would definitely be
over-kill.  ;-)

I would keep modules in directories or perhaps have one module per
package instead of one module per type, that would make it a bit more
manageable, e.g. `/lib/libchannel-audio.so'.

>> > What I wonder about is whether it's even useful to have any other
>> > than such specialized junctions...
>> 
>> I'm starting to think no.  Or at least not that is useful across
>> packages.  But there might be a need for a `generic' package that
>> deals with generic character devices, like /dev/zero, /dev/random and
>> input devices.
>
> Yeah, we probably need a "generic" package with some universal modules
> that should suffice to handle simple devices not having special needs.
> But I don't think this package will be very large: What does it require
> besides the mach device access and buffering hub types?

First a slight change in terminology, I will call an in-tee a
`broadcast' and an out-tee simply a `tee'.

I would think that broadcast and a front-end fifo might be useful to
get around exclusive access input devices and perhaps /dev/random.

>> > I don't see how one could handle network channels in any useful
>> > manner without awareness of packets.
>> 
>> Well it is obviously needed for all kinds of network junctions.  But
>> one might be able to use generic junctions if one layers a channel
>> that makes sure all reads and writes are in whole packets.
>
> Well, can this really be handled by layering? And is there any advantage
> in doing so? My impression is that packaging should be an inherent
> property of all channel interfaces. But then, I haven't thought through
> the details...

Packets only causes problems in junctions that splits a stream, where
one have to make sure that packets aren't split up and sent to
different recipients.  Assuming that the junction splits a stream
based on requests, e.g. this read here, next read there.  This could
be handled by layering a `packeter' above the junction, which makes
sure that reads and writes are in whole packets.  When read, such a
channel completes any incomplete packets by reading more from the
junction than requested, keeping the extra data in a buffer, until the
nest read on this channel.  It also makes sure no other channel can
read the junction until it has the complete packet.  When written, it
only writes any complete packets to the junction, keeping the
incomplete one in a buffer.

But layering only makes sense over a generic junction and the only
real advantage would be avoiding writing specialized junctions, in
favor of writing special packters.

At this point I'm fairly certain we want specialized junctions and
packeting would be handled by them.  So I think we can leave it at
that.

>> This would also be needed for audio channels, since a audio frame is 2
>> bytes long.
>
> In fact, it depends on the settings: A typical stereo 16 bit stream for
> example actually has 4 bytes per packet; 24 bit would have 6 bytes etc.
>
> The difference to network packets is that audio packets have a fixed
> size once the parameters are set, while network channels have a
> different size for each packet.

Ah yes.  I forget the details, but it seems you are aware of the
issue.

>> > Are you sure that doesn't require additional functionality? It seems
>> > to me that handling such sub-modules actually requires quite a lot
>> > of additional infrastructure; maybe even more than client IDs...
>> > (Aside from the fact that it's not very elegant IMHO.)
>> 
>> I'm not sure.
>> 
>> But I'm not sure it will be a problem, since our discussion seem to be
>> heading towards less generic code.
>> 
>> I'm starting to think that instead of providing generic junctions,
>> libchannel should provide code that can be shared between packages and
>> be used to piece together specialized junctions.  This would probably
>> make channelio much more user-friendly in the end.
>
> Well yes: As I have argued, the actual semantics of junctions are quite
> package specific. Yet the basic concept is generic, and the framework
> needs to provide generic mechanisms for implementing specific junctions.
> I'd say this should include generic mechanisms to faciliate implementing
> specific stream pairing variants. (input/output, data/control etc.)
>
> -antrik-

It seems we agree on this.

Cheers,
  Fredrik