Re: Filter design for nsmux

[olafBuddenhagen]

Hi,

On Thu, Apr 23, 2009 at 04:29:40PM +0300, Sergiu Ivanov wrote:
> <olafBuddenhagen@gmx.net> writes:
> > On Sun, Mar 22, 2009 at 09:37:55PM +0200, Sergiu Ivanov wrote:
> >> <olafBuddenhagen@gmx.net> writes:
> >> > On Sun, Feb 22, 2009 at 08:56:50PM +0200, Sergiu Ivanov wrote:

> > So the idea is to have only the first instance do a full translator
> > startup. It would do as much common initialization as possible; and
> > then some mystical lightweight mechanism would be used for launching
> > the other instances from there -- something that is much cheaper
> > than normal process creation...
> 
> Hm... This sounds very-very interesting... However, I'd think that
> such functionality could not be implemented without doing some
> extensions to GNU/Mach, what do you think?

Not necessarily, no. I think Mach task creation is actually rather
cheap. It's loading shared libraries, setting up various standard ports
etc. that makes POSIX process startup expensive. It seems quite possible
to optimize that without changing the Mach task/thread concept.

But I really don't know for sure -- this is all *extremely* vague ideas
still. There is a reason I called it "mystical" :-)

> >> Another question now (probably, I'm repeating myself already...):
> >> how problematic is that the filter should know about nsmux? After
> >> all, the filter's main real use case is running in a dynamic
> >> translator stack. I understand that having the filter capable of
> >> running as a normal translator would be a nice option, but I fail
> >> to find the absence of this feature a very bad thing.
> >
> > Actually, this doesn't prevent the filter from running outside
> > nsmux. It only means that the filter must be aware when it is
> > running on nsmux, and handle it specially. That's not really a big
> > deal technically -- but it's not very elegant, as it means things
> > are not as orthogonal as I'd like them to be. nsmux is not fully
> > transparent -- certain kinds of other programs need to handle it
> > specially to work correctly...
> 
> I see... So, do you think we should still try to pursue orthogonality,
> or shall we stop at this design idea for now? (I mean adding a special
> RPC for the filter to obtain the node without shadows).

I don't really think we have any other choice... I at least can't think
of any different approach.

> > See above: logically, the RPC for obtaining the underlying node
> > belongs with the existing interfaces. The operation is not really
> > filter-specific -- it could be useful in other situations as well.
> >
> > The RPC for getting a non-shadowed version of the node on the other
> > hand is very specific, so having a distinct interface only for this
> > is probably really the best approach.
> 
> OK. To sum up: do I understand it right that we shall add the RPC for
> retrieving the underlying node of a translator to fs.defs, and a
> create a new special interface expressly for the RPC for getting the
> non-shadowed version of a node?

Yes, that is what we *should* do. But as I have an irrational reluctance
of touching the existing interfaces, I'm not yet really confident that
this is really the ideal approach in practice. That's why I said that
this decision is my major concern with creating the new RPCs.

> > No. We can't obtain the untranslated version of a node we have. All
> > we can do is reopen the same *file name* -- which is a totally
> > different thing! The filter does *not* have the file name of the
> > node it filters.
> 
> Yes, this is true. I wonder now, why one can reopen a node with
> different O_READ/O_WRITE bits, but cannot do this with the O_NOTRANS
> flag...

This is actually quite obvious: reopening with different read/write
flags etc. just means obtaining a different handle for the same object.
O_NOTRANS on the other hand would require actually obtaining a different
object, served by a different translator!

However, now that you mention it, I seriously wonder whether this isn't
really what we should do. As reopening with O_NOTRANS has no meaning so
far, I guess it wouldn't break anything if we overloaded it with the
"obtain underlying node" operation. And it actually makes some sense
semantically -- it wouldn't be too ugly a hack I think...

> > The hack with O_NOTRANS on translator startup to obtain the
> > untranslated node, only works with special handling in nsmux. You
> > can't do that when running the filter on a normal file system,
> > implementing only the standard interfaces. It requires a new
> > operation -- even if this new operation is hacked as a special case
> > in an existing RPC.
> 
> I see... So, it is natural to implement this feature as a special RPC,
> without hacking any existing one, do I understand everything right?

Well, see above: I just realized that actually we might get away without
a new RPC :-) So no, that's not what I meant by natural.

I think this is missing context now: The original point was that
traversing top to bottom is not more natural than bottom to top, as both
require a new operation. (For obtaining the node directly below the
current for traversing top to bottom, or for obtaining the node at the
very bottom of the stack for traversing bottom to top.)

> >> >> I thought we could merge the functionality in a single node
> >> >> because it seemed to me that another node would mean another
> >> >> context switch...
[...]
> > What you wanted to do here is avoiding a context switch, by
> > transparently stuffing the functionality of two logically distinct
> > translators in a single process. This is *exactly* what translator
> > stacking is about...
> 
> Yes, this was exactly what I meant to achieve by merging shadow and
> proxy nodes together. Nevertheless, I am rather surprised to hear that
> translator stacking is about stuffing the functionality of two
> distinct translators in a single process... Actually, what I
> understand under ``translator stacking'' is *joining* several
> translator processes, not *merging* them.

Erm... I'm rather confused. What exactly do you mean by "joining"
translators and by "merging" translators? It never occured to me that
these terms could be used to describe different things...

Maybe I should try to restate it: The idea of translator stacking is
that we have two distinct translators, but at runtime, transparently,
the functionality of one is migrated over to the other, so they run in a
single process.

-antrik-