Filter design for nsmux (was: What shall the filter do to bottommost translators)

[Sergiu Ivanov]

Hello,

On Wed, Feb 18, 2009 at 10:01 PM,  <olafBuddenhagen@gmx.net> wrote:
> As I already pointed out, sooner or later we will problably need some
> framework to conflate simimlar translator instances in a single process
> -- while dynamic translators are likely to make the problem evident in
> many cases, the problem itself is not specific to them...

Indeed, I have also thought about reusing a single translator instance
for similar requests this week, and this does solve the
number-of-processes issue.

However, I remember you pointing out that some of the translators may
be incapable of dealing with multiple clients... Do we forget about
such possibility or shall we think about this further?..

>> I've suddenly realized that, although I remember you saying that
>> shadow nodes should be thought as separate translators, I do not fully
>> understand the reason for this. Why won't we think of shadow nodes as
>> of separate filesystem nodes, bearing a translator?..
>
> Who says we don't?...
>
> Shadow nodes are a special kind of proxy nodes. (And are meant to bear a
> dynamic translator.) As such, they are separate nodes, representing an
> alternative view on the same filesystem location.
>
> At some point I suggested implementing them as distinct translators,
> because they behave differently than the normal nsmux proxy nodes, and I
> thought it could simplify nsmux to factor them out -- but then I
> realized that nsmux would still need special handling for them, so not
> much gained here. However, I believe that it's still useful to think of
> them as separate translators conceptually, because the interaction
> between shadow nodes and other nodes becomes more explicit that way.

Aha, clear, thank you.

>> >> > And it is actually not entirely correct: Let's say we use a
>> >> > filter to skip "b" (and the rest of the stack on top of it). The
>> >> > filter would traverse the stack through all this shadow
>> >> > machinery, and would end up with a port to the root node of "a"
>> >> > that is still proxied by the various shadow nodes. The filter
>> >> > should return a "clean" node, without any shadowing on top of it;
>> >> > but the shadow nodes don't know when the filter is done and
>> >> > passes the result to the client, so they can't put themselfs out
>> >> > of the loop...
> [...]
>> The only problem I can see now (which might, however, prove
>> critical...) is that applications that do some exotic port
>> manipulations (like filters) should be designed specifically to comply
>> with nsmux standards. Could you please point out other issues?
>
> Indeed, the problem is that the dynamic translators are not entirely
> transparent -- some things that normally work, could break, unless
> specially handling dynamic translators.
>
> What other issues do you need?...

Understood; I thought there were other issues which I failed to see.

>> > Unfortunately I realized now that traversing top to bottom only
>> > partially solves this problem; we will have to address it
>> > explicitely after all... [sigh]
>>
>> Could we possibly define a new RPC (a bit frightening for me,
>> unfourtunately) to make the shadow nodes yield the real port? That is,
>> when the client should invoke this RPC on the port it obtains from the
>> filter, they should get the port directly, not involving shadow nodes?
>
> That is actually the solution I was considering. I don't really like it,
> because this way nsmux is not transparent to the filter: the filter
> needs to be aware that it is run by nsmux, and handle it specially. I
> can't think of any other approach, though.
>
> (At least this way nsmux/the shadow nodes would be transparent again
> once the filter is done.)

By saying ``transparency'' do you mean that at the invocation of this
new special RPC, nsmux will give to the client a proxy of the
translator port directly, without employing shadow nodes?

>> >> However, how do you suggest the top-to-bottom traversal should take
>> >> place (in terms of RPCs)? Shall we reuse some existing RPCs?..
>> >
>> > Not sure. Any suggestions? :-)
>>
>> I am somehow inclined to be against reusing the existing RPCs... It
>> looks a bit evil, especially if we would like some exotic use-cases to
>> be feasible (like setting nsmux, descending into the mirrored tree and
>> launching another instance of nsmux on top of a mirrored node or even
>> in a dynamic translator stack).
>
> Indeed it is "a bit evil" -- though not really different than what you
> did previously, with the kludge for obtaining the untranslated version
> of the underlying node...

Yep, that's true... And I'd rather avoid such hacky behaviour in the
future...

>> I'd probably stand for introducing and additional RPC, although it
>> would require playing with MiG, which I've tried to avoid so far...
>
> No need to fear it, writing new RPCs is almost trivial. (I did it
> recently for my KGI port.)

That's great news :-)

> My major concern here is deciding whether to add the new RPC(s) to the
> existing interfaces, or create a completely new one...

I'm not sure I can fully comprehend the implications of this
statement... Could you please explain further?..

>> Still, I cannot really discern the solution to the problem: the most
>> natural way to traverse a translator stack is bottom-to-top (since the
>> standard RPC provide the technique). As we have said not a single time
>> before, a top-to-bottom traversal cannot be implemented but with some
>> additional facilities (like proxy nodes, as far as I can understand).
>
> There is really no difference here -- we need special facilities in
> either case. When traversing bottom-to-top, we need a way to obtain the
> completely untranslated node; when traversing top-to-bottom, we need a
> way to obtain the underlying node, i.e. one translation layer lower.

Yes, and this is what I'm talking about: there is no already existing
RPC for going one translation layer lower. Isn't one of the goals of
introducing proxy and shadow nodes to make it possible for nsmux to go
downwards the translator stack?..

>> Now, what are the additional facilities for static translators? Is it
>> right that nsmux will have, for instance, to traverse the static
>> translator stack bottom-to-top to create a list which could then be
>> traversed in a kind of top-to-bottom fashion?.. I feel I'm missing
>> something fundamental...
>
> I'm sharing this feeling... ;-)
>
> Remember that to set a filter on top of the static stack, the client
> needs to look up the (statically) translated node first. Looking up a
> node with a translator stack means following retries at each level of
> the stack, until you arrive at the top. nsmux creates proxy nodes at
> each level while forwarding the retries. Each proxy node references the
> one below it. That's all the list you need...

Aha, clear :-) I used to think about using proxy nodes somehow, but I
didn't figure out this possibility...

>> > It's so much simpler to make a drawing in real time, and explain
>> > while going along... The great advantage of a face-to-face meeting.
>>
>> That's right... I wish there were no borders and people could download
>> open-source vehicles and build them at home ;-) Face-to-face meetings
>> would take place more often... :-)
>
> Well, I'm actually ambiguous about physical meetings with people coming
> from all over the world. They are very useful, but also a serious burden
> on the environment...

Hm, didn't think of that...

>> > You must be aware that we need both the shadows and the normal
>> > proxies as explicit nodes, as ports for both are handed out to
>> > external processes. (The normal proxies are handed out to clients
>> > doing a lookup, and the shadows are handed to the dynamic
>> > translators attached to them.)
> [...]
>> So, it's a matter of keeping things clear and complying with the
>> concept? I have nothing against it, I was just interested in whether
>> inclusion of proxy nodes in the internal structure of a dynamic
>> translator stack was a vital thing or a conceptual thing.
>
> I still don't understand what you are getting at. If we need the nodes
> for things to work, it's pretty vital to have them, isn't it?...
>
> What are you trying to optimize? You can't leave out the nodes. You
> *could* duplicate the proxy data in the shadow nodes, so you don't have
> to follow the reference from shadow node to proxy node in some cases --
> but what's the point? Save following one reference?...

Hm, really... I'm sorry, I thought we could merge the functionality in
a single node because it seemed to me that another node would mean
another context switch...

It looks as if I should be more attentive while reading the mail...

>> > There must be some configuration option to disable the HTML.
> [...]
>> I found some extra button and I hope it will solve the problem...
>
> It helped -- no HTML baggage this time :-)

Oh yeah :-) I'm happy :-)

>> I've never had such problems before, though :-( That's why I thought
>> it was the browser...
>
> More likely you simply didn't know of the HTML baggage before... I only
> realized it when the mails were getting absurdly large.

Yes, you're right... I couldn't imagine before that the mail client
would add some extra stuff to my mail without letting me know :-(

Regards,
scolobb