Re: What shall the filter do to bottommost translators

[Sergiu Ivanov]

Hello,

On Thu, Jan 29, 2009 at 9:19 AM,  <olafBuddenhagen@gmx.net> wrote:
> On Tue, Jan 13, 2009 at 10:55:05PM +0200, Sergiu Ivanov wrote:
>> On Fri, Jan 9, 2009 at 9:01 AM, <olafBuddenhagen@gmx.net> wrote:
>> >  On Wed, Dec 31, 2008 at 02:42:21PM +0200, Sergiu Ivanov wrote:
>>
>> Too many instances of nsmux, too many context switches, too much
>> resources consumed for things that could possibly be implemented to
>> work faster with less effort, this was what I meant.
>
> I must admit that I don't see how recursive magic is relevant in this
> context... But perhaps that's because I'm mentally blocking any serious
> considerations concerning recursive magic for now :-)

When talking about recursive magic in this context I'm mostly guided
by the thought that applying translator magic recursively is usually
more challenging for nsmux than in other cases because resource
management issues are likely to become very obvious. That's why, the
traditional approach with a single nsmux instance is probably safer.

>> However, now that I've read your mail, only the relatively large
>> number of processes troubles me.
>
> Not so large actually. For each translator in the original tree, we get
> one nsmux instance.

Hm, really... I somehow thought that there should be one instance per
each dynamic translator. And then I thought about doing something like
ls -R /usr,,x/ which might launch quite a lot of translators.

>> > Now what about the control ports? The client could do a lookup for
>> > "/n/a/b" with O_NOTRANS for example, and then invoke
>> > file_get_translator_cntl() to get the control port of the translator
>> > sitting on "b". nsmux in this case forwards the request to the
>> > original "/a/b" node, but doesn't pass the result to the client
>> > directly. Instead, it creates an new port: a proxy control port to
>> > be precise. The real control port is stored in the port structure,
>> > and the proxy port is passed to the client. Any invocations the
>> > client does on this proxy control port are forwarded to the real one
>> > as appropriate.

Aha, now I can read this text differently: I did not notice that you
were talking about proxy *ports*, not *nodes*... Yes, of course, ports
are better and are an elegant and appropriate mechanism of dealing
with proxying control ports.

>> As I've already said, I'm strongly inclined to perceive a libnetfs
>> node in a more general meaning than a filesystem node.
>
> If there are fields in the node structure that only make sense for
> actual filesystem nodes, isn't that really a pretty strong indication
> that the netfs nodes are *not* a generic concept?...

Well, libnetfs nodes *are* a bit more general than actual filesystem
nodes, because the only fields that make them similar to filesystem
nodes are nn_stat and transbox, which contain the stat information and
the information about translators on the current node
correspondingly. The fields I was talking about are declared in struct
netnode by the user and these are the fields which make a libnetfs
node less general.

However, as I've already said, I've forgotten about the port structure
which, of course, is better.

> I really wonder why you are so set on misusing them. Aside from the
> confusion this would create -- is there actually anything it would buy
> us over a plain port structure (with the required data in the hook)? Is
> there any kind of handling we need that netfs does on the netfs nodes
> and we would need to do manually otherwise? Are you sure it wouldn't
> actually cause *more* trouble, having to take care that the pseudo-nodes
> are *not* handles as real FS nodes?...

I believe that working with plain port structures is at least more
efficient that with libnetfs nodes. On the other hand, whatever might
the special handling done by libnetfs be, we can always have a look in
its code and adapt that implementation to our needs. This strategy
gives us at least two benefits: efficiency and correctness in abiding
by concepts.

>> > With the "distributed" nsmux, things would work a bit differently.
>> > Again there is a lookup for "/n/a/b/c", i.e. "a/b/c" on the proxy
>> > filesystem provided by nsmux; again it is forwarded to the root
>> > filesystem, and results in "a/b'" being returned along with a retry
>> > notification. The distributed nsmux now creates a proxy node of
>> > "a/b" (note: the untranslated b). It starts another nsmux instance,
>> > mirroring "/a/b'", and attaches this to the "a/b" proxy node.
>> >
>> > Again, the client will finish the lookup. (By doing the retry on the
>> > new nsmux instance.)
>>
>> Aha, sounds great! I've had no inkling even as to such possibility :-)
>> Very beautiful idea!
>
> It's rather ironic that you take to the idea just when I discovered that
> it's flawed and pretty much gave up on it...

Well, following what you say later in this email, this is the
disadvantage of asynchronous communication...

>> Hm... I'm thinking of the following thing: can we make nsmux behave
>> differently in different contexts? Namely, normally requests like
>> fsys_goaway should go to the translator in the real filesystem,
>> however, at some point, nsmux will treat them as directed to itself.
>>
>> One possibility to implement this would be adding a special command
>> line option which would tell nsmux to forward RPCs to the real
>> translator. Note that there are also runtime options available for a
>> translator (and these options are classically the same as simple
>> command line options) and we can modify them via
>> fsys_{get,set}_options. When a new instance of nsmux is started by an
>> already-existing instance, it will be started with this special
>> command line switch. All meaningful RPCs will be forwarded to the
>> translator in the real tree. When the parent instance of nsmux would
>> want to shut down a child, it would just reset the special option in
>> the child (fsys_set_options) and do an fsys_goaway on the child's
>> control port.
>>
>> OTOH, when a *user* (not the parent instance of nsmux) would like to
>> shutdown a child instance (which, BTW, may not be desirable, what do
>> you think?), they can use the fsysopts command to remove the
>> corresponding option and then do settrans -ga, for instance, to
>> achieve the shutdown.
>
> You are right that the child nsmux instances probably never need to be
> addressed by the user directly; they can be totally transparent. The
> main instance is different, though.
>
> Your suggestion actually wouldn't work: fsys_set_options after all is
> also just an RPC on the control port, and would have to be forwarded to
> the original translator...

Ah, really :-(

> Certainly we could come up with some other mechanism to do the switching
> -- but please, for $Deity's sake, don't.
>
> It would mean terrible usability; it would introduce some state that
> would break tools not aware of it; and if the state is global (which it
> would be in the variant you suggested), it would introduce race
> conditions as well. Quite frankly, the whole idea is an immensly ugly
> hack.

No, I won't try to come up with another mechanism because what I
realized well about fsys_set_options thing was that it looked like a
hack already, to tell nothing about something even more artificial.

> Moreover, I don't even see why we would want to do something like that.
> After all, this effectively does introduce the concept of proxy control
> ports and real control ports for the proxy server -- only they are
> multiplexed over a single port...

Sounds awfully tricky...

> So again the same conclusion: We do need the proxy control ports after
> all -- although an attempt to avoid them was precisely what made me come
> up with this distributed design in the first place...

I see...

> Having said that, the idea is not entirely without merit. Effectively it
> means that when traversing a translator stack, instead of the single
> nsmux instance creating several proxy nodes for the same filesystem
> location at different translation levels, we get only one proxy node per
> location in the main nsmux, and the other levels are handled by other
> nsmux instances. I must admit to a certain elegance in this... But I'm
> not quite sure whether it would indeed make things clearer and simpler,
> rather than more complicated.

That's a difficult question... However, I have a habit to believe that
modularity is often better than a monolithic approach, so I'm rather
inclined to think that however complicated things might seem in such
an approach, this idea could enhance the extensibility of nsmux...

> I suggest proceeding like with the shadow nodes: It could be beneficial
> to think of them as distinct translators conceptually, but until we
> fully understand the implications, better leave the actual
> implementation as it is...

I've suddenly realized that, although I remember you saying that
shadow nodes should be thought as separate translators, I do not fully
understand the reason for this. Why won't we think of shadow nodes as
of separate filesystem nodes, bearing a translator?..

>> > This process needs to be done for each filesystem server the client
>> > contacts. This is why a reauthentication needs to be done after
>> > crossing a translator boundary -- or at least that is my
>> > understanding of it. The retry port returned to the client when a
>> > translator is encountered during lookup, is obtained by the server
>> > containing the node on which the translator sits, and obviously
>> > can't have the client's authentication; the client has to
>> > authenticate to the new server itself.
>>
>> Hm... Again, the problem about RETRY_REAUTH, which seems to happen
>> only when looking up ``..''...
> [...]
>> It seems to me, anyways, that in the standard implementation of
>> netfs_S_dir_lookup RETRY_REAUTH happens when looking up ``..'' on the
>> root node of the filesystem.
>
> ... which is totally beyond me.
>
> Truth to be told, I have no clue about the implications of the auth
> mechanism on proxying...

I tried to look into some places where FS_RETRY_REAUTH is used, but I
still cannot figure out why it is not used when crossing translator
boundaries, as you have suggested... I'll try to think further about
it.

>> > BTW, I just realized there is one area we haven't considered at all
>> > so far: Who should be able to start dynamic translators, and as
>> > which user should they run?...
>>
>> nsmux starts dynamic translators using fshelp_start_translator, which
>> invokes fshelp_start_translator_long. This latter function creates a
>> new task and makes it a *child* of the process in which it runs.
>> Therefore, dynamic translators are children of nsmux and can do
>> anything the user who starts nsmux can do.
>
> Yeah, I'm aware of that.
>
>> Is this OK? Or are you thinking of the possibility that nsmux be
>> started at startup with root priveleges?..
>
> I think it's better to look at it the other way round: First decide what
> the *desired* behaviour is, and only then consider how to achieve it...
>
> But that's a completely new discussion, and I'd rather clear up the
> other points first...

I've put another item on my TODO list...

>> > And it is actually not entirely correct: Let's say we use a filter
>> > to skip "b" (and the rest of the stack on top of it). The filter
>> > would traverse the stack through all this shadow machinery, and
>> > would end up with a port to the root node of "a" that is still
>> > proxied by the various shadow nodes. The filter should return a
>> > "clean" node, without any shadowing on top of it; but the shadow
>> > nodes don't know when the filter is done and passes the result to
>> > the client, so they can't put themselfs out of the loop...
>>
>> While this may be a bit of offtopic, I would like to ask you what the
>> issue with correctness is? I can see only the sophisticated shadow
>> machinery in it's complicated action, but everything seems all right
>> about the way it handles information: nothing gets lost.
>
> Think about what happens when the filter is done, and returns the
> resulting port to the client. It skipped a couple of translators off the
> top -- returning a port to some node in the middle of the stack. The
> effect should be exactly as if someone started at the bottom of the
> translator stack, followed the first couple of retries, but then
> suddenly stopped.
>
> If there are any static translators present on this node, they could be
> traversed normally in further requests. Dynamic translators however
> should disappear: They exist only to their clients. Once we looked up
> something outside of the nodes they provide -- which we did with the
> filter -- we can't get back from there. We are no longer a client.

Aha, I see. It looks clear now.

> But that's not what happens in the scenario I described. The filter
> never gets a port to the real middle node. What it actually gets is a
> node proxied by all the shadow nodes in the stack. It is proxied by
> them, because the filter would have to see the dynamic translators if it
> continues traversal. But once the filter is done, the client should get
> a real port to the middle node, with no shadow nodes in between. (Of
> course the middle node is still mirrored by a "normal" nsmux proxy node
> if it is a directory, so that further magical lookups are possible; but
> the rest of the proxy stack should be gone.)
>
> Most of the time the still active shadow nodes would do no harm, except
> for killing performance. If another filter is applied through a magic
> lookup, this would attach a new shadow node right at the middle node;
> the traversal would be diverted right there, before ever reaching the
> offending old shadow translators.
>
> However, if someone does further translator lookups by some other means,
> the whole stack would be visible; and as soon as he arrives at the nodes
> where the old shadows sit, they would cry "here", and happily report the
> dynamic translators which should no longer be visible.
>
> Admittedly that is a quite specific case. The performance issue is
> probably more relevant in practice.

Yes, I also thought about this... I'm not trying to assess how bad is
that the shadow nodes behave like that in these circumstances... The
only problem I can see now (which might, however, prove critical...)
is that applications that do some exotic port manipulations (like
filters) should be designed specifically to comply with nsmux
standards. Could you please point out other issues?

> Unfortunately I realized now that traversing top to bottom only
> partially solves this problem; we will have to address it explicitely
> after all... [sigh]

Could we possibly define a new RPC (a bit frightening for me,
unfourtunately) to make the shadow nodes yield the real port? That is,
when the client should invoke this RPC on the port it obtains from the
filter, they should get the port directly, not involving shadow nodes?
(I mean, get a proxy node containing a direct reference to the real
node).

>> > It seems that my initial intuition about traversing top to bottom
>> > being simpler/more elegant, now proves to have been merited...
> [...]
>> > Note that unlike what I originally suggested, this probably doesn't
>> > even strictly need a new RPC implemented by all the translators, to
>> > ask for the underlying node directly: As nsmux inserts proxy nodes
>> > at each level of the translator stack, nsmux should always be able
>> > to provide the underlying node, without the help of the actual
>> > translator...
> [...]
>> Yes, this is right, nsmux can do that. However, how do you suggest the
>> top-to-bottom traversal should take place (in terms of RPCs)? Shall we
>> reuse some existing RPCs?..
>
> Not sure. Any suggestions? :-)

I am somehow inclined to be against reusing the existing RPCs... It
looks a bit evil, especially if we would like some exotic use-cases to
be feasible (like setting nsmux, descending into the mirrored tree and
launching another instance of nsmux on top of a mirrored node or even
in a dynamic translator stack). I'd probably stand for introducing and
additional RPC, although it would require playing with MiG, which I've
tried to avoid so far...

>> Hm, and another (quite important) question: what do we to static
>> translator stacks? There are no shadow nodes between static
>> translators, so nsmux has no chance of traversing the static stack in
>> top-to-bottom fashion...
>
> Seems my explanation wasn't totally clear after all...
>
> It's not the shadow nodes that inform about the next-lower node in the
> stack. The shadow nodes only divert queries regarding what translator
> sits on the node they shadow; all other RPCs are forwarded to the actual
> shadowed node, which is a normal nsmux proxy node. This one handles the
> requests for the next-lower node.

Aha, clear; I can see it now.

> For static translators, there is no shadow, but there is still the proxy
> node that can handle this request.

I see... Still, I cannot really discern the solution to the problem:
the most natural way to traverse a translator stack is bottom-to-top
(since the standard RPC provide the technique). As we have said not a
single time before, a top-to-bottom traversal cannot be implemented
but with some additional facilities (like proxy nodes, as far as I can
understand). I hope I've not said anything orthogonal to your words so
far.

Now, what are the additional facilities for static translators? Is it
right that nsmux will have, for instance, to traverse the static
translator stack bottom-to-top to create a list which could then be
traversed in a kind of top-to-bottom fashion?.. I feel I'm missing
something fundamental... The problem with the list is, of course, that
it will have to be updated at some intervals, which is bad :-(

>> As for the questions about the bottom-to-top approach I'm very
>> interested to know the following: did you manage to design all this
>> shadow-proxy machinery just in your mind, or did you put something
>> down on paper? :-) I'm asking because I had to redraw the same graph
>> for about five times before I got the idea :-)
>
> Indeed I designed it all in my mind (I have a very good visual
> imagination); but it really took me quite long, and some bits only
> became clear while writing it up.

Wow! :-)

> I'm totally aware though that this kind of stuff is very hard to
> communicate without a drawing -- which is why I suggested to start
> making drawings quite a while back...
>
> Unfortunately making drawings that make sense when studied
> asynchronously is so damn hard :-( It's so much simpler to make a
> drawing in real time, and explain while going along... The great
> advantage of a face-to-face meeting.

That's right... I wish there were no borders and people could download
open-source vehicles and build them at home ;-) Face-to-face meetings
would take place more often... :-)

>> Ah, and another thing: as far as I could notice from your
>> descriptions, the role of proxy nodes is rather humble, so my question
>> is: what is the reason of keeping them at all? I'd rather not include
>> proxy nodes in between dynamic translators. When a client invokes an
>> RPC on its port to a proxy node, nsmux knows (obviously) to which
>> shadow node the proxy node is connected, so nsmux can ignore the proxy
>> node in the matter of setting translators. My idea is to keep proxy
>> nodes as a kind of ``handles'' which are meant only to be seen to the
>> client and shall not be employed in the internal work.
>
> Not sure what you are trying to achieve here. You must be aware that we
> need both the shadows and the normal proxies as explicit nodes, as ports
> for both are handed out to external processes. (The normal proxies are
> handed out to clients doing a lookup, and the shadows are handed to the
> dynamic translators attached to them.)

Yes, I feel this is pretty obvious.

> Of course in the monolithic implementation we are free to organize the
> data internally as we like -- but I don't see any point in anything else
> than storing all data right with the respective nodes it belongs to,
> shadow or otherwise...

So, it's a matter of keeping things clear and complying with the
concept? I have nothing against it, I was just interested in whether
inclusion of proxy nodes in the internal structure of a dynamic
translator stack was a vital thing or a conceptual thing.

>> > PS. Your mails contain both plain text and HTML versions of the
>> > content. This unnecessarily bloats the messages, and is generally
>> > not seen favourably on mailing lists. Please change the
>> > configuration of your mail client to send plain text only.
>>
>> I'm sorry, it wasn't intended. I'm using web interface for mail
>> management, and when I recently tried to switch browsers, the change
>> has influenced my messages. However, I've come back to my usual
>> Firefox, so no more things like that should occur.
>>
>> Actually, I was not even aware that the other browser was putting
>> garbage in my mail :-(
>
> It's not the browser. The browser just submits a text input field. It's
> the mail client that generates the mail (with the pointless HTML part)
> from that -- a web mail client isn't any different from a normal mail
> client running on the user's PC in that regard.
>
> There must be some configuration option to disable the HTML.
>
> (Yes, this last mail has the HTML part, just like the others.)

That's bad :-( I found some extra button and I hope it will solve the
problem... I've never had such problems before, though :-( That's why
I thought it was the browser...

Regards,
scolobb