[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A New Design of nsmux (discussed on IRC on 2008/09/07)

From: Sergiu Ivanov
Subject: Re: A New Design of nsmux (discussed on IRC on 2008/09/07)
Date: Thu, 18 Sep 2008 22:34:22 +0300


On Fri, Sep 12, 2008 at 7:37 PM, <olafBuddenhagen@gmx.net> wrote:
A lot of these design considerations hinge on the question how to handle
recursion, i.e. what behaviour is desired if we have a dynamic
translator, and do further magic lookups.

When doing magic lookups on a dynamic translator exporting a directory
tree, it's rather obvious that we want further magic lookups to work:
Say we use Flavio's unzip translator to look into an OpenOffice
document, we might want to use an XML translator on the individual files
in there. Insert any number of other examples.

The trickier question is what to do about lookups a dynamic translator
does on the underlying filesystem. (This was actually my main motivation
for suggesting that shadow translators would do lookups on the nsmux
filesystem -- this way, it should work without any special handling.)

Logically I think we do want that as well: If we use nsmux, it should
work just as if we used a filesystem that introduces namespace-based
translator selection direcly in the standard lookup mechanism, i.e. we
really want *all* lookups to have magic handling.

Do you mean by this that all lookups should pass through nsmux (or the
corresponding shadow translator) and be performed on the virtual nodes
provided by nsmux? (I'm asking to see whether I understand things
OTOH, I can't think of any specific case of a translator doing lookups
on the underlying filesystem, using directly or indirectly user-provided
(and thus potentially magic) file names. I'm not saying use cases do not
exist; but they are probably rare, so I'm not sure how much we should
care... Though definitely I feel some uneasiness about just ignoring the

It seems to me that we shouldn't ignore this matter. Frankly speaking,
I think it is a good idea for a translator to use its underlying node
for lookups, too (although I don't know any examples either).

BTW, I'm not quite sure now that it's really useful to have dynamic
translators continue working on shadow nodes when nsmux goes away. nsmux
is effectively serving as the main filesystem on which the dynamic
translators are invoked; and in other situations, translators do not
have any protection against the underlying filesystem going away
either... If they don't need the underlying filesystem, they will be
fine with being orphaned; otherwise, they produce failures.

I was going to ask you several days ago how did you estimate the
probability of the situation that nsmux was asked to go away and there
were still sane and healthy clients wanting to use their dynamic
translator stacks :-) My opinion is that such functionality is not
critical, although we should probably try to implement it some time,
it being a very interesting feature.

Considering that external shadow translators do not turn out as simple
as I had first hoped either, I'm not sure anymore whether it is a useful
idea at all. I haven't abandoned the idea alltogether yet, but probably
we shouldn't spend to much thought on it right now...

I don't expect external shadow translators to be very sophisticated
(although I pretty well may have an erroneous opinion :-) ). I think
that a shadow translator should be implemented as a one-node
libnetfs-based translator; in this case many details of its
functionality can just be migrated from the existing nsmux code, with
but minor changes.

There is another point that popped up during these considerations: What
if a client for some reason does multiple lookups on the same magic
filename? With the current approach, a new dynamic translator is created
every time. While this should pose no semantic problems, if all
translators are well designed (don't have any temporary state not
reflected in shared storage), it could be quite inefficient in extreme

I wonder thus whether it wouldn't be advisable to share dynamic
translators resulting from equivalent lookups. But this would again pose
problems with badly designed translators, that behave differently with a
single instance shared between clients than individual instances for
each client...

OTOH, I already found other situations in the past where we might have a
great many instances of the same translator invoked in parallel or
successively -- so I guess this is really an issue that needs to be
addressed on a more generic level sooner or later...

I've been thinking about this problem for about a month already, and
my opinion is that we should share dynamic translators. I wonder
whether it is nsmux's responsibility to care about badly designed
translators. If dynamic translators would not be shared, it could
indeed result in considerable overheads, and the worst thing is that
these expenses would appear in the cases of well-designed translators,
In any case, I think this is an issue worth putting on a ToDo list [...]

Already done :-)
> * The nodes with translator stacks on them should not go away until
> there are clients using these translator stacks.
> * No specific node caching should be done in nsmux, but rather the
> nodes should be created on demand and destroyed, when they are not
> used already (the destruction should happen precisely when the port to
> the topmost translator in the stack will receive a no_senders
> notification)
> * In the current implementation of nsmux, the shadow nodes are
> maintained by nsmux, therefore all the translator stacks should be
> shut down *before* nsmux goes away when requested, lest the
> translators should become orphaned.
> * A netfs node cannot go away when all references to it are gone, but
> there are senders using the port to the topmost translator in the
> translator stack sitting on this node. To make this possible, there
> should be a way to keep this node alive, probably in a more explicit
> fashion, than a node cache might have done it.
> * A shadow node may go away in case the real filesystem node it is
> mirroring goes away.

I don't really understand all of these points: some of them seem to
contradict what I actually said, so I'm not sure there hasn't been some
serious misunderstanding in the original discussion... Let me restate
the requirements in my own words:

The dynamic translators are created upon client request, and should thus
never go away, unless they are not needed by the client anymore. The
only exception is the underlying node going away for a translator that
actually needs the underlying node, in which case it's best for the
translator also to go away immediately, so the clients immediately get
the information, just as if they had been using the underlying node

Sincerely, I cannot really see how the items I listed contradict what
you say here... :-) Anyway, thanks a lot for dwelling upon the
details -- it did contribute to my better understanding of the matter.

While we can make sure these policies are met by any translators
developed as part of the project, we have no control of external
translators we launch of course -- they might stay around longer than

However, all normal translators will just go away after a couple of
minutes, if there are no outstanding client connections. The delay of
several minutes is actually not necessary for dynamic translators, as
presently they are never reused, so once the original client drops the
reference, we *know* the translator won't be used anymore. It might be
useful in this case to tell the translator explicitely to go away, so it
won't waste resources sitting around -- but I don't think this is very
important, so the ability or non-ability to do so probably should not
affect any other design decisions...

I've put a corresponding item on my TODO list. I think this problem
should be addressed quite soon, because having useless processes
running makes me feel uncomfortable.

All in all, as a first approximation, I think we should *never*
explicitely destroy dynamic translators -- anything else is just

OK, I understand that, but still, shouldn't the dynamic translators go
away when nsmux goes away? (IIRC, this is what we decided to be the
best choice in case of an nsmux without external shadow translators)
> * In the current implementation of nsmux the filter will ask for the
> underlying port to be opened with O_NOTRANS. However, the port the
> filter will get will point to a netfs node, not to the real filesystem
> node. It means, that when the filter will try to trace the translator
> stack, it will find itself only (in the case of 'file,,-u', for
> instance). It will not see the static translator stack on the real
> filesystem. (This problem is, probably, only specific to the current
> implementation of nsmux).

Not sure what you mean saying that this problem is specific to the
current implementation... It is obviously wrong -- it must (and can) be
fixed, no matter what the implementation.

I used to think that we are going to abandon the existing
implementation in the favour of the architecture with external shadow
translators, and it seemed to me that this problem is specific of the
current implementation. No doubt, I will fix it.

> * Why does a shadow translator have to be invisible on the translator
> stack? I don't think a filter will ever want to ignore a shadow
> translator, since it is pretty useless. And nobody, save for a filter,
> will possibly ever require the underlying node opened in O_NOTRANS
> fashion.

I'm not sure about the possibility of cases where it is actually
necessary for the filter (or any other translator using this mechanism)
not to be aware of the fact that it sits on a shadow node...

But that is not really the point. It's not as much about hiding the
shadow translator, as about showing the original translator stack
*instead* -- which obviously requires special handling in the parent
filesystem (nsmux).

Ah, I see. This is obviously connected with the filter translator
problem I mentioned before. All right, I have several ideas, which
could be useful in achieving the required goal.

> * Suppose nsmux is asked for 'file,,x'. It will set the shadow
> translator on a copy of its root node and will tell the shadow
> translator to mirror the file 'file'. Then it will set translator 'x'
> on top of the shadow translator. Do I understand everything right?

Yes, that *was* the idea...

OK :-) May I deduce from the fact that you stress the Past Tense in
your words that this mechanism is likely to suffer modifications in
the near future?

Sorry for the delay in answering... I'm afraid, Thursday and Friday
(and a bit of Sunday) will be the only days on which I will be able to
regularly afford a decent period of time spent thinking and/or working
on nsmux... (I will try to find extra time on other days, but it will
happen on a rather accidental basis).


reply via email to

[Prev in Thread] Current Thread [Next in Thread]