On Sat, Sep 20, 2008 at 2:04 PM,
<olafBuddenhagen@gmx.net> wrote:
On Thu, Sep 18, 2008 at 10:34:22PM +0300, Sergiu Ivanov wrote:
> On Fri, Sep 12, 2008 at 7:37 PM, <
olafBuddenhagen@gmx.net> wrote:
As for the shadow translator, this is exactly what makes me sceptical
about the whole concept: If it turns out that the shadow translator
needs to implement the magic lookups as well, this would mean that it's
essentially just another nsmux instance. I wonder whether it would
actually be useful to launch another nsmux instance, or better to handle
it all in the main translator...
Hm, indeed... The shadow translator may very well become in the long
run just a variant of nsmux (maybe just a bit simpler one)...
> > BTW, I'm not quite sure now that it's really useful to have dynamic
> > translators continue working on shadow nodes when nsmux goes away.
> > nsmux is effectively serving as the main filesystem on which the
> > dynamic translators are invoked; and in other situations,
> > translators do not have any protection against the underlying
> > filesystem going away either... If they don't need the underlying
> > filesystem, they will be fine with being orphaned; otherwise, they
> > produce failures.
>
> I was going to ask you several days ago how did you estimate the
> probability of the situation that nsmux was asked to go away and there
> were still sane and healthy clients wanting to use their dynamic
> translator stacks :-) My opinion is that such functionality is not
> critical, although we should probably try to implement it some time,
> it being a very interesting feature.
The question is really not whether it's critical or not: but whether it
even makes sense. Is it really a good thing to have different behaviour
in the case of nsmux going away than a "normal" underlying filesystem
going away? I tend to doubt it.
I must acknowledge that I have been thinking about the sense of such
functionality ever since you suggested it, and I do agree with your
doubts.
> > Considering that external shadow translators do not turn out as
> > simple as I had first hoped either, I'm not sure anymore whether it
> > is a useful idea at all. I haven't abandoned the idea alltogether
> > yet, but probably we shouldn't spend to much thought on it right
> > now...
>
> I don't expect external shadow translators to be very sophisticated
Well, the problem is that even if the shadow translator itself turns out
simple, we know by now that we still need nsmux to provide some kind of
shadow nodes itself, for the shadow translators to sit on... Which makes
the whole idea rather questionable.
Yes, this fact has been embarrassing for me. As you've just said,
shadow translators become dependent of nsmux's going away.
Also, I'm not at all convinced about the shadow translator being simple
-- as pointed out above, it might be necessary to implement magic
lookups as well. Perhaps this can be avoided, but probably only at the
cost of further complicating interaction with nsmux; in which case it
might turn out that nsmux needs to be just as complicated as if it
handled all the shadowing in the main process...
Frankly speaking, I've just realized that when I said that the shadow
translator is going to be quite simple, absolutely unconsciously I was
considering the question how much code could be borrowed from nsmux
without major changes to it. Right now I do agree with the point
exposed above that in this way the shadow translator will rather
become a somewhat reduced clone of nsmux.
It might be possible though to find some smart way to factor the
functionality, so that recursion would be handled naturally by the
interaction between the translators, without need for any explicit
support in any of them. I don't think this would really mean a
simplification in the sum of code involved, but it might make things
simpler *conceptually*, which could still be a worthwhile goal...
Sounds rather tempting :-) It would be great if we had managed once to
bring things to such a wonderful conceptual level. Probably, further
development of the current implementation of nsmux can provide us with
some ideas.
> > I wonder thus whether it wouldn't be advisable to share dynamic
> > translators resulting from equivalent lookups. But this would again
> > pose problems with badly designed translators, that behave
> > differently with a single instance shared between clients than
> > individual instances for each client...
>
> I've been thinking about this problem for about a month already, and
> my opinion is that we should share dynamic translators. I wonder
> whether it is nsmux's responsibility to care about badly designed
> translators. If dynamic translators would not be shared, it could
> indeed result in considerable overheads, and the worst thing is that
> these expenses would appear in the cases of well-designed translators,
> too.
Well, the point is that it would complicate nsmux -- implementing a
specific solution to a generic problem...
I tend to think that we should just ignore this issue here; and only
when it actually poses serious problems in practice (as I'm sure it will
sooner or later, either with nsmux or in some other situation), it
should be addressed by an independent project, at the framework level.
OK, we can easily forget about this problem for now, especially since
it really does not create really serious problems right away. However,
I am inclined to think that sharing dynamic translators should be
implemented some time, because I like when resources that can be
shared are shared :-) Or are there some considerations that could show
that this idea is somehow malefic?
> > The delay of several minutes is actually not necessary for dynamic
> > translators, as presently they are never reused, so once the
> > original client drops the reference, we *know* the translator won't
> > be used anymore. It might be useful in this case to tell the
> > translator explicitely to go away, so it won't waste resources
> > sitting around -- but I don't think this is very important, so the
> > ability or non-ability to do so probably should not affect any other
> > design decisions...
>
> I've put a corresponding item on my TODO list. I think this problem
> should be addressed quite soon, because having useless processes
> running makes me feel uncomfortable.
It's not uncommon in the Hurd: You may have noticed that ls /dev will
create several dozen processes, which will only go away after a couple
of minutes.
Yes, that's right... I've already forgotten about this...
I'm not sure about Mach; but with a well-implemented kernel, idle
processes shouldn't cause any CPU overhead. The real problem is memory
use: If memory runs low, the idle translators will get swapped out --
and when the timeout passes, swapped in again only to go away...
Most likely it would be better to have a mechanism for idle translators
to go away under memory pressure, instead of being swapped out. Even if
a translator *will* be needed again at some point, I guess it wouldn't
be more expensive to start it anew, than doing all the swapping...
This is really quite the same problem as with various kinds of caches:
As long as there is plenty of memory, it's a good thing to keep things
around for possible reuse; but under memory pressure, it's usually
better to drop the caches than to do swapping... (Which is the main
point of Neal's thesis BTW.)
Oh, I hope that some time I will have the time and the knowledge to
read Neal's thesis... All this sounds interesting, it feels like both
important and fascinating :-)
Note however that all this only becomes really relevant in conjuction
with the problem discussed above, of having many individual translators
started in parallel or successively...
Yep, I understand that.
> > All in all, as a first approximation, I think we should *never*
> > explicitely destroy dynamic translators -- anything else is just
> > optimization.
>
> OK, I understand that, but still, shouldn't the dynamic translators go
> away when nsmux goes away? (IIRC, this is what we decided to be the
> best choice in case of an nsmux without external shadow translators)
Well, we arrived at the conclusion that most dynamic translators will
probably become useless when the underlying filesystem goes away, as
there is probably little point in using magic name lookups to start
translators not using the underlying node...
Yet, I don't see why we should explicitely kill them. I think they
should just deal with the underlying node going away, just as they would
have to deal with it when sitting on some other filesystem.
Hm, that's right. I've never looked at this question from this point
of view... OK, then, nsmux will never destroy dynamic translators (as
a first approximations, as you have said).
> > * Suppose nsmux is asked for 'file,,x'. It will set the shadow
> > > translator on a copy of its root node and will tell the shadow
> > > translator to mirror the file 'file'. Then it will set translator
> > > 'x' on top of the shadow translator. Do I understand everything
> > > right?
> >
> > Yes, that *was* the idea...
>
> OK :-) May I deduce from the fact that you stress the Past Tense in
> your words that this mechanism is likely to suffer modifications in
> the near future?
Not necessarily in the near future; but in view of the recursion
question, it's quite likely that it will have to change -- if we even
want a shadow translator at all anymore...
In any case, I don't consider it high priority. The only reason I
previously suggested implementing the shadow translator idea ASAP, was
that it seemed at that moment a vast simplification. But it's clear by
now that it isn't.
All right, this is clear.
> I'm afraid, Thursday and Friday (and a bit of Sunday) will be the only
> days on which I will be able to regularly afford a decent period of
> time spent thinking and/or working on nsmux... (I will try to find
> extra time on other days, but it will happen on a rather accidental
> basis).
I envy you for being able to plan it like that... :-)
In my case, I tend to start thinking about such stuff at all possible
moments, often when I'm supposed to be doing something completely
different. The result being that I get nothing done, because I spend
most of my time thinking about stuff I will never be able to implement
anyways, as I get nothing done... [sigh]
well-known, and new and interesting ideas are rare. Probably, there
be designing something new and fancy.
As for me, my University eats up much of my time and attention... I
University... The problem is that I want to maintain my grades at a
sufficient monetary resources.