Re: self-paging

[Bas Wijnen]

On Wed, Nov 30, 2005 at 03:49:48PM +0100, Marcus Brinkmann wrote:
> > Hm, we have a slightly different definition then.  What I call self-paging
> > is that the application itself determines which page will be discarded or
> > swapped out on memory pressure.  Informing the application is one way to
> > do that, but it can be done differently (as in my proposal, the
> > application just builds a prioritized list of pages, and the swapping out
> > and discarding happens without notification).
> 
> When I proposed something similar to Neal, he said that it may be important
> that the application _knows_ about how much memory is going away, so it can
> calculate for that.  It makes sense to me.  For example, a (soft) real time
> renderer could switch to a lower resolution, or a stream compressor could
> use a less memory intensive algorithm, both to avoid trashing.

That makes sense.  However, I think my idea already allows that, because the
application can request the number of pages that it still has.  That is, there
is no notification when a page is swapped out, but when the application gets a
page fault, it will probably want to know what its new limit is, and it can
find out by asking.  As long as it didn't get a page fault, it appearantly
didn't need the pages.

Anyway, I just thought about a way for the application to get the instant
notification anyway, so we may assume that it gets it: If the currently
executing code is located in the page at the end of the list, it will be the
first one to be swapped out.  So you get a page fault the moment you get a
time slice again.  Of course this paging strategy is very bad
performance-wise, but it does provide instant notifications about being
swapped out.

This doesn't work for an increase of the quota.  Also, when the quota
increases it makes sense that we do want a notification, because normal
programs may use it.  So forget about the "no notification" part.  It wasn't
essential anyway.

> > What is essential to self-paging in my view is that the decision which
> > page to swap out is made locally, not that that is done at the moment
> > there is pressure (that is, in my proposal the decision is prepared
> > beforehand and used by the system when needed).
> 
> EROS has the concept of working sets with priorities.  This gives
> advice to the paging system how you group your memory pages.
> 
> Maybe the term "advisory page mechanism" (or something like that) is
> better to describe your idea rather than self-paging.  In particular,
> you said in your original mail that the system swaps out pages.  This
> is one indication that what you describe is not really self-paging.

No, I don't think this is quite correct.  You are right that the actual paging
out of memory to disk is still done by "the system", not by the application.
So strictly speaking it may not be self paging.  But it is much stronger than
advisory.  The system is not allowed to swap out pages in a different order
than the process indicates.  This is why I call it self-paging: because the
paging decisions ("which page will be swapped out next?") are made locally.

The other type of paging decisions ("which process will have a page swapped
out _right now_?") is indeed not made by the process.  However, if it is, I
don't see how a denial of service can be prevented when a process decides not
to cooperate.  As discussed before, the problem is that it is not possible to
see if a task is not cooperating or just slow.

So I think this is the closest we can get to self paging without opening the
system to attacks.

> It seems to me that apart from this difference, your proposal boils
> down to: "Add noise to the system to reduce the bandwidth of covert
> channels", which is the typical approach if all else fails.

I wouldn't call the delays noise, but rather a low-pass filter.  Adding noise
is an additional option, which would be to randomly change the quota a bit
every now and then.

Thanks,
Bas

-- 
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://129.125.47.90/e-mail.html

signature.asc
Description: Digital signature