Re: [Fwd: [patch #2508] ext2fs support for large store (> 1.5G)]

[Ognyan Kulev]

Marco Gerards wrote:
> Ognyan Kulev <ogi@fmi.uni-sofia.bg> writes:
>
> > Marco Gerards wrote:
> No, it is not fixed.  It is just that the kernel determines the amount
> of mapped blocks (all individual mappings) and when which mapping
> should be removed.

This is the "next level".  I though about it: 
http://www.mail-archive.com/bug-hurd@gnu.org/msg07001.html .  But we 
better focus on fixed size of cache, as changing to floating mapped 
blocks is not trivial task.

> > It's just that: GNU Mach is instructed to notify us about evicted
> > pages, but it doesn't.  This is not rare case.  The code above is
> > work-around. It first touches all pages and then synces them all.
> > Fortunately this seems to always work.
>
> Does this problem cause any problems for currently use ext2fs?

No, because current ext2fs doesn't need notification on evicting page. 
This seems to happen only on precious clean pages and on eviction 
sometimes Mach "forgets" to notify us although they are marked as 
precious.  Current ext2fs doesn't rely on precious pages in any way.

> > The "starving" happens when all blocks are in use.  That is, each
> > block has reference count > 0 and it can't be replaced.  So the best
> > we can hope of is to wait for some other thread to release block.
>
> Ok, I understand.  Thinking of this (the amount of threads and the
> amount of blocks that will be referenced by just one thread) it will
> be very unlikely that this will ever happen.
>
> Because this rarely happens it would be wise to test this (perhaps by
> setting DISK_CACHE_BLOCKS to a lower value?).  Did this ever happen
> while you were testing it?

Usually I run with DISK_CACHE_BLOCKS=20.  The minimum seems to be around 
10-15.  Below that it's impossible to compile large source like Hurd's. 
 The uploaded patch has DISK_CACHE_BLOCKS=100, but it should be 
largened, of course.

So yes, on low values (10-15) I observed this "starving" and depending 
on what you are doing, you either enter endless loop, or the sleep does 
its job and ext2fs continues.

> I am thinking if it is possible to prevent this from happening
> completely.  I will come back to this later.

Probably the starving can't be avoided.  And the other case, when no 
evicted page is found, will be very, very rare when cache is 512M (for 
example).

> BTW, did you find this bug I mentioned?  I must admit that I did not
> have the time to look into it yet. :(

No, probably in the weekend, but this is not certain.  I have second 30G 
disk that will become one partition for testing.

Regards,
ogi