Re: [PATCH v3 2/6] mm: Allow dynamically requesting additional memory regions

[Daniel Axtens]

Hi Daniel,

>> I still would really prefer to bypass grub mm completely as described in
>> my other mail. If we are able to give memory back to fw, we can claim
>> 1GB chunks (on SLOF, PFW is going to be another issue) without having to
>> worry about where we put them and if we have enough memory to load a
>> kernel or initrd. It makes it much harder accidentally render your
>> system unbootable.
>
> I like your approach because you can return memory to the firmware (the
> situation in the UEFI is simpler because all memory allocated by the
> GRUB and in general by UEFI applications should be marked, usually, as
> "loader data" in UEFI memory map; then after EBS "loader data" should be
> treated as free memory; so, we do not need to return allocated memory
> regions to the UEFI explicitly; I am not sure it is possible in IEEE 1275
> firmware).

Once we leave openfirmware then we can do whatever we want with the
memory and we don't need to return it. However, this doesn't happen
until well after we get into linux*, and early parts of linux* use
openfirmware allocation methods, so unfortunately we do have to care
about the firmware's view of memory after we leave grub :(

As an aside, grub's ieee1275 linux loader iterates over the firmware's
view of available memory. It's an interesting thought experiment to
consider if we could:

 - also/alternatively look in Grub's view of the available memory for
   places to load the kernel
 
 - return a bunch of grub memory to firmware right before we jump to
   linux.

Maybe a project for another time.

> However, I think your solution have at least two problems:
>   - if you skip GRUB mm you cannot use relocator which means you cannot
>     load and use big initrds,

OK, I did not understand the role of the relocator - we don't use it to
boot linux under powerpc-ieee1275. I've done some reading up on it, I
found https://www.kernel.org/doc/html/latest/x86/boot.html very helpful.
If I understand correctly the relocator can be important if there are
things that have to be loaded within particular address ranges.

My hope was that people who used the mm-skipping were extremely diligent
to return the memory - e.g. you used it only within argon2 and then
immediately returned it. However, I understand memory management is a
hard problem and so it's reasonably not to want to add more ways to
shoot yourself in the foot.

>   - you add third family of memory management functions to the GRUB.

This is a fair criticism and I find it much more convincing now that I
know what the relocator is.

> So, I would still try to use mm. However, maybe we should improve
> algorithm which allocates memory at the GRUB init. In general we should
> have after init enough memory in the GRUB to store <handwaving> mm
> structures which describe whole or most of system memory plus an amount
> of RAM needed to run loaded core.img </handwaving>. Then I think we
> should stop seeing "firmware allocation" failures in later phases. The
> tricky part here is an algorithm which properly estimates initial memory
> requirements.

Ok so I still don't love sticking with mm, but I think I can live with
it. If we do go down this road:

 - The thing I really want for powerpc-ieee1275 is to have the platform
   code cap the amount of memory that you can acquire. I did a fair
   amount of research and maths on the edge cases of powerpc + PowerVM +
   current distro kernels and I think if grub allocates more than 1/4 of
   the RMA/OpenFirmware-visible memory we are going to really start to
   struggle to boot. I am happy to write the code to handle this on
   powerpc.

 - I'd like the grub mm to become a bit smarter: let's try to keep our
   regions from becoming fragmented and actively return entire regions
   to firmware if they become empty. Because powerpc-ieee1275 can be a
   weirdly memory-constrained platform in the early boot stages, I don't
   want to have to debug a situation where grub has chewed up a bunch of
   memory in mostly empty allocations and then can't allocate enough
   memory to boot.

   (To illustrate this with an example, say you are trying to verify a
   30MB kernel, it fails, you verify a 40MB kernel, it fails, you verify
   a 50MB kernel, it fails. Each of those verifications will require a
   completely new allocation from firmware, and if you can't give them
   back you've now locked up 120MB of memory!)

   I'm not saying we need a full-blown buddy allocator, although I
   certainly wouldn't object to a buddy allocator!

   I would like to say that I have time to do this but it would end up
   pretty far down my list. :(

 - For powerpc I would be happy to stick with the current initial memory
   claim - 32MB - or maybe consolidate with x86 and move to 64MB if that
   works for older/smaller systems - and then claim extra memory as
   required.

I don't know how much memory you want for EFI. If you want to have
memory to describe the memory map + extra for core.img, maybe you do
want to stick with a ratio, just a smaller ratio - maybe 1/8th?

Kind regards,
Daniel

* it is complex. prom_init runs as an openfirmare client, does a bunch
  of stuff, quiesces openfirmware (stops hardware DMAs etc) and then
  jumps to __start/Linux proper. Then we start the MMU, etc etc. At some
  point we take over exceptions too. (Take all of this with a grain of
  salt too, I'm not an expert in linux arch/powerpc boot process.)