Re: [PATCH] RISC-V: Place DTB at 3GB boundary instead of 4GB

[Atish Patra]

On Wed, 2021-01-06 at 08:04 +0800, Bin Meng wrote:
> Hi Atish,
> 
> On Wed, Jan 6, 2021 at 7:44 AM Atish Patra <Atish.Patra@wdc.com>
> wrote:
> > 
> > On Tue, 2021-01-05 at 11:11 +0800, Bin Meng wrote:
> > > On Fri, Dec 18, 2020 at 5:48 AM Atish Patra <atish.patra@wdc.com>
> > > wrote:
> > > > 
> > > > Currently, we place the DTB at 2MB from 4GB or end of DRAM
> > > > which
> > > > ever is
> > > > lesser. However, Linux kernel can address only 1GB of memory
> > > > for
> > > > RV32.
> > > > Thus, it can not map anything beyond 3GB (assuming 2GB is the
> > > > starting address).
> > > > As a result, it can not process DT and panic if opensbi dynamic
> > > > firmware
> > > > is used.
> > > > 
> > > > Fix this by placing the DTB at 2MB from 3GB or end of DRAM
> > > > whichever is lower.
> > > > 
> > > 
> > > Fixes: 66b1205bc5ab ("RISC-V: Copy the fdt in dram instead of
> > > ROM")
> > > 
> > 
> > Just to clarify, it is not a bug in the original patch. This bug
> > appeared in recent kernel because kernel(from v5.10) doesn't use
> > fixmap
> > for DT processing anymore. Thus, the DT has to be placed at
> > location
> > that can be addressed by kernel.
> 
> Thanks for the clarification.
> 
> > 
> > I will add the fixes tag.
> > 
> 
> Did you get a chance to trace to another possible kernel bug?
> 

Yeah. Let me explain. Apologies for long explanation in advance :)

5.10 kernel + memblock_enforce_memory_limit patch actually boots but
crashes after a point with 2GB of memory. You did not see any boot logs
as you were not using earlycon argument.

The issue was masked by the a kernel patch [3] that got merged in 5.11-
rc1. The patch[3] also throws a warning on RV32 because of a another
bug[1].

The real issue with 2GB memory in RV32 is that Linux kernel treats any
pointers within last 4K byte of addressable memory as error [2].

In the absence of the patch[3], kernel unflattens the DT towards the
end of the memory. These memory regions are absolutely fine and
addressable. I even verified the memory region in qemu and can print
the DT strings using gdb. However, kernel thinks as error by mistake
because of [2].

The kernel patch[3] masked off the 2GB problem because it tried to
allocate bunch of memblocks and failed. That lead memblock to allocate
memory for unflattend DT at a different place which helped. 

I have a fix solve the bug introduced by that patch[3] and can
reproduce the 2GB problem with 5.11-rc2 as well with the kernel fix. 

The simple solution is here to place the DTB at an address farther away
from the end of 3GB. Kernel puts some other data structure towards the
end of the addressable memory but never accesses those last 4K bytes of
memory. Thus we don't see a problem anymore. This problem manifests
only when you have a virtual address within 4K of addressable memory.
It is only possible with RV32 and 2GB combination.

fdt_addr = QEMU_ALIGN_DOWN(temp - fdtsize, X * MiB); 

where X must be > 4.

I tested with 8/16/32 MB aligned address and it works fine every time.
I think the safest option should be 32MB offset.

Kernel should also be fixed so that memblock allocation code should not
allocate anything within last 4K of addressable memory.

[1]
http://lists.infradead.org/pipermail/linux-riscv/2021-January/004086.html
[2]
https://github.com/torvalds/linux/blob/master/include/linux/err.h#L22

[3] 0f522592ab9f RISC-V: Add kernel image sections to the resource tree

> Regards,
> Bin

-- 
Regards,
Atish