Re: [PATCH v2 2/2] ppc/spapr: Initialize max_cpus limit to SPAPR_NR_IPIS.

[Philippe Mathieu-Daudé]

On 23/11/23 09:47, Cédric Le Goater wrote:
> On 11/23/23 06:03, Harsh Prateek Bora wrote:
> > Hi Philippe,
> >
> > On 11/22/23 16:46, Philippe Mathieu-Daudé wrote:
> > > Hi Harsh,
> > >
> > > On 22/11/23 10:28, Harsh Prateek Bora wrote:
> > > > Initialize the machine specific max_cpus limit as per the maximum range
> > > > of CPU IPIs available. Keeping between 4096 to 8192 will throw IRQ not
> > > > free error due to XIVE/XICS limitation and keeping beyond 8192 will hit
> > > > assert in tcg_region_init or spapr_xive_claim_irq.
> > > >
> > > > Logs:
> > > >
> > > > Without patch fix:
> > > >
> > > > [root@host build]# qemu-system-ppc64 -accel tcg -smp 10,maxcpus=4097
> > > > qemu-system-ppc64: IRQ 4096 is not free
> > > > [root@host build]#
> > > >
> > > > On LPAR:
> > > > [root@host build]# qemu-system-ppc64 -accel tcg -smp 10,maxcpus=8193
> > > > **
> > > > ERROR:../tcg/region.c:774:tcg_region_init: assertion failed:
> > > > (region_size >= 2 * page_size)
> > > > Bail out! ERROR:../tcg/region.c:774:tcg_region_init: assertion failed:
> > > > (region_size >= 2 * page_size)
> > > > Aborted (core dumped)
> > > > [root@host build]#
> > > >
> > > > On x86:
> > > > [root@host build]# qemu-system-ppc64 -accel tcg -smp 10,maxcpus=8193
> > > > qemu-system-ppc64: ../hw/intc/spapr_xive.c:596: spapr_xive_claim_irq:
> > > > Assertion `lisn < xive->nr_irqs' failed.
> > > > Aborted (core dumped)
> > > > [root@host build]#
> > > >
> > > > With patch fix:
> > > > [root@host build]# qemu-system-ppc64 -accel tcg -smp 10,maxcpus=4097
> > > > qemu-system-ppc64: Invalid SMP CPUs 4097. The max CPUs supported by
> > > > machine 'pseries-8.2' is 4096
> > > > [root@host build]#
> > > >
> > > > Signed-off-by: Harsh Prateek Bora <harshpb@linux.ibm.com>
> > > > ---
> > > >   hw/ppc/spapr.c | 9 +++------
> > > >   1 file changed, 3 insertions(+), 6 deletions(-)
> > > >
> > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > index df09aa9d6a..0de11a4458 100644
> > > > --- a/hw/ppc/spapr.c
> > > > +++ b/hw/ppc/spapr.c
> > > > @@ -4647,13 +4647,10 @@ static void 
> > > > spapr_machine_class_init(ObjectClass *oc, void *data)
> > > >       mc->block_default_type = IF_SCSI;
> > > >       /*
> > > > -     * Setting max_cpus to INT32_MAX. Both KVM and TCG max_cpus values
> > > > -     * should be limited by the host capability instead of hardcoded.
> > > > -     * max_cpus for KVM guests will be checked in kvm_init(), and TCG
> > > > -     * guests are welcome to have as many CPUs as the host are capable
> > > > -     * of emulate.
> > > > +     * While KVM determines max cpus in kvm_init() using 
> > > > kvm_max_vcpus(),
> > > > +     * In TCG the limit is restricted by the range of CPU IPIs 
> > > > available.
> > > >        */
> > > > -    mc->max_cpus = INT32_MAX;
> > > > +    mc->max_cpus = SPAPR_NR_IPIS;
> > >
> > > Is SPAPR_NR_IPIS also the upper limit for KVM?
> >
> > In KVM mode, the limit is restricted to what is supported by KVM which 
> > is checked using kvm_ioctl via wrappers in kvm_init and appears to be 
> > evaluating to 2048. So, having a larger default works for both case.
>
> QEMU sets the number of cpus with KVM ioctls :
>
>      KVM_DEV_XICS_NR_SERVERS
>      KVM_DEV_XIVE_NR_SERVERS
>
> This is important for the host since the interrupt controller is then
> configured with these values through FW.
>
> The default value is indeed 2K but this is large and wastes a lot of
> HW resources, page mappings, etc.

I was wondering if one day KVM raise its limit to 5k, then the
machine will clamp to 4k, and someone will have to debug that.
Not a big deal ;)