Re: XIVE VFIO kernel resample failure in INTx mode under heavy load

[Cédric Le Goater]

Hello,


> I've been struggling for some time with what is looking like a
> potential bug in QEMU/KVM on the POWER9 platform.  It appears that
> in XIVE mode, when the in-kernel IRQ chip is enabled, an external
> device that rapidly asserts IRQs via the legacy INTx level mechanism
> will only receive one interrupt in the KVM guest.

Indeed. I could reproduce with a pass-through PCI adapter using
'pci=nomsi'. The virtio devices operate correctly but the network
adapter only receives one event (*):


$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
CPU6       CPU7
 16:       2198       1378       1519       1216          0          0          
0          0  XIVE-IPI   0 Edge      IPI-0
 17:          0          0          0          0       2003       1936       
1335       1507  XIVE-IPI   1 Edge      IPI-1
 18:          0       6401          0          0          0          0          
0          0  XIVE-IRQ 4609 Level     virtio3, virtio0, virtio2
 19:          0          0          0          0          0        204          
0          0  XIVE-IRQ 4610 Level     virtio1
 20:          0          0          0          0          0          0          
0          0  XIVE-IRQ 4608 Level     xhci-hcd:usb1
 21:          0          1          0          0          0          0          
0          0  XIVE-IRQ 4612 Level     eth1 (*)
 23:          0          0          0          0          0          0          
0          0  XIVE-IRQ 4096 Edge      RAS_EPOW
 24:          0          0          0          0          0          0          
0          0  XIVE-IRQ 4592 Edge      hvc_console
 26:          0          0          0          0          0          0          
0          0  XIVE-IRQ 4097 Edge      RAS_HOTPLUG

> Changing any one of those items appears to avoid the glitch, e.g. XICS

XICS is very different from XIVE. The driver implements the previous
interrupt controller architecture (P5-P8) and the hypervisor mediates
the delivery to the guest. With XIVE, vCPUs are directly signaled by
the IC. When under KVM, we use different KVM devices for each mode :

* KVM XIVE is a XICS-on-XIVE implementation (P9/P10 hosts) for guests
  not using the XIVE native interface. RHEL7 for instance.
* KVM XIVE native is a XIVE implementation (P9/P10 hosts) for guests
  using the XIVE native interface. Linux > 4.14.
* KVM XICS is for P8 hosts (no XIVE HW)

VFIO adds some complexity with the source events. I think the problem
comes from the assertion state. I will talk about it later.

> mode with the in-kernel IRQ chip works (all interrupts are passed
> through),

All interrupts are passed through using XIVE also. Run 'info pic' in
the monitor. On the host, check the IRQ mapping in the debugfs file :

  /sys/kernel/debug/powerpc/kvm-xive-*

> and XIVE mode with the in-kernel IRQ chip disabled also works. 

In that case, no KVM device backs the QEMU device and all state
is in one place.

> We
> are also not seeing any problems in XIVE mode with the in-kernel
> chip from MSI/MSI-X devices.

Yes. pass-through devices are expected to operate correctly :)
 
> The device in question is a real time card that needs to raise an
> interrupt every 1ms.  It works perfectly on the host, but fails in
> the guest -- with the in-kernel IRQ chip and XIVE enabled, it
> receives exactly one interrupt, at which point the host continues to
> see INTx+ but the guest sees INTX-, and the IRQ handler in the guest
> kernel is never reentered.

ok. Same symptom as the scenario above.

> We have also seen some very rare glitches where, over a long period
> of time, we can enter a similar deadlock in XICS mode.

with the in-kernel XICS IRQ chip ?

> Disabling
> the in-kernel IRQ chip in XIVE mode will also lead to the lockup
> with this device, since the userspace IRQ emulation cannot keep up
> with the rapid interrupt firing (measurements show around 100ms
> required for processing each interrupt in the user mode).

MSI emulation in QEMU is slower indeed (35%). LSI is very slow because
it is handled as a special case in the device/driver. To maintain the
assertion state, all LSI handling is done with a special HCALL :
H_INT_ESB which is implemented in QEMU. This generates a lot of back
and forth in the KVM stack.
 
> My understanding is the resample mechanism does some clever tricks
> with level IRQs, but that QEMU needs to check if the IRQ is still
> asserted by the device on guest EOI.

Yes. the problem is in that area.

> Since a failure here would
> explain these symptoms I'm wondering if there is a bug in either
> QEMU or KVM for POWER / pSeries (SPAPr) where the IRQ is not
> resampled and therefore not re-fired in the guest?

KVM I would say. The assertion state is maintained in KVM for the KVM
XICS-on-XIVE implementation and in QEMU for the KVM XIVE native
device. These are good candidates. I will take a look.

(We might have never tested that path of the code with a passthrough
device using only INTx)
 
Thanks,

C.