Facing "PTE Read access is not set" VFIO NVMe

[Major Saheb]

Hi All,

I have an ubuntu guest running on kvm , and I am passing it 10 qemu
emulated nvme drives
    <iommu model='intel'>
      <driver intremap='on' eim='on'/>
    </iommu>
<qemu:arg value='pcie-root-port,id=pcie-root-port%d,slot=%d'/>
<qemu:arg 
value='nvme,drive=NVME%d,serial=%s_%d,id=NVME%d,bus=pcie-root-port%d'/>

kernel
Linux 5.15.0-56-generic #62-Ubuntu SMP ----- x86_64 x86_64
x86_64 GNU/Linux

kernel command line
intel_iommu=on

I have attached these drives to vfio-pcie.

when I try to send IO commands to these drives VIA a userspace nvme
driver using VFIO I get
[ 1474.752590] DMAR: DRHD: handling fault status reg 2
[ 1474.754463] DMAR: [DMA Read NO_PASID] Request device [0b:00.0]
fault addr 0xffffe000 [fault reason 0x06] PTE Read access is not set

I have narrowed it down to issuing the first command to NVMe (which is
Identify controller in this case), and it seems the DMA is failing due
to
inaccessible IO page table. I have also tried adding
'iommu_platform=on,ats=on' which I found at
https://wiki.qemu.org/Features/VT-d and it says the following

2022-12-22T11:29:07.001178Z qemu-system-x86_64: -device
nvme,drive=NVME0,serial=akqkvcl-0_0,id=NVME0,bus=pcie-root-port0,iommu_platform=on:
Property 'nvme.iommu_platform' not found
2022-12-22T11:21:48.281876Z qemu-system-x86_64: -device
nvme,drive=NVME0,serial=akqkvcl-0_0,id=NVME0,bus=pcie-root-port0,iommu_platform=on,ats=on:
Property 'nvme.ats' not found

-R