Re: [PATCH v7 1/6] memory: prevent dma-reentracy issues

[Alexander Bulekov]

On 230313 0945, Philippe Mathieu-Daudé wrote:
> Hi Alex,
> 
> Sorry for the late review, *sigh*.
> 
> On 13/3/23 09:24, Alexander Bulekov wrote:
> > Add a flag to the DeviceState, when a device is engaged in PIO/MMIO/DMA.
> > This flag is set/checked prior to calling a device's MemoryRegion
> > handlers, and set when device code initiates DMA.  The purpose of this
> > flag is to prevent two types of DMA-based reentrancy issues:
> > 
> > 1.) mmio -> dma -> mmio case
> > 2.) bh -> dma write -> mmio case
> > 
> > These issues have led to problems such as stack-exhaustion and
> > use-after-frees.
> > 
> > Summary of the problem from Peter Maydell:
> > https://lore.kernel.org/qemu-devel/CAFEAcA_23vc7hE3iaM-JVA6W38LK4hJoWae5KcknhPRD5fPBZA@mail.gmail.com
> > 
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/62
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/540
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/541
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/556
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/557
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/827
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1282
> > 
> > Reviewed-by: Darren Kenny <darren.kenny@oracle.com>
> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
> > Acked-by: Peter Xu <peterx@redhat.com>
> > ---
> >   include/hw/qdev-core.h |  7 +++++++
> >   softmmu/memory.c       | 17 +++++++++++++++++
> >   softmmu/trace-events   |  1 +
> >   3 files changed, 25 insertions(+)
> > 
> > diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
> > index bd50ad5ee1..7623703943 100644
> > --- a/include/hw/qdev-core.h
> > +++ b/include/hw/qdev-core.h
> > @@ -162,6 +162,10 @@ struct NamedClockList {
> >       QLIST_ENTRY(NamedClockList) node;
> >   };
> > +typedef struct {
> > +    bool engaged_in_io;
> 
> Do you plan to add more fields?

Not right now, but maybe some need will come up.

> > +} MemReentrancyGuard;
> > +
> >   /**
> >    * DeviceState:
> >    * @realized: Indicates whether the device has been fully constructed.
> > @@ -194,6 +198,9 @@ struct DeviceState {
> >       int alias_required_for_version;
> >       ResettableState reset;
> >       GSList *unplug_blockers;
> > +
> > +    /* Is the device currently in mmio/pio/dma? Used to prevent 
> > re-entrancy */
> > +    MemReentrancyGuard mem_reentrancy_guard;
> 
> At this point I'm not sure anymore this is a device or MR property.

It's designed to be an MR property. If it were MR specific, it wouldn't
handle the BH -> DMA case, or this one, where there are two MRs (doorbell
and oper) involed.
https://gitlab.com/qemu-project/qemu/-/issues/540

> 
> >   };
> >   struct DeviceListener {
> > diff --git a/softmmu/memory.c b/softmmu/memory.c
> > index 4699ba55ec..57bf18a257 100644
> > --- a/softmmu/memory.c
> > +++ b/softmmu/memory.c
> > @@ -533,6 +533,7 @@ static MemTxResult access_with_adjusted_size(hwaddr 
> > addr,
> >       uint64_t access_mask;
> >       unsigned access_size;
> >       unsigned i;
> > +    DeviceState *dev = NULL;
> >       MemTxResult r = MEMTX_OK;
> >       if (!access_size_min) {
> > @@ -542,6 +543,19 @@ static MemTxResult access_with_adjusted_size(hwaddr 
> > addr,
> >           access_size_max = 4;
> >       }
> > +    /* Do not allow more than one simultanous access to a device's IO 
> > Regions */
> 
> Typo "simultaneous".
> 
> 1/ access_with_adjusted_size() is complex enough and we are having hard
>    time getting it right. I'd prefer we don't intermix size adjustment
>    and re-entrancy check in the same function. This check could belong
>    to the callers.
> 

Would moving the code within this function to keep it separate from the
size adjustment be good enough? Otherwise we would end up with duplicate
code in the read/write callers.

The size-adjustment seems to be orthogonal (the MR won't change)?

> 2/ I'm not keen on calling QOM object_dynamic_cast() in this hot path;
>    and mixing QDev API within MR one. At least, can we cache this value
>    once in memory_region_do_init() since we have access to @owner?
>

Sounds like a good idea. Is it ever possible for the owner/owner's
address to change? 

Thanks
-Alex

> > +    if (mr->owner &&
> > +        !mr->ram_device && !mr->ram && !mr->rom_device && !mr->readonly) {
> > +        dev = (DeviceState *) object_dynamic_cast(mr->owner, TYPE_DEVICE);
> > +        if (dev) {
> > +            if (dev->mem_reentrancy_guard.engaged_in_io) {
> > +                trace_memory_region_reentrant_io(get_cpu_index(), mr, 
> > addr, size);
> > +                return MEMTX_ERROR;
> 
> MEMTX_ERROR is device-specific, I'm not sure it is right to return it
> from this generic path. Maybe you meant MEMTX_ACCESS_ERROR?
> 
> > +            }
> > +            dev->mem_reentrancy_guard.engaged_in_io = true;
> > +        }
> > +    }
> > +
> >       /* FIXME: support unaligned access? */
> >       access_size = MAX(MIN(size, access_size_max), access_size_min);
> >       access_mask = MAKE_64BIT_MASK(0, access_size * 8);
> > @@ -556,6 +570,9 @@ static MemTxResult access_with_adjusted_size(hwaddr 
> > addr,
> >                           access_mask, attrs);
> >           }
> >       }
> > +    if (dev) {
> > +        dev->mem_reentrancy_guard.engaged_in_io = false;
> > +    }
> >       return r;
> >   }
>