Re: [PATCH v13 0/5] UFFD write-tracking migration/snapshots

[Andrey Gruzdev]

On 22.02.2021 21:11, David Hildenbrand wrote:
> On 22.02.21 18:54, Peter Xu wrote:
> > On Mon, Feb 22, 2021 at 06:33:27PM +0100, David Hildenbrand wrote:
> > > On 22.02.21 18:29, Peter Xu wrote:
> > > > On Sat, Feb 20, 2021 at 02:59:42AM -0500, David Hildenbrand wrote:
> > > > > Live snapshotting ends up reading all guest memory (dirty bitmap 
> > > > > starts with all 1s), which is not what we want for virtio-mem - we 
> > > > > don’t want to read and migrate memory that has been discarded and 
> > > > > has no stable content.
> > > > >
> > > > > For ordinary migration we use the guest page hint API to clear 
> > > > > bits in the dirty bitmap after dirty bitmap sync. Well, if we 
> > > > > don‘t do bitmap syncs we‘ll never clear any dirty bits. That‘s the 
> > > > > problem.
> > > >
> > > > Using dirty bitmap for that information is less efficient, becase it's
> > > > definitely a larger granularity information than PAGE_SIZE. If the 
> > > > disgarded
> > > > ranges are always continuous and at the end of a memory region, we 
> > > > should have
> > > > some parameter in the ramblock showing that where we got shrinked 
> > > > then we don't
> > > > check dirty bitmap at all, rather than always assuming used_length 
> > > > is the one.
> > >
> > > They are randomly scattered across the whole RAMBlock. 
> > > Shrinking/growing
> > > will be done to some degree in the future (but it won't get rid of the
> > > general sparse layout we can produce).
> >
> > OK. Btw I think currently live snapshot should still be reading dirty 
> > bitmap,
> > so maybe it's still fine.  It's just that it's still not very clear 
> > to hide
> > virtio-mem information into dirty bitmap, imho, since that's not how we
> > interpret dirty bitmap - which is only for the sake of tracking page 
> > changes.
>
> Well, currently it is "what do we have to migrate".
Using dirty bitmap can prohibit unexpected (overwritten) content of 
pre-discarded pages, but they'll appear as zeroed on destination.
What about other virtio-baloon features like HW_POISON when poisoned 
pages should also be retained on population filled with certain pattern?

Thanks,

> > What's the granule of virtio-mem for this discard behavior? Maybe we 
> > could
>
> virtio-mem granularity is at least 1MB. This corresponds to 256 bits 
> (32 bytes) in the dirty bitmap I think.
>
> > decouple it with dirty bitmap some day; if the unit is big enough 
> > it's also a
> > gain on efficiency so we skip in chunk rather than looping over tons 
> > of pages
> > knowing that they're discarded.
>
> Yeah, it's not optimal having to go over the dirty bitmap to cross off 
> "discarded" parts and later having to find bits to migrate.
>
> At least find_next_bit() can skip whole longs (8 bytes) and is fairly 
> efficient. There is certainly room for improvement (the current guest 
> free page hinting API is certainly a hack).

--
Andrey Gruzdev, Principal Engineer
Virtuozzo GmbH  +7-903-247-6397
                virtuzzo.com