Re: towards a workable O_DIRECT outmigration to a file

[Dr. David Alan Gilbert]

* Nikolay Borisov (nborisov@suse.com) wrote:
> [adding Juan and David to cc as I had missed them. ]

Hi Nikolay,

> On 11.08.22 г. 16:47 ч., Nikolay Borisov wrote:
> > Hello,
> > 
> > I'm currently looking into implementing a 'file:' uri for migration save
> > in qemu. Ideally the solution will be O_DIRECT compatible. I'm aware of
> > the branch https://gitlab.com/berrange/qemu/-/tree/mig-file. In the
> > process of brainstorming how a solution would like the a couple of
> > questions transpired that I think warrant wider discussion in the
> > community.

OK, so this seems to be a continuation with Claudio and Daniel and co as
of a few months back.  I'd definitely be leaving libvirt sides of the
question here to Dan, and so that also means definitely looking at that
tree above.

> > First, implementing a solution which is self-contained within qemu would
> > be easy enough( famous last words) but the gist is one  has to only care
> > about the format within qemu. However, I'm being told that what libvirt
> > does is prepend its own custom header to the resulting saved file, then
> > slipstreams the migration stream from qemu. Now with the solution that I
> > envision I intend to keep all write-related logic inside qemu, this
> > means there's no way to incorporate the logic of libvirt. The reason I'd
> > like to keep the write process within qemu is to avoid an extra copy of
> > data between the two processes (qemu outging migration and libvirt),
> > with the current fd approach qemu is passed an fd, data is copied
> > between qemu/libvirt and finally the libvirt_iohelper writes the data.
> > So the question which remains to be answered is how would libvirt make
> > use of this new functionality in qemu? I was thinking something along
> > the lines of :
> > 
> > 1. Qemu writes its migration stream to a file, ideally on a filesystem
> > which supports reflink - xfs/btrfs
> > 
> > 2. Libvirt writes it's header to a separate file
> > 2.1 Reflinks the qemu's stream right after its header
> > 2.2 Writes its trailer
> > 
> > 3. Unlink() qemu's file, now only libvirt's file remains on-disk.
> > 
> > I wouldn't call this solution hacky though it definitely leaves some
> > bitter aftertaste.

Wouldn't it be simpler to tell libvirt to write it's header, then tell
qemu to append everything?

> > Another solution would be to extend the 'fd:' protocol to allow multiple
> > descriptors (for multifd) support to be passed in. The reason dup()
> > can't be used is because in order for multifd to be supported it's
> > required to be able to write to multiple, non-overlapping regions of the
> > file. And duplicated fd's share their offsets etc. But that really seems
> > more or less hacky. Alternatively it's possible that pwrite() are used
> > to write to non-overlapping regions in the file. Any feedback is
> > welcomed.

I do like the idea of letting fd: take multiple fd's.

Dave

> > 
> > 
> > Regards,
> > Nikolay
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK