[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[RFC PATCH v1 22/26] migration/multifd: Support outgoing fixed-ram strea
From: |
Fabiano Rosas |
Subject: |
[RFC PATCH v1 22/26] migration/multifd: Support outgoing fixed-ram stream format |
Date: |
Thu, 30 Mar 2023 15:03:32 -0300 |
The new fixed-ram stream format uses a file transport and puts ram
pages in the migration file at their respective offsets and can be
done in parallel by using the pwritev system call which takes iovecs
and an offset.
Add support to enabling the new format along with multifd to make use
of the threading and page handling already in place.
This requires multifd to stop sending headers and leaving the stream
format to the fixed-ram code. When it comes time to write the data, we
need to call a version of qio_channel_write that can take an offset.
Usage on HMP is:
(qemu) stop
(qemu) migrate_set_capability multifd on
(qemu) migrate_set_capability fixed-ram on
(qemu) migrate_set_parameter max-bandwidth 0
(qemu) migrate_set_parameter multifd-channels 8
(qemu) migrate file:migfile
Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
migration/migration.c | 5 -----
migration/multifd.c | 51 +++++++++++++++++++++++++++++++++++++++++--
2 files changed, 49 insertions(+), 7 deletions(-)
diff --git a/migration/migration.c b/migration/migration.c
index 258709aee1..77d24a5114 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1345,11 +1345,6 @@ static bool migrate_caps_check(bool *cap_list,
#endif
if (cap_list[MIGRATION_CAPABILITY_FIXED_RAM]) {
- if (cap_list[MIGRATION_CAPABILITY_MULTIFD]) {
- error_setg(errp, "Directly mapped memory incompatible with
multifd");
- return false;
- }
-
if (cap_list[MIGRATION_CAPABILITY_XBZRLE]) {
error_setg(errp, "Directly mapped memory incompatible with
xbzrle");
return false;
diff --git a/migration/multifd.c b/migration/multifd.c
index 20ef665218..cc70b20ff7 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -256,6 +256,19 @@ static void multifd_pages_clear(MultiFDPages_t *pages)
g_free(pages);
}
+static void multifd_set_file_bitmap(MultiFDSendParams *p, bool set)
+{
+ MultiFDPages_t *pages = p->pages;
+
+ if (!pages->block) {
+ return;
+ }
+
+ for (int i = 0; i < p->normal_num; i++) {
+ ramblock_set_shadow_bmap(pages->block, pages->offset[i], set);
+ }
+}
+
static void multifd_send_fill_packet(MultiFDSendParams *p)
{
MultiFDPacket_t *packet = p->packet;
@@ -608,6 +621,17 @@ int multifd_send_sync_main(QEMUFile *f)
}
}
+ if (!migrate_multifd_use_packets()) {
+ for (i = 0; i < migrate_multifd_channels(); i++) {
+ MultiFDSendParams *p = &multifd_send_state->params[i];
+
+ qemu_sem_post(&p->sem);
+ continue;
+ }
+
+ return 0;
+ }
+
/*
* When using zero-copy, it's necessary to flush the pages before any of
* the pages can be sent again, so we'll make sure the new version of the
@@ -692,6 +716,8 @@ static void *multifd_send_thread(void *opaque)
if (p->pending_job) {
uint32_t flags;
+ uint64_t write_base;
+
p->normal_num = 0;
if (!use_packets || use_zero_copy_send) {
@@ -716,6 +742,16 @@ static void *multifd_send_thread(void *opaque)
if (use_packets) {
multifd_send_fill_packet(p);
p->num_packets++;
+ write_base = 0;
+ } else {
+ multifd_set_file_bitmap(p, true);
+
+ /*
+ * If we subtract the host page now, we don't need to
+ * pass it into qio_channel_write_full_all() below.
+ */
+ write_base = p->pages->block->pages_offset -
+ (uint64_t)p->pages->block->host;
}
flags = p->flags;
@@ -741,8 +777,9 @@ static void *multifd_send_thread(void *opaque)
p->iov[0].iov_base = p->packet;
}
- ret = qio_channel_writev_full_all(p->c, p->iov, p->iovs_num, NULL,
- 0, p->write_flags, &local_err);
+ ret = qio_channel_write_full_all(p->c, p->iov, p->iovs_num,
+ write_base, NULL, 0,
+ p->write_flags, &local_err);
if (ret != 0) {
break;
}
@@ -758,6 +795,13 @@ static void *multifd_send_thread(void *opaque)
} else if (p->quit) {
qemu_mutex_unlock(&p->mutex);
break;
+ } else if (!use_packets) {
+ /*
+ * When migrating to a file there's not need for a SYNC
+ * packet, the channels are ready right away.
+ */
+ qemu_sem_post(&multifd_send_state->channels_ready);
+ qemu_mutex_unlock(&p->mutex);
} else {
qemu_mutex_unlock(&p->mutex);
/* sometimes there are spurious wakeups */
@@ -767,6 +811,7 @@ static void *multifd_send_thread(void *opaque)
out:
if (local_err) {
trace_multifd_send_error(p->id);
+ multifd_set_file_bitmap(p, false);
multifd_send_terminate_threads(local_err);
error_free(local_err);
}
@@ -981,6 +1026,8 @@ int multifd_save_setup(Error **errp)
if (migrate_use_zero_copy_send()) {
p->write_flags = QIO_CHANNEL_WRITE_FLAG_ZERO_COPY;
+ } else if (!use_packets) {
+ p->write_flags |= QIO_CHANNEL_WRITE_FLAG_WITH_OFFSET;
} else {
p->write_flags = 0;
}
--
2.35.3
- [RFC PATCH v1 11/26] migration: Refactor precopy ram loading code, (continued)
- [RFC PATCH v1 11/26] migration: Refactor precopy ram loading code, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 12/26] migration: Add support for 'fixed-ram' migration restore, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 13/26] tests/qtest: migration-test: Add tests for fixed-ram file-based migration, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 15/26] migration/multifd: Remove direct "socket" references, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 14/26] migration: Add completion tracepoint, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 16/26] migration/multifd: Allow multifd without packets, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 17/26] migration/multifd: Add outgoing QIOChannelFile support, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 19/26] migration/multifd: Add pages to the receiving side, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 18/26] migration/multifd: Add incoming QIOChannelFile support, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 21/26] migration/ram: Add a wrapper for fixed-ram shadow bitmap, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 22/26] migration/multifd: Support outgoing fixed-ram stream format,
Fabiano Rosas <=
- [RFC PATCH v1 20/26] io: Add a pwritev/preadv version that takes a discontiguous iovec, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 23/26] migration/multifd: Support incoming fixed-ram stream format, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 25/26] migration: Add direct-io parameter, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 24/26] tests/qtest: Add a multifd + fixed-ram migration test, Fabiano Rosas, 2023/03/30
- [RFC PATCH v1 26/26] tests/migration/guestperf: Add file, fixed-ram and direct-io support, Fabiano Rosas, 2023/03/30
- Re: [RFC PATCH v1 00/26] migration: File based migration with multifd and fixed-ram, Peter Xu, 2023/03/30