From nobody Thu Jan 1 11:16:29 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=suse.de ARC-Seal: i=1; a=rsa-sha256; t=1698093499; cv=none; d=zohomail.com; s=zohoarc; b=nZ8vBW0KGpVNEhHaIWPr/hg/Y8zePXL+rclfautG0RasHWDJlf5vT413KGX0uQVAtwzFJu9vthoSe8K2LGK260YIY4z+dm4RHo21UVYoaJLD5GEJWLaocQquCIgwtQ3hCqgOR7KbRpO4e5HHoOOs6OyhZU4kn+1/+TcHYqg8Cf8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1698093499; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=+Ab7YP+vDIwkVLvT/SbH5dSZ5HGHeC0d4KTNOwymzv4=; b=cml7iDR4tEkit88na1DAkNT0GHr/VxmK1TCdi8jdbw1mxpH9AqBZNb99aVE9dokw6n141yhLn0SzMQs7pJXfAMk+aqWw0A9G0Q9uhTq0fJDpQgcPG+QImZ34f18asYwmj9CBWa8YawpeLUVaBlj8J9sa7jroLBZSJRmObSjp0bo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1698093499692720.1924849925186; Mon, 23 Oct 2023 13:38:19 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qv1fn-0003ch-Fp; Mon, 23 Oct 2023 16:37:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qv1fe-0003Tj-8l for qemu-devel@nongnu.org; Mon, 23 Oct 2023 16:37:06 -0400 Received: from smtp-out2.suse.de ([195.135.220.29]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qv1fc-0001sK-13 for qemu-devel@nongnu.org; Mon, 23 Oct 2023 16:37:05 -0400 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9A3361FD80; Mon, 23 Oct 2023 20:37:01 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id B064B13A94; Mon, 23 Oct 2023 20:36:59 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id YIaqHmvZNmV1JQAAMHmgww (envelope-from ); Mon, 23 Oct 2023 20:36:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1698093421; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Ab7YP+vDIwkVLvT/SbH5dSZ5HGHeC0d4KTNOwymzv4=; b=QJ5eHmEaVvsREXFdaAOvx8AX8fHJTeR5bQ76u23ymHbBDhbsADvCc92kwX2/74G+PIWIvw pw1hmeaYx7IbtL1Sw1LT5cEEeIg+gFhmCJ4BDwm8fyVHO9svQ81OYvedEJQzuinqJmi1t9 9balRzUkapCBvR/A0ZjYTu8+j8P61y4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1698093421; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Ab7YP+vDIwkVLvT/SbH5dSZ5HGHeC0d4KTNOwymzv4=; b=QsOXVVO7Fiag6yRwyFSoa/xwwzu/XCMalvpfWsFUzz5wFRsby6rcfJJPTrVgD1fSoq8Hjw 1krqetJQyhR0j8Bg== From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Juan Quintela , Peter Xu , Leonardo Bras , Claudio Fontana Subject: [PATCH v2 18/29] migration/multifd: Allow multifd without packets Date: Mon, 23 Oct 2023 17:35:57 -0300 Message-Id: <20231023203608.26370-19-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20231023203608.26370-1-farosas@suse.de> References: <20231023203608.26370-1-farosas@suse.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Authentication-Results: smtp-out2.suse.de; none X-Spam-Score: -2.10 X-Spamd-Result: default: False [-2.10 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_MISSING_CHARSET(2.50)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; NEURAL_HAM_LONG(-3.00)[-1.000]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_SEVEN(0.00)[7]; MID_CONTAINS_FROM(1.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=195.135.220.29; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @suse.de) X-ZM-MESSAGEID: 1698093501228100001 Content-Type: text/plain; charset="utf-8" For the upcoming support to the new 'fixed-ram' migration stream format, we cannot use multifd packets because each write into the ramblock section in the migration file is expected to contain only the guest pages. They are written at their respective offsets relative to the ramblock section header. There is no space for the packet information and the expected gains from the new approach come partly from being able to write the pages sequentially without extraneous data in between. The new format also doesn't need the packets and all necessary information can be taken from the standard migration headers with some (future) changes to multifd code. Use the presence of the fixed-ram capability to decide whether to send packets. For now this has no effect as fixed-ram cannot yet be enabled with multifd. Signed-off-by: Fabiano Rosas --- migration/multifd.c | 119 +++++++++++++++++++++++++++----------------- migration/options.c | 5 ++ migration/options.h | 1 + 3 files changed, 80 insertions(+), 45 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index e2a45c667a..b912060b32 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -655,18 +655,22 @@ static void *multifd_send_thread(void *opaque) Error *local_err =3D NULL; int ret =3D 0; bool use_zero_copy_send =3D migrate_zero_copy_send(); + bool use_packets =3D migrate_multifd_packets(); =20 thread =3D migration_threads_add(p->name, qemu_get_thread_id()); =20 trace_multifd_send_thread_start(p->id); rcu_register_thread(); =20 - if (multifd_send_initial_packet(p, &local_err) < 0) { - ret =3D -1; - goto out; + if (use_packets) { + if (multifd_send_initial_packet(p, &local_err) < 0) { + ret =3D -1; + goto out; + } + + /* initial packet */ + p->num_packets =3D 1; } - /* initial packet */ - p->num_packets =3D 1; =20 while (true) { qemu_sem_post(&multifd_send_state->channels_ready); @@ -678,11 +682,10 @@ static void *multifd_send_thread(void *opaque) qemu_mutex_lock(&p->mutex); =20 if (p->pending_job) { - uint64_t packet_num =3D p->packet_num; uint32_t flags; p->normal_num =3D 0; =20 - if (use_zero_copy_send) { + if (!use_packets || use_zero_copy_send) { p->iovs_num =3D 0; } else { p->iovs_num =3D 1; @@ -700,16 +703,20 @@ static void *multifd_send_thread(void *opaque) break; } } - multifd_send_fill_packet(p); + + if (use_packets) { + multifd_send_fill_packet(p); + p->num_packets++; + } + flags =3D p->flags; p->flags =3D 0; - p->num_packets++; p->total_normal_pages +=3D p->normal_num; p->pages->num =3D 0; p->pages->block =3D NULL; qemu_mutex_unlock(&p->mutex); =20 - trace_multifd_send(p->id, packet_num, p->normal_num, flags, + trace_multifd_send(p->id, p->packet_num, p->normal_num, flags, p->next_packet_size); =20 if (use_zero_copy_send) { @@ -719,7 +726,7 @@ static void *multifd_send_thread(void *opaque) if (ret !=3D 0) { break; } - } else { + } else if (use_packets) { /* Send header using the same writev call */ p->iov[0].iov_len =3D p->packet_len; p->iov[0].iov_base =3D p->packet; @@ -907,6 +914,7 @@ int multifd_save_setup(Error **errp) { int thread_count; uint32_t page_count =3D MULTIFD_PACKET_SIZE / qemu_target_page_size(); + bool use_packets =3D migrate_multifd_packets(); uint8_t i; =20 if (!migrate_multifd()) { @@ -931,14 +939,20 @@ int multifd_save_setup(Error **errp) p->pending_job =3D 0; p->id =3D i; p->pages =3D multifd_pages_init(page_count); - p->packet_len =3D sizeof(MultiFDPacket_t) - + sizeof(uint64_t) * page_count; - p->packet =3D g_malloc0(p->packet_len); - p->packet->magic =3D cpu_to_be32(MULTIFD_MAGIC); - p->packet->version =3D cpu_to_be32(MULTIFD_VERSION); + + if (use_packets) { + p->packet_len =3D sizeof(MultiFDPacket_t) + + sizeof(uint64_t) * page_count; + p->packet =3D g_malloc0(p->packet_len); + p->packet->magic =3D cpu_to_be32(MULTIFD_MAGIC); + p->packet->version =3D cpu_to_be32(MULTIFD_VERSION); + + /* We need one extra place for the packet header */ + p->iov =3D g_new0(struct iovec, page_count + 1); + } else { + p->iov =3D g_new0(struct iovec, page_count); + } p->name =3D g_strdup_printf("multifdsend_%d", i); - /* We need one extra place for the packet header */ - p->iov =3D g_new0(struct iovec, page_count + 1); p->normal =3D g_new0(ram_addr_t, page_count); p->page_size =3D qemu_target_page_size(); p->page_count =3D page_count; @@ -1070,7 +1084,7 @@ void multifd_recv_sync_main(void) { int i; =20 - if (!migrate_multifd()) { + if (!migrate_multifd() || !migrate_multifd_packets()) { return; } for (i =3D 0; i < migrate_multifd_channels(); i++) { @@ -1097,6 +1111,7 @@ static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p =3D opaque; Error *local_err =3D NULL; + bool use_packets =3D migrate_multifd_packets(); int ret; =20 trace_multifd_recv_thread_start(p->id); @@ -1109,17 +1124,20 @@ static void *multifd_recv_thread(void *opaque) break; } =20 - ret =3D qio_channel_read_all_eof(p->c, (void *)p->packet, - p->packet_len, &local_err); - if (ret =3D=3D 0 || ret =3D=3D -1) { /* 0: EOF -1: Error */ - break; - } + if (use_packets) { + ret =3D qio_channel_read_all_eof(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret =3D=3D 0 || ret =3D=3D -1) { /* 0: EOF -1: Error */ + break; + } =20 - qemu_mutex_lock(&p->mutex); - ret =3D multifd_recv_unfill_packet(p, &local_err); - if (ret) { - qemu_mutex_unlock(&p->mutex); - break; + qemu_mutex_lock(&p->mutex); + ret =3D multifd_recv_unfill_packet(p, &local_err); + if (ret) { + qemu_mutex_unlock(&p->mutex); + break; + } + p->num_packets++; } =20 flags =3D p->flags; @@ -1127,7 +1145,7 @@ static void *multifd_recv_thread(void *opaque) p->flags &=3D ~MULTIFD_FLAG_SYNC; trace_multifd_recv(p->id, p->packet_num, p->normal_num, flags, p->next_packet_size); - p->num_packets++; + p->total_normal_pages +=3D p->normal_num; qemu_mutex_unlock(&p->mutex); =20 @@ -1162,6 +1180,7 @@ int multifd_load_setup(Error **errp) { int thread_count; uint32_t page_count =3D MULTIFD_PACKET_SIZE / qemu_target_page_size(); + bool use_packets =3D migrate_multifd_packets(); uint8_t i; =20 /* @@ -1186,9 +1205,12 @@ int multifd_load_setup(Error **errp) qemu_sem_init(&p->sem_sync, 0); p->quit =3D false; p->id =3D i; - p->packet_len =3D sizeof(MultiFDPacket_t) - + sizeof(uint64_t) * page_count; - p->packet =3D g_malloc0(p->packet_len); + + if (use_packets) { + p->packet_len =3D sizeof(MultiFDPacket_t) + + sizeof(uint64_t) * page_count; + p->packet =3D g_malloc0(p->packet_len); + } p->name =3D g_strdup_printf("multifdrecv_%d", i); p->iov =3D g_new0(struct iovec, page_count); p->normal =3D g_new0(ram_addr_t, page_count); @@ -1234,18 +1256,26 @@ void multifd_recv_new_channel(QIOChannel *ioc, Erro= r **errp) { MultiFDRecvParams *p; Error *local_err =3D NULL; - int id; + bool use_packets =3D migrate_multifd_packets(); + int id, num_packets =3D 0; =20 - id =3D multifd_recv_initial_packet(ioc, &local_err); - if (id < 0) { - multifd_recv_terminate_threads(local_err); - error_propagate_prepend(errp, local_err, - "failed to receive packet" - " via multifd channel %d: ", - qatomic_read(&multifd_recv_state->count)); - return; + if (use_packets) { + id =3D multifd_recv_initial_packet(ioc, &local_err); + if (id < 0) { + multifd_recv_terminate_threads(local_err); + error_propagate_prepend(errp, local_err, + "failed to receive packet" + " via multifd channel %d: ", + qatomic_read(&multifd_recv_state->coun= t)); + return; + } + trace_multifd_recv_new_channel(id); + + /* initial packet */ + num_packets =3D 1; + } else { + id =3D 0; } - trace_multifd_recv_new_channel(id); =20 p =3D &multifd_recv_state->params[id]; if (p->c !=3D NULL) { @@ -1256,9 +1286,8 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error = **errp) return; } p->c =3D ioc; + p->num_packets =3D num_packets; object_ref(OBJECT(ioc)); - /* initial packet */ - p->num_packets =3D 1; =20 p->running =3D true; qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, diff --git a/migration/options.c b/migration/options.c index 9f693d909f..bb7a2bbe06 100644 --- a/migration/options.c +++ b/migration/options.c @@ -390,6 +390,11 @@ bool migrate_multifd_flush_after_each_section(void) return s->multifd_flush_after_each_section; } =20 +bool migrate_multifd_packets(void) +{ + return !migrate_fixed_ram(); +} + bool migrate_postcopy(void) { return migrate_postcopy_ram() || migrate_dirty_bitmaps(); diff --git a/migration/options.h b/migration/options.h index 2a9e0e9e13..4a3e7e36a8 100644 --- a/migration/options.h +++ b/migration/options.h @@ -57,6 +57,7 @@ bool migrate_zero_copy_send(void); */ =20 bool migrate_multifd_flush_after_each_section(void); +bool migrate_multifd_packets(void); bool migrate_postcopy(void); bool migrate_rdma(void); bool migrate_tls(void); --=20 2.35.3