From nobody Sun Nov 24 10:42:51 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=suse.de ARC-Seal: i=1; a=rsa-sha256; t=1725454139; cv=none; d=zohomail.com; s=zohoarc; b=Cp6E4yyP11clThHtpXqNAz5enByou0LnbeVrg5DJq/KnphOWdPPGbvC5tJWGbRsnmXxWsNNaO2KwrXKyPfO7aOoN5OIu8kcIiOu8Y/Cf9KyBZxN5DPlZ0CPzJDGwlmDXpxm5nfgUAy5NiuYe4E3P2xD0Rtg60fQtZuXmTNEc5tE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1725454139; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Vu3lO01kqO9ruq0I9ow3cIczfDroX1+TjeIk0q8bgLs=; b=HqNjpZOSCjwyBwVTYvBzzrcWdJdDQLiUmkcw9GtjCA1GbS0Qiw6E+2s3lqmEcjag07Q1/we+zNW0yTLl1RjxCaj1mwm3PTwpFc6pO/2eaTttG2wDEmi33/Qh5BviDM72WmcQJ473nVBFyqB3vJHXXISZJRzAUBifodVmrWqs3/g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1725454139694878.503739042986; Wed, 4 Sep 2024 05:48:59 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1slpOi-0000mO-8X; Wed, 04 Sep 2024 08:46:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1slpNr-00068N-At for qemu-devel@nongnu.org; Wed, 04 Sep 2024 08:45:16 -0400 Received: from smtp-out2.suse.de ([195.135.223.131]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1slpNk-0005rw-QO for qemu-devel@nongnu.org; Wed, 04 Sep 2024 08:45:12 -0400 Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 611A51F38E; Wed, 4 Sep 2024 12:45:07 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 02356139D2; Wed, 4 Sep 2024 12:45:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id uPyjLlFW2GZuVQAAD6G6ig (envelope-from ); Wed, 04 Sep 2024 12:45:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1725453907; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vu3lO01kqO9ruq0I9ow3cIczfDroX1+TjeIk0q8bgLs=; b=s/BFVjc4Ztn+4wyDY44Lt9gwQidS3q2Y/du8U6KzBuq29AsjbeO0mpOY4wVx/LxR5ImdOh +MPa0HoRHbDcJsBydl80aiqLNsk6EA1tPAY/utXrUV1gRApcRJNUIJxwKwIgv2hRhRwkQ9 YQtjUTnOz2OkAdU9By4bR/rRy3/oMpM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1725453907; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vu3lO01kqO9ruq0I9ow3cIczfDroX1+TjeIk0q8bgLs=; b=WFpE3sflv7bMadmmG6YeTBLmLbVw00aomt3Cz+I/hPZ5/6+J+6Z1s1HA6nRzMnznBxXfh5 YzUeQZ4C6G29HxAQ== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1725453907; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vu3lO01kqO9ruq0I9ow3cIczfDroX1+TjeIk0q8bgLs=; b=s/BFVjc4Ztn+4wyDY44Lt9gwQidS3q2Y/du8U6KzBuq29AsjbeO0mpOY4wVx/LxR5ImdOh +MPa0HoRHbDcJsBydl80aiqLNsk6EA1tPAY/utXrUV1gRApcRJNUIJxwKwIgv2hRhRwkQ9 YQtjUTnOz2OkAdU9By4bR/rRy3/oMpM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1725453907; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vu3lO01kqO9ruq0I9ow3cIczfDroX1+TjeIk0q8bgLs=; b=WFpE3sflv7bMadmmG6YeTBLmLbVw00aomt3Cz+I/hPZ5/6+J+6Z1s1HA6nRzMnznBxXfh5 YzUeQZ4C6G29HxAQ== From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: Peter Xu , Richard Henderson Subject: [PULL 27/34] migration/multifd: Move nocomp code into multifd-nocomp.c Date: Wed, 4 Sep 2024 09:44:10 -0300 Message-Id: <20240904124417.14565-28-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240904124417.14565-1-farosas@suse.de> References: <20240904124417.14565-1-farosas@suse.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.79 X-Spamd-Result: default: False [-2.79 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.19)[-0.936]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:mid,suse.de:email]; RCVD_TLS_ALL(0.00)[] Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=195.135.223.131; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @suse.de) X-ZM-MESSAGEID: 1725454140117116600 Content-Type: text/plain; charset="utf-8" In preparation for adding new payload types to multifd, move most of the no-compression code into multifd-nocomp.c. Let's try to keep a semblance of layering by not mixing general multifd control flow with the details of transmitting pages of ram. There are still some pieces leftover, namely the p->normal, p->zero, etc variables that we use for zero page tracking and the packet allocation which is heavily dependent on the ram code. Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas --- migration/meson.build | 1 + migration/multifd-nocomp.c | 394 +++++++++++++++++++++++++++++++++++++ migration/multifd.c | 377 +---------------------------------- migration/multifd.h | 5 + 4 files changed, 402 insertions(+), 375 deletions(-) create mode 100644 migration/multifd-nocomp.c diff --git a/migration/meson.build b/migration/meson.build index 5ce2acb41e..77f3abf08e 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -21,6 +21,7 @@ system_ss.add(files( 'migration-hmp-cmds.c', 'migration.c', 'multifd.c', + 'multifd-nocomp.c', 'multifd-zlib.c', 'multifd-zero-page.c', 'options.c', diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c new file mode 100644 index 0000000000..53ea9f9c83 --- /dev/null +++ b/migration/multifd-nocomp.c @@ -0,0 +1,394 @@ +/* + * Multifd RAM migration without compression + * + * Copyright (c) 2019-2020 Red Hat Inc + * + * Authors: + * Juan Quintela + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "exec/ramblock.h" +#include "exec/target_page.h" +#include "file.h" +#include "multifd.h" +#include "options.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "trace.h" + +static MultiFDSendData *multifd_ram_send; + +size_t multifd_ram_payload_size(void) +{ + uint32_t n =3D multifd_ram_page_count(); + + /* + * We keep an array of page offsets at the end of MultiFDPages_t, + * add space for it in the allocation. + */ + return sizeof(MultiFDPages_t) + n * sizeof(ram_addr_t); +} + +void multifd_ram_save_setup(void) +{ + multifd_ram_send =3D multifd_send_data_alloc(); +} + +void multifd_ram_save_cleanup(void) +{ + g_free(multifd_ram_send); + multifd_ram_send =3D NULL; +} + +static void multifd_set_file_bitmap(MultiFDSendParams *p) +{ + MultiFDPages_t *pages =3D &p->data->u.ram; + + assert(pages->block); + + for (int i =3D 0; i < pages->normal_num; i++) { + ramblock_set_file_bmap_atomic(pages->block, pages->offset[i], true= ); + } + + for (int i =3D pages->normal_num; i < pages->num; i++) { + ramblock_set_file_bmap_atomic(pages->block, pages->offset[i], fals= e); + } +} + +static int multifd_nocomp_send_setup(MultiFDSendParams *p, Error **errp) +{ + uint32_t page_count =3D multifd_ram_page_count(); + + if (migrate_zero_copy_send()) { + p->write_flags |=3D QIO_CHANNEL_WRITE_FLAG_ZERO_COPY; + } + + if (!migrate_mapped_ram()) { + /* We need one extra place for the packet header */ + p->iov =3D g_new0(struct iovec, page_count + 1); + } else { + p->iov =3D g_new0(struct iovec, page_count); + } + + return 0; +} + +static void multifd_nocomp_send_cleanup(MultiFDSendParams *p, Error **errp) +{ + g_free(p->iov); + p->iov =3D NULL; + return; +} + +static void multifd_send_prepare_iovs(MultiFDSendParams *p) +{ + MultiFDPages_t *pages =3D &p->data->u.ram; + uint32_t page_size =3D multifd_ram_page_size(); + + for (int i =3D 0; i < pages->normal_num; i++) { + p->iov[p->iovs_num].iov_base =3D pages->block->host + pages->offse= t[i]; + p->iov[p->iovs_num].iov_len =3D page_size; + p->iovs_num++; + } + + p->next_packet_size =3D pages->normal_num * page_size; +} + +static int multifd_nocomp_send_prepare(MultiFDSendParams *p, Error **errp) +{ + bool use_zero_copy_send =3D migrate_zero_copy_send(); + int ret; + + multifd_send_zero_page_detect(p); + + if (migrate_mapped_ram()) { + multifd_send_prepare_iovs(p); + multifd_set_file_bitmap(p); + + return 0; + } + + if (!use_zero_copy_send) { + /* + * Only !zerocopy needs the header in IOV; zerocopy will + * send it separately. + */ + multifd_send_prepare_header(p); + } + + multifd_send_prepare_iovs(p); + p->flags |=3D MULTIFD_FLAG_NOCOMP; + + multifd_send_fill_packet(p); + + if (use_zero_copy_send) { + /* Send header first, without zerocopy */ + ret =3D qio_channel_write_all(p->c, (void *)p->packet, + p->packet_len, errp); + if (ret !=3D 0) { + return -1; + } + } + + return 0; +} + +static int multifd_nocomp_recv_setup(MultiFDRecvParams *p, Error **errp) +{ + p->iov =3D g_new0(struct iovec, multifd_ram_page_count()); + return 0; +} + +static void multifd_nocomp_recv_cleanup(MultiFDRecvParams *p) +{ + g_free(p->iov); + p->iov =3D NULL; +} + +static int multifd_nocomp_recv(MultiFDRecvParams *p, Error **errp) +{ + uint32_t flags; + + if (migrate_mapped_ram()) { + return multifd_file_recv_data(p, errp); + } + + flags =3D p->flags & MULTIFD_FLAG_COMPRESSION_MASK; + + if (flags !=3D MULTIFD_FLAG_NOCOMP) { + error_setg(errp, "multifd %u: flags received %x flags expected %x", + p->id, flags, MULTIFD_FLAG_NOCOMP); + return -1; + } + + multifd_recv_zero_page_process(p); + + if (!p->normal_num) { + return 0; + } + + for (int i =3D 0; i < p->normal_num; i++) { + p->iov[i].iov_base =3D p->host + p->normal[i]; + p->iov[i].iov_len =3D multifd_ram_page_size(); + ramblock_recv_bitmap_set_offset(p->block, p->normal[i]); + } + return qio_channel_readv_all(p->c, p->iov, p->normal_num, errp); +} + +static void multifd_pages_reset(MultiFDPages_t *pages) +{ + /* + * We don't need to touch offset[] array, because it will be + * overwritten later when reused. + */ + pages->num =3D 0; + pages->normal_num =3D 0; + pages->block =3D NULL; +} + +void multifd_ram_fill_packet(MultiFDSendParams *p) +{ + MultiFDPacket_t *packet =3D p->packet; + MultiFDPages_t *pages =3D &p->data->u.ram; + uint32_t zero_num =3D pages->num - pages->normal_num; + + packet->pages_alloc =3D cpu_to_be32(multifd_ram_page_count()); + packet->normal_pages =3D cpu_to_be32(pages->normal_num); + packet->zero_pages =3D cpu_to_be32(zero_num); + + if (pages->block) { + strncpy(packet->ramblock, pages->block->idstr, 256); + } + + for (int i =3D 0; i < pages->num; i++) { + /* there are architectures where ram_addr_t is 32 bit */ + uint64_t temp =3D pages->offset[i]; + + packet->offset[i] =3D cpu_to_be64(temp); + } + + trace_multifd_send_ram_fill(p->id, pages->normal_num, + zero_num); +} + +int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp) +{ + MultiFDPacket_t *packet =3D p->packet; + uint32_t page_count =3D multifd_ram_page_count(); + uint32_t page_size =3D multifd_ram_page_size(); + int i; + + packet->pages_alloc =3D be32_to_cpu(packet->pages_alloc); + /* + * If we received a packet that is 100 times bigger than expected + * just stop migration. It is a magic number. + */ + if (packet->pages_alloc > page_count) { + error_setg(errp, "multifd: received packet " + "with size %u and expected a size of %u", + packet->pages_alloc, page_count) ; + return -1; + } + + p->normal_num =3D be32_to_cpu(packet->normal_pages); + if (p->normal_num > packet->pages_alloc) { + error_setg(errp, "multifd: received packet " + "with %u normal pages and expected maximum pages are %u= ", + p->normal_num, packet->pages_alloc) ; + return -1; + } + + p->zero_num =3D be32_to_cpu(packet->zero_pages); + if (p->zero_num > packet->pages_alloc - p->normal_num) { + error_setg(errp, "multifd: received packet " + "with %u zero pages and expected maximum zero pages are= %u", + p->zero_num, packet->pages_alloc - p->normal_num) ; + return -1; + } + + if (p->normal_num =3D=3D 0 && p->zero_num =3D=3D 0) { + return 0; + } + + /* make sure that ramblock is 0 terminated */ + packet->ramblock[255] =3D 0; + p->block =3D qemu_ram_block_by_name(packet->ramblock); + if (!p->block) { + error_setg(errp, "multifd: unknown ram block %s", + packet->ramblock); + return -1; + } + + p->host =3D p->block->host; + for (i =3D 0; i < p->normal_num; i++) { + uint64_t offset =3D be64_to_cpu(packet->offset[i]); + + if (offset > (p->block->used_length - page_size)) { + error_setg(errp, "multifd: offset too long %" PRIu64 + " (max " RAM_ADDR_FMT ")", + offset, p->block->used_length); + return -1; + } + p->normal[i] =3D offset; + } + + for (i =3D 0; i < p->zero_num; i++) { + uint64_t offset =3D be64_to_cpu(packet->offset[p->normal_num + i]); + + if (offset > (p->block->used_length - page_size)) { + error_setg(errp, "multifd: offset too long %" PRIu64 + " (max " RAM_ADDR_FMT ")", + offset, p->block->used_length); + return -1; + } + p->zero[i] =3D offset; + } + + return 0; +} + +static inline bool multifd_queue_empty(MultiFDPages_t *pages) +{ + return pages->num =3D=3D 0; +} + +static inline bool multifd_queue_full(MultiFDPages_t *pages) +{ + return pages->num =3D=3D multifd_ram_page_count(); +} + +static inline void multifd_enqueue(MultiFDPages_t *pages, ram_addr_t offse= t) +{ + pages->offset[pages->num++] =3D offset; +} + +/* Returns true if enqueue successful, false otherwise */ +bool multifd_queue_page(RAMBlock *block, ram_addr_t offset) +{ + MultiFDPages_t *pages; + +retry: + pages =3D &multifd_ram_send->u.ram; + + if (multifd_payload_empty(multifd_ram_send)) { + multifd_pages_reset(pages); + multifd_set_payload_type(multifd_ram_send, MULTIFD_PAYLOAD_RAM); + } + + /* If the queue is empty, we can already enqueue now */ + if (multifd_queue_empty(pages)) { + pages->block =3D block; + multifd_enqueue(pages, offset); + return true; + } + + /* + * Not empty, meanwhile we need a flush. It can because of either: + * + * (1) The page is not on the same ramblock of previous ones, or, + * (2) The queue is full. + * + * After flush, always retry. + */ + if (pages->block !=3D block || multifd_queue_full(pages)) { + if (!multifd_send(&multifd_ram_send)) { + return false; + } + goto retry; + } + + /* Not empty, and we still have space, do it! */ + multifd_enqueue(pages, offset); + return true; +} + +int multifd_ram_flush_and_sync(void) +{ + if (!migrate_multifd()) { + return 0; + } + + if (!multifd_payload_empty(multifd_ram_send)) { + if (!multifd_send(&multifd_ram_send)) { + error_report("%s: multifd_send fail", __func__); + return -1; + } + } + + return multifd_send_sync_main(); +} + +bool multifd_send_prepare_common(MultiFDSendParams *p) +{ + MultiFDPages_t *pages =3D &p->data->u.ram; + multifd_send_zero_page_detect(p); + + if (!pages->normal_num) { + p->next_packet_size =3D 0; + return false; + } + + multifd_send_prepare_header(p); + + return true; +} + +static MultiFDMethods multifd_nocomp_ops =3D { + .send_setup =3D multifd_nocomp_send_setup, + .send_cleanup =3D multifd_nocomp_send_cleanup, + .send_prepare =3D multifd_nocomp_send_prepare, + .recv_setup =3D multifd_nocomp_recv_setup, + .recv_cleanup =3D multifd_nocomp_recv_cleanup, + .recv =3D multifd_nocomp_recv +}; + +static void multifd_nocomp_register(void) +{ + multifd_register_ops(MULTIFD_COMPRESSION_NONE, &multifd_nocomp_ops); +} + +migration_init(multifd_nocomp_register); diff --git a/migration/multifd.c b/migration/multifd.c index e100836cbe..0c07a2040b 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -96,20 +96,7 @@ struct { MultiFDMethods *ops; } *multifd_recv_state; =20 -static MultiFDSendData *multifd_ram_send; - -static size_t multifd_ram_payload_size(void) -{ - uint32_t n =3D multifd_ram_page_count(); - - /* - * We keep an array of page offsets at the end of MultiFDPages_t, - * add space for it in the allocation. - */ - return sizeof(MultiFDPages_t) + n * sizeof(ram_addr_t); -} - -static MultiFDSendData *multifd_send_data_alloc(void) +MultiFDSendData *multifd_send_data_alloc(void) { size_t max_payload_size, size_minus_payload; =20 @@ -131,17 +118,6 @@ static MultiFDSendData *multifd_send_data_alloc(void) return g_malloc0(size_minus_payload + max_payload_size); } =20 -void multifd_ram_save_setup(void) -{ - multifd_ram_send =3D multifd_send_data_alloc(); -} - -void multifd_ram_save_cleanup(void) -{ - g_free(multifd_ram_send); - multifd_ram_send =3D NULL; -} - static bool multifd_use_packets(void) { return !migrate_mapped_ram(); @@ -152,141 +128,6 @@ void multifd_send_channel_created(void) qemu_sem_post(&multifd_send_state->channels_created); } =20 -static void multifd_set_file_bitmap(MultiFDSendParams *p) -{ - MultiFDPages_t *pages =3D &p->data->u.ram; - - assert(pages->block); - - for (int i =3D 0; i < pages->normal_num; i++) { - ramblock_set_file_bmap_atomic(pages->block, pages->offset[i], true= ); - } - - for (int i =3D pages->normal_num; i < pages->num; i++) { - ramblock_set_file_bmap_atomic(pages->block, pages->offset[i], fals= e); - } -} - -static int multifd_nocomp_send_setup(MultiFDSendParams *p, Error **errp) -{ - uint32_t page_count =3D multifd_ram_page_count(); - - if (migrate_zero_copy_send()) { - p->write_flags |=3D QIO_CHANNEL_WRITE_FLAG_ZERO_COPY; - } - - if (multifd_use_packets()) { - /* We need one extra place for the packet header */ - p->iov =3D g_new0(struct iovec, page_count + 1); - } else { - p->iov =3D g_new0(struct iovec, page_count); - } - - return 0; -} - -static void multifd_nocomp_send_cleanup(MultiFDSendParams *p, Error **errp) -{ - g_free(p->iov); - p->iov =3D NULL; - return; -} - -static void multifd_send_prepare_iovs(MultiFDSendParams *p) -{ - MultiFDPages_t *pages =3D &p->data->u.ram; - uint32_t page_size =3D multifd_ram_page_size(); - - for (int i =3D 0; i < pages->normal_num; i++) { - p->iov[p->iovs_num].iov_base =3D pages->block->host + pages->offse= t[i]; - p->iov[p->iovs_num].iov_len =3D page_size; - p->iovs_num++; - } - - p->next_packet_size =3D pages->normal_num * page_size; -} - -static int multifd_nocomp_send_prepare(MultiFDSendParams *p, Error **errp) -{ - bool use_zero_copy_send =3D migrate_zero_copy_send(); - int ret; - - multifd_send_zero_page_detect(p); - - if (!multifd_use_packets()) { - multifd_send_prepare_iovs(p); - multifd_set_file_bitmap(p); - - return 0; - } - - if (!use_zero_copy_send) { - /* - * Only !zerocopy needs the header in IOV; zerocopy will - * send it separately. - */ - multifd_send_prepare_header(p); - } - - multifd_send_prepare_iovs(p); - p->flags |=3D MULTIFD_FLAG_NOCOMP; - - multifd_send_fill_packet(p); - - if (use_zero_copy_send) { - /* Send header first, without zerocopy */ - ret =3D qio_channel_write_all(p->c, (void *)p->packet, - p->packet_len, errp); - if (ret !=3D 0) { - return -1; - } - } - - return 0; -} - -static int multifd_nocomp_recv_setup(MultiFDRecvParams *p, Error **errp) -{ - p->iov =3D g_new0(struct iovec, multifd_ram_page_count()); - return 0; -} - -static void multifd_nocomp_recv_cleanup(MultiFDRecvParams *p) -{ - g_free(p->iov); - p->iov =3D NULL; -} - -static int multifd_nocomp_recv(MultiFDRecvParams *p, Error **errp) -{ - uint32_t flags; - - if (!multifd_use_packets()) { - return multifd_file_recv_data(p, errp); - } - - flags =3D p->flags & MULTIFD_FLAG_COMPRESSION_MASK; - - if (flags !=3D MULTIFD_FLAG_NOCOMP) { - error_setg(errp, "multifd %u: flags received %x flags expected %x", - p->id, flags, MULTIFD_FLAG_NOCOMP); - return -1; - } - - multifd_recv_zero_page_process(p); - - if (!p->normal_num) { - return 0; - } - - for (int i =3D 0; i < p->normal_num; i++) { - p->iov[i].iov_base =3D p->host + p->normal[i]; - p->iov[i].iov_len =3D multifd_ram_page_size(); - ramblock_recv_bitmap_set_offset(p->block, p->normal[i]); - } - return qio_channel_readv_all(p->c, p->iov, p->normal_num, errp); -} - static MultiFDMethods *multifd_ops[MULTIFD_COMPRESSION__MAX] =3D {}; =20 void multifd_register_ops(int method, MultiFDMethods *ops) @@ -296,18 +137,6 @@ void multifd_register_ops(int method, MultiFDMethods *= ops) multifd_ops[method] =3D ops; } =20 -/* Reset a MultiFDPages_t* object for the next use */ -static void multifd_pages_reset(MultiFDPages_t *pages) -{ - /* - * We don't need to touch offset[] array, because it will be - * overwritten later when reused. - */ - pages->num =3D 0; - pages->normal_num =3D 0; - pages->block =3D NULL; -} - static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) { MultiFDInit_t msg =3D {}; @@ -372,30 +201,6 @@ static int multifd_recv_initial_packet(QIOChannel *c, = Error **errp) return msg.id; } =20 -static void multifd_ram_fill_packet(MultiFDSendParams *p) -{ - MultiFDPacket_t *packet =3D p->packet; - MultiFDPages_t *pages =3D &p->data->u.ram; - uint32_t zero_num =3D pages->num - pages->normal_num; - - packet->pages_alloc =3D cpu_to_be32(multifd_ram_page_count()); - packet->normal_pages =3D cpu_to_be32(pages->normal_num); - packet->zero_pages =3D cpu_to_be32(zero_num); - - if (pages->block) { - strncpy(packet->ramblock, pages->block->idstr, 256); - } - - for (int i =3D 0; i < pages->num; i++) { - /* there are architectures where ram_addr_t is 32 bit */ - uint64_t temp =3D pages->offset[i]; - - packet->offset[i] =3D cpu_to_be64(temp); - } - - trace_multifd_send_ram_fill(p->id, pages->normal_num, zero_num); -} - void multifd_send_fill_packet(MultiFDSendParams *p) { MultiFDPacket_t *packet =3D p->packet; @@ -423,82 +228,6 @@ void multifd_send_fill_packet(MultiFDSendParams *p) p->flags, p->next_packet_size); } =20 -static int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp) -{ - MultiFDPacket_t *packet =3D p->packet; - uint32_t page_count =3D multifd_ram_page_count(); - uint32_t page_size =3D multifd_ram_page_size(); - int i; - - packet->pages_alloc =3D be32_to_cpu(packet->pages_alloc); - /* - * If we received a packet that is 100 times bigger than expected - * just stop migration. It is a magic number. - */ - if (packet->pages_alloc > page_count) { - error_setg(errp, "multifd: received packet " - "with size %u and expected a size of %u", - packet->pages_alloc, page_count) ; - return -1; - } - - p->normal_num =3D be32_to_cpu(packet->normal_pages); - if (p->normal_num > packet->pages_alloc) { - error_setg(errp, "multifd: received packet " - "with %u normal pages and expected maximum pages are %u= ", - p->normal_num, packet->pages_alloc) ; - return -1; - } - - p->zero_num =3D be32_to_cpu(packet->zero_pages); - if (p->zero_num > packet->pages_alloc - p->normal_num) { - error_setg(errp, "multifd: received packet " - "with %u zero pages and expected maximum zero pages are= %u", - p->zero_num, packet->pages_alloc - p->normal_num) ; - return -1; - } - - if (p->normal_num =3D=3D 0 && p->zero_num =3D=3D 0) { - return 0; - } - - /* make sure that ramblock is 0 terminated */ - packet->ramblock[255] =3D 0; - p->block =3D qemu_ram_block_by_name(packet->ramblock); - if (!p->block) { - error_setg(errp, "multifd: unknown ram block %s", - packet->ramblock); - return -1; - } - - p->host =3D p->block->host; - for (i =3D 0; i < p->normal_num; i++) { - uint64_t offset =3D be64_to_cpu(packet->offset[i]); - - if (offset > (p->block->used_length - page_size)) { - error_setg(errp, "multifd: offset too long %" PRIu64 - " (max " RAM_ADDR_FMT ")", - offset, p->block->used_length); - return -1; - } - p->normal[i] =3D offset; - } - - for (i =3D 0; i < p->zero_num; i++) { - uint64_t offset =3D be64_to_cpu(packet->offset[p->normal_num + i]); - - if (offset > (p->block->used_length - page_size)) { - error_setg(errp, "multifd: offset too long %" PRIu64 - " (max " RAM_ADDR_FMT ")", - offset, p->block->used_length); - return -1; - } - p->zero[i] =3D offset; - } - - return 0; -} - static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) { MultiFDPacket_t *packet =3D p->packet; @@ -571,7 +300,7 @@ static void multifd_send_kick_main(MultiFDSendParams *p) * * Returns true if succeed, false otherwise. */ -static bool multifd_send(MultiFDSendData **send_data) +bool multifd_send(MultiFDSendData **send_data) { int i; static int next_channel; @@ -632,61 +361,6 @@ static bool multifd_send(MultiFDSendData **send_data) return true; } =20 -static inline bool multifd_queue_empty(MultiFDPages_t *pages) -{ - return pages->num =3D=3D 0; -} - -static inline bool multifd_queue_full(MultiFDPages_t *pages) -{ - return pages->num =3D=3D multifd_ram_page_count(); -} - -static inline void multifd_enqueue(MultiFDPages_t *pages, ram_addr_t offse= t) -{ - pages->offset[pages->num++] =3D offset; -} - -/* Returns true if enqueue successful, false otherwise */ -bool multifd_queue_page(RAMBlock *block, ram_addr_t offset) -{ - MultiFDPages_t *pages; - -retry: - pages =3D &multifd_ram_send->u.ram; - - if (multifd_payload_empty(multifd_ram_send)) { - multifd_pages_reset(pages); - multifd_set_payload_type(multifd_ram_send, MULTIFD_PAYLOAD_RAM); - } - - /* If the queue is empty, we can already enqueue now */ - if (multifd_queue_empty(pages)) { - pages->block =3D block; - multifd_enqueue(pages, offset); - return true; - } - - /* - * Not empty, meanwhile we need a flush. It can because of either: - * - * (1) The page is not on the same ramblock of previous ones, or, - * (2) The queue is full. - * - * After flush, always retry. - */ - if (pages->block !=3D block || multifd_queue_full(pages)) { - if (!multifd_send(&multifd_ram_send)) { - return false; - } - goto retry; - } - - /* Not empty, and we still have space, do it! */ - multifd_enqueue(pages, offset); - return true; -} - /* Multifd send side hit an error; remember it and prepare to quit */ static void multifd_send_set_error(Error *err) { @@ -850,22 +524,6 @@ static int multifd_zero_copy_flush(QIOChannel *c) return ret; } =20 -int multifd_ram_flush_and_sync(void) -{ - if (!migrate_multifd()) { - return 0; - } - - if (!multifd_payload_empty(multifd_ram_send)) { - if (!multifd_send(&multifd_ram_send)) { - error_report("%s: multifd_send fail", __func__); - return -1; - } - } - - return multifd_send_sync_main(); -} - int multifd_send_sync_main(void) { int i; @@ -1676,34 +1334,3 @@ void multifd_recv_new_channel(QIOChannel *ioc, Error= **errp) QEMU_THREAD_JOINABLE); qatomic_inc(&multifd_recv_state->count); } - -bool multifd_send_prepare_common(MultiFDSendParams *p) -{ - MultiFDPages_t *pages =3D &p->data->u.ram; - multifd_send_zero_page_detect(p); - - if (!pages->normal_num) { - p->next_packet_size =3D 0; - return false; - } - - multifd_send_prepare_header(p); - - return true; -} - -static MultiFDMethods multifd_nocomp_ops =3D { - .send_setup =3D multifd_nocomp_send_setup, - .send_cleanup =3D multifd_nocomp_send_cleanup, - .send_prepare =3D multifd_nocomp_send_prepare, - .recv_setup =3D multifd_nocomp_recv_setup, - .recv_cleanup =3D multifd_nocomp_recv_cleanup, - .recv =3D multifd_nocomp_recv -}; - -static void multifd_nocomp_register(void) -{ - multifd_register_ops(MULTIFD_COMPRESSION_NONE, &multifd_nocomp_ops); -} - -migration_init(multifd_nocomp_register); diff --git a/migration/multifd.h b/migration/multifd.h index 00c872dfda..a3e35196d1 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -257,6 +257,8 @@ static inline void multifd_send_prepare_header(MultiFDS= endParams *p) } =20 void multifd_channel_connect(MultiFDSendParams *p, QIOChannel *ioc); +bool multifd_send(MultiFDSendData **send_data); +MultiFDSendData *multifd_send_data_alloc(void); =20 static inline uint32_t multifd_ram_page_size(void) { @@ -271,4 +273,7 @@ static inline uint32_t multifd_ram_page_count(void) void multifd_ram_save_setup(void); void multifd_ram_save_cleanup(void); int multifd_ram_flush_and_sync(void); +size_t multifd_ram_payload_size(void); +void multifd_ram_fill_packet(MultiFDSendParams *p); +int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp); #endif --=20 2.35.3