From nobody Tue Feb 10 20:30:50 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=bytedance.com ARC-Seal: i=1; a=rsa-sha256; t=1678415175; cv=none; d=zohomail.com; s=zohoarc; b=KvrwqNhIxzkjr+UQ6uOah3KtvnV+Zm1OqrLH9Y5DI81CGo1Xc5Aou+dQvJ60OjBV1DFJUphTZjPJsI5m2KuV8uatp4sqABFW7sB12vZw5kFbvbkskxW2kqlL8N4TTUkGzQnFSqSwktvc6syuvTFwFQCGxNfw3EWa9nBvIfhOlQo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1678415175; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=9wK7Kao4Ay8cM21jN0qeB4dy3IR6Pt8A2UW1ABAkymk=; b=YJu86nsHKYo99GRY8PP5V9XFYeEk9nLMo557VsW62mmUKdDdpt0xqYcEHSA+/S636Jzgej/RrBgJVGOpIymAoaDGaAUibSknX65bwhQIP5QCwA/JRsxlx2HzrNjspLo1OnOKZO71JJRhjRbhH7QE+1Wgy0oBCasF5Hnfd4pCsGo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1678415175694562.5952411244824; Thu, 9 Mar 2023 18:26:15 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1paSRN-0005eM-UT; Thu, 09 Mar 2023 21:25:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1paSR6-0005dp-V2 for qemu-devel@nongnu.org; Thu, 09 Mar 2023 21:24:48 -0500 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1paSR5-0001xN-9v for qemu-devel@nongnu.org; Thu, 09 Mar 2023 21:24:48 -0500 Received: by mail-pj1-x1034.google.com with SMTP id ce8-20020a17090aff0800b0023a61cff2c6so7210061pjb.0 for ; Thu, 09 Mar 2023 18:24:46 -0800 (PST) Received: from n250-032-048.byted.org ([221.194.189.28]) by smtp.gmail.com with ESMTPSA id g7-20020aa78187000000b00571f66721aesm246081pfi.42.2023.03.09.18.24.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Mar 2023 18:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678415086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9wK7Kao4Ay8cM21jN0qeB4dy3IR6Pt8A2UW1ABAkymk=; b=RSSqqrdkAhMj8GRW/4evaDHHWulxsjHj9OkLePcvcjjCOhe+EOoQ5yilgRqeLh5rhF Ym4fq5znyJajPNGnMTDPcryaBIW0Bdb3ykQGK+dkbhRiSVgO6R+qnaeSS9ytASY/pLRc 8HVCvA1jBo8biWYL1WCOIgNe4z0nc/EOEDqSo0Jhu0mE74GZBqPKIMhTyQY2heDk00MI u4wuEfLUzHkEtFm+X9afPUH5ep/926Rz1PpNQ6GN0fR7tgieJhG95RCf2xt57d+F3fy7 nrq3wy2+UYr8OsF7JbThKe69IoPPvwQ9qU9iQgAfaGIxrgACV2V1589zDRsjXjpD1Grg ZLDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678415086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9wK7Kao4Ay8cM21jN0qeB4dy3IR6Pt8A2UW1ABAkymk=; b=KvLK0Up67xNx7WJiZvk0CcThGb1rGrodobdhPF/sKGh+QxgRlaOdTcbqasNnjBg0d2 AwrhBEcRcOCcngqzwvyphQILublXjw+MNhxqiE+mRg+NshWvnmw2ntHoCmvtn6tgsyBv Q8POQPiCgE/4qajYz+WiKdlifjY1EZhrIsiuDJTW5xO8aVJJR9D36gYrrZGAogtyLt96 S2A8VrAo0UGqYDNJYjweavyKeQR+l3+lNFHw+Vaq0fqQn23h46QGhbHs38L1eBMb3B7z u6syTXJdkZVSPlbAqqv3rXVgRFBN2MaoDMfzYCCncNV+LWfR/sdbwqnZd0dPVDr6nowa qU0w== X-Gm-Message-State: AO0yUKVDAjdfUhEl2JFVZTOXPu+DybrBWdGD+sHz8xXE0sT35UXUyc96 bRl1pRpme0BKeoJA7w7bUIEyhkN0/HMB5xqywt8= X-Google-Smtp-Source: AK7set94HUnUqNaVTMoWhtiZyScsxILPYf4FtqTnUCOp+HmMQBVC/NhjlxqRNqiXC23C+Mkd+P83dw== X-Received: by 2002:a05:6a20:5489:b0:cc:120c:b259 with SMTP id i9-20020a056a20548900b000cc120cb259mr30209129pzk.39.1678415085774; Thu, 09 Mar 2023 18:24:45 -0800 (PST) From: Chuang Xu To: qemu-devel@nongnu.org Cc: dgilbert@redhat.com, quintela@redhat.com, pbonzini@redhat.com, peterx@redhat.com, david@redhat.com, philmd@linaro.org, zhouyibo@bytedance.com, Chuang Xu Subject: [PATCH v7 5/6] migration: Reduce time of loading non-iterable vmstate Date: Fri, 10 Mar 2023 10:24:24 +0800 Message-Id: <20230310022425.2992472-6-xuchuangxclwt@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20230310022425.2992472-1-xuchuangxclwt@bytedance.com> References: <20230310022425.2992472-1-xuchuangxclwt@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1034; envelope-from=xuchuangxclwt@bytedance.com; helo=mail-pj1-x1034.google.com X-Spam_score_int: 4 X-Spam_score: 0.4 X-Spam_bar: / X-Spam_report: (0.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_LOCAL_NOVOWEL=0.5, HK_RANDOM_ENVFROM=0.999, HK_RANDOM_FROM=1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @bytedance.com) X-ZM-MESSAGEID: 1678415176573100001 Content-Type: text/plain; charset="utf-8" The duration of loading non-iterable vmstate accounts for a significant portion of downtime (starting with the timestamp of source qemu stop and ending with the timestamp of target qemu start). Most of the time is spent committing memory region changes repeatedly. This patch packs all the changes to memory region during the period of loading non-iterable vmstate in a single memory transaction. With the increase of devices, this patch will greatly improve the performance. Note that the following test results are based on the application of the next patch. Without the next patch, the improvement will be reduced. Here are the test1 results: test info: - Host - Intel(R) Xeon(R) Platinum 8362 CPU - Mellanox Technologies MT28841 - VM - 32 CPUs 128GB RAM VM - 8 16-queue vhost-net device - 16 4-queue vhost-user-blk device. time of loading non-iterable vmstate downtime before about 112 ms 285 ms after about 20 ms 194 ms In test2, we keep the number of the device the same as test1, reduce the number of queues per device: Here are the test2 results: test info: - Host - Intel(R) Xeon(R) Platinum 8362 CPU - Mellanox Technologies MT28841 - VM - 32 CPUs 128GB RAM VM - 8 1-queue vhost-net device - 16 1-queue vhost-user-blk device. time of loading non-iterable vmstate downtime before about 65 ms about 151 ms after about 19 ms about 100 ms In test3, we keep the number of queues per device the same as test1, reduce the number of devices: Here are the test3 results: test info: - Host - Intel(R) Xeon(R) Platinum 8362 CPU - Mellanox Technologies MT28841 - VM - 32 CPUs 128GB RAM VM - 1 16-queue vhost-net device - 1 4-queue vhost-user-blk device. time of loading non-iterable vmstate downtime before about 24 ms about 51 ms after about 9 ms about 36 ms As we can see from the test results above, both the number of queues and the number of devices have a great impact on the time of loading non-iterab= le vmstate. The growth of the number of devices and queues will lead to more mr commits, and the time consumption caused by the flatview reconstruction will also increase. Signed-off-by: Chuang Xu Reviewed-by: Juan Quintela Reviewed-by: Peter Xu --- migration/savevm.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/migration/savevm.c b/migration/savevm.c index aa54a67fda..9a7d3e40d6 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2762,6 +2762,7 @@ out: goto retry; } } + return ret; } =20 @@ -2787,7 +2788,25 @@ int qemu_loadvm_state(QEMUFile *f) =20 cpu_synchronize_all_pre_loadvm(); =20 + /* + * Call memory_region_transaction_begin() before loading vmstate. + * This call is paired with memory_region_transaction_commit() at + * the end of qemu_loadvm_state_main(), in order to pack all the + * changes to memory region during the period of loading + * non-iterable vmstate in a single memory transaction. + * This operation will reduce time of loading non-iterable vmstate + */ + memory_region_transaction_begin(); + ret =3D qemu_loadvm_state_main(f, mis); + + /* + * Call memory_region_transaction_commit() after loading vmstate. + * At this point, qemu actually completes all the previous memory + * region transactions. + */ + memory_region_transaction_commit(); + qemu_event_set(&mis->main_thread_load_event); =20 trace_qemu_loadvm_state_post_main(ret); --=20 2.20.1