From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397611; cv=none; d=zohomail.com; s=zohoarc; b=hdzvZDUYEyX6lWX7a7t5Nk0H40+2x1zH4EtKirWVzk1hp8A38wUCwGx9qau0CENciBboITCh1eRQuifxOHK0z7aQamFQdWViey4n0olbROnqtGts6cEEDkCBL1T+wSIA+U9Ve7n1TMWbCasLC88LiqP96HCK8OflR11tmC4pPOw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397611; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=nRgsZGjFcRabgUhVMnu6mZsS+vJhVWDRtCtPb6hG3D4=; b=MsAbpKE5TYIjUXDicCyE5sSlQ6LzvH3phukztFehtFC36dLqyjALL7Esye6s1DwaoYfgn+/vScFAWThlxU0XnkqgKZVEriallkZSVl4gT0IKcbx+eEukU2shc0z85R43Mi4tNBAxlugjQBOHe5qAWBa/3amUeEcfRT3t9TxVWN4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397611230902.5846775292441; Mon, 14 Sep 2026 07:53:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683d-0002en-Nw; Mon, 14 Sep 2026 10:53:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683H-0002bt-Rr for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:00 -0400 Received: from mail-wm2-x10.google.com ([2a00:1450:4864:31::10]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683E-0005kN-Eq for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:52:58 -0400 Received: by mail-wm2-x10.google.com with SMTP id 5b1f17b1804b1-49e66390995so11019295e9.2 for ; Mon, 14 Sep 2026 07:52:56 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.52.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:52:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397575; x=1790002375; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=nRgsZGjFcRabgUhVMnu6mZsS+vJhVWDRtCtPb6hG3D4=; b=nXGqG2SiQ1wSpauXC+8fNft9d/j/IWjQPQ0YuZAk5P7E1b1No39MHsMPQHLq0CW19Y YVj3UZmr86eBGmJt3NGEAxLQ/zhXWQV4ZZ7gB52kN4tIhZUU9S44A7KTJKk6nG1+YzGi qE12nQtXc4ZEPJ5RsdWgH/1pOy8JSI4tl4EDZZs1OxQhCKwSK/FCDP0LzlSd0koQ908u qEzDGCPTvC8uhcmzmd1FzJCYmZx8nG/0H8jmAlfUT7HgSHUIq5kc3y7bZCnjD4biHB0S bRiuRcYT1UoLERpHRIwYJcxFCEpPH/UPTIvEDzq751T7qDQiGiQGhIrCMg6+UR4sjM8W OJMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397575; x=1790002375; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=nRgsZGjFcRabgUhVMnu6mZsS+vJhVWDRtCtPb6hG3D4=; b=QgPRSKGh67eQIrH1rZV3+1Tdd8C2Mrhj+sU5Sk8zLBfNrsqCrWJy5+UcA7GVdchaSD dXfMtYNybD9Zbd5oBH8hwSvRRMpWrS3RTs918bIj2Pz40EVIihV1gppu+IxddCjgbPXx rT0vB9uqYkzNNERZfkVrI9Svl4j3BmJnAgq+1rvX/wcG8Kt4zALuYrpaq5+aTEnEB8cS vmEMxFUXj8Fuo2659mJGf5i6rsnz2+ah1nZB04tghMFPWbyvVkD7bCgJk2VL68iVmmSf TUWdod+eU+vYS/Tbfv+Pa+odWBJB8Ac5asVuQISpGF5Sb50TEhM4owIKc69TC/xDvHnb CIRQ== X-Gm-Message-State: AFuF++mdQz7l/20+fVMYv1ZWd0h0ZzyaSP3QfMqGH192S7UUDcrc0M/5 znKLIB1s7SHu7CVVp5CcP7SJ3F3kzPg3mLTSQ0/UjDadT2zGu3jOcONfF6F+4JpQs+l6A4sDSe3 jaTF3 X-Gm-Gg: AYBFou0AHzqyD55/tiba6cKtEWrxIACEkCLEgYSzpuFrbBxS67vLo2Hm/P0mzA5OKgn sG5sc1OS8ttNY+a8wDWW4CJ+Z1l4dmAum+LLjTRtquCoRQ7psxDFop7VpevhMGYpvdHHRbL1HPR DnJMzVftDT3Sbf+veyR2XDqghgojC2J+wz4o4H+uzyJZuFfHTSQcs33CcxSKC6DT3ECHzHb3pJP mZ7Ze5r8z5Jppv908kGd9uN+kPRejob1AwoXmLjK9v4jMNynNLr0x/IQ5bBFojUAnt8bo0ZVC8K fhmnPs4MMq/hcsucNI62edRX9Ipi3Kjh9/6qiNCgaUrjrZVRprSe20zOKfofMTcBsk1dS5trKVT lX64OXYgmVBZhjQfh4dVmCOquqkK6+X/UYuKv8/vncHU1TblNlLef891YpGgKvj/h5OSQIsklg6 PlZYH8ah39J4EzRKTNA+7Q1ztGm79CTHjrtIymj2XrqyPp2faKzTk5+hrqzZhRUNrxlKi1 X-Received: by 2002:a05:600c:8012:b0:49c:fc6c:be08 with SMTP id 5b1f17b1804b1-49e7a67c769mr36701575e9.31.1789397574858; Mon, 14 Sep 2026 07:52:54 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Andrey Drobyshev , Vladimir Sementsov-Ogievskiy , John Snow , Eric Blake Subject: [PATCH v2 1/9] block/commit: pass BDRV_WANT_PRECISE to block-status Date: Mon, 14 Sep 2026 16:52:42 +0200 Message-ID: <20260914145250.2792879-2-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:31::10; envelope-from=den@openvz.org; helo=mail-wm2-x10.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397613210158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev c33159dec790 replaced the want_zero bool with a bitmask of BDRV_WANT_* flags. block/commit.c was missed and still passes "true", which is BDRV_BLOCK_DATA, not BDRV_WANT_PRECISE. Convert it like the other callers. Fixes: c33159dec790 ("block: Expand block status mode from bool to flags") Reviewed-by: Andrey Drobyshev Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev CC: Eric Blake --- block/commit.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/commit.c b/block/commit.c index 2d52c39594..4e0b0f9029 100644 --- a/block/commit.c +++ b/block/commit.c @@ -140,8 +140,8 @@ commit_iteration(CommitBlockJob *s, int64_t offset, /* Copy if allocated above the base */ WITH_GRAPH_RDLOCK_GUARD() { ret =3D bdrv_co_common_block_status_above(blk_bs(s->top), - s->base_overlay, true, true, offset, COMMIT_BUFFER_SIZE, - &bytes, NULL, NULL, NULL); + s->base_overlay, true, BDRV_WANT_PRECISE, offset, + COMMIT_BUFFER_SIZE, &bytes, NULL, NULL, NULL); } =20 trace_commit_one_iteration(s, offset, bytes, ret); --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397656; cv=none; d=zohomail.com; s=zohoarc; b=Novu/rEZuO41lZS+sZIc8nmjxlHzFsIxRC+Dal2Yzasw3bePJPcc2UnzV0CousrZ0iW7T8A8TvkgMOLeatAQhVJOVAZMs9xfnJShRoOXpV8Z2fQclTsupwtk1Kc6jF/mgwCogGB7Y6FfkNfnnLooE+jTijpxqOm+hugDADiuBt4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397656; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=FVFVBUMQqJn9U7YWOlxhFjuHlIQg5TGpWx/r9Z6Cy4Q=; b=Epi/cRgGOAaq59ZfOH0WR8rv3mD85nSiJa3ounhBiZQ71mO1D6v5bobCrtuMTLawgZIrLyN3ZHkDQw4SC8VoFR1fth7+tbpPmA1WJB80pOXbYxtIZP8zJS7S4lgpTmf3A4E/dPE+62M55GCBKemNV9g035BTJ8kf7LiN+Doxu5g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397656580169.09801270477305; Mon, 14 Sep 2026 07:54:16 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683k-0002pB-UP; Mon, 14 Sep 2026 10:53:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683H-0002bv-TP for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:00 -0400 Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683F-0005lI-LA for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:52:59 -0400 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-49d0da752ffso41234865e9.3 for ; Mon, 14 Sep 2026 07:52:57 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.52.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:52:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397576; x=1790002376; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=FVFVBUMQqJn9U7YWOlxhFjuHlIQg5TGpWx/r9Z6Cy4Q=; b=AK5L5La+sXb3mYzrAjW+0BWFcoXHwsD8HdVW3A/lkipIb7emEPCac7c1YyvtSO+xAk IU7zNwwjEBZD9HA2DUTXLEEQFXB6nwkd3YvQ+GNEMgKFPbODWNsqtY1NhapJIaqfb6Et Nr4SnPrWfQqeSfrAp3F3SsOOK7s6G5nfK+KQs3p0KhcTIniVMvLmtsFgCYFmJNoBHru0 6LHe6pap3wKbPaAr/6yX5rMWIkA3DkcEZJfxsaxetT8Uj6bb2iMjrlsD9UmFbdDkycry xJdJt209SBjji8DtyZ8gG2a5KBxOox9eQI8/QTTuni97v9rkDTQmVW9F2kQon9KtGcN8 2RDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397576; x=1790002376; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=FVFVBUMQqJn9U7YWOlxhFjuHlIQg5TGpWx/r9Z6Cy4Q=; b=EP3MoXq7TddGb/qb2PwpWMYodWK/PedMmf0+iZl3i0GL52qdwSnr47FR+By/AHXsSv 0zoDfjDuOGbG+5RvrY+SuiiJn6g/5C0pB1k72p/jZdfs+bp23TQDFxTPkMYH3+hunEcg roJTZ1vtchu3/xI6gaaIRF4SGP9EMoRY+RQ6lzqm3brZnDKKG+hm8XMfDT1hWxIwB+xK Woiq50haWXcnNUG2XxATN0nDjHBQGIv4i1I1BKCiLi7ynwNg4xZdb6+GdOmwojBZnDty 0fyBC4HPw7X9afEcGPMf/oIchSUo+5RblJMq8Ziuc2o4rS43FQhHgeW+GQmXDk6C8+SU EnrQ== X-Gm-Message-State: AFuF++kaCgpOGlu1a7AWspHVcqfEwTYM1N2qjfM2fJbTtjoRy2+5x9/5 vdnc1JLE2WydnMZ2NWaqLoVsA5eB2UByOOHp+zqf6KPRVakkYC0IF70RaZHULvZz5mS7YgnrHQ7 0sq1R X-Gm-Gg: AYBFou3B0lhDhk2T0somGUURYGw1Xx/2lKl4P617UxIer63Usaov/D2Y/lccqgdugMP lFJpDf/gF1zPZ6dPiizxJo2B7WD41o+J+6qkDEwdk+5LruqaXEfxpQ3wqP8GvVRP3koqdrFfX/+ WCpKMb4lhk+00NIpqUFOr5RNDlib2EYQUnHdFvN9aF22hfOBl+Rit03V+YSadX32sSd/VNyazHR YdYhQFhtobhnhD0HLHXWvIgRqiRaujuqBxU8ZCrTxT+NsABVgFXxd1q9ynf7PzcmXeGi5ZJLO1q 3te7opguAzGjKhimWCq+jU3jpFHmRBuZk2tnqWqvXdABdJm/71/OXckY5jtJVa5MDtruZvUOkUT r58NTAOt/WqnpimJOFYt31aY2EZifgJc0foxc/jeyXf6vLLqw90VAcciVOcPc3lcCSaYzIbmCuL v7umpAEgRu34ihJdWF5Mrr1pe92xkWN1L8ghOmcZvKEsVQhR08Oeogg8j/DHnb6eGy2X07 X-Received: by 2002:a05:600c:a47:b0:49c:fc6e:8cb4 with SMTP id 5b1f17b1804b1-49e7a67741cmr36331425e9.24.1789397576170; Mon, 14 Sep 2026 07:52:56 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Vladimir Sementsov-Ogievskiy , John Snow , Andrey Drobyshev Subject: [PATCH v2 2/9] iotests/040: cover large and fragmented commit runs Date: Mon, 14 Sep 2026 16:52:43 +0200 Message-ID: <20260914145250.2792879-3-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::332; envelope-from=den@openvz.org; helo=mail-wm1-x332.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397657999158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev 040 never commits runs long enough to span more than one block-status query, so the answer that commit_iteration() is about to start caching goes untested. Add two cases on a base <- mid <- active chain, committing mid so the job takes the regular commit path rather than active commit, and compare base against a snapshot of mid taken before the commit. One case mixes multi-megabyte data, zero and hole runs, the other fragments them down to single clusters with every transition off the 512K boundary. Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev Reviewed-by: Andrey Drobyshev --- tests/qemu-iotests/040 | 102 ++++++++++++++++++++++++++++++++++++- tests/qemu-iotests/040.out | 4 +- 2 files changed, 103 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040 index 5c18e413ec..e01c9385ba 100755 --- a/tests/qemu-iotests/040 +++ b/tests/qemu-iotests/040 @@ -25,13 +25,14 @@ import time import os import iotests -from iotests import qemu_img, qemu_io +from iotests import qemu_img, qemu_img_create, qemu_io, compare_images import struct import errno =20 backing_img =3D os.path.join(iotests.test_dir, 'backing.img') mid_img =3D os.path.join(iotests.test_dir, 'mid.img') test_img =3D os.path.join(iotests.test_dir, 'test.img') +reference_img =3D os.path.join(iotests.test_dir, 'reference.img') =20 class ImageCommitTestCase(iotests.QMPTestCase): '''Abstract base class for image commit test cases''' @@ -951,6 +952,105 @@ class TestCommitWithOverriddenBacking(iotests.QMPTest= Case): self.vm.qmp('block-job-complete', device=3D'commit') self.vm.event_wait('BLOCK_JOB_COMPLETED') =20 +class TestCommitLargeRuns(iotests.QMPTestCase): + """Commit runs long enough to cross commit_iteration()'s cached span."= "" + + MB =3D 1024 * 1024 + CLUSTER =3D 64 * 1024 + + # Runs several COMMIT_BUFFER_SIZE (512K) chunks long, of every kind. + SIZE =3D 32 * MB + LAYOUT =3D [ + (0, 4 * MB, 'data'), + (4 * MB, 6 * MB, 'hole'), + (10 * MB, 4 * MB, 'data'), + (14 * MB, 6 * MB, 'zero'), + (20 * MB, 4 * MB, 'data'), + (24 * MB, 8 * MB, 'hole'), + ] + + # The same, with every transition off the 512K boundary. + SIZE_FRAGMENTED =3D 384 * CLUSTER # 24M + LAYOUT_FRAGMENTED =3D [ + (0, 45 * CLUSTER, 'data'), + (45 * CLUSTER, 55 * CLUSTER, 'hole'), + (100 * CLUSTER, CLUSTER, 'data'), + (101 * CLUSTER, 49 * CLUSTER, 'zero'), + (150 * CLUSTER, 80 * CLUSTER, 'data'), + (230 * CLUSTER, CLUSTER, 'hole'), + (231 * CLUSTER, 69 * CLUSTER, 'data'), + (300 * CLUSTER, 83 * CLUSTER, 'zero'), + (383 * CLUSTER, CLUSTER, 'hole'), + ] + + def setUp(self): + self.vm =3D iotests.VM() + self.vm.launch() + + def tearDown(self): + self.vm.shutdown() + for img in (backing_img, mid_img, test_img, reference_img): + if os.path.exists(img): + os.remove(img) + + def build_images(self, layout, size): + # A pattern of its own in base, so a misplaced cluster shows up. + qemu_img_create('-f', iotests.imgfmt, backing_img, str(size)) + qemu_io('-c', f'write -P 0x11 0 {size}', backing_img) + + qemu_img_create('-f', iotests.imgfmt, '-b', backing_img, '-F', + iotests.imgfmt, mid_img, str(size)) + for offset, length, kind in layout: + if kind =3D=3D 'data': + qemu_io('-c', f'write -P 0x22 {offset} {length}', mid_img) + elif kind =3D=3D 'zero': + qemu_io('-c', f'write -z {offset} {length}', mid_img) + + # What base must equal once mid is committed into it. + qemu_img('convert', '-f', iotests.imgfmt, '-O', iotests.imgfmt, + mid_img, reference_img) + + # An empty layer above mid, so top_node=3Dmid is not the active on= e. + qemu_img_create('-f', iotests.imgfmt, '-b', mid_img, '-F', + iotests.imgfmt, test_img, str(size)) + + self.vm.cmd('blockdev-add', { + 'node-name': 'base', + 'driver': iotests.imgfmt, + 'file': {'driver': 'file', 'filename': backing_img}, + }) + self.vm.cmd('blockdev-add', { + 'node-name': 'mid', + 'driver': iotests.imgfmt, + 'file': {'driver': 'file', 'filename': mid_img}, + 'backing': 'base', + }) + self.vm.cmd('blockdev-add', { + 'node-name': 'active', + 'driver': iotests.imgfmt, + 'file': {'driver': 'file', 'filename': test_img}, + 'backing': 'mid', + }) + + def commit_and_verify(self): + self.vm.cmd('block-commit', job_id=3D'commit0', device=3D'active', + top_node=3D'mid', base_node=3D'base') + self.wait_until_completed(drive=3D'commit0') + + self.vm.cmd('blockdev-del', node_name=3D'active') + self.vm.cmd('blockdev-del', node_name=3D'mid') + self.vm.cmd('blockdev-del', node_name=3D'base') + self.assertTrue(compare_images(reference_img, backing_img)) + + def test_commit_large_runs(self): + self.build_images(self.LAYOUT, self.SIZE) + self.commit_and_verify() + + def test_commit_fragmented_runs(self): + self.build_images(self.LAYOUT_FRAGMENTED, self.SIZE_FRAGMENTED) + self.commit_and_verify() + + if __name__ =3D=3D '__main__': iotests.main(supported_fmts=3D['qcow2', 'qed'], supported_protocols=3D['file']) diff --git a/tests/qemu-iotests/040.out b/tests/qemu-iotests/040.out index 1bb1dc5f0e..d2e2a2d98f 100644 --- a/tests/qemu-iotests/040.out +++ b/tests/qemu-iotests/040.out @@ -1,5 +1,5 @@ -................................................................. +................................................................... ---------------------------------------------------------------------- -Ran 65 tests +Ran 67 tests =20 OK --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397737; cv=none; d=zohomail.com; s=zohoarc; b=Dv6a36R+vKLBnOVexTPGv9Ha4lQ1sUtNSJPFHFfH/a1+JvXB3zIytMpyL4UI17NW+wjZdSKjAjZXBzsU4zWfiNKMlY42V7H5m4DZ374SU2BkceGGWDCIohpZKnrFcWnHnfoWYFStQ9lrRQ9ie26YyAIPx4ooSIQm8OVYiKUNwQo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397737; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=oWqwUVggYdA0tHiKmfstsBnPcdY+RLvUV4CZMwRpffs=; b=FAKcLimhI1/LfsmOK5f6l5RtGriQxMtjLRKQFjZL4yC87+xHfNIN8sZvhaEspyThmSqezuYW0+oD1RFnq6/x3+zZU0z/zZCmnopqyC1x65+eO7M7Z8/+XJoUit7XL2CXeyFORJRZsCbbZyW5YZHC3wa7eyf7t3K74PMpFay7m/I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397737538569.5561768801235; Mon, 14 Sep 2026 07:55:37 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683l-0002pg-3g; Mon, 14 Sep 2026 10:53:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683J-0002cn-8D for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:02 -0400 Received: from mail-wm2-x10.google.com ([2a00:1450:4864:31::10]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683H-0005lw-DU for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:00 -0400 Received: by mail-wm2-x10.google.com with SMTP id 5b1f17b1804b1-49ccfd61ecaso20701955e9.3 for ; Mon, 14 Sep 2026 07:52:58 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.52.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:52:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397577; x=1790002377; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=oWqwUVggYdA0tHiKmfstsBnPcdY+RLvUV4CZMwRpffs=; b=rfOPj7Ik0Wu38sL33b0d4+XgldJ0/qj6CzdZvdqgY6abTt+Y8Li4y768LcYNNMVmPL 7ET53kdST9iOc1FWvz5OaR9+XBqW1VYMNKA4Cr+Y0GbCr3tv1D/AkF+Uxdc+chNbtRmr Gz1PZQ5VtxbRNTjLwpYxdcqk4ovhk/mVZSzcYi6W+Id+zxsIMC8edC+dlzhEzTpr0X+M il5nTKwGuzBI33ZZDlobpaozj41eP7r4azhRPsOdjzmMTx69cptc0/cQhUEW5orZiYqu 38ye02P4nf2XBEl5N3kZVqX1cIYe01XZEoOvNcY1FCFYR89uMrKCBVzTiMwVOfcpmBx8 yfhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397578; x=1790002378; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=oWqwUVggYdA0tHiKmfstsBnPcdY+RLvUV4CZMwRpffs=; b=KZ9CdAgrDina8aR1/6kVoi4JX1sLvY3X+nABwXK6KqIHehzf3HRXUxJWsHGIBt4f11 gbXsp7XWEzFWFchdAZZol3LdoQOZfax+v+EWFQ+6uMnEbL4xn8hZw9thnCcgNdLVlACe A8owmch343qg6f5hRVm97siw4yiNh+lU0+xLx09y0capjIrsW4gBTZz3sTQ6qK4jrgwU m5VG35oUvRw2DNduCu2IDcdMebp4UmIv5Y7ka2sdz9hNryPvFXhPI6sRrfR1Y6LAqh80 YM2loLyWKrOv6bRufwPTUWtR94Aj1RUxJboycCg8r+UTy8ulnfrejpYrUwCdhY1oc0X0 ZcXw== X-Gm-Message-State: AFuF++k66qqC86l3A3ZCJ2yVf07Wuc6JbZN58VT02H3f72KQ7D2Rz6pA D8gXLHVeqfQR32tw0l2znNYYppSZhz0xJkY/5jWNZDBVkeb41aI+vEK7hw8a2z85M8wYrn1somc Onc5i X-Gm-Gg: AYBFou1wW2Ad8YgqI1AhVEbxzwYTmvVoDNnDMLN/9rFjJ2SgHWVyMwVlz00wqOyGHny lMTbHkpHYJAGFA9foGZLG2cCLIkhnHrf64o9kxcY0onub7FwnSnACKfQ5RbPoL+xDyFlPenKB32 YFe42gs95+MZshT0i2/8JaFCvqtBJ1MTgNGglYlx4ZHwl6aYc3WLMM92fCIdEfFTynTVgUdrnes /FrHbzYyno5NZgeBEJsCLU9dy0BFc4lZn3QGL89E3QQ5HrugR+P5Tjr2mLTc2jQfOkC+KJWPE5x S571Rkc9zZ3ijb5TXsSrir1aOYcqopf4XAHVh86o2jjPT2VpqN/Tq5hnerQ3s8QMXI6Jqp25lMT hg6VW5TcQ7R9Ik49+vIETlo6sByviJ0RVfitIcb6ctUrQ9NMJRuI3EKa+18696cStQtVWE0CcwK zLyoVJZzHHvWKRwXFOHAOQIJCC28kjGsctBJJa6RsomiKA6YfE8fQawcF25b6TRVwzRyiK X-Received: by 2002:a05:600c:3ba7:b0:49c:f4e1:4c2d with SMTP id 5b1f17b1804b1-49e7a68698bmr45580575e9.16.1789397577617; Mon, 14 Sep 2026 07:52:57 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Vladimir Sementsov-Ogievskiy , John Snow , Andrey Drobyshev Subject: [PATCH v2 3/9] block/commit: batch block-status queries Date: Mon, 14 Sep 2026 16:52:44 +0200 Message-ID: <20260914145250.2792879-4-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:31::10; envelope-from=den@openvz.org; helo=mail-wm2-x10.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397738014158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev commit_iteration() asks for block status COMMIT_BUFFER_SIZE (512K) at a time, so a long run above base pays one query per 512K for an answer the whole run shares. Query the remainder of the image instead and keep the answer in a CommitStatus owned by commit_run(). Nothing can write top above base_overlay while the job runs: BLK_PERM_CONSISTENT_READ is shared only from filtered_base downwards, and the job writes below base_overlay. Copying is still bounded by the read buffer. Zeroes need no buffer, so COMMIT_ZERO_CHUNK bounds them, keeping a cancel from waiting on a huge write-zeroes, and an unallocated span is crossed in one step. Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev Reviewed-by: Andrey Drobyshev --- block/commit.c | 58 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 47 insertions(+), 11 deletions(-) diff --git a/block/commit.c b/block/commit.c index 4e0b0f9029..e8ddd46053 100644 --- a/block/commit.c +++ b/block/commit.c @@ -31,8 +31,23 @@ enum { * contiguous regions of the image is efficient. */ COMMIT_BUFFER_SIZE =3D 512 * 1024, /* in bytes */ + + /* + * Zeroes need no buffer, so they are bounded by this instead. It stays + * well under BDRV_REQUEST_MAX_BYTES and keeps a cancel from waiting on + * a multi-gigabyte write. + */ + COMMIT_ZERO_CHUNK =3D 256 * 1024 * 1024, /* in bytes */ }; =20 +/* Last block-status answer, covering [offset, end). Empty when equal. */ +typedef struct CommitStatus { + int64_t len; + int64_t offset; + int64_t end; + int ret; +} CommitStatus; + typedef struct CommitBlockJob { BlockJob common; BlockDriverState *commit_top_bs; @@ -130,26 +145,45 @@ static void commit_clean(Job *job) =20 static int coroutine_fn commit_iteration(CommitBlockJob *s, int64_t offset, - int64_t *requested_bytes, void *buf) + int64_t *requested_bytes, void *buf, CommitStatus *st) { BlockErrorAction action; - int64_t bytes =3D *requested_bytes; + int64_t bytes; int ret =3D 0; bool error_in_source =3D true; =20 - /* Copy if allocated above the base */ - WITH_GRAPH_RDLOCK_GUARD() { - ret =3D bdrv_co_common_block_status_above(blk_bs(s->top), - s->base_overlay, true, BDRV_WANT_PRECISE, offset, - COMMIT_BUFFER_SIZE, &bytes, NULL, NULL, NULL); + assert(offset >=3D st->offset); + + if (offset >=3D st->end) { + /* Copy if allocated above the base */ + WITH_GRAPH_RDLOCK_GUARD() { + ret =3D bdrv_co_common_block_status_above(blk_bs(s->top), + s->base_overlay, true, BDRV_WANT_PRECISE, offset, + st->len - offset, &bytes, NULL, NULL, NULL); + } + + if (ret < 0) { + trace_commit_one_iteration(s, offset, 0, ret); + goto fail; + } + + st->offset =3D offset; + st->end =3D offset + bytes; + st->ret =3D ret; } =20 - trace_commit_one_iteration(s, offset, bytes, ret); + ret =3D st->ret; + bytes =3D st->end - offset; =20 - if (ret < 0) { - goto fail; + /* An unallocated span costs no I/O, so it is crossed in one step. */ + if (ret & BDRV_BLOCK_ZERO) { + bytes =3D MIN(bytes, COMMIT_ZERO_CHUNK); + } else if (ret & BDRV_BLOCK_ALLOCATED) { + bytes =3D MIN(bytes, COMMIT_BUFFER_SIZE); } =20 + trace_commit_one_iteration(s, offset, bytes, ret); + if (ret & BDRV_BLOCK_ALLOCATED) { if (ret & BDRV_BLOCK_ZERO) { /* @@ -215,6 +249,7 @@ static int coroutine_fn commit_run(Job *job, Error **er= rp) int64_t n =3D 0; /* bytes */ QEMU_AUTO_VFREE void *buf =3D NULL; int64_t len, base_len; + CommitStatus st =3D { 0 }; =20 len =3D blk_co_getlength(s->top); if (len < 0) { @@ -235,6 +270,7 @@ static int coroutine_fn commit_run(Job *job, Error **er= rp) } =20 buf =3D blk_blockalign(s->top, COMMIT_BUFFER_SIZE); + st.len =3D len; =20 for (offset =3D 0; offset < len; offset +=3D n) { /* Note that even when no rate limit is applied we need to yield @@ -245,7 +281,7 @@ static int coroutine_fn commit_run(Job *job, Error **er= rp) break; } =20 - ret =3D commit_iteration(s, offset, &n, buf); + ret =3D commit_iteration(s, offset, &n, buf, &st); =20 if (ret < 0) { return ret; --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397624; cv=none; d=zohomail.com; s=zohoarc; b=XTG7AH+pPWBSrKzGY4dmbkUzbD+HsSgZyHzFt8KQUh9EBPVrWkcaRsqOhIoV3aQCrsvWHxP9FJ6kdE1mqE6xCs209LWBfLAxztiDuBEXiWSGImfsgrQ03XRZw18xxZmFu98MgPxPNCAFagFsKbVZyTDd1+sSAaCv2TnBwAZm/yI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397624; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=iP+2NOIdwPzYJO1zWMz+Ov7zV1TiXqsEYFhNlD2DqBQ=; b=n1YwQMwREfH4gQROIuxfMA+9YCNXJMrXEW95sDule6s6Vj8oOx3ou9t3R7K2uIv7hNrKqLIHdDdaHsGgjmBBnVZLUxlXBkMJhWIuW7mrBkO2x/m0vCtG+PSKNBYOWzqdNbX4z1m1Mb1ib/+13GNraXITvu57/rA9ilcfNw9vaQM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397624256851.3774257628501; Mon, 14 Sep 2026 07:53:44 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683n-0002tH-NM; Mon, 14 Sep 2026 10:53:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683S-0002eP-LT for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:12 -0400 Received: from mail-wm2-x10.google.com ([2a00:1450:4864:31::10]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683L-0005mn-3f for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:07 -0400 Received: by mail-wm2-x10.google.com with SMTP id 5b1f17b1804b1-49b912d37b6so9583295e9.0 for ; Mon, 14 Sep 2026 07:53:01 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.52.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:52:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397580; x=1790002380; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iP+2NOIdwPzYJO1zWMz+Ov7zV1TiXqsEYFhNlD2DqBQ=; b=OWfJMz6cTp60YDShIXtmeEeNb1Kbp1Em1SR7Ff6idazpHiJ3+h0uvG/DTSWYUjK+yK 5NU9ecr3w3zmb4lFp/MdebTD0fAbJuLRzH+DozcyfJ4YnT39nCpuGZOL2luTAxSydXX/ Gt5/uzsgBXJhZBgtnbTXOm9UV3f8Uts+JeyTm4H4TVNEGsvmteA1+gdtN1xtfzqZyEem +DTvKm25kSXeUE0t1mN0OidnrbwPbWkyClJCupuBSI1Qc7k+N1VZg6j7v9ZXVVbVvZzo BtO7mhnvi15P+FtkjzXPhCdyn3ZgE/VUd1GGtYzRoopeMhLub1diAWwH9K4iNLmm7KF/ A/og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397580; x=1790002380; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iP+2NOIdwPzYJO1zWMz+Ov7zV1TiXqsEYFhNlD2DqBQ=; b=RsUAC7o0pCuIgkmTI56HRHSfDv2U1mxDzDGMbzDyuQRavIMApilNRgrlr+BeTEKJrg tpfukbqoXwoQQ/mWyZMJ5motI6xFXpW5+9aYr3trBPwsbo2wengHRgNksrtM5WbfV6vy 0Mz1NTxsHEwQe+s0TqUYrHPTu7Yq526F962ICExHqDMvryIh1G3xoOnfSk7r3YVTMRNj UMoOMg3aGXDr7sVRI0Ue5vMLaWByCmahOGIRhfmJkjVZ2dczV16txA26td6G0Vf98ibI pxru6HLkw8Sdz6QQZVogx/qi7wMF05dYMQpY5xzCZgHJZ39kfRC4HSaX80wcj+TDoYCG oe2A== X-Gm-Message-State: AFuF++k9SogzoHN8wHfKSF7S7nmpS98t63pP0kkSv5jWrUffYiy715tj eRq/srAECWusFowjijHE/O8U4apj7eZdQ8/uAGqaMBjYWQmTVW0F4B57JQzvzTZ7glXsUdiOKgc 8fUXl X-Gm-Gg: AYBFou2HAmukFeT35SzDKNvB8reZE/AFneYVRQpbU3SPgcMiiwJteEFDalaCJAslxYA ybX31mGY+k6JoMeSy4AXU3CnLzes6MjL2FWW+R9vSBK0/lDhQXPlPhiIp+khAAoDmzSrckbHR4N ATWWOEIKsS0yymxMCg+Zg7/6iXOEwdrv9RzAZpSkJTpzqL/UzIjIQmBrXA9ALKn2Bp2MxGtNnau Xm+NqYW5qy1KMZ2V6HTt5Zsi4lkQXaXMA1pw6mIX4mim92bstEFGfMhDr/k2IYqDRrFBKd8Bb0g nqVS93bpQDyznIHeL6oT8anxz6xOepWN2JZxh9oRwp3Z0ZZuqUhyVLQnikn83XhF+u02N+WiaJf jC/gyqyWlmRNgMShRqWw9IFAxGcNXPoFEbo4odoODnCTXNn+hjqRVq0SBtDVq94DgKBCH7HVYaA KHTVL2mJZVwlCsuB0nmDq37vdEozpvWU8xM2vrEMQ/yqK6KhBbCIRDqtn9qxc2zSGdwGhNW5o1r di/gE0= X-Received: by 2002:a05:600c:4ecc:b0:49e:6be6:d783 with SMTP id 5b1f17b1804b1-49e7a5f419amr67189965e9.0.1789397578904; Mon, 14 Sep 2026 07:52:58 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Vladimir Sementsov-Ogievskiy , John Snow , Andrey Drobyshev Subject: [PATCH v2 4/9] iotests/124: cover backup of zero clusters and holes Date: Mon, 14 Sep 2026 16:52:45 +0200 Message-ID: <20260914145250.2792879-5-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:31::10; envelope-from=den@openvz.org; helo=mail-wm2-x10.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397625293158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev The backup tests compare content only, and their sources hold data alone, so nothing pins how backup treats zero clusters and holes. Back up a source mixing data, write-zero and holes with sync=3Dfull, sync=3Dtop and sync=3Dbitmap, and check the target with qemu-img map, which tells a copied cluster apart from a write-zero and from an untouched one. Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev Reviewed-by: Andrey Drobyshev --- tests/qemu-iotests/124 | 344 ++++++++++++++++++++++++++++++++++++- tests/qemu-iotests/124.out | 4 +- 2 files changed, 345 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124 index ab9ea4d8b5..30730e7f67 100755 --- a/tests/qemu-iotests/124 +++ b/tests/qemu-iotests/124 @@ -22,8 +22,11 @@ # =20 import os +from collections import namedtuple + import iotests -from iotests import try_remove +from iotests import (compare_images, qemu_img_create, qemu_img_map, qemu_i= o, + try_remove) from qemu.qmp.qmp_client import ExecuteError =20 =20 @@ -748,6 +751,345 @@ class TestIncrementalBackupBlkdebug(TestIncrementalBa= ckupBase): self.check_backups() =20 =20 +# Backup of sources mixing data, zero clusters and holes. + +Extent =3D namedtuple('Extent', ['start', 'length', 'kind']) +Case =3D namedtuple('Case', ['size', 'layout', 'expected'], defaults=3D[No= ne]) + +source_img =3D os.path.join(iotests.test_dir, 'source') +base_img =3D os.path.join(iotests.test_dir, 'base') +target_img =3D os.path.join(iotests.test_dir, 'target') + +SIZE =3D 64 * 1024 * 1024 +MB =3D 1024 * 1024 +CLUSTER =3D 64 * 1024 + +# Data, write-zero and holes, every combination at a cluster boundary. +MIXED =3D Case(SIZE, [ + (0, MB, 'data'), + (2 * MB, MB, 'zero'), + (5 * MB, 2 * MB, 'data'), + (8 * MB, MB, 'zero'), + (10 * MB, MB // 2, 'data'), + (20 * MB, 4 * MB, 'zero'), + (30 * MB, MB, 'data'), +]) + +# Single-cluster runs, and a zero run ending exactly at EOF. +BOUNDARY_SIZE =3D 16 * MB +BOUNDARY =3D Case(BOUNDARY_SIZE, [ + (0, CLUSTER, 'zero'), + (CLUSTER, CLUSTER, 'data'), + (2 * CLUSTER, CLUSTER, 'zero'), + (3 * CLUSTER, CLUSTER, 'data'), + (4 * CLUSTER, 8 * MB, 'zero'), + (4 * CLUSTER + 8 * MB, CLUSTER, 'data'), + (BOUNDARY_SIZE - CLUSTER, CLUSTER, 'zero'), +]) + +# A zero run past the old block_copy_chunk_size() 16M cap. +LARGE_ZERO =3D Case(48 * MB, [ + (0, MB, 'data'), + (4 * MB, 32 * MB, 'zero'), + (40 * MB, MB, 'data'), +]) + +# Image size not a multiple of the cluster size: a partial tail cluster. +TAIL_SIZE =3D 4 * MB + 4096 +TAIL_CLUSTER =3D (TAIL_SIZE // CLUSTER) * CLUSTER +TAIL_DATA =3D Case(TAIL_SIZE, [ + (0, MB, 'data'), + (2 * MB, MB, 'zero'), + (4 * MB, TAIL_SIZE - 4 * MB, 'data'), +]) +# The partial tail cluster falls out of zero_bitmap, so it copies as data. +TAIL_ZERO =3D Case(TAIL_SIZE, [ + (0, MB, 'data'), + (2 * MB, TAIL_SIZE - 2 * MB, 'zero'), +], [ + (0, MB, 'data'), + (2 * MB, TAIL_CLUSTER - 2 * MB, 'zero'), + (TAIL_CLUSTER, TAIL_SIZE - TAIL_CLUSTER, 'data'), +]) + + +def coalesce(extents): + out =3D [] + for e in extents: + prev =3D out[-1] if out else None + adjacent =3D prev is not None and prev.start + prev.length =3D=3D = e.start + if adjacent and prev.kind =3D=3D e.kind: + out[-1] =3D prev._replace(length=3Dprev.length + e.length) + else: + out.append(e) + return out + + +def layout_to_extents(layout, size, gap=3D'hole'): + extents =3D [] + pos =3D 0 + for offset, length, kind in layout: + if offset > pos: + extents.append(Extent(pos, offset - pos, gap)) + extents.append(Extent(offset, length, kind)) + pos =3D offset + length + if pos < size: + extents.append(Extent(pos, size - pos, gap)) + return coalesce(extents) + + +def create_image(path, size, backing=3DNone, opts=3DNone): + args =3D ['-f', iotests.imgfmt] + if opts: + args +=3D ['-o', opts] + if backing: + args +=3D ['-b', backing, '-F', iotests.imgfmt] + qemu_img_create(*args, path, str(size)) + + +class TestBackupZeroClusters(iotests.QMPTestCase): + def setUp(self): + self.vm =3D iotests.VM() + self.vm.launch() + + def tearDown(self): + self.vm.shutdown() + for img in (source_img, base_img, target_img): + if os.path.exists(img): + os.remove(img) + + def hmp_write(self, drive, cmd): + res =3D self.vm.hmp_qemu_io(drive, cmd) + assert 'error' not in res['return'].lower(), res + + def write_layout(self, layout): + for offset, length, kind in layout: + opt =3D '-z' if kind =3D=3D 'zero' else '-P 0x5a' + self.hmp_write('src', f'write {opt} {offset} {length}') + + def assert_map(self, case, backing=3DFalse, gap=3D'hole'): + # Content is not enough, pin the data/zero/hole split as well. + def classify(e): + is_hole =3D e['depth'] > 0 if backing else not e['present'] + return 'hole' if is_hole else ('zero' if e['zero'] else 'data') + + actual =3D coalesce([Extent(e['start'], e['length'], classify(e)) + for e in qemu_img_map(target_img)]) + layout =3D case.expected if case.expected else case.layout + + self.assertEqual(actual, layout_to_extents(layout, case.size, gap)) + + def add_source(self, case=3DMIXED, backing=3DNone, opts=3DNone): + create_image(source_img, case.size, backing, opts) + + self.vm.cmd('blockdev-add', { + 'node-name': 'src', + 'driver': iotests.imgfmt, + 'file': {'driver': 'file', 'filename': source_img}, + }) + + # Write through the node, so an attached bitmap sees it. + self.write_layout(case.layout) + + def dirty_layout(self, case): + # A new bitmap tracks nothing yet: dirty all, then re-apply. + self.vm.cmd('block-dirty-bitmap-add', node=3D'src', name=3D'bm0') + self.hmp_write('src', f'write -z 0 {case.size}') + self.write_layout(case.layout) + + def do_backup(self, sync, case, target_backing=3DNone, prefill=3DNone, + **kwargs): + create_image(target_img, case.size, target_backing) + + if prefill is not None: + # Not zero, so a skipped cluster is provably untouched. + qemu_io('-c', f'write -P {prefill} 0 {case.size}', target_img) + + self.vm.cmd('blockdev-add', { + 'node-name': 'target', + 'driver': iotests.imgfmt, + 'file': {'driver': 'file', 'filename': target_img}, + }) + + self.vm.cmd('blockdev-backup', device=3D'src', target=3D'target', + job_id=3D'bk0', sync=3Dsync, **kwargs) + self.wait_until_completed(drive=3D'bk0') + + self.vm.cmd('blockdev-del', node_name=3D'target') + self.vm.cmd('blockdev-del', node_name=3D'src') + + def backup_and_check(self, sync, case, gap=3D'zero', backing=3DFalse, + **kwargs): + self.do_backup(sync, case, **kwargs) + self.assertTrue(compare_images(source_img, target_img)) + self.assert_map(case, backing=3Dbacking, gap=3Dgap) + + def test_full(self): + self.add_source() + self.backup_and_check('full', MIXED) + + def test_full_zero_overwrite(self): + # full skips holes, so only check that zero overwrites prefill. + case =3D Case(SIZE, [(2 * MB, MB, 'zero')]) + self.add_source(case) + self.do_backup('full', case, prefill=3D0xcc) + qemu_io('-c', f'read -P 0 {2 * MB} {MB}', target_img) + + def test_bitmap(self): + self.add_source() + self.dirty_layout(MIXED) + self.backup_and_check('bitmap', MIXED, bitmap=3D'bm0', + bitmap_mode=3D'never') + + def test_top(self): + # Non-zero backing data, so a hole and an explicit zero differ. + create_image(base_img, SIZE) + qemu_io('-c', f'write -P 0x33 0 {SIZE}', base_img) + + self.add_source(backing=3Dbase_img) + self.backup_and_check('top', MIXED, gap=3D'hole', backing=3DTrue, + target_backing=3Dbase_img) + + def test_boundary_full(self): + self.add_source(BOUNDARY) + self.backup_and_check('full', BOUNDARY) + + def test_boundary_bitmap(self): + self.add_source(BOUNDARY) + self.dirty_layout(BOUNDARY) + self.backup_and_check('bitmap', BOUNDARY, bitmap=3D'bm0', + bitmap_mode=3D'never') + + def test_large_zero_full(self): + self.add_source(LARGE_ZERO) + self.backup_and_check('full', LARGE_ZERO) + + def test_large_zero_bitmap(self): + self.add_source(LARGE_ZERO) + self.dirty_layout(LARGE_ZERO) + self.backup_and_check('bitmap', LARGE_ZERO, bitmap=3D'bm0', + bitmap_mode=3D'never') + + def test_huge_zero(self): + # A zero run past the cap on a single write-zeroes request, so it + # has to be split. qemu-io caps one write at 2G, hence three. + case =3D Case(2560 * MB, [(0, 1024 * MB, 'zero'), + (1024 * MB, 1024 * MB, 'zero'), + (2048 * MB, 512 * MB, 'zero')]) + self.add_source(case) + self.backup_and_check('full', case) + + def test_tail_data(self): + self.add_source(TAIL_DATA) + self.backup_and_check('full', TAIL_DATA) + + def test_tail_zero(self): + self.add_source(TAIL_ZERO) + self.backup_and_check('full', TAIL_ZERO) + + def test_tail_zero_bitmap(self): + # Same tail rounding as test_tail_zero, via the bitmap scan. + self.add_source(TAIL_ZERO) + self.dirty_layout(TAIL_ZERO) + self.backup_and_check('bitmap', TAIL_ZERO, bitmap=3D'bm0', + bitmap_mode=3D'never') + + def test_mixed_cluster(self): + # 4K source clusters, so content varies inside one 64K cluster. + case =3D Case(2 * CLUSTER, [ + (0, 8 * 1024, 'zero'), + (8 * 1024, 8 * 1024, 'data'), + (CLUSTER, CLUSTER, 'zero'), + ], [ + # A mixed cluster must be data, or the data at [8k, 16k) is lo= st. + (0, CLUSTER, 'data'), + (CLUSTER, CLUSTER, 'zero'), + ]) + self.add_source(case, opts=3D'cluster_size=3D4k') + self.backup_and_check('full', case) + + def test_top_zero_broken(self): + # An overlay hole before an explicit-zero run: the zero prefix + # must stop at the hole, or the backing data under it is lost. + size =3D 2 * CLUSTER + create_image(base_img, size, opts=3D'cluster_size=3D4k') + qemu_io('-c', f'write -P 0x33 0 {size}', base_img) + + # [0, 4k) stays a hole, the zero run spills into cluster 1. + case =3D Case(size, [(4096, CLUSTER, 'zero')]) + self.add_source(case, backing=3Dbase_img, opts=3D'cluster_size=3D4= k') + + self.do_backup('top', case, target_backing=3Dbase_img) + self.assertTrue(compare_images(source_img, target_img)) + + def test_bitmap_straddle(self): + # One dirty run straddling a zero/data transition, both ways. + case =3D Case(SIZE, []) + self.add_source(case) + self.vm.cmd('block-dirty-bitmap-add', node=3D'src', name=3D'bm0') + + # One contiguous dirty run each: [7M, 9M) zero to data, and + # [15M, 17M) data to zero. + self.write_layout([ + (7 * MB, MB, 'zero'), + (8 * MB, MB, 'data'), + (15 * MB, MB, 'data'), + (16 * MB, MB, 'zero'), + ]) + + self.do_backup('bitmap', case, bitmap=3D'bm0', bitmap_mode=3D'neve= r') + self.assertTrue(compare_images(source_img, target_img)) + + def test_bitmap_fragmented(self): + # Isolated single-cluster dirty spots: nothing outside them + # may reach the target. + case =3D Case(SIZE, []) + self.add_source(case) + + # Written before the bitmap exists, so untracked. + self.hmp_write('src', f'write -P 0x5a 0 {16 * MB}') + + self.vm.cmd('block-dirty-bitmap-add', node=3D'src', name=3D'bm0') + + # The only dirty bit in the data region, with a pattern of its own. + data_island =3D 4 * MB + self.hmp_write('src', f'write -P 0x7b {data_island} {CLUSTER}') + + # The only dirty bit in the hole region. + zero_island =3D 24 * MB + self.hmp_write('src', f'write -z {zero_island} {CLUSTER}') + + self.do_backup('bitmap', case, bitmap=3D'bm0', bitmap_mode=3D'neve= r', + prefill=3D0xcc) + + # Only the two isolated spots should have reached target ... + qemu_io('-c', f'read -P 0x7b {data_island} {CLUSTER}', target_img) + qemu_io('-c', f'read -P 0 {zero_island} {CLUSTER}', target_img) + + # ... everything else must still hold the prefill pattern. + qemu_io('-c', f'read -P 0xcc 0 {CLUSTER}', target_img) + qemu_io('-c', f'read -P 0xcc {8 * MB} {CLUSTER}', target_img) + + def test_bitmap_clean_zero_gap(self): + # A clean zero run between two dirty ones must stay prefilled. + case =3D Case(16 * MB, []) + self.add_source(case) + self.hmp_write('src', f'write -z 0 {16 * MB}') + + self.vm.cmd('block-dirty-bitmap-add', node=3D'src', name=3D'bm0') + self.write_layout([(2 * MB, MB, 'zero'), (6 * MB, MB, 'zero')]) + + self.do_backup('bitmap', case, bitmap=3D'bm0', bitmap_mode=3D'neve= r', + prefill=3D0xcc) + + qemu_io('-c', f'read -P 0 {2 * MB} {MB}', target_img) + qemu_io('-c', f'read -P 0 {6 * MB} {MB}', target_img) + + qemu_io('-c', f'read -P 0xcc 0 {2 * MB}', target_img) + qemu_io('-c', f'read -P 0xcc {3 * MB} {3 * MB}', target_img) + qemu_io('-c', f'read -P 0xcc {7 * MB} {9 * MB}', target_img) + + if __name__ =3D=3D '__main__': iotests.main(supported_fmts=3D['qcow2'], supported_protocols=3D['file'], diff --git a/tests/qemu-iotests/124.out b/tests/qemu-iotests/124.out index fa16b5ccef..5ce2f9a2ed 100644 --- a/tests/qemu-iotests/124.out +++ b/tests/qemu-iotests/124.out @@ -1,5 +1,5 @@ -............. +.............................. ---------------------------------------------------------------------- -Ran 13 tests +Ran 30 tests =20 OK --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397726; cv=none; d=zohomail.com; s=zohoarc; b=J3ZKjdZyl6OG9t9YShzI36r4XgPtOspbTW+aA6f3uU3NeoSa4CWWHDMrUZAicL54gFQSXdEEsjZ7nE9jviX2NZ7DkeZKa8C7urS/xjC0vnlT0AbPKZZuHMemVxYUtf+qzfSZ83WJ/dYBYxq7Lm19kVxnD8uS/1KVNZMXZz5fNKE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397726; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=SJFEI1lQ8SXWXqcCjUVxKX69y0HMHBq5YIZK1wiDeeQ=; b=WtI1+VzlZxtapMJxcEgn4wU7HAaC6xg7RQGgoBz9zrO3sAMUIe0Xy1L3vLEHguumNkkTHasayUP+Tbr1IO0/LzIEkc55uKhSnw131UJFRJSD3OiD32oyvRxguxx2+TX0zGF7k1mU65hoYQh4faXdHx9yjRtCqVVjZ6KuqNONmjU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397726899488.1641610183709; Mon, 14 Sep 2026 07:55:26 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683p-0002ws-8H; Mon, 14 Sep 2026 10:53:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683P-0002e8-0D for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:08 -0400 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683L-0005nA-2G for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:06 -0400 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-49e6e69d528so21683425e9.2 for ; Mon, 14 Sep 2026 07:53:02 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.52.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:53:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397581; x=1790002381; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SJFEI1lQ8SXWXqcCjUVxKX69y0HMHBq5YIZK1wiDeeQ=; b=aNqOaKUpja4u5utFO0hvDl3rz/EZQMk6KO3Avr4zN96456tVOosacL1Bd9ZYlcXhIr xTDNa7tTKdMKcewBlfiq085tE8XdYcvQQxt9RnE651Sq92ztz1DI4/2kBrmGDLFx5f1J tEUVVwwbYDqAPETHNbgx5cEGgIiv1uosJk+u0wj5es4oNQN5eA4KCyerJ94tzqLPM6C1 yNHPv38pZiD3z4dktPbhM+IHtPdr+ByehyQtCvteuU4pe/ADlTboPESyrrctzJDlKTpQ 3nLtgvbwrSpT5hEGimCdsp3Cm+FhztBTKvnyLCU6mJfe8GFRD3aLlJaXLgSYE4icUJbn hR7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397581; x=1790002381; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=SJFEI1lQ8SXWXqcCjUVxKX69y0HMHBq5YIZK1wiDeeQ=; b=a5Wc/1f0qfUfPDRMgb+QWiDfcv/bNo7cHnzvNsA0ruqOm/bnL6ToArQWXbBv6xFcT/ EU/HUsI7BagLxD4Dcdx+7nVsNQNrrn8hmo57nT7I9hs+TyAnuFjeBfkYux+zd2lVelnb 7jJLuzzJObWMvcAGYcCmfIG+bZu3PJDBpwzA/rqUpOjU2kzos+m6imMFSUvRhbOmQl/M nQjXPY1nmbBzIk1VNPsc48SetqHdlcZCKD8h1qO2y7WXqTcy/UkQWplSQh0sym6XU8oe bHdV52iNXrrNyghQ4fgbQYyTKlN1Nx14iFlAMJeg2KnWBPrZq9XwdvVA5jSYazLxCKqr /J1A== X-Gm-Message-State: AFuF++mn2h/uTjGUxTT3njSPniZWDTOUb27+3YUOU+LzWNGV+prSGMHF Hi81fiNv6hJBbZ7+YICMcxCAzFi4PJvr0TYYd3+gomLd3ypRCRajbeSw6d4tJrylvTYrRzR2bh0 XcFor X-Gm-Gg: AYBFou0FjiXazRzkDd7e8HIwRpf2Tnv4M3Ez3teFKYixP9puS6hWPlfjVFfYZdIcYjP 6RezCUzMc3VF+64OBmO9apRa7a6avdN3WW2lFqnxvZ7b3t1uuRk8zfomhOODcFTyEMy6HFDNVpy LDS92wsxnPIMEMnJ02c6+cJO/IEQABBo7iKAvkNiK1j58g34/Z81UUOgrp9ntN8qkYHtLAXh8yW dG35+pylEeKIdZQHeBadmrk9ZAWL5pMoqNYXoNGqCUXw72UIGTEHeWBeMoECAQPOmstZyzCRyzG Rq6so1a9DkSZV08X3eWJdv3PKUKyCmsJoh+VyYSbu4TWytXmKnx2//aSTe4MTzWmlHbOtp8VHK4 z0Jlay1ulM6Y9110RRNg4KqjxnARZRwJie8s6ZKGNYSx94eDYfWcdVMTcCrlX30tth6Ko5hTkAX Y86CXjyP2Gw18eU8I4cZT7X/VIzeADteELqh4k6W/8zcVpUgy/NCr0i15ey/leC+ixr6uT X-Received: by 2002:a05:600c:1992:b0:49c:db20:da13 with SMTP id 5b1f17b1804b1-49e7a65cecfmr37370525e9.5.1789397580744; Mon, 14 Sep 2026 07:53:00 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Andrey Drobyshev , Vladimir Sementsov-Ogievskiy , John Snow Subject: [PATCH v2 5/9] block/block-copy: don't reserve memory for zero tasks Date: Mon, 14 Sep 2026 16:52:46 +0200 Message-ID: <20260914145250.2792879-6-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32f; envelope-from=den@openvz.org; helo=mail-wm1-x32f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397728246158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev block_copy_dirty_clusters() charges task->req.bytes to the shared memory pool for every task, but a COPY_WRITE_ZEROES task allocates no bounce buffer, so there is nothing to account for. Add block_copy_task_shres_bytes() and use it at all three call sites. Zero tasks lose the throttling the pool gave them incidentally and are bounded only by BLOCK_COPY_MAX_WORKERS (64). Reviewed-by: Andrey Drobyshev Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev --- block/block-copy.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index 1826c2e1c7..21ebe8aec2 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -453,6 +453,11 @@ void block_copy_set_progress_meter(BlockCopyState *s, = ProgressMeter *pm) s->progress =3D pm; } =20 +static uint64_t block_copy_task_shres_bytes(BlockCopyTask *task) +{ + return task->method =3D=3D COPY_WRITE_ZEROES ? 0 : task->req.bytes; +} + /* * Takes ownership of @task * @@ -474,7 +479,7 @@ static coroutine_fn int block_copy_task_run(AioTaskPool= *pool, =20 aio_task_pool_wait_slot(pool); if (aio_task_pool_status(pool) < 0) { - co_put_to_shres(task->s->mem, task->req.bytes); + co_put_to_shres(task->s->mem, block_copy_task_shres_bytes(task)); block_copy_task_end(task, -ECANCELED); g_free(task); return -ECANCELED; @@ -605,7 +610,7 @@ static coroutine_fn int block_copy_task_entry(AioTask *= task) progress_work_done(s->progress, t->req.bytes); } } - co_put_to_shres(s->mem, t->req.bytes); + co_put_to_shres(s->mem, block_copy_task_shres_bytes(t)); block_copy_task_end(t, ret); =20 if (s->discard_source && ret =3D=3D 0) { @@ -816,7 +821,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_stat= e) =20 trace_block_copy_process(s, task->req.offset); =20 - co_get_from_shres(s->mem, task->req.bytes); + co_get_from_shres(s->mem, block_copy_task_shres_bytes(task)); =20 offset =3D task_end(task); bytes =3D end - offset; --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397614; cv=none; d=zohomail.com; s=zohoarc; b=TOyGsdmh3ewxih9vxyozMcBlSUQcfF6mt2Tu8fxehPPgJdka16DlFfqBJDGedD0rfjUvQtQXlM5qVslNFY05C7btw0xKgq618aieKi72Pbl1jZmSn0cVTx8SzdYQupXPf+fdT3xuDfO9wacF13VMJ/WidXM9m/EjrMEhrdxQfR4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397614; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=K0kBd+JRwRmYuSzj9Gu1odey8tK1kya/yjE5FN2wYZ8=; b=YbuURKmcrfHGWvfh9MPtoioPrDnr87QlwC5t68XjVXFCPs9oh4X/gZjjv9CqT5I2ScVneTbF/0XUKOo+vmV7NB9VuLqDlXjn0JgRX13aTWFRBpVe3nfiZdPbVyf1Vc2ndr3621ED/vMzbsdWjrrPHidIXVey2G7tlJ5O+j3wF80= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397614363429.0896107377605; Mon, 14 Sep 2026 07:53:34 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683m-0002qK-54; Mon, 14 Sep 2026 10:53:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683S-0002eU-Mt for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:12 -0400 Received: from mail-wm2-x11.google.com ([2a00:1450:4864:31::11]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683O-0005nR-Nn for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:09 -0400 Received: by mail-wm2-x11.google.com with SMTP id 5b1f17b1804b1-49e79a408deso4683445e9.2 for ; Mon, 14 Sep 2026 07:53:03 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.53.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:53:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397582; x=1790002382; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=K0kBd+JRwRmYuSzj9Gu1odey8tK1kya/yjE5FN2wYZ8=; b=XFwGbNXfZM3R44knpr4kPAcU0hDUPr2+nnGhVPVDolwO05t4VauWvLkHD+VRDsdweA lkJppfJYkkuSFWodgXKEDWpTr5QmfA9dwZ8phYXPJXqeqMC7sSsf+pqLsY8R1234brTX GaPI1BeJL1v6POEfq9K+0mDpnpc7i5mHMiLJq+YnixLoKUmd2UtHQXoH+R2KH9TG0m/G MGlgPwuTcQu9iinByfpC3ngC89fBrmRYjk0AdDFWFJsQq6bSeLsUeJS1oYOeEpDxpmde Ktzzf87RQo4SkbsYYPsqgJMp7rK8BYxVQjUlCz79K1J9NU/jKwxXOM7of6J25dyIaio7 KroA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397582; x=1790002382; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=K0kBd+JRwRmYuSzj9Gu1odey8tK1kya/yjE5FN2wYZ8=; b=Dp7uwI9wnb9UouW+1Yw/h3xK7BOp5M7VcNXj6YcJlqdlBgRNJEzlmiEYfEgV8vyL4A 7WtZwBavtGYp2NnYkPP8SWg9eWcXVrL/zjjoG5EdMEcvzTTTvKzP6zs63bDvpuQNQ5Ec DUoVHU3eYULiRpHAJ8y+8EGCmNtDFvOKtInMqtL0EbrNqjfp5fTeoQpWaHXKcH31Xd+Q NpIBUqLnOJwqdpe0l2IPnsIvcaQxbXawBu/bps9wv9cS1l/2JLoBG3zHuHm5xho2r8nY jqDNRY0KcePfyYl1rpTQEaP/Qjv0tQnv5D6tnPWzX7U61MAiQxDIsEKtjNjDPxuAFXPF ajAg== X-Gm-Message-State: AFuF++k6JY5mZysRdefe9H30OPQgY4SF/6dI42yST0eHQDGN/22ywQ+c aiwO3GadxvDO0pJbd9fcdrOiWwyUlGX/bXlJksOHtUcJaxtviVZOReJDQwqwR331cbMQzKc2S/p bEl6r X-Gm-Gg: AYBFou3qNmImiNAX6TudCfMNACKNXWLabTsEIXQtYnr8jFlQwkPjFPI0Pbn6ubd0a42 grgNX+BHllhAy2obFYK/ZWLs7CKJtmZV3EW7DLltzisrnmfPze3WktHorsTtfLa5oNSLVqxj4AU xM7mRFpZ1oAsEJr0V3GOSUM//uze6gSVGDnmYrCdSvKW9n16lSDqLMP42PZcAT+gVuxdUqNiafb mPkIoPoif7gCdUeKubymLsgEuaAu63pu/ZwuOY2IG/t3igBXtgz+QUe8v7oI/FEZC8vRmNzjktf 7/LS64OhrOQHYk3gMslOZv2qc7OvBPsewz0zMKJP23AtUaJHgQ0GhIodf8Bi6f56w8POB0kAJPV pl3M4bo13eHXbTZ2kxcSuLVBHMzJV8zUq/GCQ0/yHxifyC3QLBNs95tIFNQwj5Ja7GQBrrhefIQ 90030sCbfAuHKrGIdgyabkYvkCDV1b9HadJB2rTNhQUhlPK04ZPxtnrLuxhNbBnPhLZvasHcB8/ peyE/8= X-Received: by 2002:a05:600c:4745:b0:49e:799e:c1c3 with SMTP id 5b1f17b1804b1-49e7a6868a8mr33804605e9.17.1789397582064; Mon, 14 Sep 2026 07:53:02 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Andrey Drobyshev , Vladimir Sementsov-Ogievskiy , John Snow Subject: [PATCH v2 6/9] block/block-copy: extract block_copy_set_task_method() Date: Mon, 14 Sep 2026 16:52:47 +0200 Message-ID: <20260914145250.2792879-7-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:31::11; envelope-from=den@openvz.org; helo=mail-wm2-x11.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397617337158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev Move the per-task block-status query and its skip_unallocated and BDRV_BLOCK_ZERO handling out of block_copy_dirty_clusters() into block_copy_set_task_method(). No behavior change. The next patch changes that decision, and having it in one place keeps that change to just the new logic. The block-status result no longer lands in the caller's @ret, so drop the note saying @ret may be positive at the out: label. Reviewed-by: Andrey Drobyshev Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev --- block/block-copy.c | 46 ++++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index 21ebe8aec2..d71d070dbd 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -741,6 +741,35 @@ int64_t coroutine_fn block_copy_reset_unallocated(Bloc= kCopyState *s, return ret; } =20 +/* + * Decide how @task is copied: COPY_WRITE_ZEROES if it reads as zero. May + * shrink @task. Returns false if @task is to be skipped (already ended, + * not freed). + */ +static bool coroutine_fn GRAPH_RDLOCK +block_copy_set_task_method(BlockCopyState *s, BlockCopyTask *task) +{ + int ret; + int64_t status_bytes; + + ret =3D block_copy_block_status(s, task->req.offset, task->req.bytes, + &status_bytes); + assert(ret >=3D 0); /* never fail */ + if (status_bytes < task->req.bytes) { + block_copy_task_shrink(task, status_bytes); + } + if (qatomic_read(&s->skip_unallocated) && !(ret & BDRV_BLOCK_ALLOCATED= )) { + block_copy_task_end(task, 0); + trace_block_copy_skip_range(s, task->req.offset, task->req.bytes); + return false; + } + if (ret & BDRV_BLOCK_ZERO) { + task->method =3D COPY_WRITE_ZEROES; + } + + return true; +} + /* * block_copy_dirty_clusters * @@ -773,7 +802,6 @@ block_copy_dirty_clusters(BlockCopyCallState *call_stat= e) while (bytes && aio_task_pool_status(aio) =3D=3D 0 && !qatomic_read(&call_state->cancelled)) { BlockCopyTask *task; - int64_t status_bytes; =20 task =3D block_copy_task_create(s, call_state, offset, bytes); if (!task) { @@ -787,24 +815,12 @@ block_copy_dirty_clusters(BlockCopyCallState *call_st= ate) =20 found_dirty =3D true; =20 - ret =3D block_copy_block_status(s, task->req.offset, task->req.byt= es, - &status_bytes); - assert(ret >=3D 0); /* never fail */ - if (status_bytes < task->req.bytes) { - block_copy_task_shrink(task, status_bytes); - } - if (qatomic_read(&s->skip_unallocated) && - !(ret & BDRV_BLOCK_ALLOCATED)) { - block_copy_task_end(task, 0); - trace_block_copy_skip_range(s, task->req.offset, task->req.byt= es); + if (!block_copy_set_task_method(s, task)) { offset =3D task_end(task); bytes =3D end - offset; g_free(task); continue; } - if (ret & BDRV_BLOCK_ZERO) { - task->method =3D COPY_WRITE_ZEROES; - } =20 if (!call_state->ignore_ratelimit) { uint64_t ns =3D ratelimit_calculate_delay(&s->rate_limit, 0); @@ -845,8 +861,6 @@ out: * block_copy_task_run. If it fails, it means some task already fa= iled * for real reason, let's return first failure. * Still, assert that we don't rewrite failure by success. - * - * Note: ret may be positive here because of block-status result. */ assert(ret >=3D 0 || aio_task_pool_status(aio) < 0); ret =3D aio_task_pool_status(aio); --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397721; cv=none; d=zohomail.com; s=zohoarc; b=G60BmM+Ah7NgxAwsaZOe4/KIiLEGtg3wKdpZUqTYPT72/Y4Iz5/BcDeZqY57Rx6uhjLyZOE7bAa7jJmz4Ef71+6fOAiHOT+6Fc0jMX/fjFPTd9HPfAFcyvjggnM5EagDid0mhtd2PKDz4dB2FF1AzKNpctLxaU7Cx5Qw7tke5CI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397721; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ZAQIX6p6+0pj7RUdjT3ml8ET9Xu5Dsntmwd7YXqtoog=; b=li3uIQposxVpomMpVtxHWwbsI8F9IQZC/Mrkp/bWAR9oRq0ZncCR+2WLDcCana1G9BTwaQvdxQK2XUNOirsG/sAqH0mGvhACZwyXUcWVZzO23pSu2aszhGzLfNt3c2GPm6H4NqOn75DYC2WknjbfsR/0J4IVxT43wvtDRSL6ImU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397721612560.8783062616399; Mon, 14 Sep 2026 07:55:21 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x684N-00039a-Lh; Mon, 14 Sep 2026 10:54:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683S-0002ec-OE for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:12 -0400 Received: from mail-wm2-x10.google.com ([2a00:1450:4864:31::10]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683O-0005nk-OE for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:09 -0400 Received: by mail-wm2-x10.google.com with SMTP id 5b1f17b1804b1-49b965f447cso16313305e9.3 for ; Mon, 14 Sep 2026 07:53:04 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.53.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:53:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397583; x=1790002383; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ZAQIX6p6+0pj7RUdjT3ml8ET9Xu5Dsntmwd7YXqtoog=; b=FIoN/pkHZngkIo+28R9/bCj33/DF7muL+32jTDy1uGG8bnbdVVAxtPDBf+8LYo4TCH Q2ehYKr01FzwLWdBwHjcRu1y46L7nTv5HfqkmNpdWlgQ15uLZmqFgYbqmA64MdXR8JFE /oOGYw8WniE+IAiDv9eBSBCKqEMQ/N7ihhtrVd5iCQO6owDJpsc3UEDM71EuaPIZXM5X //uIVzFPFz8hGeZGolM34Vzi2OZeb8T1PjjUpdW0yjgOqyZusdkbibKNJNe2/B70rcX0 wcbcBdM2Fl6cP3+aVXE8iBvpuDvYh0LlfONLl/dLpQ/rChEnAByO9iooUz4IucdixNAo s6ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397583; x=1790002383; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ZAQIX6p6+0pj7RUdjT3ml8ET9Xu5Dsntmwd7YXqtoog=; b=Scn0oKw+mkdlH3BlF9RvMexr6YAnnuGRRnUmckqI9Dj5UMOacVbsBPj6Cm7y1L0XGs RXCd6C8gxc3OkbzjMnJsJOBjKEmSc3r3G5SLnKmc17CprOSS6lRmQELR7B5e4yoxV6SA VxpS2vizmOzpRVR4zgPsL317slvjPJOCMTXUJU/QBvU1cJAa9AtCGhr/Xf1LRCAA2CkK PQtYSvikgdIzHVYts8FZJO+2JUdCKVgvVhD4+UjQ46QlotwGtBm+lD1L0bUrrH51LsnS R/rsshhjKfXeFonHsfslOCf9udw66i2uxIYP5SFvB+Kj4N6vFnCGbhCm6+f81ycvz1F/ 0X5g== X-Gm-Message-State: AFuF++lA/lJHyvSVhFjV91OPbUdiamj5l0TrQYHPAkJyxx1GI1rvMcdP J4Jh7cNrSgbKVSpkRvmWJ9n4MPI7Le9s8Qf83GYzfSHYGQPp42C6wJrjOej81e+o9Ggb9kvnt+z FEDy9 X-Gm-Gg: AYBFou1h+jiD+M1okCBkw0+gh5A8GitaiUxfNJHR0o1G7gTiyVYGugGPkWjHEZEmxxL 5QdjzsbbzkfgLXp31u+xIEebj+QbX7WDoyhH/f36Koo3MAZz2jIPyG3l5tv0e2uZj+lfGYB7r+Q SnqkiD64iZZeVSGyW6wm2Gk8ki822nbO3t1C3IIMCUDxFThNuJJ5QJcrRXSylCS+yfg3+YokVRK 9lkV3m4ZdZx7FotX2Z6Ukox7MsrBcsQA48t793Hlw+cOQyZbi2k9iSF0agsEWnRQ4A92BipRpBh YH80o2GO/xYG0wXOy8By1gxELhnpErcq4WOk08xb5UBQzC+j8rSijDNJhXGBrSAbXEX3hMs6JDG Bhkj0TeO+0K2JtVU6TDfi3BI19tU6tWOpaeRcXc0QcptYiRJgHerayrG63++juRCuoiw7sOVsf7 hwcisExCWPXiaeK141pK5SQSmXvy5eWyI8HoT5n/jz5ufJMy9Y9Er61E2F5wR7prHepv9NkQ== X-Received: by 2002:a05:600c:4686:b0:49c:fa20:cc05 with SMTP id 5b1f17b1804b1-49e7a67bc37mr36966655e9.28.1789397583265; Mon, 14 Sep 2026 07:53:03 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Vladimir Sementsov-Ogievskiy , John Snow , Andrey Drobyshev Subject: [PATCH v2 7/9] block/block-copy: track known-zero source clusters Date: Mon, 14 Sep 2026 16:52:48 +0200 Message-ID: <20260914145250.2792879-8-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:31::10; envelope-from=den@openvz.org; helo=mail-wm2-x10.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397724113158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev block_copy_reset_unallocated()'s up-front scan already queries the source, and the same query reports zero-ness. The copy loop re-queries per task and, having no answer up front, can only size a task by the copy buffer, so a mostly-zero image is cut into thousands of pieces that each turn out to read as zero. Cache the answer in zero_bitmap, published by zero_bitmap_valid once the scan is done. block_copy_task_create() then picks the method from the bitmap and ends the task where the answer changes, so a zero task never reaches into data and a copy task never swallows a zero run. The per-task query stays as the fallback until the flag is set, since CBW intercepts guest writes while the scan is still running. The scan and the query it replaces must resolve BDRV_BLOCK_ZERO against the same part of the chain, so the scan moves from bdrv_co_is_allocated() to bdrv_co_block_status_above() and the base selection is factored out into block_copy_status_base(). zero_bitmap is a plain HBitmap, not a BdrvDirtyBitmap: an internal cache has no business in query-named-block-nodes. One writer and readers gated by the flag need no mutex, but the publish needs ordering, hence store-release and load-acquire. It is allocated on demand, so sync=3Dnone and a standalone copy-before-write filter do not pay for it. Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev Reviewed-by: Andrey Drobyshev --- block/backup.c | 1 + block/block-copy.c | 108 ++++++++++++++++++++++++++++++++----- include/block/block-copy.h | 1 + 3 files changed, 98 insertions(+), 12 deletions(-) diff --git a/block/backup.c b/block/backup.c index d4713fa1cd..11d70243e2 100644 --- a/block/backup.c +++ b/block/backup.c @@ -280,6 +280,7 @@ static int coroutine_fn backup_run(Job *job, Error **er= rp) offset +=3D count; } block_copy_set_skip_unallocated(s->bcs, false); + block_copy_set_zero_bitmap_valid(s->bcs); } =20 if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_NONE) { diff --git a/block/block-copy.c b/block/block-copy.c index d71d070dbd..94d4f3dd69 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -20,6 +20,8 @@ #include "block/block_int-io.h" #include "block/dirty-bitmap.h" #include "block/reqlist.h" +#include "qemu/hbitmap.h" +#include "qemu/host-utils.h" #include "system/block-backend.h" #include "qemu/units.h" #include "qemu/co-shared-resource.h" @@ -157,6 +159,10 @@ typedef struct BlockCopyState { bool skip_unallocated; /* atomic */ /* State fields that use a thread-safe API */ BdrvDirtyBitmap *copy_bitmap; + /* Clusters reading as zero; allocated on demand, frozen once valid. */ + HBitmap *zero_bitmap; + /* Published only after the scan, with skip_unallocated already false.= */ + bool zero_bitmap_valid; /* atomic, store-release/load-acquire */ ProgressMeter *progress; SharedResource *mem; RateLimit rate_limit; @@ -190,6 +196,7 @@ block_copy_task_create(BlockCopyState *s, BlockCopyCall= State *call_state, int64_t offset, int64_t bytes) { BlockCopyTask *task; + BlockCopyMethod method; int64_t max_chunk; =20 QEMU_LOCK_GUARD(&s->lock); @@ -201,6 +208,27 @@ block_copy_task_create(BlockCopyState *s, BlockCopyCal= lState *call_state, return NULL; } =20 + method =3D s->method; + + /* + * The scan already knows how this range reads: pick the method here a= nd + * stop the task where the answer changes. + */ + if (qatomic_load_acquire(&s->zero_bitmap_valid)) { + int64_t boundary; + + if (hbitmap_get(s->zero_bitmap, offset)) { + method =3D COPY_WRITE_ZEROES; + boundary =3D hbitmap_next_zero(s->zero_bitmap, offset, bytes); + } else { + boundary =3D hbitmap_next_dirty(s->zero_bitmap, offset, bytes); + } + + if (boundary >=3D 0) { + bytes =3D boundary - offset; + } + } + assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); bytes =3D QEMU_ALIGN_UP(bytes, s->cluster_size); =20 @@ -215,7 +243,7 @@ block_copy_task_create(BlockCopyState *s, BlockCopyCall= State *call_state, .task.func =3D block_copy_task_entry, .s =3D s, .call_state =3D call_state, - .method =3D s->method, + .method =3D method, }; reqlist_init_req(&s->reqs, &task->req, offset, bytes); =20 @@ -271,6 +299,9 @@ void block_copy_state_free(BlockCopyState *s) =20 ratelimit_destroy(&s->rate_limit); bdrv_release_dirty_bitmap(s->copy_bitmap); + if (s->zero_bitmap) { + hbitmap_free(s->zero_bitmap); + } shres_destroy(s->mem); g_free(s); } @@ -624,20 +655,24 @@ static coroutine_fn int block_copy_task_entry(AioTask= *task) return ret; } =20 +/* The scan and the per-task query must resolve BDRV_BLOCK_ZERO alike. */ +static GRAPH_RDLOCK BlockDriverState *block_copy_status_base(BlockCopyStat= e *s) +{ + if (qatomic_read(&s->skip_unallocated)) { + return bdrv_backing_chain_next(s->source->bs); + } + + return NULL; +} + static coroutine_fn GRAPH_RDLOCK int block_copy_block_status(BlockCopyState *s, int64_t offset, int64_t byt= es, int64_t *pnum) { int64_t num; - BlockDriverState *base; + BlockDriverState *base =3D block_copy_status_base(s); int ret; =20 - if (qatomic_read(&s->skip_unallocated)) { - base =3D bdrv_backing_chain_next(s->source->bs); - } else { - base =3D NULL; - } - ret =3D bdrv_co_block_status_above(s->source->bs, base, offset, bytes,= &num, NULL, NULL); if (ret < 0 || num < s->cluster_size) { @@ -657,16 +692,41 @@ int block_copy_block_status(BlockCopyState *s, int64_= t offset, int64_t bytes, return ret; } =20 +/* Only the scan allocates, and it runs before zero_bitmap_valid. */ +static HBitmap *block_copy_zero_bitmap(BlockCopyState *s) +{ + if (!s->zero_bitmap) { + s->zero_bitmap =3D hbitmap_alloc(s->len, ctz32(s->cluster_size)); + } + + return s->zero_bitmap; +} + +static void block_copy_mark_zero_prefix(BlockCopyState *s, int64_t offset, + int64_t zero_count) +{ + int64_t zero_bytes =3D QEMU_ALIGN_DOWN(zero_count, s->cluster_size); + + if (zero_bytes > 0) { + hbitmap_set(block_copy_zero_bitmap(s), offset, zero_bytes); + } +} + /* * Check if the cluster starting at offset is allocated or not. * return via pnum the number of contiguous clusters sharing this allocati= on. + * Also marks the zero prefix of the range in zero_bitmap. */ static int coroutine_fn GRAPH_RDLOCK block_copy_is_cluster_allocated(BlockCopyState *s, int64_t offset, int64_t *pnum) { BlockDriverState *bs =3D s->source->bs; + BlockDriverState *base =3D block_copy_status_base(s); + int64_t orig_offset =3D offset; int64_t count, total_count =3D 0; + int64_t zero_count =3D 0; + bool zero_broken =3D false; int64_t bytes =3D s->len - offset; int ret; =20 @@ -674,25 +734,37 @@ block_copy_is_cluster_allocated(BlockCopyState *s, in= t64_t offset, =20 while (true) { /* protected in backup_run() */ - ret =3D bdrv_co_is_allocated(bs, offset, bytes, &count); + ret =3D bdrv_co_block_status_above(bs, base, offset, bytes, &count, + NULL, NULL); if (ret < 0) { return ret; } =20 + if (!zero_broken) { + if (ret & BDRV_BLOCK_ZERO) { + zero_count +=3D count; + } else { + zero_broken =3D true; + } + } + total_count +=3D count; =20 - if (ret || count =3D=3D 0) { + if ((ret & BDRV_BLOCK_ALLOCATED) || count =3D=3D 0) { /* - * ret: partial segment(s) are considered allocated. + * BDRV_BLOCK_ALLOCATED: partial segment(s) are considered + * allocated. * otherwise: unallocated tail is treated as an entire segment. */ *pnum =3D DIV_ROUND_UP(total_count, s->cluster_size); - return ret; + block_copy_mark_zero_prefix(s, orig_offset, zero_count); + return !!(ret & BDRV_BLOCK_ALLOCATED); } =20 /* Unallocated segment(s) with uncertain following segment(s) */ if (total_count >=3D s->cluster_size) { *pnum =3D total_count / s->cluster_size; + block_copy_mark_zero_prefix(s, orig_offset, zero_count); return 0; } =20 @@ -752,6 +824,12 @@ block_copy_set_task_method(BlockCopyState *s, BlockCop= yTask *task) int ret; int64_t status_bytes; =20 + /* block_copy_task_create() already decided, from zero_bitmap. */ + if (qatomic_load_acquire(&s->zero_bitmap_valid)) { + return true; + } + + /* CBW filter could call this early. */ ret =3D block_copy_block_status(s, task->req.offset, task->req.bytes, &status_bytes); assert(ret >=3D 0); /* never fail */ @@ -1081,6 +1159,12 @@ void block_copy_set_skip_unallocated(BlockCopyState = *s, bool skip) qatomic_set(&s->skip_unallocated, skip); } =20 +void block_copy_set_zero_bitmap_valid(BlockCopyState *s) +{ + block_copy_zero_bitmap(s); + qatomic_store_release(&s->zero_bitmap_valid, true); +} + void block_copy_set_speed(BlockCopyState *s, uint64_t speed) { ratelimit_set_speed(&s->rate_limit, speed, BLOCK_COPY_SLICE_TIME); diff --git a/include/block/block-copy.h b/include/block/block-copy.h index 0df2771181..e4e5b56753 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -101,5 +101,6 @@ void block_copy_call_cancel(BlockCopyCallState *call_st= ate); BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s); int64_t block_copy_cluster_size(BlockCopyState *s); void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip); +void block_copy_set_zero_bitmap_valid(BlockCopyState *s); =20 #endif /* BLOCK_COPY_H */ --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397731; cv=none; d=zohomail.com; s=zohoarc; b=MUMZpCM8L0wsde8jxP4L5j3bgU/mhWByok61eliSlc6ye8HO7KPbWpjK/OKN6HZoAHmmuKi4dzzQblgfFTro8MuUqZx1+X7Sq+TXKZCYDqTzqHR27ogyoDttx5rb8sN3LveHNgBuSTD0MNc2wi/sTPdZEgSE6yluTCj5FyMmxX4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397731; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=thMNzpR9hfmP2UrqepCGOnSZLkYfUI9v4LXa+JCQASs=; b=ZeqtA+lmMTydoLzA79YBB7RS3B5MteEg2ejdo4+AgLbklBSITpaM2tIO/mLwB2x3NhFhB4my7cZ0Rtvop9AsJ+3yBnwWSWI4ZgMSkXjuCwn9aiK7+wa5LwjrEront6y8CDZyqMKr+rWbBSUzKyltmdGXU/OK7Ndc/RI13t4jxGw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397731939952.5797179735736; Mon, 14 Sep 2026 07:55:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683l-0002pr-DB; Mon, 14 Sep 2026 10:53:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683S-0002eT-Mk for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:12 -0400 Received: from mail-wr2-x10.google.com ([2a00:1450:4864:30::10]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683O-0005o1-QD for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:09 -0400 Received: by mail-wr2-x10.google.com with SMTP id ffacd0b85a97d-485b1d2874aso1710342f8f.1 for ; Mon, 14 Sep 2026 07:53:06 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.53.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:53:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397585; x=1790002385; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=thMNzpR9hfmP2UrqepCGOnSZLkYfUI9v4LXa+JCQASs=; b=AObpbK28oOD5B8ssrlK0J0SfmE4rSanzQE1jj+TXPw4124HnD+N0zsU8cuB/fPde74 5wBXlB4fqp+v8/zsLMGpbww4pAVLe9WFR4wnjLUfzwtgycEcQ77ESrwbm9gZVyAFd9zJ cEEx6vdWbFWAN4h7EFuBTR48MJ/tr3VyYYbjglygNiIL1nU4GWOggqSx9ELBokvu2c6f g9r2POu8P3bG7h3u4Z5Z6X6lkJj4+rlkgqAcgKjLQQkAx5jEIwI6BpkD/EhsESMyN8Ty G68mAib9eiE9GcUYH7idJeHOVEsAwkg7K9/TnHRw5sULvxS43TmabXan2USL/NquJDh/ Xmmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397585; x=1790002385; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=thMNzpR9hfmP2UrqepCGOnSZLkYfUI9v4LXa+JCQASs=; b=jN55RzE/h8CeN+arhhz88S++JZCC1doHOghMEYolfxHt6Amy4SvccTSNME27DfYYfC 7+W+PGaZvuW12pBUK0vVREN30uxu2M0gAjaOeIhW+t5aqXJgzQiGU3SBSnGlY3qWHT0e UqTOODNgpYj6Pt0SERDoA8QZAmuosHxMxnPl3/pDiHcEnJT2a5wMUWZ4EEV6kQxdFfer Fg8N6alTA8SCoeQ+495UrZHZc8haotioU2pJJMZVhRABZqTfvapuSrfE0jEwSJKFf9+p iRyBJ9vO6N/SCmukBg2/nGoOnOMoVod/ICpk8YINJEd1qusel6UijyndqhfvXWlgaSsu gHaQ== X-Gm-Message-State: AFuF++krvGDH6RUy2mP7TsVeyQpZSANyXe8mEea/2ayRPy0Bsms3W5aM fRXACxnADRi1LDVFqSxsDexUlGZRMK7suz/uM1NbYDdG99PsFlqNh65++rmIP2nd8UdHKoO3mAN NqWYa X-Gm-Gg: AYBFou3A7UONMfNUpbQMGuLga1xtiyXA7aD5cJRUin7zdh+yTbotX1ZQqUmmYv9GO8O /d0FEXMuobgBKmcI/Mrt3D+XgquGCZacnFa+QHjp43Rzm1HGlCnG2LQsDicrhvIwIUFu18Og2Tm zOiILUwPaWFaz7OFZ55ZV5i6I0zz+Vj/we19MeZqkKiWo+1cBV+JRGSroT+8f7Ze8N2eLrUAz4o /dazbx2IBHqc+m+UpRFLuCEqogZzfBqevdy0ebZgmmWAUfDt+YpIWrOXmPVwXJO0lxnfhyzFEcw ryvFHzs9tQYclIB0i1j4vWp1iqwYzrRwM/n1ImxcDPCXq9jaV6fkkhuHW5C3m/XOYJvBhDI1Nb4 ynUm2qGGMq7auE7HbwANc3q9e/BJmLb2kE4J6xyT6sZNnd4vWl8ug7XN3xiwDkcItA235s+E9wC Q1qMrszpCxashM4FTPxxbaDHU+UGUaPtoXHWp4U0fpUuUirrZCZpBl0wAry6VeUafi4CdM X-Received: by 2002:a05:600c:4703:b0:49c:f512:2361 with SMTP id 5b1f17b1804b1-49e7a67e2e1mr48499455e9.14.1789397584392; Mon, 14 Sep 2026 07:53:04 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Andrey Drobyshev , Vladimir Sementsov-Ogievskiy , John Snow Subject: [PATCH v2 8/9] block/backup: pre-fill zero_bitmap for full/bitmap Date: Mon, 14 Sep 2026 16:52:49 +0200 Message-ID: <20260914145250.2792879-9-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:30::10; envelope-from=den@openvz.org; helo=mail-wr2-x10.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397733995158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev Only sync=3Dtop runs an up-front scan, so the previous patch did nothing for the other modes. FULL and BITMAP know their complete copy_bitmap before copying starts, and the same copy-before-write guarantee holds: a cluster still dirty when the copy loop reaches it has not been written since the scan. Add block_copy_calculate_zero_bitmap(), driven one query at a time from backup_scan_source() so the graph rdlock and the pause point are taken per query, and query each confirmed span once however many dirty areas it covers. The scan only fills zero_bitmap and never clears copy_bitmap, so sync=3Dfull still writes zeroes for clusters the source never allocated; it just stops re-querying to find that out. Unlike the per-task queries it replaces the scan is serial, so a sparsely dirtied job against a slow source may spend more time here than it saves. The sync=3Dtop loop had the same shape, so both fold into backup_scan_source(). sync=3Dnone keeps the per-task query. Its bitmap says "anything may be copied", not "this will be copied", and it only ever copies what the guest writes during the fleecing window, so scanning the whole image at attach is the wrong trade. Reviewed-by: Andrey Drobyshev Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev --- block/backup.c | 94 ++++++++++++++++++++++++-------------- block/block-copy.c | 41 +++++++++++++++++ include/block/block-copy.h | 4 ++ 3 files changed, 105 insertions(+), 34 deletions(-) diff --git a/block/backup.c b/block/backup.c index 11d70243e2..38b0838d69 100644 --- a/block/backup.c +++ b/block/backup.c @@ -247,59 +247,85 @@ static void backup_init_bcs_bitmap(BackupBlockJob *jo= b) job_progress_set_remaining(&job->common.job, estimate); } =20 -static int coroutine_fn backup_run(Job *job, Error **errp) +/* + * Walk the source before copying starts: fill zero_bitmap and, for sync= =3Dtop, + * drop the clusters that are not allocated above the backing. + */ +static int coroutine_fn backup_scan_source(BackupBlockJob *s) { - BackupBlockJob *s =3D container_of(job, BackupBlockJob, common.job); - int ret; - - backup_init_bcs_bitmap(s); - - if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_TOP) { - int64_t offset =3D 0; - int64_t count; + Job *job =3D &s->common.job; + bool top =3D s->sync_mode =3D=3D MIRROR_SYNC_MODE_TOP; + int64_t offset, cached_end =3D 0, count; + int ret =3D 0; =20 - for (offset =3D 0; offset < s->len; ) { - if (job_is_cancelled(job)) { - return -ECANCELED; - } + for (offset =3D 0; offset < s->len; offset +=3D count) { + if (job_is_cancelled(job)) { + return -ECANCELED; + } =20 - job_pause_point(job); + job_pause_point(job); =20 - if (job_is_cancelled(job)) { - return -ECANCELED; - } + if (job_is_cancelled(job)) { + return -ECANCELED; + } =20 - /* rdlock protects the subsequent call to bdrv_is_allocated() = */ - bdrv_graph_co_rdlock(); + /* rdlock protects the block-status queries below */ + bdrv_graph_co_rdlock(); + if (top) { ret =3D block_copy_reset_unallocated(s->bcs, offset, &count); - bdrv_graph_co_rdunlock(); - if (ret < 0) { - return ret; - } + } else { + block_copy_calculate_zero_bitmap(s->bcs, offset, &cached_end, + &count); + } + bdrv_graph_co_rdunlock(); =20 - offset +=3D count; + if (ret < 0) { + return ret; } + } + + if (top) { block_copy_set_skip_unallocated(s->bcs, false); - block_copy_set_zero_bitmap_valid(s->bcs); } + block_copy_set_zero_bitmap_valid(s->bcs); =20 - if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_NONE) { + return 0; +} + +static int coroutine_fn backup_run(Job *job, Error **errp) +{ + BackupBlockJob *s =3D container_of(job, BackupBlockJob, common.job); + int ret; + + backup_init_bcs_bitmap(s); + + switch (s->sync_mode) { + case MIRROR_SYNC_MODE_TOP: + case MIRROR_SYNC_MODE_FULL: + case MIRROR_SYNC_MODE_BITMAP: + ret =3D backup_scan_source(s); + if (ret < 0) { + return ret; + } + break; + + case MIRROR_SYNC_MODE_NONE: /* * All bits are set in bcs bitmap to allow any cluster to be copie= d. - * This does not actually require them to be copied. + * This does not actually require them to be copied. Yield until t= he + * job is cancelled and let the before_write notify callback servi= ce + * CoW requests. */ while (!job_is_cancelled(job)) { - /* - * Yield until the job is cancelled. We just let our before_w= rite - * notify callback service CoW requests. - */ job_yield(job); } - } else { - return backup_loop(s); + return 0; + + default: + g_assert_not_reached(); } =20 - return 0; + return backup_loop(s); } =20 static void coroutine_fn backup_pause(Job *job) diff --git a/block/block-copy.c b/block/block-copy.c index 94d4f3dd69..c0a3359938 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -813,6 +813,47 @@ int64_t coroutine_fn block_copy_reset_unallocated(Bloc= kCopyState *s, return ret; } =20 +/* + * One scan step, like block_copy_reset_unallocated(); @cached_end + * tracks progress across calls, @count how far @offset should move. + */ +void coroutine_fn GRAPH_RDLOCK +block_copy_calculate_zero_bitmap(BlockCopyState *s, int64_t offset, + int64_t *cached_end, int64_t *count) +{ + int64_t dirty_offset, dirty_bytes, dirty_end; + bool found; + + /* CBW mutates copy_bitmap from its own AioContext meanwhile. */ + WITH_QEMU_LOCK_GUARD(&s->lock) { + found =3D bdrv_dirty_bitmap_next_dirty_area(s->copy_bitmap, offset, + s->len, INT64_MAX, + &dirty_offset, &dirty_by= tes); + } + + if (!found) { + *count =3D s->len - offset; + return; + } + dirty_end =3D dirty_offset + dirty_bytes; + + if (*cached_end < dirty_end) { + int64_t clusters; + int64_t query_offset =3D MAX(dirty_offset, *cached_end); + int ret =3D block_copy_is_cluster_allocated(s, query_offset, &clus= ters); + + if (ret >=3D 0) { + *cached_end =3D query_offset + clusters * s->cluster_size; + } else { + /* Best-effort: leave this area unmarked; it copies as data. */ + *cached_end =3D dirty_end; + } + } + + /* 0 if the query above didn't yet reach dirty_end: try again next cal= l. */ + *count =3D *cached_end >=3D dirty_end ? dirty_end - offset : 0; +} + /* * Decide how @task is copied: COPY_WRITE_ZEROES if it reads as zero. May * shrink @task. Returns false if @task is to be skipped (already ended, diff --git a/include/block/block-copy.h b/include/block/block-copy.h index e4e5b56753..3a61066b55 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -44,6 +44,10 @@ void block_copy_reset(BlockCopyState *s, int64_t offset,= int64_t bytes); int64_t coroutine_fn GRAPH_RDLOCK block_copy_reset_unallocated(BlockCopyState *s, int64_t offset, int64_t *c= ount); =20 +void coroutine_fn GRAPH_RDLOCK +block_copy_calculate_zero_bitmap(BlockCopyState *s, int64_t offset, + int64_t *cached_end, int64_t *count); + int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t byt= es, bool ignore_ratelimit, uint64_t timeout_ns, BlockCopyAsyncCallbackFunc cb, --=20 2.53.0 From nobody Sat Sep 26 20:51:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=openvz.org ARC-Seal: i=1; a=rsa-sha256; t=1789397721; cv=none; d=zohomail.com; s=zohoarc; b=V7MHbFzSVjpb31FmdYXFU1y5UbyNFKmVRumAf3CWZNCkz9Yzfo3tUGfyytsXVxrBqTNBxxoiKm00+yuhijNkCPZcoRb87Yftlo68ZnpSNDJwXZI3d34akSGwFDyU0C12whZRVHvMe5O3wmH+WN3e4EE1IXDWqTqguBEbI+xjibs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789397721; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=liBSdTP5ff8LcPYB2vyoXB7PN69e2uBoRDzt5wyNAtk=; b=iH16PKKsk1as6n8FlwNI0Mt2NDlQV5xywdqeorcF3MJwINJslb1AErLW+GVmZr28SAjU7MKE1Jz+4Ar2Fd1o2JHmqK6stP+ZMAcRQHVxDknTldDi1gIF9QkDZDniAvwVBYW4HykMBI3hHhoa4U7CGhylQq9O4fIyxlVigyGmjbw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1789397721419734.6666838512906; Mon, 14 Sep 2026 07:55:21 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x683l-0002qH-SO; Mon, 14 Sep 2026 10:53:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x683V-0002fA-J2 for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:20 -0400 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1x683R-0005oB-Ld for qemu-devel@nongnu.org; Mon, 14 Sep 2026 10:53:12 -0400 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-49e6bad7b79so19388075e9.1 for ; Mon, 14 Sep 2026 07:53:07 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:aec6:5a37:bc54:89b7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e63c9a7c9sm540230595e9.13.2026.09.14.07.53.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 07:53:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789397586; x=1790002386; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=liBSdTP5ff8LcPYB2vyoXB7PN69e2uBoRDzt5wyNAtk=; b=haSwEJgXjfAkYIfwQhjN685G7Cvo53rxJ8p+0YalHD3cDSc5840+kA2I9LJWzsgbqM amZnffnsJae+QfMDRjYUkGG6QkXSI0WQ1Zr4bL3dyRFErkCPa8nnpowspr5VCbKOoLRl KtSFfQa2y2ByFrfjvW3oBJC7T/3HSjqVFZy7HSvQI3hOmQqzGEcxBYLJk4/KUiH2OCZV cBQSaCF8aIRO3eTsAqFPR98Llu1dtaAWxbw1el/7/ECX2/ZJKINh9c+7JjtR1IoSbFDL RRBbjQIM/nakI+VmXYMFklYueGa7BjwcXyvo6QU8NpKhHvncw87fKaqpAyDPiBZ0e9I5 WLNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789397586; x=1790002386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=liBSdTP5ff8LcPYB2vyoXB7PN69e2uBoRDzt5wyNAtk=; b=TXhpFiz0tRpxU7QEoJ0VWCvfQJ3dbadBeCO7SAm2EzYBQQaOh2nUPxn/OSOVsIpNVu iv4QNhM2QqfFPT0bYj7ja0QDUc5fwi4MID2o0ySDv9U9d+x1c2DVGrqdxUngKHMt/How AwOW8FwB2WbeBYnnL27MLwwK0yM7QvXgEQkJmgk4vLzpwb1ziL6QXmS0bmiB95kHVq8/ r+fOXA4dRpT91Sznhg7JsVMbGo8fi1f6JdOyGF3CnRcSR5PMIeEEjglGpUJRzirpHsKQ GslO19dQtYYJSl0dEUDizlLpvXEhMbYEWVomyUkF0QpE+Bqu48rp9ZxTI96CVQ9XZ2o6 Typw== X-Gm-Message-State: AFuF++ka2VUGfVE87RVGqnh9g35KZZzQ5EzlqV2EcLdacFnSM0qB02ra sdwE7ghJBNopFkcnSuo1C+fwiD5nuaR1S9DsrT1otrl20azL20Cq9EAVMI/g4zaAqozOLrLx0cL deiBb X-Gm-Gg: AYBFou3Fiy6BxGLKSLMDoJ3iI+mY23iOSVAV2US8FD1tht/87246C/AFh26xv68A137 yHi2ctz+5EqF/KTDFcWEvphV2mktpRMDZaU+VGzVRJFrRjy4qqJxa3nPtWIAwRSrpyVrR4LvQZK p5Enb+gIHI2ZAS+/i9dHbV4nOzlTb5iMLptMIhQEcDamFhnskAZaQCM7hcGGkVWMghOOMxQmhW+ nZDkAa/nqt4JCfSqf+mGruZy+gKOTNqDS7xGKZpQypWOAbiwfn0CgHbbbFQauPbQ00z2AZNLy5W qb/tCjZkfwVAu9Tj8V8PaCvruuFlj370GrF5qPDSlQwIWx+hFkL6ryaoOU+qysd4gk6GdJDdQnY HVGXQ9r6FCjiZmjawd7yPeoExfPpEEziiVpuc0ojETsWBGNHjnFwFtAXQTEjapJD/42TN2Xzs0U jwccI9va8b+lw4eG8T9YiQy2xBbRQOawBaG5QFQAlguwwTELZQ3H1JeHmO39kcaA/SZO1i4ODGf YZ1MdI= X-Received: by 2002:a05:600c:190d:b0:49e:7a00:b9a5 with SMTP id 5b1f17b1804b1-49e7a63a894mr37881775e9.4.1789397586155; Mon, 14 Sep 2026 07:53:06 -0700 (PDT) From: "Denis V. Lunev" To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, "Denis V. Lunev" , Vladimir Sementsov-Ogievskiy , John Snow , Andrey Drobyshev Subject: [PATCH v2 9/9] block/block-copy: coalesce write-zeroes tasks Date: Mon, 14 Sep 2026 16:52:50 +0200 Message-ID: <20260914145250.2792879-10-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914145250.2792879-1-den@openvz.org> References: <20260914145250.2792879-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::333; envelope-from=den@openvz.org; helo=mail-wm1-x333.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @openvz.org) X-ZM-MESSAGEID: 1789397722022158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev Task size was capped to block_copy_chunk_size() regardless of method, which sizes a task by the copy buffer. A write-zeroes task carries no buffer, so that cap only splits one long known-zero run into a crowd of small COPY_WRITE_ZEROES tasks, each with its own request against the target. block_copy_task_create() already decides from zero_bitmap how far a task may run, so add block_copy_widen_zero_area() to that decision: for a write-zeroes task, re-search the dirty area under BDRV_REQUEST_MAX_BYTES instead of the buffer chunk size, and let the existing clamp cut it back to where the zero run ends. That bound is INT_MAX rounded down to a sector, so it is aligned down at the use site, nothing bounding cluster_size. Widening is safe: an overlapping caller waits on the task's BlockReq via reqlist_wait_one() rather than observing it mid-flight. A request that large is only cheap where the target zeroes by metadata. BDRV_REQ_NO_FALLBACK in supported_zero_flags rules out the targets which cannot, qcow2 v2 and iscsi among them, but it is optimistic for the rest: file-posix advertises it at open and only learns from a failing fallocate. So the first widened request also asks for it, which costs nothing when the target obliges and fails without writing when it does not. The run then keeps every write-zeroes request at the buffer chunk size, and the range that found out is written in those chunks. Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Andrey Drobyshev --- block/block-copy.c | 99 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 92 insertions(+), 7 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index c0a3359938..16f3d04819 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -38,6 +38,12 @@ #define BLOCK_COPY_SLICE_TIME 100000000ULL /* ns */ #define BLOCK_COPY_CLUSTER_SIZE_DEFAULT (1 << 16) =20 +typedef enum { + ZERO_WIDEN_UNPROBED, + ZERO_WIDEN_ON, + ZERO_WIDEN_OFF, +} BlockCopyZeroWiden; + typedef enum { COPY_READ_WRITE_CLUSTER, COPY_READ_WRITE, @@ -159,6 +165,8 @@ typedef struct BlockCopyState { bool skip_unallocated; /* atomic */ /* State fields that use a thread-safe API */ BdrvDirtyBitmap *copy_bitmap; + /* Whether the target zeroes by metadata; see block_copy_write_zeroes(= ). */ + BlockCopyZeroWiden zero_widen; /* atomic */ /* Clusters reading as zero; allocated on demand, frozen once valid. */ HBitmap *zero_bitmap; /* Published only after the scan, with skip_unallocated already false.= */ @@ -187,6 +195,34 @@ static int64_t block_copy_chunk_size(BlockCopyState *s) } } =20 +/* + * A write-zeroes task carries no buffer, so it may cover far more than + * block_copy_chunk_size(). Return how far it may run; the caller clamps it + * to where the zero run ends. + */ +static int64_t block_copy_widen_zero_area(BlockCopyState *s, + BlockCopyCallState *call_state, + int64_t offset, int64_t search_e= nd, + int64_t bytes) +{ + int64_t aligned =3D QEMU_ALIGN_DOWN(BDRV_REQUEST_MAX_BYTES, + s->cluster_size); + int64_t zero_chunk =3D MIN_NON_ZERO(MAX(aligned, s->cluster_size), + call_state->max_chunk); + int64_t wide_offset, wide_bytes; + + if (!bdrv_dirty_bitmap_next_dirty_area(s->copy_bitmap, offset, search_= end, + zero_chunk, &wide_offset, + &wide_bytes)) { + return bytes; + } + + /* @offset is dirty, so the search cannot have moved past it. */ + assert(wide_offset =3D=3D offset); + + return wide_bytes; +} + /* * Search for the first dirty area in offset/bytes range and create task at * the beginning of it. @@ -198,6 +234,7 @@ block_copy_task_create(BlockCopyState *s, BlockCopyCall= State *call_state, BlockCopyTask *task; BlockCopyMethod method; int64_t max_chunk; + int64_t search_end =3D offset + bytes; =20 QEMU_LOCK_GUARD(&s->lock); max_chunk =3D MIN_NON_ZERO(block_copy_chunk_size(s), call_state->max_c= hunk); @@ -219,6 +256,10 @@ block_copy_task_create(BlockCopyState *s, BlockCopyCal= lState *call_state, =20 if (hbitmap_get(s->zero_bitmap, offset)) { method =3D COPY_WRITE_ZEROES; + if (qatomic_read(&s->zero_widen) !=3D ZERO_WIDEN_OFF) { + bytes =3D block_copy_widen_zero_area(s, call_state, offset, + search_end, bytes); + } boundary =3D hbitmap_next_zero(s->zero_bitmap, offset, bytes); } else { boundary =3D hbitmap_next_dirty(s->zero_bitmap, offset, bytes); @@ -465,6 +506,8 @@ BlockCopyState *block_copy_state_new(BdrvChild *source,= BdrvChild *target, .max_transfer =3D QEMU_ALIGN_DOWN( block_copy_max_transfer(source, target= ), cluster_size), + .zero_widen =3D target->bs->supported_zero_flags & BDRV_REQ_NO_FAL= LBACK ? + ZERO_WIDEN_UNPROBED : ZERO_WIDEN_OFF, }; =20 s->discard_source =3D discard_source; @@ -521,6 +564,54 @@ static coroutine_fn int block_copy_task_run(AioTaskPoo= l *pool, return 0; } =20 +/* + * Widening a write-zeroes request only pays off where the target zeroes by + * metadata. Ask the first one to fail instead of falling back to writing = the + * zeroes out: it then costs nothing, and a target which would have written + * them keeps its requests small for the rest of the run. + */ +static int coroutine_fn GRAPH_RDLOCK +block_copy_write_zeroes(BlockCopyState *s, int64_t offset, int64_t bytes, + bool *error_is_read) +{ + BdrvRequestFlags flags =3D s->write_flags & ~BDRV_REQ_WRITE_COMPRESSED; + int64_t chunk =3D bytes; + int ret =3D 0; + + if (qatomic_read(&s->zero_widen) =3D=3D ZERO_WIDEN_UNPROBED) { + ret =3D bdrv_co_pwrite_zeroes(s->target, offset, bytes, + flags | BDRV_REQ_NO_FALLBACK); + if (ret !=3D -ENOTSUP) { + qatomic_set(&s->zero_widen, ZERO_WIDEN_ON); + goto out; + } + + /* Nothing has been written, so the whole range is still to do. */ + qatomic_set(&s->zero_widen, ZERO_WIDEN_OFF); + chunk =3D block_copy_chunk_size(s); + } + + while (bytes) { + int64_t n =3D MIN(bytes, chunk); + + ret =3D bdrv_co_pwrite_zeroes(s->target, offset, n, flags); + if (ret < 0) { + break; + } + + offset +=3D n; + bytes -=3D n; + } + +out: + if (ret < 0) { + trace_block_copy_write_zeroes_fail(s, offset, ret); + *error_is_read =3D false; + } + + return ret; +} + /* * block_copy_do_copy * @@ -552,13 +643,7 @@ block_copy_do_copy(BlockCopyState *s, int64_t offset, = int64_t bytes, =20 switch (*method) { case COPY_WRITE_ZEROES: - ret =3D bdrv_co_pwrite_zeroes(s->target, offset, nbytes, s->write_= flags & - ~BDRV_REQ_WRITE_COMPRESSED); - if (ret < 0) { - trace_block_copy_write_zeroes_fail(s, offset, ret); - *error_is_read =3D false; - } - return ret; + return block_copy_write_zeroes(s, offset, nbytes, error_is_read); =20 case COPY_RANGE_SMALL: case COPY_RANGE_FULL: --=20 2.53.0