From nobody Mon May 6 07:22:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15380170015591014.3950768460704; Wed, 26 Sep 2018 19:56:41 -0700 (PDT) Received: from localhost ([::1]:33503 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g5MTk-0007ch-TV for importer@patchew.org; Wed, 26 Sep 2018 22:56:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58104) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g5MRq-0006WS-CH for qemu-devel@nongnu.org; Wed, 26 Sep 2018 22:54:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g5MRp-00085F-DN for qemu-devel@nongnu.org; Wed, 26 Sep 2018 22:54:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46842) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g5MRl-000848-L9; Wed, 26 Sep 2018 22:54:33 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 02483307D913; Thu, 27 Sep 2018 02:54:33 +0000 (UTC) Received: from red.redhat.com (ovpn-126-36.rdu2.redhat.com [10.10.126.36]) by smtp.corp.redhat.com (Postfix) with ESMTP id D3FED5C541; Thu, 27 Sep 2018 02:54:31 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Wed, 26 Sep 2018 21:54:25 -0500 Message-Id: <20180927025427.101853-2-eblake@redhat.com> In-Reply-To: <20180927025427.101853-1-eblake@redhat.com> References: <20180927025427.101853-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.48]); Thu, 27 Sep 2018 02:54:33 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 1/3] nbd/server: fix bitmap export X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Vladimir Sementsov-Ogievskiy , qemu-stable@nongnu.org, "open list:Network Block Dev..." Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy bitmap_to_extents function is broken: it switches dirty variable after every iteration, however it can process only part of dirty (or zero) area during one iteration in case when this area is too large for one extent. Fortunately, the bug doesn't produce wrong extent flags: it just inserts a zero-length extent between sequential extents representing large dirty (or zero) area. However, zero-length extents are forbidden by the NBD protocol. So, a careful client should consider such a reply as a server fault, while a less-careful will likely ignore zero-length extents. The bug can only be triggered by a client that requests block status for nearly 4G at once (a request of 4G and larger is impossible per the protocol, and requests smaller than 4G less the bitmap granularity cause the loop to quit iterating rather than revisit the tail of the large area); it also cannot trigger if the client used the NBD_CMD_FLAG_REQ_ONE flag. Since qemu 3.0 as client (using the x-dirty-bitmap extension) always passes the flag, it is immune; and we are not aware of other open-source clients that know how to request qemu:dirty-bitmap:FOO contexts. Clients that want to avoid the bug could cap block status requests to a smaller length, such as 2G or 3G. Fix this by more careful handling of dirty variable. Bug was introduced in 3d068aff16 "nbd/server: implement dirty bitmap export", with the whole function. and is present in v3.0.0 release. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20180914165116.23182-1-vsementsov@virtuozzo.com> CC: qemu-stable@nongnu.org Reviewed-by: Eric Blake [eblake: improved commit message] Signed-off-by: Eric Blake --- nbd/server.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/nbd/server.c b/nbd/server.c index ea5fe0eb336..12f721482dd 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -1951,6 +1951,8 @@ static unsigned int bitmap_to_extents(BdrvDirtyBitmap= *bitmap, uint64_t offset, assert(begin < overall_end && nb_extents); while (begin < overall_end && i < nb_extents) { + bool next_dirty =3D !dirty; + if (dirty) { end =3D bdrv_dirty_bitmap_next_zero(bitmap, begin); } else { @@ -1962,6 +1964,7 @@ static unsigned int bitmap_to_extents(BdrvDirtyBitmap= *bitmap, uint64_t offset, end =3D MIN(bdrv_dirty_bitmap_size(bitmap), begin + UINT32_MAX + 1 - bdrv_dirty_bitmap_granularity(bitmap)); + next_dirty =3D dirty; } if (dont_fragment && end > overall_end) { end =3D overall_end; @@ -1971,7 +1974,7 @@ static unsigned int bitmap_to_extents(BdrvDirtyBitmap= *bitmap, uint64_t offset, extents[i].flags =3D cpu_to_be32(dirty ? NBD_STATE_DIRTY : 0); i++; begin =3D end; - dirty =3D !dirty; + dirty =3D next_dirty; } bdrv_dirty_iter_free(it); --=20 2.17.1 From nobody Mon May 6 07:22:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1538017018881728.3960705261353; Wed, 26 Sep 2018 19:56:58 -0700 (PDT) Received: from localhost ([::1]:33504 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g5MU5-0007q4-Pr for importer@patchew.org; Wed, 26 Sep 2018 22:56:57 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58148) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g5MRx-0006a7-Mm for qemu-devel@nongnu.org; Wed, 26 Sep 2018 22:54:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g5MRw-00087j-Cd for qemu-devel@nongnu.org; Wed, 26 Sep 2018 22:54:45 -0400 Received: from mx1.redhat.com ([209.132.183.28]:48840) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g5MRw-000872-7D for qemu-devel@nongnu.org; Wed, 26 Sep 2018 22:54:44 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 80778308AA0E for ; Thu, 27 Sep 2018 02:54:43 +0000 (UTC) Received: from red.redhat.com (ovpn-126-36.rdu2.redhat.com [10.10.126.36]) by smtp.corp.redhat.com (Postfix) with ESMTP id 406EA6148C; Thu, 27 Sep 2018 02:54:33 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Wed, 26 Sep 2018 21:54:26 -0500 Message-Id: <20180927025427.101853-3-eblake@redhat.com> In-Reply-To: <20180927025427.101853-1-eblake@redhat.com> References: <20180927025427.101853-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Thu, 27 Sep 2018 02:54:43 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 2/3] qapi: bitmap-merge: document name change X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Markus Armbruster Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: John Snow We named these using underscores instead of the preferred dash, document this nearby so we cannot possibly forget to rectify this when we remove the 'x-' prefixes when the feature becomes stable. We do not implement the change ahead of time to avoid more work for libvirt to do in order to figure out how to use the beta version of the API needlessly. Reported-by: Eric Blake Signed-off-by: John Snow Message-Id: <20180919190934.16284-1-jsnow@redhat.com> Reviewed-by: Eric Blake [eblake: typo fix] Signed-off-by: Eric Blake --- qapi/block-core.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/qapi/block-core.json b/qapi/block-core.json index ac3b48ee54e..58ec9931c7f 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -1935,6 +1935,8 @@ ## # @x-block-dirty-bitmap-merge: # +# FIXME: Rename @src_name and @dst_name to src-name and dst-name. +# # Merge @src_name dirty bitmap to @dst_name dirty bitmap. @src_name dirty # bitmap is unchanged. On error, @dst_name is unchanged. # --=20 2.17.1 From nobody Mon May 6 07:22:11 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1538017117287693.3542673156705; Wed, 26 Sep 2018 19:58:37 -0700 (PDT) Received: from localhost ([::1]:33512 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g5MVg-0001ek-9F for importer@patchew.org; Wed, 26 Sep 2018 22:58:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58268) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g5MS6-0006fP-O5 for qemu-devel@nongnu.org; Wed, 26 Sep 2018 22:54:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g5MS3-0008AX-DE for qemu-devel@nongnu.org; Wed, 26 Sep 2018 22:54:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35972) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g5MRy-00088q-Oq; Wed, 26 Sep 2018 22:54:46 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 16FC53082128; Thu, 27 Sep 2018 02:54:46 +0000 (UTC) Received: from red.redhat.com (ovpn-126-36.rdu2.redhat.com [10.10.126.36]) by smtp.corp.redhat.com (Postfix) with ESMTP id BD7024BE; Thu, 27 Sep 2018 02:54:43 +0000 (UTC) From: Eric Blake To: qemu-devel@nongnu.org Date: Wed, 26 Sep 2018 21:54:27 -0500 Message-Id: <20180927025427.101853-4-eblake@redhat.com> In-Reply-To: <20180927025427.101853-1-eblake@redhat.com> References: <20180927025427.101853-1-eblake@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.42]); Thu, 27 Sep 2018 02:54:46 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 3/3] nbd/server: send more than one extent of base:allocation context X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Vladimir Sementsov-Ogievskiy , "open list:Network Block Dev..." Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RDMRC_1 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy This is necessary for efficient block-status export, for clients which support it. (qemu is not yet such a client, but could become one.) Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20180704112302.471456-3-vsementsov@virtuozzo.com> [eblake: grammar tweaks] Signed-off-by: Eric Blake --- nbd/server.c | 87 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 23 deletions(-) diff --git a/nbd/server.c b/nbd/server.c index 12f721482dd..c3dd402b45e 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -1844,37 +1844,68 @@ static int coroutine_fn nbd_co_send_sparse_read(NBD= Client *client, return ret; } -static int blockstatus_to_extent_be(BlockDriverState *bs, uint64_t offset, - uint64_t bytes, NBDExtent *extent) +/* + * Populate @extents from block status. Update @bytes to be the actual + * length encoded (which may be smaller than the original), and update + * @nb_extents to the number of extents used. + * + * Returns zero on success and -errno on bdrv_block_status_above failure. + */ +static int blockstatus_to_extents(BlockDriverState *bs, uint64_t offset, + uint64_t *bytes, NBDExtent *extents, + unsigned int *nb_extents) { - uint64_t remaining_bytes =3D bytes; + uint64_t remaining_bytes =3D *bytes; + NBDExtent *extent =3D extents, *extents_end =3D extents + *nb_extents; + bool first_extent =3D true; + assert(*nb_extents); while (remaining_bytes) { uint32_t flags; int64_t num; int ret =3D bdrv_block_status_above(bs, NULL, offset, remaining_by= tes, &num, NULL, NULL); + if (ret < 0) { return ret; } flags =3D (ret & BDRV_BLOCK_ALLOCATED ? 0 : NBD_STATE_HOLE) | (ret & BDRV_BLOCK_ZERO ? NBD_STATE_ZERO : 0); - - if (remaining_bytes =3D=3D bytes) { - extent->flags =3D flags; - } - - if (flags !=3D extent->flags) { - break; - } - offset +=3D num; remaining_bytes -=3D num; + + if (first_extent) { + extent->flags =3D flags; + extent->length =3D num; + first_extent =3D false; + continue; + } + + if (flags =3D=3D extent->flags) { + /* extend current extent */ + extent->length +=3D num; + } else { + if (extent + 1 =3D=3D extents_end) { + break; + } + + /* start new extent */ + extent++; + extent->flags =3D flags; + extent->length =3D num; + } + } + + extents_end =3D extent + 1; + + for (extent =3D extents; extent < extents_end; extent++) { + cpu_to_be32s(&extent->flags); + cpu_to_be32s(&extent->length); } - cpu_to_be32s(&extent->flags); - extent->length =3D cpu_to_be32(bytes - remaining_bytes); + *bytes -=3D remaining_bytes; + *nb_extents =3D extents_end - extents; return 0; } @@ -1910,21 +1941,29 @@ static int nbd_co_send_extents(NBDClient *client, u= int64_t handle, /* Get block status from the exported device and send it to the client */ static int nbd_co_send_block_status(NBDClient *client, uint64_t handle, BlockDriverState *bs, uint64_t offset, - uint32_t length, bool last, - uint32_t context_id, Error **errp) + uint32_t length, bool dont_fragment, + bool last, uint32_t context_id, + Error **errp) { int ret; - NBDExtent extent; + unsigned int nb_extents =3D dont_fragment ? 1 : NBD_MAX_BITMAP_EXTENTS; + NBDExtent *extents =3D g_new(NBDExtent, nb_extents); + uint64_t final_length =3D length; - ret =3D blockstatus_to_extent_be(bs, offset, length, &extent); + ret =3D blockstatus_to_extents(bs, offset, &final_length, extents, + &nb_extents); if (ret < 0) { + g_free(extents); return nbd_co_send_structured_error( client, handle, -ret, "can't get block status", errp); } - return nbd_co_send_extents(client, handle, &extent, 1, - be32_to_cpu(extent.length), last, - context_id, errp); + ret =3D nbd_co_send_extents(client, handle, extents, nb_extents, + final_length, last, context_id, errp); + + g_free(extents); + + return ret; } /* @@ -2231,10 +2270,12 @@ static coroutine_fn int nbd_handle_request(NBDClien= t *client, (client->export_meta.base_allocation || client->export_meta.bitmap)) { + bool dont_fragment =3D request->flags & NBD_CMD_FLAG_REQ_ONE; + if (client->export_meta.base_allocation) { ret =3D nbd_co_send_block_status(client, request->handle, blk_bs(exp->blk), request->= from, - request->len, + request->len, dont_fragment, !client->export_meta.bitmap, NBD_META_ID_BASE_ALLOCATION, errp); @@ -2247,7 +2288,7 @@ static coroutine_fn int nbd_handle_request(NBDClient = *client, ret =3D nbd_co_send_bitmap(client, request->handle, client->exp->export_bitmap, request->from, request->len, - request->flags & NBD_CMD_FLAG_REQ= _ONE, + dont_fragment, true, NBD_META_ID_DIRTY_BITMAP, e= rrp); if (ret < 0) { return ret; --=20 2.17.1