From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595948852; cv=none; d=zohomail.com; s=zohoarc; b=oIxoq6YG5wTXGO6dwLKbnnPBfTnyYvtzKodN6s1UZZ5QAyqtsKqGpPo43sRMK2OYqQyckMEMsKqQwgsMOpJm0Fw+w+y+jT+T6Rl3ttG8arnHCoAnQczSGabr4Xh9bRwCqtWcrN++hJQ40oRhUj3Xee1EherOdLUCoy/zY4Zr+r4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595948852; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=zy+mW3gkiHJc9gLQSi1r6UafpwHJFUTOjM3IIN10LYg=; b=DGWn81cQEzKlCa668jR93o5YjjTGaqdI7y6zyPMe5iHvMU9CLSBkQBFIQKj9Ah4dSo3+RL4NvOifQAfhHrxPOYirTey6w2RtLQgXjJ5Iso0A2ACBoCkU/oirlQ14x3yELMZWmpsDk2duxiNzLbTtH7KKtzJw6SL6GJNYuWL9cH0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595948852091905.1615521642591; Tue, 28 Jul 2020 08:07:32 -0700 (PDT) Received: from localhost ([::1]:52004 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RCU-0002KR-8a for importer@patchew.org; Tue, 28 Jul 2020 11:07:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39806) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9S-00065I-Fo for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:22 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:31508 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9N-0006s3-Tg for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:22 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-372-C1kIXqHrMKKXKNC5QS722g-1; Tue, 28 Jul 2020 11:04:12 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8AF7718A1DFB; Tue, 28 Jul 2020 15:04:11 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id D120A1992D; Tue, 28 Jul 2020 15:04:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zy+mW3gkiHJc9gLQSi1r6UafpwHJFUTOjM3IIN10LYg=; b=g9y7noM7nvT/hWuxG2JnQft+wGNoQd06WVOwWbgjh/NF+5QHiWLpd74rdsyzT0hVrTZqEm skYDSheS32xQhkcnt/Lm1PjrrdzkHQTtGmT2BUHczurXj3P5UrkyleVO6FrG9nbQhcCL7W 6KrcN2gi/FiV6iVDsi8mRxmbVnzrg5o= X-MC-Unique: C1kIXqHrMKKXKNC5QS722g-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 1/9] nbd: Fix large trim/zero requests Date: Tue, 28 Jul 2020 10:03:59 -0500 Message-Id: <20200728150408.291299-2-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:31:20 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Vladimir Sementsov-Ogievskiy , qemu-stable@nongnu.org, "open list:Network Block Dev..." Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Although qemu as NBD client limits requests to <2G, the NBD protocol allows clients to send requests almost all the way up to 4G. But because our block layer is not yet 64-bit clean, we accidentally wrap such requests into a negative size, and fail with EIO instead of performing the intended operation. The bug is visible in modern systems with something as simple as: $ qemu-img create -f qcow2 /tmp/image.img 5G $ sudo qemu-nbd --connect=3D/dev/nbd0 /tmp/image.img $ sudo blkdiscard /dev/nbd0 or with user-space only: $ truncate --size=3D3G file $ qemu-nbd -f raw file $ nbdsh -u nbd://localhost:10809 -c 'h.trim(3*1024*1024*1024,0)' Although both blk_co_pdiscard and blk_pwrite_zeroes currently return 0 on success, this is also a good time to fix our code to a more robust paradigm that treats all non-negative values as success. Alas, our iotests do not currently make it easy to add external dependencies on blkdiscard or nbdsh, so we have to rely on manual testing for now. This patch can be reverted when we later improve the overall block layer to be 64-bit clean, but for now, a minimal fix was deemed less risky prior to release. CC: qemu-stable@nongnu.org Fixes: 1f4d6d18ed Fixes: 1c6c4bb7f0 Fixes: https://github.com/systemd/systemd/issues/16242 Signed-off-by: Eric Blake Message-Id: <20200722212231.535072-1-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy [eblake: rework success tests to use >=3D0] --- nbd/server.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/nbd/server.c b/nbd/server.c index 4752a6c8bc07..c5d71cff1001 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -2378,8 +2378,17 @@ static coroutine_fn int nbd_handle_request(NBDClient= *client, if (request->flags & NBD_CMD_FLAG_FAST_ZERO) { flags |=3D BDRV_REQ_NO_FALLBACK; } - ret =3D blk_pwrite_zeroes(exp->blk, request->from + exp->dev_offse= t, - request->len, flags); + ret =3D 0; + /* FIXME simplify this when blk_pwrite_zeroes switches to 64-bit */ + while (ret >=3D 0 && request->len) { + int align =3D client->check_align ?: 1; + int len =3D MIN(request->len, QEMU_ALIGN_DOWN(BDRV_REQUEST_MAX= _BYTES, + align)); + ret =3D blk_pwrite_zeroes(exp->blk, request->from + exp->dev_o= ffset, + len, flags); + request->len -=3D len; + request->from +=3D len; + } return nbd_send_generic_reply(client, request->handle, ret, "writing to file failed", errp); @@ -2393,9 +2402,18 @@ static coroutine_fn int nbd_handle_request(NBDClient= *client, "flush failed", errp); case NBD_CMD_TRIM: - ret =3D blk_co_pdiscard(exp->blk, request->from + exp->dev_offset, - request->len); - if (ret =3D=3D 0 && request->flags & NBD_CMD_FLAG_FUA) { + ret =3D 0; + /* FIXME simplify this when blk_co_pdiscard switches to 64-bit */ + while (ret >=3D 0 && request->len) { + int align =3D client->check_align ?: 1; + int len =3D MIN(request->len, QEMU_ALIGN_DOWN(BDRV_REQUEST_MAX= _BYTES, + align)); + ret =3D blk_co_pdiscard(exp->blk, request->from + exp->dev_off= set, + len); + request->len -=3D len; + request->from +=3D len; + } + if (ret >=3D 0 && request->flags & NBD_CMD_FLAG_FUA) { ret =3D blk_co_flush(exp->blk); } return nbd_send_generic_reply(client, request->handle, ret, --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595948747; cv=none; d=zohomail.com; s=zohoarc; b=WBn7Le9xJoIQ7OuGSg6uQxOaDeBpPe+lN528nkD4JC9ZfLH7CgAwJ3+FUIlcFeapX9NtP84f2KdU80NihGcEwPhy0Tj0OhPYjLRNldzrwt+r6aCZlb2Y2oDIL+xD7wDM2eo/6s9nT+9plkd31FKdi0xkARfuHH6b413/H9c2p0Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595948747; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=V8k5wikXvPVmhblRldgoJMU2RI2i2mycBnjay7fXFw4=; b=gWpwCbGVN2uiJ9nUn9CiCFj3H5eDnexcehYvS6m9ET97Z5k9dmewXf3zvBe1ANHJ9edCu6ZYJsMzbFLM5o9XTwsfh30py7mqI3Hgw7G1hPDGu84JnrEyBkXgWAhQ+l90sHiROibJy3Mj88ymaMPbazBWOUJSNJ3DV65qzY6GZh8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595948747220767.4446989094389; Tue, 28 Jul 2020 08:05:47 -0700 (PDT) Received: from localhost ([::1]:44932 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RAn-0007cF-FQ for importer@patchew.org; Tue, 28 Jul 2020 11:05:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39764) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9R-00064C-40 for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:21 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:28048 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9N-0006s0-V6 for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:20 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-85-IKHxsBRGNXm6ZfUCvEiLfQ-1; Tue, 28 Jul 2020 11:04:14 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EBB3A10059B2; Tue, 28 Jul 2020 15:04:12 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id D69DD1992D; Tue, 28 Jul 2020 15:04:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V8k5wikXvPVmhblRldgoJMU2RI2i2mycBnjay7fXFw4=; b=d6SlI/AoTrLJCyqv7kUrQNwf//p85kwPQYbWWG7lucs9vL+1CYxIsHzxlhI6sqTUZPMvqF v4llxAhnN6NcOl38s1qfoBTQ0srXPrpTPcaeCBC0WN4IwCZWB7Koov78dPgFClCNRtOtxT YLOTKAL0xhtNRrNIrWtH7IZBJocQ7MY= X-MC-Unique: IKHxsBRGNXm6ZfUCvEiLfQ-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 2/9] block: nbd: Fix convert qcow2 compressed to nbd Date: Tue, 28 Jul 2020 10:04:00 -0500 Message-Id: <20200728150408.291299-3-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eblake@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:31:13 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , Nir Soffer , "open list:Network Block Dev..." , Max Reitz , Nir Soffer Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Nir Soffer When converting to qcow2 compressed format, the last step is a special zero length compressed write, ending in a call to bdrv_co_truncate(). This call always fails for the nbd driver since it does not implement bdrv_co_truncate(). For block devices, which have the same limits, the call succeeds since the file driver implements bdrv_co_truncate(). If the caller asked to truncate to the same or smaller size with exact=3Dfalse, the truncate succeeds. Implement the same logic for nbd. Example failing without this change: In one shell start qemu-nbd: $ truncate -s 1g test.tar $ qemu-nbd --socket=3D/tmp/nbd.sock --persistent --format=3Draw --offset 15= 36 test.tar In another shell convert an image to qcow2 compressed via NBD: $ echo "disk data" > disk.raw $ truncate -s 1g disk.raw $ qemu-img convert -f raw -O qcow2 -c disk1.raw nbd+unix:///?socket=3D/tmp/= nbd.sock; echo $? 1 qemu-img failed, but the conversion was successful: $ qemu-img info nbd+unix:///?socket=3D/tmp/nbd.sock image: nbd+unix://?socket=3D/tmp/nbd.sock file format: qcow2 virtual size: 1 GiB (1073741824 bytes) ... $ qemu-img check nbd+unix:///?socket=3D/tmp/nbd.sock No errors were found on the image. 1/16384 =3D 0.01% allocated, 100.00% fragmented, 100.00% compressed clusters Image end offset: 393216 $ qemu-img compare disk.raw nbd+unix:///?socket=3D/tmp/nbd.sock Images are identical. Fixes: https://bugzilla.redhat.com/1860627 Signed-off-by: Nir Soffer Message-Id: <20200727215846.395443-2-nsoffer@redhat.com> Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy [eblake: typo fixes] Signed-off-by: Eric Blake --- block/nbd.c | 30 ++++++++++++++++++++++++++++++ qemu-io-cmds.c | 2 +- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/block/nbd.c b/block/nbd.c index 65a4f56924ec..8934bcb47901 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -1966,6 +1966,33 @@ static void nbd_close(BlockDriverState *bs) nbd_clear_bdrvstate(s); } +/* + * NBD cannot truncate, but if the caller asks to truncate to the same siz= e, or + * to a smaller size with exact=3Dfalse, there is no reason to fail the + * operation. + * + * Preallocation mode is ignored since it does not seems useful to fail wh= en + * we never change anything. + */ +static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offs= et, + bool exact, PreallocMode prealloc, + BdrvRequestFlags flags, Error **er= rp) +{ + BDRVNBDState *s =3D bs->opaque; + + if (offset !=3D s->info.size && exact) { + error_setg(errp, "Cannot resize NBD nodes"); + return -ENOTSUP; + } + + if (offset > s->info.size) { + error_setg(errp, "Cannot grow NBD nodes"); + return -EINVAL; + } + + return 0; +} + static int64_t nbd_getlength(BlockDriverState *bs) { BDRVNBDState *s =3D bs->opaque; @@ -2045,6 +2072,7 @@ static BlockDriver bdrv_nbd =3D { .bdrv_co_flush_to_os =3D nbd_co_flush, .bdrv_co_pdiscard =3D nbd_client_co_pdiscard, .bdrv_refresh_limits =3D nbd_refresh_limits, + .bdrv_co_truncate =3D nbd_co_truncate, .bdrv_getlength =3D nbd_getlength, .bdrv_detach_aio_context =3D nbd_client_detach_aio_context, .bdrv_attach_aio_context =3D nbd_client_attach_aio_context, @@ -2072,6 +2100,7 @@ static BlockDriver bdrv_nbd_tcp =3D { .bdrv_co_flush_to_os =3D nbd_co_flush, .bdrv_co_pdiscard =3D nbd_client_co_pdiscard, .bdrv_refresh_limits =3D nbd_refresh_limits, + .bdrv_co_truncate =3D nbd_co_truncate, .bdrv_getlength =3D nbd_getlength, .bdrv_detach_aio_context =3D nbd_client_detach_aio_context, .bdrv_attach_aio_context =3D nbd_client_attach_aio_context, @@ -2099,6 +2128,7 @@ static BlockDriver bdrv_nbd_unix =3D { .bdrv_co_flush_to_os =3D nbd_co_flush, .bdrv_co_pdiscard =3D nbd_client_co_pdiscard, .bdrv_refresh_limits =3D nbd_refresh_limits, + .bdrv_co_truncate =3D nbd_co_truncate, .bdrv_getlength =3D nbd_getlength, .bdrv_detach_aio_context =3D nbd_client_detach_aio_context, .bdrv_attach_aio_context =3D nbd_client_attach_aio_context, diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c index 851f07e8f8b9..baeae86d8c85 100644 --- a/qemu-io-cmds.c +++ b/qemu-io-cmds.c @@ -1715,7 +1715,7 @@ static int truncate_f(BlockBackend *blk, int argc, ch= ar **argv) * exact=3Dtrue. It is better to err on the "emit more errors" side * than to be overly permissive. */ - ret =3D blk_truncate(blk, offset, true, PREALLOC_MODE_OFF, 0, &local_e= rr); + ret =3D blk_truncate(blk, offset, false, PREALLOC_MODE_OFF, 0, &local_= err); if (ret < 0) { error_report_err(local_err); return ret; --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595948870; cv=none; d=zohomail.com; s=zohoarc; b=Mx0+xzEQwpJdF0SrShisu0st1DMUZnuQdHhsBKzVoMQGNs2yxxBJsBaoQ0sEjeesoIYZW/17+U/3PUsLqK6yRfBY9Vp3AhBdF8PZFOOJrcWQljL8CHlzzuGSGbi9TFwD9EZM0SAzOayQEaSGj8oo1+uqhnFZgXlh8EAbuSM1Wpw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595948870; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=9gHPdIJ5NKGQI04PaXppH9DfzdHxKKOt0eEbsUplSgQ=; b=XCj+y6FWtUDNM893OsHUZg82hYBl2AM+yS0YQPIyEGmJJnQyVK/EaBuWFkvCZuUpa7K/ZIr+2lkgXkhaxxgLjEZ/prfbZU0JWpHiogG816Wwaxht/gGvjbKpKmuD+L8BtDx4V4Rix4p7EmvGTINWvxyN5fWfEMN8JyJUqHVVPoE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595948870580801.4397884138392; Tue, 28 Jul 2020 08:07:50 -0700 (PDT) Received: from localhost ([::1]:52648 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RCn-0002bG-B2 for importer@patchew.org; Tue, 28 Jul 2020 11:07:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39838) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9T-00067M-NP for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:23 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:56572 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9Q-0006se-Vy for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:23 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-249-X_HjwuNTMZyXpdAxRgMWTQ-1; Tue, 28 Jul 2020 11:04:15 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 54EC710059B6; Tue, 28 Jul 2020 15:04:14 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id 43E2219741; Tue, 28 Jul 2020 15:04:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948659; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9gHPdIJ5NKGQI04PaXppH9DfzdHxKKOt0eEbsUplSgQ=; b=AlNJ/+divCBdP6m6cv3pSDVdedbRZxANwncRtYG9dHGNiYA1igrLb82PzQtWTf7PezG/a8 WKpVPii5LBhCLy9k6h8i5Dl/b72Va2M2xc5v1tgh2B79QzlyfMO9YE1s04sg/1F41bXpcM pS2OW9HP90L7BzCSgU2DvxYV7NCkEVU= X-MC-Unique: X_HjwuNTMZyXpdAxRgMWTQ-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 3/9] iotests: Make qemu_nbd_popen() a contextmanager Date: Tue, 28 Jul 2020 10:04:01 -0500 Message-Id: <20200728150408.291299-4-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:28:27 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , Nir Soffer , "open list:Block layer core" , Max Reitz , Nir Soffer Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Nir Soffer Instead of duplicating the code to wait until the server is ready and remember to terminate the server and wait for it, make it possible to use like this: with qemu_nbd_popen('-k', sock, image): # Access image via qemu-nbd socket... Only test 264 used this helper, but I had to modify the output since it did not consistently when starting and stopping qemu-nbd. Signed-off-by: Nir Soffer Message-Id: <20200727215846.395443-3-nsoffer@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Eric Blake --- tests/qemu-iotests/264 | 72 +++++++++++++---------------------- tests/qemu-iotests/264.out | 2 + tests/qemu-iotests/iotests.py | 28 +++++++++++++- 3 files changed, 54 insertions(+), 48 deletions(-) diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264 index 304a7443d77f..666f164ed8f4 100755 --- a/tests/qemu-iotests/264 +++ b/tests/qemu-iotests/264 @@ -36,48 +36,32 @@ wait_step =3D 0.2 qemu_img_create('-f', iotests.imgfmt, disk_a, str(size)) qemu_img_create('-f', iotests.imgfmt, disk_b, str(size)) -srv =3D qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b) -# Wait for NBD server availability -t =3D 0 -ok =3D False -while t < wait_limit: - ok =3D qemu_io_silent_check('-f', 'raw', '-c', 'read 0 512', nbd_uri) - if ok: - break - time.sleep(wait_step) - t +=3D wait_step +with qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b): + vm =3D iotests.VM().add_drive(disk_a) + vm.launch() + vm.hmp_qemu_io('drive0', 'write 0 {}'.format(size)) -assert ok + vm.qmp_log('blockdev-add', filters=3D[iotests.filter_qmp_testfiles], + **{'node_name': 'backup0', + 'driver': 'raw', + 'file': {'driver': 'nbd', + 'server': {'type': 'unix', 'path': nbd_sock}, + 'reconnect-delay': 10}}) + vm.qmp_log('blockdev-backup', device=3D'drive0', sync=3D'full', target= =3D'backup0', + speed=3D(1 * 1024 * 1024)) -vm =3D iotests.VM().add_drive(disk_a) -vm.launch() -vm.hmp_qemu_io('drive0', 'write 0 {}'.format(size)) + # Wait for some progress + t =3D 0 + while t < wait_limit: + jobs =3D vm.qmp('query-block-jobs')['return'] + if jobs and jobs[0]['offset'] > 0: + break + time.sleep(wait_step) + t +=3D wait_step -vm.qmp_log('blockdev-add', filters=3D[iotests.filter_qmp_testfiles], - **{'node_name': 'backup0', - 'driver': 'raw', - 'file': {'driver': 'nbd', - 'server': {'type': 'unix', 'path': nbd_sock}, - 'reconnect-delay': 10}}) -vm.qmp_log('blockdev-backup', device=3D'drive0', sync=3D'full', target=3D'= backup0', - speed=3D(1 * 1024 * 1024)) - -# Wait for some progress -t =3D 0 -while t < wait_limit: - jobs =3D vm.qmp('query-block-jobs')['return'] if jobs and jobs[0]['offset'] > 0: - break - time.sleep(wait_step) - t +=3D wait_step - -if jobs and jobs[0]['offset'] > 0: - log('Backup job is started') - -log('Kill NBD server') -srv.kill() -srv.wait() + log('Backup job is started') jobs =3D vm.qmp('query-block-jobs')['return'] if jobs and jobs[0]['offset'] < jobs[0]['len']: @@ -88,12 +72,8 @@ vm.qmp_log('block-job-set-speed', device=3D'drive0', spe= ed=3D0) # Emulate server down time for 1 second time.sleep(1) -log('Start NBD server') -srv =3D qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b) - -e =3D vm.event_wait('BLOCK_JOB_COMPLETED') -log('Backup completed: {}'.format(e['data']['offset'])) - -vm.qmp_log('blockdev-del', node_name=3D'backup0') -srv.kill() -vm.shutdown() +with qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk_b): + e =3D vm.event_wait('BLOCK_JOB_COMPLETED') + log('Backup completed: {}'.format(e['data']['offset'])) + vm.qmp_log('blockdev-del', node_name=3D'backup0') + vm.shutdown() diff --git a/tests/qemu-iotests/264.out b/tests/qemu-iotests/264.out index 3000944b099a..c45b1e81ef26 100644 --- a/tests/qemu-iotests/264.out +++ b/tests/qemu-iotests/264.out @@ -1,3 +1,4 @@ +Start NBD server {"execute": "blockdev-add", "arguments": {"driver": "raw", "file": {"drive= r": "nbd", "reconnect-delay": 10, "server": {"path": "TEST_DIR/PID-nbd-sock= ", "type": "unix"}}, "node-name": "backup0"}} {"return": {}} {"execute": "blockdev-backup", "arguments": {"device": "drive0", "speed": = 1048576, "sync": "full", "target": "backup0"}} @@ -11,3 +12,4 @@ Start NBD server Backup completed: 5242880 {"execute": "blockdev-del", "arguments": {"node-name": "backup0"}} {"return": {}} +Kill NBD server diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 3590ed78a0e4..8f79668435a0 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -28,10 +28,13 @@ import signal import struct import subprocess import sys +import time from typing import (Any, Callable, Dict, Iterable, List, Optional, Sequence, Tuple, TypeVar) import unittest +from contextlib import contextmanager + # pylint: disable=3Dimport-error, wrong-import-position sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'pytho= n')) from qemu import qtest @@ -270,9 +273,30 @@ def qemu_nbd_early_pipe(*args): return subp.returncode, output if subp.returncode else '' +@contextmanager def qemu_nbd_popen(*args): - '''Run qemu-nbd in daemon mode and return the parent's exit code''' - return subprocess.Popen(qemu_nbd_args + ['--persistent'] + list(args)) + '''Context manager running qemu-nbd within the context''' + pid_file =3D file_path("pid") + + cmd =3D list(qemu_nbd_args) + cmd.extend(('--persistent', '--pid-file', pid_file)) + cmd.extend(args) + + log('Start NBD server') + p =3D subprocess.Popen(cmd) + try: + while not os.path.exists(pid_file): + if p.poll() is not None: + raise RuntimeError( + "qemu-nbd terminated with exit code {}: {}" + .format(p.returncode, ' '.join(cmd))) + + time.sleep(0.01) + yield + finally: + log('Kill NBD server') + p.kill() + p.wait() def compare_images(img1, img2, fmt1=3Dimgfmt, fmt2=3Dimgfmt): '''Return True if two image files are identical''' --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595949169; cv=none; d=zohomail.com; s=zohoarc; b=mcbzsc/VE9mGHjzOh6brDqrzms05+nVtsxfkJXffW5pLa2GbP4D5PstkOD8BtQb2tWcGlYoDMvfcvekTACQVE0xSHzo2ju3Eyr6GoBMB5eafBnnEkS/jaXVBokwqBzobeBYJCr+ZRJIjUvtXrsHZ5tMrQjgY4dEcho3AkaYS6Og= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595949169; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=kEro/WGEDa7FLpObt0GBhXJnpiF9wp9HuVFS+fPX2vU=; b=Dxb/OUbjZDjTSQddwwq4GvL6wYL/IAR2eVHfODEhltNWg9OXEUcv2mXQn/S9npTX2w8xNRQr+5vpcb9hBk036bMRkHkKO4JH3ehKKBClFZyYDl2qndqhv9hLBWYZ1gpe0XkT1JruqRGjLU31+AWlYIE35yEMZdI2i/18hf69r6o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595949169692979.8409670054969; Tue, 28 Jul 2020 08:12:49 -0700 (PDT) Received: from localhost ([::1]:44902 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RHc-00038S-6k for importer@patchew.org; Tue, 28 Jul 2020 11:12:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39990) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9Z-0006IU-30 for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:29 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:21203 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9V-0006uE-UF for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:28 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-472-arlqmFcUPm6xpcDegXiJqA-1; Tue, 28 Jul 2020 11:04:16 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A9346800460; Tue, 28 Jul 2020 15:04:15 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id 968FB19C71; Tue, 28 Jul 2020 15:04:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948665; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kEro/WGEDa7FLpObt0GBhXJnpiF9wp9HuVFS+fPX2vU=; b=AhDdQ+iDADqz+amOEYn11uLvXqczSer8KaZ4unc3jAmFX73lThE8kBHwpVL+PObFpUysYV 5PWMgnxg9buNNsTBc3KCriJ4Qx8lzZOJs/23cArpY9+onJqWtdzuDw4CKbgFWhAKrj0LVN hgXdcU5+a2JgtyV5jitY+UsD/KtkuBU= X-MC-Unique: arlqmFcUPm6xpcDegXiJqA-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 4/9] iotests: Add more qemu_img helpers Date: Tue, 28 Jul 2020 10:04:02 -0500 Message-Id: <20200728150408.291299-5-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 11:04:16 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , Nir Soffer , "open list:Block layer core" , Max Reitz , Nir Soffer Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Nir Soffer Add 2 helpers for measuring and checking images: - qemu_img_measure() - qemu_img_check() Both use --output-json and parse the returned json to make easy to use in other tests. I'm going to use them in a new test, and I hope they will be useful in may other tests. Signed-off-by: Nir Soffer Message-Id: <20200727215846.395443-4-nsoffer@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Eric Blake --- tests/qemu-iotests/iotests.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 8f79668435a0..717b5b652c45 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -141,6 +141,12 @@ def qemu_img_create(*args): return qemu_img(*args) +def qemu_img_measure(*args): + return json.loads(qemu_img_pipe("measure", "--output", "json", *args)) + +def qemu_img_check(*args): + return json.loads(qemu_img_pipe("check", "--output", "json", *args)) + def qemu_img_verbose(*args): '''Run qemu-img without suppressing its output and return the exit cod= e''' exitcode =3D subprocess.call(qemu_img_args + list(args)) --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595949032; cv=none; d=zohomail.com; s=zohoarc; b=LfunIjj68Mg56ek9w9YSn8T1UkLGemDZ8YfHYNncfwmVHnCBYgICB0RKip0EV+3rUFqiDMGMt+1pwxK96cRwH3iW5qtpz1UwTIICSAhOuDZ4eykzkU6L7cRC75RnbRuq59zWgr58jFHZkaGc7g5vi5SEGc0sAfsXTOVjn74qo6Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595949032; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ah1BNHGRzQCMR0vyXLGcIrWGVHRIu4rYiTqWfrIuzH4=; b=ScopbF83D4p5yKpF1bf61OE33OWa08ISfxQI694NnpYNOH6nX+8uL09zZadBjaSMb8BurBDdGLBvCsKG3MWvGBPtEAAu7+8ywj3cDxTfHwtTzhDNUqQa0gxXalh6STIGEO0kLA+sDooY0DVJNTKF29h9+9CWWcVsmrj+dhOLpMM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595949032178163.27454614787143; Tue, 28 Jul 2020 08:10:32 -0700 (PDT) Received: from localhost ([::1]:35620 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RFO-0007MD-Iu for importer@patchew.org; Tue, 28 Jul 2020 11:10:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39912) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9W-0006D0-C5 for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:26 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:49648 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9T-0006tb-TB for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:25 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-374-7NmkATP3NsOxTOiwYVeDKA-1; Tue, 28 Jul 2020 11:04:17 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id ED01F10059BB; Tue, 28 Jul 2020 15:04:16 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id 02DD819C71; Tue, 28 Jul 2020 15:04:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ah1BNHGRzQCMR0vyXLGcIrWGVHRIu4rYiTqWfrIuzH4=; b=KmFE7c2jWY4yYUg/4+PvxBcQ+HwmFgBVl5isJjtE0XidRQ758Z3HHBOY3ZmgKWey7qXIx7 0J/dUhpOFg7WKY2Ziri83zGTPHgWh217AN35v96LO+LU1sHHGHlh3rnrhGM7vFoCxb7lFX rZaOGfveGK6Fu3j3vTfo2TPWvNHyMuc= X-MC-Unique: 7NmkATP3NsOxTOiwYVeDKA-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 5/9] iotests: Test convert to qcow2 compressed to NBD Date: Tue, 28 Jul 2020 10:04:03 -0500 Message-Id: <20200728150408.291299-6-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:31:20 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nir Soffer , Kevin Wolf , "open list:Block layer core" , Nir Soffer , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: Nir Soffer Add test for "qemu-img convert -O qcow2 -c" to NBD target. The tests =C2=A0= =C2=A0 create a OVA file and write compressed qcow2 disk content directly into the OVA file via qemu-nbd. Signed-off-by: Nir Soffer Message-Id: <20200727215846.395443-5-nsoffer@redhat.com> Tested-by: Eric Blake Reviewed-by: Eric Blake Signed-off-by: Eric Blake --- tests/qemu-iotests/302 | 127 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/302.out | 31 +++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 159 insertions(+) create mode 100755 tests/qemu-iotests/302 create mode 100644 tests/qemu-iotests/302.out diff --git a/tests/qemu-iotests/302 b/tests/qemu-iotests/302 new file mode 100755 index 000000000000..a8506bda1592 --- /dev/null +++ b/tests/qemu-iotests/302 @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +# +# Tests converting qcow2 compressed to NBD +# +# Copyright (c) 2020 Nir Soffer +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +# owner=3Dnirsof@gmail.com + +import io +import tarfile + +import iotests + +from iotests import ( + file_path, + qemu_img, + qemu_img_check, + qemu_img_create, + qemu_img_log, + qemu_img_measure, + qemu_io, + qemu_nbd_popen, +) + +iotests.script_initialize(supported_fmts=3D["qcow2"]) + +# Create source disk. Using qcow2 to enable strict comparing later, and +# avoid issues with random filesystem on CI environment. +src_disk =3D file_path("disk.qcow2") +qemu_img_create("-f", iotests.imgfmt, src_disk, "1g") +qemu_io("-f", iotests.imgfmt, "-c", "write 1m 64k", src_disk) + +# The use case is writing qcow2 image directly into an ova file, which +# is a tar file with specific layout. This is tricky since we don't know t= he +# size of the image before compressing, so we have to do: +# 1. Add an ovf file. +# 2. Find the offset of the next member data. +# 3. Make room for image data, allocating for the worst case. +# 4. Write compressed image data into the tar. +# 5. Add a tar entry with the actual image size. +# 6. Shrink the tar to the actual size, aligned to 512 bytes. + +tar_file =3D file_path("test.ova") + +with tarfile.open(tar_file, "w") as tar: + + # 1. Add an ovf file. + + ovf_data =3D b"" + ovf =3D tarfile.TarInfo("vm.ovf") + ovf.size =3D len(ovf_data) + tar.addfile(ovf, io.BytesIO(ovf_data)) + + # 2. Find the offset of the next member data. + + offset =3D tar.fileobj.tell() + 512 + + # 3. Make room for image data, allocating for the worst case. + + measure =3D qemu_img_measure("-O", "qcow2", src_disk) + tar.fileobj.truncate(offset + measure["required"]) + + # 4. Write compressed image data into the tar. + + nbd_sock =3D file_path("nbd-sock", base_dir=3Diotests.sock_dir) + nbd_uri =3D "nbd+unix:///exp?socket=3D" + nbd_sock + + # Use raw format to allow creating qcow2 directly into tar file. + with qemu_nbd_popen( + "--socket", nbd_sock, + "--export-name", "exp", + "--format", "raw", + "--offset", str(offset), + tar_file): + + iotests.log("=3D=3D=3D Target image info =3D=3D=3D") + qemu_img_log("info", nbd_uri) + + qemu_img( + "convert", + "-f", iotests.imgfmt, + "-O", "qcow2", + "-c", + src_disk, + nbd_uri) + + iotests.log("=3D=3D=3D Converted image info =3D=3D=3D") + qemu_img_log("info", nbd_uri) + + iotests.log("=3D=3D=3D Converted image check =3D=3D=3D") + qemu_img_log("check", nbd_uri) + + iotests.log("=3D=3D=3D Comparing to source disk =3D=3D=3D") + qemu_img_log("compare", src_disk, nbd_uri) + + actual_size =3D qemu_img_check(nbd_uri)["image-end-offset"] + + # 5. Add a tar entry with the actual image size. + + disk =3D tarfile.TarInfo("disk") + disk.size =3D actual_size + tar.addfile(disk) + + # 6. Shrink the tar to the actual size, aligned to 512 bytes. + + tar_size =3D offset + (disk.size + 511) & ~511 + tar.fileobj.seek(tar_size) + tar.fileobj.truncate(tar_size) + +with tarfile.open(tar_file) as tar: + members =3D [{"name": m.name, "size": m.size, "offset": m.offset_data} + for m in tar] + iotests.log("=3D=3D=3D OVA file contents =3D=3D=3D") + iotests.log(members) diff --git a/tests/qemu-iotests/302.out b/tests/qemu-iotests/302.out new file mode 100644 index 000000000000..e37d3a103082 --- /dev/null +++ b/tests/qemu-iotests/302.out @@ -0,0 +1,31 @@ +Start NBD server +=3D=3D=3D Target image info =3D=3D=3D +image: nbd+unix:///exp?socket=3DSOCK_DIR/PID-nbd-sock +file format: raw +virtual size: 448 KiB (458752 bytes) +disk size: unavailable + +=3D=3D=3D Converted image info =3D=3D=3D +image: nbd+unix:///exp?socket=3DSOCK_DIR/PID-nbd-sock +file format: qcow2 +virtual size: 1 GiB (1073741824 bytes) +disk size: unavailable +cluster_size: 65536 +Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=3D=3D=3D Converted image check =3D=3D=3D +No errors were found on the image. +1/16384 =3D 0.01% allocated, 100.00% fragmented, 100.00% compressed cluste= rs +Image end offset: 393216 + +=3D=3D=3D Comparing to source disk =3D=3D=3D +Images are identical. + +Kill NBD server +=3D=3D=3D OVA file contents =3D=3D=3D +[{"name": "vm.ovf", "offset": 512, "size": 6}, {"name": "disk", "offset": = 1536, "size": 393216}] diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 1d0252e1f051..1e1cb27bc8e3 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -308,3 +308,4 @@ 297 meta 299 auto quick 301 backing quick +302 quick --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595948761; cv=none; d=zohomail.com; s=zohoarc; b=Cf14SVAfSK+i3ETudzinh+QDOHQYeEOWb2bV2YATGcp3gnTNrHdfdmh3rWft1WB8K0IofmJk2dUMxxr1d+Pq55yrHJGTI+/BEAgRXnN1U93d4qyjzN1zNXxIM6fmHtWHLQ7sz0xzhrCX/TBSwUyf56t5XrScOr8xd7QelLKOUuc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595948761; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=SsgrxZXzr054DCT3tacjg72ARLC7VEvLE+gSU5BZj1k=; b=a79D+hhiDix0tuXOcqwmnLb+5hekQw44stKsa8dF4bitYV5ZcTWM3LxvqbygJ/GahtfKnpd9jXrCtY4eSQIa78M50gztmM0MbzDkjWuEmuWIZxUX6aBxezBIMjFvR4CG/+pU9VPEKclzNlW3NYEsP0Zt0F9mq425Vh7Hwd0oBWw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595948761006342.6700875714273; Tue, 28 Jul 2020 08:06:01 -0700 (PDT) Received: from localhost ([::1]:46002 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RB1-0008Bn-Ck for importer@patchew.org; Tue, 28 Jul 2020 11:05:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39916) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9W-0006DE-DU for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:26 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:22703 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9U-0006tm-6C for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:26 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-478-HK2DxJgPNXOqlkZAYXep-w-1; Tue, 28 Jul 2020 11:04:19 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 19D1358; Tue, 28 Jul 2020 15:04:18 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4124319C71; Tue, 28 Jul 2020 15:04:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SsgrxZXzr054DCT3tacjg72ARLC7VEvLE+gSU5BZj1k=; b=NwrmcFVMQZuy4k3H2SbadIbNX6FGNydDvZY41GOIUaYY+ydqG2KLmGiO6JoX4R9OBofOMZ 0qPdkeW5qLdP77Cu2R0639tJGlNO6yzGCmKsJeSGtoxj56SJxywQkczUoWMVXsFrSv/NiK tfWXoGMphfttJCYtnh/hFYVHykgy0OU= X-MC-Unique: HK2DxJgPNXOqlkZAYXep-w-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 6/9] block/nbd: split nbd_establish_connection out of nbd_client_connect Date: Tue, 28 Jul 2020 10:04:04 -0500 Message-Id: <20200728150408.291299-7-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:31:20 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -40 X-Spam_score: -4.1 X-Spam_bar: ---- X-Spam_report: (-4.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Network Block Dev..." , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy We are going to implement non-blocking version of nbd_establish_connection, which for a while will be used only for nbd_reconnect_attempt, not for nbd_open, so we need to call it separately. Refactor nbd_reconnect_attempt in a way which makes next commit simpler. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200727184751.15704-2-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake Signed-off-by: Eric Blake --- block/nbd.c | 60 +++++++++++++++++++++++++++------------------- block/trace-events | 4 ++-- 2 files changed, 38 insertions(+), 26 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index 8934bcb47901..3558c173e34b 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -93,7 +93,10 @@ typedef struct BDRVNBDState { char *x_dirty_bitmap; } BDRVNBDState; -static int nbd_client_connect(BlockDriverState *bs, Error **errp); +static QIOChannelSocket *nbd_establish_connection(SocketAddress *saddr, + Error **errp); +static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *si= oc, + Error **errp); static void nbd_clear_bdrvstate(BDRVNBDState *s) { @@ -241,7 +244,9 @@ static bool nbd_client_connecting_wait(BDRVNBDState *s) static coroutine_fn void nbd_reconnect_attempt(BDRVNBDState *s) { + int ret; Error *local_err =3D NULL; + QIOChannelSocket *sioc; if (!nbd_client_connecting(s)) { return; @@ -280,19 +285,25 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNB= DState *s) s->ioc =3D NULL; } - s->connect_status =3D nbd_client_connect(s->bs, &local_err); + sioc =3D nbd_establish_connection(s->saddr, &local_err); + if (!sioc) { + ret =3D -ECONNREFUSED; + goto out; + } + + ret =3D nbd_client_handshake(s->bs, sioc, &local_err); + +out: + s->connect_status =3D ret; error_free(s->connect_err); s->connect_err =3D NULL; error_propagate(&s->connect_err, local_err); - if (s->connect_status < 0) { - /* failed attempt */ - return; + if (ret >=3D 0) { + /* successfully connected */ + s->state =3D NBD_CLIENT_CONNECTED; + qemu_co_queue_restart_all(&s->free_sema); } - - /* successfully connected */ - s->state =3D NBD_CLIENT_CONNECTED; - qemu_co_queue_restart_all(&s->free_sema); } static coroutine_fn void nbd_co_reconnect_loop(BDRVNBDState *s) @@ -1425,24 +1436,15 @@ static QIOChannelSocket *nbd_establish_connection(S= ocketAddress *saddr, return sioc; } -static int nbd_client_connect(BlockDriverState *bs, Error **errp) +/* nbd_client_handshake takes ownership on sioc. On failure it is unref'ed= . */ +static int nbd_client_handshake(BlockDriverState *bs, QIOChannelSocket *si= oc, + Error **errp) { BDRVNBDState *s =3D (BDRVNBDState *)bs->opaque; AioContext *aio_context =3D bdrv_get_aio_context(bs); int ret; - /* - * establish TCP connection, return error if it fails - * TODO: Configurable retry-until-timeout behaviour. - */ - QIOChannelSocket *sioc =3D nbd_establish_connection(s->saddr, errp); - - if (!sioc) { - return -ECONNREFUSED; - } - - /* NBD handshake */ - trace_nbd_client_connect(s->export); + trace_nbd_client_handshake(s->export); qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL); qio_channel_attach_aio_context(QIO_CHANNEL(sioc), aio_context); @@ -1489,7 +1491,7 @@ static int nbd_client_connect(BlockDriverState *bs, E= rror **errp) object_ref(OBJECT(s->ioc)); } - trace_nbd_client_connect_success(s->export); + trace_nbd_client_handshake_success(s->export); return 0; @@ -1894,6 +1896,7 @@ static int nbd_open(BlockDriverState *bs, QDict *opti= ons, int flags, { int ret; BDRVNBDState *s =3D (BDRVNBDState *)bs->opaque; + QIOChannelSocket *sioc; ret =3D nbd_process_options(bs, options, errp); if (ret < 0) { @@ -1904,7 +1907,16 @@ static int nbd_open(BlockDriverState *bs, QDict *opt= ions, int flags, qemu_co_mutex_init(&s->send_mutex); qemu_co_queue_init(&s->free_sema); - ret =3D nbd_client_connect(bs, errp); + /* + * establish TCP connection, return error if it fails + * TODO: Configurable retry-until-timeout behaviour. + */ + sioc =3D nbd_establish_connection(s->saddr, errp); + if (!sioc) { + return -ECONNREFUSED; + } + + ret =3D nbd_client_handshake(bs, sioc, errp); if (ret < 0) { nbd_clear_bdrvstate(s); return ret; diff --git a/block/trace-events b/block/trace-events index d3533ca8963c..915833506189 100644 --- a/block/trace-events +++ b/block/trace-events @@ -168,8 +168,8 @@ nbd_parse_blockstatus_compliance(const char *err) "igno= ring extra data from non- nbd_structured_read_compliance(const char *type) "server sent non-complian= t unaligned read %s chunk" nbd_read_reply_entry_fail(int ret, const char *err) "ret =3D %d, err: %s" nbd_co_request_fail(uint64_t from, uint32_t len, uint64_t handle, uint16_t= flags, uint16_t type, const char *name, int ret, const char *err) "Request= failed { .from =3D %" PRIu64", .len =3D %" PRIu32 ", .handle =3D %" PRIu64= ", .flags =3D 0x%" PRIx16 ", .type =3D %" PRIu16 " (%s) } ret =3D %d, err:= %s" -nbd_client_connect(const char *export_name) "export '%s'" -nbd_client_connect_success(const char *export_name) "export '%s'" +nbd_client_handshake(const char *export_name) "export '%s'" +nbd_client_handshake_success(const char *export_name) "export '%s'" # ssh.c ssh_restart_coroutine(void *co) "co=3D%p" --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595948876; cv=none; d=zohomail.com; s=zohoarc; b=ZTdM1fwk9XEjFzkI3FQFx97BcIuLSUFV5QFxytHY5Qhd0501eJIm7Wd4RlYBci6p5EB9vnj891pQ5WMUoFnoLPASsDSO5Ql6UxiF4DTvMM5VWzs9+n7kS4RLWA+PwkiEVlWEtDb5YFxJt7qSX8z9CO+ESfLAcJpev+vFsMnOHyo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595948876; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=r/a1rGqjYWIyOxl6RfY6IwBJxZ5hlPtmnidW1TBR+C0=; b=ieDeOunh91FKB7esa3amHXC/y4zBaoi+JHiJzXxGnQMR2ungqHhatRthI6ryZEKl1I6MokVuNN6V8UFpdjFl9g5XVRthvZlf0erAyPTnPJeVGnLASmyYR6sfyqkFZdscnil4EfBssSkj/3VeHN6WCcSwwMXJx5clCJuSn761Tl0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595948876745208.41915947365942; Tue, 28 Jul 2020 08:07:56 -0700 (PDT) Received: from localhost ([::1]:53024 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RCs-0002ks-FQ for importer@patchew.org; Tue, 28 Jul 2020 11:07:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39946) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9X-0006Ft-Qm for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:27 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:20205 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9V-0006u1-2C for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:27 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-345-x95emFrtNk67EN07YTod7A-1; Tue, 28 Jul 2020 11:04:20 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3BC741030C22; Tue, 28 Jul 2020 15:04:19 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id 634F719C71; Tue, 28 Jul 2020 15:04:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948664; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r/a1rGqjYWIyOxl6RfY6IwBJxZ5hlPtmnidW1TBR+C0=; b=DYevB8bevnY/QKT7vGiDPF85PUwfXgYBtzdq7ZXzKHpmoYrL8tDD2e8/zpFKk8IVQGOv8U cLYU5ew2Hlma9LWdf7S6UkI24jbaBMUyFoMXmEt85U1M0OF8iU3DHd2aXqKVajzi4n7J56 0IWltmMvaS+rgGFPK62Nup2OjXsLQwA= X-MC-Unique: x95emFrtNk67EN07YTod7A-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 7/9] block/nbd: allow drain during reconnect attempt Date: Tue, 28 Jul 2020 10:04:05 -0500 Message-Id: <20200728150408.291299-8-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:31:13 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Network Block Dev..." , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy It should be safe to reenter qio_channel_yield() on io/channel read/write path, so it's safe to reduce in_flight and allow attaching new aio context. And no problem to allow drain itself: connection attempt is not a guest request. Moreover, if remote server is down, we can hang in negotiation, blocking drain section and provoking a dead lock. How to reproduce the dead lock: 1. Create nbd-fault-injector.conf with the following contents: [inject-error "mega1"] event=3Ddata io=3Dreadwrite when=3Dbefore 2. In one terminal run nbd-fault-injector in a loop, like this: n=3D1; while true; do echo $n; ((n++)); ./nbd-fault-injector.py 127.0.0.1:10000 nbd-fault-injector.conf; done 3. In another terminal run qemu-io in a loop, like this: n=3D1; while true; do echo $n; ((n++)); ./qemu-io -c 'read 0 512' nbd://127.0.0.1:10000; done After some time, qemu-io will hang trying to drain, for example, like this: #3 aio_poll (ctx=3D0x55f006bdd890, blocking=3Dtrue) at util/aio-posix.c:600 #4 bdrv_do_drained_begin (bs=3D0x55f006bea710, recursive=3Dfalse, parent=3D0x0, ignore_bds_parents=3Dfalse, poll=3Dtrue) at block/io.c:427 #5 bdrv_drained_begin (bs=3D0x55f006bea710) at block/io.c:433 #6 blk_drain (blk=3D0x55f006befc80) at block/block-backend.c:1710 #7 blk_unref (blk=3D0x55f006befc80) at block/block-backend.c:498 #8 bdrv_open_inherit (filename=3D0x7fffba1563bc "nbd+tcp://127.0.0.1:10000", reference=3D0x0, options=3D0x55f006be86d0, flags=3D24578, parent=3D0x0, child_class=3D0x0, child_role=3D0, errp=3D0x7fffba154620) at block.c:3491 #9 bdrv_open (filename=3D0x7fffba1563bc "nbd+tcp://127.0.0.1:10000", reference=3D0x0, options=3D0x0, flags=3D16386, errp=3D0x7fffba154620) at block.c:3513 #10 blk_new_open (filename=3D0x7fffba1563bc "nbd+tcp://127.0.0.1:10000", reference=3D0x0, options=3D0x0, flags=3D16386, errp=3D0x7fffba154620) at block/block-backend.c:421 And connection_co stack like this: #0 qemu_coroutine_switch (from_=3D0x55f006bf2650, to_=3D0x7fe96e07d918, action=3DCOROUTINE_YIELD) at util/coroutine-ucontext.c:302 #1 qemu_coroutine_yield () at util/qemu-coroutine.c:193 #2 qio_channel_yield (ioc=3D0x55f006bb3c20, condition=3DG_IO_IN) at io/channel.c:472 #3 qio_channel_readv_all_eof (ioc=3D0x55f006bb3c20, iov=3D0x7fe96d729bf0, niov=3D1, errp=3D0x7fe96d729eb0) at io/channel.c:110 #4 qio_channel_readv_all (ioc=3D0x55f006bb3c20, iov=3D0x7fe96d729bf0, niov=3D1, errp=3D0x7fe96d729eb0) at io/channel.c:143 #5 qio_channel_read_all (ioc=3D0x55f006bb3c20, buf=3D0x7fe96d729d28 "\300.\366\004\360U", buflen=3D8, errp=3D0x7fe96d729eb0) at io/channel.c:247 #6 nbd_read (ioc=3D0x55f006bb3c20, buffer=3D0x7fe96d729d28, size=3D8, desc=3D0x55f004f69644 "initial magic", errp=3D0x7fe96d729eb0) at /work/src/qemu/master/include/block/nbd.h:365 #7 nbd_read64 (ioc=3D0x55f006bb3c20, val=3D0x7fe96d729d28, desc=3D0x55f004f69644 "initial magic", errp=3D0x7fe96d729eb0) at /work/src/qemu/master/include/block/nbd.h:391 #8 nbd_start_negotiate (aio_context=3D0x55f006bdd890, ioc=3D0x55f006bb3c20, tlscreds=3D0x0, hostname=3D0x0, outioc=3D0x55f006bf19f8, structured_reply=3Dtrue, zeroes=3D0x7fe96d729dca, errp=3D0x7fe96d729eb0) at nbd/client.c:904 #9 nbd_receive_negotiate (aio_context=3D0x55f006bdd890, ioc=3D0x55f006bb3c20, tlscreds=3D0x0, hostname=3D0x0, outioc=3D0x55f006bf19f8, info=3D0x55f006bf1a00, errp=3D0x7fe96d729eb0) = at nbd/client.c:1032 #10 nbd_client_connect (bs=3D0x55f006bea710, errp=3D0x7fe96d729eb0) at block/nbd.c:1460 #11 nbd_reconnect_attempt (s=3D0x55f006bf19f0) at block/nbd.c:287 #12 nbd_co_reconnect_loop (s=3D0x55f006bf19f0) at block/nbd.c:309 #13 nbd_connection_entry (opaque=3D0x55f006bf19f0) at block/nbd.c:360 #14 coroutine_trampoline (i0=3D113190480, i1=3D22000) at util/coroutine-ucontext.c:173 Note, that the hang may be triggered by another bug, so the whole case is fixed only together with commit "block/nbd: on shutdown terminate connection attempt". Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200727184751.15704-3-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake Signed-off-by: Eric Blake --- block/nbd.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/block/nbd.c b/block/nbd.c index 3558c173e34b..ee9ab7512b65 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -291,8 +291,22 @@ static coroutine_fn void nbd_reconnect_attempt(BDRVNBD= State *s) goto out; } + bdrv_dec_in_flight(s->bs); + ret =3D nbd_client_handshake(s->bs, sioc, &local_err); + if (s->drained) { + s->wait_drained_end =3D true; + while (s->drained) { + /* + * We may be entered once from nbd_client_attach_aio_context_bh + * and then from nbd_client_co_drain_end. So here is a loop. + */ + qemu_coroutine_yield(); + } + } + bdrv_inc_in_flight(s->bs); + out: s->connect_status =3D ret; error_free(s->connect_err); --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595948997; cv=none; d=zohomail.com; s=zohoarc; b=K0MzumaBXI7ZJOTZqgdGxRRLZ8SF4eNeGfbNig2DaOWRtStl6+j7eRjBEtsJfD7poNR5nvDh0XZ9cj/IfvexNe7mRoqw75ZuAHn+YErCz9PkcA7jkxzu6ztmg0gAywZP0f/ne7HWB+MSMx9k5GrguHlv26Kshvq0eFiiBzV4Qd4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595948997; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=SB4Hc2q6uuPHOSGYKf6pIASqeyKKgufXtcN8atc9kFs=; b=OLBAn9x4Pjop+IkGBEcQApcwc9EC4mB/zk0a5kINa85CDk5gErSrnjgmlRDrDhNGhRbxIzvO3+O5NQA9C7or+z/6St8SCOQy6uy7B5amwt92u4UbQLf+djFn/JdYani+sbB+UyRtMRAXOBOqH5x6fC5XqLpYl1Xt/Rq4vNyb4FQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595948997276949.1060717510792; Tue, 28 Jul 2020 08:09:57 -0700 (PDT) Received: from localhost ([::1]:32970 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0REn-0006ER-B5 for importer@patchew.org; Tue, 28 Jul 2020 11:09:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40050) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9b-0006OV-Hz for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:31 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:47816 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9Y-0006uy-MF for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:31 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-435-QrHGTC0FONunJbziWCHV0g-1; Tue, 28 Jul 2020 11:04:23 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2DB9A18A1DFE; Tue, 28 Jul 2020 15:04:22 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7EA0E19C71; Tue, 28 Jul 2020 15:04:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948667; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SB4Hc2q6uuPHOSGYKf6pIASqeyKKgufXtcN8atc9kFs=; b=VZ5bPJInfSNvdu2EIvAi1E6SdN/wzte/sdqDfS8q/OujeQAyfsnpE+GwK8RyVm+zI1r+Av NDAW5seWmuGK+nerPh9QbJxjQKTB3IsUdIZoiz6jFmxHFIJkpqt4bdI4p28kblaem6RYPx PmeQnV02PPKslrU8uocV39n1zpIE59s= X-MC-Unique: QrHGTC0FONunJbziWCHV0g-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 8/9] block/nbd: on shutdown terminate connection attempt Date: Tue, 28 Jul 2020 10:04:06 -0500 Message-Id: <20200728150408.291299-9-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:31:13 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Network Block Dev..." , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy On shutdown nbd driver may be in a connecting state. We should shutdown it as well, otherwise we may hang in nbd_teardown_connection, waiting for conneciton_co to finish in BDRV_POLL_WHILE(bs, s->connection_co) loop if remote server is down. How to reproduce the dead lock: 1. Create nbd-fault-injector.conf with the following contents: [inject-error "mega1"] event=3Ddata io=3Dreadwrite when=3Dbefore 2. In one terminal run nbd-fault-injector in a loop, like this: n=3D1; while true; do echo $n; ((n++)); ./nbd-fault-injector.py 127.0.0.1:10000 nbd-fault-injector.conf; done 3. In another terminal run qemu-io in a loop, like this: n=3D1; while true; do echo $n; ((n++)); ./qemu-io -c 'read 0 512' nbd://127.0.0.1:10000; done After some time, qemu-io will hang. Note, that this hang may be triggered by another bug, so the whole case is fixed only together with commit "block/nbd: allow drain during reconnect attempt". Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Message-Id: <20200727184751.15704-4-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- block/nbd.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index ee9ab7512b65..620c97be6ba2 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -209,11 +209,15 @@ static void nbd_teardown_connection(BlockDriverState = *bs) { BDRVNBDState *s =3D (BDRVNBDState *)bs->opaque; - if (s->state =3D=3D NBD_CLIENT_CONNECTED) { + if (s->ioc) { /* finish any pending coroutines */ - assert(s->ioc); qio_channel_shutdown(s->ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL); + } else if (s->sioc) { + /* abort negotiation */ + qio_channel_shutdown(QIO_CHANNEL(s->sioc), QIO_CHANNEL_SHUTDOWN_BO= TH, + NULL); } + s->state =3D NBD_CLIENT_QUIT; if (s->connection_co) { if (s->connection_co_sleep_ns_state) { @@ -1459,6 +1463,9 @@ static int nbd_client_handshake(BlockDriverState *bs,= QIOChannelSocket *sioc, int ret; trace_nbd_client_handshake(s->export); + + s->sioc =3D sioc; + qio_channel_set_blocking(QIO_CHANNEL(sioc), false, NULL); qio_channel_attach_aio_context(QIO_CHANNEL(sioc), aio_context); @@ -1473,6 +1480,7 @@ static int nbd_client_handshake(BlockDriverState *bs,= QIOChannelSocket *sioc, g_free(s->info.name); if (ret < 0) { object_unref(OBJECT(sioc)); + s->sioc =3D NULL; return ret; } if (s->x_dirty_bitmap && !s->info.base_allocation) { @@ -1498,8 +1506,6 @@ static int nbd_client_handshake(BlockDriverState *bs,= QIOChannelSocket *sioc, } } - s->sioc =3D sioc; - if (!s->ioc) { s->ioc =3D QIO_CHANNEL(sioc); object_ref(OBJECT(s->ioc)); @@ -1520,6 +1526,7 @@ static int nbd_client_handshake(BlockDriverState *bs,= QIOChannelSocket *sioc, nbd_send_request(s->ioc ?: QIO_CHANNEL(sioc), &request); object_unref(OBJECT(sioc)); + s->sioc =3D NULL; return ret; } --=20 2.27.0 From nobody Fri Apr 26 14:22:01 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1595949262; cv=none; d=zohomail.com; s=zohoarc; b=HTCsWTl18nONwYcWtfba68Fja3AxkP2QHJBYGvExj4+0b9ccacknvcKbfg6OPdrO1cPeo37DGslFZVtj/B5NgQOUhtJwNU/tn+/BeGhi/6HAFv85TKUCEZb7Ao3gS60A/OqRlD/6q4LxxAz3dRjKRn0dTRG99RgDSYG0rmiY/Ec= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595949262; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ud/U6DKMNFT/byf9RgFHSOlNS0KW7MtFl8y7LTDHIqw=; b=RM8A0asc1dQdw2EK7ygvq5tTSkIsnDC0vQeg2QS8j9+j5EZ6pN682xgwPbqazLdcpbhScVwAuacX+5p2Zr7GEXd1+8TwwQEPwUONvcq2j7bD/rM/9DudyqwvGm8hl5ofzz3UzbwibsKd30LtahaFniCLbvbu8yt2bY+RTtnOYFM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1595949262251269.62627708520256; Tue, 28 Jul 2020 08:14:22 -0700 (PDT) Received: from localhost ([::1]:53260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k0RJ6-0006cU-R2 for importer@patchew.org; Tue, 28 Jul 2020 11:14:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40078) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k0R9d-0006S2-5b for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:33 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:58350 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1k0R9b-0006vt-FE for qemu-devel@nongnu.org; Tue, 28 Jul 2020 11:04:32 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-165-2Js8r9QKM2-YhU9KrQGpMQ-1; Tue, 28 Jul 2020 11:04:27 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1D7CA800460; Tue, 28 Jul 2020 15:04:26 +0000 (UTC) Received: from blue.redhat.com (ovpn-118-248.rdu2.redhat.com [10.10.118.248]) by smtp.corp.redhat.com (Postfix) with ESMTP id D2EDB19C71; Tue, 28 Jul 2020 15:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595948670; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ud/U6DKMNFT/byf9RgFHSOlNS0KW7MtFl8y7LTDHIqw=; b=dXKQBtSmvE03oj8jc1lVuDyF33RDVTGSlDMwmdMlXxSxZYPl98wRnBiQtxwbVEIRBgLVpj D6SeT40wpd6QpKIOwlg2TbBufIogRITTMAhcdRppc2t1ks1VikZ2RB626Sbwj5i2g1APZn BU2EZrcfCZz+p/WpAWXjmn5FIiBLl6U= X-MC-Unique: 2Js8r9QKM2-YhU9KrQGpMQ-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 9/9] block/nbd: nbd_co_reconnect_loop(): don't sleep if drained Date: Tue, 28 Jul 2020 10:04:07 -0500 Message-Id: <20200728150408.291299-10-eblake@redhat.com> In-Reply-To: <20200728150408.291299-1-eblake@redhat.com> References: <20200728150408.291299-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/28 10:28:27 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Network Block Dev..." , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy We try to go to wakeable sleep, so that, if drain begins it will break the sleep. But what if nbd_client_co_drain_begin() already called and s->drained is already true? We'll go to sleep, and drain will have to wait for the whole timeout. Let's improve it. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Message-Id: <20200727184751.15704-5-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- block/nbd.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index 620c97be6ba2..7bb881fef49c 100644 --- a/block/nbd.c +++ b/block/nbd.c @@ -341,8 +341,6 @@ static coroutine_fn void nbd_co_reconnect_loop(BDRVNBDS= tate *s) qemu_co_queue_restart_all(&s->free_sema); } - qemu_co_sleep_ns_wakeable(QEMU_CLOCK_REALTIME, timeout, - &s->connection_co_sleep_ns_state); if (s->drained) { bdrv_dec_in_flight(s->bs); s->wait_drained_end =3D true; @@ -354,9 +352,12 @@ static coroutine_fn void nbd_co_reconnect_loop(BDRVNBD= State *s) qemu_coroutine_yield(); } bdrv_inc_in_flight(s->bs); - } - if (timeout < max_timeout) { - timeout *=3D 2; + } else { + qemu_co_sleep_ns_wakeable(QEMU_CLOCK_REALTIME, timeout, + &s->connection_co_sleep_ns_state); + if (timeout < max_timeout) { + timeout *=3D 2; + } } nbd_reconnect_attempt(s); --=20 2.27.0