From nobody Sun May 5 08:47:53 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1630077126383524.9961523421343; Fri, 27 Aug 2021 08:12:06 -0700 (PDT) Received: from localhost ([::1]:35590 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mJdWW-0004ox-OO for importer@patchew.org; Fri, 27 Aug 2021 11:12:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54340) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mJdU5-0003IB-LR for qemu-devel@nongnu.org; Fri, 27 Aug 2021 11:09:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:52103) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mJdU1-0004xB-VH for qemu-devel@nongnu.org; Fri, 27 Aug 2021 11:09:33 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-372-yTFrWzX1OqWXOLcqdRyK8A-1; Fri, 27 Aug 2021 11:09:23 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6DD451082922; Fri, 27 Aug 2021 15:09:22 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-96.phx2.redhat.com [10.3.112.96]) by smtp.corp.redhat.com (Postfix) with ESMTP id C68F460BD9; Fri, 27 Aug 2021 15:09:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630076968; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=FG3bCkqWxQVSEzIavrFuJdg8rr9M8qOOL17NqmnRPFQ=; b=C0FkIWZIcM0iPtwD2ePloQS8PEAmDFNv3tXAfdtu7FMlRuu26Z4vBorfIcZh5foIeWcaoP uVC8iCDanoZPCfF4kHkfSDY+2w0grmLHZV/HWTvZvAOof63wY1s5FMdGB4P5pk1GHmHb6u yXYqO9x9kg6HPIUQVE3PURSTIHa7Ng4= X-MC-Unique: yTFrWzX1OqWXOLcqdRyK8A-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PATCH] nbd/server: Advertise MULTI_CONN for shared writable exports Date: Fri, 27 Aug 2021 10:09:16 -0500 Message-Id: <20210827150916.532260-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eblake@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -34 X-Spam_score: -3.5 X-Spam_bar: --- X-Spam_report: (-3.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.743, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Vladimir Sementsov-Ogievskiy , qemu-block@nongnu.org, rjones@redhat.com, Markus Armbruster , nsoffer@redhat.com, Hanna Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1630077128634100001 Content-Type: text/plain; charset="utf-8" According to the NBD spec, a server advertising NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush. We satisfy these conditions in qemu because our block layer serializes any overlapping operations (see bdrv_find_conflicting_request and friends): no matter which client performs a flush, parallel requests coming from distinct NBD clients will still be well-ordered by the time they are passed on to the underlying device, with no caching in qemu proper to allow stale results to leak after a flush. We don't want to advertise MULTI_CONN when we know that a second client can connect (which is the default for qemu-nbd, but not for QMP nbd-server-add), so it does require a QAPI addition. But other than that, the actual change to advertise the bit for writable servers is fairly small. The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but concisely managing that in shell is painful. I found it easier to do by relying on the libnbd project's nbdsh, which means this test will be skipped on platforms where that is not available. Signed-off-by: Eric Blake Reviewed-by: Richard W.M. Jones --- docs/interop/nbd.txt | 1 + docs/tools/qemu-nbd.rst | 3 +- qapi/block-export.json | 6 +- blockdev-nbd.c | 4 + nbd/server.c | 10 +-- qemu-nbd.c | 2 + MAINTAINERS | 1 + tests/qemu-iotests/tests/nbd-multiconn | 91 ++++++++++++++++++++++ tests/qemu-iotests/tests/nbd-multiconn.out | 14 ++++ 9 files changed, 124 insertions(+), 8 deletions(-) create mode 100755 tests/qemu-iotests/tests/nbd-multiconn create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out diff --git a/docs/interop/nbd.txt b/docs/interop/nbd.txt index 10ce098a29bf..d03910f1e9eb 100644 --- a/docs/interop/nbd.txt +++ b/docs/interop/nbd.txt @@ -68,3 +68,4 @@ NBD_CMD_BLOCK_STATUS for "qemu:dirty-bitmap:", NBD_CMD_CA= CHE * 4.2: NBD_FLAG_CAN_MULTI_CONN for shareable read-only exports, NBD_CMD_FLAG_FAST_ZERO * 5.2: NBD_CMD_BLOCK_STATUS for "qemu:allocation-depth" +* 6.2: NBD_FLAG_CAN_MULTI_CONN for shareable writable exports diff --git a/docs/tools/qemu-nbd.rst b/docs/tools/qemu-nbd.rst index 5643da26e982..81be32164a55 100644 --- a/docs/tools/qemu-nbd.rst +++ b/docs/tools/qemu-nbd.rst @@ -138,8 +138,7 @@ driver options if ``--image-opts`` is specified. .. option:: -e, --shared=3DNUM Allow up to *NUM* clients to share the device (default - ``1``), 0 for unlimited. Safe for readers, but for now, - consistency is not guaranteed between multiple writers. + ``1``), 0 for unlimited. .. option:: -t, --persistent diff --git a/qapi/block-export.json b/qapi/block-export.json index 0ed63442a819..b2085a9fdd4c 100644 --- a/qapi/block-export.json +++ b/qapi/block-export.json @@ -95,11 +95,15 @@ # the metadata context name "qemu:allocation-depth" to # inspect allocation details. (since 5.2) # +# @shared: True if the server should advertise that multiple clients may +# connect, default false. (since 6.2) +# # Since: 5.2 ## { 'struct': 'BlockExportOptionsNbd', 'base': 'BlockExportOptionsNbdBase', - 'data': { '*bitmaps': ['str'], '*allocation-depth': 'bool' } } + 'data': { '*bitmaps': ['str'], '*allocation-depth': 'bool', + '*shared': 'bool' } } ## # @BlockExportOptionsVhostUserBlk: diff --git a/blockdev-nbd.c b/blockdev-nbd.c index bdfa7ed3a5a9..258feaa76e02 100644 --- a/blockdev-nbd.c +++ b/blockdev-nbd.c @@ -221,6 +221,10 @@ void qmp_nbd_server_add(NbdServerAddOptions *arg, Erro= r **errp) QAPI_LIST_PREPEND(export_opts->u.nbd.bitmaps, g_strdup(arg->bitmap= )); } + /* nbd-server-add always permits parallel clients */ + export_opts->u.nbd.has_shared =3D true; + export_opts->u.nbd.shared =3D true; + /* * nbd-server-add doesn't complain when a read-only device should be * exported as writable, but simply downgrades it. This is an error wi= th diff --git a/nbd/server.c b/nbd/server.c index 3927f7789dcf..1646796a4798 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -1,5 +1,5 @@ /* - * Copyright (C) 2016-2020 Red Hat, Inc. + * Copyright (C) 2016-2021 Red Hat, Inc. * Copyright (C) 2005 Anthony Liguori * * Network Block Device Server Side @@ -1634,7 +1634,7 @@ static int nbd_export_create(BlockExport *blk_exp, Bl= ockExportOptions *exp_args, int64_t size; uint64_t perm, shared_perm; bool readonly =3D !exp_args->writable; - bool shared =3D !exp_args->writable; + bool shared =3D arg->shared; strList *bitmaps; size_t i; int ret; @@ -1685,11 +1685,11 @@ static int nbd_export_create(BlockExport *blk_exp, = BlockExportOptions *exp_args, exp->description =3D g_strdup(arg->description); exp->nbdflags =3D (NBD_FLAG_HAS_FLAGS | NBD_FLAG_SEND_FLUSH | NBD_FLAG_SEND_FUA | NBD_FLAG_SEND_CACHE); + if (shared) { + exp->nbdflags |=3D NBD_FLAG_CAN_MULTI_CONN; + } if (readonly) { exp->nbdflags |=3D NBD_FLAG_READ_ONLY; - if (shared) { - exp->nbdflags |=3D NBD_FLAG_CAN_MULTI_CONN; - } } else { exp->nbdflags |=3D (NBD_FLAG_SEND_TRIM | NBD_FLAG_SEND_WRITE_ZEROE= S | NBD_FLAG_SEND_FAST_ZERO); diff --git a/qemu-nbd.c b/qemu-nbd.c index f68fcceadd9e..198b1c55729d 100644 --- a/qemu-nbd.c +++ b/qemu-nbd.c @@ -1084,6 +1084,8 @@ int main(int argc, char **argv) .bitmaps =3D bitmaps, .has_allocation_depth =3D alloc_depth, .allocation_depth =3D alloc_depth, + .has_shared =3D true, + .shared =3D shared !=3D 1, }, }; blk_exp_add(export_opts, &error_fatal); diff --git a/MAINTAINERS b/MAINTAINERS index 6b3697962c1b..6eab82e6b982 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3142,6 +3142,7 @@ F: qemu-nbd.* F: blockdev-nbd.c F: docs/interop/nbd.txt F: docs/tools/qemu-nbd.rst +F: tests/qemu-iotests/tests/*nbd* T: git https://repo.or.cz/qemu/ericb.git nbd T: git https://src.openvz.org/scm/~vsementsov/qemu.git nbd diff --git a/tests/qemu-iotests/tests/nbd-multiconn b/tests/qemu-iotests/te= sts/nbd-multiconn new file mode 100755 index 000000000000..66b410546e2f --- /dev/null +++ b/tests/qemu-iotests/tests/nbd-multiconn @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# group: rw auto quick +# +# Test that qemu-nbd MULTI_CONN works +# +# Copyright (C) 2018-2021 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +seq=3D"$(basename $0)" +echo "QA output created by $seq" + +status=3D1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img + nbd_server_stop +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +cd .. +. ./common.rc +. ./common.filter +. ./common.nbd + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +_require_command QEMU_NBD + +# Parallel client connections are easier with libnbd than qemu-io: +if ! (nbdsh --version) >/dev/null 2>&1; then + _notrun "nbdsh utility required, skipped this test" +fi + +echo +echo "=3D=3D=3D Initial image setup =3D=3D=3D" +echo + +_make_test_img 4M +$QEMU_IO -c 'w -P 1 0 2M' -c 'w -P 2 2M 2M' "$TEST_IMG" | _filter_qemu_io +IMG=3D"driver=3Dnbd,server.type=3Dunix,server.path=3D$nbd_unix_socket" +nbd_server_start_unix_socket -f qcow2 -e 5 "$TEST_IMG" + +echo +echo "=3D=3D=3D Demonstrate parallel clients =3D=3D=3D" +echo + +export nbd_unix_socket +nbdsh -c ' +import os +sock =3D os.getenv("nbd_unix_socket") +h =3D [] + +for i in range(3): + h.append(nbd.NBD()) + h[i].connect_unix(sock) + assert h[i].can_multi_conn() + +buf1 =3D h[0].pread(1024 * 1024, 0) +if buf1 !=3D b"\x01" * 1024 * 1024: + print("Unexpected initial read") +buf2 =3D b"\x03" * 1024 * 1024 +h[1].pwrite(buf2, 0) +h[2].flush() +buf1 =3D h[0].pread(1024 * 1024, 0) +if buf1 =3D=3D buf2: + print("Flush appears to be consistent across connections") + +for i in range(3): + h[i].shutdown() +' + +# success, all done +echo '*** done' +rm -f $seq.full +status=3D0 diff --git a/tests/qemu-iotests/tests/nbd-multiconn.out b/tests/qemu-iotest= s/tests/nbd-multiconn.out new file mode 100644 index 000000000000..4554099e4d62 --- /dev/null +++ b/tests/qemu-iotests/tests/nbd-multiconn.out @@ -0,0 +1,14 @@ +QA output created by nbd-multiconn + +=3D=3D=3D Initial image setup =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D4194304 +wrote 2097152/2097152 bytes at offset 0 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 2097152/2097152 bytes at offset 2097152 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=3D=3D=3D Demonstrate parallel clients =3D=3D=3D + +Flush appears to be consistent across connections +*** done --=20 2.31.1