From nobody Wed May 8 20:29:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1505816823484365.83992100755313; Tue, 19 Sep 2017 03:27:03 -0700 (PDT) Received: from localhost ([::1]:41293 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFk6-0005db-Dn for importer@patchew.org; Tue, 19 Sep 2017 06:27:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45335) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFhz-00049r-IK for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1duFhx-0004dV-3q for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:51 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50680) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1duFhm-0004Vc-BN; Tue, 19 Sep 2017 06:24:38 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 758D7267CA; Tue, 19 Sep 2017 10:24:37 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-117-61.ams2.redhat.com [10.36.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 738D55D6A4; Tue, 19 Sep 2017 10:24:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 758D7267CA Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=pbonzini@redhat.com From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 19 Sep 2017 12:24:31 +0200 Message-Id: <20170919102434.21147-2-pbonzini@redhat.com> In-Reply-To: <20170919102434.21147-1-pbonzini@redhat.com> References: <20170919102434.21147-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Tue, 19 Sep 2017 10:24:37 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 1/4] scsi, file-posix: add support for persistent reservation management X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" It is a common requirement for virtual machine to send persistent reservations, but this currently requires either running QEMU with CAP_SYS_RAWIO, or using out-of-tree patches that let an unprivileged QEMU bypass Linux's filter on SG_IO commands. As an alternative mechanism, the next patches will introduce a privileged helper to run persistent reservation commands without expanding QEMU's attack surface unnecessarily. The helper is invoked through a "pr-manager" QOM object, to which file-posix.c passes SG_IO requests for PERSISTENT RESERVE OUT and PERSISTENT RESERVE IN commands. For example: $ qemu-system-x86_64 -device virtio-scsi \ -object pr-manager-helper,id=3Dhelper0,path=3D/var/run/qemu-pr-helper= .sock -drive if=3Dnone,id=3Dhd,driver=3Draw,file.filename=3D/dev/sdb,file.p= r-manager=3Dhelper0 -device scsi-block,drive=3Dhd or: $ qemu-system-x86_64 -device virtio-scsi \ -object pr-manager-helper,id=3Dhelper0,path=3D/var/run/qemu-pr-helper= .sock -blockdev node-name=3Dhd,driver=3Draw,file.driver=3Dhost_device,file.= filename=3D/dev/sdb,file.pr-manager=3Dhelper0 -device scsi-block,drive=3Dhd Multiple pr-manager implementations are conceivable and possible, though only one is implemented right now. For example, a pr-manager could: - talk directly to the multipath daemon from a privileged QEMU (i.e. QEMU links to libmpathpersist); this makes reservation work properly with multipath, but still requires CAP_SYS_RAWIO - use the Linux IOC_PR_* ioctls (they require CAP_SYS_ADMIN though) - more interestingly, implement reservations directly in QEMU through file system locks or a shared database (e.g. sqlite) Signed-off-by: Paolo Bonzini Reviewed-by: Stefan Hajnoczi --- Makefile.objs | 1 + block/file-posix.c | 30 +++++++++++++ docs/pr-manager.rst | 51 ++++++++++++++++++++++ include/scsi/pr-manager.h | 56 ++++++++++++++++++++++++ qapi/block-core.json | 4 ++ scsi/Makefile.objs | 2 + scsi/pr-manager.c | 109 ++++++++++++++++++++++++++++++++++++++++++= ++++ vl.c | 3 +- 8 files changed, 255 insertions(+), 1 deletion(-) create mode 100644 docs/pr-manager.rst create mode 100644 include/scsi/pr-manager.h create mode 100644 scsi/pr-manager.c diff --git a/Makefile.objs b/Makefile.objs index 0caa8a5cf8..12abaa6191 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -170,6 +170,7 @@ trace-events-subdirs +=3D qapi trace-events-subdirs +=3D accel/tcg trace-events-subdirs +=3D accel/kvm trace-events-subdirs +=3D nbd +trace-events-subdirs +=3D scsi =20 trace-events-files =3D $(SRC_PATH)/trace-events $(trace-events-subdirs:%= =3D$(SRC_PATH)/%/trace-events) =20 diff --git a/block/file-posix.c b/block/file-posix.c index 6acbd56238..ab12a2b591 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -33,6 +33,9 @@ #include "block/raw-aio.h" #include "qapi/qmp/qstring.h" =20 +#include "scsi/pr-manager.h" +#include "scsi/constants.h" + #if defined(__APPLE__) && (__MACH__) #include #include @@ -155,6 +158,8 @@ typedef struct BDRVRawState { bool page_cache_inconsistent:1; bool has_fallocate; bool needs_alignment; + + PRManager *pr_mgr; } BDRVRawState; =20 typedef struct BDRVRawReopenState { @@ -402,6 +407,11 @@ static QemuOptsList raw_runtime_opts =3D { .type =3D QEMU_OPT_STRING, .help =3D "file locking mode (on/off/auto, default: auto)", }, + { + .name =3D "pr-manager", + .type =3D QEMU_OPT_STRING, + .help =3D "id of persistent reservation manager object (defaul= t: none)", + }, { /* end of list */ } }, }; @@ -413,6 +423,7 @@ static int raw_open_common(BlockDriverState *bs, QDict = *options, QemuOpts *opts; Error *local_err =3D NULL; const char *filename =3D NULL; + const char *str; BlockdevAioOptions aio, aio_default; int fd, ret; struct stat st; @@ -476,6 +487,16 @@ static int raw_open_common(BlockDriverState *bs, QDict= *options, abort(); } =20 + str =3D qemu_opt_get(opts, "pr-manager"); + if (str) { + s->pr_mgr =3D pr_manager_lookup(str, &local_err); + if (local_err) { + error_propagate(errp, local_err); + ret =3D -EINVAL; + goto fail; + } + } + s->open_flags =3D open_flags; raw_parse_flags(bdrv_flags, &s->open_flags); =20 @@ -2597,6 +2618,15 @@ static BlockAIOCB *hdev_aio_ioctl(BlockDriverState *= bs, if (fd_open(bs) < 0) return NULL; =20 + if (req =3D=3D SG_IO && s->pr_mgr) { + struct sg_io_hdr *io_hdr =3D buf; + if (io_hdr->cmdp[0] =3D=3D PERSISTENT_RESERVE_OUT || + io_hdr->cmdp[0] =3D=3D PERSISTENT_RESERVE_IN) { + return pr_manager_execute(s->pr_mgr, bdrv_get_aio_context(bs), + s->fd, io_hdr, cb, opaque); + } + } + acb =3D g_new(RawPosixAIOData, 1); acb->bs =3D bs; acb->aio_type =3D QEMU_AIO_IOCTL; diff --git a/docs/pr-manager.rst b/docs/pr-manager.rst new file mode 100644 index 0000000000..b6089fb57c --- /dev/null +++ b/docs/pr-manager.rst @@ -0,0 +1,51 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Persistent reservation managers +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +SCSI persistent Reservations allow restricting access to block devices +to specific initiators in a shared storage setup. When implementing +clustering of virtual machines, it is a common requirement for virtual +machines to send persistent reservation SCSI commands. However, +the operating system restricts sending these commands to unprivileged +programs because incorrect usage can disrupt regular operation of the +storage fabric. + +For this reason, QEMU's SCSI passthrough devices, ``scsi-block`` +and ``scsi-generic`` (both are only available on Linux) can delegate +implementation of persistent reservations to a separate object, +the "persistent reservation manager". Only PERSISTENT RESERVE OUT and +PERSISTENT RESERVE IN commands are passed to the persistent reservation +manager object; other commands are processed by QEMU as usual. + +----------------------------------------- +Defining a persistent reservation manager +----------------------------------------- + +A persistent reservation manager is an instance of a subclass of the +"pr-manager" QOM class. + +Right now only one subclass is defined, ``pr-manager-helper``, which +forwards the commands to an external privileged helper program +over Unix sockets. The helper program only allows sending persistent +reservation commands to devices for which QEMU has a file descriptor, +so that QEMU will not be able to effect persistent reservations +unless it has access to both the socket and the device. + +``pr-manager-helper`` has a single string property, ``path``, which +accepts the path to the helper program's Unix socket. For example, +the following command line defines a ``pr-manager-helper`` object and +attaches it to a SCSI passthrough device:: + + $ qemu-system-x86_64 + -device virtio-scsi \ + -object pr-manager-helper,id=3Dhelper0,path=3D/var/run/qemu-pr-h= elper.sock + -drive if=3Dnone,id=3Dhd,driver=3Draw,file.filename=3D/dev/sdb,f= ile.pr-manager=3Dhelper0 + -device scsi-block,drive=3Dhd + +Alternatively, using ``-blockdev``:: + + $ qemu-system-x86_64 + -device virtio-scsi \ + -object pr-manager-helper,id=3Dhelper0,path=3D/var/run/qemu-pr-h= elper.sock + -blockdev node-name=3Dhd,driver=3Draw,file.driver=3Dhost_device,= file.filename=3D/dev/sdb,file.pr-manager=3Dhelper0 + -device scsi-block,drive=3Dhd diff --git a/include/scsi/pr-manager.h b/include/scsi/pr-manager.h new file mode 100644 index 0000000000..b2b37d63bc --- /dev/null +++ b/include/scsi/pr-manager.h @@ -0,0 +1,56 @@ +#ifndef PR_MANAGER_H +#define PR_MANAGER_H + +#include "qom/object.h" +#include "qapi/qmp/qdict.h" +#include "qapi/visitor.h" +#include "qom/object_interfaces.h" +#include "block/aio.h" + +#define TYPE_PR_MANAGER "pr-manager" + +#define PR_MANAGER_CLASS(klass) \ + OBJECT_CLASS_CHECK(PRManagerClass, (klass), TYPE_PR_MANAGER) +#define PR_MANAGER_GET_CLASS(obj) \ + OBJECT_GET_CLASS(PRManagerClass, (obj), TYPE_PR_MANAGER) +#define PR_MANAGER(obj) \ + OBJECT_CHECK(PRManager, (obj), TYPE_PR_MANAGER) + +struct sg_io_hdr; + +typedef struct PRManager { + /* */ + Object parent; +} PRManager; + +/** + * PRManagerClass: + * @parent_class: the base class + * @run: callback invoked in thread pool context + */ +typedef struct PRManagerClass { + /* */ + ObjectClass parent_class; + + /* */ + int (*run)(PRManager *pr_mgr, int fd, struct sg_io_hdr *hdr); +} PRManagerClass; + +BlockAIOCB *pr_manager_execute(PRManager *pr_mgr, + AioContext *ctx, int fd, + struct sg_io_hdr *hdr, + BlockCompletionFunc *complete, + void *opaque); + +#ifdef CONFIG_LINUX +PRManager *pr_manager_lookup(const char *id, Error **errp); +#else +static inline PRManager *pr_manager_lookup(const char *id, Error **errp) +{ + /* The classes do not exist at all! */ + error_setg(errp, "No persistent reservation manager with id '%s'", id); + return NULL; +} +#endif + +#endif diff --git a/qapi/block-core.json b/qapi/block-core.json index bb11815608..c69a395804 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -2241,6 +2241,9 @@ # Driver specific block device options for the file backend. # # @filename: path to the image file +# @pr-manager: the id for the object that will handle persistent reservat= ions +# for this device (default: none, forward the commands via S= G_IO; +# since 2.11) # @aio: AIO backend (default: threads) (since: 2.8) # @locking: whether to enable file locking. If set to 'auto', only ena= ble # when Open File Descriptor (OFD) locking API is available @@ -2250,6 +2253,7 @@ ## { 'struct': 'BlockdevOptionsFile', 'data': { 'filename': 'str', + '*pr-manager': 'str', '*locking': 'OnOffAuto', '*aio': 'BlockdevAioOptions' } } =20 diff --git a/scsi/Makefile.objs b/scsi/Makefile.objs index 31b82a5a36..5496d2ae6a 100644 --- a/scsi/Makefile.objs +++ b/scsi/Makefile.objs @@ -1 +1,3 @@ block-obj-y +=3D utils.o + +block-obj-$(CONFIG_LINUX) +=3D pr-manager.o diff --git a/scsi/pr-manager.c b/scsi/pr-manager.c new file mode 100644 index 0000000000..cde19b87c3 --- /dev/null +++ b/scsi/pr-manager.c @@ -0,0 +1,109 @@ +/* + * Persistent reservation manager abstract class + * + * Copyright (c) 2017 Red Hat, Inc. + * + * Author: Paolo Bonzini + * + * This code is licensed under the LGPL. + * + */ + +#include "qemu/osdep.h" +#include + +#include "qapi/error.h" +#include "block/aio.h" +#include "block/thread-pool.h" +#include "scsi/pr-manager.h" +#include "trace.h" + +typedef struct PRManagerData { + PRManager *pr_mgr; + struct sg_io_hdr *hdr; + int fd; +} PRManagerData; + +static int pr_manager_worker(void *opaque) +{ + PRManagerData *data =3D opaque; + PRManager *pr_mgr =3D data->pr_mgr; + PRManagerClass *pr_mgr_class =3D + PR_MANAGER_GET_CLASS(pr_mgr); + struct sg_io_hdr *hdr =3D data->hdr; + int fd =3D data->fd; + int r; + + g_free(data); + trace_pr_manager_run(fd, hdr->cmdp[0], hdr->cmdp[1]); + + /* The is was taken in pr_manager_execute. */ + r =3D pr_mgr_class->run(pr_mgr, fd, hdr); + object_unref(OBJECT(pr_mgr)); + return r; +} + + +BlockAIOCB *pr_manager_execute(PRManager *pr_mgr, + AioContext *ctx, int fd, + struct sg_io_hdr *hdr, + BlockCompletionFunc *complete, + void *opaque) +{ + PRManagerData *data =3D g_new(PRManagerData, 1); + ThreadPool *pool =3D aio_get_thread_pool(ctx); + + trace_pr_manager_execute(fd, hdr->cmdp[0], hdr->cmdp[1], opaque); + data->pr_mgr =3D pr_mgr; + data->fd =3D fd; + data->hdr =3D hdr; + + /* The matching object_unref is in pr_manager_worker. */ + object_ref(OBJECT(pr_mgr)); + return thread_pool_submit_aio(pool, pr_manager_worker, + data, complete, opaque); +} + +static const TypeInfo pr_manager_info =3D { + .parent =3D TYPE_OBJECT, + .name =3D TYPE_PR_MANAGER, + .class_size =3D sizeof(PRManagerClass), + .abstract =3D true, + .interfaces =3D (InterfaceInfo[]) { + { TYPE_USER_CREATABLE }, + { } + } +}; + +PRManager *pr_manager_lookup(const char *id, Error **errp) +{ + Object *obj; + PRManager *pr_mgr; + + obj =3D object_resolve_path_component(object_get_objects_root(), id); + if (!obj) { + error_setg(errp, "No persistent reservation manager with id '%s'",= id); + return NULL; + } + + pr_mgr =3D (PRManager *) + object_dynamic_cast(obj, + TYPE_PR_MANAGER); + if (!pr_mgr) { + error_setg(errp, + "Object with id '%s' is not a persistent reservation ma= nager", + id); + return NULL; + } + + return pr_mgr; +} + +static void +pr_manager_register_types(void) +{ + type_register_static(&pr_manager_info); +} + + +type_init(pr_manager_register_types); diff --git a/vl.c b/vl.c index 9e62e92aea..bfee61053b 100644 --- a/vl.c +++ b/vl.c @@ -2893,7 +2893,8 @@ static int machine_set_property(void *opaque, */ static bool object_create_initial(const char *type) { - if (g_str_equal(type, "rng-egd")) { + if (g_str_equal(type, "rng-egd") || + g_str_has_prefix(type, "pr-manager-")) { return false; } =20 --=20 2.13.5 From nobody Wed May 8 20:29:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150581681836438.151850104711; Tue, 19 Sep 2017 03:26:58 -0700 (PDT) Received: from localhost ([::1]:41292 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFk1-0005Xv-4n for importer@patchew.org; Tue, 19 Sep 2017 06:26:57 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45373) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFi2-0004C9-3c for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1duFhv-0004cU-IJ for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:48114) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1duFhn-0004WS-V1; Tue, 19 Sep 2017 06:24:40 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0049781DE6; Tue, 19 Sep 2017 10:24:39 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-117-61.ams2.redhat.com [10.36.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id CA1F05D6A4; Tue, 19 Sep 2017 10:24:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 0049781DE6 Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=pbonzini@redhat.com From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 19 Sep 2017 12:24:32 +0200 Message-Id: <20170919102434.21147-3-pbonzini@redhat.com> In-Reply-To: <20170919102434.21147-1-pbonzini@redhat.com> References: <20170919102434.21147-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 19 Sep 2017 10:24:39 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 2/4] scsi: build qemu-pr-helper X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce a privileged helper to run persistent reservation commands. This lets virtual machines send persistent reservations without using CAP_SYS_RAWIO or out-of-tree patches. The helper uses Unix permissions and SCM_RIGHTS to restrict access to processes that can access its socket and prove that they have an open file descriptor for a raw SCSI device. The next patch will also correct the usage of persistent reservations with multipath devices. It would also be possible to support for Linux's IOC_PR_* ioctls in the future, to support NVMe devices. For now, however, only SCSI is supported. Signed-off-by: Paolo Bonzini Reviewed-by: Stefan Hajnoczi --- v1->v2: do not install man page on !Linux systems fix typos in documentation clarify initialization protocol clarify concurrency added BSD license to protocol header added payload size to reply, to handle residual (buffer underrun) fixed closing of file descriptors on error block PERSISTENT RESERVE OUT for read-only file descriptors do not use g_assert use EXIT_SUCCESS/EXIT_FAILURE consistently removed all CONFIG_MPATH usage from this patch Makefile | 2 + configure | 2 +- docs/interop/pr-helper.rst | 83 ++++++ docs/pr-manager.rst | 33 +++ scsi/pr-helper.h | 41 +++ scsi/qemu-pr-helper.c | 698 +++++++++++++++++++++++++++++++++++++++++= ++++ 6 files changed, 866 insertions(+), 1 deletions(-) create mode 100644 docs/interop/pr-helper.rst create mode 100644 scsi/pr-helper.h create mode 100644 scsi/qemu-pr-helper.c diff --git a/Makefile b/Makefile index b53fc69a60..51503ee8ad 100644 --- a/Makefile +++ b/Makefile @@ -373,6 +376,8 @@ qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COM= MON_LDADDS) fsdev/virtfs-proxy-helper$(EXESUF): fsdev/virtfs-proxy-helper.o fsdev/9p-m= arshal.o fsdev/9p-iov-marshal.o $(COMMON_LDADDS) fsdev/virtfs-proxy-helper$(EXESUF): LIBS +=3D -lcap =20 +scsi/qemu-pr-helper$(EXESUF): scsi/qemu-pr-helper.o scsi/utils.o $(crypto-= obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) + qemu-img-cmds.h: $(SRC_PATH)/qemu-img-cmds.hx $(SRC_PATH)/scripts/hxtool $(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -h < $< > $@,"GEN","$@= ") =20 diff --git a/configure b/configure index 94db2d103e..fba27cf0fc 100755 --- a/configure +++ b/configure @@ -6506,7 +6506,7 @@ fi =20 # build tree in object directory in case the source is not in the current = directory DIRS=3D"tests tests/tcg tests/tcg/cris tests/tcg/lm32 tests/libqos tests/q= api-schema tests/tcg/xtensa tests/qemu-iotests" -DIRS=3D"$DIRS docs docs/interop fsdev" +DIRS=3D"$DIRS docs docs/interop fsdev scsi" DIRS=3D"$DIRS pc-bios/optionrom pc-bios/spapr-rtas pc-bios/s390-ccw" DIRS=3D"$DIRS roms/seabios roms/vgabios" DIRS=3D"$DIRS qapi-generated" diff --git a/docs/interop/pr-helper.rst b/docs/interop/pr-helper.rst new file mode 100644 index 0000000000..9f76d5bcf9 --- /dev/null +++ b/docs/interop/pr-helper.rst @@ -0,0 +1,83 @@ +.. + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Persistent reservation helper protocol +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +QEMU's SCSI passthrough devices, ``scsi-block`` and ``scsi-generic``, +can delegate implementation of persistent reservations to an external +(and typically privileged) program. Persistent Reservations allow +restricting access to block devices to specific initiators in a shared +storage setup. + +For a more detailed reference please refer the the SCSI Primary +Commands standard, specifically the section on Reservations and the +"PERSISTENT RESERVE IN" and "PERSISTENT RESERVE OUT" commands. + +This document describes the socket protocol used between QEMU's +``pr-manager-helper`` object and the external program. + +.. contents:: + +Connection and initialization +----------------------------- + +All data transmitted on the socket is big-endian. + +After connecting to the helper program's socket, the helper starts a simple +feature negotiation process by writing four bytes corresponding to +the features it exposes (``supported_features``). QEMU reads it, +then writes four bytes corresponding to the desired features of the +helper program (``requested_features``). + +If a bit is 1 in ``requested_features`` and 0 in ``supported_features``, +the corresponding feature is not supported by the helper and the connection +is closed. On the other hand, it is acceptable for a bit to be 0 in +``requested_features`` and 1 in ``supported_features``; in this case, +the helper will not enable the feature. + +Right now no feature is defined, so the two parties always write four +zero bytes. + +Command format +-------------- + +It is invalid to send multiple commands concurrently on the same +socket. It is however possible to connect multiple sockets to the +helper and send multiple commands to the helper for one or more +file descriptors. + +A command consists of a request and a response. A request consists +of a 16-byte SCSI CDB. A file descriptor must be passed to the helper +together with the SCSI CDB using ancillary data. + +The CDB has the following limitations: + +- the command (stored in the first byte) must be one of 0x5E + (PERSISTENT RESERVE IN) or 0x5F (PERSISTENT RESERVE OUT). + +- the allocation length (stored in bytes 7-8 of the CDB for PERSISTENT + RESERVE IN) or parameter list length (stored in bytes 5-8 of the CDB + for PERSISTENT RESERVE OUT) is limited to 8 KiB. + +For PERSISTENT RESERVE OUT, the parameter list is sent right after the +CDB. The length of the parameter list is taken from the CDB itself. + +The helper's reply has the following structure: + +- 4 bytes for the SCSI status + +- 4 bytes for the payload size (nonzero only for PERSISTENT RESERVE IN + and only if the SCSI status is 0x00, i.e. GOOD) + +- 96 bytes for the SCSI sense data + +- if the size is nonzero, the payload follows + +The sense data is always sent to keep the protocol simple, even though +it is only valid if the SCSI status is CHECK CONDITION (0x02). + +The payload size is always less than or equal to the allocation length +specified in the CDB for the PERSISTENT RESERVE IN command. + +If the protocol is violated, the helper closes the socket. diff --git a/docs/pr-manager.rst b/docs/pr-manager.rst index b6089fb57c..7107e59fb8 100644 --- a/docs/pr-manager.rst +++ b/docs/pr-manager.rst @@ -49,3 +49,36 @@ Alternatively, using ``-blockdev``:: -object pr-manager-helper,id=3Dhelper0,path=3D/var/run/qemu-pr-h= elper.sock -blockdev node-name=3Dhd,driver=3Draw,file.driver=3Dhost_device,= file.filename=3D/dev/sdb,file.pr-manager=3Dhelper0 -device scsi-block,drive=3Dhd + +---------------------------------- +Invoking :program:`qemu-pr-helper` +---------------------------------- + +QEMU provides an implementation of the persistent reservation helper, +called :program:`qemu-pr-helper`. The helper should be started as a +system service and supports the following option: + +-d, --daemon run in the background +-q, --quiet decrease verbosity +-f, --pidfile=3Dpath PID file when running as a daemon +-k, --socket=3Dpath path to the socket +-T, --trace=3Dtrace-opts tracing options + +By default, the socket and PID file are placed in the runtime state +directory, for example :file:`/var/run/qemu-pr-helper.sock` and +:file:`/var/run/qemu-pr-helper.pid`. The PID file is not created +unless :option:`-d` is passed too. + +:program:`qemu-pr-helper` can also use the systemd socket activation +protocol. In this case, the systemd socket unit should specify a +Unix stream socket, like this:: + + [Socket] + ListenStream=3D/var/run/qemu-pr-helper.sock + +After connecting to the socket, :program:`qemu-pr-helper`` can optionally = drop +root privileges, except for those capabilities that are needed for +its operation. To do this, add the following options: + +-u, --user=3Duser user to drop privileges to +-g, --group=3Dgroup group to drop privileges to diff --git a/scsi/pr-helper.h b/scsi/pr-helper.h new file mode 100644 index 0000000000..96c50a9e5f --- /dev/null +++ b/scsi/pr-helper.h @@ -0,0 +1,41 @@ +/* Definitions for QEMU's persistent reservation helper daemon + * + * Copyright (C) 2017 Red Hat, Inc. + * + * Author: + * Paolo Bonzini + * + * Permission is hereby granted, free of charge, to any person obtaining a= copy + * of this software and associated documentation files (the "Software"), to + * deal in the Software without restriction, including without limitation = the + * rights to use, copy, modify, merge, publish, distribute, sublicense, an= d/or + * sell copies of the Software, and to permit persons to whom the Software= is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included= in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS= OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL= THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEA= LINGS + * IN THE SOFTWARE. + */ +#ifndef QEMU_PR_HELPER_H +#define QEMU_PR_HELPER_H 1 + +#include + +#define PR_HELPER_CDB_SIZE 16 +#define PR_HELPER_SENSE_SIZE 96 +#define PR_HELPER_DATA_SIZE 8192 + +typedef struct PRHelperResponse { + int32_t result; + int32_t sz; + uint8_t sense[PR_HELPER_SENSE_SIZE]; +} PRHelperResponse; + +#endif diff --git a/scsi/qemu-pr-helper.c b/scsi/qemu-pr-helper.c new file mode 100644 index 0000000000..9662ede753 --- /dev/null +++ b/scsi/qemu-pr-helper.c @@ -0,0 +1,698 @@ +/* + * Privileged helper to handle persistent reservation commands for QEMU + * + * Copyright (C) 2017 Red Hat, Inc. + * + * Author: Paolo Bonzini + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; under version 2 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#include "qemu/osdep.h" +#include +#include +#include +#include + +#ifdef CONFIG_LIBCAP +#include +#endif +#include +#include + +#include "qapi/error.h" +#include "qemu-common.h" +#include "qemu/cutils.h" +#include "qemu/main-loop.h" +#include "qemu/error-report.h" +#include "qemu/config-file.h" +#include "qemu/bswap.h" +#include "qemu/log.h" +#include "qemu/systemd.h" +#include "qapi/util.h" +#include "qapi/qmp/qstring.h" +#include "io/channel-socket.h" +#include "trace/control.h" +#include "qemu-version.h" + +#include "block/aio.h" +#include "block/thread-pool.h" + +#include "scsi/constants.h" +#include "scsi/utils.h" +#include "pr-helper.h" + +#define PR_OUT_FIXED_PARAM_SIZE 24 + +static char *socket_path; +static char *pidfile; +static enum { RUNNING, TERMINATE, TERMINATING } state; +static QIOChannelSocket *server_ioc; +static int server_watch; +static int num_active_sockets =3D 1; +static int verbose; + +#ifdef CONFIG_LIBCAP +static int uid =3D -1; +static int gid =3D -1; +#endif + +static void usage(const char *name) +{ + (printf) ( +"Usage: %s [OPTIONS] FILE\n" +"Persistent Reservation helper program for QEMU\n" +"\n" +" -h, --help display this help and exit\n" +" -V, --version output version information and exit\n" +"\n" +" -d, --daemon run in the background\n" +" -f, --pidfile=3DPATH PID file when running as a daemon\n" +" (default '%s')\n" +" -k, --socket=3DPATH path to the unix socket\n" +" (default '%s')\n" +" -T, --trace [[enable=3D]][,events=3D][,file=3D]\n" +" specify tracing options\n" +#ifdef CONFIG_LIBCAP +" -u, --user=3DUSER user to drop privileges to\n" +" -g, --group=3DGROUP group to drop privileges to\n" +#endif +"\n" +QEMU_HELP_BOTTOM "\n" + , name, pidfile, socket_path); +} + +static void version(const char *name) +{ + printf( +"%s " QEMU_VERSION QEMU_PKGVERSION "\n" +"Written by Paolo Bonzini.\n" +"\n" +QEMU_COPYRIGHT "\n" +"This is free software; see the source for copying conditions. There is N= O\n" +"warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOS= E.\n" + , name); +} + +/* SG_IO support */ + +typedef struct PRHelperSGIOData { + int fd; + const uint8_t *cdb; + uint8_t *sense; + uint8_t *buf; + int sz; /* input/output */ + int dir; +} PRHelperSGIOData; + +static int do_sgio_worker(void *opaque) +{ + PRHelperSGIOData *data =3D opaque; + struct sg_io_hdr io_hdr; + int ret; + int status; + SCSISense sense_code; + + memset(data->sense, 0, PR_HELPER_SENSE_SIZE); + memset(&io_hdr, 0, sizeof(io_hdr)); + io_hdr.interface_id =3D 'S'; + io_hdr.cmd_len =3D PR_HELPER_CDB_SIZE; + io_hdr.cmdp =3D (uint8_t *)data->cdb; + io_hdr.sbp =3D data->sense; + io_hdr.mx_sb_len =3D PR_HELPER_SENSE_SIZE; + io_hdr.timeout =3D 1; + io_hdr.dxfer_direction =3D data->dir; + io_hdr.dxferp =3D (char *)data->buf; + io_hdr.dxfer_len =3D data->sz; + ret =3D ioctl(data->fd, SG_IO, &io_hdr); + status =3D sg_io_sense_from_errno(ret < 0 ? errno : 0, &io_hdr, + &sense_code); + if (status =3D=3D GOOD) { + data->sz -=3D io_hdr.resid; + } else { + data->sz =3D 0; + } + + if (status =3D=3D CHECK_CONDITION && + !(io_hdr.driver_status & SG_ERR_DRIVER_SENSE)) { + scsi_build_sense(data->sense, sense_code); + } + + return status; +} + +static int do_sgio(int fd, const uint8_t *cdb, uint8_t *sense, + uint8_t *buf, int *sz, int dir) +{ + ThreadPool *pool =3D aio_get_thread_pool(qemu_get_aio_context()); + int r; + + PRHelperSGIOData data =3D { + .fd =3D fd, + .cdb =3D cdb, + .sense =3D sense, + .buf =3D buf, + .sz =3D *sz, + .dir =3D dir, + }; + + r =3D thread_pool_submit_co(pool, do_sgio_worker, &data); + *sz =3D data.sz; + return r; +} + +static int do_pr_in(int fd, const uint8_t *cdb, uint8_t *sense, + uint8_t *data, int *resp_sz) +{ + return do_sgio(fd, cdb, sense, data, resp_sz, + SG_DXFER_FROM_DEV); +} + +static int do_pr_out(int fd, const uint8_t *cdb, uint8_t *sense, + const uint8_t *param, int sz) +{ + int resp_sz =3D sz; + return do_sgio(fd, cdb, sense, (uint8_t *)param, &resp_sz, + SG_DXFER_TO_DEV); +} + +/* Client */ + +typedef struct PRHelperClient { + QIOChannelSocket *ioc; + Coroutine *co; + int fd; + uint8_t data[PR_HELPER_DATA_SIZE]; +} PRHelperClient; + +typedef struct PRHelperRequest { + int fd; + size_t sz; + uint8_t cdb[PR_HELPER_CDB_SIZE]; +} PRHelperRequest; + +static int coroutine_fn prh_read(PRHelperClient *client, void *buf, int sz, + Error **errp) +{ + int ret =3D 0; + + while (sz > 0) { + int *fds =3D NULL; + size_t nfds =3D 0; + int i; + struct iovec iov; + ssize_t n_read; + + iov.iov_base =3D buf; + iov.iov_len =3D sz; + n_read =3D qio_channel_readv_full(QIO_CHANNEL(client->ioc), &iov, = 1, + &fds, &nfds, errp); + + if (n_read =3D=3D QIO_CHANNEL_ERR_BLOCK) { + qio_channel_yield(QIO_CHANNEL(client->ioc), G_IO_IN); + continue; + } + if (n_read <=3D 0) { + ret =3D n_read ? n_read : -1; + goto err; + } + + /* Stash one file descriptor per request. */ + if (nfds) { + bool too_many =3D false; + for (i =3D 0; i < nfds; i++) { + if (client->fd =3D=3D -1) { + client->fd =3D fds[i]; + } else { + close(fds[i]); + too_many =3D true; + } + } + g_free(fds); + if (too_many) { + ret =3D -1; + goto err; + } + } + + buf +=3D n_read; + sz -=3D n_read; + } + + return 0; + +err: + if (client->fd !=3D -1) { + close(client->fd); + client->fd =3D -1; + } + return ret; +} + +static int coroutine_fn prh_read_request(PRHelperClient *client, + PRHelperRequest *req, + PRHelperResponse *resp, Error **e= rrp) +{ + uint32_t sz; + + if (prh_read(client, req->cdb, sizeof(req->cdb), NULL) < 0) { + return -1; + } + + if (client->fd =3D=3D -1) { + error_setg(errp, "No file descriptor in request."); + return -1; + } + + if (req->cdb[0] !=3D PERSISTENT_RESERVE_OUT && + req->cdb[0] !=3D PERSISTENT_RESERVE_IN) { + error_setg(errp, "Invalid CDB, closing socket."); + goto out_close; + } + + sz =3D scsi_cdb_xfer(req->cdb); + if (sz > sizeof(client->data)) { + goto out_close; + } + + if (req->cdb[0] =3D=3D PERSISTENT_RESERVE_OUT) { + if (qio_channel_read_all(QIO_CHANNEL(client->ioc), + (char *)client->data, sz, + errp) < 0) { + goto out_close; + } + if ((fcntl(client->fd, F_GETFL) & O_ACCMODE) =3D=3D O_RDONLY) { + scsi_build_sense(resp->sense, SENSE_CODE(INVALID_OPCODE)); + sz =3D 0; + } else if (sz < PR_OUT_FIXED_PARAM_SIZE) { + /* Illegal request, Parameter list length error. This isn't f= atal; + * we have read the data, send an error without closing the so= cket. + */ + scsi_build_sense(resp->sense, SENSE_CODE(INVALID_PARAM_LEN)); + sz =3D 0; + } + if (sz =3D=3D 0) { + resp->result =3D CHECK_CONDITION; + close(client->fd); + client->fd =3D -1; + } + } + + req->fd =3D client->fd; + req->sz =3D sz; + client->fd =3D -1; + return sz; + +out_close: + close(client->fd); + client->fd =3D -1; + return -1; +} + +static int coroutine_fn prh_write_response(PRHelperClient *client, + PRHelperRequest *req, + PRHelperResponse *resp, Error *= *errp) +{ + ssize_t r; + size_t sz; + + if (req->cdb[0] =3D=3D PERSISTENT_RESERVE_IN && resp->result =3D=3D GO= OD) { + assert(resp->sz <=3D req->sz && resp->sz <=3D sizeof(client->data)= ); + } else { + assert(resp->sz =3D=3D 0); + } + + sz =3D resp->sz; + + resp->result =3D cpu_to_be32(resp->result); + resp->sz =3D cpu_to_be32(resp->sz); + r =3D qio_channel_write_all(QIO_CHANNEL(client->ioc), + (char *) resp, sizeof(*resp), errp); + if (r < 0) { + return r; + } + + r =3D qio_channel_write_all(QIO_CHANNEL(client->ioc), + (char *) client->data, + sz, errp); + return r < 0 ? r : 0; +} + +static void coroutine_fn prh_co_entry(void *opaque) +{ + PRHelperClient *client =3D opaque; + Error *local_err =3D NULL; + uint32_t flags; + int r; + + qio_channel_set_blocking(QIO_CHANNEL(client->ioc), + false, NULL); + qio_channel_attach_aio_context(QIO_CHANNEL(client->ioc), + qemu_get_aio_context()); + + /* A very simple negotiation for future extensibility. No features + * are defined so write 0. + */ + flags =3D cpu_to_be32(0); + r =3D qio_channel_write_all(QIO_CHANNEL(client->ioc), + (char *) &flags, sizeof(flags), NULL); + if (r < 0) { + goto out; + } + + r =3D qio_channel_read_all(QIO_CHANNEL(client->ioc), + (char *) &flags, sizeof(flags), NULL); + if (be32_to_cpu(flags) !=3D 0 || r < 0) { + goto out; + } + + while (atomic_read(&state) =3D=3D RUNNING) { + PRHelperRequest req; + PRHelperResponse resp; + int sz; + + sz =3D prh_read_request(client, &req, &resp, &local_err); + if (sz < 0) { + break; + } + + if (sz > 0) { + num_active_sockets++; + if (req.cdb[0] =3D=3D PERSISTENT_RESERVE_OUT) { + r =3D do_pr_out(req.fd, req.cdb, resp.sense, + client->data, sz); + resp.sz =3D 0; + } else { + resp.sz =3D sizeof(client->data); + r =3D do_pr_in(req.fd, req.cdb, resp.sense, + client->data, &resp.sz); + resp.sz =3D MIN(resp.sz, sz); + } + num_active_sockets--; + close(req.fd); + if (r =3D=3D -1) { + break; + } + resp.result =3D r; + } + + if (prh_write_response(client, &req, &resp, &local_err) < 0) { + break; + } + } + + if (local_err) { + if (verbose =3D=3D 0) { + error_free(local_err); + } else { + error_report_err(local_err); + } + } + +out: + qio_channel_detach_aio_context(QIO_CHANNEL(client->ioc)); + object_unref(OBJECT(client->ioc)); + g_free(client); +} + +static gboolean accept_client(QIOChannel *ioc, GIOCondition cond, gpointer= opaque) +{ + QIOChannelSocket *cioc; + PRHelperClient *prh; + + cioc =3D qio_channel_socket_accept(QIO_CHANNEL_SOCKET(ioc), + NULL); + if (!cioc) { + return TRUE; + } + + prh =3D g_new(PRHelperClient, 1); + prh->ioc =3D cioc; + prh->fd =3D -1; + prh->co =3D qemu_coroutine_create(prh_co_entry, prh); + qemu_coroutine_enter(prh->co); + + return TRUE; +} + + +/* + * Check socket parameters compatibility when socket activation is used. + */ +static const char *socket_activation_validate_opts(void) +{ + if (socket_path !=3D NULL) { + return "Unix socket can't be set when using socket activation"; + } + + return NULL; +} + +static void compute_default_paths(void) +{ + socket_path =3D qemu_get_local_state_pathname("run/qemu-pr-helper.sock= "); + pidfile =3D qemu_get_local_state_pathname("run/qemu-pr-helper.pid"); +} + +static void termsig_handler(int signum) +{ + atomic_cmpxchg(&state, RUNNING, TERMINATE); + qemu_notify_event(); +} + +static void close_server_socket(void) +{ + assert(server_ioc); + + g_source_remove(server_watch); + server_watch =3D -1; + object_unref(OBJECT(server_ioc)); + num_active_sockets--; +} + +#ifdef CONFIG_LIBCAP +static int drop_privileges(void) +{ + /* clear all capabilities */ + capng_clear(CAPNG_SELECT_BOTH); + + if (capng_update(CAPNG_ADD, CAPNG_EFFECTIVE | CAPNG_PERMITTED, + CAP_SYS_RAWIO) < 0) { + return -1; + } + + /* Change user/group id, retaining the capabilities. Because file des= criptors + * are passed via SCM_RIGHTS, we don't need supplementary groups (and = in + * fact the helper can run as "nobody"). + */ + if (capng_change_id(uid !=3D -1 ? uid : getuid(), + gid !=3D -1 ? gid : getgid(), + CAPNG_DROP_SUPP_GRP | CAPNG_CLEAR_BOUNDING)) { + return -1; + } + + return 0; +} +#endif + +int main(int argc, char **argv) +{ + const char *sopt =3D "hVk:fdT:u:g:q"; + struct option lopt[] =3D { + { "help", no_argument, NULL, 'h' }, + { "version", no_argument, NULL, 'V' }, + { "socket", required_argument, NULL, 'k' }, + { "pidfile", no_argument, NULL, 'f' }, + { "daemon", no_argument, NULL, 'd' }, + { "trace", required_argument, NULL, 'T' }, + { "user", required_argument, NULL, 'u' }, + { "group", required_argument, NULL, 'g' }, + { "quiet", no_argument, NULL, 'q' }, + { NULL, 0, NULL, 0 } + }; + int opt_ind =3D 0; + int quiet =3D 0; + char ch; + Error *local_err =3D NULL; + char *trace_file =3D NULL; + bool daemonize =3D false; + unsigned socket_activation; + + struct sigaction sa_sigterm; + memset(&sa_sigterm, 0, sizeof(sa_sigterm)); + sa_sigterm.sa_handler =3D termsig_handler; + sigaction(SIGTERM, &sa_sigterm, NULL); + sigaction(SIGINT, &sa_sigterm, NULL); + sigaction(SIGHUP, &sa_sigterm, NULL); + + signal(SIGPIPE, SIG_IGN); + + compute_default_paths(); + + module_call_init(MODULE_INIT_TRACE); + module_call_init(MODULE_INIT_QOM); + qemu_add_opts(&qemu_trace_opts); + qemu_init_exec_dir(argv[0]); + + while ((ch =3D getopt_long(argc, argv, sopt, lopt, &opt_ind)) !=3D -1)= { + switch (ch) { + case 'k': + socket_path =3D optarg; + if (socket_path[0] !=3D '/') { + error_report("socket path must be absolute"); + exit(EXIT_FAILURE); + } + break; + case 'f': + pidfile =3D optarg; + break; +#ifdef CONFIG_LIBCAP + case 'u': { + unsigned long res; + struct passwd *userinfo =3D getpwnam(optarg); + if (userinfo) { + uid =3D userinfo->pw_uid; + } else if (qemu_strtoul(optarg, NULL, 10, &res) =3D=3D 0 && + (uid_t)res =3D=3D res) { + uid =3D res; + } else { + error_report("invalid user '%s'", optarg); + exit(EXIT_FAILURE); + } + break; + } + case 'g': { + unsigned long res; + struct group *groupinfo =3D getgrnam(optarg); + if (groupinfo) { + gid =3D groupinfo->gr_gid; + } else if (qemu_strtoul(optarg, NULL, 10, &res) =3D=3D 0 && + (gid_t)res =3D=3D res) { + gid =3D res; + } else { + error_report("invalid group '%s'", optarg); + exit(EXIT_FAILURE); + } + break; + } +#else + case 'u': + case 'g': + error_report("-%c not supported by this %s", ch, argv[0]); + exit(1); +#endif + case 'd': + daemonize =3D true; + break; + case 'q': + quiet =3D 1; + break; + case 'T': + g_free(trace_file); + trace_file =3D trace_opt_parse(optarg); + break; + case 'V': + version(argv[0]); + exit(EXIT_SUCCESS); + break; + case 'h': + usage(argv[0]); + exit(EXIT_SUCCESS); + break; + case '?': + error_report("Try `%s --help' for more information.", argv[0]); + exit(EXIT_FAILURE); + } + } + + /* set verbosity */ + verbose =3D !quiet; + + if (!trace_init_backends()) { + exit(EXIT_FAILURE); + } + trace_init_file(trace_file); + qemu_set_log(LOG_TRACE); + + socket_activation =3D check_socket_activation(); + if (socket_activation =3D=3D 0) { + SocketAddress saddr =3D { + .type =3D SOCKET_ADDRESS_TYPE_UNIX, + .u.q_unix.path =3D g_strdup(socket_path) + }; + server_ioc =3D qio_channel_socket_new(); + if (qio_channel_socket_listen_sync(server_ioc, &saddr, &local_err)= < 0) { + object_unref(OBJECT(server_ioc)); + error_report_err(local_err); + return 1; + } + g_free(saddr.u.q_unix.path); + } else { + /* Using socket activation - check user didn't use -p etc. */ + const char *err_msg =3D socket_activation_validate_opts(); + if (err_msg !=3D NULL) { + error_report("%s", err_msg); + exit(EXIT_FAILURE); + } + + /* Can only listen on a single socket. */ + if (socket_activation > 1) { + error_report("%s does not support socket activation with LISTE= N_FDS > 1", + argv[0]); + exit(EXIT_FAILURE); + } + server_ioc =3D qio_channel_socket_new_fd(FIRST_SOCKET_ACTIVATION_F= D, + &local_err); + if (server_ioc =3D=3D NULL) { + error_report("Failed to use socket activation: %s", + error_get_pretty(local_err)); + exit(EXIT_FAILURE); + } + socket_path =3D NULL; + } + + if (qemu_init_main_loop(&local_err)) { + error_report_err(local_err); + exit(EXIT_FAILURE); + } + + server_watch =3D qio_channel_add_watch(QIO_CHANNEL(server_ioc), + G_IO_IN, + accept_client, + NULL, NULL); + +#ifdef CONFIG_LIBCAP + if (drop_privileges() < 0) { + error_report("Failed to drop privileges: %s", strerror(errno)); + exit(EXIT_FAILURE); + } +#endif + + if (daemonize) { + if (daemon(0, 0) < 0) { + error_report("Failed to daemonize: %s", strerror(errno)); + exit(EXIT_FAILURE); + } + } + + state =3D RUNNING; + do { + main_loop_wait(false); + if (state =3D=3D TERMINATE) { + state =3D TERMINATING; + close_server_socket(); + } + } while (num_active_sockets > 0); + + exit(EXIT_SUCCESS); +} --=20 2.13.5 From nobody Wed May 8 20:29:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150581681492214.659244303763671; Tue, 19 Sep 2017 03:26:54 -0700 (PDT) Received: from localhost ([::1]:41291 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFjw-0005Uf-Sv for importer@patchew.org; Tue, 19 Sep 2017 06:26:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45337) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFhz-00049s-J5 for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1duFhv-0004cN-Cl for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:51 -0400 Received: from mx1.redhat.com ([209.132.183.28]:48136) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1duFhp-0004XO-Dy; Tue, 19 Sep 2017 06:24:41 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8428A81DE6; Tue, 19 Sep 2017 10:24:40 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-117-61.ams2.redhat.com [10.36.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id 55B245D6A4; Tue, 19 Sep 2017 10:24:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8428A81DE6 Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=pbonzini@redhat.com From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 19 Sep 2017 12:24:33 +0200 Message-Id: <20170919102434.21147-4-pbonzini@redhat.com> In-Reply-To: <20170919102434.21147-1-pbonzini@redhat.com> References: <20170919102434.21147-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Tue, 19 Sep 2017 10:24:40 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 3/4] scsi: add multipath support to qemu-pr-helper X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Proper support of persistent reservation for multipath devices requires communication with the multipath daemon, so that the reservation is registered and applied when a path comes up. The device mapper utilities provide a library to do so; this patch makes qemu-pr-helper.c detect multipath devices and, when one is found, delegate the operation to libmpathpersist. Signed-off-by: Paolo Bonzini Reviewed-by: Stefan Hajnoczi --- v1->v2: moved dm_init/multipath_pr_init to this patch drop CAP_SYS_ADMIN if multipath not compiled in simplify buffer size handling in multipath PERSISTENT RESERVE IN block REGISTER AND MOVE operation for multipath PERSISTENT RESERVE = OUT fixed transport id handling in multipath PERSISTENT RESERVE OUT Makefile | 3 + configure | 57 ++++++++- docs/pr-manager.rst | 27 ++++ include/scsi/utils.h | 4 + scsi/qemu-pr-helper.c | 346 ++++++++++++++++++++++++++++++++++++++++++++++= +++- scsi/utils.c | 10 ++ 6 files changed, 441 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 51503ee8ad..39ec5112b6 100644 --- a/Makefile +++ b/Makefile @@ -377,6 +377,9 @@ fsdev/virtfs-proxy-helper$(EXESUF): fsdev/virtfs-proxy-= helper.o fsdev/9p-marshal fsdev/virtfs-proxy-helper$(EXESUF): LIBS +=3D -lcap =20 scsi/qemu-pr-helper$(EXESUF): scsi/qemu-pr-helper.o scsi/utils.o $(crypto-= obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS) +ifdef CONFIG_MPATH +scsi/qemu-pr-helper$(EXESUF): LIBS +=3D -ludev -lmultipath -lmpathpersist +endif =20 qemu-img-cmds.h: $(SRC_PATH)/qemu-img-cmds.hx $(SRC_PATH)/scripts/hxtool $(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -h < $< > $@,"GEN","$@= ") diff --git a/configure b/configure index fba27cf0fc..bea343c701 100755 --- a/configure +++ b/configure @@ -290,6 +290,7 @@ netmap=3D"no" sdl=3D"" sdlabi=3D"" virtfs=3D"" +mpath=3D"" vnc=3D"yes" sparse=3D"no" vde=3D"" @@ -936,6 +937,10 @@ for opt do ;; --enable-virtfs) virtfs=3D"yes" ;; + --disable-mpath) mpath=3D"no" + ;; + --enable-mpath) mpath=3D"yes" + ;; --disable-vnc) vnc=3D"no" ;; --enable-vnc) vnc=3D"yes" @@ -1479,6 +1484,7 @@ disabled with --disable-FEATURE, default is enabled i= f available: vnc-png PNG compression for VNC server cocoa Cocoa UI (Mac OS X only) virtfs VirtFS + mpath Multipath persistent reservation passthrough xen xen backend driver support xen-pci-passthrough brlapi BrlAPI (Braile) @@ -3300,6 +3306,29 @@ else fi =20 ########################################## +# libmpathpersist probe + +if test "$mpath" !=3D "no" ; then + cat > $TMPC < +#include +unsigned mpath_mx_alloc_len =3D 1024; +int logsink; +int main(void) { + struct udev *udev =3D udev_new(); + mpath_lib_init(udev); +} +EOF + if compile_prog "" "-ludev -lmultipath -lmpathpersist" ; then + mpathpersist=3Dyes + else + mpathpersist=3Dno + fi +else + mpathpersist=3Dno +fi + +########################################## # libcap probe =20 if test "$cap" !=3D "no" ; then @@ -5034,16 +5063,34 @@ if test "$want_tools" =3D "yes" ; then fi fi if test "$softmmu" =3D yes ; then - if test "$virtfs" !=3D no ; then - if test "$cap" =3D yes && test "$linux" =3D yes && test "$attr" =3D ye= s ; then + if test "$linux" =3D yes; then + if test "$virtfs" !=3D no && test "$cap" =3D yes && test "$attr" =3D y= es ; then virtfs=3Dyes tools=3D"$tools fsdev/virtfs-proxy-helper\$(EXESUF)" else if test "$virtfs" =3D yes; then - error_exit "VirtFS is supported only on Linux and requires libcap = devel and libattr devel" + error_exit "VirtFS requires libcap devel and libattr devel" fi virtfs=3Dno fi + if test "$mpath" !=3D no && test "$mpathpersist" =3D yes ; then + mpath=3Dyes + tools=3D"$tools mpath/qemu-mpath-helper\$(EXESUF)" + else + if test "$mpath" =3D yes; then + error_exit "Multipath requires libmpathpersist devel" + fi + mpath=3Dno + fi + else + if test "$virtfs" =3D yes; then + error_exit "VirtFS is supported only on Linux" + fi + virtfs=3Dno + if test "$mpath" =3D yes; then + error_exit "Multipath is supported only on Linux" + fi + mpath=3Dno fi fi =20 @@ -5289,6 +5336,7 @@ echo "Audio drivers $audio_drv_list" echo "Block whitelist (rw) $block_drv_rw_whitelist" echo "Block whitelist (ro) $block_drv_ro_whitelist" echo "VirtFS support $virtfs" +echo "Multipath support $mpath" echo "VNC support $vnc" if test "$vnc" =3D "yes" ; then echo "VNC SASL support $vnc_sasl" @@ -5732,6 +5780,9 @@ fi if test "$virtfs" =3D "yes" ; then echo "CONFIG_VIRTFS=3Dy" >> $config_host_mak fi +if test "$mpath" =3D "yes" ; then + echo "CONFIG_MPATH=3Dy" >> $config_host_mak +fi if test "$vhost_scsi" =3D "yes" ; then echo "CONFIG_VHOST_SCSI=3Dy" >> $config_host_mak fi diff --git a/docs/pr-manager.rst b/docs/pr-manager.rst index 7107e59fb8..9b1de198b1 100644 --- a/docs/pr-manager.rst +++ b/docs/pr-manager.rst @@ -60,6 +60,7 @@ system service and supports the following option: =20 -d, --daemon run in the background -q, --quiet decrease verbosity +-v, --verbose increase verbosity -f, --pidfile=3Dpath PID file when running as a daemon -k, --socket=3Dpath path to the socket -T, --trace=3Dtrace-opts tracing options @@ -82,3 +83,29 @@ its operation. To do this, add the following options: =20 -u, --user=3Duser user to drop privileges to -g, --group=3Dgroup group to drop privileges to + +--------------------------------------------- +Multipath devices and persistent reservations +--------------------------------------------- + +Proper support of persistent reservation for multipath devices requires +communication with the multipath daemon, so that the reservation is +registered and applied when a path is newly discovered or becomes online +again. :command:`qemu-pr-helper` can do this if the ``libmpathpersist`` +library was available on the system at build time. + +As of August 2017, a reservation key must be specified in ``multipath.conf= `` +for ``multipathd`` to check for persistent reservation for newly +discovered paths or reinstated paths. The attribute can be added +to the ``defaults`` section or the ``multipaths`` section; for example:: + + multipaths { + multipath { + wwid XXXXXXXXXXXXXXXX + alias yellow + reservation_key 0x123abc + } + } + +Linking :program:`qemu-pr-helper` to ``libmpathpersist`` does not impede +its usage on regular SCSI devices. diff --git a/include/scsi/utils.h b/include/scsi/utils.h index d301b31768..00a4bdb080 100644 --- a/include/scsi/utils.h +++ b/include/scsi/utils.h @@ -72,10 +72,14 @@ extern const struct SCSISense sense_code_IO_ERROR; extern const struct SCSISense sense_code_I_T_NEXUS_LOSS; /* Command aborted, Logical Unit failure */ extern const struct SCSISense sense_code_LUN_FAILURE; +/* Command aborted, LUN Communication failure */ +extern const struct SCSISense sense_code_LUN_COMM_FAILURE; /* Command aborted, Overlapped Commands Attempted */ extern const struct SCSISense sense_code_OVERLAPPED_COMMANDS; /* LUN not ready, Capacity data has changed */ extern const struct SCSISense sense_code_CAPACITY_CHANGED; +/* Unit attention, SCSI bus reset */ +extern const struct SCSISense sense_code_SCSI_BUS_RESET; /* LUN not ready, Medium not present */ extern const struct SCSISense sense_code_UNIT_ATTENTION_NO_MEDIUM; /* Unit attention, Power on, reset or bus device reset occurred */ diff --git a/scsi/qemu-pr-helper.c b/scsi/qemu-pr-helper.c index 9662ede753..3f08ab12b4 100644 --- a/scsi/qemu-pr-helper.c +++ b/scsi/qemu-pr-helper.c @@ -30,6 +30,12 @@ #include #include =20 +#ifdef CONFIG_MPATH +#include +#include +#include +#endif + #include "qapi/error.h" #include "qemu-common.h" #include "qemu/cutils.h" @@ -60,6 +66,7 @@ static enum { RUNNING, TERMINATE, TERMINATING } state; static QIOChannelSocket *server_ioc; static int server_watch; static int num_active_sockets =3D 1; +static int noisy; static int verbose; =20 #ifdef CONFIG_LIBCAP @@ -171,9 +178,316 @@ static int do_sgio(int fd, const uint8_t *cdb, uint8_= t *sense, return r; } =20 +/* Device mapper interface */ + +#ifdef CONFIG_MPATH +#define CONTROL_PATH "/dev/mapper/control" + +typedef struct DMData { + struct dm_ioctl dm; + uint8_t data[1024]; +} DMData; + +static int control_fd; + +static void *dm_ioctl(int ioc, struct dm_ioctl *dm) +{ + static DMData d; + memcpy(&d.dm, dm, sizeof(d.dm)); + QEMU_BUILD_BUG_ON(sizeof(d.data) < sizeof(struct dm_target_spec)); + + d.dm.version[0] =3D DM_VERSION_MAJOR; + d.dm.version[1] =3D 0; + d.dm.version[2] =3D 0; + d.dm.data_size =3D 1024; + d.dm.data_start =3D offsetof(DMData, data); + if (ioctl(control_fd, ioc, &d) < 0) { + return NULL; + } + memcpy(dm, &d.dm, sizeof(d.dm)); + return &d.data; +} + +static void *dm_dev_ioctl(int fd, int ioc, struct dm_ioctl *dm) +{ + struct stat st; + int r; + + r =3D fstat(fd, &st); + if (r < 0) { + perror("fstat"); + exit(1); + } + + dm->dev =3D st.st_rdev; + return dm_ioctl(ioc, dm); +} + +static void dm_init(void) +{ + control_fd =3D open(CONTROL_PATH, O_RDWR); + if (control_fd < 0) { + perror("Cannot open " CONTROL_PATH); + exit(1); + } + struct dm_ioctl dm =3D { 0 }; + if (!dm_ioctl(DM_VERSION, &dm)) { + perror("ioctl"); + exit(1); + } + if (dm.version[0] !=3D DM_VERSION_MAJOR) { + fprintf(stderr, "Unsupported device mapper interface"); + exit(1); + } +} + +/* Variables required by libmultipath and libmpathpersist. */ +QEMU_BUILD_BUG_ON(PR_HELPER_DATA_SIZE > MPATH_MAX_PARAM_LEN); +unsigned mpath_mx_alloc_len =3D PR_HELPER_DATA_SIZE; +int logsink; + +static void multipath_pr_init(void) +{ + static struct udev *udev; + + udev =3D udev_new(); + mpath_lib_init(udev); +} + +static int is_mpath(int fd) +{ + struct dm_ioctl dm =3D { .flags =3D DM_NOFLUSH_FLAG }; + struct dm_target_spec *tgt; + + tgt =3D dm_dev_ioctl(fd, DM_TABLE_STATUS, &dm); + if (!tgt) { + if (errno =3D=3D ENXIO) { + return 0; + } + perror("ioctl"); + exit(EXIT_FAILURE); + } + return !strncmp(tgt->target_type, "multipath", DM_MAX_TYPE_NAME); +} + +static int mpath_reconstruct_sense(int fd, int r, uint8_t *sense) +{ + switch (r) { + case MPATH_PR_SUCCESS: + return GOOD; + case MPATH_PR_SENSE_NOT_READY: + case MPATH_PR_SENSE_MEDIUM_ERROR: + case MPATH_PR_SENSE_HARDWARE_ERROR: + case MPATH_PR_SENSE_ABORTED_COMMAND: + { + /* libmpathpersist ate the exact sense. Try to find it by + * issuing TEST UNIT READY. + */ + uint8_t cdb[6] =3D { TEST_UNIT_READY }; + int sz =3D 0; + return do_sgio(fd, cdb, sense, NULL, &sz, SG_DXFER_NONE); + } + + case MPATH_PR_SENSE_UNIT_ATTENTION: + /* Congratulations libmpathpersist, you ruined the Unit Attention.= .. + * Return a heavyweight one. + */ + scsi_build_sense(sense, SENSE_CODE(SCSI_BUS_RESET)); + return CHECK_CONDITION; + case MPATH_PR_SENSE_INVALID_OP: + /* Only one valid sense. */ + scsi_build_sense(sense, SENSE_CODE(INVALID_OPCODE)); + return CHECK_CONDITION; + case MPATH_PR_ILLEGAL_REQ: + /* Guess. */ + scsi_build_sense(sense, SENSE_CODE(INVALID_PARAM)); + return CHECK_CONDITION; + case MPATH_PR_NO_SENSE: + scsi_build_sense(sense, SENSE_CODE(NO_SENSE)); + return CHECK_CONDITION; + + case MPATH_PR_RESERV_CONFLICT: + return RESERVATION_CONFLICT; + + case MPATH_PR_OTHER: + default: + scsi_build_sense(sense, SENSE_CODE(LUN_COMM_FAILURE)); + return CHECK_CONDITION; + } +} + +static int multipath_pr_in(int fd, const uint8_t *cdb, uint8_t *sense, + uint8_t *data, int sz) +{ + int rq_servact =3D cdb[1]; + struct prin_resp resp; + size_t written; + int r; + + switch (rq_servact) { + case MPATH_PRIN_RKEY_SA: + case MPATH_PRIN_RRES_SA: + case MPATH_PRIN_RCAP_SA: + break; + case MPATH_PRIN_RFSTAT_SA: + /* Nobody implements it anyway, so bail out. */ + default: + /* Cannot parse any other output. */ + scsi_build_sense(sense, SENSE_CODE(INVALID_FIELD)); + return CHECK_CONDITION; + } + + r =3D mpath_persistent_reserve_in(fd, rq_servact, &resp, noisy, verbos= e); + if (r =3D=3D MPATH_PR_SUCCESS) { + switch (rq_servact) { + case MPATH_PRIN_RKEY_SA: + case MPATH_PRIN_RRES_SA: { + struct prin_readdescr *out =3D &resp.prin_descriptor.prin_read= keys; + assert(sz >=3D 8); + written =3D MIN(out->additional_length + 8, sz); + stl_be_p(&data[0], out->prgeneration); + stl_be_p(&data[4], out->additional_length); + memcpy(&data[8], out->key_list, written - 8); + break; + } + case MPATH_PRIN_RCAP_SA: { + struct prin_capdescr *out =3D &resp.prin_descriptor.prin_readc= ap; + assert(sz >=3D 6); + written =3D 6; + stw_be_p(&data[0], out->length); + data[2] =3D out->flags[0]; + data[3] =3D out->flags[1]; + stw_be_p(&data[4], out->pr_type_mask); + break; + } + default: + scsi_build_sense(sense, SENSE_CODE(INVALID_OPCODE)); + return CHECK_CONDITION; + } + assert(written <=3D sz); + memset(data + written, 0, sz - written); + } + + return mpath_reconstruct_sense(fd, r, sense); +} + +static int multipath_pr_out(int fd, const uint8_t *cdb, uint8_t *sense, + const uint8_t *param, int sz) +{ + int rq_servact =3D cdb[1]; + int rq_scope =3D cdb[2] >> 4; + int rq_type =3D cdb[2] & 0xf; + struct prout_param_descriptor paramp; + char transportids[PR_HELPER_DATA_SIZE]; + int r; + + switch (rq_servact) { + case MPATH_PROUT_REG_SA: + case MPATH_PROUT_RES_SA: + case MPATH_PROUT_REL_SA: + case MPATH_PROUT_CLEAR_SA: + case MPATH_PROUT_PREE_SA: + case MPATH_PROUT_PREE_AB_SA: + case MPATH_PROUT_REG_IGN_SA: + break; + case MPATH_PROUT_REG_MOV_SA: + /* Not supported by struct prout_param_descriptor. */ + default: + /* Cannot parse any other input. */ + scsi_build_sense(sense, SENSE_CODE(INVALID_FIELD)); + return CHECK_CONDITION; + } + + /* Convert input data, especially transport IDs, to the structs + * used by libmpathpersist (which, of course, will immediately + * do the opposite). + */ + memset(¶mp, 0, sizeof(paramp)); + memcpy(¶mp.key, ¶m[0], 8); + memcpy(¶mp.sa_key, ¶m[8], 8); + paramp.sa_flags =3D param[10]; + if (sz > PR_OUT_FIXED_PARAM_SIZE) { + size_t transportid_len; + int i, j; + if (sz < PR_OUT_FIXED_PARAM_SIZE + 4) { + scsi_build_sense(sense, SENSE_CODE(INVALID_PARAM_LEN)); + return CHECK_CONDITION; + } + transportid_len =3D ldl_be_p(¶m[24]) + PR_OUT_FIXED_PARAM_SIZE= + 4; + if (transportid_len > sz) { + scsi_build_sense(sense, SENSE_CODE(INVALID_PARAM)); + return CHECK_CONDITION; + } + for (i =3D PR_OUT_FIXED_PARAM_SIZE + 4, j =3D 0; i < transportid_l= en; ) { + struct transportid *id =3D (struct transportid *) &transportid= s[j]; + int len; + + id->format_code =3D param[i] & 0xc0; + id->protocol_id =3D param[i] & 0x0f; + switch (param[i] & 0xcf) { + case 0: + /* FC transport. */ + if (i + 24 > transportid_len) { + goto illegal_req; + } + memcpy(id->n_port_name, ¶m[i + 8], 8); + j +=3D offsetof(struct transportid, n_port_name[8]); + i +=3D 24; + break; + case 3: + case 0x43: + /* iSCSI transport. */ + len =3D lduw_be_p(¶m[i + 2]); + if (len > 252 || (len & 3) || i + len + 4 > transportid_le= n) { + /* For format code 00, the standard says the maximum i= s 223 + * plus the NUL terminator. For format code 01 there = is no + * maximum length, but libmpathpersist ignores the fir= st + * byte of id->iscsi_name so our maximum is 252. + */ + goto illegal_req; + } + if (memchr(¶m[i + 4], 0, len) =3D=3D NULL) { + goto illegal_req; + } + memcpy(id->iscsi_name, ¶m[i + 2], len + 2); + j +=3D offsetof(struct transportid, iscsi_name[len + 2]); + i +=3D len + 4; + break; + case 6: + /* SAS transport. */ + if (i + 24 > transportid_len) { + goto illegal_req; + } + memcpy(id->sas_address, ¶m[i + 4], 8); + j +=3D offsetof(struct transportid, sas_address[8]); + i +=3D 24; + break; + default: + illegal_req: + scsi_build_sense(sense, SENSE_CODE(INVALID_PARAM)); + return CHECK_CONDITION; + } + + paramp.trnptid_list[paramp.num_transportid++] =3D id; + } + } + + r =3D mpath_persistent_reserve_out(fd, rq_servact, rq_scope, rq_type, + ¶mp, noisy, verbose); + return mpath_reconstruct_sense(fd, r, sense); +} +#endif + static int do_pr_in(int fd, const uint8_t *cdb, uint8_t *sense, uint8_t *data, int *resp_sz) { +#ifdef CONFIG_MPATH + if (is_mpath(fd)) { + /* multipath_pr_in fills the whole input buffer. */ + return multipath_pr_in(fd, cdb, sense, data, *resp_sz); + } +#endif + return do_sgio(fd, cdb, sense, data, resp_sz, SG_DXFER_FROM_DEV); } @@ -181,7 +495,14 @@ static int do_pr_in(int fd, const uint8_t *cdb, uint8_= t *sense, static int do_pr_out(int fd, const uint8_t *cdb, uint8_t *sense, const uint8_t *param, int sz) { - int resp_sz =3D sz; + int resp_sz; +#ifdef CONFIG_MPATH + if (is_mpath(fd)) { + return multipath_pr_out(fd, cdb, sense, param, sz); + } +#endif + + resp_sz =3D sz; return do_sgio(fd, cdb, sense, (uint8_t *)param, &resp_sz, SG_DXFER_TO_DEV); } @@ -491,6 +812,14 @@ static int drop_privileges(void) return -1; } =20 +#ifdef CONFIG_MPATH + /* For /dev/mapper/control ioctls */ + if (capng_update(CAPNG_ADD, CAPNG_EFFECTIVE | CAPNG_PERMITTED, + CAP_SYS_ADMIN) < 0) { + return -1; + } +#endif + /* Change user/group id, retaining the capabilities. Because file des= criptors * are passed via SCM_RIGHTS, we don't need supplementary groups (and = in * fact the helper can run as "nobody"). @@ -507,7 +836,7 @@ static int drop_privileges(void) =20 int main(int argc, char **argv) { - const char *sopt =3D "hVk:fdT:u:g:q"; + const char *sopt =3D "hVk:fdT:u:g:vq"; struct option lopt[] =3D { { "help", no_argument, NULL, 'h' }, { "version", no_argument, NULL, 'V' }, @@ -517,10 +846,12 @@ int main(int argc, char **argv) { "trace", required_argument, NULL, 'T' }, { "user", required_argument, NULL, 'u' }, { "group", required_argument, NULL, 'g' }, + { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, { NULL, 0, NULL, 0 } }; int opt_ind =3D 0; + int loglevel =3D 1; int quiet =3D 0; char ch; Error *local_err =3D NULL; @@ -597,6 +928,9 @@ int main(int argc, char **argv) case 'q': quiet =3D 1; break; + case 'v': + ++loglevel; + break; case 'T': g_free(trace_file); trace_file =3D trace_opt_parse(optarg); @@ -616,7 +950,8 @@ int main(int argc, char **argv) } =20 /* set verbosity */ - verbose =3D !quiet; + noisy =3D !quiet && (loglevel >=3D 3); + verbose =3D quiet ? 0 : MIN(loglevel, 3); =20 if (!trace_init_backends()) { exit(EXIT_FAILURE); @@ -624,6 +959,11 @@ int main(int argc, char **argv) trace_init_file(trace_file); qemu_set_log(LOG_TRACE); =20 +#ifdef CONFIG_MPATH + dm_init(); + multipath_pr_init(); +#endif + socket_activation =3D check_socket_activation(); if (socket_activation =3D=3D 0) { SocketAddress saddr =3D { diff --git a/scsi/utils.c b/scsi/utils.c index fab60bdf20..5684951b12 100644 --- a/scsi/utils.c +++ b/scsi/utils.c @@ -206,6 +206,11 @@ const struct SCSISense sense_code_OVERLAPPED_COMMANDS = =3D { .key =3D ABORTED_COMMAND, .asc =3D 0x4e, .ascq =3D 0x00 }; =20 +/* Command aborted, LUN Communication Failure */ +const struct SCSISense sense_code_LUN_COMM_FAILURE =3D { + .key =3D ABORTED_COMMAND, .asc =3D 0x08, .ascq =3D 0x00 +}; + /* Unit attention, Capacity data has changed */ const struct SCSISense sense_code_CAPACITY_CHANGED =3D { .key =3D UNIT_ATTENTION, .asc =3D 0x2a, .ascq =3D 0x09 @@ -216,6 +221,11 @@ const struct SCSISense sense_code_RESET =3D { .key =3D UNIT_ATTENTION, .asc =3D 0x29, .ascq =3D 0x00 }; =20 +/* Unit attention, SCSI bus reset */ +const struct SCSISense sense_code_SCSI_BUS_RESET =3D { + .key =3D UNIT_ATTENTION, .asc =3D 0x29, .ascq =3D 0x02 +}; + /* Unit attention, No medium */ const struct SCSISense sense_code_UNIT_ATTENTION_NO_MEDIUM =3D { .key =3D UNIT_ATTENTION, .asc =3D 0x3a, .ascq =3D 0x00 --=20 2.13.5 From nobody Wed May 8 20:29:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1505816925958836.8609549774847; Tue, 19 Sep 2017 03:28:45 -0700 (PDT) Received: from localhost ([::1]:41311 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFll-0007Gc-8A for importer@patchew.org; Tue, 19 Sep 2017 06:28:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45358) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duFi1-0004BN-6X for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1duFhz-0004f7-Nl for qemu-devel@nongnu.org; Tue, 19 Sep 2017 06:24:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60956) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1duFhq-0004Y5-Ig; Tue, 19 Sep 2017 06:24:42 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A498E5AFED; Tue, 19 Sep 2017 10:24:41 +0000 (UTC) Received: from donizetti.redhat.com (ovpn-117-61.ams2.redhat.com [10.36.117.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id D9F165D6A4; Tue, 19 Sep 2017 10:24:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A498E5AFED Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=pbonzini@redhat.com From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 19 Sep 2017 12:24:34 +0200 Message-Id: <20170919102434.21147-5-pbonzini@redhat.com> In-Reply-To: <20170919102434.21147-1-pbonzini@redhat.com> References: <20170919102434.21147-1-pbonzini@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Tue, 19 Sep 2017 10:24:41 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 4/4] scsi: add persistent reservation manager using qemu-pr-helper X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This adds a concrete subclass of pr-manager that talks to qemu-pr-helper. Signed-off-by: Paolo Bonzini Reviewed-by: Stefan Hajnoczi --- v1->v2: fixed string property double-free fixed/cleaned up error handling handle buffer underrun scsi/Makefile.objs | 2 +- scsi/pr-manager-helper.c | 302 +++++++++++++++++++++++++++++++++++++++++++= ++++ 2 files changed, 303 insertions(+), 1 deletion(-) create mode 100644 scsi/pr-manager-helper.c diff --git a/scsi/Makefile.objs b/scsi/Makefile.objs index 5496d2ae6a..4d25e476cf 100644 --- a/scsi/Makefile.objs +++ b/scsi/Makefile.objs @@ -1,3 +1,3 @@ block-obj-y +=3D utils.o =20 -block-obj-$(CONFIG_LINUX) +=3D pr-manager.o +block-obj-$(CONFIG_LINUX) +=3D pr-manager.o pr-manager-helper.o diff --git a/scsi/pr-manager-helper.c b/scsi/pr-manager-helper.c new file mode 100644 index 0000000000..063fd35dee --- /dev/null +++ b/scsi/pr-manager-helper.c @@ -0,0 +1,302 @@ +/* + * Persistent reservation manager that talks to qemu-pr-helper + * + * Copyright (c) 2017 Red Hat, Inc. + * + * Author: Paolo Bonzini + * + * This code is licensed under the LGPL v2.1 or later. + * + */ + +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "scsi/constants.h" +#include "scsi/pr-manager.h" +#include "scsi/utils.h" +#include "io/channel.h" +#include "io/channel-socket.h" +#include "pr-helper.h" + +#include + +#define PR_MAX_RECONNECT_ATTEMPTS 5 + +#define TYPE_PR_MANAGER_HELPER "pr-manager-helper" + +#define PR_MANAGER_HELPER(obj) \ + OBJECT_CHECK(PRManagerHelper, (obj), \ + TYPE_PR_MANAGER_HELPER) + +typedef struct PRManagerHelper { + /* */ + PRManager parent; + + char *path; + + QemuMutex lock; + QIOChannel *ioc; +} PRManagerHelper; + +/* Called with lock held. */ +static int pr_manager_helper_read(PRManagerHelper *pr_mgr, + void *buf, int sz, Error **errp) +{ + ssize_t r =3D qio_channel_read_all(pr_mgr->ioc, buf, sz, errp); + + if (r < 0) { + object_unref(OBJECT(pr_mgr->ioc)); + pr_mgr->ioc =3D NULL; + return -EINVAL; + } + + return 0; +} + +/* Called with lock held. */ +static int pr_manager_helper_write(PRManagerHelper *pr_mgr, + int fd, + const void *buf, int sz, Error **errp) +{ + size_t nfds =3D (fd !=3D -1); + while (sz > 0) { + struct iovec iov; + ssize_t n_written; + + iov.iov_base =3D (void *)buf; + iov.iov_len =3D sz; + n_written =3D qio_channel_writev_full(QIO_CHANNEL(pr_mgr->ioc), &i= ov, 1, + nfds ? &fd : NULL, nfds, errp); + + if (n_written <=3D 0) { + assert(n_written !=3D QIO_CHANNEL_ERR_BLOCK); + object_unref(OBJECT(pr_mgr->ioc)); + return n_written < 0 ? -EINVAL : 0; + } + + nfds =3D 0; + buf +=3D n_written; + sz -=3D n_written; + } + + return 0; +} + +/* Called with lock held. */ +static int pr_manager_helper_initialize(PRManagerHelper *pr_mgr, + Error **errp) +{ + char *path =3D g_strdup(pr_mgr->path); + SocketAddress saddr =3D { + .type =3D SOCKET_ADDRESS_TYPE_UNIX, + .u.q_unix.path =3D path + }; + QIOChannelSocket *sioc =3D qio_channel_socket_new(); + Error *local_err =3D NULL; + + uint32_t flags; + int r; + + assert(!pr_mgr->ioc); + qio_channel_set_name(QIO_CHANNEL(sioc), "pr-manager-helper"); + qio_channel_socket_connect_sync(sioc, + &saddr, + &local_err); + g_free(path); + if (local_err) { + object_unref(OBJECT(sioc)); + error_propagate(errp, local_err); + return -ENOTCONN; + } + + qio_channel_set_delay(QIO_CHANNEL(sioc), false); + pr_mgr->ioc =3D QIO_CHANNEL(sioc); + + /* A simple feature negotation protocol, even though there is + * no optional feature right now. + */ + r =3D pr_manager_helper_read(pr_mgr, &flags, sizeof(flags), errp); + if (r < 0) { + goto out_close; + } + + flags =3D 0; + r =3D pr_manager_helper_write(pr_mgr, -1, &flags, sizeof(flags), errp); + if (r < 0) { + goto out_close; + } + + return 0; + +out_close: + object_unref(OBJECT(pr_mgr->ioc)); + pr_mgr->ioc =3D NULL; + return r; +} + +static int pr_manager_helper_run(PRManager *p, + int fd, struct sg_io_hdr *io_hdr) +{ + PRManagerHelper *pr_mgr =3D PR_MANAGER_HELPER(p); + + uint32_t len; + PRHelperResponse resp; + int ret; + int expected_dir; + int attempts; + uint8_t cdb[PR_HELPER_CDB_SIZE] =3D { 0 }; + + if (!io_hdr->cmd_len || io_hdr->cmd_len > PR_HELPER_CDB_SIZE) { + return -EINVAL; + } + + memcpy(cdb, io_hdr->cmdp, io_hdr->cmd_len); + assert(cdb[0] =3D=3D PERSISTENT_RESERVE_OUT || cdb[0] =3D=3D PERSISTEN= T_RESERVE_IN); + expected_dir =3D + (cdb[0] =3D=3D PERSISTENT_RESERVE_OUT ? SG_DXFER_TO_DEV : SG_DXFER= _FROM_DEV); + if (io_hdr->dxfer_direction !=3D expected_dir) { + return -EINVAL; + } + + len =3D scsi_cdb_xfer(cdb); + if (io_hdr->dxfer_len < len || len > PR_HELPER_DATA_SIZE) { + return -EINVAL; + } + + qemu_mutex_lock(&pr_mgr->lock); + + /* Try to reconnect while sending the CDB. */ + for (attempts =3D 0; attempts < PR_MAX_RECONNECT_ATTEMPTS; attempts++)= { + if (!pr_mgr->ioc) { + ret =3D pr_manager_helper_initialize(pr_mgr, NULL); + if (ret < 0) { + qemu_mutex_unlock(&pr_mgr->lock); + g_usleep(G_USEC_PER_SEC); + qemu_mutex_lock(&pr_mgr->lock); + continue; + } + } + + ret =3D pr_manager_helper_write(pr_mgr, fd, cdb, ARRAY_SIZE(cdb), = NULL); + if (ret >=3D 0) { + break; + } + } + if (ret < 0) { + goto out; + } + + /* After sending the CDB, any communications failure causes the + * command to fail. The failure is transient, retrying the command + * will invoke pr_manager_helper_initialize again. + */ + if (expected_dir =3D=3D SG_DXFER_TO_DEV) { + io_hdr->resid =3D io_hdr->dxfer_len - len; + ret =3D pr_manager_helper_write(pr_mgr, -1, io_hdr->dxferp, len, N= ULL); + if (ret < 0) { + goto out; + } + } + ret =3D pr_manager_helper_read(pr_mgr, &resp, sizeof(resp), NULL); + if (ret < 0) { + goto out; + } + + resp.result =3D be32_to_cpu(resp.result); + resp.sz =3D be32_to_cpu(resp.sz); + if (io_hdr->dxfer_direction =3D=3D SG_DXFER_FROM_DEV) { + assert(resp.sz <=3D io_hdr->dxfer_len); + ret =3D pr_manager_helper_read(pr_mgr, io_hdr->dxferp, resp.sz, NU= LL); + if (ret < 0) { + goto out; + } + io_hdr->resid =3D io_hdr->dxfer_len - resp.sz; + } else { + assert(resp.sz =3D=3D 0); + } + + io_hdr->status =3D resp.result; + if (resp.result =3D=3D CHECK_CONDITION) { + io_hdr->driver_status =3D SG_ERR_DRIVER_SENSE; + io_hdr->sb_len_wr =3D MIN(io_hdr->mx_sb_len, PR_HELPER_SENSE_SIZE); + memcpy(io_hdr->sbp, resp.sense, io_hdr->sb_len_wr); + } + +out: + if (ret < 0) { + int sense_len =3D scsi_build_sense(io_hdr->sbp, + SENSE_CODE(LUN_COMM_FAILURE)); + io_hdr->driver_status =3D SG_ERR_DRIVER_SENSE; + io_hdr->sb_len_wr =3D MIN(io_hdr->mx_sb_len, sense_len); + io_hdr->status =3D CHECK_CONDITION; + } + qemu_mutex_unlock(&pr_mgr->lock); + return ret; +} + +static void pr_manager_helper_complete(UserCreatable *uc, Error **errp) +{ + PRManagerHelper *pr_mgr =3D PR_MANAGER_HELPER(uc); + + qemu_mutex_lock(&pr_mgr->lock); + pr_manager_helper_initialize(pr_mgr, errp); + qemu_mutex_unlock(&pr_mgr->lock); +} + +static char *get_path(Object *obj, Error **errp) +{ + PRManagerHelper *pr_mgr =3D PR_MANAGER_HELPER(obj); + + return g_strdup(pr_mgr->path); +} + +static void set_path(Object *obj, const char *str, Error **errp) +{ + PRManagerHelper *pr_mgr =3D PR_MANAGER_HELPER(obj); + + g_free(pr_mgr->path); + pr_mgr->path =3D g_strdup(str); +} + +static void pr_manager_helper_instance_finalize(Object *obj) +{ + PRManagerHelper *pr_mgr =3D PR_MANAGER_HELPER(obj); + + object_unref(OBJECT(pr_mgr->ioc)); + qemu_mutex_destroy(&pr_mgr->lock); +} + +static void pr_manager_helper_instance_init(Object *obj) +{ + PRManagerHelper *pr_mgr =3D PR_MANAGER_HELPER(obj); + + qemu_mutex_init(&pr_mgr->lock); +} + +static void pr_manager_helper_class_init(ObjectClass *klass, + void *class_data G_GNUC_UNUSED) +{ + PRManagerClass *prmgr_klass =3D PR_MANAGER_CLASS(klass); + UserCreatableClass *uc_klass =3D USER_CREATABLE_CLASS(klass); + + object_class_property_add_str(klass, "path", get_path, set_path, + &error_abort); + uc_klass->complete =3D pr_manager_helper_complete; + prmgr_klass->run =3D pr_manager_helper_run; +} + +static const TypeInfo pr_manager_helper_info =3D { + .parent =3D TYPE_PR_MANAGER, + .name =3D TYPE_PR_MANAGER_HELPER, + .instance_size =3D sizeof(PRManagerHelper), + .instance_init =3D pr_manager_helper_instance_init, + .instance_finalize =3D pr_manager_helper_instance_finalize, + .class_init =3D pr_manager_helper_class_init, +}; + +static void pr_manager_helper_register_types(void) +{ + type_register_static(&pr_manager_helper_info); +} + +type_init(pr_manager_helper_register_types); --=20 2.13.5