From nobody Sun Apr 28 13:46:23 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; dkim=fail spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1491277840418234.1845495245716; Mon, 3 Apr 2017 20:50:40 -0700 (PDT) Received: from localhost ([::1]:33813 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cvFUM-0006md-QV for importer@patchew.org; Mon, 03 Apr 2017 23:50:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34030) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cvFSl-0005l4-Tp for qemu-devel@nongnu.org; Mon, 03 Apr 2017 23:49:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cvFSi-0005rf-W5 for qemu-devel@nongnu.org; Mon, 03 Apr 2017 23:48:59 -0400 Received: from mail-pg0-x242.google.com ([2607:f8b0:400e:c05::242]:32795) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cvFSi-0005qP-LZ for qemu-devel@nongnu.org; Mon, 03 Apr 2017 23:48:56 -0400 Received: by mail-pg0-x242.google.com with SMTP id 79so34208454pgf.0 for ; Mon, 03 Apr 2017 20:48:56 -0700 (PDT) Received: from localhost.localdomain.localdomain ([172.56.38.27]) by smtp.gmail.com with ESMTPSA id u198sm28331074pgb.45.2017.04.03.20.48.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Apr 2017 20:48:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QBduXCTHUB2Uc/8dJN06haG0aRbv01Vs5/o4H+Nx9GI=; b=LscHYwUTOEWmlDY3Q0IMiVR+pcoNxe0LsYZj6iqsucIDSgvAKXb2GSVMNPUcW7bCYo oP0lXvPhAutbxcSKQI9Vqd8ANvA/45ZWHKF7Y+Ywf3D3GSjtV8wHUECNpwMlaUdK7+i4 +xTCS7+g08LJSCpsTiZ/ds0boR/Zta56s0BdGTFeSA4QqXO8463SPDNtXoJoq63V1WNz tE9GlGrxqBADihXceBH60N5kMEMU9DQZcG8mfj+20r3EnfoUebftXc6nF6f6wcKTWt3m rVL5lpK6a4yExEJK2Y5gRkTn41LBk60hOFzUHHrWw+lXaQoeobeOee2dl+LXSsC78gSy RrTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=QBduXCTHUB2Uc/8dJN06haG0aRbv01Vs5/o4H+Nx9GI=; b=g0vZ+enJWEAooc60na/bmRGFvPgWcYOTFJ2CnO1of0kctgzRWOmthmZ5hcPS7UX3lO 7DhsXXGoiuLOhtzhSQuFEZEB2KprU5P3K03ltaAnhT4SFbsrhUOBdCBLY2SJVQmMAhps 7C1/1JoSrX2xqj5mDSgW1W8M4ms3+aKL8VYwr/aVW9H8Jm0qwkZLnTp50WqV/3j7IxGk +DN2LVwyvL0x0O1b/A9a3Fh83pKdHlN/eTzEenHN0YP9UeS6sONuamctoD40HNTK4qmA Rp/s2c8+GK5o/uGVn13k/pKElx30DcyA1V5Kj07YQAzAaB46Hl/Xqmx55yv68aOAaVKC saTg== X-Gm-Message-State: AFeK/H2XRZXM9jfJ1y+gf2TlB3Hi0Xzl50P1HeosiofbYQ6J4YGPzqs2lY2uGpMSJ7C2HA== X-Received: by 10.98.56.215 with SMTP id f206mr20797537pfa.71.1491277735504; Mon, 03 Apr 2017 20:48:55 -0700 (PDT) From: Ashish Mittal X-Google-Original-From: Ashish Mittal To: qemu-devel@nongnu.org, pbonzini@redhat.com, kwolf@redhat.com, armbru@redhat.com, berrange@redhat.com, jcody@redhat.com, famz@redhat.com, ashish.mittal@veritas.com, stefanha@gmail.com, Ketan.Nilangekar@veritas.com, jferlan@redhat.com, Buddhi.Madhav@veritas.com, Suraj.Singh@veritas.com, Nitin.Jerath@veritas.com, peter.maydell@linaro.org, venkatesha.mg@veritas.com, Rakesh.Ranjan@veritas.com, eblake@redhat.com Date: Mon, 3 Apr 2017 20:48:08 -0700 Message-Id: <1491277689-24949-2-git-send-email-Ashish.Mittal@veritas.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1491277689-24949-1-git-send-email-Ashish.Mittal@veritas.com> References: <1491277689-24949-1-git-send-email-Ashish.Mittal@veritas.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2607:f8b0:400e:c05::242 Subject: [Qemu-devel] [PATCH v11 1/2] block/vxhs.c: Add support for a new block device type called "vxhs" X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ashish Mittal , Abhijit.Dey@veritas.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Source code for the qnio library that this code loads can be downloaded fro= m: https://github.com/VeritasHyperScale/libqnio.git Sample command line using JSON syntax: ./x86_64-softmmu/qemu-system-x86_64 -name instance-00000008 -S -vnc 0.0.0.0= :0 -k en-us -vga cirrus -device virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,a= ddr=3D0x5 -msg timestamp=3Don 'json:{"driver":"vxhs","vdisk-id":"c3e9095a-a5ee-4dce-afeb-2a59fb387410", "server":{"host":"172.172.17.4","port":"9999"}}' Sample command line using URI syntax: qemu-img convert -f raw -O raw -n /var/lib/nova/instances/_base/0c5eacd5ebea5ed914b6a3e7b18f1ce734c386ad vxhs://192.168.0.1:9999/c6718f6b-0401-441d-a8c3-1f0064d75ee0 Sample command line using TLS credentials (run in secure mode): ./qemu-io --object tls-creds-x509,id=3Dtls0,dir=3D/etc/pki/qemu/vxhs,endpoint=3Dclient -c 'read -v 66000 2.5k' 'json:{"server.host": "127.0.0.1", "server.port": "9999", "vdisk-id": "/test.raw", "driver": "vxhs", "tls-creds":"tls0"}' Signed-off-by: Ashish Mittal Reviewed-by: Stefan Hajnoczi --- v11 changelog: (1) Replaced InetSocketAddress with InetSocketAddressBase. (2) Removed access to qemu_uuid. (3) Removed unnecessary g_strdup()/g_free(). (4) Removed unused acb->qiov. (5) Changed vxhs_init_and_ref() and vxhs_unref() per suggestion. (6) Removed unnecessary initializations from local variables. v10 changelog: (1) Implemented accepting TLS creds per block device via the CLI (see 3rd e.g in commit log). Corresponding changes made to the libqnio library. (2) iio_open() changed to accept TLS creds and use these internally to set up SSL connections. (3) Got rid of hard-coded VXHS_UUID_DEF. qemu_uuid is no longer used for authentication in any way. (4) Removed unnecessary qdict_del(backing_options, str). (5) Added '*tls-creds' to BlockdevOptionsVxHS. v9 changelog: (1) Fixes for all the review comments from v8. I have left the definition of VXHS_UUID_DEF unchanged pending a better suggestion. (2) qcow2 tests now pass on the vxhs test server. (3) Packaging changes for libvxhs will be checked in to the git repo soon. (4) I have not moved extern QemuUUID qemu_uuid to a separate header file. v8 changelog: (1) Security implementation for libqnio present in branch 'securify'. Please use 'securify' branch for building libqnio and testing with this patch. (2) Renamed libqnio to libvxhs. (3) Pass instance ID to libvxhs for SSL authentication. v7 changelog: (1) IO failover code has moved out to the libqnio library. (2) Fixes for issues reported by Stefan on v6. (3) Incorporated the QEMUBH patch provided by Stefan. This is a replacement for the pipe mechanism used earlier. (4) Fixes to the buffer overflows reported in libqnio. (5) Input validations in vxhs.c to prevent any buffer overflows for=20 arguments passed to libqnio. v6 changelog: (1) Added qemu-iotests for VxHS as a new patch in the series. (2) Replaced release version from 2.8 to 2.9 in block-core.json. v5 changelog: (1) Incorporated v4 review comments. v4 changelog: (1) Incorporated v3 review comments on QAPI changes. (2) Added refcounting for device open/close. Free library resources on last device close. v3 changelog: (1) Added QAPI schema for the VxHS driver. v2 changelog: (1) Changes done in response to v1 comments. block/Makefile.objs | 2 + block/trace-events | 17 ++ block/vxhs.c | 575 +++++++++++++++++++++++++++++++++++++++++++++++= ++++ configure | 39 ++++ qapi/block-core.json | 23 ++- 5 files changed, 654 insertions(+), 2 deletions(-) create mode 100644 block/vxhs.c diff --git a/block/Makefile.objs b/block/Makefile.objs index de96f8e..ea95530 100644 --- a/block/Makefile.objs +++ b/block/Makefile.objs @@ -19,6 +19,7 @@ block-obj-$(CONFIG_LIBNFS) +=3D nfs.o block-obj-$(CONFIG_CURL) +=3D curl.o block-obj-$(CONFIG_RBD) +=3D rbd.o block-obj-$(CONFIG_GLUSTERFS) +=3D gluster.o +block-obj-$(CONFIG_VXHS) +=3D vxhs.o block-obj-$(CONFIG_LIBSSH2) +=3D ssh.o block-obj-y +=3D accounting.o dirty-bitmap.o block-obj-y +=3D write-threshold.o @@ -38,6 +39,7 @@ rbd.o-cflags :=3D $(RBD_CFLAGS) rbd.o-libs :=3D $(RBD_LIBS) gluster.o-cflags :=3D $(GLUSTERFS_CFLAGS) gluster.o-libs :=3D $(GLUSTERFS_LIBS) +vxhs.o-libs :=3D $(VXHS_LIBS) ssh.o-cflags :=3D $(LIBSSH2_CFLAGS) ssh.o-libs :=3D $(LIBSSH2_LIBS) block-obj-$(if $(CONFIG_BZIP2),m,n) +=3D dmg-bz2.o diff --git a/block/trace-events b/block/trace-events index 0bc5c0a..7758ec3 100644 --- a/block/trace-events +++ b/block/trace-events @@ -110,3 +110,20 @@ qed_aio_write_data(void *s, void *acb, int ret, uint64= _t offset, size_t len) "s qed_aio_write_prefill(void *s, void *acb, uint64_t start, size_t len, uint= 64_t offset) "s %p acb %p start %"PRIu64" len %zu offset %"PRIu64 qed_aio_write_postfill(void *s, void *acb, uint64_t start, size_t len, uin= t64_t offset) "s %p acb %p start %"PRIu64" len %zu offset %"PRIu64 qed_aio_write_main(void *s, void *acb, int ret, uint64_t offset, size_t le= n) "s %p acb %p ret %d offset %"PRIu64" len %zu" + +# block/vxhs.c +vxhs_iio_callback(int error) "ctx is NULL: error %d" +vxhs_iio_callback_chnfail(int err, int error) "QNIO channel failed, no i/o= %d, %d" +vxhs_iio_callback_unknwn(int opcode, int err) "unexpected opcode %d, errno= %d" +vxhs_aio_rw_invalid(int req) "Invalid I/O request iodir %d" +vxhs_aio_rw_ioerr(char *guid, int iodir, uint64_t size, uint64_t off, void= *acb, int ret, int err) "IO ERROR (vDisk %s) FOR : Read/Write =3D %d size = =3D %lu offset =3D %lu ACB =3D %p. Error =3D %d, errno =3D %d" +vxhs_get_vdisk_stat_err(char *guid, int ret, int err) "vDisk (%s) stat ioc= tl failed, ret =3D %d, errno =3D %d" +vxhs_get_vdisk_stat(char *vdisk_guid, uint64_t vdisk_size) "vDisk %s stat = ioctl returned size %lu" +vxhs_complete_aio(void *acb, uint64_t ret) "aio failed acb %p ret %ld" +vxhs_parse_uri_filename(const char *filename) "URI passed via bdrv_parse_f= ilename %s" +vxhs_open_vdiskid(const char *vdisk_id) "Opening vdisk-id %s" +vxhs_open_hostinfo(char *of_vsa_addr, int port) "Adding host %s:%d to BDRV= VXHSState" +vxhs_open_iio_open(const char *host) "Failed to connect to storage agent o= n host %s" +vxhs_parse_uri_hostinfo(char *host, int port) "Host: IP %s, Port %d" +vxhs_close(char *vdisk_guid) "Closing vdisk %s" +vxhs_get_creds(const char *cacert, const char *client_key, const char *cli= ent_cert) "cacert %s, client_key %s, client_cert %s" diff --git a/block/vxhs.c b/block/vxhs.c new file mode 100644 index 0000000..9ffe9d3 --- /dev/null +++ b/block/vxhs.c @@ -0,0 +1,575 @@ +/* + * QEMU Block driver for Veritas HyperScale (VxHS) + * + * Copyright (c) 2017 Veritas Technologies LLC. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include +#include +#include "block/block_int.h" +#include "qapi/qmp/qerror.h" +#include "qapi/qmp/qdict.h" +#include "qapi/qmp/qstring.h" +#include "trace.h" +#include "qemu/uri.h" +#include "qapi/error.h" +#include "qemu/uuid.h" +#include "crypto/tlscredsx509.h" + +#define VXHS_OPT_FILENAME "filename" +#define VXHS_OPT_VDISK_ID "vdisk-id" +#define VXHS_OPT_SERVER "server" +#define VXHS_OPT_HOST "host" +#define VXHS_OPT_PORT "port" + +/* Only accessed under QEMU global mutex */ +static uint32_t vxhs_ref; + +typedef enum { + VDISK_AIO_READ, + VDISK_AIO_WRITE, +} VDISKAIOCmd; + +/* + * HyperScale AIO callbacks structure + */ +typedef struct VXHSAIOCB { + BlockAIOCB common; + int err; +} VXHSAIOCB; + +typedef struct VXHSvDiskHostsInfo { + void *dev_handle; /* Device handle */ + char *host; /* Host name or IP */ + int port; /* Host's port number */ +} VXHSvDiskHostsInfo; + +/* + * Structure per vDisk maintained for state + */ +typedef struct BDRVVXHSState { + VXHSvDiskHostsInfo vdisk_hostinfo; /* Per host info */ + char *vdisk_guid; + char *tlscredsid; /* tlscredsid */ +} BDRVVXHSState; + +static void vxhs_complete_aio_bh(void *opaque) +{ + VXHSAIOCB *acb =3D opaque; + BlockCompletionFunc *cb =3D acb->common.cb; + void *cb_opaque =3D acb->common.opaque; + int ret =3D 0; + + if (acb->err !=3D 0) { + trace_vxhs_complete_aio(acb, acb->err); + ret =3D (-EIO); + } + + qemu_aio_unref(acb); + cb(cb_opaque, ret); +} + +/* + * Called from a libqnio thread + */ +static void vxhs_iio_callback(void *ctx, uint32_t opcode, uint32_t error) +{ + VXHSAIOCB *acb =3D NULL; + + switch (opcode) { + case IRP_READ_REQUEST: + case IRP_WRITE_REQUEST: + + /* + * ctx is VXHSAIOCB* + * ctx is NULL if error is QNIOERROR_CHANNEL_HUP + */ + if (ctx) { + acb =3D ctx; + } else { + trace_vxhs_iio_callback(error); + goto out; + } + + if (error) { + if (!acb->err) { + acb->err =3D error; + } + trace_vxhs_iio_callback(error); + } + + aio_bh_schedule_oneshot(bdrv_get_aio_context(acb->common.bs), + vxhs_complete_aio_bh, acb); + break; + + default: + if (error =3D=3D QNIOERROR_HUP) { + /* + * Channel failed, spontaneous notification, + * not in response to I/O + */ + trace_vxhs_iio_callback_chnfail(error, errno); + } else { + trace_vxhs_iio_callback_unknwn(opcode, error); + } + break; + } +out: + return; +} + +static QemuOptsList runtime_opts =3D { + .name =3D "vxhs", + .head =3D QTAILQ_HEAD_INITIALIZER(runtime_opts.head), + .desc =3D { + { + .name =3D VXHS_OPT_FILENAME, + .type =3D QEMU_OPT_STRING, + .help =3D "URI to the Veritas HyperScale image", + }, + { + .name =3D VXHS_OPT_VDISK_ID, + .type =3D QEMU_OPT_STRING, + .help =3D "UUID of the VxHS vdisk", + }, + { + .name =3D "tls-creds", + .type =3D QEMU_OPT_STRING, + .help =3D "ID of the TLS/SSL credentials to use", + }, + { /* end of list */ } + }, +}; + +static QemuOptsList runtime_tcp_opts =3D { + .name =3D "vxhs_tcp", + .head =3D QTAILQ_HEAD_INITIALIZER(runtime_tcp_opts.head), + .desc =3D { + { + .name =3D VXHS_OPT_HOST, + .type =3D QEMU_OPT_STRING, + .help =3D "host address (ipv4 addresses)", + }, + { + .name =3D VXHS_OPT_PORT, + .type =3D QEMU_OPT_NUMBER, + .help =3D "port number on which VxHSD is listening (default 99= 99)", + .def_value_str =3D "9999" + }, + { /* end of list */ } + }, +}; + +/* + * Parse incoming URI and populate *options with the host + * and device information + */ +static int vxhs_parse_uri(const char *filename, QDict *options) +{ + URI *uri =3D NULL; + char *port; + int ret =3D 0; + + trace_vxhs_parse_uri_filename(filename); + uri =3D uri_parse(filename); + if (!uri || !uri->server || !uri->path) { + uri_free(uri); + return -EINVAL; + } + + qdict_put(options, VXHS_OPT_SERVER".host", qstring_from_str(uri->serve= r)); + + if (uri->port) { + port =3D g_strdup_printf("%d", uri->port); + qdict_put(options, VXHS_OPT_SERVER".port", qstring_from_str(port)); + g_free(port); + } + + qdict_put(options, "vdisk-id", qstring_from_str(uri->path)); + + trace_vxhs_parse_uri_hostinfo(uri->server, uri->port); + uri_free(uri); + + return ret; +} + +static void vxhs_parse_filename(const char *filename, QDict *options, + Error **errp) +{ + if (qdict_haskey(options, "vdisk-id") || qdict_haskey(options, "server= ")) { + error_setg(errp, "vdisk-id/server and a file name may not be speci= fied " + "at the same time"); + return; + } + + if (strstr(filename, "://")) { + int ret =3D vxhs_parse_uri(filename, options); + if (ret < 0) { + error_setg(errp, "Invalid URI. URI should be of the form " + " vxhs://:/"); + } + } +} + +static int vxhs_init_and_ref(void) +{ + if (vxhs_ref++ =3D=3D 0) { + if (iio_init(QNIO_VERSION, vxhs_iio_callback)) { + return -ENODEV; + } + } + return 0; +} + +static void vxhs_unref(void) +{ + if (--vxhs_ref =3D=3D 0) { + iio_fini(); + } +} + +static void vxhs_get_tls_creds(const char *id, char **cacert, + char **key, char **cert, Error **errp) +{ + Object *obj; + QCryptoTLSCreds *creds; + QCryptoTLSCredsX509 *creds_x509; + + obj =3D object_resolve_path_component( + object_get_objects_root(), id); + + if (!obj) { + error_setg(errp, "No TLS credentials with id '%s'", + id); + return; + } + + creds_x509 =3D (QCryptoTLSCredsX509 *) + object_dynamic_cast(obj, TYPE_QCRYPTO_TLS_CREDS_X509); + + if (!creds_x509) { + error_setg(errp, "Object with id '%s' is not TLS credentials", + id); + return; + } + + creds =3D &creds_x509->parent_obj; + + if (creds->endpoint !=3D QCRYPTO_TLS_CREDS_ENDPOINT_CLIENT) { + error_setg(errp, + "Expecting TLS credentials with a client endpoint"); + return; + } + + /* + * Get the cacert, client_cert and client_key file names. + */ + if (!creds->dir) { + error_setg(errp, "TLS object missing 'dir' property value"); + return; + } + + *cacert =3D g_strdup_printf("%s/%s", creds->dir, + QCRYPTO_TLS_CREDS_X509_CA_CERT); + *cert =3D g_strdup_printf("%s/%s", creds->dir, + QCRYPTO_TLS_CREDS_X509_CLIENT_CERT); + *key =3D g_strdup_printf("%s/%s", creds->dir, + QCRYPTO_TLS_CREDS_X509_CLIENT_KEY); +} + +static int vxhs_open(BlockDriverState *bs, QDict *options, + int bdrv_flags, Error **errp) +{ + BDRVVXHSState *s =3D bs->opaque; + void *dev_handlep; + QDict *backing_options =3D NULL; + QemuOpts *opts =3D NULL; + QemuOpts *tcp_opts =3D NULL; + char *of_vsa_addr =3D NULL; + Error *local_err =3D NULL; + const char *vdisk_id_opt; + const char *server_host_opt; + int ret =3D 0; + char *cacert =3D NULL; + char *client_key =3D NULL; + char *client_cert =3D NULL; + + ret =3D vxhs_init_and_ref(); + if (ret < 0) { + ret =3D -EINVAL; + goto out; + } + + /* Create opts info from runtime_opts and runtime_tcp_opts list */ + opts =3D qemu_opts_create(&runtime_opts, NULL, 0, &error_abort); + tcp_opts =3D qemu_opts_create(&runtime_tcp_opts, NULL, 0, &error_abort= ); + + qemu_opts_absorb_qdict(opts, options, &local_err); + if (local_err) { + ret =3D -EINVAL; + goto out; + } + + /* vdisk-id is the disk UUID */ + vdisk_id_opt =3D qemu_opt_get(opts, VXHS_OPT_VDISK_ID); + if (!vdisk_id_opt) { + error_setg(&local_err, QERR_MISSING_PARAMETER, VXHS_OPT_VDISK_ID); + ret =3D -EINVAL; + goto out; + } + + /* vdisk-id may contain a leading '/' */ + if (strlen(vdisk_id_opt) > UUID_FMT_LEN + 1) { + error_setg(&local_err, "vdisk-id cannot be more than %d characters= ", + UUID_FMT_LEN); + ret =3D -EINVAL; + goto out; + } + + s->vdisk_guid =3D g_strdup(vdisk_id_opt); + trace_vxhs_open_vdiskid(vdisk_id_opt); + + /* get the 'server.' arguments */ + qdict_extract_subqdict(options, &backing_options, VXHS_OPT_SERVER"."); + + qemu_opts_absorb_qdict(tcp_opts, backing_options, &local_err); + if (local_err !=3D NULL) { + ret =3D -EINVAL; + goto out; + } + + server_host_opt =3D qemu_opt_get(tcp_opts, VXHS_OPT_HOST); + if (!server_host_opt) { + error_setg(&local_err, QERR_MISSING_PARAMETER, + VXHS_OPT_SERVER"."VXHS_OPT_HOST); + ret =3D -EINVAL; + goto out; + } + + if (strlen(server_host_opt) > MAXHOSTNAMELEN) { + error_setg(&local_err, "server.host cannot be more than %d charact= ers", + MAXHOSTNAMELEN); + ret =3D -EINVAL; + goto out; + } + + /* check if we got tls-creds via the --object argument */ + s->tlscredsid =3D g_strdup(qemu_opt_get(opts, "tls-creds")); + if (s->tlscredsid) { + vxhs_get_tls_creds(s->tlscredsid, &cacert, &client_key, + &client_cert, &local_err); + if (local_err !=3D NULL) { + ret =3D -EINVAL; + goto out; + } + trace_vxhs_get_creds(cacert, client_key, client_cert); + } + + s->vdisk_hostinfo.host =3D g_strdup(server_host_opt); + s->vdisk_hostinfo.port =3D g_ascii_strtoll(qemu_opt_get(tcp_opts, + VXHS_OPT_PORT), + NULL, 0); + + trace_vxhs_open_hostinfo(s->vdisk_hostinfo.host, + s->vdisk_hostinfo.port); + + of_vsa_addr =3D g_strdup_printf("of://%s:%d", + s->vdisk_hostinfo.host, + s->vdisk_hostinfo.port); + + /* + * Open qnio channel to storage agent if not opened before + */ + dev_handlep =3D iio_open(of_vsa_addr, s->vdisk_guid, 0, + cacert, client_key, client_cert); + if (dev_handlep =3D=3D NULL) { + trace_vxhs_open_iio_open(of_vsa_addr); + ret =3D -ENODEV; + goto out; + } + s->vdisk_hostinfo.dev_handle =3D dev_handlep; + +out: + g_free(of_vsa_addr); + QDECREF(backing_options); + qemu_opts_del(tcp_opts); + qemu_opts_del(opts); + g_free(cacert); + g_free(client_key); + g_free(client_cert); + + if (ret < 0) { + vxhs_unref(); + error_propagate(errp, local_err); + g_free(s->vdisk_hostinfo.host); + g_free(s->vdisk_guid); + g_free(s->tlscredsid); + s->vdisk_guid =3D NULL; + } + + return ret; +} + +static const AIOCBInfo vxhs_aiocb_info =3D { + .aiocb_size =3D sizeof(VXHSAIOCB) +}; + +/* + * This allocates QEMU-VXHS callback for each IO + * and is passed to QNIO. When QNIO completes the work, + * it will be passed back through the callback. + */ +static BlockAIOCB *vxhs_aio_rw(BlockDriverState *bs, int64_t sector_num, + QEMUIOVector *qiov, int nb_sectors, + BlockCompletionFunc *cb, void *opaque, + VDISKAIOCmd iodir) +{ + VXHSAIOCB *acb =3D NULL; + BDRVVXHSState *s =3D bs->opaque; + size_t size; + uint64_t offset; + int iio_flags =3D 0; + int ret =3D 0; + void *dev_handle =3D s->vdisk_hostinfo.dev_handle; + + offset =3D sector_num * BDRV_SECTOR_SIZE; + size =3D nb_sectors * BDRV_SECTOR_SIZE; + acb =3D qemu_aio_get(&vxhs_aiocb_info, bs, cb, opaque); + + /* + * Initialize VXHSAIOCB. + */ + acb->err =3D 0; + + iio_flags =3D IIO_FLAG_ASYNC; + + switch (iodir) { + case VDISK_AIO_WRITE: + ret =3D iio_writev(dev_handle, acb, qiov->iov, qiov->niov, + offset, (uint64_t)size, iio_flags); + break; + case VDISK_AIO_READ: + ret =3D iio_readv(dev_handle, acb, qiov->iov, qiov->niov, + offset, (uint64_t)size, iio_flags); + break; + default: + trace_vxhs_aio_rw_invalid(iodir); + goto errout; + } + + if (ret !=3D 0) { + trace_vxhs_aio_rw_ioerr(s->vdisk_guid, iodir, size, offset, + acb, ret, errno); + goto errout; + } + return &acb->common; + +errout: + qemu_aio_unref(acb); + return NULL; +} + +static BlockAIOCB *vxhs_aio_readv(BlockDriverState *bs, + int64_t sector_num, QEMUIOVector *qiov, + int nb_sectors, + BlockCompletionFunc *cb, void *opaque) +{ + return vxhs_aio_rw(bs, sector_num, qiov, nb_sectors, cb, + opaque, VDISK_AIO_READ); +} + +static BlockAIOCB *vxhs_aio_writev(BlockDriverState *bs, + int64_t sector_num, QEMUIOVector *qiov, + int nb_sectors, + BlockCompletionFunc *cb, void *opaque) +{ + return vxhs_aio_rw(bs, sector_num, qiov, nb_sectors, + cb, opaque, VDISK_AIO_WRITE); +} + +static void vxhs_close(BlockDriverState *bs) +{ + BDRVVXHSState *s =3D bs->opaque; + + trace_vxhs_close(s->vdisk_guid); + + g_free(s->vdisk_guid); + s->vdisk_guid =3D NULL; + + /* + * Close vDisk device + */ + if (s->vdisk_hostinfo.dev_handle) { + iio_close(s->vdisk_hostinfo.dev_handle); + s->vdisk_hostinfo.dev_handle =3D NULL; + } + + vxhs_unref(); + + /* + * Free the dynamically allocated host string etc + */ + g_free(s->vdisk_hostinfo.host); + g_free(s->tlscredsid); + s->tlscredsid =3D NULL; + s->vdisk_hostinfo.host =3D NULL; + s->vdisk_hostinfo.port =3D 0; +} + +static int64_t vxhs_get_vdisk_stat(BDRVVXHSState *s) +{ + int64_t vdisk_size =3D -1; + int ret =3D 0; + void *dev_handle =3D s->vdisk_hostinfo.dev_handle; + + ret =3D iio_ioctl(dev_handle, IOR_VDISK_STAT, &vdisk_size, 0); + if (ret < 0) { + trace_vxhs_get_vdisk_stat_err(s->vdisk_guid, ret, errno); + return -EIO; + } + + trace_vxhs_get_vdisk_stat(s->vdisk_guid, vdisk_size); + return vdisk_size; +} + +/* + * Returns the size of vDisk in bytes. This is required + * by QEMU block upper block layer so that it is visible + * to guest. + */ +static int64_t vxhs_getlength(BlockDriverState *bs) +{ + BDRVVXHSState *s =3D bs->opaque; + int64_t vdisk_size; + + vdisk_size =3D vxhs_get_vdisk_stat(s); + if (vdisk_size < 0) { + return -EIO; + } + + return vdisk_size; +} + +static BlockDriver bdrv_vxhs =3D { + .format_name =3D "vxhs", + .protocol_name =3D "vxhs", + .instance_size =3D sizeof(BDRVVXHSState), + .bdrv_file_open =3D vxhs_open, + .bdrv_parse_filename =3D vxhs_parse_filename, + .bdrv_close =3D vxhs_close, + .bdrv_getlength =3D vxhs_getlength, + .bdrv_aio_readv =3D vxhs_aio_readv, + .bdrv_aio_writev =3D vxhs_aio_writev, +}; + +static void bdrv_vxhs_init(void) +{ + bdrv_register(&bdrv_vxhs); +} + +block_init(bdrv_vxhs_init); diff --git a/configure b/configure index 4b3b5cd..94f0818 100755 --- a/configure +++ b/configure @@ -320,6 +320,7 @@ numa=3D"" tcmalloc=3D"no" jemalloc=3D"no" replication=3D"yes" +vxhs=3D"" =20 supported_cpu=3D"no" supported_os=3D"no" @@ -1183,6 +1184,10 @@ for opt do ;; --enable-replication) replication=3D"yes" ;; + --disable-vxhs) vxhs=3D"no" + ;; + --enable-vxhs) vxhs=3D"yes" + ;; *) echo "ERROR: unknown option $opt" echo "Try '$0 --help' for more information" @@ -1427,6 +1432,7 @@ disabled with --disable-FEATURE, default is enabled i= f available: xfsctl xfsctl support qom-cast-debug cast debugging support tools build qemu-io, qemu-nbd and qemu-image tools + vxhs Veritas HyperScale vDisk backend support =20 NOTE: The object files are built at the place where configure is launched EOF @@ -4777,6 +4783,33 @@ if compile_prog "" "" ; then fi =20 ########################################## +# Veritas HyperScale block driver VxHS +# Check if libvxhs is installed + +if test "$vxhs" !=3D "no" ; then + cat > $TMPC < +#include + +void *vxhs_callback; + +int main(void) { + iio_init(QNIO_VERSION, vxhs_callback); + return 0; +} +EOF + vxhs_libs=3D"-lvxhs -lssl" + if compile_prog "" "$vxhs_libs" ; then + vxhs=3Dyes + else + if test "$vxhs" =3D "yes" ; then + feature_not_found "vxhs block device" "Install libvxhs See github" + fi + vxhs=3Dno + fi +fi + +########################################## # End of CC checks # After here, no more $cc or $ld runs =20 @@ -5142,6 +5175,7 @@ echo "tcmalloc support $tcmalloc" echo "jemalloc support $jemalloc" echo "avx2 optimization $avx2_opt" echo "replication support $replication" +echo "VxHS block device $vxhs" =20 if test "$sdl_too_old" =3D "yes"; then echo "-> Your SDL version is too old - please upgrade to have SDL support" @@ -5781,6 +5815,11 @@ if test "$pthread_setname_np" =3D "yes" ; then echo "CONFIG_PTHREAD_SETNAME_NP=3Dy" >> $config_host_mak fi =20 +if test "$vxhs" =3D "yes" ; then + echo "CONFIG_VXHS=3Dy" >> $config_host_mak + echo "VXHS_LIBS=3D$vxhs_libs" >> $config_host_mak +fi + if test "$tcg_interpreter" =3D "yes"; then QEMU_INCLUDES=3D"-I\$(SRC_PATH)/tcg/tci $QEMU_INCLUDES" elif test "$ARCH" =3D "sparc64" ; then diff --git a/qapi/block-core.json b/qapi/block-core.json index 033457c..87fb747 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -2108,6 +2108,8 @@ # # Drivers that are supported in block device operations. # +# @vxhs: Since 2.10 +# # Since: 2.9 ## { 'enum': 'BlockdevDriver', @@ -2116,7 +2118,7 @@ 'host_device', 'http', 'https', 'iscsi', 'luks', 'nbd', 'nfs', 'null-aio', 'null-co', 'parallels', 'qcow', 'qcow2', 'qed', 'quorum', 'raw', 'rbd', 'replication', 'sheepdog', 'ssh', - 'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat' ] } + 'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] } =20 ## # @BlockdevOptionsFile: @@ -2866,6 +2868,22 @@ 'data': { '*offset': 'int', '*size': 'int' } } =20 ## +# @BlockdevOptionsVxHS: +# +# Driver specific block device options for VxHS +# +# @vdisk-id: UUID of VxHS volume +# @server: vxhs server IP, port +# @tls-creds: TLS credentials ID +# +# Since: 2.10 +## +{ 'struct': 'BlockdevOptionsVxHS', + 'data': { 'vdisk-id': 'str', + 'server': 'InetSocketAddressBase', + '*tls-creds': 'str' } } + +## # @BlockdevOptions: # # Options for creating a block device. Many options are available for all @@ -2927,7 +2945,8 @@ 'vhdx': 'BlockdevOptionsGenericFormat', 'vmdk': 'BlockdevOptionsGenericCOWFormat', 'vpc': 'BlockdevOptionsGenericFormat', - 'vvfat': 'BlockdevOptionsVVFAT' + 'vvfat': 'BlockdevOptionsVVFAT', + 'vxhs': 'BlockdevOptionsVxHS' } } =20 ## --=20 2.5.5 From nobody Sun Apr 28 13:46:23 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; dkim=fail spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1491277875820565.0742815192974; Mon, 3 Apr 2017 20:51:15 -0700 (PDT) Received: from localhost ([::1]:33814 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cvFUw-0007BE-On for importer@patchew.org; Mon, 03 Apr 2017 23:51:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34033) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cvFSm-0005lF-7I for qemu-devel@nongnu.org; Mon, 03 Apr 2017 23:49:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cvFSl-0005vP-0o for qemu-devel@nongnu.org; Mon, 03 Apr 2017 23:49:00 -0400 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]:35071) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cvFSk-0005u4-Qa for qemu-devel@nongnu.org; Mon, 03 Apr 2017 23:48:58 -0400 Received: by mail-pg0-x244.google.com with SMTP id g2so34086357pge.2 for ; Mon, 03 Apr 2017 20:48:58 -0700 (PDT) Received: from localhost.localdomain.localdomain ([172.56.38.27]) by smtp.gmail.com with ESMTPSA id u198sm28331074pgb.45.2017.04.03.20.48.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Apr 2017 20:48:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TSg3pgf8JD2QoGN6Z880SoCYOhUh6LCx4/UDO+cutRU=; b=uKsQm3SIrlJ1NFUhI+g8BfCmR1GONUtNdL/zE3VPclSr4rL3Pm/zhJiEQhDRf04KkJ g9vkJ9esRe/TYS7HZNu+8dd40kPaKDY2TkIDxQBB2rceB5b04Xr/vxmRG4Bh/nFvToQt SoSsIF+l6jDkO1NVV45I09IQQgL2sk+sGic8zQs4wjeuQjEc+lkkp3IiXCnfi+LHtJNS wXw/CZevWNJTqD6qJztHZwR07Qa3wn3RGR3iZWsHYVsyWz6OIHGssiC0KNXujT9KWp47 foOS8u09Maj4d9EoKxpMF2SKFsUVC3tUzIYFiiSt8MkOe0xmsmW7RsC1iFN6Z8iLaGSC aGDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TSg3pgf8JD2QoGN6Z880SoCYOhUh6LCx4/UDO+cutRU=; b=K+Ne+aS0esZyidXKEBn7oC6zC30sgWibKvk+P7XkCZAntgrs9OU2qTLthwpvzaVNrZ ppQPgmpuu4ZBDtHj9pFnQW8oPoKH9iPpttMQwl3cTv7GqREnL90C+VPi7UBpiyxp6bQi Um28EeXj8avIS4WJnxu7HmGKyvqsWlxhS6zBxVrz3mdqjeasUthbzTof81KAJS0xJaPb kapIiO8Bckwk2PlxOPrlkkGLZM3P3Y00KhkNhr14LhbvS5kzhucjbUypMgpkG0rN2Oue 0Q/zJgdOt4CvSVkpB3876P5rUbrvaddhA4qgQ3ZCR7s/mJ2t9YBd+5UDoH/X7gJPQ4yw 7K7A== X-Gm-Message-State: AFeK/H1iq5VfZgHxaZGK0y7a8sztAD5Balmew/al9FaABJb1Sn00h4o9rGgukYLKBw0hWA== X-Received: by 10.98.57.153 with SMTP id u25mr20022829pfj.111.1491277737868; Mon, 03 Apr 2017 20:48:57 -0700 (PDT) From: Ashish Mittal X-Google-Original-From: Ashish Mittal To: qemu-devel@nongnu.org, pbonzini@redhat.com, kwolf@redhat.com, armbru@redhat.com, berrange@redhat.com, jcody@redhat.com, famz@redhat.com, ashish.mittal@veritas.com, stefanha@gmail.com, Ketan.Nilangekar@veritas.com, jferlan@redhat.com, Buddhi.Madhav@veritas.com, Suraj.Singh@veritas.com, Nitin.Jerath@veritas.com, peter.maydell@linaro.org, venkatesha.mg@veritas.com, Rakesh.Ranjan@veritas.com, eblake@redhat.com Date: Mon, 3 Apr 2017 20:48:09 -0700 Message-Id: <1491277689-24949-3-git-send-email-Ashish.Mittal@veritas.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1491277689-24949-1-git-send-email-Ashish.Mittal@veritas.com> References: <1491277689-24949-1-git-send-email-Ashish.Mittal@veritas.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2607:f8b0:400e:c05::244 Subject: [Qemu-devel] [PATCH v11 2/2] block/vxhs.c: Add qemu-iotests for new block device type "vxhs" X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ashish Mittal , Abhijit.Dey@veritas.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" These changes use a vxhs test server that is a part of the following repository: https://github.com/VeritasHyperScale/libqnio.git Signed-off-by: Ashish Mittal Reviewed-by: Stefan Hajnoczi --- v11 changelog: (1) No changes. v10 changelog: (1) Redirect o/p of "$QEMU_VXHS -d $TEST_DIR" to /dev/null v9 changelog: (1) Dropped second argument to set_prog_path(). We will pick up the test server location from the user's PATH env setting. v8/v7 changelog: (1) No changes. v6 changelog: (1) Added iotests for VxHS block device. tests/qemu-iotests/common | 6 ++++++ tests/qemu-iotests/common.config | 13 +++++++++++++ tests/qemu-iotests/common.filter | 1 + tests/qemu-iotests/common.rc | 19 +++++++++++++++++++ 4 files changed, 39 insertions(+) diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common index 4d5650d..9c6f972 100644 --- a/tests/qemu-iotests/common +++ b/tests/qemu-iotests/common @@ -157,6 +157,7 @@ check options -ssh test ssh -nfs test nfs -luks test luks + -vxhs test vxhs -xdiff graphical mode diff -nocache use O_DIRECT on backing file -misalign misalign memory allocations @@ -260,6 +261,11 @@ testlist options xpand=3Dfalse ;; =20 + -vxhs) + IMGPROTO=3Dvxhs + xpand=3Dfalse + ;; + -ssh) IMGPROTO=3Dssh xpand=3Dfalse diff --git a/tests/qemu-iotests/common.config b/tests/qemu-iotests/common.c= onfig index 55527aa..c4b51b3 100644 --- a/tests/qemu-iotests/common.config +++ b/tests/qemu-iotests/common.config @@ -105,6 +105,10 @@ if [ -z "$QEMU_NBD_PROG" ]; then export QEMU_NBD_PROG=3D"`set_prog_path qemu-nbd`" fi =20 +if [ -z "$QEMU_VXHS_PROG" ]; then + export QEMU_VXHS_PROG=3D"`set_prog_path qnio_server`" +fi + _qemu_wrapper() { ( @@ -156,10 +160,19 @@ _qemu_nbd_wrapper() ) } =20 +_qemu_vxhs_wrapper() +{ + ( + echo $BASHPID > "${TEST_DIR}/qemu-vxhs.pid" + exec "$QEMU_VXHS_PROG" $QEMU_VXHS_OPTIONS "$@" + ) +} + export QEMU=3D_qemu_wrapper export QEMU_IMG=3D_qemu_img_wrapper export QEMU_IO=3D_qemu_io_wrapper export QEMU_NBD=3D_qemu_nbd_wrapper +export QEMU_VXHS=3D_qemu_vxhs_wrapper =20 QEMU_IMG_EXTRA_ARGS=3D if [ "$IMGOPTSSYNTAX" =3D "true" ]; then diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.f= ilter index 1040013..c9a2d5c 100644 --- a/tests/qemu-iotests/common.filter +++ b/tests/qemu-iotests/common.filter @@ -122,6 +122,7 @@ _filter_img_info() -e "s#$TEST_DIR#TEST_DIR#g" \ -e "s#$IMGFMT#IMGFMT#g" \ -e 's#nbd://127.0.0.1:10810$#TEST_DIR/t.IMGFMT#g' \ + -e 's#json.*vdisk-id.*vxhs"}}#TEST_DIR/t.IMGFMT#' \ -e "/encrypted: yes/d" \ -e "/cluster_size: [0-9]\\+/d" \ -e "/table_size: [0-9]\\+/d" \ diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc index 7d4781d..62529ee 100644 --- a/tests/qemu-iotests/common.rc +++ b/tests/qemu-iotests/common.rc @@ -85,6 +85,9 @@ else elif [ "$IMGPROTO" =3D "nfs" ]; then TEST_DIR=3D"nfs://127.0.0.1/$TEST_DIR" TEST_IMG=3D$TEST_DIR/t.$IMGFMT + elif [ "$IMGPROTO" =3D "vxhs" ]; then + TEST_IMG_FILE=3D$TEST_DIR/t.$IMGFMT + TEST_IMG=3D"vxhs://127.0.0.1:9999/t.$IMGFMT" else TEST_IMG=3D$IMGPROTO:$TEST_DIR/t.$IMGFMT fi @@ -171,6 +174,12 @@ _make_test_img() eval "$QEMU_NBD -v -t -b 127.0.0.1 -p 10810 -f $IMGFMT $TEST_IMG_= FILE >/dev/null &" sleep 1 # FIXME: qemu-nbd needs to be listening before we continue fi + + # Start QNIO server on image directory for vxhs protocol + if [ $IMGPROTO =3D "vxhs" ]; then + eval "$QEMU_VXHS -d $TEST_DIR > /dev/null &" + sleep 1 # Wait for server to come up. + fi } =20 _rm_test_img() @@ -197,6 +206,16 @@ _cleanup_test_img() fi rm -f "$TEST_IMG_FILE" ;; + vxhs) + if [ -f "${TEST_DIR}/qemu-vxhs.pid" ]; then + local QEMU_VXHS_PID + read QEMU_VXHS_PID < "${TEST_DIR}/qemu-vxhs.pid" + kill ${QEMU_VXHS_PID} >/dev/null 2>&1 + rm -f "${TEST_DIR}/qemu-vxhs.pid" + fi + rm -f "$TEST_IMG_FILE" + ;; + file) _rm_test_img "$TEST_DIR/t.$IMGFMT" _rm_test_img "$TEST_DIR/t.$IMGFMT.orig" --=20 2.5.5