Series comparison

-[Qemu-devel] [PULL 00/15] Block patches
+[PULL 00/14] Block patches
-The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:
+The following changes since commit 754f756cc4c6d9d14b7230c62b5bb20f9d655888:
-  Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)
+  Merge tag 'pull-target-arm-20220422-1' of https://git.linaro.org/people/pmaydell/qemu-arm into staging (2022-04-22 08:03:18 -0700)
 are available in the Git repository at:
-  https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27
+  https://gitlab.com/hreitz/qemu.git tags/pull-block-2022-04-25
-for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:
+for you to fetch changes up to 348a0740afc5b313599533eb69bbb2b95d2f1bba:
-  iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)
+  iotests/108: Fix when missing user_allow_other (2022-04-25 14:46:45 +0200)
 ----------------------------------------------------------------
 Block patches:
-- qemu-io now accepts a file to read a write pattern from
+- New @force parameter for blockdev-change-medium
-- Ensure that raw files have their first block allocated so we can probe
+- Improvements to the iotests to help with debugging
-  the O_DIRECT alignment if necessary
+- Fix iotest 108 on systems without user_allow_other in fuse.conf
 - Various fixes
 ----------------------------------------------------------------
-Denis Plotnikov (1):
+Denis V. Lunev (1):
-  qemu-io: add pattern file for write command
+  block: add 'force' parameter to 'blockdev-change-medium' command
-Max Reitz (7):
+Hanna Reitz (1):
-  iotests: Fix _filter_img_create()
+  iotests/108: Fix when missing user_allow_other
   vmdk: Use bdrv_dirname() for relative extent paths
   iotests: Keep testing broken relative extent paths
   vmdk: Reject invalid compressed writes
   iotests: Disable broken streamOptimized tests
   iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
   iotests: Disable 126 for flat vmdk subformats
-Nir Soffer (3):
+John Snow (12):
-  block: posix: Always allocate the first block
+  iotests: replace calls to log(qemu_io(...)) with qemu_io_log()
-  iotests: Test allocate_first_block() with O_DIRECT
+  iotests/163: Fix broken qemu-io invocation
-  iotests: Unify cache mode quoting
+  iotests: Don't check qemu_io() output for specific error strings
   iotests/040: Don't check image pattern on zero-length image
   iotests/040: Fix TestCommitWithFilters test
   iotests: create generic qemu_tool() function
   iotests: rebase qemu_io() on top of qemu_tool()
   iotests/migration-permissions: use assertRaises() for qemu_io()
     negative test
   iotests/image-fleecing: switch to qemu_io()
   iotests: remove qemu_io_pipe_and_status()
   iotests: remove qemu_io_silent() and qemu_io_silent_check().
   iotests: make qemu_io_log() check return codes by default
-Stefan Hajnoczi (1):
+ qapi/block.json                               |  6 ++
-  file-posix: fix request_alignment typo
+ block/qapi-sysemu.c                           |  3 +-
+ monitor/hmp-cmds.c                            |  4 +-
-Thomas Huth (2):
+ hmp-commands.hx                               | 11 ++-
-  iotests: Check for enabled drivers before testing them
+ tests/qemu-iotests/030                        | 85 +++++++++++--------
-  tests/check-block: Skip iotests when sanitizers are enabled
+ tests/qemu-iotests/040                        | 53 +++++++-----
+ tests/qemu-iotests/056                        |  2 +-
-Vladimir Sementsov-Ogievskiy (1):
+ tests/qemu-iotests/108                        |  2 +-
-  block: fix permission update in bdrv_replace_node
+ tests/qemu-iotests/149                        |  6 +-
+ tests/qemu-iotests/163                        |  5 +-
- block.c                                       |  5 +-
+ tests/qemu-iotests/205                        |  4 +-
- block/file-posix.c                            | 53 +++++++++-
+ tests/qemu-iotests/216                        | 12 +--
- block/vmdk.c                                  | 64 ++++++++----
+ tests/qemu-iotests/218                        |  5 +-
- qemu-io-cmds.c                                | 99 +++++++++++++++++--
+ tests/qemu-iotests/224                        |  4 +-
- tests/check-block.sh                          |  5 +
+ tests/qemu-iotests/242                        |  6 +-
- tests/qemu-iotests/002                        |  1 +
+ tests/qemu-iotests/245                        | 17 ++--
- tests/qemu-iotests/003                        |  1 +
+ tests/qemu-iotests/255                        |  4 +-
- tests/qemu-iotests/005                        |  3 +-
+ tests/qemu-iotests/258                        | 11 ++-
- tests/qemu-iotests/009                        |  1 +
+ tests/qemu-iotests/298                        | 17 ++--
- tests/qemu-iotests/010                        |  1 +
+ tests/qemu-iotests/303                        |  4 +-
- tests/qemu-iotests/011                        |  1 +
+ tests/qemu-iotests/310                        | 22 ++---
- tests/qemu-iotests/017                        |  3 +-
+ tests/qemu-iotests/iotests.py                 | 69 ++++++++-------
- tests/qemu-iotests/018                        |  3 +-
+ tests/qemu-iotests/tests/image-fleecing       | 30 ++++---
- tests/qemu-iotests/019                        |  3 +-
+ .../qemu-iotests/tests/migration-permissions  | 28 +++---
- tests/qemu-iotests/020                        |  3 +-
+ .../tests/mirror-ready-cancel-error           |  2 +-
- tests/qemu-iotests/026                        |  4 +-
+ .../qemu-iotests/tests/nbd-reconnect-on-open  |  2 +-
- tests/qemu-iotests/027                        |  1 +
+ .../qemu-iotests/tests/stream-error-on-reset  |  4 +-
- tests/qemu-iotests/032                        |  1 +
+ ui/cocoa.m                                    |  1 +
- tests/qemu-iotests/033                        |  1 +
+files changed, 231 insertions(+), 188 deletions(-)
  tests/qemu-iotests/034                        |  3 +-
  tests/qemu-iotests/037                        |  3 +-
  tests/qemu-iotests/039                        |  4 +-
  tests/qemu-iotests/052                        |  2 +-
  tests/qemu-iotests/059                        | 34 ++++++-
  tests/qemu-iotests/059.out                    | 26 +++--
  tests/qemu-iotests/063                        |  3 +-
  tests/qemu-iotests/071                        |  1 +
  tests/qemu-iotests/072                        |  1 +
  tests/qemu-iotests/081                        |  4 +-
  tests/qemu-iotests/091                        |  4 +-
  tests/qemu-iotests/099                        |  1 +
  tests/qemu-iotests/105                        |  3 +-
  tests/qemu-iotests/110                        |  3 +-
  tests/qemu-iotests/120                        |  1 +
  tests/qemu-iotests/126                        |  2 +
  tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
  tests/qemu-iotests/150.out.raw                | 12 +++
  tests/qemu-iotests/162                        |  4 +-
  tests/qemu-iotests/175                        | 47 +++++++--
  tests/qemu-iotests/175.out                    | 16 ++-
  tests/qemu-iotests/178.out.qcow2              |  4 +-
  tests/qemu-iotests/184                        |  1 +
  tests/qemu-iotests/186                        |  1 +
  tests/qemu-iotests/197                        |  1 +
  tests/qemu-iotests/215                        |  1 +
  tests/qemu-iotests/221.out                    | 12 ++-
  tests/qemu-iotests/251                        |  1 +
  tests/qemu-iotests/253.out                    | 12 ++-
  tests/qemu-iotests/common.filter              |  4 +-
  tests/qemu-iotests/common.rc                  | 14 +++
 files changed, 391 insertions(+), 87 deletions(-)
  rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
  create mode 100644 tests/qemu-iotests/150.out.raw
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 01/15] qemu-io: add pattern file for write command
+Deleted patch
-From: Denis Plotnikov <dplotnikov@virtuozzo.com>
-The patch allows to provide a pattern file for write
-command. There was no similar ability before.
-Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
-Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
-Reviewed-by: Eric Blake <eblake@redhat.com>
-[mreitz: Keep optstring in alphabetical order]
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
-file changed, 93 insertions(+), 6 deletions(-)
-diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
-index XXXXXXX..XXXXXXX 100644
---- a/qemu-io-cmds.c
-+++ b/qemu-io-cmds.c
-@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
-     qemu_vfree(p);
- }
-+/*
-+ * qemu_io_alloc_from_file()
-+ *
-+ * Allocates the buffer and populates it with the content of the given file
-+ * up to @len bytes. If the file length is less than @len, then the buffer
-+ * is populated with the file content cyclically.
-+ *
-+ * @blk - the block backend where the buffer content is going to be written to
-+ * @len - the buffer length
-+ * @file_name - the file to read the content from
-+ *
-+ * Returns: the buffer pointer on success
-+ *          NULL on error
-+ */
-+static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
-+                                     const char *file_name)
-+{
-+    char *buf, *buf_origin;
-+    FILE *f = fopen(file_name, "r");
-+    int pattern_len;
-+
-+    if (!f) {
-+        perror(file_name);
-+        return NULL;
-+    }
-+
-+    if (qemuio_misalign) {
-+        len += MISALIGN_OFFSET;
-+    }
-+
-+    buf_origin = buf = blk_blockalign(blk, len);
-+
-+    if (qemuio_misalign) {
-+        buf_origin += MISALIGN_OFFSET;
-+        buf += MISALIGN_OFFSET;
-+        len -= MISALIGN_OFFSET;
-+    }
-+
-+    pattern_len = fread(buf_origin, 1, len, f);
-+
-+    if (ferror(f)) {
-+        perror(file_name);
-+        goto error;
-+    }
-+
-+    if (pattern_len == 0) {
-+        fprintf(stderr, "%s: file is empty\n", file_name);
-+        goto error;
-+    }
-+
-+    fclose(f);
-+
-+    if (len > pattern_len) {
-+        len -= pattern_len;
-+        buf += pattern_len;
-+
-+        while (len > 0) {
-+            size_t len_to_copy = MIN(pattern_len, len);
-+
-+            memcpy(buf, buf_origin, len_to_copy);
-+
-+            len -= len_to_copy;
-+            buf += len_to_copy;
-+        }
-+    }
-+
-+    return buf_origin;
-+
-+error:
-+    qemu_io_free(buf_origin);
-+    return NULL;
-+}
-+
- static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
- {
-     uint64_t i;
-@@ -XXX,XX +XXX,XX @@ static void write_help(void)
- " -n, -- with -z, don't allow slow fallback\n"
- " -p, -- ignored for backwards compatibility\n"
- " -P, -- use different pattern to fill file\n"
-+" -s, -- use a pattern file to fill the write buffer\n"
- " -C, -- report statistics in a machine parsable format\n"
- " -q, -- quiet mode, do not show I/O statistics\n"
- " -u, -- with -z, allow unmapping\n"
-@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
-     .perm       = BLK_PERM_WRITE,
-     .argmin     = 2,
-     .argmax     = -1,
--    .args       = "[-bcCfnquz] [-P pattern] off len",
-+    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
-     .oneline    = "writes a number of bytes at a specified offset",
-     .help       = write_help,
- };
-@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
- {
-     struct timespec t1, t2;
-     bool Cflag = false, qflag = false, bflag = false;
--    bool Pflag = false, zflag = false, cflag = false;
-+    bool Pflag = false, zflag = false, cflag = false, sflag = false;
-     int flags = 0;
-     int c, cnt, ret;
-     char *buf = NULL;
-@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
-     /* Some compilers get confused and warn if this is not initialized.  */
-     int64_t total = 0;
-     int pattern = 0xcd;
-+    const char *file_name = NULL;
--    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
-+    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
-         switch (c) {
-         case 'b':
-             bflag = true;
-@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
-         case 'q':
-             qflag = true;
-             break;
-+        case 's':
-+            sflag = true;
-+            file_name = optarg;
-+            break;
-         case 'u':
-             flags |= BDRV_REQ_MAY_UNMAP;
-             break;
-@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
-         return -EINVAL;
-     }
--    if (zflag && Pflag) {
--        printf("-z and -P cannot be specified at the same time\n");
-+    if (zflag + Pflag + sflag > 1) {
-+        printf("Only one of -z, -P, and -s "
-+               "can be specified at the same time\n");
-         return -EINVAL;
-     }
-@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
-     }
-     if (!zflag) {
--        buf = qemu_io_alloc(blk, count, pattern);
-+        if (sflag) {
-+            buf = qemu_io_alloc_from_file(blk, count, file_name);
-+            if (!buf) {
-+                return -EINVAL;
-+            }
-+        } else {
-+            buf = qemu_io_alloc(blk, count, pattern);
-+        }
-     }
-     clock_gettime(CLOCK_MONOTONIC, &t1);
---
-.21.0

-[Qemu-devel] [PULL 06/15] vmdk: Use bdrv_dirname() for relative extent paths
+[PULL 01/14] block: add 'force' parameter to 'blockdev-change-medium' command
-This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
+From: "Denis V. Lunev" <den@openvz.org>
 turns a former error in 059 into success.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+'blockdev-change-medium' is a convinient wrapper for the following
-Message-id: 20190815153638.4600-3-mreitz@redhat.com
+sequence of commands:
-Reviewed-by: John Snow <jsnow@redhat.com>
+ * blockdev-open-tray
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+ * blockdev-remove-medium
  * blockdev-insert-medium
  * blockdev-close-tray
 and should be used f.e. to change ISO image inside the CD-ROM tray.
 Though the guest could lock the tray and some linux guests like
 CentOS 8.5 actually does that. In this case the execution if this
 command results in the error like the following:
   Device 'scsi0-0-1-0' is locked and force was not specified,
   wait for tray to open and try again.
 This situation is could be resolved 'blockdev-open-tray' by passing
 flag 'force' inside. Thus is seems reasonable to add the same
 capability for 'blockdev-change-medium' too.
 Signed-off-by: Denis V. Lunev <den@openvz.org>
 Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@openvz.org>
 Acked-by: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
 CC: Kevin Wolf <kwolf@redhat.com>
 CC: Hanna Reitz <hreitz@redhat.com>
 CC: Eric Blake <eblake@redhat.com>
 CC: Markus Armbruster <armbru@redhat.com>
 Message-Id: <20220412221846.280723-1-den@openvz.org>
 Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
+ qapi/block.json     |  6 ++++++
- tests/qemu-iotests/059     |  7 +++--
+ block/qapi-sysemu.c |  3 ++-
- tests/qemu-iotests/059.out |  4 ++-
+ monitor/hmp-cmds.c  |  4 +++-
-files changed, 42 insertions(+), 23 deletions(-)
+ hmp-commands.hx     | 11 +++++++----
  ui/cocoa.m          |  1 +
 files changed, 19 insertions(+), 6 deletions(-)
-diff --git a/block/vmdk.c b/block/vmdk.c
+diff --git a/qapi/block.json b/qapi/block.json
 index XXXXXXX..XXXXXXX 100644
---- a/block/vmdk.c
+--- a/qapi/block.json
-+++ b/block/vmdk.c
++++ b/qapi/block.json
-@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
+@@ -XXX,XX +XXX,XX @@
- }
+ # @read-only-mode: change the read-only mode of the device; defaults
+ #                  to 'retain'
- static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
+ #
--                              const char *desc_file_path, QDict *options,
++# @force: if false (the default), an eject request through blockdev-open-tray
--                              Error **errp)
++#         will be sent to the guest if it has locked the tray (and the tray
-+                              QDict *options, Error **errp)
++#         will not be opened immediately); if true, the tray will be opened
- {
++#         regardless of whether it is locked. (since 7.1)
-     int ret;
++#
-     int matches;
+ # Features:
-@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
+ # @deprecated: Member @device is deprecated.  Use @id instead.
-     const char *p, *np;
+ #
-     int64_t sectors = 0;
+@@ -XXX,XX +XXX,XX @@
-     int64_t flat_offset;
+             '*id': 'str',
-+    char *desc_file_dir = NULL;
+             'filename': 'str',
-     char *extent_path;
+             '*format': 'str',
-     BdrvChild *extent_file;
++            '*force': 'bool',
-     BDRVVmdkState *s = bs->opaque;
+             '*read-only-mode': 'BlockdevChangeReadOnlyMode' } }
-@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-             continue;
 diff --git a/block/qapi-sysemu.c b/block/qapi-sysemu.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/qapi-sysemu.c
 +++ b/block/qapi-sysemu.c
@@ -XXX,XX +XXX,XX @@ void qmp_blockdev_change_medium(bool has_device, const char *device,
                                  bool has_id, const char *id,
                                  const char *filename,
                                  bool has_format, const char *format,
 +                                bool has_force, bool force,
                                  bool has_read_only,
                                  BlockdevChangeReadOnlyMode read_only,
                                  Error **errp)
@@ -XXX,XX +XXX,XX @@ void qmp_blockdev_change_medium(bool has_device, const char *device,
      rc = do_open_tray(has_device ? device : NULL,
                        has_id ? id : NULL,
 -                      false, &err);
 +                      force, &err);
      if (rc && rc != -ENOSYS) {
          error_propagate(errp, err);
          goto fail;
 diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
 index XXXXXXX..XXXXXXX 100644
 --- a/monitor/hmp-cmds.c
 +++ b/monitor/hmp-cmds.c
@@ -XXX,XX +XXX,XX @@ void hmp_change(Monitor *mon, const QDict *qdict)
      const char *target = qdict_get_str(qdict, "target");
      const char *arg = qdict_get_try_str(qdict, "arg");
      const char *read_only = qdict_get_try_str(qdict, "read-only-mode");
 +    bool force = qdict_get_try_bool(qdict, "force", false);
      BlockdevChangeReadOnlyMode read_only_mode = 0;
      Error *err = NULL;
@@ -XXX,XX +XXX,XX @@ void hmp_change(Monitor *mon, const QDict *qdict)
          }
--        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
+         qmp_blockdev_change_medium(true, device, false, NULL, target,
--            !desc_file_path[0])
+-                                   !!arg, arg, !!read_only, read_only_mode,
--        {
++                                   !!arg, arg, true, force,
--            bdrv_refresh_filename(bs->file->bs);
++                                   !!read_only, read_only_mode,
--            error_setg(errp, "Cannot use relative extent paths with VMDK "
+                                    &err);
 -                       "descriptor file '%s'", bs->file->bs->filename);
 -            return -EINVAL;
 -        }
 +        if (path_is_absolute(fname)) {
 +            extent_path = g_strdup(fname);
 +        } else {
 +            if (!desc_file_dir) {
 +                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
 +                if (!desc_file_dir) {
 +                    bdrv_refresh_filename(bs->file->bs);
 +                    error_prepend(errp, "Cannot use relative paths with VMDK "
 +                                  "descriptor file '%s': ",
 +                                  bs->file->bs->filename);
 +                    ret = -EINVAL;
 +                    goto out;
 +                }
 +            }
 -        extent_path = path_combine(desc_file_path, fname);
 +            extent_path = g_strconcat(desc_file_dir, fname, NULL);
 +        }
          ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
          assert(ret < 32);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          g_free(extent_path);
          if (local_err) {
              error_propagate(errp, local_err);
 -            return -EINVAL;
 +            ret = -EINVAL;
 +            goto out;
          }
          /* save to extents array */
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
 , 0, 0, 0, 0, &extent, errp);
              if (ret < 0) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent->flat_start_offset = flat_offset << 9;
          } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
              g_free(buf);
              if (ret) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent = &s->extents[s->num_extents - 1];
          } else if (!strcmp(type, "SESPARSE")) {
              ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
              if (ret) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent = &s->extents[s->num_extents - 1];
          } else {
              error_setg(errp, "Unsupported extent type '%s'", type);
              bdrv_unref_child(bs, extent_file);
 -            return -ENOTSUP;
 +            ret = -ENOTSUP;
 +            goto out;
          }
          extent->type = g_strdup(type);
      }
--    return 0;
 diff --git a/hmp-commands.hx b/hmp-commands.hx
 index XXXXXXX..XXXXXXX 100644
 --- a/hmp-commands.hx
 +++ b/hmp-commands.hx
@@ -XXX,XX +XXX,XX @@ ERST
      {
          .name       = "change",
 -        .args_type  = "device:B,target:F,arg:s?,read-only-mode:s?",
 -        .params     = "device filename [format [read-only-mode]]",
 -        .help       = "change a removable medium, optional format",
 +        .args_type  = "device:B,force:-f,target:F,arg:s?,read-only-mode:s?",
 +        .params     = "device [-f] filename [format [read-only-mode]]",
 +        .help       = "change a removable medium, optional format, use -f to force the operation",
          .cmd        = hmp_change,
      },
@@ -XXX,XX +XXX,XX @@ SRST
  ``change`` *device* *setting*
    Change the configuration of a device.
 -  ``change`` *diskdevice* *filename* [*format* [*read-only-mode*]]
 +  ``change`` *diskdevice* [-f] *filename* [*format* [*read-only-mode*]]
      Change the medium for a removable disk device to point to *filename*. eg::
        (qemu) change ide1-cd0 /path/to/some.iso
 +    ``-f``
 +      forces the operation even if the guest has locked the tray.
 +
-+    ret = 0;
+     *format* is optional.
-+    goto out;
+     *read-only-mode* may be used to change the read-only status of the device.
- invalid:
+diff --git a/ui/cocoa.m b/ui/cocoa.m
      np = next_line(p);
@@ -XXX,XX +XXX,XX @@ invalid:
          np--;
      }
      error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
 -    return -EINVAL;
 +    ret = -EINVAL;
 +
 +out:
 +    g_free(desc_file_dir);
 +    return ret;
  }
  static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
      }
      s->create_type = g_strdup(ct);
      s->desc_offset = 0;
 -    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
 -                             errp);
 +    ret = vmdk_parse_extents(buf, bs, options, errp);
  exit:
      return ret;
  }
 diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/059
 +++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
  echo
  echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 +# Should work, because bdrv_dirname() works fine with blkdebug
  IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 -$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
 -    | _filter_testdir | _filter_imgfmt
 +$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
 +         -c info \
 +    2>&1 \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
  echo
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/059.out
+--- a/ui/cocoa.m
-+++ b/tests/qemu-iotests/059.out
++++ b/ui/cocoa.m
-@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
+@@ -XXX,XX +XXX,XX @@ - (void)changeDeviceMedia:(id)sender
+                                        [file cStringUsingEncoding:
- === Testing monolithicFlat with internally generated JSON file name ===
+                                                  NSASCIIStringEncoding],
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+                                        true, "raw",
--qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
++                                       true, false,
-+format name: IMGFMT
+                                        false, 0,
-+cluster size: 0 bytes
+                                        &err);
-+vm state offset: 0 bytes
+         });
  === Testing version 3 ===
  image: TEST_DIR/iotest-version3.IMGFMT
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 14/15] tests/check-block: Skip iotests when sanitizers are enabled
+[PULL 02/14] iotests: replace calls to log(qemu_io(...)) with qemu_io_log()
-From: Thomas Huth <thuth@redhat.com>
+From: John Snow <jsnow@redhat.com>
-The sanitizers (especially the address sanitizer from Clang) are
+This makes these callsites a little simpler, but the real motivation is
-sometimes printing out warnings or false positives - this spoils
+a forthcoming commit will change the return type of qemu_io(), so removing
-the output of the iotests, causing some of the tests to fail.
+users of the return value now is helpful.
 Thus let's skip the automatic iotests during "make check" when the
 user configured QEMU with --enable-sanitizers.
-Signed-off-by: Thomas Huth <thuth@redhat.com>
+Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190823084203.29734-1-thuth@redhat.com
+Reviewed-by: Eric Blake <eblake@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-2-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- tests/check-block.sh | 5 +++++
+ tests/qemu-iotests/242 | 6 +++---
-file changed, 5 insertions(+)
+ tests/qemu-iotests/255 | 4 +---
  tests/qemu-iotests/303 | 4 ++--
 files changed, 6 insertions(+), 8 deletions(-)
-diff --git a/tests/check-block.sh b/tests/check-block.sh
+diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242
 index XXXXXXX..XXXXXXX 100755
---- a/tests/check-block.sh
+--- a/tests/qemu-iotests/242
-+++ b/tests/check-block.sh
++++ b/tests/qemu-iotests/242
-@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
+@@ -XXX,XX +XXX,XX @@
-     exit 0
+ import iotests
- fi
+ import json
+ import struct
-+if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
+-from iotests import qemu_img_create, qemu_io, qemu_img_info, \
-+    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
+-    file_path, img_info_log, log, filter_qemu_io
-+    exit 0
++from iotests import qemu_img_create, qemu_io_log, qemu_img_info, \
-+fi
++    file_path, img_info_log, log
-+
- if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
+ iotests.script_initialize(supported_fmts=['qcow2'],
-     echo "No qemu-system binary available ==> Not running the qemu-iotests."
+                           supported_protocols=['file'],
-     exit 0
+@@ -XXX,XX +XXX,XX @@ def add_bitmap(bitmap_number, persistent, disabled):
  def write_to_disk(offset, size):
      write = 'write {} {}'.format(offset, size)
 -    log(qemu_io('-c', write, disk), filters=[filter_qemu_io])
 +    qemu_io_log('-c', write, disk)
  def toggle_flag(offset):
 diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/255
 +++ b/tests/qemu-iotests/255
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('src.qcow2') as src_path, \
      iotests.qemu_img_create('-f', iotests.imgfmt, src_path, size_str)
      iotests.qemu_img_create('-f', iotests.imgfmt, dst_path, size_str)
 -    iotests.log(iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write 0 1M',
 -                                src_path),
 -                filters=[iotests.filter_test_dir, iotests.filter_qemu_io])
 +    iotests.qemu_io_log('-f', iotests.imgfmt, '-c', 'write 0 1M', src_path),
      vm.add_object('throttle-group,x-bps-read=4096,id=throttle0')
 diff --git a/tests/qemu-iotests/303 b/tests/qemu-iotests/303
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/303
 +++ b/tests/qemu-iotests/303
@@ -XXX,XX +XXX,XX @@
  import iotests
  import subprocess
 -from iotests import qemu_img_create, qemu_io, file_path, log, filter_qemu_io, \
 +from iotests import qemu_img_create, qemu_io_log, file_path, log, \
          verify_qcow2_zstd_compression
  iotests.script_initialize(supported_fmts=['qcow2'],
@@ -XXX,XX +XXX,XX @@ def create_bitmap(bitmap_number, disabled):
  def write_to_disk(offset, size):
      write = f'write {offset} {size}'
 -    log(qemu_io('-c', write, disk), filters=[filter_qemu_io])
 +    qemu_io_log('-c', write, disk)
  def add_bitmap(num, begin, end, disabled):
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 13/15] iotests: Check for enabled drivers before testing them
+[PULL 03/14] iotests/163: Fix broken qemu-io invocation
-From: Thomas Huth <thuth@redhat.com>
+From: John Snow <jsnow@redhat.com>
-It is possible to enable only a subset of the block drivers with the
+The 'read' commands to qemu-io were malformed, and this invocation only
-"--block-drv-rw-whitelist" option of the "configure" script. All other
+worked by coincidence because the error messages were identical. Oops.
 drivers are marked as unusable (or only included as read-only with the
 "--block-drv-ro-whitelist" option). If an iotest is now using such a
 disabled block driver, it is failing - which is bad, since at least the
 tests in the "auto" group should be able to deal with this situation.
 Thus let's introduce a "_require_drivers" function that can be used by
 the shell tests to check for the availability of certain drivers first,
 and marks the test as "not run" if one of the drivers is missing.
-This patch mainly targets the test in the "auto" group which should
+There's no point in checking the patterning of the reference image, so
-never fail in such a case, but also improves some of the other tests
+just check the empty image by itself instead.
 along the way. Note that we also assume that the "qcow2" and "file"
 drivers are always available - otherwise it does not make sense to
 run "make check-block" at all (which only tests with qcow2 by default).
-Signed-off-by: Thomas Huth <thuth@redhat.com>
+(Note: as of this commit, nothing actually enforces that this command
-Message-id: 20190823133552.11680-1-thuth@redhat.com
+completes successfully, but a forthcoming commit in this series will
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+enforce that qemu_io() must have a zero status code.)
 Signed-off-by: John Snow <jsnow@redhat.com>
 Reviewed-by: Eric Blake <eblake@redhat.com>
 Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-3-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- tests/qemu-iotests/071       |  1 +
+ tests/qemu-iotests/163 | 5 +----
- tests/qemu-iotests/081       |  4 +---
+file changed, 1 insertion(+), 4 deletions(-)
  tests/qemu-iotests/099       |  1 +
  tests/qemu-iotests/120       |  1 +
  tests/qemu-iotests/162       |  4 +---
  tests/qemu-iotests/184       |  1 +
  tests/qemu-iotests/186       |  1 +
  tests/qemu-iotests/common.rc | 14 ++++++++++++++
 files changed, 21 insertions(+), 6 deletions(-)
-diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
+diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
 index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/071
+--- a/tests/qemu-iotests/163
-+++ b/tests/qemu-iotests/071
++++ b/tests/qemu-iotests/163
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ class ShrinkBaseClass(iotests.QMPTestCase):
+         qemu_img('resize',  '-f', iotests.imgfmt, '--shrink', test_img,
- _supported_fmt qcow2
+                  self.shrink_size)
- _supported_proto file
-+_require_drivers blkdebug blkverify
+-        self.assertEqual(
+-            qemu_io('-c', 'read -P 0x00 %s'%self.shrink_size, test_img),
- do_run_qemu()
+-            qemu_io('-c', 'read -P 0x00 %s'%self.shrink_size, check_img),
- {
+-            "Verifying image content")
-diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
++        qemu_io('-c', f"read -P 0x00 0 {self.shrink_size}", test_img)
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/081
+         self.image_verify()
-+++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt raw
  _supported_proto file
  _supported_os Linux
 +_require_drivers quorum
  do_run_qemu()
  {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                            | _filter_qemu_io | _filter_generated_node_ids
  }
 -test_quorum=$($QEMU_IMG --help|grep quorum)
 -[ "$test_quorum" = "" ] && _supported_fmt quorum
 -
  quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
  quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
  quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
 diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/099
 +++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
  _supported_proto file
  _supported_os Linux
 +_require_drivers blkdebug blkverify
  _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
      "subformat=twoGbMaxExtentSparse"
 diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/120
 +++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto file
  _unsupported_fmt luks
 +_require_drivers raw
  _make_test_img 64M
 diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/162
 +++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_fmt generic
 -
 -test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh\( \|$\)')
 -[ "$test_ssh" = "" ] && _notrun "ssh support required"
 +_require_drivers ssh
  echo
  echo '=== NBD ==='
 diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/184
 +++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_os Linux
 +_require_drivers throttle
  do_run_qemu()
  {
 diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/186
 +++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
 +_require_drivers null-co
  if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
      _notrun "Requires a PC machine"
 diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/common.rc
 +++ b/tests/qemu-iotests/common.rc
@@ -XXX,XX +XXX,XX @@ _require_command()
      [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
  }
 +# Check that a set of drivers has been whitelisted in the QEMU binary
 +#
 +_require_drivers()
 +{
 +    available=$($QEMU -drive format=help | \
 +                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
 +    for driver
 +    do
 +        if ! echo "$available" | grep -q " $driver\( \|$\)"; then
 +            _notrun "$driver not available"
 +        fi
 +    done
 +}
 +
  # make sure this script returns success
  true
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 11/15] iotests: Disable 126 for flat vmdk subformats
+[PULL 04/14] iotests: Don't check qemu_io() output for specific error strings
-iotest 126 requires backing file support, which flat vmdks cannot offer.
+From: John Snow <jsnow@redhat.com>
 Skip this test for such subformats.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+A forthcoming commit updates qemu_io() to raise an exception on non-zero
-Message-id: 20190815153638.4600-8-mreitz@redhat.com
+return by default, and changes its return type.
-Reviewed-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+In preparation, simplify some calls to qemu_io() that assert that
 specific error message strings do not appear in qemu-io's
 output. Asserting that all of these calls return a status code of zero
 will be a more robust way to guard against failure.
 Signed-off-by: John Snow <jsnow@redhat.com>
 Reviewed-by: Eric Blake <eblake@redhat.com>
 Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-4-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- tests/qemu-iotests/126 | 2 ++
+ tests/qemu-iotests/040 | 33 ++++++++++++++++-----------------
-file changed, 2 insertions(+)
+ tests/qemu-iotests/056 |  2 +-
 files changed, 17 insertions(+), 18 deletions(-)
-diff --git a/tests/qemu-iotests/126 b/tests/qemu-iotests/126
+diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
 index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/126
+--- a/tests/qemu-iotests/040
-+++ b/tests/qemu-iotests/126
++++ b/tests/qemu-iotests/040
-@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
- # Needs backing file support
+     def test_commit(self):
- _supported_fmt qcow qcow2 qed vmdk
+         self.run_commit_test(mid_img, backing_img)
-+_unsupported_imgopts "subformat=monolithicFlat" \
+-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
-+                     "subformat=twoGbMaxExtentFlat"
+-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
- # This is the default protocol (and we want to test the difference between
++        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
- # colons which separate a protocol prefix from the rest and colons which are
++        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
- # just part of the filename, so we cannot test protocols which require a prefix)
      def test_commit_node(self):
          self.run_commit_test("mid", "base", node_names=True)
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
      @iotests.skip_if_unsupported(['throttle'])
      def test_commit_with_filter_and_quit(self):
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
      def test_top_is_active(self):
          self.run_commit_test(test_img, backing_img, need_ready=True)
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
      def test_top_is_default_active(self):
          self.run_default_commit_test()
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
      def test_top_and_base_reversed(self):
          self.assert_no_active_block_jobs()
@@ -XXX,XX +XXX,XX @@ class TestRelativePaths(ImageCommitTestCase):
      def test_commit(self):
          self.run_commit_test(self.mid_img, self.backing_img)
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs).find("verification failed"))
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs).find("verification failed"))
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs)
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs)
      def test_device_not_found(self):
          result = self.vm.qmp('block-commit', device='nonexistent', top='%s' % self.mid_img)
@@ -XXX,XX +XXX,XX @@ class TestRelativePaths(ImageCommitTestCase):
      def test_top_is_active(self):
          self.run_commit_test(self.test_img, self.backing_img)
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs).find("verification failed"))
 -        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs).find("verification failed"))
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs)
 +        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs)
      def test_top_and_base_reversed(self):
          self.assert_no_active_block_jobs()
@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
      def do_test_io(self, read_or_write):
          for index, pattern_file in enumerate(self.pattern_files):
 -            result = qemu_io('-f', iotests.imgfmt,
 -                             '-c',
 -                             f'{read_or_write} -P {index + 1} {index}M 1M',
 -                             pattern_file)
 -            self.assertFalse('Pattern verification failed' in result)
 +            qemu_io('-f', iotests.imgfmt,
 +                    '-c',
 +                    f'{read_or_write} -P {index + 1} {index}M 1M',
 +                    pattern_file)
      @iotests.skip_if_unsupported(['throttle'])
      def setUp(self):
 diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/056
 +++ b/tests/qemu-iotests/056
@@ -XXX,XX +XXX,XX @@ class TestSyncModesNoneAndTop(iotests.QMPTestCase):
          self.vm.shutdown()
          time.sleep(1)
 -        self.assertEqual(-1, qemu_io('-c', 'read -P0x41 0 512', target_img).find("verification failed"))
 +        qemu_io('-c', 'read -P0x41 0 512', target_img)
  class TestBeforeWriteNotifier(iotests.QMPTestCase):
      def setUp(self):
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 15/15] iotests: Unify cache mode quoting
+[PULL 05/14] iotests/040: Don't check image pattern on zero-length image
-From: Nir Soffer <nirsof@gmail.com>
+From: John Snow <jsnow@redhat.com>
-Quoting cache mode is not needed, and most tests use unquoted values.
+qemu-io fails on read/write beyond end-of-file on raw images, so skip
-Unify all test to use the same style.
+these invocations when running the zero-length image tests.
-Message-id: 20190827173432.7656-1-nsoffer@redhat.com
+Signed-off-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Nir Soffer <nsoffer@redhat.com>
+Reviewed-by: Eric Blake <eblake@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-5-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- tests/qemu-iotests/026 | 4 ++--
+ tests/qemu-iotests/040 | 14 ++++++++++++--
- tests/qemu-iotests/039 | 4 ++--
+file changed, 12 insertions(+), 2 deletions(-)
  tests/qemu-iotests/052 | 2 +-
  tests/qemu-iotests/091 | 4 ++--
 files changed, 7 insertions(+), 7 deletions(-)
-diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
+diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
 index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/026
+--- a/tests/qemu-iotests/040
-+++ b/tests/qemu-iotests/026
++++ b/tests/qemu-iotests/040
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
- # Currently only qcow2 supports rebasing
+         qemu_img('create', '-f', iotests.imgfmt,
- _supported_fmt qcow2
+                  '-o', 'backing_file=%s' % mid_img,
- _supported_proto file
+                  '-F', iotests.imgfmt, test_img)
--_default_cache_mode "writethrough"
+-        qemu_io('-f', 'raw', '-c', 'write -P 0xab 0 524288', backing_img)
--_supported_cache_modes "writethrough" "none"
+-        qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xef 524288 524288', mid_img)
-+_default_cache_mode writethrough
++        if self.image_len:
-+_supported_cache_modes writethrough none
++            qemu_io('-f', 'raw', '-c', 'write -P 0xab 0 524288', backing_img)
- # The refcount table tests expect a certain minimum width for refcount entries
++            qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xef 524288 524288',
- # (so that the refcount table actually needs to grow); that minimum is 16 bits,
++                    mid_img)
- # being the default refcount entry width.
+         self.vm = iotests.VM().add_drive(test_img, "node-name=top,backing.node-name=mid,backing.backing.node-name=base", interface="none")
-diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
+         self.vm.add_device('virtio-scsi')
-index XXXXXXX..XXXXXXX 100755
+         self.vm.add_device("scsi-hd,id=scsi0,drive=drive0")
---- a/tests/qemu-iotests/039
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
-+++ b/tests/qemu-iotests/039
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+     def test_commit(self):
- _supported_fmt qcow2
+         self.run_commit_test(mid_img, backing_img)
- _supported_proto file
++        if not self.image_len:
- _supported_os Linux
++            return
--_default_cache_mode "writethrough"
+         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
--_supported_cache_modes "writethrough"
+         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
-+_default_cache_mode writethrough
-+_supported_cache_modes writethrough
+     def test_commit_node(self):
+         self.run_commit_test("mid", "base", node_names=True)
- size=128M
++        if not self.image_len:
++            return
-diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
+         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
-index XXXXXXX..XXXXXXX 100755
+         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
---- a/tests/qemu-iotests/052
-+++ b/tests/qemu-iotests/052
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
-@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
- _supported_proto file
+     def test_top_is_active(self):
+         self.run_commit_test(test_img, backing_img, need_ready=True)
- # Don't do O_DIRECT on tmpfs
++        if not self.image_len:
--_supported_cache_modes "writeback" "writethrough" "unsafe"
++            return
-+_supported_cache_modes writeback writethrough unsafe
+         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
+         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
- size=128M
- _make_test_img $size
+     def test_top_is_default_active(self):
-diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
+         self.run_default_commit_test()
-index XXXXXXX..XXXXXXX 100755
++        if not self.image_len:
---- a/tests/qemu-iotests/091
++            return
-+++ b/tests/qemu-iotests/091
+         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
  _supported_fmt qcow2
  _supported_proto file
  _supported_os Linux
 -_default_cache_mode "none"
 -_supported_cache_modes "writethrough" "none" "writeback"
 +_default_cache_mode none
 +_supported_cache_modes writethrough none writeback
  size=1G
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 10/15] iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
+[PULL 06/14] iotests/040: Fix TestCommitWithFilters test
-The error message for the test case where we have a quorum node for
+From: John Snow <jsnow@redhat.com>
 which no directory name can be generated is different: For
 twoGbMaxExtentSparse, it complains that it cannot open the extent file.
 For other (sub)formats, it just notes that it cannot determine the
 backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
 for simplicity's sake.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Without this change, asserting that qemu_io always returns 0 causes this
-Reviewed-by: John Snow <jsnow@redhat.com>
+test to fail in a way we happened not to be catching previously:
-Message-id: 20190815153638.4600-7-mreitz@redhat.com
-Reviewed-by: John Snow <jsnow@redhat.com>
+ qemu.utils.VerboseProcessError: Command
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+  '('/home/jsnow/src/qemu/bin/git/tests/qemu-iotests/../../qemu-io',
   '--cache', 'writeback', '--aio', 'threads', '-f', 'qcow2', '-c',
   'read -P 4 3M 1M',
   '/home/jsnow/src/qemu/bin/git/tests/qemu-iotests/scratch/3.img')'
   returned non-zero exit status 1.
   ┏━ output ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   ┃ qemu-io: can't open device
   ┃ /home/jsnow/src/qemu/bin/git/tests/qemu-iotests/scratch/3.img:
   ┃ Could not open backing file: Could not open backing file: Throttle
   ┃ group 'tg' does not exist
   ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 The commit jobs changes the backing file string stored in the image file
 header belonging to the node above the commit’s top node to point to the
 commit target (the base node).  QEMU tries to be as accurate as
 possible, and so in these test cases will include the filter that is
 part of the block graph in that backing file string (by virtue of making
 it a json:{} description of the post-commit subgraph).  This makes
 little sense outside of QEMU, though: Specifically, the throttle node in
 that subgraph will dearly miss its supposedly associated throttle group
 object.
 When starting the commit job, we can specify a custom backing file
 string to write into said image file, so let’s use that feature to write
 the plain filename of the backing chain’s next actual image file there.
 Explicitly provide the backing file so that opening the file outside of
 QEMU (Where we will not have throttle groups) will succeed.
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 Signed-off-by: John Snow <jsnow@redhat.com>
 Reviewed-by: Eric Blake <eblake@redhat.com>
 Message-Id: <20220418211504.943969-6-jsnow@redhat.com>
 ---
- tests/qemu-iotests/110 | 3 ++-
+ tests/qemu-iotests/040 | 6 ++++--
-file changed, 2 insertions(+), 1 deletion(-)
+file changed, 4 insertions(+), 2 deletions(-)
-diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
+diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
 index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/110
+--- a/tests/qemu-iotests/040
-+++ b/tests/qemu-iotests/110
++++ b/tests/qemu-iotests/040
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
- # Any format supporting backing files
+                              job_id='commit',
- _supported_fmt qed qcow qcow2 vmdk
+                              device='top-filter',
- _supported_proto file
+                              top_node='cow-2',
--_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+-                             base_node='cow-1')
-+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
++                             base_node='cow-1',
-+                     "subformat=twoGbMaxExtentSparse"
++                             backing_file=self.img1)
+         self.assert_qmp(result, 'return', {})
- TEST_IMG_REL=$(basename "$TEST_IMG")
+         self.wait_until_completed(drive='commit')
@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
                               job_id='commit',
                               device='top-filter',
                               top_node='cow-1',
 -                             base_node='cow-0')
 +                             base_node='cow-0',
 +                             backing_file=self.img0)
          self.assert_qmp(result, 'return', {})
          self.wait_until_completed(drive='commit')
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 07/15] iotests: Keep testing broken relative extent paths
+[PULL 07/14] iotests: create generic qemu_tool() function
-We had a test for a case where relative extent paths did not work, but
+From: John Snow <jsnow@redhat.com>
 unfortunately we just fixed the underlying problem, so it works now.
 This patch adds a new test case that still fails.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+reimplement qemu_img() in terms of qemu_tool() in preparation for doing
-Reviewed-by: John Snow <jsnow@redhat.com>
+the same with qemu_io().
-Message-id: 20190815153638.4600-4-mreitz@redhat.com
-Reviewed-by: John Snow <jsnow@redhat.com>
+Signed-off-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Reviewed-by: Eric Blake <eblake@redhat.com>
 Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-7-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
+ tests/qemu-iotests/iotests.py | 32 +++++++++++++++++++++-----------
- tests/qemu-iotests/059.out |  4 ++++
+file changed, 21 insertions(+), 11 deletions(-)
 files changed, 31 insertions(+)
-diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
+diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/059
+--- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/059
++++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
+@@ -XXX,XX +XXX,XX @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
- echo
+     return result
- echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 -def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
 -             ) -> 'subprocess.CompletedProcess[str]':
 -    """
 -    Run qemu_img and return the status code and console output.
 -    This function always prepends QEMU_IMG_OPTIONS and may further alter
 -    the args for 'create' commands.
 +def qemu_tool(*args: str, check: bool = True, combine_stdio: bool = True
 +              ) -> 'subprocess.CompletedProcess[str]':
 +    """
 +    Run a qemu tool and return its status code and console output.
 -    :param args: command-line arguments to qemu-img.
 +    :param args: full command line to run.
      :param check: Enforce a return code of zero.
      :param combine_stdio: set to False to keep stdout/stderr separated.
@@ -XXX,XX +XXX,XX @@ def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
          properties. If streams are not combined, it will also have a
          stderr property.
      """
 -    full_args = qemu_img_args + qemu_img_create_prepare_args(list(args))
 -
      subp = subprocess.run(
 -        full_args,
 +        args,
          stdout=subprocess.PIPE,
          stderr=subprocess.STDOUT if combine_stdio else subprocess.PIPE,
          universal_newlines=True,
@@ -XXX,XX +XXX,XX @@ def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
      if check and subp.returncode or (subp.returncode < 0):
          raise VerboseProcessError(
 -            subp.returncode, full_args,
 +            subp.returncode, args,
              output=subp.stdout,
              stderr=subp.stderr,
          )
@@ -XXX,XX +XXX,XX @@ def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
      return subp
 +def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
 +             ) -> 'subprocess.CompletedProcess[str]':
 +    """
 +    Run QEMU_IMG_PROG and return its status code and console output.
 +
-+echo '--- blkdebug ---'
++    This function always prepends QEMU_IMG_OPTIONS and may further alter
- # Should work, because bdrv_dirname() works fine with blkdebug
++    the args for 'create' commands.
  IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
  $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
      | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
 +echo '--- quorum ---'
 +# Should not work, because bdrv_dirname() does not work with quorum
 +IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 +cp "$TEST_IMG" "$TEST_IMG.orig"
 +
-+filename="json:{
++    See `qemu_tool()` for greater detail.
-+    \"driver\": \"$IMGFMT\",
++    """
-+    \"file\": {
++    full_args = qemu_img_args + qemu_img_create_prepare_args(list(args))
-+        \"driver\": \"quorum\",
++    return qemu_tool(*full_args, check=check, combine_stdio=combine_stdio)
 +        \"children\": [ {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG\"
 +        }, {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG.orig\"
 +        } ],
 +        \"vote-threshold\": 1
 +    } }"
 +
 +filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
 +$QEMU_IMG info "$filename" 2>&1 \
 +    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
 +
 +
- echo
+ def ordered_qmp(qmsg, conv_keys=True):
- echo "=== Testing version 3 ==="
+     # Dictionaries are not ordered prior to 3.6, therefore:
- _use_sample_img iotest-version3.vmdk.bz2
+     if isinstance(qmsg, list):
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/059.out
 +++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing monolithicFlat with internally generated JSON file name ===
 +--- blkdebug ---
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
  format name: IMGFMT
  cluster size: 0 bytes
  vm state offset: 0 bytes
 +--- quorum ---
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 +qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
  === Testing version 3 ===
  image: TEST_DIR/iotest-version3.IMGFMT
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 09/15] iotests: Disable broken streamOptimized tests
+[PULL 08/14] iotests: rebase qemu_io() on top of qemu_tool()
-streamOptimized does not support writes that do not span exactly one
+From: John Snow <jsnow@redhat.com>
-cluster.  Furthermore, it cannot rewrite already allocated clusters.
-As such, many iotests do not work with it.  Disable them.
+Rework qemu_io() to be analogous to qemu_img(); a function that requires
+a return code of zero by default unless disabled explicitly.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20190815153638.4600-6-mreitz@redhat.com
+Tests that use qemu_io():
-Reviewed-by: John Snow <jsnow@redhat.com>
+040 041 044 055 056 093 124 129 132 136 148 149 151 152 163 165 205
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+219 236 245 248 254 255 257 260 264 280 298 300 302 304
 image-fleecing migrate-bitmaps-postcopy-test migrate-bitmaps-test
 migrate-during-backup migration-permissions
 Test that use qemu_io_log():
 245 255 274 303 307 nbd-reconnect-on-open
 Copy-pastables for testing/verification:
 ./check -qcow2 030 040 041 044 055 056 124 129 132 151 152 163 165 209 \
 236 242 245 248 254 255 257 260 264 274 \
 298 300 302 303 304 307 image-fleecing \
                migrate-bitmaps-postcopy-test migrate-bitmaps-test \
                migrate-during-backup nbd-reconnect-on-open
 ./check -raw 093 136 148 migration-permissions
 ./check -nbd 205
 # ./configure configure --disable-gnutls --enable-gcrypt
 # this ALSO requires passwordless sudo.
 ./check -luks 149
 # Just the tests that were edited in this commit:
 ./check -qcow2 030 040 242 245
 ./check -raw migration-permissions
 ./check -nbd 205
 ./check -luks 149
 Signed-off-by: John Snow <jsnow@redhat.com>
 Message-Id: <20220418211504.943969-8-jsnow@redhat.com>
 Reviewed-by: Eric Blake <eblake@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- tests/qemu-iotests/002 | 1 +
+ tests/qemu-iotests/030                        | 85 +++++++++++--------
- tests/qemu-iotests/003 | 1 +
+ tests/qemu-iotests/149                        |  6 +-
- tests/qemu-iotests/005 | 3 ++-
+ tests/qemu-iotests/205                        |  4 +-
- tests/qemu-iotests/009 | 1 +
+ tests/qemu-iotests/245                        | 17 ++--
- tests/qemu-iotests/010 | 1 +
+ tests/qemu-iotests/iotests.py                 | 19 +++--
- tests/qemu-iotests/011 | 1 +
+ .../qemu-iotests/tests/migration-permissions  |  4 +-
- tests/qemu-iotests/017 | 3 ++-
+files changed, 81 insertions(+), 54 deletions(-)
- tests/qemu-iotests/018 | 3 ++-
- tests/qemu-iotests/019 | 3 ++-
+diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
- tests/qemu-iotests/020 | 3 ++-
+index XXXXXXX..XXXXXXX 100755
- tests/qemu-iotests/027 | 1 +
+--- a/tests/qemu-iotests/030
- tests/qemu-iotests/032 | 1 +
++++ b/tests/qemu-iotests/030
- tests/qemu-iotests/033 | 1 +
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
- tests/qemu-iotests/034 | 3 ++-
+         self.assert_no_active_block_jobs()
- tests/qemu-iotests/037 | 3 ++-
+         self.vm.shutdown()
- tests/qemu-iotests/063 | 3 ++-
- tests/qemu-iotests/072 | 1 +
+-        self.assertEqual(qemu_io('-f', 'raw', '-c', 'map', backing_img),
- tests/qemu-iotests/105 | 3 ++-
+-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
- tests/qemu-iotests/197 | 1 +
+-                         'image file map does not match backing file after streaming')
- tests/qemu-iotests/215 | 1 +
++        self.assertEqual(
- tests/qemu-iotests/251 | 1 +
++            qemu_io('-f', 'raw', '-c', 'map', backing_img).stdout,
-files changed, 30 insertions(+), 9 deletions(-)
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
++            'image file map does not match backing file after streaming')
-diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
-index XXXXXXX..XXXXXXX 100755
+     def test_stream_intermediate(self):
---- a/tests/qemu-iotests/002
+         self.assert_no_active_block_jobs()
-+++ b/tests/qemu-iotests/002
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+-        self.assertNotEqual(qemu_io('-f', 'raw', '-rU', '-c', 'map', backing_img),
+-                            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', mid_img),
- _supported_fmt generic
+-                            'image file map matches backing file before streaming')
- _supported_proto generic
++        self.assertNotEqual(
-+_unsupported_imgopts "subformat=streamOptimized"
++            qemu_io('-f', 'raw', '-rU', '-c', 'map', backing_img).stdout,
++            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', mid_img).stdout,
++            'image file map matches backing file before streaming')
- size=128M
-diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
+         result = self.vm.qmp('block-stream', device='mid', job_id='stream-mid')
-index XXXXXXX..XXXXXXX 100755
+         self.assert_qmp(result, 'return', {})
---- a/tests/qemu-iotests/003
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
-+++ b/tests/qemu-iotests/003
+         self.assert_no_active_block_jobs()
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+         self.vm.shutdown()
- _supported_fmt generic
+-        self.assertEqual(qemu_io('-f', 'raw', '-c', 'map', backing_img),
- _supported_proto generic
+-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img),
-+_unsupported_imgopts "subformat=streamOptimized"
+-                         'image file map does not match backing file after streaming')
++        self.assertEqual(
- size=128M
++            qemu_io('-f', 'raw', '-c', 'map', backing_img).stdout,
- offset=67M
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img).stdout,
-diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
++            'image file map does not match backing file after streaming')
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/005
+     def test_stream_pause(self):
-+++ b/tests/qemu-iotests/005
+         self.assert_no_active_block_jobs()
-@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
- _supported_proto generic
+         self.assert_no_active_block_jobs()
- _supported_os Linux
+         self.vm.shutdown()
- _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
--                     "subformat=twoGbMaxExtentSparse"
+-        self.assertEqual(qemu_io('-f', 'raw', '-c', 'map', backing_img),
-+                     "subformat=twoGbMaxExtentSparse" \
+-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
-+                     "subformat=streamOptimized"
+-                         'image file map does not match backing file after streaming')
++        self.assertEqual(
- # vpc is limited to 127GB, so we can't test it here
++            qemu_io('-f', 'raw', '-c', 'map', backing_img).stdout,
- if [ "$IMGFMT" = "vpc" ]; then
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
-diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
++            'image file map does not match backing file after streaming')
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/009
+     def test_stream_no_op(self):
-+++ b/tests/qemu-iotests/009
+         self.assert_no_active_block_jobs()
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+         # The image map is empty before the operation
- _supported_fmt generic
+-        empty_map = qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', test_img)
- _supported_proto generic
++        empty_map = qemu_io(
-+_unsupported_imgopts "subformat=streamOptimized"
++            '-f', iotests.imgfmt, '-rU', '-c', 'map', test_img).stdout
+         # This is a no-op: no data should ever be copied from the base image
- size=6G
+         result = self.vm.qmp('block-stream', device='drive0', base=mid_img)
-diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
-index XXXXXXX..XXXXXXX 100755
+         self.assert_no_active_block_jobs()
---- a/tests/qemu-iotests/010
+         self.vm.shutdown()
-+++ b/tests/qemu-iotests/010
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
+-                         empty_map, 'image file map changed after a no-op')
- _supported_fmt generic
++        self.assertEqual(
- _supported_proto generic
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
-+_unsupported_imgopts "subformat=streamOptimized"
++            empty_map, 'image file map changed after a no-op')
+     def test_stream_partial(self):
- size=6G
+         self.assert_no_active_block_jobs()
-diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
+@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
-index XXXXXXX..XXXXXXX 100755
+         self.assert_no_active_block_jobs()
---- a/tests/qemu-iotests/011
+         self.vm.shutdown()
-+++ b/tests/qemu-iotests/011
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img),
+-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
- _supported_fmt generic
+-                         'image file map does not match backing file after streaming')
- _supported_proto generic
++        self.assertEqual(
-+_unsupported_imgopts "subformat=streamOptimized"
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img).stdout,
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
++            'image file map does not match backing file after streaming')
- size=6G
-diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
+     def test_device_not_found(self):
-index XXXXXXX..XXXXXXX 100755
+         result = self.vm.qmp('block-stream', device='nonexistent')
---- a/tests/qemu-iotests/017
+@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
-+++ b/tests/qemu-iotests/017
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+         # Check that the maps don't match before the streaming operations
- _supported_fmt qcow qcow2 vmdk qed
+         for i in range(2, self.num_imgs, 2):
- _supported_proto generic
+-            self.assertNotEqual(qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i]),
- _unsupported_proto vxhs
+-                                qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i-1]),
--_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+-                                'image file map matches backing file before streaming')
-+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
++            self.assertNotEqual(
-+                     "subformat=streamOptimized"
++                qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i]).stdout,
++                qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i-1]).stdout,
- TEST_OFFSETS="0 4294967296"
++                'image file map matches backing file before streaming')
-diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
+         # Create all streaming jobs
-index XXXXXXX..XXXXXXX 100755
+         pending_jobs = []
---- a/tests/qemu-iotests/018
+@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
-+++ b/tests/qemu-iotests/018
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+         # Check that all maps match now
- _supported_fmt qcow qcow2 vmdk qed
+         for i in range(2, self.num_imgs, 2):
- _supported_proto file
+-            self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i]),
- _supported_os Linux
+-                             qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i-1]),
--_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+-                             'image file map does not match backing file after streaming')
-+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
++            self.assertEqual(
-+                     "streamOptimized"
++                qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i]).stdout,
++                qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i-1]).stdout,
- TEST_OFFSETS="0 4294967296"
++                'image file map does not match backing file after streaming')
-diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
+     # Test that it's not possible to perform two block-stream
-index XXXXXXX..XXXXXXX 100755
+     # operations if there are nodes involved in both.
---- a/tests/qemu-iotests/019
+@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
-+++ b/tests/qemu-iotests/019
+     def test_stream_base_node_name(self):
-@@ -XXX,XX +XXX,XX @@ _supported_proto file
+         self.assert_no_active_block_jobs()
- _supported_os Linux
- _unsupported_imgopts "subformat=monolithicFlat" \
+-        self.assertNotEqual(qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[4]),
-                      "subformat=twoGbMaxExtentFlat" \
+-                            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[3]),
--                     "subformat=twoGbMaxExtentSparse"
+-                            'image file map matches backing file before streaming')
-+                     "subformat=twoGbMaxExtentSparse" \
++        self.assertNotEqual(
-+                     "subformat=streamOptimized"
++            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[4]).stdout,
++            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[3]).stdout,
- TEST_OFFSETS="0 4294967296"
++            'image file map matches backing file before streaming')
- CLUSTER_SIZE=65536
-diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
+         # Error: the base node does not exist
-index XXXXXXX..XXXXXXX 100755
+         result = self.vm.qmp('block-stream', device='node4', base_node='none', job_id='stream')
---- a/tests/qemu-iotests/020
+@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
-+++ b/tests/qemu-iotests/020
+         self.assert_no_active_block_jobs()
-@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
+         self.vm.shutdown()
- _supported_proto file
- _unsupported_imgopts "subformat=monolithicFlat" \
+-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[4]),
-                      "subformat=twoGbMaxExtentFlat" \
+-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[3]),
--                     "subformat=twoGbMaxExtentSparse"
+-                         'image file map matches backing file after streaming')
-+                     "subformat=twoGbMaxExtentSparse" \
++        self.assertEqual(
-+                     "subformat=streamOptimized"
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[4]).stdout,
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[3]).stdout,
- TEST_OFFSETS="0 4294967296"
++            'image file map matches backing file after streaming')
-diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
+ class TestQuorum(iotests.QMPTestCase):
-index XXXXXXX..XXXXXXX 100755
+     num_children = 3
---- a/tests/qemu-iotests/027
+@@ -XXX,XX +XXX,XX @@ class TestQuorum(iotests.QMPTestCase):
-+++ b/tests/qemu-iotests/027
+             os.remove(img)
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+     def test_stream_quorum(self):
- _supported_fmt vmdk qcow qcow2 qed
+-        self.assertNotEqual(qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.children[0]),
- _supported_proto generic
+-                            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.backing[0]),
-+_unsupported_imgopts "subformat=streamOptimized"
+-                            'image file map matches backing file before streaming')
++        self.assertNotEqual(
++            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.children[0]).stdout,
- size=128M
++            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.backing[0]).stdout,
-diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
++            'image file map matches backing file before streaming')
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/032
+         self.assert_no_active_block_jobs()
-+++ b/tests/qemu-iotests/032
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ class TestQuorum(iotests.QMPTestCase):
- # This works for any image format (though unlikely to segfault for raw)
+         self.assert_no_active_block_jobs()
- _supported_fmt generic
+         self.vm.shutdown()
- _supported_proto generic
-+_unsupported_imgopts "subformat=streamOptimized"
+-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', self.children[0]),
+-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', self.backing[0]),
- echo
+-                         'image file map does not match backing file after streaming')
- echo === Prepare image ===
++        self.assertEqual(
-diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.children[0]).stdout,
-index XXXXXXX..XXXXXXX 100755
++            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.backing[0]).stdout,
---- a/tests/qemu-iotests/033
++            'image file map does not match backing file after streaming')
-+++ b/tests/qemu-iotests/033
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+ class TestSmallerBackingFile(iotests.QMPTestCase):
+     backing_len = 1 * 1024 * 1024 # MB
- _supported_fmt generic
+diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
- _supported_proto generic
+index XXXXXXX..XXXXXXX 100755
-+_unsupported_imgopts "subformat=streamOptimized"
+--- a/tests/qemu-iotests/149
++++ b/tests/qemu-iotests/149
+@@ -XXX,XX +XXX,XX @@ def qemu_io_write_pattern(config, pattern, offset_mb, size_mb, dev=False):
- size=128M
+     args = ["-c", "write -P 0x%x %dM %dM" % (pattern, offset_mb, size_mb)]
-diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
+     args.extend(qemu_io_image_args(config, dev))
-index XXXXXXX..XXXXXXX 100755
+     iotests.log("qemu-io " + " ".join(args), filters=[iotests.filter_test_dir])
---- a/tests/qemu-iotests/034
+-    iotests.log(check_cipher_support(config, iotests.qemu_io(*args)),
-+++ b/tests/qemu-iotests/034
++    output = iotests.qemu_io(*args, check=False).stdout
-@@ -XXX,XX +XXX,XX @@ _supported_proto file
++    iotests.log(check_cipher_support(config, output),
- _supported_os Linux
+                 filters=[iotests.filter_test_dir, iotests.filter_qemu_io])
- _unsupported_imgopts "subformat=monolithicFlat" \
-                      "subformat=twoGbMaxExtentFlat" \
--                     "subformat=twoGbMaxExtentSparse"
+@@ -XXX,XX +XXX,XX @@ def qemu_io_read_pattern(config, pattern, offset_mb, size_mb, dev=False):
-+                     "subformat=twoGbMaxExtentSparse" \
+     args = ["-c", "read -P 0x%x %dM %dM" % (pattern, offset_mb, size_mb)]
-+                     "subformat=streamOptimized"
+     args.extend(qemu_io_image_args(config, dev))
+     iotests.log("qemu-io " + " ".join(args), filters=[iotests.filter_test_dir])
- CLUSTER_SIZE=4k
+-    iotests.log(check_cipher_support(config, iotests.qemu_io(*args)),
- size=128M
++    output = iotests.qemu_io(*args, check=False).stdout
-diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
++    iotests.log(check_cipher_support(config, output),
-index XXXXXXX..XXXXXXX 100755
+                 filters=[iotests.filter_test_dir, iotests.filter_qemu_io])
---- a/tests/qemu-iotests/037
-+++ b/tests/qemu-iotests/037
-@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
+diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
- _supported_proto file
+index XXXXXXX..XXXXXXX 100755
- _unsupported_imgopts "subformat=monolithicFlat" \
+--- a/tests/qemu-iotests/205
-                      "subformat=twoGbMaxExtentFlat" \
++++ b/tests/qemu-iotests/205
--                     "subformat=twoGbMaxExtentSparse"
+@@ -XXX,XX +XXX,XX @@ class TestNbdServerRemove(iotests.QMPTestCase):
-+                     "subformat=twoGbMaxExtentSparse" \
-+                     "subformat=streamOptimized"
+     def do_test_connect_after_remove(self, mode=None):
+         args = ('-r', '-f', 'raw', '-c', 'read 0 512', nbd_uri)
- CLUSTER_SIZE=4k
+-        self.assertReadOk(qemu_io(*args))
- size=128M
++        self.assertReadOk(qemu_io(*args).stdout)
-diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
-index XXXXXXX..XXXXXXX 100755
+         result = self.remove_export('exp', mode)
---- a/tests/qemu-iotests/063
+         self.assert_qmp(result, 'return', {})
-+++ b/tests/qemu-iotests/063
-@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
+         self.assertExportNotFound('exp')
- _supported_proto file
+-        self.assertConnectFailed(qemu_io(*args))
- _unsupported_imgopts "subformat=monolithicFlat" \
++        self.assertConnectFailed(qemu_io(*args, check=False).stdout)
-                      "subformat=twoGbMaxExtentFlat" \
--                     "subformat=twoGbMaxExtentSparse"
+     def test_connect_after_remove_default(self):
-+                     "subformat=twoGbMaxExtentSparse" \
+         self.do_test_connect_after_remove()
-+                     "subformat=streamOptimized"
+diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245
+index XXXXXXX..XXXXXXX 100755
- _make_test_img 4M
+--- a/tests/qemu-iotests/245
++++ b/tests/qemu-iotests/245
-diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
+@@ -XXX,XX +XXX,XX @@
-index XXXXXXX..XXXXXXX 100755
+ # along with this program.  If not, see <http://www.gnu.org/licenses/>.
---- a/tests/qemu-iotests/072
+ #
-+++ b/tests/qemu-iotests/072
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
++import copy
++import json
- _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
+ import os
- _supported_proto file
+ import re
-+_unsupported_imgopts "subformat=streamOptimized"
++from subprocess import CalledProcessError
++
- IMG_SIZE=64M
+ import iotests
+-import copy
-diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
+-import json
-index XXXXXXX..XXXXXXX 100755
+ from iotests import qemu_img, qemu_io
---- a/tests/qemu-iotests/105
-+++ b/tests/qemu-iotests/105
+ hd_path = [
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ class TestBlockdevReopen(iotests.QMPTestCase):
- _supported_fmt qcow2 vmdk vhdx qed
- _supported_proto generic
+     # Reopen an image several times changing some of its options
- _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
+     def test_reopen(self):
--                     "subformat=twoGbMaxExtentSparse"
+-        # Check whether the filesystem supports O_DIRECT
-+                     "subformat=twoGbMaxExtentSparse" \
+-        if 'O_DIRECT' in qemu_io('-f', 'raw', '-t', 'none', '-c', 'quit', hd_path[0]):
-+                     "subformat=streamOptimized"
+-            supports_direct = False
+-        else:
- echo
++        try:
- echo "creating large image"
++            qemu_io('-f', 'raw', '-t', 'none', '-c', 'quit', hd_path[0])
-diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
+             supports_direct = True
-index XXXXXXX..XXXXXXX 100755
++        except CalledProcessError as exc:
---- a/tests/qemu-iotests/197
++            if 'O_DIRECT' in exc.stdout:
-+++ b/tests/qemu-iotests/197
++                supports_direct = False
-@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
++            else:
- _supported_proto generic
++                raise
- # LUKS support may be possible, but it complicates things.
- _unsupported_fmt luks
+         # Open the hd1 image passing all backing options
-+_unsupported_imgopts "subformat=streamOptimized"
+         opts = hd_opts(1)
+diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
- echo
+index XXXXXXX..XXXXXXX 100644
- echo '=== Copy-on-read ==='
+--- a/tests/qemu-iotests/iotests.py
-diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
++++ b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100755
+@@ -XXX,XX +XXX,XX @@ def qemu_io_wrap_args(args: Sequence[str]) -> List[str]:
---- a/tests/qemu-iotests/215
+ def qemu_io_popen(*args):
-+++ b/tests/qemu-iotests/215
+     return qemu_tool_popen(qemu_io_wrap_args(args))
-@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
- _supported_proto generic
+-def qemu_io(*args):
- # LUKS support may be possible, but it complicates things.
+-    '''Run qemu-io and return the stdout data'''
- _unsupported_fmt luks
+-    return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))[0]
-+_unsupported_imgopts "subformat=streamOptimized"
++def qemu_io(*args: str, check: bool = True, combine_stdio: bool = True
++            ) -> 'subprocess.CompletedProcess[str]':
- echo
++    """
- echo '=== Copy-on-read ==='
++    Run QEMU_IO_PROG and return the status code and console output.
-diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
++
-index XXXXXXX..XXXXXXX 100755
++    This function always prepends either QEMU_IO_OPTIONS or
---- a/tests/qemu-iotests/251
++    QEMU_IO_OPTIONS_NO_FMT.
-+++ b/tests/qemu-iotests/251
++    """
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
++    return qemu_tool(*qemu_io_wrap_args(args),
- _supported_fmt generic
++                     check=check, combine_stdio=combine_stdio)
- _supported_proto file
- _supported_os Linux
+ def qemu_io_pipe_and_status(*args):
-+_unsupported_imgopts "subformat=streamOptimized"
+     return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))
- if [ "$IMGOPTSSYNTAX" = "true" ]; then
+-def qemu_io_log(*args):
-     # We use json:{} filenames here, so we cannot work with additional options.
+-    result = qemu_io(*args)
 -    log(result, filters=[filter_testfiles, filter_qemu_io])
 +def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
 +    result = qemu_io(*args, check=False)
 +    log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
      return result
  def qemu_io_silent(*args):
 diff --git a/tests/qemu-iotests/tests/migration-permissions b/tests/qemu-iotests/tests/migration-permissions
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/tests/migration-permissions
 +++ b/tests/qemu-iotests/tests/migration-permissions
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
      def test_post_migration_permissions(self):
          # Try to access the image R/W, which should fail because virtio-blk
          # has not been configured with share-rw=on
 -        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img)
 +        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
          if not log.strip():
              print('ERROR (pre-migration): qemu-io should not be able to '
                    'access this image, but it reported no error')
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
          # Try the same qemu-io access again, verifying that the WRITE
          # permission remains unshared
 -        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img)
 +        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
          if not log.strip():
              print('ERROR (post-migration): qemu-io should not be able to '
                    'access this image, but it reported no error')
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 08/15] vmdk: Reject invalid compressed writes
+[PULL 09/14] iotests/migration-permissions: use assertRaises() for qemu_io() negative test
-Compressed writes generally have to write full clusters, not just in
+From: John Snow <jsnow@redhat.com>
 theory but also in practice when it comes to vmdk's streamOptimized
 subformat.  It currently is just silently broken for writes with
 non-zero in-cluster offsets:
-$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
+Modify this test to use assertRaises for its negative testing of
-$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
+qemu_io. If the exception raised does not match the one we tell it to
-wrote 4096/4096 bytes at offset 4096
+expect, we get *that* exception unhandled. If we get no exception, we
-KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
+get a unittest assertion failure and the provided emsg printed to
-read failed: Invalid argument
+screen.
-(The technical reason is that vmdk_write_extent() just writes the
+If we get the CalledProcessError exception but the output is not what we
-incomplete compressed data actually to offset 4k.  When reading the
+expect, we re-raise the original CalledProcessError.
 data, vmdk_read_extent() looks at offset 0 and finds the compressed data
 size to be 0, because that is what it reads from there.  This yields an
 error.)
-For incomplete writes with zero in-cluster offsets, the error path when
+Tidy.
 reading the rest of the cluster is a bit different, but the result is
 the same:
-$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
+(Note: Yes, you can reference "with" objects after that block ends; it
-$ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
+just means that ctx.__exit__(...) will have been called on it. It does
-wrote 4096/4096 bytes at offset 0
+not *actually* go out of scope. unittests expects you to want to inspect
-KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
+the Exception object, so they leave it defined post-exit.)
 read failed: Invalid argument
-(Here, vmdk_read_extent() finds the data and then sees that the
+Signed-off-by: John Snow <jsnow@redhat.com>
-uncompressed data is short.)
+Reviewed-by: Eric Blake <eblake@redhat.com>
 Tested-by: Eric Blake <eblake@redhat.com>
 Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-9-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
  .../qemu-iotests/tests/migration-permissions  | 28 +++++++++----------
 file changed, 14 insertions(+), 14 deletions(-)
-It is better to reject invalid writes than to make the user believe they
+diff --git a/tests/qemu-iotests/tests/migration-permissions b/tests/qemu-iotests/tests/migration-permissions
-might have succeeded and then fail when trying to read it back.
+index XXXXXXX..XXXXXXX 100755
+--- a/tests/qemu-iotests/tests/migration-permissions
-Signed-off-by: Max Reitz <mreitz@redhat.com>
++++ b/tests/qemu-iotests/tests/migration-permissions
-Reviewed-by: John Snow <jsnow@redhat.com>
+@@ -XXX,XX +XXX,XX @@
-Message-id: 20190815153638.4600-5-mreitz@redhat.com
+ #
-Reviewed-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+ import os
----
++from subprocess import CalledProcessError
  block/vmdk.c | 10 ++++++++++
 file changed, 10 insertions(+)
 diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/vmdk.c
 +++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
      if (extent->compressed) {
          void *compressed_data;
 +        /* Only whole clusters */
 +        if (offset_in_cluster ||
 +            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
 +            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
 +             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
 +        {
 +            ret = -EINVAL;
 +            goto out;
 +        }
 +
-         if (!extent->has_marker) {
+ import iotests
-             ret = -EINVAL;
+ from iotests import imgfmt, qemu_img_create, qemu_io
-             goto out;
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
      def test_post_migration_permissions(self):
          # Try to access the image R/W, which should fail because virtio-blk
          # has not been configured with share-rw=on
 -        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
 -        if not log.strip():
 -            print('ERROR (pre-migration): qemu-io should not be able to '
 -                  'access this image, but it reported no error')
 -        else:
 -            # This is the expected output
 -            assert 'Is another process using the image' in log
 +        emsg = ('ERROR (pre-migration): qemu-io should not be able to '
 +                'access this image, but it reported no error')
 +        with self.assertRaises(CalledProcessError, msg=emsg) as ctx:
 +            qemu_io('-f', imgfmt, '-c', 'quit', test_img)
 +        if 'Is another process using the image' not in ctx.exception.stdout:
 +            raise ctx.exception
          # Now migrate the VM
          self.vm_s.qmp('migrate', uri=f'unix:{mig_sock}')
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
          # Try the same qemu-io access again, verifying that the WRITE
          # permission remains unshared
 -        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
 -        if not log.strip():
 -            print('ERROR (post-migration): qemu-io should not be able to '
 -                  'access this image, but it reported no error')
 -        else:
 -            # This is the expected output
 -            assert 'Is another process using the image' in log
 +        emsg = ('ERROR (post-migration): qemu-io should not be able to '
 +                'access this image, but it reported no error')
 +        with self.assertRaises(CalledProcessError, msg=emsg) as ctx:
 +            qemu_io('-f', imgfmt, '-c', 'quit', test_img)
 +        if 'Is another process using the image' not in ctx.exception.stdout:
 +            raise ctx.exception
  if __name__ == '__main__':
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 05/15] iotests: Fix _filter_img_create()
+[PULL 10/14] iotests/image-fleecing: switch to qemu_io()
-fe646693acc changed qemu-img create's output so that it no longer prints
+From: John Snow <jsnow@redhat.com>
 single quotes around parameter values.  The subformat and adapter_type
 filters in _filter_img_create() have never been adapted to that change.
-Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
+This test expects failure ... but only sometimes. When? Why?
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: John Snow <jsnow@redhat.com>
+It's for reads of a region not defined by a bitmap. Adjust the test to
-Message-id: 20190815153638.4600-2-mreitz@redhat.com
+be more explicit about what it expects to fail and why.
-Reviewed-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Signed-off-by: John Snow <jsnow@redhat.com>
 Reviewed-by: Eric Blake <eblake@redhat.com>
 Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-10-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- tests/qemu-iotests/059.out       | 16 ++++++++--------
+ tests/qemu-iotests/tests/image-fleecing | 28 +++++++++++++++++--------
- tests/qemu-iotests/common.filter |  4 ++--
+file changed, 19 insertions(+), 9 deletions(-)
 files changed, 10 insertions(+), 10 deletions(-)
-diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
+diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/059.out
+--- a/tests/qemu-iotests/tests/image-fleecing
-+++ b/tests/qemu-iotests/059.out
++++ b/tests/qemu-iotests/tests/image-fleecing
-@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+@@ -XXX,XX +XXX,XX @@
- qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
+ #
+ # Creator/Owner: John Snow <jsnow@redhat.com>
- === Testing monolithicFlat creation and opening ===
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
++from subprocess import CalledProcessError
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
++
- image: TEST_DIR/t.IMGFMT
+ import iotests
- file format: IMGFMT
+-from iotests import log, qemu_img, qemu_io, qemu_io_silent, \
- virtual size: 2 GiB (2147483648 bytes)
+-    qemu_io_pipe_and_status
++from iotests import log, qemu_img, qemu_io, qemu_io_silent
- === Testing monolithicFlat with zeroed_grain ===
- qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
+ iotests.script_initialize(
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+     supported_fmts=['qcow2'],
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
+@@ -XXX,XX +XXX,XX @@ def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
+         for p in patterns + zeroes:
- === Testing big twoGbMaxExtentFlat ===
+             cmd = 'read -P%s %s %s' % p
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
+             log(cmd)
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
+-            out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
- image: TEST_DIR/t.vmdk
+-                                               nbd_uri)
- file format: vmdk
+-            if ret != 0:
- virtual size: 0.977 TiB (1073741824000 bytes)
+-                print(out)
-@@ -XXX,XX +XXX,XX @@ Format specific information:
++
- qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
++            try:
++                qemu_io('-r', '-f', 'raw', '-c', cmd, nbd_uri)
- === Testing truncated sparse ===
++            except CalledProcessError as exc:
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
++                if bitmap and p in zeroes:
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
++                    log(exc.stdout)
- qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
++                else:
++                    raise
- === Converting to streamOptimized from image with small cluster size===
-@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
+     log('')
-bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+     log('--- Testing COW ---')
+@@ -XXX,XX +XXX,XX @@ def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
- === Testing monolithicFlat with internally generated JSON file name ===
+             args += [target_img_path]
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
+         else:
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+             args += ['-f', 'raw', nbd_uri]
- qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
+-        out, ret = qemu_io_pipe_and_status(*args)
+-        if ret != 0:
- === Testing version 3 ===
+-            print(out)
-@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
++
-bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
++        try:
++            qemu_io(*args)
- === Testing 4TB monolithicFlat creation and IO ===
++        except CalledProcessError as exc:
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
++            if bitmap and p in zeroes:
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
++                log(exc.stdout)
- image: TEST_DIR/t.IMGFMT
++            else:
- file format: IMGFMT
++                raise
- virtual size: 4 TiB (4398046511104 bytes)
-@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
+     log('')
-KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+     log('--- Cleanup ---')
  === Testing qemu-img map on extents ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0x20000         0x3f0000        TEST_DIR/t.vmdk
 x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
 diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/common.filter
 +++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
          -e "s# compat6=\\(on\\|off\\)##g" \
          -e "s# static=\\(on\\|off\\)##g" \
          -e "s# zeroed_grain=\\(on\\|off\\)##g" \
 -        -e "s# subformat='[^']*'##g" \
 -        -e "s# adapter_type='[^']*'##g" \
 +        -e "s# subformat=[^ ]*##g" \
 +        -e "s# adapter_type=[^ ]*##g" \
          -e "s# hwversion=[^ ]*##g" \
          -e "s# lazy_refcounts=\\(on\\|off\\)##g" \
          -e "s# block_size=[0-9]\\+##g" \
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 04/15] iotests: Test allocate_first_block() with O_DIRECT
+[PULL 11/14] iotests: remove qemu_io_pipe_and_status()
-From: Nir Soffer <nirsof@gmail.com>
+From: John Snow <jsnow@redhat.com>
-Using block_resize we can test allocate_first_block() with file
+I know we just added it, sorry. This is done in favor of qemu_io() which
-descriptor opened with O_DIRECT, ensuring that it works for any size
+*also* returns the console output and status, but with more robust error
-larger than 4096 bytes.
+handling on failure.
-Testing smaller sizes is tricky as the result depends on the filesystem
+Signed-off-by: John Snow <jsnow@redhat.com>
-used for testing. For example on NFS any size will work since O_DIRECT
+Reviewed-by: Eric Blake <eblake@redhat.com>
-does not require any alignment.
+Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-11-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
  tests/qemu-iotests/iotests.py | 3 ---
 file changed, 3 deletions(-)
-Signed-off-by: Nir Soffer <nsoffer@redhat.com>
+diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Message-id: 20190827010528.8818-3-nsoffer@redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
  tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
  tests/qemu-iotests/175.out |  8 ++++++++
 files changed, 36 insertions(+)
 diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/175
 +++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ _filter_blocks()
          -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
  }
 +# Resize image using block_resize.
 +# Parameter 1: image path
 +# Parameter 2: new size
 +_block_resize()
 +{
 +    local path=$1
 +    local size=$2
 +
 +    $QEMU -qmp stdio -nographic -nodefaults \
 +        -blockdev file,node-name=file,filename=$path,cache.direct=on \
 +        <<EOF
 +{'execute': 'qmp_capabilities'}
 +{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
 +{'execute': 'quit'}
 +EOF
 +}
 +
  # get standard environment, filters and checks
  . ./common.rc
  . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
  _supported_proto file
  _supported_os Linux
 +_default_cache_mode none
 +_supported_cache_modes none directsync
 +
  size=$((1 * 1024 * 1024))
  touch "$TEST_DIR/empty"
@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
      stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  done
 +for new_size in 4096 1048576; do
 +    echo
 +    echo "== resize empty image with block_resize =="
 +    _make_test_img 0 | _filter_imgfmt
 +    _block_resize $TEST_IMG $new_size >/dev/null
 +    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
 +done
 +
  # success, all done
  echo "*** done"
  rm -f $seq.full
 diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/175.out
+--- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/175.out
++++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
+@@ -XXX,XX +XXX,XX @@ def qemu_io(*args: str, check: bool = True, combine_stdio: bool = True
- == creating image with preallocation falloc ==
+     return qemu_tool(*qemu_io_wrap_args(args),
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
+                      check=check, combine_stdio=combine_stdio)
- size=1048576, max allocation
-+
+-def qemu_io_pipe_and_status(*args):
-+== resize empty image with block_resize ==
+-    return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+-
-+size=4096, min allocation
+ def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
-+
+     result = qemu_io(*args, check=False)
-+== resize empty image with block_resize ==
+     log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
 +size=1048576, min allocation
   *** done
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 03/15] block: posix: Always allocate the first block
+[PULL 12/14] iotests: remove qemu_io_silent() and qemu_io_silent_check().
-From: Nir Soffer <nirsof@gmail.com>
+From: John Snow <jsnow@redhat.com>
-When creating an image with preallocation "off" or "falloc", the first
+Like qemu-img, qemu-io returning 0 should be the norm and not the
-block of the image is typically not allocated. When using Gluster
+exception. Remove all calls to qemu_io_silent that just assert the
-storage backed by XFS filesystem, reading this block using direct I/O
+return code is zero (That's every last call, as it turns out), and
-succeeds regardless of request length, fooling alignment detection.
+replace them with a normal qemu_io() call.
-In this case we fallback to a safe value (4096) instead of the optimal
+qemu_io_silent_check() appeared to have been unused already.
 value (512), which may lead to unneeded data copying when aligning
 requests.  Allocating the first block avoids the fallback.
-Since we allocate the first block even with preallocation=off, we no
+Signed-off-by: John Snow <jsnow@redhat.com>
-longer create images with zero disk size:
+Reviewed-by: Eric Blake <eblake@redhat.com>
 Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-12-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
  tests/qemu-iotests/216                        | 12 +++++-----
  tests/qemu-iotests/218                        |  5 ++---
  tests/qemu-iotests/224                        |  4 ++--
  tests/qemu-iotests/258                        | 11 +++++-----
  tests/qemu-iotests/298                        | 17 ++++++--------
  tests/qemu-iotests/310                        | 22 +++++++++----------
  tests/qemu-iotests/iotests.py                 | 16 --------------
  tests/qemu-iotests/tests/image-fleecing       |  4 ++--
  .../tests/mirror-ready-cancel-error           |  2 +-
  .../qemu-iotests/tests/stream-error-on-reset  |  4 ++--
 files changed, 38 insertions(+), 59 deletions(-)
-    $ ./qemu-img create -f raw test.raw 1g
+diff --git a/tests/qemu-iotests/216 b/tests/qemu-iotests/216
-    Formatting 'test.raw', fmt=raw size=1073741824
+index XXXXXXX..XXXXXXX 100755
+--- a/tests/qemu-iotests/216
-    $ ls -lhs test.raw
++++ b/tests/qemu-iotests/216
-.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
+@@ -XXX,XX +XXX,XX @@
+ # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
-And converting the image requires additional cluster:
+ import iotests
-    $ ./qemu-img measure -f raw -O qcow2 test.raw
+-from iotests import log, qemu_img, qemu_io_silent
-    required size: 458752
++from iotests import log, qemu_img, qemu_io
-    fully allocated size: 1074135040
+ # Need backing file support
-When using format like vmdk with multiple files per image, we allocate
+ iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk'],
-one block per file:
+@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
+     log('')
-    $ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
-    Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat
+     qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
+-    assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
-    $ ls -lhs test*.vmdk
++    qemu_io(base_img_path, '-c', 'write -P 1 0M 1M')
-.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
+     qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
+              '-F', iotests.imgfmt, top_img_path)
-.0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk
+-    assert qemu_io_silent(top_img_path,  '-c', 'write -P 2 1M 1M') == 0
++    qemu_io(top_img_path,  '-c', 'write -P 2 1M 1M')
-I did quick performance test for copying disks with qemu-img convert to
-new raw target image to Gluster storage with sector size of 512 bytes:
+     log('Done')
-    for i in $(seq 10); do
+@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
-        rm -f dst.raw
+     log('--- Checking COR result ---')
-        sleep 10
+     log('')
-        time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
-    done
+-    assert qemu_io_silent(base_img_path, '-c', 'discard 0 64M') == 0
+-    assert qemu_io_silent(top_img_path,  '-c', 'read -P 1 0M 1M') == 0
-Here is a table comparing the total time spent:
+-    assert qemu_io_silent(top_img_path,  '-c', 'read -P 2 1M 1M') == 0
++    qemu_io(base_img_path, '-c', 'discard 0 64M')
-Type    Before(s)   After(s)    Diff(%)
++    qemu_io(top_img_path,  '-c', 'read -P 1 0M 1M')
----------------------------------------
++    qemu_io(top_img_path,  '-c', 'read -P 2 1M 1M')
-real      530.028    469.123      -11.4
-user       17.204     10.768      -37.4
+     log('Done')
-sys        17.881      7.011      -60.7
+diff --git a/tests/qemu-iotests/218 b/tests/qemu-iotests/218
+index XXXXXXX..XXXXXXX 100755
-We can see very clear improvement in CPU usage.
+--- a/tests/qemu-iotests/218
++++ b/tests/qemu-iotests/218
-Signed-off-by: Nir Soffer <nsoffer@redhat.com>
+@@ -XXX,XX +XXX,XX @@
-Message-id: 20190827010528.8818-2-nsoffer@redhat.com
+ # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+ import iotests
----
+-from iotests import log, qemu_img, qemu_io_silent
- block/file-posix.c                            | 51 +++++++++++++++++++
++from iotests import log, qemu_img, qemu_io
- tests/qemu-iotests/059.out                    |  2 +-
- tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
+ iotests.script_initialize(supported_fmts=['qcow2', 'raw'])
- tests/qemu-iotests/150.out.raw                | 12 +++++
- tests/qemu-iotests/175                        | 19 ++++---
+@@ -XXX,XX +XXX,XX @@ with iotests.VM() as vm, \
- tests/qemu-iotests/175.out                    |  8 +--
+      iotests.FilePath('src.img') as src_img_path:
- tests/qemu-iotests/178.out.qcow2              |  4 +-
- tests/qemu-iotests/221.out                    | 12 +++--
+     qemu_img('create', '-f', iotests.imgfmt, src_img_path, '64M')
- tests/qemu-iotests/253.out                    | 12 +++--
+-    assert qemu_io_silent('-f', iotests.imgfmt, src_img_path,
-files changed, 99 insertions(+), 21 deletions(-)
+-                          '-c', 'write -P 42 0M 64M') == 0
- rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
++    qemu_io('-f', iotests.imgfmt, src_img_path, '-c', 'write -P 42 0M 64M')
- create mode 100644 tests/qemu-iotests/150.out.raw
+     vm.launch()
-diff --git a/block/file-posix.c b/block/file-posix.c
 diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/224
 +++ b/tests/qemu-iotests/224
@@ -XXX,XX +XXX,XX @@
  # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
  import iotests
 -from iotests import log, qemu_img, qemu_io_silent, filter_qmp_testfiles, \
 +from iotests import log, qemu_img, qemu_io, filter_qmp_testfiles, \
                      filter_qmp_imgfmt
  import json
@@ -XXX,XX +XXX,XX @@ for filter_node_name in False, True:
                   '-F', iotests.imgfmt, top_img_path)
          # Something to commit
 -        assert qemu_io_silent(mid_img_path, '-c', 'write -P 1 0 1M') == 0
 +        qemu_io(mid_img_path, '-c', 'write -P 1 0 1M')
          vm.launch()
 diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/258
 +++ b/tests/qemu-iotests/258
@@ -XXX,XX +XXX,XX @@
  # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
  import iotests
 -from iotests import log, qemu_img, qemu_io_silent, \
 +from iotests import log, qemu_img, qemu_io, \
          filter_qmp_testfiles, filter_qmp_imgfmt
  # Returns a node for blockdev-add
@@ -XXX,XX +XXX,XX @@ def test_concurrent_finish(write_to_stream_node):
          if write_to_stream_node:
              # This is what (most of the time) makes commit finish
              # earlier and then pull in stream
 -            assert qemu_io_silent(node2_path,
 -                                  '-c', 'write %iK 64K' % (65536 - 192),
 -                                  '-c', 'write %iK 64K' % (65536 -  64)) == 0
 +            qemu_io(node2_path,
 +                    '-c', 'write %iK 64K' % (65536 - 192),
 +                    '-c', 'write %iK 64K' % (65536 -  64))
              stream_throttle='tg'
          else:
              # And this makes stream finish earlier
 -            assert qemu_io_silent(node1_path,
 -                                  '-c', 'write %iK 64K' % (65536 - 64)) == 0
 +            qemu_io(node1_path, '-c', 'write %iK 64K' % (65536 - 64))
              commit_throttle='tg'
 diff --git a/tests/qemu-iotests/298 b/tests/qemu-iotests/298
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/298
 +++ b/tests/qemu-iotests/298
@@ -XXX,XX +XXX,XX @@ class TestTruncate(iotests.QMPTestCase):
          os.remove(refdisk)
      def do_test(self, prealloc_mode, new_size):
 -        ret = iotests.qemu_io_silent('--image-opts', '-c', 'write 0 10M', '-c',
 -                                     f'truncate -m {prealloc_mode} {new_size}',
 -                                     drive_opts)
 -        self.assertEqual(ret, 0)
 -
 -        ret = iotests.qemu_io_silent('-f', iotests.imgfmt, '-c', 'write 0 10M',
 -                                     '-c',
 -                                     f'truncate -m {prealloc_mode} {new_size}',
 -                                     refdisk)
 -        self.assertEqual(ret, 0)
 +        iotests.qemu_io('--image-opts', '-c', 'write 0 10M', '-c',
 +                        f'truncate -m {prealloc_mode} {new_size}',
 +                        drive_opts)
 +
 +        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write 0 10M',
 +                        '-c', f'truncate -m {prealloc_mode} {new_size}',
 +                        refdisk)
          stat = os.stat(disk)
          refstat = os.stat(refdisk)
 diff --git a/tests/qemu-iotests/310 b/tests/qemu-iotests/310
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/310
 +++ b/tests/qemu-iotests/310
@@ -XXX,XX +XXX,XX @@
  #
  import iotests
 -from iotests import log, qemu_img, qemu_io_silent
 +from iotests import log, qemu_img, qemu_io
  # Need backing file support
  iotests.script_initialize(supported_fmts=['qcow2'],
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
      log('')
      qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
 -    assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
 -    assert qemu_io_silent(base_img_path, '-c', 'write -P 1 3M 1M') == 0
 +    qemu_io(base_img_path, '-c', 'write -P 1 0M 1M')
 +    qemu_io(base_img_path, '-c', 'write -P 1 3M 1M')
      qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
               '-F', iotests.imgfmt, mid_img_path)
 -    assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 2M 1M') == 0
 -    assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 4M 1M') == 0
 +    qemu_io(mid_img_path, '-c', 'write -P 3 2M 1M')
 +    qemu_io(mid_img_path, '-c', 'write -P 3 4M 1M')
      qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
               '-F', iotests.imgfmt, top_img_path)
 -    assert qemu_io_silent(top_img_path, '-c', 'write -P 2 1M 1M') == 0
 +    qemu_io(top_img_path, '-c', 'write -P 2 1M 1M')
  #      0 1 2 3 4
  # top    2
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
      # Detach backing to check that we can read the data from the top level now
      qemu_img('rebase', '-u', '-b', '', '-f', iotests.imgfmt, top_img_path)
 -    assert qemu_io_silent(top_img_path, '-c', 'read -P 0 0 1M') == 0
 -    assert qemu_io_silent(top_img_path, '-c', 'read -P 2 1M 1M') == 0
 -    assert qemu_io_silent(top_img_path, '-c', 'read -P 3 2M 1M') == 0
 -    assert qemu_io_silent(top_img_path, '-c', 'read -P 0 3M 1M') == 0
 -    assert qemu_io_silent(top_img_path, '-c', 'read -P 3 4M 1M') == 0
 +    qemu_io(top_img_path, '-c', 'read -P 0 0 1M')
 +    qemu_io(top_img_path, '-c', 'read -P 2 1M 1M')
 +    qemu_io(top_img_path, '-c', 'read -P 3 2M 1M')
 +    qemu_io(top_img_path, '-c', 'read -P 0 3M 1M')
 +    qemu_io(top_img_path, '-c', 'read -P 3 4M 1M')
      log('Done')
 diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
 index XXXXXXX..XXXXXXX 100644
---- a/block/file-posix.c
+--- a/tests/qemu-iotests/iotests.py
-+++ b/block/file-posix.c
++++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
+@@ -XXX,XX +XXX,XX @@ def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
-     return ret;
+     log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
- }
+     return result
-+/*
+-def qemu_io_silent(*args):
-+ * Help alignment probing by allocating the first block.
+-    '''Run qemu-io and return the exit code, suppressing stdout'''
-+ *
+-    args = qemu_io_wrap_args(args)
-+ * When reading with direct I/O from unallocated area on Gluster backed by XFS,
+-    result = subprocess.run(args, stdout=subprocess.DEVNULL, check=False)
-+ * reading succeeds regardless of request length. In this case we fallback to
+-    if result.returncode < 0:
-+ * safe alignment which is not optimal. Allocating the first block avoids this
+-        sys.stderr.write('qemu-io received signal %i: %s\n' %
-+ * fallback.
+-                         (-result.returncode, ' '.join(args)))
-+ *
+-    return result.returncode
-+ * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
+-
-+ * request alignment, so we use safe values.
+-def qemu_io_silent_check(*args):
-+ *
+-    '''Run qemu-io and return the true if subprocess returned 0'''
-+ * Returns: 0 on success, -errno on failure. Since this is an optimization,
+-    args = qemu_io_wrap_args(args)
-+ * caller may ignore failures.
+-    result = subprocess.run(args, stdout=subprocess.DEVNULL,
-+ */
+-                            stderr=subprocess.STDOUT, check=False)
-+static int allocate_first_block(int fd, size_t max_size)
+-    return result.returncode == 0
-+{
+-
-+    size_t write_size = (max_size < MAX_BLOCKSIZE)
+ class QemuIoInteractive:
-+        ? BDRV_SECTOR_SIZE
+     def __init__(self, *args):
-+        : MAX_BLOCKSIZE;
+         self.args = qemu_io_wrap_args(args)
-+    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
+diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
-+    void *buf;
+index XXXXXXX..XXXXXXX 100755
-+    ssize_t n;
+--- a/tests/qemu-iotests/tests/image-fleecing
-+    int ret;
++++ b/tests/qemu-iotests/tests/image-fleecing
-+
+@@ -XXX,XX +XXX,XX @@
-+    buf = qemu_memalign(max_align, write_size);
+ from subprocess import CalledProcessError
-+    memset(buf, 0, write_size);
-+
+ import iotests
-+    do {
+-from iotests import log, qemu_img, qemu_io, qemu_io_silent
-+        n = pwrite(fd, buf, write_size, 0);
++from iotests import log, qemu_img, qemu_io
-+    } while (n == -1 && errno == EINTR);
-+
+ iotests.script_initialize(
-+    ret = (n == -1) ? -errno : 0;
+     supported_fmts=['qcow2'],
-+
+@@ -XXX,XX +XXX,XX @@ def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
-+    qemu_vfree(buf);
+     for p in overwrite + remainder:
-+    return ret;
+         cmd = 'read -P%s %s %s' % p
-+}
+         log(cmd)
-+
+-        assert qemu_io_silent(base_img_path, '-c', cmd) == 0
- static int handle_aiocb_truncate(void *opaque)
++        qemu_io(base_img_path, '-c', cmd)
- {
-     RawPosixAIOData *aiocb = opaque;
+     log('')
-@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
+     log('Done')
-                 /* posix_fallocate() doesn't set errno. */
+diff --git a/tests/qemu-iotests/tests/mirror-ready-cancel-error b/tests/qemu-iotests/tests/mirror-ready-cancel-error
-                 error_setg_errno(errp, -result,
+index XXXXXXX..XXXXXXX 100755
-                                  "Could not preallocate new data");
+--- a/tests/qemu-iotests/tests/mirror-ready-cancel-error
-+            } else if (current_length == 0) {
++++ b/tests/qemu-iotests/tests/mirror-ready-cancel-error
-+                /*
+@@ -XXX,XX +XXX,XX @@ class TestMirrorReadyCancelError(iotests.QMPTestCase):
-+                 * posix_fallocate() uses fallocate() if the filesystem
+         # Ensure that mirror will copy something before READY so the
-+                 * supports it, or fallback to manually writing zeroes. If
+         # target format layer will forward the pre-READY flush to its
-+                 * fallocate() was used, unaligned reads from the fallocated
+         # file child
-+                 * area in raw_probe_alignment() will succeed, hence we need to
+-        assert iotests.qemu_io_silent('-c', 'write -P 1 0 64k', source) == 0
-+                 * allocate the first block.
++        iotests.qemu_io('-c', 'write -P 1 0 64k', source)
-+                 *
-+                 * Optimize future alignment probing; ignore failures.
+         self.vm = iotests.VM()
-+                 */
+         self.vm.launch()
-+                allocate_first_block(fd, offset);
+diff --git a/tests/qemu-iotests/tests/stream-error-on-reset b/tests/qemu-iotests/tests/stream-error-on-reset
-             }
+index XXXXXXX..XXXXXXX 100755
-         } else {
+--- a/tests/qemu-iotests/tests/stream-error-on-reset
-             result = 0;
++++ b/tests/qemu-iotests/tests/stream-error-on-reset
-@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
+@@ -XXX,XX +XXX,XX @@
-         if (ftruncate(fd, offset) != 0) {
-             result = -errno;
+ import os
-             error_setg_errno(errp, -result, "Could not resize file");
+ import iotests
-+        } else if (current_length == 0 && offset > current_length) {
+-from iotests import imgfmt, qemu_img_create, qemu_io_silent, QMPTestCase
-+            /* Optimize future alignment probing; ignore failures. */
++from iotests import imgfmt, qemu_img_create, qemu_io, QMPTestCase
-+            allocate_first_block(fd, offset);
-         }
-         return result;
+ image_size = 1 * 1024 * 1024
-     default:
+@@ -XXX,XX +XXX,XX @@ class TestStreamErrorOnReset(QMPTestCase):
-diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
+         - top image is attached to a virtio-scsi device
-index XXXXXXX..XXXXXXX 100644
+         """
---- a/tests/qemu-iotests/059.out
+         qemu_img_create('-f', imgfmt, base, str(image_size))
-+++ b/tests/qemu-iotests/059.out
+-        assert qemu_io_silent('-c', f'write 0 {data_size}', base) == 0
-@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
++        qemu_io('-c', f'write 0 {data_size}', base)
- image: TEST_DIR/t.vmdk
+         qemu_img_create('-f', imgfmt, top, str(image_size))
- file format: vmdk
- virtual size: 0.977 TiB (1073741824000 bytes)
+         self.vm = iotests.VM()
 -disk size: 16 KiB
 +disk size: 1.97 MiB
  Format specific information:
      cid: XXXXXXXX
      parent cid: XXXXXXXX
 diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
 similarity index 100%
 rename from tests/qemu-iotests/150.out
 rename to tests/qemu-iotests/150.out.qcow2
 diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/tests/qemu-iotests/150.out.raw
@@ -XXX,XX +XXX,XX @@
 +QA output created by 150
 +
 +=== Mapping sparse conversion ===
 +
 +Offset          Length          File
 +0               0x1000          TEST_DIR/t.IMGFMT
 +
 +=== Mapping non-sparse conversion ===
 +
 +Offset          Length          File
 +0               0x100000        TEST_DIR/t.IMGFMT
 +*** done
 diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/175
 +++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  # the file size.  This function hides the resulting difference in the
  # stat -c '%b' output.
  # Parameter 1: Number of blocks an empty file occupies
 -# Parameter 2: Image size in bytes
 +# Parameter 2: Minimal number of blocks in an image
 +# Parameter 3: Image size in bytes
  _filter_blocks()
  {
      extra_blocks=$1
 -    img_size=$2
 +    min_blocks=$2
 +    img_size=$3
 -    sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
 -        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
 +    sed -e "s/blocks=$min_blocks\\(\$\\|[^0-9]\\)/min allocation/" \
 +        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
  }
  # get standard environment, filters and checks
@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
  touch "$TEST_DIR/empty"
  extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 +# We always write the first byte; check how many blocks this filesystem
 +# allocates to match empty image alloation.
 +printf "\0" > "$TEST_DIR/empty"
 +min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 +
  echo
  echo "== creating image with default preallocation =="
  _make_test_img $size | _filter_imgfmt
 -stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 +stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  for mode in off full falloc; do
      echo
      echo "== creating image with preallocation $mode =="
      IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
 -    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 +    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  done
  # success, all done
 diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/175.out
 +++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
  == creating image with default preallocation ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
 -size=1048576, nothing allocated
 +size=1048576, min allocation
  == creating image with preallocation off ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
 -size=1048576, nothing allocated
 +size=1048576, min allocation
  == creating image with preallocation full ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
 -size=1048576, everything allocated
 +size=1048576, max allocation
  == creating image with preallocation falloc ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
 -size=1048576, everything allocated
 +size=1048576, max allocation
   *** done
 diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/178.out.qcow2
 +++ b/tests/qemu-iotests/178.out.qcow2
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
  == raw input image with data (human) ==
  Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
 -required size: 393216
 +required size: 458752
  fully allocated size: 1074135040
  wrote 512/512 bytes at offset 512
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
  Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
  {
 -    "required": 393216,
 +    "required": 458752,
      "fully-allocated": 1074135040
  }
  wrote 512/512 bytes at offset 512
 diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/221.out
 +++ b/tests/qemu-iotests/221.out
@@ -XXX,XX +XXX,XX @@ QA output created by 221
  === Check mapping of unaligned raw image ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  wrote 1/1 bytes at offset 65536
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
  { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
  { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  *** done
 diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/253.out
 +++ b/tests/qemu-iotests/253.out
@@ -XXX,XX +XXX,XX @@ QA output created by 253
  === Check mapping of unaligned raw image ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  wrote 65535/65535 bytes at offset 983040
 .999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
  *** done
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 02/15] block: fix permission update in bdrv_replace_node
+[PULL 13/14] iotests: make qemu_io_log() check return codes by default
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: John Snow <jsnow@redhat.com>
-It's wrong to OR shared permissions. It may lead to crash on further
+Just like qemu_img_log(), upgrade qemu_io_log() to enforce a return code
-permission updates.
+of zero by default.
 Also, no needs to consider previously calculated permissions, as at
 this point we already bind all new parents and bdrv_get_cumulative_perm
 result is enough. So fix the bug by just set permissions by
 bdrv_get_cumulative_perm result.
-Bug was introduced in long ago 234ac1a9025, in 2.9.
+Tests that use qemu_io_log(): 242 245 255 274 303 307 nbd-reconnect-on-open
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
+Reviewed-by: Eric Blake <eblake@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Reviewed-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220418211504.943969-13-jsnow@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 ---
- block.c | 5 ++---
+ tests/qemu-iotests/iotests.py                  | 5 +++--
-file changed, 2 insertions(+), 3 deletions(-)
+ tests/qemu-iotests/tests/nbd-reconnect-on-open | 2 +-
 files changed, 4 insertions(+), 3 deletions(-)
-diff --git a/block.c b/block.c
+diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
 index XXXXXXX..XXXXXXX 100644
---- a/block.c
+--- a/tests/qemu-iotests/iotests.py
-+++ b/block.c
++++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
+@@ -XXX,XX +XXX,XX @@ def qemu_io(*args: str, check: bool = True, combine_stdio: bool = True
- {
+     return qemu_tool(*qemu_io_wrap_args(args),
-     BdrvChild *c, *next;
+                      check=check, combine_stdio=combine_stdio)
-     GSList *list = NULL, *p;
--    uint64_t old_perm, old_shared;
+-def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
-     uint64_t perm = 0, shared = BLK_PERM_ALL;
+-    result = qemu_io(*args, check=False)
-     int ret;
++def qemu_io_log(*args: str, check: bool = True
++                ) -> 'subprocess.CompletedProcess[str]':
-@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
++    result = qemu_io(*args, check=check)
-         bdrv_unref(from);
+     log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
-     }
+     return result
--    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
+diff --git a/tests/qemu-iotests/tests/nbd-reconnect-on-open b/tests/qemu-iotests/tests/nbd-reconnect-on-open
--    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
+index XXXXXXX..XXXXXXX 100755
-+    bdrv_get_cumulative_perm(to, &perm, &shared);
+--- a/tests/qemu-iotests/tests/nbd-reconnect-on-open
-+    bdrv_set_perm(to, perm, shared);
++++ b/tests/qemu-iotests/tests/nbd-reconnect-on-open
+@@ -XXX,XX +XXX,XX @@ def check_fail_to_connect(open_timeout):
- out:
+     log(f'Check fail to connect with {open_timeout} seconds of timeout')
-     g_slist_free(list);
      start_t = time.time()
 -    qemu_io_log(*create_args(open_timeout))
 +    qemu_io_log(*create_args(open_timeout), check=False)
      delta_t = time.time() - start_t
      max_delta = open_timeout + 0.2
 --
-.21.0
+.35.1

-[Qemu-devel] [PULL 12/15] file-posix: fix request_alignment typo
+[PULL 14/14] iotests/108: Fix when missing user_allow_other
-From: Stefan Hajnoczi <stefanha@redhat.com>
+FUSE exports' allow-other option defaults to "auto", which means that it
 will try passing allow_other as a mount option, and fall back to not
 using it when an error occurs.  We make no effort to hide fusermount's
 error message (because it would be difficult, and because users might
 want to know about the fallback occurring), and so when allow_other does
 not work (primarily when /etc/fuse.conf does not contain
 user_allow_other), this error message will appear and break the
 reference output.
-Fixes: a6b257a08e3d72219f03e461a52152672fec0612
+We do not need allow_other here, though, so we can just pass
-       ("file-posix: Handle undetectable alignment")
+allow-other=off to fix that.
-Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
-Message-id: 20190827101328.4062-1-stefanha@redhat.com
+Reported-by: Markus Armbruster <armbru@redhat.com>
-Reviewed-by: Thomas Huth <thuth@redhat.com>
+Signed-off-by: Hanna Reitz <hreitz@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Message-Id: <20220421142435.569600-1-hreitz@redhat.com>
 Tested-by: Markus Armbruster <armbru@redhat.com>
 Tested-by: Eric Blake <eblake@redhat.com>
 ---
- block/file-posix.c | 2 +-
+ tests/qemu-iotests/108 | 2 +-
 file changed, 1 insertion(+), 1 deletion(-)
-diff --git a/block/file-posix.c b/block/file-posix.c
+diff --git a/tests/qemu-iotests/108 b/tests/qemu-iotests/108
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/block/file-posix.c
+--- a/tests/qemu-iotests/108
-+++ b/block/file-posix.c
++++ b/tests/qemu-iotests/108
-@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
+@@ -XXX,XX +XXX,XX @@ else
-         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
-             align = alignments[i];
+     $QSD \
-             if (raw_is_io_aligned(fd, buf + align, max_align)) {
+         --blockdev file,node-name=export-node,filename="$TEST_IMG" \
--                /* Fallback to request_aligment. */
+-        --export fuse,id=fuse-export,node-name=export-node,mountpoint="$export_mp",writable=on,growable=off \
-+                /* Fallback to request_alignment. */
++        --export fuse,id=fuse-export,node-name=export-node,mountpoint="$export_mp",writable=on,growable=off,allow-other=off \
-                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
+         --pidfile "$TEST_DIR/qsd.pid" \
-                 break;
+         --daemonize
-             }
+ fi
 --
-.21.0
+.35.1

The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:

Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27

for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:

iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)

----------------------------------------------------------------
Block patches:
- qemu-io now accepts a file to read a write pattern from
- Ensure that raw files have their first block allocated so we can probe
  the O_DIRECT alignment if necessary
- Various fixes

----------------------------------------------------------------
Denis Plotnikov (1):
  qemu-io: add pattern file for write command

Max Reitz (7):
  iotests: Fix _filter_img_create()
  vmdk: Use bdrv_dirname() for relative extent paths
  iotests: Keep testing broken relative extent paths
  vmdk: Reject invalid compressed writes
  iotests: Disable broken streamOptimized tests
  iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
  iotests: Disable 126 for flat vmdk subformats

Nir Soffer (3):
  block: posix: Always allocate the first block
  iotests: Test allocate_first_block() with O_DIRECT
  iotests: Unify cache mode quoting

Stefan Hajnoczi (1):
  file-posix: fix request_alignment typo

Thomas Huth (2):
  iotests: Check for enabled drivers before testing them
  tests/check-block: Skip iotests when sanitizers are enabled

Vladimir Sementsov-Ogievskiy (1):
  block: fix permission update in bdrv_replace_node

block.c                                       |  5 +-
 block/file-posix.c                            | 53 +++++++++-
 block/vmdk.c                                  | 64 ++++++++----
 qemu-io-cmds.c                                | 99 +++++++++++++++++--
 tests/check-block.sh                          |  5 +
 tests/qemu-iotests/002                        |  1 +
 tests/qemu-iotests/003                        |  1 +
 tests/qemu-iotests/005                        |  3 +-
 tests/qemu-iotests/009                        |  1 +
 tests/qemu-iotests/010                        |  1 +
 tests/qemu-iotests/011                        |  1 +
 tests/qemu-iotests/017                        |  3 +-
 tests/qemu-iotests/018                        |  3 +-
 tests/qemu-iotests/019                        |  3 +-
 tests/qemu-iotests/020                        |  3 +-
 tests/qemu-iotests/026                        |  4 +-
 tests/qemu-iotests/027                        |  1 +
 tests/qemu-iotests/032                        |  1 +
 tests/qemu-iotests/033                        |  1 +
 tests/qemu-iotests/034                        |  3 +-
 tests/qemu-iotests/037                        |  3 +-
 tests/qemu-iotests/039                        |  4 +-
 tests/qemu-iotests/052                        |  2 +-
 tests/qemu-iotests/059                        | 34 ++++++-
 tests/qemu-iotests/059.out                    | 26 +++--
 tests/qemu-iotests/063                        |  3 +-
 tests/qemu-iotests/071                        |  1 +
 tests/qemu-iotests/072                        |  1 +
 tests/qemu-iotests/081                        |  4 +-
 tests/qemu-iotests/091                        |  4 +-
 tests/qemu-iotests/099                        |  1 +
 tests/qemu-iotests/105                        |  3 +-
 tests/qemu-iotests/110                        |  3 +-
 tests/qemu-iotests/120                        |  1 +
 tests/qemu-iotests/126                        |  2 +
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++
 tests/qemu-iotests/162                        |  4 +-
 tests/qemu-iotests/175                        | 47 +++++++--
 tests/qemu-iotests/175.out                    | 16 ++-
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/184                        |  1 +
 tests/qemu-iotests/186                        |  1 +
 tests/qemu-iotests/197                        |  1 +
 tests/qemu-iotests/215                        |  1 +
 tests/qemu-iotests/221.out                    | 12 ++-
 tests/qemu-iotests/251                        |  1 +
 tests/qemu-iotests/253.out                    | 12 ++-
 tests/qemu-iotests/common.filter              |  4 +-
 tests/qemu-iotests/common.rc                  | 14 +++
 50 files changed, 391 insertions(+), 87 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

-- 
2.21.0

From: Denis Plotnikov <dplotnikov@virtuozzo.com>

The patch allows to provide a pattern file for write
command. There was no similar ability before.

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
Reviewed-by: Eric Blake <eblake@redhat.com>
[mreitz: Keep optstring in alphabetical order]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 6 deletions(-)

diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
     qemu_vfree(p);
 }
 
+/*
+ * qemu_io_alloc_from_file()
+ *
+ * Allocates the buffer and populates it with the content of the given file
+ * up to @len bytes. If the file length is less than @len, then the buffer
+ * is populated with the file content cyclically.
+ *
+ * @blk - the block backend where the buffer content is going to be written to
+ * @len - the buffer length
+ * @file_name - the file to read the content from
+ *
+ * Returns: the buffer pointer on success
+ *          NULL on error
+ */
+static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
+                                     const char *file_name)
+{
+    char *buf, *buf_origin;
+    FILE *f = fopen(file_name, "r");
+    int pattern_len;
+
+    if (!f) {
+        perror(file_name);
+        return NULL;
+    }
+
+    if (qemuio_misalign) {
+        len += MISALIGN_OFFSET;
+    }
+
+    buf_origin = buf = blk_blockalign(blk, len);
+
+    if (qemuio_misalign) {
+        buf_origin += MISALIGN_OFFSET;
+        buf += MISALIGN_OFFSET;
+        len -= MISALIGN_OFFSET;
+    }
+
+    pattern_len = fread(buf_origin, 1, len, f);
+
+    if (ferror(f)) {
+        perror(file_name);
+        goto error;
+    }
+
+    if (pattern_len == 0) {
+        fprintf(stderr, "%s: file is empty\n", file_name);
+        goto error;
+    }
+
+    fclose(f);
+
+    if (len > pattern_len) {
+        len -= pattern_len;
+        buf += pattern_len;
+
+        while (len > 0) {
+            size_t len_to_copy = MIN(pattern_len, len);
+
+            memcpy(buf, buf_origin, len_to_copy);
+
+            len -= len_to_copy;
+            buf += len_to_copy;
+        }
+    }
+
+    return buf_origin;
+
+error:
+    qemu_io_free(buf_origin);
+    return NULL;
+}
+
 static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
 {
     uint64_t i;
@@ -XXX,XX +XXX,XX @@ static void write_help(void)
 " -n, -- with -z, don't allow slow fallback\n"
 " -p, -- ignored for backwards compatibility\n"
 " -P, -- use different pattern to fill file\n"
+" -s, -- use a pattern file to fill the write buffer\n"
 " -C, -- report statistics in a machine parsable format\n"
 " -q, -- quiet mode, do not show I/O statistics\n"
 " -u, -- with -z, allow unmapping\n"
@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
     .perm       = BLK_PERM_WRITE,
     .argmin     = 2,
     .argmax     = -1,
-    .args       = "[-bcCfnquz] [-P pattern] off len",
+    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
     .oneline    = "writes a number of bytes at a specified offset",
     .help       = write_help,
 };
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
 {
     struct timespec t1, t2;
     bool Cflag = false, qflag = false, bflag = false;
-    bool Pflag = false, zflag = false, cflag = false;
+    bool Pflag = false, zflag = false, cflag = false, sflag = false;
     int flags = 0;
     int c, cnt, ret;
     char *buf = NULL;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     /* Some compilers get confused and warn if this is not initialized.  */
     int64_t total = 0;
     int pattern = 0xcd;
+    const char *file_name = NULL;
 
-    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
+    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
         switch (c) {
         case 'b':
             bflag = true;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         case 'q':
             qflag = true;
             break;
+        case 's':
+            sflag = true;
+            file_name = optarg;
+            break;
         case 'u':
             flags |= BDRV_REQ_MAY_UNMAP;
             break;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         return -EINVAL;
     }
 
-    if (zflag && Pflag) {
-        printf("-z and -P cannot be specified at the same time\n");
+    if (zflag + Pflag + sflag > 1) {
+        printf("Only one of -z, -P, and -s "
+               "can be specified at the same time\n");
         return -EINVAL;
     }
 
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     }
 
     if (!zflag) {
-        buf = qemu_io_alloc(blk, count, pattern);
+        if (sflag) {
+            buf = qemu_io_alloc_from_file(blk, count, file_name);
+            if (!buf) {
+                return -EINVAL;
+            }
+        } else {
+            buf = qemu_io_alloc(blk, count, pattern);
+        }
     }
 
     clock_gettime(CLOCK_MONOTONIC, &t1);
-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

It's wrong to OR shared permissions. It may lead to crash on further
permission updates.
Also, no needs to consider previously calculated permissions, as at
this point we already bind all new parents and bdrv_get_cumulative_perm
result is enough. So fix the bug by just set permissions by
bdrv_get_cumulative_perm result.

Bug was introduced in long ago 234ac1a9025, in 2.9.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
 {
     BdrvChild *c, *next;
     GSList *list = NULL, *p;
-    uint64_t old_perm, old_shared;
     uint64_t perm = 0, shared = BLK_PERM_ALL;
     int ret;
 
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
         bdrv_unref(from);
     }
 
-    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
-    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
+    bdrv_get_cumulative_perm(to, &perm, &shared);
+    bdrv_set_perm(to, perm, shared);
 
 out:
     g_slist_free(list);
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

When creating an image with preallocation "off" or "falloc", the first
block of the image is typically not allocated. When using Gluster
storage backed by XFS filesystem, reading this block using direct I/O
succeeds regardless of request length, fooling alignment detection.

In this case we fallback to a safe value (4096) instead of the optimal
value (512), which may lead to unneeded data copying when aligning
requests.  Allocating the first block avoids the fallback.

Since we allocate the first block even with preallocation=off, we no
longer create images with zero disk size:

$ ./qemu-img create -f raw test.raw 1g
    Formatting 'test.raw', fmt=raw size=1073741824

$ ls -lhs test.raw
    4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw

And converting the image requires additional cluster:

$ ./qemu-img measure -f raw -O qcow2 test.raw
    required size: 458752
    fully allocated size: 1074135040

When using format like vmdk with multiple files per image, we allocate
one block per file:

$ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
    Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat

$ ls -lhs test*.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk

I did quick performance test for copying disks with qemu-img convert to
new raw target image to Gluster storage with sector size of 512 bytes:

for i in $(seq 10); do
        rm -f dst.raw
        sleep 10
        time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
    done

Here is a table comparing the total time spent:

Type    Before(s)   After(s)    Diff(%)
---------------------------------------
real      530.028    469.123      -11.4
user       17.204     10.768      -37.4
sys        17.881      7.011      -60.7

We can see very clear improvement in CPU usage.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190827010528.8818-2-nsoffer@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c                            | 51 +++++++++++++++++++
 tests/qemu-iotests/059.out                    |  2 +-
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++++
 tests/qemu-iotests/175                        | 19 ++++---
 tests/qemu-iotests/175.out                    |  8 +--
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/221.out                    | 12 +++--
 tests/qemu-iotests/253.out                    | 12 +++--
 9 files changed, 99 insertions(+), 21 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
     return ret;
 }
 
+/*
+ * Help alignment probing by allocating the first block.
+ *
+ * When reading with direct I/O from unallocated area on Gluster backed by XFS,
+ * reading succeeds regardless of request length. In this case we fallback to
+ * safe alignment which is not optimal. Allocating the first block avoids this
+ * fallback.
+ *
+ * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
+ * request alignment, so we use safe values.
+ *
+ * Returns: 0 on success, -errno on failure. Since this is an optimization,
+ * caller may ignore failures.
+ */
+static int allocate_first_block(int fd, size_t max_size)
+{
+    size_t write_size = (max_size < MAX_BLOCKSIZE)
+        ? BDRV_SECTOR_SIZE
+        : MAX_BLOCKSIZE;
+    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
+    void *buf;
+    ssize_t n;
+    int ret;
+
+    buf = qemu_memalign(max_align, write_size);
+    memset(buf, 0, write_size);
+
+    do {
+        n = pwrite(fd, buf, write_size, 0);
+    } while (n == -1 && errno == EINTR);
+
+    ret = (n == -1) ? -errno : 0;
+
+    qemu_vfree(buf);
+    return ret;
+}
+
 static int handle_aiocb_truncate(void *opaque)
 {
     RawPosixAIOData *aiocb = opaque;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
                 /* posix_fallocate() doesn't set errno. */
                 error_setg_errno(errp, -result,
                                  "Could not preallocate new data");
+            } else if (current_length == 0) {
+                /*
+                 * posix_fallocate() uses fallocate() if the filesystem
+                 * supports it, or fallback to manually writing zeroes. If
+                 * fallocate() was used, unaligned reads from the fallocated
+                 * area in raw_probe_alignment() will succeed, hence we need to
+                 * allocate the first block.
+                 *
+                 * Optimize future alignment probing; ignore failures.
+                 */
+                allocate_first_block(fd, offset);
             }
         } else {
             result = 0;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
         if (ftruncate(fd, offset) != 0) {
             result = -errno;
             error_setg_errno(errp, -result, "Could not resize file");
+        } else if (current_length == 0 && offset > current_length) {
+            /* Optimize future alignment probing; ignore failures. */
+            allocate_first_block(fd, offset);
         }
         return result;
     default:
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
-disk size: 16 KiB
+disk size: 1.97 MiB
 Format specific information:
     cid: XXXXXXXX
     parent cid: XXXXXXXX
diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
similarity index 100%
rename from tests/qemu-iotests/150.out
rename to tests/qemu-iotests/150.out.qcow2
diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/150.out.raw
@@ -XXX,XX +XXX,XX @@
+QA output created by 150
+
+=== Mapping sparse conversion ===
+
+Offset          Length          File
+0               0x1000          TEST_DIR/t.IMGFMT
+
+=== Mapping non-sparse conversion ===
+
+Offset          Length          File
+0               0x100000        TEST_DIR/t.IMGFMT
+*** done
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # the file size.  This function hides the resulting difference in the
 # stat -c '%b' output.
 # Parameter 1: Number of blocks an empty file occupies
-# Parameter 2: Image size in bytes
+# Parameter 2: Minimal number of blocks in an image
+# Parameter 3: Image size in bytes
 _filter_blocks()
 {
     extra_blocks=$1
-    img_size=$2
+    min_blocks=$2
+    img_size=$3
 
-    sed -e "s/blocks=$extra_blocks\$\$\\|[^0-9]\$/nothing allocated/" \
-        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/everything allocated/"
+    sed -e "s/blocks=$min_blocks\$\$\\|[^0-9]\$/min allocation/" \
+        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
 # get standard environment, filters and checks
@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
 touch "$TEST_DIR/empty"
 extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 
+# We always write the first byte; check how many blocks this filesystem
+# allocates to match empty image alloation.
+printf "\0" > "$TEST_DIR/empty"
+min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
+
 echo
 echo "== creating image with default preallocation =="
 _make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 
 for mode in off full falloc; do
     echo
     echo "== creating image with preallocation $mode =="
     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
-    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
 # success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
 
 == creating image with default preallocation ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation off ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation full ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, everything allocated
+size=1048576, max allocation
 
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, everything allocated
+size=1048576, max allocation
  *** done
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 == raw input image with data (human) ==
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
-required size: 393216
+required size: 458752
 fully allocated size: 1074135040
 wrote 512/512 bytes at offset 512
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
 {
-    "required": 393216,
+    "required": 458752,
     "fully-allocated": 1074135040
 }
 wrote 512/512 bytes at offset 512
diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/221.out
+++ b/tests/qemu-iotests/221.out
@@ -XXX,XX +XXX,XX @@ QA output created by 221
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 1/1 bytes at offset 65536
 1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 *** done
diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/253.out
+++ b/tests/qemu-iotests/253.out
@@ -XXX,XX +XXX,XX @@ QA output created by 253
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 65535/65535 bytes at offset 983040
 63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Using block_resize we can test allocate_first_block() with file
descriptor opened with O_DIRECT, ensuring that it works for any size
larger than 4096 bytes.

Testing smaller sizes is tricky as the result depends on the filesystem
used for testing. For example on NFS any size will work since O_DIRECT
does not require any alignment.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190827010528.8818-3-nsoffer@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  8 ++++++++
 2 files changed, 36 insertions(+)

diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ _filter_blocks()
         -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
+# Resize image using block_resize.
+# Parameter 1: image path
+# Parameter 2: new size
+_block_resize()
+{
+    local path=$1
+    local size=$2
+
+    $QEMU -qmp stdio -nographic -nodefaults \
+        -blockdev file,node-name=file,filename=$path,cache.direct=on \
+        <<EOF
+{'execute': 'qmp_capabilities'}
+{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
+{'execute': 'quit'}
+EOF
+}
+
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
 _supported_proto file
 _supported_os Linux
 
+_default_cache_mode none
+_supported_cache_modes none directsync
+
 size=$((1 * 1024 * 1024))
 
 touch "$TEST_DIR/empty"
@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
     stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
+for new_size in 4096 1048576; do
+    echo
+    echo "== resize empty image with block_resize =="
+    _make_test_img 0 | _filter_imgfmt
+    _block_resize $TEST_IMG $new_size >/dev/null
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
+done
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
 size=1048576, max allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=4096, min allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=1048576, min allocation
  *** done
-- 
2.21.0

fe646693acc changed qemu-img create's output so that it no longer prints
single quotes around parameter values.  The subformat and adapter_type
filters in _filter_img_create() have never been adapted to that change.

Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059.out       | 16 ++++++++--------
 tests/qemu-iotests/common.filter |  4 ++--
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
 
 === Testing monolithicFlat creation and opening ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 2 GiB (2147483648 bytes)
 
 === Testing monolithicFlat with zeroed_grain ===
 qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 
 === Testing big twoGbMaxExtentFlat ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
@@ -XXX,XX +XXX,XX @@ Format specific information:
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
 
 === Testing truncated sparse ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
 
 === Converting to streamOptimized from image with small cluster size===
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
 
 === Testing version 3 ===
@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing 4TB monolithicFlat creation and IO ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 4 TiB (4398046511104 bytes)
@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing qemu-img map on extents ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0               0x20000         0x3f0000        TEST_DIR/t.vmdk
 0x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 0x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
         -e "s# compat6=\$on\\|off\$##g" \
         -e "s# static=\$on\\|off\$##g" \
         -e "s# zeroed_grain=\$on\\|off\$##g" \
-        -e "s# subformat='[^']*'##g" \
-        -e "s# adapter_type='[^']*'##g" \
+        -e "s# subformat=[^ ]*##g" \
+        -e "s# adapter_type=[^ ]*##g" \
         -e "s# hwversion=[^ ]*##g" \
         -e "s# lazy_refcounts=\$on\\|off\$##g" \
         -e "s# block_size=[0-9]\\+##g" \
-- 
2.21.0

This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
turns a former error in 059 into success.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190815153638.4600-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
 tests/qemu-iotests/059     |  7 +++--
 tests/qemu-iotests/059.out |  4 ++-
 3 files changed, 42 insertions(+), 23 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
 }
 
 static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-                              const char *desc_file_path, QDict *options,
-                              Error **errp)
+                              QDict *options, Error **errp)
 {
     int ret;
     int matches;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
     const char *p, *np;
     int64_t sectors = 0;
     int64_t flat_offset;
+    char *desc_file_dir = NULL;
     char *extent_path;
     BdrvChild *extent_file;
     BDRVVmdkState *s = bs->opaque;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             continue;
         }
 
-        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
-            !desc_file_path[0])
-        {
-            bdrv_refresh_filename(bs->file->bs);
-            error_setg(errp, "Cannot use relative extent paths with VMDK "
-                       "descriptor file '%s'", bs->file->bs->filename);
-            return -EINVAL;
-        }
+        if (path_is_absolute(fname)) {
+            extent_path = g_strdup(fname);
+        } else {
+            if (!desc_file_dir) {
+                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
+                if (!desc_file_dir) {
+                    bdrv_refresh_filename(bs->file->bs);
+                    error_prepend(errp, "Cannot use relative paths with VMDK "
+                                  "descriptor file '%s': ",
+                                  bs->file->bs->filename);
+                    ret = -EINVAL;
+                    goto out;
+                }
+            }
 
-        extent_path = path_combine(desc_file_path, fname);
+            extent_path = g_strconcat(desc_file_dir, fname, NULL);
+        }
 
         ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
         assert(ret < 32);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
         g_free(extent_path);
         if (local_err) {
             error_propagate(errp, local_err);
-            return -EINVAL;
+            ret = -EINVAL;
+            goto out;
         }
 
         /* save to extents array */
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                             0, 0, 0, 0, 0, &extent, errp);
             if (ret < 0) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent->flat_start_offset = flat_offset << 9;
         } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             g_free(buf);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else if (!strcmp(type, "SESPARSE")) {
             ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else {
             error_setg(errp, "Unsupported extent type '%s'", type);
             bdrv_unref_child(bs, extent_file);
-            return -ENOTSUP;
+            ret = -ENOTSUP;
+            goto out;
         }
         extent->type = g_strdup(type);
     }
-    return 0;
+
+    ret = 0;
+    goto out;
 
 invalid:
     np = next_line(p);
@@ -XXX,XX +XXX,XX @@ invalid:
         np--;
     }
     error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
-    return -EINVAL;
+    ret = -EINVAL;
+
+out:
+    g_free(desc_file_dir);
+    return ret;
 }
 
 static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
     }
     s->create_type = g_strdup(ct);
     s->desc_offset = 0;
-    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
-                             errp);
+    ret = vmdk_parse_extents(buf, bs, options, errp);
 exit:
     return ret;
 }
diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+# Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
-$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
-    | _filter_testdir | _filter_imgfmt
+$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
+         -c info \
+    2>&1 \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
 echo
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 
 === Testing monolithicFlat with internally generated JSON file name ===
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
+format name: IMGFMT
+cluster size: 0 bytes
+vm state offset: 0 bytes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

We had a test for a case where relative extent paths did not work, but
unfortunately we just fixed the underlying problem, so it works now.
This patch adds a new test case that still fails.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-4-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
 tests/qemu-iotests/059.out |  4 ++++
 2 files changed, 31 insertions(+)

diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+
+echo '--- blkdebug ---'
 # Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
     | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
+echo '--- quorum ---'
+# Should not work, because bdrv_dirname() does not work with quorum
+IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
+cp "$TEST_IMG" "$TEST_IMG.orig"
+
+filename="json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"quorum\",
+        \"children\": [ {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG\"
+        }, {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG.orig\"
+        } ],
+        \"vote-threshold\": 1
+    } }"
+
+filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
+$QEMU_IMG info "$filename" 2>&1 \
+    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
+
+
 echo
 echo "=== Testing version 3 ==="
 _use_sample_img iotest-version3.vmdk.bz2
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
+--- blkdebug ---
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 format name: IMGFMT
 cluster size: 0 bytes
 vm state offset: 0 bytes
+--- quorum ---
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

Compressed writes generally have to write full clusters, not just in
theory but also in practice when it comes to vmdk's streamOptimized
subformat.  It currently is just silently broken for writes with
non-zero in-cluster offsets:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 4096
4 KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
read failed: Invalid argument

(The technical reason is that vmdk_write_extent() just writes the
incomplete compressed data actually to offset 4k.  When reading the
data, vmdk_read_extent() looks at offset 0 and finds the compressed data
size to be 0, because that is what it reads from there.  This yields an
error.)

For incomplete writes with zero in-cluster offsets, the error path when
reading the rest of the cluster is a bit different, but the result is
the same:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 0
4 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
read failed: Invalid argument

(Here, vmdk_read_extent() finds the data and then sees that the
uncompressed data is short.)

It is better to reject invalid writes than to make the user believe they
might have succeeded and then fail when trying to read it back.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-5-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
     if (extent->compressed) {
         void *compressed_data;
 
+        /* Only whole clusters */
+        if (offset_in_cluster ||
+            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
+            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
+             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
+        {
+            ret = -EINVAL;
+            goto out;
+        }
+
         if (!extent->has_marker) {
             ret = -EINVAL;
             goto out;
-- 
2.21.0

streamOptimized does not support writes that do not span exactly one
cluster.  Furthermore, it cannot rewrite already allocated clusters.
As such, many iotests do not work with it.  Disable them.

diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/002
+++ b/tests/qemu-iotests/002
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/003
+++ b/tests/qemu-iotests/003
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 size=128M
 offset=67M
diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/005
+++ b/tests/qemu-iotests/005
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 _supported_os Linux
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 # vpc is limited to 127GB, so we can't test it here
 if [ "$IMGFMT" = "vpc" ]; then
diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/009
+++ b/tests/qemu-iotests/009
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/010
+++ b/tests/qemu-iotests/010
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/011
+++ b/tests/qemu-iotests/011
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto generic
 _unsupported_proto vxhs
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/018
+++ b/tests/qemu-iotests/018
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _supported_os Linux
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/019
+++ b/tests/qemu-iotests/019
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 CLUSTER_SIZE=65536
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/027
+++ b/tests/qemu-iotests/027
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vmdk qcow qcow2 qed
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/032
+++ b/tests/qemu-iotests/032
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # This works for any image format (though unlikely to segfault for raw)
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo === Prepare image ===
diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/033
+++ b/tests/qemu-iotests/033
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/034
+++ b/tests/qemu-iotests/034
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/037
+++ b/tests/qemu-iotests/037
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/063
+++ b/tests/qemu-iotests/063
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 _make_test_img 4M
 
diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/072
+++ b/tests/qemu-iotests/072
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
 _supported_proto file
+_unsupported_imgopts "subformat=streamOptimized"
 
 IMG_SIZE=64M
 
diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/105
+++ b/tests/qemu-iotests/105
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2 vmdk vhdx qed
 _supported_proto generic
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 echo
 echo "creating large image"
diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/197
+++ b/tests/qemu-iotests/197
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/215
+++ b/tests/qemu-iotests/215
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/251
+++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _supported_os Linux
+_unsupported_imgopts "subformat=streamOptimized"
 
 if [ "$IMGOPTSSYNTAX" = "true" ]; then
     # We use json:{} filenames here, so we cannot work with additional options.
-- 
2.21.0

The error message for the test case where we have a quorum node for
which no directory name can be generated is different: For
twoGbMaxExtentSparse, it complains that it cannot open the extent file.
For other (sub)formats, it just notes that it cannot determine the
backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
for simplicity's sake.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-7-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/110 | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Any format supporting backing files
 _supported_fmt qed qcow qcow2 vmdk
 _supported_proto file
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=twoGbMaxExtentSparse"
 
 TEST_IMG_REL=$(basename "$TEST_IMG")
 
-- 
2.21.0

From: Stefan Hajnoczi <stefanha@redhat.com>

Fixes: a6b257a08e3d72219f03e461a52152672fec0612
       ("file-posix: Handle undetectable alignment")
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190827101328.4062-1-stefanha@redhat.com
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
             align = alignments[i];
             if (raw_is_io_aligned(fd, buf + align, max_align)) {
-                /* Fallback to request_aligment. */
+                /* Fallback to request_alignment. */
                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
                 break;
             }
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

It is possible to enable only a subset of the block drivers with the
"--block-drv-rw-whitelist" option of the "configure" script. All other
drivers are marked as unusable (or only included as read-only with the
"--block-drv-ro-whitelist" option). If an iotest is now using such a
disabled block driver, it is failing - which is bad, since at least the
tests in the "auto" group should be able to deal with this situation.
Thus let's introduce a "_require_drivers" function that can be used by
the shell tests to check for the availability of certain drivers first,
and marks the test as "not run" if one of the drivers is missing.

This patch mainly targets the test in the "auto" group which should
never fail in such a case, but also improves some of the other tests
along the way. Note that we also assume that the "qcow2" and "file"
drivers are always available - otherwise it does not make sense to
run "make check-block" at all (which only tests with qcow2 by default).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823133552.11680-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/071       |  1 +
 tests/qemu-iotests/081       |  4 +---
 tests/qemu-iotests/099       |  1 +
 tests/qemu-iotests/120       |  1 +
 tests/qemu-iotests/162       |  4 +---
 tests/qemu-iotests/184       |  1 +
 tests/qemu-iotests/186       |  1 +
 tests/qemu-iotests/common.rc | 14 ++++++++++++++
 8 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/071
+++ b/tests/qemu-iotests/071
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers blkdebug blkverify
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/081
+++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt raw
 _supported_proto file
 _supported_os Linux
+_require_drivers quorum
 
 do_run_qemu()
 {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                           | _filter_qemu_io | _filter_generated_node_ids
 }
 
-test_quorum=$($QEMU_IMG --help|grep quorum)
-[ "$test_quorum" = "" ] && _supported_fmt quorum
-
 quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
 quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
 quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/099
+++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
 _supported_proto file
 _supported_os Linux
+_require_drivers blkdebug blkverify
 _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
     "subformat=twoGbMaxExtentSparse"
 
diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/120
+++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _unsupported_fmt luks
+_require_drivers raw
 
 _make_test_img 64M
 
diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/162
+++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
-
-test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh$ \|$$')
-[ "$test_ssh" = "" ] && _notrun "ssh support required"
+_require_drivers ssh
 
 echo
 echo '=== NBD ==='
diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/184
+++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_os Linux
+_require_drivers throttle
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/186
+++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers null-co
 
 if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
     _notrun "Requires a PC machine"
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -XXX,XX +XXX,XX @@ _require_command()
     [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
 }
 
+# Check that a set of drivers has been whitelisted in the QEMU binary
+#
+_require_drivers()
+{
+    available=$($QEMU -drive format=help | \
+                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
+    for driver
+    do
+        if ! echo "$available" | grep -q " $driver$ \|$$"; then
+            _notrun "$driver not available"
+        fi
+    done
+}
+
 # make sure this script returns success
 true
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

The sanitizers (especially the address sanitizer from Clang) are
sometimes printing out warnings or false positives - this spoils
the output of the iotests, causing some of the tests to fail.
Thus let's skip the automatic iotests during "make check" when the
user configured QEMU with --enable-sanitizers.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823084203.29734-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/check-block.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tests/check-block.sh b/tests/check-block.sh
index XXXXXXX..XXXXXXX 100755
--- a/tests/check-block.sh
+++ b/tests/check-block.sh
@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
     exit 0
 fi
 
+if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
+    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
+    exit 0
+fi
+
 if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
     echo "No qemu-system binary available ==> Not running the qemu-iotests."
     exit 0
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Quoting cache mode is not needed, and most tests use unquoted values.
Unify all test to use the same style.

Message-id: 20190827173432.7656-1-nsoffer@redhat.com
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/026 | 4 ++--
 tests/qemu-iotests/039 | 4 ++--
 tests/qemu-iotests/052 | 2 +-
 tests/qemu-iotests/091 | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/026
+++ b/tests/qemu-iotests/026
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Currently only qcow2 supports rebasing
 _supported_fmt qcow2
 _supported_proto file
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough" "none"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough none
 # The refcount table tests expect a certain minimum width for refcount entries
 # (so that the refcount table actually needs to grow); that minimum is 16 bits,
 # being the default refcount entry width.
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough
 
 size=128M
 
diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/052
+++ b/tests/qemu-iotests/052
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto file
 
 # Don't do O_DIRECT on tmpfs
-_supported_cache_modes "writeback" "writethrough" "unsafe"
+_supported_cache_modes writeback writethrough unsafe
 
 size=128M
 _make_test_img $size
diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/091
+++ b/tests/qemu-iotests/091
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "none"
-_supported_cache_modes "writethrough" "none" "writeback"
+_default_cache_mode none
+_supported_cache_modes writethrough none writeback
 
 size=1G
 
-- 
2.21.0

The following changes since commit 754f756cc4c6d9d14b7230c62b5bb20f9d655888:

Merge tag 'pull-target-arm-20220422-1' of https://git.linaro.org/people/pmaydell/qemu-arm into staging (2022-04-22 08:03:18 -0700)

are available in the Git repository at:

https://gitlab.com/hreitz/qemu.git tags/pull-block-2022-04-25

for you to fetch changes up to 348a0740afc5b313599533eb69bbb2b95d2f1bba:

iotests/108: Fix when missing user_allow_other (2022-04-25 14:46:45 +0200)

----------------------------------------------------------------
Block patches:
- New @force parameter for blockdev-change-medium
- Improvements to the iotests to help with debugging
- Fix iotest 108 on systems without user_allow_other in fuse.conf

----------------------------------------------------------------
Denis V. Lunev (1):
  block: add 'force' parameter to 'blockdev-change-medium' command

Hanna Reitz (1):
  iotests/108: Fix when missing user_allow_other

John Snow (12):
  iotests: replace calls to log(qemu_io(...)) with qemu_io_log()
  iotests/163: Fix broken qemu-io invocation
  iotests: Don't check qemu_io() output for specific error strings
  iotests/040: Don't check image pattern on zero-length image
  iotests/040: Fix TestCommitWithFilters test
  iotests: create generic qemu_tool() function
  iotests: rebase qemu_io() on top of qemu_tool()
  iotests/migration-permissions: use assertRaises() for qemu_io()
    negative test
  iotests/image-fleecing: switch to qemu_io()
  iotests: remove qemu_io_pipe_and_status()
  iotests: remove qemu_io_silent() and qemu_io_silent_check().
  iotests: make qemu_io_log() check return codes by default

-- 
2.35.1

From: "Denis V. Lunev" <den@openvz.org>

'blockdev-change-medium' is a convinient wrapper for the following
sequence of commands:
 * blockdev-open-tray
 * blockdev-remove-medium
 * blockdev-insert-medium
 * blockdev-close-tray
and should be used f.e. to change ISO image inside the CD-ROM tray.
Though the guest could lock the tray and some linux guests like
CentOS 8.5 actually does that. In this case the execution if this
command results in the error like the following:
  Device 'scsi0-0-1-0' is locked and force was not specified,
  wait for tray to open and try again.

This situation is could be resolved 'blockdev-open-tray' by passing
flag 'force' inside. Thus is seems reasonable to add the same
capability for 'blockdev-change-medium' too.

Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@openvz.org>
Acked-by: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
CC: Kevin Wolf <kwolf@redhat.com>
CC: Hanna Reitz <hreitz@redhat.com>
CC: Eric Blake <eblake@redhat.com>
CC: Markus Armbruster <armbru@redhat.com>
Message-Id: <20220412221846.280723-1-den@openvz.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 qapi/block.json     |  6 ++++++
 block/qapi-sysemu.c |  3 ++-
 monitor/hmp-cmds.c  |  4 +++-
 hmp-commands.hx     | 11 +++++++----
 ui/cocoa.m          |  1 +
 5 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/qapi/block.json b/qapi/block.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -XXX,XX +XXX,XX @@
 # @read-only-mode: change the read-only mode of the device; defaults
 #                  to 'retain'
 #
+# @force: if false (the default), an eject request through blockdev-open-tray
+#         will be sent to the guest if it has locked the tray (and the tray
+#         will not be opened immediately); if true, the tray will be opened
+#         regardless of whether it is locked. (since 7.1)
+#
 # Features:
 # @deprecated: Member @device is deprecated.  Use @id instead.
 #
@@ -XXX,XX +XXX,XX @@
             '*id': 'str',
             'filename': 'str',
             '*format': 'str',
+            '*force': 'bool',
             '*read-only-mode': 'BlockdevChangeReadOnlyMode' } }
 
 
diff --git a/block/qapi-sysemu.c b/block/qapi-sysemu.c
index XXXXXXX..XXXXXXX 100644
--- a/block/qapi-sysemu.c
+++ b/block/qapi-sysemu.c
@@ -XXX,XX +XXX,XX @@ void qmp_blockdev_change_medium(bool has_device, const char *device,
                                 bool has_id, const char *id,
                                 const char *filename,
                                 bool has_format, const char *format,
+                                bool has_force, bool force,
                                 bool has_read_only,
                                 BlockdevChangeReadOnlyMode read_only,
                                 Error **errp)
@@ -XXX,XX +XXX,XX @@ void qmp_blockdev_change_medium(bool has_device, const char *device,
 
     rc = do_open_tray(has_device ? device : NULL,
                       has_id ? id : NULL,
-                      false, &err);
+                      force, &err);
     if (rc && rc != -ENOSYS) {
         error_propagate(errp, err);
         goto fail;
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -XXX,XX +XXX,XX @@ void hmp_change(Monitor *mon, const QDict *qdict)
     const char *target = qdict_get_str(qdict, "target");
     const char *arg = qdict_get_try_str(qdict, "arg");
     const char *read_only = qdict_get_try_str(qdict, "read-only-mode");
+    bool force = qdict_get_try_bool(qdict, "force", false);
     BlockdevChangeReadOnlyMode read_only_mode = 0;
     Error *err = NULL;
 
@@ -XXX,XX +XXX,XX @@ void hmp_change(Monitor *mon, const QDict *qdict)
         }
 
         qmp_blockdev_change_medium(true, device, false, NULL, target,
-                                   !!arg, arg, !!read_only, read_only_mode,
+                                   !!arg, arg, true, force,
+                                   !!read_only, read_only_mode,
                                    &err);
     }
 
diff --git a/hmp-commands.hx b/hmp-commands.hx
index XXXXXXX..XXXXXXX 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -XXX,XX +XXX,XX @@ ERST
 
     {
         .name       = "change",
-        .args_type  = "device:B,target:F,arg:s?,read-only-mode:s?",
-        .params     = "device filename [format [read-only-mode]]",
-        .help       = "change a removable medium, optional format",
+        .args_type  = "device:B,force:-f,target:F,arg:s?,read-only-mode:s?",
+        .params     = "device [-f] filename [format [read-only-mode]]",
+        .help       = "change a removable medium, optional format, use -f to force the operation",
         .cmd        = hmp_change,
     },
 
@@ -XXX,XX +XXX,XX @@ SRST
 ``change`` *device* *setting*
   Change the configuration of a device.
 
-  ``change`` *diskdevice* *filename* [*format* [*read-only-mode*]]
+  ``change`` *diskdevice* [-f] *filename* [*format* [*read-only-mode*]]
     Change the medium for a removable disk device to point to *filename*. eg::
 
       (qemu) change ide1-cd0 /path/to/some.iso
 
+    ``-f``
+      forces the operation even if the guest has locked the tray.
+
     *format* is optional.
 
     *read-only-mode* may be used to change the read-only status of the device.
diff --git a/ui/cocoa.m b/ui/cocoa.m
index XXXXXXX..XXXXXXX 100644
--- a/ui/cocoa.m
+++ b/ui/cocoa.m
@@ -XXX,XX +XXX,XX @@ - (void)changeDeviceMedia:(id)sender
                                        [file cStringUsingEncoding:
                                                  NSASCIIStringEncoding],
                                        true, "raw",
+                                       true, false,
                                        false, 0,
                                        &err);
         });
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

This makes these callsites a little simpler, but the real motivation is
a forthcoming commit will change the return type of qemu_io(), so removing
users of the return value now is helpful.

diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/242
+++ b/tests/qemu-iotests/242
@@ -XXX,XX +XXX,XX @@
 import iotests
 import json
 import struct
-from iotests import qemu_img_create, qemu_io, qemu_img_info, \
-    file_path, img_info_log, log, filter_qemu_io
+from iotests import qemu_img_create, qemu_io_log, qemu_img_info, \
+    file_path, img_info_log, log
 
 iotests.script_initialize(supported_fmts=['qcow2'],
                           supported_protocols=['file'],
@@ -XXX,XX +XXX,XX @@ def add_bitmap(bitmap_number, persistent, disabled):
 
 def write_to_disk(offset, size):
     write = 'write {} {}'.format(offset, size)
-    log(qemu_io('-c', write, disk), filters=[filter_qemu_io])
+    qemu_io_log('-c', write, disk)
 
 
 def toggle_flag(offset):
diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/255
+++ b/tests/qemu-iotests/255
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('src.qcow2') as src_path, \
     iotests.qemu_img_create('-f', iotests.imgfmt, src_path, size_str)
     iotests.qemu_img_create('-f', iotests.imgfmt, dst_path, size_str)
 
-    iotests.log(iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write 0 1M',
-                                src_path),
-                filters=[iotests.filter_test_dir, iotests.filter_qemu_io])
+    iotests.qemu_io_log('-f', iotests.imgfmt, '-c', 'write 0 1M', src_path),
 
     vm.add_object('throttle-group,x-bps-read=4096,id=throttle0')
 
diff --git a/tests/qemu-iotests/303 b/tests/qemu-iotests/303
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/303
+++ b/tests/qemu-iotests/303
@@ -XXX,XX +XXX,XX @@
 
 import iotests
 import subprocess
-from iotests import qemu_img_create, qemu_io, file_path, log, filter_qemu_io, \
+from iotests import qemu_img_create, qemu_io_log, file_path, log, \
         verify_qcow2_zstd_compression
 
 iotests.script_initialize(supported_fmts=['qcow2'],
@@ -XXX,XX +XXX,XX @@ def create_bitmap(bitmap_number, disabled):
 
 def write_to_disk(offset, size):
     write = f'write {offset} {size}'
-    log(qemu_io('-c', write, disk), filters=[filter_qemu_io])
+    qemu_io_log('-c', write, disk)
 
 
 def add_bitmap(num, begin, end, disabled):
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

The 'read' commands to qemu-io were malformed, and this invocation only
worked by coincidence because the error messages were identical. Oops.

There's no point in checking the patterning of the reference image, so
just check the empty image by itself instead.

(Note: as of this commit, nothing actually enforces that this command
completes successfully, but a forthcoming commit in this series will
enforce that qemu_io() must have a zero status code.)

diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -XXX,XX +XXX,XX @@ class ShrinkBaseClass(iotests.QMPTestCase):
         qemu_img('resize',  '-f', iotests.imgfmt, '--shrink', test_img,
                  self.shrink_size)
 
-        self.assertEqual(
-            qemu_io('-c', 'read -P 0x00 %s'%self.shrink_size, test_img),
-            qemu_io('-c', 'read -P 0x00 %s'%self.shrink_size, check_img),
-            "Verifying image content")
+        qemu_io('-c', f"read -P 0x00 0 {self.shrink_size}", test_img)
 
         self.image_verify()
 
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

A forthcoming commit updates qemu_io() to raise an exception on non-zero
return by default, and changes its return type.

In preparation, simplify some calls to qemu_io() that assert that
specific error message strings do not appear in qemu-io's
output. Asserting that all of these calls return a status code of zero
will be a more robust way to guard against failure.

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
 
     def test_commit(self):
         self.run_commit_test(mid_img, backing_img)
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
+        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
+        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
     def test_commit_node(self):
         self.run_commit_test("mid", "base", node_names=True)
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
+        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
+        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
     @iotests.skip_if_unsupported(['throttle'])
     def test_commit_with_filter_and_quit(self):
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
 
     def test_top_is_active(self):
         self.run_commit_test(test_img, backing_img, need_ready=True)
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
+        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
+        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
     def test_top_is_default_active(self):
         self.run_default_commit_test()
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img).find("verification failed"))
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img).find("verification failed"))
+        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
+        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
     def test_top_and_base_reversed(self):
         self.assert_no_active_block_jobs()
@@ -XXX,XX +XXX,XX @@ class TestRelativePaths(ImageCommitTestCase):
 
     def test_commit(self):
         self.run_commit_test(self.mid_img, self.backing_img)
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs).find("verification failed"))
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs).find("verification failed"))
+        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs)
+        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs)
 
     def test_device_not_found(self):
         result = self.vm.qmp('block-commit', device='nonexistent', top='%s' % self.mid_img)
@@ -XXX,XX +XXX,XX @@ class TestRelativePaths(ImageCommitTestCase):
 
     def test_top_is_active(self):
         self.run_commit_test(self.test_img, self.backing_img)
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs).find("verification failed"))
-        self.assertEqual(-1, qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs).find("verification failed"))
+        qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', self.backing_img_abs)
+        qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', self.backing_img_abs)
 
     def test_top_and_base_reversed(self):
         self.assert_no_active_block_jobs()
@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
 
     def do_test_io(self, read_or_write):
         for index, pattern_file in enumerate(self.pattern_files):
-            result = qemu_io('-f', iotests.imgfmt,
-                             '-c',
-                             f'{read_or_write} -P {index + 1} {index}M 1M',
-                             pattern_file)
-            self.assertFalse('Pattern verification failed' in result)
+            qemu_io('-f', iotests.imgfmt,
+                    '-c',
+                    f'{read_or_write} -P {index + 1} {index}M 1M',
+                    pattern_file)
 
     @iotests.skip_if_unsupported(['throttle'])
     def setUp(self):
diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -XXX,XX +XXX,XX @@ class TestSyncModesNoneAndTop(iotests.QMPTestCase):
 
         self.vm.shutdown()
         time.sleep(1)
-        self.assertEqual(-1, qemu_io('-c', 'read -P0x41 0 512', target_img).find("verification failed"))
+        qemu_io('-c', 'read -P0x41 0 512', target_img)
 
 class TestBeforeWriteNotifier(iotests.QMPTestCase):
     def setUp(self):
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

qemu-io fails on read/write beyond end-of-file on raw images, so skip
these invocations when running the zero-length image tests.

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
         qemu_img('create', '-f', iotests.imgfmt,
                  '-o', 'backing_file=%s' % mid_img,
                  '-F', iotests.imgfmt, test_img)
-        qemu_io('-f', 'raw', '-c', 'write -P 0xab 0 524288', backing_img)
-        qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xef 524288 524288', mid_img)
+        if self.image_len:
+            qemu_io('-f', 'raw', '-c', 'write -P 0xab 0 524288', backing_img)
+            qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xef 524288 524288',
+                    mid_img)
         self.vm = iotests.VM().add_drive(test_img, "node-name=top,backing.node-name=mid,backing.backing.node-name=base", interface="none")
         self.vm.add_device('virtio-scsi')
         self.vm.add_device("scsi-hd,id=scsi0,drive=drive0")
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
 
     def test_commit(self):
         self.run_commit_test(mid_img, backing_img)
+        if not self.image_len:
+            return
         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
     def test_commit_node(self):
         self.run_commit_test("mid", "base", node_names=True)
+        if not self.image_len:
+            return
         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(ImageCommitTestCase):
 
     def test_top_is_active(self):
         self.run_commit_test(test_img, backing_img, need_ready=True)
+        if not self.image_len:
+            return
         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
     def test_top_is_default_active(self):
         self.run_default_commit_test()
+        if not self.image_len:
+            return
         qemu_io('-f', 'raw', '-c', 'read -P 0xab 0 524288', backing_img)
         qemu_io('-f', 'raw', '-c', 'read -P 0xef 524288 524288', backing_img)
 
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

Without this change, asserting that qemu_io always returns 0 causes this
test to fail in a way we happened not to be catching previously:

qemu.utils.VerboseProcessError: Command
  '('/home/jsnow/src/qemu/bin/git/tests/qemu-iotests/../../qemu-io',
  '--cache', 'writeback', '--aio', 'threads', '-f', 'qcow2', '-c',
  'read -P 4 3M 1M',
  '/home/jsnow/src/qemu/bin/git/tests/qemu-iotests/scratch/3.img')'
  returned non-zero exit status 1.
  ┏━ output ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ┃ qemu-io: can't open device
  ┃ /home/jsnow/src/qemu/bin/git/tests/qemu-iotests/scratch/3.img:
  ┃ Could not open backing file: Could not open backing file: Throttle
  ┃ group 'tg' does not exist
  ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The commit jobs changes the backing file string stored in the image file
header belonging to the node above the commit’s top node to point to the
commit target (the base node).  QEMU tries to be as accurate as
possible, and so in these test cases will include the filter that is
part of the block graph in that backing file string (by virtue of making
it a json:{} description of the post-commit subgraph).  This makes
little sense outside of QEMU, though: Specifically, the throttle node in
that subgraph will dearly miss its supposedly associated throttle group
object.

When starting the commit job, we can specify a custom backing file
string to write into said image file, so let’s use that feature to write
the plain filename of the backing chain’s next actual image file there.

Explicitly provide the backing file so that opening the file outside of
QEMU (Where we will not have throttle groups) will succeed.

Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20220418211504.943969-6-jsnow@redhat.com>
---
 tests/qemu-iotests/040 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
                              job_id='commit',
                              device='top-filter',
                              top_node='cow-2',
-                             base_node='cow-1')
+                             base_node='cow-1',
+                             backing_file=self.img1)
         self.assert_qmp(result, 'return', {})
         self.wait_until_completed(drive='commit')
 
@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
                              job_id='commit',
                              device='top-filter',
                              top_node='cow-1',
-                             base_node='cow-0')
+                             base_node='cow-0',
+                             backing_file=self.img0)
         self.assert_qmp(result, 'return', {})
         self.wait_until_completed(drive='commit')
 
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

reimplement qemu_img() in terms of qemu_tool() in preparation for doing
the same with qemu_io().

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def qemu_img_create_prepare_args(args: List[str]) -> List[str]:
 
     return result
 
-def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
-             ) -> 'subprocess.CompletedProcess[str]':
-    """
-    Run qemu_img and return the status code and console output.
 
-    This function always prepends QEMU_IMG_OPTIONS and may further alter
-    the args for 'create' commands.
+def qemu_tool(*args: str, check: bool = True, combine_stdio: bool = True
+              ) -> 'subprocess.CompletedProcess[str]':
+    """
+    Run a qemu tool and return its status code and console output.
 
-    :param args: command-line arguments to qemu-img.
+    :param args: full command line to run.
     :param check: Enforce a return code of zero.
     :param combine_stdio: set to False to keep stdout/stderr separated.
 
@@ -XXX,XX +XXX,XX @@ def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
         properties. If streams are not combined, it will also have a
         stderr property.
     """
-    full_args = qemu_img_args + qemu_img_create_prepare_args(list(args))
-
     subp = subprocess.run(
-        full_args,
+        args,
         stdout=subprocess.PIPE,
         stderr=subprocess.STDOUT if combine_stdio else subprocess.PIPE,
         universal_newlines=True,
@@ -XXX,XX +XXX,XX @@ def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
 
     if check and subp.returncode or (subp.returncode < 0):
         raise VerboseProcessError(
-            subp.returncode, full_args,
+            subp.returncode, args,
             output=subp.stdout,
             stderr=subp.stderr,
         )
@@ -XXX,XX +XXX,XX @@ def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
     return subp
 
 
+def qemu_img(*args: str, check: bool = True, combine_stdio: bool = True
+             ) -> 'subprocess.CompletedProcess[str]':
+    """
+    Run QEMU_IMG_PROG and return its status code and console output.
+
+    This function always prepends QEMU_IMG_OPTIONS and may further alter
+    the args for 'create' commands.
+
+    See `qemu_tool()` for greater detail.
+    """
+    full_args = qemu_img_args + qemu_img_create_prepare_args(list(args))
+    return qemu_tool(*full_args, check=check, combine_stdio=combine_stdio)
+
+
 def ordered_qmp(qmsg, conv_keys=True):
     # Dictionaries are not ordered prior to 3.6, therefore:
     if isinstance(qmsg, list):
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

Rework qemu_io() to be analogous to qemu_img(); a function that requires
a return code of zero by default unless disabled explicitly.

Tests that use qemu_io():
030 040 041 044 055 056 093 124 129 132 136 148 149 151 152 163 165 205
209 219 236 245 248 254 255 257 260 264 280 298 300 302 304
image-fleecing migrate-bitmaps-postcopy-test migrate-bitmaps-test
migrate-during-backup migration-permissions

Test that use qemu_io_log():
242 245 255 274 303 307 nbd-reconnect-on-open

Copy-pastables for testing/verification:

./check -qcow2 030 040 041 044 055 056 124 129 132 151 152 163 165 209 \
               219 236 242 245 248 254 255 257 260 264 274 \
               280 298 300 302 303 304 307 image-fleecing \
               migrate-bitmaps-postcopy-test migrate-bitmaps-test \
               migrate-during-backup nbd-reconnect-on-open
./check -raw 093 136 148 migration-permissions
./check -nbd 205

# ./configure configure --disable-gnutls --enable-gcrypt
# this ALSO requires passwordless sudo.
./check -luks 149

# Just the tests that were edited in this commit:
./check -qcow2 030 040 242 245
./check -raw migration-permissions
./check -nbd 205
./check -luks 149

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20220418211504.943969-8-jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/030                        | 85 +++++++++++--------
 tests/qemu-iotests/149                        |  6 +-
 tests/qemu-iotests/205                        |  4 +-
 tests/qemu-iotests/245                        | 17 ++--
 tests/qemu-iotests/iotests.py                 | 19 +++--
 .../qemu-iotests/tests/migration-permissions  |  4 +-
 6 files changed, 81 insertions(+), 54 deletions(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-        self.assertEqual(qemu_io('-f', 'raw', '-c', 'map', backing_img),
-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
-                         'image file map does not match backing file after streaming')
+        self.assertEqual(
+            qemu_io('-f', 'raw', '-c', 'map', backing_img).stdout,
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
+            'image file map does not match backing file after streaming')
 
     def test_stream_intermediate(self):
         self.assert_no_active_block_jobs()
 
-        self.assertNotEqual(qemu_io('-f', 'raw', '-rU', '-c', 'map', backing_img),
-                            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', mid_img),
-                            'image file map matches backing file before streaming')
+        self.assertNotEqual(
+            qemu_io('-f', 'raw', '-rU', '-c', 'map', backing_img).stdout,
+            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', mid_img).stdout,
+            'image file map matches backing file before streaming')
 
         result = self.vm.qmp('block-stream', device='mid', job_id='stream-mid')
         self.assert_qmp(result, 'return', {})
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-        self.assertEqual(qemu_io('-f', 'raw', '-c', 'map', backing_img),
-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img),
-                         'image file map does not match backing file after streaming')
+        self.assertEqual(
+            qemu_io('-f', 'raw', '-c', 'map', backing_img).stdout,
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img).stdout,
+            'image file map does not match backing file after streaming')
 
     def test_stream_pause(self):
         self.assert_no_active_block_jobs()
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-        self.assertEqual(qemu_io('-f', 'raw', '-c', 'map', backing_img),
-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
-                         'image file map does not match backing file after streaming')
+        self.assertEqual(
+            qemu_io('-f', 'raw', '-c', 'map', backing_img).stdout,
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
+            'image file map does not match backing file after streaming')
 
     def test_stream_no_op(self):
         self.assert_no_active_block_jobs()
 
         # The image map is empty before the operation
-        empty_map = qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', test_img)
+        empty_map = qemu_io(
+            '-f', iotests.imgfmt, '-rU', '-c', 'map', test_img).stdout
 
         # This is a no-op: no data should ever be copied from the base image
         result = self.vm.qmp('block-stream', device='drive0', base=mid_img)
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
-                         empty_map, 'image file map changed after a no-op')
+        self.assertEqual(
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
+            empty_map, 'image file map changed after a no-op')
 
     def test_stream_partial(self):
         self.assert_no_active_block_jobs()
@@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img),
-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img),
-                         'image file map does not match backing file after streaming')
+        self.assertEqual(
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', mid_img).stdout,
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img).stdout,
+            'image file map does not match backing file after streaming')
 
     def test_device_not_found(self):
         result = self.vm.qmp('block-stream', device='nonexistent')
@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
 
         # Check that the maps don't match before the streaming operations
         for i in range(2, self.num_imgs, 2):
-            self.assertNotEqual(qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i]),
-                                qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i-1]),
-                                'image file map matches backing file before streaming')
+            self.assertNotEqual(
+                qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i]).stdout,
+                qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[i-1]).stdout,
+                'image file map matches backing file before streaming')
 
         # Create all streaming jobs
         pending_jobs = []
@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
 
         # Check that all maps match now
         for i in range(2, self.num_imgs, 2):
-            self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i]),
-                             qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i-1]),
-                             'image file map does not match backing file after streaming')
+            self.assertEqual(
+                qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i]).stdout,
+                qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[i-1]).stdout,
+                'image file map does not match backing file after streaming')
 
     # Test that it's not possible to perform two block-stream
     # operations if there are nodes involved in both.
@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
     def test_stream_base_node_name(self):
         self.assert_no_active_block_jobs()
 
-        self.assertNotEqual(qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[4]),
-                            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[3]),
-                            'image file map matches backing file before streaming')
+        self.assertNotEqual(
+            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[4]).stdout,
+            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.imgs[3]).stdout,
+            'image file map matches backing file before streaming')
 
         # Error: the base node does not exist
         result = self.vm.qmp('block-stream', device='node4', base_node='none', job_id='stream')
@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[4]),
-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[3]),
-                         'image file map matches backing file after streaming')
+        self.assertEqual(
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[4]).stdout,
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.imgs[3]).stdout,
+            'image file map matches backing file after streaming')
 
 class TestQuorum(iotests.QMPTestCase):
     num_children = 3
@@ -XXX,XX +XXX,XX @@ class TestQuorum(iotests.QMPTestCase):
             os.remove(img)
 
     def test_stream_quorum(self):
-        self.assertNotEqual(qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.children[0]),
-                            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.backing[0]),
-                            'image file map matches backing file before streaming')
+        self.assertNotEqual(
+            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.children[0]).stdout,
+            qemu_io('-f', iotests.imgfmt, '-rU', '-c', 'map', self.backing[0]).stdout,
+            'image file map matches backing file before streaming')
 
         self.assert_no_active_block_jobs()
 
@@ -XXX,XX +XXX,XX @@ class TestQuorum(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
         self.vm.shutdown()
 
-        self.assertEqual(qemu_io('-f', iotests.imgfmt, '-c', 'map', self.children[0]),
-                         qemu_io('-f', iotests.imgfmt, '-c', 'map', self.backing[0]),
-                         'image file map does not match backing file after streaming')
+        self.assertEqual(
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.children[0]).stdout,
+            qemu_io('-f', iotests.imgfmt, '-c', 'map', self.backing[0]).stdout,
+            'image file map does not match backing file after streaming')
 
 class TestSmallerBackingFile(iotests.QMPTestCase):
     backing_len = 1 * 1024 * 1024 # MB
diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -XXX,XX +XXX,XX @@ def qemu_io_write_pattern(config, pattern, offset_mb, size_mb, dev=False):
     args = ["-c", "write -P 0x%x %dM %dM" % (pattern, offset_mb, size_mb)]
     args.extend(qemu_io_image_args(config, dev))
     iotests.log("qemu-io " + " ".join(args), filters=[iotests.filter_test_dir])
-    iotests.log(check_cipher_support(config, iotests.qemu_io(*args)),
+    output = iotests.qemu_io(*args, check=False).stdout
+    iotests.log(check_cipher_support(config, output),
                 filters=[iotests.filter_test_dir, iotests.filter_qemu_io])
 
 
@@ -XXX,XX +XXX,XX @@ def qemu_io_read_pattern(config, pattern, offset_mb, size_mb, dev=False):
     args = ["-c", "read -P 0x%x %dM %dM" % (pattern, offset_mb, size_mb)]
     args.extend(qemu_io_image_args(config, dev))
     iotests.log("qemu-io " + " ".join(args), filters=[iotests.filter_test_dir])
-    iotests.log(check_cipher_support(config, iotests.qemu_io(*args)),
+    output = iotests.qemu_io(*args, check=False).stdout
+    iotests.log(check_cipher_support(config, output),
                 filters=[iotests.filter_test_dir, iotests.filter_qemu_io])
 
 
diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/205
+++ b/tests/qemu-iotests/205
@@ -XXX,XX +XXX,XX @@ class TestNbdServerRemove(iotests.QMPTestCase):
 
     def do_test_connect_after_remove(self, mode=None):
         args = ('-r', '-f', 'raw', '-c', 'read 0 512', nbd_uri)
-        self.assertReadOk(qemu_io(*args))
+        self.assertReadOk(qemu_io(*args).stdout)
 
         result = self.remove_export('exp', mode)
         self.assert_qmp(result, 'return', {})
 
         self.assertExportNotFound('exp')
-        self.assertConnectFailed(qemu_io(*args))
+        self.assertConnectFailed(qemu_io(*args, check=False).stdout)
 
     def test_connect_after_remove_default(self):
         self.do_test_connect_after_remove()
diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/245
+++ b/tests/qemu-iotests/245
@@ -XXX,XX +XXX,XX @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+import copy
+import json
 import os
 import re
+from subprocess import CalledProcessError
+
 import iotests
-import copy
-import json
 from iotests import qemu_img, qemu_io
 
 hd_path = [
@@ -XXX,XX +XXX,XX @@ class TestBlockdevReopen(iotests.QMPTestCase):
 
     # Reopen an image several times changing some of its options
     def test_reopen(self):
-        # Check whether the filesystem supports O_DIRECT
-        if 'O_DIRECT' in qemu_io('-f', 'raw', '-t', 'none', '-c', 'quit', hd_path[0]):
-            supports_direct = False
-        else:
+        try:
+            qemu_io('-f', 'raw', '-t', 'none', '-c', 'quit', hd_path[0])
             supports_direct = True
+        except CalledProcessError as exc:
+            if 'O_DIRECT' in exc.stdout:
+                supports_direct = False
+            else:
+                raise
 
         # Open the hd1 image passing all backing options
         opts = hd_opts(1)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def qemu_io_wrap_args(args: Sequence[str]) -> List[str]:
 def qemu_io_popen(*args):
     return qemu_tool_popen(qemu_io_wrap_args(args))
 
-def qemu_io(*args):
-    '''Run qemu-io and return the stdout data'''
-    return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))[0]
+def qemu_io(*args: str, check: bool = True, combine_stdio: bool = True
+            ) -> 'subprocess.CompletedProcess[str]':
+    """
+    Run QEMU_IO_PROG and return the status code and console output.
+
+    This function always prepends either QEMU_IO_OPTIONS or
+    QEMU_IO_OPTIONS_NO_FMT.
+    """
+    return qemu_tool(*qemu_io_wrap_args(args),
+                     check=check, combine_stdio=combine_stdio)
 
 def qemu_io_pipe_and_status(*args):
     return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))
 
-def qemu_io_log(*args):
-    result = qemu_io(*args)
-    log(result, filters=[filter_testfiles, filter_qemu_io])
+def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
+    result = qemu_io(*args, check=False)
+    log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
     return result
 
 def qemu_io_silent(*args):
diff --git a/tests/qemu-iotests/tests/migration-permissions b/tests/qemu-iotests/tests/migration-permissions
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/migration-permissions
+++ b/tests/qemu-iotests/tests/migration-permissions
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
     def test_post_migration_permissions(self):
         # Try to access the image R/W, which should fail because virtio-blk
         # has not been configured with share-rw=on
-        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img)
+        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
         if not log.strip():
             print('ERROR (pre-migration): qemu-io should not be able to '
                   'access this image, but it reported no error')
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
 
         # Try the same qemu-io access again, verifying that the WRITE
         # permission remains unshared
-        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img)
+        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
         if not log.strip():
             print('ERROR (post-migration): qemu-io should not be able to '
                   'access this image, but it reported no error')
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

Modify this test to use assertRaises for its negative testing of
qemu_io. If the exception raised does not match the one we tell it to
expect, we get *that* exception unhandled. If we get no exception, we
get a unittest assertion failure and the provided emsg printed to
screen.

If we get the CalledProcessError exception but the output is not what we
expect, we re-raise the original CalledProcessError.

Tidy.

(Note: Yes, you can reference "with" objects after that block ends; it
just means that ctx.__exit__(...) will have been called on it. It does
not *actually* go out of scope. unittests expects you to want to inspect
the Exception object, so they leave it defined post-exit.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Tested-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220418211504.943969-9-jsnow@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 .../qemu-iotests/tests/migration-permissions  | 28 +++++++++----------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/tests/qemu-iotests/tests/migration-permissions b/tests/qemu-iotests/tests/migration-permissions
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/migration-permissions
+++ b/tests/qemu-iotests/tests/migration-permissions
@@ -XXX,XX +XXX,XX @@
 #
 
 import os
+from subprocess import CalledProcessError
+
 import iotests
 from iotests import imgfmt, qemu_img_create, qemu_io
 
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
     def test_post_migration_permissions(self):
         # Try to access the image R/W, which should fail because virtio-blk
         # has not been configured with share-rw=on
-        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
-        if not log.strip():
-            print('ERROR (pre-migration): qemu-io should not be able to '
-                  'access this image, but it reported no error')
-        else:
-            # This is the expected output
-            assert 'Is another process using the image' in log
+        emsg = ('ERROR (pre-migration): qemu-io should not be able to '
+                'access this image, but it reported no error')
+        with self.assertRaises(CalledProcessError, msg=emsg) as ctx:
+            qemu_io('-f', imgfmt, '-c', 'quit', test_img)
+        if 'Is another process using the image' not in ctx.exception.stdout:
+            raise ctx.exception
 
         # Now migrate the VM
         self.vm_s.qmp('migrate', uri=f'unix:{mig_sock}')
@@ -XXX,XX +XXX,XX @@ class TestMigrationPermissions(iotests.QMPTestCase):
 
         # Try the same qemu-io access again, verifying that the WRITE
         # permission remains unshared
-        log = qemu_io('-f', imgfmt, '-c', 'quit', test_img, check=False).stdout
-        if not log.strip():
-            print('ERROR (post-migration): qemu-io should not be able to '
-                  'access this image, but it reported no error')
-        else:
-            # This is the expected output
-            assert 'Is another process using the image' in log
+        emsg = ('ERROR (post-migration): qemu-io should not be able to '
+                'access this image, but it reported no error')
+        with self.assertRaises(CalledProcessError, msg=emsg) as ctx:
+            qemu_io('-f', imgfmt, '-c', 'quit', test_img)
+        if 'Is another process using the image' not in ctx.exception.stdout:
+            raise ctx.exception
 
 
 if __name__ == '__main__':
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

This test expects failure ... but only sometimes. When? Why?

It's for reads of a region not defined by a bitmap. Adjust the test to
be more explicit about what it expects to fail and why.

diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/image-fleecing
+++ b/tests/qemu-iotests/tests/image-fleecing
@@ -XXX,XX +XXX,XX @@
 #
 # Creator/Owner: John Snow <jsnow@redhat.com>
 
+from subprocess import CalledProcessError
+
 import iotests
-from iotests import log, qemu_img, qemu_io, qemu_io_silent, \
-    qemu_io_pipe_and_status
+from iotests import log, qemu_img, qemu_io, qemu_io_silent
 
 iotests.script_initialize(
     supported_fmts=['qcow2'],
@@ -XXX,XX +XXX,XX @@ def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
         for p in patterns + zeroes:
             cmd = 'read -P%s %s %s' % p
             log(cmd)
-            out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
-                                               nbd_uri)
-            if ret != 0:
-                print(out)
+
+            try:
+                qemu_io('-r', '-f', 'raw', '-c', cmd, nbd_uri)
+            except CalledProcessError as exc:
+                if bitmap and p in zeroes:
+                    log(exc.stdout)
+                else:
+                    raise
 
     log('')
     log('--- Testing COW ---')
@@ -XXX,XX +XXX,XX @@ def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
             args += [target_img_path]
         else:
             args += ['-f', 'raw', nbd_uri]
-        out, ret = qemu_io_pipe_and_status(*args)
-        if ret != 0:
-            print(out)
+
+        try:
+            qemu_io(*args)
+        except CalledProcessError as exc:
+            if bitmap and p in zeroes:
+                log(exc.stdout)
+            else:
+                raise
 
     log('')
     log('--- Cleanup ---')
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

I know we just added it, sorry. This is done in favor of qemu_io() which
*also* returns the console output and status, but with more robust error
handling on failure.

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def qemu_io(*args: str, check: bool = True, combine_stdio: bool = True
     return qemu_tool(*qemu_io_wrap_args(args),
                      check=check, combine_stdio=combine_stdio)
 
-def qemu_io_pipe_and_status(*args):
-    return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))
-
 def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
     result = qemu_io(*args, check=False)
     log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

Like qemu-img, qemu-io returning 0 should be the norm and not the
exception. Remove all calls to qemu_io_silent that just assert the
return code is zero (That's every last call, as it turns out), and
replace them with a normal qemu_io() call.

qemu_io_silent_check() appeared to have been unused already.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220418211504.943969-12-jsnow@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/216                        | 12 +++++-----
 tests/qemu-iotests/218                        |  5 ++---
 tests/qemu-iotests/224                        |  4 ++--
 tests/qemu-iotests/258                        | 11 +++++-----
 tests/qemu-iotests/298                        | 17 ++++++--------
 tests/qemu-iotests/310                        | 22 +++++++++----------
 tests/qemu-iotests/iotests.py                 | 16 --------------
 tests/qemu-iotests/tests/image-fleecing       |  4 ++--
 .../tests/mirror-ready-cancel-error           |  2 +-
 .../qemu-iotests/tests/stream-error-on-reset  |  4 ++--
 10 files changed, 38 insertions(+), 59 deletions(-)

diff --git a/tests/qemu-iotests/216 b/tests/qemu-iotests/216
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/216
+++ b/tests/qemu-iotests/216
@@ -XXX,XX +XXX,XX @@
 # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent
+from iotests import log, qemu_img, qemu_io
 
 # Need backing file support
 iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk'],
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
     log('')
 
     qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
-    assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
+    qemu_io(base_img_path, '-c', 'write -P 1 0M 1M')
     qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
              '-F', iotests.imgfmt, top_img_path)
-    assert qemu_io_silent(top_img_path,  '-c', 'write -P 2 1M 1M') == 0
+    qemu_io(top_img_path,  '-c', 'write -P 2 1M 1M')
 
     log('Done')
 
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
     log('--- Checking COR result ---')
     log('')
 
-    assert qemu_io_silent(base_img_path, '-c', 'discard 0 64M') == 0
-    assert qemu_io_silent(top_img_path,  '-c', 'read -P 1 0M 1M') == 0
-    assert qemu_io_silent(top_img_path,  '-c', 'read -P 2 1M 1M') == 0
+    qemu_io(base_img_path, '-c', 'discard 0 64M')
+    qemu_io(top_img_path,  '-c', 'read -P 1 0M 1M')
+    qemu_io(top_img_path,  '-c', 'read -P 2 1M 1M')
 
     log('Done')
diff --git a/tests/qemu-iotests/218 b/tests/qemu-iotests/218
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/218
+++ b/tests/qemu-iotests/218
@@ -XXX,XX +XXX,XX @@
 # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent
+from iotests import log, qemu_img, qemu_io
 
 iotests.script_initialize(supported_fmts=['qcow2', 'raw'])
 
@@ -XXX,XX +XXX,XX @@ with iotests.VM() as vm, \
      iotests.FilePath('src.img') as src_img_path:
 
     qemu_img('create', '-f', iotests.imgfmt, src_img_path, '64M')
-    assert qemu_io_silent('-f', iotests.imgfmt, src_img_path,
-                          '-c', 'write -P 42 0M 64M') == 0
+    qemu_io('-f', iotests.imgfmt, src_img_path, '-c', 'write -P 42 0M 64M')
 
     vm.launch()
 
diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/224
+++ b/tests/qemu-iotests/224
@@ -XXX,XX +XXX,XX @@
 # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent, filter_qmp_testfiles, \
+from iotests import log, qemu_img, qemu_io, filter_qmp_testfiles, \
                     filter_qmp_imgfmt
 import json
 
@@ -XXX,XX +XXX,XX @@ for filter_node_name in False, True:
                  '-F', iotests.imgfmt, top_img_path)
 
         # Something to commit
-        assert qemu_io_silent(mid_img_path, '-c', 'write -P 1 0 1M') == 0
+        qemu_io(mid_img_path, '-c', 'write -P 1 0 1M')
 
         vm.launch()
 
diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/258
+++ b/tests/qemu-iotests/258
@@ -XXX,XX +XXX,XX @@
 # Creator/Owner: Hanna Reitz <hreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent, \
+from iotests import log, qemu_img, qemu_io, \
         filter_qmp_testfiles, filter_qmp_imgfmt
 
 # Returns a node for blockdev-add
@@ -XXX,XX +XXX,XX @@ def test_concurrent_finish(write_to_stream_node):
         if write_to_stream_node:
             # This is what (most of the time) makes commit finish
             # earlier and then pull in stream
-            assert qemu_io_silent(node2_path,
-                                  '-c', 'write %iK 64K' % (65536 - 192),
-                                  '-c', 'write %iK 64K' % (65536 -  64)) == 0
+            qemu_io(node2_path,
+                    '-c', 'write %iK 64K' % (65536 - 192),
+                    '-c', 'write %iK 64K' % (65536 -  64))
 
             stream_throttle='tg'
         else:
             # And this makes stream finish earlier
-            assert qemu_io_silent(node1_path,
-                                  '-c', 'write %iK 64K' % (65536 - 64)) == 0
+            qemu_io(node1_path, '-c', 'write %iK 64K' % (65536 - 64))
 
             commit_throttle='tg'
 
diff --git a/tests/qemu-iotests/298 b/tests/qemu-iotests/298
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/298
+++ b/tests/qemu-iotests/298
@@ -XXX,XX +XXX,XX @@ class TestTruncate(iotests.QMPTestCase):
         os.remove(refdisk)
 
     def do_test(self, prealloc_mode, new_size):
-        ret = iotests.qemu_io_silent('--image-opts', '-c', 'write 0 10M', '-c',
-                                     f'truncate -m {prealloc_mode} {new_size}',
-                                     drive_opts)
-        self.assertEqual(ret, 0)
-
-        ret = iotests.qemu_io_silent('-f', iotests.imgfmt, '-c', 'write 0 10M',
-                                     '-c',
-                                     f'truncate -m {prealloc_mode} {new_size}',
-                                     refdisk)
-        self.assertEqual(ret, 0)
+        iotests.qemu_io('--image-opts', '-c', 'write 0 10M', '-c',
+                        f'truncate -m {prealloc_mode} {new_size}',
+                        drive_opts)
+
+        iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write 0 10M',
+                        '-c', f'truncate -m {prealloc_mode} {new_size}',
+                        refdisk)
 
         stat = os.stat(disk)
         refstat = os.stat(refdisk)
diff --git a/tests/qemu-iotests/310 b/tests/qemu-iotests/310
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/310
+++ b/tests/qemu-iotests/310
@@ -XXX,XX +XXX,XX @@
 #
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent
+from iotests import log, qemu_img, qemu_io
 
 # Need backing file support
 iotests.script_initialize(supported_fmts=['qcow2'],
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
     log('')
 
     qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M')
-    assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
-    assert qemu_io_silent(base_img_path, '-c', 'write -P 1 3M 1M') == 0
+    qemu_io(base_img_path, '-c', 'write -P 1 0M 1M')
+    qemu_io(base_img_path, '-c', 'write -P 1 3M 1M')
     qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
              '-F', iotests.imgfmt, mid_img_path)
-    assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 2M 1M') == 0
-    assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 4M 1M') == 0
+    qemu_io(mid_img_path, '-c', 'write -P 3 2M 1M')
+    qemu_io(mid_img_path, '-c', 'write -P 3 4M 1M')
     qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
              '-F', iotests.imgfmt, top_img_path)
-    assert qemu_io_silent(top_img_path, '-c', 'write -P 2 1M 1M') == 0
+    qemu_io(top_img_path, '-c', 'write -P 2 1M 1M')
 
 #      0 1 2 3 4
 # top    2
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('base.img') as base_img_path, \
     # Detach backing to check that we can read the data from the top level now
     qemu_img('rebase', '-u', '-b', '', '-f', iotests.imgfmt, top_img_path)
 
-    assert qemu_io_silent(top_img_path, '-c', 'read -P 0 0 1M') == 0
-    assert qemu_io_silent(top_img_path, '-c', 'read -P 2 1M 1M') == 0
-    assert qemu_io_silent(top_img_path, '-c', 'read -P 3 2M 1M') == 0
-    assert qemu_io_silent(top_img_path, '-c', 'read -P 0 3M 1M') == 0
-    assert qemu_io_silent(top_img_path, '-c', 'read -P 3 4M 1M') == 0
+    qemu_io(top_img_path, '-c', 'read -P 0 0 1M')
+    qemu_io(top_img_path, '-c', 'read -P 2 1M 1M')
+    qemu_io(top_img_path, '-c', 'read -P 3 2M 1M')
+    qemu_io(top_img_path, '-c', 'read -P 0 3M 1M')
+    qemu_io(top_img_path, '-c', 'read -P 3 4M 1M')
 
     log('Done')
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
     log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
     return result
 
-def qemu_io_silent(*args):
-    '''Run qemu-io and return the exit code, suppressing stdout'''
-    args = qemu_io_wrap_args(args)
-    result = subprocess.run(args, stdout=subprocess.DEVNULL, check=False)
-    if result.returncode < 0:
-        sys.stderr.write('qemu-io received signal %i: %s\n' %
-                         (-result.returncode, ' '.join(args)))
-    return result.returncode
-
-def qemu_io_silent_check(*args):
-    '''Run qemu-io and return the true if subprocess returned 0'''
-    args = qemu_io_wrap_args(args)
-    result = subprocess.run(args, stdout=subprocess.DEVNULL,
-                            stderr=subprocess.STDOUT, check=False)
-    return result.returncode == 0
-
 class QemuIoInteractive:
     def __init__(self, *args):
         self.args = qemu_io_wrap_args(args)
diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/image-fleecing
+++ b/tests/qemu-iotests/tests/image-fleecing
@@ -XXX,XX +XXX,XX @@
 from subprocess import CalledProcessError
 
 import iotests
-from iotests import log, qemu_img, qemu_io, qemu_io_silent
+from iotests import log, qemu_img, qemu_io
 
 iotests.script_initialize(
     supported_fmts=['qcow2'],
@@ -XXX,XX +XXX,XX @@ def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
     for p in overwrite + remainder:
         cmd = 'read -P%s %s %s' % p
         log(cmd)
-        assert qemu_io_silent(base_img_path, '-c', cmd) == 0
+        qemu_io(base_img_path, '-c', cmd)
 
     log('')
     log('Done')
diff --git a/tests/qemu-iotests/tests/mirror-ready-cancel-error b/tests/qemu-iotests/tests/mirror-ready-cancel-error
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/mirror-ready-cancel-error
+++ b/tests/qemu-iotests/tests/mirror-ready-cancel-error
@@ -XXX,XX +XXX,XX @@ class TestMirrorReadyCancelError(iotests.QMPTestCase):
         # Ensure that mirror will copy something before READY so the
         # target format layer will forward the pre-READY flush to its
         # file child
-        assert iotests.qemu_io_silent('-c', 'write -P 1 0 64k', source) == 0
+        iotests.qemu_io('-c', 'write -P 1 0 64k', source)
 
         self.vm = iotests.VM()
         self.vm.launch()
diff --git a/tests/qemu-iotests/tests/stream-error-on-reset b/tests/qemu-iotests/tests/stream-error-on-reset
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/stream-error-on-reset
+++ b/tests/qemu-iotests/tests/stream-error-on-reset
@@ -XXX,XX +XXX,XX @@
 
 import os
 import iotests
-from iotests import imgfmt, qemu_img_create, qemu_io_silent, QMPTestCase
+from iotests import imgfmt, qemu_img_create, qemu_io, QMPTestCase
 
 
 image_size = 1 * 1024 * 1024
@@ -XXX,XX +XXX,XX @@ class TestStreamErrorOnReset(QMPTestCase):
         - top image is attached to a virtio-scsi device
         """
         qemu_img_create('-f', imgfmt, base, str(image_size))
-        assert qemu_io_silent('-c', f'write 0 {data_size}', base) == 0
+        qemu_io('-c', f'write 0 {data_size}', base)
         qemu_img_create('-f', imgfmt, top, str(image_size))
 
         self.vm = iotests.VM()
-- 
2.35.1

From: John Snow <jsnow@redhat.com>

Just like qemu_img_log(), upgrade qemu_io_log() to enforce a return code
of zero by default.

Tests that use qemu_io_log(): 242 245 255 274 303 307 nbd-reconnect-on-open

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def qemu_io(*args: str, check: bool = True, combine_stdio: bool = True
     return qemu_tool(*qemu_io_wrap_args(args),
                      check=check, combine_stdio=combine_stdio)
 
-def qemu_io_log(*args: str) -> 'subprocess.CompletedProcess[str]':
-    result = qemu_io(*args, check=False)
+def qemu_io_log(*args: str, check: bool = True
+                ) -> 'subprocess.CompletedProcess[str]':
+    result = qemu_io(*args, check=check)
     log(result.stdout, filters=[filter_testfiles, filter_qemu_io])
     return result
 
diff --git a/tests/qemu-iotests/tests/nbd-reconnect-on-open b/tests/qemu-iotests/tests/nbd-reconnect-on-open
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/nbd-reconnect-on-open
+++ b/tests/qemu-iotests/tests/nbd-reconnect-on-open
@@ -XXX,XX +XXX,XX @@ def check_fail_to_connect(open_timeout):
     log(f'Check fail to connect with {open_timeout} seconds of timeout')
 
     start_t = time.time()
-    qemu_io_log(*create_args(open_timeout))
+    qemu_io_log(*create_args(open_timeout), check=False)
     delta_t = time.time() - start_t
 
     max_delta = open_timeout + 0.2
-- 
2.35.1

FUSE exports' allow-other option defaults to "auto", which means that it
will try passing allow_other as a mount option, and fall back to not
using it when an error occurs.  We make no effort to hide fusermount's
error message (because it would be difficult, and because users might
want to know about the fallback occurring), and so when allow_other does
not work (primarily when /etc/fuse.conf does not contain
user_allow_other), this error message will appear and break the
reference output.

We do not need allow_other here, though, so we can just pass
allow-other=off to fix that.

Reported-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220421142435.569600-1-hreitz@redhat.com>
Tested-by: Markus Armbruster <armbru@redhat.com>
Tested-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/108 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/108 b/tests/qemu-iotests/108
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/108
+++ b/tests/qemu-iotests/108
@@ -XXX,XX +XXX,XX @@ else
 
     $QSD \
         --blockdev file,node-name=export-node,filename="$TEST_IMG" \
-        --export fuse,id=fuse-export,node-name=export-node,mountpoint="$export_mp",writable=on,growable=off \
+        --export fuse,id=fuse-export,node-name=export-node,mountpoint="$export_mp",writable=on,growable=off,allow-other=off \
         --pidfile "$TEST_DIR/qsd.pid" \
         --daemonize
 fi
-- 
2.35.1