Series comparison

-[Qemu-devel] [PULL 00/20] Block patches
+[Qemu-devel] [PULL 00/15] Block patches
-The following changes since commit 5ec2eca83dc478ddf24077e02a8b34dd26cd3ff9:
+The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:
-  Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20190613.0' into staging (2019-06-14 09:33:55 +0100)
+  Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)
 are available in the Git repository at:
-  https://github.com/XanClic/qemu.git tags/pull-block-2019-06-14
+  https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27
-for you to fetch changes up to 21c1ce592a144188dfe59b9e156a97da412a59a2:
+for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:
-  iotests: Test qemu-img convert -C --salvage (2019-06-14 15:09:42 +0200)
+  iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)
 ----------------------------------------------------------------
 Block patches:
-- Allow blockdev-backup from nodes that are not in qemu's main AIO
+- qemu-io now accepts a file to read a write pattern from
-  context to newly added nodes
+- Ensure that raw files have their first block allocated so we can probe
-- Add salvaging mode to qemu-img convert
+  the O_DIRECT alignment if necessary
-- Minor fixes to tests, documentation, and for less Valgrind annoyance
+- Various fixes
 ----------------------------------------------------------------
-Andrey Shinkevich (1):
+Denis Plotnikov (1):
-  hw/block/fdc: floppy command FIFO memory initialization
+  qemu-io: add pattern file for write command
-John Snow (6):
+Max Reitz (7):
-  blockdev-backup: don't check aio_context too early
+  iotests: Fix _filter_img_create()
-  iotests.py: do not use infinite waits
+  vmdk: Use bdrv_dirname() for relative extent paths
-  QEMUMachine: add events_wait method
+  iotests: Keep testing broken relative extent paths
-  iotests.py: rewrite run_job to be pickier
+  vmdk: Reject invalid compressed writes
-  iotests: add iotest 256 for testing blockdev-backup across iothread
+  iotests: Disable broken streamOptimized tests
-    contexts
+  iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
-  event_match: always match on None value
+  iotests: Disable 126 for flat vmdk subformats
-Max Reitz (12):
+Nir Soffer (3):
-  iotests: Filter 175's allocation information
+  block: posix: Always allocate the first block
-  iotests: Fix intermittent failure in 219
+  iotests: Test allocate_first_block() with O_DIRECT
-  qemu-img: Fix options leakage in img_rebase()
+  iotests: Unify cache mode quoting
-  qapi/block-core: Overlays are not snapshots
-  blockdev: Overlays are not snapshots
+Stefan Hajnoczi (1):
-  qemu-img: Move quiet into ImgConvertState
+  file-posix: fix request_alignment typo
-  qemu-img: Add salvaging mode to convert
-  blkdebug: Add @iotype error option
+Thomas Huth (2):
-  blkdebug: Add "none" event
+  iotests: Check for enabled drivers before testing them
-  blkdebug: Inject errors on .bdrv_co_block_status()
+  tests/check-block: Skip iotests when sanitizers are enabled
   iotests: Test qemu-img convert --salvage
   iotests: Test qemu-img convert -C --salvage
 Vladimir Sementsov-Ogievskiy (1):
-  iotests: restrict 254 to support only qcow2
+  block: fix permission update in bdrv_replace_node
- qapi/block-core.json          |  53 ++++++++---
+ block.c                                       |  5 +-
- block/blkdebug.c              |  60 ++++++++++--
+ block/file-posix.c                            | 53 +++++++++-
- blockdev.c                    |  14 +--
+ block/vmdk.c                                  | 64 ++++++++----
- hw/block/fdc.c                |   1 +
+ qemu-io-cmds.c                                | 99 +++++++++++++++++--
- qemu-img.c                    | 106 +++++++++++++++------
+ tests/check-block.sh                          |  5 +
- python/qemu/__init__.py       |  67 ++++++++++----
+ tests/qemu-iotests/002                        |  1 +
- qemu-img-cmds.hx              |   4 +-
+ tests/qemu-iotests/003                        |  1 +
- qemu-img.texi                 |   4 +
+ tests/qemu-iotests/005                        |  3 +-
- tests/qemu-iotests/082        |   1 +
+ tests/qemu-iotests/009                        |  1 +
- tests/qemu-iotests/082.out    |   3 +
+ tests/qemu-iotests/010                        |  1 +
- tests/qemu-iotests/085.out    |  10 +-
+ tests/qemu-iotests/011                        |  1 +
- tests/qemu-iotests/175        |  26 +++++-
+ tests/qemu-iotests/017                        |  3 +-
- tests/qemu-iotests/175.out    |   8 +-
+ tests/qemu-iotests/018                        |  3 +-
- tests/qemu-iotests/219        |  13 ++-
+ tests/qemu-iotests/019                        |  3 +-
- tests/qemu-iotests/251        | 170 ++++++++++++++++++++++++++++++++++
+ tests/qemu-iotests/020                        |  3 +-
- tests/qemu-iotests/251.out    |  43 +++++++++
+ tests/qemu-iotests/026                        |  4 +-
- tests/qemu-iotests/254        |   2 +
+ tests/qemu-iotests/027                        |  1 +
- tests/qemu-iotests/256        | 122 ++++++++++++++++++++++++
+ tests/qemu-iotests/032                        |  1 +
- tests/qemu-iotests/256.out    | 119 ++++++++++++++++++++++++
+ tests/qemu-iotests/033                        |  1 +
- tests/qemu-iotests/group      |   2 +
+ tests/qemu-iotests/034                        |  3 +-
- tests/qemu-iotests/iotests.py |  60 +++++++-----
+ tests/qemu-iotests/037                        |  3 +-
-files changed, 772 insertions(+), 116 deletions(-)
+ tests/qemu-iotests/039                        |  4 +-
- create mode 100755 tests/qemu-iotests/251
+ tests/qemu-iotests/052                        |  2 +-
- create mode 100644 tests/qemu-iotests/251.out
+ tests/qemu-iotests/059                        | 34 ++++++-
- create mode 100755 tests/qemu-iotests/256
+ tests/qemu-iotests/059.out                    | 26 +++--
- create mode 100644 tests/qemu-iotests/256.out
+ tests/qemu-iotests/063                        |  3 +-
  tests/qemu-iotests/071                        |  1 +
  tests/qemu-iotests/072                        |  1 +
  tests/qemu-iotests/081                        |  4 +-
  tests/qemu-iotests/091                        |  4 +-
  tests/qemu-iotests/099                        |  1 +
  tests/qemu-iotests/105                        |  3 +-
  tests/qemu-iotests/110                        |  3 +-
  tests/qemu-iotests/120                        |  1 +
  tests/qemu-iotests/126                        |  2 +
  tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
  tests/qemu-iotests/150.out.raw                | 12 +++
  tests/qemu-iotests/162                        |  4 +-
  tests/qemu-iotests/175                        | 47 +++++++--
  tests/qemu-iotests/175.out                    | 16 ++-
  tests/qemu-iotests/178.out.qcow2              |  4 +-
  tests/qemu-iotests/184                        |  1 +
  tests/qemu-iotests/186                        |  1 +
  tests/qemu-iotests/197                        |  1 +
  tests/qemu-iotests/215                        |  1 +
  tests/qemu-iotests/221.out                    | 12 ++-
  tests/qemu-iotests/251                        |  1 +
  tests/qemu-iotests/253.out                    | 12 ++-
  tests/qemu-iotests/common.filter              |  4 +-
  tests/qemu-iotests/common.rc                  | 14 +++
 files changed, 391 insertions(+), 87 deletions(-)
  rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
  create mode 100644 tests/qemu-iotests/150.out.raw
 --
 .21.0

-[Qemu-devel] [PULL 13/20] blockdev: Overlays are not snapshots
+[Qemu-devel] [PULL 01/15] qemu-io: add pattern file for write command
-There are error messages which refer to an overlay node as the snapshot.
+From: Denis Plotnikov <dplotnikov@virtuozzo.com>
 That is wrong, those are two different things.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+The patch allows to provide a pattern file for write
 command. There was no similar ability before.
 Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
 Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
 Reviewed-by: Eric Blake <eblake@redhat.com>
-Message-id: 20190603202236.1342-3-mreitz@redhat.com
+[mreitz: Keep optstring in alphabetical order]
 Reviewed-by: John Snow <jsnow@redhat.com>
 Reviewed-by: Alberto Garcia <berto@igalia.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- blockdev.c                 | 10 +++++-----
+ qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
- tests/qemu-iotests/085.out | 10 +++++-----
+file changed, 93 insertions(+), 6 deletions(-)
 files changed, 10 insertions(+), 10 deletions(-)
-diff --git a/blockdev.c b/blockdev.c
+diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
 index XXXXXXX..XXXXXXX 100644
---- a/blockdev.c
+--- a/qemu-io-cmds.c
-+++ b/blockdev.c
++++ b/qemu-io-cmds.c
-@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
+@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
-             s->has_snapshot_node_name ? s->snapshot_node_name : NULL;
+     qemu_vfree(p);
+ }
-         if (node_name && !snapshot_node_name) {
--            error_setg(errp, "New snapshot node name missing");
++/*
-+            error_setg(errp, "New overlay node name missing");
++ * qemu_io_alloc_from_file()
-             goto out;
++ *
-         }
++ * Allocates the buffer and populates it with the content of the given file
++ * up to @len bytes. If the file length is less than @len, then the buffer
-         if (snapshot_node_name &&
++ * is populated with the file content cyclically.
-             bdrv_lookup_bs(snapshot_node_name, snapshot_node_name, NULL)) {
++ *
--            error_setg(errp, "New snapshot node name already in use");
++ * @blk - the block backend where the buffer content is going to be written to
-+            error_setg(errp, "New overlay node name already in use");
++ * @len - the buffer length
-             goto out;
++ * @file_name - the file to read the content from
-         }
++ *
++ * Returns: the buffer pointer on success
-@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
++ *          NULL on error
 + */
 +static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
 +                                     const char *file_name)
 +{
 +    char *buf, *buf_origin;
 +    FILE *f = fopen(file_name, "r");
 +    int pattern_len;
 +
 +    if (!f) {
 +        perror(file_name);
 +        return NULL;
 +    }
 +
 +    if (qemuio_misalign) {
 +        len += MISALIGN_OFFSET;
 +    }
 +
 +    buf_origin = buf = blk_blockalign(blk, len);
 +
 +    if (qemuio_misalign) {
 +        buf_origin += MISALIGN_OFFSET;
 +        buf += MISALIGN_OFFSET;
 +        len -= MISALIGN_OFFSET;
 +    }
 +
 +    pattern_len = fread(buf_origin, 1, len, f);
 +
 +    if (ferror(f)) {
 +        perror(file_name);
 +        goto error;
 +    }
 +
 +    if (pattern_len == 0) {
 +        fprintf(stderr, "%s: file is empty\n", file_name);
 +        goto error;
 +    }
 +
 +    fclose(f);
 +
 +    if (len > pattern_len) {
 +        len -= pattern_len;
 +        buf += pattern_len;
 +
 +        while (len > 0) {
 +            size_t len_to_copy = MIN(pattern_len, len);
 +
 +            memcpy(buf, buf_origin, len_to_copy);
 +
 +            len -= len_to_copy;
 +            buf += len_to_copy;
 +        }
 +    }
 +
 +    return buf_origin;
 +
 +error:
 +    qemu_io_free(buf_origin);
 +    return NULL;
 +}
 +
  static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
  {
      uint64_t i;
@@ -XXX,XX +XXX,XX @@ static void write_help(void)
  " -n, -- with -z, don't allow slow fallback\n"
  " -p, -- ignored for backwards compatibility\n"
  " -P, -- use different pattern to fill file\n"
 +" -s, -- use a pattern file to fill the write buffer\n"
  " -C, -- report statistics in a machine parsable format\n"
  " -q, -- quiet mode, do not show I/O statistics\n"
  " -u, -- with -z, allow unmapping\n"
@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
      .perm       = BLK_PERM_WRITE,
      .argmin     = 2,
      .argmax     = -1,
 -    .args       = "[-bcCfnquz] [-P pattern] off len",
 +    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
      .oneline    = "writes a number of bytes at a specified offset",
      .help       = write_help,
  };
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
  {
      struct timespec t1, t2;
      bool Cflag = false, qflag = false, bflag = false;
 -    bool Pflag = false, zflag = false, cflag = false;
 +    bool Pflag = false, zflag = false, cflag = false, sflag = false;
      int flags = 0;
      int c, cnt, ret;
      char *buf = NULL;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
      /* Some compilers get confused and warn if this is not initialized.  */
      int64_t total = 0;
      int pattern = 0xcd;
 +    const char *file_name = NULL;
 -    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
 +    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
          switch (c) {
          case 'b':
              bflag = true;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
          case 'q':
              qflag = true;
              break;
 +        case 's':
 +            sflag = true;
 +            file_name = optarg;
 +            break;
          case 'u':
              flags |= BDRV_REQ_MAY_UNMAP;
              break;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
          return -EINVAL;
      }
-     if (bdrv_has_blk(state->new_bs)) {
+-    if (zflag && Pflag) {
--        error_setg(errp, "The snapshot is already in use");
+-        printf("-z and -P cannot be specified at the same time\n");
-+        error_setg(errp, "The overlay is already in use");
++    if (zflag + Pflag + sflag > 1) {
-         goto out;
++        printf("Only one of -z, -P, and -s "
 +               "can be specified at the same time\n");
          return -EINVAL;
      }
-@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
+@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
      }
-     if (state->new_bs->backing != NULL) {
+     if (!zflag) {
--        error_setg(errp, "The snapshot already has a backing image");
+-        buf = qemu_io_alloc(blk, count, pattern);
-+        error_setg(errp, "The overlay already has a backing image");
++        if (sflag) {
-         goto out;
++            buf = qemu_io_alloc_from_file(blk, count, file_name);
 +            if (!buf) {
 +                return -EINVAL;
 +            }
 +        } else {
 +            buf = qemu_io_alloc(blk, count, pattern);
 +        }
      }
-     if (!state->new_bs->drv->supports_backing) {
+     clock_gettime(CLOCK_MONOTONIC, &t1);
 -        error_setg(errp, "The snapshot does not support backing images");
 +        error_setg(errp, "The overlay does not support backing images");
          goto out;
      }
 diff --git a/tests/qemu-iotests/085.out b/tests/qemu-iotests/085.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/085.out
 +++ b/tests/qemu-iotests/085.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
  === Invalid command - cannot create a snapshot using a file BDS ===
 -{"error": {"class": "GenericError", "desc": "The snapshot does not support backing images"}}
 +{"error": {"class": "GenericError", "desc": "The overlay does not support backing images"}}
  === Invalid command - snapshot node used as active layer ===
 -{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
 -{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
 -{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
 +{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
 +{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
 +{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
  === Invalid command - snapshot node used as backing hd ===
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
  Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base
  {"return": {}}
 -{"error": {"class": "GenericError", "desc": "The snapshot already has a backing image"}}
 +{"error": {"class": "GenericError", "desc": "The overlay already has a backing image"}}
  === Invalid command - The node does not exist ===
 --
 .21.0

-[Qemu-devel] [PULL 10/20] iotests: restrict 254 to support only qcow2
+[Qemu-devel] [PULL 02/15] block: fix permission update in bdrv_replace_node
 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Test fails at least for qcow, because of different cluster sizes in
+It's wrong to OR shared permissions. It may lead to crash on further
-base and top (and therefore different granularities of bitmaps we are
+permission updates.
-trying to merge).
+Also, no needs to consider previously calculated permissions, as at
 this point we already bind all new parents and bdrv_get_cumulative_perm
 result is enough. So fix the bug by just set permissions by
 bdrv_get_cumulative_perm result.
-The test aim is to check block-dirty-bitmap-merge between different
+Bug was introduced in long ago 234ac1a9025, in 2.9.
 nodes functionality, no needs to check all formats. So, let's just drop
 support for anything except qcow2.
-Reported-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20190605155405.104384-1-vsementsov@virtuozzo.com
+Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/254 | 2 ++
+ block.c | 5 ++---
-file changed, 2 insertions(+)
+file changed, 2 insertions(+), 3 deletions(-)
-diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
+diff --git a/block.c b/block.c
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/254
+--- a/block.c
-+++ b/tests/qemu-iotests/254
++++ b/block.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
- import iotests
+ {
- from iotests import qemu_img_create, file_path, log
+     BdrvChild *c, *next;
+     GSList *list = NULL, *p;
-+iotests.verify_image_format(supported_fmts=['qcow2'])
+-    uint64_t old_perm, old_shared;
-+
+     uint64_t perm = 0, shared = BLK_PERM_ALL;
- disk, top = file_path('disk', 'top')
+     int ret;
- size = 1024 * 1024
+@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
          bdrv_unref(from);
      }
 -    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
 -    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
 +    bdrv_get_cumulative_perm(to, &perm, &shared);
 +    bdrv_set_perm(to, perm, shared);
  out:
      g_slist_free(list);
 --
 .21.0

-[Qemu-devel] [PULL 07/20] iotests: Filter 175's allocation information
+[Qemu-devel] [PULL 03/15] block: posix: Always allocate the first block
-It is possible for an empty file to take up blocks on a filesystem, for
+From: Nir Soffer <nirsof@gmail.com>
-example:
+When creating an image with preallocation "off" or "falloc", the first
-$ qemu-img create -f raw test.img 1G
+block of the image is typically not allocated. When using Gluster
-Formatting 'test.img', fmt=raw size=1073741824
+storage backed by XFS filesystem, reading this block using direct I/O
-$ mkfs.ext4 -I 128 -q test.img
+succeeds regardless of request length, fooling alignment detection.
-$ mkdir test-mount
-$ sudo mount -o loop test.img test-mount
+In this case we fallback to a safe value (4096) instead of the optimal
-$ sudo touch test-mount/test-file
+value (512), which may lead to unneeded data copying when aligning
-$ stat -c 'blocks=%b' test-mount/test-file
+requests.  Allocating the first block avoids the fallback.
-blocks=8
+Since we allocate the first block even with preallocation=off, we no
-These extra blocks (one cluster) are apparently used for metadata,
+longer create images with zero disk size:
-because they are always there, on top of blocks used for data:
+    $ ./qemu-img create -f raw test.raw 1g
-$ sudo dd if=/dev/zero of=test-mount/test-file bs=1M count=1
+    Formatting 'test.raw', fmt=raw size=1073741824
-+0 records in
-+0 records out
+    $ ls -lhs test.raw
-1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00135339 s, 775 MB/s
+.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
-$ stat -c 'blocks=%b' test-mount/test-file
-blocks=2056
+And converting the image requires additional cluster:
-Make iotest 175 take this into account.
+    $ ./qemu-img measure -f raw -O qcow2 test.raw
+    required size: 458752
-Reported-by: Thomas Huth <thuth@redhat.com>
+    fully allocated size: 1074135040
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
+When using format like vmdk with multiple files per image, we allocate
-Reviewed-by: Nir Soffer <nsoffer@redhat.com>
+one block per file:
-Message-id: 20190516144319.12570-1-mreitz@redhat.com
     $ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
     Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat
     $ ls -lhs test*.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk
 I did quick performance test for copying disks with qemu-img convert to
 new raw target image to Gluster storage with sector size of 512 bytes:
     for i in $(seq 10); do
         rm -f dst.raw
         sleep 10
         time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
     done
 Here is a table comparing the total time spent:
 Type    Before(s)   After(s)    Diff(%)
 ---------------------------------------
 real      530.028    469.123      -11.4
 user       17.204     10.768      -37.4
 sys        17.881      7.011      -60.7
 We can see very clear improvement in CPU usage.
 Signed-off-by: Nir Soffer <nsoffer@redhat.com>
 Message-id: 20190827010528.8818-2-nsoffer@redhat.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/175     | 26 ++++++++++++++++++++++----
+ block/file-posix.c                            | 51 +++++++++++++++++++
- tests/qemu-iotests/175.out |  8 ++++----
+ tests/qemu-iotests/059.out                    |  2 +-
-files changed, 26 insertions(+), 8 deletions(-)
+ tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
+ tests/qemu-iotests/150.out.raw                | 12 +++++
  tests/qemu-iotests/175                        | 19 ++++---
  tests/qemu-iotests/175.out                    |  8 +--
  tests/qemu-iotests/178.out.qcow2              |  4 +-
  tests/qemu-iotests/221.out                    | 12 +++--
  tests/qemu-iotests/253.out                    | 12 +++--
 files changed, 99 insertions(+), 21 deletions(-)
  rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
  create mode 100644 tests/qemu-iotests/150.out.raw
 diff --git a/block/file-posix.c b/block/file-posix.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/file-posix.c
 +++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
      return ret;
  }
 +/*
 + * Help alignment probing by allocating the first block.
 + *
 + * When reading with direct I/O from unallocated area on Gluster backed by XFS,
 + * reading succeeds regardless of request length. In this case we fallback to
 + * safe alignment which is not optimal. Allocating the first block avoids this
 + * fallback.
 + *
 + * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
 + * request alignment, so we use safe values.
 + *
 + * Returns: 0 on success, -errno on failure. Since this is an optimization,
 + * caller may ignore failures.
 + */
 +static int allocate_first_block(int fd, size_t max_size)
 +{
 +    size_t write_size = (max_size < MAX_BLOCKSIZE)
 +        ? BDRV_SECTOR_SIZE
 +        : MAX_BLOCKSIZE;
 +    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
 +    void *buf;
 +    ssize_t n;
 +    int ret;
 +
 +    buf = qemu_memalign(max_align, write_size);
 +    memset(buf, 0, write_size);
 +
 +    do {
 +        n = pwrite(fd, buf, write_size, 0);
 +    } while (n == -1 && errno == EINTR);
 +
 +    ret = (n == -1) ? -errno : 0;
 +
 +    qemu_vfree(buf);
 +    return ret;
 +}
 +
  static int handle_aiocb_truncate(void *opaque)
  {
      RawPosixAIOData *aiocb = opaque;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
                  /* posix_fallocate() doesn't set errno. */
                  error_setg_errno(errp, -result,
                                   "Could not preallocate new data");
 +            } else if (current_length == 0) {
 +                /*
 +                 * posix_fallocate() uses fallocate() if the filesystem
 +                 * supports it, or fallback to manually writing zeroes. If
 +                 * fallocate() was used, unaligned reads from the fallocated
 +                 * area in raw_probe_alignment() will succeed, hence we need to
 +                 * allocate the first block.
 +                 *
 +                 * Optimize future alignment probing; ignore failures.
 +                 */
 +                allocate_first_block(fd, offset);
              }
          } else {
              result = 0;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
          if (ftruncate(fd, offset) != 0) {
              result = -errno;
              error_setg_errno(errp, -result, "Could not resize file");
 +        } else if (current_length == 0 && offset > current_length) {
 +            /* Optimize future alignment probing; ignore failures. */
 +            allocate_first_block(fd, offset);
          }
          return result;
      default:
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/059.out
 +++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
  image: TEST_DIR/t.vmdk
  file format: vmdk
  virtual size: 0.977 TiB (1073741824000 bytes)
 -disk size: 16 KiB
 +disk size: 1.97 MiB
  Format specific information:
      cid: XXXXXXXX
      parent cid: XXXXXXXX
 diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
 similarity index 100%
 rename from tests/qemu-iotests/150.out
 rename to tests/qemu-iotests/150.out.qcow2
 diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/tests/qemu-iotests/150.out.raw
@@ -XXX,XX +XXX,XX @@
 +QA output created by 150
 +
 +=== Mapping sparse conversion ===
 +
 +Offset          Length          File
 +0               0x1000          TEST_DIR/t.IMGFMT
 +
 +=== Mapping non-sparse conversion ===
 +
 +Offset          Length          File
 +0               0x100000        TEST_DIR/t.IMGFMT
 +*** done
 diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/175
 +++ b/tests/qemu-iotests/175
-@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+ # the file size.  This function hides the resulting difference in the
- _cleanup()
+ # stat -c '%b' output.
  # Parameter 1: Number of blocks an empty file occupies
 -# Parameter 2: Image size in bytes
 +# Parameter 2: Minimal number of blocks in an image
 +# Parameter 3: Image size in bytes
  _filter_blocks()
  {
--    _cleanup_test_img
+     extra_blocks=$1
-+    _cleanup_test_img
+-    img_size=$2
-+    rm -f "$TEST_DIR/empty"
++    min_blocks=$2
 +    img_size=$3
 -    sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
 -        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
 +    sed -e "s/blocks=$min_blocks\\(\$\\|[^0-9]\\)/min allocation/" \
 +        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
  }
- trap "_cleanup; exit \$status" 0 1 2 3 15
 +# Some file systems sometimes allocate extra blocks independently of
 +# the file size.  This function hides the resulting difference in the
 +# stat -c '%b' output.
 +# Parameter 1: Number of blocks an empty file occupies
 +# Parameter 2: Image size in bytes
 +_filter_blocks()
 +{
 +    extra_blocks=$1
 +    img_size=$2
 +
 +    sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
 +        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
 +}
 +
  # get standard environment, filters and checks
- . ./common.rc
+@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
- . ./common.filter
+ touch "$TEST_DIR/empty"
-@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
+ extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
- _supported_proto file
- _supported_os Linux
++# We always write the first byte; check how many blocks this filesystem
++# allocates to match empty image alloation.
--size=1m
++printf "\0" > "$TEST_DIR/empty"
-+size=$((1 * 1024 * 1024))
++min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 +
 +touch "$TEST_DIR/empty"
 +extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
  echo
  echo "== creating image with default preallocation =="
  _make_test_img $size | _filter_imgfmt
--stat -c "size=%s, blocks=%b" $TEST_IMG
+-stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
-+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
++stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  for mode in off full falloc; do
      echo
      echo "== creating image with preallocation $mode =="
      IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
--    stat -c "size=%s, blocks=%b" $TEST_IMG
+-    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
-+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
++    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  done
  # success, all done
 diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/175.out
 +++ b/tests/qemu-iotests/175.out
 @@ -XXX,XX +XXX,XX @@ QA output created by 175
  == creating image with default preallocation ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
--size=1048576, blocks=0
+-size=1048576, nothing allocated
-+size=1048576, nothing allocated
++size=1048576, min allocation
  == creating image with preallocation off ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
--size=1048576, blocks=0
+-size=1048576, nothing allocated
-+size=1048576, nothing allocated
++size=1048576, min allocation
  == creating image with preallocation full ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
--size=1048576, blocks=2048
+-size=1048576, everything allocated
-+size=1048576, everything allocated
++size=1048576, max allocation
  == creating image with preallocation falloc ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
--size=1048576, blocks=2048
+-size=1048576, everything allocated
-+size=1048576, everything allocated
++size=1048576, max allocation
   *** done
+diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
+index XXXXXXX..XXXXXXX 100644
+--- a/tests/qemu-iotests/178.out.qcow2
++++ b/tests/qemu-iotests/178.out.qcow2
+@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
+ == raw input image with data (human) ==
+ Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
+-required size: 393216
++required size: 458752
+ fully allocated size: 1074135040
+ wrote 512/512 bytes at offset 512
+bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
+ Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
+ {
+-    "required": 393216,
++    "required": 458752,
+     "fully-allocated": 1074135040
+ }
+ wrote 512/512 bytes at offset 512
+diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
+index XXXXXXX..XXXXXXX 100644
+--- a/tests/qemu-iotests/221.out
++++ b/tests/qemu-iotests/221.out
+@@ -XXX,XX +XXX,XX @@ QA output created by 221
+ === Check mapping of unaligned raw image ===
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
+-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+ wrote 1/1 bytes at offset 65536
+bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+ { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+ { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+ { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+ { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+ *** done
+diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
+index XXXXXXX..XXXXXXX 100644
+--- a/tests/qemu-iotests/253.out
++++ b/tests/qemu-iotests/253.out
+@@ -XXX,XX +XXX,XX @@ QA output created by 253
+ === Check mapping of unaligned raw image ===
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
+-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+ wrote 65535/65535 bytes at offset 983040
+.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+ { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
++[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
++{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+ { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+ *** done
 --
 .21.0

-[Qemu-devel] [PULL 18/20] blkdebug: Inject errors on .bdrv_co_block_status()
+[Qemu-devel] [PULL 04/15] iotests: Test allocate_first_block() with O_DIRECT
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+From: Nir Soffer <nirsof@gmail.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Using block_resize we can test allocate_first_block() with file
-Message-id: 20190507203508.18026-6-mreitz@redhat.com
+descriptor opened with O_DIRECT, ensuring that it works for any size
 larger than 4096 bytes.
 Testing smaller sizes is tricky as the result depends on the filesystem
 used for testing. For example on NFS any size will work since O_DIRECT
 does not require any alignment.
 Signed-off-by: Nir Soffer <nsoffer@redhat.com>
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Message-id: 20190827010528.8818-3-nsoffer@redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json | 5 ++++-
+ tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
- block/blkdebug.c     | 8 ++++++++
+ tests/qemu-iotests/175.out |  8 ++++++++
-files changed, 12 insertions(+), 1 deletion(-)
+files changed, 36 insertions(+)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
+diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/175
 +++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ _filter_blocks()
          -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
  }
 +# Resize image using block_resize.
 +# Parameter 1: image path
 +# Parameter 2: new size
 +_block_resize()
 +{
 +    local path=$1
 +    local size=$2
 +
 +    $QEMU -qmp stdio -nographic -nodefaults \
 +        -blockdev file,node-name=file,filename=$path,cache.direct=on \
 +        <<EOF
 +{'execute': 'qmp_capabilities'}
 +{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
 +{'execute': 'quit'}
 +EOF
 +}
 +
  # get standard environment, filters and checks
  . ./common.rc
  . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
  _supported_proto file
  _supported_os Linux
 +_default_cache_mode none
 +_supported_cache_modes none directsync
 +
  size=$((1 * 1024 * 1024))
  touch "$TEST_DIR/empty"
@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
      stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  done
 +for new_size in 4096 1048576; do
 +    echo
 +    echo "== resize empty image with block_resize =="
 +    _make_test_img 0 | _filter_imgfmt
 +    _block_resize $TEST_IMG $new_size >/dev/null
 +    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
 +done
 +
  # success, all done
  echo "*** done"
  rm -f $seq.full
 diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
 index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
+--- a/tests/qemu-iotests/175.out
-+++ b/qapi/block-core.json
++++ b/tests/qemu-iotests/175.out
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
- #
+ == creating image with preallocation falloc ==
- # @flush: .bdrv_co_flush_to_disk()
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
- #
+ size=1048576, max allocation
 +# @block-status: .bdrv_co_block_status()
 +#
  # Since: 4.1
  ##
  { 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
 -  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
 +  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush',
 +            'block-status' ] }
  ##
  # @BlkdebugInjectErrorOptions:
 diff --git a/block/blkdebug.c b/block/blkdebug.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/blkdebug.c
 +++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,
                                                   int64_t *map,
                                                   BlockDriverState **file)
  {
 +    int err;
 +
-     assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment));
++== resize empty image with block_resize ==
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
 +size=4096, min allocation
 +
-+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_BLOCK_STATUS);
++== resize empty image with block_resize ==
-+    if (err) {
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
-+        return err;
++size=1048576, min allocation
-+    }
+  *** done
 +
      return bdrv_co_block_status_from_file(bs, want_zero, offset, bytes,
                                            pnum, map, file);
  }
 --
 .21.0

-[Qemu-devel] [PULL 11/20] qemu-img: Fix options leakage in img_rebase()
+[Qemu-devel] [PULL 05/15] iotests: Fix _filter_img_create()
-img_rebase() can leak a QDict in two occasions.  Fix it.
+fe646693acc changed qemu-img create's output so that it no longer prints
 single quotes around parameter values.  The subformat and adapter_type
 filters in _filter_img_create() have never been adapted to that change.
-Coverity: CID 1401416
+Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
 Fixes: d16699b64671466b42079c45b89127aeea1ca565
 Fixes: 330c72957196e0ae382abcaa97ebf4eb9bc8574f
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20190528195338.12376-1-mreitz@redhat.com
+Reviewed-by: John Snow <jsnow@redhat.com>
 Message-id: 20190815153638.4600-2-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qemu-img.c | 3 +++
+ tests/qemu-iotests/059.out       | 16 ++++++++--------
-file changed, 3 insertions(+)
+ tests/qemu-iotests/common.filter |  4 ++--
 files changed, 10 insertions(+), 10 deletions(-)
-diff --git a/qemu-img.c b/qemu-img.c
+diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
---- a/qemu-img.c
+--- a/tests/qemu-iotests/059.out
-+++ b/qemu-img.c
++++ b/tests/qemu-iotests/059.out
-@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
+@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-                                                              out_baseimg,
+ qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
-                                                              &local_err);
-             if (local_err) {
+ === Testing monolithicFlat creation and opening ===
-+                qobject_unref(options);
+-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
-                 error_reportf_err(local_err,
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
-                                   "Could not resolve backing filename: ");
+ image: TEST_DIR/t.IMGFMT
-                 ret = -1;
+ file format: IMGFMT
-@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
+ virtual size: 2 GiB (2147483648 bytes)
-              */
-             prefix_chain_bs = bdrv_find_backing_image(bs, out_real_path);
+ === Testing monolithicFlat with zeroed_grain ===
-             if (prefix_chain_bs) {
+ qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
-+                qobject_unref(options);
+-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
-                 g_free(out_real_path);
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
-+
-                 blk_new_backing = blk_new(qemu_get_aio_context(),
+ === Testing big twoGbMaxExtentFlat ===
-                                           BLK_PERM_CONSISTENT_READ,
+-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
-                                           BLK_PERM_ALL);
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
  image: TEST_DIR/t.vmdk
  file format: vmdk
  virtual size: 0.977 TiB (1073741824000 bytes)
@@ -XXX,XX +XXX,XX @@ Format specific information:
  qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
  === Testing truncated sparse ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
  qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
  === Converting to streamOptimized from image with small cluster size===
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing monolithicFlat with internally generated JSON file name ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
  qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
  === Testing version 3 ===
@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing 4TB monolithicFlat creation and IO ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
  image: TEST_DIR/t.IMGFMT
  file format: IMGFMT
  virtual size: 4 TiB (4398046511104 bytes)
@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing qemu-img map on extents ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0x20000         0x3f0000        TEST_DIR/t.vmdk
 x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
 diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/common.filter
 +++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
          -e "s# compat6=\\(on\\|off\\)##g" \
          -e "s# static=\\(on\\|off\\)##g" \
          -e "s# zeroed_grain=\\(on\\|off\\)##g" \
 -        -e "s# subformat='[^']*'##g" \
 -        -e "s# adapter_type='[^']*'##g" \
 +        -e "s# subformat=[^ ]*##g" \
 +        -e "s# adapter_type=[^ ]*##g" \
          -e "s# hwversion=[^ ]*##g" \
          -e "s# lazy_refcounts=\\(on\\|off\\)##g" \
          -e "s# block_size=[0-9]\\+##g" \
 --
 .21.0

-[Qemu-devel] [PULL 15/20] qemu-img: Add salvaging mode to convert
+[Qemu-devel] [PULL 06/15] vmdk: Use bdrv_dirname() for relative extent paths
-This adds a salvaging mode (--salvage) to qemu-img convert which ignores
+This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
-read errors and treats the respective areas as containing only zeroes.
+turns a former error in 059 into success.
 This can be used for instance to at least partially recover the data
 from terminally corrupted qcow2 images.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Message-id: 20190815153638.4600-3-mreitz@redhat.com
-Message-id: 20190507203508.18026-3-mreitz@redhat.com
+Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qemu-img.c       | 90 +++++++++++++++++++++++++++++++++++++-----------
+ block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
- qemu-img-cmds.hx |  4 +--
+ tests/qemu-iotests/059     |  7 +++--
- qemu-img.texi    |  4 +++
+ tests/qemu-iotests/059.out |  4 ++-
-files changed, 75 insertions(+), 23 deletions(-)
+files changed, 42 insertions(+), 23 deletions(-)
-diff --git a/qemu-img.c b/qemu-img.c
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/qemu-img.c
+--- a/block/vmdk.c
-+++ b/qemu-img.c
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ enum {
+@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
-     OPTION_SIZE = 264,
+ }
-     OPTION_PREALLOCATION = 265,
-     OPTION_SHRINK = 266,
+ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-+    OPTION_SALVAGE = 267,
+-                              const char *desc_file_path, QDict *options,
- };
+-                              Error **errp)
++                              QDict *options, Error **errp)
- typedef enum OutputFormat {
+ {
-@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
+     int ret;
-     int64_t target_backing_sectors; /* negative if unknown */
+     int matches;
-     bool wr_in_order;
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-     bool copy_range;
+     const char *p, *np;
-+    bool salvage;
+     int64_t sectors = 0;
-     bool quiet;
+     int64_t flat_offset;
-     int min_sparse;
++    char *desc_file_dir = NULL;
-     int alignment;
+     char *extent_path;
-@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
+     BdrvChild *extent_file;
-     }
+     BDRVVmdkState *s = bs->opaque;
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-     if (s->sector_next_status <= sector_num) {
+             continue;
--        int64_t count = n * BDRV_SECTOR_SIZE;
+         }
-+        uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
-+        int64_t count;
+-        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
+-            !desc_file_path[0])
--        if (s->target_has_backing) {
+-        {
-+        do {
+-            bdrv_refresh_filename(bs->file->bs);
-+            count = n * BDRV_SECTOR_SIZE;
+-            error_setg(errp, "Cannot use relative extent paths with VMDK "
-+
+-                       "descriptor file '%s'", bs->file->bs->filename);
-+            if (s->target_has_backing) {
+-            return -EINVAL;
-+                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
+-        }
-+                                        count, &count, NULL, NULL);
++        if (path_is_absolute(fname)) {
-+            } else {
++            extent_path = g_strdup(fname);
-+                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
++        } else {
-+                                              offset, count, &count, NULL,
++            if (!desc_file_dir) {
-+                                              NULL);
++                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
-+            }
++                if (!desc_file_dir) {
-+
++                    bdrv_refresh_filename(bs->file->bs);
-+            if (ret < 0) {
++                    error_prepend(errp, "Cannot use relative paths with VMDK "
-+                if (s->salvage) {
++                                  "descriptor file '%s': ",
-+                    if (n == 1) {
++                                  bs->file->bs->filename);
-+                        if (!s->quiet) {
++                    ret = -EINVAL;
-+                            warn_report("error while reading block status at "
++                    goto out;
 +                                        "offset %" PRIu64 ": %s", offset,
 +                                        strerror(-ret));
 +                        }
 +                        /* Just try to read the data, then */
 +                        ret = BDRV_BLOCK_DATA;
 +                        count = BDRV_SECTOR_SIZE;
 +                    } else {
 +                        /* Retry on a shorter range */
 +                        n = DIV_ROUND_UP(n, 4);
 +                    }
 +                } else {
 +                    error_report("error while reading block status at offset "
 +                                 "%" PRIu64 ": %s", offset, strerror(-ret));
 +                    return ret;
 +                }
 +            }
-+        } while (ret < 0);
+-        extent_path = path_combine(desc_file_path, fname);
--            ret = bdrv_block_status(blk_bs(s->src[src_cur]),
++            extent_path = g_strconcat(desc_file_dir, fname, NULL);
--                                    (sector_num - src_cur_offset) *
++        }
--                                    BDRV_SECTOR_SIZE,
--                                    count, &count, NULL, NULL);
+         ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
--        } else {
+         assert(ret < 32);
--            ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
--                                          (sector_num - src_cur_offset) *
+         g_free(extent_path);
--                                          BDRV_SECTOR_SIZE,
+         if (local_err) {
--                                          count, &count, NULL, NULL);
+             error_propagate(errp, local_err);
--        }
+-            return -EINVAL;
--        if (ret < 0) {
++            ret = -EINVAL;
--            error_report("error while reading block status of sector %" PRId64
++            goto out;
--                         ": %s", sector_num, strerror(-ret));
+         }
--            return ret;
--        }
+         /* save to extents array */
-         n = DIV_ROUND_UP(count, BDRV_SECTOR_SIZE);
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
+, 0, 0, 0, 0, &extent, errp);
-         if (ret & BDRV_BLOCK_ZERO) {
+             if (ret < 0) {
-@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
+                 bdrv_unref_child(bs, extent_file);
- static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
+-                return ret;
-                                         int nb_sectors, uint8_t *buf)
++                goto out;
- {
+             }
-+    uint64_t single_read_until = 0;
+             extent->flat_start_offset = flat_offset << 9;
-     int n, ret;
+         } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-     assert(nb_sectors <= s->buf_sectors);
+             g_free(buf);
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
+             if (ret) {
-         BlockBackend *blk;
+                 bdrv_unref_child(bs, extent_file);
-         int src_cur;
+-                return ret;
-         int64_t bs_sectors, src_cur_offset;
++                goto out;
-+        uint64_t offset;
+             }
+             extent = &s->extents[s->num_extents - 1];
-         /* In the case of compression with multiple source files, we can get a
+         } else if (!strcmp(type, "SESPARSE")) {
-          * nb_sectors that spreads into the next part. So we must be able to
+             ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
+             if (ret) {
-         blk = s->src[src_cur];
+                 bdrv_unref_child(bs, extent_file);
-         bs_sectors = s->src_sectors[src_cur];
+-                return ret;
++                goto out;
-+        offset = (sector_num - src_cur_offset) << BDRV_SECTOR_BITS;
+             }
              extent = &s->extents[s->num_extents - 1];
          } else {
              error_setg(errp, "Unsupported extent type '%s'", type);
              bdrv_unref_child(bs, extent_file);
 -            return -ENOTSUP;
 +            ret = -ENOTSUP;
 +            goto out;
          }
          extent->type = g_strdup(type);
      }
 -    return 0;
 +
-         n = MIN(nb_sectors, bs_sectors - (sector_num - src_cur_offset));
++    ret = 0;
-+        if (single_read_until > offset) {
++    goto out;
-+            n = 1;
-+        }
+ invalid:
+     np = next_line(p);
--        ret = blk_co_pread(
+@@ -XXX,XX +XXX,XX @@ invalid:
--                blk, (sector_num - src_cur_offset) << BDRV_SECTOR_BITS,
+         np--;
 -                n << BDRV_SECTOR_BITS, buf, 0);
 +        ret = blk_co_pread(blk, offset, n << BDRV_SECTOR_BITS, buf, 0);
          if (ret < 0) {
 -            return ret;
 +            if (s->salvage) {
 +                if (n > 1) {
 +                    single_read_until = offset + (n << BDRV_SECTOR_BITS);
 +                    continue;
 +                } else {
 +                    if (!s->quiet) {
 +                        warn_report("error while reading offset %" PRIu64
 +                                    ": %s", offset, strerror(-ret));
 +                    }
 +                    memset(buf, 0, BDRV_SECTOR_SIZE);
 +                }
 +            } else {
 +                return ret;
 +            }
          }
          sector_num += n;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
              {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
              {"force-share", no_argument, 0, 'U'},
              {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
 +            {"salvage", no_argument, 0, OPTION_SALVAGE},
              {0, 0, 0, 0}
          };
          c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
          case OPTION_IMAGE_OPTS:
              image_opts = true;
              break;
 +        case OPTION_SALVAGE:
 +            s.salvage = true;
 +            break;
          case OPTION_TARGET_IMAGE_OPTS:
              tgt_image_opts = true;
              break;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
          goto fail_getopt;
      }
+     error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
-+    if (s.copy_range && s.salvage) {
+-    return -EINVAL;
-+        error_report("Cannot use copy offloading in salvaging mode");
++    ret = -EINVAL;
 +        goto fail_getopt;
 +    }
 +
-     if (tgt_image_opts && !skip_create) {
++out:
-         error_report("--target-image-opts requires use of -n flag");
++    g_free(desc_file_dir);
-         goto fail_getopt;
++    return ret;
-diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
+ }
  static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
      }
      s->create_type = g_strdup(ct);
      s->desc_offset = 0;
 -    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
 -                             errp);
 +    ret = vmdk_parse_extents(buf, bs, options, errp);
  exit:
      return ret;
  }
 diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/059
 +++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
  echo
  echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 +# Should work, because bdrv_dirname() works fine with blkdebug
  IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 -$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
 -    | _filter_testdir | _filter_imgfmt
 +$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
 +         -c info \
 +    2>&1 \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
  echo
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
---- a/qemu-img-cmds.hx
+--- a/tests/qemu-iotests/059.out
-+++ b/qemu-img-cmds.hx
++++ b/tests/qemu-iotests/059.out
-@@ -XXX,XX +XXX,XX @@ STEXI
+@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
- ETEXI
+ === Testing monolithicFlat with internally generated JSON file name ===
- DEF("convert", img_convert,
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
--    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename")
+-qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
-+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
++format name: IMGFMT
- STEXI
++cluster size: 0 bytes
--@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename}
++vm state offset: 0 bytes
-+@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] [--salvage] @var{filename} [@var{filename2} [...]] @var{output_filename}
- ETEXI
+ === Testing version 3 ===
+ image: TEST_DIR/iotest-version3.IMGFMT
  DEF("create", img_create,
 diff --git a/qemu-img.texi b/qemu-img.texi
 index XXXXXXX..XXXXXXX 100644
 --- a/qemu-img.texi
 +++ b/qemu-img.texi
@@ -XXX,XX +XXX,XX @@ improve performance if the data is remote, such as with NFS or iSCSI backends,
  but will not automatically sparsify zero sectors, and may result in a fully
  allocated target image depending on the host support for getting allocation
  information.
 +@item --salvage
 +Try to ignore I/O errors when reading.  Unless in quiet mode (@code{-q}), errors
 +will still be printed.  Areas that cannot be read from the source will be
 +treated as containing only zeroes.
  @end table
  Parameters to dd subcommand:
 --
 .21.0

-[Qemu-devel] [PULL 17/20] blkdebug: Add "none" event
+[Qemu-devel] [PULL 07/15] iotests: Keep testing broken relative extent paths
-Together with @iotypes and @sector, this can be used to trap e.g. the
+We had a test for a case where relative extent paths did not work, but
-first read or write access to a certain sector without having to know
+unfortunately we just fixed the underlying problem, so it works now.
-what happens internally in the block layer, i.e. which "real" events
+This patch adds a new test case that still fails.
 happen right before such an access.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
-Message-id: 20190507203508.18026-5-mreitz@redhat.com
+Message-id: 20190815153638.4600-4-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json | 4 +++-
+ tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
- block/blkdebug.c     | 2 ++
+ tests/qemu-iotests/059.out |  4 ++++
-files changed, 5 insertions(+), 1 deletion(-)
+files changed, 31 insertions(+)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
+diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/059
 +++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
  echo
  echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 +
 +echo '--- blkdebug ---'
  # Should work, because bdrv_dirname() works fine with blkdebug
  IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
  $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
      | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
 +echo '--- quorum ---'
 +# Should not work, because bdrv_dirname() does not work with quorum
 +IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 +cp "$TEST_IMG" "$TEST_IMG.orig"
 +
 +filename="json:{
 +    \"driver\": \"$IMGFMT\",
 +    \"file\": {
 +        \"driver\": \"quorum\",
 +        \"children\": [ {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG\"
 +        }, {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG.orig\"
 +        } ],
 +        \"vote-threshold\": 1
 +    } }"
 +
 +filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
 +$QEMU_IMG info "$filename" 2>&1 \
 +    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
 +
 +
  echo
  echo "=== Testing version 3 ==="
  _use_sample_img iotest-version3.vmdk.bz2
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
+--- a/tests/qemu-iotests/059.out
-+++ b/qapi/block-core.json
++++ b/tests/qemu-iotests/059.out
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
- #
+bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
- # @cluster_alloc_space: an allocation of file space for a cluster (since 4.1)
- #
+ === Testing monolithicFlat with internally generated JSON file name ===
-+# @none: triggers once at creation of the blkdebug node (since 4.1)
++--- blkdebug ---
-+#
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
- # Since: 2.9
+ format name: IMGFMT
- ##
+ cluster size: 0 bytes
- { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
+ vm state offset: 0 bytes
-@@ -XXX,XX +XXX,XX @@
++--- quorum ---
-             'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev',
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-             'pwritev_zero', 'pwritev_done', 'empty_image_prepare',
++qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
-             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
--            'cor_write', 'cluster_alloc_space'] }
+ === Testing version 3 ===
-+            'cor_write', 'cluster_alloc_space', 'none'] }
+ image: TEST_DIR/iotest-version3.IMGFMT
  ##
  # @BlkdebugIOType:
 diff --git a/block/blkdebug.c b/block/blkdebug.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/blkdebug.c
 +++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ static int blkdebug_open(BlockDriverState *bs, QDict *options, int flags,
          goto out;
      }
 +    bdrv_debug_event(bs, BLKDBG_NONE);
 +
      ret = 0;
  out:
      if (ret < 0) {
 --
 .21.0

-[Qemu-devel] [PULL 16/20] blkdebug: Add @iotype error option
+[Qemu-devel] [PULL 08/15] vmdk: Reject invalid compressed writes
-This new error option allows users of blkdebug to inject errors only on
+Compressed writes generally have to write full clusters, not just in
-certain kinds of I/O operations.  Users usually want to make a very
+theory but also in practice when it comes to vmdk's streamOptimized
-specific operation fail, not just any; but right now they simply hope
+subformat.  It currently is just silently broken for writes with
-that the event that triggers the error injection is followed up with
+non-zero in-cluster offsets:
 that very operation.  That may not be true, however, because the block
 layer is changing (including blkdebug, which may increase the number of
 types of I/O operations on which to inject errors).
-The new option's default has been chosen to keep backwards
+$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
-compatibility.
+$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
 wrote 4096/4096 bytes at offset 4096
 KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
 read failed: Invalid argument
-Note that similar to the internal representation, we could choose to
+(The technical reason is that vmdk_write_extent() just writes the
-expose this option as a list of I/O types.  But there is no practical
+incomplete compressed data actually to offset 4k.  When reading the
-use for this, because as described above, users usually know exactly
+data, vmdk_read_extent() looks at offset 0 and finds the compressed data
-which kind of operation they want to make fail, so there is no need to
+size to be 0, because that is what it reads from there.  This yields an
-specify multiple I/O types at once.  In addition, exposing this option
+error.)
-as a list would require non-trivial changes to qemu_opts_absorb_qdict().
 For incomplete writes with zero in-cluster offsets, the error path when
 reading the rest of the cluster is a bit different, but the result is
 the same:
 $ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
 $ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
 wrote 4096/4096 bytes at offset 0
 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
 read failed: Invalid argument
 (Here, vmdk_read_extent() finds the data and then sees that the
 uncompressed data is short.)
 It is better to reject invalid writes than to make the user believe they
 might have succeeded and then fail when trying to read it back.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
-Message-id: 20190507203508.18026-4-mreitz@redhat.com
+Message-id: 20190815153638.4600-5-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json | 26 +++++++++++++++++++++++
+ block/vmdk.c | 10 ++++++++++
- block/blkdebug.c     | 50 ++++++++++++++++++++++++++++++++++++--------
+file changed, 10 insertions(+)
 files changed, 67 insertions(+), 9 deletions(-)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
+--- a/block/vmdk.c
-+++ b/qapi/block-core.json
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
-             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
+     if (extent->compressed) {
-             'cor_write', 'cluster_alloc_space'] }
+         void *compressed_data;
-+##
++        /* Only whole clusters */
-+# @BlkdebugIOType:
++        if (offset_in_cluster ||
-+#
++            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
-+# Kinds of I/O that blkdebug can inject errors in.
++            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
-+#
++             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
 +# @read: .bdrv_co_preadv()
 +#
 +# @write: .bdrv_co_pwritev()
 +#
 +# @write-zeroes: .bdrv_co_pwrite_zeroes()
 +#
 +# @discard: .bdrv_co_pdiscard()
 +#
 +# @flush: .bdrv_co_flush_to_disk()
 +#
 +# Since: 4.1
 +##
 +{ 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
 +  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
 +
  ##
  # @BlkdebugInjectErrorOptions:
  #
@@ -XXX,XX +XXX,XX @@
  # @state:       the state identifier blkdebug needs to be in to
  #               actually trigger the event; defaults to "any"
  #
 +# @iotype:      the type of I/O operations on which this error should
 +#               be injected; defaults to "all read, write,
 +#               write-zeroes, discard, and flush operations"
 +#               (since: 4.1)
 +#
  # @errno:       error identifier (errno) to be returned; defaults to
  #               EIO
  #
@@ -XXX,XX +XXX,XX @@
  { 'struct': 'BlkdebugInjectErrorOptions',
    'data': { 'event': 'BlkdebugEvent',
              '*state': 'int',
 +            '*iotype': 'BlkdebugIOType',
              '*errno': 'int',
              '*sector': 'int',
              '*once': 'bool',
 diff --git a/block/blkdebug.c b/block/blkdebug.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/blkdebug.c
 +++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
      int state;
      union {
          struct {
 +            uint64_t iotype_mask;
              int error;
              int immediately;
              int once;
@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
      QSIMPLEQ_ENTRY(BlkdebugRule) active_next;
  } BlkdebugRule;
 +QEMU_BUILD_BUG_MSG(BLKDEBUG_IO_TYPE__MAX > 64,
 +                   "BlkdebugIOType mask does not fit into an uint64_t");
 +
  static QemuOptsList inject_error_opts = {
      .name = "inject-error",
      .head = QTAILQ_HEAD_INITIALIZER(inject_error_opts.head),
@@ -XXX,XX +XXX,XX @@ static QemuOptsList inject_error_opts = {
              .name = "state",
              .type = QEMU_OPT_NUMBER,
          },
 +        {
-+            .name = "iotype",
++            ret = -EINVAL;
-+            .type = QEMU_OPT_STRING,
++            goto out;
 +        },
          {
              .name = "errno",
              .type = QEMU_OPT_NUMBER,
@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
      int event;
      struct BlkdebugRule *rule;
      int64_t sector;
 +    BlkdebugIOType iotype;
 +    Error *local_error = NULL;
      /* Find the right event for the rule */
      event_name = qemu_opt_get(opts, "event");
@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
          sector = qemu_opt_get_number(opts, "sector", -1);
          rule->options.inject.offset =
              sector == -1 ? -1 : sector * BDRV_SECTOR_SIZE;
 +
 +        iotype = qapi_enum_parse(&BlkdebugIOType_lookup,
 +                                 qemu_opt_get(opts, "iotype"),
 +                                 BLKDEBUG_IO_TYPE__MAX, &local_error);
 +        if (local_error) {
 +            error_propagate(errp, local_error);
 +            return -1;
 +        }
 +        if (iotype != BLKDEBUG_IO_TYPE__MAX) {
 +            rule->options.inject.iotype_mask = (1ull << iotype);
 +        } else {
 +            /* Apply the default */
 +            rule->options.inject.iotype_mask =
 +                (1ull << BLKDEBUG_IO_TYPE_READ)
 +                | (1ull << BLKDEBUG_IO_TYPE_WRITE)
 +                | (1ull << BLKDEBUG_IO_TYPE_WRITE_ZEROES)
 +                | (1ull << BLKDEBUG_IO_TYPE_DISCARD)
 +                | (1ull << BLKDEBUG_IO_TYPE_FLUSH);
 +        }
 +
-         break;
+         if (!extent->has_marker) {
+             ret = -EINVAL;
-     case ACTION_SET_STATE:
+             goto out;
@@ -XXX,XX +XXX,XX @@ out:
      return ret;
  }
 -static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
 +static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
 +                      BlkdebugIOType iotype)
  {
      BDRVBlkdebugState *s = bs->opaque;
      BlkdebugRule *rule = NULL;
@@ -XXX,XX +XXX,XX @@ static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
      QSIMPLEQ_FOREACH(rule, &s->active_rules, active_next) {
          uint64_t inject_offset = rule->options.inject.offset;
 -        if (inject_offset == -1 ||
 -            (bytes && inject_offset >= offset &&
 -             inject_offset < offset + bytes))
 +        if ((inject_offset == -1 ||
 +             (bytes && inject_offset >= offset &&
 +              inject_offset < offset + bytes)) &&
 +            (rule->options.inject.iotype_mask & (1ull << iotype)))
          {
              break;
          }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
          assert(bytes <= bs->bl.max_transfer);
      }
 -    err = rule_check(bs, offset, bytes);
 +    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_READ);
      if (err) {
          return err;
      }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
          assert(bytes <= bs->bl.max_transfer);
      }
 -    err = rule_check(bs, offset, bytes);
 +    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE);
      if (err) {
          return err;
      }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
  static int blkdebug_co_flush(BlockDriverState *bs)
  {
 -    int err = rule_check(bs, 0, 0);
 +    int err = rule_check(bs, 0, 0, BLKDEBUG_IO_TYPE_FLUSH);
      if (err) {
          return err;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pwrite_zeroes(BlockDriverState *bs,
          assert(bytes <= bs->bl.max_pwrite_zeroes);
      }
 -    err = rule_check(bs, offset, bytes);
 +    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE_ZEROES);
      if (err) {
          return err;
      }
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pdiscard(BlockDriverState *bs,
          assert(bytes <= bs->bl.max_pdiscard);
      }
 -    err = rule_check(bs, offset, bytes);
 +    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_DISCARD);
      if (err) {
          return err;
      }
 --
 .21.0

-[Qemu-devel] [PULL 14/20] qemu-img: Move quiet into ImgConvertState
+[Qemu-devel] [PULL 09/15] iotests: Disable broken streamOptimized tests
-Move img_convert()'s quiet flag into the ImgConvertState so it is
+streamOptimized does not support writes that do not span exactly one
-accessible by nested functions.  -q dictates that it suppresses anything
+cluster.  Furthermore, it cannot rewrite already allocated clusters.
-but errors, so if those functions want to emit warnings, they need to
+As such, many iotests do not work with it.  Disable them.
 query this flag first.  (There currently are no such warnings, but there
 will be as of the next patch.)
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
+Message-id: 20190815153638.4600-6-mreitz@redhat.com
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
 Message-id: 20190507203508.18026-2-mreitz@redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qemu-img.c | 13 +++++++------
+ tests/qemu-iotests/002 | 1 +
-file changed, 7 insertions(+), 6 deletions(-)
+ tests/qemu-iotests/003 | 1 +
  tests/qemu-iotests/005 | 3 ++-
  tests/qemu-iotests/009 | 1 +
  tests/qemu-iotests/010 | 1 +
  tests/qemu-iotests/011 | 1 +
  tests/qemu-iotests/017 | 3 ++-
  tests/qemu-iotests/018 | 3 ++-
  tests/qemu-iotests/019 | 3 ++-
  tests/qemu-iotests/020 | 3 ++-
  tests/qemu-iotests/027 | 1 +
  tests/qemu-iotests/032 | 1 +
  tests/qemu-iotests/033 | 1 +
  tests/qemu-iotests/034 | 3 ++-
  tests/qemu-iotests/037 | 3 ++-
  tests/qemu-iotests/063 | 3 ++-
  tests/qemu-iotests/072 | 1 +
  tests/qemu-iotests/105 | 3 ++-
  tests/qemu-iotests/197 | 1 +
  tests/qemu-iotests/215 | 1 +
  tests/qemu-iotests/251 | 1 +
 files changed, 30 insertions(+), 9 deletions(-)
-diff --git a/qemu-img.c b/qemu-img.c
+diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/qemu-img.c
+--- a/tests/qemu-iotests/002
-+++ b/qemu-img.c
++++ b/tests/qemu-iotests/002
-@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-     int64_t target_backing_sectors; /* negative if unknown */
-     bool wr_in_order;
+ _supported_fmt generic
-     bool copy_range;
+ _supported_proto generic
-+    bool quiet;
++_unsupported_imgopts "subformat=streamOptimized"
-     int min_sparse;
-     int alignment;
-     size_t cluster_sectors;
+ size=128M
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
+diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
-     QDict *open_opts = NULL;
+index XXXXXXX..XXXXXXX 100755
-     char *options = NULL;
+--- a/tests/qemu-iotests/003
-     Error *local_err = NULL;
++++ b/tests/qemu-iotests/003
--    bool writethrough, src_writethrough, quiet = false, image_opts = false,
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-+    bool writethrough, src_writethrough, image_opts = false,
-          skip_create = false, progress = false, tgt_image_opts = false;
+ _supported_fmt generic
-     int64_t ret = -EINVAL;
+ _supported_proto generic
-     bool force_share = false;
++_unsupported_imgopts "subformat=streamOptimized"
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-             src_cache = optarg;
+ size=128M
-             break;
+ offset=67M
-         case 'q':
+diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
--            quiet = true;
+index XXXXXXX..XXXXXXX 100755
-+            s.quiet = true;
+--- a/tests/qemu-iotests/005
-             break;
++++ b/tests/qemu-iotests/005
-         case 'n':
+@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
-             skip_create = true;
+ _supported_proto generic
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
+ _supported_os Linux
-     }
+ _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
+-                     "subformat=twoGbMaxExtentSparse"
-     /* Initialize before goto out */
++                     "subformat=twoGbMaxExtentSparse" \
--    if (quiet) {
++                     "subformat=streamOptimized"
-+    if (s.quiet) {
-         progress = false;
+ # vpc is limited to 127GB, so we can't test it here
-     }
+ if [ "$IMGFMT" = "vpc" ]; then
-     qemu_progress_init(progress, 1.0);
+diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
+index XXXXXXX..XXXXXXX 100755
+--- a/tests/qemu-iotests/009
-     for (bs_i = 0; bs_i < s.src_num; bs_i++) {
++++ b/tests/qemu-iotests/009
-         s.src[bs_i] = img_open(image_opts, argv[optind + bs_i],
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
--                               fmt, src_flags, src_writethrough, quiet,
-+                               fmt, src_flags, src_writethrough, s.quiet,
+ _supported_fmt generic
-                                force_share);
+ _supported_proto generic
-         if (!s.src[bs_i]) {
++_unsupported_imgopts "subformat=streamOptimized"
-             ret = -1;
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
+ size=6G
-     if (skip_create) {
+diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
-         s.target = img_open(tgt_image_opts, out_filename, out_fmt,
+index XXXXXXX..XXXXXXX 100755
--                            flags, writethrough, quiet, false);
+--- a/tests/qemu-iotests/010
-+                            flags, writethrough, s.quiet, false);
++++ b/tests/qemu-iotests/010
-     } else {
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-         /* TODO ultimately we should allow --target-image-opts
-          * to be used even when -n is not given.
+ _supported_fmt generic
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
+ _supported_proto generic
-          * to allow filenames in option syntax
++_unsupported_imgopts "subformat=streamOptimized"
-          */
-         s.target = img_open_file(out_filename, open_opts, out_fmt,
--                                 flags, writethrough, quiet, false);
+ size=6G
-+                                 flags, writethrough, s.quiet, false);
+diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
-         open_opts = NULL; /* blk_new_open will have freed it */
+index XXXXXXX..XXXXXXX 100755
-     }
+--- a/tests/qemu-iotests/011
-     if (!s.target) {
++++ b/tests/qemu-iotests/011
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=6G
 diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/017
 +++ b/tests/qemu-iotests/017
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 vmdk qed
  _supported_proto generic
  _unsupported_proto vxhs
 -_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
 +_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
 diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/018
 +++ b/tests/qemu-iotests/018
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _supported_os Linux
 -_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
 +_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
 +                     "streamOptimized"
  TEST_OFFSETS="0 4294967296"
 diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/019
 +++ b/tests/qemu-iotests/019
@@ -XXX,XX +XXX,XX @@ _supported_proto file
  _supported_os Linux
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
  CLUSTER_SIZE=65536
 diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/020
 +++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
 diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/027
 +++ b/tests/qemu-iotests/027
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt vmdk qcow qcow2 qed
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=128M
 diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/032
 +++ b/tests/qemu-iotests/032
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  # This works for any image format (though unlikely to segfault for raw)
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo === Prepare image ===
 diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/033
 +++ b/tests/qemu-iotests/033
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=128M
 diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/034
 +++ b/tests/qemu-iotests/034
@@ -XXX,XX +XXX,XX @@ _supported_proto file
  _supported_os Linux
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  CLUSTER_SIZE=4k
  size=128M
 diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/037
 +++ b/tests/qemu-iotests/037
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  CLUSTER_SIZE=4k
  size=128M
 diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/063
 +++ b/tests/qemu-iotests/063
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  _make_test_img 4M
 diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/072
 +++ b/tests/qemu-iotests/072
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
  _supported_proto file
 +_unsupported_imgopts "subformat=streamOptimized"
  IMG_SIZE=64M
 diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/105
 +++ b/tests/qemu-iotests/105
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2 vmdk vhdx qed
  _supported_proto generic
  _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  echo
  echo "creating large image"
 diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/197
 +++ b/tests/qemu-iotests/197
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto generic
  # LUKS support may be possible, but it complicates things.
  _unsupported_fmt luks
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo '=== Copy-on-read ==='
 diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/215
 +++ b/tests/qemu-iotests/215
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto generic
  # LUKS support may be possible, but it complicates things.
  _unsupported_fmt luks
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo '=== Copy-on-read ==='
 diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/251
 +++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto file
  _supported_os Linux
 +_unsupported_imgopts "subformat=streamOptimized"
  if [ "$IMGOPTSSYNTAX" = "true" ]; then
      # We use json:{} filenames here, so we cannot work with additional options.
 --
 .21.0

-[Qemu-devel] [PULL 12/20] qapi/block-core: Overlays are not snapshots
+[Qemu-devel] [PULL 10/15] iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
-A snapshot is something that reflects the state of something at a
+The error message for the test case where we have a quorum node for
-certain point in time.  It does not change.
+which no directory name can be generated is different: For
+twoGbMaxExtentSparse, it complains that it cannot open the extent file.
-The file our snapshot commands create (or the node they install) is not
+For other (sub)formats, it just notes that it cannot determine the
-a snapshot, as it does change over time.  It is an overlay.  We cannot
+backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
-do anything about the parameter names, but we can at least adjust the
+for simplicity's sake.
 descriptions to reflect that fact.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Message-id: 20190603202236.1342-2-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
-Reviewed-by: Alberto Garcia <berto@igalia.com>
+Message-id: 20190815153638.4600-7-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json | 20 ++++++++++----------
+ tests/qemu-iotests/110 | 3 ++-
-file changed, 10 insertions(+), 10 deletions(-)
+file changed, 2 insertions(+), 1 deletion(-)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
+diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/qapi/block-core.json
+--- a/tests/qemu-iotests/110
-+++ b/qapi/block-core.json
++++ b/tests/qemu-iotests/110
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
- #
+ # Any format supporting backing files
- # Either @device or @node-name must be set but not both.
+ _supported_fmt qed qcow qcow2 vmdk
- #
+ _supported_proto file
--# @device: the name of the device to generate the snapshot from.
+-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
-+# @device: the name of the device to take a snapshot of.
++_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
- #
++                     "subformat=twoGbMaxExtentSparse"
- # @node-name: graph node name to generate the snapshot from (Since 2.0)
- #
+ TEST_IMG_REL=$(basename "$TEST_IMG")
--# @snapshot-file: the target of the new image. If the file exists, or
 -# if it is a device, the snapshot will be created in the existing
 -# file/device. Otherwise, a new file will be created.
 +# @snapshot-file: the target of the new overlay image. If the file
 +# exists, or if it is a device, the overlay will be created in the
 +# existing file/device. Otherwise, a new file will be created.
  #
  # @snapshot-node-name: the graph node name of the new image (Since 2.0)
  #
 -# @format: the format of the snapshot image, default is 'qcow2'.
 +# @format: the format of the overlay image, default is 'qcow2'.
  #
  # @mode: whether and how QEMU should create a new image, default is
  #        'absolute-paths'.
@@ -XXX,XX +XXX,XX @@
  ##
  # @BlockdevSnapshot:
  #
 -# @node: device or node name that will have a snapshot created.
 +# @node: device or node name that will have a snapshot taken.
  #
  # @overlay: reference to the existing block device that will become
 -#           the overlay of @node, as part of creating the snapshot.
 +#           the overlay of @node, as part of taking the snapshot.
  #           It must not have a current backing file (this can be
  #           achieved by passing "backing": null to blockdev-add).
  #
@@ -XXX,XX +XXX,XX @@
  ##
  # @blockdev-snapshot-sync:
  #
 -# Generates a synchronous snapshot of a block device.
 +# Takes a synchronous snapshot of a block device.
  #
  # For the arguments, see the documentation of BlockdevSnapshotSync.
  #
@@ -XXX,XX +XXX,XX @@
  ##
  # @blockdev-snapshot:
  #
 -# Generates a snapshot of a block device.
 +# Takes a snapshot of a block device.
  #
 -# Create a snapshot, by installing 'node' as the backing image of
 +# Take a snapshot, by installing 'node' as the backing image of
  # 'overlay'. Additionally, if 'node' is associated with a block
  # device, the block device changes to using 'overlay' as its new active
  # image.
 --
 .21.0

-[Qemu-devel] [PULL 09/20] hw/block/fdc: floppy command FIFO memory initialization
+[Qemu-devel] [PULL 11/15] iotests: Disable 126 for flat vmdk subformats
-From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
+iotest 126 requires backing file support, which flat vmdks cannot offer.
 Skip this test for such subformats.
-The uninitialized memory allocated for the command FIFO of the
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-floppy controller during the VM hardware initialization incurs
+Message-id: 20190815153638.4600-8-mreitz@redhat.com
 many unwanted reports by Valgrind when VM state is being saved.
 That verbosity hardens a search for the real memory issues when
 the iotests run. Particularly, the patch eliminates 20 unnecessary
 reports of the Valgrind tool in the iotest #169.
 Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
 Message-id: 1559154027-282547-1-git-send-email-andrey.shinkevich@virtuozzo.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- hw/block/fdc.c | 1 +
+ tests/qemu-iotests/126 | 2 ++
-file changed, 1 insertion(+)
+file changed, 2 insertions(+)
-diff --git a/hw/block/fdc.c b/hw/block/fdc.c
+diff --git a/tests/qemu-iotests/126 b/tests/qemu-iotests/126
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/hw/block/fdc.c
+--- a/tests/qemu-iotests/126
-+++ b/hw/block/fdc.c
++++ b/tests/qemu-iotests/126
-@@ -XXX,XX +XXX,XX @@ static void fdctrl_realize_common(DeviceState *dev, FDCtrl *fdctrl,
+@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
-     FLOPPY_DPRINTF("init controller\n");
+ # Needs backing file support
-     fdctrl->fifo = qemu_memalign(512, FD_SECTOR_LEN);
+ _supported_fmt qcow qcow2 qed vmdk
-+    memset(fdctrl->fifo, 0, FD_SECTOR_LEN);
++_unsupported_imgopts "subformat=monolithicFlat" \
-     fdctrl->fifo_size = 512;
++                     "subformat=twoGbMaxExtentFlat"
-     fdctrl->result_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
+ # This is the default protocol (and we want to test the difference between
-                                              fdctrl_result_timer, fdctrl);
+ # colons which separate a protocol prefix from the rest and colons which are
  # just part of the filename, so we cannot test protocols which require a prefix)
 --
 .21.0

-[Qemu-devel] [PULL 08/20] iotests: Fix intermittent failure in 219
+[Qemu-devel] [PULL 12/15] file-posix: fix request_alignment typo
-In 219, we wait for the job to make progress before we emit its status.
+From: Stefan Hajnoczi <stefanha@redhat.com>
 This makes the output reliable.  We do not wait for any more progress if
 the job's current-progress already matches its total-progress.
-Unfortunately, there is a bug: Right after the job has been started,
+Fixes: a6b257a08e3d72219f03e461a52152672fec0612
-it's possible that total-progress is still 0.  In that case, we may skip
+       ("file-posix: Handle undetectable alignment")
-the first progress-making step and keep ending up 64 kB short.
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+Message-id: 20190827101328.4062-1-stefanha@redhat.com
-To fix that bug, we can simply wait for total-progress to reach 4 MB
+Reviewed-by: Thomas Huth <thuth@redhat.com>
 (the image size) after starting the job.
 Reported-by: Karen Mezick <kmezick@redhat.com>
 Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1686651
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 Message-id: 20190516161114.27596-1-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 [mreitz: Adjusted commit message as per John's proposal]
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/219 | 13 ++++++++++---
+ block/file-posix.c | 2 +-
-file changed, 10 insertions(+), 3 deletions(-)
+file changed, 1 insertion(+), 1 deletion(-)
-diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
+diff --git a/block/file-posix.c b/block/file-posix.c
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/219
+--- a/block/file-posix.c
-+++ b/tests/qemu-iotests/219
++++ b/block/file-posix.c
-@@ -XXX,XX +XXX,XX @@ import iotests
+@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
+         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
- iotests.verify_image_format(supported_fmts=['qcow2'])
+             align = alignments[i];
+             if (raw_is_io_aligned(fd, buf + align, max_align)) {
-+img_size = 4 * 1024 * 1024
+-                /* Fallback to request_aligment. */
-+
++                /* Fallback to request_alignment. */
- def pause_wait(vm, job_id):
+                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
-     with iotests.Timeout(3, "Timeout waiting for job to pause"):
+                 break;
-         while True:
+             }
@@ -XXX,XX +XXX,XX @@ def test_pause_resume(vm):
                  iotests.log(vm.qmp('query-jobs'))
  def test_job_lifecycle(vm, job, job_args, has_ready=False):
 +    global img_size
 +
      iotests.log('')
      iotests.log('')
      iotests.log('Starting block job: %s (auto-finalize: %s; auto-dismiss: %s)' %
@@ -XXX,XX +XXX,XX @@ def test_job_lifecycle(vm, job, job_args, has_ready=False):
      iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
      iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
 +    # Wait for total-progress to stabilize
 +    while vm.qmp('query-jobs')['return'][0]['total-progress'] < img_size:
 +        pass
 +
      # RUNNING state:
      # pause/resume should work, complete/finalize/dismiss should error out
      iotests.log('')
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('disk.img') as disk_path, \
       iotests.FilePath('copy.img') as copy_path, \
       iotests.VM() as vm:
 -    img_size = '4M'
 -    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, img_size)
 -    iotests.qemu_io('-c', 'write 0 %s' % (img_size),
 +    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, str(img_size))
 +    iotests.qemu_io('-c', 'write 0 %i' % (img_size),
                      '-f', iotests.imgfmt, disk_path)
      iotests.log('Launching VM...')
 --
 .21.0

-[Qemu-devel] [PULL 20/20] iotests: Test qemu-img convert -C --salvage
+[Qemu-devel] [PULL 13/15] iotests: Check for enabled drivers before testing them
-We do not support this combination (yet), so this should yield an error
+From: Thomas Huth <thuth@redhat.com>
 message.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+It is possible to enable only a subset of the block drivers with the
-Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+"--block-drv-rw-whitelist" option of the "configure" script. All other
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+drivers are marked as unusable (or only included as read-only with the
-Message-id: 20190507203508.18026-8-mreitz@redhat.com
+"--block-drv-ro-whitelist" option). If an iotest is now using such a
 disabled block driver, it is failing - which is bad, since at least the
 tests in the "auto" group should be able to deal with this situation.
 Thus let's introduce a "_require_drivers" function that can be used by
 the shell tests to check for the availability of certain drivers first,
 and marks the test as "not run" if one of the drivers is missing.
 This patch mainly targets the test in the "auto" group which should
 never fail in such a case, but also improves some of the other tests
 along the way. Note that we also assume that the "qcow2" and "file"
 drivers are always available - otherwise it does not make sense to
 run "make check-block" at all (which only tests with qcow2 by default).
 Signed-off-by: Thomas Huth <thuth@redhat.com>
 Message-id: 20190823133552.11680-1-thuth@redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/082     | 1 +
+ tests/qemu-iotests/071       |  1 +
- tests/qemu-iotests/082.out | 3 +++
+ tests/qemu-iotests/081       |  4 +---
-files changed, 4 insertions(+)
+ tests/qemu-iotests/099       |  1 +
  tests/qemu-iotests/120       |  1 +
  tests/qemu-iotests/162       |  4 +---
  tests/qemu-iotests/184       |  1 +
  tests/qemu-iotests/186       |  1 +
  tests/qemu-iotests/common.rc | 14 ++++++++++++++
 files changed, 21 insertions(+), 6 deletions(-)
-diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082
+diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
 index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/082
+--- a/tests/qemu-iotests/071
-+++ b/tests/qemu-iotests/082
++++ b/tests/qemu-iotests/071
-@@ -XXX,XX +XXX,XX @@ echo === convert: -C and other options ===
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
- run_qemu_img convert -C -S 4k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
- run_qemu_img convert -C -S 8k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
+ _supported_fmt qcow2
- run_qemu_img convert -C -c -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
+ _supported_proto file
-+run_qemu_img convert -C --salvage -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
++_require_drivers blkdebug blkverify
  do_run_qemu()
  {
 diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/081
 +++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt raw
  _supported_proto file
  _supported_os Linux
 +_require_drivers quorum
  do_run_qemu()
  {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                            | _filter_qemu_io | _filter_generated_node_ids
  }
 -test_quorum=$($QEMU_IMG --help|grep quorum)
 -[ "$test_quorum" = "" ] && _supported_fmt quorum
 -
  quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
  quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
  quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
 diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/099
 +++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
  _supported_proto file
  _supported_os Linux
 +_require_drivers blkdebug blkverify
  _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
      "subformat=twoGbMaxExtentSparse"
 diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/120
 +++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto file
  _unsupported_fmt luks
 +_require_drivers raw
  _make_test_img 64M
 diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/162
 +++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_fmt generic
 -
 -test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh\( \|$\)')
 -[ "$test_ssh" = "" ] && _notrun "ssh support required"
 +_require_drivers ssh
  echo
- echo === amend: Options specified more than once ===
+ echo '=== NBD ==='
-diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
+diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/184
 +++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_os Linux
 +_require_drivers throttle
  do_run_qemu()
  {
 diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/186
 +++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
 +_require_drivers null-co
  if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
      _notrun "Requires a PC machine"
 diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/082.out
+--- a/tests/qemu-iotests/common.rc
-+++ b/tests/qemu-iotests/082.out
++++ b/tests/qemu-iotests/common.rc
-@@ -XXX,XX +XXX,XX @@ qemu-img: Cannot enable copy offloading when -S is used
+@@ -XXX,XX +XXX,XX @@ _require_command()
- Testing: convert -C -c -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
+     [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
- qemu-img: Cannot enable copy offloading when -c is used
+ }
-+Testing: convert -C --salvage -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
++# Check that a set of drivers has been whitelisted in the QEMU binary
-+qemu-img: Cannot use copy offloading in salvaging mode
++#
 +_require_drivers()
 +{
 +    available=$($QEMU -drive format=help | \
 +                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
 +    for driver
 +    do
 +        if ! echo "$available" | grep -q " $driver\( \|$\)"; then
 +            _notrun "$driver not available"
 +        fi
 +    done
 +}
 +
- === amend: Options specified more than once ===
+ # make sure this script returns success
+ true
  Testing: amend -f foo -f qcow2 -o lazy_refcounts=on TEST_DIR/t.qcow2
 --
 .21.0

-[Qemu-devel] [PULL 19/20] iotests: Test qemu-img convert --salvage
+[Qemu-devel] [PULL 14/15] tests/check-block: Skip iotests when sanitizers are enabled
-This test converts a simple image to another, but blkdebug injects
+From: Thomas Huth <thuth@redhat.com>
 block_status and read faults at some offsets.  The resulting image
 should be the same as the input image, except that sectors that could
 not be read have to be 0.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+The sanitizers (especially the address sanitizer from Clang) are
-Message-id: 20190507203508.18026-7-mreitz@redhat.com
+sometimes printing out warnings or false positives - this spoils
-Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+the output of the iotests, causing some of the tests to fail.
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Thus let's skip the automatic iotests during "make check" when the
-[mreitz: Dropped superfluous printf from _filter_offsets, as suggested
+user configured QEMU with --enable-sanitizers.
-         by Vladimir; disable test for VDI and IMGOPTSSYNTAX]
 Signed-off-by: Thomas Huth <thuth@redhat.com>
 Message-id: 20190823084203.29734-1-thuth@redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/251     | 170 +++++++++++++++++++++++++++++++++++++
+ tests/check-block.sh | 5 +++++
- tests/qemu-iotests/251.out |  43 ++++++++++
+file changed, 5 insertions(+)
  tests/qemu-iotests/group   |   1 +
 files changed, 214 insertions(+)
  create mode 100755 tests/qemu-iotests/251
  create mode 100644 tests/qemu-iotests/251.out
-diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
+diff --git a/tests/check-block.sh b/tests/check-block.sh
-new file mode 100755
+index XXXXXXX..XXXXXXX 100755
-index XXXXXXX..XXXXXXX
+--- a/tests/check-block.sh
---- /dev/null
++++ b/tests/check-block.sh
-+++ b/tests/qemu-iotests/251
+@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
-@@ -XXX,XX +XXX,XX @@
+     exit 0
-+#!/usr/bin/env bash
+ fi
-+#
-+# Test qemu-img convert --salvage
++if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
-+#
++    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
-+# Copyright (C) 2019 Red Hat, Inc.
++    exit 0
 +#
 +# This program is free software; you can redistribute it and/or modify
 +# it under the terms of the GNU General Public License as published by
 +# the Free Software Foundation; either version 2 of the License, or
 +# (at your option) any later version.
 +#
 +# This program is distributed in the hope that it will be useful,
 +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 +# GNU General Public License for more details.
 +#
 +# You should have received a copy of the GNU General Public License
 +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 +#
 +
 +# creator
 +owner=mreitz@redhat.com
 +
 +seq=$(basename $0)
 +echo "QA output created by $seq"
 +
 +status=1    # failure is the default!
 +
 +_cleanup()
 +{
 +    _cleanup_test_img
 +}
 +trap "_cleanup; exit \$status" 0 1 2 3 15
 +
 +# get standard environment, filters and checks
 +. ./common.rc
 +. ./common.filter
 +. ./common.qemu
 +
 +_supported_fmt generic
 +_supported_proto file
 +_supported_os Linux
 +
 +if [ "$IMGOPTSSYNTAX" = "true" ]; then
 +    # We use json:{} filenames here, so we cannot work with additional options.
 +    _unsupported_fmt $IMGFMT
 +else
 +    # With VDI, the output is ordered differently.  Just disable it.
 +    _unsupported_fmt vdi
 +fi
 +
-+
+ if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
-+TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
+     echo "No qemu-system binary available ==> Not running the qemu-iotests."
-+
+     exit 0
 +$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
 +
 +
 +sector_size=512
 +
 +# Offsets on which to fail block-status.  Keep in ascending order so
 +# the indexing done by _filter_offsets will appear in ascending order
 +# in the output as well.
 +status_fail_offsets="$((16 * 1024 * 1024 + 8192))
 +                     $((33 * 1024 * 1024 + 512))"
 +
 +# Offsets on which to fail reads.  Keep in ascending order for the
 +# same reason.
 +# The second element is shared with $status_fail_offsets on purpose.
 +# Starting with the third element, we test what happens when a
 +# continuous range of sectors is inaccessible.
 +read_fail_offsets="$((32 * 1024 * 1024 - 65536))
 +                   $((33 * 1024 * 1024 + 512))
 +                   $(seq $((34 * 1024 * 1024)) $sector_size \
 +                         $((34 * 1024 * 1024 + 4096 - $sector_size)))"
 +
 +
 +# blkdebug must be above the format layer so it can intercept all
 +# block-status events
 +source_img="json:{'driver': 'blkdebug',
 +                  'image': {
 +                      'driver': '$IMGFMT',
 +                      'file': {
 +                          'driver': 'file',
 +                          'filename': '$TEST_IMG.orig'
 +                      }
 +                  },
 +                  'inject-error': ["
 +
 +for ofs in $status_fail_offsets
 +do
 +    source_img+="{ 'event': 'none',
 +                   'iotype': 'block-status',
 +                   'errno': 5,
 +                   'sector': $((ofs / sector_size)) },"
 +done
 +
 +for ofs in $read_fail_offsets
 +do
 +    source_img+="{ 'event': 'none',
 +                   'iotype': 'read',
 +                   'errno': 5,
 +                   'sector': $((ofs / sector_size)) },"
 +done
 +
 +# Remove the trailing comma and terminate @inject-error and json:{}
 +source_img="${source_img%,} ] }"
 +
 +
 +echo
 +
 +
 +_filter_offsets() {
 +    filters=
 +
 +    index=0
 +    for ofs in $1
 +    do
 +        filters+=" -e s/$ofs/status_fail_offset_$index/"
 +        index=$((index + 1))
 +    done
 +
 +    index=0
 +    for ofs in $2
 +    do
 +        filters+=" -e s/$ofs/read_fail_offset_$index/"
 +        index=$((index + 1))
 +    done
 +
 +    sed $filters
 +}
 +
 +# While determining the number of allocated sectors in the input
 +# image, we should see one block status warning per element of
 +# $status_fail_offsets.
 +#
 +# Then, the image is read.  Since the block status is queried in
 +# basically the same way, the same warnings as in the previous step
 +# should reappear.  Interleaved with those we should see a read
 +# warning per element of $read_fail_offsets.
 +# Note that $read_fail_offsets and $status_fail_offsets share an
 +# element (read_fail_offset_1 == status_fail_offset_1), so
 +# "status_fail_offset_1" in the output is the same as
 +# "read_fail_offset_1".
 +$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
 +    | _filter_offsets "$status_fail_offsets" "$read_fail_offsets"
 +
 +echo
 +
 +# The offsets where the block status could not be determined should
 +# have been treated as containing data and thus should be correct in
 +# the output image.
 +# The offsets where reading failed altogether should be 0.  Make them
 +# 0 in the input image, too, so we can compare both images.
 +for ofs in $read_fail_offsets
 +do
 +    $QEMU_IO -c "write -z $ofs $sector_size" "$TEST_IMG.orig" \
 +        | _filter_qemu_io \
 +        | _filter_offsets '' "$read_fail_offsets"
 +done
 +
 +echo
 +
 +# These should be equal now.
 +$QEMU_IMG compare "$TEST_IMG.orig" "$TEST_IMG"
 +
 +
 +# success, all done
 +echo "*** done"
 +rm -f $seq.full
 +status=0
 diff --git a/tests/qemu-iotests/251.out b/tests/qemu-iotests/251.out
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/tests/qemu-iotests/251.out
@@ -XXX,XX +XXX,XX @@
 +QA output created by 251
 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
 +wrote 67108864/67108864 bytes at offset 0
 +64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
 +qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
 +qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
 +qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_0: Input/output error
 +qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
 +qemu-img: warning: error while reading offset status_fail_offset_1: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_2: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_3: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_4: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_5: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_6: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_7: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_8: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_9: Input/output error
 +
 +wrote 512/512 bytes at offset read_fail_offset_0
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_1
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_2
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_3
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_4
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_5
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_6
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_7
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_8
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_9
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
 +Images are identical.
 +*** done
 diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/group
 +++ b/tests/qemu-iotests/group
@@ -XXX,XX +XXX,XX @@
 rw quick
 rw auto quick
 rw auto quick
 +251 rw auto quick
 rw auto backing quick
 rw auto quick
 rw auto backing quick
 --
 .21.0

-[Qemu-devel] [PULL 01/20] blockdev-backup: don't check aio_context too early
+[Qemu-devel] [PULL 15/15] iotests: Unify cache mode quoting
-From: John Snow <jsnow@redhat.com>
+From: Nir Soffer <nirsof@gmail.com>
-in blockdev_backup_prepare, we check to make sure that the target is
+Quoting cache mode is not needed, and most tests use unquoted values.
-associated with a compatible aio context. However, do_blockdev_backup is
+Unify all test to use the same style.
 called later and has some logic to move the target to a compatible
 aio_context. The transaction version will fail certain commands
 needlessly early as a result.
-Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
+Message-id: 20190827173432.7656-1-nsoffer@redhat.com
-will ultimately decide if the contexts are compatible or not.
+Signed-off-by: Nir Soffer <nsoffer@redhat.com>
 Note: the transaction version has always disallowed this operation since
 its initial commit bd8baecd (2014), whereas the version of
 qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
 enforce the aio_context switch instead. It's not clear, and I can't see
 from the mailing list archives at the time, why the two functions take a
 different approach. It wasn't until later in efd7556708b (2016) that the
 standalone version tried to determine if it could set the context or
 not.
 Reported-by: aihua liang <aliang@redhat.com>
 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
 Signed-off-by: John Snow <jsnow@redhat.com>
 Message-id: 20190523170643.20794-2-jsnow@redhat.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- blockdev.c | 4 ----
+ tests/qemu-iotests/026 | 4 ++--
-file changed, 4 deletions(-)
+ tests/qemu-iotests/039 | 4 ++--
  tests/qemu-iotests/052 | 2 +-
  tests/qemu-iotests/091 | 4 ++--
 files changed, 7 insertions(+), 7 deletions(-)
-diff --git a/blockdev.c b/blockdev.c
+diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/blockdev.c
+--- a/tests/qemu-iotests/026
-+++ b/blockdev.c
++++ b/tests/qemu-iotests/026
-@@ -XXX,XX +XXX,XX @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-     }
+ # Currently only qcow2 supports rebasing
+ _supported_fmt qcow2
-     aio_context = bdrv_get_aio_context(bs);
+ _supported_proto file
--    if (aio_context != bdrv_get_aio_context(target)) {
+-_default_cache_mode "writethrough"
--        error_setg(errp, "Backup between two IO threads is not implemented");
+-_supported_cache_modes "writethrough" "none"
--        return;
++_default_cache_mode writethrough
--    }
++_supported_cache_modes writethrough none
-     aio_context_acquire(aio_context);
+ # The refcount table tests expect a certain minimum width for refcount entries
-     state->bs = bs;
+ # (so that the refcount table actually needs to grow); that minimum is 16 bits,
  # being the default refcount entry width.
 diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/039
 +++ b/tests/qemu-iotests/039
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
  _supported_os Linux
 -_default_cache_mode "writethrough"
 -_supported_cache_modes "writethrough"
 +_default_cache_mode writethrough
 +_supported_cache_modes writethrough
  size=128M
 diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/052
 +++ b/tests/qemu-iotests/052
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto file
  # Don't do O_DIRECT on tmpfs
 -_supported_cache_modes "writeback" "writethrough" "unsafe"
 +_supported_cache_modes writeback writethrough unsafe
  size=128M
  _make_test_img $size
 diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/091
 +++ b/tests/qemu-iotests/091
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
  _supported_os Linux
 -_default_cache_mode "none"
 -_supported_cache_modes "writethrough" "none" "writeback"
 +_default_cache_mode none
 +_supported_cache_modes writethrough none writeback
  size=1G
 --
 .21.0

-[Qemu-devel] [PULL 02/20] iotests.py: do not use infinite waits
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Cap waits to 60 seconds so that iotests can fail gracefully if something
-goes wrong.
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190523170643.20794-3-jsnow@redhat.com
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/iotests.py | 14 +++++++-------
-file changed, 7 insertions(+), 7 deletions(-)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
-             output_list += [key + '=' + obj[key]]
-         return ','.join(output_list)
--    def get_qmp_events_filtered(self, wait=True):
-+    def get_qmp_events_filtered(self, wait=60.0):
-         result = []
-         for ev in self.get_qmp_events(wait=wait):
-             result.append(filter_qmp_event(ev))
-@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
-     # Returns None on success, and an error string on failure
-     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
--                pre_finalize=None):
-+                pre_finalize=None, wait=60.0):
-         error = None
-         while True:
--            for ev in self.get_qmp_events_filtered(wait=True):
-+            for ev in self.get_qmp_events_filtered(wait=wait):
-                 if ev['event'] == 'JOB_STATUS_CHANGE':
-                     status = ev['data']['status']
-                     if status == 'aborting':
-@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
-         self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])),
-                          self.vm.flatten_qmp_object(reference))
--    def cancel_and_wait(self, drive='drive0', force=False, resume=False):
-+    def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0):
-         '''Cancel a block job and wait for it to finish, returning the event'''
-         result = self.vm.qmp('block-job-cancel', device=drive, force=force)
-         self.assert_qmp(result, 'return', {})
-@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
-         cancelled = False
-         result = None
-         while not cancelled:
--            for event in self.vm.get_qmp_events(wait=True):
-+            for event in self.vm.get_qmp_events(wait=wait):
-                 if event['event'] == 'BLOCK_JOB_COMPLETED' or \
-                    event['event'] == 'BLOCK_JOB_CANCELLED':
-                     self.assert_qmp(event, 'data/device', drive)
-@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
-         self.assert_no_active_block_jobs()
-         return result
--    def wait_until_completed(self, drive='drive0', check_offset=True):
-+    def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0):
-         '''Wait for a block job to finish, returning the event'''
-         while True:
--            for event in self.vm.get_qmp_events(wait=True):
-+            for event in self.vm.get_qmp_events(wait=wait):
-                 if event['event'] == 'BLOCK_JOB_COMPLETED':
-                     self.assert_qmp(event, 'data/device', drive)
-                     self.assert_qmp_absent(event, 'data/error')
---
-.21.0

-[Qemu-devel] [PULL 03/20] QEMUMachine: add events_wait method
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Instead of event_wait which looks for a single event, add an events_wait
-which can look for any number of events simultaneously. However, it
-will still only return one at a time, whichever happens first.
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190523170643.20794-4-jsnow@redhat.com
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
-file changed, 49 insertions(+), 20 deletions(-)
-diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
-index XXXXXXX..XXXXXXX 100644
---- a/python/qemu/__init__.py
-+++ b/python/qemu/__init__.py
-@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
-         self._qmp.clear_events()
-         return events
--    def event_wait(self, name, timeout=60.0, match=None):
-+    @staticmethod
-+    def event_match(event, match=None):
-         """
--        Wait for specified timeout on named event in QMP; optionally filter
--        results by match.
-+        Check if an event matches optional match criteria.
--        The 'match' is checked to be a recursive subset of the 'event'; skips
--        branch processing on match's value None
--           {"foo": {"bar": 1}} matches {"foo": None}
--           {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
-+        The match criteria takes the form of a matching subdict. The event is
-+        checked to be a superset of the subdict, recursively, with matching
-+        values whenever those values are not None.
-+
-+        Examples, with the subdict queries on the left:
-+         - None matches any object.
-+         - {"foo": None} matches {"foo": {"bar": 1}}
-+         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
-+         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
-         """
--        def event_match(event, match=None):
--            if match is None:
--                return True
-+        if match is None:
-+            return True
--            for key in match:
--                if key in event:
--                    if isinstance(event[key], dict):
--                        if not event_match(event[key], match[key]):
--                            return False
--                    elif event[key] != match[key]:
-+        for key in match:
-+            if key in event:
-+                if isinstance(event[key], dict):
-+                    if not QEMUMachine.event_match(event[key], match[key]):
-                         return False
--                else:
-+                elif event[key] != match[key]:
-                     return False
-+            else:
-+                return False
-+        return True
--            return True
-+    def event_wait(self, name, timeout=60.0, match=None):
-+        """
-+        event_wait waits for and returns a named event from QMP with a timeout.
-+
-+        name: The event to wait for.
-+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
-+        match: Optional match criteria. See event_match for details.
-+        """
-+        return self.events_wait([(name, match)], timeout)
-+
-+    def events_wait(self, events, timeout=60.0):
-+        """
-+        events_wait waits for and returns a named event from QMP with a timeout.
-+
-+        events: a sequence of (name, match_criteria) tuples.
-+                The match criteria are optional and may be None.
-+                See event_match for details.
-+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
-+        """
-+        def _match(event):
-+            for name, match in events:
-+                if (event['event'] == name and
-+                    self.event_match(event, match)):
-+                    return True
-+            return False
-         # Search cached events
-         for event in self._events:
--            if (event['event'] == name) and event_match(event, match):
-+            if _match(event):
-                 self._events.remove(event)
-                 return event
-         # Poll for new events
-         while True:
-             event = self._qmp.pull_event(wait=timeout)
--            if (event['event'] == name) and event_match(event, match):
-+            if _match(event):
-                 return event
-             self._events.append(event)
---
-.21.0

-[Qemu-devel] [PULL 04/20] iotests.py: rewrite run_job to be pickier
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Don't pull events out of the queue that don't belong to us;
-be choosier so that we can use this method to drive jobs that
-were launched by transactions that may have more jobs.
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190523170643.20794-5-jsnow@redhat.com
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++--------------
-file changed, 29 insertions(+), 19 deletions(-)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
-     # Returns None on success, and an error string on failure
-     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
-                 pre_finalize=None, wait=60.0):
-+        match_device = {'data': {'device': job}}
-+        match_id = {'data': {'id': job}}
-+        events = [
-+            ('BLOCK_JOB_COMPLETED', match_device),
-+            ('BLOCK_JOB_CANCELLED', match_device),
-+            ('BLOCK_JOB_ERROR', match_device),
-+            ('BLOCK_JOB_READY', match_device),
-+            ('BLOCK_JOB_PENDING', match_id),
-+            ('JOB_STATUS_CHANGE', match_id)
-+        ]
-         error = None
-         while True:
--            for ev in self.get_qmp_events_filtered(wait=wait):
--                if ev['event'] == 'JOB_STATUS_CHANGE':
--                    status = ev['data']['status']
--                    if status == 'aborting':
--                        result = self.qmp('query-jobs')
--                        for j in result['return']:
--                            if j['id'] == job:
--                                error = j['error']
--                                log('Job failed: %s' % (j['error']))
--                    elif status == 'pending' and not auto_finalize:
--                        if pre_finalize:
--                            pre_finalize()
--                        self.qmp_log('job-finalize', id=job)
--                    elif status == 'concluded' and not auto_dismiss:
--                        self.qmp_log('job-dismiss', id=job)
--                    elif status == 'null':
--                        return error
--                else:
--                    log(ev)
-+            ev = filter_qmp_event(self.events_wait(events))
-+            if ev['event'] != 'JOB_STATUS_CHANGE':
-+                log(ev)
-+                continue
-+            status = ev['data']['status']
-+            if status == 'aborting':
-+                result = self.qmp('query-jobs')
-+                for j in result['return']:
-+                    if j['id'] == job:
-+                        error = j['error']
-+                        log('Job failed: %s' % (j['error']))
-+            elif status == 'pending' and not auto_finalize:
-+                if pre_finalize:
-+                    pre_finalize()
-+                self.qmp_log('job-finalize', id=job)
-+            elif status == 'concluded' and not auto_dismiss:
-+                self.qmp_log('job-dismiss', id=job)
-+            elif status == 'null':
-+                return error
-     def node_info(self, node_name):
-         nodes = self.qmp('query-named-block-nodes')
---
-.21.0

-[Qemu-devel] [PULL 05/20] iotests: add iotest 256 for testing blockdev-backup across iothread contexts
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190523170643.20794-6-jsnow@redhat.com
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-[mreitz: Moved from 250 to 256]
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/256     | 122 +++++++++++++++++++++++++++++++++++++
- tests/qemu-iotests/256.out | 119 ++++++++++++++++++++++++++++++++++++
- tests/qemu-iotests/group   |   1 +
-files changed, 242 insertions(+)
- create mode 100755 tests/qemu-iotests/256
- create mode 100644 tests/qemu-iotests/256.out
-diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
-new file mode 100755
-index XXXXXXX..XXXXXXX
---- /dev/null
-+++ b/tests/qemu-iotests/256
-@@ -XXX,XX +XXX,XX @@
-+#!/usr/bin/env python
-+#
-+# Test incremental/backup across iothread contexts
-+#
-+# Copyright (c) 2019 John Snow for Red Hat, Inc.
-+#
-+# This program is free software; you can redistribute it and/or modify
-+# it under the terms of the GNU General Public License as published by
-+# the Free Software Foundation; either version 2 of the License, or
-+# (at your option) any later version.
-+#
-+# This program is distributed in the hope that it will be useful,
-+# but WITHOUT ANY WARRANTY; without even the implied warranty of
-+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-+# GNU General Public License for more details.
-+#
-+# You should have received a copy of the GNU General Public License
-+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-+#
-+# owner=jsnow@redhat.com
-+
-+import os
-+import iotests
-+from iotests import log
-+
-+iotests.verify_image_format(supported_fmts=['qcow2'])
-+size = 64 * 1024 * 1024
-+
-+with iotests.FilePath('img0') as img0_path, \
-+     iotests.FilePath('img1') as img1_path, \
-+     iotests.FilePath('img0-full') as img0_full_path, \
-+     iotests.FilePath('img1-full') as img1_full_path, \
-+     iotests.FilePath('img0-incr') as img0_incr_path, \
-+     iotests.FilePath('img1-incr') as img1_incr_path, \
-+     iotests.VM() as vm:
-+
-+    def create_target(filepath, name, size):
-+        basename = os.path.basename(filepath)
-+        nodename = "file_{}".format(basename)
-+        log(vm.command('blockdev-create', job_id='job1',
-+                       options={
-+                           'driver': 'file',
-+                           'filename': filepath,
-+                           'size': 0,
-+                       }))
-+        vm.run_job('job1')
-+        log(vm.command('blockdev-add', driver='file',
-+                       node_name=nodename, filename=filepath))
-+        log(vm.command('blockdev-create', job_id='job2',
-+                       options={
-+                           'driver': iotests.imgfmt,
-+                           'file': nodename,
-+                           'size': size,
-+                       }))
-+        vm.run_job('job2')
-+        log(vm.command('blockdev-add', driver=iotests.imgfmt,
-+                       node_name=name,
-+                       file=nodename))
-+
-+    log('--- Preparing images & VM ---\n')
-+    vm.add_object('iothread,id=iothread0')
-+    vm.add_object('iothread,id=iothread1')
-+    vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
-+    vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
-+    iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
-+    iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
-+    vm.add_drive(img0_path, interface='none')
-+    vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
-+    vm.add_drive(img1_path, interface='none')
-+    vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0')
-+
-+    log('--- Starting VM ---\n')
-+    vm.launch()
-+
-+    log('--- Create Targets & Full Backups ---\n')
-+    create_target(img0_full_path, 'img0-full', size)
-+    create_target(img1_full_path, 'img1-full', size)
-+    ret = vm.qmp_log('transaction', indent=2, actions=[
-+        { 'type': 'block-dirty-bitmap-add',
-+          'data': { 'node': 'drive0', 'name': 'bitmap0' }},
-+        { 'type': 'block-dirty-bitmap-add',
-+          'data': { 'node': 'drive1', 'name': 'bitmap1' }},
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive0',
-+                    'target': 'img0-full',
-+                    'sync': 'full',
-+                    'job-id': 'j0' }},
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive1',
-+                    'target': 'img1-full',
-+                    'sync': 'full',
-+                    'job-id': 'j1' }}
-+    ])
-+    if "error" in ret:
-+        raise Exception(ret['error']['desc'])
-+    vm.run_job('j0', auto_dismiss=True)
-+    vm.run_job('j1', auto_dismiss=True)
-+
-+    log('\n--- Create Targets & Incremental Backups ---\n')
-+    create_target(img0_incr_path, 'img0-incr', size)
-+    create_target(img1_incr_path, 'img1-incr', size)
-+    ret = vm.qmp_log('transaction', indent=2, actions=[
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive0',
-+                    'target': 'img0-incr',
-+                    'sync': 'incremental',
-+                    'bitmap': 'bitmap0',
-+                    'job-id': 'j2' }},
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive1',
-+                    'target': 'img1-incr',
-+                    'sync': 'incremental',
-+                    'bitmap': 'bitmap1',
-+                    'job-id': 'j3' }}
-+    ])
-+    if "error" in ret:
-+        raise Exception(ret['error']['desc'])
-+    vm.run_job('j2', auto_dismiss=True)
-+    vm.run_job('j3', auto_dismiss=True)
-+
-+    log('\n--- Done ---')
-+    vm.shutdown()
-diff --git a/tests/qemu-iotests/256.out b/tests/qemu-iotests/256.out
-new file mode 100644
-index XXXXXXX..XXXXXXX
---- /dev/null
-+++ b/tests/qemu-iotests/256.out
-@@ -XXX,XX +XXX,XX @@
-+--- Preparing images & VM ---
-+
-+--- Starting VM ---
-+
-+--- Create Targets & Full Backups ---
-+
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{
-+  "execute": "transaction",
-+  "arguments": {
-+    "actions": [
-+      {
-+        "data": {
-+          "name": "bitmap0",
-+          "node": "drive0"
-+        },
-+        "type": "block-dirty-bitmap-add"
-+      },
-+      {
-+        "data": {
-+          "name": "bitmap1",
-+          "node": "drive1"
-+        },
-+        "type": "block-dirty-bitmap-add"
-+      },
-+      {
-+        "data": {
-+          "device": "drive0",
-+          "job-id": "j0",
-+          "sync": "full",
-+          "target": "img0-full"
-+        },
-+        "type": "blockdev-backup"
-+      },
-+      {
-+        "data": {
-+          "device": "drive1",
-+          "job-id": "j1",
-+          "sync": "full",
-+          "target": "img1-full"
-+        },
-+        "type": "blockdev-backup"
-+      }
-+    ]
-+  }
-+}
-+{
-+  "return": {}
-+}
-+{"data": {"device": "j0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+{"data": {"device": "j1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+
-+--- Create Targets & Incremental Backups ---
-+
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{
-+  "execute": "transaction",
-+  "arguments": {
-+    "actions": [
-+      {
-+        "data": {
-+          "bitmap": "bitmap0",
-+          "device": "drive0",
-+          "job-id": "j2",
-+          "sync": "incremental",
-+          "target": "img0-incr"
-+        },
-+        "type": "blockdev-backup"
-+      },
-+      {
-+        "data": {
-+          "bitmap": "bitmap1",
-+          "device": "drive1",
-+          "job-id": "j3",
-+          "sync": "incremental",
-+          "target": "img1-incr"
-+        },
-+        "type": "blockdev-backup"
-+      }
-+    ]
-+  }
-+}
-+{
-+  "return": {}
-+}
-+{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+
-+--- Done ---
-diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/group
-+++ b/tests/qemu-iotests/group
-@@ -XXX,XX +XXX,XX @@
-rw auto quick
-rw auto backing quick
-rw auto quick
-+256 rw auto quick
---
-.21.0

-[Qemu-devel] [PULL 06/20] event_match: always match on None value
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Before, event_match didn't always recurse if the event value was not a
-dictionary, and would instead check for equality immediately.
-By delaying equality checking to post-recursion, we can allow leaf
-values like "5" to match "None" and take advantage of the generic
-None-returns-True clause.
-This makes the matching a little more obviously consistent at the
-expense of being able to check for explicit None values, which is
-probably not that important given what this function is used for.
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190528183857.26167-1-jsnow@redhat.com
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- python/qemu/__init__.py | 24 ++++++++++++++----------
-file changed, 14 insertions(+), 10 deletions(-)
-diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
-index XXXXXXX..XXXXXXX 100644
---- a/python/qemu/__init__.py
-+++ b/python/qemu/__init__.py
-@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
-         The match criteria takes the form of a matching subdict. The event is
-         checked to be a superset of the subdict, recursively, with matching
--        values whenever those values are not None.
-+        values whenever the subdict values are not None.
-+
-+        This has a limitation that you cannot explicitly check for None values.
-         Examples, with the subdict queries on the left:
-          - None matches any object.
-          - {"foo": None} matches {"foo": {"bar": 1}}
--         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
--         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
-+         - {"foo": None} matches {"foo": 5}
-+         - {"foo": {"abc": None}} does not match {"foo": {"bar": 1}}
-+         - {"foo": {"rab": 2}} matches {"foo": {"bar": 1, "rab": 2}}
-         """
-         if match is None:
-             return True
--        for key in match:
--            if key in event:
--                if isinstance(event[key], dict):
-+        try:
-+            for key in match:
-+                if key in event:
-                     if not QEMUMachine.event_match(event[key], match[key]):
-                         return False
--                elif event[key] != match[key]:
-+                else:
-                     return False
--            else:
--                return False
--        return True
-+            return True
-+        except TypeError:
-+            # either match or event wasn't iterable (not a dict)
-+            return match == event
-     def event_wait(self, name, timeout=60.0, match=None):
-         """
---
-.21.0

The following changes since commit 5ec2eca83dc478ddf24077e02a8b34dd26cd3ff9:

Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20190613.0' into staging (2019-06-14 09:33:55 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-06-14

for you to fetch changes up to 21c1ce592a144188dfe59b9e156a97da412a59a2:

iotests: Test qemu-img convert -C --salvage (2019-06-14 15:09:42 +0200)

----------------------------------------------------------------
Block patches:
- Allow blockdev-backup from nodes that are not in qemu's main AIO
  context to newly added nodes
- Add salvaging mode to qemu-img convert
- Minor fixes to tests, documentation, and for less Valgrind annoyance

----------------------------------------------------------------
Andrey Shinkevich (1):
  hw/block/fdc: floppy command FIFO memory initialization

John Snow (6):
  blockdev-backup: don't check aio_context too early
  iotests.py: do not use infinite waits
  QEMUMachine: add events_wait method
  iotests.py: rewrite run_job to be pickier
  iotests: add iotest 256 for testing blockdev-backup across iothread
    contexts
  event_match: always match on None value

Max Reitz (12):
  iotests: Filter 175's allocation information
  iotests: Fix intermittent failure in 219
  qemu-img: Fix options leakage in img_rebase()
  qapi/block-core: Overlays are not snapshots
  blockdev: Overlays are not snapshots
  qemu-img: Move quiet into ImgConvertState
  qemu-img: Add salvaging mode to convert
  blkdebug: Add @iotype error option
  blkdebug: Add "none" event
  blkdebug: Inject errors on .bdrv_co_block_status()
  iotests: Test qemu-img convert --salvage
  iotests: Test qemu-img convert -C --salvage

Vladimir Sementsov-Ogievskiy (1):
  iotests: restrict 254 to support only qcow2

qapi/block-core.json          |  53 ++++++++---
 block/blkdebug.c              |  60 ++++++++++--
 blockdev.c                    |  14 +--
 hw/block/fdc.c                |   1 +
 qemu-img.c                    | 106 +++++++++++++++------
 python/qemu/__init__.py       |  67 ++++++++++----
 qemu-img-cmds.hx              |   4 +-
 qemu-img.texi                 |   4 +
 tests/qemu-iotests/082        |   1 +
 tests/qemu-iotests/082.out    |   3 +
 tests/qemu-iotests/085.out    |  10 +-
 tests/qemu-iotests/175        |  26 +++++-
 tests/qemu-iotests/175.out    |   8 +-
 tests/qemu-iotests/219        |  13 ++-
 tests/qemu-iotests/251        | 170 ++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out    |  43 +++++++++
 tests/qemu-iotests/254        |   2 +
 tests/qemu-iotests/256        | 122 ++++++++++++++++++++++++
 tests/qemu-iotests/256.out    | 119 ++++++++++++++++++++++++
 tests/qemu-iotests/group      |   2 +
 tests/qemu-iotests/iotests.py |  60 +++++++-----
 21 files changed, 772 insertions(+), 116 deletions(-)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out
 create mode 100755 tests/qemu-iotests/256
 create mode 100644 tests/qemu-iotests/256.out

-- 
2.21.0

From: John Snow <jsnow@redhat.com>

in blockdev_backup_prepare, we check to make sure that the target is
associated with a compatible aio context. However, do_blockdev_backup is
called later and has some logic to move the target to a compatible
aio_context. The transaction version will fail certain commands
needlessly early as a result.

Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
will ultimately decide if the contexts are compatible or not.

Note: the transaction version has always disallowed this operation since
its initial commit bd8baecd (2014), whereas the version of
qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
enforce the aio_context switch instead. It's not clear, and I can't see
from the mailing list archives at the time, why the two functions take a
different approach. It wasn't until later in efd7556708b (2016) that the
standalone version tried to determine if it could set the context or
not.

Reported-by: aihua liang <aliang@redhat.com>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-2-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
     }
 
     aio_context = bdrv_get_aio_context(bs);
-    if (aio_context != bdrv_get_aio_context(target)) {
-        error_setg(errp, "Backup between two IO threads is not implemented");
-        return;
-    }
     aio_context_acquire(aio_context);
     state->bs = bs;
 
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Cap waits to 60 seconds so that iotests can fail gracefully if something
goes wrong.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-3-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
             output_list += [key + '=' + obj[key]]
         return ','.join(output_list)
 
-    def get_qmp_events_filtered(self, wait=True):
+    def get_qmp_events_filtered(self, wait=60.0):
         result = []
         for ev in self.get_qmp_events(wait=wait):
             result.append(filter_qmp_event(ev))
@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
 
     # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
-                pre_finalize=None):
+                pre_finalize=None, wait=60.0):
         error = None
         while True:
-            for ev in self.get_qmp_events_filtered(wait=True):
+            for ev in self.get_qmp_events_filtered(wait=wait):
                 if ev['event'] == 'JOB_STATUS_CHANGE':
                     status = ev['data']['status']
                     if status == 'aborting':
@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
         self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])),
                          self.vm.flatten_qmp_object(reference))
 
-    def cancel_and_wait(self, drive='drive0', force=False, resume=False):
+    def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0):
         '''Cancel a block job and wait for it to finish, returning the event'''
         result = self.vm.qmp('block-job-cancel', device=drive, force=force)
         self.assert_qmp(result, 'return', {})
@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
         cancelled = False
         result = None
         while not cancelled:
-            for event in self.vm.get_qmp_events(wait=True):
+            for event in self.vm.get_qmp_events(wait=wait):
                 if event['event'] == 'BLOCK_JOB_COMPLETED' or \
                    event['event'] == 'BLOCK_JOB_CANCELLED':
                     self.assert_qmp(event, 'data/device', drive)
@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
         self.assert_no_active_block_jobs()
         return result
 
-    def wait_until_completed(self, drive='drive0', check_offset=True):
+    def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0):
         '''Wait for a block job to finish, returning the event'''
         while True:
-            for event in self.vm.get_qmp_events(wait=True):
+            for event in self.vm.get_qmp_events(wait=wait):
                 if event['event'] == 'BLOCK_JOB_COMPLETED':
                     self.assert_qmp(event, 'data/device', drive)
                     self.assert_qmp_absent(event, 'data/error')
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Instead of event_wait which looks for a single event, add an events_wait
which can look for any number of events simultaneously. However, it
will still only return one at a time, whichever happens first.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-4-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
 1 file changed, 49 insertions(+), 20 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index XXXXXXX..XXXXXXX 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
         self._qmp.clear_events()
         return events
 
-    def event_wait(self, name, timeout=60.0, match=None):
+    @staticmethod
+    def event_match(event, match=None):
         """
-        Wait for specified timeout on named event in QMP; optionally filter
-        results by match.
+        Check if an event matches optional match criteria.
 
-        The 'match' is checked to be a recursive subset of the 'event'; skips
-        branch processing on match's value None
-           {"foo": {"bar": 1}} matches {"foo": None}
-           {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
+        The match criteria takes the form of a matching subdict. The event is
+        checked to be a superset of the subdict, recursively, with matching
+        values whenever those values are not None.
+
+        Examples, with the subdict queries on the left:
+         - None matches any object.
+         - {"foo": None} matches {"foo": {"bar": 1}}
+         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
+         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
         """
-        def event_match(event, match=None):
-            if match is None:
-                return True
+        if match is None:
+            return True
 
-            for key in match:
-                if key in event:
-                    if isinstance(event[key], dict):
-                        if not event_match(event[key], match[key]):
-                            return False
-                    elif event[key] != match[key]:
+        for key in match:
+            if key in event:
+                if isinstance(event[key], dict):
+                    if not QEMUMachine.event_match(event[key], match[key]):
                         return False
-                else:
+                elif event[key] != match[key]:
                     return False
+            else:
+                return False
+        return True
 
-            return True
+    def event_wait(self, name, timeout=60.0, match=None):
+        """
+        event_wait waits for and returns a named event from QMP with a timeout.
+
+        name: The event to wait for.
+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+        match: Optional match criteria. See event_match for details.
+        """
+        return self.events_wait([(name, match)], timeout)
+
+    def events_wait(self, events, timeout=60.0):
+        """
+        events_wait waits for and returns a named event from QMP with a timeout.
+
+        events: a sequence of (name, match_criteria) tuples.
+                The match criteria are optional and may be None.
+                See event_match for details.
+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+        """
+        def _match(event):
+            for name, match in events:
+                if (event['event'] == name and
+                    self.event_match(event, match)):
+                    return True
+            return False
 
         # Search cached events
         for event in self._events:
-            if (event['event'] == name) and event_match(event, match):
+            if _match(event):
                 self._events.remove(event)
                 return event
 
         # Poll for new events
         while True:
             event = self._qmp.pull_event(wait=timeout)
-            if (event['event'] == name) and event_match(event, match):
+            if _match(event):
                 return event
             self._events.append(event)
 
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Don't pull events out of the queue that don't belong to us;
be choosier so that we can use this method to drive jobs that
were launched by transactions that may have more jobs.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-5-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++--------------
 1 file changed, 29 insertions(+), 19 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
     # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
                 pre_finalize=None, wait=60.0):
+        match_device = {'data': {'device': job}}
+        match_id = {'data': {'id': job}}
+        events = [
+            ('BLOCK_JOB_COMPLETED', match_device),
+            ('BLOCK_JOB_CANCELLED', match_device),
+            ('BLOCK_JOB_ERROR', match_device),
+            ('BLOCK_JOB_READY', match_device),
+            ('BLOCK_JOB_PENDING', match_id),
+            ('JOB_STATUS_CHANGE', match_id)
+        ]
         error = None
         while True:
-            for ev in self.get_qmp_events_filtered(wait=wait):
-                if ev['event'] == 'JOB_STATUS_CHANGE':
-                    status = ev['data']['status']
-                    if status == 'aborting':
-                        result = self.qmp('query-jobs')
-                        for j in result['return']:
-                            if j['id'] == job:
-                                error = j['error']
-                                log('Job failed: %s' % (j['error']))
-                    elif status == 'pending' and not auto_finalize:
-                        if pre_finalize:
-                            pre_finalize()
-                        self.qmp_log('job-finalize', id=job)
-                    elif status == 'concluded' and not auto_dismiss:
-                        self.qmp_log('job-dismiss', id=job)
-                    elif status == 'null':
-                        return error
-                else:
-                    log(ev)
+            ev = filter_qmp_event(self.events_wait(events))
+            if ev['event'] != 'JOB_STATUS_CHANGE':
+                log(ev)
+                continue
+            status = ev['data']['status']
+            if status == 'aborting':
+                result = self.qmp('query-jobs')
+                for j in result['return']:
+                    if j['id'] == job:
+                        error = j['error']
+                        log('Job failed: %s' % (j['error']))
+            elif status == 'pending' and not auto_finalize:
+                if pre_finalize:
+                    pre_finalize()
+                self.qmp_log('job-finalize', id=job)
+            elif status == 'concluded' and not auto_dismiss:
+                self.qmp_log('job-dismiss', id=job)
+            elif status == 'null':
+                return error
 
     def node_info(self, node_name):
         nodes = self.qmp('query-named-block-nodes')
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-6-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
[mreitz: Moved from 250 to 256]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/256     | 122 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/256.out | 119 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 242 insertions(+)
 create mode 100755 tests/qemu-iotests/256
 create mode 100644 tests/qemu-iotests/256.out

diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
new file mode 100755
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/256
@@ -XXX,XX +XXX,XX @@
+#!/usr/bin/env python
+#
+# Test incremental/backup across iothread contexts
+#
+# Copyright (c) 2019 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# owner=jsnow@redhat.com
+
+import os
+import iotests
+from iotests import log
+
+iotests.verify_image_format(supported_fmts=['qcow2'])
+size = 64 * 1024 * 1024
+
+with iotests.FilePath('img0') as img0_path, \
+     iotests.FilePath('img1') as img1_path, \
+     iotests.FilePath('img0-full') as img0_full_path, \
+     iotests.FilePath('img1-full') as img1_full_path, \
+     iotests.FilePath('img0-incr') as img0_incr_path, \
+     iotests.FilePath('img1-incr') as img1_incr_path, \
+     iotests.VM() as vm:
+
+    def create_target(filepath, name, size):
+        basename = os.path.basename(filepath)
+        nodename = "file_{}".format(basename)
+        log(vm.command('blockdev-create', job_id='job1',
+                       options={
+                           'driver': 'file',
+                           'filename': filepath,
+                           'size': 0,
+                       }))
+        vm.run_job('job1')
+        log(vm.command('blockdev-add', driver='file',
+                       node_name=nodename, filename=filepath))
+        log(vm.command('blockdev-create', job_id='job2',
+                       options={
+                           'driver': iotests.imgfmt,
+                           'file': nodename,
+                           'size': size,
+                       }))
+        vm.run_job('job2')
+        log(vm.command('blockdev-add', driver=iotests.imgfmt,
+                       node_name=name,
+                       file=nodename))
+
+    log('--- Preparing images & VM ---\n')
+    vm.add_object('iothread,id=iothread0')
+    vm.add_object('iothread,id=iothread1')
+    vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
+    vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
+    iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
+    iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
+    vm.add_drive(img0_path, interface='none')
+    vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
+    vm.add_drive(img1_path, interface='none')
+    vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0')
+
+    log('--- Starting VM ---\n')
+    vm.launch()
+
+    log('--- Create Targets & Full Backups ---\n')
+    create_target(img0_full_path, 'img0-full', size)
+    create_target(img1_full_path, 'img1-full', size)
+    ret = vm.qmp_log('transaction', indent=2, actions=[
+        { 'type': 'block-dirty-bitmap-add',
+          'data': { 'node': 'drive0', 'name': 'bitmap0' }},
+        { 'type': 'block-dirty-bitmap-add',
+          'data': { 'node': 'drive1', 'name': 'bitmap1' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive0',
+                    'target': 'img0-full',
+                    'sync': 'full',
+                    'job-id': 'j0' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive1',
+                    'target': 'img1-full',
+                    'sync': 'full',
+                    'job-id': 'j1' }}
+    ])
+    if "error" in ret:
+        raise Exception(ret['error']['desc'])
+    vm.run_job('j0', auto_dismiss=True)
+    vm.run_job('j1', auto_dismiss=True)
+
+    log('\n--- Create Targets & Incremental Backups ---\n')
+    create_target(img0_incr_path, 'img0-incr', size)
+    create_target(img1_incr_path, 'img1-incr', size)
+    ret = vm.qmp_log('transaction', indent=2, actions=[
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive0',
+                    'target': 'img0-incr',
+                    'sync': 'incremental',
+                    'bitmap': 'bitmap0',
+                    'job-id': 'j2' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive1',
+                    'target': 'img1-incr',
+                    'sync': 'incremental',
+                    'bitmap': 'bitmap1',
+                    'job-id': 'j3' }}
+    ])
+    if "error" in ret:
+        raise Exception(ret['error']['desc'])
+    vm.run_job('j2', auto_dismiss=True)
+    vm.run_job('j3', auto_dismiss=True)
+
+    log('\n--- Done ---')
+    vm.shutdown()
diff --git a/tests/qemu-iotests/256.out b/tests/qemu-iotests/256.out
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/256.out
@@ -XXX,XX +XXX,XX @@
+--- Preparing images & VM ---
+
+--- Starting VM ---
+
+--- Create Targets & Full Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+  "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {
+        "data": {
+          "name": "bitmap0",
+          "node": "drive0"
+        },
+        "type": "block-dirty-bitmap-add"
+      },
+      {
+        "data": {
+          "name": "bitmap1",
+          "node": "drive1"
+        },
+        "type": "block-dirty-bitmap-add"
+      },
+      {
+        "data": {
+          "device": "drive0",
+          "job-id": "j0",
+          "sync": "full",
+          "target": "img0-full"
+        },
+        "type": "blockdev-backup"
+      },
+      {
+        "data": {
+          "device": "drive1",
+          "job-id": "j1",
+          "sync": "full",
+          "target": "img1-full"
+        },
+        "type": "blockdev-backup"
+      }
+    ]
+  }
+}
+{
+  "return": {}
+}
+{"data": {"device": "j0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Create Targets & Incremental Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+  "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {
+        "data": {
+          "bitmap": "bitmap0",
+          "device": "drive0",
+          "job-id": "j2",
+          "sync": "incremental",
+          "target": "img0-incr"
+        },
+        "type": "blockdev-backup"
+      },
+      {
+        "data": {
+          "bitmap": "bitmap1",
+          "device": "drive1",
+          "job-id": "j3",
+          "sync": "incremental",
+          "target": "img1-incr"
+        },
+        "type": "blockdev-backup"
+      }
+    ]
+  }
+}
+{
+  "return": {}
+}
+{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Done ---
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -XXX,XX +XXX,XX @@
 253 rw auto quick
 254 rw auto backing quick
 255 rw auto quick
+256 rw auto quick
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Before, event_match didn't always recurse if the event value was not a
dictionary, and would instead check for equality immediately.

By delaying equality checking to post-recursion, we can allow leaf
values like "5" to match "None" and take advantage of the generic
None-returns-True clause.

This makes the matching a little more obviously consistent at the
expense of being able to check for explicit None values, which is
probably not that important given what this function is used for.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190528183857.26167-1-jsnow@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 python/qemu/__init__.py | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index XXXXXXX..XXXXXXX 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
 
         The match criteria takes the form of a matching subdict. The event is
         checked to be a superset of the subdict, recursively, with matching
-        values whenever those values are not None.
+        values whenever the subdict values are not None.
+
+        This has a limitation that you cannot explicitly check for None values.
 
         Examples, with the subdict queries on the left:
          - None matches any object.
          - {"foo": None} matches {"foo": {"bar": 1}}
-         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
-         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
+         - {"foo": None} matches {"foo": 5}
+         - {"foo": {"abc": None}} does not match {"foo": {"bar": 1}}
+         - {"foo": {"rab": 2}} matches {"foo": {"bar": 1, "rab": 2}}
         """
         if match is None:
             return True
 
-        for key in match:
-            if key in event:
-                if isinstance(event[key], dict):
+        try:
+            for key in match:
+                if key in event:
                     if not QEMUMachine.event_match(event[key], match[key]):
                         return False
-                elif event[key] != match[key]:
+                else:
                     return False
-            else:
-                return False
-        return True
+            return True
+        except TypeError:
+            # either match or event wasn't iterable (not a dict)
+            return match == event
 
     def event_wait(self, name, timeout=60.0, match=None):
         """
-- 
2.21.0

It is possible for an empty file to take up blocks on a filesystem, for
example:

$ qemu-img create -f raw test.img 1G
Formatting 'test.img', fmt=raw size=1073741824
$ mkfs.ext4 -I 128 -q test.img
$ mkdir test-mount
$ sudo mount -o loop test.img test-mount
$ sudo touch test-mount/test-file
$ stat -c 'blocks=%b' test-mount/test-file
blocks=8

These extra blocks (one cluster) are apparently used for metadata,
because they are always there, on top of blocks used for data:

$ sudo dd if=/dev/zero of=test-mount/test-file bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00135339 s, 775 MB/s
$ stat -c 'blocks=%b' test-mount/test-file
blocks=2056

Make iotest 175 take this into account.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190516144319.12570-1-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/175     | 26 ++++++++++++++++++++++----
 tests/qemu-iotests/175.out |  8 ++++----
 2 files changed, 26 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
-	_cleanup_test_img
+    _cleanup_test_img
+    rm -f "$TEST_DIR/empty"
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
+# Some file systems sometimes allocate extra blocks independently of
+# the file size.  This function hides the resulting difference in the
+# stat -c '%b' output.
+# Parameter 1: Number of blocks an empty file occupies
+# Parameter 2: Image size in bytes
+_filter_blocks()
+{
+    extra_blocks=$1
+    img_size=$2
+
+    sed -e "s/blocks=$extra_blocks\$\$\\|[^0-9]\$/nothing allocated/" \
+        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/everything allocated/"
+}
+
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
 _supported_proto file
 _supported_os Linux
 
-size=1m
+size=$((1 * 1024 * 1024))
+
+touch "$TEST_DIR/empty"
+extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 
 echo
 echo "== creating image with default preallocation =="
 _make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 
 for mode in off full falloc; do
     echo
     echo "== creating image with preallocation $mode =="
     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
-    stat -c "size=%s, blocks=%b" $TEST_IMG
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 done
 
 # success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
 
 == creating image with default preallocation ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, blocks=0
+size=1048576, nothing allocated
 
 == creating image with preallocation off ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, blocks=0
+size=1048576, nothing allocated
 
 == creating image with preallocation full ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, blocks=2048
+size=1048576, everything allocated
 
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, blocks=2048
+size=1048576, everything allocated
  *** done
-- 
2.21.0

In 219, we wait for the job to make progress before we emit its status.
This makes the output reliable.  We do not wait for any more progress if
the job's current-progress already matches its total-progress.

Unfortunately, there is a bug: Right after the job has been started,
it's possible that total-progress is still 0.  In that case, we may skip
the first progress-making step and keep ending up 64 kB short.

To fix that bug, we can simply wait for total-progress to reach 4 MB
(the image size) after starting the job.

Reported-by: Karen Mezick <kmezick@redhat.com>
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1686651
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190516161114.27596-1-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
[mreitz: Adjusted commit message as per John's proposal]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/219 | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/219
+++ b/tests/qemu-iotests/219
@@ -XXX,XX +XXX,XX @@ import iotests
 
 iotests.verify_image_format(supported_fmts=['qcow2'])
 
+img_size = 4 * 1024 * 1024
+
 def pause_wait(vm, job_id):
     with iotests.Timeout(3, "Timeout waiting for job to pause"):
         while True:
@@ -XXX,XX +XXX,XX @@ def test_pause_resume(vm):
                 iotests.log(vm.qmp('query-jobs'))
 
 def test_job_lifecycle(vm, job, job_args, has_ready=False):
+    global img_size
+
     iotests.log('')
     iotests.log('')
     iotests.log('Starting block job: %s (auto-finalize: %s; auto-dismiss: %s)' %
@@ -XXX,XX +XXX,XX @@ def test_job_lifecycle(vm, job, job_args, has_ready=False):
     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
 
+    # Wait for total-progress to stabilize
+    while vm.qmp('query-jobs')['return'][0]['total-progress'] < img_size:
+        pass
+
     # RUNNING state:
     # pause/resume should work, complete/finalize/dismiss should error out
     iotests.log('')
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('disk.img') as disk_path, \
      iotests.FilePath('copy.img') as copy_path, \
      iotests.VM() as vm:
 
-    img_size = '4M'
-    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, img_size)
-    iotests.qemu_io('-c', 'write 0 %s' % (img_size),
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, str(img_size))
+    iotests.qemu_io('-c', 'write 0 %i' % (img_size),
                     '-f', iotests.imgfmt, disk_path)
 
     iotests.log('Launching VM...')
-- 
2.21.0

From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>

The uninitialized memory allocated for the command FIFO of the
floppy controller during the VM hardware initialization incurs
many unwanted reports by Valgrind when VM state is being saved.
That verbosity hardens a search for the real memory issues when
the iotests run. Particularly, the patch eliminates 20 unnecessary
reports of the Valgrind tool in the iotest #169.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-id: 1559154027-282547-1-git-send-email-andrey.shinkevich@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 hw/block/fdc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/block/fdc.c b/hw/block/fdc.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/block/fdc.c
+++ b/hw/block/fdc.c
@@ -XXX,XX +XXX,XX @@ static void fdctrl_realize_common(DeviceState *dev, FDCtrl *fdctrl,
 
     FLOPPY_DPRINTF("init controller\n");
     fdctrl->fifo = qemu_memalign(512, FD_SECTOR_LEN);
+    memset(fdctrl->fifo, 0, FD_SECTOR_LEN);
     fdctrl->fifo_size = 512;
     fdctrl->result_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
                                              fdctrl_result_timer, fdctrl);
-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Test fails at least for qcow, because of different cluster sizes in
base and top (and therefore different granularities of bitmaps we are
trying to merge).

The test aim is to check block-dirty-bitmap-merge between different
nodes functionality, no needs to check all formats. So, let's just drop
support for anything except qcow2.

Reported-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190605155405.104384-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/254 | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/254
+++ b/tests/qemu-iotests/254
@@ -XXX,XX +XXX,XX @@
 import iotests
 from iotests import qemu_img_create, file_path, log
 
+iotests.verify_image_format(supported_fmts=['qcow2'])
+
 disk, top = file_path('disk', 'top')
 size = 1024 * 1024
 
-- 
2.21.0

img_rebase() can leak a QDict in two occasions.  Fix it.

Coverity: CID 1401416
Fixes: d16699b64671466b42079c45b89127aeea1ca565
Fixes: 330c72957196e0ae382abcaa97ebf4eb9bc8574f
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190528195338.12376-1-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
                                                              out_baseimg,
                                                              &local_err);
             if (local_err) {
+                qobject_unref(options);
                 error_reportf_err(local_err,
                                   "Could not resolve backing filename: ");
                 ret = -1;
@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
              */
             prefix_chain_bs = bdrv_find_backing_image(bs, out_real_path);
             if (prefix_chain_bs) {
+                qobject_unref(options);
                 g_free(out_real_path);
+
                 blk_new_backing = blk_new(qemu_get_aio_context(),
                                           BLK_PERM_CONSISTENT_READ,
                                           BLK_PERM_ALL);
-- 
2.21.0

A snapshot is something that reflects the state of something at a
certain point in time.  It does not change.

The file our snapshot commands create (or the node they install) is not
a snapshot, as it does change over time.  It is an overlay.  We cannot
do anything about the parameter names, but we can at least adjust the
descriptions to reflect that fact.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190603202236.1342-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # Either @device or @node-name must be set but not both.
 #
-# @device: the name of the device to generate the snapshot from.
+# @device: the name of the device to take a snapshot of.
 #
 # @node-name: graph node name to generate the snapshot from (Since 2.0)
 #
-# @snapshot-file: the target of the new image. If the file exists, or
-# if it is a device, the snapshot will be created in the existing
-# file/device. Otherwise, a new file will be created.
+# @snapshot-file: the target of the new overlay image. If the file
+# exists, or if it is a device, the overlay will be created in the
+# existing file/device. Otherwise, a new file will be created.
 #
 # @snapshot-node-name: the graph node name of the new image (Since 2.0)
 #
-# @format: the format of the snapshot image, default is 'qcow2'.
+# @format: the format of the overlay image, default is 'qcow2'.
 #
 # @mode: whether and how QEMU should create a new image, default is
 #        'absolute-paths'.
@@ -XXX,XX +XXX,XX @@
 ##
 # @BlockdevSnapshot:
 #
-# @node: device or node name that will have a snapshot created.
+# @node: device or node name that will have a snapshot taken.
 #
 # @overlay: reference to the existing block device that will become
-#           the overlay of @node, as part of creating the snapshot.
+#           the overlay of @node, as part of taking the snapshot.
 #           It must not have a current backing file (this can be
 #           achieved by passing "backing": null to blockdev-add).
 #
@@ -XXX,XX +XXX,XX @@
 ##
 # @blockdev-snapshot-sync:
 #
-# Generates a synchronous snapshot of a block device.
+# Takes a synchronous snapshot of a block device.
 #
 # For the arguments, see the documentation of BlockdevSnapshotSync.
 #
@@ -XXX,XX +XXX,XX @@
 ##
 # @blockdev-snapshot:
 #
-# Generates a snapshot of a block device.
+# Takes a snapshot of a block device.
 #
-# Create a snapshot, by installing 'node' as the backing image of
+# Take a snapshot, by installing 'node' as the backing image of
 # 'overlay'. Additionally, if 'node' is associated with a block
 # device, the block device changes to using 'overlay' as its new active
 # image.
-- 
2.21.0

There are error messages which refer to an overlay node as the snapshot.
That is wrong, those are two different things.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190603202236.1342-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c                 | 10 +++++-----
 tests/qemu-iotests/085.out | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
             s->has_snapshot_node_name ? s->snapshot_node_name : NULL;
 
         if (node_name && !snapshot_node_name) {
-            error_setg(errp, "New snapshot node name missing");
+            error_setg(errp, "New overlay node name missing");
             goto out;
         }
 
         if (snapshot_node_name &&
             bdrv_lookup_bs(snapshot_node_name, snapshot_node_name, NULL)) {
-            error_setg(errp, "New snapshot node name already in use");
+            error_setg(errp, "New overlay node name already in use");
             goto out;
         }
 
@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
     }
 
     if (bdrv_has_blk(state->new_bs)) {
-        error_setg(errp, "The snapshot is already in use");
+        error_setg(errp, "The overlay is already in use");
         goto out;
     }
 
@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
     }
 
     if (state->new_bs->backing != NULL) {
-        error_setg(errp, "The snapshot already has a backing image");
+        error_setg(errp, "The overlay already has a backing image");
         goto out;
     }
 
     if (!state->new_bs->drv->supports_backing) {
-        error_setg(errp, "The snapshot does not support backing images");
+        error_setg(errp, "The overlay does not support backing images");
         goto out;
     }
 
diff --git a/tests/qemu-iotests/085.out b/tests/qemu-iotests/085.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/085.out
+++ b/tests/qemu-iotests/085.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
 
 === Invalid command - cannot create a snapshot using a file BDS ===
 
-{"error": {"class": "GenericError", "desc": "The snapshot does not support backing images"}}
+{"error": {"class": "GenericError", "desc": "The overlay does not support backing images"}}
 
 === Invalid command - snapshot node used as active layer ===
 
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
 
 === Invalid command - snapshot node used as backing hd ===
 
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
 Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base
 {"return": {}}
-{"error": {"class": "GenericError", "desc": "The snapshot already has a backing image"}}
+{"error": {"class": "GenericError", "desc": "The overlay already has a backing image"}}
 
 === Invalid command - The node does not exist ===
 
-- 
2.21.0

Move img_convert()'s quiet flag into the ImgConvertState so it is
accessible by nested functions.  -q dictates that it suppresses anything
but errors, so if those functions want to emit warnings, they need to
query this flag first.  (There currently are no such warnings, but there
will be as of the next patch.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-2-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool quiet;
     int min_sparse;
     int alignment;
     size_t cluster_sectors;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
     QDict *open_opts = NULL;
     char *options = NULL;
     Error *local_err = NULL;
-    bool writethrough, src_writethrough, quiet = false, image_opts = false,
+    bool writethrough, src_writethrough, image_opts = false,
          skip_create = false, progress = false, tgt_image_opts = false;
     int64_t ret = -EINVAL;
     bool force_share = false;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
             src_cache = optarg;
             break;
         case 'q':
-            quiet = true;
+            s.quiet = true;
             break;
         case 'n':
             skip_create = true;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
     }
 
     /* Initialize before goto out */
-    if (quiet) {
+    if (s.quiet) {
         progress = false;
     }
     qemu_progress_init(progress, 1.0);
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
 
     for (bs_i = 0; bs_i < s.src_num; bs_i++) {
         s.src[bs_i] = img_open(image_opts, argv[optind + bs_i],
-                               fmt, src_flags, src_writethrough, quiet,
+                               fmt, src_flags, src_writethrough, s.quiet,
                                force_share);
         if (!s.src[bs_i]) {
             ret = -1;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
 
     if (skip_create) {
         s.target = img_open(tgt_image_opts, out_filename, out_fmt,
-                            flags, writethrough, quiet, false);
+                            flags, writethrough, s.quiet, false);
     } else {
         /* TODO ultimately we should allow --target-image-opts
          * to be used even when -n is not given.
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
          * to allow filenames in option syntax
          */
         s.target = img_open_file(out_filename, open_opts, out_fmt,
-                                 flags, writethrough, quiet, false);
+                                 flags, writethrough, s.quiet, false);
         open_opts = NULL; /* blk_new_open will have freed it */
     }
     if (!s.target) {
-- 
2.21.0

This adds a salvaging mode (--salvage) to qemu-img convert which ignores
read errors and treats the respective areas as containing only zeroes.
This can be used for instance to at least partially recover the data
from terminally corrupted qcow2 images.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-3-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c       | 90 +++++++++++++++++++++++++++++++++++++-----------
 qemu-img-cmds.hx |  4 +--
 qemu-img.texi    |  4 +++
 3 files changed, 75 insertions(+), 23 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ enum {
     OPTION_SIZE = 264,
     OPTION_PREALLOCATION = 265,
     OPTION_SHRINK = 266,
+    OPTION_SALVAGE = 267,
 };
 
 typedef enum OutputFormat {
@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool salvage;
     bool quiet;
     int min_sparse;
     int alignment;
@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
     }
 
     if (s->sector_next_status <= sector_num) {
-        int64_t count = n * BDRV_SECTOR_SIZE;
+        uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
+        int64_t count;
 
-        if (s->target_has_backing) {
+        do {
+            count = n * BDRV_SECTOR_SIZE;
+
+            if (s->target_has_backing) {
+                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
+                                        count, &count, NULL, NULL);
+            } else {
+                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
+                                              offset, count, &count, NULL,
+                                              NULL);
+            }
+
+            if (ret < 0) {
+                if (s->salvage) {
+                    if (n == 1) {
+                        if (!s->quiet) {
+                            warn_report("error while reading block status at "
+                                        "offset %" PRIu64 ": %s", offset,
+                                        strerror(-ret));
+                        }
+                        /* Just try to read the data, then */
+                        ret = BDRV_BLOCK_DATA;
+                        count = BDRV_SECTOR_SIZE;
+                    } else {
+                        /* Retry on a shorter range */
+                        n = DIV_ROUND_UP(n, 4);
+                    }
+                } else {
+                    error_report("error while reading block status at offset "
+                                 "%" PRIu64 ": %s", offset, strerror(-ret));
+                    return ret;
+                }
+            }
+        } while (ret < 0);
 
-            ret = bdrv_block_status(blk_bs(s->src[src_cur]),
-                                    (sector_num - src_cur_offset) *
-                                    BDRV_SECTOR_SIZE,
-                                    count, &count, NULL, NULL);
-        } else {
-            ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                          (sector_num - src_cur_offset) *
-                                          BDRV_SECTOR_SIZE,
-                                          count, &count, NULL, NULL);
-        }
-        if (ret < 0) {
-            error_report("error while reading block status of sector %" PRId64
-                         ": %s", sector_num, strerror(-ret));
-            return ret;
-        }
         n = DIV_ROUND_UP(count, BDRV_SECTOR_SIZE);
 
         if (ret & BDRV_BLOCK_ZERO) {
@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
 static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
                                         int nb_sectors, uint8_t *buf)
 {
+    uint64_t single_read_until = 0;
     int n, ret;
 
     assert(nb_sectors <= s->buf_sectors);
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         BlockBackend *blk;
         int src_cur;
         int64_t bs_sectors, src_cur_offset;
+        uint64_t offset;
 
         /* In the case of compression with multiple source files, we can get a
          * nb_sectors that spreads into the next part. So we must be able to
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         blk = s->src[src_cur];
         bs_sectors = s->src_sectors[src_cur];
 
+        offset = (sector_num - src_cur_offset) << BDRV_SECTOR_BITS;
+
         n = MIN(nb_sectors, bs_sectors - (sector_num - src_cur_offset));
+        if (single_read_until > offset) {
+            n = 1;
+        }
 
-        ret = blk_co_pread(
-                blk, (sector_num - src_cur_offset) << BDRV_SECTOR_BITS,
-                n << BDRV_SECTOR_BITS, buf, 0);
+        ret = blk_co_pread(blk, offset, n << BDRV_SECTOR_BITS, buf, 0);
         if (ret < 0) {
-            return ret;
+            if (s->salvage) {
+                if (n > 1) {
+                    single_read_until = offset + (n << BDRV_SECTOR_BITS);
+                    continue;
+                } else {
+                    if (!s->quiet) {
+                        warn_report("error while reading offset %" PRIu64
+                                    ": %s", offset, strerror(-ret));
+                    }
+                    memset(buf, 0, BDRV_SECTOR_SIZE);
+                }
+            } else {
+                return ret;
+            }
         }
 
         sector_num += n;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
             {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
             {"force-share", no_argument, 0, 'U'},
             {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
+            {"salvage", no_argument, 0, OPTION_SALVAGE},
             {0, 0, 0, 0}
         };
         c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
         case OPTION_IMAGE_OPTS:
             image_opts = true;
             break;
+        case OPTION_SALVAGE:
+            s.salvage = true;
+            break;
         case OPTION_TARGET_IMAGE_OPTS:
             tgt_image_opts = true;
             break;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
         goto fail_getopt;
     }
 
+    if (s.copy_range && s.salvage) {
+        error_report("Cannot use copy offloading in salvaging mode");
+        goto fail_getopt;
+    }
+
     if (tgt_image_opts && !skip_create) {
         error_report("--target-image-opts requires use of -n flag");
         goto fail_getopt;
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -XXX,XX +XXX,XX @@ STEXI
 ETEXI
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
 STEXI
-@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename}
+@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] [--salvage] @var{filename} [@var{filename2} [...]] @var{output_filename}
 ETEXI
 
 DEF("create", img_create,
diff --git a/qemu-img.texi b/qemu-img.texi
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.texi
+++ b/qemu-img.texi
@@ -XXX,XX +XXX,XX @@ improve performance if the data is remote, such as with NFS or iSCSI backends,
 but will not automatically sparsify zero sectors, and may result in a fully
 allocated target image depending on the host support for getting allocation
 information.
+@item --salvage
+Try to ignore I/O errors when reading.  Unless in quiet mode (@code{-q}), errors
+will still be printed.  Areas that cannot be read from the source will be
+treated as containing only zeroes.
 @end table
 
 Parameters to dd subcommand:
-- 
2.21.0

This new error option allows users of blkdebug to inject errors only on
certain kinds of I/O operations.  Users usually want to make a very
specific operation fail, not just any; but right now they simply hope
that the event that triggers the error injection is followed up with
that very operation.  That may not be true, however, because the block
layer is changing (including blkdebug, which may increase the number of
types of I/O operations on which to inject errors).

The new option's default has been chosen to keep backwards
compatibility.

Note that similar to the internal representation, we could choose to
expose this option as a list of I/O types.  But there is no practical
use for this, because as described above, users usually know exactly
which kind of operation they want to make fail, so there is no need to
specify multiple I/O types at once.  In addition, exposing this option
as a list would require non-trivial changes to qemu_opts_absorb_qdict().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-4-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 26 +++++++++++++++++++++++
 block/blkdebug.c     | 50 ++++++++++++++++++++++++++++++++++++--------
 2 files changed, 67 insertions(+), 9 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
             'cor_write', 'cluster_alloc_space'] }
 
+##
+# @BlkdebugIOType:
+#
+# Kinds of I/O that blkdebug can inject errors in.
+#
+# @read: .bdrv_co_preadv()
+#
+# @write: .bdrv_co_pwritev()
+#
+# @write-zeroes: .bdrv_co_pwrite_zeroes()
+#
+# @discard: .bdrv_co_pdiscard()
+#
+# @flush: .bdrv_co_flush_to_disk()
+#
+# Since: 4.1
+##
+{ 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+
 ##
 # @BlkdebugInjectErrorOptions:
 #
@@ -XXX,XX +XXX,XX @@
 # @state:       the state identifier blkdebug needs to be in to
 #               actually trigger the event; defaults to "any"
 #
+# @iotype:      the type of I/O operations on which this error should
+#               be injected; defaults to "all read, write,
+#               write-zeroes, discard, and flush operations"
+#               (since: 4.1)
+#
 # @errno:       error identifier (errno) to be returned; defaults to
 #               EIO
 #
@@ -XXX,XX +XXX,XX @@
 { 'struct': 'BlkdebugInjectErrorOptions',
   'data': { 'event': 'BlkdebugEvent',
             '*state': 'int',
+            '*iotype': 'BlkdebugIOType',
             '*errno': 'int',
             '*sector': 'int',
             '*once': 'bool',
diff --git a/block/blkdebug.c b/block/blkdebug.c
index XXXXXXX..XXXXXXX 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
     int state;
     union {
         struct {
+            uint64_t iotype_mask;
             int error;
             int immediately;
             int once;
@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
     QSIMPLEQ_ENTRY(BlkdebugRule) active_next;
 } BlkdebugRule;
 
+QEMU_BUILD_BUG_MSG(BLKDEBUG_IO_TYPE__MAX > 64,
+                   "BlkdebugIOType mask does not fit into an uint64_t");
+
 static QemuOptsList inject_error_opts = {
     .name = "inject-error",
     .head = QTAILQ_HEAD_INITIALIZER(inject_error_opts.head),
@@ -XXX,XX +XXX,XX @@ static QemuOptsList inject_error_opts = {
             .name = "state",
             .type = QEMU_OPT_NUMBER,
         },
+        {
+            .name = "iotype",
+            .type = QEMU_OPT_STRING,
+        },
         {
             .name = "errno",
             .type = QEMU_OPT_NUMBER,
@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
     int event;
     struct BlkdebugRule *rule;
     int64_t sector;
+    BlkdebugIOType iotype;
+    Error *local_error = NULL;
 
     /* Find the right event for the rule */
     event_name = qemu_opt_get(opts, "event");
@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
         sector = qemu_opt_get_number(opts, "sector", -1);
         rule->options.inject.offset =
             sector == -1 ? -1 : sector * BDRV_SECTOR_SIZE;
+
+        iotype = qapi_enum_parse(&BlkdebugIOType_lookup,
+                                 qemu_opt_get(opts, "iotype"),
+                                 BLKDEBUG_IO_TYPE__MAX, &local_error);
+        if (local_error) {
+            error_propagate(errp, local_error);
+            return -1;
+        }
+        if (iotype != BLKDEBUG_IO_TYPE__MAX) {
+            rule->options.inject.iotype_mask = (1ull << iotype);
+        } else {
+            /* Apply the default */
+            rule->options.inject.iotype_mask =
+                (1ull << BLKDEBUG_IO_TYPE_READ)
+                | (1ull << BLKDEBUG_IO_TYPE_WRITE)
+                | (1ull << BLKDEBUG_IO_TYPE_WRITE_ZEROES)
+                | (1ull << BLKDEBUG_IO_TYPE_DISCARD)
+                | (1ull << BLKDEBUG_IO_TYPE_FLUSH);
+        }
+
         break;
 
     case ACTION_SET_STATE:
@@ -XXX,XX +XXX,XX @@ out:
     return ret;
 }
 
-static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
+static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
+                      BlkdebugIOType iotype)
 {
     BDRVBlkdebugState *s = bs->opaque;
     BlkdebugRule *rule = NULL;
@@ -XXX,XX +XXX,XX @@ static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
     QSIMPLEQ_FOREACH(rule, &s->active_rules, active_next) {
         uint64_t inject_offset = rule->options.inject.offset;
 
-        if (inject_offset == -1 ||
-            (bytes && inject_offset >= offset &&
-             inject_offset < offset + bytes))
+        if ((inject_offset == -1 ||
+             (bytes && inject_offset >= offset &&
+              inject_offset < offset + bytes)) &&
+            (rule->options.inject.iotype_mask & (1ull << iotype)))
         {
             break;
         }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
         assert(bytes <= bs->bl.max_transfer);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_READ);
     if (err) {
         return err;
     }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
         assert(bytes <= bs->bl.max_transfer);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE);
     if (err) {
         return err;
     }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
 
 static int blkdebug_co_flush(BlockDriverState *bs)
 {
-    int err = rule_check(bs, 0, 0);
+    int err = rule_check(bs, 0, 0, BLKDEBUG_IO_TYPE_FLUSH);
 
     if (err) {
         return err;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pwrite_zeroes(BlockDriverState *bs,
         assert(bytes <= bs->bl.max_pwrite_zeroes);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE_ZEROES);
     if (err) {
         return err;
     }
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pdiscard(BlockDriverState *bs,
         assert(bytes <= bs->bl.max_pdiscard);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_DISCARD);
     if (err) {
         return err;
     }
-- 
2.21.0

Together with @iotypes and @sector, this can be used to trap e.g. the
first read or write access to a certain sector without having to know
what happens internally in the block layer, i.e. which "real" events
happen right before such an access.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-5-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 4 +++-
 block/blkdebug.c     | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # @cluster_alloc_space: an allocation of file space for a cluster (since 4.1)
 #
+# @none: triggers once at creation of the blkdebug node (since 4.1)
+#
 # Since: 2.9
 ##
 { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
@@ -XXX,XX +XXX,XX @@
             'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev',
             'pwritev_zero', 'pwritev_done', 'empty_image_prepare',
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
-            'cor_write', 'cluster_alloc_space'] }
+            'cor_write', 'cluster_alloc_space', 'none'] }
 
 ##
 # @BlkdebugIOType:
diff --git a/block/blkdebug.c b/block/blkdebug.c
index XXXXXXX..XXXXXXX 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ static int blkdebug_open(BlockDriverState *bs, QDict *options, int flags,
         goto out;
     }
 
+    bdrv_debug_event(bs, BLKDBG_NONE);
+
     ret = 0;
 out:
     if (ret < 0) {
-- 
2.21.0

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-6-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 5 ++++-
 block/blkdebug.c     | 8 ++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # @flush: .bdrv_co_flush_to_disk()
 #
+# @block-status: .bdrv_co_block_status()
+#
 # Since: 4.1
 ##
 { 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
-  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush',
+            'block-status' ] }
 
 ##
 # @BlkdebugInjectErrorOptions:
diff --git a/block/blkdebug.c b/block/blkdebug.c
index XXXXXXX..XXXXXXX 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,
                                                  int64_t *map,
                                                  BlockDriverState **file)
 {
+    int err;
+
     assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment));
+
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_BLOCK_STATUS);
+    if (err) {
+        return err;
+    }
+
     return bdrv_co_block_status_from_file(bs, want_zero, offset, bytes,
                                           pnum, map, file);
 }
-- 
2.21.0

This test converts a simple image to another, but blkdebug injects
block_status and read faults at some offsets.  The resulting image
should be the same as the input image, except that sectors that could
not be read have to be 0.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190507203508.18026-7-mreitz@redhat.com
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[mreitz: Dropped superfluous printf from _filter_offsets, as suggested
         by Vladimir; disable test for VDI and IMGOPTSSYNTAX]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/251     | 170 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out |  43 ++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 214 insertions(+)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out

diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
new file mode 100755
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@
+#!/usr/bin/env bash
+#
+# Test qemu-img convert --salvage
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mreitz@redhat.com
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt generic
+_supported_proto file
+_supported_os Linux
+
+if [ "$IMGOPTSSYNTAX" = "true" ]; then
+    # We use json:{} filenames here, so we cannot work with additional options.
+    _unsupported_fmt $IMGFMT
+else
+    # With VDI, the output is ordered differently.  Just disable it.
+    _unsupported_fmt vdi
+fi
+
+
+TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
+
+
+sector_size=512
+
+# Offsets on which to fail block-status.  Keep in ascending order so
+# the indexing done by _filter_offsets will appear in ascending order
+# in the output as well.
+status_fail_offsets="$((16 * 1024 * 1024 + 8192))
+                     $((33 * 1024 * 1024 + 512))"
+
+# Offsets on which to fail reads.  Keep in ascending order for the
+# same reason.
+# The second element is shared with $status_fail_offsets on purpose.
+# Starting with the third element, we test what happens when a
+# continuous range of sectors is inaccessible.
+read_fail_offsets="$((32 * 1024 * 1024 - 65536))
+                   $((33 * 1024 * 1024 + 512))
+                   $(seq $((34 * 1024 * 1024)) $sector_size \
+                         $((34 * 1024 * 1024 + 4096 - $sector_size)))"
+
+
+# blkdebug must be above the format layer so it can intercept all
+# block-status events
+source_img="json:{'driver': 'blkdebug',
+                  'image': {
+                      'driver': '$IMGFMT',
+                      'file': {
+                          'driver': 'file',
+                          'filename': '$TEST_IMG.orig'
+                      }
+                  },
+                  'inject-error': ["
+
+for ofs in $status_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'block-status',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+for ofs in $read_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'read',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+# Remove the trailing comma and terminate @inject-error and json:{}
+source_img="${source_img%,} ] }"
+
+
+echo
+
+
+_filter_offsets() {
+    filters=
+
+    index=0
+    for ofs in $1
+    do
+        filters+=" -e s/$ofs/status_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    index=0
+    for ofs in $2
+    do
+        filters+=" -e s/$ofs/read_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    sed $filters
+}
+
+# While determining the number of allocated sectors in the input
+# image, we should see one block status warning per element of
+# $status_fail_offsets.
+#
+# Then, the image is read.  Since the block status is queried in
+# basically the same way, the same warnings as in the previous step
+# should reappear.  Interleaved with those we should see a read
+# warning per element of $read_fail_offsets.
+# Note that $read_fail_offsets and $status_fail_offsets share an
+# element (read_fail_offset_1 == status_fail_offset_1), so
+# "status_fail_offset_1" in the output is the same as
+# "read_fail_offset_1".
+$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
+    | _filter_offsets "$status_fail_offsets" "$read_fail_offsets"
+
+echo
+
+# The offsets where the block status could not be determined should
+# have been treated as containing data and thus should be correct in
+# the output image.
+# The offsets where reading failed altogether should be 0.  Make them
+# 0 in the input image, too, so we can compare both images.
+for ofs in $read_fail_offsets
+do
+    $QEMU_IO -c "write -z $ofs $sector_size" "$TEST_IMG.orig" \
+        | _filter_qemu_io \
+        | _filter_offsets '' "$read_fail_offsets"
+done
+
+echo
+
+# These should be equal now.
+$QEMU_IMG compare "$TEST_IMG.orig" "$TEST_IMG"
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/251.out b/tests/qemu-iotests/251.out
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/251.out
@@ -XXX,XX +XXX,XX @@
+QA output created by 251
+Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
+wrote 67108864/67108864 bytes at offset 0
+64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_2: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_3: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_4: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_5: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_6: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_7: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_8: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_9: Input/output error
+
+wrote 512/512 bytes at offset read_fail_offset_0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_1
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_2
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_3
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_4
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_5
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_6
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_7
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_8
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_9
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+Images are identical.
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -XXX,XX +XXX,XX @@
 248 rw quick
 249 rw auto quick
 250 rw auto quick
+251 rw auto quick
 252 rw auto backing quick
 253 rw auto quick
 254 rw auto backing quick
-- 
2.21.0

We do not support this combination (yet), so this should yield an error
message.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-8-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/082     | 1 +
 tests/qemu-iotests/082.out | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/082
+++ b/tests/qemu-iotests/082
@@ -XXX,XX +XXX,XX @@ echo === convert: -C and other options ===
 run_qemu_img convert -C -S 4k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -S 8k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -c -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
+run_qemu_img convert -C --salvage -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 
 echo
 echo === amend: Options specified more than once ===
diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -XXX,XX +XXX,XX @@ qemu-img: Cannot enable copy offloading when -S is used
 Testing: convert -C -c -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
 qemu-img: Cannot enable copy offloading when -c is used
 
+Testing: convert -C --salvage -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
+qemu-img: Cannot use copy offloading in salvaging mode
+
 === amend: Options specified more than once ===
 
 Testing: amend -f foo -f qcow2 -o lazy_refcounts=on TEST_DIR/t.qcow2
-- 
2.21.0

The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:

Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27

for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:

iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)

----------------------------------------------------------------
Block patches:
- qemu-io now accepts a file to read a write pattern from
- Ensure that raw files have their first block allocated so we can probe
  the O_DIRECT alignment if necessary
- Various fixes

----------------------------------------------------------------
Denis Plotnikov (1):
  qemu-io: add pattern file for write command

Max Reitz (7):
  iotests: Fix _filter_img_create()
  vmdk: Use bdrv_dirname() for relative extent paths
  iotests: Keep testing broken relative extent paths
  vmdk: Reject invalid compressed writes
  iotests: Disable broken streamOptimized tests
  iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
  iotests: Disable 126 for flat vmdk subformats

Nir Soffer (3):
  block: posix: Always allocate the first block
  iotests: Test allocate_first_block() with O_DIRECT
  iotests: Unify cache mode quoting

Stefan Hajnoczi (1):
  file-posix: fix request_alignment typo

Thomas Huth (2):
  iotests: Check for enabled drivers before testing them
  tests/check-block: Skip iotests when sanitizers are enabled

Vladimir Sementsov-Ogievskiy (1):
  block: fix permission update in bdrv_replace_node

block.c                                       |  5 +-
 block/file-posix.c                            | 53 +++++++++-
 block/vmdk.c                                  | 64 ++++++++----
 qemu-io-cmds.c                                | 99 +++++++++++++++++--
 tests/check-block.sh                          |  5 +
 tests/qemu-iotests/002                        |  1 +
 tests/qemu-iotests/003                        |  1 +
 tests/qemu-iotests/005                        |  3 +-
 tests/qemu-iotests/009                        |  1 +
 tests/qemu-iotests/010                        |  1 +
 tests/qemu-iotests/011                        |  1 +
 tests/qemu-iotests/017                        |  3 +-
 tests/qemu-iotests/018                        |  3 +-
 tests/qemu-iotests/019                        |  3 +-
 tests/qemu-iotests/020                        |  3 +-
 tests/qemu-iotests/026                        |  4 +-
 tests/qemu-iotests/027                        |  1 +
 tests/qemu-iotests/032                        |  1 +
 tests/qemu-iotests/033                        |  1 +
 tests/qemu-iotests/034                        |  3 +-
 tests/qemu-iotests/037                        |  3 +-
 tests/qemu-iotests/039                        |  4 +-
 tests/qemu-iotests/052                        |  2 +-
 tests/qemu-iotests/059                        | 34 ++++++-
 tests/qemu-iotests/059.out                    | 26 +++--
 tests/qemu-iotests/063                        |  3 +-
 tests/qemu-iotests/071                        |  1 +
 tests/qemu-iotests/072                        |  1 +
 tests/qemu-iotests/081                        |  4 +-
 tests/qemu-iotests/091                        |  4 +-
 tests/qemu-iotests/099                        |  1 +
 tests/qemu-iotests/105                        |  3 +-
 tests/qemu-iotests/110                        |  3 +-
 tests/qemu-iotests/120                        |  1 +
 tests/qemu-iotests/126                        |  2 +
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++
 tests/qemu-iotests/162                        |  4 +-
 tests/qemu-iotests/175                        | 47 +++++++--
 tests/qemu-iotests/175.out                    | 16 ++-
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/184                        |  1 +
 tests/qemu-iotests/186                        |  1 +
 tests/qemu-iotests/197                        |  1 +
 tests/qemu-iotests/215                        |  1 +
 tests/qemu-iotests/221.out                    | 12 ++-
 tests/qemu-iotests/251                        |  1 +
 tests/qemu-iotests/253.out                    | 12 ++-
 tests/qemu-iotests/common.filter              |  4 +-
 tests/qemu-iotests/common.rc                  | 14 +++
 50 files changed, 391 insertions(+), 87 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

-- 
2.21.0

From: Denis Plotnikov <dplotnikov@virtuozzo.com>

The patch allows to provide a pattern file for write
command. There was no similar ability before.

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
Reviewed-by: Eric Blake <eblake@redhat.com>
[mreitz: Keep optstring in alphabetical order]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 6 deletions(-)

diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
     qemu_vfree(p);
 }
 
+/*
+ * qemu_io_alloc_from_file()
+ *
+ * Allocates the buffer and populates it with the content of the given file
+ * up to @len bytes. If the file length is less than @len, then the buffer
+ * is populated with the file content cyclically.
+ *
+ * @blk - the block backend where the buffer content is going to be written to
+ * @len - the buffer length
+ * @file_name - the file to read the content from
+ *
+ * Returns: the buffer pointer on success
+ *          NULL on error
+ */
+static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
+                                     const char *file_name)
+{
+    char *buf, *buf_origin;
+    FILE *f = fopen(file_name, "r");
+    int pattern_len;
+
+    if (!f) {
+        perror(file_name);
+        return NULL;
+    }
+
+    if (qemuio_misalign) {
+        len += MISALIGN_OFFSET;
+    }
+
+    buf_origin = buf = blk_blockalign(blk, len);
+
+    if (qemuio_misalign) {
+        buf_origin += MISALIGN_OFFSET;
+        buf += MISALIGN_OFFSET;
+        len -= MISALIGN_OFFSET;
+    }
+
+    pattern_len = fread(buf_origin, 1, len, f);
+
+    if (ferror(f)) {
+        perror(file_name);
+        goto error;
+    }
+
+    if (pattern_len == 0) {
+        fprintf(stderr, "%s: file is empty\n", file_name);
+        goto error;
+    }
+
+    fclose(f);
+
+    if (len > pattern_len) {
+        len -= pattern_len;
+        buf += pattern_len;
+
+        while (len > 0) {
+            size_t len_to_copy = MIN(pattern_len, len);
+
+            memcpy(buf, buf_origin, len_to_copy);
+
+            len -= len_to_copy;
+            buf += len_to_copy;
+        }
+    }
+
+    return buf_origin;
+
+error:
+    qemu_io_free(buf_origin);
+    return NULL;
+}
+
 static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
 {
     uint64_t i;
@@ -XXX,XX +XXX,XX @@ static void write_help(void)
 " -n, -- with -z, don't allow slow fallback\n"
 " -p, -- ignored for backwards compatibility\n"
 " -P, -- use different pattern to fill file\n"
+" -s, -- use a pattern file to fill the write buffer\n"
 " -C, -- report statistics in a machine parsable format\n"
 " -q, -- quiet mode, do not show I/O statistics\n"
 " -u, -- with -z, allow unmapping\n"
@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
     .perm       = BLK_PERM_WRITE,
     .argmin     = 2,
     .argmax     = -1,
-    .args       = "[-bcCfnquz] [-P pattern] off len",
+    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
     .oneline    = "writes a number of bytes at a specified offset",
     .help       = write_help,
 };
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
 {
     struct timespec t1, t2;
     bool Cflag = false, qflag = false, bflag = false;
-    bool Pflag = false, zflag = false, cflag = false;
+    bool Pflag = false, zflag = false, cflag = false, sflag = false;
     int flags = 0;
     int c, cnt, ret;
     char *buf = NULL;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     /* Some compilers get confused and warn if this is not initialized.  */
     int64_t total = 0;
     int pattern = 0xcd;
+    const char *file_name = NULL;
 
-    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
+    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
         switch (c) {
         case 'b':
             bflag = true;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         case 'q':
             qflag = true;
             break;
+        case 's':
+            sflag = true;
+            file_name = optarg;
+            break;
         case 'u':
             flags |= BDRV_REQ_MAY_UNMAP;
             break;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         return -EINVAL;
     }
 
-    if (zflag && Pflag) {
-        printf("-z and -P cannot be specified at the same time\n");
+    if (zflag + Pflag + sflag > 1) {
+        printf("Only one of -z, -P, and -s "
+               "can be specified at the same time\n");
         return -EINVAL;
     }
 
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     }
 
     if (!zflag) {
-        buf = qemu_io_alloc(blk, count, pattern);
+        if (sflag) {
+            buf = qemu_io_alloc_from_file(blk, count, file_name);
+            if (!buf) {
+                return -EINVAL;
+            }
+        } else {
+            buf = qemu_io_alloc(blk, count, pattern);
+        }
     }
 
     clock_gettime(CLOCK_MONOTONIC, &t1);
-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

It's wrong to OR shared permissions. It may lead to crash on further
permission updates.
Also, no needs to consider previously calculated permissions, as at
this point we already bind all new parents and bdrv_get_cumulative_perm
result is enough. So fix the bug by just set permissions by
bdrv_get_cumulative_perm result.

Bug was introduced in long ago 234ac1a9025, in 2.9.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
 {
     BdrvChild *c, *next;
     GSList *list = NULL, *p;
-    uint64_t old_perm, old_shared;
     uint64_t perm = 0, shared = BLK_PERM_ALL;
     int ret;
 
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
         bdrv_unref(from);
     }
 
-    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
-    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
+    bdrv_get_cumulative_perm(to, &perm, &shared);
+    bdrv_set_perm(to, perm, shared);
 
 out:
     g_slist_free(list);
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

When creating an image with preallocation "off" or "falloc", the first
block of the image is typically not allocated. When using Gluster
storage backed by XFS filesystem, reading this block using direct I/O
succeeds regardless of request length, fooling alignment detection.

In this case we fallback to a safe value (4096) instead of the optimal
value (512), which may lead to unneeded data copying when aligning
requests.  Allocating the first block avoids the fallback.

Since we allocate the first block even with preallocation=off, we no
longer create images with zero disk size:

$ ./qemu-img create -f raw test.raw 1g
    Formatting 'test.raw', fmt=raw size=1073741824

$ ls -lhs test.raw
    4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw

And converting the image requires additional cluster:

$ ./qemu-img measure -f raw -O qcow2 test.raw
    required size: 458752
    fully allocated size: 1074135040

When using format like vmdk with multiple files per image, we allocate
one block per file:

$ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
    Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat

$ ls -lhs test*.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk

I did quick performance test for copying disks with qemu-img convert to
new raw target image to Gluster storage with sector size of 512 bytes:

for i in $(seq 10); do
        rm -f dst.raw
        sleep 10
        time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
    done

Here is a table comparing the total time spent:

Type    Before(s)   After(s)    Diff(%)
---------------------------------------
real      530.028    469.123      -11.4
user       17.204     10.768      -37.4
sys        17.881      7.011      -60.7

We can see very clear improvement in CPU usage.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190827010528.8818-2-nsoffer@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c                            | 51 +++++++++++++++++++
 tests/qemu-iotests/059.out                    |  2 +-
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++++
 tests/qemu-iotests/175                        | 19 ++++---
 tests/qemu-iotests/175.out                    |  8 +--
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/221.out                    | 12 +++--
 tests/qemu-iotests/253.out                    | 12 +++--
 9 files changed, 99 insertions(+), 21 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
     return ret;
 }
 
+/*
+ * Help alignment probing by allocating the first block.
+ *
+ * When reading with direct I/O from unallocated area on Gluster backed by XFS,
+ * reading succeeds regardless of request length. In this case we fallback to
+ * safe alignment which is not optimal. Allocating the first block avoids this
+ * fallback.
+ *
+ * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
+ * request alignment, so we use safe values.
+ *
+ * Returns: 0 on success, -errno on failure. Since this is an optimization,
+ * caller may ignore failures.
+ */
+static int allocate_first_block(int fd, size_t max_size)
+{
+    size_t write_size = (max_size < MAX_BLOCKSIZE)
+        ? BDRV_SECTOR_SIZE
+        : MAX_BLOCKSIZE;
+    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
+    void *buf;
+    ssize_t n;
+    int ret;
+
+    buf = qemu_memalign(max_align, write_size);
+    memset(buf, 0, write_size);
+
+    do {
+        n = pwrite(fd, buf, write_size, 0);
+    } while (n == -1 && errno == EINTR);
+
+    ret = (n == -1) ? -errno : 0;
+
+    qemu_vfree(buf);
+    return ret;
+}
+
 static int handle_aiocb_truncate(void *opaque)
 {
     RawPosixAIOData *aiocb = opaque;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
                 /* posix_fallocate() doesn't set errno. */
                 error_setg_errno(errp, -result,
                                  "Could not preallocate new data");
+            } else if (current_length == 0) {
+                /*
+                 * posix_fallocate() uses fallocate() if the filesystem
+                 * supports it, or fallback to manually writing zeroes. If
+                 * fallocate() was used, unaligned reads from the fallocated
+                 * area in raw_probe_alignment() will succeed, hence we need to
+                 * allocate the first block.
+                 *
+                 * Optimize future alignment probing; ignore failures.
+                 */
+                allocate_first_block(fd, offset);
             }
         } else {
             result = 0;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
         if (ftruncate(fd, offset) != 0) {
             result = -errno;
             error_setg_errno(errp, -result, "Could not resize file");
+        } else if (current_length == 0 && offset > current_length) {
+            /* Optimize future alignment probing; ignore failures. */
+            allocate_first_block(fd, offset);
         }
         return result;
     default:
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
-disk size: 16 KiB
+disk size: 1.97 MiB
 Format specific information:
     cid: XXXXXXXX
     parent cid: XXXXXXXX
diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
similarity index 100%
rename from tests/qemu-iotests/150.out
rename to tests/qemu-iotests/150.out.qcow2
diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/150.out.raw
@@ -XXX,XX +XXX,XX @@
+QA output created by 150
+
+=== Mapping sparse conversion ===
+
+Offset          Length          File
+0               0x1000          TEST_DIR/t.IMGFMT
+
+=== Mapping non-sparse conversion ===
+
+Offset          Length          File
+0               0x100000        TEST_DIR/t.IMGFMT
+*** done
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # the file size.  This function hides the resulting difference in the
 # stat -c '%b' output.
 # Parameter 1: Number of blocks an empty file occupies
-# Parameter 2: Image size in bytes
+# Parameter 2: Minimal number of blocks in an image
+# Parameter 3: Image size in bytes
 _filter_blocks()
 {
     extra_blocks=$1
-    img_size=$2
+    min_blocks=$2
+    img_size=$3
 
-    sed -e "s/blocks=$extra_blocks\$\$\\|[^0-9]\$/nothing allocated/" \
-        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/everything allocated/"
+    sed -e "s/blocks=$min_blocks\$\$\\|[^0-9]\$/min allocation/" \
+        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
 # get standard environment, filters and checks
@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
 touch "$TEST_DIR/empty"
 extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 
+# We always write the first byte; check how many blocks this filesystem
+# allocates to match empty image alloation.
+printf "\0" > "$TEST_DIR/empty"
+min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
+
 echo
 echo "== creating image with default preallocation =="
 _make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 
 for mode in off full falloc; do
     echo
     echo "== creating image with preallocation $mode =="
     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
-    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
 # success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
 
 == creating image with default preallocation ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation off ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation full ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, everything allocated
+size=1048576, max allocation
 
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, everything allocated
+size=1048576, max allocation
  *** done
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 == raw input image with data (human) ==
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
-required size: 393216
+required size: 458752
 fully allocated size: 1074135040
 wrote 512/512 bytes at offset 512
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
 {
-    "required": 393216,
+    "required": 458752,
     "fully-allocated": 1074135040
 }
 wrote 512/512 bytes at offset 512
diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/221.out
+++ b/tests/qemu-iotests/221.out
@@ -XXX,XX +XXX,XX @@ QA output created by 221
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 1/1 bytes at offset 65536
 1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 *** done
diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/253.out
+++ b/tests/qemu-iotests/253.out
@@ -XXX,XX +XXX,XX @@ QA output created by 253
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 65535/65535 bytes at offset 983040
 63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Using block_resize we can test allocate_first_block() with file
descriptor opened with O_DIRECT, ensuring that it works for any size
larger than 4096 bytes.

Testing smaller sizes is tricky as the result depends on the filesystem
used for testing. For example on NFS any size will work since O_DIRECT
does not require any alignment.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190827010528.8818-3-nsoffer@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  8 ++++++++
 2 files changed, 36 insertions(+)

diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ _filter_blocks()
         -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
+# Resize image using block_resize.
+# Parameter 1: image path
+# Parameter 2: new size
+_block_resize()
+{
+    local path=$1
+    local size=$2
+
+    $QEMU -qmp stdio -nographic -nodefaults \
+        -blockdev file,node-name=file,filename=$path,cache.direct=on \
+        <<EOF
+{'execute': 'qmp_capabilities'}
+{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
+{'execute': 'quit'}
+EOF
+}
+
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
 _supported_proto file
 _supported_os Linux
 
+_default_cache_mode none
+_supported_cache_modes none directsync
+
 size=$((1 * 1024 * 1024))
 
 touch "$TEST_DIR/empty"
@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
     stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
+for new_size in 4096 1048576; do
+    echo
+    echo "== resize empty image with block_resize =="
+    _make_test_img 0 | _filter_imgfmt
+    _block_resize $TEST_IMG $new_size >/dev/null
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
+done
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
 size=1048576, max allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=4096, min allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=1048576, min allocation
  *** done
-- 
2.21.0

fe646693acc changed qemu-img create's output so that it no longer prints
single quotes around parameter values.  The subformat and adapter_type
filters in _filter_img_create() have never been adapted to that change.

Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059.out       | 16 ++++++++--------
 tests/qemu-iotests/common.filter |  4 ++--
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
 
 === Testing monolithicFlat creation and opening ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 2 GiB (2147483648 bytes)
 
 === Testing monolithicFlat with zeroed_grain ===
 qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 
 === Testing big twoGbMaxExtentFlat ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
@@ -XXX,XX +XXX,XX @@ Format specific information:
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
 
 === Testing truncated sparse ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
 
 === Converting to streamOptimized from image with small cluster size===
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
 
 === Testing version 3 ===
@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing 4TB monolithicFlat creation and IO ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 4 TiB (4398046511104 bytes)
@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing qemu-img map on extents ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0               0x20000         0x3f0000        TEST_DIR/t.vmdk
 0x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 0x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
         -e "s# compat6=\$on\\|off\$##g" \
         -e "s# static=\$on\\|off\$##g" \
         -e "s# zeroed_grain=\$on\\|off\$##g" \
-        -e "s# subformat='[^']*'##g" \
-        -e "s# adapter_type='[^']*'##g" \
+        -e "s# subformat=[^ ]*##g" \
+        -e "s# adapter_type=[^ ]*##g" \
         -e "s# hwversion=[^ ]*##g" \
         -e "s# lazy_refcounts=\$on\\|off\$##g" \
         -e "s# block_size=[0-9]\\+##g" \
-- 
2.21.0

This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
turns a former error in 059 into success.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190815153638.4600-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
 tests/qemu-iotests/059     |  7 +++--
 tests/qemu-iotests/059.out |  4 ++-
 3 files changed, 42 insertions(+), 23 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
 }
 
 static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-                              const char *desc_file_path, QDict *options,
-                              Error **errp)
+                              QDict *options, Error **errp)
 {
     int ret;
     int matches;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
     const char *p, *np;
     int64_t sectors = 0;
     int64_t flat_offset;
+    char *desc_file_dir = NULL;
     char *extent_path;
     BdrvChild *extent_file;
     BDRVVmdkState *s = bs->opaque;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             continue;
         }
 
-        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
-            !desc_file_path[0])
-        {
-            bdrv_refresh_filename(bs->file->bs);
-            error_setg(errp, "Cannot use relative extent paths with VMDK "
-                       "descriptor file '%s'", bs->file->bs->filename);
-            return -EINVAL;
-        }
+        if (path_is_absolute(fname)) {
+            extent_path = g_strdup(fname);
+        } else {
+            if (!desc_file_dir) {
+                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
+                if (!desc_file_dir) {
+                    bdrv_refresh_filename(bs->file->bs);
+                    error_prepend(errp, "Cannot use relative paths with VMDK "
+                                  "descriptor file '%s': ",
+                                  bs->file->bs->filename);
+                    ret = -EINVAL;
+                    goto out;
+                }
+            }
 
-        extent_path = path_combine(desc_file_path, fname);
+            extent_path = g_strconcat(desc_file_dir, fname, NULL);
+        }
 
         ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
         assert(ret < 32);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
         g_free(extent_path);
         if (local_err) {
             error_propagate(errp, local_err);
-            return -EINVAL;
+            ret = -EINVAL;
+            goto out;
         }
 
         /* save to extents array */
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                             0, 0, 0, 0, 0, &extent, errp);
             if (ret < 0) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent->flat_start_offset = flat_offset << 9;
         } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             g_free(buf);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else if (!strcmp(type, "SESPARSE")) {
             ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else {
             error_setg(errp, "Unsupported extent type '%s'", type);
             bdrv_unref_child(bs, extent_file);
-            return -ENOTSUP;
+            ret = -ENOTSUP;
+            goto out;
         }
         extent->type = g_strdup(type);
     }
-    return 0;
+
+    ret = 0;
+    goto out;
 
 invalid:
     np = next_line(p);
@@ -XXX,XX +XXX,XX @@ invalid:
         np--;
     }
     error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
-    return -EINVAL;
+    ret = -EINVAL;
+
+out:
+    g_free(desc_file_dir);
+    return ret;
 }
 
 static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
     }
     s->create_type = g_strdup(ct);
     s->desc_offset = 0;
-    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
-                             errp);
+    ret = vmdk_parse_extents(buf, bs, options, errp);
 exit:
     return ret;
 }
diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+# Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
-$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
-    | _filter_testdir | _filter_imgfmt
+$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
+         -c info \
+    2>&1 \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
 echo
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 
 === Testing monolithicFlat with internally generated JSON file name ===
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
+format name: IMGFMT
+cluster size: 0 bytes
+vm state offset: 0 bytes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

We had a test for a case where relative extent paths did not work, but
unfortunately we just fixed the underlying problem, so it works now.
This patch adds a new test case that still fails.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-4-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
 tests/qemu-iotests/059.out |  4 ++++
 2 files changed, 31 insertions(+)

diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+
+echo '--- blkdebug ---'
 # Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
     | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
+echo '--- quorum ---'
+# Should not work, because bdrv_dirname() does not work with quorum
+IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
+cp "$TEST_IMG" "$TEST_IMG.orig"
+
+filename="json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"quorum\",
+        \"children\": [ {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG\"
+        }, {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG.orig\"
+        } ],
+        \"vote-threshold\": 1
+    } }"
+
+filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
+$QEMU_IMG info "$filename" 2>&1 \
+    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
+
+
 echo
 echo "=== Testing version 3 ==="
 _use_sample_img iotest-version3.vmdk.bz2
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
+--- blkdebug ---
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 format name: IMGFMT
 cluster size: 0 bytes
 vm state offset: 0 bytes
+--- quorum ---
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

Compressed writes generally have to write full clusters, not just in
theory but also in practice when it comes to vmdk's streamOptimized
subformat.  It currently is just silently broken for writes with
non-zero in-cluster offsets:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 4096
4 KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
read failed: Invalid argument

(The technical reason is that vmdk_write_extent() just writes the
incomplete compressed data actually to offset 4k.  When reading the
data, vmdk_read_extent() looks at offset 0 and finds the compressed data
size to be 0, because that is what it reads from there.  This yields an
error.)

For incomplete writes with zero in-cluster offsets, the error path when
reading the rest of the cluster is a bit different, but the result is
the same:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 0
4 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
read failed: Invalid argument

(Here, vmdk_read_extent() finds the data and then sees that the
uncompressed data is short.)

It is better to reject invalid writes than to make the user believe they
might have succeeded and then fail when trying to read it back.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-5-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
     if (extent->compressed) {
         void *compressed_data;
 
+        /* Only whole clusters */
+        if (offset_in_cluster ||
+            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
+            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
+             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
+        {
+            ret = -EINVAL;
+            goto out;
+        }
+
         if (!extent->has_marker) {
             ret = -EINVAL;
             goto out;
-- 
2.21.0

streamOptimized does not support writes that do not span exactly one
cluster.  Furthermore, it cannot rewrite already allocated clusters.
As such, many iotests do not work with it.  Disable them.

diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/002
+++ b/tests/qemu-iotests/002
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/003
+++ b/tests/qemu-iotests/003
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 size=128M
 offset=67M
diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/005
+++ b/tests/qemu-iotests/005
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 _supported_os Linux
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 # vpc is limited to 127GB, so we can't test it here
 if [ "$IMGFMT" = "vpc" ]; then
diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/009
+++ b/tests/qemu-iotests/009
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/010
+++ b/tests/qemu-iotests/010
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/011
+++ b/tests/qemu-iotests/011
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto generic
 _unsupported_proto vxhs
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/018
+++ b/tests/qemu-iotests/018
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _supported_os Linux
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/019
+++ b/tests/qemu-iotests/019
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 CLUSTER_SIZE=65536
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/027
+++ b/tests/qemu-iotests/027
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vmdk qcow qcow2 qed
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/032
+++ b/tests/qemu-iotests/032
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # This works for any image format (though unlikely to segfault for raw)
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo === Prepare image ===
diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/033
+++ b/tests/qemu-iotests/033
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/034
+++ b/tests/qemu-iotests/034
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/037
+++ b/tests/qemu-iotests/037
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/063
+++ b/tests/qemu-iotests/063
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 _make_test_img 4M
 
diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/072
+++ b/tests/qemu-iotests/072
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
 _supported_proto file
+_unsupported_imgopts "subformat=streamOptimized"
 
 IMG_SIZE=64M
 
diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/105
+++ b/tests/qemu-iotests/105
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2 vmdk vhdx qed
 _supported_proto generic
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 echo
 echo "creating large image"
diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/197
+++ b/tests/qemu-iotests/197
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/215
+++ b/tests/qemu-iotests/215
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/251
+++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _supported_os Linux
+_unsupported_imgopts "subformat=streamOptimized"
 
 if [ "$IMGOPTSSYNTAX" = "true" ]; then
     # We use json:{} filenames here, so we cannot work with additional options.
-- 
2.21.0

The error message for the test case where we have a quorum node for
which no directory name can be generated is different: For
twoGbMaxExtentSparse, it complains that it cannot open the extent file.
For other (sub)formats, it just notes that it cannot determine the
backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
for simplicity's sake.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-7-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/110 | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Any format supporting backing files
 _supported_fmt qed qcow qcow2 vmdk
 _supported_proto file
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=twoGbMaxExtentSparse"
 
 TEST_IMG_REL=$(basename "$TEST_IMG")
 
-- 
2.21.0

From: Stefan Hajnoczi <stefanha@redhat.com>

Fixes: a6b257a08e3d72219f03e461a52152672fec0612
       ("file-posix: Handle undetectable alignment")
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190827101328.4062-1-stefanha@redhat.com
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
             align = alignments[i];
             if (raw_is_io_aligned(fd, buf + align, max_align)) {
-                /* Fallback to request_aligment. */
+                /* Fallback to request_alignment. */
                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
                 break;
             }
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

It is possible to enable only a subset of the block drivers with the
"--block-drv-rw-whitelist" option of the "configure" script. All other
drivers are marked as unusable (or only included as read-only with the
"--block-drv-ro-whitelist" option). If an iotest is now using such a
disabled block driver, it is failing - which is bad, since at least the
tests in the "auto" group should be able to deal with this situation.
Thus let's introduce a "_require_drivers" function that can be used by
the shell tests to check for the availability of certain drivers first,
and marks the test as "not run" if one of the drivers is missing.

This patch mainly targets the test in the "auto" group which should
never fail in such a case, but also improves some of the other tests
along the way. Note that we also assume that the "qcow2" and "file"
drivers are always available - otherwise it does not make sense to
run "make check-block" at all (which only tests with qcow2 by default).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823133552.11680-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/071       |  1 +
 tests/qemu-iotests/081       |  4 +---
 tests/qemu-iotests/099       |  1 +
 tests/qemu-iotests/120       |  1 +
 tests/qemu-iotests/162       |  4 +---
 tests/qemu-iotests/184       |  1 +
 tests/qemu-iotests/186       |  1 +
 tests/qemu-iotests/common.rc | 14 ++++++++++++++
 8 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/071
+++ b/tests/qemu-iotests/071
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers blkdebug blkverify
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/081
+++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt raw
 _supported_proto file
 _supported_os Linux
+_require_drivers quorum
 
 do_run_qemu()
 {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                           | _filter_qemu_io | _filter_generated_node_ids
 }
 
-test_quorum=$($QEMU_IMG --help|grep quorum)
-[ "$test_quorum" = "" ] && _supported_fmt quorum
-
 quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
 quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
 quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/099
+++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
 _supported_proto file
 _supported_os Linux
+_require_drivers blkdebug blkverify
 _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
     "subformat=twoGbMaxExtentSparse"
 
diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/120
+++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _unsupported_fmt luks
+_require_drivers raw
 
 _make_test_img 64M
 
diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/162
+++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
-
-test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh$ \|$$')
-[ "$test_ssh" = "" ] && _notrun "ssh support required"
+_require_drivers ssh
 
 echo
 echo '=== NBD ==='
diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/184
+++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_os Linux
+_require_drivers throttle
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/186
+++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers null-co
 
 if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
     _notrun "Requires a PC machine"
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -XXX,XX +XXX,XX @@ _require_command()
     [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
 }
 
+# Check that a set of drivers has been whitelisted in the QEMU binary
+#
+_require_drivers()
+{
+    available=$($QEMU -drive format=help | \
+                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
+    for driver
+    do
+        if ! echo "$available" | grep -q " $driver$ \|$$"; then
+            _notrun "$driver not available"
+        fi
+    done
+}
+
 # make sure this script returns success
 true
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

The sanitizers (especially the address sanitizer from Clang) are
sometimes printing out warnings or false positives - this spoils
the output of the iotests, causing some of the tests to fail.
Thus let's skip the automatic iotests during "make check" when the
user configured QEMU with --enable-sanitizers.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823084203.29734-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/check-block.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tests/check-block.sh b/tests/check-block.sh
index XXXXXXX..XXXXXXX 100755
--- a/tests/check-block.sh
+++ b/tests/check-block.sh
@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
     exit 0
 fi
 
+if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
+    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
+    exit 0
+fi
+
 if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
     echo "No qemu-system binary available ==> Not running the qemu-iotests."
     exit 0
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Quoting cache mode is not needed, and most tests use unquoted values.
Unify all test to use the same style.

Message-id: 20190827173432.7656-1-nsoffer@redhat.com
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/026 | 4 ++--
 tests/qemu-iotests/039 | 4 ++--
 tests/qemu-iotests/052 | 2 +-
 tests/qemu-iotests/091 | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/026
+++ b/tests/qemu-iotests/026
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Currently only qcow2 supports rebasing
 _supported_fmt qcow2
 _supported_proto file
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough" "none"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough none
 # The refcount table tests expect a certain minimum width for refcount entries
 # (so that the refcount table actually needs to grow); that minimum is 16 bits,
 # being the default refcount entry width.
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough
 
 size=128M
 
diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/052
+++ b/tests/qemu-iotests/052
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto file
 
 # Don't do O_DIRECT on tmpfs
-_supported_cache_modes "writeback" "writethrough" "unsafe"
+_supported_cache_modes writeback writethrough unsafe
 
 size=128M
 _make_test_img $size
diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/091
+++ b/tests/qemu-iotests/091
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "none"
-_supported_cache_modes "writethrough" "none" "writeback"
+_default_cache_mode none
+_supported_cache_modes writethrough none writeback
 
 size=1G
 
-- 
2.21.0