Series comparison

-[Qemu-devel] [PULL 00/13] Block patches
+[Qemu-devel] [PULL 00/15] Block patches
-The following changes since commit b4fbe1f65a4769c09e6bf2d79fc84360f840f40e:
+The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:
-  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20190129' into staging (2019-01-29 12:00:19 +0000)
+  Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)
 are available in the Git repository at:
-  https://git.xanclic.moe/XanClic/qemu.git tags/pull-block-2019-01-31
+  https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27
-for you to fetch changes up to 908b30164bbffad7430d551b2a03a8fbcaa631ef:
+for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:
-  iotests: Allow 147 to be run concurrently (2019-01-31 00:44:55 +0100)
+  iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)
 ----------------------------------------------------------------
 Block patches:
-- New debugging QMP command to explore block graphs
+- qemu-io now accepts a file to read a write pattern from
-- Converted DPRINTF()s to trace events
+- Ensure that raw files have their first block allocated so we can probe
-- Fixed qemu-io's use of getopt() for systems with optreset
+  the O_DIRECT alignment if necessary
-- Minor NVMe emulation fixes
+- Various fixes
 - An iotest fix
 ----------------------------------------------------------------
-Laurent Vivier (4):
+Denis Plotnikov (1):
-  block/ssh: Convert from DPRINTF() macro to trace events
+  qemu-io: add pattern file for write command
   block/curl: Convert from DPRINTF() macro to trace events
   block/file-posix: Convert from DPRINTF() macro to trace events
   block/sheepdog: Convert from DPRINTF() macro to trace events
-Li Qiang (3):
+Max Reitz (7):
-  nvme: use TYPE_NVME instead of constant string
+  iotests: Fix _filter_img_create()
-  nvme: ensure the num_queues is not zero
+  vmdk: Use bdrv_dirname() for relative extent paths
-  nvme: use pci_dev directly in nvme_realize
+  iotests: Keep testing broken relative extent paths
   vmdk: Reject invalid compressed writes
   iotests: Disable broken streamOptimized tests
   iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
   iotests: Disable 126 for flat vmdk subformats
-Max Reitz (3):
+Nir Soffer (3):
-  iotests.py: Add qemu_nbd_pipe()
+  block: posix: Always allocate the first block
-  iotests: Bind qemu-nbd to localhost in 147
+  iotests: Test allocate_first_block() with O_DIRECT
-  iotests: Allow 147 to be run concurrently
+  iotests: Unify cache mode quoting
-Richard W.M. Jones (1):
+Stefan Hajnoczi (1):
-  qemu-io: Add generic function for reinitializing optind.
+  file-posix: fix request_alignment typo
-Vladimir Sementsov-Ogievskiy (2):
+Thomas Huth (2):
-  qapi: add x-debug-query-block-graph
+  iotests: Check for enabled drivers before testing them
-  scripts: add render_block_graph function for QEMUMachine
+  tests/check-block: Skip iotests when sanitizers are enabled
- configure                      |  14 ++++
+Vladimir Sementsov-Ogievskiy (1):
- qapi/block-core.json           | 108 ++++++++++++++++++++++++
+  block: fix permission update in bdrv_replace_node
- include/block/block.h          |   1 +
- include/qemu/osdep.h           |  16 ++++
+ block.c                                       |  5 +-
- include/sysemu/block-backend.h |   2 +
+ block/file-posix.c                            | 53 +++++++++-
- block.c                        | 148 +++++++++++++++++++++++++++++++++
+ block/vmdk.c                                  | 64 ++++++++----
- block/block-backend.c          |   5 ++
+ qemu-io-cmds.c                                | 99 +++++++++++++++++--
- block/curl.c                   |  29 ++-----
+ tests/check-block.sh                          |  5 +
- block/file-posix.c             |  25 ++----
+ tests/qemu-iotests/002                        |  1 +
- block/sheepdog.c               |  47 ++++-------
+ tests/qemu-iotests/003                        |  1 +
- block/ssh.c                    |  46 ++++------
+ tests/qemu-iotests/005                        |  3 +-
- blockdev.c                     |   5 ++
+ tests/qemu-iotests/009                        |  1 +
- hw/block/nvme.c                |  15 ++--
+ tests/qemu-iotests/010                        |  1 +
- qemu-img.c                     |   2 +-
+ tests/qemu-iotests/011                        |  1 +
- qemu-io-cmds.c                 |   2 +-
+ tests/qemu-iotests/017                        |  3 +-
- block/trace-events             |  47 +++++++++++
+ tests/qemu-iotests/018                        |  3 +-
- scripts/render_block_graph.py  | 120 ++++++++++++++++++++++++++
+ tests/qemu-iotests/019                        |  3 +-
- tests/qemu-iotests/147         |  98 +++++++++++++++-------
+ tests/qemu-iotests/020                        |  3 +-
- tests/qemu-iotests/iotests.py  |  14 ++++
+ tests/qemu-iotests/026                        |  4 +-
-files changed, 608 insertions(+), 136 deletions(-)
+ tests/qemu-iotests/027                        |  1 +
- create mode 100755 scripts/render_block_graph.py
+ tests/qemu-iotests/032                        |  1 +
  tests/qemu-iotests/033                        |  1 +
  tests/qemu-iotests/034                        |  3 +-
  tests/qemu-iotests/037                        |  3 +-
  tests/qemu-iotests/039                        |  4 +-
  tests/qemu-iotests/052                        |  2 +-
  tests/qemu-iotests/059                        | 34 ++++++-
  tests/qemu-iotests/059.out                    | 26 +++--
  tests/qemu-iotests/063                        |  3 +-
  tests/qemu-iotests/071                        |  1 +
  tests/qemu-iotests/072                        |  1 +
  tests/qemu-iotests/081                        |  4 +-
  tests/qemu-iotests/091                        |  4 +-
  tests/qemu-iotests/099                        |  1 +
  tests/qemu-iotests/105                        |  3 +-
  tests/qemu-iotests/110                        |  3 +-
  tests/qemu-iotests/120                        |  1 +
  tests/qemu-iotests/126                        |  2 +
  tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
  tests/qemu-iotests/150.out.raw                | 12 +++
  tests/qemu-iotests/162                        |  4 +-
  tests/qemu-iotests/175                        | 47 +++++++--
  tests/qemu-iotests/175.out                    | 16 ++-
  tests/qemu-iotests/178.out.qcow2              |  4 +-
  tests/qemu-iotests/184                        |  1 +
  tests/qemu-iotests/186                        |  1 +
  tests/qemu-iotests/197                        |  1 +
  tests/qemu-iotests/215                        |  1 +
  tests/qemu-iotests/221.out                    | 12 ++-
  tests/qemu-iotests/251                        |  1 +
  tests/qemu-iotests/253.out                    | 12 ++-
  tests/qemu-iotests/common.filter              |  4 +-
  tests/qemu-iotests/common.rc                  | 14 +++
 files changed, 391 insertions(+), 87 deletions(-)
  rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
  create mode 100644 tests/qemu-iotests/150.out.raw
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 06/13] block/sheepdog: Convert from DPRINTF() macro to trace events
+[Qemu-devel] [PULL 01/15] qemu-io: add pattern file for write command
-From: Laurent Vivier <lvivier@redhat.com>
+From: Denis Plotnikov <dplotnikov@virtuozzo.com>
-Signed-off-by: Laurent Vivier <lvivier@redhat.com>
+The patch allows to provide a pattern file for write
-Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
+command. There was no similar ability before.
-Message-id: 20181213162727.17438-5-lvivier@redhat.com
-[mreitz: Fixed sheepdog_snapshot_create_inode's format string to use
+Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
-         PRIx32 for uint32_ts]
+Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
 Reviewed-by: Eric Blake <eblake@redhat.com>
 [mreitz: Keep optstring in alphabetical order]
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/sheepdog.c   | 47 +++++++++++++++++-----------------------------
+ qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
- block/trace-events | 14 ++++++++++++++
+file changed, 93 insertions(+), 6 deletions(-)
 files changed, 31 insertions(+), 30 deletions(-)
-diff --git a/block/sheepdog.c b/block/sheepdog.c
+diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/sheepdog.c
+--- a/qemu-io-cmds.c
-+++ b/block/sheepdog.c
++++ b/qemu-io-cmds.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
- #include "sysemu/block-backend.h"
+     qemu_vfree(p);
  #include "qemu/bitops.h"
  #include "qemu/cutils.h"
 +#include "trace.h"
  #define SD_PROTO_VER 0x01
@@ -XXX,XX +XXX,XX @@ static inline size_t count_data_objs(const struct SheepdogInode *inode)
                          (1UL << inode->block_size_shift));
  }
--#undef DPRINTF
++/*
--#ifdef DEBUG_SDOG
++ * qemu_io_alloc_from_file()
--#define DEBUG_SDOG_PRINT 1
++ *
--#else
++ * Allocates the buffer and populates it with the content of the given file
--#define DEBUG_SDOG_PRINT 0
++ * up to @len bytes. If the file length is less than @len, then the buffer
--#endif
++ * is populated with the file content cyclically.
--#define DPRINTF(fmt, args...)                                           \
++ *
--    do {                                                                \
++ * @blk - the block backend where the buffer content is going to be written to
--        if (DEBUG_SDOG_PRINT) {                                         \
++ * @len - the buffer length
--            fprintf(stderr, "%s %d: " fmt, __func__, __LINE__, ##args); \
++ * @file_name - the file to read the content from
--        }                                                               \
++ *
--    } while (0)
++ * Returns: the buffer pointer on success
--
++ *          NULL on error
- typedef struct SheepdogAIOCB SheepdogAIOCB;
++ */
- typedef struct BDRVSheepdogState BDRVSheepdogState;
++static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
++                                     const char *file_name)
-@@ -XXX,XX +XXX,XX @@ static coroutine_fn void reconnect_to_sdog(void *opaque)
++{
-         Error *local_err = NULL;
++    char *buf, *buf_origin;
-         s->fd = get_sheep_fd(s, &local_err);
++    FILE *f = fopen(file_name, "r");
-         if (s->fd < 0) {
++    int pattern_len;
--            DPRINTF("Wait for connection to be established\n");
++
-+            trace_sheepdog_reconnect_to_sdog();
++    if (!f) {
-             error_report_err(local_err);
++        perror(file_name);
-             qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000000ULL);
++        return NULL;
-         }
++    }
-@@ -XXX,XX +XXX,XX @@ static void coroutine_fn aio_read_response(void *opaque)
++
-         break;
++    if (qemuio_misalign) {
-     case AIOCB_FLUSH_CACHE:
++        len += MISALIGN_OFFSET;
-         if (rsp.result == SD_RES_INVALID_PARMS) {
++    }
--            DPRINTF("disable cache since the server doesn't support it\n");
++
-+            trace_sheepdog_aio_read_response();
++    buf_origin = buf = blk_blockalign(blk, len);
-             s->cache_flags = SD_FLAG_CMD_DIRECT;
++
-             rsp.result = SD_RES_SUCCESS;
++    if (qemuio_misalign) {
-         }
++        buf_origin += MISALIGN_OFFSET;
-@@ -XXX,XX +XXX,XX @@ static int sd_open(BlockDriverState *bs, QDict *options, int flags,
++        buf += MISALIGN_OFFSET;
-     s->discard_supported = true;
++        len -= MISALIGN_OFFSET;
++    }
-     if (snap_id || tag[0]) {
++
--        DPRINTF("%" PRIx32 " snapshot inode was open.\n", vid);
++    pattern_len = fread(buf_origin, 1, len, f);
-+        trace_sheepdog_open(vid);
++
-         s->is_snapshot = true;
++    if (ferror(f)) {
-     }
++        perror(file_name);
++        goto error;
-@@ -XXX,XX +XXX,XX @@ static void sd_close(BlockDriverState *bs)
++    }
-     unsigned int wlen, rlen = 0;
++
-     int fd, ret;
++    if (pattern_len == 0) {
++        fprintf(stderr, "%s: file is empty\n", file_name);
--    DPRINTF("%s\n", s->name);
++        goto error;
-+    trace_sheepdog_close(s->name);
++    }
++
-     fd = connect_to_sdog(s, &local_err);
++    fclose(f);
-     if (fd < 0) {
++
-@@ -XXX,XX +XXX,XX @@ static int sd_create_branch(BDRVSheepdogState *s)
++    if (len > pattern_len) {
-     char *buf;
++        len -= pattern_len;
-     bool deleted;
++        buf += pattern_len;
++
--    DPRINTF("%" PRIx32 " is snapshot.\n", s->inode.vdi_id);
++        while (len > 0) {
-+    trace_sheepdog_create_branch_snapshot(s->inode.vdi_id);
++            size_t len_to_copy = MIN(pattern_len, len);
++
-     buf = g_malloc(SD_INODE_SIZE);
++            memcpy(buf, buf_origin, len_to_copy);
++
-@@ -XXX,XX +XXX,XX @@ static int sd_create_branch(BDRVSheepdogState *s)
++            len -= len_to_copy;
-         goto out;
++            buf += len_to_copy;
-     }
++        }
++    }
--    DPRINTF("%" PRIx32 " is created.\n", vid);
++
-+    trace_sheepdog_create_branch_created(vid);
++    return buf_origin;
++
-     fd = connect_to_sdog(s, &local_err);
++error:
-     if (fd < 0) {
++    qemu_io_free(buf_origin);
-@@ -XXX,XX +XXX,XX @@ static int sd_create_branch(BDRVSheepdogState *s)
++    return NULL;
++}
-     s->is_snapshot = false;
++
-     ret = 0;
+ static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
--    DPRINTF("%" PRIx32 " was newly created.\n", s->inode.vdi_id);
+ {
-+    trace_sheepdog_create_branch_new(s->inode.vdi_id);
+     uint64_t i;
+@@ -XXX,XX +XXX,XX @@ static void write_help(void)
- out:
+ " -n, -- with -z, don't allow slow fallback\n"
-     g_free(buf);
+ " -p, -- ignored for backwards compatibility\n"
-@@ -XXX,XX +XXX,XX @@ static void coroutine_fn sd_co_rw_vector(SheepdogAIOCB *acb)
+ " -P, -- use different pattern to fill file\n"
-         }
++" -s, -- use a pattern file to fill the write buffer\n"
+ " -C, -- report statistics in a machine parsable format\n"
-         if (create) {
+ " -q, -- quiet mode, do not show I/O statistics\n"
--            DPRINTF("update ino (%" PRIu32 ") %" PRIu64 " %" PRIu64 " %ld\n",
+ " -u, -- with -z, allow unmapping\n"
--                    inode->vdi_id, oid,
+@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
--                    vid_to_data_oid(inode->data_vdi_id[idx], idx), idx);
+     .perm       = BLK_PERM_WRITE,
-+            trace_sheepdog_co_rw_vector_update(inode->vdi_id, oid,
+     .argmin     = 2,
-+                                  vid_to_data_oid(inode->data_vdi_id[idx], idx),
+     .argmax     = -1,
-+                                  idx);
+-    .args       = "[-bcCfnquz] [-P pattern] off len",
-             oid = vid_to_data_oid(inode->vdi_id, idx);
++    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
--            DPRINTF("new oid %" PRIx64 "\n", oid);
+     .oneline    = "writes a number of bytes at a specified offset",
-+            trace_sheepdog_co_rw_vector_new(oid);
+     .help       = write_help,
-         }
+ };
+@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
-         aio_req = alloc_aio_req(s, acb, oid, len, offset, flags, create,
+ {
-@@ -XXX,XX +XXX,XX @@ static int sd_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
+     struct timespec t1, t2;
-     SheepdogInode *inode;
+     bool Cflag = false, qflag = false, bflag = false;
-     unsigned int datalen;
+-    bool Pflag = false, zflag = false, cflag = false;
++    bool Pflag = false, zflag = false, cflag = false, sflag = false;
--    DPRINTF("sn_info: name %s id_str %s s: name %s vm_state_size %" PRId64 " "
+     int flags = 0;
--            "is_snapshot %d\n", sn_info->name, sn_info->id_str,
+     int c, cnt, ret;
--            s->name, sn_info->vm_state_size, s->is_snapshot);
+     char *buf = NULL;
-+    trace_sheepdog_snapshot_create_info(sn_info->name, sn_info->id_str, s->name,
+@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
-+                                        sn_info->vm_state_size, s->is_snapshot);
+     /* Some compilers get confused and warn if this is not initialized.  */
+     int64_t total = 0;
-     if (s->is_snapshot) {
+     int pattern = 0xcd;
-         error_report("You can't create a snapshot of a snapshot VDI, "
++    const char *file_name = NULL;
-@@ -XXX,XX +XXX,XX @@ static int sd_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
 -    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
 +    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
          switch (c) {
          case 'b':
              bflag = true;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
          case 'q':
              qflag = true;
              break;
 +        case 's':
 +            sflag = true;
 +            file_name = optarg;
 +            break;
          case 'u':
              flags |= BDRV_REQ_MAY_UNMAP;
              break;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
          return -EINVAL;
      }
--    DPRINTF("%s %s\n", sn_info->name, sn_info->id_str);
+-    if (zflag && Pflag) {
-+    trace_sheepdog_snapshot_create(sn_info->name, sn_info->id_str);
+-        printf("-z and -P cannot be specified at the same time\n");
++    if (zflag + Pflag + sflag > 1) {
-     s->inode.vm_state_size = sn_info->vm_state_size;
++        printf("Only one of -z, -P, and -s "
-     s->inode.vm_clock_nsec = sn_info->vm_clock_nsec;
++               "can be specified at the same time\n");
-@@ -XXX,XX +XXX,XX @@ static int sd_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
+         return -EINVAL;
      }
-     memcpy(&s->inode, inode, datalen);
+@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
--    DPRINTF("s->inode: name %s snap_id %x oid %x\n",
+     }
--            s->inode.name, s->inode.snap_id, s->inode.vdi_id);
-+    trace_sheepdog_snapshot_create_inode(s->inode.name, s->inode.snap_id,
+     if (!zflag) {
-+                                         s->inode.vdi_id);
+-        buf = qemu_io_alloc(blk, count, pattern);
++        if (sflag) {
- cleanup:
++            buf = qemu_io_alloc_from_file(blk, count, file_name);
-     g_free(inode);
++            if (!buf) {
-diff --git a/block/trace-events b/block/trace-events
++                return -EINVAL;
-index XXXXXXX..XXXXXXX 100644
++            }
---- a/block/trace-events
++        } else {
-+++ b/block/trace-events
++            buf = qemu_io_alloc(blk, count, pattern);
-@@ -XXX,XX +XXX,XX @@ file_xfs_discard(const char *error) "cannot punch hole (%s)"
++        }
- file_FindEjectableOpticalMedia(const char *media) "Matching using %s"
+     }
- file_setup_cdrom(const char *partition) "Using %s as optical disc"
- file_hdev_is_sg(int type, int version) "SG device found: type=%d, version=%d"
+     clock_gettime(CLOCK_MONOTONIC, &t1);
 +
 +# block/sheepdog.c
 +sheepdog_reconnect_to_sdog(void) "Wait for connection to be established"
 +sheepdog_aio_read_response(void) "disable cache since the server doesn't support it"
 +sheepdog_open(uint32_t vid) "0x%" PRIx32 " snapshot inode was open"
 +sheepdog_close(const char *name) "%s"
 +sheepdog_create_branch_snapshot(uint32_t vdi) "0x%" PRIx32 " is snapshot"
 +sheepdog_create_branch_created(uint32_t vdi) "0x%" PRIx32 " is created"
 +sheepdog_create_branch_new(uint32_t vdi) "0x%" PRIx32 " was newly created"
 +sheepdog_co_rw_vector_update(uint32_t vdi, uint64_t oid, uint64_t data, long idx) "update ino (%" PRIu32 ") %" PRIu64 " %" PRIu64 " %ld"
 +sheepdog_co_rw_vector_new(uint64_t oid) "new oid 0x%" PRIx64
 +sheepdog_snapshot_create_info(const char *sn_name, const char *id, const char *name, int64_t size, int is_snapshot) "sn_info: name %s id_str %s s: name %s vm_state_size %" PRId64 " " "is_snapshot %d"
 +sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
 +sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 01/13] qapi: add x-debug-query-block-graph
+[Qemu-devel] [PULL 02/15] block: fix permission update in bdrv_replace_node
 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Add a new command, returning block nodes (and their users) graph.
+It's wrong to OR shared permissions. It may lead to crash on further
 permission updates.
 Also, no needs to consider previously calculated permissions, as at
 this point we already bind all new parents and bdrv_get_cumulative_perm
 result is enough. So fix the bug by just set permissions by
 bdrv_get_cumulative_perm result.
 Bug was introduced in long ago 234ac1a9025, in 2.9.
 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20181221170909.25584-2-vsementsov@virtuozzo.com
+Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json           | 108 ++++++++++++++++++++++++
+ block.c | 5 ++---
- include/block/block.h          |   1 +
+file changed, 2 insertions(+), 3 deletions(-)
  include/sysemu/block-backend.h |   2 +
  block.c                        | 148 +++++++++++++++++++++++++++++++++
  block/block-backend.c          |   5 ++
  blockdev.c                     |   5 ++
 files changed, 269 insertions(+)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
-+++ b/qapi/block-core.json
-@@ -XXX,XX +XXX,XX @@
- ##
- { 'command': 'query-named-block-nodes', 'returns': [ 'BlockDeviceInfo' ] }
-+##
-+# @XDbgBlockGraphNodeType:
-+#
-+# @block-backend: corresponds to BlockBackend
-+#
-+# @block-job: corresonds to BlockJob
-+#
-+# @block-driver: corresponds to BlockDriverState
-+#
-+# Since: 4.0
-+##
-+{ 'enum': 'XDbgBlockGraphNodeType',
-+  'data': [ 'block-backend', 'block-job', 'block-driver' ] }
-+
-+##
-+# @XDbgBlockGraphNode:
-+#
-+# @id: Block graph node identifier. This @id is generated only for
-+#      x-debug-query-block-graph and does not relate to any other identifiers in
-+#      Qemu.
-+#
-+# @type: Type of graph node. Can be one of block-backend, block-job or
-+#        block-driver-state.
-+#
-+# @name: Human readable name of the node. Corresponds to node-name for
-+#        block-driver-state nodes; is not guaranteed to be unique in the whole
-+#        graph (with block-jobs and block-backends).
-+#
-+# Since: 4.0
-+##
-+{ 'struct': 'XDbgBlockGraphNode',
-+  'data': { 'id': 'uint64', 'type': 'XDbgBlockGraphNodeType', 'name': 'str' } }
-+
-+##
-+# @BlockPermission:
-+#
-+# Enum of base block permissions.
-+#
-+# @consistent-read: A user that has the "permission" of consistent reads is
-+#                   guaranteed that their view of the contents of the block
-+#                   device is complete and self-consistent, representing the
-+#                   contents of a disk at a specific point.
-+#                   For most block devices (including their backing files) this
-+#                   is true, but the property cannot be maintained in a few
-+#                   situations like for intermediate nodes of a commit block
-+#                   job.
-+#
-+# @write: This permission is required to change the visible disk contents.
-+#
-+# @write-unchanged: This permission (which is weaker than BLK_PERM_WRITE) is
-+#                   both enough and required for writes to the block node when
-+#                   the caller promises that the visible disk content doesn't
-+#                   change.
-+#                   As the BLK_PERM_WRITE permission is strictly stronger,
-+#                   either is sufficient to perform an unchanging write.
-+#
-+# @resize: This permission is required to change the size of a block node.
-+#
-+# @graph-mod: This permission is required to change the node that this
-+#             BdrvChild points to.
-+#
-+# Since: 4.0
-+##
-+  { 'enum': 'BlockPermission',
-+    'data': [ 'consistent-read', 'write', 'write-unchanged', 'resize',
-+              'graph-mod' ] }
-+##
-+# @XDbgBlockGraphEdge:
-+#
-+# Block Graph edge description for x-debug-query-block-graph.
-+#
-+# @parent: parent id
-+#
-+# @child: child id
-+#
-+# @name: name of the relation (examples are 'file' and 'backing')
-+#
-+# @perm: granted permissions for the parent operating on the child
-+#
-+# @shared-perm: permissions that can still be granted to other users of the
-+#               child while it is still attached to this parent
-+#
-+# Since: 4.0
-+##
-+{ 'struct': 'XDbgBlockGraphEdge',
-+  'data': { 'parent': 'uint64', 'child': 'uint64',
-+            'name': 'str', 'perm': [ 'BlockPermission' ],
-+            'shared-perm': [ 'BlockPermission' ] } }
-+
-+##
-+# @XDbgBlockGraph:
-+#
-+# Block Graph - list of nodes and list of edges.
-+#
-+# Since: 4.0
-+##
-+{ 'struct': 'XDbgBlockGraph',
-+  'data': { 'nodes': ['XDbgBlockGraphNode'], 'edges': ['XDbgBlockGraphEdge'] } }
-+
-+##
-+# @x-debug-query-block-graph:
-+#
-+# Get the block graph.
-+#
-+# Since: 4.0
-+##
-+{ 'command': 'x-debug-query-block-graph', 'returns': 'XDbgBlockGraph' }
-+
- ##
- # @drive-mirror:
- #
-diff --git a/include/block/block.h b/include/block/block.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block.h
-+++ b/include/block/block.h
-@@ -XXX,XX +XXX,XX @@ void bdrv_eject(BlockDriverState *bs, bool eject_flag);
- const char *bdrv_get_format_name(BlockDriverState *bs);
- BlockDriverState *bdrv_find_node(const char *node_name);
- BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp);
-+XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp);
- BlockDriverState *bdrv_lookup_bs(const char *device,
-                                  const char *node_name,
-                                  Error **errp);
-diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/sysemu/block-backend.h
-+++ b/include/sysemu/block-backend.h
-@@ -XXX,XX +XXX,XX @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in,
-                                    int bytes, BdrvRequestFlags read_flags,
-                                    BdrvRequestFlags write_flags);
-+const BdrvChild *blk_root(BlockBackend *blk);
-+
- #endif
 diff --git a/block.c b/block.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block.c
 +++ b/block.c
-@@ -XXX,XX +XXX,XX @@ BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp)
+@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
      return list;
  }
 +#define QAPI_LIST_ADD(list, element) do { \
 +    typeof(list) _tmp = g_new(typeof(*(list)), 1); \
 +    _tmp->value = (element); \
 +    _tmp->next = (list); \
 +    (list) = _tmp; \
 +} while (0)
 +
 +typedef struct XDbgBlockGraphConstructor {
 +    XDbgBlockGraph *graph;
 +    GHashTable *graph_nodes;
 +} XDbgBlockGraphConstructor;
 +
 +static XDbgBlockGraphConstructor *xdbg_graph_new(void)
 +{
 +    XDbgBlockGraphConstructor *gr = g_new(XDbgBlockGraphConstructor, 1);
 +
 +    gr->graph = g_new0(XDbgBlockGraph, 1);
 +    gr->graph_nodes = g_hash_table_new(NULL, NULL);
 +
 +    return gr;
 +}
 +
 +static XDbgBlockGraph *xdbg_graph_finalize(XDbgBlockGraphConstructor *gr)
 +{
 +    XDbgBlockGraph *graph = gr->graph;
 +
 +    g_hash_table_destroy(gr->graph_nodes);
 +    g_free(gr);
 +
 +    return graph;
 +}
 +
 +static uintptr_t xdbg_graph_node_num(XDbgBlockGraphConstructor *gr, void *node)
 +{
 +    uintptr_t ret = (uintptr_t)g_hash_table_lookup(gr->graph_nodes, node);
 +
 +    if (ret != 0) {
 +        return ret;
 +    }
 +
 +    /*
 +     * Start counting from 1, not 0, because 0 interferes with not-found (NULL)
 +     * answer of g_hash_table_lookup.
 +     */
 +    ret = g_hash_table_size(gr->graph_nodes) + 1;
 +    g_hash_table_insert(gr->graph_nodes, node, (void *)ret);
 +
 +    return ret;
 +}
 +
 +static void xdbg_graph_add_node(XDbgBlockGraphConstructor *gr, void *node,
 +                                XDbgBlockGraphNodeType type, const char *name)
 +{
 +    XDbgBlockGraphNode *n;
 +
 +    n = g_new0(XDbgBlockGraphNode, 1);
 +
 +    n->id = xdbg_graph_node_num(gr, node);
 +    n->type = type;
 +    n->name = g_strdup(name);
 +
 +    QAPI_LIST_ADD(gr->graph->nodes, n);
 +}
 +
 +static void xdbg_graph_add_edge(XDbgBlockGraphConstructor *gr, void *parent,
 +                                const BdrvChild *child)
 +{
 +    typedef struct {
 +        unsigned int flag;
 +        BlockPermission num;
 +    } PermissionMap;
 +
 +    static const PermissionMap permissions[] = {
 +        { BLK_PERM_CONSISTENT_READ, BLOCK_PERMISSION_CONSISTENT_READ },
 +        { BLK_PERM_WRITE,           BLOCK_PERMISSION_WRITE },
 +        { BLK_PERM_WRITE_UNCHANGED, BLOCK_PERMISSION_WRITE_UNCHANGED },
 +        { BLK_PERM_RESIZE,          BLOCK_PERMISSION_RESIZE },
 +        { BLK_PERM_GRAPH_MOD,       BLOCK_PERMISSION_GRAPH_MOD },
 +        { 0, 0 }
 +    };
 +    const PermissionMap *p;
 +    XDbgBlockGraphEdge *edge;
 +
 +    QEMU_BUILD_BUG_ON(1UL << (ARRAY_SIZE(permissions) - 1) != BLK_PERM_ALL + 1);
 +
 +    edge = g_new0(XDbgBlockGraphEdge, 1);
 +
 +    edge->parent = xdbg_graph_node_num(gr, parent);
 +    edge->child = xdbg_graph_node_num(gr, child->bs);
 +    edge->name = g_strdup(child->name);
 +
 +    for (p = permissions; p->flag; p++) {
 +        if (p->flag & child->perm) {
 +            QAPI_LIST_ADD(edge->perm, p->num);
 +        }
 +        if (p->flag & child->shared_perm) {
 +            QAPI_LIST_ADD(edge->shared_perm, p->num);
 +        }
 +    }
 +
 +    QAPI_LIST_ADD(gr->graph->edges, edge);
 +}
 +
 +
 +XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
 +{
 +    BlockBackend *blk;
 +    BlockJob *job;
 +    BlockDriverState *bs;
 +    BdrvChild *child;
 +    XDbgBlockGraphConstructor *gr = xdbg_graph_new();
 +
 +    for (blk = blk_all_next(NULL); blk; blk = blk_all_next(blk)) {
 +        char *allocated_name = NULL;
 +        const char *name = blk_name(blk);
 +
 +        if (!*name) {
 +            name = allocated_name = blk_get_attached_dev_id(blk);
 +        }
 +        xdbg_graph_add_node(gr, blk, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_BACKEND,
 +                           name);
 +        g_free(allocated_name);
 +        if (blk_root(blk)) {
 +            xdbg_graph_add_edge(gr, blk, blk_root(blk));
 +        }
 +    }
 +
 +    for (job = block_job_next(NULL); job; job = block_job_next(job)) {
 +        GSList *el;
 +
 +        xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
 +                           job->job.id);
 +        for (el = job->nodes; el; el = el->next) {
 +            xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
 +        }
 +    }
 +
 +    QTAILQ_FOREACH(bs, &graph_bdrv_states, node_list) {
 +        xdbg_graph_add_node(gr, bs, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_DRIVER,
 +                           bs->node_name);
 +        QLIST_FOREACH(child, &bs->children, next) {
 +            xdbg_graph_add_edge(gr, bs, child);
 +        }
 +    }
 +
 +    return xdbg_graph_finalize(gr);
 +}
 +
  BlockDriverState *bdrv_lookup_bs(const char *device,
                                   const char *node_name,
                                   Error **errp)
 diff --git a/block/block-backend.c b/block/block-backend.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/block-backend.c
 +++ b/block/block-backend.c
@@ -XXX,XX +XXX,XX @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in,
                                blk_out->root, off_out,
                                bytes, read_flags, write_flags);
  }
 +
 +const BdrvChild *blk_root(BlockBackend *blk)
 +{
 +    return blk->root;
 +}
 diff --git a/blockdev.c b/blockdev.c
 index XXXXXXX..XXXXXXX 100644
 --- a/blockdev.c
 +++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ BlockDeviceInfoList *qmp_query_named_block_nodes(Error **errp)
      return bdrv_named_nodes_list(errp);
  }
 +XDbgBlockGraph *qmp_x_debug_query_block_graph(Error **errp)
 +{
 +    return bdrv_get_xdbg_block_graph(errp);
 +}
 +
  BlockJob *do_blockdev_backup(BlockdevBackup *backup, JobTxn *txn,
                               Error **errp)
  {
+     BdrvChild *c, *next;
+     GSList *list = NULL, *p;
+-    uint64_t old_perm, old_shared;
+     uint64_t perm = 0, shared = BLK_PERM_ALL;
+     int ret;
+@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
+         bdrv_unref(from);
+     }
+-    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
+-    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
++    bdrv_get_cumulative_perm(to, &perm, &shared);
++    bdrv_set_perm(to, perm, shared);
+ out:
+     g_slist_free(list);
 --
-.20.1
+.21.0

-New patch
+[Qemu-devel] [PULL 03/15] block: posix: Always allocate the first block
+From: Nir Soffer <nirsof@gmail.com>
 When creating an image with preallocation "off" or "falloc", the first
 block of the image is typically not allocated. When using Gluster
 storage backed by XFS filesystem, reading this block using direct I/O
 succeeds regardless of request length, fooling alignment detection.
 In this case we fallback to a safe value (4096) instead of the optimal
 value (512), which may lead to unneeded data copying when aligning
 requests.  Allocating the first block avoids the fallback.
 Since we allocate the first block even with preallocation=off, we no
 longer create images with zero disk size:
     $ ./qemu-img create -f raw test.raw 1g
     Formatting 'test.raw', fmt=raw size=1073741824
     $ ls -lhs test.raw
 .0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
 And converting the image requires additional cluster:
     $ ./qemu-img measure -f raw -O qcow2 test.raw
     required size: 458752
     fully allocated size: 1074135040
 When using format like vmdk with multiple files per image, we allocate
 one block per file:
     $ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
     Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat
     $ ls -lhs test*.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk
 I did quick performance test for copying disks with qemu-img convert to
 new raw target image to Gluster storage with sector size of 512 bytes:
     for i in $(seq 10); do
         rm -f dst.raw
         sleep 10
         time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
     done
 Here is a table comparing the total time spent:
 Type    Before(s)   After(s)    Diff(%)
 ---------------------------------------
 real      530.028    469.123      -11.4
 user       17.204     10.768      -37.4
 sys        17.881      7.011      -60.7
 We can see very clear improvement in CPU usage.
 Signed-off-by: Nir Soffer <nsoffer@redhat.com>
 Message-id: 20190827010528.8818-2-nsoffer@redhat.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
  block/file-posix.c                            | 51 +++++++++++++++++++
  tests/qemu-iotests/059.out                    |  2 +-
  tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
  tests/qemu-iotests/150.out.raw                | 12 +++++
  tests/qemu-iotests/175                        | 19 ++++---
  tests/qemu-iotests/175.out                    |  8 +--
  tests/qemu-iotests/178.out.qcow2              |  4 +-
  tests/qemu-iotests/221.out                    | 12 +++--
  tests/qemu-iotests/253.out                    | 12 +++--
 files changed, 99 insertions(+), 21 deletions(-)
  rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
  create mode 100644 tests/qemu-iotests/150.out.raw
 diff --git a/block/file-posix.c b/block/file-posix.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/file-posix.c
 +++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
      return ret;
  }
 +/*
 + * Help alignment probing by allocating the first block.
 + *
 + * When reading with direct I/O from unallocated area on Gluster backed by XFS,
 + * reading succeeds regardless of request length. In this case we fallback to
 + * safe alignment which is not optimal. Allocating the first block avoids this
 + * fallback.
 + *
 + * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
 + * request alignment, so we use safe values.
 + *
 + * Returns: 0 on success, -errno on failure. Since this is an optimization,
 + * caller may ignore failures.
 + */
 +static int allocate_first_block(int fd, size_t max_size)
 +{
 +    size_t write_size = (max_size < MAX_BLOCKSIZE)
 +        ? BDRV_SECTOR_SIZE
 +        : MAX_BLOCKSIZE;
 +    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
 +    void *buf;
 +    ssize_t n;
 +    int ret;
 +
 +    buf = qemu_memalign(max_align, write_size);
 +    memset(buf, 0, write_size);
 +
 +    do {
 +        n = pwrite(fd, buf, write_size, 0);
 +    } while (n == -1 && errno == EINTR);
 +
 +    ret = (n == -1) ? -errno : 0;
 +
 +    qemu_vfree(buf);
 +    return ret;
 +}
 +
  static int handle_aiocb_truncate(void *opaque)
  {
      RawPosixAIOData *aiocb = opaque;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
                  /* posix_fallocate() doesn't set errno. */
                  error_setg_errno(errp, -result,
                                   "Could not preallocate new data");
 +            } else if (current_length == 0) {
 +                /*
 +                 * posix_fallocate() uses fallocate() if the filesystem
 +                 * supports it, or fallback to manually writing zeroes. If
 +                 * fallocate() was used, unaligned reads from the fallocated
 +                 * area in raw_probe_alignment() will succeed, hence we need to
 +                 * allocate the first block.
 +                 *
 +                 * Optimize future alignment probing; ignore failures.
 +                 */
 +                allocate_first_block(fd, offset);
              }
          } else {
              result = 0;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
          if (ftruncate(fd, offset) != 0) {
              result = -errno;
              error_setg_errno(errp, -result, "Could not resize file");
 +        } else if (current_length == 0 && offset > current_length) {
 +            /* Optimize future alignment probing; ignore failures. */
 +            allocate_first_block(fd, offset);
          }
          return result;
      default:
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/059.out
 +++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
  image: TEST_DIR/t.vmdk
  file format: vmdk
  virtual size: 0.977 TiB (1073741824000 bytes)
 -disk size: 16 KiB
 +disk size: 1.97 MiB
  Format specific information:
      cid: XXXXXXXX
      parent cid: XXXXXXXX
 diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
 similarity index 100%
 rename from tests/qemu-iotests/150.out
 rename to tests/qemu-iotests/150.out.qcow2
 diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/tests/qemu-iotests/150.out.raw
@@ -XXX,XX +XXX,XX @@
 +QA output created by 150
 +
 +=== Mapping sparse conversion ===
 +
 +Offset          Length          File
 +0               0x1000          TEST_DIR/t.IMGFMT
 +
 +=== Mapping non-sparse conversion ===
 +
 +Offset          Length          File
 +0               0x100000        TEST_DIR/t.IMGFMT
 +*** done
 diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/175
 +++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  # the file size.  This function hides the resulting difference in the
  # stat -c '%b' output.
  # Parameter 1: Number of blocks an empty file occupies
 -# Parameter 2: Image size in bytes
 +# Parameter 2: Minimal number of blocks in an image
 +# Parameter 3: Image size in bytes
  _filter_blocks()
  {
      extra_blocks=$1
 -    img_size=$2
 +    min_blocks=$2
 +    img_size=$3
 -    sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
 -        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
 +    sed -e "s/blocks=$min_blocks\\(\$\\|[^0-9]\\)/min allocation/" \
 +        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
  }
  # get standard environment, filters and checks
@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
  touch "$TEST_DIR/empty"
  extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 +# We always write the first byte; check how many blocks this filesystem
 +# allocates to match empty image alloation.
 +printf "\0" > "$TEST_DIR/empty"
 +min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 +
  echo
  echo "== creating image with default preallocation =="
  _make_test_img $size | _filter_imgfmt
 -stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 +stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  for mode in off full falloc; do
      echo
      echo "== creating image with preallocation $mode =="
      IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
 -    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 +    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  done
  # success, all done
 diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/175.out
 +++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
  == creating image with default preallocation ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
 -size=1048576, nothing allocated
 +size=1048576, min allocation
  == creating image with preallocation off ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
 -size=1048576, nothing allocated
 +size=1048576, min allocation
  == creating image with preallocation full ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
 -size=1048576, everything allocated
 +size=1048576, max allocation
  == creating image with preallocation falloc ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
 -size=1048576, everything allocated
 +size=1048576, max allocation
   *** done
 diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/178.out.qcow2
 +++ b/tests/qemu-iotests/178.out.qcow2
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
  == raw input image with data (human) ==
  Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
 -required size: 393216
 +required size: 458752
  fully allocated size: 1074135040
  wrote 512/512 bytes at offset 512
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
  Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
  {
 -    "required": 393216,
 +    "required": 458752,
      "fully-allocated": 1074135040
  }
  wrote 512/512 bytes at offset 512
 diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/221.out
 +++ b/tests/qemu-iotests/221.out
@@ -XXX,XX +XXX,XX @@ QA output created by 221
  === Check mapping of unaligned raw image ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  wrote 1/1 bytes at offset 65536
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
  { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
  { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  *** done
 diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/253.out
 +++ b/tests/qemu-iotests/253.out
@@ -XXX,XX +XXX,XX @@ QA output created by 253
  === Check mapping of unaligned raw image ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  wrote 65535/65535 bytes at offset 983040
 .999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
  *** done
 --
 .21.0

-New patch
+[Qemu-devel] [PULL 04/15] iotests: Test allocate_first_block() with O_DIRECT
+From: Nir Soffer <nirsof@gmail.com>
+Using block_resize we can test allocate_first_block() with file
+descriptor opened with O_DIRECT, ensuring that it works for any size
+larger than 4096 bytes.
+Testing smaller sizes is tricky as the result depends on the filesystem
+used for testing. For example on NFS any size will work since O_DIRECT
+does not require any alignment.
+Signed-off-by: Nir Soffer <nsoffer@redhat.com>
+Reviewed-by: Max Reitz <mreitz@redhat.com>
+Message-id: 20190827010528.8818-3-nsoffer@redhat.com
+Signed-off-by: Max Reitz <mreitz@redhat.com>
+---
+ tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
+ tests/qemu-iotests/175.out |  8 ++++++++
+files changed, 36 insertions(+)
+diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
+index XXXXXXX..XXXXXXX 100755
+--- a/tests/qemu-iotests/175
++++ b/tests/qemu-iotests/175
+@@ -XXX,XX +XXX,XX @@ _filter_blocks()
+         -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
+ }
++# Resize image using block_resize.
++# Parameter 1: image path
++# Parameter 2: new size
++_block_resize()
++{
++    local path=$1
++    local size=$2
++
++    $QEMU -qmp stdio -nographic -nodefaults \
++        -blockdev file,node-name=file,filename=$path,cache.direct=on \
++        <<EOF
++{'execute': 'qmp_capabilities'}
++{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
++{'execute': 'quit'}
++EOF
++}
++
+ # get standard environment, filters and checks
+ . ./common.rc
+ . ./common.filter
+@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
+ _supported_proto file
+ _supported_os Linux
++_default_cache_mode none
++_supported_cache_modes none directsync
++
+ size=$((1 * 1024 * 1024))
+ touch "$TEST_DIR/empty"
+@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
+     stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
+ done
++for new_size in 4096 1048576; do
++    echo
++    echo "== resize empty image with block_resize =="
++    _make_test_img 0 | _filter_imgfmt
++    _block_resize $TEST_IMG $new_size >/dev/null
++    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
++done
++
+ # success, all done
+ echo "*** done"
+ rm -f $seq.full
+diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
+index XXXXXXX..XXXXXXX 100644
+--- a/tests/qemu-iotests/175.out
++++ b/tests/qemu-iotests/175.out
+@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
+ == creating image with preallocation falloc ==
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
+ size=1048576, max allocation
++
++== resize empty image with block_resize ==
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
++size=4096, min allocation
++
++== resize empty image with block_resize ==
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
++size=1048576, min allocation
+  *** done
+--
+.21.0

-[Qemu-devel] [PULL 09/13] nvme: ensure the num_queues is not zero
+[Qemu-devel] [PULL 05/15] iotests: Fix _filter_img_create()
-From: Li Qiang <liq3ea@163.com>
+fe646693acc changed qemu-img create's output so that it no longer prints
 single quotes around parameter values.  The subformat and adapter_type
 filters in _filter_img_create() have never been adapted to that change.
-When it is zero, it causes segv.
+Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
-Using following command:
+Signed-off-by: Max Reitz <mreitz@redhat.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
-"-drive file=//home/test/test1.img,if=none,id=id0
+Message-id: 20190815153638.4600-2-mreitz@redhat.com
--device nvme,drive=id0,serial=test,num_queues=0"
+Reviewed-by: John Snow <jsnow@redhat.com>
 causes following Backtrack:
 Thread 4 "qemu-system-x86" received signal SIGSEGV, Segmentation fault.
 [Switching to Thread 0x7fffe9735700 (LWP 30952)]
 x0000555555a7a77c in nvme_start_ctrl (n=0x5555577473f0) at hw/block/nvme.c:825
         if (unlikely(n->cq[0])) {
 (gdb) bt
 0x0000555555a7a77c in nvme_start_ctrl (n=0x5555577473f0)
     at hw/block/nvme.c:825
 0x0000555555a7af7f in nvme_write_bar (n=0x5555577473f0, offset=20,
     data=4587521, size=4) at hw/block/nvme.c:969
 0x0000555555a7b81a in nvme_mmio_write (opaque=0x5555577473f0, addr=20,
     data=4587521, size=4) at hw/block/nvme.c:1163
 0x0000555555869236 in memory_region_write_accessor (mr=0x555557747cd0,
     addr=20, value=0x7fffe97320f8, size=4, shift=0, mask=4294967295, attrs=...)
     at /home/test/qemu1/qemu/memory.c:502
 0x0000555555869446 in access_with_adjusted_size (addr=20,
     value=0x7fffe97320f8, size=4, access_size_min=2, access_size_max=8,
     access_fn=0x55555586914d <memory_region_write_accessor>,
     mr=0x555557747cd0, attrs=...) at /home/test/qemu1/qemu/memory.c:568
 0x000055555586c479 in memory_region_dispatch_write (mr=0x555557747cd0,
     addr=20, data=4587521, size=4, attrs=...)
     at /home/test/qemu1/qemu/memory.c:1499
 0x00005555558030af in flatview_write_continue (fv=0x7fffe0061130,
     addr=4273930260, attrs=..., buf=0x7ffff7ff0028 "\001", len=4, addr1=20,
     l=4, mr=0x555557747cd0) at /home/test/qemu1/qemu/exec.c:3234
 0x00005555558031f9 in flatview_write (fv=0x7fffe0061130, addr=4273930260,
     attrs=..., buf=0x7ffff7ff0028 "\001", len=4)
     at /home/test/qemu1/qemu/exec.c:3273
 0x00005555558034ff in address_space_write (
 ---Type <return> to continue, or q <return> to quit---
     as=0x555556758480 <address_space_memory>, addr=4273930260, attrs=...,
     buf=0x7ffff7ff0028 "\001", len=4) at /home/test/qemu1/qemu/exec.c:3363
 0x0000555555803550 in address_space_rw (
     as=0x555556758480 <address_space_memory>, addr=4273930260, attrs=...,
     buf=0x7ffff7ff0028 "\001", len=4, is_write=true)
     at /home/test/qemu1/qemu/exec.c:3374
 0x00005555558884a1 in kvm_cpu_exec (cpu=0x555556920e40)
     at /home/test/qemu1/qemu/accel/kvm/kvm-all.c:2031
 0x000055555584cd9d in qemu_kvm_cpu_thread_fn (arg=0x555556920e40)
     at /home/test/qemu1/qemu/cpus.c:1281
 0x0000555555dbaf6d in qemu_thread_start (args=0x5555569438a0)
     at util/qemu-thread-posix.c:502
 0x00007ffff5dc86db in start_thread (arg=0x7fffe9735700)
     at pthread_create.c:463
 0x00007ffff5af188f in clone ()
     at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 Signed-off-by: Li Qiang <liq3ea@163.com>
 Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
 Message-id: 20190120055558.32984-3-liq3ea@163.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- hw/block/nvme.c | 5 +++++
+ tests/qemu-iotests/059.out       | 16 ++++++++--------
-file changed, 5 insertions(+)
+ tests/qemu-iotests/common.filter |  4 ++--
 files changed, 10 insertions(+), 10 deletions(-)
-diff --git a/hw/block/nvme.c b/hw/block/nvme.c
+diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
---- a/hw/block/nvme.c
+--- a/tests/qemu-iotests/059.out
-+++ b/hw/block/nvme.c
++++ b/tests/qemu-iotests/059.out
-@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
+@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-     int64_t bs_size;
+ qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
-     uint8_t *pci_conf;
+ === Testing monolithicFlat creation and opening ===
-+    if (!n->num_queues) {
+-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
-+        error_setg(errp, "num_queues can't be zero");
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
-+        return;
+ image: TEST_DIR/t.IMGFMT
-+    }
+ file format: IMGFMT
-+
+ virtual size: 2 GiB (2147483648 bytes)
-     if (!n->conf.blk) {
-         error_setg(errp, "drive property not set");
+ === Testing monolithicFlat with zeroed_grain ===
-         return;
+ qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
  === Testing big twoGbMaxExtentFlat ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
  image: TEST_DIR/t.vmdk
  file format: vmdk
  virtual size: 0.977 TiB (1073741824000 bytes)
@@ -XXX,XX +XXX,XX @@ Format specific information:
  qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
  === Testing truncated sparse ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
  qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
  === Converting to streamOptimized from image with small cluster size===
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing monolithicFlat with internally generated JSON file name ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
  qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
  === Testing version 3 ===
@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing 4TB monolithicFlat creation and IO ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
  image: TEST_DIR/t.IMGFMT
  file format: IMGFMT
  virtual size: 4 TiB (4398046511104 bytes)
@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing qemu-img map on extents ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0x20000         0x3f0000        TEST_DIR/t.vmdk
 x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
 diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/common.filter
 +++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
          -e "s# compat6=\\(on\\|off\\)##g" \
          -e "s# static=\\(on\\|off\\)##g" \
          -e "s# zeroed_grain=\\(on\\|off\\)##g" \
 -        -e "s# subformat='[^']*'##g" \
 -        -e "s# adapter_type='[^']*'##g" \
 +        -e "s# subformat=[^ ]*##g" \
 +        -e "s# adapter_type=[^ ]*##g" \
          -e "s# hwversion=[^ ]*##g" \
          -e "s# lazy_refcounts=\\(on\\|off\\)##g" \
          -e "s# block_size=[0-9]\\+##g" \
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 08/13] nvme: use TYPE_NVME instead of constant string
+[Qemu-devel] [PULL 06/15] vmdk: Use bdrv_dirname() for relative extent paths
-From: Li Qiang <liq3ea@163.com>
+This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
 turns a former error in 059 into success.
-Signed-off-by: Li Qiang <liq3ea@163.com>
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Max Reitz <mreitz@redhat.com>
+Message-id: 20190815153638.4600-3-mreitz@redhat.com
-Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
 Message-id: 20190120055558.32984-2-liq3ea@163.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- hw/block/nvme.c | 2 +-
+ block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
-file changed, 1 insertion(+), 1 deletion(-)
+ tests/qemu-iotests/059     |  7 +++--
  tests/qemu-iotests/059.out |  4 ++-
 files changed, 42 insertions(+), 23 deletions(-)
-diff --git a/hw/block/nvme.c b/hw/block/nvme.c
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/hw/block/nvme.c
+--- a/block/vmdk.c
-+++ b/hw/block/nvme.c
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ static void nvme_instance_init(Object *obj)
+@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
  }
- static const TypeInfo nvme_info = {
+ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
--    .name          = "nvme",
+-                              const char *desc_file_path, QDict *options,
-+    .name          = TYPE_NVME,
+-                              Error **errp)
-     .parent        = TYPE_PCI_DEVICE,
++                              QDict *options, Error **errp)
-     .instance_size = sizeof(NvmeCtrl),
+ {
-     .class_init    = nvme_class_init,
+     int ret;
      int matches;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
      const char *p, *np;
      int64_t sectors = 0;
      int64_t flat_offset;
 +    char *desc_file_dir = NULL;
      char *extent_path;
      BdrvChild *extent_file;
      BDRVVmdkState *s = bs->opaque;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
              continue;
          }
 -        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
 -            !desc_file_path[0])
 -        {
 -            bdrv_refresh_filename(bs->file->bs);
 -            error_setg(errp, "Cannot use relative extent paths with VMDK "
 -                       "descriptor file '%s'", bs->file->bs->filename);
 -            return -EINVAL;
 -        }
 +        if (path_is_absolute(fname)) {
 +            extent_path = g_strdup(fname);
 +        } else {
 +            if (!desc_file_dir) {
 +                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
 +                if (!desc_file_dir) {
 +                    bdrv_refresh_filename(bs->file->bs);
 +                    error_prepend(errp, "Cannot use relative paths with VMDK "
 +                                  "descriptor file '%s': ",
 +                                  bs->file->bs->filename);
 +                    ret = -EINVAL;
 +                    goto out;
 +                }
 +            }
 -        extent_path = path_combine(desc_file_path, fname);
 +            extent_path = g_strconcat(desc_file_dir, fname, NULL);
 +        }
          ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
          assert(ret < 32);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          g_free(extent_path);
          if (local_err) {
              error_propagate(errp, local_err);
 -            return -EINVAL;
 +            ret = -EINVAL;
 +            goto out;
          }
          /* save to extents array */
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
 , 0, 0, 0, 0, &extent, errp);
              if (ret < 0) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent->flat_start_offset = flat_offset << 9;
          } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
              g_free(buf);
              if (ret) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent = &s->extents[s->num_extents - 1];
          } else if (!strcmp(type, "SESPARSE")) {
              ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
              if (ret) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent = &s->extents[s->num_extents - 1];
          } else {
              error_setg(errp, "Unsupported extent type '%s'", type);
              bdrv_unref_child(bs, extent_file);
 -            return -ENOTSUP;
 +            ret = -ENOTSUP;
 +            goto out;
          }
          extent->type = g_strdup(type);
      }
 -    return 0;
 +
 +    ret = 0;
 +    goto out;
  invalid:
      np = next_line(p);
@@ -XXX,XX +XXX,XX @@ invalid:
          np--;
      }
      error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
 -    return -EINVAL;
 +    ret = -EINVAL;
 +
 +out:
 +    g_free(desc_file_dir);
 +    return ret;
  }
  static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
      }
      s->create_type = g_strdup(ct);
      s->desc_offset = 0;
 -    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
 -                             errp);
 +    ret = vmdk_parse_extents(buf, bs, options, errp);
  exit:
      return ret;
  }
 diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/059
 +++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
  echo
  echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 +# Should work, because bdrv_dirname() works fine with blkdebug
  IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 -$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
 -    | _filter_testdir | _filter_imgfmt
 +$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
 +         -c info \
 +    2>&1 \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
  echo
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/059.out
 +++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
  === Testing monolithicFlat with internally generated JSON file name ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 -qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
 +format name: IMGFMT
 +cluster size: 0 bytes
 +vm state offset: 0 bytes
  === Testing version 3 ===
  image: TEST_DIR/iotest-version3.IMGFMT
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 02/13] scripts: add render_block_graph function for QEMUMachine
+[Qemu-devel] [PULL 07/15] iotests: Keep testing broken relative extent paths
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+We had a test for a case where relative extent paths did not work, but
 unfortunately we just fixed the underlying problem, so it works now.
 This patch adds a new test case that still fails.
-Render block nodes graph with help of graphviz. This new function is
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-for debugging, so there is no sense to put it into qemu.py as a method
+Reviewed-by: John Snow <jsnow@redhat.com>
-of QEMUMachine. Let's instead put it separately.
+Message-id: 20190815153638.4600-4-mreitz@redhat.com
+Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
 Acked-by: Eduardo Habkost <ehabkost@redhat.com>
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Message-id: 20181221170909.25584-3-vsementsov@virtuozzo.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- scripts/render_block_graph.py | 120 ++++++++++++++++++++++++++++++++++
+ tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
-file changed, 120 insertions(+)
+ tests/qemu-iotests/059.out |  4 ++++
- create mode 100755 scripts/render_block_graph.py
+files changed, 31 insertions(+)
-diff --git a/scripts/render_block_graph.py b/scripts/render_block_graph.py
+diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
-new file mode 100755
+index XXXXXXX..XXXXXXX 100755
-index XXXXXXX..XXXXXXX
+--- a/tests/qemu-iotests/059
---- /dev/null
++++ b/tests/qemu-iotests/059
-+++ b/scripts/render_block_graph.py
+@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
-@@ -XXX,XX +XXX,XX @@
-+#!/usr/bin/env python
+ echo
-+#
+ echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 +# Render Qemu Block Graph
 +#
 +# Copyright (c) 2018 Virtuozzo International GmbH. All rights reserved.
 +#
 +# This program is free software; you can redistribute it and/or modify
 +# it under the terms of the GNU General Public License as published by
 +# the Free Software Foundation; either version 2 of the License, or
 +# (at your option) any later version.
 +#
 +# This program is distributed in the hope that it will be useful,
 +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 +# GNU General Public License for more details.
 +#
 +# You should have received a copy of the GNU General Public License
 +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 +#
 +
-+import os
++echo '--- blkdebug ---'
-+import sys
+ # Should work, because bdrv_dirname() works fine with blkdebug
-+import subprocess
+ IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
-+import json
+ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
-+from graphviz import Digraph
+@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
-+from qemu import MonitorResponseError
+     | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
 +echo '--- quorum ---'
 +# Should not work, because bdrv_dirname() does not work with quorum
 +IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 +cp "$TEST_IMG" "$TEST_IMG.orig"
 +
 +filename="json:{
 +    \"driver\": \"$IMGFMT\",
 +    \"file\": {
 +        \"driver\": \"quorum\",
 +        \"children\": [ {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG\"
 +        }, {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG.orig\"
 +        } ],
 +        \"vote-threshold\": 1
 +    } }"
 +
 +filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
 +$QEMU_IMG info "$filename" 2>&1 \
 +    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
 +
 +
-+def perm(arr):
+ echo
-+    s = 'w' if 'write' in arr else '_'
+ echo "=== Testing version 3 ==="
-+    s += 'r' if 'consistent-read' in arr else '_'
+ _use_sample_img iotest-version3.vmdk.bz2
-+    s += 'u' if 'write-unchanged' in arr else '_'
+diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
-+    s += 'g' if 'graph-mod' in arr else '_'
+index XXXXXXX..XXXXXXX 100644
-+    s += 's' if 'resize' in arr else '_'
+--- a/tests/qemu-iotests/059.out
-+    return s
++++ b/tests/qemu-iotests/059.out
-+
+@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
-+
+bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-+def render_block_graph(qmp, filename, format='png'):
-+    '''
+ === Testing monolithicFlat with internally generated JSON file name ===
-+    Render graph in text (dot) representation into "@filename" and
++--- blkdebug ---
-+    representation in @format into "@filename.@format"
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-+    '''
+ format name: IMGFMT
-+
+ cluster size: 0 bytes
-+    bds_nodes = qmp.command('query-named-block-nodes')
+ vm state offset: 0 bytes
-+    bds_nodes = {n['node-name']: n for n in bds_nodes}
++--- quorum ---
-+
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-+    job_nodes = qmp.command('query-block-jobs')
++qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
-+    job_nodes = {n['device']: n for n in job_nodes}
-+
+ === Testing version 3 ===
-+    block_graph = qmp.command('x-debug-query-block-graph')
+ image: TEST_DIR/iotest-version3.IMGFMT
 +
 +    graph = Digraph(comment='Block Nodes Graph')
 +    graph.format = format
 +    graph.node('permission symbols:\l'
 +               '  w - Write\l'
 +               '  r - consistent-Read\l'
 +               '  u - write - Unchanged\l'
 +               '  g - Graph-mod\l'
 +               '  s - reSize\l'
 +               'edge label scheme:\l'
 +               '  <child type>\l'
 +               '  <perm>\l'
 +               '  <shared_perm>\l', shape='none')
 +
 +    for n in block_graph['nodes']:
 +        if n['type'] == 'block-driver':
 +            info = bds_nodes[n['name']]
 +            label = n['name'] + ' [' + info['drv'] + ']'
 +            if info['drv'] == 'file':
 +                label += '\n' + os.path.basename(info['file'])
 +            shape = 'ellipse'
 +        elif n['type'] == 'block-job':
 +            info = job_nodes[n['name']]
 +            label = info['type'] + ' job (' + n['name'] + ')'
 +            shape = 'box'
 +        else:
 +            assert n['type'] == 'block-backend'
 +            label = n['name'] if n['name'] else 'unnamed blk'
 +            shape = 'box'
 +
 +        graph.node(str(n['id']), label, shape=shape)
 +
 +    for e in block_graph['edges']:
 +        label = '%s\l%s\l%s\l' % (e['name'], perm(e['perm']),
 +                                  perm(e['shared-perm']))
 +        graph.edge(str(e['parent']), str(e['child']), label=label)
 +
 +    graph.render(filename)
 +
 +
 +class LibvirtGuest():
 +    def __init__(self, name):
 +        self.name = name
 +
 +    def command(self, cmd):
 +        # only supports qmp commands without parameters
 +        m = {'execute': cmd}
 +        ar = ['virsh', 'qemu-monitor-command', self.name, json.dumps(m)]
 +
 +        reply = json.loads(subprocess.check_output(ar))
 +
 +        if 'error' in reply:
 +            raise MonitorResponseError(reply)
 +
 +        return reply['return']
 +
 +
 +if __name__ == '__main__':
 +    obj = sys.argv[1]
 +    out = sys.argv[2]
 +
 +    if os.path.exists(obj):
 +        # assume unix socket
 +        qmp = QEMUMonitorProtocol(obj)
 +        qmp.connect()
 +    else:
 +        # assume libvirt guest name
 +        qmp = LibvirtGuest(obj)
 +
 +    render_block_graph(qmp, out)
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 12/13] iotests: Bind qemu-nbd to localhost in 147
+[Qemu-devel] [PULL 08/15] vmdk: Reject invalid compressed writes
-By default, qemu-nbd binds to 0.0.0.0.  However, we then proceed to
+Compressed writes generally have to write full clusters, not just in
-connect to "localhost".  Usually, this works out fine; but if this test
+theory but also in practice when it comes to vmdk's streamOptimized
-is run concurrently, some other test function may have bound a different
+subformat.  It currently is just silently broken for writes with
-server to ::1 (on the same port -- you can bind different serves to the
+non-zero in-cluster offsets:
 same port, as long as one is on IPv4 and the other on IPv6).
-So running qemu-nbd works, it can bind to 0.0.0.0:NBD_PORT.  But
+$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
-potentially a concurrent test has successfully taken [::1]:NBD_PORT.  In
+$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
-this case, trying to connect to "localhost" will lead us to the IPv6
+wrote 4096/4096 bytes at offset 4096
-instance, where we do not want to end up.
+KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
 read failed: Invalid argument
-Fix this by just binding to "localhost".  This will make qemu-nbd error
+(The technical reason is that vmdk_write_extent() just writes the
-out immediately and not give us cryptic errors later.
+incomplete compressed data actually to offset 4k.  When reading the
 data, vmdk_read_extent() looks at offset 0 and finds the compressed data
 size to be 0, because that is what it reads from there.  This yields an
 error.)
-(Also, it will allow us to just try a different port as of a future
+For incomplete writes with zero in-cluster offsets, the error path when
-patch.)
+reading the rest of the cluster is a bit different, but the result is
 the same:
 $ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
 $ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
 wrote 4096/4096 bytes at offset 0
 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
 read failed: Invalid argument
 (Here, vmdk_read_extent() finds the data and then sees that the
 uncompressed data is short.)
 It is better to reject invalid writes than to make the user believe they
 might have succeeded and then fail when trying to read it back.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20181221234750.23577-3-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
+Message-id: 20190815153638.4600-5-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/147 | 2 +-
+ block/vmdk.c | 10 ++++++++++
-file changed, 1 insertion(+), 1 deletion(-)
+file changed, 10 insertions(+)
-diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
+diff --git a/block/vmdk.c b/block/vmdk.c
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/147
+--- a/block/vmdk.c
-+++ b/tests/qemu-iotests/147
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ class QemuNBD(NBDBlockdevAddBase):
+@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
-         self.assertEqual(qemu_nbd('-f', imgfmt, test_img, *args), 0)
+     if (extent->compressed) {
+         void *compressed_data;
-     def test_inet(self):
--        self._server_up('-p', str(NBD_PORT))
++        /* Only whole clusters */
-+        self._server_up('-b', 'localhost', '-p', str(NBD_PORT))
++        if (offset_in_cluster ||
-         address = { 'type': 'inet',
++            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
-                     'data': {
++            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
-                         'host': 'localhost',
++             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
 +        {
 +            ret = -EINVAL;
 +            goto out;
 +        }
 +
          if (!extent->has_marker) {
              ret = -EINVAL;
              goto out;
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 04/13] block/curl: Convert from DPRINTF() macro to trace events
+[Qemu-devel] [PULL 09/15] iotests: Disable broken streamOptimized tests
-From: Laurent Vivier <lvivier@redhat.com>
+streamOptimized does not support writes that do not span exactly one
 cluster.  Furthermore, it cannot rewrite already allocated clusters.
 As such, many iotests do not work with it.  Disable them.
-Signed-off-by: Laurent Vivier <lvivier@redhat.com>
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
+Message-id: 20190815153638.4600-6-mreitz@redhat.com
-Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
 Message-id: 20181213162727.17438-3-lvivier@redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/curl.c       | 29 ++++++++---------------------
+ tests/qemu-iotests/002 | 1 +
- block/trace-events |  9 +++++++++
+ tests/qemu-iotests/003 | 1 +
-files changed, 17 insertions(+), 21 deletions(-)
+ tests/qemu-iotests/005 | 3 ++-
  tests/qemu-iotests/009 | 1 +
  tests/qemu-iotests/010 | 1 +
  tests/qemu-iotests/011 | 1 +
  tests/qemu-iotests/017 | 3 ++-
  tests/qemu-iotests/018 | 3 ++-
  tests/qemu-iotests/019 | 3 ++-
  tests/qemu-iotests/020 | 3 ++-
  tests/qemu-iotests/027 | 1 +
  tests/qemu-iotests/032 | 1 +
  tests/qemu-iotests/033 | 1 +
  tests/qemu-iotests/034 | 3 ++-
  tests/qemu-iotests/037 | 3 ++-
  tests/qemu-iotests/063 | 3 ++-
  tests/qemu-iotests/072 | 1 +
  tests/qemu-iotests/105 | 3 ++-
  tests/qemu-iotests/197 | 1 +
  tests/qemu-iotests/215 | 1 +
  tests/qemu-iotests/251 | 1 +
 files changed, 30 insertions(+), 9 deletions(-)
-diff --git a/block/curl.c b/block/curl.c
+diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/block/curl.c
+--- a/tests/qemu-iotests/002
-+++ b/block/curl.c
++++ b/tests/qemu-iotests/002
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
- #include "crypto/secret.h"
- #include <curl/curl.h>
+ _supported_fmt generic
- #include "qemu/cutils.h"
+ _supported_proto generic
-+#include "trace.h"
++_unsupported_imgopts "subformat=streamOptimized"
--// #define DEBUG_CURL
- // #define DEBUG_VERBOSE
+ size=128M
+diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
--#ifdef DEBUG_CURL
+index XXXXXXX..XXXXXXX 100755
--#define DEBUG_CURL_PRINT 1
+--- a/tests/qemu-iotests/003
--#else
++++ b/tests/qemu-iotests/003
--#define DEBUG_CURL_PRINT 0
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
--#endif
--#define DPRINTF(fmt, ...)                                            \
+ _supported_fmt generic
--    do {                                                             \
+ _supported_proto generic
--        if (DEBUG_CURL_PRINT) {                                      \
++_unsupported_imgopts "subformat=streamOptimized"
--            fprintf(stderr, fmt, ## __VA_ARGS__);                    \
--        }                                                            \
+ size=128M
--    } while (0)
+ offset=67M
--
+diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
- #if LIBCURL_VERSION_NUM >= 0x071000
+index XXXXXXX..XXXXXXX 100755
- /* The multi interface timer callback was introduced in 7.16.0 */
+--- a/tests/qemu-iotests/005
- #define NEED_CURL_TIMER_CALLBACK
++++ b/tests/qemu-iotests/005
-@@ -XXX,XX +XXX,XX @@ static int curl_timer_cb(CURLM *multi, long timeout_ms, void *opaque)
+@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
- {
+ _supported_proto generic
-     BDRVCURLState *s = opaque;
+ _supported_os Linux
+ _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
--    DPRINTF("CURL: timer callback timeout_ms %ld\n", timeout_ms);
+-                     "subformat=twoGbMaxExtentSparse"
-+    trace_curl_timer_cb(timeout_ms);
++                     "subformat=twoGbMaxExtentSparse" \
-     if (timeout_ms == -1) {
++                     "subformat=streamOptimized"
-         timer_del(&s->timer);
-     } else {
+ # vpc is limited to 127GB, so we can't test it here
-@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
+ if [ "$IMGFMT" = "vpc" ]; then
-     }
+diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
-     socket = NULL;
+index XXXXXXX..XXXXXXX 100755
+--- a/tests/qemu-iotests/009
--    DPRINTF("CURL (AIO): Sock action %d on fd %d\n", action, (int)fd);
++++ b/tests/qemu-iotests/009
-+    trace_curl_sock_cb(action, (int)fd);
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-     switch (action) {
-         case CURL_POLL_IN:
+ _supported_fmt generic
-             aio_set_fd_handler(s->aio_context, fd, false,
+ _supported_proto generic
-@@ -XXX,XX +XXX,XX @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
++_unsupported_imgopts "subformat=streamOptimized"
-     size_t realsize = size * nmemb;
-     int i;
+ size=6G
--    DPRINTF("CURL: Just reading %zd bytes\n", realsize);
+diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
-+    trace_curl_read_cb(realsize);
+index XXXXXXX..XXXXXXX 100755
+--- a/tests/qemu-iotests/010
-     if (!s || !s->orig_buf) {
++++ b/tests/qemu-iotests/010
-         goto read_end;
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
-         }
+ _supported_fmt generic
-     }
+ _supported_proto generic
++_unsupported_imgopts "subformat=streamOptimized"
--    DPRINTF("CURL: Opening %s\n", file);
-+    trace_curl_open(file);
-     qemu_co_queue_init(&s->free_state_waitq);
+ size=6G
-     s->aio_context = bdrv_get_aio_context(bs);
+diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
-     s->url = g_strdup(file);
+index XXXXXXX..XXXXXXX 100755
-@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
+--- a/tests/qemu-iotests/011
-                 "Server does not support 'range' (byte ranges).");
++++ b/tests/qemu-iotests/011
-         goto out;
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-     }
--    DPRINTF("CURL: Size = %" PRIu64 "\n", s->len);
+ _supported_fmt generic
-+    trace_curl_open_size(s->len);
+ _supported_proto generic
++_unsupported_imgopts "subformat=streamOptimized"
-     qemu_mutex_lock(&s->mutex);
-     curl_clean_state(state);
-@@ -XXX,XX +XXX,XX @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
+ size=6G
-     state->acb[0] = acb;
+diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
+index XXXXXXX..XXXXXXX 100755
-     snprintf(state->range, 127, "%" PRIu64 "-%" PRIu64, start, end);
+--- a/tests/qemu-iotests/017
--    DPRINTF("CURL (AIO): Reading %" PRIu64 " at %" PRIu64 " (%s)\n",
++++ b/tests/qemu-iotests/017
--            acb->bytes, start, state->range);
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-+    trace_curl_setup_preadv(acb->bytes, start, state->range);
+ _supported_fmt qcow qcow2 vmdk qed
-     curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range);
+ _supported_proto generic
+ _unsupported_proto vxhs
-     curl_multi_add_handle(s->multi, state->curl);
+-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
-@@ -XXX,XX +XXX,XX @@ static void curl_close(BlockDriverState *bs)
++_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
- {
++                     "subformat=streamOptimized"
-     BDRVCURLState *s = bs->opaque;
+ TEST_OFFSETS="0 4294967296"
--    DPRINTF("CURL: Close\n");
-+    trace_curl_close();
+diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
-     curl_detach_aio_context(bs);
+index XXXXXXX..XXXXXXX 100755
-     qemu_mutex_destroy(&s->mutex);
+--- a/tests/qemu-iotests/018
++++ b/tests/qemu-iotests/018
-diff --git a/block/trace-events b/block/trace-events
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-index XXXXXXX..XXXXXXX 100644
+ _supported_fmt qcow qcow2 vmdk qed
---- a/block/trace-events
+ _supported_proto file
-+++ b/block/trace-events
+ _supported_os Linux
-@@ -XXX,XX +XXX,XX @@ ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
+-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
- ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
++_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
- ssh_write_return(ssize_t ret) "sftp_write returned %zd"
++                     "streamOptimized"
- ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
-+
+ TEST_OFFSETS="0 4294967296"
-+# block/curl.c
-+curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
+diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
-+curl_sock_cb(int action, int fd) "sock action %d on fd %d"
+index XXXXXXX..XXXXXXX 100755
-+curl_read_cb(size_t realsize) "just reading %zu bytes"
+--- a/tests/qemu-iotests/019
-+curl_open(const char *file) "opening %s"
++++ b/tests/qemu-iotests/019
-+curl_open_size(uint64_t size) "size = %" PRIu64
+@@ -XXX,XX +XXX,XX @@ _supported_proto file
-+curl_setup_preadv(uint64_t bytes, uint64_t start, const char *range) "reading %" PRIu64 " at %" PRIu64 " (%s)"
+ _supported_os Linux
-+curl_close(void) "close"
+ _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
  CLUSTER_SIZE=65536
 diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/020
 +++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
 diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/027
 +++ b/tests/qemu-iotests/027
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt vmdk qcow qcow2 qed
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=128M
 diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/032
 +++ b/tests/qemu-iotests/032
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  # This works for any image format (though unlikely to segfault for raw)
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo === Prepare image ===
 diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/033
 +++ b/tests/qemu-iotests/033
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=128M
 diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/034
 +++ b/tests/qemu-iotests/034
@@ -XXX,XX +XXX,XX @@ _supported_proto file
  _supported_os Linux
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  CLUSTER_SIZE=4k
  size=128M
 diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/037
 +++ b/tests/qemu-iotests/037
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  CLUSTER_SIZE=4k
  size=128M
 diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/063
 +++ b/tests/qemu-iotests/063
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  _make_test_img 4M
 diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/072
 +++ b/tests/qemu-iotests/072
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
  _supported_proto file
 +_unsupported_imgopts "subformat=streamOptimized"
  IMG_SIZE=64M
 diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/105
 +++ b/tests/qemu-iotests/105
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2 vmdk vhdx qed
  _supported_proto generic
  _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  echo
  echo "creating large image"
 diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/197
 +++ b/tests/qemu-iotests/197
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto generic
  # LUKS support may be possible, but it complicates things.
  _unsupported_fmt luks
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo '=== Copy-on-read ==='
 diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/215
 +++ b/tests/qemu-iotests/215
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto generic
  # LUKS support may be possible, but it complicates things.
  _unsupported_fmt luks
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo '=== Copy-on-read ==='
 diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/251
 +++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto file
  _supported_os Linux
 +_unsupported_imgopts "subformat=streamOptimized"
  if [ "$IMGOPTSSYNTAX" = "true" ]; then
      # We use json:{} filenames here, so we cannot work with additional options.
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 13/13] iotests: Allow 147 to be run concurrently
+[Qemu-devel] [PULL 10/15] iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
-To do this, we need to allow creating the NBD server on various ports
+The error message for the test case where we have a quorum node for
-instead of a single one (which may not even work if you run just one
+which no directory name can be generated is different: For
-instance, because something entirely else might be using that port).
+twoGbMaxExtentSparse, it complains that it cannot open the extent file.
+For other (sub)formats, it just notes that it cannot determine the
-So we just pick a random port in [32768, 32768 + 1024) and try to create
+backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
-a server there.  If that fails, we just retry until something sticks.
+for simplicity's sake.
 For the IPv6 test, we need a different range, though (just above that
 one).  This is because "localhost" resolves to both 127.0.0.1 and ::1.
 This means that if you bind to it, it will bind to both, if possible, or
 just one if the other is already in use.  Therefore, if the IPv6 test
 has already taken [::1]:some_port and we then try to take
 localhost:some_port, that will work -- only the second server will be
 bound to 127.0.0.1:some_port alone and not [::1]:some_port in addition.
 So we have two different servers on the same port, one for IPv4 and one
 for IPv6.
 But when we then try to connect to the server through
 localhost:some_port, we will always end up at the IPv6 one (as long as
 it is up), and this may not be the one we want.
 Thus, we must make sure not to create an IPv6-only NBD server on the
 same port as a normal "dual-stack" NBD server -- which is done by using
 distinct port ranges, as explained above.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20181221234750.23577-4-mreitz@redhat.com
+Reviewed-by: John Snow <jsnow@redhat.com>
 Message-id: 20190815153638.4600-7-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/147 | 98 +++++++++++++++++++++++++++++-------------
+ tests/qemu-iotests/110 | 3 ++-
-file changed, 68 insertions(+), 30 deletions(-)
+file changed, 2 insertions(+), 1 deletion(-)
-diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
+diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
 index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/147
+--- a/tests/qemu-iotests/110
-+++ b/tests/qemu-iotests/147
++++ b/tests/qemu-iotests/110
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
- #
+ # Any format supporting backing files
+ _supported_fmt qed qcow qcow2 vmdk
- import os
+ _supported_proto file
-+import random
+-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
- import socket
++_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
- import stat
++                     "subformat=twoGbMaxExtentSparse"
- import time
- import iotests
+ TEST_IMG_REL=$(basename "$TEST_IMG")
 -from iotests import cachemode, imgfmt, qemu_img, qemu_nbd
 +from iotests import cachemode, imgfmt, qemu_img, qemu_nbd, qemu_nbd_pipe
 -NBD_PORT = 10811
 +NBD_PORT_START      = 32768
 +NBD_PORT_END        = NBD_PORT_START + 1024
 +NBD_IPV6_PORT_START = NBD_PORT_END
 +NBD_IPV6_PORT_END   = NBD_IPV6_PORT_START + 1024
  test_img = os.path.join(iotests.test_dir, 'test.img')
  unix_socket = os.path.join(iotests.test_dir, 'nbd.socket')
@@ -XXX,XX +XXX,XX @@ class QemuNBD(NBDBlockdevAddBase):
          except OSError:
              pass
 +    def _try_server_up(self, *args):
 +        status, msg = qemu_nbd_pipe('-f', imgfmt, test_img, *args)
 +        if status == 0:
 +            return True
 +        if 'Address already in use' in msg:
 +            return False
 +        self.fail(msg)
 +
      def _server_up(self, *args):
 -        self.assertEqual(qemu_nbd('-f', imgfmt, test_img, *args), 0)
 +        self.assertTrue(self._try_server_up(*args))
      def test_inet(self):
 -        self._server_up('-b', 'localhost', '-p', str(NBD_PORT))
 +        while True:
 +            nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END)
 +            if self._try_server_up('-b', 'localhost', '-p', str(nbd_port)):
 +                break
 +
          address = { 'type': 'inet',
                      'data': {
                          'host': 'localhost',
 -                        'port': str(NBD_PORT)
 +                        'port': str(nbd_port)
                      } }
 -        self.client_test('nbd://localhost:%i' % NBD_PORT,
 +        self.client_test('nbd://localhost:%i' % nbd_port,
                           flatten_sock_addr(address))
      def test_unix(self):
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
          except OSError:
              pass
 -    def _server_up(self, address, export_name=None, export_name2=None):
 +    # Returns False on EADDRINUSE; fails an assertion on other errors.
 +    # Returns True on success.
 +    def _try_server_up(self, address, export_name=None, export_name2=None):
          result = self.server.qmp('nbd-server-start', addr=address)
 +        if 'error' in result and \
 +           'Address already in use' in result['error']['desc']:
 +            return False
          self.assert_qmp(result, 'return', {})
          if export_name is None:
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
                                       name=export_name2)
              self.assert_qmp(result, 'return', {})
 +        return True
 +
 +    def _server_up(self, address, export_name=None, export_name2=None):
 +        self.assertTrue(self._try_server_up(address, export_name, export_name2))
      def _server_down(self):
          result = self.server.qmp('nbd-server-stop')
          self.assert_qmp(result, 'return', {})
      def do_test_inet(self, export_name=None):
 -        address = { 'type': 'inet',
 -                    'data': {
 -                        'host': 'localhost',
 -                        'port': str(NBD_PORT)
 -                    } }
 -        self._server_up(address, export_name)
 +        while True:
 +            nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END)
 +            address = { 'type': 'inet',
 +                        'data': {
 +                            'host': 'localhost',
 +                            'port': str(nbd_port)
 +                        } }
 +            if self._try_server_up(address, export_name):
 +                break
 +
          export_name = export_name or 'nbd-export'
 -        self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, export_name),
 +        self.client_test('nbd://localhost:%i/%s' % (nbd_port, export_name),
                           flatten_sock_addr(address), export_name)
          self._server_down()
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
          self.do_test_inet('shadow')
      def test_inet_two_exports(self):
 -        address = { 'type': 'inet',
 -                    'data': {
 -                        'host': 'localhost',
 -                        'port': str(NBD_PORT)
 -                    } }
 -        self._server_up(address, 'exp1', 'exp2')
 -        self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, 'exp1'),
 +        while True:
 +            nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END)
 +            address = { 'type': 'inet',
 +                        'data': {
 +                            'host': 'localhost',
 +                            'port': str(nbd_port)
 +                        } }
 +            if self._try_server_up(address, 'exp1', 'exp2'):
 +                break
 +
 +        self.client_test('nbd://localhost:%i/%s' % (nbd_port, 'exp1'),
                           flatten_sock_addr(address), 'exp1', 'node1', False)
 -        self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, 'exp2'),
 +        self.client_test('nbd://localhost:%i/%s' % (nbd_port, 'exp2'),
                           flatten_sock_addr(address), 'exp2', 'node2', False)
          result = self.vm.qmp('blockdev-del', node_name='node1')
          self.assert_qmp(result, 'return', {})
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
          except socket.gaierror:
              # IPv6 not available, skip
              return
 -        address = { 'type': 'inet',
 -                    'data': {
 -                        'host': '::1',
 -                        'port': str(NBD_PORT),
 -                        'ipv4': False,
 -                        'ipv6': True
 -                    } }
 +
 +        while True:
 +            nbd_port = random.randrange(NBD_IPV6_PORT_START, NBD_IPV6_PORT_END)
 +            address = { 'type': 'inet',
 +                        'data': {
 +                            'host': '::1',
 +                            'port': str(nbd_port),
 +                            'ipv4': False,
 +                            'ipv6': True
 +                        } }
 +            if self._try_server_up(address):
 +                break
 +
          filename = { 'driver': 'raw',
                       'file': {
                           'driver': 'nbd',
                           'export': 'nbd-export',
                           'server': flatten_sock_addr(address)
                       } }
 -        self._server_up(address)
          self.client_test(filename, flatten_sock_addr(address), 'nbd-export')
          self._server_down()
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 11/13] iotests.py: Add qemu_nbd_pipe()
+[Qemu-devel] [PULL 11/15] iotests: Disable 126 for flat vmdk subformats
-In some cases, we may want to deal with qemu-nbd errors (e.g. by
+iotest 126 requires backing file support, which flat vmdks cannot offer.
-launching it in a different configuration until it no longer throws
+Skip this test for such subformats.
 any).  In that case, we do not want its output ending up in the test
 output.
 It may still be useful for handling the error, though, so add a new
 function that works basically like qemu_nbd(), only that it returns the
 qemu-nbd output instead of making it end up in the log.  In contrast to
 qemu_img_pipe(), it does still return the exit code as well, though,
 because that is even more important for error handling.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20181221234750.23577-2-mreitz@redhat.com
+Message-id: 20190815153638.4600-8-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/iotests.py | 14 ++++++++++++++
+ tests/qemu-iotests/126 | 2 ++
-file changed, 14 insertions(+)
+file changed, 2 insertions(+)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
+diff --git a/tests/qemu-iotests/126 b/tests/qemu-iotests/126
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/iotests.py
+--- a/tests/qemu-iotests/126
-+++ b/tests/qemu-iotests/iotests.py
++++ b/tests/qemu-iotests/126
-@@ -XXX,XX +XXX,XX @@ def qemu_nbd(*args):
+@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
-     '''Run qemu-nbd in daemon mode and return the parent's exit code'''
-     return subprocess.call(qemu_nbd_args + ['--fork'] + list(args))
+ # Needs backing file support
+ _supported_fmt qcow qcow2 qed vmdk
-+def qemu_nbd_pipe(*args):
++_unsupported_imgopts "subformat=monolithicFlat" \
-+    '''Run qemu-nbd in daemon mode and return both the parent's exit code
++                     "subformat=twoGbMaxExtentFlat"
-+       and its output'''
+ # This is the default protocol (and we want to test the difference between
-+    subp = subprocess.Popen(qemu_nbd_args + ['--fork'] + list(args),
+ # colons which separate a protocol prefix from the rest and colons which are
-+                            stdout=subprocess.PIPE,
+ # just part of the filename, so we cannot test protocols which require a prefix)
 +                            stderr=subprocess.STDOUT,
 +                            universal_newlines=True)
 +    exitcode = subp.wait()
 +    if exitcode < 0:
 +        sys.stderr.write('qemu-nbd received signal %i: %s\n' %
 +                         (-exitcode,
 +                          ' '.join(qemu_nbd_args + ['--fork'] + list(args))))
 +    return exitcode, subp.communicate()[0]
 +
  def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
      '''Return True if two image files are identical'''
      return qemu_img('compare', '-f', fmt1,
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 05/13] block/file-posix: Convert from DPRINTF() macro to trace events
+[Qemu-devel] [PULL 12/15] file-posix: fix request_alignment typo
-From: Laurent Vivier <lvivier@redhat.com>
+From: Stefan Hajnoczi <stefanha@redhat.com>
-Signed-off-by: Laurent Vivier <lvivier@redhat.com>
+Fixes: a6b257a08e3d72219f03e461a52152672fec0612
-Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
+       ("file-posix: Handle undetectable alignment")
-Message-id: 20181213162727.17438-4-lvivier@redhat.com
+Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
 Message-id: 20190827101328.4062-1-stefanha@redhat.com
 Reviewed-by: Thomas Huth <thuth@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/file-posix.c | 25 ++++++-------------------
+ block/file-posix.c | 2 +-
- block/trace-events |  7 +++++++
+file changed, 1 insertion(+), 1 deletion(-)
 files changed, 13 insertions(+), 19 deletions(-)
 diff --git a/block/file-posix.c b/block/file-posix.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/file-posix.c
 +++ b/block/file-posix.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
- #include <xfs/xfs.h>
+         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
- #endif
+             align = alignments[i];
+             if (raw_is_io_aligned(fd, buf + align, max_align)) {
--//#define DEBUG_BLOCK
+-                /* Fallback to request_aligment. */
--
++                /* Fallback to request_alignment. */
--#ifdef DEBUG_BLOCK
+                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
--# define DEBUG_BLOCK_PRINT 1
+                 break;
--#else
+             }
 -# define DEBUG_BLOCK_PRINT 0
 -#endif
 -#define DPRINTF(fmt, ...) \
 -do { \
 -    if (DEBUG_BLOCK_PRINT) { \
 -        printf(fmt, ## __VA_ARGS__); \
 -    } \
 -} while (0)
 +#include "trace.h"
  /* OS X does not have O_DSYNC */
  #ifndef O_DSYNC
@@ -XXX,XX +XXX,XX @@ static int xfs_write_zeroes(BDRVRawState *s, int64_t offset, uint64_t bytes)
      if (xfsctl(NULL, s->fd, XFS_IOC_ZERO_RANGE, &fl) < 0) {
          err = errno;
 -        DPRINTF("cannot write zero range (%s)\n", strerror(errno));
 +        trace_file_xfs_write_zeroes(strerror(errno));
          return -err;
      }
@@ -XXX,XX +XXX,XX @@ static int xfs_discard(BDRVRawState *s, int64_t offset, uint64_t bytes)
      if (xfsctl(NULL, s->fd, XFS_IOC_UNRESVSP64, &fl) < 0) {
          err = errno;
 -        DPRINTF("cannot punch hole (%s)\n", strerror(errno));
 +        trace_file_xfs_discard(strerror(errno));
          return -err;
      }
@@ -XXX,XX +XXX,XX @@ static char *FindEjectableOpticalMedia(io_iterator_t *mediaIterator)
          /* If a match was found, leave the loop */
          if (*mediaIterator != 0) {
 -            DPRINTF("Matching using %s\n", matching_array[index]);
 +            trace_file_FindEjectableOpticalMedia(matching_array[index]);
              mediaType = g_strdup(matching_array[index]);
              break;
          }
@@ -XXX,XX +XXX,XX @@ static bool setup_cdrom(char *bsd_path, Error **errp)
      if (partition_found == false) {
          error_setg(errp, "Failed to find a working partition on disc");
      } else {
 -        DPRINTF("Using %s as optical disc\n", test_partition);
 +        trace_file_setup_cdrom(test_partition);
          pstrcpy(bsd_path, MAXPATHLEN, test_partition);
      }
      return partition_found;
@@ -XXX,XX +XXX,XX @@ static bool hdev_is_sg(BlockDriverState *bs)
      ret = ioctl(s->fd, SG_GET_SCSI_ID, &scsiid);
      if (ret >= 0) {
 -        DPRINTF("SG device found: type=%d, version=%d\n",
 -            scsiid.scsi_type, sg_version);
 +        trace_file_hdev_is_sg(scsiid.scsi_type, sg_version);
          return true;
      }
 diff --git a/block/trace-events b/block/trace-events
 index XXXXXXX..XXXXXXX 100644
 --- a/block/trace-events
 +++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ curl_open(const char *file) "opening %s"
  curl_open_size(uint64_t size) "size = %" PRIu64
  curl_setup_preadv(uint64_t bytes, uint64_t start, const char *range) "reading %" PRIu64 " at %" PRIu64 " (%s)"
  curl_close(void) "close"
 +
 +# block/file-posix.c
 +file_xfs_write_zeroes(const char *error) "cannot write zero range (%s)"
 +file_xfs_discard(const char *error) "cannot punch hole (%s)"
 +file_FindEjectableOpticalMedia(const char *media) "Matching using %s"
 +file_setup_cdrom(const char *partition) "Using %s as optical disc"
 +file_hdev_is_sg(int type, int version) "SG device found: type=%d, version=%d"
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 03/13] block/ssh: Convert from DPRINTF() macro to trace events
+[Qemu-devel] [PULL 13/15] iotests: Check for enabled drivers before testing them
-From: Laurent Vivier <lvivier@redhat.com>
+From: Thomas Huth <thuth@redhat.com>
-Signed-off-by: Laurent Vivier <lvivier@redhat.com>
+It is possible to enable only a subset of the block drivers with the
-Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
+"--block-drv-rw-whitelist" option of the "configure" script. All other
-Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
+drivers are marked as unusable (or only included as read-only with the
-Message-id: 20181213162727.17438-2-lvivier@redhat.com
+"--block-drv-ro-whitelist" option). If an iotest is now using such a
-[mreitz: Fixed type of ssh_{read,write}_return's parameter to be ssize_t
+disabled block driver, it is failing - which is bad, since at least the
-         instead of size_t]
+tests in the "auto" group should be able to deal with this situation.
 Thus let's introduce a "_require_drivers" function that can be used by
 the shell tests to check for the availability of certain drivers first,
 and marks the test as "not run" if one of the drivers is missing.
 This patch mainly targets the test in the "auto" group which should
 never fail in such a case, but also improves some of the other tests
 along the way. Note that we also assume that the "qcow2" and "file"
 drivers are always available - otherwise it does not make sense to
 run "make check-block" at all (which only tests with qcow2 by default).
 Signed-off-by: Thomas Huth <thuth@redhat.com>
 Message-id: 20190823133552.11680-1-thuth@redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/ssh.c        | 46 +++++++++++++++++-----------------------------
+ tests/qemu-iotests/071       |  1 +
- block/trace-events | 17 +++++++++++++++++
+ tests/qemu-iotests/081       |  4 +---
-files changed, 34 insertions(+), 29 deletions(-)
+ tests/qemu-iotests/099       |  1 +
  tests/qemu-iotests/120       |  1 +
  tests/qemu-iotests/162       |  4 +---
  tests/qemu-iotests/184       |  1 +
  tests/qemu-iotests/186       |  1 +
  tests/qemu-iotests/common.rc | 14 ++++++++++++++
 files changed, 21 insertions(+), 6 deletions(-)
-diff --git a/block/ssh.c b/block/ssh.c
+diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/071
 +++ b/tests/qemu-iotests/071
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
 +_require_drivers blkdebug blkverify
  do_run_qemu()
  {
 diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/081
 +++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt raw
  _supported_proto file
  _supported_os Linux
 +_require_drivers quorum
  do_run_qemu()
  {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                            | _filter_qemu_io | _filter_generated_node_ids
  }
 -test_quorum=$($QEMU_IMG --help|grep quorum)
 -[ "$test_quorum" = "" ] && _supported_fmt quorum
 -
  quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
  quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
  quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
 diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/099
 +++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
  _supported_proto file
  _supported_os Linux
 +_require_drivers blkdebug blkverify
  _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
      "subformat=twoGbMaxExtentSparse"
 diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/120
 +++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto file
  _unsupported_fmt luks
 +_require_drivers raw
  _make_test_img 64M
 diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/162
 +++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_fmt generic
 -
 -test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh\( \|$\)')
 -[ "$test_ssh" = "" ] && _notrun "ssh support required"
 +_require_drivers ssh
  echo
  echo '=== NBD ==='
 diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/184
 +++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_os Linux
 +_require_drivers throttle
  do_run_qemu()
  {
 diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/186
 +++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
 +_require_drivers null-co
  if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
      _notrun "Requires a PC machine"
 diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
 index XXXXXXX..XXXXXXX 100644
---- a/block/ssh.c
+--- a/tests/qemu-iotests/common.rc
-+++ b/block/ssh.c
++++ b/tests/qemu-iotests/common.rc
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ _require_command()
- #include "qapi/qmp/qstring.h"
+     [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
  #include "qapi/qobject-input-visitor.h"
  #include "qapi/qobject-output-visitor.h"
 +#include "trace.h"
 -/* DEBUG_SSH=1 enables the DPRINTF (debugging printf) statements in
 - * this block driver code.
 - *
 +/*
   * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
   * that this requires that libssh2 was specially compiled with the
   * `./configure --enable-debug' option, so most likely you will have
   * to compile it yourself.  The meaning of <bitmask> is described
   * here: http://www.libssh2.org/libssh2_trace.html
   */
 -#define DEBUG_SSH     0
  #define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
 -#define DPRINTF(fmt, ...)                           \
 -    do {                                            \
 -        if (DEBUG_SSH) {                            \
 -            fprintf(stderr, "ssh: %-15s " fmt "\n", \
 -                    __func__, ##__VA_ARGS__);       \
 -        }                                           \
 -    } while (0)
 -
  typedef struct BDRVSSHState {
      /* Coroutine. */
      CoMutex lock;
@@ -XXX,XX +XXX,XX @@ static int check_host_key_knownhosts(BDRVSSHState *s,
      switch (r) {
      case LIBSSH2_KNOWNHOST_CHECK_MATCH:
          /* OK */
 -        DPRINTF("host key OK: %s", found->key);
 +        trace_ssh_check_host_key_knownhosts(found->key);
          break;
      case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
          ret = -EINVAL;
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
      }
      /* Open the remote file. */
 -    DPRINTF("opening file %s flags=0x%x creat_mode=0%o",
 -            opts->path, ssh_flags, creat_mode);
 +    trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
      s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
                                         creat_mode);
      if (!s->sftp_handle) {
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_create_opts(const char *filename, QemuOpts *opts,
      /* Get desired file size. */
      ssh_opts->size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
                                BDRV_SECTOR_SIZE);
 -    DPRINTF("total_size=%" PRIi64, ssh_opts->size);
 +    trace_ssh_co_create_opts(ssh_opts->size);
      uri_options = qdict_new();
      ret = parse_uri(filename, uri_options, errp);
@@ -XXX,XX +XXX,XX @@ static void restart_coroutine(void *opaque)
      BDRVSSHState *s = bs->opaque;
      AioContext *ctx = bdrv_get_aio_context(bs);
 -    DPRINTF("co=%p", restart->co);
 +    trace_ssh_restart_coroutine(restart->co);
      aio_set_fd_handler(ctx, s->sock, false, NULL, NULL, NULL, NULL);
      aio_co_wake(restart->co);
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
          wr_handler = restart_coroutine;
      }
 -    DPRINTF("s->sock=%d rd_handler=%p wr_handler=%p", s->sock,
 -            rd_handler, wr_handler);
 +    trace_ssh_co_yield(s->sock, rd_handler, wr_handler);
      aio_set_fd_handler(bdrv_get_aio_context(bs), s->sock,
                         false, rd_handler, wr_handler, NULL, &restart);
      qemu_coroutine_yield();
 -    DPRINTF("s->sock=%d - back", s->sock);
 +    trace_ssh_co_yield_back(s->sock);
  }
- /* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
++# Check that a set of drivers has been whitelisted in the QEMU binary
-@@ -XXX,XX +XXX,XX @@ static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
++#
-     bool force = (flags & SSH_SEEK_FORCE) != 0;
++_require_drivers()
++{
-     if (force || op_read != s->offset_op_read || offset != s->offset) {
++    available=$($QEMU -drive format=help | \
--        DPRINTF("seeking to offset=%" PRIi64, offset);
++                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
-+        trace_ssh_seek(offset);
++    for driver
-         libssh2_sftp_seek64(s->sftp_handle, offset);
++    do
-         s->offset = offset;
++        if ! echo "$available" | grep -q " $driver\( \|$\)"; then
-         s->offset_op_read = op_read;
++            _notrun "$driver not available"
-@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
++        fi
-     char *buf, *end_of_vec;
++    done
-     struct iovec *i;
++}
 -    DPRINTF("offset=%" PRIi64 " size=%zu", offset, size);
 +    trace_ssh_read(offset, size);
      ssh_seek(s, offset, SSH_SEEK_READ);
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
       */
      for (got = 0; got < size; ) {
      again:
 -        DPRINTF("sftp_read buf=%p size=%zu", buf, end_of_vec - buf);
 +        trace_ssh_read_buf(buf, end_of_vec - buf);
          r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
 -        DPRINTF("sftp_read returned %zd", r);
 +        trace_ssh_read_return(r);
          if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
              co_yield(s, bs);
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
      char *buf, *end_of_vec;
      struct iovec *i;
 -    DPRINTF("offset=%" PRIi64 " size=%zu", offset, size);
 +    trace_ssh_write(offset, size);
      ssh_seek(s, offset, SSH_SEEK_WRITE);
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
      for (written = 0; written < size; ) {
      again:
 -        DPRINTF("sftp_write buf=%p size=%zu", buf, end_of_vec - buf);
 +        trace_ssh_write_buf(buf, end_of_vec - buf);
          r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
 -        DPRINTF("sftp_write returned %zd", r);
 +        trace_ssh_write_return(r);
          if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
              co_yield(s, bs);
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
  {
      int r;
 -    DPRINTF("fsync");
 +    trace_ssh_flush();
   again:
      r = libssh2_sftp_fsync(s->sftp_handle);
      if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
@@ -XXX,XX +XXX,XX @@ static int64_t ssh_getlength(BlockDriverState *bs)
      /* Note we cannot make a libssh2 call here. */
      length = (int64_t) s->attrs.filesize;
 -    DPRINTF("length=%" PRIi64, length);
 +    trace_ssh_getlength(length);
      return length;
  }
 diff --git a/block/trace-events b/block/trace-events
 index XXXXXXX..XXXXXXX 100644
 --- a/block/trace-events
 +++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ iscsi_xcopy(void *src_lun, uint64_t src_off, void *dst_lun, uint64_t dst_off, ui
  # block/nbd-client.c
  nbd_read_reply_entry_fail(int ret, const char *err) "ret = %d, err: %s"
  nbd_co_request_fail(uint64_t from, uint32_t len, uint64_t handle, uint16_t flags, uint16_t type, const char *name, int ret, const char *err) "Request failed { .from = %" PRIu64", .len = %" PRIu32 ", .handle = %" PRIu64 ", .flags = 0x%" PRIx16 ", .type = %" PRIu16 " (%s) } ret = %d, err: %s"
 +
-+# block/ssh.c
+ # make sure this script returns success
-+ssh_restart_coroutine(void *co) "co=%p"
+ true
 +ssh_flush(void) "fsync"
 +ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
 +ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
 +ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
 +ssh_co_yield_back(int sock) "s->sock=%d - back"
 +ssh_getlength(int64_t length) "length=%" PRIi64
 +ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
 +ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
 +ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
 +ssh_read_return(ssize_t ret) "sftp_read returned %zd"
 +ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
 +ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
 +ssh_write_return(ssize_t ret) "sftp_write returned %zd"
 +ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 07/13] qemu-io: Add generic function for reinitializing optind.
+[Qemu-devel] [PULL 14/15] tests/check-block: Skip iotests when sanitizers are enabled
-From: "Richard W.M. Jones" <rjones@redhat.com>
+From: Thomas Huth <thuth@redhat.com>
-On FreeBSD 11.2:
+The sanitizers (especially the address sanitizer from Clang) are
 sometimes printing out warnings or false positives - this spoils
 the output of the iotests, causing some of the tests to fail.
 Thus let's skip the automatic iotests during "make check" when the
 user configured QEMU with --enable-sanitizers.
-  $ nbdkit memory size=1M --run './qemu-io -f raw -c "aio_write 0 512" $nbd'
+Signed-off-by: Thomas Huth <thuth@redhat.com>
-  Parsing error: non-numeric argument, or extraneous/unrecognized suffix -- aio_write
+Message-id: 20190823084203.29734-1-thuth@redhat.com
 After main option parsing, we reinitialize optind so we can parse each
 command.  However reinitializing optind to 0 does not work on FreeBSD.
 What happens when you do this is optind remains 0 after the option
 parsing loop, and the result is we try to parse argv[optind] ==
 argv[0] == "aio_write" as if it was the first parameter.
 The FreeBSD manual page says:
   In order to use getopt() to evaluate multiple sets of arguments, or to
   evaluate a single set of arguments multiple times, the variable optreset
   must be set to 1 before the second and each additional set of calls to
   getopt(), and the variable optind must be reinitialized.
 (From the rest of the man page it is clear that optind must be
 reinitialized to 1).
 The glibc man page says:
   A program that scans multiple argument vectors,  or  rescans  the  same
   vector  more than once, and wants to make use of GNU extensions such as
   '+' and '-' at  the  start  of  optstring,  or  changes  the  value  of
   POSIXLY_CORRECT  between scans, must reinitialize getopt() by resetting
   optind to 0, rather than the traditional value of 1.  (Resetting  to  0
   forces  the  invocation  of  an  internal  initialization  routine that
   rechecks POSIXLY_CORRECT and checks for GNU extensions in optstring.)
 This commit introduces an OS-portability function called
 qemu_reset_optind which provides a way of resetting optind that works
 on FreeBSD and platforms that use optreset, while keeping it the same
 as now on other platforms.
 Note that the qemu codebase sets optind in many other places, but in
 those other places it's setting a local variable and not using getopt.
 This change is only needed in places where we are using getopt and the
 associated global variable optind.
 Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
 Message-id: 20190118101114.11759-2-rjones@redhat.com
 Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
 Reviewed-by: Eric Blake <eblake@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- configure            | 14 ++++++++++++++
+ tests/check-block.sh | 5 +++++
- include/qemu/osdep.h | 16 ++++++++++++++++
+file changed, 5 insertions(+)
  qemu-img.c           |  2 +-
  qemu-io-cmds.c       |  2 +-
 files changed, 32 insertions(+), 2 deletions(-)
-diff --git a/configure b/configure
+diff --git a/tests/check-block.sh b/tests/check-block.sh
 index XXXXXXX..XXXXXXX 100755
---- a/configure
+--- a/tests/check-block.sh
-+++ b/configure
++++ b/tests/check-block.sh
-@@ -XXX,XX +XXX,XX @@ if compile_prog "" "" ; then
+@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
-   signalfd=yes
+     exit 0
  fi
-+# check if optreset global is declared by <getopt.h>
++if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
-+optreset="no"
++    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
-+cat > $TMPC << EOF
++    exit 0
 +#include <getopt.h>
 +int main(void) { return optreset; }
 +EOF
 +
 +if compile_prog "" "" ; then
 +  optreset=yes
 +fi
 +
- # check if eventfd is supported
+ if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
- eventfd=no
+     echo "No qemu-system binary available ==> Not running the qemu-iotests."
- cat > $TMPC << EOF
+     exit 0
@@ -XXX,XX +XXX,XX @@ fi
  if test "$signalfd" = "yes" ; then
    echo "CONFIG_SIGNALFD=y" >> $config_host_mak
  fi
 +if test "$optreset" = "yes" ; then
 +  echo "HAVE_OPTRESET=y" >> $config_host_mak
 +fi
  if test "$tcg" = "yes"; then
    echo "CONFIG_TCG=y" >> $config_host_mak
    if test "$tcg_interpreter" = "yes" ; then
 diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/qemu/osdep.h
 +++ b/include/qemu/osdep.h
@@ -XXX,XX +XXX,XX @@ extern int daemon(int, int);
  #include <ctype.h>
  #include <errno.h>
  #include <fcntl.h>
 +#include <getopt.h>
  #include <sys/stat.h>
  #include <sys/time.h>
  #include <assert.h>
@@ -XXX,XX +XXX,XX @@ extern int qemu_icache_linesize_log;
  extern int qemu_dcache_linesize;
  extern int qemu_dcache_linesize_log;
 +/*
 + * After using getopt or getopt_long, if you need to parse another set
 + * of options, then you must reset optind.  Unfortunately the way to
 + * do this varies between implementations of getopt.
 + */
 +static inline void qemu_reset_optind(void)
 +{
 +#ifdef HAVE_OPTRESET
 +    optind = 1;
 +    optreset = 1;
 +#else
 +    optind = 0;
 +#endif
 +}
 +
  #endif
 diff --git a/qemu-img.c b/qemu-img.c
 index XXXXXXX..XXXXXXX 100644
 --- a/qemu-img.c
 +++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
          return 0;
      }
      argv += optind;
 -    optind = 0;
 +    qemu_reset_optind();
      if (!trace_init_backends()) {
          exit(1);
 diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
 index XXXXXXX..XXXXXXX 100644
 --- a/qemu-io-cmds.c
 +++ b/qemu-io-cmds.c
@@ -XXX,XX +XXX,XX @@ static int command(BlockBackend *blk, const cmdinfo_t *ct, int argc,
          }
      }
 -    optind = 0;
 +    qemu_reset_optind();
      return ct->cfunc(blk, argc, argv);
  }
 --
-.20.1
+.21.0

-[Qemu-devel] [PULL 10/13] nvme: use pci_dev directly in nvme_realize
+[Qemu-devel] [PULL 15/15] iotests: Unify cache mode quoting
-From: Li Qiang <liq3ea@163.com>
+From: Nir Soffer <nirsof@gmail.com>
-There is no need to make another reference.
+Quoting cache mode is not needed, and most tests use unquoted values.
 Unify all test to use the same style.
-Signed-off-by: Li Qiang <liq3ea@163.com>
+Message-id: 20190827173432.7656-1-nsoffer@redhat.com
-Reviewed-by: Max Reitz <mreitz@redhat.com>
+Signed-off-by: Nir Soffer <nsoffer@redhat.com>
 Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
 Message-id: 20190120055558.32984-4-liq3ea@163.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- hw/block/nvme.c | 8 ++++----
+ tests/qemu-iotests/026 | 4 ++--
-file changed, 4 insertions(+), 4 deletions(-)
+ tests/qemu-iotests/039 | 4 ++--
  tests/qemu-iotests/052 | 2 +-
  tests/qemu-iotests/091 | 4 ++--
 files changed, 7 insertions(+), 7 deletions(-)
-diff --git a/hw/block/nvme.c b/hw/block/nvme.c
+diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/hw/block/nvme.c
+--- a/tests/qemu-iotests/026
-+++ b/hw/block/nvme.c
++++ b/tests/qemu-iotests/026
-@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-     pci_conf[PCI_INTERRUPT_PIN] = 1;
+ # Currently only qcow2 supports rebasing
-     pci_config_set_prog_interface(pci_dev->config, 0x2);
+ _supported_fmt qcow2
-     pci_config_set_class(pci_dev->config, PCI_CLASS_STORAGE_EXPRESS);
+ _supported_proto file
--    pcie_endpoint_cap_init(&n->parent_obj, 0x80);
+-_default_cache_mode "writethrough"
-+    pcie_endpoint_cap_init(pci_dev, 0x80);
+-_supported_cache_modes "writethrough" "none"
++_default_cache_mode writethrough
-     n->num_namespaces = 1;
++_supported_cache_modes writethrough none
-     n->reg_size = pow2ceil(0x1004 + 2 * (n->num_queues + 1) * 4);
+ # The refcount table tests expect a certain minimum width for refcount entries
-@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
+ # (so that the refcount table actually needs to grow); that minimum is 16 bits,
+ # being the default refcount entry width.
-     memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n,
+diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
-                           "nvme", n->reg_size);
+index XXXXXXX..XXXXXXX 100755
--    pci_register_bar(&n->parent_obj, 0,
+--- a/tests/qemu-iotests/039
-+    pci_register_bar(pci_dev, 0,
++++ b/tests/qemu-iotests/039
-         PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64,
+@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
-         &n->iomem);
+ _supported_fmt qcow2
--    msix_init_exclusive_bar(&n->parent_obj, n->num_queues, 4, NULL);
+ _supported_proto file
-+    msix_init_exclusive_bar(pci_dev, n->num_queues, 4, NULL);
+ _supported_os Linux
+-_default_cache_mode "writethrough"
-     id->vid = cpu_to_le16(pci_get_word(pci_conf + PCI_VENDOR_ID));
+-_supported_cache_modes "writethrough"
-     id->ssvid = cpu_to_le16(pci_get_word(pci_conf + PCI_SUBSYSTEM_VENDOR_ID));
++_default_cache_mode writethrough
-@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
++_supported_cache_modes writethrough
-         n->cmbuf = g_malloc0(NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
-         memory_region_init_io(&n->ctrl_mem, OBJECT(n), &nvme_cmb_ops, n,
+ size=128M
-                               "nvme-cmb", NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
--        pci_register_bar(&n->parent_obj, NVME_CMBLOC_BIR(n->bar.cmbloc),
+diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
-+        pci_register_bar(pci_dev, NVME_CMBLOC_BIR(n->bar.cmbloc),
+index XXXXXXX..XXXXXXX 100755
-             PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64 |
+--- a/tests/qemu-iotests/052
-             PCI_BASE_ADDRESS_MEM_PREFETCH, &n->ctrl_mem);
++++ b/tests/qemu-iotests/052
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto file
  # Don't do O_DIRECT on tmpfs
 -_supported_cache_modes "writeback" "writethrough" "unsafe"
 +_supported_cache_modes writeback writethrough unsafe
  size=128M
  _make_test_img $size
 diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/091
 +++ b/tests/qemu-iotests/091
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
  _supported_os Linux
 -_default_cache_mode "none"
 -_supported_cache_modes "writethrough" "none" "writeback"
 +_default_cache_mode none
 +_supported_cache_modes writethrough none writeback
  size=1G
 --
-.20.1
+.21.0

The following changes since commit b4fbe1f65a4769c09e6bf2d79fc84360f840f40e:

Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20190129' into staging (2019-01-29 12:00:19 +0000)

are available in the Git repository at:

https://git.xanclic.moe/XanClic/qemu.git tags/pull-block-2019-01-31

for you to fetch changes up to 908b30164bbffad7430d551b2a03a8fbcaa631ef:

iotests: Allow 147 to be run concurrently (2019-01-31 00:44:55 +0100)

----------------------------------------------------------------
Block patches:
- New debugging QMP command to explore block graphs
- Converted DPRINTF()s to trace events
- Fixed qemu-io's use of getopt() for systems with optreset
- Minor NVMe emulation fixes
- An iotest fix

----------------------------------------------------------------
Laurent Vivier (4):
  block/ssh: Convert from DPRINTF() macro to trace events
  block/curl: Convert from DPRINTF() macro to trace events
  block/file-posix: Convert from DPRINTF() macro to trace events
  block/sheepdog: Convert from DPRINTF() macro to trace events

Li Qiang (3):
  nvme: use TYPE_NVME instead of constant string
  nvme: ensure the num_queues is not zero
  nvme: use pci_dev directly in nvme_realize

Max Reitz (3):
  iotests.py: Add qemu_nbd_pipe()
  iotests: Bind qemu-nbd to localhost in 147
  iotests: Allow 147 to be run concurrently

Richard W.M. Jones (1):
  qemu-io: Add generic function for reinitializing optind.

Vladimir Sementsov-Ogievskiy (2):
  qapi: add x-debug-query-block-graph
  scripts: add render_block_graph function for QEMUMachine

-- 
2.20.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Add a new command, returning block nodes (and their users) graph.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20181221170909.25584-2-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json           | 108 ++++++++++++++++++++++++
 include/block/block.h          |   1 +
 include/sysemu/block-backend.h |   2 +
 block.c                        | 148 +++++++++++++++++++++++++++++++++
 block/block-backend.c          |   5 ++
 blockdev.c                     |   5 ++
 6 files changed, 269 insertions(+)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 ##
 { 'command': 'query-named-block-nodes', 'returns': [ 'BlockDeviceInfo' ] }
 
+##
+# @XDbgBlockGraphNodeType:
+#
+# @block-backend: corresponds to BlockBackend
+#
+# @block-job: corresonds to BlockJob
+#
+# @block-driver: corresponds to BlockDriverState
+#
+# Since: 4.0
+##
+{ 'enum': 'XDbgBlockGraphNodeType',
+  'data': [ 'block-backend', 'block-job', 'block-driver' ] }
+
+##
+# @XDbgBlockGraphNode:
+#
+# @id: Block graph node identifier. This @id is generated only for
+#      x-debug-query-block-graph and does not relate to any other identifiers in
+#      Qemu.
+#
+# @type: Type of graph node. Can be one of block-backend, block-job or
+#        block-driver-state.
+#
+# @name: Human readable name of the node. Corresponds to node-name for
+#        block-driver-state nodes; is not guaranteed to be unique in the whole
+#        graph (with block-jobs and block-backends).
+#
+# Since: 4.0
+##
+{ 'struct': 'XDbgBlockGraphNode',
+  'data': { 'id': 'uint64', 'type': 'XDbgBlockGraphNodeType', 'name': 'str' } }
+
+##
+# @BlockPermission:
+#
+# Enum of base block permissions.
+#
+# @consistent-read: A user that has the "permission" of consistent reads is
+#                   guaranteed that their view of the contents of the block
+#                   device is complete and self-consistent, representing the
+#                   contents of a disk at a specific point.
+#                   For most block devices (including their backing files) this
+#                   is true, but the property cannot be maintained in a few
+#                   situations like for intermediate nodes of a commit block
+#                   job.
+#
+# @write: This permission is required to change the visible disk contents.
+#
+# @write-unchanged: This permission (which is weaker than BLK_PERM_WRITE) is
+#                   both enough and required for writes to the block node when
+#                   the caller promises that the visible disk content doesn't
+#                   change.
+#                   As the BLK_PERM_WRITE permission is strictly stronger,
+#                   either is sufficient to perform an unchanging write.
+#
+# @resize: This permission is required to change the size of a block node.
+#
+# @graph-mod: This permission is required to change the node that this
+#             BdrvChild points to.
+#
+# Since: 4.0
+##
+  { 'enum': 'BlockPermission',
+    'data': [ 'consistent-read', 'write', 'write-unchanged', 'resize',
+              'graph-mod' ] }
+##
+# @XDbgBlockGraphEdge:
+#
+# Block Graph edge description for x-debug-query-block-graph.
+#
+# @parent: parent id
+#
+# @child: child id
+#
+# @name: name of the relation (examples are 'file' and 'backing')
+#
+# @perm: granted permissions for the parent operating on the child
+#
+# @shared-perm: permissions that can still be granted to other users of the
+#               child while it is still attached to this parent
+#
+# Since: 4.0
+##
+{ 'struct': 'XDbgBlockGraphEdge',
+  'data': { 'parent': 'uint64', 'child': 'uint64',
+            'name': 'str', 'perm': [ 'BlockPermission' ],
+            'shared-perm': [ 'BlockPermission' ] } }
+
+##
+# @XDbgBlockGraph:
+#
+# Block Graph - list of nodes and list of edges.
+#
+# Since: 4.0
+##
+{ 'struct': 'XDbgBlockGraph',
+  'data': { 'nodes': ['XDbgBlockGraphNode'], 'edges': ['XDbgBlockGraphEdge'] } }
+
+##
+# @x-debug-query-block-graph:
+#
+# Get the block graph.
+#
+# Since: 4.0
+##
+{ 'command': 'x-debug-query-block-graph', 'returns': 'XDbgBlockGraph' }
+
 ##
 # @drive-mirror:
 #
diff --git a/include/block/block.h b/include/block/block.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ void bdrv_eject(BlockDriverState *bs, bool eject_flag);
 const char *bdrv_get_format_name(BlockDriverState *bs);
 BlockDriverState *bdrv_find_node(const char *node_name);
 BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp);
+XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp);
 BlockDriverState *bdrv_lookup_bs(const char *device,
                                  const char *node_name,
                                  Error **errp);
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index XXXXXXX..XXXXXXX 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -XXX,XX +XXX,XX @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in,
                                    int bytes, BdrvRequestFlags read_flags,
                                    BdrvRequestFlags write_flags);
 
+const BdrvChild *blk_root(BlockBackend *blk);
+
 #endif
diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp)
     return list;
 }
 
+#define QAPI_LIST_ADD(list, element) do { \
+    typeof(list) _tmp = g_new(typeof(*(list)), 1); \
+    _tmp->value = (element); \
+    _tmp->next = (list); \
+    (list) = _tmp; \
+} while (0)
+
+typedef struct XDbgBlockGraphConstructor {
+    XDbgBlockGraph *graph;
+    GHashTable *graph_nodes;
+} XDbgBlockGraphConstructor;
+
+static XDbgBlockGraphConstructor *xdbg_graph_new(void)
+{
+    XDbgBlockGraphConstructor *gr = g_new(XDbgBlockGraphConstructor, 1);
+
+    gr->graph = g_new0(XDbgBlockGraph, 1);
+    gr->graph_nodes = g_hash_table_new(NULL, NULL);
+
+    return gr;
+}
+
+static XDbgBlockGraph *xdbg_graph_finalize(XDbgBlockGraphConstructor *gr)
+{
+    XDbgBlockGraph *graph = gr->graph;
+
+    g_hash_table_destroy(gr->graph_nodes);
+    g_free(gr);
+
+    return graph;
+}
+
+static uintptr_t xdbg_graph_node_num(XDbgBlockGraphConstructor *gr, void *node)
+{
+    uintptr_t ret = (uintptr_t)g_hash_table_lookup(gr->graph_nodes, node);
+
+    if (ret != 0) {
+        return ret;
+    }
+
+    /*
+     * Start counting from 1, not 0, because 0 interferes with not-found (NULL)
+     * answer of g_hash_table_lookup.
+     */
+    ret = g_hash_table_size(gr->graph_nodes) + 1;
+    g_hash_table_insert(gr->graph_nodes, node, (void *)ret);
+
+    return ret;
+}
+
+static void xdbg_graph_add_node(XDbgBlockGraphConstructor *gr, void *node,
+                                XDbgBlockGraphNodeType type, const char *name)
+{
+    XDbgBlockGraphNode *n;
+
+    n = g_new0(XDbgBlockGraphNode, 1);
+
+    n->id = xdbg_graph_node_num(gr, node);
+    n->type = type;
+    n->name = g_strdup(name);
+
+    QAPI_LIST_ADD(gr->graph->nodes, n);
+}
+
+static void xdbg_graph_add_edge(XDbgBlockGraphConstructor *gr, void *parent,
+                                const BdrvChild *child)
+{
+    typedef struct {
+        unsigned int flag;
+        BlockPermission num;
+    } PermissionMap;
+
+    static const PermissionMap permissions[] = {
+        { BLK_PERM_CONSISTENT_READ, BLOCK_PERMISSION_CONSISTENT_READ },
+        { BLK_PERM_WRITE,           BLOCK_PERMISSION_WRITE },
+        { BLK_PERM_WRITE_UNCHANGED, BLOCK_PERMISSION_WRITE_UNCHANGED },
+        { BLK_PERM_RESIZE,          BLOCK_PERMISSION_RESIZE },
+        { BLK_PERM_GRAPH_MOD,       BLOCK_PERMISSION_GRAPH_MOD },
+        { 0, 0 }
+    };
+    const PermissionMap *p;
+    XDbgBlockGraphEdge *edge;
+
+    QEMU_BUILD_BUG_ON(1UL << (ARRAY_SIZE(permissions) - 1) != BLK_PERM_ALL + 1);
+
+    edge = g_new0(XDbgBlockGraphEdge, 1);
+
+    edge->parent = xdbg_graph_node_num(gr, parent);
+    edge->child = xdbg_graph_node_num(gr, child->bs);
+    edge->name = g_strdup(child->name);
+
+    for (p = permissions; p->flag; p++) {
+        if (p->flag & child->perm) {
+            QAPI_LIST_ADD(edge->perm, p->num);
+        }
+        if (p->flag & child->shared_perm) {
+            QAPI_LIST_ADD(edge->shared_perm, p->num);
+        }
+    }
+
+    QAPI_LIST_ADD(gr->graph->edges, edge);
+}
+
+
+XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
+{
+    BlockBackend *blk;
+    BlockJob *job;
+    BlockDriverState *bs;
+    BdrvChild *child;
+    XDbgBlockGraphConstructor *gr = xdbg_graph_new();
+
+    for (blk = blk_all_next(NULL); blk; blk = blk_all_next(blk)) {
+        char *allocated_name = NULL;
+        const char *name = blk_name(blk);
+
+        if (!*name) {
+            name = allocated_name = blk_get_attached_dev_id(blk);
+        }
+        xdbg_graph_add_node(gr, blk, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_BACKEND,
+                           name);
+        g_free(allocated_name);
+        if (blk_root(blk)) {
+            xdbg_graph_add_edge(gr, blk, blk_root(blk));
+        }
+    }
+
+    for (job = block_job_next(NULL); job; job = block_job_next(job)) {
+        GSList *el;
+
+        xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
+                           job->job.id);
+        for (el = job->nodes; el; el = el->next) {
+            xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
+        }
+    }
+
+    QTAILQ_FOREACH(bs, &graph_bdrv_states, node_list) {
+        xdbg_graph_add_node(gr, bs, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_DRIVER,
+                           bs->node_name);
+        QLIST_FOREACH(child, &bs->children, next) {
+            xdbg_graph_add_edge(gr, bs, child);
+        }
+    }
+
+    return xdbg_graph_finalize(gr);
+}
+
 BlockDriverState *bdrv_lookup_bs(const char *device,
                                  const char *node_name,
                                  Error **errp)
diff --git a/block/block-backend.c b/block/block-backend.c
index XXXXXXX..XXXXXXX 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -XXX,XX +XXX,XX @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in,
                               blk_out->root, off_out,
                               bytes, read_flags, write_flags);
 }
+
+const BdrvChild *blk_root(BlockBackend *blk)
+{
+    return blk->root;
+}
diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ BlockDeviceInfoList *qmp_query_named_block_nodes(Error **errp)
     return bdrv_named_nodes_list(errp);
 }
 
+XDbgBlockGraph *qmp_x_debug_query_block_graph(Error **errp)
+{
+    return bdrv_get_xdbg_block_graph(errp);
+}
+
 BlockJob *do_blockdev_backup(BlockdevBackup *backup, JobTxn *txn,
                              Error **errp)
 {
-- 
2.20.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Render block nodes graph with help of graphviz. This new function is
for debugging, so there is no sense to put it into qemu.py as a method
of QEMUMachine. Let's instead put it separately.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Acked-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20181221170909.25584-3-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 scripts/render_block_graph.py | 120 ++++++++++++++++++++++++++++++++++
 1 file changed, 120 insertions(+)
 create mode 100755 scripts/render_block_graph.py

diff --git a/scripts/render_block_graph.py b/scripts/render_block_graph.py
new file mode 100755
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/scripts/render_block_graph.py
@@ -XXX,XX +XXX,XX @@
+#!/usr/bin/env python
+#
+# Render Qemu Block Graph
+#
+# Copyright (c) 2018 Virtuozzo International GmbH. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import sys
+import subprocess
+import json
+from graphviz import Digraph
+from qemu import MonitorResponseError
+
+
+def perm(arr):
+    s = 'w' if 'write' in arr else '_'
+    s += 'r' if 'consistent-read' in arr else '_'
+    s += 'u' if 'write-unchanged' in arr else '_'
+    s += 'g' if 'graph-mod' in arr else '_'
+    s += 's' if 'resize' in arr else '_'
+    return s
+
+
+def render_block_graph(qmp, filename, format='png'):
+    '''
+    Render graph in text (dot) representation into "@filename" and
+    representation in @format into "@filename.@format"
+    '''
+
+    bds_nodes = qmp.command('query-named-block-nodes')
+    bds_nodes = {n['node-name']: n for n in bds_nodes}
+
+    job_nodes = qmp.command('query-block-jobs')
+    job_nodes = {n['device']: n for n in job_nodes}
+
+    block_graph = qmp.command('x-debug-query-block-graph')
+
+    graph = Digraph(comment='Block Nodes Graph')
+    graph.format = format
+    graph.node('permission symbols:\l'
+               '  w - Write\l'
+               '  r - consistent-Read\l'
+               '  u - write - Unchanged\l'
+               '  g - Graph-mod\l'
+               '  s - reSize\l'
+               'edge label scheme:\l'
+               '  <child type>\l'
+               '  <perm>\l'
+               '  <shared_perm>\l', shape='none')
+
+    for n in block_graph['nodes']:
+        if n['type'] == 'block-driver':
+            info = bds_nodes[n['name']]
+            label = n['name'] + ' [' + info['drv'] + ']'
+            if info['drv'] == 'file':
+                label += '\n' + os.path.basename(info['file'])
+            shape = 'ellipse'
+        elif n['type'] == 'block-job':
+            info = job_nodes[n['name']]
+            label = info['type'] + ' job (' + n['name'] + ')'
+            shape = 'box'
+        else:
+            assert n['type'] == 'block-backend'
+            label = n['name'] if n['name'] else 'unnamed blk'
+            shape = 'box'
+
+        graph.node(str(n['id']), label, shape=shape)
+
+    for e in block_graph['edges']:
+        label = '%s\l%s\l%s\l' % (e['name'], perm(e['perm']),
+                                  perm(e['shared-perm']))
+        graph.edge(str(e['parent']), str(e['child']), label=label)
+
+    graph.render(filename)
+
+
+class LibvirtGuest():
+    def __init__(self, name):
+        self.name = name
+
+    def command(self, cmd):
+        # only supports qmp commands without parameters
+        m = {'execute': cmd}
+        ar = ['virsh', 'qemu-monitor-command', self.name, json.dumps(m)]
+
+        reply = json.loads(subprocess.check_output(ar))
+
+        if 'error' in reply:
+            raise MonitorResponseError(reply)
+
+        return reply['return']
+
+
+if __name__ == '__main__':
+    obj = sys.argv[1]
+    out = sys.argv[2]
+
+    if os.path.exists(obj):
+        # assume unix socket
+        qmp = QEMUMonitorProtocol(obj)
+        qmp.connect()
+    else:
+        # assume libvirt guest name
+        qmp = LibvirtGuest(obj)
+
+    render_block_graph(qmp, out)
-- 
2.20.1

From: Laurent Vivier <lvivier@redhat.com>

Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20181213162727.17438-2-lvivier@redhat.com
[mreitz: Fixed type of ssh_{read,write}_return's parameter to be ssize_t
         instead of size_t]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/ssh.c        | 46 +++++++++++++++++-----------------------------
 block/trace-events | 17 +++++++++++++++++
 2 files changed, 34 insertions(+), 29 deletions(-)

diff --git a/block/ssh.c b/block/ssh.c
index XXXXXXX..XXXXXXX 100644
--- a/block/ssh.c
+++ b/block/ssh.c
@@ -XXX,XX +XXX,XX @@
 #include "qapi/qmp/qstring.h"
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/qobject-output-visitor.h"
+#include "trace.h"
 
-/* DEBUG_SSH=1 enables the DPRINTF (debugging printf) statements in
- * this block driver code.
- *
+/*
  * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
  * that this requires that libssh2 was specially compiled with the
  * `./configure --enable-debug' option, so most likely you will have
  * to compile it yourself.  The meaning of <bitmask> is described
  * here: http://www.libssh2.org/libssh2_trace.html
  */
-#define DEBUG_SSH     0
 #define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
 
-#define DPRINTF(fmt, ...)                           \
-    do {                                            \
-        if (DEBUG_SSH) {                            \
-            fprintf(stderr, "ssh: %-15s " fmt "\n", \
-                    __func__, ##__VA_ARGS__);       \
-        }                                           \
-    } while (0)
-
 typedef struct BDRVSSHState {
     /* Coroutine. */
     CoMutex lock;
@@ -XXX,XX +XXX,XX @@ static int check_host_key_knownhosts(BDRVSSHState *s,
     switch (r) {
     case LIBSSH2_KNOWNHOST_CHECK_MATCH:
         /* OK */
-        DPRINTF("host key OK: %s", found->key);
+        trace_ssh_check_host_key_knownhosts(found->key);
         break;
     case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
         ret = -EINVAL;
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
     }
 
     /* Open the remote file. */
-    DPRINTF("opening file %s flags=0x%x creat_mode=0%o",
-            opts->path, ssh_flags, creat_mode);
+    trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
     s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
                                        creat_mode);
     if (!s->sftp_handle) {
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_create_opts(const char *filename, QemuOpts *opts,
     /* Get desired file size. */
     ssh_opts->size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
                               BDRV_SECTOR_SIZE);
-    DPRINTF("total_size=%" PRIi64, ssh_opts->size);
+    trace_ssh_co_create_opts(ssh_opts->size);
 
     uri_options = qdict_new();
     ret = parse_uri(filename, uri_options, errp);
@@ -XXX,XX +XXX,XX @@ static void restart_coroutine(void *opaque)
     BDRVSSHState *s = bs->opaque;
     AioContext *ctx = bdrv_get_aio_context(bs);
 
-    DPRINTF("co=%p", restart->co);
+    trace_ssh_restart_coroutine(restart->co);
     aio_set_fd_handler(ctx, s->sock, false, NULL, NULL, NULL, NULL);
 
     aio_co_wake(restart->co);
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
         wr_handler = restart_coroutine;
     }
 
-    DPRINTF("s->sock=%d rd_handler=%p wr_handler=%p", s->sock,
-            rd_handler, wr_handler);
+    trace_ssh_co_yield(s->sock, rd_handler, wr_handler);
 
     aio_set_fd_handler(bdrv_get_aio_context(bs), s->sock,
                        false, rd_handler, wr_handler, NULL, &restart);
     qemu_coroutine_yield();
-    DPRINTF("s->sock=%d - back", s->sock);
+    trace_ssh_co_yield_back(s->sock);
 }
 
 /* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
@@ -XXX,XX +XXX,XX @@ static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
     bool force = (flags & SSH_SEEK_FORCE) != 0;
 
     if (force || op_read != s->offset_op_read || offset != s->offset) {
-        DPRINTF("seeking to offset=%" PRIi64, offset);
+        trace_ssh_seek(offset);
         libssh2_sftp_seek64(s->sftp_handle, offset);
         s->offset = offset;
         s->offset_op_read = op_read;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
     char *buf, *end_of_vec;
     struct iovec *i;
 
-    DPRINTF("offset=%" PRIi64 " size=%zu", offset, size);
+    trace_ssh_read(offset, size);
 
     ssh_seek(s, offset, SSH_SEEK_READ);
 
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
      */
     for (got = 0; got < size; ) {
     again:
-        DPRINTF("sftp_read buf=%p size=%zu", buf, end_of_vec - buf);
+        trace_ssh_read_buf(buf, end_of_vec - buf);
         r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
-        DPRINTF("sftp_read returned %zd", r);
+        trace_ssh_read_return(r);
 
         if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
             co_yield(s, bs);
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
     char *buf, *end_of_vec;
     struct iovec *i;
 
-    DPRINTF("offset=%" PRIi64 " size=%zu", offset, size);
+    trace_ssh_write(offset, size);
 
     ssh_seek(s, offset, SSH_SEEK_WRITE);
 
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
 
     for (written = 0; written < size; ) {
     again:
-        DPRINTF("sftp_write buf=%p size=%zu", buf, end_of_vec - buf);
+        trace_ssh_write_buf(buf, end_of_vec - buf);
         r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
-        DPRINTF("sftp_write returned %zd", r);
+        trace_ssh_write_return(r);
 
         if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
             co_yield(s, bs);
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
 {
     int r;
 
-    DPRINTF("fsync");
+    trace_ssh_flush();
  again:
     r = libssh2_sftp_fsync(s->sftp_handle);
     if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
@@ -XXX,XX +XXX,XX @@ static int64_t ssh_getlength(BlockDriverState *bs)
 
     /* Note we cannot make a libssh2 call here. */
     length = (int64_t) s->attrs.filesize;
-    DPRINTF("length=%" PRIi64, length);
+    trace_ssh_getlength(length);
 
     return length;
 }
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ iscsi_xcopy(void *src_lun, uint64_t src_off, void *dst_lun, uint64_t dst_off, ui
 # block/nbd-client.c
 nbd_read_reply_entry_fail(int ret, const char *err) "ret = %d, err: %s"
 nbd_co_request_fail(uint64_t from, uint32_t len, uint64_t handle, uint16_t flags, uint16_t type, const char *name, int ret, const char *err) "Request failed { .from = %" PRIu64", .len = %" PRIu32 ", .handle = %" PRIu64 ", .flags = 0x%" PRIx16 ", .type = %" PRIu16 " (%s) } ret = %d, err: %s"
+
+# block/ssh.c
+ssh_restart_coroutine(void *co) "co=%p"
+ssh_flush(void) "fsync"
+ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
+ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
+ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
+ssh_co_yield_back(int sock) "s->sock=%d - back"
+ssh_getlength(int64_t length) "length=%" PRIi64
+ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
+ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
+ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
+ssh_read_return(ssize_t ret) "sftp_read returned %zd"
+ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
+ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
+ssh_write_return(ssize_t ret) "sftp_write returned %zd"
+ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
-- 
2.20.1

From: Laurent Vivier <lvivier@redhat.com>

Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Richard W.M. Jones <rjones@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20181213162727.17438-3-lvivier@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c       | 29 ++++++++---------------------
 block/trace-events |  9 +++++++++
 2 files changed, 17 insertions(+), 21 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@
 #include "crypto/secret.h"
 #include <curl/curl.h>
 #include "qemu/cutils.h"
+#include "trace.h"
 
-// #define DEBUG_CURL
 // #define DEBUG_VERBOSE
 
-#ifdef DEBUG_CURL
-#define DEBUG_CURL_PRINT 1
-#else
-#define DEBUG_CURL_PRINT 0
-#endif
-#define DPRINTF(fmt, ...)                                            \
-    do {                                                             \
-        if (DEBUG_CURL_PRINT) {                                      \
-            fprintf(stderr, fmt, ## __VA_ARGS__);                    \
-        }                                                            \
-    } while (0)
-
 #if LIBCURL_VERSION_NUM >= 0x071000
 /* The multi interface timer callback was introduced in 7.16.0 */
 #define NEED_CURL_TIMER_CALLBACK
@@ -XXX,XX +XXX,XX @@ static int curl_timer_cb(CURLM *multi, long timeout_ms, void *opaque)
 {
     BDRVCURLState *s = opaque;
 
-    DPRINTF("CURL: timer callback timeout_ms %ld\n", timeout_ms);
+    trace_curl_timer_cb(timeout_ms);
     if (timeout_ms == -1) {
         timer_del(&s->timer);
     } else {
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
     }
     socket = NULL;
 
-    DPRINTF("CURL (AIO): Sock action %d on fd %d\n", action, (int)fd);
+    trace_curl_sock_cb(action, (int)fd);
     switch (action) {
         case CURL_POLL_IN:
             aio_set_fd_handler(s->aio_context, fd, false,
@@ -XXX,XX +XXX,XX @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
     size_t realsize = size * nmemb;
     int i;
 
-    DPRINTF("CURL: Just reading %zd bytes\n", realsize);
+    trace_curl_read_cb(realsize);
 
     if (!s || !s->orig_buf) {
         goto read_end;
@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
         }
     }
 
-    DPRINTF("CURL: Opening %s\n", file);
+    trace_curl_open(file);
     qemu_co_queue_init(&s->free_state_waitq);
     s->aio_context = bdrv_get_aio_context(bs);
     s->url = g_strdup(file);
@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
                 "Server does not support 'range' (byte ranges).");
         goto out;
     }
-    DPRINTF("CURL: Size = %" PRIu64 "\n", s->len);
+    trace_curl_open_size(s->len);
 
     qemu_mutex_lock(&s->mutex);
     curl_clean_state(state);
@@ -XXX,XX +XXX,XX @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
     state->acb[0] = acb;
 
     snprintf(state->range, 127, "%" PRIu64 "-%" PRIu64, start, end);
-    DPRINTF("CURL (AIO): Reading %" PRIu64 " at %" PRIu64 " (%s)\n",
-            acb->bytes, start, state->range);
+    trace_curl_setup_preadv(acb->bytes, start, state->range);
     curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range);
 
     curl_multi_add_handle(s->multi, state->curl);
@@ -XXX,XX +XXX,XX @@ static void curl_close(BlockDriverState *bs)
 {
     BDRVCURLState *s = bs->opaque;
 
-    DPRINTF("CURL: Close\n");
+    trace_curl_close();
     curl_detach_aio_context(bs);
     qemu_mutex_destroy(&s->mutex);
 
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
 ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
 ssh_write_return(ssize_t ret) "sftp_write returned %zd"
 ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
+
+# block/curl.c
+curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
+curl_sock_cb(int action, int fd) "sock action %d on fd %d"
+curl_read_cb(size_t realsize) "just reading %zu bytes"
+curl_open(const char *file) "opening %s"
+curl_open_size(uint64_t size) "size = %" PRIu64
+curl_setup_preadv(uint64_t bytes, uint64_t start, const char *range) "reading %" PRIu64 " at %" PRIu64 " (%s)"
+curl_close(void) "close"
-- 
2.20.1

From: Laurent Vivier <lvivier@redhat.com>

Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20181213162727.17438-4-lvivier@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c | 25 ++++++-------------------
 block/trace-events |  7 +++++++
 2 files changed, 13 insertions(+), 19 deletions(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@
 #include <xfs/xfs.h>
 #endif
 
-//#define DEBUG_BLOCK
-
-#ifdef DEBUG_BLOCK
-# define DEBUG_BLOCK_PRINT 1
-#else
-# define DEBUG_BLOCK_PRINT 0
-#endif
-#define DPRINTF(fmt, ...) \
-do { \
-    if (DEBUG_BLOCK_PRINT) { \
-        printf(fmt, ## __VA_ARGS__); \
-    } \
-} while (0)
+#include "trace.h"
 
 /* OS X does not have O_DSYNC */
 #ifndef O_DSYNC
@@ -XXX,XX +XXX,XX @@ static int xfs_write_zeroes(BDRVRawState *s, int64_t offset, uint64_t bytes)
 
     if (xfsctl(NULL, s->fd, XFS_IOC_ZERO_RANGE, &fl) < 0) {
         err = errno;
-        DPRINTF("cannot write zero range (%s)\n", strerror(errno));
+        trace_file_xfs_write_zeroes(strerror(errno));
         return -err;
     }
 
@@ -XXX,XX +XXX,XX @@ static int xfs_discard(BDRVRawState *s, int64_t offset, uint64_t bytes)
 
     if (xfsctl(NULL, s->fd, XFS_IOC_UNRESVSP64, &fl) < 0) {
         err = errno;
-        DPRINTF("cannot punch hole (%s)\n", strerror(errno));
+        trace_file_xfs_discard(strerror(errno));
         return -err;
     }
 
@@ -XXX,XX +XXX,XX @@ static char *FindEjectableOpticalMedia(io_iterator_t *mediaIterator)
 
         /* If a match was found, leave the loop */
         if (*mediaIterator != 0) {
-            DPRINTF("Matching using %s\n", matching_array[index]);
+            trace_file_FindEjectableOpticalMedia(matching_array[index]);
             mediaType = g_strdup(matching_array[index]);
             break;
         }
@@ -XXX,XX +XXX,XX @@ static bool setup_cdrom(char *bsd_path, Error **errp)
     if (partition_found == false) {
         error_setg(errp, "Failed to find a working partition on disc");
     } else {
-        DPRINTF("Using %s as optical disc\n", test_partition);
+        trace_file_setup_cdrom(test_partition);
         pstrcpy(bsd_path, MAXPATHLEN, test_partition);
     }
     return partition_found;
@@ -XXX,XX +XXX,XX @@ static bool hdev_is_sg(BlockDriverState *bs)
 
     ret = ioctl(s->fd, SG_GET_SCSI_ID, &scsiid);
     if (ret >= 0) {
-        DPRINTF("SG device found: type=%d, version=%d\n",
-            scsiid.scsi_type, sg_version);
+        trace_file_hdev_is_sg(scsiid.scsi_type, sg_version);
         return true;
     }
 
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ curl_open(const char *file) "opening %s"
 curl_open_size(uint64_t size) "size = %" PRIu64
 curl_setup_preadv(uint64_t bytes, uint64_t start, const char *range) "reading %" PRIu64 " at %" PRIu64 " (%s)"
 curl_close(void) "close"
+
+# block/file-posix.c
+file_xfs_write_zeroes(const char *error) "cannot write zero range (%s)"
+file_xfs_discard(const char *error) "cannot punch hole (%s)"
+file_FindEjectableOpticalMedia(const char *media) "Matching using %s"
+file_setup_cdrom(const char *partition) "Using %s as optical disc"
+file_hdev_is_sg(int type, int version) "SG device found: type=%d, version=%d"
-- 
2.20.1

From: Laurent Vivier <lvivier@redhat.com>

Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20181213162727.17438-5-lvivier@redhat.com
[mreitz: Fixed sheepdog_snapshot_create_inode's format string to use
         PRIx32 for uint32_ts]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/sheepdog.c   | 47 +++++++++++++++++-----------------------------
 block/trace-events | 14 ++++++++++++++
 2 files changed, 31 insertions(+), 30 deletions(-)

diff --git a/block/sheepdog.c b/block/sheepdog.c
index XXXXXXX..XXXXXXX 100644
--- a/block/sheepdog.c
+++ b/block/sheepdog.c
@@ -XXX,XX +XXX,XX @@
 #include "sysemu/block-backend.h"
 #include "qemu/bitops.h"
 #include "qemu/cutils.h"
+#include "trace.h"
 
 #define SD_PROTO_VER 0x01
 
@@ -XXX,XX +XXX,XX @@ static inline size_t count_data_objs(const struct SheepdogInode *inode)
                         (1UL << inode->block_size_shift));
 }
 
-#undef DPRINTF
-#ifdef DEBUG_SDOG
-#define DEBUG_SDOG_PRINT 1
-#else
-#define DEBUG_SDOG_PRINT 0
-#endif
-#define DPRINTF(fmt, args...)                                           \
-    do {                                                                \
-        if (DEBUG_SDOG_PRINT) {                                         \
-            fprintf(stderr, "%s %d: " fmt, __func__, __LINE__, ##args); \
-        }                                                               \
-    } while (0)
-
 typedef struct SheepdogAIOCB SheepdogAIOCB;
 typedef struct BDRVSheepdogState BDRVSheepdogState;
 
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void reconnect_to_sdog(void *opaque)
         Error *local_err = NULL;
         s->fd = get_sheep_fd(s, &local_err);
         if (s->fd < 0) {
-            DPRINTF("Wait for connection to be established\n");
+            trace_sheepdog_reconnect_to_sdog();
             error_report_err(local_err);
             qemu_co_sleep_ns(QEMU_CLOCK_REALTIME, 1000000000ULL);
         }
@@ -XXX,XX +XXX,XX @@ static void coroutine_fn aio_read_response(void *opaque)
         break;
     case AIOCB_FLUSH_CACHE:
         if (rsp.result == SD_RES_INVALID_PARMS) {
-            DPRINTF("disable cache since the server doesn't support it\n");
+            trace_sheepdog_aio_read_response();
             s->cache_flags = SD_FLAG_CMD_DIRECT;
             rsp.result = SD_RES_SUCCESS;
         }
@@ -XXX,XX +XXX,XX @@ static int sd_open(BlockDriverState *bs, QDict *options, int flags,
     s->discard_supported = true;
 
     if (snap_id || tag[0]) {
-        DPRINTF("%" PRIx32 " snapshot inode was open.\n", vid);
+        trace_sheepdog_open(vid);
         s->is_snapshot = true;
     }
 
@@ -XXX,XX +XXX,XX @@ static void sd_close(BlockDriverState *bs)
     unsigned int wlen, rlen = 0;
     int fd, ret;
 
-    DPRINTF("%s\n", s->name);
+    trace_sheepdog_close(s->name);
 
     fd = connect_to_sdog(s, &local_err);
     if (fd < 0) {
@@ -XXX,XX +XXX,XX @@ static int sd_create_branch(BDRVSheepdogState *s)
     char *buf;
     bool deleted;
 
-    DPRINTF("%" PRIx32 " is snapshot.\n", s->inode.vdi_id);
+    trace_sheepdog_create_branch_snapshot(s->inode.vdi_id);
 
     buf = g_malloc(SD_INODE_SIZE);
 
@@ -XXX,XX +XXX,XX @@ static int sd_create_branch(BDRVSheepdogState *s)
         goto out;
     }
 
-    DPRINTF("%" PRIx32 " is created.\n", vid);
+    trace_sheepdog_create_branch_created(vid);
 
     fd = connect_to_sdog(s, &local_err);
     if (fd < 0) {
@@ -XXX,XX +XXX,XX @@ static int sd_create_branch(BDRVSheepdogState *s)
 
     s->is_snapshot = false;
     ret = 0;
-    DPRINTF("%" PRIx32 " was newly created.\n", s->inode.vdi_id);
+    trace_sheepdog_create_branch_new(s->inode.vdi_id);
 
 out:
     g_free(buf);
@@ -XXX,XX +XXX,XX @@ static void coroutine_fn sd_co_rw_vector(SheepdogAIOCB *acb)
         }
 
         if (create) {
-            DPRINTF("update ino (%" PRIu32 ") %" PRIu64 " %" PRIu64 " %ld\n",
-                    inode->vdi_id, oid,
-                    vid_to_data_oid(inode->data_vdi_id[idx], idx), idx);
+            trace_sheepdog_co_rw_vector_update(inode->vdi_id, oid,
+                                  vid_to_data_oid(inode->data_vdi_id[idx], idx),
+                                  idx);
             oid = vid_to_data_oid(inode->vdi_id, idx);
-            DPRINTF("new oid %" PRIx64 "\n", oid);
+            trace_sheepdog_co_rw_vector_new(oid);
         }
 
         aio_req = alloc_aio_req(s, acb, oid, len, offset, flags, create,
@@ -XXX,XX +XXX,XX @@ static int sd_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
     SheepdogInode *inode;
     unsigned int datalen;
 
-    DPRINTF("sn_info: name %s id_str %s s: name %s vm_state_size %" PRId64 " "
-            "is_snapshot %d\n", sn_info->name, sn_info->id_str,
-            s->name, sn_info->vm_state_size, s->is_snapshot);
+    trace_sheepdog_snapshot_create_info(sn_info->name, sn_info->id_str, s->name,
+                                        sn_info->vm_state_size, s->is_snapshot);
 
     if (s->is_snapshot) {
         error_report("You can't create a snapshot of a snapshot VDI, "
@@ -XXX,XX +XXX,XX @@ static int sd_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
         return -EINVAL;
     }
 
-    DPRINTF("%s %s\n", sn_info->name, sn_info->id_str);
+    trace_sheepdog_snapshot_create(sn_info->name, sn_info->id_str);
 
     s->inode.vm_state_size = sn_info->vm_state_size;
     s->inode.vm_clock_nsec = sn_info->vm_clock_nsec;
@@ -XXX,XX +XXX,XX @@ static int sd_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
     }
 
     memcpy(&s->inode, inode, datalen);
-    DPRINTF("s->inode: name %s snap_id %x oid %x\n",
-            s->inode.name, s->inode.snap_id, s->inode.vdi_id);
+    trace_sheepdog_snapshot_create_inode(s->inode.name, s->inode.snap_id,
+                                         s->inode.vdi_id);
 
 cleanup:
     g_free(inode);
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ file_xfs_discard(const char *error) "cannot punch hole (%s)"
 file_FindEjectableOpticalMedia(const char *media) "Matching using %s"
 file_setup_cdrom(const char *partition) "Using %s as optical disc"
 file_hdev_is_sg(int type, int version) "SG device found: type=%d, version=%d"
+
+# block/sheepdog.c
+sheepdog_reconnect_to_sdog(void) "Wait for connection to be established"
+sheepdog_aio_read_response(void) "disable cache since the server doesn't support it"
+sheepdog_open(uint32_t vid) "0x%" PRIx32 " snapshot inode was open"
+sheepdog_close(const char *name) "%s"
+sheepdog_create_branch_snapshot(uint32_t vdi) "0x%" PRIx32 " is snapshot"
+sheepdog_create_branch_created(uint32_t vdi) "0x%" PRIx32 " is created"
+sheepdog_create_branch_new(uint32_t vdi) "0x%" PRIx32 " was newly created"
+sheepdog_co_rw_vector_update(uint32_t vdi, uint64_t oid, uint64_t data, long idx) "update ino (%" PRIu32 ") %" PRIu64 " %" PRIu64 " %ld"
+sheepdog_co_rw_vector_new(uint64_t oid) "new oid 0x%" PRIx64
+sheepdog_snapshot_create_info(const char *sn_name, const char *id, const char *name, int64_t size, int is_snapshot) "sn_info: name %s id_str %s s: name %s vm_state_size %" PRId64 " " "is_snapshot %d"
+sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
+sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
-- 
2.20.1

From: "Richard W.M. Jones" <rjones@redhat.com>

On FreeBSD 11.2:

$ nbdkit memory size=1M --run './qemu-io -f raw -c "aio_write 0 512" $nbd'
  Parsing error: non-numeric argument, or extraneous/unrecognized suffix -- aio_write

After main option parsing, we reinitialize optind so we can parse each
command.  However reinitializing optind to 0 does not work on FreeBSD.
What happens when you do this is optind remains 0 after the option
parsing loop, and the result is we try to parse argv[optind] ==
argv[0] == "aio_write" as if it was the first parameter.

The FreeBSD manual page says:

In order to use getopt() to evaluate multiple sets of arguments, or to
  evaluate a single set of arguments multiple times, the variable optreset
  must be set to 1 before the second and each additional set of calls to
  getopt(), and the variable optind must be reinitialized.

(From the rest of the man page it is clear that optind must be
reinitialized to 1).

The glibc man page says:

A program that scans multiple argument vectors,  or  rescans  the  same
  vector  more than once, and wants to make use of GNU extensions such as
  '+' and '-' at  the  start  of  optstring,  or  changes  the  value  of
  POSIXLY_CORRECT  between scans, must reinitialize getopt() by resetting
  optind to 0, rather than the traditional value of 1.  (Resetting  to  0
  forces  the  invocation  of  an  internal  initialization  routine that
  rechecks POSIXLY_CORRECT and checks for GNU extensions in optstring.)

This commit introduces an OS-portability function called
qemu_reset_optind which provides a way of resetting optind that works
on FreeBSD and platforms that use optreset, while keeping it the same
as now on other platforms.

Note that the qemu codebase sets optind in many other places, but in
those other places it's setting a local variable and not using getopt.
This change is only needed in places where we are using getopt and the
associated global variable optind.

Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
Message-id: 20190118101114.11759-2-rjones@redhat.com
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 configure            | 14 ++++++++++++++
 include/qemu/osdep.h | 16 ++++++++++++++++
 qemu-img.c           |  2 +-
 qemu-io-cmds.c       |  2 +-
 4 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/configure b/configure
index XXXXXXX..XXXXXXX 100755
--- a/configure
+++ b/configure
@@ -XXX,XX +XXX,XX @@ if compile_prog "" "" ; then
   signalfd=yes
 fi
 
+# check if optreset global is declared by <getopt.h>
+optreset="no"
+cat > $TMPC << EOF
+#include <getopt.h>
+int main(void) { return optreset; }
+EOF
+
+if compile_prog "" "" ; then
+  optreset=yes
+fi
+
 # check if eventfd is supported
 eventfd=no
 cat > $TMPC << EOF
@@ -XXX,XX +XXX,XX @@ fi
 if test "$signalfd" = "yes" ; then
   echo "CONFIG_SIGNALFD=y" >> $config_host_mak
 fi
+if test "$optreset" = "yes" ; then
+  echo "HAVE_OPTRESET=y" >> $config_host_mak
+fi
 if test "$tcg" = "yes"; then
   echo "CONFIG_TCG=y" >> $config_host_mak
   if test "$tcg_interpreter" = "yes" ; then
diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -XXX,XX +XXX,XX @@ extern int daemon(int, int);
 #include <ctype.h>
 #include <errno.h>
 #include <fcntl.h>
+#include <getopt.h>
 #include <sys/stat.h>
 #include <sys/time.h>
 #include <assert.h>
@@ -XXX,XX +XXX,XX @@ extern int qemu_icache_linesize_log;
 extern int qemu_dcache_linesize;
 extern int qemu_dcache_linesize_log;
 
+/*
+ * After using getopt or getopt_long, if you need to parse another set
+ * of options, then you must reset optind.  Unfortunately the way to
+ * do this varies between implementations of getopt.
+ */
+static inline void qemu_reset_optind(void)
+{
+#ifdef HAVE_OPTRESET
+    optind = 1;
+    optreset = 1;
+#else
+    optind = 0;
+#endif
+}
+
 #endif
diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
         return 0;
     }
     argv += optind;
-    optind = 0;
+    qemu_reset_optind();
 
     if (!trace_init_backends()) {
         exit(1);
diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -XXX,XX +XXX,XX @@ static int command(BlockBackend *blk, const cmdinfo_t *ct, int argc,
         }
     }
 
-    optind = 0;
+    qemu_reset_optind();
     return ct->cfunc(blk, argc, argv);
 }
 
-- 
2.20.1

From: Li Qiang <liq3ea@163.com>

When it is zero, it causes segv.
Using following command:

"-drive file=//home/test/test1.img,if=none,id=id0
-device nvme,drive=id0,serial=test,num_queues=0"
causes following Backtrack:

Thread 4 "qemu-system-x86" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe9735700 (LWP 30952)]
0x0000555555a7a77c in nvme_start_ctrl (n=0x5555577473f0) at hw/block/nvme.c:825
825	    if (unlikely(n->cq[0])) {
(gdb) bt
0  0x0000555555a7a77c in nvme_start_ctrl (n=0x5555577473f0)
    at hw/block/nvme.c:825
1  0x0000555555a7af7f in nvme_write_bar (n=0x5555577473f0, offset=20,
    data=4587521, size=4) at hw/block/nvme.c:969
2  0x0000555555a7b81a in nvme_mmio_write (opaque=0x5555577473f0, addr=20,
    data=4587521, size=4) at hw/block/nvme.c:1163
3  0x0000555555869236 in memory_region_write_accessor (mr=0x555557747cd0,
    addr=20, value=0x7fffe97320f8, size=4, shift=0, mask=4294967295, attrs=...)
    at /home/test/qemu1/qemu/memory.c:502
4  0x0000555555869446 in access_with_adjusted_size (addr=20,
    value=0x7fffe97320f8, size=4, access_size_min=2, access_size_max=8,
    access_fn=0x55555586914d <memory_region_write_accessor>,
    mr=0x555557747cd0, attrs=...) at /home/test/qemu1/qemu/memory.c:568
5  0x000055555586c479 in memory_region_dispatch_write (mr=0x555557747cd0,
    addr=20, data=4587521, size=4, attrs=...)
    at /home/test/qemu1/qemu/memory.c:1499
6  0x00005555558030af in flatview_write_continue (fv=0x7fffe0061130,
    addr=4273930260, attrs=..., buf=0x7ffff7ff0028 "\001", len=4, addr1=20,
    l=4, mr=0x555557747cd0) at /home/test/qemu1/qemu/exec.c:3234
7  0x00005555558031f9 in flatview_write (fv=0x7fffe0061130, addr=4273930260,
    attrs=..., buf=0x7ffff7ff0028 "\001", len=4)
    at /home/test/qemu1/qemu/exec.c:3273
8  0x00005555558034ff in address_space_write (
---Type <return> to continue, or q <return> to quit---
    as=0x555556758480 <address_space_memory>, addr=4273930260, attrs=...,
    buf=0x7ffff7ff0028 "\001", len=4) at /home/test/qemu1/qemu/exec.c:3363
9  0x0000555555803550 in address_space_rw (
    as=0x555556758480 <address_space_memory>, addr=4273930260, attrs=...,
    buf=0x7ffff7ff0028 "\001", len=4, is_write=true)
    at /home/test/qemu1/qemu/exec.c:3374
10 0x00005555558884a1 in kvm_cpu_exec (cpu=0x555556920e40)
    at /home/test/qemu1/qemu/accel/kvm/kvm-all.c:2031
11 0x000055555584cd9d in qemu_kvm_cpu_thread_fn (arg=0x555556920e40)
    at /home/test/qemu1/qemu/cpus.c:1281
12 0x0000555555dbaf6d in qemu_thread_start (args=0x5555569438a0)
    at util/qemu-thread-posix.c:502
13 0x00007ffff5dc86db in start_thread (arg=0x7fffe9735700)
    at pthread_create.c:463
14 0x00007ffff5af188f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Signed-off-by: Li Qiang <liq3ea@163.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190120055558.32984-3-liq3ea@163.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 hw/block/nvme.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
     int64_t bs_size;
     uint8_t *pci_conf;
 
+    if (!n->num_queues) {
+        error_setg(errp, "num_queues can't be zero");
+        return;
+    }
+
     if (!n->conf.blk) {
         error_setg(errp, "drive property not set");
         return;
-- 
2.20.1

From: Li Qiang <liq3ea@163.com>

There is no need to make another reference.

Signed-off-by: Li Qiang <liq3ea@163.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190120055558.32984-4-liq3ea@163.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 hw/block/nvme.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
     pci_conf[PCI_INTERRUPT_PIN] = 1;
     pci_config_set_prog_interface(pci_dev->config, 0x2);
     pci_config_set_class(pci_dev->config, PCI_CLASS_STORAGE_EXPRESS);
-    pcie_endpoint_cap_init(&n->parent_obj, 0x80);
+    pcie_endpoint_cap_init(pci_dev, 0x80);
 
     n->num_namespaces = 1;
     n->reg_size = pow2ceil(0x1004 + 2 * (n->num_queues + 1) * 4);
@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
 
     memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n,
                           "nvme", n->reg_size);
-    pci_register_bar(&n->parent_obj, 0,
+    pci_register_bar(pci_dev, 0,
         PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64,
         &n->iomem);
-    msix_init_exclusive_bar(&n->parent_obj, n->num_queues, 4, NULL);
+    msix_init_exclusive_bar(pci_dev, n->num_queues, 4, NULL);
 
     id->vid = cpu_to_le16(pci_get_word(pci_conf + PCI_VENDOR_ID));
     id->ssvid = cpu_to_le16(pci_get_word(pci_conf + PCI_SUBSYSTEM_VENDOR_ID));
@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
         n->cmbuf = g_malloc0(NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
         memory_region_init_io(&n->ctrl_mem, OBJECT(n), &nvme_cmb_ops, n,
                               "nvme-cmb", NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
-        pci_register_bar(&n->parent_obj, NVME_CMBLOC_BIR(n->bar.cmbloc),
+        pci_register_bar(pci_dev, NVME_CMBLOC_BIR(n->bar.cmbloc),
             PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64 |
             PCI_BASE_ADDRESS_MEM_PREFETCH, &n->ctrl_mem);
 
-- 
2.20.1

In some cases, we may want to deal with qemu-nbd errors (e.g. by
launching it in a different configuration until it no longer throws
any).  In that case, we do not want its output ending up in the test
output.

It may still be useful for handling the error, though, so add a new
function that works basically like qemu_nbd(), only that it returns the
qemu-nbd output instead of making it end up in the log.  In contrast to
qemu_img_pipe(), it does still return the exit code as well, though,
because that is even more important for error handling.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20181221234750.23577-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def qemu_nbd(*args):
     '''Run qemu-nbd in daemon mode and return the parent's exit code'''
     return subprocess.call(qemu_nbd_args + ['--fork'] + list(args))
 
+def qemu_nbd_pipe(*args):
+    '''Run qemu-nbd in daemon mode and return both the parent's exit code
+       and its output'''
+    subp = subprocess.Popen(qemu_nbd_args + ['--fork'] + list(args),
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
+    exitcode = subp.wait()
+    if exitcode < 0:
+        sys.stderr.write('qemu-nbd received signal %i: %s\n' %
+                         (-exitcode,
+                          ' '.join(qemu_nbd_args + ['--fork'] + list(args))))
+    return exitcode, subp.communicate()[0]
+
 def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
     '''Return True if two image files are identical'''
     return qemu_img('compare', '-f', fmt1,
-- 
2.20.1

By default, qemu-nbd binds to 0.0.0.0.  However, we then proceed to
connect to "localhost".  Usually, this works out fine; but if this test
is run concurrently, some other test function may have bound a different
server to ::1 (on the same port -- you can bind different serves to the
same port, as long as one is on IPv4 and the other on IPv6).

So running qemu-nbd works, it can bind to 0.0.0.0:NBD_PORT.  But
potentially a concurrent test has successfully taken [::1]:NBD_PORT.  In
this case, trying to connect to "localhost" will lead us to the IPv6
instance, where we do not want to end up.

Fix this by just binding to "localhost".  This will make qemu-nbd error
out immediately and not give us cryptic errors later.

(Also, it will allow us to just try a different port as of a future
patch.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20181221234750.23577-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/147 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -XXX,XX +XXX,XX @@ class QemuNBD(NBDBlockdevAddBase):
         self.assertEqual(qemu_nbd('-f', imgfmt, test_img, *args), 0)
 
     def test_inet(self):
-        self._server_up('-p', str(NBD_PORT))
+        self._server_up('-b', 'localhost', '-p', str(NBD_PORT))
         address = { 'type': 'inet',
                     'data': {
                         'host': 'localhost',
-- 
2.20.1

To do this, we need to allow creating the NBD server on various ports
instead of a single one (which may not even work if you run just one
instance, because something entirely else might be using that port).

So we just pick a random port in [32768, 32768 + 1024) and try to create
a server there.  If that fails, we just retry until something sticks.

For the IPv6 test, we need a different range, though (just above that
one).  This is because "localhost" resolves to both 127.0.0.1 and ::1.
This means that if you bind to it, it will bind to both, if possible, or
just one if the other is already in use.  Therefore, if the IPv6 test
has already taken [::1]:some_port and we then try to take
localhost:some_port, that will work -- only the second server will be
bound to 127.0.0.1:some_port alone and not [::1]:some_port in addition.
So we have two different servers on the same port, one for IPv4 and one
for IPv6.

But when we then try to connect to the server through
localhost:some_port, we will always end up at the IPv6 one (as long as
it is up), and this may not be the one we want.

Thus, we must make sure not to create an IPv6-only NBD server on the
same port as a normal "dual-stack" NBD server -- which is done by using
distinct port ranges, as explained above.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20181221234750.23577-4-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/147 | 98 +++++++++++++++++++++++++++++-------------
 1 file changed, 68 insertions(+), 30 deletions(-)

diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -XXX,XX +XXX,XX @@
 #
 
 import os
+import random
 import socket
 import stat
 import time
 import iotests
-from iotests import cachemode, imgfmt, qemu_img, qemu_nbd
+from iotests import cachemode, imgfmt, qemu_img, qemu_nbd, qemu_nbd_pipe
 
-NBD_PORT = 10811
+NBD_PORT_START      = 32768
+NBD_PORT_END        = NBD_PORT_START + 1024
+NBD_IPV6_PORT_START = NBD_PORT_END
+NBD_IPV6_PORT_END   = NBD_IPV6_PORT_START + 1024
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 unix_socket = os.path.join(iotests.test_dir, 'nbd.socket')
@@ -XXX,XX +XXX,XX @@ class QemuNBD(NBDBlockdevAddBase):
         except OSError:
             pass
 
+    def _try_server_up(self, *args):
+        status, msg = qemu_nbd_pipe('-f', imgfmt, test_img, *args)
+        if status == 0:
+            return True
+        if 'Address already in use' in msg:
+            return False
+        self.fail(msg)
+
     def _server_up(self, *args):
-        self.assertEqual(qemu_nbd('-f', imgfmt, test_img, *args), 0)
+        self.assertTrue(self._try_server_up(*args))
 
     def test_inet(self):
-        self._server_up('-b', 'localhost', '-p', str(NBD_PORT))
+        while True:
+            nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END)
+            if self._try_server_up('-b', 'localhost', '-p', str(nbd_port)):
+                break
+
         address = { 'type': 'inet',
                     'data': {
                         'host': 'localhost',
-                        'port': str(NBD_PORT)
+                        'port': str(nbd_port)
                     } }
-        self.client_test('nbd://localhost:%i' % NBD_PORT,
+        self.client_test('nbd://localhost:%i' % nbd_port,
                          flatten_sock_addr(address))
 
     def test_unix(self):
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
         except OSError:
             pass
 
-    def _server_up(self, address, export_name=None, export_name2=None):
+    # Returns False on EADDRINUSE; fails an assertion on other errors.
+    # Returns True on success.
+    def _try_server_up(self, address, export_name=None, export_name2=None):
         result = self.server.qmp('nbd-server-start', addr=address)
+        if 'error' in result and \
+           'Address already in use' in result['error']['desc']:
+            return False
         self.assert_qmp(result, 'return', {})
 
         if export_name is None:
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
                                      name=export_name2)
             self.assert_qmp(result, 'return', {})
 
+        return True
+
+    def _server_up(self, address, export_name=None, export_name2=None):
+        self.assertTrue(self._try_server_up(address, export_name, export_name2))
 
     def _server_down(self):
         result = self.server.qmp('nbd-server-stop')
         self.assert_qmp(result, 'return', {})
 
     def do_test_inet(self, export_name=None):
-        address = { 'type': 'inet',
-                    'data': {
-                        'host': 'localhost',
-                        'port': str(NBD_PORT)
-                    } }
-        self._server_up(address, export_name)
+        while True:
+            nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END)
+            address = { 'type': 'inet',
+                        'data': {
+                            'host': 'localhost',
+                            'port': str(nbd_port)
+                        } }
+            if self._try_server_up(address, export_name):
+                break
+
         export_name = export_name or 'nbd-export'
-        self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, export_name),
+        self.client_test('nbd://localhost:%i/%s' % (nbd_port, export_name),
                          flatten_sock_addr(address), export_name)
         self._server_down()
 
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
         self.do_test_inet('shadow')
 
     def test_inet_two_exports(self):
-        address = { 'type': 'inet',
-                    'data': {
-                        'host': 'localhost',
-                        'port': str(NBD_PORT)
-                    } }
-        self._server_up(address, 'exp1', 'exp2')
-        self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, 'exp1'),
+        while True:
+            nbd_port = random.randrange(NBD_PORT_START, NBD_PORT_END)
+            address = { 'type': 'inet',
+                        'data': {
+                            'host': 'localhost',
+                            'port': str(nbd_port)
+                        } }
+            if self._try_server_up(address, 'exp1', 'exp2'):
+                break
+
+        self.client_test('nbd://localhost:%i/%s' % (nbd_port, 'exp1'),
                          flatten_sock_addr(address), 'exp1', 'node1', False)
-        self.client_test('nbd://localhost:%i/%s' % (NBD_PORT, 'exp2'),
+        self.client_test('nbd://localhost:%i/%s' % (nbd_port, 'exp2'),
                          flatten_sock_addr(address), 'exp2', 'node2', False)
         result = self.vm.qmp('blockdev-del', node_name='node1')
         self.assert_qmp(result, 'return', {})
@@ -XXX,XX +XXX,XX @@ class BuiltinNBD(NBDBlockdevAddBase):
         except socket.gaierror:
             # IPv6 not available, skip
             return
-        address = { 'type': 'inet',
-                    'data': {
-                        'host': '::1',
-                        'port': str(NBD_PORT),
-                        'ipv4': False,
-                        'ipv6': True
-                    } }
+
+        while True:
+            nbd_port = random.randrange(NBD_IPV6_PORT_START, NBD_IPV6_PORT_END)
+            address = { 'type': 'inet',
+                        'data': {
+                            'host': '::1',
+                            'port': str(nbd_port),
+                            'ipv4': False,
+                            'ipv6': True
+                        } }
+            if self._try_server_up(address):
+                break
+
         filename = { 'driver': 'raw',
                      'file': {
                          'driver': 'nbd',
                          'export': 'nbd-export',
                          'server': flatten_sock_addr(address)
                      } }
-        self._server_up(address)
         self.client_test(filename, flatten_sock_addr(address), 'nbd-export')
         self._server_down()
 
-- 
2.20.1

The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:

Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27

for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:

iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)

----------------------------------------------------------------
Block patches:
- qemu-io now accepts a file to read a write pattern from
- Ensure that raw files have their first block allocated so we can probe
  the O_DIRECT alignment if necessary
- Various fixes

----------------------------------------------------------------
Denis Plotnikov (1):
  qemu-io: add pattern file for write command

Max Reitz (7):
  iotests: Fix _filter_img_create()
  vmdk: Use bdrv_dirname() for relative extent paths
  iotests: Keep testing broken relative extent paths
  vmdk: Reject invalid compressed writes
  iotests: Disable broken streamOptimized tests
  iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
  iotests: Disable 126 for flat vmdk subformats

Nir Soffer (3):
  block: posix: Always allocate the first block
  iotests: Test allocate_first_block() with O_DIRECT
  iotests: Unify cache mode quoting

Stefan Hajnoczi (1):
  file-posix: fix request_alignment typo

Thomas Huth (2):
  iotests: Check for enabled drivers before testing them
  tests/check-block: Skip iotests when sanitizers are enabled

Vladimir Sementsov-Ogievskiy (1):
  block: fix permission update in bdrv_replace_node

block.c                                       |  5 +-
 block/file-posix.c                            | 53 +++++++++-
 block/vmdk.c                                  | 64 ++++++++----
 qemu-io-cmds.c                                | 99 +++++++++++++++++--
 tests/check-block.sh                          |  5 +
 tests/qemu-iotests/002                        |  1 +
 tests/qemu-iotests/003                        |  1 +
 tests/qemu-iotests/005                        |  3 +-
 tests/qemu-iotests/009                        |  1 +
 tests/qemu-iotests/010                        |  1 +
 tests/qemu-iotests/011                        |  1 +
 tests/qemu-iotests/017                        |  3 +-
 tests/qemu-iotests/018                        |  3 +-
 tests/qemu-iotests/019                        |  3 +-
 tests/qemu-iotests/020                        |  3 +-
 tests/qemu-iotests/026                        |  4 +-
 tests/qemu-iotests/027                        |  1 +
 tests/qemu-iotests/032                        |  1 +
 tests/qemu-iotests/033                        |  1 +
 tests/qemu-iotests/034                        |  3 +-
 tests/qemu-iotests/037                        |  3 +-
 tests/qemu-iotests/039                        |  4 +-
 tests/qemu-iotests/052                        |  2 +-
 tests/qemu-iotests/059                        | 34 ++++++-
 tests/qemu-iotests/059.out                    | 26 +++--
 tests/qemu-iotests/063                        |  3 +-
 tests/qemu-iotests/071                        |  1 +
 tests/qemu-iotests/072                        |  1 +
 tests/qemu-iotests/081                        |  4 +-
 tests/qemu-iotests/091                        |  4 +-
 tests/qemu-iotests/099                        |  1 +
 tests/qemu-iotests/105                        |  3 +-
 tests/qemu-iotests/110                        |  3 +-
 tests/qemu-iotests/120                        |  1 +
 tests/qemu-iotests/126                        |  2 +
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++
 tests/qemu-iotests/162                        |  4 +-
 tests/qemu-iotests/175                        | 47 +++++++--
 tests/qemu-iotests/175.out                    | 16 ++-
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/184                        |  1 +
 tests/qemu-iotests/186                        |  1 +
 tests/qemu-iotests/197                        |  1 +
 tests/qemu-iotests/215                        |  1 +
 tests/qemu-iotests/221.out                    | 12 ++-
 tests/qemu-iotests/251                        |  1 +
 tests/qemu-iotests/253.out                    | 12 ++-
 tests/qemu-iotests/common.filter              |  4 +-
 tests/qemu-iotests/common.rc                  | 14 +++
 50 files changed, 391 insertions(+), 87 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

-- 
2.21.0

From: Denis Plotnikov <dplotnikov@virtuozzo.com>

The patch allows to provide a pattern file for write
command. There was no similar ability before.

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
Reviewed-by: Eric Blake <eblake@redhat.com>
[mreitz: Keep optstring in alphabetical order]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 6 deletions(-)

diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
     qemu_vfree(p);
 }
 
+/*
+ * qemu_io_alloc_from_file()
+ *
+ * Allocates the buffer and populates it with the content of the given file
+ * up to @len bytes. If the file length is less than @len, then the buffer
+ * is populated with the file content cyclically.
+ *
+ * @blk - the block backend where the buffer content is going to be written to
+ * @len - the buffer length
+ * @file_name - the file to read the content from
+ *
+ * Returns: the buffer pointer on success
+ *          NULL on error
+ */
+static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
+                                     const char *file_name)
+{
+    char *buf, *buf_origin;
+    FILE *f = fopen(file_name, "r");
+    int pattern_len;
+
+    if (!f) {
+        perror(file_name);
+        return NULL;
+    }
+
+    if (qemuio_misalign) {
+        len += MISALIGN_OFFSET;
+    }
+
+    buf_origin = buf = blk_blockalign(blk, len);
+
+    if (qemuio_misalign) {
+        buf_origin += MISALIGN_OFFSET;
+        buf += MISALIGN_OFFSET;
+        len -= MISALIGN_OFFSET;
+    }
+
+    pattern_len = fread(buf_origin, 1, len, f);
+
+    if (ferror(f)) {
+        perror(file_name);
+        goto error;
+    }
+
+    if (pattern_len == 0) {
+        fprintf(stderr, "%s: file is empty\n", file_name);
+        goto error;
+    }
+
+    fclose(f);
+
+    if (len > pattern_len) {
+        len -= pattern_len;
+        buf += pattern_len;
+
+        while (len > 0) {
+            size_t len_to_copy = MIN(pattern_len, len);
+
+            memcpy(buf, buf_origin, len_to_copy);
+
+            len -= len_to_copy;
+            buf += len_to_copy;
+        }
+    }
+
+    return buf_origin;
+
+error:
+    qemu_io_free(buf_origin);
+    return NULL;
+}
+
 static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
 {
     uint64_t i;
@@ -XXX,XX +XXX,XX @@ static void write_help(void)
 " -n, -- with -z, don't allow slow fallback\n"
 " -p, -- ignored for backwards compatibility\n"
 " -P, -- use different pattern to fill file\n"
+" -s, -- use a pattern file to fill the write buffer\n"
 " -C, -- report statistics in a machine parsable format\n"
 " -q, -- quiet mode, do not show I/O statistics\n"
 " -u, -- with -z, allow unmapping\n"
@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
     .perm       = BLK_PERM_WRITE,
     .argmin     = 2,
     .argmax     = -1,
-    .args       = "[-bcCfnquz] [-P pattern] off len",
+    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
     .oneline    = "writes a number of bytes at a specified offset",
     .help       = write_help,
 };
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
 {
     struct timespec t1, t2;
     bool Cflag = false, qflag = false, bflag = false;
-    bool Pflag = false, zflag = false, cflag = false;
+    bool Pflag = false, zflag = false, cflag = false, sflag = false;
     int flags = 0;
     int c, cnt, ret;
     char *buf = NULL;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     /* Some compilers get confused and warn if this is not initialized.  */
     int64_t total = 0;
     int pattern = 0xcd;
+    const char *file_name = NULL;
 
-    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
+    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
         switch (c) {
         case 'b':
             bflag = true;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         case 'q':
             qflag = true;
             break;
+        case 's':
+            sflag = true;
+            file_name = optarg;
+            break;
         case 'u':
             flags |= BDRV_REQ_MAY_UNMAP;
             break;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         return -EINVAL;
     }
 
-    if (zflag && Pflag) {
-        printf("-z and -P cannot be specified at the same time\n");
+    if (zflag + Pflag + sflag > 1) {
+        printf("Only one of -z, -P, and -s "
+               "can be specified at the same time\n");
         return -EINVAL;
     }
 
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     }
 
     if (!zflag) {
-        buf = qemu_io_alloc(blk, count, pattern);
+        if (sflag) {
+            buf = qemu_io_alloc_from_file(blk, count, file_name);
+            if (!buf) {
+                return -EINVAL;
+            }
+        } else {
+            buf = qemu_io_alloc(blk, count, pattern);
+        }
     }
 
     clock_gettime(CLOCK_MONOTONIC, &t1);
-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

It's wrong to OR shared permissions. It may lead to crash on further
permission updates.
Also, no needs to consider previously calculated permissions, as at
this point we already bind all new parents and bdrv_get_cumulative_perm
result is enough. So fix the bug by just set permissions by
bdrv_get_cumulative_perm result.

Bug was introduced in long ago 234ac1a9025, in 2.9.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
 {
     BdrvChild *c, *next;
     GSList *list = NULL, *p;
-    uint64_t old_perm, old_shared;
     uint64_t perm = 0, shared = BLK_PERM_ALL;
     int ret;
 
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
         bdrv_unref(from);
     }
 
-    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
-    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
+    bdrv_get_cumulative_perm(to, &perm, &shared);
+    bdrv_set_perm(to, perm, shared);
 
 out:
     g_slist_free(list);
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

When creating an image with preallocation "off" or "falloc", the first
block of the image is typically not allocated. When using Gluster
storage backed by XFS filesystem, reading this block using direct I/O
succeeds regardless of request length, fooling alignment detection.

In this case we fallback to a safe value (4096) instead of the optimal
value (512), which may lead to unneeded data copying when aligning
requests.  Allocating the first block avoids the fallback.

Since we allocate the first block even with preallocation=off, we no
longer create images with zero disk size:

$ ./qemu-img create -f raw test.raw 1g
    Formatting 'test.raw', fmt=raw size=1073741824

$ ls -lhs test.raw
    4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw

And converting the image requires additional cluster:

$ ./qemu-img measure -f raw -O qcow2 test.raw
    required size: 458752
    fully allocated size: 1074135040

When using format like vmdk with multiple files per image, we allocate
one block per file:

$ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
    Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat

$ ls -lhs test*.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk

I did quick performance test for copying disks with qemu-img convert to
new raw target image to Gluster storage with sector size of 512 bytes:

for i in $(seq 10); do
        rm -f dst.raw
        sleep 10
        time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
    done

Here is a table comparing the total time spent:

Type    Before(s)   After(s)    Diff(%)
---------------------------------------
real      530.028    469.123      -11.4
user       17.204     10.768      -37.4
sys        17.881      7.011      -60.7

We can see very clear improvement in CPU usage.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190827010528.8818-2-nsoffer@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c                            | 51 +++++++++++++++++++
 tests/qemu-iotests/059.out                    |  2 +-
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++++
 tests/qemu-iotests/175                        | 19 ++++---
 tests/qemu-iotests/175.out                    |  8 +--
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/221.out                    | 12 +++--
 tests/qemu-iotests/253.out                    | 12 +++--
 9 files changed, 99 insertions(+), 21 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
     return ret;
 }
 
+/*
+ * Help alignment probing by allocating the first block.
+ *
+ * When reading with direct I/O from unallocated area on Gluster backed by XFS,
+ * reading succeeds regardless of request length. In this case we fallback to
+ * safe alignment which is not optimal. Allocating the first block avoids this
+ * fallback.
+ *
+ * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
+ * request alignment, so we use safe values.
+ *
+ * Returns: 0 on success, -errno on failure. Since this is an optimization,
+ * caller may ignore failures.
+ */
+static int allocate_first_block(int fd, size_t max_size)
+{
+    size_t write_size = (max_size < MAX_BLOCKSIZE)
+        ? BDRV_SECTOR_SIZE
+        : MAX_BLOCKSIZE;
+    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
+    void *buf;
+    ssize_t n;
+    int ret;
+
+    buf = qemu_memalign(max_align, write_size);
+    memset(buf, 0, write_size);
+
+    do {
+        n = pwrite(fd, buf, write_size, 0);
+    } while (n == -1 && errno == EINTR);
+
+    ret = (n == -1) ? -errno : 0;
+
+    qemu_vfree(buf);
+    return ret;
+}
+
 static int handle_aiocb_truncate(void *opaque)
 {
     RawPosixAIOData *aiocb = opaque;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
                 /* posix_fallocate() doesn't set errno. */
                 error_setg_errno(errp, -result,
                                  "Could not preallocate new data");
+            } else if (current_length == 0) {
+                /*
+                 * posix_fallocate() uses fallocate() if the filesystem
+                 * supports it, or fallback to manually writing zeroes. If
+                 * fallocate() was used, unaligned reads from the fallocated
+                 * area in raw_probe_alignment() will succeed, hence we need to
+                 * allocate the first block.
+                 *
+                 * Optimize future alignment probing; ignore failures.
+                 */
+                allocate_first_block(fd, offset);
             }
         } else {
             result = 0;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
         if (ftruncate(fd, offset) != 0) {
             result = -errno;
             error_setg_errno(errp, -result, "Could not resize file");
+        } else if (current_length == 0 && offset > current_length) {
+            /* Optimize future alignment probing; ignore failures. */
+            allocate_first_block(fd, offset);
         }
         return result;
     default:
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
-disk size: 16 KiB
+disk size: 1.97 MiB
 Format specific information:
     cid: XXXXXXXX
     parent cid: XXXXXXXX
diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
similarity index 100%
rename from tests/qemu-iotests/150.out
rename to tests/qemu-iotests/150.out.qcow2
diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/150.out.raw
@@ -XXX,XX +XXX,XX @@
+QA output created by 150
+
+=== Mapping sparse conversion ===
+
+Offset          Length          File
+0               0x1000          TEST_DIR/t.IMGFMT
+
+=== Mapping non-sparse conversion ===
+
+Offset          Length          File
+0               0x100000        TEST_DIR/t.IMGFMT
+*** done
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # the file size.  This function hides the resulting difference in the
 # stat -c '%b' output.
 # Parameter 1: Number of blocks an empty file occupies
-# Parameter 2: Image size in bytes
+# Parameter 2: Minimal number of blocks in an image
+# Parameter 3: Image size in bytes
 _filter_blocks()
 {
     extra_blocks=$1
-    img_size=$2
+    min_blocks=$2
+    img_size=$3
 
-    sed -e "s/blocks=$extra_blocks\$\$\\|[^0-9]\$/nothing allocated/" \
-        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/everything allocated/"
+    sed -e "s/blocks=$min_blocks\$\$\\|[^0-9]\$/min allocation/" \
+        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
 # get standard environment, filters and checks
@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
 touch "$TEST_DIR/empty"
 extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 
+# We always write the first byte; check how many blocks this filesystem
+# allocates to match empty image alloation.
+printf "\0" > "$TEST_DIR/empty"
+min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
+
 echo
 echo "== creating image with default preallocation =="
 _make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 
 for mode in off full falloc; do
     echo
     echo "== creating image with preallocation $mode =="
     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
-    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
 # success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
 
 == creating image with default preallocation ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation off ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation full ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, everything allocated
+size=1048576, max allocation
 
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, everything allocated
+size=1048576, max allocation
  *** done
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 == raw input image with data (human) ==
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
-required size: 393216
+required size: 458752
 fully allocated size: 1074135040
 wrote 512/512 bytes at offset 512
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
 {
-    "required": 393216,
+    "required": 458752,
     "fully-allocated": 1074135040
 }
 wrote 512/512 bytes at offset 512
diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/221.out
+++ b/tests/qemu-iotests/221.out
@@ -XXX,XX +XXX,XX @@ QA output created by 221
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 1/1 bytes at offset 65536
 1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 *** done
diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/253.out
+++ b/tests/qemu-iotests/253.out
@@ -XXX,XX +XXX,XX @@ QA output created by 253
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 65535/65535 bytes at offset 983040
 63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Using block_resize we can test allocate_first_block() with file
descriptor opened with O_DIRECT, ensuring that it works for any size
larger than 4096 bytes.

Testing smaller sizes is tricky as the result depends on the filesystem
used for testing. For example on NFS any size will work since O_DIRECT
does not require any alignment.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190827010528.8818-3-nsoffer@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  8 ++++++++
 2 files changed, 36 insertions(+)

diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ _filter_blocks()
         -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
+# Resize image using block_resize.
+# Parameter 1: image path
+# Parameter 2: new size
+_block_resize()
+{
+    local path=$1
+    local size=$2
+
+    $QEMU -qmp stdio -nographic -nodefaults \
+        -blockdev file,node-name=file,filename=$path,cache.direct=on \
+        <<EOF
+{'execute': 'qmp_capabilities'}
+{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
+{'execute': 'quit'}
+EOF
+}
+
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
 _supported_proto file
 _supported_os Linux
 
+_default_cache_mode none
+_supported_cache_modes none directsync
+
 size=$((1 * 1024 * 1024))
 
 touch "$TEST_DIR/empty"
@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
     stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
+for new_size in 4096 1048576; do
+    echo
+    echo "== resize empty image with block_resize =="
+    _make_test_img 0 | _filter_imgfmt
+    _block_resize $TEST_IMG $new_size >/dev/null
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
+done
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
 size=1048576, max allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=4096, min allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=1048576, min allocation
  *** done
-- 
2.21.0

fe646693acc changed qemu-img create's output so that it no longer prints
single quotes around parameter values.  The subformat and adapter_type
filters in _filter_img_create() have never been adapted to that change.

Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059.out       | 16 ++++++++--------
 tests/qemu-iotests/common.filter |  4 ++--
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
 
 === Testing monolithicFlat creation and opening ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 2 GiB (2147483648 bytes)
 
 === Testing monolithicFlat with zeroed_grain ===
 qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 
 === Testing big twoGbMaxExtentFlat ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
@@ -XXX,XX +XXX,XX @@ Format specific information:
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
 
 === Testing truncated sparse ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
 
 === Converting to streamOptimized from image with small cluster size===
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
 
 === Testing version 3 ===
@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing 4TB monolithicFlat creation and IO ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 4 TiB (4398046511104 bytes)
@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing qemu-img map on extents ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0               0x20000         0x3f0000        TEST_DIR/t.vmdk
 0x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 0x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
         -e "s# compat6=\$on\\|off\$##g" \
         -e "s# static=\$on\\|off\$##g" \
         -e "s# zeroed_grain=\$on\\|off\$##g" \
-        -e "s# subformat='[^']*'##g" \
-        -e "s# adapter_type='[^']*'##g" \
+        -e "s# subformat=[^ ]*##g" \
+        -e "s# adapter_type=[^ ]*##g" \
         -e "s# hwversion=[^ ]*##g" \
         -e "s# lazy_refcounts=\$on\\|off\$##g" \
         -e "s# block_size=[0-9]\\+##g" \
-- 
2.21.0

This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
turns a former error in 059 into success.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190815153638.4600-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
 tests/qemu-iotests/059     |  7 +++--
 tests/qemu-iotests/059.out |  4 ++-
 3 files changed, 42 insertions(+), 23 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
 }
 
 static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-                              const char *desc_file_path, QDict *options,
-                              Error **errp)
+                              QDict *options, Error **errp)
 {
     int ret;
     int matches;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
     const char *p, *np;
     int64_t sectors = 0;
     int64_t flat_offset;
+    char *desc_file_dir = NULL;
     char *extent_path;
     BdrvChild *extent_file;
     BDRVVmdkState *s = bs->opaque;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             continue;
         }
 
-        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
-            !desc_file_path[0])
-        {
-            bdrv_refresh_filename(bs->file->bs);
-            error_setg(errp, "Cannot use relative extent paths with VMDK "
-                       "descriptor file '%s'", bs->file->bs->filename);
-            return -EINVAL;
-        }
+        if (path_is_absolute(fname)) {
+            extent_path = g_strdup(fname);
+        } else {
+            if (!desc_file_dir) {
+                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
+                if (!desc_file_dir) {
+                    bdrv_refresh_filename(bs->file->bs);
+                    error_prepend(errp, "Cannot use relative paths with VMDK "
+                                  "descriptor file '%s': ",
+                                  bs->file->bs->filename);
+                    ret = -EINVAL;
+                    goto out;
+                }
+            }
 
-        extent_path = path_combine(desc_file_path, fname);
+            extent_path = g_strconcat(desc_file_dir, fname, NULL);
+        }
 
         ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
         assert(ret < 32);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
         g_free(extent_path);
         if (local_err) {
             error_propagate(errp, local_err);
-            return -EINVAL;
+            ret = -EINVAL;
+            goto out;
         }
 
         /* save to extents array */
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                             0, 0, 0, 0, 0, &extent, errp);
             if (ret < 0) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent->flat_start_offset = flat_offset << 9;
         } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             g_free(buf);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else if (!strcmp(type, "SESPARSE")) {
             ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else {
             error_setg(errp, "Unsupported extent type '%s'", type);
             bdrv_unref_child(bs, extent_file);
-            return -ENOTSUP;
+            ret = -ENOTSUP;
+            goto out;
         }
         extent->type = g_strdup(type);
     }
-    return 0;
+
+    ret = 0;
+    goto out;
 
 invalid:
     np = next_line(p);
@@ -XXX,XX +XXX,XX @@ invalid:
         np--;
     }
     error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
-    return -EINVAL;
+    ret = -EINVAL;
+
+out:
+    g_free(desc_file_dir);
+    return ret;
 }
 
 static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
     }
     s->create_type = g_strdup(ct);
     s->desc_offset = 0;
-    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
-                             errp);
+    ret = vmdk_parse_extents(buf, bs, options, errp);
 exit:
     return ret;
 }
diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+# Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
-$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
-    | _filter_testdir | _filter_imgfmt
+$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
+         -c info \
+    2>&1 \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
 echo
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 
 === Testing monolithicFlat with internally generated JSON file name ===
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
+format name: IMGFMT
+cluster size: 0 bytes
+vm state offset: 0 bytes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

We had a test for a case where relative extent paths did not work, but
unfortunately we just fixed the underlying problem, so it works now.
This patch adds a new test case that still fails.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-4-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
 tests/qemu-iotests/059.out |  4 ++++
 2 files changed, 31 insertions(+)

diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+
+echo '--- blkdebug ---'
 # Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
     | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
+echo '--- quorum ---'
+# Should not work, because bdrv_dirname() does not work with quorum
+IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
+cp "$TEST_IMG" "$TEST_IMG.orig"
+
+filename="json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"quorum\",
+        \"children\": [ {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG\"
+        }, {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG.orig\"
+        } ],
+        \"vote-threshold\": 1
+    } }"
+
+filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
+$QEMU_IMG info "$filename" 2>&1 \
+    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
+
+
 echo
 echo "=== Testing version 3 ==="
 _use_sample_img iotest-version3.vmdk.bz2
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
+--- blkdebug ---
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 format name: IMGFMT
 cluster size: 0 bytes
 vm state offset: 0 bytes
+--- quorum ---
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

Compressed writes generally have to write full clusters, not just in
theory but also in practice when it comes to vmdk's streamOptimized
subformat.  It currently is just silently broken for writes with
non-zero in-cluster offsets:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 4096
4 KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
read failed: Invalid argument

(The technical reason is that vmdk_write_extent() just writes the
incomplete compressed data actually to offset 4k.  When reading the
data, vmdk_read_extent() looks at offset 0 and finds the compressed data
size to be 0, because that is what it reads from there.  This yields an
error.)

For incomplete writes with zero in-cluster offsets, the error path when
reading the rest of the cluster is a bit different, but the result is
the same:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 0
4 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
read failed: Invalid argument

(Here, vmdk_read_extent() finds the data and then sees that the
uncompressed data is short.)

It is better to reject invalid writes than to make the user believe they
might have succeeded and then fail when trying to read it back.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-5-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
     if (extent->compressed) {
         void *compressed_data;
 
+        /* Only whole clusters */
+        if (offset_in_cluster ||
+            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
+            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
+             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
+        {
+            ret = -EINVAL;
+            goto out;
+        }
+
         if (!extent->has_marker) {
             ret = -EINVAL;
             goto out;
-- 
2.21.0

streamOptimized does not support writes that do not span exactly one
cluster.  Furthermore, it cannot rewrite already allocated clusters.
As such, many iotests do not work with it.  Disable them.

diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/002
+++ b/tests/qemu-iotests/002
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/003
+++ b/tests/qemu-iotests/003
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 size=128M
 offset=67M
diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/005
+++ b/tests/qemu-iotests/005
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 _supported_os Linux
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 # vpc is limited to 127GB, so we can't test it here
 if [ "$IMGFMT" = "vpc" ]; then
diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/009
+++ b/tests/qemu-iotests/009
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/010
+++ b/tests/qemu-iotests/010
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/011
+++ b/tests/qemu-iotests/011
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto generic
 _unsupported_proto vxhs
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/018
+++ b/tests/qemu-iotests/018
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _supported_os Linux
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/019
+++ b/tests/qemu-iotests/019
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 CLUSTER_SIZE=65536
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/027
+++ b/tests/qemu-iotests/027
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vmdk qcow qcow2 qed
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/032
+++ b/tests/qemu-iotests/032
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # This works for any image format (though unlikely to segfault for raw)
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo === Prepare image ===
diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/033
+++ b/tests/qemu-iotests/033
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/034
+++ b/tests/qemu-iotests/034
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/037
+++ b/tests/qemu-iotests/037
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/063
+++ b/tests/qemu-iotests/063
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 _make_test_img 4M
 
diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/072
+++ b/tests/qemu-iotests/072
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
 _supported_proto file
+_unsupported_imgopts "subformat=streamOptimized"
 
 IMG_SIZE=64M
 
diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/105
+++ b/tests/qemu-iotests/105
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2 vmdk vhdx qed
 _supported_proto generic
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 echo
 echo "creating large image"
diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/197
+++ b/tests/qemu-iotests/197
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/215
+++ b/tests/qemu-iotests/215
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/251
+++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _supported_os Linux
+_unsupported_imgopts "subformat=streamOptimized"
 
 if [ "$IMGOPTSSYNTAX" = "true" ]; then
     # We use json:{} filenames here, so we cannot work with additional options.
-- 
2.21.0

The error message for the test case where we have a quorum node for
which no directory name can be generated is different: For
twoGbMaxExtentSparse, it complains that it cannot open the extent file.
For other (sub)formats, it just notes that it cannot determine the
backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
for simplicity's sake.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-7-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/110 | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Any format supporting backing files
 _supported_fmt qed qcow qcow2 vmdk
 _supported_proto file
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=twoGbMaxExtentSparse"
 
 TEST_IMG_REL=$(basename "$TEST_IMG")
 
-- 
2.21.0

From: Stefan Hajnoczi <stefanha@redhat.com>

Fixes: a6b257a08e3d72219f03e461a52152672fec0612
       ("file-posix: Handle undetectable alignment")
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190827101328.4062-1-stefanha@redhat.com
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
             align = alignments[i];
             if (raw_is_io_aligned(fd, buf + align, max_align)) {
-                /* Fallback to request_aligment. */
+                /* Fallback to request_alignment. */
                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
                 break;
             }
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

It is possible to enable only a subset of the block drivers with the
"--block-drv-rw-whitelist" option of the "configure" script. All other
drivers are marked as unusable (or only included as read-only with the
"--block-drv-ro-whitelist" option). If an iotest is now using such a
disabled block driver, it is failing - which is bad, since at least the
tests in the "auto" group should be able to deal with this situation.
Thus let's introduce a "_require_drivers" function that can be used by
the shell tests to check for the availability of certain drivers first,
and marks the test as "not run" if one of the drivers is missing.

This patch mainly targets the test in the "auto" group which should
never fail in such a case, but also improves some of the other tests
along the way. Note that we also assume that the "qcow2" and "file"
drivers are always available - otherwise it does not make sense to
run "make check-block" at all (which only tests with qcow2 by default).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823133552.11680-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/071       |  1 +
 tests/qemu-iotests/081       |  4 +---
 tests/qemu-iotests/099       |  1 +
 tests/qemu-iotests/120       |  1 +
 tests/qemu-iotests/162       |  4 +---
 tests/qemu-iotests/184       |  1 +
 tests/qemu-iotests/186       |  1 +
 tests/qemu-iotests/common.rc | 14 ++++++++++++++
 8 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/071
+++ b/tests/qemu-iotests/071
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers blkdebug blkverify
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/081
+++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt raw
 _supported_proto file
 _supported_os Linux
+_require_drivers quorum
 
 do_run_qemu()
 {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                           | _filter_qemu_io | _filter_generated_node_ids
 }
 
-test_quorum=$($QEMU_IMG --help|grep quorum)
-[ "$test_quorum" = "" ] && _supported_fmt quorum
-
 quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
 quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
 quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/099
+++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
 _supported_proto file
 _supported_os Linux
+_require_drivers blkdebug blkverify
 _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
     "subformat=twoGbMaxExtentSparse"
 
diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/120
+++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _unsupported_fmt luks
+_require_drivers raw
 
 _make_test_img 64M
 
diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/162
+++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
-
-test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh$ \|$$')
-[ "$test_ssh" = "" ] && _notrun "ssh support required"
+_require_drivers ssh
 
 echo
 echo '=== NBD ==='
diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/184
+++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_os Linux
+_require_drivers throttle
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/186
+++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers null-co
 
 if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
     _notrun "Requires a PC machine"
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -XXX,XX +XXX,XX @@ _require_command()
     [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
 }
 
+# Check that a set of drivers has been whitelisted in the QEMU binary
+#
+_require_drivers()
+{
+    available=$($QEMU -drive format=help | \
+                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
+    for driver
+    do
+        if ! echo "$available" | grep -q " $driver$ \|$$"; then
+            _notrun "$driver not available"
+        fi
+    done
+}
+
 # make sure this script returns success
 true
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

The sanitizers (especially the address sanitizer from Clang) are
sometimes printing out warnings or false positives - this spoils
the output of the iotests, causing some of the tests to fail.
Thus let's skip the automatic iotests during "make check" when the
user configured QEMU with --enable-sanitizers.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823084203.29734-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/check-block.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tests/check-block.sh b/tests/check-block.sh
index XXXXXXX..XXXXXXX 100755
--- a/tests/check-block.sh
+++ b/tests/check-block.sh
@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
     exit 0
 fi
 
+if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
+    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
+    exit 0
+fi
+
 if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
     echo "No qemu-system binary available ==> Not running the qemu-iotests."
     exit 0
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Quoting cache mode is not needed, and most tests use unquoted values.
Unify all test to use the same style.

Message-id: 20190827173432.7656-1-nsoffer@redhat.com
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/026 | 4 ++--
 tests/qemu-iotests/039 | 4 ++--
 tests/qemu-iotests/052 | 2 +-
 tests/qemu-iotests/091 | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/026
+++ b/tests/qemu-iotests/026
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Currently only qcow2 supports rebasing
 _supported_fmt qcow2
 _supported_proto file
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough" "none"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough none
 # The refcount table tests expect a certain minimum width for refcount entries
 # (so that the refcount table actually needs to grow); that minimum is 16 bits,
 # being the default refcount entry width.
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough
 
 size=128M
 
diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/052
+++ b/tests/qemu-iotests/052
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto file
 
 # Don't do O_DIRECT on tmpfs
-_supported_cache_modes "writeback" "writethrough" "unsafe"
+_supported_cache_modes writeback writethrough unsafe
 
 size=128M
 _make_test_img $size
diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/091
+++ b/tests/qemu-iotests/091
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "none"
-_supported_cache_modes "writethrough" "none" "writeback"
+_default_cache_mode none
+_supported_cache_modes writethrough none writeback
 
 size=1G
 
-- 
2.21.0