Series comparison

-[Qemu-devel] [PULL 00/15] Block patches
+[Qemu-devel] [PULL 00/16] Block patches
-The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:
+The following changes since commit dd25f97c66a75d1508f1d4c6478ed2c95bec428f:
-  Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)
+  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20190913' into staging (2019-09-16 10:15:15 +0100)
 are available in the Git repository at:
-  https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27
+  https://github.com/XanClic/qemu.git tags/pull-block-2019-09-16
-for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:
+for you to fetch changes up to 1825cc0783ccf0ec5d9f0b225a99b340bdd4c68f:
-  iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)
+  qemu-iotests: Add test for bz #1745922 (2019-09-16 15:37:12 +0200)
 ----------------------------------------------------------------
 Block patches:
-- qemu-io now accepts a file to read a write pattern from
+- Fix for block jobs when used with I/O threads
-- Ensure that raw files have their first block allocated so we can probe
+- Fix for a corruption when using qcow2's LUKS encryption mode
-  the O_DIRECT alignment if necessary
+- cURL fix
-- Various fixes
+- check-block.sh cleanups (for make check)
 - Refactoring
 ----------------------------------------------------------------
-Denis Plotnikov (1):
+Max Reitz (7):
-  qemu-io: add pattern file for write command
+  curl: Keep pointer to the CURLState in CURLSocket
   curl: Keep *socket until the end of curl_sock_cb()
   curl: Check completion in curl_multi_do()
   curl: Pass CURLSocket to curl_multi_do()
   curl: Report only ready sockets
   curl: Handle success in multi_check_completion
   curl: Check curl_multi_add_handle()'s return code
-Max Reitz (7):
+Maxim Levitsky (3):
-  iotests: Fix _filter_img_create()
+  block/qcow2: Fix corruption introduced by commit 8ac0f15f335
-  vmdk: Use bdrv_dirname() for relative extent paths
+  block/qcow2: refactor encryption code
-  iotests: Keep testing broken relative extent paths
+  qemu-iotests: Add test for bz #1745922
   vmdk: Reject invalid compressed writes
   iotests: Disable broken streamOptimized tests
   iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
   iotests: Disable 126 for flat vmdk subformats
-Nir Soffer (3):
+Nir Soffer (2):
-  block: posix: Always allocate the first block
+  block: Use QEMU_IS_ALIGNED
-  iotests: Test allocate_first_block() with O_DIRECT
+  block: Remove unused masks
   iotests: Unify cache mode quoting
-Stefan Hajnoczi (1):
+Sergio Lopez (1):
-  file-posix: fix request_alignment typo
+  blockjob: update nodes head while removing all bdrv
 Thomas Huth (2):
-  iotests: Check for enabled drivers before testing them
+  tests/qemu-iotests/check: Replace "tests" with "iotests" in final
-  tests/check-block: Skip iotests when sanitizers are enabled
+    status text
   tests/Makefile: Do not print the name of the check-block.sh shell
     script
 Vladimir Sementsov-Ogievskiy (1):
-  block: fix permission update in bdrv_replace_node
+  tests/qemu-iotests: Fix qemu-io related output in 026.out.nocache
- block.c                                       |  5 +-
+ tests/Makefile.include             |   2 +-
- block/file-posix.c                            | 53 +++++++++-
+ block/qcow2.h                      |   8 +-
- block/vmdk.c                                  | 64 ++++++++----
+ include/block/block.h              |   2 -
- qemu-io-cmds.c                                | 99 +++++++++++++++++--
+ block/bochs.c                      |   4 +-
- tests/check-block.sh                          |  5 +
+ block/cloop.c                      |   4 +-
- tests/qemu-iotests/002                        |  1 +
+ block/curl.c                       | 133 ++++++++++-------------
- tests/qemu-iotests/003                        |  1 +
+ block/dmg.c                        |   4 +-
- tests/qemu-iotests/005                        |  3 +-
+ block/io.c                         |   8 +-
- tests/qemu-iotests/009                        |  1 +
+ block/qcow2-cluster.c              |  40 +++----
- tests/qemu-iotests/010                        |  1 +
+ block/qcow2-threads.c              |  63 ++++++++---
- tests/qemu-iotests/011                        |  1 +
+ block/qcow2.c                      |   9 +-
- tests/qemu-iotests/017                        |  3 +-
+ block/vvfat.c                      |   8 +-
- tests/qemu-iotests/018                        |  3 +-
+ blockjob.c                         |  17 ++-
- tests/qemu-iotests/019                        |  3 +-
+ migration/block.c                  |   2 +-
- tests/qemu-iotests/020                        |  3 +-
+ qemu-img.c                         |   2 +-
- tests/qemu-iotests/026                        |  4 +-
+ tests/qemu-iotests/026.out.nocache | 168 ++++++++++++++---------------
- tests/qemu-iotests/027                        |  1 +
+ tests/qemu-iotests/263             |  91 ++++++++++++++++
- tests/qemu-iotests/032                        |  1 +
+ tests/qemu-iotests/263.out         |  40 +++++++
- tests/qemu-iotests/033                        |  1 +
+ tests/qemu-iotests/check           |   8 +-
- tests/qemu-iotests/034                        |  3 +-
+ tests/qemu-iotests/group           |   1 +
- tests/qemu-iotests/037                        |  3 +-
+files changed, 380 insertions(+), 234 deletions(-)
- tests/qemu-iotests/039                        |  4 +-
+ create mode 100755 tests/qemu-iotests/263
- tests/qemu-iotests/052                        |  2 +-
+ create mode 100644 tests/qemu-iotests/263.out
  tests/qemu-iotests/059                        | 34 ++++++-
  tests/qemu-iotests/059.out                    | 26 +++--
  tests/qemu-iotests/063                        |  3 +-
  tests/qemu-iotests/071                        |  1 +
  tests/qemu-iotests/072                        |  1 +
  tests/qemu-iotests/081                        |  4 +-
  tests/qemu-iotests/091                        |  4 +-
  tests/qemu-iotests/099                        |  1 +
  tests/qemu-iotests/105                        |  3 +-
  tests/qemu-iotests/110                        |  3 +-
  tests/qemu-iotests/120                        |  1 +
  tests/qemu-iotests/126                        |  2 +
  tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
  tests/qemu-iotests/150.out.raw                | 12 +++
  tests/qemu-iotests/162                        |  4 +-
  tests/qemu-iotests/175                        | 47 +++++++--
  tests/qemu-iotests/175.out                    | 16 ++-
  tests/qemu-iotests/178.out.qcow2              |  4 +-
  tests/qemu-iotests/184                        |  1 +
  tests/qemu-iotests/186                        |  1 +
  tests/qemu-iotests/197                        |  1 +
  tests/qemu-iotests/215                        |  1 +
  tests/qemu-iotests/221.out                    | 12 ++-
  tests/qemu-iotests/251                        |  1 +
  tests/qemu-iotests/253.out                    | 12 ++-
  tests/qemu-iotests/common.filter              |  4 +-
  tests/qemu-iotests/common.rc                  | 14 +++
 files changed, 391 insertions(+), 87 deletions(-)
  rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
  create mode 100644 tests/qemu-iotests/150.out.raw
 --
 .21.0

-[Qemu-devel] [PULL 04/15] iotests: Test allocate_first_block() with O_DIRECT
+[Qemu-devel] [PULL 01/16] block: Use QEMU_IS_ALIGNED
 From: Nir Soffer <nirsof@gmail.com>
-Using block_resize we can test allocate_first_block() with file
+Replace instances of:
 descriptor opened with O_DIRECT, ensuring that it works for any size
 larger than 4096 bytes.
-Testing smaller sizes is tricky as the result depends on the filesystem
+    (n & (BDRV_SECTOR_SIZE - 1)) == 0
-used for testing. For example on NFS any size will work since O_DIRECT
-does not require any alignment.
+And:
    (n & ~BDRV_SECTOR_MASK) == 0
 With:
     QEMU_IS_ALIGNED(n, BDRV_SECTOR_SIZE)
 Which reveals the intent of the code better, and makes it easier to
 locate the code checking alignment.
 Signed-off-by: Nir Soffer <nsoffer@redhat.com>
-Reviewed-by: Max Reitz <mreitz@redhat.com>
+Message-id: 20190827185913.27427-2-nsoffer@redhat.com
-Message-id: 20190827010528.8818-3-nsoffer@redhat.com
+Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
+ block/bochs.c         | 4 ++--
- tests/qemu-iotests/175.out |  8 ++++++++
+ block/cloop.c         | 4 ++--
-files changed, 36 insertions(+)
+ block/dmg.c           | 4 ++--
  block/io.c            | 8 ++++----
  block/qcow2-cluster.c | 4 ++--
  block/qcow2.c         | 4 ++--
  block/vvfat.c         | 8 ++++----
  qemu-img.c            | 2 +-
 files changed, 19 insertions(+), 19 deletions(-)
-diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
+diff --git a/block/bochs.c b/block/bochs.c
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/175
 +++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ _filter_blocks()
          -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
  }
 +# Resize image using block_resize.
 +# Parameter 1: image path
 +# Parameter 2: new size
 +_block_resize()
 +{
 +    local path=$1
 +    local size=$2
 +
 +    $QEMU -qmp stdio -nographic -nodefaults \
 +        -blockdev file,node-name=file,filename=$path,cache.direct=on \
 +        <<EOF
 +{'execute': 'qmp_capabilities'}
 +{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
 +{'execute': 'quit'}
 +EOF
 +}
 +
  # get standard environment, filters and checks
  . ./common.rc
  . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
  _supported_proto file
  _supported_os Linux
 +_default_cache_mode none
 +_supported_cache_modes none directsync
 +
  size=$((1 * 1024 * 1024))
  touch "$TEST_DIR/empty"
@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
      stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  done
 +for new_size in 4096 1048576; do
 +    echo
 +    echo "== resize empty image with block_resize =="
 +    _make_test_img 0 | _filter_imgfmt
 +    _block_resize $TEST_IMG $new_size >/dev/null
 +    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
 +done
 +
  # success, all done
  echo "*** done"
  rm -f $seq.full
 diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/175.out
+--- a/block/bochs.c
-+++ b/tests/qemu-iotests/175.out
++++ b/block/bochs.c
-@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
+@@ -XXX,XX +XXX,XX @@ bochs_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
- == creating image with preallocation falloc ==
+     QEMUIOVector local_qiov;
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
+     int ret;
- size=1048576, max allocation
-+
+-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-+== resize empty image with block_resize ==
+-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
++    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
-+size=4096, min allocation
++    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
-+
-+== resize empty image with block_resize ==
+     qemu_iovec_init(&local_qiov, qiov->niov);
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+     qemu_co_mutex_lock(&s->lock);
-+size=1048576, min allocation
+diff --git a/block/cloop.c b/block/cloop.c
-  *** done
+index XXXXXXX..XXXXXXX 100644
 --- a/block/cloop.c
 +++ b/block/cloop.c
@@ -XXX,XX +XXX,XX @@ cloop_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
      int nb_sectors = bytes >> BDRV_SECTOR_BITS;
      int ret, i;
 -    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
 -    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
 +    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
      qemu_co_mutex_lock(&s->lock);
 diff --git a/block/dmg.c b/block/dmg.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/dmg.c
 +++ b/block/dmg.c
@@ -XXX,XX +XXX,XX @@ dmg_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
      int nb_sectors = bytes >> BDRV_SECTOR_BITS;
      int ret, i;
 -    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
 -    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
 +    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
      qemu_co_mutex_lock(&s->lock);
 diff --git a/block/io.c b/block/io.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/io.c
 +++ b/block/io.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_driver_preadv(BlockDriverState *bs,
      sector_num = offset >> BDRV_SECTOR_BITS;
      nb_sectors = bytes >> BDRV_SECTOR_BITS;
 -    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
 -    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
 +    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
      assert(bytes <= BDRV_REQUEST_MAX_BYTES);
      assert(drv->bdrv_co_readv);
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_driver_pwritev(BlockDriverState *bs,
      sector_num = offset >> BDRV_SECTOR_BITS;
      nb_sectors = bytes >> BDRV_SECTOR_BITS;
 -    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
 -    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
 +    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
      assert(bytes <= BDRV_REQUEST_MAX_BYTES);
      assert(drv->bdrv_co_writev);
 diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/qcow2-cluster.c
 +++ b/block/qcow2-cluster.c
@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
  {
      if (bytes && bs->encrypted) {
          BDRVQcow2State *s = bs->opaque;
 -        assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
 -        assert((bytes & ~BDRV_SECTOR_MASK) == 0);
 +        assert(QEMU_IS_ALIGNED(offset_in_cluster, BDRV_SECTOR_SIZE));
 +        assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
          assert(s->crypto);
          if (qcow2_co_encrypt(bs, cluster_offset,
                               src_cluster_offset + offset_in_cluster,
 diff --git a/block/qcow2.c b/block/qcow2.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/qcow2.c
 +++ b/block/qcow2.c
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv_part(BlockDriverState *bs,
                      goto fail;
                  }
 -                assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
 -                assert((cur_bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
 +                assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
 +                assert(QEMU_IS_ALIGNED(cur_bytes, BDRV_SECTOR_SIZE));
                  if (qcow2_co_decrypt(bs, cluster_offset, offset,
                                       cluster_data, cur_bytes) < 0) {
                      ret = -EIO;
 diff --git a/block/vvfat.c b/block/vvfat.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/vvfat.c
 +++ b/block/vvfat.c
@@ -XXX,XX +XXX,XX @@ vvfat_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
      int nb_sectors = bytes >> BDRV_SECTOR_BITS;
      void *buf;
 -    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
 -    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
 +    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
      buf = g_try_malloc(bytes);
      if (bytes && buf == NULL) {
@@ -XXX,XX +XXX,XX @@ vvfat_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
      int nb_sectors = bytes >> BDRV_SECTOR_BITS;
      void *buf;
 -    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
 -    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
 +    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
      buf = g_try_malloc(bytes);
      if (bytes && buf == NULL) {
 diff --git a/qemu-img.c b/qemu-img.c
 index XXXXXXX..XXXXXXX 100644
 --- a/qemu-img.c
 +++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
              int64_t sval;
              sval = cvtnum(optarg);
 -            if (sval < 0 || sval & (BDRV_SECTOR_SIZE - 1) ||
 +            if (sval < 0 || !QEMU_IS_ALIGNED(sval, BDRV_SECTOR_SIZE) ||
                  sval / BDRV_SECTOR_SIZE > MAX_BUF_SECTORS) {
                  error_report("Invalid buffer size for sparse output specified. "
                      "Valid sizes are multiples of %llu up to %llu. Select "
 --
 .21.0

-[Qemu-devel] [PULL 15/15] iotests: Unify cache mode quoting
+[Qemu-devel] [PULL 02/16] block: Remove unused masks
 From: Nir Soffer <nirsof@gmail.com>
-Quoting cache mode is not needed, and most tests use unquoted values.
+Replace confusing usage:
 Unify all test to use the same style.
-Message-id: 20190827173432.7656-1-nsoffer@redhat.com
+    ~BDRV_SECTOR_MASK
 With more clear:
     (BDRV_SECTOR_SIZE - 1)
 Remove BDRV_SECTOR_MASK and the unused BDRV_BLOCK_OFFSET_MASK which was
 it's last user.
 Signed-off-by: Nir Soffer <nsoffer@redhat.com>
+Message-id: 20190827185913.27427-3-nsoffer@redhat.com
+Reviewed-by: Juan Quintela <quintela@redhat.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/026 | 4 ++--
+ include/block/block.h | 2 --
- tests/qemu-iotests/039 | 4 ++--
+ migration/block.c     | 2 +-
- tests/qemu-iotests/052 | 2 +-
+files changed, 1 insertion(+), 3 deletions(-)
  tests/qemu-iotests/091 | 4 ++--
 files changed, 7 insertions(+), 7 deletions(-)
-diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
+diff --git a/include/block/block.h b/include/block/block.h
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/026
+--- a/include/block/block.h
-+++ b/tests/qemu-iotests/026
++++ b/include/block/block.h
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ typedef struct HDGeometry {
- # Currently only qcow2 supports rebasing
- _supported_fmt qcow2
+ #define BDRV_SECTOR_BITS   9
- _supported_proto file
+ #define BDRV_SECTOR_SIZE   (1ULL << BDRV_SECTOR_BITS)
--_default_cache_mode "writethrough"
+-#define BDRV_SECTOR_MASK   ~(BDRV_SECTOR_SIZE - 1)
--_supported_cache_modes "writethrough" "none"
-+_default_cache_mode writethrough
+ #define BDRV_REQUEST_MAX_SECTORS MIN(SIZE_MAX >> BDRV_SECTOR_BITS, \
-+_supported_cache_modes writethrough none
+                                      INT_MAX >> BDRV_SECTOR_BITS)
- # The refcount table tests expect a certain minimum width for refcount entries
+@@ -XXX,XX +XXX,XX @@ typedef struct HDGeometry {
- # (so that the refcount table actually needs to grow); that minimum is 16 bits,
+ #define BDRV_BLOCK_ALLOCATED    0x10
- # being the default refcount entry width.
+ #define BDRV_BLOCK_EOF          0x20
-diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
+ #define BDRV_BLOCK_RECURSE      0x40
-index XXXXXXX..XXXXXXX 100755
+-#define BDRV_BLOCK_OFFSET_MASK  BDRV_SECTOR_MASK
---- a/tests/qemu-iotests/039
-+++ b/tests/qemu-iotests/039
+ typedef QSIMPLEQ_HEAD(BlockReopenQueue, BlockReopenQueueEntry) BlockReopenQueue;
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
- _supported_fmt qcow2
+diff --git a/migration/block.c b/migration/block.c
- _supported_proto file
+index XXXXXXX..XXXXXXX 100644
- _supported_os Linux
+--- a/migration/block.c
--_default_cache_mode "writethrough"
++++ b/migration/block.c
--_supported_cache_modes "writethrough"
+@@ -XXX,XX +XXX,XX @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
-+_default_cache_mode writethrough
+     do {
-+_supported_cache_modes writethrough
+         addr = qemu_get_be64(f);
- size=128M
+-        flags = addr & ~BDRV_SECTOR_MASK;
++        flags = addr & (BDRV_SECTOR_SIZE - 1);
-diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
+         addr >>= BDRV_SECTOR_BITS;
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/052
+         if (flags & BLK_MIG_FLAG_DEVICE_BLOCK) {
 +++ b/tests/qemu-iotests/052
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto file
  # Don't do O_DIRECT on tmpfs
 -_supported_cache_modes "writeback" "writethrough" "unsafe"
 +_supported_cache_modes writeback writethrough unsafe
  size=128M
  _make_test_img $size
 diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/091
 +++ b/tests/qemu-iotests/091
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
  _supported_os Linux
 -_default_cache_mode "none"
 -_supported_cache_modes "writethrough" "none" "writeback"
 +_default_cache_mode none
 +_supported_cache_modes writethrough none writeback
  size=1G
 --
 .21.0

-[Qemu-devel] [PULL 14/15] tests/check-block: Skip iotests when sanitizers are enabled
+[Qemu-devel] [PULL 03/16] tests/qemu-iotests/check: Replace "tests" with "iotests" in final status text
 From: Thomas Huth <thuth@redhat.com>
-The sanitizers (especially the address sanitizer from Clang) are
+When running "make check -j8" or something similar, the iotests are
-sometimes printing out warnings or false positives - this spoils
+running in parallel with the other tests. So when they are printing
-the output of the iotests, causing some of the tests to fail.
+out "Passed all xx tests" or a similar status message at the end,
-Thus let's skip the automatic iotests during "make check" when the
+it might not be quite clear that this message belongs to the iotests,
-user configured QEMU with --enable-sanitizers.
+since the output might be mixed with the other tests. Thus change the
 word "tests" here to "iotests" instead to avoid confusion.
 Signed-off-by: Thomas Huth <thuth@redhat.com>
-Message-id: 20190823084203.29734-1-thuth@redhat.com
+Message-id: 20190906113920.11271-1-thuth@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/check-block.sh | 5 +++++
+ tests/qemu-iotests/check | 8 ++++----
-file changed, 5 insertions(+)
+file changed, 4 insertions(+), 4 deletions(-)
-diff --git a/tests/check-block.sh b/tests/check-block.sh
+diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
 index XXXXXXX..XXXXXXX 100755
---- a/tests/check-block.sh
+--- a/tests/qemu-iotests/check
-+++ b/tests/check-block.sh
++++ b/tests/qemu-iotests/check
-@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
+@@ -XXX,XX +XXX,XX @@ END        { if (NR > 0) {
-     exit 0
+         if [ ! -z "$n_bad" -a $n_bad != 0 ]
- fi
+         then
+             echo "Failures:$bad"
-+if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
+-            echo "Failed $n_bad of $try tests"
-+    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
++            echo "Failed $n_bad of $try iotests"
-+    exit 0
+             echo "Failures:$bad" | fmt >>check.log
-+fi
+-            echo "Failed $n_bad of $try tests" >>check.log
-+
++            echo "Failed $n_bad of $try iotests" >>check.log
- if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
+         else
-     echo "No qemu-system binary available ==> Not running the qemu-iotests."
+-            echo "Passed all $try tests"
-     exit 0
+-            echo "Passed all $try tests" >>check.log
 +            echo "Passed all $try iotests"
 +            echo "Passed all $try iotests" >>check.log
          fi
          needwrap=false
      fi
 --
 .21.0

-[Qemu-devel] [PULL 13/15] iotests: Check for enabled drivers before testing them
+[Qemu-devel] [PULL 04/16] tests/Makefile: Do not print the name of the check-block.sh shell script
 From: Thomas Huth <thuth@redhat.com>
-It is possible to enable only a subset of the block drivers with the
+The check script is already printing out which iotest is currently
-"--block-drv-rw-whitelist" option of the "configure" script. All other
+running, so printing out the name of the check-block.sh shell script
-drivers are marked as unusable (or only included as read-only with the
+looks superfluous here.
 "--block-drv-ro-whitelist" option). If an iotest is now using such a
 disabled block driver, it is failing - which is bad, since at least the
 tests in the "auto" group should be able to deal with this situation.
 Thus let's introduce a "_require_drivers" function that can be used by
 the shell tests to check for the availability of certain drivers first,
 and marks the test as "not run" if one of the drivers is missing.
 This patch mainly targets the test in the "auto" group which should
 never fail in such a case, but also improves some of the other tests
 along the way. Note that we also assume that the "qcow2" and "file"
 drivers are always available - otherwise it does not make sense to
 run "make check-block" at all (which only tests with qcow2 by default).
 Signed-off-by: Thomas Huth <thuth@redhat.com>
-Message-id: 20190823133552.11680-1-thuth@redhat.com
+Message-id: 20190906113534.10907-1-thuth@redhat.com
 Acked-by: John Snow <jsnow@redhat.com>
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/071       |  1 +
+ tests/Makefile.include | 2 +-
- tests/qemu-iotests/081       |  4 +---
+file changed, 1 insertion(+), 1 deletion(-)
  tests/qemu-iotests/099       |  1 +
  tests/qemu-iotests/120       |  1 +
  tests/qemu-iotests/162       |  4 +---
  tests/qemu-iotests/184       |  1 +
  tests/qemu-iotests/186       |  1 +
  tests/qemu-iotests/common.rc | 14 ++++++++++++++
 files changed, 21 insertions(+), 6 deletions(-)
-diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
+diff --git a/tests/Makefile.include b/tests/Makefile.include
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/071
 +++ b/tests/qemu-iotests/071
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
 +_require_drivers blkdebug blkverify
  do_run_qemu()
  {
 diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/081
 +++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt raw
  _supported_proto file
  _supported_os Linux
 +_require_drivers quorum
  do_run_qemu()
  {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                            | _filter_qemu_io | _filter_generated_node_ids
  }
 -test_quorum=$($QEMU_IMG --help|grep quorum)
 -[ "$test_quorum" = "" ] && _supported_fmt quorum
 -
  quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
  quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
  quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
 diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/099
 +++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
  _supported_proto file
  _supported_os Linux
 +_require_drivers blkdebug blkverify
  _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
      "subformat=twoGbMaxExtentSparse"
 diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/120
 +++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto file
  _unsupported_fmt luks
 +_require_drivers raw
  _make_test_img 64M
 diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/162
 +++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_fmt generic
 -
 -test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh\( \|$\)')
 -[ "$test_ssh" = "" ] && _notrun "ssh support required"
 +_require_drivers ssh
  echo
  echo '=== NBD ==='
 diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/184
 +++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
  . ./common.filter
  _supported_os Linux
 +_require_drivers throttle
  do_run_qemu()
  {
 diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/186
 +++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
 +_require_drivers null-co
  if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
      _notrun "Requires a PC machine"
 diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/common.rc
+--- a/tests/Makefile.include
-+++ b/tests/qemu-iotests/common.rc
++++ b/tests/Makefile.include
-@@ -XXX,XX +XXX,XX @@ _require_command()
+@@ -XXX,XX +XXX,XX @@ QEMU_IOTESTS_HELPERS-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_LINUX)) = tests/qemu
-     [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
+ check-tests/check-block.sh: tests/check-block.sh qemu-img$(EXESUF) \
- }
+         qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) \
+         $(patsubst %,%/all,$(filter %-softmmu,$(TARGET_DIRS)))
-+# Check that a set of drivers has been whitelisted in the QEMU binary
+-    $<
-+#
++    @$<
-+_require_drivers()
-+{
+ .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
-+    available=$($QEMU -drive format=help | \
+ $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 +                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
 +    for driver
 +    do
 +        if ! echo "$available" | grep -q " $driver\( \|$\)"; then
 +            _notrun "$driver not available"
 +        fi
 +    done
 +}
 +
  # make sure this script returns success
  true
 --
 .21.0

-[Qemu-devel] [PULL 02/15] block: fix permission update in bdrv_replace_node
+[Qemu-devel] [PULL 05/16] tests/qemu-iotests: Fix qemu-io related output in 026.out.nocache
 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-It's wrong to OR shared permissions. It may lead to crash on further
+qemu-io now prefixes its error and warnings with "qemu-io:".
-permission updates.
+b9986b08787019e fixed a lot of iotests output but forget about
-Also, no needs to consider previously calculated permissions, as at
+.out.nocache. Fix it too.
 this point we already bind all new parents and bdrv_get_cumulative_perm
 result is enough. So fix the bug by just set permissions by
 bdrv_get_cumulative_perm result.
-Bug was introduced in long ago 234ac1a9025, in 2.9.
+Fixes: 99e98d7c9fc1a1639fad ("qemu-io: Use error_[gs]et_progname()")
 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
+Message-id: 20190816153015.447957-2-vsementsov@virtuozzo.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block.c | 5 ++---
+ tests/qemu-iotests/026.out.nocache | 168 ++++++++++++++---------------
-file changed, 2 insertions(+), 3 deletions(-)
+file changed, 84 insertions(+), 84 deletions(-)
-diff --git a/block.c b/block.c
+diff --git a/tests/qemu-iotests/026.out.nocache b/tests/qemu-iotests/026.out.nocache
 index XXXXXXX..XXXXXXX 100644
---- a/block.c
+--- a/tests/qemu-iotests/026.out.nocache
-+++ b/block.c
++++ b/tests/qemu-iotests/026.out.nocache
-@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
+@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
- {
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
-     BdrvChild *c, *next;
-     GSList *list = NULL, *p;
+ Event: l1_update; errno: 5; imm: off; once: off; write
--    uint64_t old_perm, old_shared;
+-Failed to flush the L2 table cache: Input/output error
-     uint64_t perm = 0, shared = BLK_PERM_ALL;
+-Failed to flush the refcount block cache: Input/output error
-     int ret;
++qemu-io: Failed to flush the L2 table cache: Input/output error
++qemu-io: Failed to flush the refcount block cache: Input/output error
-@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
+ write failed: Input/output error
-         bdrv_unref(from);
-     }
+leaked clusters were found on the image.
+@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
--    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
+ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
--    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
-+    bdrv_get_cumulative_perm(to, &perm, &shared);
+ Event: l1_update; errno: 5; imm: off; once: off; write -b
-+    bdrv_set_perm(to, perm, shared);
+-Failed to flush the L2 table cache: Input/output error
+-Failed to flush the refcount block cache: Input/output error
- out:
++qemu-io: Failed to flush the L2 table cache: Input/output error
-     g_slist_free(list);
++qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l1_update; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l1_update; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_update; errno: 5; imm: off; once: off; write
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  wrote 131072/131072 bytes at offset 0
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_update; errno: 5; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  wrote 131072/131072 bytes at offset 0
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_update; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  wrote 131072/131072 bytes at offset 0
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_update; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  wrote 131072/131072 bytes at offset 0
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_alloc_write; errno: 5; imm: off; once: off; write
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_alloc_write; errno: 5; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_alloc_write; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l2_alloc_write; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: write_aio; errno: 5; imm: off; once: off; write
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: write_aio; errno: 5; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: write_aio; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: write_aio; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_load; errno: 5; imm: off; once: off; write
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_load; errno: 5; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_load; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_load; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_update_part; errno: 5; imm: off; once: off; write
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_update_part; errno: 5; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_update_part; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_update_part; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc; errno: 5; imm: off; once: off; write
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc; errno: 5; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_hookup; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_hookup; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_write; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_write; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_write_blocks; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_write_blocks; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_write_table; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_write_table; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_switch_table; errno: 28; imm: off; once: off; write
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: refblock_alloc_switch_table; errno: 28; imm: off; once: off; write -b
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l1_grow_write_table; errno: 5; imm: off; once: off
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l1_grow_write_table; errno: 28; imm: off; once: off
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
  No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l1_grow_activate_table; errno: 5; imm: off; once: off
 -Failed to flush the L2 table cache: Input/output error
 -Failed to flush the refcount block cache: Input/output error
 +qemu-io: Failed to flush the L2 table cache: Input/output error
 +qemu-io: Failed to flush the refcount block cache: Input/output error
  write failed: Input/output error
 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
  Event: l1_grow_activate_table; errno: 28; imm: off; once: off
 -Failed to flush the L2 table cache: No space left on device
 -Failed to flush the refcount block cache: No space left on device
 +qemu-io: Failed to flush the L2 table cache: No space left on device
 +qemu-io: Failed to flush the refcount block cache: No space left on device
  write failed: No space left on device
 leaked clusters were found on the image.
 --
 .21.0

-[Qemu-devel] [PULL 07/15] iotests: Keep testing broken relative extent paths
+[Qemu-devel] [PULL 06/16] curl: Keep pointer to the CURLState in CURLSocket
-We had a test for a case where relative extent paths did not work, but
+A follow-up patch will make curl_multi_do() and curl_multi_read() take a
-unfortunately we just fixed the underlying problem, so it works now.
+CURLSocket instead of the CURLState.  They still need the latter,
-This patch adds a new test case that still fails.
+though, so add a pointer to it to the former.
+Cc: qemu-stable@nongnu.org
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 Reviewed-by: John Snow <jsnow@redhat.com>
-Message-id: 20190815153638.4600-4-mreitz@redhat.com
+Message-id: 20190910124136.10565-2-mreitz@redhat.com
-Reviewed-by: John Snow <jsnow@redhat.com>
+Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
+ block/curl.c | 3 +++
- tests/qemu-iotests/059.out |  4 ++++
+file changed, 3 insertions(+)
 files changed, 31 insertions(+)
-diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
+diff --git a/block/curl.c b/block/curl.c
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/059
 +++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
  echo
  echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 +
 +echo '--- blkdebug ---'
  # Should work, because bdrv_dirname() works fine with blkdebug
  IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
  $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
      | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
 +echo '--- quorum ---'
 +# Should not work, because bdrv_dirname() does not work with quorum
 +IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 +cp "$TEST_IMG" "$TEST_IMG.orig"
 +
 +filename="json:{
 +    \"driver\": \"$IMGFMT\",
 +    \"file\": {
 +        \"driver\": \"quorum\",
 +        \"children\": [ {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG\"
 +        }, {
 +            \"driver\": \"file\",
 +            \"filename\": \"$TEST_IMG.orig\"
 +        } ],
 +        \"vote-threshold\": 1
 +    } }"
 +
 +filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
 +$QEMU_IMG info "$filename" 2>&1 \
 +    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
 +
 +
  echo
  echo "=== Testing version 3 ==="
  _use_sample_img iotest-version3.vmdk.bz2
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/059.out
+--- a/block/curl.c
-+++ b/tests/qemu-iotests/059.out
++++ b/block/curl.c
-@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
+@@ -XXX,XX +XXX,XX @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
-bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+ #define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5
- === Testing monolithicFlat with internally generated JSON file name ===
+ struct BDRVCURLState;
-+--- blkdebug ---
++struct CURLState;
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
- format name: IMGFMT
+ static bool libcurl_initialized;
- cluster size: 0 bytes
- vm state offset: 0 bytes
+@@ -XXX,XX +XXX,XX @@ typedef struct CURLAIOCB {
-+--- quorum ---
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+ typedef struct CURLSocket {
-+qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
+     int fd;
++    struct CURLState *state;
- === Testing version 3 ===
+     QLIST_ENTRY(CURLSocket) next;
- image: TEST_DIR/iotest-version3.IMGFMT
+ } CURLSocket;
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
      if (!socket) {
          socket = g_new0(CURLSocket, 1);
          socket->fd = fd;
 +        socket->state = state;
          QLIST_INSERT_HEAD(&state->sockets, socket, next);
      }
      socket = NULL;
 --
 .21.0

-[Qemu-devel] [PULL 10/15] iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
+[Qemu-devel] [PULL 07/16] curl: Keep *socket until the end of curl_sock_cb()
-The error message for the test case where we have a quorum node for
+This does not really change anything, but it makes the code a bit easier
-which no directory name can be generated is different: For
+to follow once we use @socket as the opaque pointer for
-twoGbMaxExtentSparse, it complains that it cannot open the extent file.
+aio_set_fd_handler().
 For other (sub)formats, it just notes that it cannot determine the
 backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
 for simplicity's sake.
+Cc: qemu-stable@nongnu.org
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: John Snow <jsnow@redhat.com>
+Message-id: 20190910124136.10565-3-mreitz@redhat.com
-Message-id: 20190815153638.4600-7-mreitz@redhat.com
+Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/110 | 3 ++-
+ block/curl.c | 10 +++++-----
-file changed, 2 insertions(+), 1 deletion(-)
+file changed, 5 insertions(+), 5 deletions(-)
-diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
+diff --git a/block/curl.c b/block/curl.c
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/110
+--- a/block/curl.c
-+++ b/tests/qemu-iotests/110
++++ b/block/curl.c
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
- # Any format supporting backing files
- _supported_fmt qed qcow qcow2 vmdk
+     QLIST_FOREACH(socket, &state->sockets, next) {
- _supported_proto file
+         if (socket->fd == fd) {
--_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+-            if (action == CURL_POLL_REMOVE) {
-+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+-                QLIST_REMOVE(socket, next);
-+                     "subformat=twoGbMaxExtentSparse"
+-                g_free(socket);
+-            }
- TEST_IMG_REL=$(basename "$TEST_IMG")
+             break;
          }
      }
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
          socket->state = state;
          QLIST_INSERT_HEAD(&state->sockets, socket, next);
      }
 -    socket = NULL;
      trace_curl_sock_cb(action, (int)fd);
      switch (action) {
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
              break;
      }
 +    if (action == CURL_POLL_REMOVE) {
 +        QLIST_REMOVE(socket, next);
 +        g_free(socket);
 +    }
 +
      return 0;
  }
 --
 .21.0

-[Qemu-devel] [PULL 11/15] iotests: Disable 126 for flat vmdk subformats
+[Qemu-devel] [PULL 08/16] curl: Check completion in curl_multi_do()
-iotest 126 requires backing file support, which flat vmdks cannot offer.
+While it is more likely that transfers complete after some file
-Skip this test for such subformats.
+descriptor has data ready to read, we probably should not rely on it.
 Better be safe than sorry and call curl_multi_check_completion() in
 curl_multi_do(), too, just like it is done in curl_multi_read().
 With this change, curl_multi_do() and curl_multi_read() are actually the
 same, so drop curl_multi_read() and use curl_multi_do() as the sole FD
 handler.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20190815153638.4600-8-mreitz@redhat.com
+Message-id: 20190910124136.10565-4-mreitz@redhat.com
 Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/126 | 2 ++
+ block/curl.c | 14 ++------------
-file changed, 2 insertions(+)
+file changed, 2 insertions(+), 12 deletions(-)
-diff --git a/tests/qemu-iotests/126 b/tests/qemu-iotests/126
+diff --git a/block/curl.c b/block/curl.c
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/126
+--- a/block/curl.c
-+++ b/tests/qemu-iotests/126
++++ b/block/curl.c
-@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
+@@ -XXX,XX +XXX,XX @@ typedef struct BDRVCURLState {
- # Needs backing file support
+ static void curl_clean_state(CURLState *s);
- _supported_fmt qcow qcow2 qed vmdk
+ static void curl_multi_do(void *arg);
-+_unsupported_imgopts "subformat=monolithicFlat" \
+-static void curl_multi_read(void *arg);
-+                     "subformat=twoGbMaxExtentFlat"
- # This is the default protocol (and we want to test the difference between
+ #ifdef NEED_CURL_TIMER_CALLBACK
- # colons which separate a protocol prefix from the rest and colons which are
+ /* Called from curl_multi_do_locked, with s->mutex held.  */
- # just part of the filename, so we cannot test protocols which require a prefix)
+@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
      switch (action) {
          case CURL_POLL_IN:
              aio_set_fd_handler(s->aio_context, fd, false,
 -                               curl_multi_read, NULL, NULL, state);
 +                               curl_multi_do, NULL, NULL, state);
              break;
          case CURL_POLL_OUT:
              aio_set_fd_handler(s->aio_context, fd, false,
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
              break;
          case CURL_POLL_INOUT:
              aio_set_fd_handler(s->aio_context, fd, false,
 -                               curl_multi_read, curl_multi_do, NULL, state);
 +                               curl_multi_do, curl_multi_do, NULL, state);
              break;
          case CURL_POLL_REMOVE:
              aio_set_fd_handler(s->aio_context, fd, false,
@@ -XXX,XX +XXX,XX @@ static void curl_multi_do(void *arg)
  {
      CURLState *s = (CURLState *)arg;
 -    qemu_mutex_lock(&s->s->mutex);
 -    curl_multi_do_locked(s);
 -    qemu_mutex_unlock(&s->s->mutex);
 -}
 -
 -static void curl_multi_read(void *arg)
 -{
 -    CURLState *s = (CURLState *)arg;
 -
      qemu_mutex_lock(&s->s->mutex);
      curl_multi_do_locked(s);
      curl_multi_check_completion(s->s);
 --
 .21.0

-[Qemu-devel] [PULL 06/15] vmdk: Use bdrv_dirname() for relative extent paths
+[Qemu-devel] [PULL 09/16] curl: Pass CURLSocket to curl_multi_do()
-This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
+curl_multi_do_locked() currently marks all sockets as ready.  That is
-turns a former error in 059 into success.
+not only inefficient, but in fact unsafe (the loop is).  A follow-up
 patch will change that, but to do so, curl_multi_do_locked() needs to
 know exactly which socket is ready; and that is accomplished by this
 patch here.
+Cc: qemu-stable@nongnu.org
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20190815153638.4600-3-mreitz@redhat.com
+Message-id: 20190910124136.10565-5-mreitz@redhat.com
 Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
+ block/curl.c | 20 +++++++++++---------
- tests/qemu-iotests/059     |  7 +++--
+file changed, 11 insertions(+), 9 deletions(-)
  tests/qemu-iotests/059.out |  4 ++-
 files changed, 42 insertions(+), 23 deletions(-)
-diff --git a/block/vmdk.c b/block/vmdk.c
+diff --git a/block/curl.c b/block/curl.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/vmdk.c
+--- a/block/curl.c
-+++ b/block/vmdk.c
++++ b/block/curl.c
-@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
+@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
      switch (action) {
          case CURL_POLL_IN:
              aio_set_fd_handler(s->aio_context, fd, false,
 -                               curl_multi_do, NULL, NULL, state);
 +                               curl_multi_do, NULL, NULL, socket);
              break;
          case CURL_POLL_OUT:
              aio_set_fd_handler(s->aio_context, fd, false,
 -                               NULL, curl_multi_do, NULL, state);
 +                               NULL, curl_multi_do, NULL, socket);
              break;
          case CURL_POLL_INOUT:
              aio_set_fd_handler(s->aio_context, fd, false,
 -                               curl_multi_do, curl_multi_do, NULL, state);
 +                               curl_multi_do, curl_multi_do, NULL, socket);
              break;
          case CURL_POLL_REMOVE:
              aio_set_fd_handler(s->aio_context, fd, false,
@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
  }
- static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
+ /* Called with s->mutex held.  */
--                              const char *desc_file_path, QDict *options,
+-static void curl_multi_do_locked(CURLState *s)
--                              Error **errp)
++static void curl_multi_do_locked(CURLSocket *ready_socket)
 +                              QDict *options, Error **errp)
  {
-     int ret;
+     CURLSocket *socket, *next_socket;
-     int matches;
++    CURLState *s = ready_socket->state;
-@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
+     int running;
-     const char *p, *np;
+     int r;
-     int64_t sectors = 0;
-     int64_t flat_offset;
+@@ -XXX,XX +XXX,XX @@ static void curl_multi_do_locked(CURLState *s)
-+    char *desc_file_dir = NULL;
-     char *extent_path;
+ static void curl_multi_do(void *arg)
-     BdrvChild *extent_file;
+ {
-     BDRVVmdkState *s = bs->opaque;
+-    CURLState *s = (CURLState *)arg;
-@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
++    CURLSocket *socket = arg;
-             continue;
++    BDRVCURLState *s = socket->state->s;
-         }
+-    qemu_mutex_lock(&s->s->mutex);
--        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
+-    curl_multi_do_locked(s);
--            !desc_file_path[0])
+-    curl_multi_check_completion(s->s);
--        {
+-    qemu_mutex_unlock(&s->s->mutex);
--            bdrv_refresh_filename(bs->file->bs);
++    qemu_mutex_lock(&s->mutex);
--            error_setg(errp, "Cannot use relative extent paths with VMDK "
++    curl_multi_do_locked(socket);
--                       "descriptor file '%s'", bs->file->bs->filename);
++    curl_multi_check_completion(s);
--            return -EINVAL;
++    qemu_mutex_unlock(&s->mutex);
 -        }
 +        if (path_is_absolute(fname)) {
 +            extent_path = g_strdup(fname);
 +        } else {
 +            if (!desc_file_dir) {
 +                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
 +                if (!desc_file_dir) {
 +                    bdrv_refresh_filename(bs->file->bs);
 +                    error_prepend(errp, "Cannot use relative paths with VMDK "
 +                                  "descriptor file '%s': ",
 +                                  bs->file->bs->filename);
 +                    ret = -EINVAL;
 +                    goto out;
 +                }
 +            }
 -        extent_path = path_combine(desc_file_path, fname);
 +            extent_path = g_strconcat(desc_file_dir, fname, NULL);
 +        }
          ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
          assert(ret < 32);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          g_free(extent_path);
          if (local_err) {
              error_propagate(errp, local_err);
 -            return -EINVAL;
 +            ret = -EINVAL;
 +            goto out;
          }
          /* save to extents array */
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
 , 0, 0, 0, 0, &extent, errp);
              if (ret < 0) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent->flat_start_offset = flat_offset << 9;
          } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
              g_free(buf);
              if (ret) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent = &s->extents[s->num_extents - 1];
          } else if (!strcmp(type, "SESPARSE")) {
              ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
              if (ret) {
                  bdrv_unref_child(bs, extent_file);
 -                return ret;
 +                goto out;
              }
              extent = &s->extents[s->num_extents - 1];
          } else {
              error_setg(errp, "Unsupported extent type '%s'", type);
              bdrv_unref_child(bs, extent_file);
 -            return -ENOTSUP;
 +            ret = -ENOTSUP;
 +            goto out;
          }
          extent->type = g_strdup(type);
      }
 -    return 0;
 +
 +    ret = 0;
 +    goto out;
  invalid:
      np = next_line(p);
@@ -XXX,XX +XXX,XX @@ invalid:
          np--;
      }
      error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
 -    return -EINVAL;
 +    ret = -EINVAL;
 +
 +out:
 +    g_free(desc_file_dir);
 +    return ret;
  }
- static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
+ static void curl_multi_timeout_do(void *arg)
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
      }
      s->create_type = g_strdup(ct);
      s->desc_offset = 0;
 -    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
 -                             errp);
 +    ret = vmdk_parse_extents(buf, bs, options, errp);
  exit:
      return ret;
  }
 diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/059
 +++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
  echo
  echo "=== Testing monolithicFlat with internally generated JSON file name ==="
 +# Should work, because bdrv_dirname() works fine with blkdebug
  IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 -$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
 -    | _filter_testdir | _filter_imgfmt
 +$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
 +         -c info \
 +    2>&1 \
 +    | _filter_testdir | _filter_imgfmt | _filter_img_info
  _cleanup_test_img
  echo
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/059.out
 +++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
  === Testing monolithicFlat with internally generated JSON file name ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 -qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
 +format name: IMGFMT
 +cluster size: 0 bytes
 +vm state offset: 0 bytes
  === Testing version 3 ===
  image: TEST_DIR/iotest-version3.IMGFMT
 --
 .21.0

-[Qemu-devel] [PULL 01/15] qemu-io: add pattern file for write command
+[Qemu-devel] [PULL 10/16] curl: Report only ready sockets
-From: Denis Plotnikov <dplotnikov@virtuozzo.com>
+Instead of reporting all sockets to cURL, only report the one that has
 caused curl_multi_do_locked() to be called.  This lets us get rid of the
 QLIST_FOREACH_SAFE() list, which was actually wrong: SAFE foreaches are
 only safe when the current element is removed in each iteration.  If it
 possible for the list to be concurrently modified, we cannot guarantee
 that only the current element will be removed.  Therefore, we must not
 use QLIST_FOREACH_SAFE() here.
-The patch allows to provide a pattern file for write
+Fixes: ff5ca1664af85b24a4180d595ea6873fd3deac57
-command. There was no similar ability before.
+Cc: qemu-stable@nongnu.org
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
+Message-id: 20190910124136.10565-6-mreitz@redhat.com
-Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
+Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
 [mreitz: Keep optstring in alphabetical order]
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
+ block/curl.c | 17 ++++++-----------
-file changed, 93 insertions(+), 6 deletions(-)
+file changed, 6 insertions(+), 11 deletions(-)
-diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
+diff --git a/block/curl.c b/block/curl.c
 index XXXXXXX..XXXXXXX 100644
---- a/qemu-io-cmds.c
+--- a/block/curl.c
-+++ b/qemu-io-cmds.c
++++ b/block/curl.c
-@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
+@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
      qemu_vfree(p);
  }
-+/*
+ /* Called with s->mutex held.  */
-+ * qemu_io_alloc_from_file()
+-static void curl_multi_do_locked(CURLSocket *ready_socket)
-+ *
++static void curl_multi_do_locked(CURLSocket *socket)
 + * Allocates the buffer and populates it with the content of the given file
 + * up to @len bytes. If the file length is less than @len, then the buffer
 + * is populated with the file content cyclically.
 + *
 + * @blk - the block backend where the buffer content is going to be written to
 + * @len - the buffer length
 + * @file_name - the file to read the content from
 + *
 + * Returns: the buffer pointer on success
 + *          NULL on error
 + */
 +static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
 +                                     const char *file_name)
 +{
 +    char *buf, *buf_origin;
 +    FILE *f = fopen(file_name, "r");
 +    int pattern_len;
 +
 +    if (!f) {
 +        perror(file_name);
 +        return NULL;
 +    }
 +
 +    if (qemuio_misalign) {
 +        len += MISALIGN_OFFSET;
 +    }
 +
 +    buf_origin = buf = blk_blockalign(blk, len);
 +
 +    if (qemuio_misalign) {
 +        buf_origin += MISALIGN_OFFSET;
 +        buf += MISALIGN_OFFSET;
 +        len -= MISALIGN_OFFSET;
 +    }
 +
 +    pattern_len = fread(buf_origin, 1, len, f);
 +
 +    if (ferror(f)) {
 +        perror(file_name);
 +        goto error;
 +    }
 +
 +    if (pattern_len == 0) {
 +        fprintf(stderr, "%s: file is empty\n", file_name);
 +        goto error;
 +    }
 +
 +    fclose(f);
 +
 +    if (len > pattern_len) {
 +        len -= pattern_len;
 +        buf += pattern_len;
 +
 +        while (len > 0) {
 +            size_t len_to_copy = MIN(pattern_len, len);
 +
 +            memcpy(buf, buf_origin, len_to_copy);
 +
 +            len -= len_to_copy;
 +            buf += len_to_copy;
 +        }
 +    }
 +
 +    return buf_origin;
 +
 +error:
 +    qemu_io_free(buf_origin);
 +    return NULL;
 +}
 +
  static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
  {
-     uint64_t i;
+-    CURLSocket *socket, *next_socket;
-@@ -XXX,XX +XXX,XX @@ static void write_help(void)
+-    CURLState *s = ready_socket->state;
- " -n, -- with -z, don't allow slow fallback\n"
++    BDRVCURLState *s = socket->state->s;
- " -p, -- ignored for backwards compatibility\n"
+     int running;
- " -P, -- use different pattern to fill file\n"
+     int r;
-+" -s, -- use a pattern file to fill the write buffer\n"
- " -C, -- report statistics in a machine parsable format\n"
+-    if (!s->s->multi) {
- " -q, -- quiet mode, do not show I/O statistics\n"
++    if (!s->multi) {
- " -u, -- with -z, allow unmapping\n"
+         return;
@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
      .perm       = BLK_PERM_WRITE,
      .argmin     = 2,
      .argmax     = -1,
 -    .args       = "[-bcCfnquz] [-P pattern] off len",
 +    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
      .oneline    = "writes a number of bytes at a specified offset",
      .help       = write_help,
  };
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
  {
      struct timespec t1, t2;
      bool Cflag = false, qflag = false, bflag = false;
 -    bool Pflag = false, zflag = false, cflag = false;
 +    bool Pflag = false, zflag = false, cflag = false, sflag = false;
      int flags = 0;
      int c, cnt, ret;
      char *buf = NULL;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
      /* Some compilers get confused and warn if this is not initialized.  */
      int64_t total = 0;
      int pattern = 0xcd;
 +    const char *file_name = NULL;
 -    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
 +    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
          switch (c) {
          case 'b':
              bflag = true;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
          case 'q':
              qflag = true;
              break;
 +        case 's':
 +            sflag = true;
 +            file_name = optarg;
 +            break;
          case 'u':
              flags |= BDRV_REQ_MAY_UNMAP;
              break;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
          return -EINVAL;
      }
--    if (zflag && Pflag) {
+-    /* Need to use _SAFE because curl_multi_socket_action() may trigger
--        printf("-z and -P cannot be specified at the same time\n");
+-     * curl_sock_cb() which might modify this list */
-+    if (zflag + Pflag + sflag > 1) {
+-    QLIST_FOREACH_SAFE(socket, &s->sockets, next, next_socket) {
-+        printf("Only one of -z, -P, and -s "
+-        do {
-+               "can be specified at the same time\n");
+-            r = curl_multi_socket_action(s->s->multi, socket->fd, 0, &running);
-         return -EINVAL;
+-        } while (r == CURLM_CALL_MULTI_PERFORM);
-     }
+-    }
++    do {
-@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
++        r = curl_multi_socket_action(s->multi, socket->fd, 0, &running);
-     }
++    } while (r == CURLM_CALL_MULTI_PERFORM);
+ }
-     if (!zflag) {
--        buf = qemu_io_alloc(blk, count, pattern);
+ static void curl_multi_do(void *arg)
 +        if (sflag) {
 +            buf = qemu_io_alloc_from_file(blk, count, file_name);
 +            if (!buf) {
 +                return -EINVAL;
 +            }
 +        } else {
 +            buf = qemu_io_alloc(blk, count, pattern);
 +        }
      }
      clock_gettime(CLOCK_MONOTONIC, &t1);
 --
 .21.0

-New patch
+[Qemu-devel] [PULL 11/16] curl: Handle success in multi_check_completion
+Background: As of cURL 7.59.0, it verifies that several functions are
+not called from within a callback.  Among these functions is
+curl_multi_add_handle().
+curl_read_cb() is a callback from cURL and not a coroutine.  Waking up
+acb->co will lead to entering it then and there, which means the current
+request will settle and the caller (if it runs in the same coroutine)
+may then issue the next request.  In such a case, we will enter
+curl_setup_preadv() effectively from within curl_read_cb().
+Calling curl_multi_add_handle() will then fail and the new request will
+not be processed.
+Fix this by not letting curl_read_cb() wake up acb->co.  Instead, leave
+the whole business of settling the AIOCB objects to
+curl_multi_check_completion() (which is called from our timer callback
+and our FD handler, so not from any cURL callbacks).
+Reported-by: Natalie Gavrielov <ngavrilo@redhat.com>
+Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1740193
+Cc: qemu-stable@nongnu.org
+Signed-off-by: Max Reitz <mreitz@redhat.com>
+Message-id: 20190910124136.10565-7-mreitz@redhat.com
+Reviewed-by: John Snow <jsnow@redhat.com>
+Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
+Signed-off-by: Max Reitz <mreitz@redhat.com>
+---
+ block/curl.c | 69 ++++++++++++++++++++++------------------------------
+file changed, 29 insertions(+), 40 deletions(-)
+diff --git a/block/curl.c b/block/curl.c
+index XXXXXXX..XXXXXXX 100644
+--- a/block/curl.c
++++ b/block/curl.c
+@@ -XXX,XX +XXX,XX @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
+ {
+     CURLState *s = ((CURLState*)opaque);
+     size_t realsize = size * nmemb;
+-    int i;
+     trace_curl_read_cb(realsize);
+@@ -XXX,XX +XXX,XX @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
+     memcpy(s->orig_buf + s->buf_off, ptr, realsize);
+     s->buf_off += realsize;
+-    for(i=0; i<CURL_NUM_ACB; i++) {
+-        CURLAIOCB *acb = s->acb[i];
+-
+-        if (!acb)
+-            continue;
+-
+-        if ((s->buf_off >= acb->end)) {
+-            size_t request_length = acb->bytes;
+-
+-            qemu_iovec_from_buf(acb->qiov, 0, s->orig_buf + acb->start,
+-                                acb->end - acb->start);
+-
+-            if (acb->end - acb->start < request_length) {
+-                size_t offset = acb->end - acb->start;
+-                qemu_iovec_memset(acb->qiov, offset, 0,
+-                                  request_length - offset);
+-            }
+-
+-            acb->ret = 0;
+-            s->acb[i] = NULL;
+-            qemu_mutex_unlock(&s->s->mutex);
+-            aio_co_wake(acb->co);
+-            qemu_mutex_lock(&s->s->mutex);
+-        }
+-    }
+-
+ read_end:
+     /* curl will error out if we do not return this value */
+     return size * nmemb;
+@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
+             break;
+         if (msg->msg == CURLMSG_DONE) {
++            int i;
+             CURLState *state = NULL;
++            bool error = msg->data.result != CURLE_OK;
++
+             curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE,
+                               (char **)&state);
+-            /* ACBs for successful messages get completed in curl_read_cb */
+-            if (msg->data.result != CURLE_OK) {
+-                int i;
++            if (error) {
+                 static int errcount = 100;
+                 /* Don't lose the original error message from curl, since
+@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
+                         error_report("curl: further errors suppressed");
+                     }
+                 }
++            }
+-                for (i = 0; i < CURL_NUM_ACB; i++) {
+-                    CURLAIOCB *acb = state->acb[i];
++            for (i = 0; i < CURL_NUM_ACB; i++) {
++                CURLAIOCB *acb = state->acb[i];
+-                    if (acb == NULL) {
+-                        continue;
+-                    }
++                if (acb == NULL) {
++                    continue;
++                }
++
++                if (!error) {
++                    /* Assert that we have read all data */
++                    assert(state->buf_off >= acb->end);
++
++                    qemu_iovec_from_buf(acb->qiov, 0,
++                                        state->orig_buf + acb->start,
++                                        acb->end - acb->start);
+-                    acb->ret = -EIO;
+-                    state->acb[i] = NULL;
+-                    qemu_mutex_unlock(&s->mutex);
+-                    aio_co_wake(acb->co);
+-                    qemu_mutex_lock(&s->mutex);
++                    if (acb->end - acb->start < acb->bytes) {
++                        size_t offset = acb->end - acb->start;
++                        qemu_iovec_memset(acb->qiov, offset, 0,
++                                          acb->bytes - offset);
++                    }
+                 }
++
++                acb->ret = error ? -EIO : 0;
++                state->acb[i] = NULL;
++                qemu_mutex_unlock(&s->mutex);
++                aio_co_wake(acb->co);
++                qemu_mutex_lock(&s->mutex);
+             }
+             curl_clean_state(state);
+--
+.21.0

-[Qemu-devel] [PULL 09/15] iotests: Disable broken streamOptimized tests
+[Qemu-devel] [PULL 12/16] curl: Check curl_multi_add_handle()'s return code
-streamOptimized does not support writes that do not span exactly one
+If we had done that all along, debugging would have been much simpler.
-cluster.  Furthermore, it cannot rewrite already allocated clusters.
+(Also, I/O errors are better than hangs.)
 As such, many iotests do not work with it.  Disable them.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20190815153638.4600-6-mreitz@redhat.com
+Message-id: 20190910124136.10565-8-mreitz@redhat.com
 Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/002 | 1 +
+ block/curl.c | 8 +++++++-
- tests/qemu-iotests/003 | 1 +
+file changed, 7 insertions(+), 1 deletion(-)
  tests/qemu-iotests/005 | 3 ++-
  tests/qemu-iotests/009 | 1 +
  tests/qemu-iotests/010 | 1 +
  tests/qemu-iotests/011 | 1 +
  tests/qemu-iotests/017 | 3 ++-
  tests/qemu-iotests/018 | 3 ++-
  tests/qemu-iotests/019 | 3 ++-
  tests/qemu-iotests/020 | 3 ++-
  tests/qemu-iotests/027 | 1 +
  tests/qemu-iotests/032 | 1 +
  tests/qemu-iotests/033 | 1 +
  tests/qemu-iotests/034 | 3 ++-
  tests/qemu-iotests/037 | 3 ++-
  tests/qemu-iotests/063 | 3 ++-
  tests/qemu-iotests/072 | 1 +
  tests/qemu-iotests/105 | 3 ++-
  tests/qemu-iotests/197 | 1 +
  tests/qemu-iotests/215 | 1 +
  tests/qemu-iotests/251 | 1 +
 files changed, 30 insertions(+), 9 deletions(-)
-diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
+diff --git a/block/curl.c b/block/curl.c
-index XXXXXXX..XXXXXXX 100755
+index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/002
+--- a/block/curl.c
-+++ b/tests/qemu-iotests/002
++++ b/block/curl.c
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+@@ -XXX,XX +XXX,XX @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
+     trace_curl_setup_preadv(acb->bytes, start, state->range);
- _supported_fmt generic
+     curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range);
- _supported_proto generic
-+_unsupported_imgopts "subformat=streamOptimized"
+-    curl_multi_add_handle(s->multi, state->curl);
++    if (curl_multi_add_handle(s->multi, state->curl) != CURLM_OK) {
++        state->acb[0] = NULL;
- size=128M
++        acb->ret = -EIO;
-diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
++
-index XXXXXXX..XXXXXXX 100755
++        curl_clean_state(state);
---- a/tests/qemu-iotests/003
++        goto out;
-+++ b/tests/qemu-iotests/003
++    }
-@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
+     /* Tell curl it needs to kick things off */
- _supported_fmt generic
+     curl_multi_socket_action(s->multi, CURL_SOCKET_TIMEOUT, 0, &running);
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=128M
  offset=67M
 diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/005
 +++ b/tests/qemu-iotests/005
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto generic
  _supported_os Linux
  _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  # vpc is limited to 127GB, so we can't test it here
  if [ "$IMGFMT" = "vpc" ]; then
 diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/009
 +++ b/tests/qemu-iotests/009
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=6G
 diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/010
 +++ b/tests/qemu-iotests/010
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=6G
 diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/011
 +++ b/tests/qemu-iotests/011
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=6G
 diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/017
 +++ b/tests/qemu-iotests/017
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 vmdk qed
  _supported_proto generic
  _unsupported_proto vxhs
 -_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
 +_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
 diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/018
 +++ b/tests/qemu-iotests/018
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _supported_os Linux
 -_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
 +_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
 +                     "streamOptimized"
  TEST_OFFSETS="0 4294967296"
 diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/019
 +++ b/tests/qemu-iotests/019
@@ -XXX,XX +XXX,XX @@ _supported_proto file
  _supported_os Linux
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
  CLUSTER_SIZE=65536
 diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/020
 +++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  TEST_OFFSETS="0 4294967296"
 diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/027
 +++ b/tests/qemu-iotests/027
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt vmdk qcow qcow2 qed
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=128M
 diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/032
 +++ b/tests/qemu-iotests/032
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  # This works for any image format (though unlikely to segfault for raw)
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo === Prepare image ===
 diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/033
 +++ b/tests/qemu-iotests/033
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto generic
 +_unsupported_imgopts "subformat=streamOptimized"
  size=128M
 diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/034
 +++ b/tests/qemu-iotests/034
@@ -XXX,XX +XXX,XX @@ _supported_proto file
  _supported_os Linux
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  CLUSTER_SIZE=4k
  size=128M
 diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/037
 +++ b/tests/qemu-iotests/037
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  CLUSTER_SIZE=4k
  size=128M
 diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/063
 +++ b/tests/qemu-iotests/063
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
  _supported_proto file
  _unsupported_imgopts "subformat=monolithicFlat" \
                       "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  _make_test_img 4M
 diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/072
 +++ b/tests/qemu-iotests/072
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
  _supported_proto file
 +_unsupported_imgopts "subformat=streamOptimized"
  IMG_SIZE=64M
 diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/105
 +++ b/tests/qemu-iotests/105
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2 vmdk vhdx qed
  _supported_proto generic
  _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
 -                     "subformat=twoGbMaxExtentSparse"
 +                     "subformat=twoGbMaxExtentSparse" \
 +                     "subformat=streamOptimized"
  echo
  echo "creating large image"
 diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/197
 +++ b/tests/qemu-iotests/197
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto generic
  # LUKS support may be possible, but it complicates things.
  _unsupported_fmt luks
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo '=== Copy-on-read ==='
 diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/215
 +++ b/tests/qemu-iotests/215
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
  _supported_proto generic
  # LUKS support may be possible, but it complicates things.
  _unsupported_fmt luks
 +_unsupported_imgopts "subformat=streamOptimized"
  echo
  echo '=== Copy-on-read ==='
 diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/251
 +++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt generic
  _supported_proto file
  _supported_os Linux
 +_unsupported_imgopts "subformat=streamOptimized"
  if [ "$IMGOPTSSYNTAX" = "true" ]; then
      # We use json:{} filenames here, so we cannot work with additional options.
 --
 .21.0

-[Qemu-devel] [PULL 12/15] file-posix: fix request_alignment typo
+[Qemu-devel] [PULL 13/16] blockjob: update nodes head while removing all bdrv
-From: Stefan Hajnoczi <stefanha@redhat.com>
+From: Sergio Lopez <slp@redhat.com>
-Fixes: a6b257a08e3d72219f03e461a52152672fec0612
+block_job_remove_all_bdrv() iterates through job->nodes, calling
-       ("file-posix: Handle undetectable alignment")
+bdrv_root_unref_child() for each entry. The call to the latter may
-Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
+reach child_job_[can_]set_aio_ctx(), which will also attempt to
-Message-id: 20190827101328.4062-1-stefanha@redhat.com
+traverse job->nodes, potentially finding entries that where freed
-Reviewed-by: Thomas Huth <thuth@redhat.com>
+on previous iterations.
 To avoid this situation, update job->nodes head on each iteration to
 ensure that already freed entries are no longer linked to the list.
 RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1746631
 Signed-off-by: Sergio Lopez <slp@redhat.com>
 Cc: qemu-stable@nongnu.org
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 Message-id: 20190911100316.32282-1-mreitz@redhat.com
 Reviewed-by: Sergio Lopez <slp@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/file-posix.c | 2 +-
+ blockjob.c | 17 +++++++++++++----
-file changed, 1 insertion(+), 1 deletion(-)
+file changed, 13 insertions(+), 4 deletions(-)
-diff --git a/block/file-posix.c b/block/file-posix.c
+diff --git a/blockjob.c b/blockjob.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/file-posix.c
+--- a/blockjob.c
-+++ b/block/file-posix.c
++++ b/blockjob.c
-@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
+@@ -XXX,XX +XXX,XX @@ static const BdrvChildRole child_job = {
-         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
-             align = alignments[i];
+ void block_job_remove_all_bdrv(BlockJob *job)
-             if (raw_is_io_aligned(fd, buf + align, max_align)) {
+ {
--                /* Fallback to request_aligment. */
+-    GSList *l;
-+                /* Fallback to request_alignment. */
+-    for (l = job->nodes; l; l = l->next) {
-                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
++    /*
-                 break;
++     * bdrv_root_unref_child() may reach child_job_[can_]set_aio_ctx(),
-             }
++     * which will also traverse job->nodes, so consume the list one by
 +     * one to make sure that such a concurrent access does not attempt
 +     * to process an already freed BdrvChild.
 +     */
 +    while (job->nodes) {
 +        GSList *l = job->nodes;
          BdrvChild *c = l->data;
 +
 +        job->nodes = l->next;
 +
          bdrv_op_unblock_all(c->bs, job->blocker);
          bdrv_root_unref_child(c);
 +
 +        g_slist_free_1(l);
      }
 -    g_slist_free(job->nodes);
 -    job->nodes = NULL;
  }
  bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs)
 --
 .21.0

-[Qemu-devel] [PULL 05/15] iotests: Fix _filter_img_create()
+[Qemu-devel] [PULL 14/16] block/qcow2: Fix corruption introduced by commit 8ac0f15f335
-fe646693acc changed qemu-img create's output so that it no longer prints
+From: Maxim Levitsky <mlevitsk@redhat.com>
 single quotes around parameter values.  The subformat and adapter_type
 filters in _filter_img_create() have never been adapted to that change.
-Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
+This fixes subtle corruption introduced by luks threaded encryption
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+in commit 8ac0f15f335
-Reviewed-by: John Snow <jsnow@redhat.com>
-Message-id: 20190815153638.4600-2-mreitz@redhat.com
+Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922
-Reviewed-by: John Snow <jsnow@redhat.com>
 The corruption happens when we do a write that
    * writes to two or more unallocated clusters at once
    * doesn't fully cover the first sector
    * doesn't fully cover the last sector
    * uses luks encryption
 In this case, when allocating the new clusters we COW both areas
 prior to the write and after the write, and we encrypt them.
 The above mentioned commit accidentally made it so we encrypt the
 second COW area using the physical cluster offset of the first area.
 The problem is that offset_in_cluster in do_perform_cow_encrypt
 can be larger that the cluster size, thus cluster_offset
 will no longer point to the start of the cluster at which encrypted
 area starts.
 Next patch in this series will refactor the code to avoid all these
 assumptions.
 In the bugreport that was triggered by rebasing a luks image to new,
 zero filled base, which lot of such writes, and causes some files
 with zero areas to contain garbage there instead.
 But as described above it can happen elsewhere as well
 Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
 Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
 Message-id: 20190915203655.21638-2-mlevitsk@redhat.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/059.out       | 16 ++++++++--------
+ block/qcow2-cluster.c | 7 ++++---
- tests/qemu-iotests/common.filter |  4 ++--
+file changed, 4 insertions(+), 3 deletions(-)
 files changed, 10 insertions(+), 10 deletions(-)
-diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
+diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/059.out
+--- a/block/qcow2-cluster.c
-+++ b/tests/qemu-iotests/059.out
++++ b/block/qcow2-cluster.c
-@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
- qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
+         assert(QEMU_IS_ALIGNED(offset_in_cluster, BDRV_SECTOR_SIZE));
+         assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
- === Testing monolithicFlat creation and opening ===
+         assert(s->crypto);
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+-        if (qcow2_co_encrypt(bs, cluster_offset,
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
+-                             src_cluster_offset + offset_in_cluster,
- image: TEST_DIR/t.IMGFMT
+-                             buffer, bytes) < 0) {
- file format: IMGFMT
++        if (qcow2_co_encrypt(bs,
- virtual size: 2 GiB (2147483648 bytes)
++                start_of_cluster(s, cluster_offset + offset_in_cluster),
++                src_cluster_offset + offset_in_cluster,
- === Testing monolithicFlat with zeroed_grain ===
++                buffer, bytes) < 0) {
- qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
+             return false;
--Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+         }
-+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
+     }
  === Testing big twoGbMaxExtentFlat ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
  image: TEST_DIR/t.vmdk
  file format: vmdk
  virtual size: 0.977 TiB (1073741824000 bytes)
@@ -XXX,XX +XXX,XX @@ Format specific information:
  qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
  === Testing truncated sparse ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
  qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
  === Converting to streamOptimized from image with small cluster size===
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing monolithicFlat with internally generated JSON file name ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
  qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
  === Testing version 3 ===
@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing 4TB monolithicFlat creation and IO ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
  image: TEST_DIR/t.IMGFMT
  file format: IMGFMT
  virtual size: 4 TiB (4398046511104 bytes)
@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  === Testing qemu-img map on extents ===
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0x20000         0x3f0000        TEST_DIR/t.vmdk
 x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
 -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
  wrote 1024/1024 bytes at offset 65024
 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  wrote 1024/1024 bytes at offset 2147483136
 diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/common.filter
 +++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
          -e "s# compat6=\\(on\\|off\\)##g" \
          -e "s# static=\\(on\\|off\\)##g" \
          -e "s# zeroed_grain=\\(on\\|off\\)##g" \
 -        -e "s# subformat='[^']*'##g" \
 -        -e "s# adapter_type='[^']*'##g" \
 +        -e "s# subformat=[^ ]*##g" \
 +        -e "s# adapter_type=[^ ]*##g" \
          -e "s# hwversion=[^ ]*##g" \
          -e "s# lazy_refcounts=\\(on\\|off\\)##g" \
          -e "s# block_size=[0-9]\\+##g" \
 --
 .21.0

-[Qemu-devel] [PULL 08/15] vmdk: Reject invalid compressed writes
+[Qemu-devel] [PULL 15/16] block/qcow2: refactor encryption code
-Compressed writes generally have to write full clusters, not just in
+From: Maxim Levitsky <mlevitsk@redhat.com>
-theory but also in practice when it comes to vmdk's streamOptimized
-subformat.  It currently is just silently broken for writes with
+* Change the qcow2_co_{encrypt|decrypt} to just receive full host and
-non-zero in-cluster offsets:
+  guest offsets and use this function directly instead of calling
+  do_perform_cow_encrypt (which is removed by that patch).
-$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
-$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
+* Adjust qcow2_co_encdec to take full host and guest offsets as well.
-wrote 4096/4096 bytes at offset 4096
-KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
+* Document the qcow2_co_{encrypt|decrypt} arguments
-read failed: Invalid argument
+  to prevent the bug fixed in former commit from hopefully
+  happening again.
-(The technical reason is that vmdk_write_extent() just writes the
-incomplete compressed data actually to offset 4k.  When reading the
+Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
-data, vmdk_read_extent() looks at offset 0 and finds the compressed data
+Message-id: 20190915203655.21638-3-mlevitsk@redhat.com
-size to be 0, because that is what it reads from there.  This yields an
+Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-error.)
+[mreitz: Let perform_cow() return the error value returned by
+         qcow2_co_encrypt(), as proposed by Vladimir]
 For incomplete writes with zero in-cluster offsets, the error path when
 reading the rest of the cluster is a bit different, but the result is
 the same:
 $ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
 $ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
 wrote 4096/4096 bytes at offset 0
 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
 read failed: Invalid argument
 (Here, vmdk_read_extent() finds the data and then sees that the
 uncompressed data is short.)
 It is better to reject invalid writes than to make the user believe they
 might have succeeded and then fail when trying to read it back.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 Reviewed-by: John Snow <jsnow@redhat.com>
 Message-id: 20190815153638.4600-5-mreitz@redhat.com
 Reviewed-by: John Snow <jsnow@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/vmdk.c | 10 ++++++++++
+ block/qcow2.h         |  8 +++---
-file changed, 10 insertions(+)
+ block/qcow2-cluster.c | 41 +++++++++-------------------
+ block/qcow2-threads.c | 63 +++++++++++++++++++++++++++++++++----------
-diff --git a/block/vmdk.c b/block/vmdk.c
+ block/qcow2.c         |  5 ++--
-index XXXXXXX..XXXXXXX 100644
+files changed, 69 insertions(+), 48 deletions(-)
---- a/block/vmdk.c
-+++ b/block/vmdk.c
+diff --git a/block/qcow2.h b/block/qcow2.h
-@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
+index XXXXXXX..XXXXXXX 100644
-     if (extent->compressed) {
+--- a/block/qcow2.h
-         void *compressed_data;
++++ b/block/qcow2.h
+@@ -XXX,XX +XXX,XX @@ ssize_t coroutine_fn
-+        /* Only whole clusters */
+ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
-+        if (offset_in_cluster ||
+                     const void *src, size_t src_size);
-+            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
+ int coroutine_fn
-+            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
+-qcow2_co_encrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
-+             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
+-                 uint64_t offset, void *buf, size_t len);
-+        {
++qcow2_co_encrypt(BlockDriverState *bs, uint64_t host_offset,
-+            ret = -EINVAL;
++                 uint64_t guest_offset, void *buf, size_t len);
-+            goto out;
+ int coroutine_fn
 -qcow2_co_decrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
 -                 uint64_t offset, void *buf, size_t len);
 +qcow2_co_decrypt(BlockDriverState *bs, uint64_t host_offset,
 +                 uint64_t guest_offset, void *buf, size_t len);
  #endif
 diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/qcow2-cluster.c
 +++ b/block/qcow2-cluster.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn do_perform_cow_read(BlockDriverState *bs,
      return 0;
  }
 -static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
 -                                                uint64_t src_cluster_offset,
 -                                                uint64_t cluster_offset,
 -                                                unsigned offset_in_cluster,
 -                                                uint8_t *buffer,
 -                                                unsigned bytes)
 -{
 -    if (bytes && bs->encrypted) {
 -        BDRVQcow2State *s = bs->opaque;
 -        assert(QEMU_IS_ALIGNED(offset_in_cluster, BDRV_SECTOR_SIZE));
 -        assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
 -        assert(s->crypto);
 -        if (qcow2_co_encrypt(bs,
 -                start_of_cluster(s, cluster_offset + offset_in_cluster),
 -                src_cluster_offset + offset_in_cluster,
 -                buffer, bytes) < 0) {
 -            return false;
 -        }
 -    }
 -    return true;
 -}
 -
  static int coroutine_fn do_perform_cow_write(BlockDriverState *bs,
                                               uint64_t cluster_offset,
                                               unsigned offset_in_cluster,
@@ -XXX,XX +XXX,XX @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta *m)
      /* Encrypt the data if necessary before writing it */
      if (bs->encrypted) {
 -        if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
 -                                    start->offset, start_buffer,
 -                                    start->nb_bytes) ||
 -            !do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
 -                                    end->offset, end_buffer, end->nb_bytes)) {
 -            ret = -EIO;
 +        ret = qcow2_co_encrypt(bs,
 +                               m->alloc_offset + start->offset,
 +                               m->offset + start->offset,
 +                               start_buffer, start->nb_bytes);
 +        if (ret < 0) {
 +            goto fail;
 +        }
 +
-         if (!extent->has_marker) {
++        ret = qcow2_co_encrypt(bs,
-             ret = -EINVAL;
++                               m->alloc_offset + end->offset,
-             goto out;
++                               m->offset + end->offset,
 +                               end_buffer, end->nb_bytes);
 +        if (ret < 0) {
              goto fail;
          }
      }
 diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/qcow2-threads.c
 +++ b/block/qcow2-threads.c
@@ -XXX,XX +XXX,XX @@ static int qcow2_encdec_pool_func(void *opaque)
  }
  static int coroutine_fn
 -qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
 -                  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc func)
 +qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
 +                uint64_t guest_offset, void *buf, size_t len,
 +                Qcow2EncDecFunc func)
  {
      BDRVQcow2State *s = bs->opaque;
      Qcow2EncDecData arg = {
          .block = s->crypto,
 -        .offset = s->crypt_physical_offset ?
 -                      file_cluster_offset + offset_into_cluster(s, offset) :
 -                      offset,
 +        .offset = s->crypt_physical_offset ? host_offset : guest_offset,
          .buf = buf,
          .len = len,
          .func = func,
      };
 -    return qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
 +    assert(QEMU_IS_ALIGNED(guest_offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(host_offset, BDRV_SECTOR_SIZE));
 +    assert(QEMU_IS_ALIGNED(len, BDRV_SECTOR_SIZE));
 +    assert(s->crypto);
 +
 +    return len == 0 ? 0 : qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
  }
 +/*
 + * qcow2_co_encrypt()
 + *
 + * Encrypts one or more contiguous aligned sectors
 + *
 + * @host_offset - underlying storage offset of the first sector of the
 + * data to be encrypted
 + *
 + * @guest_offset - guest (virtual) offset of the first sector of the
 + * data to be encrypted
 + *
 + * @buf - buffer with the data to encrypt, that after encryption
 + *        will be written to the underlying storage device at
 + *        @host_offset
 + *
 + * @len - length of the buffer (must be a BDRV_SECTOR_SIZE multiple)
 + *
 + * Depending on the encryption method, @host_offset and/or @guest_offset
 + * may be used for generating the initialization vector for
 + * encryption.
 + *
 + * Note that while the whole range must be aligned on sectors, it
 + * does not have to be aligned on clusters and can also cross cluster
 + * boundaries
 + */
  int coroutine_fn
 -qcow2_co_encrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
 -                 uint64_t offset, void *buf, size_t len)
 +qcow2_co_encrypt(BlockDriverState *bs, uint64_t host_offset,
 +                 uint64_t guest_offset, void *buf, size_t len)
  {
 -    return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len,
 -                             qcrypto_block_encrypt);
 +    return qcow2_co_encdec(bs, host_offset, guest_offset, buf, len,
 +                           qcrypto_block_encrypt);
  }
 +/*
 + * qcow2_co_decrypt()
 + *
 + * Decrypts one or more contiguous aligned sectors
 + * Similar to qcow2_co_encrypt
 + */
  int coroutine_fn
 -qcow2_co_decrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
 -                 uint64_t offset, void *buf, size_t len)
 +qcow2_co_decrypt(BlockDriverState *bs, uint64_t host_offset,
 +                 uint64_t guest_offset, void *buf, size_t len)
  {
 -    return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len,
 -                             qcrypto_block_decrypt);
 +    return qcow2_co_encdec(bs, host_offset, guest_offset, buf, len,
 +                           qcrypto_block_decrypt);
  }
 diff --git a/block/qcow2.c b/block/qcow2.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/qcow2.c
 +++ b/block/qcow2.c
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv_part(BlockDriverState *bs,
                  assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
                  assert(QEMU_IS_ALIGNED(cur_bytes, BDRV_SECTOR_SIZE));
 -                if (qcow2_co_decrypt(bs, cluster_offset, offset,
 +                if (qcow2_co_decrypt(bs, cluster_offset + offset_in_cluster,
 +                                     offset,
                                       cluster_data, cur_bytes) < 0) {
                      ret = -EIO;
                      goto fail;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev_part(
              qemu_iovec_to_buf(qiov, qiov_offset + bytes_done,
                                cluster_data, cur_bytes);
 -            if (qcow2_co_encrypt(bs, cluster_offset, offset,
 +            if (qcow2_co_encrypt(bs, cluster_offset + offset_in_cluster, offset,
                                   cluster_data, cur_bytes) < 0) {
                  ret = -EIO;
                  goto out_unlocked;
 --
 .21.0

-[Qemu-devel] [PULL 03/15] block: posix: Always allocate the first block
+[Qemu-devel] [PULL 16/16] qemu-iotests: Add test for bz #1745922
-From: Nir Soffer <nirsof@gmail.com>
+From: Maxim Levitsky <mlevitsk@redhat.com>
-When creating an image with preallocation "off" or "falloc", the first
+Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
-block of the image is typically not allocated. When using Gluster
+Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-storage backed by XFS filesystem, reading this block using direct I/O
+Message-id: 20190915203655.21638-4-mlevitsk@redhat.com
 succeeds regardless of request length, fooling alignment detection.
 In this case we fallback to a safe value (4096) instead of the optimal
 value (512), which may lead to unneeded data copying when aligning
 requests.  Allocating the first block avoids the fallback.
 Since we allocate the first block even with preallocation=off, we no
 longer create images with zero disk size:
     $ ./qemu-img create -f raw test.raw 1g
     Formatting 'test.raw', fmt=raw size=1073741824
     $ ls -lhs test.raw
 .0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw
 And converting the image requires additional cluster:
     $ ./qemu-img measure -f raw -O qcow2 test.raw
     required size: 458752
     fully allocated size: 1074135040
 When using format like vmdk with multiple files per image, we allocate
 one block per file:
     $ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
     Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat
     $ ls -lhs test*.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
 .0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk
 I did quick performance test for copying disks with qemu-img convert to
 new raw target image to Gluster storage with sector size of 512 bytes:
     for i in $(seq 10); do
         rm -f dst.raw
         sleep 10
         time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
     done
 Here is a table comparing the total time spent:
 Type    Before(s)   After(s)    Diff(%)
 ---------------------------------------
 real      530.028    469.123      -11.4
 user       17.204     10.768      -37.4
 sys        17.881      7.011      -60.7
 We can see very clear improvement in CPU usage.
 Signed-off-by: Nir Soffer <nsoffer@redhat.com>
 Message-id: 20190827010528.8818-2-nsoffer@redhat.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/file-posix.c                            | 51 +++++++++++++++++++
+ tests/qemu-iotests/263     | 91 ++++++++++++++++++++++++++++++++++++++
- tests/qemu-iotests/059.out                    |  2 +-
+ tests/qemu-iotests/263.out | 40 +++++++++++++++++
- tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
+ tests/qemu-iotests/group   |  1 +
- tests/qemu-iotests/150.out.raw                | 12 +++++
+files changed, 132 insertions(+)
- tests/qemu-iotests/175                        | 19 ++++---
+ create mode 100755 tests/qemu-iotests/263
- tests/qemu-iotests/175.out                    |  8 +--
+ create mode 100644 tests/qemu-iotests/263.out
  tests/qemu-iotests/178.out.qcow2              |  4 +-
  tests/qemu-iotests/221.out                    | 12 +++--
  tests/qemu-iotests/253.out                    | 12 +++--
 files changed, 99 insertions(+), 21 deletions(-)
  rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
  create mode 100644 tests/qemu-iotests/150.out.raw
-diff --git a/block/file-posix.c b/block/file-posix.c
+diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
-index XXXXXXX..XXXXXXX 100644
+new file mode 100755
---- a/block/file-posix.c
+index XXXXXXX..XXXXXXX
-+++ b/block/file-posix.c
+--- /dev/null
-@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
++++ b/tests/qemu-iotests/263
-     return ret;
+@@ -XXX,XX +XXX,XX @@
- }
++#!/usr/bin/env bash
++#
-+/*
++# Test encrypted write that crosses cluster boundary of two unallocated clusters
-+ * Help alignment probing by allocating the first block.
++# Based on 188
-+ *
++#
-+ * When reading with direct I/O from unallocated area on Gluster backed by XFS,
++# Copyright (C) 2019 Red Hat, Inc.
-+ * reading succeeds regardless of request length. In this case we fallback to
++#
-+ * safe alignment which is not optimal. Allocating the first block avoids this
++# This program is free software; you can redistribute it and/or modify
-+ * fallback.
++# it under the terms of the GNU General Public License as published by
-+ *
++# the Free Software Foundation; either version 2 of the License, or
-+ * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
++# (at your option) any later version.
-+ * request alignment, so we use safe values.
++#
-+ *
++# This program is distributed in the hope that it will be useful,
-+ * Returns: 0 on success, -errno on failure. Since this is an optimization,
++# but WITHOUT ANY WARRANTY; without even the implied warranty of
-+ * caller may ignore failures.
++# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-+ */
++# GNU General Public License for more details.
-+static int allocate_first_block(int fd, size_t max_size)
++#
 +# You should have received a copy of the GNU General Public License
 +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 +#
 +
 +# creator
 +owner=mlevitsk@redhat.com
 +
 +seq=`basename $0`
 +echo "QA output created by $seq"
 +
 +status=1    # failure is the default!
 +
 +_cleanup()
 +{
-+    size_t write_size = (max_size < MAX_BLOCKSIZE)
++    _cleanup_test_img
-+        ? BDRV_SECTOR_SIZE
++}
-+        : MAX_BLOCKSIZE;
++trap "_cleanup; exit \$status" 0 1 2 3 15
 +    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
 +    void *buf;
 +    ssize_t n;
 +    int ret;
 +
-+    buf = qemu_memalign(max_align, write_size);
++# get standard environment, filters and checks
-+    memset(buf, 0, write_size);
++. ./common.rc
 +. ./common.filter
 +
-+    do {
++_supported_fmt qcow2
-+        n = pwrite(fd, buf, write_size, 0);
++_supported_proto generic
-+    } while (n == -1 && errno == EINTR);
++_supported_os Linux
 +
-+    ret = (n == -1) ? -errno : 0;
 +
-+    qemu_vfree(buf);
++size=1M
-+    return ret;
++
 +SECRET="secret,id=sec0,data=astrochicken"
 +QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT
 +
 +
 +_run_test()
 +{
 +    echo "== reading the whole image =="
 +    $QEMU_IO --object $SECRET -c "read -P 0 0 $size" --image-opts "$1" | _filter_qemu_io | _filter_testdir
 +
 +    echo
 +    echo "== write two 512 byte sectors on a cluster boundary =="
 +    $QEMU_IO --object $SECRET -c "write -P 0xAA 0xFE00 0x400" --image-opts "$1" | _filter_qemu_io | _filter_testdir
 +
 +    echo
 +    echo "== verify that the rest of the image is not changed =="
 +    $QEMU_IO --object $SECRET -c "read -P 0x00 0x00000 0xFE00" --image-opts "$1" | _filter_qemu_io | _filter_testdir
 +    $QEMU_IO --object $SECRET -c "read -P 0xAA 0x0FE00 0x400" --image-opts "$1" | _filter_qemu_io | _filter_testdir
 +    $QEMU_IO --object $SECRET -c "read -P 0x00 0x10200 0xEFE00" --image-opts "$1" | _filter_qemu_io | _filter_testdir
 +
 +}
 +
- static int handle_aiocb_truncate(void *opaque)
++
- {
++echo
-     RawPosixAIOData *aiocb = opaque;
++echo "testing LUKS qcow2 encryption"
-@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
++echo
-                 /* posix_fallocate() doesn't set errno. */
++
-                 error_setg_errno(errp, -result,
++_make_test_img --object $SECRET -o "encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10,cluster_size=64K" $size
-                                  "Could not preallocate new data");
++_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
-+            } else if (current_length == 0) {
++_cleanup_test_img
-+                /*
++
-+                 * posix_fallocate() uses fallocate() if the filesystem
++echo
-+                 * supports it, or fallback to manually writing zeroes. If
++echo "testing legacy AES qcow2 encryption"
-+                 * fallocate() was used, unaligned reads from the fallocated
++echo
-+                 * area in raw_probe_alignment() will succeed, hence we need to
++
-+                 * allocate the first block.
++
-+                 *
++_make_test_img --object $SECRET -o "encrypt.format=aes,encrypt.key-secret=sec0,cluster_size=64K" $size
-+                 * Optimize future alignment probing; ignore failures.
++_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
-+                 */
++_cleanup_test_img
-+                allocate_first_block(fd, offset);
++
-             }
++
-         } else {
++
-             result = 0;
++# success, all done
-@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
++echo "*** done"
-         if (ftruncate(fd, offset) != 0) {
++rm -f $seq.full
-             result = -errno;
++status=0
-             error_setg_errno(errp, -result, "Could not resize file");
+diff --git a/tests/qemu-iotests/263.out b/tests/qemu-iotests/263.out
 +        } else if (current_length == 0 && offset > current_length) {
 +            /* Optimize future alignment probing; ignore failures. */
 +            allocate_first_block(fd, offset);
          }
          return result;
      default:
 diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/059.out
 +++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
  image: TEST_DIR/t.vmdk
  file format: vmdk
  virtual size: 0.977 TiB (1073741824000 bytes)
 -disk size: 16 KiB
 +disk size: 1.97 MiB
  Format specific information:
      cid: XXXXXXXX
      parent cid: XXXXXXXX
 diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
 similarity index 100%
 rename from tests/qemu-iotests/150.out
 rename to tests/qemu-iotests/150.out.qcow2
 diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
-+++ b/tests/qemu-iotests/150.out.raw
++++ b/tests/qemu-iotests/263.out
 @@ -XXX,XX +XXX,XX @@
-+QA output created by 150
++QA output created by 263
 +
-+=== Mapping sparse conversion ===
++testing LUKS qcow2 encryption
 +
-+Offset          Length          File
++Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
-+0               0x1000          TEST_DIR/t.IMGFMT
++== reading the whole image ==
 +read 1048576/1048576 bytes at offset 0
 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
-+=== Mapping non-sparse conversion ===
++== write two 512 byte sectors on a cluster boundary ==
 +wrote 1024/1024 bytes at offset 65024
 +1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
-+Offset          Length          File
++== verify that the rest of the image is not changed ==
-+0               0x100000        TEST_DIR/t.IMGFMT
++read 65024/65024 bytes at offset 0
 +63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +read 1024/1024 bytes at offset 65024
 +1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +read 982528/982528 bytes at offset 66048
 +959.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
 +testing legacy AES qcow2 encryption
 +
 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=aes encrypt.key-secret=sec0
 +== reading the whole image ==
 +read 1048576/1048576 bytes at offset 0
 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
 +== write two 512 byte sectors on a cluster boundary ==
 +wrote 1024/1024 bytes at offset 65024
 +1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
 +== verify that the rest of the image is not changed ==
 +read 65024/65024 bytes at offset 0
 +63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +read 1024/1024 bytes at offset 65024
 +1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +read 982528/982528 bytes at offset 66048
 +959.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +*** done
-diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
+diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/175
 +++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  # the file size.  This function hides the resulting difference in the
  # stat -c '%b' output.
  # Parameter 1: Number of blocks an empty file occupies
 -# Parameter 2: Image size in bytes
 +# Parameter 2: Minimal number of blocks in an image
 +# Parameter 3: Image size in bytes
  _filter_blocks()
  {
      extra_blocks=$1
 -    img_size=$2
 +    min_blocks=$2
 +    img_size=$3
 -    sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
 -        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
 +    sed -e "s/blocks=$min_blocks\\(\$\\|[^0-9]\\)/min allocation/" \
 +        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/max allocation/"
  }
  # get standard environment, filters and checks
@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
  touch "$TEST_DIR/empty"
  extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 +# We always write the first byte; check how many blocks this filesystem
 +# allocates to match empty image alloation.
 +printf "\0" > "$TEST_DIR/empty"
 +min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 +
  echo
  echo "== creating image with default preallocation =="
  _make_test_img $size | _filter_imgfmt
 -stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 +stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  for mode in off full falloc; do
      echo
      echo "== creating image with preallocation $mode =="
      IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
 -    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 +    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
  done
  # success, all done
 diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/175.out
+--- a/tests/qemu-iotests/group
-+++ b/tests/qemu-iotests/175.out
++++ b/tests/qemu-iotests/group
-@@ -XXX,XX +XXX,XX @@ QA output created by 175
+@@ -XXX,XX +XXX,XX @@
+rw
- == creating image with default preallocation ==
+rw quick
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
+rw quick migration
--size=1048576, nothing allocated
++263 rw quick
-+size=1048576, min allocation
+rw auto quick
+rw quick
  == creating image with preallocation off ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
 -size=1048576, nothing allocated
 +size=1048576, min allocation
  == creating image with preallocation full ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
 -size=1048576, everything allocated
 +size=1048576, max allocation
  == creating image with preallocation falloc ==
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
 -size=1048576, everything allocated
 +size=1048576, max allocation
   *** done
 diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/178.out.qcow2
 +++ b/tests/qemu-iotests/178.out.qcow2
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
  == raw input image with data (human) ==
  Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
 -required size: 393216
 +required size: 458752
  fully allocated size: 1074135040
  wrote 512/512 bytes at offset 512
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
  Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
  {
 -    "required": 393216,
 +    "required": 458752,
      "fully-allocated": 1074135040
  }
  wrote 512/512 bytes at offset 512
 diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/221.out
 +++ b/tests/qemu-iotests/221.out
@@ -XXX,XX +XXX,XX @@ QA output created by 221
  === Check mapping of unaligned raw image ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  wrote 1/1 bytes at offset 65536
 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
  { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
  { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  *** done
 diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/253.out
 +++ b/tests/qemu-iotests/253.out
@@ -XXX,XX +XXX,XX @@ QA output created by 253
  === Check mapping of unaligned raw image ===
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 -[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
  wrote 65535/65535 bytes at offset 983040
 .999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 -[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 +[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 +{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
  { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
  *** done
 --
 .21.0

The following changes since commit 23919ddfd56135cad3cb468a8f54d5a595f024f4:

Merge remote-tracking branch 'remotes/aperard/tags/pull-xen-20190827' into staging (2019-08-27 15:52:36 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-08-27

for you to fetch changes up to bb043c056cffcc2f3ce88bfdaf2e76e455c09e2c:

iotests: Unify cache mode quoting (2019-08-27 19:48:44 +0200)

----------------------------------------------------------------
Block patches:
- qemu-io now accepts a file to read a write pattern from
- Ensure that raw files have their first block allocated so we can probe
  the O_DIRECT alignment if necessary
- Various fixes

----------------------------------------------------------------
Denis Plotnikov (1):
  qemu-io: add pattern file for write command

Max Reitz (7):
  iotests: Fix _filter_img_create()
  vmdk: Use bdrv_dirname() for relative extent paths
  iotests: Keep testing broken relative extent paths
  vmdk: Reject invalid compressed writes
  iotests: Disable broken streamOptimized tests
  iotests: Disable 110 for vmdk.twoGbMaxExtentSparse
  iotests: Disable 126 for flat vmdk subformats

Nir Soffer (3):
  block: posix: Always allocate the first block
  iotests: Test allocate_first_block() with O_DIRECT
  iotests: Unify cache mode quoting

Stefan Hajnoczi (1):
  file-posix: fix request_alignment typo

Thomas Huth (2):
  iotests: Check for enabled drivers before testing them
  tests/check-block: Skip iotests when sanitizers are enabled

Vladimir Sementsov-Ogievskiy (1):
  block: fix permission update in bdrv_replace_node

block.c                                       |  5 +-
 block/file-posix.c                            | 53 +++++++++-
 block/vmdk.c                                  | 64 ++++++++----
 qemu-io-cmds.c                                | 99 +++++++++++++++++--
 tests/check-block.sh                          |  5 +
 tests/qemu-iotests/002                        |  1 +
 tests/qemu-iotests/003                        |  1 +
 tests/qemu-iotests/005                        |  3 +-
 tests/qemu-iotests/009                        |  1 +
 tests/qemu-iotests/010                        |  1 +
 tests/qemu-iotests/011                        |  1 +
 tests/qemu-iotests/017                        |  3 +-
 tests/qemu-iotests/018                        |  3 +-
 tests/qemu-iotests/019                        |  3 +-
 tests/qemu-iotests/020                        |  3 +-
 tests/qemu-iotests/026                        |  4 +-
 tests/qemu-iotests/027                        |  1 +
 tests/qemu-iotests/032                        |  1 +
 tests/qemu-iotests/033                        |  1 +
 tests/qemu-iotests/034                        |  3 +-
 tests/qemu-iotests/037                        |  3 +-
 tests/qemu-iotests/039                        |  4 +-
 tests/qemu-iotests/052                        |  2 +-
 tests/qemu-iotests/059                        | 34 ++++++-
 tests/qemu-iotests/059.out                    | 26 +++--
 tests/qemu-iotests/063                        |  3 +-
 tests/qemu-iotests/071                        |  1 +
 tests/qemu-iotests/072                        |  1 +
 tests/qemu-iotests/081                        |  4 +-
 tests/qemu-iotests/091                        |  4 +-
 tests/qemu-iotests/099                        |  1 +
 tests/qemu-iotests/105                        |  3 +-
 tests/qemu-iotests/110                        |  3 +-
 tests/qemu-iotests/120                        |  1 +
 tests/qemu-iotests/126                        |  2 +
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++
 tests/qemu-iotests/162                        |  4 +-
 tests/qemu-iotests/175                        | 47 +++++++--
 tests/qemu-iotests/175.out                    | 16 ++-
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/184                        |  1 +
 tests/qemu-iotests/186                        |  1 +
 tests/qemu-iotests/197                        |  1 +
 tests/qemu-iotests/215                        |  1 +
 tests/qemu-iotests/221.out                    | 12 ++-
 tests/qemu-iotests/251                        |  1 +
 tests/qemu-iotests/253.out                    | 12 ++-
 tests/qemu-iotests/common.filter              |  4 +-
 tests/qemu-iotests/common.rc                  | 14 +++
 50 files changed, 391 insertions(+), 87 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

-- 
2.21.0

From: Denis Plotnikov <dplotnikov@virtuozzo.com>

The patch allows to provide a pattern file for write
command. There was no similar ability before.

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Message-id: 20190820164616.4072-1-dplotnikov@virtuozzo.com
Reviewed-by: Eric Blake <eblake@redhat.com>
[mreitz: Keep optstring in alphabetical order]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-io-cmds.c | 99 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 6 deletions(-)

diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -XXX,XX +XXX,XX @@ static void qemu_io_free(void *p)
     qemu_vfree(p);
 }
 
+/*
+ * qemu_io_alloc_from_file()
+ *
+ * Allocates the buffer and populates it with the content of the given file
+ * up to @len bytes. If the file length is less than @len, then the buffer
+ * is populated with the file content cyclically.
+ *
+ * @blk - the block backend where the buffer content is going to be written to
+ * @len - the buffer length
+ * @file_name - the file to read the content from
+ *
+ * Returns: the buffer pointer on success
+ *          NULL on error
+ */
+static void *qemu_io_alloc_from_file(BlockBackend *blk, size_t len,
+                                     const char *file_name)
+{
+    char *buf, *buf_origin;
+    FILE *f = fopen(file_name, "r");
+    int pattern_len;
+
+    if (!f) {
+        perror(file_name);
+        return NULL;
+    }
+
+    if (qemuio_misalign) {
+        len += MISALIGN_OFFSET;
+    }
+
+    buf_origin = buf = blk_blockalign(blk, len);
+
+    if (qemuio_misalign) {
+        buf_origin += MISALIGN_OFFSET;
+        buf += MISALIGN_OFFSET;
+        len -= MISALIGN_OFFSET;
+    }
+
+    pattern_len = fread(buf_origin, 1, len, f);
+
+    if (ferror(f)) {
+        perror(file_name);
+        goto error;
+    }
+
+    if (pattern_len == 0) {
+        fprintf(stderr, "%s: file is empty\n", file_name);
+        goto error;
+    }
+
+    fclose(f);
+
+    if (len > pattern_len) {
+        len -= pattern_len;
+        buf += pattern_len;
+
+        while (len > 0) {
+            size_t len_to_copy = MIN(pattern_len, len);
+
+            memcpy(buf, buf_origin, len_to_copy);
+
+            len -= len_to_copy;
+            buf += len_to_copy;
+        }
+    }
+
+    return buf_origin;
+
+error:
+    qemu_io_free(buf_origin);
+    return NULL;
+}
+
 static void dump_buffer(const void *buffer, int64_t offset, int64_t len)
 {
     uint64_t i;
@@ -XXX,XX +XXX,XX @@ static void write_help(void)
 " -n, -- with -z, don't allow slow fallback\n"
 " -p, -- ignored for backwards compatibility\n"
 " -P, -- use different pattern to fill file\n"
+" -s, -- use a pattern file to fill the write buffer\n"
 " -C, -- report statistics in a machine parsable format\n"
 " -q, -- quiet mode, do not show I/O statistics\n"
 " -u, -- with -z, allow unmapping\n"
@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t write_cmd = {
     .perm       = BLK_PERM_WRITE,
     .argmin     = 2,
     .argmax     = -1,
-    .args       = "[-bcCfnquz] [-P pattern] off len",
+    .args       = "[-bcCfnquz] [-P pattern | -s source_file] off len",
     .oneline    = "writes a number of bytes at a specified offset",
     .help       = write_help,
 };
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
 {
     struct timespec t1, t2;
     bool Cflag = false, qflag = false, bflag = false;
-    bool Pflag = false, zflag = false, cflag = false;
+    bool Pflag = false, zflag = false, cflag = false, sflag = false;
     int flags = 0;
     int c, cnt, ret;
     char *buf = NULL;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     /* Some compilers get confused and warn if this is not initialized.  */
     int64_t total = 0;
     int pattern = 0xcd;
+    const char *file_name = NULL;
 
-    while ((c = getopt(argc, argv, "bcCfnpP:quz")) != -1) {
+    while ((c = getopt(argc, argv, "bcCfnpP:qs:uz")) != -1) {
         switch (c) {
         case 'b':
             bflag = true;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         case 'q':
             qflag = true;
             break;
+        case 's':
+            sflag = true;
+            file_name = optarg;
+            break;
         case 'u':
             flags |= BDRV_REQ_MAY_UNMAP;
             break;
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
         return -EINVAL;
     }
 
-    if (zflag && Pflag) {
-        printf("-z and -P cannot be specified at the same time\n");
+    if (zflag + Pflag + sflag > 1) {
+        printf("Only one of -z, -P, and -s "
+               "can be specified at the same time\n");
         return -EINVAL;
     }
 
@@ -XXX,XX +XXX,XX @@ static int write_f(BlockBackend *blk, int argc, char **argv)
     }
 
     if (!zflag) {
-        buf = qemu_io_alloc(blk, count, pattern);
+        if (sflag) {
+            buf = qemu_io_alloc_from_file(blk, count, file_name);
+            if (!buf) {
+                return -EINVAL;
+            }
+        } else {
+            buf = qemu_io_alloc(blk, count, pattern);
+        }
     }
 
     clock_gettime(CLOCK_MONOTONIC, &t1);
-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

It's wrong to OR shared permissions. It may lead to crash on further
permission updates.
Also, no needs to consider previously calculated permissions, as at
this point we already bind all new parents and bdrv_get_cumulative_perm
result is enough. So fix the bug by just set permissions by
bdrv_get_cumulative_perm result.

Bug was introduced in long ago 234ac1a9025, in 2.9.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190824100740.61635-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
 {
     BdrvChild *c, *next;
     GSList *list = NULL, *p;
-    uint64_t old_perm, old_shared;
     uint64_t perm = 0, shared = BLK_PERM_ALL;
     int ret;
 
@@ -XXX,XX +XXX,XX @@ void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to,
         bdrv_unref(from);
     }
 
-    bdrv_get_cumulative_perm(to, &old_perm, &old_shared);
-    bdrv_set_perm(to, old_perm | perm, old_shared | shared);
+    bdrv_get_cumulative_perm(to, &perm, &shared);
+    bdrv_set_perm(to, perm, shared);
 
 out:
     g_slist_free(list);
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

When creating an image with preallocation "off" or "falloc", the first
block of the image is typically not allocated. When using Gluster
storage backed by XFS filesystem, reading this block using direct I/O
succeeds regardless of request length, fooling alignment detection.

In this case we fallback to a safe value (4096) instead of the optimal
value (512), which may lead to unneeded data copying when aligning
requests.  Allocating the first block avoids the fallback.

Since we allocate the first block even with preallocation=off, we no
longer create images with zero disk size:

$ ./qemu-img create -f raw test.raw 1g
    Formatting 'test.raw', fmt=raw size=1073741824

$ ls -lhs test.raw
    4.0K -rw-r--r--. 1 nsoffer nsoffer 1.0G Aug 16 23:48 test.raw

And converting the image requires additional cluster:

$ ./qemu-img measure -f raw -O qcow2 test.raw
    required size: 458752
    fully allocated size: 1074135040

When using format like vmdk with multiple files per image, we allocate
one block per file:

$ ./qemu-img create -f vmdk -o subformat=twoGbMaxExtentFlat test.vmdk 4g
    Formatting 'test.vmdk', fmt=vmdk size=4294967296 compat6=off hwversion=undefined subformat=twoGbMaxExtentFlat

$ ls -lhs test*.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f001.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer 2.0G Aug 27 03:23 test-f002.vmdk
    4.0K -rw-r--r--. 1 nsoffer nsoffer  353 Aug 27 03:23 test.vmdk

I did quick performance test for copying disks with qemu-img convert to
new raw target image to Gluster storage with sector size of 512 bytes:

for i in $(seq 10); do
        rm -f dst.raw
        sleep 10
        time ./qemu-img convert -f raw -O raw -t none -T none src.raw dst.raw
    done

Here is a table comparing the total time spent:

Type    Before(s)   After(s)    Diff(%)
---------------------------------------
real      530.028    469.123      -11.4
user       17.204     10.768      -37.4
sys        17.881      7.011      -60.7

We can see very clear improvement in CPU usage.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190827010528.8818-2-nsoffer@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c                            | 51 +++++++++++++++++++
 tests/qemu-iotests/059.out                    |  2 +-
 tests/qemu-iotests/{150.out => 150.out.qcow2} |  0
 tests/qemu-iotests/150.out.raw                | 12 +++++
 tests/qemu-iotests/175                        | 19 ++++---
 tests/qemu-iotests/175.out                    |  8 +--
 tests/qemu-iotests/178.out.qcow2              |  4 +-
 tests/qemu-iotests/221.out                    | 12 +++--
 tests/qemu-iotests/253.out                    | 12 +++--
 9 files changed, 99 insertions(+), 21 deletions(-)
 rename tests/qemu-iotests/{150.out => 150.out.qcow2} (100%)
 create mode 100644 tests/qemu-iotests/150.out.raw

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_discard(void *opaque)
     return ret;
 }
 
+/*
+ * Help alignment probing by allocating the first block.
+ *
+ * When reading with direct I/O from unallocated area on Gluster backed by XFS,
+ * reading succeeds regardless of request length. In this case we fallback to
+ * safe alignment which is not optimal. Allocating the first block avoids this
+ * fallback.
+ *
+ * fd may be opened with O_DIRECT, but we don't know the buffer alignment or
+ * request alignment, so we use safe values.
+ *
+ * Returns: 0 on success, -errno on failure. Since this is an optimization,
+ * caller may ignore failures.
+ */
+static int allocate_first_block(int fd, size_t max_size)
+{
+    size_t write_size = (max_size < MAX_BLOCKSIZE)
+        ? BDRV_SECTOR_SIZE
+        : MAX_BLOCKSIZE;
+    size_t max_align = MAX(MAX_BLOCKSIZE, getpagesize());
+    void *buf;
+    ssize_t n;
+    int ret;
+
+    buf = qemu_memalign(max_align, write_size);
+    memset(buf, 0, write_size);
+
+    do {
+        n = pwrite(fd, buf, write_size, 0);
+    } while (n == -1 && errno == EINTR);
+
+    ret = (n == -1) ? -errno : 0;
+
+    qemu_vfree(buf);
+    return ret;
+}
+
 static int handle_aiocb_truncate(void *opaque)
 {
     RawPosixAIOData *aiocb = opaque;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
                 /* posix_fallocate() doesn't set errno. */
                 error_setg_errno(errp, -result,
                                  "Could not preallocate new data");
+            } else if (current_length == 0) {
+                /*
+                 * posix_fallocate() uses fallocate() if the filesystem
+                 * supports it, or fallback to manually writing zeroes. If
+                 * fallocate() was used, unaligned reads from the fallocated
+                 * area in raw_probe_alignment() will succeed, hence we need to
+                 * allocate the first block.
+                 *
+                 * Optimize future alignment probing; ignore failures.
+                 */
+                allocate_first_block(fd, offset);
             }
         } else {
             result = 0;
@@ -XXX,XX +XXX,XX @@ static int handle_aiocb_truncate(void *opaque)
         if (ftruncate(fd, offset) != 0) {
             result = -errno;
             error_setg_errno(errp, -result, "Could not resize file");
+        } else if (current_length == 0 && offset > current_length) {
+            /* Optimize future alignment probing; ignore failures. */
+            allocate_first_block(fd, offset);
         }
         return result;
     default:
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMax
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
-disk size: 16 KiB
+disk size: 1.97 MiB
 Format specific information:
     cid: XXXXXXXX
     parent cid: XXXXXXXX
diff --git a/tests/qemu-iotests/150.out b/tests/qemu-iotests/150.out.qcow2
similarity index 100%
rename from tests/qemu-iotests/150.out
rename to tests/qemu-iotests/150.out.qcow2
diff --git a/tests/qemu-iotests/150.out.raw b/tests/qemu-iotests/150.out.raw
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/150.out.raw
@@ -XXX,XX +XXX,XX @@
+QA output created by 150
+
+=== Mapping sparse conversion ===
+
+Offset          Length          File
+0               0x1000          TEST_DIR/t.IMGFMT
+
+=== Mapping non-sparse conversion ===
+
+Offset          Length          File
+0               0x100000        TEST_DIR/t.IMGFMT
+*** done
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # the file size.  This function hides the resulting difference in the
 # stat -c '%b' output.
 # Parameter 1: Number of blocks an empty file occupies
-# Parameter 2: Image size in bytes
+# Parameter 2: Minimal number of blocks in an image
+# Parameter 3: Image size in bytes
 _filter_blocks()
 {
     extra_blocks=$1
-    img_size=$2
+    min_blocks=$2
+    img_size=$3
 
-    sed -e "s/blocks=$extra_blocks\$\$\\|[^0-9]\$/nothing allocated/" \
-        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/everything allocated/"
+    sed -e "s/blocks=$min_blocks\$\$\\|[^0-9]\$/min allocation/" \
+        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
 # get standard environment, filters and checks
@@ -XXX,XX +XXX,XX @@ size=$((1 * 1024 * 1024))
 touch "$TEST_DIR/empty"
 extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 
+# We always write the first byte; check how many blocks this filesystem
+# allocates to match empty image alloation.
+printf "\0" > "$TEST_DIR/empty"
+min_blocks=$(stat -c '%b' "$TEST_DIR/empty")
+
 echo
 echo "== creating image with default preallocation =="
 _make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 
 for mode in off full falloc; do
     echo
     echo "== creating image with preallocation $mode =="
     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
-    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
 # success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
 
 == creating image with default preallocation ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation off ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, nothing allocated
+size=1048576, min allocation
 
 == creating image with preallocation full ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, everything allocated
+size=1048576, max allocation
 
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, everything allocated
+size=1048576, max allocation
  *** done
diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/178.out.qcow2
+++ b/tests/qemu-iotests/178.out.qcow2
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 == raw input image with data (human) ==
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
-required size: 393216
+required size: 458752
 fully allocated size: 1074135040
 wrote 512/512 bytes at offset 512
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
@@ -XXX,XX +XXX,XX @@ converted image file size in bytes: 196608
 
 Formatting 'TEST_DIR/t.qcow2', fmt=IMGFMT size=1073741824
 {
-    "required": 393216,
+    "required": 458752,
     "fully-allocated": 1074135040
 }
 wrote 512/512 bytes at offset 512
diff --git a/tests/qemu-iotests/221.out b/tests/qemu-iotests/221.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/221.out
+++ b/tests/qemu-iotests/221.out
@@ -XXX,XX +XXX,XX @@ QA output created by 221
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65537
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 66048, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61952, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 1/1 bytes at offset 65536
 1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 61440, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 65536, "length": 1, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 65537, "length": 511, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 *** done
diff --git a/tests/qemu-iotests/253.out b/tests/qemu-iotests/253.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/253.out
+++ b/tests/qemu-iotests/253.out
@@ -XXX,XX +XXX,XX @@ QA output created by 253
 === Check mapping of unaligned raw image ===
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048575
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
-[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 1044480, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
 wrote 65535/65535 bytes at offset 983040
 63.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
-[{ "start": 0, "length": 983040, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
+[{ "start": 0, "length": 4096, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 4096, "length": 978944, "depth": 0, "zero": true, "data": false, "offset": OFFSET},
 { "start": 983040, "length": 65536, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Using block_resize we can test allocate_first_block() with file
descriptor opened with O_DIRECT, ensuring that it works for any size
larger than 4096 bytes.

Testing smaller sizes is tricky as the result depends on the filesystem
used for testing. For example on NFS any size will work since O_DIRECT
does not require any alignment.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190827010528.8818-3-nsoffer@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/175     | 28 ++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  8 ++++++++
 2 files changed, 36 insertions(+)

diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ _filter_blocks()
         -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/max allocation/"
 }
 
+# Resize image using block_resize.
+# Parameter 1: image path
+# Parameter 2: new size
+_block_resize()
+{
+    local path=$1
+    local size=$2
+
+    $QEMU -qmp stdio -nographic -nodefaults \
+        -blockdev file,node-name=file,filename=$path,cache.direct=on \
+        <<EOF
+{'execute': 'qmp_capabilities'}
+{'execute': 'block_resize', 'arguments': {'node-name': 'file', 'size': $size}}
+{'execute': 'quit'}
+EOF
+}
+
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
 _supported_proto file
 _supported_os Linux
 
+_default_cache_mode none
+_supported_cache_modes none directsync
+
 size=$((1 * 1024 * 1024))
 
 touch "$TEST_DIR/empty"
@@ -XXX,XX +XXX,XX @@ for mode in off full falloc; do
     stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $size
 done
 
+for new_size in 4096 1048576; do
+    echo
+    echo "== resize empty image with block_resize =="
+    _make_test_img 0 | _filter_imgfmt
+    _block_resize $TEST_IMG $new_size >/dev/null
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $min_blocks $new_size
+done
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ size=1048576, max allocation
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
 size=1048576, max allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=4096, min allocation
+
+== resize empty image with block_resize ==
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
+size=1048576, min allocation
  *** done
-- 
2.21.0

fe646693acc changed qemu-img create's output so that it no longer prints
single quotes around parameter values.  The subformat and adapter_type
filters in _filter_img_create() have never been adapted to that change.

Fixes: fe646693acc13ac48b98435d14149ab04dc597bc
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059.out       | 16 ++++++++--------
 tests/qemu-iotests/common.filter |  4 ++--
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open device TEST_DIR/t.vmdk: L1 size too big
 
 === Testing monolithicFlat creation and opening ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 2 GiB (2147483648 bytes)
 
 === Testing monolithicFlat with zeroed_grain ===
 qemu-img: TEST_DIR/t.IMGFMT: Flat image can't enable zeroed grain
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2147483648
 
 === Testing big twoGbMaxExtentFlat ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000 subformat=twoGbMaxExtentFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824000
 image: TEST_DIR/t.vmdk
 file format: vmdk
 virtual size: 0.977 TiB (1073741824000 bytes)
@@ -XXX,XX +XXX,XX @@ Format specific information:
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Invalid extent line: RW 12582912 VMFS "dummy.IMGFMT" 1
 
 === Testing truncated sparse ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=107374182400
 qemu-img: Could not open 'TEST_DIR/t.IMGFMT': File truncated, expecting at least 13172736 bytes
 
 === Converting to streamOptimized from image with small cluster size===
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
 
 === Testing version 3 ===
@@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 64931328
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing 4TB monolithicFlat creation and IO ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104 subformat=monolithicFlat
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4398046511104
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 4 TiB (4398046511104 bytes)
@@ -XXX,XX +XXX,XX @@ read 1024/1024 bytes at offset 966367641600
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing qemu-img map on extents ===
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=monolithicSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0               0x20000         0x3f0000        TEST_DIR/t.vmdk
 0x7fff0000      0x20000         0x410000        TEST_DIR/t.vmdk
 0x140000000     0x10000         0x430000        TEST_DIR/t.vmdk
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33285996544
 wrote 1024/1024 bytes at offset 65024
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 1024/1024 bytes at offset 2147483136
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -XXX,XX +XXX,XX @@ _filter_img_create()
         -e "s# compat6=\$on\\|off\$##g" \
         -e "s# static=\$on\\|off\$##g" \
         -e "s# zeroed_grain=\$on\\|off\$##g" \
-        -e "s# subformat='[^']*'##g" \
-        -e "s# adapter_type='[^']*'##g" \
+        -e "s# subformat=[^ ]*##g" \
+        -e "s# adapter_type=[^ ]*##g" \
         -e "s# hwversion=[^ ]*##g" \
         -e "s# lazy_refcounts=\$on\\|off\$##g" \
         -e "s# block_size=[0-9]\\+##g" \
-- 
2.21.0

This makes iotest 033 pass with e.g. subformat=monolithicFlat.  It also
turns a former error in 059 into success.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190815153638.4600-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c               | 54 ++++++++++++++++++++++++--------------
 tests/qemu-iotests/059     |  7 +++--
 tests/qemu-iotests/059.out |  4 ++-
 3 files changed, 42 insertions(+), 23 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static const char *next_line(const char *s)
 }
 
 static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-                              const char *desc_file_path, QDict *options,
-                              Error **errp)
+                              QDict *options, Error **errp)
 {
     int ret;
     int matches;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
     const char *p, *np;
     int64_t sectors = 0;
     int64_t flat_offset;
+    char *desc_file_dir = NULL;
     char *extent_path;
     BdrvChild *extent_file;
     BDRVVmdkState *s = bs->opaque;
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             continue;
         }
 
-        if (!path_is_absolute(fname) && !path_has_protocol(fname) &&
-            !desc_file_path[0])
-        {
-            bdrv_refresh_filename(bs->file->bs);
-            error_setg(errp, "Cannot use relative extent paths with VMDK "
-                       "descriptor file '%s'", bs->file->bs->filename);
-            return -EINVAL;
-        }
+        if (path_is_absolute(fname)) {
+            extent_path = g_strdup(fname);
+        } else {
+            if (!desc_file_dir) {
+                desc_file_dir = bdrv_dirname(bs->file->bs, errp);
+                if (!desc_file_dir) {
+                    bdrv_refresh_filename(bs->file->bs);
+                    error_prepend(errp, "Cannot use relative paths with VMDK "
+                                  "descriptor file '%s': ",
+                                  bs->file->bs->filename);
+                    ret = -EINVAL;
+                    goto out;
+                }
+            }
 
-        extent_path = path_combine(desc_file_path, fname);
+            extent_path = g_strconcat(desc_file_dir, fname, NULL);
+        }
 
         ret = snprintf(extent_opt_prefix, 32, "extents.%d", s->num_extents);
         assert(ret < 32);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
         g_free(extent_path);
         if (local_err) {
             error_propagate(errp, local_err);
-            return -EINVAL;
+            ret = -EINVAL;
+            goto out;
         }
 
         /* save to extents array */
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                             0, 0, 0, 0, 0, &extent, errp);
             if (ret < 0) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent->flat_start_offset = flat_offset << 9;
         } else if (!strcmp(type, "SPARSE") || !strcmp(type, "VMFSSPARSE")) {
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
             g_free(buf);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else if (!strcmp(type, "SESPARSE")) {
             ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
             if (ret) {
                 bdrv_unref_child(bs, extent_file);
-                return ret;
+                goto out;
             }
             extent = &s->extents[s->num_extents - 1];
         } else {
             error_setg(errp, "Unsupported extent type '%s'", type);
             bdrv_unref_child(bs, extent_file);
-            return -ENOTSUP;
+            ret = -ENOTSUP;
+            goto out;
         }
         extent->type = g_strdup(type);
     }
-    return 0;
+
+    ret = 0;
+    goto out;
 
 invalid:
     np = next_line(p);
@@ -XXX,XX +XXX,XX @@ invalid:
         np--;
     }
     error_setg(errp, "Invalid extent line: %.*s", (int)(np - p), p);
-    return -EINVAL;
+    ret = -EINVAL;
+
+out:
+    g_free(desc_file_dir);
+    return ret;
 }
 
 static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
     }
     s->create_type = g_strdup(ct);
     s->desc_offset = 0;
-    ret = vmdk_parse_extents(buf, bs, bs->file->bs->exact_filename, options,
-                             errp);
+    ret = vmdk_parse_extents(buf, bs, options, errp);
 exit:
     return ret;
 }
diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+# Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
-$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" 2>&1 \
-    | _filter_testdir | _filter_imgfmt
+$QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
+         -c info \
+    2>&1 \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
 echo
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 
 === Testing monolithicFlat with internally generated JSON file name ===
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
-qemu-io: can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
+format name: IMGFMT
+cluster size: 0 bytes
+vm state offset: 0 bytes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

We had a test for a case where relative extent paths did not work, but
unfortunately we just fixed the underlying problem, so it works now.
This patch adds a new test case that still fails.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-4-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/059     | 27 +++++++++++++++++++++++++++
 tests/qemu-iotests/059.out |  4 ++++
 2 files changed, 31 insertions(+)

diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -XXX,XX +XXX,XX @@ $QEMU_IMG convert -f qcow2 -O vmdk -o subformat=streamOptimized "$TEST_IMG.qcow2
 
 echo
 echo "=== Testing monolithicFlat with internally generated JSON file name ==="
+
+echo '--- blkdebug ---'
 # Should work, because bdrv_dirname() works fine with blkdebug
 IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
 $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TEST_IMG,file.inject-error.0.event=read_aio" \
@@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o driver=$IMGFMT,file.driver=blkdebug,file.image.filename=$TE
     | _filter_testdir | _filter_imgfmt | _filter_img_info
 _cleanup_test_img
 
+echo '--- quorum ---'
+# Should not work, because bdrv_dirname() does not work with quorum
+IMGOPTS="subformat=monolithicFlat" _make_test_img 64M
+cp "$TEST_IMG" "$TEST_IMG.orig"
+
+filename="json:{
+    \"driver\": \"$IMGFMT\",
+    \"file\": {
+        \"driver\": \"quorum\",
+        \"children\": [ {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG\"
+        }, {
+            \"driver\": \"file\",
+            \"filename\": \"$TEST_IMG.orig\"
+        } ],
+        \"vote-threshold\": 1
+    } }"
+
+filename=$(echo "$filename" | tr '\n' ' ' | sed -e 's/\s\+/ /g')
+$QEMU_IMG info "$filename" 2>&1 \
+    | sed -e "s/'json:[^']*'/\$QUORUM_FILE/g" \
+    | _filter_testdir | _filter_imgfmt | _filter_img_info
+
+
 echo
 echo "=== Testing version 3 ==="
 _use_sample_img iotest-version3.vmdk.bz2
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 10240
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 === Testing monolithicFlat with internally generated JSON file name ===
+--- blkdebug ---
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 format name: IMGFMT
 cluster size: 0 bytes
 vm state offset: 0 bytes
+--- quorum ---
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+qemu-img: Could not open $QUORUM_FILE: Cannot use relative paths with VMDK descriptor file $QUORUM_FILE: Cannot generate a base directory for quorum nodes
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
-- 
2.21.0

Compressed writes generally have to write full clusters, not just in
theory but also in practice when it comes to vmdk's streamOptimized
subformat.  It currently is just silently broken for writes with
non-zero in-cluster offsets:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 4k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 4096
4 KiB, 1 ops; 00.01 sec (443.724 KiB/sec and 110.9309 ops/sec)
read failed: Invalid argument

(The technical reason is that vmdk_write_extent() just writes the
incomplete compressed data actually to offset 4k.  When reading the
data, vmdk_read_extent() looks at offset 0 and finds the compressed data
size to be 0, because that is what it reads from there.  This yields an
error.)

For incomplete writes with zero in-cluster offsets, the error path when
reading the rest of the cluster is a bit different, but the result is
the same:

$ qemu-img create -f vmdk -o subformat=streamOptimized foo.vmdk 1M
$ qemu-io -c 'write 0k 4k' -c 'read 4k 4k' foo.vmdk
wrote 4096/4096 bytes at offset 0
4 KiB, 1 ops; 00.01 sec (362.641 KiB/sec and 90.6603 ops/sec)
read failed: Invalid argument

(Here, vmdk_read_extent() finds the data and then sees that the
uncompressed data is short.)

It is better to reject invalid writes than to make the user believe they
might have succeeded and then fail when trying to read it back.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-5-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_write_extent(VmdkExtent *extent, int64_t cluster_offset,
     if (extent->compressed) {
         void *compressed_data;
 
+        /* Only whole clusters */
+        if (offset_in_cluster ||
+            n_bytes > (extent->cluster_sectors * SECTOR_SIZE) ||
+            (n_bytes < (extent->cluster_sectors * SECTOR_SIZE) &&
+             offset + n_bytes != extent->end_sector * SECTOR_SIZE))
+        {
+            ret = -EINVAL;
+            goto out;
+        }
+
         if (!extent->has_marker) {
             ret = -EINVAL;
             goto out;
-- 
2.21.0

streamOptimized does not support writes that do not span exactly one
cluster.  Furthermore, it cannot rewrite already allocated clusters.
As such, many iotests do not work with it.  Disable them.

diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/002
+++ b/tests/qemu-iotests/002
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/003
+++ b/tests/qemu-iotests/003
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 size=128M
 offset=67M
diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/005
+++ b/tests/qemu-iotests/005
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 _supported_os Linux
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 # vpc is limited to 127GB, so we can't test it here
 if [ "$IMGFMT" = "vpc" ]; then
diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/009
+++ b/tests/qemu-iotests/009
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/010
+++ b/tests/qemu-iotests/010
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/011
+++ b/tests/qemu-iotests/011
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=6G
diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto generic
 _unsupported_proto vxhs
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/018
+++ b/tests/qemu-iotests/018
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _supported_os Linux
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/019
+++ b/tests/qemu-iotests/019
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 CLUSTER_SIZE=65536
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 TEST_OFFSETS="0 4294967296"
 
diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/027
+++ b/tests/qemu-iotests/027
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vmdk qcow qcow2 qed
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/032
+++ b/tests/qemu-iotests/032
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # This works for any image format (though unlikely to segfault for raw)
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo === Prepare image ===
diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/033
+++ b/tests/qemu-iotests/033
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt generic
 _supported_proto generic
+_unsupported_imgopts "subformat=streamOptimized"
 
 
 size=128M
diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/034
+++ b/tests/qemu-iotests/034
@@ -XXX,XX +XXX,XX @@ _supported_proto file
 _supported_os Linux
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/037
+++ b/tests/qemu-iotests/037
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 CLUSTER_SIZE=4k
 size=128M
diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/063
+++ b/tests/qemu-iotests/063
@@ -XXX,XX +XXX,XX @@ _supported_fmt qcow qcow2 vmdk qed raw
 _supported_proto file
 _unsupported_imgopts "subformat=monolithicFlat" \
                      "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 _make_test_img 4M
 
diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/072
+++ b/tests/qemu-iotests/072
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt vpc vmdk vhdx vdi qed qcow2 qcow
 _supported_proto file
+_unsupported_imgopts "subformat=streamOptimized"
 
 IMG_SIZE=64M
 
diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/105
+++ b/tests/qemu-iotests/105
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2 vmdk vhdx qed
 _supported_proto generic
 _unsupported_imgopts "subformat=twoGbMaxExtentFlat" \
-                     "subformat=twoGbMaxExtentSparse"
+                     "subformat=twoGbMaxExtentSparse" \
+                     "subformat=streamOptimized"
 
 echo
 echo "creating large image"
diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/197
+++ b/tests/qemu-iotests/197
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/215
+++ b/tests/qemu-iotests/215
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto generic
 # LUKS support may be possible, but it complicates things.
 _unsupported_fmt luks
+_unsupported_imgopts "subformat=streamOptimized"
 
 echo
 echo '=== Copy-on-read ==='
diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/251
+++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _supported_os Linux
+_unsupported_imgopts "subformat=streamOptimized"
 
 if [ "$IMGOPTSSYNTAX" = "true" ]; then
     # We use json:{} filenames here, so we cannot work with additional options.
-- 
2.21.0

The error message for the test case where we have a quorum node for
which no directory name can be generated is different: For
twoGbMaxExtentSparse, it complains that it cannot open the extent file.
For other (sub)formats, it just notes that it cannot determine the
backing file path.  Both are fine, but just disable twoGbMaxExtentSparse
for simplicity's sake.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190815153638.4600-7-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/110 | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/110
+++ b/tests/qemu-iotests/110
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Any format supporting backing files
 _supported_fmt qed qcow qcow2 vmdk
 _supported_proto file
-_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat"
+_unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
+                     "subformat=twoGbMaxExtentSparse"
 
 TEST_IMG_REL=$(basename "$TEST_IMG")
 
-- 
2.21.0

From: Stefan Hajnoczi <stefanha@redhat.com>

Fixes: a6b257a08e3d72219f03e461a52152672fec0612
       ("file-posix: Handle undetectable alignment")
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190827101328.4062-1-stefanha@redhat.com
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index XXXXXXX..XXXXXXX 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -XXX,XX +XXX,XX @@ static void raw_probe_alignment(BlockDriverState *bs, int fd, Error **errp)
         for (i = 0; i < ARRAY_SIZE(alignments); i++) {
             align = alignments[i];
             if (raw_is_io_aligned(fd, buf + align, max_align)) {
-                /* Fallback to request_aligment. */
+                /* Fallback to request_alignment. */
                 s->buf_align = (align != 1) ? align : bs->bl.request_alignment;
                 break;
             }
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

It is possible to enable only a subset of the block drivers with the
"--block-drv-rw-whitelist" option of the "configure" script. All other
drivers are marked as unusable (or only included as read-only with the
"--block-drv-ro-whitelist" option). If an iotest is now using such a
disabled block driver, it is failing - which is bad, since at least the
tests in the "auto" group should be able to deal with this situation.
Thus let's introduce a "_require_drivers" function that can be used by
the shell tests to check for the availability of certain drivers first,
and marks the test as "not run" if one of the drivers is missing.

This patch mainly targets the test in the "auto" group which should
never fail in such a case, but also improves some of the other tests
along the way. Note that we also assume that the "qcow2" and "file"
drivers are always available - otherwise it does not make sense to
run "make check-block" at all (which only tests with qcow2 by default).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823133552.11680-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/071       |  1 +
 tests/qemu-iotests/081       |  4 +---
 tests/qemu-iotests/099       |  1 +
 tests/qemu-iotests/120       |  1 +
 tests/qemu-iotests/162       |  4 +---
 tests/qemu-iotests/184       |  1 +
 tests/qemu-iotests/186       |  1 +
 tests/qemu-iotests/common.rc | 14 ++++++++++++++
 8 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/071
+++ b/tests/qemu-iotests/071
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers blkdebug blkverify
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/081
+++ b/tests/qemu-iotests/081
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt raw
 _supported_proto file
 _supported_os Linux
+_require_drivers quorum
 
 do_run_qemu()
 {
@@ -XXX,XX +XXX,XX @@ run_qemu()
                           | _filter_qemu_io | _filter_generated_node_ids
 }
 
-test_quorum=$($QEMU_IMG --help|grep quorum)
-[ "$test_quorum" = "" ] && _supported_fmt quorum
-
 quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
 quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
 quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/099
+++ b/tests/qemu-iotests/099
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 qed vdi vhdx vmdk vpc
 _supported_proto file
 _supported_os Linux
+_require_drivers blkdebug blkverify
 _unsupported_imgopts "subformat=monolithicFlat" "subformat=twoGbMaxExtentFlat" \
     "subformat=twoGbMaxExtentSparse"
 
diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/120
+++ b/tests/qemu-iotests/120
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt generic
 _supported_proto file
 _unsupported_fmt luks
+_require_drivers raw
 
 _make_test_img 64M
 
diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/162
+++ b/tests/qemu-iotests/162
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_fmt generic
-
-test_ssh=$($QEMU_IMG --help | grep '^Supported formats:.* ssh$ \|$$')
-[ "$test_ssh" = "" ] && _notrun "ssh support required"
+_require_drivers ssh
 
 echo
 echo '=== NBD ==='
diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/184
+++ b/tests/qemu-iotests/184
@@ -XXX,XX +XXX,XX @@ trap "exit \$status" 0 1 2 3 15
 . ./common.filter
 
 _supported_os Linux
+_require_drivers throttle
 
 do_run_qemu()
 {
diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/186
+++ b/tests/qemu-iotests/186
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _supported_fmt qcow2
 _supported_proto file
+_require_drivers null-co
 
 if [ "$QEMU_DEFAULT_MACHINE" != "pc" ]; then
     _notrun "Requires a PC machine"
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -XXX,XX +XXX,XX @@ _require_command()
     [ -x "$c" ] || _notrun "$1 utility required, skipped this test"
 }
 
+# Check that a set of drivers has been whitelisted in the QEMU binary
+#
+_require_drivers()
+{
+    available=$($QEMU -drive format=help | \
+                sed -e '/Supported formats:/!d' -e 's/Supported formats://')
+    for driver
+    do
+        if ! echo "$available" | grep -q " $driver$ \|$$"; then
+            _notrun "$driver not available"
+        fi
+    done
+}
+
 # make sure this script returns success
 true
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

The sanitizers (especially the address sanitizer from Clang) are
sometimes printing out warnings or false positives - this spoils
the output of the iotests, causing some of the tests to fail.
Thus let's skip the automatic iotests during "make check" when the
user configured QEMU with --enable-sanitizers.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190823084203.29734-1-thuth@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/check-block.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tests/check-block.sh b/tests/check-block.sh
index XXXXXXX..XXXXXXX 100755
--- a/tests/check-block.sh
+++ b/tests/check-block.sh
@@ -XXX,XX +XXX,XX @@ if grep -q "TARGET_GPROF=y" *-softmmu/config-target.mak 2>/dev/null ; then
     exit 0
 fi
 
+if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
+    echo "Sanitizers are enabled ==> Not running the qemu-iotests."
+    exit 0
+fi
+
 if [ -z "$(find . -name 'qemu-system-*' -print)" ]; then
     echo "No qemu-system binary available ==> Not running the qemu-iotests."
     exit 0
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Quoting cache mode is not needed, and most tests use unquoted values.
Unify all test to use the same style.

Message-id: 20190827173432.7656-1-nsoffer@redhat.com
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/026 | 4 ++--
 tests/qemu-iotests/039 | 4 ++--
 tests/qemu-iotests/052 | 2 +-
 tests/qemu-iotests/091 | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/026
+++ b/tests/qemu-iotests/026
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # Currently only qcow2 supports rebasing
 _supported_fmt qcow2
 _supported_proto file
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough" "none"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough none
 # The refcount table tests expect a certain minimum width for refcount entries
 # (so that the refcount table actually needs to grow); that minimum is 16 bits,
 # being the default refcount entry width.
diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/039
+++ b/tests/qemu-iotests/039
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "writethrough"
-_supported_cache_modes "writethrough"
+_default_cache_mode writethrough
+_supported_cache_modes writethrough
 
 size=128M
 
diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/052
+++ b/tests/qemu-iotests/052
@@ -XXX,XX +XXX,XX @@ _supported_fmt generic
 _supported_proto file
 
 # Don't do O_DIRECT on tmpfs
-_supported_cache_modes "writeback" "writethrough" "unsafe"
+_supported_cache_modes writeback writethrough unsafe
 
 size=128M
 _make_test_img $size
diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/091
+++ b/tests/qemu-iotests/091
@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow2
 _supported_proto file
 _supported_os Linux
-_default_cache_mode "none"
-_supported_cache_modes "writethrough" "none" "writeback"
+_default_cache_mode none
+_supported_cache_modes writethrough none writeback
 
 size=1G
 
-- 
2.21.0

The following changes since commit dd25f97c66a75d1508f1d4c6478ed2c95bec428f:

Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20190913' into staging (2019-09-16 10:15:15 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-09-16

for you to fetch changes up to 1825cc0783ccf0ec5d9f0b225a99b340bdd4c68f:

qemu-iotests: Add test for bz #1745922 (2019-09-16 15:37:12 +0200)

----------------------------------------------------------------
Block patches:
- Fix for block jobs when used with I/O threads
- Fix for a corruption when using qcow2's LUKS encryption mode
- cURL fix
- check-block.sh cleanups (for make check)
- Refactoring

----------------------------------------------------------------
Max Reitz (7):
  curl: Keep pointer to the CURLState in CURLSocket
  curl: Keep *socket until the end of curl_sock_cb()
  curl: Check completion in curl_multi_do()
  curl: Pass CURLSocket to curl_multi_do()
  curl: Report only ready sockets
  curl: Handle success in multi_check_completion
  curl: Check curl_multi_add_handle()'s return code

Maxim Levitsky (3):
  block/qcow2: Fix corruption introduced by commit 8ac0f15f335
  block/qcow2: refactor encryption code
  qemu-iotests: Add test for bz #1745922

Nir Soffer (2):
  block: Use QEMU_IS_ALIGNED
  block: Remove unused masks

Sergio Lopez (1):
  blockjob: update nodes head while removing all bdrv

Thomas Huth (2):
  tests/qemu-iotests/check: Replace "tests" with "iotests" in final
    status text
  tests/Makefile: Do not print the name of the check-block.sh shell
    script

Vladimir Sementsov-Ogievskiy (1):
  tests/qemu-iotests: Fix qemu-io related output in 026.out.nocache

-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Replace instances of:

(n & (BDRV_SECTOR_SIZE - 1)) == 0

And:

(n & ~BDRV_SECTOR_MASK) == 0

With:

QEMU_IS_ALIGNED(n, BDRV_SECTOR_SIZE)

Which reveals the intent of the code better, and makes it easier to
locate the code checking alignment.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190827185913.27427-2-nsoffer@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/bochs.c         | 4 ++--
 block/cloop.c         | 4 ++--
 block/dmg.c           | 4 ++--
 block/io.c            | 8 ++++----
 block/qcow2-cluster.c | 4 ++--
 block/qcow2.c         | 4 ++--
 block/vvfat.c         | 8 ++++----
 qemu-img.c            | 2 +-
 8 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/block/bochs.c b/block/bochs.c
index XXXXXXX..XXXXXXX 100644
--- a/block/bochs.c
+++ b/block/bochs.c
@@ -XXX,XX +XXX,XX @@ bochs_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
     QEMUIOVector local_qiov;
     int ret;
 
-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
 
     qemu_iovec_init(&local_qiov, qiov->niov);
     qemu_co_mutex_lock(&s->lock);
diff --git a/block/cloop.c b/block/cloop.c
index XXXXXXX..XXXXXXX 100644
--- a/block/cloop.c
+++ b/block/cloop.c
@@ -XXX,XX +XXX,XX @@ cloop_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
     int nb_sectors = bytes >> BDRV_SECTOR_BITS;
     int ret, i;
 
-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
 
     qemu_co_mutex_lock(&s->lock);
 
diff --git a/block/dmg.c b/block/dmg.c
index XXXXXXX..XXXXXXX 100644
--- a/block/dmg.c
+++ b/block/dmg.c
@@ -XXX,XX +XXX,XX @@ dmg_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
     int nb_sectors = bytes >> BDRV_SECTOR_BITS;
     int ret, i;
 
-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
 
     qemu_co_mutex_lock(&s->lock);
 
diff --git a/block/io.c b/block/io.c
index XXXXXXX..XXXXXXX 100644
--- a/block/io.c
+++ b/block/io.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_driver_preadv(BlockDriverState *bs,
     sector_num = offset >> BDRV_SECTOR_BITS;
     nb_sectors = bytes >> BDRV_SECTOR_BITS;
 
-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
     assert(bytes <= BDRV_REQUEST_MAX_BYTES);
     assert(drv->bdrv_co_readv);
 
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_driver_pwritev(BlockDriverState *bs,
     sector_num = offset >> BDRV_SECTOR_BITS;
     nb_sectors = bytes >> BDRV_SECTOR_BITS;
 
-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
     assert(bytes <= BDRV_REQUEST_MAX_BYTES);
 
     assert(drv->bdrv_co_writev);
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index XXXXXXX..XXXXXXX 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
 {
     if (bytes && bs->encrypted) {
         BDRVQcow2State *s = bs->opaque;
-        assert((offset_in_cluster & ~BDRV_SECTOR_MASK) == 0);
-        assert((bytes & ~BDRV_SECTOR_MASK) == 0);
+        assert(QEMU_IS_ALIGNED(offset_in_cluster, BDRV_SECTOR_SIZE));
+        assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
         assert(s->crypto);
         if (qcow2_co_encrypt(bs, cluster_offset,
                              src_cluster_offset + offset_in_cluster,
diff --git a/block/qcow2.c b/block/qcow2.c
index XXXXXXX..XXXXXXX 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv_part(BlockDriverState *bs,
                     goto fail;
                 }
 
-                assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-                assert((cur_bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+                assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+                assert(QEMU_IS_ALIGNED(cur_bytes, BDRV_SECTOR_SIZE));
                 if (qcow2_co_decrypt(bs, cluster_offset, offset,
                                      cluster_data, cur_bytes) < 0) {
                     ret = -EIO;
diff --git a/block/vvfat.c b/block/vvfat.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vvfat.c
+++ b/block/vvfat.c
@@ -XXX,XX +XXX,XX @@ vvfat_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
     int nb_sectors = bytes >> BDRV_SECTOR_BITS;
     void *buf;
 
-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
 
     buf = g_try_malloc(bytes);
     if (bytes && buf == NULL) {
@@ -XXX,XX +XXX,XX @@ vvfat_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
     int nb_sectors = bytes >> BDRV_SECTOR_BITS;
     void *buf;
 
-    assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0);
-    assert((bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
 
     buf = g_try_malloc(bytes);
     if (bytes && buf == NULL) {
diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
             int64_t sval;
 
             sval = cvtnum(optarg);
-            if (sval < 0 || sval & (BDRV_SECTOR_SIZE - 1) ||
+            if (sval < 0 || !QEMU_IS_ALIGNED(sval, BDRV_SECTOR_SIZE) ||
                 sval / BDRV_SECTOR_SIZE > MAX_BUF_SECTORS) {
                 error_report("Invalid buffer size for sparse output specified. "
                     "Valid sizes are multiples of %llu up to %llu. Select "
-- 
2.21.0

From: Nir Soffer <nirsof@gmail.com>

Replace confusing usage:

~BDRV_SECTOR_MASK

With more clear:

(BDRV_SECTOR_SIZE - 1)

Remove BDRV_SECTOR_MASK and the unused BDRV_BLOCK_OFFSET_MASK which was
it's last user.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190827185913.27427-3-nsoffer@redhat.com
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block.h | 2 --
 migration/block.c     | 2 +-
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/include/block/block.h b/include/block/block.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -XXX,XX +XXX,XX @@ typedef struct HDGeometry {
 
 #define BDRV_SECTOR_BITS   9
 #define BDRV_SECTOR_SIZE   (1ULL << BDRV_SECTOR_BITS)
-#define BDRV_SECTOR_MASK   ~(BDRV_SECTOR_SIZE - 1)
 
 #define BDRV_REQUEST_MAX_SECTORS MIN(SIZE_MAX >> BDRV_SECTOR_BITS, \
                                      INT_MAX >> BDRV_SECTOR_BITS)
@@ -XXX,XX +XXX,XX @@ typedef struct HDGeometry {
 #define BDRV_BLOCK_ALLOCATED    0x10
 #define BDRV_BLOCK_EOF          0x20
 #define BDRV_BLOCK_RECURSE      0x40
-#define BDRV_BLOCK_OFFSET_MASK  BDRV_SECTOR_MASK
 
 typedef QSIMPLEQ_HEAD(BlockReopenQueue, BlockReopenQueueEntry) BlockReopenQueue;
 
diff --git a/migration/block.c b/migration/block.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -XXX,XX +XXX,XX @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
     do {
         addr = qemu_get_be64(f);
 
-        flags = addr & ~BDRV_SECTOR_MASK;
+        flags = addr & (BDRV_SECTOR_SIZE - 1);
         addr >>= BDRV_SECTOR_BITS;
 
         if (flags & BLK_MIG_FLAG_DEVICE_BLOCK) {
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

When running "make check -j8" or something similar, the iotests are
running in parallel with the other tests. So when they are printing
out "Passed all xx tests" or a similar status message at the end,
it might not be quite clear that this message belongs to the iotests,
since the output might be mixed with the other tests. Thus change the
word "tests" here to "iotests" instead to avoid confusion.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190906113920.11271-1-thuth@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/check | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -XXX,XX +XXX,XX @@ END        { if (NR > 0) {
         if [ ! -z "$n_bad" -a $n_bad != 0 ]
         then
             echo "Failures:$bad"
-            echo "Failed $n_bad of $try tests"
+            echo "Failed $n_bad of $try iotests"
             echo "Failures:$bad" | fmt >>check.log
-            echo "Failed $n_bad of $try tests" >>check.log
+            echo "Failed $n_bad of $try iotests" >>check.log
         else
-            echo "Passed all $try tests"
-            echo "Passed all $try tests" >>check.log
+            echo "Passed all $try iotests"
+            echo "Passed all $try iotests" >>check.log
         fi
         needwrap=false
     fi
-- 
2.21.0

From: Thomas Huth <thuth@redhat.com>

The check script is already printing out which iotest is currently
running, so printing out the name of the check-block.sh shell script
looks superfluous here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-id: 20190906113534.10907-1-thuth@redhat.com
Acked-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index XXXXXXX..XXXXXXX 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -XXX,XX +XXX,XX @@ QEMU_IOTESTS_HELPERS-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_LINUX)) = tests/qemu
 check-tests/check-block.sh: tests/check-block.sh qemu-img$(EXESUF) \
 		qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) \
 		$(patsubst %,%/all,$(filter %-softmmu,$(TARGET_DIRS)))
-	$<
+	@$<
 
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

qemu-io now prefixes its error and warnings with "qemu-io:".
36b9986b08787019e fixed a lot of iotests output but forget about
026.out.nocache. Fix it too.

Fixes: 99e98d7c9fc1a1639fad ("qemu-io: Use error_[gs]et_progname()")
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190816153015.447957-2-vsementsov@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/026.out.nocache | 168 ++++++++++++++---------------
 1 file changed, 84 insertions(+), 84 deletions(-)

diff --git a/tests/qemu-iotests/026.out.nocache b/tests/qemu-iotests/026.out.nocache
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/026.out.nocache
+++ b/tests/qemu-iotests/026.out.nocache
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_update; errno: 5; imm: off; once: off; write 
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 
 1 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_update; errno: 5; imm: off; once: off; write -b
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 
 1 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_update; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 1 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_update; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 1 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
 
 Event: l2_update; errno: 5; imm: off; once: off; write 
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 wrote 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l2_update; errno: 5; imm: off; once: off; write -b
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 wrote 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
 
 Event: l2_update; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 wrote 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l2_update; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 wrote 131072/131072 bytes at offset 0
 128 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l2_alloc_write; errno: 5; imm: off; once: off; write 
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l2_alloc_write; errno: 5; imm: off; once: off; write -b
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 
 1 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l2_alloc_write; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l2_alloc_write; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 1 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: write_aio; errno: 5; imm: off; once: off; write 
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: write_aio; errno: 5; imm: off; once: off; write -b
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: write_aio; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: write_aio; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_load; errno: 5; imm: off; once: off; write 
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_load; errno: 5; imm: off; once: off; write -b
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_load; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_load; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_update_part; errno: 5; imm: off; once: off; write 
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_update_part; errno: 5; imm: off; once: off; write -b
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_update_part; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_update_part; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc; errno: 5; imm: off; once: off; write 
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc; errno: 5; imm: off; once: off; write -b
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_hookup; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 55 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_hookup; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 251 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_write; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_write; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_write_blocks; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 10 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_write_blocks; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 23 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_write_table; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 10 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_write_table; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 23 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_switch_table; errno: 28; imm: off; once: off; write 
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 10 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ This means waste of disk space, but no harm to data.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: refblock_alloc_switch_table; errno: 28; imm: off; once: off; write -b
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 23 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_grow_write_table; errno: 5; imm: off; once: off
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_grow_write_table; errno: 28; imm: off; once: off
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_grow_activate_table; errno: 5; imm: off; once: off
-Failed to flush the L2 table cache: Input/output error
-Failed to flush the refcount block cache: Input/output error
+qemu-io: Failed to flush the L2 table cache: Input/output error
+qemu-io: Failed to flush the refcount block cache: Input/output error
 write failed: Input/output error
 
 96 leaked clusters were found on the image.
@@ -XXX,XX +XXX,XX @@ No errors were found on the image.
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 
 
 Event: l1_grow_activate_table; errno: 28; imm: off; once: off
-Failed to flush the L2 table cache: No space left on device
-Failed to flush the refcount block cache: No space left on device
+qemu-io: Failed to flush the L2 table cache: No space left on device
+qemu-io: Failed to flush the refcount block cache: No space left on device
 write failed: No space left on device
 
 96 leaked clusters were found on the image.
-- 
2.21.0

A follow-up patch will make curl_multi_do() and curl_multi_read() take a
CURLSocket instead of the CURLState.  They still need the latter,
though, so add a pointer to it to the former.

Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190910124136.10565-2-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static CURLMcode __curl_multi_socket_action(CURLM *multi_handle,
 #define CURL_BLOCK_OPT_TIMEOUT_DEFAULT 5
 
 struct BDRVCURLState;
+struct CURLState;
 
 static bool libcurl_initialized;
 
@@ -XXX,XX +XXX,XX @@ typedef struct CURLAIOCB {
 
 typedef struct CURLSocket {
     int fd;
+    struct CURLState *state;
     QLIST_ENTRY(CURLSocket) next;
 } CURLSocket;
 
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
     if (!socket) {
         socket = g_new0(CURLSocket, 1);
         socket->fd = fd;
+        socket->state = state;
         QLIST_INSERT_HEAD(&state->sockets, socket, next);
     }
     socket = NULL;
-- 
2.21.0

This does not really change anything, but it makes the code a bit easier
to follow once we use @socket as the opaque pointer for
aio_set_fd_handler().

Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-3-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
 
     QLIST_FOREACH(socket, &state->sockets, next) {
         if (socket->fd == fd) {
-            if (action == CURL_POLL_REMOVE) {
-                QLIST_REMOVE(socket, next);
-                g_free(socket);
-            }
             break;
         }
     }
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
         socket->state = state;
         QLIST_INSERT_HEAD(&state->sockets, socket, next);
     }
-    socket = NULL;
 
     trace_curl_sock_cb(action, (int)fd);
     switch (action) {
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
             break;
     }
 
+    if (action == CURL_POLL_REMOVE) {
+        QLIST_REMOVE(socket, next);
+        g_free(socket);
+    }
+
     return 0;
 }
 
-- 
2.21.0

While it is more likely that transfers complete after some file
descriptor has data ready to read, we probably should not rely on it.
Better be safe than sorry and call curl_multi_check_completion() in
curl_multi_do(), too, just like it is done in curl_multi_read().

With this change, curl_multi_do() and curl_multi_read() are actually the
same, so drop curl_multi_read() and use curl_multi_do() as the sole FD
handler.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-4-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVCURLState {
 
 static void curl_clean_state(CURLState *s);
 static void curl_multi_do(void *arg);
-static void curl_multi_read(void *arg);
 
 #ifdef NEED_CURL_TIMER_CALLBACK
 /* Called from curl_multi_do_locked, with s->mutex held.  */
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
     switch (action) {
         case CURL_POLL_IN:
             aio_set_fd_handler(s->aio_context, fd, false,
-                               curl_multi_read, NULL, NULL, state);
+                               curl_multi_do, NULL, NULL, state);
             break;
         case CURL_POLL_OUT:
             aio_set_fd_handler(s->aio_context, fd, false,
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
             break;
         case CURL_POLL_INOUT:
             aio_set_fd_handler(s->aio_context, fd, false,
-                               curl_multi_read, curl_multi_do, NULL, state);
+                               curl_multi_do, curl_multi_do, NULL, state);
             break;
         case CURL_POLL_REMOVE:
             aio_set_fd_handler(s->aio_context, fd, false,
@@ -XXX,XX +XXX,XX @@ static void curl_multi_do(void *arg)
 {
     CURLState *s = (CURLState *)arg;
 
-    qemu_mutex_lock(&s->s->mutex);
-    curl_multi_do_locked(s);
-    qemu_mutex_unlock(&s->s->mutex);
-}
-
-static void curl_multi_read(void *arg)
-{
-    CURLState *s = (CURLState *)arg;
-
     qemu_mutex_lock(&s->s->mutex);
     curl_multi_do_locked(s);
     curl_multi_check_completion(s->s);
-- 
2.21.0

curl_multi_do_locked() currently marks all sockets as ready.  That is
not only inefficient, but in fact unsafe (the loop is).  A follow-up
patch will change that, but to do so, curl_multi_do_locked() needs to
know exactly which socket is ready; and that is accomplished by this
patch here.

Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-5-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static int curl_sock_cb(CURL *curl, curl_socket_t fd, int action,
     switch (action) {
         case CURL_POLL_IN:
             aio_set_fd_handler(s->aio_context, fd, false,
-                               curl_multi_do, NULL, NULL, state);
+                               curl_multi_do, NULL, NULL, socket);
             break;
         case CURL_POLL_OUT:
             aio_set_fd_handler(s->aio_context, fd, false,
-                               NULL, curl_multi_do, NULL, state);
+                               NULL, curl_multi_do, NULL, socket);
             break;
         case CURL_POLL_INOUT:
             aio_set_fd_handler(s->aio_context, fd, false,
-                               curl_multi_do, curl_multi_do, NULL, state);
+                               curl_multi_do, curl_multi_do, NULL, socket);
             break;
         case CURL_POLL_REMOVE:
             aio_set_fd_handler(s->aio_context, fd, false,
@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
 }
 
 /* Called with s->mutex held.  */
-static void curl_multi_do_locked(CURLState *s)
+static void curl_multi_do_locked(CURLSocket *ready_socket)
 {
     CURLSocket *socket, *next_socket;
+    CURLState *s = ready_socket->state;
     int running;
     int r;
 
@@ -XXX,XX +XXX,XX @@ static void curl_multi_do_locked(CURLState *s)
 
 static void curl_multi_do(void *arg)
 {
-    CURLState *s = (CURLState *)arg;
+    CURLSocket *socket = arg;
+    BDRVCURLState *s = socket->state->s;
 
-    qemu_mutex_lock(&s->s->mutex);
-    curl_multi_do_locked(s);
-    curl_multi_check_completion(s->s);
-    qemu_mutex_unlock(&s->s->mutex);
+    qemu_mutex_lock(&s->mutex);
+    curl_multi_do_locked(socket);
+    curl_multi_check_completion(s);
+    qemu_mutex_unlock(&s->mutex);
 }
 
 static void curl_multi_timeout_do(void *arg)
-- 
2.21.0

Instead of reporting all sockets to cURL, only report the one that has
caused curl_multi_do_locked() to be called.  This lets us get rid of the
QLIST_FOREACH_SAFE() list, which was actually wrong: SAFE foreaches are
only safe when the current element is removed in each iteration.  If it
possible for the list to be concurrently modified, we cannot guarantee
that only the current element will be removed.  Therefore, we must not
use QLIST_FOREACH_SAFE() here.

Fixes: ff5ca1664af85b24a4180d595ea6873fd3deac57
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-6-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
 }
 
 /* Called with s->mutex held.  */
-static void curl_multi_do_locked(CURLSocket *ready_socket)
+static void curl_multi_do_locked(CURLSocket *socket)
 {
-    CURLSocket *socket, *next_socket;
-    CURLState *s = ready_socket->state;
+    BDRVCURLState *s = socket->state->s;
     int running;
     int r;
 
-    if (!s->s->multi) {
+    if (!s->multi) {
         return;
     }
 
-    /* Need to use _SAFE because curl_multi_socket_action() may trigger
-     * curl_sock_cb() which might modify this list */
-    QLIST_FOREACH_SAFE(socket, &s->sockets, next, next_socket) {
-        do {
-            r = curl_multi_socket_action(s->s->multi, socket->fd, 0, &running);
-        } while (r == CURLM_CALL_MULTI_PERFORM);
-    }
+    do {
+        r = curl_multi_socket_action(s->multi, socket->fd, 0, &running);
+    } while (r == CURLM_CALL_MULTI_PERFORM);
 }
 
 static void curl_multi_do(void *arg)
-- 
2.21.0

Background: As of cURL 7.59.0, it verifies that several functions are
not called from within a callback.  Among these functions is
curl_multi_add_handle().

curl_read_cb() is a callback from cURL and not a coroutine.  Waking up
acb->co will lead to entering it then and there, which means the current
request will settle and the caller (if it runs in the same coroutine)
may then issue the next request.  In such a case, we will enter
curl_setup_preadv() effectively from within curl_read_cb().

Calling curl_multi_add_handle() will then fail and the new request will
not be processed.

Fix this by not letting curl_read_cb() wake up acb->co.  Instead, leave
the whole business of settling the AIOCB objects to
curl_multi_check_completion() (which is called from our timer callback
and our FD handler, so not from any cURL callbacks).

Reported-by: Natalie Gavrielov <ngavrilo@redhat.com>
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1740193
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-7-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 69 ++++++++++++++++++++++------------------------------
 1 file changed, 29 insertions(+), 40 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
 {
     CURLState *s = ((CURLState*)opaque);
     size_t realsize = size * nmemb;
-    int i;
 
     trace_curl_read_cb(realsize);
 
@@ -XXX,XX +XXX,XX @@ static size_t curl_read_cb(void *ptr, size_t size, size_t nmemb, void *opaque)
     memcpy(s->orig_buf + s->buf_off, ptr, realsize);
     s->buf_off += realsize;
 
-    for(i=0; i<CURL_NUM_ACB; i++) {
-        CURLAIOCB *acb = s->acb[i];
-
-        if (!acb)
-            continue;
-
-        if ((s->buf_off >= acb->end)) {
-            size_t request_length = acb->bytes;
-
-            qemu_iovec_from_buf(acb->qiov, 0, s->orig_buf + acb->start,
-                                acb->end - acb->start);
-
-            if (acb->end - acb->start < request_length) {
-                size_t offset = acb->end - acb->start;
-                qemu_iovec_memset(acb->qiov, offset, 0,
-                                  request_length - offset);
-            }
-
-            acb->ret = 0;
-            s->acb[i] = NULL;
-            qemu_mutex_unlock(&s->s->mutex);
-            aio_co_wake(acb->co);
-            qemu_mutex_lock(&s->s->mutex);
-        }
-    }
-
 read_end:
     /* curl will error out if we do not return this value */
     return size * nmemb;
@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
             break;
 
         if (msg->msg == CURLMSG_DONE) {
+            int i;
             CURLState *state = NULL;
+            bool error = msg->data.result != CURLE_OK;
+
             curl_easy_getinfo(msg->easy_handle, CURLINFO_PRIVATE,
                               (char **)&state);
 
-            /* ACBs for successful messages get completed in curl_read_cb */
-            if (msg->data.result != CURLE_OK) {
-                int i;
+            if (error) {
                 static int errcount = 100;
 
                 /* Don't lose the original error message from curl, since
@@ -XXX,XX +XXX,XX @@ static void curl_multi_check_completion(BDRVCURLState *s)
                         error_report("curl: further errors suppressed");
                     }
                 }
+            }
 
-                for (i = 0; i < CURL_NUM_ACB; i++) {
-                    CURLAIOCB *acb = state->acb[i];
+            for (i = 0; i < CURL_NUM_ACB; i++) {
+                CURLAIOCB *acb = state->acb[i];
 
-                    if (acb == NULL) {
-                        continue;
-                    }
+                if (acb == NULL) {
+                    continue;
+                }
+
+                if (!error) {
+                    /* Assert that we have read all data */
+                    assert(state->buf_off >= acb->end);
+
+                    qemu_iovec_from_buf(acb->qiov, 0,
+                                        state->orig_buf + acb->start,
+                                        acb->end - acb->start);
 
-                    acb->ret = -EIO;
-                    state->acb[i] = NULL;
-                    qemu_mutex_unlock(&s->mutex);
-                    aio_co_wake(acb->co);
-                    qemu_mutex_lock(&s->mutex);
+                    if (acb->end - acb->start < acb->bytes) {
+                        size_t offset = acb->end - acb->start;
+                        qemu_iovec_memset(acb->qiov, offset, 0,
+                                          acb->bytes - offset);
+                    }
                 }
+
+                acb->ret = error ? -EIO : 0;
+                state->acb[i] = NULL;
+                qemu_mutex_unlock(&s->mutex);
+                aio_co_wake(acb->co);
+                qemu_mutex_lock(&s->mutex);
             }
 
             curl_clean_state(state);
-- 
2.21.0

If we had done that all along, debugging would have been much simpler.
(Also, I/O errors are better than hangs.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190910124136.10565-8-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/curl.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
     trace_curl_setup_preadv(acb->bytes, start, state->range);
     curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range);
 
-    curl_multi_add_handle(s->multi, state->curl);
+    if (curl_multi_add_handle(s->multi, state->curl) != CURLM_OK) {
+        state->acb[0] = NULL;
+        acb->ret = -EIO;
+
+        curl_clean_state(state);
+        goto out;
+    }
 
     /* Tell curl it needs to kick things off */
     curl_multi_socket_action(s->multi, CURL_SOCKET_TIMEOUT, 0, &running);
-- 
2.21.0

From: Sergio Lopez <slp@redhat.com>

block_job_remove_all_bdrv() iterates through job->nodes, calling
bdrv_root_unref_child() for each entry. The call to the latter may
reach child_job_[can_]set_aio_ctx(), which will also attempt to
traverse job->nodes, potentially finding entries that where freed
on previous iterations.

To avoid this situation, update job->nodes head on each iteration to
ensure that already freed entries are no longer linked to the list.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1746631
Signed-off-by: Sergio Lopez <slp@redhat.com>
Cc: qemu-stable@nongnu.org
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190911100316.32282-1-mreitz@redhat.com
Reviewed-by: Sergio Lopez <slp@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockjob.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/blockjob.c b/blockjob.c
index XXXXXXX..XXXXXXX 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -XXX,XX +XXX,XX @@ static const BdrvChildRole child_job = {
 
 void block_job_remove_all_bdrv(BlockJob *job)
 {
-    GSList *l;
-    for (l = job->nodes; l; l = l->next) {
+    /*
+     * bdrv_root_unref_child() may reach child_job_[can_]set_aio_ctx(),
+     * which will also traverse job->nodes, so consume the list one by
+     * one to make sure that such a concurrent access does not attempt
+     * to process an already freed BdrvChild.
+     */
+    while (job->nodes) {
+        GSList *l = job->nodes;
         BdrvChild *c = l->data;
+
+        job->nodes = l->next;
+
         bdrv_op_unblock_all(c->bs, job->blocker);
         bdrv_root_unref_child(c);
+
+        g_slist_free_1(l);
     }
-    g_slist_free(job->nodes);
-    job->nodes = NULL;
 }
 
 bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs)
-- 
2.21.0

From: Maxim Levitsky <mlevitsk@redhat.com>

This fixes subtle corruption introduced by luks threaded encryption
in commit 8ac0f15f335

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1745922

The corruption happens when we do a write that
   * writes to two or more unallocated clusters at once
   * doesn't fully cover the first sector
   * doesn't fully cover the last sector
   * uses luks encryption

In this case, when allocating the new clusters we COW both areas
prior to the write and after the write, and we encrypt them.

The above mentioned commit accidentally made it so we encrypt the
second COW area using the physical cluster offset of the first area.

The problem is that offset_in_cluster in do_perform_cow_encrypt
can be larger that the cluster size, thus cluster_offset
will no longer point to the start of the cluster at which encrypted
area starts.

Next patch in this series will refactor the code to avoid all these
assumptions.

In the bugreport that was triggered by rebasing a luks image to new,
zero filled base, which lot of such writes, and causes some files
with zero areas to contain garbage there instead.
But as described above it can happen elsewhere as well

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190915203655.21638-2-mlevitsk@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/qcow2-cluster.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index XXXXXXX..XXXXXXX 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
         assert(QEMU_IS_ALIGNED(offset_in_cluster, BDRV_SECTOR_SIZE));
         assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
         assert(s->crypto);
-        if (qcow2_co_encrypt(bs, cluster_offset,
-                             src_cluster_offset + offset_in_cluster,
-                             buffer, bytes) < 0) {
+        if (qcow2_co_encrypt(bs,
+                start_of_cluster(s, cluster_offset + offset_in_cluster),
+                src_cluster_offset + offset_in_cluster,
+                buffer, bytes) < 0) {
             return false;
         }
     }
-- 
2.21.0

From: Maxim Levitsky <mlevitsk@redhat.com>

* Change the qcow2_co_{encrypt|decrypt} to just receive full host and
  guest offsets and use this function directly instead of calling
  do_perform_cow_encrypt (which is removed by that patch).

* Adjust qcow2_co_encdec to take full host and guest offsets as well.

* Document the qcow2_co_{encrypt|decrypt} arguments
  to prevent the bug fixed in former commit from hopefully
  happening again.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-id: 20190915203655.21638-3-mlevitsk@redhat.com
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[mreitz: Let perform_cow() return the error value returned by
         qcow2_co_encrypt(), as proposed by Vladimir]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/qcow2.h         |  8 +++---
 block/qcow2-cluster.c | 41 +++++++++-------------------
 block/qcow2-threads.c | 63 +++++++++++++++++++++++++++++++++----------
 block/qcow2.c         |  5 ++--
 4 files changed, 69 insertions(+), 48 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index XXXXXXX..XXXXXXX 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -XXX,XX +XXX,XX @@ ssize_t coroutine_fn
 qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
                     const void *src, size_t src_size);
 int coroutine_fn
-qcow2_co_encrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
-                 uint64_t offset, void *buf, size_t len);
+qcow2_co_encrypt(BlockDriverState *bs, uint64_t host_offset,
+                 uint64_t guest_offset, void *buf, size_t len);
 int coroutine_fn
-qcow2_co_decrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
-                 uint64_t offset, void *buf, size_t len);
+qcow2_co_decrypt(BlockDriverState *bs, uint64_t host_offset,
+                 uint64_t guest_offset, void *buf, size_t len);
 
 #endif
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index XXXXXXX..XXXXXXX 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn do_perform_cow_read(BlockDriverState *bs,
     return 0;
 }
 
-static bool coroutine_fn do_perform_cow_encrypt(BlockDriverState *bs,
-                                                uint64_t src_cluster_offset,
-                                                uint64_t cluster_offset,
-                                                unsigned offset_in_cluster,
-                                                uint8_t *buffer,
-                                                unsigned bytes)
-{
-    if (bytes && bs->encrypted) {
-        BDRVQcow2State *s = bs->opaque;
-        assert(QEMU_IS_ALIGNED(offset_in_cluster, BDRV_SECTOR_SIZE));
-        assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE));
-        assert(s->crypto);
-        if (qcow2_co_encrypt(bs,
-                start_of_cluster(s, cluster_offset + offset_in_cluster),
-                src_cluster_offset + offset_in_cluster,
-                buffer, bytes) < 0) {
-            return false;
-        }
-    }
-    return true;
-}
-
 static int coroutine_fn do_perform_cow_write(BlockDriverState *bs,
                                              uint64_t cluster_offset,
                                              unsigned offset_in_cluster,
@@ -XXX,XX +XXX,XX @@ static int perform_cow(BlockDriverState *bs, QCowL2Meta *m)
 
     /* Encrypt the data if necessary before writing it */
     if (bs->encrypted) {
-        if (!do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
-                                    start->offset, start_buffer,
-                                    start->nb_bytes) ||
-            !do_perform_cow_encrypt(bs, m->offset, m->alloc_offset,
-                                    end->offset, end_buffer, end->nb_bytes)) {
-            ret = -EIO;
+        ret = qcow2_co_encrypt(bs,
+                               m->alloc_offset + start->offset,
+                               m->offset + start->offset,
+                               start_buffer, start->nb_bytes);
+        if (ret < 0) {
+            goto fail;
+        }
+
+        ret = qcow2_co_encrypt(bs,
+                               m->alloc_offset + end->offset,
+                               m->offset + end->offset,
+                               end_buffer, end->nb_bytes);
+        if (ret < 0) {
             goto fail;
         }
     }
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index XXXXXXX..XXXXXXX 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -XXX,XX +XXX,XX @@ static int qcow2_encdec_pool_func(void *opaque)
 }
 
 static int coroutine_fn
-qcow2_co_encdec(BlockDriverState *bs, uint64_t file_cluster_offset,
-                  uint64_t offset, void *buf, size_t len, Qcow2EncDecFunc func)
+qcow2_co_encdec(BlockDriverState *bs, uint64_t host_offset,
+                uint64_t guest_offset, void *buf, size_t len,
+                Qcow2EncDecFunc func)
 {
     BDRVQcow2State *s = bs->opaque;
     Qcow2EncDecData arg = {
         .block = s->crypto,
-        .offset = s->crypt_physical_offset ?
-                      file_cluster_offset + offset_into_cluster(s, offset) :
-                      offset,
+        .offset = s->crypt_physical_offset ? host_offset : guest_offset,
         .buf = buf,
         .len = len,
         .func = func,
     };
 
-    return qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
+    assert(QEMU_IS_ALIGNED(guest_offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(host_offset, BDRV_SECTOR_SIZE));
+    assert(QEMU_IS_ALIGNED(len, BDRV_SECTOR_SIZE));
+    assert(s->crypto);
+
+    return len == 0 ? 0 : qcow2_co_process(bs, qcow2_encdec_pool_func, &arg);
 }
 
+/*
+ * qcow2_co_encrypt()
+ *
+ * Encrypts one or more contiguous aligned sectors
+ *
+ * @host_offset - underlying storage offset of the first sector of the
+ * data to be encrypted
+ *
+ * @guest_offset - guest (virtual) offset of the first sector of the
+ * data to be encrypted
+ *
+ * @buf - buffer with the data to encrypt, that after encryption
+ *        will be written to the underlying storage device at
+ *        @host_offset
+ *
+ * @len - length of the buffer (must be a BDRV_SECTOR_SIZE multiple)
+ *
+ * Depending on the encryption method, @host_offset and/or @guest_offset
+ * may be used for generating the initialization vector for
+ * encryption.
+ *
+ * Note that while the whole range must be aligned on sectors, it
+ * does not have to be aligned on clusters and can also cross cluster
+ * boundaries
+ */
 int coroutine_fn
-qcow2_co_encrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
-                 uint64_t offset, void *buf, size_t len)
+qcow2_co_encrypt(BlockDriverState *bs, uint64_t host_offset,
+                 uint64_t guest_offset, void *buf, size_t len)
 {
-    return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len,
-                             qcrypto_block_encrypt);
+    return qcow2_co_encdec(bs, host_offset, guest_offset, buf, len,
+                           qcrypto_block_encrypt);
 }
 
+/*
+ * qcow2_co_decrypt()
+ *
+ * Decrypts one or more contiguous aligned sectors
+ * Similar to qcow2_co_encrypt
+ */
 int coroutine_fn
-qcow2_co_decrypt(BlockDriverState *bs, uint64_t file_cluster_offset,
-                 uint64_t offset, void *buf, size_t len)
+qcow2_co_decrypt(BlockDriverState *bs, uint64_t host_offset,
+                 uint64_t guest_offset, void *buf, size_t len)
 {
-    return qcow2_co_encdec(bs, file_cluster_offset, offset, buf, len,
-                             qcrypto_block_decrypt);
+    return qcow2_co_encdec(bs, host_offset, guest_offset, buf, len,
+                           qcrypto_block_decrypt);
 }
diff --git a/block/qcow2.c b/block/qcow2.c
index XXXXXXX..XXXXXXX 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv_part(BlockDriverState *bs,
 
                 assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE));
                 assert(QEMU_IS_ALIGNED(cur_bytes, BDRV_SECTOR_SIZE));
-                if (qcow2_co_decrypt(bs, cluster_offset, offset,
+                if (qcow2_co_decrypt(bs, cluster_offset + offset_in_cluster,
+                                     offset,
                                      cluster_data, cur_bytes) < 0) {
                     ret = -EIO;
                     goto fail;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev_part(
             qemu_iovec_to_buf(qiov, qiov_offset + bytes_done,
                               cluster_data, cur_bytes);
 
-            if (qcow2_co_encrypt(bs, cluster_offset, offset,
+            if (qcow2_co_encrypt(bs, cluster_offset + offset_in_cluster, offset,
                                  cluster_data, cur_bytes) < 0) {
                 ret = -EIO;
                 goto out_unlocked;
-- 
2.21.0

From: Maxim Levitsky <mlevitsk@redhat.com>

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190915203655.21638-4-mlevitsk@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/263     | 91 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/263.out | 40 +++++++++++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 132 insertions(+)
 create mode 100755 tests/qemu-iotests/263
 create mode 100644 tests/qemu-iotests/263.out

diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263
new file mode 100755
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/263
@@ -XXX,XX +XXX,XX @@
+#!/usr/bin/env bash
+#
+# Test encrypted write that crosses cluster boundary of two unallocated clusters
+# Based on 188
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mlevitsk@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto generic
+_supported_os Linux
+
+
+size=1M
+
+SECRET="secret,id=sec0,data=astrochicken"
+QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT
+
+
+_run_test()
+{
+	echo "== reading the whole image =="
+	$QEMU_IO --object $SECRET -c "read -P 0 0 $size" --image-opts "$1" | _filter_qemu_io | _filter_testdir
+
+	echo
+	echo "== write two 512 byte sectors on a cluster boundary =="
+	$QEMU_IO --object $SECRET -c "write -P 0xAA 0xFE00 0x400" --image-opts "$1" | _filter_qemu_io | _filter_testdir
+
+	echo
+	echo "== verify that the rest of the image is not changed =="
+	$QEMU_IO --object $SECRET -c "read -P 0x00 0x00000 0xFE00" --image-opts "$1" | _filter_qemu_io | _filter_testdir
+	$QEMU_IO --object $SECRET -c "read -P 0xAA 0x0FE00 0x400" --image-opts "$1" | _filter_qemu_io | _filter_testdir
+	$QEMU_IO --object $SECRET -c "read -P 0x00 0x10200 0xEFE00" --image-opts "$1" | _filter_qemu_io | _filter_testdir
+
+}
+
+
+echo
+echo "testing LUKS qcow2 encryption"
+echo
+
+_make_test_img --object $SECRET -o "encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10,cluster_size=64K" $size
+_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
+_cleanup_test_img
+
+echo
+echo "testing legacy AES qcow2 encryption"
+echo
+
+
+_make_test_img --object $SECRET -o "encrypt.format=aes,encrypt.key-secret=sec0,cluster_size=64K" $size
+_run_test "driver=$IMGFMT,encrypt.key-secret=sec0,file.filename=$TEST_IMG"
+_cleanup_test_img
+
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/263.out b/tests/qemu-iotests/263.out
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/263.out
@@ -XXX,XX +XXX,XX @@
+QA output created by 263
+
+testing LUKS qcow2 encryption
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
+== reading the whole image ==
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== write two 512 byte sectors on a cluster boundary ==
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify that the rest of the image is not changed ==
+read 65024/65024 bytes at offset 0
+63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 982528/982528 bytes at offset 66048
+959.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+testing legacy AES qcow2 encryption
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=aes encrypt.key-secret=sec0
+== reading the whole image ==
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== write two 512 byte sectors on a cluster boundary ==
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify that the rest of the image is not changed ==
+read 65024/65024 bytes at offset 0
+63.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 982528/982528 bytes at offset 66048
+959.500 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -XXX,XX +XXX,XX @@
 257 rw
 258 rw quick
 262 rw quick migration
+263 rw quick
 265 rw auto quick
 266 rw quick
-- 
2.21.0