Series comparison

-[Qemu-devel] [PULL 00/20] Block patches
+[Qemu-devel] [PULL v2 0/8] Block patches
-The following changes since commit 5ec2eca83dc478ddf24077e02a8b34dd26cd3ff9:
+The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:
-  Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20190613.0' into staging (2019-06-14 09:33:55 +0100)
+  Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)
 are available in the Git repository at:
-  https://github.com/XanClic/qemu.git tags/pull-block-2019-06-14
+  https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24
-for you to fetch changes up to 21c1ce592a144188dfe59b9e156a97da412a59a2:
+for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:
-  iotests: Test qemu-img convert -C --salvage (2019-06-14 15:09:42 +0200)
+  iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)
 ----------------------------------------------------------------
 Block patches:
-- Allow blockdev-backup from nodes that are not in qemu's main AIO
+- The SSH block driver now uses libssh instead of libssh2
-  context to newly added nodes
+- The VMDK block driver gets read-only support for the seSparse
-- Add salvaging mode to qemu-img convert
+  subformat
-- Minor fixes to tests, documentation, and for less Valgrind annoyance
+- Various fixes
 ---
 v2:
 - Squashed Pino's fix for pre-0.8 libssh into the libssh patch
 ----------------------------------------------------------------
-Andrey Shinkevich (1):
+Anton Nefedov (1):
-  hw/block/fdc: floppy command FIFO memory initialization
+  iotest 134: test cluster-misaligned encrypted write
-John Snow (6):
+Klaus Birkelund Jensen (1):
-  blockdev-backup: don't check aio_context too early
+  nvme: do not advertise support for unsupported arbitration mechanism
   iotests.py: do not use infinite waits
   QEMUMachine: add events_wait method
   iotests.py: rewrite run_job to be pickier
   iotests: add iotest 256 for testing blockdev-backup across iothread
     contexts
   event_match: always match on None value
-Max Reitz (12):
+Max Reitz (1):
-  iotests: Filter 175's allocation information
+  iotests: Fix 205 for concurrent runs
-  iotests: Fix intermittent failure in 219
-  qemu-img: Fix options leakage in img_rebase()
+Pino Toscano (1):
-  qapi/block-core: Overlays are not snapshots
+  ssh: switch from libssh2 to libssh
-  blockdev: Overlays are not snapshots
-  qemu-img: Move quiet into ImgConvertState
+Sam Eiderman (3):
-  qemu-img: Add salvaging mode to convert
+  vmdk: Fix comment regarding max l1_size coverage
-  blkdebug: Add @iotype error option
+  vmdk: Reduce the max bound for L1 table size
-  blkdebug: Add "none" event
+  vmdk: Add read-only support for seSparse snapshots
   blkdebug: Inject errors on .bdrv_co_block_status()
   iotests: Test qemu-img convert --salvage
   iotests: Test qemu-img convert -C --salvage
 Vladimir Sementsov-Ogievskiy (1):
-  iotests: restrict 254 to support only qcow2
+  blockdev: enable non-root nodes for transaction drive-backup source
- qapi/block-core.json          |  53 ++++++++---
+ configure                                     |  65 +-
- block/blkdebug.c              |  60 ++++++++++--
+ block/Makefile.objs                           |   6 +-
- blockdev.c                    |  14 +--
+ block/ssh.c                                   | 652 ++++++++++--------
- hw/block/fdc.c                |   1 +
+ block/vmdk.c                                  | 372 +++++++++-
- qemu-img.c                    | 106 +++++++++++++++------
+ blockdev.c                                    |   2 +-
- python/qemu/__init__.py       |  67 ++++++++++----
+ hw/block/nvme.c                               |   1 -
- qemu-img-cmds.hx              |   4 +-
+ .travis.yml                                   |   4 +-
- qemu-img.texi                 |   4 +
+ block/trace-events                            |  14 +-
- tests/qemu-iotests/082        |   1 +
+ docs/qemu-block-drivers.texi                  |   2 +-
- tests/qemu-iotests/082.out    |   3 +
+ .../dockerfiles/debian-win32-cross.docker     |   1 -
- tests/qemu-iotests/085.out    |  10 +-
+ .../dockerfiles/debian-win64-cross.docker     |   1 -
- tests/qemu-iotests/175        |  26 +++++-
+ tests/docker/dockerfiles/fedora.docker        |   4 +-
- tests/qemu-iotests/175.out    |   8 +-
+ tests/docker/dockerfiles/ubuntu.docker        |   2 +-
- tests/qemu-iotests/219        |  13 ++-
+ tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
- tests/qemu-iotests/251        | 170 ++++++++++++++++++++++++++++++++++
+ tests/qemu-iotests/059.out                    |   2 +-
- tests/qemu-iotests/251.out    |  43 +++++++++
+ tests/qemu-iotests/134                        |   9 +
- tests/qemu-iotests/254        |   2 +
+ tests/qemu-iotests/134.out                    |  10 +
- tests/qemu-iotests/256        | 122 ++++++++++++++++++++++++
+ tests/qemu-iotests/205                        |   2 +-
- tests/qemu-iotests/256.out    | 119 ++++++++++++++++++++++++
+ tests/qemu-iotests/207                        |  54 +-
- tests/qemu-iotests/group      |   2 +
+ tests/qemu-iotests/207.out                    |   2 +-
- tests/qemu-iotests/iotests.py |  60 +++++++-----
+files changed, 823 insertions(+), 384 deletions(-)
 files changed, 772 insertions(+), 116 deletions(-)
  create mode 100755 tests/qemu-iotests/251
  create mode 100644 tests/qemu-iotests/251.out
  create mode 100755 tests/qemu-iotests/256
  create mode 100644 tests/qemu-iotests/256.out
 --
 .21.0

-[Qemu-devel] [PULL 20/20] iotests: Test qemu-img convert -C --salvage
+[Qemu-devel] [PULL v2 1/8] nvme: do not advertise support for unsupported arbitration mechanism
-We do not support this combination (yet), so this should yield an error
+From: Klaus Birkelund Jensen <klaus@birkelund.eu>
 message.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+The device mistakenly reports that the Weighted Round Robin with Urgent
-Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Priority Class arbitration mechanism is supported.
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20190507203508.18026-8-mreitz@redhat.com
+It is not.
 Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
 Message-id: 20190606092530.14206-1-klaus@birkelund.eu
 Acked-by: Maxim Levitsky <mlevitsk@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/082     | 1 +
+ hw/block/nvme.c | 1 -
- tests/qemu-iotests/082.out | 3 +++
+file changed, 1 deletion(-)
 files changed, 4 insertions(+)
-diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082
+diff --git a/hw/block/nvme.c b/hw/block/nvme.c
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/082
 +++ b/tests/qemu-iotests/082
@@ -XXX,XX +XXX,XX @@ echo === convert: -C and other options ===
  run_qemu_img convert -C -S 4k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
  run_qemu_img convert -C -S 8k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
  run_qemu_img convert -C -c -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 +run_qemu_img convert -C --salvage -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
  echo
  echo === amend: Options specified more than once ===
 diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/082.out
+--- a/hw/block/nvme.c
-+++ b/tests/qemu-iotests/082.out
++++ b/hw/block/nvme.c
-@@ -XXX,XX +XXX,XX @@ qemu-img: Cannot enable copy offloading when -S is used
+@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
- Testing: convert -C -c -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
+     n->bar.cap = 0;
- qemu-img: Cannot enable copy offloading when -c is used
+     NVME_CAP_SET_MQES(n->bar.cap, 0x7ff);
+     NVME_CAP_SET_CQR(n->bar.cap, 1);
-+Testing: convert -C --salvage -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
+-    NVME_CAP_SET_AMS(n->bar.cap, 1);
-+qemu-img: Cannot use copy offloading in salvaging mode
+     NVME_CAP_SET_TO(n->bar.cap, 0xf);
-+
+     NVME_CAP_SET_CSS(n->bar.cap, 1);
- === amend: Options specified more than once ===
+     NVME_CAP_SET_MPSMAX(n->bar.cap, 4);
  Testing: amend -f foo -f qcow2 -o lazy_refcounts=on TEST_DIR/t.qcow2
 --
 .21.0

-[Qemu-devel] [PULL 01/20] blockdev-backup: don't check aio_context too early
+[Qemu-devel] [PULL v2 2/8] blockdev: enable non-root nodes for transaction drive-backup source
-From: John Snow <jsnow@redhat.com>
+From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-in blockdev_backup_prepare, we check to make sure that the target is
+We forget to enable it for transaction .prepare, while it is already
-associated with a compatible aio context. However, do_blockdev_backup is
+enabled in do_drive_backup since commit a2d665c1bc362
-called later and has some logic to move the target to a compatible
+    "blockdev: loosen restrictions on drive-backup source node"
 aio_context. The transaction version will fail certain commands
 needlessly early as a result.
-Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
+Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-will ultimately decide if the contexts are compatible or not.
+Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com
+Reviewed-by: John Snow <jsnow@redhat.com>
 Note: the transaction version has always disallowed this operation since
 its initial commit bd8baecd (2014), whereas the version of
 qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
 enforce the aio_context switch instead. It's not clear, and I can't see
 from the mailing list archives at the time, why the two functions take a
 different approach. It wasn't until later in efd7556708b (2016) that the
 standalone version tried to determine if it could set the context or
 not.
 Reported-by: aihua liang <aliang@redhat.com>
 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
 Signed-off-by: John Snow <jsnow@redhat.com>
 Message-id: 20190523170643.20794-2-jsnow@redhat.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- blockdev.c | 4 ----
+ blockdev.c | 2 +-
-file changed, 4 deletions(-)
+file changed, 1 insertion(+), 1 deletion(-)
 diff --git a/blockdev.c b/blockdev.c
 index XXXXXXX..XXXXXXX 100644
 --- a/blockdev.c
 +++ b/blockdev.c
-@@ -XXX,XX +XXX,XX @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
+@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
      assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);
      backup = common->action->u.drive_backup.data;
 -    bs = qmp_get_root_bs(backup->device, errp);
 +    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
      if (!bs) {
          return;
      }
-     aio_context = bdrv_get_aio_context(bs);
--    if (aio_context != bdrv_get_aio_context(target)) {
--        error_setg(errp, "Backup between two IO threads is not implemented");
--        return;
--    }
-     aio_context_acquire(aio_context);
-     state->bs = bs;
 --
 .21.0

-[Qemu-devel] [PULL 19/20] iotests: Test qemu-img convert --salvage
+[Qemu-devel] [PULL v2 3/8] iotest 134: test cluster-misaligned encrypted write
-This test converts a simple image to another, but blkdebug injects
+From: Anton Nefedov <anton.nefedov@virtuozzo.com>
 block_status and read faults at some offsets.  The resulting image
 should be the same as the input image, except that sectors that could
 not be read have to be 0.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+COW (even empty/zero) areas require encryption too
-Message-id: 20190507203508.18026-7-mreitz@redhat.com
-Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Reviewed-by: Eric Blake <eblake@redhat.com>
-[mreitz: Dropped superfluous printf from _filter_offsets, as suggested
+Reviewed-by: Max Reitz <mreitz@redhat.com>
-         by Vladimir; disable test for VDI and IMGOPTSSYNTAX]
+Reviewed-by: Alberto Garcia <berto@igalia.com>
 Message-id: 20190516143028.81155-1-anton.nefedov@virtuozzo.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/251     | 170 +++++++++++++++++++++++++++++++++++++
+ tests/qemu-iotests/134     |  9 +++++++++
- tests/qemu-iotests/251.out |  43 ++++++++++
+ tests/qemu-iotests/134.out | 10 ++++++++++
- tests/qemu-iotests/group   |   1 +
+files changed, 19 insertions(+)
 files changed, 214 insertions(+)
  create mode 100755 tests/qemu-iotests/251
  create mode 100644 tests/qemu-iotests/251.out
-diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
+diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
-new file mode 100755
+index XXXXXXX..XXXXXXX 100755
-index XXXXXXX..XXXXXXX
+--- a/tests/qemu-iotests/134
---- /dev/null
++++ b/tests/qemu-iotests/134
-+++ b/tests/qemu-iotests/251
+@@ -XXX,XX +XXX,XX @@ echo
-@@ -XXX,XX +XXX,XX @@
+ echo "== reading whole image =="
-+#!/usr/bin/env bash
+ $QEMU_IO --object $SECRET -c "read 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
-+#
-+# Test qemu-img convert --salvage
++echo
-+#
++echo "== rewriting cluster part =="
-+# Copyright (C) 2019 Red Hat, Inc.
++$QEMU_IO --object $SECRET -c "write -P 0xb 512 512" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 +#
 +# This program is free software; you can redistribute it and/or modify
 +# it under the terms of the GNU General Public License as published by
 +# the Free Software Foundation; either version 2 of the License, or
 +# (at your option) any later version.
 +#
 +# This program is distributed in the hope that it will be useful,
 +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 +# GNU General Public License for more details.
 +#
 +# You should have received a copy of the GNU General Public License
 +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 +#
 +
 +# creator
 +owner=mreitz@redhat.com
 +
 +seq=$(basename $0)
 +echo "QA output created by $seq"
 +
 +status=1    # failure is the default!
 +
 +_cleanup()
 +{
 +    _cleanup_test_img
 +}
 +trap "_cleanup; exit \$status" 0 1 2 3 15
 +
 +# get standard environment, filters and checks
 +. ./common.rc
 +. ./common.filter
 +. ./common.qemu
 +
 +_supported_fmt generic
 +_supported_proto file
 +_supported_os Linux
 +
 +if [ "$IMGOPTSSYNTAX" = "true" ]; then
 +    # We use json:{} filenames here, so we cannot work with additional options.
 +    _unsupported_fmt $IMGFMT
 +else
 +    # With VDI, the output is ordered differently.  Just disable it.
 +    _unsupported_fmt vdi
 +fi
 +
 +
 +TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
 +
 +$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
 +
 +
 +sector_size=512
 +
 +# Offsets on which to fail block-status.  Keep in ascending order so
 +# the indexing done by _filter_offsets will appear in ascending order
 +# in the output as well.
 +status_fail_offsets="$((16 * 1024 * 1024 + 8192))
 +                     $((33 * 1024 * 1024 + 512))"
 +
 +# Offsets on which to fail reads.  Keep in ascending order for the
 +# same reason.
 +# The second element is shared with $status_fail_offsets on purpose.
 +# Starting with the third element, we test what happens when a
 +# continuous range of sectors is inaccessible.
 +read_fail_offsets="$((32 * 1024 * 1024 - 65536))
 +                   $((33 * 1024 * 1024 + 512))
 +                   $(seq $((34 * 1024 * 1024)) $sector_size \
 +                         $((34 * 1024 * 1024 + 4096 - $sector_size)))"
 +
 +
 +# blkdebug must be above the format layer so it can intercept all
 +# block-status events
 +source_img="json:{'driver': 'blkdebug',
 +                  'image': {
 +                      'driver': '$IMGFMT',
 +                      'file': {
 +                          'driver': 'file',
 +                          'filename': '$TEST_IMG.orig'
 +                      }
 +                  },
 +                  'inject-error': ["
 +
 +for ofs in $status_fail_offsets
 +do
 +    source_img+="{ 'event': 'none',
 +                   'iotype': 'block-status',
 +                   'errno': 5,
 +                   'sector': $((ofs / sector_size)) },"
 +done
 +
 +for ofs in $read_fail_offsets
 +do
 +    source_img+="{ 'event': 'none',
 +                   'iotype': 'read',
 +                   'errno': 5,
 +                   'sector': $((ofs / sector_size)) },"
 +done
 +
 +# Remove the trailing comma and terminate @inject-error and json:{}
 +source_img="${source_img%,} ] }"
 +
 +
 +echo
++echo "== verify pattern =="
++$QEMU_IO --object $SECRET -c "read -P 0 0 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
++$QEMU_IO --object $SECRET -c "read -P 0xb 512 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 +
-+
+ echo
-+_filter_offsets() {
+ echo "== rewriting whole image =="
-+    filters=
+ $QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
-+
+diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out
-+    index=0
+index XXXXXXX..XXXXXXX 100644
-+    for ofs in $1
+--- a/tests/qemu-iotests/134.out
-+    do
++++ b/tests/qemu-iotests/134.out
-+        filters+=" -e s/$ofs/status_fail_offset_$index/"
+@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.
-+        index=$((index + 1))
+ read 134217728/134217728 bytes at offset 0
-+    done
+MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-+
-+    index=0
++== rewriting cluster part ==
-+    for ofs in $2
++wrote 512/512 bytes at offset 512
 +    do
 +        filters+=" -e s/$ofs/read_fail_offset_$index/"
 +        index=$((index + 1))
 +    done
 +
 +    sed $filters
 +}
 +
 +# While determining the number of allocated sectors in the input
 +# image, we should see one block status warning per element of
 +# $status_fail_offsets.
 +#
 +# Then, the image is read.  Since the block status is queried in
 +# basically the same way, the same warnings as in the previous step
 +# should reappear.  Interleaved with those we should see a read
 +# warning per element of $read_fail_offsets.
 +# Note that $read_fail_offsets and $status_fail_offsets share an
 +# element (read_fail_offset_1 == status_fail_offset_1), so
 +# "status_fail_offset_1" in the output is the same as
 +# "read_fail_offset_1".
 +$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
 +    | _filter_offsets "$status_fail_offsets" "$read_fail_offsets"
 +
 +echo
 +
 +# The offsets where the block status could not be determined should
 +# have been treated as containing data and thus should be correct in
 +# the output image.
 +# The offsets where reading failed altogether should be 0.  Make them
 +# 0 in the input image, too, so we can compare both images.
 +for ofs in $read_fail_offsets
 +do
 +    $QEMU_IO -c "write -z $ofs $sector_size" "$TEST_IMG.orig" \
 +        | _filter_qemu_io \
 +        | _filter_offsets '' "$read_fail_offsets"
 +done
 +
 +echo
 +
 +# These should be equal now.
 +$QEMU_IMG compare "$TEST_IMG.orig" "$TEST_IMG"
 +
 +
 +# success, all done
 +echo "*** done"
 +rm -f $seq.full
 +status=0
 diff --git a/tests/qemu-iotests/251.out b/tests/qemu-iotests/251.out
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/tests/qemu-iotests/251.out
@@ -XXX,XX +XXX,XX @@
 +QA output created by 251
 +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
 +wrote 67108864/67108864 bytes at offset 0
 +64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
 +qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
 +qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
 +qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_0: Input/output error
 +qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
 +qemu-img: warning: error while reading offset status_fail_offset_1: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_2: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_3: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_4: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_5: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_6: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_7: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_8: Input/output error
 +qemu-img: warning: error while reading offset read_fail_offset_9: Input/output error
 +
 +wrote 512/512 bytes at offset read_fail_offset_0
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_1
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_2
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_3
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_4
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_5
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_6
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_7
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_8
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +wrote 512/512 bytes at offset read_fail_offset_9
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
-+Images are identical.
++== verify pattern ==
-+*** done
++read 512/512 bytes at offset 0
-diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
++512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-index XXXXXXX..XXXXXXX 100644
++read 512/512 bytes at offset 512
---- a/tests/qemu-iotests/group
++512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-+++ b/tests/qemu-iotests/group
++
-@@ -XXX,XX +XXX,XX @@
+ == rewriting whole image ==
-rw quick
+ wrote 134217728/134217728 bytes at offset 0
-rw auto quick
+MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 rw auto quick
 +251 rw auto quick
 rw auto backing quick
 rw auto quick
 rw auto backing quick
 --
 .21.0

-[Qemu-devel] [PULL 03/20] QEMUMachine: add events_wait method
+[Qemu-devel] [PULL v2 4/8] vmdk: Fix comment regarding max l1_size coverage
-From: John Snow <jsnow@redhat.com>
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-Instead of event_wait which looks for a single event, add an events_wait
+Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")
-which can look for any number of events simultaneously. However, it
+extended the l1_size check from VMDK4 to VMDK3 but did not update the
-will still only return one at a time, whichever happens first.
+default coverage in the moved comment.
-Signed-off-by: John Snow <jsnow@redhat.com>
+The previous vmdk4 calculation:
-Message-id: 20190523170643.20794-4-jsnow@redhat.com
     (512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB
 The added vmdk3 calculation:
     (512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB
 Adding the calculation of vmdk3 to the comment.
 In any case, VMware does not offer virtual disks more than 2TB for
 vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is
 not implemented yet in qemu.
 Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
 Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
 Reviewed-by: Liran Alon <liran.alon@oracle.com>
 Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
 Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
 Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com
 Reviewed-by: yuchenlin <yuchenlin@synology.com>
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
+ block/vmdk.c | 11 ++++++++---
-file changed, 49 insertions(+), 20 deletions(-)
+file changed, 8 insertions(+), 3 deletions(-)
-diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/python/qemu/__init__.py
+--- a/block/vmdk.c
-+++ b/python/qemu/__init__.py
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-         self._qmp.clear_events()
+         return -EFBIG;
-         return events
+     }
+     if (l1_size > 512 * 1024 * 1024) {
--    def event_wait(self, name, timeout=60.0, match=None):
+-        /* Although with big capacity and small l1_entry_sectors, we can get a
-+    @staticmethod
++        /*
-+    def event_match(event, match=None):
++         * Although with big capacity and small l1_entry_sectors, we can get a
-         """
+          * big l1_size, we don't want unbounded value to allocate the table.
--        Wait for specified timeout on named event in QMP; optionally filter
+-         * Limit it to 512M, which is 16PB for default cluster and L2 table
--        results by match.
+-         * size */
-+        Check if an event matches optional match criteria.
++         * Limit it to 512M, which is:
++         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
--        The 'match' is checked to be a recursive subset of the 'event'; skips
++         *            cluster size: 64KB, L2 table size: 512 entries
--        branch processing on match's value None
++         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
--           {"foo": {"bar": 1}} matches {"foo": None}
++         *            cluster size: 512B, L2 table size: 4096 entries
--           {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
++         */
-+        The match criteria takes the form of a matching subdict. The event is
+         error_setg(errp, "L1 size too big");
-+        checked to be a superset of the subdict, recursively, with matching
+         return -EFBIG;
-+        values whenever those values are not None.
+     }
 +
 +        Examples, with the subdict queries on the left:
 +         - None matches any object.
 +         - {"foo": None} matches {"foo": {"bar": 1}}
 +         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
 +         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
          """
 -        def event_match(event, match=None):
 -            if match is None:
 -                return True
 +        if match is None:
 +            return True
 -            for key in match:
 -                if key in event:
 -                    if isinstance(event[key], dict):
 -                        if not event_match(event[key], match[key]):
 -                            return False
 -                    elif event[key] != match[key]:
 +        for key in match:
 +            if key in event:
 +                if isinstance(event[key], dict):
 +                    if not QEMUMachine.event_match(event[key], match[key]):
                          return False
 -                else:
 +                elif event[key] != match[key]:
                      return False
 +            else:
 +                return False
 +        return True
 -            return True
 +    def event_wait(self, name, timeout=60.0, match=None):
 +        """
 +        event_wait waits for and returns a named event from QMP with a timeout.
 +
 +        name: The event to wait for.
 +        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
 +        match: Optional match criteria. See event_match for details.
 +        """
 +        return self.events_wait([(name, match)], timeout)
 +
 +    def events_wait(self, events, timeout=60.0):
 +        """
 +        events_wait waits for and returns a named event from QMP with a timeout.
 +
 +        events: a sequence of (name, match_criteria) tuples.
 +                The match criteria are optional and may be None.
 +                See event_match for details.
 +        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
 +        """
 +        def _match(event):
 +            for name, match in events:
 +                if (event['event'] == name and
 +                    self.event_match(event, match)):
 +                    return True
 +            return False
          # Search cached events
          for event in self._events:
 -            if (event['event'] == name) and event_match(event, match):
 +            if _match(event):
                  self._events.remove(event)
                  return event
          # Poll for new events
          while True:
              event = self._qmp.pull_event(wait=timeout)
 -            if (event['event'] == name) and event_match(event, match):
 +            if _match(event):
                  return event
              self._events.append(event)
 --
 .21.0

-[Qemu-devel] [PULL 02/20] iotests.py: do not use infinite waits
+[Qemu-devel] [PULL v2 5/8] vmdk: Reduce the max bound for L1 table size
-From: John Snow <jsnow@redhat.com>
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-Cap waits to 60 seconds so that iotests can fail gracefully if something
+M of L1 entries is a very loose bound, only 32M are required to store
-goes wrong.
+the maximal supported VMDK file size of 2TB.
-Signed-off-by: John Snow <jsnow@redhat.com>
+Fixed qemu-iotest 59# - now failure occures before on impossible L1
-Message-id: 20190523170643.20794-3-jsnow@redhat.com
+table size.
 Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
 Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
 Reviewed-by: Liran Alon <liran.alon@oracle.com>
 Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
 Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
 Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/iotests.py | 14 +++++++-------
+ block/vmdk.c               | 13 +++++++------
-file changed, 7 insertions(+), 7 deletions(-)
+ tests/qemu-iotests/059.out |  2 +-
 files changed, 8 insertions(+), 7 deletions(-)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
+--- a/block/vmdk.c
-+++ b/tests/qemu-iotests/iotests.py
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-             output_list += [key + '=' + obj[key]]
+         error_setg(errp, "Invalid granularity, image may be corrupt");
-         return ','.join(output_list)
+         return -EFBIG;
+     }
--    def get_qmp_events_filtered(self, wait=True):
+-    if (l1_size > 512 * 1024 * 1024) {
-+    def get_qmp_events_filtered(self, wait=60.0):
++    if (l1_size > 32 * 1024 * 1024) {
-         result = []
+         /*
-         for ev in self.get_qmp_events(wait=wait):
+          * Although with big capacity and small l1_entry_sectors, we can get a
-             result.append(filter_qmp_event(ev))
+          * big l1_size, we don't want unbounded value to allocate the table.
-@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
+-         * Limit it to 512M, which is:
+-         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
-     # Returns None on success, and an error string on failure
+-         *            cluster size: 64KB, L2 table size: 512 entries
-     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
+-         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
--                pre_finalize=None):
+-         *            cluster size: 512B, L2 table size: 4096 entries
-+                pre_finalize=None, wait=60.0):
++         * Limit it to 32M, which is enough to store:
-         error = None
++         *     8TB  - for both VMDK3 & VMDK4 with
-         while True:
++         *            minimal cluster size: 512B
--            for ev in self.get_qmp_events_filtered(wait=True):
++         *            minimal L2 table size: 512 entries
-+            for ev in self.get_qmp_events_filtered(wait=wait):
++         *            8 TB is still more than the maximal value supported for
-                 if ev['event'] == 'JOB_STATUS_CHANGE':
++         *            VMDK3 & VMDK4 which is 2TB.
-                     status = ev['data']['status']
+          */
-                     if status == 'aborting':
+         error_setg(errp, "L1 size too big");
-@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
+         return -EFBIG;
-         self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])),
+diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
-                          self.vm.flatten_qmp_object(reference))
+index XXXXXXX..XXXXXXX 100644
+--- a/tests/qemu-iotests/059.out
--    def cancel_and_wait(self, drive='drive0', force=False, resume=False):
++++ b/tests/qemu-iotests/059.out
-+    def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0):
+@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
-         '''Cancel a block job and wait for it to finish, returning the event'''
+x140000000     0x10000         0x50000         TEST_DIR/t-s003.vmdk
-         result = self.vm.qmp('block-job-cancel', device=drive, force=force)
-         self.assert_qmp(result, 'return', {})
+ === Testing afl image with a very large capacity ===
-@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
+-qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large
-         cancelled = False
++qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big
-         result = None
+ *** done
          while not cancelled:
 -            for event in self.vm.get_qmp_events(wait=True):
 +            for event in self.vm.get_qmp_events(wait=wait):
                  if event['event'] == 'BLOCK_JOB_COMPLETED' or \
                     event['event'] == 'BLOCK_JOB_CANCELLED':
                      self.assert_qmp(event, 'data/device', drive)
@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
          self.assert_no_active_block_jobs()
          return result
 -    def wait_until_completed(self, drive='drive0', check_offset=True):
 +    def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0):
          '''Wait for a block job to finish, returning the event'''
          while True:
 -            for event in self.vm.get_qmp_events(wait=True):
 +            for event in self.vm.get_qmp_events(wait=wait):
                  if event['event'] == 'BLOCK_JOB_COMPLETED':
                      self.assert_qmp(event, 'data/device', drive)
                      self.assert_qmp_absent(event, 'data/error')
 --
 .21.0

-[Qemu-devel] [PULL 04/20] iotests.py: rewrite run_job to be pickier
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Don't pull events out of the queue that don't belong to us;
-be choosier so that we can use this method to drive jobs that
-were launched by transactions that may have more jobs.
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190523170643.20794-5-jsnow@redhat.com
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++--------------
-file changed, 29 insertions(+), 19 deletions(-)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
-     # Returns None on success, and an error string on failure
-     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
-                 pre_finalize=None, wait=60.0):
-+        match_device = {'data': {'device': job}}
-+        match_id = {'data': {'id': job}}
-+        events = [
-+            ('BLOCK_JOB_COMPLETED', match_device),
-+            ('BLOCK_JOB_CANCELLED', match_device),
-+            ('BLOCK_JOB_ERROR', match_device),
-+            ('BLOCK_JOB_READY', match_device),
-+            ('BLOCK_JOB_PENDING', match_id),
-+            ('JOB_STATUS_CHANGE', match_id)
-+        ]
-         error = None
-         while True:
--            for ev in self.get_qmp_events_filtered(wait=wait):
--                if ev['event'] == 'JOB_STATUS_CHANGE':
--                    status = ev['data']['status']
--                    if status == 'aborting':
--                        result = self.qmp('query-jobs')
--                        for j in result['return']:
--                            if j['id'] == job:
--                                error = j['error']
--                                log('Job failed: %s' % (j['error']))
--                    elif status == 'pending' and not auto_finalize:
--                        if pre_finalize:
--                            pre_finalize()
--                        self.qmp_log('job-finalize', id=job)
--                    elif status == 'concluded' and not auto_dismiss:
--                        self.qmp_log('job-dismiss', id=job)
--                    elif status == 'null':
--                        return error
--                else:
--                    log(ev)
-+            ev = filter_qmp_event(self.events_wait(events))
-+            if ev['event'] != 'JOB_STATUS_CHANGE':
-+                log(ev)
-+                continue
-+            status = ev['data']['status']
-+            if status == 'aborting':
-+                result = self.qmp('query-jobs')
-+                for j in result['return']:
-+                    if j['id'] == job:
-+                        error = j['error']
-+                        log('Job failed: %s' % (j['error']))
-+            elif status == 'pending' and not auto_finalize:
-+                if pre_finalize:
-+                    pre_finalize()
-+                self.qmp_log('job-finalize', id=job)
-+            elif status == 'concluded' and not auto_dismiss:
-+                self.qmp_log('job-dismiss', id=job)
-+            elif status == 'null':
-+                return error
-     def node_info(self, node_name):
-         nodes = self.qmp('query-named-block-nodes')
---
-.21.0

-[Qemu-devel] [PULL 05/20] iotests: add iotest 256 for testing blockdev-backup across iothread contexts
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190523170643.20794-6-jsnow@redhat.com
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-[mreitz: Moved from 250 to 256]
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/256     | 122 +++++++++++++++++++++++++++++++++++++
- tests/qemu-iotests/256.out | 119 ++++++++++++++++++++++++++++++++++++
- tests/qemu-iotests/group   |   1 +
-files changed, 242 insertions(+)
- create mode 100755 tests/qemu-iotests/256
- create mode 100644 tests/qemu-iotests/256.out
-diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
-new file mode 100755
-index XXXXXXX..XXXXXXX
---- /dev/null
-+++ b/tests/qemu-iotests/256
-@@ -XXX,XX +XXX,XX @@
-+#!/usr/bin/env python
-+#
-+# Test incremental/backup across iothread contexts
-+#
-+# Copyright (c) 2019 John Snow for Red Hat, Inc.
-+#
-+# This program is free software; you can redistribute it and/or modify
-+# it under the terms of the GNU General Public License as published by
-+# the Free Software Foundation; either version 2 of the License, or
-+# (at your option) any later version.
-+#
-+# This program is distributed in the hope that it will be useful,
-+# but WITHOUT ANY WARRANTY; without even the implied warranty of
-+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-+# GNU General Public License for more details.
-+#
-+# You should have received a copy of the GNU General Public License
-+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-+#
-+# owner=jsnow@redhat.com
-+
-+import os
-+import iotests
-+from iotests import log
-+
-+iotests.verify_image_format(supported_fmts=['qcow2'])
-+size = 64 * 1024 * 1024
-+
-+with iotests.FilePath('img0') as img0_path, \
-+     iotests.FilePath('img1') as img1_path, \
-+     iotests.FilePath('img0-full') as img0_full_path, \
-+     iotests.FilePath('img1-full') as img1_full_path, \
-+     iotests.FilePath('img0-incr') as img0_incr_path, \
-+     iotests.FilePath('img1-incr') as img1_incr_path, \
-+     iotests.VM() as vm:
-+
-+    def create_target(filepath, name, size):
-+        basename = os.path.basename(filepath)
-+        nodename = "file_{}".format(basename)
-+        log(vm.command('blockdev-create', job_id='job1',
-+                       options={
-+                           'driver': 'file',
-+                           'filename': filepath,
-+                           'size': 0,
-+                       }))
-+        vm.run_job('job1')
-+        log(vm.command('blockdev-add', driver='file',
-+                       node_name=nodename, filename=filepath))
-+        log(vm.command('blockdev-create', job_id='job2',
-+                       options={
-+                           'driver': iotests.imgfmt,
-+                           'file': nodename,
-+                           'size': size,
-+                       }))
-+        vm.run_job('job2')
-+        log(vm.command('blockdev-add', driver=iotests.imgfmt,
-+                       node_name=name,
-+                       file=nodename))
-+
-+    log('--- Preparing images & VM ---\n')
-+    vm.add_object('iothread,id=iothread0')
-+    vm.add_object('iothread,id=iothread1')
-+    vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
-+    vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
-+    iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
-+    iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
-+    vm.add_drive(img0_path, interface='none')
-+    vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
-+    vm.add_drive(img1_path, interface='none')
-+    vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0')
-+
-+    log('--- Starting VM ---\n')
-+    vm.launch()
-+
-+    log('--- Create Targets & Full Backups ---\n')
-+    create_target(img0_full_path, 'img0-full', size)
-+    create_target(img1_full_path, 'img1-full', size)
-+    ret = vm.qmp_log('transaction', indent=2, actions=[
-+        { 'type': 'block-dirty-bitmap-add',
-+          'data': { 'node': 'drive0', 'name': 'bitmap0' }},
-+        { 'type': 'block-dirty-bitmap-add',
-+          'data': { 'node': 'drive1', 'name': 'bitmap1' }},
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive0',
-+                    'target': 'img0-full',
-+                    'sync': 'full',
-+                    'job-id': 'j0' }},
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive1',
-+                    'target': 'img1-full',
-+                    'sync': 'full',
-+                    'job-id': 'j1' }}
-+    ])
-+    if "error" in ret:
-+        raise Exception(ret['error']['desc'])
-+    vm.run_job('j0', auto_dismiss=True)
-+    vm.run_job('j1', auto_dismiss=True)
-+
-+    log('\n--- Create Targets & Incremental Backups ---\n')
-+    create_target(img0_incr_path, 'img0-incr', size)
-+    create_target(img1_incr_path, 'img1-incr', size)
-+    ret = vm.qmp_log('transaction', indent=2, actions=[
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive0',
-+                    'target': 'img0-incr',
-+                    'sync': 'incremental',
-+                    'bitmap': 'bitmap0',
-+                    'job-id': 'j2' }},
-+        { 'type': 'blockdev-backup',
-+          'data': { 'device': 'drive1',
-+                    'target': 'img1-incr',
-+                    'sync': 'incremental',
-+                    'bitmap': 'bitmap1',
-+                    'job-id': 'j3' }}
-+    ])
-+    if "error" in ret:
-+        raise Exception(ret['error']['desc'])
-+    vm.run_job('j2', auto_dismiss=True)
-+    vm.run_job('j3', auto_dismiss=True)
-+
-+    log('\n--- Done ---')
-+    vm.shutdown()
-diff --git a/tests/qemu-iotests/256.out b/tests/qemu-iotests/256.out
-new file mode 100644
-index XXXXXXX..XXXXXXX
---- /dev/null
-+++ b/tests/qemu-iotests/256.out
-@@ -XXX,XX +XXX,XX @@
-+--- Preparing images & VM ---
-+
-+--- Starting VM ---
-+
-+--- Create Targets & Full Backups ---
-+
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{
-+  "execute": "transaction",
-+  "arguments": {
-+    "actions": [
-+      {
-+        "data": {
-+          "name": "bitmap0",
-+          "node": "drive0"
-+        },
-+        "type": "block-dirty-bitmap-add"
-+      },
-+      {
-+        "data": {
-+          "name": "bitmap1",
-+          "node": "drive1"
-+        },
-+        "type": "block-dirty-bitmap-add"
-+      },
-+      {
-+        "data": {
-+          "device": "drive0",
-+          "job-id": "j0",
-+          "sync": "full",
-+          "target": "img0-full"
-+        },
-+        "type": "blockdev-backup"
-+      },
-+      {
-+        "data": {
-+          "device": "drive1",
-+          "job-id": "j1",
-+          "sync": "full",
-+          "target": "img1-full"
-+        },
-+        "type": "blockdev-backup"
-+      }
-+    ]
-+  }
-+}
-+{
-+  "return": {}
-+}
-+{"data": {"device": "j0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+{"data": {"device": "j1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+
-+--- Create Targets & Incremental Backups ---
-+
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
-+{"return": {}}
-+{}
-+{}
-+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
-+{"return": {}}
-+{}
-+{
-+  "execute": "transaction",
-+  "arguments": {
-+    "actions": [
-+      {
-+        "data": {
-+          "bitmap": "bitmap0",
-+          "device": "drive0",
-+          "job-id": "j2",
-+          "sync": "incremental",
-+          "target": "img0-incr"
-+        },
-+        "type": "blockdev-backup"
-+      },
-+      {
-+        "data": {
-+          "bitmap": "bitmap1",
-+          "device": "drive1",
-+          "job-id": "j3",
-+          "sync": "incremental",
-+          "target": "img1-incr"
-+        },
-+        "type": "blockdev-backup"
-+      }
-+    ]
-+  }
-+}
-+{
-+  "return": {}
-+}
-+{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-+
-+--- Done ---
-diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/group
-+++ b/tests/qemu-iotests/group
-@@ -XXX,XX +XXX,XX @@
-rw auto quick
-rw auto backing quick
-rw auto quick
-+256 rw auto quick
---
-.21.0

-[Qemu-devel] [PULL 06/20] event_match: always match on None value
+Deleted patch
-From: John Snow <jsnow@redhat.com>
-Before, event_match didn't always recurse if the event value was not a
-dictionary, and would instead check for equality immediately.
-By delaying equality checking to post-recursion, we can allow leaf
-values like "5" to match "None" and take advantage of the generic
-None-returns-True clause.
-This makes the matching a little more obviously consistent at the
-expense of being able to check for explicit None values, which is
-probably not that important given what this function is used for.
-Signed-off-by: John Snow <jsnow@redhat.com>
-Message-id: 20190528183857.26167-1-jsnow@redhat.com
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- python/qemu/__init__.py | 24 ++++++++++++++----------
-file changed, 14 insertions(+), 10 deletions(-)
-diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
-index XXXXXXX..XXXXXXX 100644
---- a/python/qemu/__init__.py
-+++ b/python/qemu/__init__.py
-@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
-         The match criteria takes the form of a matching subdict. The event is
-         checked to be a superset of the subdict, recursively, with matching
--        values whenever those values are not None.
-+        values whenever the subdict values are not None.
-+
-+        This has a limitation that you cannot explicitly check for None values.
-         Examples, with the subdict queries on the left:
-          - None matches any object.
-          - {"foo": None} matches {"foo": {"bar": 1}}
--         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
--         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
-+         - {"foo": None} matches {"foo": 5}
-+         - {"foo": {"abc": None}} does not match {"foo": {"bar": 1}}
-+         - {"foo": {"rab": 2}} matches {"foo": {"bar": 1, "rab": 2}}
-         """
-         if match is None:
-             return True
--        for key in match:
--            if key in event:
--                if isinstance(event[key], dict):
-+        try:
-+            for key in match:
-+                if key in event:
-                     if not QEMUMachine.event_match(event[key], match[key]):
-                         return False
--                elif event[key] != match[key]:
-+                else:
-                     return False
--            else:
--                return False
--        return True
-+            return True
-+        except TypeError:
-+            # either match or event wasn't iterable (not a dict)
-+            return match == event
-     def event_wait(self, name, timeout=60.0, match=None):
-         """
---
-.21.0

-[Qemu-devel] [PULL 07/20] iotests: Filter 175's allocation information
+Deleted patch
-It is possible for an empty file to take up blocks on a filesystem, for
-example:
-$ qemu-img create -f raw test.img 1G
-Formatting 'test.img', fmt=raw size=1073741824
-$ mkfs.ext4 -I 128 -q test.img
-$ mkdir test-mount
-$ sudo mount -o loop test.img test-mount
-$ sudo touch test-mount/test-file
-$ stat -c 'blocks=%b' test-mount/test-file
-blocks=8
-These extra blocks (one cluster) are apparently used for metadata,
-because they are always there, on top of blocks used for data:
-$ sudo dd if=/dev/zero of=test-mount/test-file bs=1M count=1
-+0 records in
-+0 records out
-1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00135339 s, 775 MB/s
-$ stat -c 'blocks=%b' test-mount/test-file
-blocks=2056
-Make iotest 175 take this into account.
-Reported-by: Thomas Huth <thuth@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Reviewed-by: Nir Soffer <nsoffer@redhat.com>
-Message-id: 20190516144319.12570-1-mreitz@redhat.com
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/175     | 26 ++++++++++++++++++++++----
- tests/qemu-iotests/175.out |  8 ++++----
-files changed, 26 insertions(+), 8 deletions(-)
-diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/175
-+++ b/tests/qemu-iotests/175
-@@ -XXX,XX +XXX,XX @@ status=1    # failure is the default!
- _cleanup()
- {
--    _cleanup_test_img
-+    _cleanup_test_img
-+    rm -f "$TEST_DIR/empty"
- }
- trap "_cleanup; exit \$status" 0 1 2 3 15
-+# Some file systems sometimes allocate extra blocks independently of
-+# the file size.  This function hides the resulting difference in the
-+# stat -c '%b' output.
-+# Parameter 1: Number of blocks an empty file occupies
-+# Parameter 2: Image size in bytes
-+_filter_blocks()
-+{
-+    extra_blocks=$1
-+    img_size=$2
-+
-+    sed -e "s/blocks=$extra_blocks\\(\$\\|[^0-9]\\)/nothing allocated/" \
-+        -e "s/blocks=$((extra_blocks + img_size / 512))\\(\$\\|[^0-9]\\)/everything allocated/"
-+}
-+
- # get standard environment, filters and checks
- . ./common.rc
- . ./common.filter
-@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
- _supported_proto file
- _supported_os Linux
--size=1m
-+size=$((1 * 1024 * 1024))
-+
-+touch "$TEST_DIR/empty"
-+extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
- echo
- echo "== creating image with default preallocation =="
- _make_test_img $size | _filter_imgfmt
--stat -c "size=%s, blocks=%b" $TEST_IMG
-+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
- for mode in off full falloc; do
-     echo
-     echo "== creating image with preallocation $mode =="
-     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
--    stat -c "size=%s, blocks=%b" $TEST_IMG
-+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
- done
- # success, all done
-diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/175.out
-+++ b/tests/qemu-iotests/175.out
-@@ -XXX,XX +XXX,XX @@ QA output created by 175
- == creating image with default preallocation ==
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
--size=1048576, blocks=0
-+size=1048576, nothing allocated
- == creating image with preallocation off ==
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
--size=1048576, blocks=0
-+size=1048576, nothing allocated
- == creating image with preallocation full ==
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
--size=1048576, blocks=2048
-+size=1048576, everything allocated
- == creating image with preallocation falloc ==
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
--size=1048576, blocks=2048
-+size=1048576, everything allocated
-  *** done
---
-.21.0

-[Qemu-devel] [PULL 08/20] iotests: Fix intermittent failure in 219
+Deleted patch
-In 219, we wait for the job to make progress before we emit its status.
-This makes the output reliable.  We do not wait for any more progress if
-the job's current-progress already matches its total-progress.
-Unfortunately, there is a bug: Right after the job has been started,
-it's possible that total-progress is still 0.  In that case, we may skip
-the first progress-making step and keep ending up 64 kB short.
-To fix that bug, we can simply wait for total-progress to reach 4 MB
-(the image size) after starting the job.
-Reported-by: Karen Mezick <kmezick@redhat.com>
-Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1686651
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20190516161114.27596-1-mreitz@redhat.com
-Reviewed-by: John Snow <jsnow@redhat.com>
-[mreitz: Adjusted commit message as per John's proposal]
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/219 | 13 ++++++++++---
-file changed, 10 insertions(+), 3 deletions(-)
-diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/219
-+++ b/tests/qemu-iotests/219
-@@ -XXX,XX +XXX,XX @@ import iotests
- iotests.verify_image_format(supported_fmts=['qcow2'])
-+img_size = 4 * 1024 * 1024
-+
- def pause_wait(vm, job_id):
-     with iotests.Timeout(3, "Timeout waiting for job to pause"):
-         while True:
-@@ -XXX,XX +XXX,XX @@ def test_pause_resume(vm):
-                 iotests.log(vm.qmp('query-jobs'))
- def test_job_lifecycle(vm, job, job_args, has_ready=False):
-+    global img_size
-+
-     iotests.log('')
-     iotests.log('')
-     iotests.log('Starting block job: %s (auto-finalize: %s; auto-dismiss: %s)' %
-@@ -XXX,XX +XXX,XX @@ def test_job_lifecycle(vm, job, job_args, has_ready=False):
-     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
-     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
-+    # Wait for total-progress to stabilize
-+    while vm.qmp('query-jobs')['return'][0]['total-progress'] < img_size:
-+        pass
-+
-     # RUNNING state:
-     # pause/resume should work, complete/finalize/dismiss should error out
-     iotests.log('')
-@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('disk.img') as disk_path, \
-      iotests.FilePath('copy.img') as copy_path, \
-      iotests.VM() as vm:
--    img_size = '4M'
--    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, img_size)
--    iotests.qemu_io('-c', 'write 0 %s' % (img_size),
-+    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, str(img_size))
-+    iotests.qemu_io('-c', 'write 0 %i' % (img_size),
-                     '-f', iotests.imgfmt, disk_path)
-     iotests.log('Launching VM...')
---
-.21.0

-[Qemu-devel] [PULL 09/20] hw/block/fdc: floppy command FIFO memory initialization
+Deleted patch
-From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
-The uninitialized memory allocated for the command FIFO of the
-floppy controller during the VM hardware initialization incurs
-many unwanted reports by Valgrind when VM state is being saved.
-That verbosity hardens a search for the real memory issues when
-the iotests run. Particularly, the patch eliminates 20 unnecessary
-reports of the Valgrind tool in the iotest #169.
-Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
-Message-id: 1559154027-282547-1-git-send-email-andrey.shinkevich@virtuozzo.com
-Reviewed-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- hw/block/fdc.c | 1 +
-file changed, 1 insertion(+)
-diff --git a/hw/block/fdc.c b/hw/block/fdc.c
-index XXXXXXX..XXXXXXX 100644
---- a/hw/block/fdc.c
-+++ b/hw/block/fdc.c
-@@ -XXX,XX +XXX,XX @@ static void fdctrl_realize_common(DeviceState *dev, FDCtrl *fdctrl,
-     FLOPPY_DPRINTF("init controller\n");
-     fdctrl->fifo = qemu_memalign(512, FD_SECTOR_LEN);
-+    memset(fdctrl->fifo, 0, FD_SECTOR_LEN);
-     fdctrl->fifo_size = 512;
-     fdctrl->result_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
-                                              fdctrl_result_timer, fdctrl);
---
-.21.0

-[Qemu-devel] [PULL 10/20] iotests: restrict 254 to support only qcow2
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Test fails at least for qcow, because of different cluster sizes in
-base and top (and therefore different granularities of bitmaps we are
-trying to merge).
-The test aim is to check block-dirty-bitmap-merge between different
-nodes functionality, no needs to check all formats. So, let's just drop
-support for anything except qcow2.
-Reported-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20190605155405.104384-1-vsementsov@virtuozzo.com
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/254 | 2 ++
-file changed, 2 insertions(+)
-diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/254
-+++ b/tests/qemu-iotests/254
-@@ -XXX,XX +XXX,XX @@
- import iotests
- from iotests import qemu_img_create, file_path, log
-+iotests.verify_image_format(supported_fmts=['qcow2'])
-+
- disk, top = file_path('disk', 'top')
- size = 1024 * 1024
---
-.21.0

-[Qemu-devel] [PULL 11/20] qemu-img: Fix options leakage in img_rebase()
+Deleted patch
-img_rebase() can leak a QDict in two occasions.  Fix it.
-Coverity: CID 1401416
-Fixes: d16699b64671466b42079c45b89127aeea1ca565
-Fixes: 330c72957196e0ae382abcaa97ebf4eb9bc8574f
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Message-id: 20190528195338.12376-1-mreitz@redhat.com
-Reviewed-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- qemu-img.c | 3 +++
-file changed, 3 insertions(+)
-diff --git a/qemu-img.c b/qemu-img.c
-index XXXXXXX..XXXXXXX 100644
---- a/qemu-img.c
-+++ b/qemu-img.c
-@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
-                                                              out_baseimg,
-                                                              &local_err);
-             if (local_err) {
-+                qobject_unref(options);
-                 error_reportf_err(local_err,
-                                   "Could not resolve backing filename: ");
-                 ret = -1;
-@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
-              */
-             prefix_chain_bs = bdrv_find_backing_image(bs, out_real_path);
-             if (prefix_chain_bs) {
-+                qobject_unref(options);
-                 g_free(out_real_path);
-+
-                 blk_new_backing = blk_new(qemu_get_aio_context(),
-                                           BLK_PERM_CONSISTENT_READ,
-                                           BLK_PERM_ALL);
---
-.21.0

-[Qemu-devel] [PULL 12/20] qapi/block-core: Overlays are not snapshots
+Deleted patch
-A snapshot is something that reflects the state of something at a
-certain point in time.  It does not change.
-The file our snapshot commands create (or the node they install) is not
-a snapshot, as it does change over time.  It is an overlay.  We cannot
-do anything about the parameter names, but we can at least adjust the
-descriptions to reflect that fact.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Message-id: 20190603202236.1342-2-mreitz@redhat.com
-Reviewed-by: John Snow <jsnow@redhat.com>
-Reviewed-by: Alberto Garcia <berto@igalia.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- qapi/block-core.json | 20 ++++++++++----------
-file changed, 10 insertions(+), 10 deletions(-)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
-+++ b/qapi/block-core.json
-@@ -XXX,XX +XXX,XX @@
- #
- # Either @device or @node-name must be set but not both.
- #
--# @device: the name of the device to generate the snapshot from.
-+# @device: the name of the device to take a snapshot of.
- #
- # @node-name: graph node name to generate the snapshot from (Since 2.0)
- #
--# @snapshot-file: the target of the new image. If the file exists, or
--# if it is a device, the snapshot will be created in the existing
--# file/device. Otherwise, a new file will be created.
-+# @snapshot-file: the target of the new overlay image. If the file
-+# exists, or if it is a device, the overlay will be created in the
-+# existing file/device. Otherwise, a new file will be created.
- #
- # @snapshot-node-name: the graph node name of the new image (Since 2.0)
- #
--# @format: the format of the snapshot image, default is 'qcow2'.
-+# @format: the format of the overlay image, default is 'qcow2'.
- #
- # @mode: whether and how QEMU should create a new image, default is
- #        'absolute-paths'.
-@@ -XXX,XX +XXX,XX @@
- ##
- # @BlockdevSnapshot:
- #
--# @node: device or node name that will have a snapshot created.
-+# @node: device or node name that will have a snapshot taken.
- #
- # @overlay: reference to the existing block device that will become
--#           the overlay of @node, as part of creating the snapshot.
-+#           the overlay of @node, as part of taking the snapshot.
- #           It must not have a current backing file (this can be
- #           achieved by passing "backing": null to blockdev-add).
- #
-@@ -XXX,XX +XXX,XX @@
- ##
- # @blockdev-snapshot-sync:
- #
--# Generates a synchronous snapshot of a block device.
-+# Takes a synchronous snapshot of a block device.
- #
- # For the arguments, see the documentation of BlockdevSnapshotSync.
- #
-@@ -XXX,XX +XXX,XX @@
- ##
- # @blockdev-snapshot:
- #
--# Generates a snapshot of a block device.
-+# Takes a snapshot of a block device.
- #
--# Create a snapshot, by installing 'node' as the backing image of
-+# Take a snapshot, by installing 'node' as the backing image of
- # 'overlay'. Additionally, if 'node' is associated with a block
- # device, the block device changes to using 'overlay' as its new active
- # image.
---
-.21.0

-[Qemu-devel] [PULL 16/20] blkdebug: Add @iotype error option
+[Qemu-devel] [PULL v2 6/8] vmdk: Add read-only support for seSparse snapshots
-This new error option allows users of blkdebug to inject errors only on
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-certain kinds of I/O operations.  Users usually want to make a very
-specific operation fail, not just any; but right now they simply hope
+Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in
-that the event that triggers the error injection is followed up with
+QEMU).
-that very operation.  That may not be true, however, because the block
-layer is changing (including blkdebug, which may increase the number of
+This format was lacking in the following:
-types of I/O operations on which to inject errors).
+    * Grain directory (L1) and grain table (L2) entries were 32-bit,
-The new option's default has been chosen to keep backwards
+      allowing access to only 2TB (slightly less) of data.
-compatibility.
+    * The grain size (default) was 512 bytes - leading to data
+      fragmentation and many grain tables.
-Note that similar to the internal representation, we could choose to
+    * For space reclamation purposes, it was necessary to find all the
-expose this option as a list of I/O types.  But there is no practical
+      grains which are not pointed to by any grain table - so a reverse
-use for this, because as described above, users usually know exactly
+      mapping of "offset of grain in vmdk" to "grain table" must be
-which kind of operation they want to make fail, so there is no need to
+      constructed - which takes large amounts of CPU/RAM.
-specify multiple I/O types at once.  In addition, exposing this option
-as a list would require non-trivial changes to qemu_opts_absorb_qdict().
+The format specification can be found in VMware's documentation:
+https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+In ESXi 6.5, to support snapshot files larger than 2TB, a new format was
-Message-id: 20190507203508.18026-4-mreitz@redhat.com
+introduced: SESparse (Space Efficient).
 This format fixes the above issues:
     * All entries are now 64-bit.
     * The grain size (default) is 4KB.
     * Grain directory and grain tables are now located at the beginning
       of the file.
       + seSparse format reserves space for all grain tables.
       + Grain tables can be addressed using an index.
       + Grains are located in the end of the file and can also be
         addressed with an index.
       - seSparse vmdks of large disks (64TB) have huge preallocated
         headers - mainly due to L2 tables, even for empty snapshots.
     * The header contains a reverse mapping ("backmap") of "offset of
       grain in vmdk" to "grain table" and a bitmap ("free bitmap") which
       specifies for each grain - whether it is allocated or not.
       Using these data structures we can implement space reclamation
       efficiently.
     * Due to the fact that the header now maintains two mappings:
         * The regular one (grain directory & grain tables)
         * A reverse one (backmap and free bitmap)
       These data structures can lose consistency upon crash and result
       in a corrupted VMDK.
       Therefore, a journal is also added to the VMDK and is replayed
       when the VMware reopens the file after a crash.
 Since ESXi 6.7 - SESparse is the only snapshot format available.
 Unfortunately, VMware does not provide documentation regarding the new
 seSparse format.
 This commit is based on black-box research of the seSparse format.
 Various in-guest block operations and their effect on the snapshot file
 were tested.
 The only VMware provided source of information (regarding the underlying
 implementation) was a log file on the ESXi:
     /var/log/hostd.log
 Whenever an seSparse snapshot is created - the log is being populated
 with seSparse records.
 Relevant log records are of the form:
 [...] Const Header:
 [...]  constMagic     = 0xcafebabe
 [...]  version        = 2.1
 [...]  capacity       = 204800
 [...]  grainSize      = 8
 [...]  grainTableSize = 64
 [...]  flags          = 0
 [...] Extents:
 [...]  Header         : <1 : 1>
 [...]  JournalHdr     : <2 : 2>
 [...]  Journal        : <2048 : 2048>
 [...]  GrainDirectory : <4096 : 2048>
 [...]  GrainTables    : <6144 : 2048>
 [...]  FreeBitmap     : <8192 : 2048>
 [...]  BackMap        : <10240 : 2048>
 [...]  Grain          : <12288 : 204800>
 [...] Volatile Header:
 [...] volatileMagic     = 0xcafecafe
 [...] FreeGTNumber      = 0
 [...] nextTxnSeqNumber  = 0
 [...] replayJournal     = 0
 The sizes that are seen in the log file are in sectors.
 Extents are of the following format: <offset : size>
 This commit is a strict implementation which enforces:
     * magics
     * version number 2.1
     * grain size of 8 sectors  (4KB)
     * grain table size of 64 sectors
     * zero flags
     * extent locations
 Additionally, this commit proivdes only a subset of the functionality
 offered by seSparse's format:
     * Read-only
     * No journal replay
     * No space reclamation
     * No unmap support
 Hence, journal header, journal, free bitmap and backmap extents are
 unused, only the "classic" (L1 -> L2 -> data) grain access is
 implemented.
 However there are several differences in the grain access itself.
 Grain directory (L1):
     * Grain directory entries are indexes (not offsets) to grain
       tables.
     * Valid grain directory entries have their highest nibble set to
 x1.
     * Since grain tables are always located in the beginning of the
       file - the index can fit into 32 bits - so we can use its low
       part if it's valid.
 Grain table (L2):
     * Grain table entries are indexes (not offsets) to grains.
     * If the highest nibble of the entry is:
 x0:
             The grain in not allocated.
             The rest of the bytes are 0.
 x1:
             The grain is unmapped - guest sees a zero grain.
             The rest of the bits point to the previously mapped grain,
             see 0x3 case.
 x2:
             The grain is zero.
 x3:
             The grain is allocated - to get the index calculate:
             ((entry & 0x0fff000000000000) >> 48) |
             ((entry & 0x0000ffffffffffff) << 12)
     * The difference between 0x1 and 0x2 is that 0x1 is an unallocated
       grain which results from the guest using sg_unmap to unmap the
       grain - but the grain itself still exists in the grain extent - a
       space reclamation procedure should delete it.
       Unmapping a zero grain has no effect (0x2 will not change to 0x1)
       but unmapping an unallocated grain will (0x0 to 0x1) - naturally.
 In order to implement seSparse some fields had to be changed to support
 both 32-bit and 64-bit entry sizes.
 Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
 Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
 Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
 Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
 Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json | 26 +++++++++++++++++++++++
+ block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---
- block/blkdebug.c     | 50 ++++++++++++++++++++++++++++++++++++--------
+file changed, 342 insertions(+), 16 deletions(-)
-files changed, 67 insertions(+), 9 deletions(-)
+diff --git a/block/vmdk.c b/block/vmdk.c
 diff --git a/qapi/block-core.json b/qapi/block-core.json
 index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
+--- a/block/vmdk.c
-+++ b/qapi/block-core.json
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ typedef struct {
-             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
+     uint16_t compressAlgorithm;
-             'cor_write', 'cluster_alloc_space'] }
+ } QEMU_PACKED VMDK4Header;
-+##
++typedef struct VMDKSESparseConstHeader {
-+# @BlkdebugIOType:
++    uint64_t magic;
-+#
++    uint64_t version;
-+# Kinds of I/O that blkdebug can inject errors in.
++    uint64_t capacity;
-+#
++    uint64_t grain_size;
-+# @read: .bdrv_co_preadv()
++    uint64_t grain_table_size;
-+#
++    uint64_t flags;
-+# @write: .bdrv_co_pwritev()
++    uint64_t reserved1;
-+#
++    uint64_t reserved2;
-+# @write-zeroes: .bdrv_co_pwrite_zeroes()
++    uint64_t reserved3;
-+#
++    uint64_t reserved4;
-+# @discard: .bdrv_co_pdiscard()
++    uint64_t volatile_header_offset;
-+#
++    uint64_t volatile_header_size;
-+# @flush: .bdrv_co_flush_to_disk()
++    uint64_t journal_header_offset;
-+#
++    uint64_t journal_header_size;
-+# Since: 4.1
++    uint64_t journal_offset;
-+##
++    uint64_t journal_size;
-+{ 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
++    uint64_t grain_dir_offset;
-+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
++    uint64_t grain_dir_size;
-+
++    uint64_t grain_tables_offset;
- ##
++    uint64_t grain_tables_size;
- # @BlkdebugInjectErrorOptions:
++    uint64_t free_bitmap_offset;
- #
++    uint64_t free_bitmap_size;
-@@ -XXX,XX +XXX,XX @@
++    uint64_t backmap_offset;
- # @state:       the state identifier blkdebug needs to be in to
++    uint64_t backmap_size;
- #               actually trigger the event; defaults to "any"
++    uint64_t grains_offset;
- #
++    uint64_t grains_size;
-+# @iotype:      the type of I/O operations on which this error should
++    uint8_t pad[304];
-+#               be injected; defaults to "all read, write,
++} QEMU_PACKED VMDKSESparseConstHeader;
-+#               write-zeroes, discard, and flush operations"
++
-+#               (since: 4.1)
++typedef struct VMDKSESparseVolatileHeader {
-+#
++    uint64_t magic;
- # @errno:       error identifier (errno) to be returned; defaults to
++    uint64_t free_gt_number;
- #               EIO
++    uint64_t next_txn_seq_number;
- #
++    uint64_t replay_journal;
-@@ -XXX,XX +XXX,XX @@
++    uint8_t pad[480];
- { 'struct': 'BlkdebugInjectErrorOptions',
++} QEMU_PACKED VMDKSESparseVolatileHeader;
-   'data': { 'event': 'BlkdebugEvent',
++
-             '*state': 'int',
+ #define L2_CACHE_SIZE 16
-+            '*iotype': 'BlkdebugIOType',
-             '*errno': 'int',
+ typedef struct VmdkExtent {
-             '*sector': 'int',
+@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {
-             '*once': 'bool',
+     bool compressed;
-diff --git a/block/blkdebug.c b/block/blkdebug.c
+     bool has_marker;
-index XXXXXXX..XXXXXXX 100644
+     bool has_zero_grain;
---- a/block/blkdebug.c
++    bool sesparse;
-+++ b/block/blkdebug.c
++    uint64_t sesparse_l2_tables_offset;
-@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
++    uint64_t sesparse_clusters_offset;
-     int state;
++    int32_t entry_size;
-     union {
+     int version;
-         struct {
+     int64_t sectors;
-+            uint64_t iotype_mask;
+     int64_t end_sector;
-             int error;
+     int64_t flat_start_offset;
-             int immediately;
+     int64_t l1_table_offset;
-             int once;
+     int64_t l1_backup_table_offset;
-@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
+-    uint32_t *l1_table;
-     QSIMPLEQ_ENTRY(BlkdebugRule) active_next;
++    void *l1_table;
- } BlkdebugRule;
+     uint32_t *l1_backup_table;
+     unsigned int l1_size;
-+QEMU_BUILD_BUG_MSG(BLKDEBUG_IO_TYPE__MAX > 64,
+     uint32_t l1_entry_sectors;
-+                   "BlkdebugIOType mask does not fit into an uint64_t");
-+
+     unsigned int l2_size;
- static QemuOptsList inject_error_opts = {
+-    uint32_t *l2_cache;
-     .name = "inject-error",
++    void *l2_cache;
-     .head = QTAILQ_HEAD_INITIALIZER(inject_error_opts.head),
+     uint32_t l2_cache_offsets[L2_CACHE_SIZE];
-@@ -XXX,XX +XXX,XX @@ static QemuOptsList inject_error_opts = {
+     uint32_t l2_cache_counts[L2_CACHE_SIZE];
-             .name = "state",
-             .type = QEMU_OPT_NUMBER,
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-         },
+          *            minimal L2 table size: 512 entries
-+        {
+          *            8 TB is still more than the maximal value supported for
-+            .name = "iotype",
+          *            VMDK3 & VMDK4 which is 2TB.
-+            .type = QEMU_OPT_STRING,
++         *     64TB - for "ESXi seSparse Extent"
-+        },
++         *            minimal cluster size: 512B (default is 4KB)
-         {
++         *            L2 table size: 4096 entries (const).
-             .name = "errno",
++         *            64TB is more than the maximal value supported for
-             .type = QEMU_OPT_NUMBER,
++         *            seSparse VMDKs (which is slightly less than 64TB)
-@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
+          */
-     int event;
+         error_setg(errp, "L1 size too big");
-     struct BlkdebugRule *rule;
+         return -EFBIG;
-     int64_t sector;
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-+    BlkdebugIOType iotype;
+     extent->l2_size = l2_size;
-+    Error *local_error = NULL;
+     extent->cluster_sectors = flat ? sectors : cluster_sectors;
+     extent->next_cluster_sector = ROUND_UP(nb_sectors, cluster_sectors);
-     /* Find the right event for the rule */
++    extent->entry_size = sizeof(uint32_t);
-     event_name = qemu_opt_get(opts, "event");
-@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
+     if (s->num_extents > 1) {
-         sector = qemu_opt_get_number(opts, "sector", -1);
+         extent->end_sector = (*(extent - 1)).end_sector + extent->sectors;
-         rule->options.inject.offset =
+@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
-             sector == -1 ? -1 : sector * BDRV_SECTOR_SIZE;
+     int i;
-+
-+        iotype = qapi_enum_parse(&BlkdebugIOType_lookup,
+     /* read the L1 table */
-+                                 qemu_opt_get(opts, "iotype"),
+-    l1_size = extent->l1_size * sizeof(uint32_t);
-+                                 BLKDEBUG_IO_TYPE__MAX, &local_error);
++    l1_size = extent->l1_size * extent->entry_size;
-+        if (local_error) {
+     extent->l1_table = g_try_malloc(l1_size);
-+            error_propagate(errp, local_error);
+     if (l1_size && extent->l1_table == NULL) {
-+            return -1;
+         return -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
          goto fail_l1;
      }
      for (i = 0; i < extent->l1_size; i++) {
 -        le32_to_cpus(&extent->l1_table[i]);
 +        if (extent->entry_size == sizeof(uint64_t)) {
 +            le64_to_cpus((uint64_t *)extent->l1_table + i);
 +        } else {
 +            assert(extent->entry_size == sizeof(uint32_t));
 +            le32_to_cpus((uint32_t *)extent->l1_table + i);
 +        }
-+        if (iotype != BLKDEBUG_IO_TYPE__MAX) {
+     }
-+            rule->options.inject.iotype_mask = (1ull << iotype);
-+        } else {
+     if (extent->l1_backup_table_offset) {
-+            /* Apply the default */
++        assert(!extent->sesparse);
-+            rule->options.inject.iotype_mask =
+         extent->l1_backup_table = g_try_malloc(l1_size);
-+                (1ull << BLKDEBUG_IO_TYPE_READ)
+         if (l1_size && extent->l1_backup_table == NULL) {
-+                | (1ull << BLKDEBUG_IO_TYPE_WRITE)
+             ret = -ENOMEM;
-+                | (1ull << BLKDEBUG_IO_TYPE_WRITE_ZEROES)
+@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
-+                | (1ull << BLKDEBUG_IO_TYPE_DISCARD)
+     }
-+                | (1ull << BLKDEBUG_IO_TYPE_FLUSH);
-+        }
+     extent->l2_cache =
-+
+-        g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE);
-         break;
++        g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE);
+     return 0;
-     case ACTION_SET_STATE:
+  fail_l1b:
-@@ -XXX,XX +XXX,XX @@ out:
+     g_free(extent->l1_backup_table);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
      return ret;
  }
--static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
++#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe)
-+static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
++#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe)
-+                      BlkdebugIOType iotype)
++
 +/* Strict checks - format not officially documented */
 +static int check_se_sparse_const_header(VMDKSESparseConstHeader *header,
 +                                        Error **errp)
 +{
 +    header->magic = le64_to_cpu(header->magic);
 +    header->version = le64_to_cpu(header->version);
 +    header->grain_size = le64_to_cpu(header->grain_size);
 +    header->grain_table_size = le64_to_cpu(header->grain_table_size);
 +    header->flags = le64_to_cpu(header->flags);
 +    header->reserved1 = le64_to_cpu(header->reserved1);
 +    header->reserved2 = le64_to_cpu(header->reserved2);
 +    header->reserved3 = le64_to_cpu(header->reserved3);
 +    header->reserved4 = le64_to_cpu(header->reserved4);
 +
 +    header->volatile_header_offset =
 +        le64_to_cpu(header->volatile_header_offset);
 +    header->volatile_header_size = le64_to_cpu(header->volatile_header_size);
 +
 +    header->journal_header_offset = le64_to_cpu(header->journal_header_offset);
 +    header->journal_header_size = le64_to_cpu(header->journal_header_size);
 +
 +    header->journal_offset = le64_to_cpu(header->journal_offset);
 +    header->journal_size = le64_to_cpu(header->journal_size);
 +
 +    header->grain_dir_offset = le64_to_cpu(header->grain_dir_offset);
 +    header->grain_dir_size = le64_to_cpu(header->grain_dir_size);
 +
 +    header->grain_tables_offset = le64_to_cpu(header->grain_tables_offset);
 +    header->grain_tables_size = le64_to_cpu(header->grain_tables_size);
 +
 +    header->free_bitmap_offset = le64_to_cpu(header->free_bitmap_offset);
 +    header->free_bitmap_size = le64_to_cpu(header->free_bitmap_size);
 +
 +    header->backmap_offset = le64_to_cpu(header->backmap_offset);
 +    header->backmap_size = le64_to_cpu(header->backmap_size);
 +
 +    header->grains_offset = le64_to_cpu(header->grains_offset);
 +    header->grains_size = le64_to_cpu(header->grains_size);
 +
 +    if (header->magic != SESPARSE_CONST_HEADER_MAGIC) {
 +        error_setg(errp, "Bad const header magic: 0x%016" PRIx64,
 +                   header->magic);
 +        return -EINVAL;
 +    }
 +
 +    if (header->version != 0x0000000200000001) {
 +        error_setg(errp, "Unsupported version: 0x%016" PRIx64,
 +                   header->version);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->grain_size != 8) {
 +        error_setg(errp, "Unsupported grain size: %" PRIu64,
 +                   header->grain_size);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->grain_table_size != 64) {
 +        error_setg(errp, "Unsupported grain table size: %" PRIu64,
 +                   header->grain_table_size);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->flags != 0) {
 +        error_setg(errp, "Unsupported flags: 0x%016" PRIx64,
 +                   header->flags);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->reserved1 != 0 || header->reserved2 != 0 ||
 +        header->reserved3 != 0 || header->reserved4 != 0) {
 +        error_setg(errp, "Unsupported reserved bits:"
 +                   " 0x%016" PRIx64 " 0x%016" PRIx64
 +                   " 0x%016" PRIx64 " 0x%016" PRIx64,
 +                   header->reserved1, header->reserved2,
 +                   header->reserved3, header->reserved4);
 +        return -ENOTSUP;
 +    }
 +
 +    /* check that padding is 0 */
 +    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
 +        error_setg(errp, "Unsupported non-zero const header padding");
 +        return -ENOTSUP;
 +    }
 +
 +    return 0;
 +}
 +
 +static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *header,
 +                                           Error **errp)
 +{
 +    header->magic = le64_to_cpu(header->magic);
 +    header->free_gt_number = le64_to_cpu(header->free_gt_number);
 +    header->next_txn_seq_number = le64_to_cpu(header->next_txn_seq_number);
 +    header->replay_journal = le64_to_cpu(header->replay_journal);
 +
 +    if (header->magic != SESPARSE_VOLATILE_HEADER_MAGIC) {
 +        error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64,
 +                   header->magic);
 +        return -EINVAL;
 +    }
 +
 +    if (header->replay_journal) {
 +        error_setg(errp, "Image is dirty, Replaying journal not supported");
 +        return -ENOTSUP;
 +    }
 +
 +    /* check that padding is 0 */
 +    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
 +        error_setg(errp, "Unsupported non-zero volatile header padding");
 +        return -ENOTSUP;
 +    }
 +
 +    return 0;
 +}
 +
 +static int vmdk_open_se_sparse(BlockDriverState *bs,
 +                               BdrvChild *file,
 +                               int flags, Error **errp)
 +{
 +    int ret;
 +    VMDKSESparseConstHeader const_header;
 +    VMDKSESparseVolatileHeader volatile_header;
 +    VmdkExtent *extent;
 +
 +    ret = bdrv_apply_auto_read_only(bs,
 +            "No write support for seSparse images available", errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    assert(sizeof(const_header) == SECTOR_SIZE);
 +
 +    ret = bdrv_pread(file, 0, &const_header, sizeof(const_header));
 +    if (ret < 0) {
 +        bdrv_refresh_filename(file->bs);
 +        error_setg_errno(errp, -ret,
 +                         "Could not read const header from file '%s'",
 +                         file->bs->filename);
 +        return ret;
 +    }
 +
 +    /* check const header */
 +    ret = check_se_sparse_const_header(&const_header, errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    assert(sizeof(volatile_header) == SECTOR_SIZE);
 +
 +    ret = bdrv_pread(file,
 +                     const_header.volatile_header_offset * SECTOR_SIZE,
 +                     &volatile_header, sizeof(volatile_header));
 +    if (ret < 0) {
 +        bdrv_refresh_filename(file->bs);
 +        error_setg_errno(errp, -ret,
 +                         "Could not read volatile header from file '%s'",
 +                         file->bs->filename);
 +        return ret;
 +    }
 +
 +    /* check volatile header */
 +    ret = check_se_sparse_volatile_header(&volatile_header, errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    ret = vmdk_add_extent(bs, file, false,
 +                          const_header.capacity,
 +                          const_header.grain_dir_offset * SECTOR_SIZE,
 +                          0,
 +                          const_header.grain_dir_size *
 +                          SECTOR_SIZE / sizeof(uint64_t),
 +                          const_header.grain_table_size *
 +                          SECTOR_SIZE / sizeof(uint64_t),
 +                          const_header.grain_size,
 +                          &extent,
 +                          errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    extent->sesparse = true;
 +    extent->sesparse_l2_tables_offset = const_header.grain_tables_offset;
 +    extent->sesparse_clusters_offset = const_header.grains_offset;
 +    extent->entry_size = sizeof(uint64_t);
 +
 +    ret = vmdk_init_tables(bs, extent, errp);
 +    if (ret) {
 +        /* free extent allocated by vmdk_add_extent */
 +        vmdk_free_last_extent(bs);
 +    }
 +
 +    return ret;
 +}
 +
  static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
                                 QDict *options, Error **errp);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
           * RW [size in sectors] SPARSE "file-name.vmdk"
           * RW [size in sectors] VMFS "file-name.vmdk"
           * RW [size in sectors] VMFSSPARSE "file-name.vmdk"
 +         * RW [size in sectors] SESPARSE "file-name.vmdk"
           */
          flat_offset = -1;
          matches = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          if (sectors <= 0 ||
              (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
 -             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) ||
 +             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") &&
 +             strcmp(type, "SESPARSE")) ||
              (strcmp(access, "RW"))) {
              continue;
          }
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                  return ret;
              }
              extent = &s->extents[s->num_extents - 1];
 +        } else if (!strcmp(type, "SESPARSE")) {
 +            ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
 +            if (ret) {
 +                bdrv_unref_child(bs, extent_file);
 +                return ret;
 +            }
 +            extent = &s->extents[s->num_extents - 1];
          } else {
              error_setg(errp, "Unsupported extent type '%s'", type);
              bdrv_unref_child(bs, extent_file);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
      if (strcmp(ct, "monolithicFlat") &&
          strcmp(ct, "vmfs") &&
          strcmp(ct, "vmfsSparse") &&
 +        strcmp(ct, "seSparse") &&
          strcmp(ct, "twoGbMaxExtentSparse") &&
          strcmp(ct, "twoGbMaxExtentFlat")) {
          error_setg(errp, "Unsupported image type '%s'", ct);
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
  {
-     BDRVBlkdebugState *s = bs->opaque;
+     unsigned int l1_index, l2_offset, l2_index;
-     BlkdebugRule *rule = NULL;
+     int min_index, i, j;
-@@ -XXX,XX +XXX,XX @@ static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
+-    uint32_t min_count, *l2_table;
-     QSIMPLEQ_FOREACH(rule, &s->active_rules, active_next) {
++    uint32_t min_count;
-         uint64_t inject_offset = rule->options.inject.offset;
++    void *l2_table;
+     bool zeroed = false;
--        if (inject_offset == -1 ||
+     int64_t ret;
--            (bytes && inject_offset >= offset &&
+     int64_t cluster_sector;
--             inject_offset < offset + bytes))
++    unsigned int l2_size_bytes = extent->l2_size * extent->entry_size;
-+        if ((inject_offset == -1 ||
-+             (bytes && inject_offset >= offset &&
+     if (m_data) {
-+              inject_offset < offset + bytes)) &&
+         m_data->valid = 0;
-+            (rule->options.inject.iotype_mask & (1ull << iotype)))
+@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
-         {
+     if (l1_index >= extent->l1_size) {
-             break;
+         return VMDK_ERROR;
      }
 -    l2_offset = extent->l1_table[l1_index];
 +    if (extent->sesparse) {
 +        uint64_t l2_offset_u64;
 +
 +        assert(extent->entry_size == sizeof(uint64_t));
 +
 +        l2_offset_u64 = ((uint64_t *)extent->l1_table)[l1_index];
 +        if (l2_offset_u64 == 0) {
 +            l2_offset = 0;
 +        } else if ((l2_offset_u64 & 0xffffffff00000000) != 0x1000000000000000) {
 +            /*
 +             * Top most nibble is 0x1 if grain table is allocated.
 +             * strict check - top most 4 bytes must be 0x10000000 since max
 +             * supported size is 64TB for disk - so no more than 64TB / 16MB
 +             * grain directories which is smaller than uint32,
 +             * where 16MB is the only supported default grain table coverage.
 +             */
 +            return VMDK_ERROR;
 +        } else {
 +            l2_offset_u64 = l2_offset_u64 & 0x00000000ffffffff;
 +            l2_offset_u64 = extent->sesparse_l2_tables_offset +
 +                l2_offset_u64 * l2_size_bytes / SECTOR_SIZE;
 +            if (l2_offset_u64 > 0x00000000ffffffff) {
 +                return VMDK_ERROR;
 +            }
 +            l2_offset = (unsigned int)(l2_offset_u64);
 +        }
 +    } else {
 +        assert(extent->entry_size == sizeof(uint32_t));
 +        l2_offset = ((uint32_t *)extent->l1_table)[l1_index];
 +    }
      if (!l2_offset) {
          return VMDK_UNALLOC;
      }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
                      extent->l2_cache_counts[j] >>= 1;
                  }
              }
 -            l2_table = extent->l2_cache + (i * extent->l2_size);
 +            l2_table = (char *)extent->l2_cache + (i * l2_size_bytes);
              goto found;
          }
-@@ -XXX,XX +XXX,XX @@ blkdebug_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
+     }
-         assert(bytes <= bs->bl.max_transfer);
+@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
-     }
+             min_index = i;
+         }
--    err = rule_check(bs, offset, bytes);
+     }
-+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_READ);
+-    l2_table = extent->l2_cache + (min_index * extent->l2_size);
-     if (err) {
++    l2_table = (char *)extent->l2_cache + (min_index * l2_size_bytes);
-         return err;
+     BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD);
-     }
+     if (bdrv_pread(extent->file,
-@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
+                 (int64_t)l2_offset * 512,
-         assert(bytes <= bs->bl.max_transfer);
+                 l2_table,
-     }
+-                extent->l2_size * sizeof(uint32_t)
+-            ) != extent->l2_size * sizeof(uint32_t)) {
--    err = rule_check(bs, offset, bytes);
++                l2_size_bytes
-+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE);
++            ) != l2_size_bytes) {
-     if (err) {
+         return VMDK_ERROR;
-         return err;
+     }
-     }
-@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
+@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
+     extent->l2_cache_counts[min_index] = 1;
- static int blkdebug_co_flush(BlockDriverState *bs)
+  found:
- {
+     l2_index = ((offset >> 9) / extent->cluster_sectors) % extent->l2_size;
--    int err = rule_check(bs, 0, 0);
+-    cluster_sector = le32_to_cpu(l2_table[l2_index]);
-+    int err = rule_check(bs, 0, 0, BLKDEBUG_IO_TYPE_FLUSH);
+-    if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
-     if (err) {
+-        zeroed = true;
-         return err;
++    if (extent->sesparse) {
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pwrite_zeroes(BlockDriverState *bs,
++        cluster_sector = le64_to_cpu(((uint64_t *)l2_table)[l2_index]);
-         assert(bytes <= bs->bl.max_pwrite_zeroes);
++        switch (cluster_sector & 0xf000000000000000) {
-     }
++        case 0x0000000000000000:
++            /* unallocated grain */
--    err = rule_check(bs, offset, bytes);
++            if (cluster_sector != 0) {
-+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE_ZEROES);
++                return VMDK_ERROR;
-     if (err) {
++            }
-         return err;
++            break;
-     }
++        case 0x1000000000000000:
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pdiscard(BlockDriverState *bs,
++            /* scsi-unmapped grain - fallthrough */
-         assert(bytes <= bs->bl.max_pdiscard);
++        case 0x2000000000000000:
-     }
++            /* zero grain */
++            zeroed = true;
--    err = rule_check(bs, offset, bytes);
++            break;
-+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_DISCARD);
++        case 0x3000000000000000:
-     if (err) {
++            /* allocated grain */
-         return err;
++            cluster_sector = (((cluster_sector & 0x0fff000000000000) >> 48) |
-     }
++                              ((cluster_sector & 0x0000ffffffffffff) << 12));
 +            cluster_sector = extent->sesparse_clusters_offset +
 +                cluster_sector * extent->cluster_sectors;
 +            break;
 +        default:
 +            return VMDK_ERROR;
 +        }
 +    } else {
 +        cluster_sector = le32_to_cpu(((uint32_t *)l2_table)[l2_index]);
 +
 +        if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
 +            zeroed = true;
 +        }
      }
      if (!cluster_sector || zeroed) {
          if (!allocate) {
              return zeroed ? VMDK_ZEROED : VMDK_UNALLOC;
          }
 +        assert(!extent->sesparse);
          if (extent->next_cluster_sector >= VMDK_EXTENT_MAX_SECTORS) {
              return VMDK_ERROR;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
              m_data->l1_index = l1_index;
              m_data->l2_index = l2_index;
              m_data->l2_offset = l2_offset;
 -            m_data->l2_cache_entry = &l2_table[l2_index];
 +            m_data->l2_cache_entry = ((uint32_t *)l2_table) + l2_index;
          }
      }
      *cluster_offset = cluster_sector << BDRV_SECTOR_BITS;
@@ -XXX,XX +XXX,XX @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_t offset,
          if (!extent) {
              return -EIO;
          }
 +        if (extent->sesparse) {
 +            return -ENOTSUP;
 +        }
          offset_in_cluster = vmdk_find_offset_in_cluster(extent, offset);
          n_bytes = MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE
                               - offset_in_cluster);
 --
 .21.0

-[Qemu-devel] [PULL 13/20] blockdev: Overlays are not snapshots
+[Qemu-devel] [PULL v2 7/8] ssh: switch from libssh2 to libssh
-There are error messages which refer to an overlay node as the snapshot.
+From: Pino Toscano <ptoscano@redhat.com>
 That is wrong, those are two different things.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
+Rewrite the implementation of the ssh block driver to use libssh instead
-Reviewed-by: Eric Blake <eblake@redhat.com>
+of libssh2.  The libssh library has various advantages over libssh2:
-Message-id: 20190603202236.1342-3-mreitz@redhat.com
+- easier API for authentication (for example for using ssh-agent)
-Reviewed-by: John Snow <jsnow@redhat.com>
+- easier API for known_hosts handling
-Reviewed-by: Alberto Garcia <berto@igalia.com>
+- supports newer types of keys in known_hosts
 Use APIs/features available in libssh 0.8 conditionally, to support
 older versions (which are not recommended though).
 Adjust the iotest 207 according to the different error message, and to
 find the default key type for localhost (to properly compare the
 fingerprint with).
 Contributed-by: Max Reitz <mreitz@redhat.com>
 Adjust the various Docker/Travis scripts to use libssh when available
 instead of libssh2. The mingw/mxe testing is dropped for now, as there
 are no packages for it.
 Signed-off-by: Pino Toscano <ptoscano@redhat.com>
 Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
 Acked-by: Alex Bennée <alex.bennee@linaro.org>
 Message-id: 20190620200840.17655-1-ptoscano@redhat.com
 Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
 Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- blockdev.c                 | 10 +++++-----
+ configure                                     |  65 +-
- tests/qemu-iotests/085.out | 10 +++++-----
+ block/Makefile.objs                           |   6 +-
-files changed, 10 insertions(+), 10 deletions(-)
+ block/ssh.c                                   | 652 ++++++++++--------
  .travis.yml                                   |   4 +-
  block/trace-events                            |  14 +-
  docs/qemu-block-drivers.texi                  |   2 +-
  .../dockerfiles/debian-win32-cross.docker     |   1 -
  .../dockerfiles/debian-win64-cross.docker     |   1 -
  tests/docker/dockerfiles/fedora.docker        |   4 +-
  tests/docker/dockerfiles/ubuntu.docker        |   2 +-
  tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
  tests/qemu-iotests/207                        |  54 +-
  tests/qemu-iotests/207.out                    |   2 +-
 files changed, 449 insertions(+), 360 deletions(-)
-diff --git a/blockdev.c b/blockdev.c
+diff --git a/configure b/configure
 index XXXXXXX..XXXXXXX 100755
 --- a/configure
 +++ b/configure
@@ -XXX,XX +XXX,XX @@ auth_pam=""
  vte=""
  virglrenderer=""
  tpm=""
 -libssh2=""
 +libssh=""
  live_block_migration="yes"
  numa=""
  tcmalloc="no"
@@ -XXX,XX +XXX,XX @@ for opt do
    ;;
    --enable-tpm) tpm="yes"
    ;;
 -  --disable-libssh2) libssh2="no"
 +  --disable-libssh) libssh="no"
    ;;
 -  --enable-libssh2) libssh2="yes"
 +  --enable-libssh) libssh="yes"
    ;;
    --disable-live-block-migration) live_block_migration="no"
    ;;
@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
    coroutine-pool  coroutine freelist (better performance)
    glusterfs       GlusterFS backend
    tpm             TPM support
 -  libssh2         ssh block device support
 +  libssh          ssh block device support
    numa            libnuma support
    libxml2         for Parallels image format
    tcmalloc        tcmalloc support
@@ -XXX,XX +XXX,XX @@ EOF
  fi
  ##########################################
 -# libssh2 probe
 -min_libssh2_version=1.2.8
 -if test "$libssh2" != "no" ; then
 -  if $pkg_config --atleast-version=$min_libssh2_version libssh2; then
 -    libssh2_cflags=$($pkg_config libssh2 --cflags)
 -    libssh2_libs=$($pkg_config libssh2 --libs)
 -    libssh2=yes
 +# libssh probe
 +if test "$libssh" != "no" ; then
 +  if $pkg_config --exists libssh; then
 +    libssh_cflags=$($pkg_config libssh --cflags)
 +    libssh_libs=$($pkg_config libssh --libs)
 +    libssh=yes
    else
 -    if test "$libssh2" = "yes" ; then
 -      error_exit "libssh2 >= $min_libssh2_version required for --enable-libssh2"
 +    if test "$libssh" = "yes" ; then
 +      error_exit "libssh required for --enable-libssh"
      fi
 -    libssh2=no
 +    libssh=no
    fi
  fi
  ##########################################
 -# libssh2_sftp_fsync probe
 +# Check for libssh 0.8
 +# This is done like this instead of using the LIBSSH_VERSION_* and
 +# SSH_VERSION_* macros because some distributions in the past shipped
 +# snapshots of the future 0.8 from Git, and those snapshots did not
 +# have updated version numbers (still referring to 0.7.0).
 -if test "$libssh2" = "yes"; then
 +if test "$libssh" = "yes"; then
    cat > $TMPC <<EOF
 -#include <stdio.h>
 -#include <libssh2.h>
 -#include <libssh2_sftp.h>
 -int main(void) {
 -    LIBSSH2_SESSION *session;
 -    LIBSSH2_SFTP *sftp;
 -    LIBSSH2_SFTP_HANDLE *sftp_handle;
 -    session = libssh2_session_init ();
 -    sftp = libssh2_sftp_init (session);
 -    sftp_handle = libssh2_sftp_open (sftp, "/", 0, 0);
 -    libssh2_sftp_fsync (sftp_handle);
 -    return 0;
 -}
 +#include <libssh/libssh.h>
 +int main(void) { return ssh_get_server_publickey(NULL, NULL); }
  EOF
 -  # libssh2_cflags/libssh2_libs defined in previous test.
 -  if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then
 -    QEMU_CFLAGS="-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS"
 +  if compile_prog "$libssh_cflags" "$libssh_libs"; then
 +    libssh_cflags="-DHAVE_LIBSSH_0_8 $libssh_cflags"
    fi
  fi
@@ -XXX,XX +XXX,XX @@ echo "GlusterFS support $glusterfs"
  echo "gcov              $gcov_tool"
  echo "gcov enabled      $gcov"
  echo "TPM support       $tpm"
 -echo "libssh2 support   $libssh2"
 +echo "libssh support    $libssh"
  echo "QOM debugging     $qom_cast_debug"
  echo "Live block migration $live_block_migration"
  echo "lzo support       $lzo"
@@ -XXX,XX +XXX,XX @@ if test "$glusterfs_iocb_has_stat" = "yes" ; then
    echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=y" >> $config_host_mak
  fi
 -if test "$libssh2" = "yes" ; then
 -  echo "CONFIG_LIBSSH2=m" >> $config_host_mak
 -  echo "LIBSSH2_CFLAGS=$libssh2_cflags" >> $config_host_mak
 -  echo "LIBSSH2_LIBS=$libssh2_libs" >> $config_host_mak
 +if test "$libssh" = "yes" ; then
 +  echo "CONFIG_LIBSSH=m" >> $config_host_mak
 +  echo "LIBSSH_CFLAGS=$libssh_cflags" >> $config_host_mak
 +  echo "LIBSSH_LIBS=$libssh_libs" >> $config_host_mak
  fi
  if test "$live_block_migration" = "yes" ; then
 diff --git a/block/Makefile.objs b/block/Makefile.objs
 index XXXXXXX..XXXXXXX 100644
---- a/blockdev.c
+--- a/block/Makefile.objs
-+++ b/blockdev.c
++++ b/block/Makefile.objs
-@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
+@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_CURL) += curl.o
-             s->has_snapshot_node_name ? s->snapshot_node_name : NULL;
+ block-obj-$(CONFIG_RBD) += rbd.o
+ block-obj-$(CONFIG_GLUSTERFS) += gluster.o
-         if (node_name && !snapshot_node_name) {
+ block-obj-$(CONFIG_VXHS) += vxhs.o
--            error_setg(errp, "New snapshot node name missing");
+-block-obj-$(CONFIG_LIBSSH2) += ssh.o
-+            error_setg(errp, "New overlay node name missing");
++block-obj-$(CONFIG_LIBSSH) += ssh.o
  block-obj-y += accounting.o dirty-bitmap.o
  block-obj-y += write-threshold.o
  block-obj-y += backup.o
@@ -XXX,XX +XXX,XX @@ rbd.o-libs         := $(RBD_LIBS)
  gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
  gluster.o-libs     := $(GLUSTERFS_LIBS)
  vxhs.o-libs        := $(VXHS_LIBS)
 -ssh.o-cflags       := $(LIBSSH2_CFLAGS)
 -ssh.o-libs         := $(LIBSSH2_LIBS)
 +ssh.o-cflags       := $(LIBSSH_CFLAGS)
 +ssh.o-libs         := $(LIBSSH_LIBS)
  block-obj-dmg-bz2-$(CONFIG_BZIP2) += dmg-bz2.o
  block-obj-$(if $(CONFIG_DMG),m,n) += $(block-obj-dmg-bz2-y)
  dmg-bz2.o-libs     := $(BZIP2_LIBS)
 diff --git a/block/ssh.c b/block/ssh.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/ssh.c
 +++ b/block/ssh.c
@@ -XXX,XX +XXX,XX @@
  #include "qemu/osdep.h"
 -#include <libssh2.h>
 -#include <libssh2_sftp.h>
 +#include <libssh/libssh.h>
 +#include <libssh/sftp.h>
  #include "block/block_int.h"
  #include "block/qdict.h"
@@ -XXX,XX +XXX,XX @@
  #include "trace.h"
  /*
 - * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
 - * that this requires that libssh2 was specially compiled with the
 - * `./configure --enable-debug' option, so most likely you will have
 - * to compile it yourself.  The meaning of <bitmask> is described
 - * here: http://www.libssh2.org/libssh2_trace.html
 + * TRACE_LIBSSH=<level> enables tracing in libssh itself.
 + * The meaning of <level> is described here:
 + * http://api.libssh.org/master/group__libssh__log.html
   */
 -#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
 +#define TRACE_LIBSSH  0 /* see: SSH_LOG_* */
  typedef struct BDRVSSHState {
      /* Coroutine. */
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVSSHState {
      /* SSH connection. */
      int sock;                         /* socket */
 -    LIBSSH2_SESSION *session;         /* ssh session */
 -    LIBSSH2_SFTP *sftp;               /* sftp session */
 -    LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */
 +    ssh_session session;              /* ssh session */
 +    sftp_session sftp;                /* sftp session */
 +    sftp_file sftp_handle;            /* sftp remote file handle */
 -    /* See ssh_seek() function below. */
 -    int64_t offset;
 -    bool offset_op_read;
 -
 -    /* File attributes at open.  We try to keep the .filesize field
 +    /*
 +     * File attributes at open.  We try to keep the .size field
       * updated if it changes (eg by writing at the end of the file).
       */
 -    LIBSSH2_SFTP_ATTRIBUTES attrs;
 +    sftp_attributes attrs;
      InetSocketAddress *inet;
@@ -XXX,XX +XXX,XX @@ static void ssh_state_init(BDRVSSHState *s)
  {
      memset(s, 0, sizeof *s);
      s->sock = -1;
 -    s->offset = -1;
      qemu_co_mutex_init(&s->lock);
  }
@@ -XXX,XX +XXX,XX @@ static void ssh_state_free(BDRVSSHState *s)
  {
      g_free(s->user);
 +    if (s->attrs) {
 +        sftp_attributes_free(s->attrs);
 +    }
      if (s->sftp_handle) {
 -        libssh2_sftp_close(s->sftp_handle);
 +        sftp_close(s->sftp_handle);
      }
      if (s->sftp) {
 -        libssh2_sftp_shutdown(s->sftp);
 +        sftp_free(s->sftp);
      }
      if (s->session) {
 -        libssh2_session_disconnect(s->session,
 -                                   "from qemu ssh client: "
 -                                   "user closed the connection");
 -        libssh2_session_free(s->session);
 -    }
 -    if (s->sock >= 0) {
 -        close(s->sock);
 +        ssh_disconnect(s->session);
 +        ssh_free(s->session); /* This frees s->sock */
      }
  }
@@ -XXX,XX +XXX,XX @@ session_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
      va_end(args);
      if (s->session) {
 -        char *ssh_err;
 +        const char *ssh_err;
          int ssh_err_code;
 -        /* This is not an errno.  See <libssh2.h>. */
 -        ssh_err_code = libssh2_session_last_error(s->session,
 -                                                  &ssh_err, NULL, 0);
 -        error_setg(errp, "%s: %s (libssh2 error code: %d)",
 +        /* This is not an errno.  See <libssh/libssh.h>. */
 +        ssh_err = ssh_get_error(s->session);
 +        ssh_err_code = ssh_get_error_code(s->session);
 +        error_setg(errp, "%s: %s (libssh error code: %d)",
                     msg, ssh_err, ssh_err_code);
      } else {
          error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
      va_end(args);
      if (s->sftp) {
 -        char *ssh_err;
 +        const char *ssh_err;
          int ssh_err_code;
 -        unsigned long sftp_err_code;
 +        int sftp_err_code;
 -        /* This is not an errno.  See <libssh2.h>. */
 -        ssh_err_code = libssh2_session_last_error(s->session,
 -                                                  &ssh_err, NULL, 0);
 -        /* See <libssh2_sftp.h>. */
 -        sftp_err_code = libssh2_sftp_last_error((s)->sftp);
 +        /* This is not an errno.  See <libssh/libssh.h>. */
 +        ssh_err = ssh_get_error(s->session);
 +        ssh_err_code = ssh_get_error_code(s->session);
 +        /* See <libssh/sftp.h>. */
 +        sftp_err_code = sftp_get_error(s->sftp);
          error_setg(errp,
 -                   "%s: %s (libssh2 error code: %d, sftp error code: %lu)",
 +                   "%s: %s (libssh error code: %d, sftp error code: %d)",
                     msg, ssh_err, ssh_err_code, sftp_err_code);
      } else {
          error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
  static void sftp_error_trace(BDRVSSHState *s, const char *op)
  {
 -    char *ssh_err;
 +    const char *ssh_err;
      int ssh_err_code;
 -    unsigned long sftp_err_code;
 +    int sftp_err_code;
 -    /* This is not an errno.  See <libssh2.h>. */
 -    ssh_err_code = libssh2_session_last_error(s->session,
 -                                              &ssh_err, NULL, 0);
 -    /* See <libssh2_sftp.h>. */
 -    sftp_err_code = libssh2_sftp_last_error((s)->sftp);
 +    /* This is not an errno.  See <libssh/libssh.h>. */
 +    ssh_err = ssh_get_error(s->session);
 +    ssh_err_code = ssh_get_error_code(s->session);
 +    /* See <libssh/sftp.h>. */
 +    sftp_err_code = sftp_get_error(s->sftp);
      trace_sftp_error(op, ssh_err, ssh_err_code, sftp_err_code);
  }
@@ -XXX,XX +XXX,XX @@ static void ssh_parse_filename(const char *filename, QDict *options,
      parse_uri(filename, options, errp);
  }
 -static int check_host_key_knownhosts(BDRVSSHState *s,
 -                                     const char *host, int port, Error **errp)
 +static int check_host_key_knownhosts(BDRVSSHState *s, Error **errp)
  {
 -    const char *home;
 -    char *knh_file = NULL;
 -    LIBSSH2_KNOWNHOSTS *knh = NULL;
 -    struct libssh2_knownhost *found;
 -    int ret, r;
 -    const char *hostkey;
 -    size_t len;
 -    int type;
 -
 -    hostkey = libssh2_session_hostkey(s->session, &len, &type);
 -    if (!hostkey) {
 +    int ret;
 +#ifdef HAVE_LIBSSH_0_8
 +    enum ssh_known_hosts_e state;
 +    int r;
 +    ssh_key pubkey;
 +    enum ssh_keytypes_e pubkey_type;
 +    unsigned char *server_hash = NULL;
 +    size_t server_hash_len;
 +    char *fingerprint = NULL;
 +
 +    state = ssh_session_is_known_server(s->session);
 +    trace_ssh_server_status(state);
 +
 +    switch (state) {
 +    case SSH_KNOWN_HOSTS_OK:
 +        /* OK */
 +        trace_ssh_check_host_key_knownhosts();
 +        break;
 +    case SSH_KNOWN_HOSTS_CHANGED:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to read remote host key");
 +        r = ssh_get_server_publickey(s->session, &pubkey);
 +        if (r == 0) {
 +            r = ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256,
 +                                       &server_hash, &server_hash_len);
 +            pubkey_type = ssh_key_type(pubkey);
 +            ssh_key_free(pubkey);
 +        }
 +        if (r == 0) {
 +            fingerprint = ssh_get_fingerprint_hash(SSH_PUBLICKEY_HASH_SHA256,
 +                                                   server_hash,
 +                                                   server_hash_len);
 +            ssh_clean_pubkey_hash(&server_hash);
 +        }
 +        if (fingerprint) {
 +            error_setg(errp,
 +                       "host key (%s key with fingerprint %s) does not match "
 +                       "the one in known_hosts; this may be a possible attack",
 +                       ssh_key_type_to_char(pubkey_type), fingerprint);
 +            ssh_string_free_char(fingerprint);
 +        } else  {
 +            error_setg(errp,
 +                       "host key does not match the one in known_hosts; this "
 +                       "may be a possible attack");
 +        }
          goto out;
 -    }
 -
 -    knh = libssh2_knownhost_init(s->session);
 -    if (!knh) {
 +    case SSH_KNOWN_HOSTS_OTHER:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                           "failed to initialize known hosts support");
 +        error_setg(errp,
 +                   "host key for this server not found, another type exists");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_UNKNOWN:
 +        ret = -EINVAL;
 +        error_setg(errp, "no host key was found in known_hosts");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_NOT_FOUND:
 +        ret = -ENOENT;
 +        error_setg(errp, "known_hosts file not found");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_ERROR:
 +        ret = -EINVAL;
 +        error_setg(errp, "error while checking the host");
 +        goto out;
 +    default:
 +        ret = -EINVAL;
 +        error_setg(errp, "error while checking for known server (%d)", state);
          goto out;
      }
 +#else /* !HAVE_LIBSSH_0_8 */
 +    int state;
 -    home = getenv("HOME");
 -    if (home) {
 -        knh_file = g_strdup_printf("%s/.ssh/known_hosts", home);
 -    } else {
 -        knh_file = g_strdup_printf("/root/.ssh/known_hosts");
 -    }
 -
 -    /* Read all known hosts from OpenSSH-style known_hosts file. */
 -    libssh2_knownhost_readfile(knh, knh_file, LIBSSH2_KNOWNHOST_FILE_OPENSSH);
 +    state = ssh_is_server_known(s->session);
 +    trace_ssh_server_status(state);
 -    r = libssh2_knownhost_checkp(knh, host, port, hostkey, len,
 -                                 LIBSSH2_KNOWNHOST_TYPE_PLAIN|
 -                                 LIBSSH2_KNOWNHOST_KEYENC_RAW,
 -                                 &found);
 -    switch (r) {
 -    case LIBSSH2_KNOWNHOST_CHECK_MATCH:
 +    switch (state) {
 +    case SSH_SERVER_KNOWN_OK:
          /* OK */
 -        trace_ssh_check_host_key_knownhosts(found->key);
 +        trace_ssh_check_host_key_knownhosts();
          break;
 -    case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
 +    case SSH_SERVER_KNOWN_CHANGED:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                      "host key does not match the one in known_hosts"
 -                      " (found key %s)", found->key);
 +        error_setg(errp,
 +                   "host key does not match the one in known_hosts; this "
 +                   "may be a possible attack");
          goto out;
 -    case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND:
 +    case SSH_SERVER_FOUND_OTHER:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "no host key was found in known_hosts");
 +        error_setg(errp,
 +                   "host key for this server not found, another type exists");
 +        goto out;
 +    case SSH_SERVER_FILE_NOT_FOUND:
 +        ret = -ENOENT;
 +        error_setg(errp, "known_hosts file not found");
          goto out;
 -    case LIBSSH2_KNOWNHOST_CHECK_FAILURE:
 +    case SSH_SERVER_NOT_KNOWN:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                      "failure matching the host key with known_hosts");
 +        error_setg(errp, "no host key was found in known_hosts");
 +        goto out;
 +    case SSH_SERVER_ERROR:
 +        ret = -EINVAL;
 +        error_setg(errp, "server error");
          goto out;
      default:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "unknown error matching the host key"
 -                      " with known_hosts (%d)", r);
 +        error_setg(errp, "error while checking for known server (%d)", state);
          goto out;
      }
 +#endif /* !HAVE_LIBSSH_0_8 */
      /* known_hosts checking successful. */
      ret = 0;
   out:
 -    if (knh != NULL) {
 -        libssh2_knownhost_free(knh);
 -    }
 -    g_free(knh_file);
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int compare_fingerprint(const unsigned char *fingerprint, size_t len,
  static int
  check_host_key_hash(BDRVSSHState *s, const char *hash,
 -                    int hash_type, size_t fingerprint_len, Error **errp)
 +                    enum ssh_publickey_hash_type type, Error **errp)
  {
 -    const char *fingerprint;
 -
 -    fingerprint = libssh2_hostkey_hash(s->session, hash_type);
 -    if (!fingerprint) {
 +    int r;
 +    ssh_key pubkey;
 +    unsigned char *server_hash;
 +    size_t server_hash_len;
 +
 +#ifdef HAVE_LIBSSH_0_8
 +    r = ssh_get_server_publickey(s->session, &pubkey);
 +#else
 +    r = ssh_get_publickey(s->session, &pubkey);
 +#endif
 +    if (r != SSH_OK) {
          session_error_setg(errp, s, "failed to read remote host key");
          return -EINVAL;
      }
 -    if(compare_fingerprint((unsigned char *) fingerprint, fingerprint_len,
 -                           hash) != 0) {
 +    r = ssh_get_publickey_hash(pubkey, type, &server_hash, &server_hash_len);
 +    ssh_key_free(pubkey);
 +    if (r != 0) {
 +        session_error_setg(errp, s,
 +                           "failed reading the hash of the server SSH key");
 +        return -EINVAL;
 +    }
 +
 +    r = compare_fingerprint(server_hash, server_hash_len, hash);
 +    ssh_clean_pubkey_hash(&server_hash);
 +    if (r != 0) {
          error_setg(errp, "remote host key does not match host_key_check '%s'",
                     hash);
          return -EPERM;
@@ -XXX,XX +XXX,XX @@ check_host_key_hash(BDRVSSHState *s, const char *hash,
      return 0;
  }
 -static int check_host_key(BDRVSSHState *s, const char *host, int port,
 -                          SshHostKeyCheck *hkc, Error **errp)
 +static int check_host_key(BDRVSSHState *s, SshHostKeyCheck *hkc, Error **errp)
  {
      SshHostKeyCheckMode mode;
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
      case SSH_HOST_KEY_CHECK_MODE_HASH:
          if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) {
              return check_host_key_hash(s, hkc->u.hash.hash,
 -                                       LIBSSH2_HOSTKEY_HASH_MD5, 16, errp);
 +                                       SSH_PUBLICKEY_HASH_MD5, errp);
          } else if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_SHA1) {
              return check_host_key_hash(s, hkc->u.hash.hash,
 -                                       LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp);
 +                                       SSH_PUBLICKEY_HASH_SHA1, errp);
          }
          g_assert_not_reached();
          break;
      case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS:
 -        return check_host_key_knownhosts(s, host, port, errp);
 +        return check_host_key_knownhosts(s, errp);
      default:
          g_assert_not_reached();
      }
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
      return -EINVAL;
  }
 -static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
 +static int authenticate(BDRVSSHState *s, Error **errp)
  {
      int r, ret;
 -    const char *userauthlist;
 -    LIBSSH2_AGENT *agent = NULL;
 -    struct libssh2_agent_publickey *identity;
 -    struct libssh2_agent_publickey *prev_identity = NULL;
 +    int method;
 -    userauthlist = libssh2_userauth_list(s->session, user, strlen(user));
 -    if (strstr(userauthlist, "publickey") == NULL) {
 +    /* Try to authenticate with the "none" method. */
 +    r = ssh_userauth_none(s->session, NULL);
 +    if (r == SSH_AUTH_ERROR) {
          ret = -EPERM;
 -        error_setg(errp,
 -                "remote server does not support \"publickey\" authentication");
 +        session_error_setg(errp, s, "failed to authenticate using none "
 +                                    "authentication");
          goto out;
 -    }
 -
 -    /* Connect to ssh-agent and try each identity in turn. */
 -    agent = libssh2_agent_init(s->session);
 -    if (!agent) {
 -        ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to initialize ssh-agent support");
 -        goto out;
 -    }
 -    if (libssh2_agent_connect(agent)) {
 -        ret = -ECONNREFUSED;
 -        session_error_setg(errp, s, "failed to connect to ssh-agent");
 -        goto out;
 -    }
 -    if (libssh2_agent_list_identities(agent)) {
 -        ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                           "failed requesting identities from ssh-agent");
 +    } else if (r == SSH_AUTH_SUCCESS) {
 +        /* Authenticated! */
 +        ret = 0;
          goto out;
      }
 -    for(;;) {
 -        r = libssh2_agent_get_identity(agent, &identity, prev_identity);
 -        if (r == 1) {           /* end of list */
 -            break;
 -        }
 -        if (r < 0) {
 +    method = ssh_userauth_list(s->session, NULL);
 +    trace_ssh_auth_methods(method);
 +
 +    /*
 +     * Try to authenticate with publickey, using the ssh-agent
 +     * if available.
 +     */
 +    if (method & SSH_AUTH_METHOD_PUBLICKEY) {
 +        r = ssh_userauth_publickey_auto(s->session, NULL, NULL);
 +        if (r == SSH_AUTH_ERROR) {
              ret = -EINVAL;
 -            session_error_setg(errp, s,
 -                               "failed to obtain identity from ssh-agent");
 +            session_error_setg(errp, s, "failed to authenticate using "
 +                                        "publickey authentication");
              goto out;
 -        }
 -        r = libssh2_agent_userauth(agent, user, identity);
 -        if (r == 0) {
 +        } else if (r == SSH_AUTH_SUCCESS) {
              /* Authenticated! */
              ret = 0;
              goto out;
          }
+-        /* Failed to authenticate with this identity, try the next one. */
-         if (snapshot_node_name &&
+-        prev_identity = identity;
-             bdrv_lookup_bs(snapshot_node_name, snapshot_node_name, NULL)) {
+     }
--            error_setg(errp, "New snapshot node name already in use");
-+            error_setg(errp, "New overlay node name already in use");
+     ret = -EPERM;
-             goto out;
+@@ -XXX,XX +XXX,XX @@ static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
                 "and the identities held by your ssh-agent");
   out:
 -    if (agent != NULL) {
 -        /* Note: libssh2 implementation implicitly calls
 -         * libssh2_agent_disconnect if necessary.
 -         */
 -        libssh2_agent_free(agent);
 -    }
 -
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
                            int ssh_flags, int creat_mode, Error **errp)
  {
      int r, ret;
 -    long port = 0;
 +    unsigned int port = 0;
 +    int new_sock = -1;
      if (opts->has_user) {
          s->user = g_strdup(opts->user);
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
      s->inet = opts->server;
      opts->server = NULL;
 -    if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) {
 +    if (qemu_strtoui(s->inet->port, NULL, 10, &port) < 0) {
          error_setg(errp, "Use only numeric port value");
          ret = -EINVAL;
          goto err;
      }
      /* Open the socket and connect. */
 -    s->sock = inet_connect_saddr(s->inet, errp);
 -    if (s->sock < 0) {
 +    new_sock = inet_connect_saddr(s->inet, errp);
 +    if (new_sock < 0) {
          ret = -EIO;
          goto err;
      }
 +    /*
 +     * Try to disable the Nagle algorithm on TCP sockets to reduce latency,
 +     * but do not fail if it cannot be disabled.
 +     */
 +    r = socket_set_nodelay(new_sock);
 +    if (r < 0) {
 +        warn_report("can't set TCP_NODELAY for the ssh server %s: %s",
 +                    s->inet->host, strerror(errno));
 +    }
 +
      /* Create SSH session. */
 -    s->session = libssh2_session_init();
 +    s->session = ssh_new();
      if (!s->session) {
          ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to initialize libssh2 session");
 +        session_error_setg(errp, s, "failed to initialize libssh session");
          goto err;
      }
 -#if TRACE_LIBSSH2 != 0
 -    libssh2_trace(s->session, TRACE_LIBSSH2);
 -#endif
 +    /*
 +     * Make sure we are in blocking mode during the connection and
 +     * authentication phases.
 +     */
 +    ssh_set_blocking(s->session, 1);
 -    r = libssh2_session_handshake(s->session, s->sock);
 -    if (r != 0) {
 +    r = ssh_options_set(s->session, SSH_OPTIONS_USER, s->user);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the user in the libssh session");
 +        goto err;
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_HOST, s->inet->host);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the host in the libssh session");
 +        goto err;
 +    }
 +
 +    if (port > 0) {
 +        r = ssh_options_set(s->session, SSH_OPTIONS_PORT, &port);
 +        if (r < 0) {
 +            ret = -EINVAL;
 +            session_error_setg(errp, s,
 +                               "failed to set the port in the libssh session");
 +            goto err;
 +        }
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_COMPRESSION, "none");
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to disable the compression in the libssh "
 +                           "session");
 +        goto err;
 +    }
 +
 +    /* Read ~/.ssh/config. */
 +    r = ssh_options_parse_config(s->session, NULL);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s, "failed to parse ~/.ssh/config");
 +        goto err;
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_FD, &new_sock);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the socket in the libssh session");
 +        goto err;
 +    }
 +    /* libssh took ownership of the socket. */
 +    s->sock = new_sock;
 +    new_sock = -1;
 +
 +    /* Connect. */
 +    r = ssh_connect(s->session);
 +    if (r != SSH_OK) {
          ret = -EINVAL;
          session_error_setg(errp, s, "failed to establish SSH session");
          goto err;
      }
      /* Check the remote host's key against known_hosts. */
 -    ret = check_host_key(s, s->inet->host, port, opts->host_key_check, errp);
 +    ret = check_host_key(s, opts->host_key_check, errp);
      if (ret < 0) {
          goto err;
      }
      /* Authenticate. */
 -    ret = authenticate(s, s->user, errp);
 +    ret = authenticate(s, errp);
      if (ret < 0) {
          goto err;
      }
      /* Start SFTP. */
 -    s->sftp = libssh2_sftp_init(s->session);
 +    s->sftp = sftp_new(s->session);
      if (!s->sftp) {
 -        session_error_setg(errp, s, "failed to initialize sftp handle");
 +        session_error_setg(errp, s, "failed to create sftp handle");
 +        ret = -EINVAL;
 +        goto err;
 +    }
 +
 +    r = sftp_init(s->sftp);
 +    if (r < 0) {
 +        sftp_error_setg(errp, s, "failed to initialize sftp handle");
          ret = -EINVAL;
          goto err;
      }
      /* Open the remote file. */
      trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
 -    s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
 -                                       creat_mode);
 +    s->sftp_handle = sftp_open(s->sftp, opts->path, ssh_flags, creat_mode);
      if (!s->sftp_handle) {
 -        session_error_setg(errp, s, "failed to open remote file '%s'",
 -                           opts->path);
 +        sftp_error_setg(errp, s, "failed to open remote file '%s'",
 +                        opts->path);
          ret = -EINVAL;
          goto err;
      }
 -    r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs);
 -    if (r < 0) {
 +    /* Make sure the SFTP file is handled in blocking mode. */
 +    sftp_file_set_blocking(s->sftp_handle);
 +
 +    s->attrs = sftp_fstat(s->sftp_handle);
 +    if (!s->attrs) {
          sftp_error_setg(errp, s, "failed to read file attributes");
          return -EINVAL;
      }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
      return 0;
   err:
 +    if (s->attrs) {
 +        sftp_attributes_free(s->attrs);
 +    }
 +    s->attrs = NULL;
      if (s->sftp_handle) {
 -        libssh2_sftp_close(s->sftp_handle);
 +        sftp_close(s->sftp_handle);
      }
      s->sftp_handle = NULL;
      if (s->sftp) {
 -        libssh2_sftp_shutdown(s->sftp);
 +        sftp_free(s->sftp);
      }
      s->sftp = NULL;
      if (s->session) {
 -        libssh2_session_disconnect(s->session,
 -                                   "from qemu ssh client: "
 -                                   "error opening connection");
 -        libssh2_session_free(s->session);
 +        ssh_disconnect(s->session);
 +        ssh_free(s->session);
      }
      s->session = NULL;
 +    s->sock = -1;
 +    if (new_sock >= 0) {
 +        close(new_sock);
 +    }
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
      ssh_state_init(s);
 -    ssh_flags = LIBSSH2_FXF_READ;
 +    ssh_flags = 0;
      if (bdrv_flags & BDRV_O_RDWR) {
 -        ssh_flags |= LIBSSH2_FXF_WRITE;
 +        ssh_flags |= O_RDWR;
 +    } else {
 +        ssh_flags |= O_RDONLY;
      }
      opts = ssh_parse_options(options, errp);
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
      }
      /* Go non-blocking. */
 -    libssh2_session_set_blocking(s->session, 0);
 +    ssh_set_blocking(s->session, 0);
      qapi_free_BlockdevOptionsSsh(opts);
      return 0;
   err:
 -    if (s->sock >= 0) {
 -        close(s->sock);
 -    }
 -    s->sock = -1;
 -
      qapi_free_BlockdevOptionsSsh(opts);
      return ret;
@@ -XXX,XX +XXX,XX @@ static int ssh_grow_file(BDRVSSHState *s, int64_t offset, Error **errp)
  {
      ssize_t ret;
      char c[1] = { '\0' };
 -    int was_blocking = libssh2_session_get_blocking(s->session);
 +    int was_blocking = ssh_is_blocking(s->session);
      /* offset must be strictly greater than the current size so we do
       * not overwrite anything */
 -    assert(offset > 0 && offset > s->attrs.filesize);
 +    assert(offset > 0 && offset > s->attrs->size);
 -    libssh2_session_set_blocking(s->session, 1);
 +    ssh_set_blocking(s->session, 1);
 -    libssh2_sftp_seek64(s->sftp_handle, offset - 1);
 -    ret = libssh2_sftp_write(s->sftp_handle, c, 1);
 +    sftp_seek64(s->sftp_handle, offset - 1);
 +    ret = sftp_write(s->sftp_handle, c, 1);
 -    libssh2_session_set_blocking(s->session, was_blocking);
 +    ssh_set_blocking(s->session, was_blocking);
      if (ret < 0) {
          sftp_error_setg(errp, s, "Failed to grow file");
          return -EIO;
      }
 -    s->attrs.filesize = offset;
 +    s->attrs->size = offset;
      return 0;
  }
@@ -XXX,XX +XXX,XX @@ static int ssh_co_create(BlockdevCreateOptions *options, Error **errp)
      ssh_state_init(&s);
      ret = connect_to_ssh(&s, opts->location,
 -                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
 -                         LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
 +                         O_RDWR | O_CREAT | O_TRUNC,
 , errp);
      if (ret < 0) {
          goto fail;
@@ -XXX,XX +XXX,XX @@ static int ssh_has_zero_init(BlockDriverState *bs)
      /* Assume false, unless we can positively prove it's true. */
      int has_zero_init = 0;
 -    if (s->attrs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) {
 -        if (s->attrs.permissions & LIBSSH2_SFTP_S_IFREG) {
 -            has_zero_init = 1;
 -        }
 +    if (s->attrs->type == SSH_FILEXFER_TYPE_REGULAR) {
 +        has_zero_init = 1;
      }
      return has_zero_init;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
          .co = qemu_coroutine_self()
      };
 -    r = libssh2_session_block_directions(s->session);
 +    r = ssh_get_poll_flags(s->session);
 -    if (r & LIBSSH2_SESSION_BLOCK_INBOUND) {
 +    if (r & SSH_READ_PENDING) {
          rd_handler = restart_coroutine;
      }
 -    if (r & LIBSSH2_SESSION_BLOCK_OUTBOUND) {
 +    if (r & SSH_WRITE_PENDING) {
          wr_handler = restart_coroutine;
      }
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
      trace_ssh_co_yield_back(s->sock);
  }
 -/* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
 - * in the remote file.  Notice that it just updates a field in the
 - * sftp_handle structure, so there is no network traffic and it cannot
 - * fail.
 - *
 - * However, `libssh2_sftp_seek64' does have a catastrophic effect on
 - * performance since it causes the handle to throw away all in-flight
 - * reads and buffered readahead data.  Therefore this function tries
 - * to be intelligent about when to call the underlying libssh2 function.
 - */
 -#define SSH_SEEK_WRITE 0
 -#define SSH_SEEK_READ  1
 -#define SSH_SEEK_FORCE 2
 -
 -static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
 -{
 -    bool op_read = (flags & SSH_SEEK_READ) != 0;
 -    bool force = (flags & SSH_SEEK_FORCE) != 0;
 -
 -    if (force || op_read != s->offset_op_read || offset != s->offset) {
 -        trace_ssh_seek(offset);
 -        libssh2_sftp_seek64(s->sftp_handle, offset);
 -        s->offset = offset;
 -        s->offset_op_read = op_read;
 -    }
 -}
 -
  static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
                                   int64_t offset, size_t size,
                                   QEMUIOVector *qiov)
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
      trace_ssh_read(offset, size);
 -    ssh_seek(s, offset, SSH_SEEK_READ);
 +    trace_ssh_seek(offset);
 +    sftp_seek64(s->sftp_handle, offset);
      /* This keeps track of the current iovec element ('i'), where we
       * will write to next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
      buf = i->iov_base;
      end_of_vec = i->iov_base + i->iov_len;
 -    /* libssh2 has a hard-coded limit of 2000 bytes per request,
 -     * although it will also do readahead behind our backs.  Therefore
 -     * we may have to do repeated reads here until we have read 'size'
 -     * bytes.
 -     */
      for (got = 0; got < size; ) {
 +        size_t request_read_size;
      again:
 -        trace_ssh_read_buf(buf, end_of_vec - buf);
 -        r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
 -        trace_ssh_read_return(r);
 +        /*
 +         * The size of SFTP packets is limited to 32K bytes, so limit
 +         * the amount of data requested to 16K, as libssh currently
 +         * does not handle multiple requests on its own.
 +         */
 +        request_read_size = MIN(end_of_vec - buf, 16384);
 +        trace_ssh_read_buf(buf, end_of_vec - buf, request_read_size);
 +        r = sftp_read(s->sftp_handle, buf, request_read_size);
 +        trace_ssh_read_return(r, sftp_get_error(s->sftp));
 -        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +        if (r == SSH_AGAIN) {
              co_yield(s, bs);
              goto again;
          }
+-        if (r < 0) {
-@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
+-            sftp_error_trace(s, "read");
-     }
+-            s->offset = -1;
+-            return -EIO;
-     if (bdrv_has_blk(state->new_bs)) {
+-        }
--        error_setg(errp, "The snapshot is already in use");
+-        if (r == 0) {
-+        error_setg(errp, "The overlay is already in use");
++        if (r == SSH_EOF || (r == 0 && sftp_get_error(s->sftp) == SSH_FX_EOF)) {
-         goto out;
+             /* EOF: Short read so pad the buffer with zeroes and return it. */
-     }
+             qemu_iovec_memset(qiov, got, 0, size - got);
+             return 0;
-@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
+         }
-     }
++        if (r <= 0) {
++            sftp_error_trace(s, "read");
-     if (state->new_bs->backing != NULL) {
++            return -EIO;
--        error_setg(errp, "The snapshot already has a backing image");
++        }
-+        error_setg(errp, "The overlay already has a backing image");
-         goto out;
+         got += r;
-     }
+         buf += r;
+-        s->offset += r;
-     if (!state->new_bs->drv->supports_backing) {
+         if (buf >= end_of_vec && got < size) {
--        error_setg(errp, "The snapshot does not support backing images");
+             i++;
-+        error_setg(errp, "The overlay does not support backing images");
+             buf = i->iov_base;
-         goto out;
+@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
-     }
+     trace_ssh_write(offset, size);
-diff --git a/tests/qemu-iotests/085.out b/tests/qemu-iotests/085.out
 -    ssh_seek(s, offset, SSH_SEEK_WRITE);
 +    trace_ssh_seek(offset);
 +    sftp_seek64(s->sftp_handle, offset);
      /* This keeps track of the current iovec element ('i'), where we
       * will read from next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
      end_of_vec = i->iov_base + i->iov_len;
      for (written = 0; written < size; ) {
 +        size_t request_write_size;
      again:
 -        trace_ssh_write_buf(buf, end_of_vec - buf);
 -        r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
 -        trace_ssh_write_return(r);
 +        /*
 +         * Avoid too large data packets, as libssh currently does not
 +         * handle multiple requests on its own.
 +         */
 +        request_write_size = MIN(end_of_vec - buf, 131072);
 +        trace_ssh_write_buf(buf, end_of_vec - buf, request_write_size);
 +        r = sftp_write(s->sftp_handle, buf, request_write_size);
 +        trace_ssh_write_return(r, sftp_get_error(s->sftp));
 -        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +        if (r == SSH_AGAIN) {
              co_yield(s, bs);
              goto again;
          }
          if (r < 0) {
              sftp_error_trace(s, "write");
 -            s->offset = -1;
              return -EIO;
          }
 -        /* The libssh2 API is very unclear about this.  A comment in
 -         * the code says "nothing was acked, and no EAGAIN was
 -         * received!" which apparently means that no data got sent
 -         * out, and the underlying channel didn't return any EAGAIN
 -         * indication.  I think this is a bug in either libssh2 or
 -         * OpenSSH (server-side).  In any case, forcing a seek (to
 -         * discard libssh2 internal buffers), and then trying again
 -         * works for me.
 -         */
 -        if (r == 0) {
 -            ssh_seek(s, offset + written, SSH_SEEK_WRITE|SSH_SEEK_FORCE);
 -            co_yield(s, bs);
 -            goto again;
 -        }
          written += r;
          buf += r;
 -        s->offset += r;
          if (buf >= end_of_vec && written < size) {
              i++;
              buf = i->iov_base;
              end_of_vec = i->iov_base + i->iov_len;
          }
 -        if (offset + written > s->attrs.filesize)
 -            s->attrs.filesize = offset + written;
 +        if (offset + written > s->attrs->size) {
 +            s->attrs->size = offset + written;
 +        }
      }
      return 0;
@@ -XXX,XX +XXX,XX @@ static void unsafe_flush_warning(BDRVSSHState *s, const char *what)
      }
  }
 -#ifdef HAS_LIBSSH2_SFTP_FSYNC
 +#ifdef HAVE_LIBSSH_0_8
  static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
  {
      int r;
      trace_ssh_flush();
 +
 +    if (!sftp_extension_supported(s->sftp, "fsync@openssh.com", "1")) {
 +        unsafe_flush_warning(s, "OpenSSH >= 6.3");
 +        return 0;
 +    }
   again:
 -    r = libssh2_sftp_fsync(s->sftp_handle);
 -    if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +    r = sftp_fsync(s->sftp_handle);
 +    if (r == SSH_AGAIN) {
          co_yield(s, bs);
          goto again;
      }
 -    if (r == LIBSSH2_ERROR_SFTP_PROTOCOL &&
 -        libssh2_sftp_last_error(s->sftp) == LIBSSH2_FX_OP_UNSUPPORTED) {
 -        unsafe_flush_warning(s, "OpenSSH >= 6.3");
 -        return 0;
 -    }
      if (r < 0) {
          sftp_error_trace(s, "fsync");
          return -EIO;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
      return ret;
  }
 -#else /* !HAS_LIBSSH2_SFTP_FSYNC */
 +#else /* !HAVE_LIBSSH_0_8 */
  static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
  {
      BDRVSSHState *s = bs->opaque;
 -    unsafe_flush_warning(s, "libssh2 >= 1.4.4");
 +    unsafe_flush_warning(s, "libssh >= 0.8.0");
      return 0;
  }
 -#endif /* !HAS_LIBSSH2_SFTP_FSYNC */
 +#endif /* !HAVE_LIBSSH_0_8 */
  static int64_t ssh_getlength(BlockDriverState *bs)
  {
      BDRVSSHState *s = bs->opaque;
      int64_t length;
 -    /* Note we cannot make a libssh2 call here. */
 -    length = (int64_t) s->attrs.filesize;
 +    /* Note we cannot make a libssh call here. */
 +    length = (int64_t) s->attrs->size;
      trace_ssh_getlength(length);
      return length;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_truncate(BlockDriverState *bs, int64_t offset,
          return -ENOTSUP;
      }
 -    if (offset < s->attrs.filesize) {
 +    if (offset < s->attrs->size) {
          error_setg(errp, "ssh driver does not support shrinking files");
          return -ENOTSUP;
      }
 -    if (offset == s->attrs.filesize) {
 +    if (offset == s->attrs->size) {
          return 0;
      }
@@ -XXX,XX +XXX,XX @@ static void bdrv_ssh_init(void)
  {
      int r;
 -    r = libssh2_init(0);
 +    r = ssh_init();
      if (r != 0) {
 -        fprintf(stderr, "libssh2 initialization failed, %d\n", r);
 +        fprintf(stderr, "libssh initialization failed, %d\n", r);
          exit(EXIT_FAILURE);
      }
 +#if TRACE_LIBSSH != 0
 +    ssh_set_log_level(TRACE_LIBSSH);
 +#endif
 +
      bdrv_register(&bdrv_ssh);
  }
 diff --git a/.travis.yml b/.travis.yml
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/085.out
+--- a/.travis.yml
-+++ b/tests/qemu-iotests/085.out
++++ b/.travis.yml
-@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
+@@ -XXX,XX +XXX,XX @@ addons:
+       - libseccomp-dev
- === Invalid command - cannot create a snapshot using a file BDS ===
+       - libspice-protocol-dev
+       - libspice-server-dev
--{"error": {"class": "GenericError", "desc": "The snapshot does not support backing images"}}
+-      - libssh2-1-dev
-+{"error": {"class": "GenericError", "desc": "The overlay does not support backing images"}}
++      - libssh-dev
+       - liburcu-dev
- === Invalid command - snapshot node used as active layer ===
+       - libusb-1.0-0-dev
+       - libvte-2.91-dev
--{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
+@@ -XXX,XX +XXX,XX @@ matrix:
--{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
+             - libseccomp-dev
--{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
+             - libspice-protocol-dev
-+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+             - libspice-server-dev
-+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+-            - libssh2-1-dev
-+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
++            - libssh-dev
+             - liburcu-dev
- === Invalid command - snapshot node used as backing hd ===
+             - libusb-1.0-0-dev
+             - libvte-2.91-dev
-@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
+diff --git a/block/trace-events b/block/trace-events
- Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728
+index XXXXXXX..XXXXXXX 100644
- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base
+--- a/block/trace-events
 +++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ nbd_client_connect_success(const char *export_name) "export '%s'"
  # ssh.c
  ssh_restart_coroutine(void *co) "co=%p"
  ssh_flush(void) "fsync"
 -ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
 +ssh_check_host_key_knownhosts(void) "host key OK"
  ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
  ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
  ssh_co_yield_back(int sock) "s->sock=%d - back"
  ssh_getlength(int64_t length) "length=%" PRIi64
  ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
  ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
 -ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
 -ssh_read_return(ssize_t ret) "sftp_read returned %zd"
 +ssh_read_buf(void *buf, size_t size, size_t actual_size) "sftp_read buf=%p size=%zu (actual size=%zu)"
 +ssh_read_return(ssize_t ret, int sftp_err) "sftp_read returned %zd (sftp error=%d)"
  ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
 -ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
 -ssh_write_return(ssize_t ret) "sftp_write returned %zd"
 +ssh_write_buf(void *buf, size_t size, size_t actual_size) "sftp_write buf=%p size=%zu (actual size=%zu)"
 +ssh_write_return(ssize_t ret, int sftp_err) "sftp_write returned %zd (sftp error=%d)"
  ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
 +ssh_auth_methods(int methods) "auth methods=0x%x"
 +ssh_server_status(int status) "server status=%d"
  # curl.c
  curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
@@ -XXX,XX +XXX,XX @@ sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
  sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
  # ssh.c
 -sftp_error(const char *op, const char *ssh_err, int ssh_err_code, unsigned long sftp_err_code) "%s failed: %s (libssh2 error code: %d, sftp error code: %lu)"
 +sftp_error(const char *op, const char *ssh_err, int ssh_err_code, int sftp_err_code) "%s failed: %s (libssh error code: %d, sftp error code: %d)"
 diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi
 index XXXXXXX..XXXXXXX 100644
 --- a/docs/qemu-block-drivers.texi
 +++ b/docs/qemu-block-drivers.texi
@@ -XXX,XX +XXX,XX @@ print a warning when @code{fsync} is not supported:
  warning: ssh server @code{ssh.example.com:22} does not support fsync
 -With sufficiently new versions of libssh2 and OpenSSH, @code{fsync} is
 +With sufficiently new versions of libssh and OpenSSH, @code{fsync} is
  supported.
  @node disk_images_nvme
 diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/debian-win32-cross.docker
 +++ b/tests/docker/dockerfiles/debian-win32-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
          mxe-$TARGET-w64-mingw32.shared-curl \
          mxe-$TARGET-w64-mingw32.shared-glib \
          mxe-$TARGET-w64-mingw32.shared-libgcrypt \
 -        mxe-$TARGET-w64-mingw32.shared-libssh2 \
          mxe-$TARGET-w64-mingw32.shared-libusb1 \
          mxe-$TARGET-w64-mingw32.shared-lzo \
          mxe-$TARGET-w64-mingw32.shared-nettle \
 diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/debian-win64-cross.docker
 +++ b/tests/docker/dockerfiles/debian-win64-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
          mxe-$TARGET-w64-mingw32.shared-curl \
          mxe-$TARGET-w64-mingw32.shared-glib \
          mxe-$TARGET-w64-mingw32.shared-libgcrypt \
 -        mxe-$TARGET-w64-mingw32.shared-libssh2 \
          mxe-$TARGET-w64-mingw32.shared-libusb1 \
          mxe-$TARGET-w64-mingw32.shared-lzo \
          mxe-$TARGET-w64-mingw32.shared-nettle \
 diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/fedora.docker
 +++ b/tests/docker/dockerfiles/fedora.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
      libpng-devel \
      librbd-devel \
      libseccomp-devel \
 -    libssh2-devel \
 +    libssh-devel \
      libubsan \
      libusbx-devel \
      libxml2-devel \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
      mingw32-gtk3 \
      mingw32-libjpeg-turbo \
      mingw32-libpng \
 -    mingw32-libssh2 \
      mingw32-libtasn1 \
      mingw32-nettle \
      mingw32-pixman \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
      mingw64-gtk3 \
      mingw64-libjpeg-turbo \
      mingw64-libpng \
 -    mingw64-libssh2 \
      mingw64-libtasn1 \
      mingw64-nettle \
      mingw64-pixman \
 diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/ubuntu.docker
 +++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
      libsnappy-dev \
      libspice-protocol-dev \
      libspice-server-dev \
 -    libssh2-1-dev \
 +    libssh-dev \
      libusb-1.0-0-dev \
      libusbredirhost-dev \
      libvdeplug-dev \
 diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/ubuntu1804.docker
 +++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
      libsnappy-dev \
      libspice-protocol-dev \
      libspice-server-dev \
 -    libssh2-1-dev \
 +    libssh-dev \
      libusb-1.0-0-dev \
      libusbredirhost-dev \
      libvdeplug-dev \
 diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/207
 +++ b/tests/qemu-iotests/207
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
      iotests.img_info_log(remote_path)
 -    md5_key = subprocess.check_output(
 -        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
 -        'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
 -        shell=True).rstrip().decode('ascii')
 +    keys = subprocess.check_output(
 +        'ssh-keyscan 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
 +        'cut -d" " -f3',
 +        shell=True).rstrip().decode('ascii').split('\n')
 +
 +    # Mappings of base64 representations to digests
 +    md5_keys = {}
 +    sha1_keys = {}
 +
 +    for key in keys:
 +        md5_keys[key] = subprocess.check_output(
 +            'echo %s | base64 -d | md5sum -b | cut -d" " -f1' % key,
 +            shell=True).rstrip().decode('ascii')
 +
 +        sha1_keys[key] = subprocess.check_output(
 +            'echo %s | base64 -d | sha1sum -b | cut -d" " -f1' % key,
 +            shell=True).rstrip().decode('ascii')
      vm.launch()
 +
 +    # Find correct key first
 +    matching_key = None
 +    for key in keys:
 +        result = vm.qmp('blockdev-add',
 +                        driver='ssh', node_name='node0', path=disk_path,
 +                        server={
 +                             'host': '127.0.0.1',
 +                             'port': '22',
 +                        }, host_key_check={
 +                             'mode': 'hash',
 +                             'type': 'md5',
 +                             'hash': md5_keys[key],
 +                        })
 +
 +        if 'error' not in result:
 +            vm.qmp('blockdev-del', node_name='node0')
 +            matching_key = key
 +            break
 +
 +    if matching_key is None:
 +        vm.shutdown()
 +        iotests.notrun('Did not find a key that fits 127.0.0.1')
 +
      blockdev_create(vm, { 'driver': 'ssh',
                            'location': {
                                'path': disk_path,
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                                'host-key-check': {
                                    'mode': 'hash',
                                    'type': 'md5',
 -                                  'hash': md5_key,
 +                                  'hash': md5_keys[matching_key],
                                }
                            },
                            'size': 8388608 })
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
      iotests.img_info_log(remote_path)
 -    sha1_key = subprocess.check_output(
 -        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
 -        'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
 -        shell=True).rstrip().decode('ascii')
 -
      vm.launch()
      blockdev_create(vm, { 'driver': 'ssh',
                            'location': {
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                                'host-key-check': {
                                    'mode': 'hash',
                                    'type': 'sha1',
 -                                  'hash': sha1_key,
 +                                  'hash': sha1_keys[matching_key],
                                }
                            },
                            'size': 4194304 })
 diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/207.out
 +++ b/tests/qemu-iotests/207.out
@@ -XXX,XX +XXX,XX @@ virtual size: 4 MiB (4194304 bytes)
  {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
  {"return": {}}
--{"error": {"class": "GenericError", "desc": "The snapshot already has a backing image"}}
+-Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
-+{"error": {"class": "GenericError", "desc": "The overlay already has a backing image"}}
++Job failed: failed to open remote file '/this/is/not/an/existing/path': SFTP server: No such file (libssh error code: 1, sftp error code: 2)
+ {"execute": "job-dismiss", "arguments": {"id": "job0"}}
- === Invalid command - The node does not exist ===
+ {"return": {}}
 --
 .21.0

-[Qemu-devel] [PULL 14/20] qemu-img: Move quiet into ImgConvertState
+Deleted patch
-Move img_convert()'s quiet flag into the ImgConvertState so it is
-accessible by nested functions.  -q dictates that it suppresses anything
-but errors, so if those functions want to emit warnings, they need to
-query this flag first.  (There currently are no such warnings, but there
-will be as of the next patch.)
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20190507203508.18026-2-mreitz@redhat.com
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- qemu-img.c | 13 +++++++------
-file changed, 7 insertions(+), 6 deletions(-)
-diff --git a/qemu-img.c b/qemu-img.c
-index XXXXXXX..XXXXXXX 100644
---- a/qemu-img.c
-+++ b/qemu-img.c
-@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
-     int64_t target_backing_sectors; /* negative if unknown */
-     bool wr_in_order;
-     bool copy_range;
-+    bool quiet;
-     int min_sparse;
-     int alignment;
-     size_t cluster_sectors;
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-     QDict *open_opts = NULL;
-     char *options = NULL;
-     Error *local_err = NULL;
--    bool writethrough, src_writethrough, quiet = false, image_opts = false,
-+    bool writethrough, src_writethrough, image_opts = false,
-          skip_create = false, progress = false, tgt_image_opts = false;
-     int64_t ret = -EINVAL;
-     bool force_share = false;
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-             src_cache = optarg;
-             break;
-         case 'q':
--            quiet = true;
-+            s.quiet = true;
-             break;
-         case 'n':
-             skip_create = true;
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-     }
-     /* Initialize before goto out */
--    if (quiet) {
-+    if (s.quiet) {
-         progress = false;
-     }
-     qemu_progress_init(progress, 1.0);
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-     for (bs_i = 0; bs_i < s.src_num; bs_i++) {
-         s.src[bs_i] = img_open(image_opts, argv[optind + bs_i],
--                               fmt, src_flags, src_writethrough, quiet,
-+                               fmt, src_flags, src_writethrough, s.quiet,
-                                force_share);
-         if (!s.src[bs_i]) {
-             ret = -1;
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-     if (skip_create) {
-         s.target = img_open(tgt_image_opts, out_filename, out_fmt,
--                            flags, writethrough, quiet, false);
-+                            flags, writethrough, s.quiet, false);
-     } else {
-         /* TODO ultimately we should allow --target-image-opts
-          * to be used even when -n is not given.
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-          * to allow filenames in option syntax
-          */
-         s.target = img_open_file(out_filename, open_opts, out_fmt,
--                                 flags, writethrough, quiet, false);
-+                                 flags, writethrough, s.quiet, false);
-         open_opts = NULL; /* blk_new_open will have freed it */
-     }
-     if (!s.target) {
---
-.21.0

-[Qemu-devel] [PULL 15/20] qemu-img: Add salvaging mode to convert
+Deleted patch
-This adds a salvaging mode (--salvage) to qemu-img convert which ignores
-read errors and treats the respective areas as containing only zeroes.
-This can be used for instance to at least partially recover the data
-from terminally corrupted qcow2 images.
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20190507203508.18026-3-mreitz@redhat.com
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- qemu-img.c       | 90 +++++++++++++++++++++++++++++++++++++-----------
- qemu-img-cmds.hx |  4 +--
- qemu-img.texi    |  4 +++
-files changed, 75 insertions(+), 23 deletions(-)
-diff --git a/qemu-img.c b/qemu-img.c
-index XXXXXXX..XXXXXXX 100644
---- a/qemu-img.c
-+++ b/qemu-img.c
-@@ -XXX,XX +XXX,XX @@ enum {
-     OPTION_SIZE = 264,
-     OPTION_PREALLOCATION = 265,
-     OPTION_SHRINK = 266,
-+    OPTION_SALVAGE = 267,
- };
- typedef enum OutputFormat {
-@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
-     int64_t target_backing_sectors; /* negative if unknown */
-     bool wr_in_order;
-     bool copy_range;
-+    bool salvage;
-     bool quiet;
-     int min_sparse;
-     int alignment;
-@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
-     }
-     if (s->sector_next_status <= sector_num) {
--        int64_t count = n * BDRV_SECTOR_SIZE;
-+        uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
-+        int64_t count;
--        if (s->target_has_backing) {
-+        do {
-+            count = n * BDRV_SECTOR_SIZE;
-+
-+            if (s->target_has_backing) {
-+                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
-+                                        count, &count, NULL, NULL);
-+            } else {
-+                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-+                                              offset, count, &count, NULL,
-+                                              NULL);
-+            }
-+
-+            if (ret < 0) {
-+                if (s->salvage) {
-+                    if (n == 1) {
-+                        if (!s->quiet) {
-+                            warn_report("error while reading block status at "
-+                                        "offset %" PRIu64 ": %s", offset,
-+                                        strerror(-ret));
-+                        }
-+                        /* Just try to read the data, then */
-+                        ret = BDRV_BLOCK_DATA;
-+                        count = BDRV_SECTOR_SIZE;
-+                    } else {
-+                        /* Retry on a shorter range */
-+                        n = DIV_ROUND_UP(n, 4);
-+                    }
-+                } else {
-+                    error_report("error while reading block status at offset "
-+                                 "%" PRIu64 ": %s", offset, strerror(-ret));
-+                    return ret;
-+                }
-+            }
-+        } while (ret < 0);
--            ret = bdrv_block_status(blk_bs(s->src[src_cur]),
--                                    (sector_num - src_cur_offset) *
--                                    BDRV_SECTOR_SIZE,
--                                    count, &count, NULL, NULL);
--        } else {
--            ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
--                                          (sector_num - src_cur_offset) *
--                                          BDRV_SECTOR_SIZE,
--                                          count, &count, NULL, NULL);
--        }
--        if (ret < 0) {
--            error_report("error while reading block status of sector %" PRId64
--                         ": %s", sector_num, strerror(-ret));
--            return ret;
--        }
-         n = DIV_ROUND_UP(count, BDRV_SECTOR_SIZE);
-         if (ret & BDRV_BLOCK_ZERO) {
-@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
- static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
-                                         int nb_sectors, uint8_t *buf)
- {
-+    uint64_t single_read_until = 0;
-     int n, ret;
-     assert(nb_sectors <= s->buf_sectors);
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
-         BlockBackend *blk;
-         int src_cur;
-         int64_t bs_sectors, src_cur_offset;
-+        uint64_t offset;
-         /* In the case of compression with multiple source files, we can get a
-          * nb_sectors that spreads into the next part. So we must be able to
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
-         blk = s->src[src_cur];
-         bs_sectors = s->src_sectors[src_cur];
-+        offset = (sector_num - src_cur_offset) << BDRV_SECTOR_BITS;
-+
-         n = MIN(nb_sectors, bs_sectors - (sector_num - src_cur_offset));
-+        if (single_read_until > offset) {
-+            n = 1;
-+        }
--        ret = blk_co_pread(
--                blk, (sector_num - src_cur_offset) << BDRV_SECTOR_BITS,
--                n << BDRV_SECTOR_BITS, buf, 0);
-+        ret = blk_co_pread(blk, offset, n << BDRV_SECTOR_BITS, buf, 0);
-         if (ret < 0) {
--            return ret;
-+            if (s->salvage) {
-+                if (n > 1) {
-+                    single_read_until = offset + (n << BDRV_SECTOR_BITS);
-+                    continue;
-+                } else {
-+                    if (!s->quiet) {
-+                        warn_report("error while reading offset %" PRIu64
-+                                    ": %s", offset, strerror(-ret));
-+                    }
-+                    memset(buf, 0, BDRV_SECTOR_SIZE);
-+                }
-+            } else {
-+                return ret;
-+            }
-         }
-         sector_num += n;
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-             {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
-             {"force-share", no_argument, 0, 'U'},
-             {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
-+            {"salvage", no_argument, 0, OPTION_SALVAGE},
-             {0, 0, 0, 0}
-         };
-         c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-         case OPTION_IMAGE_OPTS:
-             image_opts = true;
-             break;
-+        case OPTION_SALVAGE:
-+            s.salvage = true;
-+            break;
-         case OPTION_TARGET_IMAGE_OPTS:
-             tgt_image_opts = true;
-             break;
-@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
-         goto fail_getopt;
-     }
-+    if (s.copy_range && s.salvage) {
-+        error_report("Cannot use copy offloading in salvaging mode");
-+        goto fail_getopt;
-+    }
-+
-     if (tgt_image_opts && !skip_create) {
-         error_report("--target-image-opts requires use of -n flag");
-         goto fail_getopt;
-diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
-index XXXXXXX..XXXXXXX 100644
---- a/qemu-img-cmds.hx
-+++ b/qemu-img-cmds.hx
-@@ -XXX,XX +XXX,XX @@ STEXI
- ETEXI
- DEF("convert", img_convert,
--    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename")
-+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
- STEXI
--@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename}
-+@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] [--salvage] @var{filename} [@var{filename2} [...]] @var{output_filename}
- ETEXI
- DEF("create", img_create,
-diff --git a/qemu-img.texi b/qemu-img.texi
-index XXXXXXX..XXXXXXX 100644
---- a/qemu-img.texi
-+++ b/qemu-img.texi
-@@ -XXX,XX +XXX,XX @@ improve performance if the data is remote, such as with NFS or iSCSI backends,
- but will not automatically sparsify zero sectors, and may result in a fully
- allocated target image depending on the host support for getting allocation
- information.
-+@item --salvage
-+Try to ignore I/O errors when reading.  Unless in quiet mode (@code{-q}), errors
-+will still be printed.  Areas that cannot be read from the source will be
-+treated as containing only zeroes.
- @end table
- Parameters to dd subcommand:
---
-.21.0

-[Qemu-devel] [PULL 17/20] blkdebug: Add "none" event
+[Qemu-devel] [PULL v2 8/8] iotests: Fix 205 for concurrent runs
-Together with @iotypes and @sector, this can be used to trap e.g. the
+Tests should place their files into the test directory.  This includes
-first read or write access to a certain sector without having to know
+Unix sockets.  205 currently fails to do so, which prevents it from
-what happens internally in the block layer, i.e. which "real" events
+being run concurrently.
 happen right before such an access.
 Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Message-id: 20190618210238.9524-1-mreitz@redhat.com
-Message-id: 20190507203508.18026-5-mreitz@redhat.com
+Reviewed-by: Eric Blake <eblake@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json | 4 +++-
+ tests/qemu-iotests/205 | 2 +-
- block/blkdebug.c     | 2 ++
+file changed, 1 insertion(+), 1 deletion(-)
 files changed, 5 insertions(+), 1 deletion(-)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
+diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/qapi/block-core.json
+--- a/tests/qemu-iotests/205
-+++ b/qapi/block-core.json
++++ b/tests/qemu-iotests/205
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ import iotests
- #
+ import time
- # @cluster_alloc_space: an allocation of file space for a cluster (since 4.1)
+ from iotests import qemu_img_create, qemu_io, filter_qemu_io, QemuIoInteractive
- #
-+# @none: triggers once at creation of the blkdebug node (since 4.1)
+-nbd_sock = 'nbd_sock'
-+#
++nbd_sock = os.path.join(iotests.test_dir, 'nbd_sock')
- # Since: 2.9
+ nbd_uri = 'nbd+unix:///exp?socket=' + nbd_sock
- ##
+ disk = os.path.join(iotests.test_dir, 'disk')
- { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
@@ -XXX,XX +XXX,XX @@
              'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev',
              'pwritev_zero', 'pwritev_done', 'empty_image_prepare',
              'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
 -            'cor_write', 'cluster_alloc_space'] }
 +            'cor_write', 'cluster_alloc_space', 'none'] }
  ##
  # @BlkdebugIOType:
 diff --git a/block/blkdebug.c b/block/blkdebug.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/blkdebug.c
 +++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ static int blkdebug_open(BlockDriverState *bs, QDict *options, int flags,
          goto out;
      }
 +    bdrv_debug_event(bs, BLKDBG_NONE);
 +
      ret = 0;
  out:
      if (ret < 0) {
 --
 .21.0

-[Qemu-devel] [PULL 18/20] blkdebug: Inject errors on .bdrv_co_block_status()
+Deleted patch
-Signed-off-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-id: 20190507203508.18026-6-mreitz@redhat.com
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- qapi/block-core.json | 5 ++++-
- block/blkdebug.c     | 8 ++++++++
-files changed, 12 insertions(+), 1 deletion(-)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
-+++ b/qapi/block-core.json
-@@ -XXX,XX +XXX,XX @@
- #
- # @flush: .bdrv_co_flush_to_disk()
- #
-+# @block-status: .bdrv_co_block_status()
-+#
- # Since: 4.1
- ##
- { 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
--  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
-+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush',
-+            'block-status' ] }
- ##
- # @BlkdebugInjectErrorOptions:
-diff --git a/block/blkdebug.c b/block/blkdebug.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/blkdebug.c
-+++ b/block/blkdebug.c
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,
-                                                  int64_t *map,
-                                                  BlockDriverState **file)
- {
-+    int err;
-+
-     assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment));
-+
-+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_BLOCK_STATUS);
-+    if (err) {
-+        return err;
-+    }
-+
-     return bdrv_co_block_status_from_file(bs, want_zero, offset, bytes,
-                                           pnum, map, file);
- }
---
-.21.0

The following changes since commit 5ec2eca83dc478ddf24077e02a8b34dd26cd3ff9:

Merge remote-tracking branch 'remotes/awilliam/tags/vfio-updates-20190613.0' into staging (2019-06-14 09:33:55 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-06-14

for you to fetch changes up to 21c1ce592a144188dfe59b9e156a97da412a59a2:

iotests: Test qemu-img convert -C --salvage (2019-06-14 15:09:42 +0200)

----------------------------------------------------------------
Block patches:
- Allow blockdev-backup from nodes that are not in qemu's main AIO
  context to newly added nodes
- Add salvaging mode to qemu-img convert
- Minor fixes to tests, documentation, and for less Valgrind annoyance

----------------------------------------------------------------
Andrey Shinkevich (1):
  hw/block/fdc: floppy command FIFO memory initialization

John Snow (6):
  blockdev-backup: don't check aio_context too early
  iotests.py: do not use infinite waits
  QEMUMachine: add events_wait method
  iotests.py: rewrite run_job to be pickier
  iotests: add iotest 256 for testing blockdev-backup across iothread
    contexts
  event_match: always match on None value

Max Reitz (12):
  iotests: Filter 175's allocation information
  iotests: Fix intermittent failure in 219
  qemu-img: Fix options leakage in img_rebase()
  qapi/block-core: Overlays are not snapshots
  blockdev: Overlays are not snapshots
  qemu-img: Move quiet into ImgConvertState
  qemu-img: Add salvaging mode to convert
  blkdebug: Add @iotype error option
  blkdebug: Add "none" event
  blkdebug: Inject errors on .bdrv_co_block_status()
  iotests: Test qemu-img convert --salvage
  iotests: Test qemu-img convert -C --salvage

Vladimir Sementsov-Ogievskiy (1):
  iotests: restrict 254 to support only qcow2

qapi/block-core.json          |  53 ++++++++---
 block/blkdebug.c              |  60 ++++++++++--
 blockdev.c                    |  14 +--
 hw/block/fdc.c                |   1 +
 qemu-img.c                    | 106 +++++++++++++++------
 python/qemu/__init__.py       |  67 ++++++++++----
 qemu-img-cmds.hx              |   4 +-
 qemu-img.texi                 |   4 +
 tests/qemu-iotests/082        |   1 +
 tests/qemu-iotests/082.out    |   3 +
 tests/qemu-iotests/085.out    |  10 +-
 tests/qemu-iotests/175        |  26 +++++-
 tests/qemu-iotests/175.out    |   8 +-
 tests/qemu-iotests/219        |  13 ++-
 tests/qemu-iotests/251        | 170 ++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out    |  43 +++++++++
 tests/qemu-iotests/254        |   2 +
 tests/qemu-iotests/256        | 122 ++++++++++++++++++++++++
 tests/qemu-iotests/256.out    | 119 ++++++++++++++++++++++++
 tests/qemu-iotests/group      |   2 +
 tests/qemu-iotests/iotests.py |  60 +++++++-----
 21 files changed, 772 insertions(+), 116 deletions(-)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out
 create mode 100755 tests/qemu-iotests/256
 create mode 100644 tests/qemu-iotests/256.out

-- 
2.21.0

From: John Snow <jsnow@redhat.com>

in blockdev_backup_prepare, we check to make sure that the target is
associated with a compatible aio context. However, do_blockdev_backup is
called later and has some logic to move the target to a compatible
aio_context. The transaction version will fail certain commands
needlessly early as a result.

Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
will ultimately decide if the contexts are compatible or not.

Note: the transaction version has always disallowed this operation since
its initial commit bd8baecd (2014), whereas the version of
qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
enforce the aio_context switch instead. It's not clear, and I can't see
from the mailing list archives at the time, why the two functions take a
different approach. It wasn't until later in efd7556708b (2016) that the
standalone version tried to determine if it could set the context or
not.

Reported-by: aihua liang <aliang@redhat.com>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-2-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
     }
 
     aio_context = bdrv_get_aio_context(bs);
-    if (aio_context != bdrv_get_aio_context(target)) {
-        error_setg(errp, "Backup between two IO threads is not implemented");
-        return;
-    }
     aio_context_acquire(aio_context);
     state->bs = bs;
 
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Cap waits to 60 seconds so that iotests can fail gracefully if something
goes wrong.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-3-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
             output_list += [key + '=' + obj[key]]
         return ','.join(output_list)
 
-    def get_qmp_events_filtered(self, wait=True):
+    def get_qmp_events_filtered(self, wait=60.0):
         result = []
         for ev in self.get_qmp_events(wait=wait):
             result.append(filter_qmp_event(ev))
@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
 
     # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
-                pre_finalize=None):
+                pre_finalize=None, wait=60.0):
         error = None
         while True:
-            for ev in self.get_qmp_events_filtered(wait=True):
+            for ev in self.get_qmp_events_filtered(wait=wait):
                 if ev['event'] == 'JOB_STATUS_CHANGE':
                     status = ev['data']['status']
                     if status == 'aborting':
@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
         self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])),
                          self.vm.flatten_qmp_object(reference))
 
-    def cancel_and_wait(self, drive='drive0', force=False, resume=False):
+    def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0):
         '''Cancel a block job and wait for it to finish, returning the event'''
         result = self.vm.qmp('block-job-cancel', device=drive, force=force)
         self.assert_qmp(result, 'return', {})
@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
         cancelled = False
         result = None
         while not cancelled:
-            for event in self.vm.get_qmp_events(wait=True):
+            for event in self.vm.get_qmp_events(wait=wait):
                 if event['event'] == 'BLOCK_JOB_COMPLETED' or \
                    event['event'] == 'BLOCK_JOB_CANCELLED':
                     self.assert_qmp(event, 'data/device', drive)
@@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase):
         self.assert_no_active_block_jobs()
         return result
 
-    def wait_until_completed(self, drive='drive0', check_offset=True):
+    def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0):
         '''Wait for a block job to finish, returning the event'''
         while True:
-            for event in self.vm.get_qmp_events(wait=True):
+            for event in self.vm.get_qmp_events(wait=wait):
                 if event['event'] == 'BLOCK_JOB_COMPLETED':
                     self.assert_qmp(event, 'data/device', drive)
                     self.assert_qmp_absent(event, 'data/error')
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Instead of event_wait which looks for a single event, add an events_wait
which can look for any number of events simultaneously. However, it
will still only return one at a time, whichever happens first.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-4-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 python/qemu/__init__.py | 69 +++++++++++++++++++++++++++++------------
 1 file changed, 49 insertions(+), 20 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index XXXXXXX..XXXXXXX 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
         self._qmp.clear_events()
         return events
 
-    def event_wait(self, name, timeout=60.0, match=None):
+    @staticmethod
+    def event_match(event, match=None):
         """
-        Wait for specified timeout on named event in QMP; optionally filter
-        results by match.
+        Check if an event matches optional match criteria.
 
-        The 'match' is checked to be a recursive subset of the 'event'; skips
-        branch processing on match's value None
-           {"foo": {"bar": 1}} matches {"foo": None}
-           {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
+        The match criteria takes the form of a matching subdict. The event is
+        checked to be a superset of the subdict, recursively, with matching
+        values whenever those values are not None.
+
+        Examples, with the subdict queries on the left:
+         - None matches any object.
+         - {"foo": None} matches {"foo": {"bar": 1}}
+         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
+         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
         """
-        def event_match(event, match=None):
-            if match is None:
-                return True
+        if match is None:
+            return True
 
-            for key in match:
-                if key in event:
-                    if isinstance(event[key], dict):
-                        if not event_match(event[key], match[key]):
-                            return False
-                    elif event[key] != match[key]:
+        for key in match:
+            if key in event:
+                if isinstance(event[key], dict):
+                    if not QEMUMachine.event_match(event[key], match[key]):
                         return False
-                else:
+                elif event[key] != match[key]:
                     return False
+            else:
+                return False
+        return True
 
-            return True
+    def event_wait(self, name, timeout=60.0, match=None):
+        """
+        event_wait waits for and returns a named event from QMP with a timeout.
+
+        name: The event to wait for.
+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+        match: Optional match criteria. See event_match for details.
+        """
+        return self.events_wait([(name, match)], timeout)
+
+    def events_wait(self, events, timeout=60.0):
+        """
+        events_wait waits for and returns a named event from QMP with a timeout.
+
+        events: a sequence of (name, match_criteria) tuples.
+                The match criteria are optional and may be None.
+                See event_match for details.
+        timeout: QEMUMonitorProtocol.pull_event timeout parameter.
+        """
+        def _match(event):
+            for name, match in events:
+                if (event['event'] == name and
+                    self.event_match(event, match)):
+                    return True
+            return False
 
         # Search cached events
         for event in self._events:
-            if (event['event'] == name) and event_match(event, match):
+            if _match(event):
                 self._events.remove(event)
                 return event
 
         # Poll for new events
         while True:
             event = self._qmp.pull_event(wait=timeout)
-            if (event['event'] == name) and event_match(event, match):
+            if _match(event):
                 return event
             self._events.append(event)
 
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Don't pull events out of the queue that don't belong to us;
be choosier so that we can use this method to drive jobs that
were launched by transactions that may have more jobs.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-5-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 48 +++++++++++++++++++++--------------
 1 file changed, 29 insertions(+), 19 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):
     # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
                 pre_finalize=None, wait=60.0):
+        match_device = {'data': {'device': job}}
+        match_id = {'data': {'id': job}}
+        events = [
+            ('BLOCK_JOB_COMPLETED', match_device),
+            ('BLOCK_JOB_CANCELLED', match_device),
+            ('BLOCK_JOB_ERROR', match_device),
+            ('BLOCK_JOB_READY', match_device),
+            ('BLOCK_JOB_PENDING', match_id),
+            ('JOB_STATUS_CHANGE', match_id)
+        ]
         error = None
         while True:
-            for ev in self.get_qmp_events_filtered(wait=wait):
-                if ev['event'] == 'JOB_STATUS_CHANGE':
-                    status = ev['data']['status']
-                    if status == 'aborting':
-                        result = self.qmp('query-jobs')
-                        for j in result['return']:
-                            if j['id'] == job:
-                                error = j['error']
-                                log('Job failed: %s' % (j['error']))
-                    elif status == 'pending' and not auto_finalize:
-                        if pre_finalize:
-                            pre_finalize()
-                        self.qmp_log('job-finalize', id=job)
-                    elif status == 'concluded' and not auto_dismiss:
-                        self.qmp_log('job-dismiss', id=job)
-                    elif status == 'null':
-                        return error
-                else:
-                    log(ev)
+            ev = filter_qmp_event(self.events_wait(events))
+            if ev['event'] != 'JOB_STATUS_CHANGE':
+                log(ev)
+                continue
+            status = ev['data']['status']
+            if status == 'aborting':
+                result = self.qmp('query-jobs')
+                for j in result['return']:
+                    if j['id'] == job:
+                        error = j['error']
+                        log('Job failed: %s' % (j['error']))
+            elif status == 'pending' and not auto_finalize:
+                if pre_finalize:
+                    pre_finalize()
+                self.qmp_log('job-finalize', id=job)
+            elif status == 'concluded' and not auto_dismiss:
+                self.qmp_log('job-dismiss', id=job)
+            elif status == 'null':
+                return error
 
     def node_info(self, node_name):
         nodes = self.qmp('query-named-block-nodes')
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190523170643.20794-6-jsnow@redhat.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
[mreitz: Moved from 250 to 256]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/256     | 122 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/256.out | 119 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 242 insertions(+)
 create mode 100755 tests/qemu-iotests/256
 create mode 100644 tests/qemu-iotests/256.out

diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
new file mode 100755
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/256
@@ -XXX,XX +XXX,XX @@
+#!/usr/bin/env python
+#
+# Test incremental/backup across iothread contexts
+#
+# Copyright (c) 2019 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# owner=jsnow@redhat.com
+
+import os
+import iotests
+from iotests import log
+
+iotests.verify_image_format(supported_fmts=['qcow2'])
+size = 64 * 1024 * 1024
+
+with iotests.FilePath('img0') as img0_path, \
+     iotests.FilePath('img1') as img1_path, \
+     iotests.FilePath('img0-full') as img0_full_path, \
+     iotests.FilePath('img1-full') as img1_full_path, \
+     iotests.FilePath('img0-incr') as img0_incr_path, \
+     iotests.FilePath('img1-incr') as img1_incr_path, \
+     iotests.VM() as vm:
+
+    def create_target(filepath, name, size):
+        basename = os.path.basename(filepath)
+        nodename = "file_{}".format(basename)
+        log(vm.command('blockdev-create', job_id='job1',
+                       options={
+                           'driver': 'file',
+                           'filename': filepath,
+                           'size': 0,
+                       }))
+        vm.run_job('job1')
+        log(vm.command('blockdev-add', driver='file',
+                       node_name=nodename, filename=filepath))
+        log(vm.command('blockdev-create', job_id='job2',
+                       options={
+                           'driver': iotests.imgfmt,
+                           'file': nodename,
+                           'size': size,
+                       }))
+        vm.run_job('job2')
+        log(vm.command('blockdev-add', driver=iotests.imgfmt,
+                       node_name=name,
+                       file=nodename))
+
+    log('--- Preparing images & VM ---\n')
+    vm.add_object('iothread,id=iothread0')
+    vm.add_object('iothread,id=iothread1')
+    vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
+    vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
+    iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
+    iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
+    vm.add_drive(img0_path, interface='none')
+    vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
+    vm.add_drive(img1_path, interface='none')
+    vm.add_device('scsi-hd,id=device1,drive=drive1,bus=scsi1.0')
+
+    log('--- Starting VM ---\n')
+    vm.launch()
+
+    log('--- Create Targets & Full Backups ---\n')
+    create_target(img0_full_path, 'img0-full', size)
+    create_target(img1_full_path, 'img1-full', size)
+    ret = vm.qmp_log('transaction', indent=2, actions=[
+        { 'type': 'block-dirty-bitmap-add',
+          'data': { 'node': 'drive0', 'name': 'bitmap0' }},
+        { 'type': 'block-dirty-bitmap-add',
+          'data': { 'node': 'drive1', 'name': 'bitmap1' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive0',
+                    'target': 'img0-full',
+                    'sync': 'full',
+                    'job-id': 'j0' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive1',
+                    'target': 'img1-full',
+                    'sync': 'full',
+                    'job-id': 'j1' }}
+    ])
+    if "error" in ret:
+        raise Exception(ret['error']['desc'])
+    vm.run_job('j0', auto_dismiss=True)
+    vm.run_job('j1', auto_dismiss=True)
+
+    log('\n--- Create Targets & Incremental Backups ---\n')
+    create_target(img0_incr_path, 'img0-incr', size)
+    create_target(img1_incr_path, 'img1-incr', size)
+    ret = vm.qmp_log('transaction', indent=2, actions=[
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive0',
+                    'target': 'img0-incr',
+                    'sync': 'incremental',
+                    'bitmap': 'bitmap0',
+                    'job-id': 'j2' }},
+        { 'type': 'blockdev-backup',
+          'data': { 'device': 'drive1',
+                    'target': 'img1-incr',
+                    'sync': 'incremental',
+                    'bitmap': 'bitmap1',
+                    'job-id': 'j3' }}
+    ])
+    if "error" in ret:
+        raise Exception(ret['error']['desc'])
+    vm.run_job('j2', auto_dismiss=True)
+    vm.run_job('j3', auto_dismiss=True)
+
+    log('\n--- Done ---')
+    vm.shutdown()
diff --git a/tests/qemu-iotests/256.out b/tests/qemu-iotests/256.out
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/256.out
@@ -XXX,XX +XXX,XX @@
+--- Preparing images & VM ---
+
+--- Starting VM ---
+
+--- Create Targets & Full Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+  "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {
+        "data": {
+          "name": "bitmap0",
+          "node": "drive0"
+        },
+        "type": "block-dirty-bitmap-add"
+      },
+      {
+        "data": {
+          "name": "bitmap1",
+          "node": "drive1"
+        },
+        "type": "block-dirty-bitmap-add"
+      },
+      {
+        "data": {
+          "device": "drive0",
+          "job-id": "j0",
+          "sync": "full",
+          "target": "img0-full"
+        },
+        "type": "blockdev-backup"
+      },
+      {
+        "data": {
+          "device": "drive1",
+          "job-id": "j1",
+          "sync": "full",
+          "target": "img1-full"
+        },
+        "type": "blockdev-backup"
+      }
+    ]
+  }
+}
+{
+  "return": {}
+}
+{"data": {"device": "j0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Create Targets & Incremental Backups ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job1"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "job2"}}
+{"return": {}}
+{}
+{
+  "execute": "transaction",
+  "arguments": {
+    "actions": [
+      {
+        "data": {
+          "bitmap": "bitmap0",
+          "device": "drive0",
+          "job-id": "j2",
+          "sync": "incremental",
+          "target": "img0-incr"
+        },
+        "type": "blockdev-backup"
+      },
+      {
+        "data": {
+          "bitmap": "bitmap1",
+          "device": "drive1",
+          "job-id": "j3",
+          "sync": "incremental",
+          "target": "img1-incr"
+        },
+        "type": "blockdev-backup"
+      }
+    ]
+  }
+}
+{
+  "return": {}
+}
+{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Done ---
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -XXX,XX +XXX,XX @@
 253 rw auto quick
 254 rw auto backing quick
 255 rw auto quick
+256 rw auto quick
-- 
2.21.0

From: John Snow <jsnow@redhat.com>

Before, event_match didn't always recurse if the event value was not a
dictionary, and would instead check for equality immediately.

By delaying equality checking to post-recursion, we can allow leaf
values like "5" to match "None" and take advantage of the generic
None-returns-True clause.

This makes the matching a little more obviously consistent at the
expense of being able to check for explicit None values, which is
probably not that important given what this function is used for.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20190528183857.26167-1-jsnow@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 python/qemu/__init__.py | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/python/qemu/__init__.py b/python/qemu/__init__.py
index XXXXXXX..XXXXXXX 100644
--- a/python/qemu/__init__.py
+++ b/python/qemu/__init__.py
@@ -XXX,XX +XXX,XX @@ class QEMUMachine(object):
 
         The match criteria takes the form of a matching subdict. The event is
         checked to be a superset of the subdict, recursively, with matching
-        values whenever those values are not None.
+        values whenever the subdict values are not None.
+
+        This has a limitation that you cannot explicitly check for None values.
 
         Examples, with the subdict queries on the left:
          - None matches any object.
          - {"foo": None} matches {"foo": {"bar": 1}}
-         - {"foo": {"baz": None}} does not match {"foo": {"bar": 1}}
-         - {"foo": {"baz": 2}} matches {"foo": {"bar": 1, "baz": 2}}
+         - {"foo": None} matches {"foo": 5}
+         - {"foo": {"abc": None}} does not match {"foo": {"bar": 1}}
+         - {"foo": {"rab": 2}} matches {"foo": {"bar": 1, "rab": 2}}
         """
         if match is None:
             return True
 
-        for key in match:
-            if key in event:
-                if isinstance(event[key], dict):
+        try:
+            for key in match:
+                if key in event:
                     if not QEMUMachine.event_match(event[key], match[key]):
                         return False
-                elif event[key] != match[key]:
+                else:
                     return False
-            else:
-                return False
-        return True
+            return True
+        except TypeError:
+            # either match or event wasn't iterable (not a dict)
+            return match == event
 
     def event_wait(self, name, timeout=60.0, match=None):
         """
-- 
2.21.0

It is possible for an empty file to take up blocks on a filesystem, for
example:

$ qemu-img create -f raw test.img 1G
Formatting 'test.img', fmt=raw size=1073741824
$ mkfs.ext4 -I 128 -q test.img
$ mkdir test-mount
$ sudo mount -o loop test.img test-mount
$ sudo touch test-mount/test-file
$ stat -c 'blocks=%b' test-mount/test-file
blocks=8

These extra blocks (one cluster) are apparently used for metadata,
because they are always there, on top of blocks used for data:

$ sudo dd if=/dev/zero of=test-mount/test-file bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00135339 s, 775 MB/s
$ stat -c 'blocks=%b' test-mount/test-file
blocks=2056

Make iotest 175 take this into account.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Nir Soffer <nsoffer@redhat.com>
Message-id: 20190516144319.12570-1-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/175     | 26 ++++++++++++++++++++++----
 tests/qemu-iotests/175.out |  8 ++++----
 2 files changed, 26 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -XXX,XX +XXX,XX @@ status=1	# failure is the default!
 
 _cleanup()
 {
-	_cleanup_test_img
+    _cleanup_test_img
+    rm -f "$TEST_DIR/empty"
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
+# Some file systems sometimes allocate extra blocks independently of
+# the file size.  This function hides the resulting difference in the
+# stat -c '%b' output.
+# Parameter 1: Number of blocks an empty file occupies
+# Parameter 2: Image size in bytes
+_filter_blocks()
+{
+    extra_blocks=$1
+    img_size=$2
+
+    sed -e "s/blocks=$extra_blocks\$\$\\|[^0-9]\$/nothing allocated/" \
+        -e "s/blocks=$((extra_blocks + img_size / 512))\$\$\\|[^0-9]\$/everything allocated/"
+}
+
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
@@ -XXX,XX +XXX,XX @@ _supported_fmt raw
 _supported_proto file
 _supported_os Linux
 
-size=1m
+size=$((1 * 1024 * 1024))
+
+touch "$TEST_DIR/empty"
+extra_blocks=$(stat -c '%b' "$TEST_DIR/empty")
 
 echo
 echo "== creating image with default preallocation =="
 _make_test_img $size | _filter_imgfmt
-stat -c "size=%s, blocks=%b" $TEST_IMG
+stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 
 for mode in off full falloc; do
     echo
     echo "== creating image with preallocation $mode =="
     IMGOPTS=preallocation=$mode _make_test_img $size | _filter_imgfmt
-    stat -c "size=%s, blocks=%b" $TEST_IMG
+    stat -c "size=%s, blocks=%b" $TEST_IMG | _filter_blocks $extra_blocks $size
 done
 
 # success, all done
diff --git a/tests/qemu-iotests/175.out b/tests/qemu-iotests/175.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/175.out
+++ b/tests/qemu-iotests/175.out
@@ -XXX,XX +XXX,XX @@ QA output created by 175
 
 == creating image with default preallocation ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576
-size=1048576, blocks=0
+size=1048576, nothing allocated
 
 == creating image with preallocation off ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=off
-size=1048576, blocks=0
+size=1048576, nothing allocated
 
 == creating image with preallocation full ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=full
-size=1048576, blocks=2048
+size=1048576, everything allocated
 
 == creating image with preallocation falloc ==
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 preallocation=falloc
-size=1048576, blocks=2048
+size=1048576, everything allocated
  *** done
-- 
2.21.0

In 219, we wait for the job to make progress before we emit its status.
This makes the output reliable.  We do not wait for any more progress if
the job's current-progress already matches its total-progress.

Unfortunately, there is a bug: Right after the job has been started,
it's possible that total-progress is still 0.  In that case, we may skip
the first progress-making step and keep ending up 64 kB short.

To fix that bug, we can simply wait for total-progress to reach 4 MB
(the image size) after starting the job.

Reported-by: Karen Mezick <kmezick@redhat.com>
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1686651
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190516161114.27596-1-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
[mreitz: Adjusted commit message as per John's proposal]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/219 | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/219
+++ b/tests/qemu-iotests/219
@@ -XXX,XX +XXX,XX @@ import iotests
 
 iotests.verify_image_format(supported_fmts=['qcow2'])
 
+img_size = 4 * 1024 * 1024
+
 def pause_wait(vm, job_id):
     with iotests.Timeout(3, "Timeout waiting for job to pause"):
         while True:
@@ -XXX,XX +XXX,XX @@ def test_pause_resume(vm):
                 iotests.log(vm.qmp('query-jobs'))
 
 def test_job_lifecycle(vm, job, job_args, has_ready=False):
+    global img_size
+
     iotests.log('')
     iotests.log('')
     iotests.log('Starting block job: %s (auto-finalize: %s; auto-dismiss: %s)' %
@@ -XXX,XX +XXX,XX @@ def test_job_lifecycle(vm, job, job_args, has_ready=False):
     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
     iotests.log(iotests.filter_qmp_event(vm.event_wait('JOB_STATUS_CHANGE')))
 
+    # Wait for total-progress to stabilize
+    while vm.qmp('query-jobs')['return'][0]['total-progress'] < img_size:
+        pass
+
     # RUNNING state:
     # pause/resume should work, complete/finalize/dismiss should error out
     iotests.log('')
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('disk.img') as disk_path, \
      iotests.FilePath('copy.img') as copy_path, \
      iotests.VM() as vm:
 
-    img_size = '4M'
-    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, img_size)
-    iotests.qemu_io('-c', 'write 0 %s' % (img_size),
+    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, str(img_size))
+    iotests.qemu_io('-c', 'write 0 %i' % (img_size),
                     '-f', iotests.imgfmt, disk_path)
 
     iotests.log('Launching VM...')
-- 
2.21.0

From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>

The uninitialized memory allocated for the command FIFO of the
floppy controller during the VM hardware initialization incurs
many unwanted reports by Valgrind when VM state is being saved.
That verbosity hardens a search for the real memory issues when
the iotests run. Particularly, the patch eliminates 20 unnecessary
reports of the Valgrind tool in the iotest #169.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-id: 1559154027-282547-1-git-send-email-andrey.shinkevich@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 hw/block/fdc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/block/fdc.c b/hw/block/fdc.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/block/fdc.c
+++ b/hw/block/fdc.c
@@ -XXX,XX +XXX,XX @@ static void fdctrl_realize_common(DeviceState *dev, FDCtrl *fdctrl,
 
     FLOPPY_DPRINTF("init controller\n");
     fdctrl->fifo = qemu_memalign(512, FD_SECTOR_LEN);
+    memset(fdctrl->fifo, 0, FD_SECTOR_LEN);
     fdctrl->fifo_size = 512;
     fdctrl->result_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
                                              fdctrl_result_timer, fdctrl);
-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Test fails at least for qcow, because of different cluster sizes in
base and top (and therefore different granularities of bitmaps we are
trying to merge).

The test aim is to check block-dirty-bitmap-merge between different
nodes functionality, no needs to check all formats. So, let's just drop
support for anything except qcow2.

Reported-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190605155405.104384-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/254 | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/254
+++ b/tests/qemu-iotests/254
@@ -XXX,XX +XXX,XX @@
 import iotests
 from iotests import qemu_img_create, file_path, log
 
+iotests.verify_image_format(supported_fmts=['qcow2'])
+
 disk, top = file_path('disk', 'top')
 size = 1024 * 1024
 
-- 
2.21.0

img_rebase() can leak a QDict in two occasions.  Fix it.

Coverity: CID 1401416
Fixes: d16699b64671466b42079c45b89127aeea1ca565
Fixes: 330c72957196e0ae382abcaa97ebf4eb9bc8574f
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190528195338.12376-1-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
                                                              out_baseimg,
                                                              &local_err);
             if (local_err) {
+                qobject_unref(options);
                 error_reportf_err(local_err,
                                   "Could not resolve backing filename: ");
                 ret = -1;
@@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv)
              */
             prefix_chain_bs = bdrv_find_backing_image(bs, out_real_path);
             if (prefix_chain_bs) {
+                qobject_unref(options);
                 g_free(out_real_path);
+
                 blk_new_backing = blk_new(qemu_get_aio_context(),
                                           BLK_PERM_CONSISTENT_READ,
                                           BLK_PERM_ALL);
-- 
2.21.0

A snapshot is something that reflects the state of something at a
certain point in time.  It does not change.

The file our snapshot commands create (or the node they install) is not
a snapshot, as it does change over time.  It is an overlay.  We cannot
do anything about the parameter names, but we can at least adjust the
descriptions to reflect that fact.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190603202236.1342-2-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # Either @device or @node-name must be set but not both.
 #
-# @device: the name of the device to generate the snapshot from.
+# @device: the name of the device to take a snapshot of.
 #
 # @node-name: graph node name to generate the snapshot from (Since 2.0)
 #
-# @snapshot-file: the target of the new image. If the file exists, or
-# if it is a device, the snapshot will be created in the existing
-# file/device. Otherwise, a new file will be created.
+# @snapshot-file: the target of the new overlay image. If the file
+# exists, or if it is a device, the overlay will be created in the
+# existing file/device. Otherwise, a new file will be created.
 #
 # @snapshot-node-name: the graph node name of the new image (Since 2.0)
 #
-# @format: the format of the snapshot image, default is 'qcow2'.
+# @format: the format of the overlay image, default is 'qcow2'.
 #
 # @mode: whether and how QEMU should create a new image, default is
 #        'absolute-paths'.
@@ -XXX,XX +XXX,XX @@
 ##
 # @BlockdevSnapshot:
 #
-# @node: device or node name that will have a snapshot created.
+# @node: device or node name that will have a snapshot taken.
 #
 # @overlay: reference to the existing block device that will become
-#           the overlay of @node, as part of creating the snapshot.
+#           the overlay of @node, as part of taking the snapshot.
 #           It must not have a current backing file (this can be
 #           achieved by passing "backing": null to blockdev-add).
 #
@@ -XXX,XX +XXX,XX @@
 ##
 # @blockdev-snapshot-sync:
 #
-# Generates a synchronous snapshot of a block device.
+# Takes a synchronous snapshot of a block device.
 #
 # For the arguments, see the documentation of BlockdevSnapshotSync.
 #
@@ -XXX,XX +XXX,XX @@
 ##
 # @blockdev-snapshot:
 #
-# Generates a snapshot of a block device.
+# Takes a snapshot of a block device.
 #
-# Create a snapshot, by installing 'node' as the backing image of
+# Take a snapshot, by installing 'node' as the backing image of
 # 'overlay'. Additionally, if 'node' is associated with a block
 # device, the block device changes to using 'overlay' as its new active
 # image.
-- 
2.21.0

There are error messages which refer to an overlay node as the snapshot.
That is wrong, those are two different things.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 20190603202236.1342-3-mreitz@redhat.com
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c                 | 10 +++++-----
 tests/qemu-iotests/085.out | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
             s->has_snapshot_node_name ? s->snapshot_node_name : NULL;
 
         if (node_name && !snapshot_node_name) {
-            error_setg(errp, "New snapshot node name missing");
+            error_setg(errp, "New overlay node name missing");
             goto out;
         }
 
         if (snapshot_node_name &&
             bdrv_lookup_bs(snapshot_node_name, snapshot_node_name, NULL)) {
-            error_setg(errp, "New snapshot node name already in use");
+            error_setg(errp, "New overlay node name already in use");
             goto out;
         }
 
@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
     }
 
     if (bdrv_has_blk(state->new_bs)) {
-        error_setg(errp, "The snapshot is already in use");
+        error_setg(errp, "The overlay is already in use");
         goto out;
     }
 
@@ -XXX,XX +XXX,XX @@ static void external_snapshot_prepare(BlkActionState *common,
     }
 
     if (state->new_bs->backing != NULL) {
-        error_setg(errp, "The snapshot already has a backing image");
+        error_setg(errp, "The overlay already has a backing image");
         goto out;
     }
 
     if (!state->new_bs->drv->supports_backing) {
-        error_setg(errp, "The snapshot does not support backing images");
+        error_setg(errp, "The overlay does not support backing images");
         goto out;
     }
 
diff --git a/tests/qemu-iotests/085.out b/tests/qemu-iotests/085.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/085.out
+++ b/tests/qemu-iotests/085.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
 
 === Invalid command - cannot create a snapshot using a file BDS ===
 
-{"error": {"class": "GenericError", "desc": "The snapshot does not support backing images"}}
+{"error": {"class": "GenericError", "desc": "The overlay does not support backing images"}}
 
 === Invalid command - snapshot node used as active layer ===
 
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
-{"error": {"class": "GenericError", "desc": "The snapshot is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
+{"error": {"class": "GenericError", "desc": "The overlay is already in use"}}
 
 === Invalid command - snapshot node used as backing hd ===
 
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/
 Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base
 {"return": {}}
-{"error": {"class": "GenericError", "desc": "The snapshot already has a backing image"}}
+{"error": {"class": "GenericError", "desc": "The overlay already has a backing image"}}
 
 === Invalid command - The node does not exist ===
 
-- 
2.21.0

Move img_convert()'s quiet flag into the ImgConvertState so it is
accessible by nested functions.  -q dictates that it suppresses anything
but errors, so if those functions want to emit warnings, they need to
query this flag first.  (There currently are no such warnings, but there
will be as of the next patch.)

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-2-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool quiet;
     int min_sparse;
     int alignment;
     size_t cluster_sectors;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
     QDict *open_opts = NULL;
     char *options = NULL;
     Error *local_err = NULL;
-    bool writethrough, src_writethrough, quiet = false, image_opts = false,
+    bool writethrough, src_writethrough, image_opts = false,
          skip_create = false, progress = false, tgt_image_opts = false;
     int64_t ret = -EINVAL;
     bool force_share = false;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
             src_cache = optarg;
             break;
         case 'q':
-            quiet = true;
+            s.quiet = true;
             break;
         case 'n':
             skip_create = true;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
     }
 
     /* Initialize before goto out */
-    if (quiet) {
+    if (s.quiet) {
         progress = false;
     }
     qemu_progress_init(progress, 1.0);
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
 
     for (bs_i = 0; bs_i < s.src_num; bs_i++) {
         s.src[bs_i] = img_open(image_opts, argv[optind + bs_i],
-                               fmt, src_flags, src_writethrough, quiet,
+                               fmt, src_flags, src_writethrough, s.quiet,
                                force_share);
         if (!s.src[bs_i]) {
             ret = -1;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
 
     if (skip_create) {
         s.target = img_open(tgt_image_opts, out_filename, out_fmt,
-                            flags, writethrough, quiet, false);
+                            flags, writethrough, s.quiet, false);
     } else {
         /* TODO ultimately we should allow --target-image-opts
          * to be used even when -n is not given.
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
          * to allow filenames in option syntax
          */
         s.target = img_open_file(out_filename, open_opts, out_fmt,
-                                 flags, writethrough, quiet, false);
+                                 flags, writethrough, s.quiet, false);
         open_opts = NULL; /* blk_new_open will have freed it */
     }
     if (!s.target) {
-- 
2.21.0

This adds a salvaging mode (--salvage) to qemu-img convert which ignores
read errors and treats the respective areas as containing only zeroes.
This can be used for instance to at least partially recover the data
from terminally corrupted qcow2 images.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-3-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c       | 90 +++++++++++++++++++++++++++++++++++++-----------
 qemu-img-cmds.hx |  4 +--
 qemu-img.texi    |  4 +++
 3 files changed, 75 insertions(+), 23 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -XXX,XX +XXX,XX @@ enum {
     OPTION_SIZE = 264,
     OPTION_PREALLOCATION = 265,
     OPTION_SHRINK = 266,
+    OPTION_SALVAGE = 267,
 };
 
 typedef enum OutputFormat {
@@ -XXX,XX +XXX,XX @@ typedef struct ImgConvertState {
     int64_t target_backing_sectors; /* negative if unknown */
     bool wr_in_order;
     bool copy_range;
+    bool salvage;
     bool quiet;
     int min_sparse;
     int alignment;
@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
     }
 
     if (s->sector_next_status <= sector_num) {
-        int64_t count = n * BDRV_SECTOR_SIZE;
+        uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
+        int64_t count;
 
-        if (s->target_has_backing) {
+        do {
+            count = n * BDRV_SECTOR_SIZE;
+
+            if (s->target_has_backing) {
+                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
+                                        count, &count, NULL, NULL);
+            } else {
+                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
+                                              offset, count, &count, NULL,
+                                              NULL);
+            }
+
+            if (ret < 0) {
+                if (s->salvage) {
+                    if (n == 1) {
+                        if (!s->quiet) {
+                            warn_report("error while reading block status at "
+                                        "offset %" PRIu64 ": %s", offset,
+                                        strerror(-ret));
+                        }
+                        /* Just try to read the data, then */
+                        ret = BDRV_BLOCK_DATA;
+                        count = BDRV_SECTOR_SIZE;
+                    } else {
+                        /* Retry on a shorter range */
+                        n = DIV_ROUND_UP(n, 4);
+                    }
+                } else {
+                    error_report("error while reading block status at offset "
+                                 "%" PRIu64 ": %s", offset, strerror(-ret));
+                    return ret;
+                }
+            }
+        } while (ret < 0);
 
-            ret = bdrv_block_status(blk_bs(s->src[src_cur]),
-                                    (sector_num - src_cur_offset) *
-                                    BDRV_SECTOR_SIZE,
-                                    count, &count, NULL, NULL);
-        } else {
-            ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                          (sector_num - src_cur_offset) *
-                                          BDRV_SECTOR_SIZE,
-                                          count, &count, NULL, NULL);
-        }
-        if (ret < 0) {
-            error_report("error while reading block status of sector %" PRId64
-                         ": %s", sector_num, strerror(-ret));
-            return ret;
-        }
         n = DIV_ROUND_UP(count, BDRV_SECTOR_SIZE);
 
         if (ret & BDRV_BLOCK_ZERO) {
@@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
 static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
                                         int nb_sectors, uint8_t *buf)
 {
+    uint64_t single_read_until = 0;
     int n, ret;
 
     assert(nb_sectors <= s->buf_sectors);
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         BlockBackend *blk;
         int src_cur;
         int64_t bs_sectors, src_cur_offset;
+        uint64_t offset;
 
         /* In the case of compression with multiple source files, we can get a
          * nb_sectors that spreads into the next part. So we must be able to
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn convert_co_read(ImgConvertState *s, int64_t sector_num,
         blk = s->src[src_cur];
         bs_sectors = s->src_sectors[src_cur];
 
+        offset = (sector_num - src_cur_offset) << BDRV_SECTOR_BITS;
+
         n = MIN(nb_sectors, bs_sectors - (sector_num - src_cur_offset));
+        if (single_read_until > offset) {
+            n = 1;
+        }
 
-        ret = blk_co_pread(
-                blk, (sector_num - src_cur_offset) << BDRV_SECTOR_BITS,
-                n << BDRV_SECTOR_BITS, buf, 0);
+        ret = blk_co_pread(blk, offset, n << BDRV_SECTOR_BITS, buf, 0);
         if (ret < 0) {
-            return ret;
+            if (s->salvage) {
+                if (n > 1) {
+                    single_read_until = offset + (n << BDRV_SECTOR_BITS);
+                    continue;
+                } else {
+                    if (!s->quiet) {
+                        warn_report("error while reading offset %" PRIu64
+                                    ": %s", offset, strerror(-ret));
+                    }
+                    memset(buf, 0, BDRV_SECTOR_SIZE);
+                }
+            } else {
+                return ret;
+            }
         }
 
         sector_num += n;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
             {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
             {"force-share", no_argument, 0, 'U'},
             {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
+            {"salvage", no_argument, 0, OPTION_SALVAGE},
             {0, 0, 0, 0}
         };
         c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
         case OPTION_IMAGE_OPTS:
             image_opts = true;
             break;
+        case OPTION_SALVAGE:
+            s.salvage = true;
+            break;
         case OPTION_TARGET_IMAGE_OPTS:
             tgt_image_opts = true;
             break;
@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)
         goto fail_getopt;
     }
 
+    if (s.copy_range && s.salvage) {
+        error_report("Cannot use copy offloading in salvaging mode");
+        goto fail_getopt;
+    }
+
     if (tgt_image_opts && !skip_create) {
         error_report("--target-image-opts requires use of -n flag");
         goto fail_getopt;
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -XXX,XX +XXX,XX @@ STEXI
 ETEXI
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
 STEXI
-@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] @var{filename} [@var{filename2} [...]] @var{output_filename}
+@item convert [--object @var{objectdef}] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-B @var{backing_file}] [-o @var{options}] [-l @var{snapshot_param}] [-S @var{sparse_size}] [-m @var{num_coroutines}] [-W] [--salvage] @var{filename} [@var{filename2} [...]] @var{output_filename}
 ETEXI
 
 DEF("create", img_create,
diff --git a/qemu-img.texi b/qemu-img.texi
index XXXXXXX..XXXXXXX 100644
--- a/qemu-img.texi
+++ b/qemu-img.texi
@@ -XXX,XX +XXX,XX @@ improve performance if the data is remote, such as with NFS or iSCSI backends,
 but will not automatically sparsify zero sectors, and may result in a fully
 allocated target image depending on the host support for getting allocation
 information.
+@item --salvage
+Try to ignore I/O errors when reading.  Unless in quiet mode (@code{-q}), errors
+will still be printed.  Areas that cannot be read from the source will be
+treated as containing only zeroes.
 @end table
 
 Parameters to dd subcommand:
-- 
2.21.0

This new error option allows users of blkdebug to inject errors only on
certain kinds of I/O operations.  Users usually want to make a very
specific operation fail, not just any; but right now they simply hope
that the event that triggers the error injection is followed up with
that very operation.  That may not be true, however, because the block
layer is changing (including blkdebug, which may increase the number of
types of I/O operations on which to inject errors).

The new option's default has been chosen to keep backwards
compatibility.

Note that similar to the internal representation, we could choose to
expose this option as a list of I/O types.  But there is no practical
use for this, because as described above, users usually know exactly
which kind of operation they want to make fail, so there is no need to
specify multiple I/O types at once.  In addition, exposing this option
as a list would require non-trivial changes to qemu_opts_absorb_qdict().

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-4-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 26 +++++++++++++++++++++++
 block/blkdebug.c     | 50 ++++++++++++++++++++++++++++++++++++--------
 2 files changed, 67 insertions(+), 9 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
             'cor_write', 'cluster_alloc_space'] }
 
+##
+# @BlkdebugIOType:
+#
+# Kinds of I/O that blkdebug can inject errors in.
+#
+# @read: .bdrv_co_preadv()
+#
+# @write: .bdrv_co_pwritev()
+#
+# @write-zeroes: .bdrv_co_pwrite_zeroes()
+#
+# @discard: .bdrv_co_pdiscard()
+#
+# @flush: .bdrv_co_flush_to_disk()
+#
+# Since: 4.1
+##
+{ 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+
 ##
 # @BlkdebugInjectErrorOptions:
 #
@@ -XXX,XX +XXX,XX @@
 # @state:       the state identifier blkdebug needs to be in to
 #               actually trigger the event; defaults to "any"
 #
+# @iotype:      the type of I/O operations on which this error should
+#               be injected; defaults to "all read, write,
+#               write-zeroes, discard, and flush operations"
+#               (since: 4.1)
+#
 # @errno:       error identifier (errno) to be returned; defaults to
 #               EIO
 #
@@ -XXX,XX +XXX,XX @@
 { 'struct': 'BlkdebugInjectErrorOptions',
   'data': { 'event': 'BlkdebugEvent',
             '*state': 'int',
+            '*iotype': 'BlkdebugIOType',
             '*errno': 'int',
             '*sector': 'int',
             '*once': 'bool',
diff --git a/block/blkdebug.c b/block/blkdebug.c
index XXXXXXX..XXXXXXX 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
     int state;
     union {
         struct {
+            uint64_t iotype_mask;
             int error;
             int immediately;
             int once;
@@ -XXX,XX +XXX,XX @@ typedef struct BlkdebugRule {
     QSIMPLEQ_ENTRY(BlkdebugRule) active_next;
 } BlkdebugRule;
 
+QEMU_BUILD_BUG_MSG(BLKDEBUG_IO_TYPE__MAX > 64,
+                   "BlkdebugIOType mask does not fit into an uint64_t");
+
 static QemuOptsList inject_error_opts = {
     .name = "inject-error",
     .head = QTAILQ_HEAD_INITIALIZER(inject_error_opts.head),
@@ -XXX,XX +XXX,XX @@ static QemuOptsList inject_error_opts = {
             .name = "state",
             .type = QEMU_OPT_NUMBER,
         },
+        {
+            .name = "iotype",
+            .type = QEMU_OPT_STRING,
+        },
         {
             .name = "errno",
             .type = QEMU_OPT_NUMBER,
@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
     int event;
     struct BlkdebugRule *rule;
     int64_t sector;
+    BlkdebugIOType iotype;
+    Error *local_error = NULL;
 
     /* Find the right event for the rule */
     event_name = qemu_opt_get(opts, "event");
@@ -XXX,XX +XXX,XX @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
         sector = qemu_opt_get_number(opts, "sector", -1);
         rule->options.inject.offset =
             sector == -1 ? -1 : sector * BDRV_SECTOR_SIZE;
+
+        iotype = qapi_enum_parse(&BlkdebugIOType_lookup,
+                                 qemu_opt_get(opts, "iotype"),
+                                 BLKDEBUG_IO_TYPE__MAX, &local_error);
+        if (local_error) {
+            error_propagate(errp, local_error);
+            return -1;
+        }
+        if (iotype != BLKDEBUG_IO_TYPE__MAX) {
+            rule->options.inject.iotype_mask = (1ull << iotype);
+        } else {
+            /* Apply the default */
+            rule->options.inject.iotype_mask =
+                (1ull << BLKDEBUG_IO_TYPE_READ)
+                | (1ull << BLKDEBUG_IO_TYPE_WRITE)
+                | (1ull << BLKDEBUG_IO_TYPE_WRITE_ZEROES)
+                | (1ull << BLKDEBUG_IO_TYPE_DISCARD)
+                | (1ull << BLKDEBUG_IO_TYPE_FLUSH);
+        }
+
         break;
 
     case ACTION_SET_STATE:
@@ -XXX,XX +XXX,XX @@ out:
     return ret;
 }
 
-static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
+static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
+                      BlkdebugIOType iotype)
 {
     BDRVBlkdebugState *s = bs->opaque;
     BlkdebugRule *rule = NULL;
@@ -XXX,XX +XXX,XX @@ static int rule_check(BlockDriverState *bs, uint64_t offset, uint64_t bytes)
     QSIMPLEQ_FOREACH(rule, &s->active_rules, active_next) {
         uint64_t inject_offset = rule->options.inject.offset;
 
-        if (inject_offset == -1 ||
-            (bytes && inject_offset >= offset &&
-             inject_offset < offset + bytes))
+        if ((inject_offset == -1 ||
+             (bytes && inject_offset >= offset &&
+              inject_offset < offset + bytes)) &&
+            (rule->options.inject.iotype_mask & (1ull << iotype)))
         {
             break;
         }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
         assert(bytes <= bs->bl.max_transfer);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_READ);
     if (err) {
         return err;
     }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
         assert(bytes <= bs->bl.max_transfer);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE);
     if (err) {
         return err;
     }
@@ -XXX,XX +XXX,XX @@ blkdebug_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
 
 static int blkdebug_co_flush(BlockDriverState *bs)
 {
-    int err = rule_check(bs, 0, 0);
+    int err = rule_check(bs, 0, 0, BLKDEBUG_IO_TYPE_FLUSH);
 
     if (err) {
         return err;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pwrite_zeroes(BlockDriverState *bs,
         assert(bytes <= bs->bl.max_pwrite_zeroes);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_WRITE_ZEROES);
     if (err) {
         return err;
     }
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pdiscard(BlockDriverState *bs,
         assert(bytes <= bs->bl.max_pdiscard);
     }
 
-    err = rule_check(bs, offset, bytes);
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_DISCARD);
     if (err) {
         return err;
     }
-- 
2.21.0

Together with @iotypes and @sector, this can be used to trap e.g. the
first read or write access to a certain sector without having to know
what happens internally in the block layer, i.e. which "real" events
happen right before such an access.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-5-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 4 +++-
 block/blkdebug.c     | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # @cluster_alloc_space: an allocation of file space for a cluster (since 4.1)
 #
+# @none: triggers once at creation of the blkdebug node (since 4.1)
+#
 # Since: 2.9
 ##
 { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
@@ -XXX,XX +XXX,XX @@
             'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev',
             'pwritev_zero', 'pwritev_done', 'empty_image_prepare',
             'l1_shrink_write_table', 'l1_shrink_free_l2_clusters',
-            'cor_write', 'cluster_alloc_space'] }
+            'cor_write', 'cluster_alloc_space', 'none'] }
 
 ##
 # @BlkdebugIOType:
diff --git a/block/blkdebug.c b/block/blkdebug.c
index XXXXXXX..XXXXXXX 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ static int blkdebug_open(BlockDriverState *bs, QDict *options, int flags,
         goto out;
     }
 
+    bdrv_debug_event(bs, BLKDBG_NONE);
+
     ret = 0;
 out:
     if (ret < 0) {
-- 
2.21.0

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-6-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qapi/block-core.json | 5 ++++-
 block/blkdebug.c     | 8 ++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # @flush: .bdrv_co_flush_to_disk()
 #
+# @block-status: .bdrv_co_block_status()
+#
 # Since: 4.1
 ##
 { 'enum': 'BlkdebugIOType', 'prefix': 'BLKDEBUG_IO_TYPE',
-  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush' ] }
+  'data': [ 'read', 'write', 'write-zeroes', 'discard', 'flush',
+            'block-status' ] }
 
 ##
 # @BlkdebugInjectErrorOptions:
diff --git a/block/blkdebug.c b/block/blkdebug.c
index XXXXXXX..XXXXXXX 100644
--- a/block/blkdebug.c
+++ b/block/blkdebug.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,
                                                  int64_t *map,
                                                  BlockDriverState **file)
 {
+    int err;
+
     assert(QEMU_IS_ALIGNED(offset | bytes, bs->bl.request_alignment));
+
+    err = rule_check(bs, offset, bytes, BLKDEBUG_IO_TYPE_BLOCK_STATUS);
+    if (err) {
+        return err;
+    }
+
     return bdrv_co_block_status_from_file(bs, want_zero, offset, bytes,
                                           pnum, map, file);
 }
-- 
2.21.0

This test converts a simple image to another, but blkdebug injects
block_status and read faults at some offsets.  The resulting image
should be the same as the input image, except that sectors that could
not be read have to be 0.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190507203508.18026-7-mreitz@redhat.com
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[mreitz: Dropped superfluous printf from _filter_offsets, as suggested
         by Vladimir; disable test for VDI and IMGOPTSSYNTAX]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/251     | 170 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/251.out |  43 ++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 214 insertions(+)
 create mode 100755 tests/qemu-iotests/251
 create mode 100644 tests/qemu-iotests/251.out

diff --git a/tests/qemu-iotests/251 b/tests/qemu-iotests/251
new file mode 100755
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/251
@@ -XXX,XX +XXX,XX @@
+#!/usr/bin/env bash
+#
+# Test qemu-img convert --salvage
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mreitz@redhat.com
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt generic
+_supported_proto file
+_supported_os Linux
+
+if [ "$IMGOPTSSYNTAX" = "true" ]; then
+    # We use json:{} filenames here, so we cannot work with additional options.
+    _unsupported_fmt $IMGFMT
+else
+    # With VDI, the output is ordered differently.  Just disable it.
+    _unsupported_fmt vdi
+fi
+
+
+TEST_IMG="$TEST_IMG.orig" _make_test_img 64M
+
+$QEMU_IO -c 'write -P 42 0 64M' "$TEST_IMG.orig" | _filter_qemu_io
+
+
+sector_size=512
+
+# Offsets on which to fail block-status.  Keep in ascending order so
+# the indexing done by _filter_offsets will appear in ascending order
+# in the output as well.
+status_fail_offsets="$((16 * 1024 * 1024 + 8192))
+                     $((33 * 1024 * 1024 + 512))"
+
+# Offsets on which to fail reads.  Keep in ascending order for the
+# same reason.
+# The second element is shared with $status_fail_offsets on purpose.
+# Starting with the third element, we test what happens when a
+# continuous range of sectors is inaccessible.
+read_fail_offsets="$((32 * 1024 * 1024 - 65536))
+                   $((33 * 1024 * 1024 + 512))
+                   $(seq $((34 * 1024 * 1024)) $sector_size \
+                         $((34 * 1024 * 1024 + 4096 - $sector_size)))"
+
+
+# blkdebug must be above the format layer so it can intercept all
+# block-status events
+source_img="json:{'driver': 'blkdebug',
+                  'image': {
+                      'driver': '$IMGFMT',
+                      'file': {
+                          'driver': 'file',
+                          'filename': '$TEST_IMG.orig'
+                      }
+                  },
+                  'inject-error': ["
+
+for ofs in $status_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'block-status',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+for ofs in $read_fail_offsets
+do
+    source_img+="{ 'event': 'none',
+                   'iotype': 'read',
+                   'errno': 5,
+                   'sector': $((ofs / sector_size)) },"
+done
+
+# Remove the trailing comma and terminate @inject-error and json:{}
+source_img="${source_img%,} ] }"
+
+
+echo
+
+
+_filter_offsets() {
+    filters=
+
+    index=0
+    for ofs in $1
+    do
+        filters+=" -e s/$ofs/status_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    index=0
+    for ofs in $2
+    do
+        filters+=" -e s/$ofs/read_fail_offset_$index/"
+        index=$((index + 1))
+    done
+
+    sed $filters
+}
+
+# While determining the number of allocated sectors in the input
+# image, we should see one block status warning per element of
+# $status_fail_offsets.
+#
+# Then, the image is read.  Since the block status is queried in
+# basically the same way, the same warnings as in the previous step
+# should reappear.  Interleaved with those we should see a read
+# warning per element of $read_fail_offsets.
+# Note that $read_fail_offsets and $status_fail_offsets share an
+# element (read_fail_offset_1 == status_fail_offset_1), so
+# "status_fail_offset_1" in the output is the same as
+# "read_fail_offset_1".
+$QEMU_IMG convert --salvage "$source_img" "$TEST_IMG" 2>&1 \
+    | _filter_offsets "$status_fail_offsets" "$read_fail_offsets"
+
+echo
+
+# The offsets where the block status could not be determined should
+# have been treated as containing data and thus should be correct in
+# the output image.
+# The offsets where reading failed altogether should be 0.  Make them
+# 0 in the input image, too, so we can compare both images.
+for ofs in $read_fail_offsets
+do
+    $QEMU_IO -c "write -z $ofs $sector_size" "$TEST_IMG.orig" \
+        | _filter_qemu_io \
+        | _filter_offsets '' "$read_fail_offsets"
+done
+
+echo
+
+# These should be equal now.
+$QEMU_IMG compare "$TEST_IMG.orig" "$TEST_IMG"
+
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/251.out b/tests/qemu-iotests/251.out
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qemu-iotests/251.out
@@ -XXX,XX +XXX,XX @@
+QA output created by 251
+Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
+wrote 67108864/67108864 bytes at offset 0
+64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_0: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_0: Input/output error
+qemu-img: warning: error while reading block status at offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset status_fail_offset_1: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_2: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_3: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_4: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_5: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_6: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_7: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_8: Input/output error
+qemu-img: warning: error while reading offset read_fail_offset_9: Input/output error
+
+wrote 512/512 bytes at offset read_fail_offset_0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_1
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_2
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_3
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_4
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_5
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_6
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_7
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_8
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 512/512 bytes at offset read_fail_offset_9
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+Images are identical.
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -XXX,XX +XXX,XX @@
 248 rw quick
 249 rw auto quick
 250 rw auto quick
+251 rw auto quick
 252 rw auto backing quick
 253 rw auto quick
 254 rw auto backing quick
-- 
2.21.0

We do not support this combination (yet), so this should yield an error
message.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190507203508.18026-8-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/082     | 1 +
 tests/qemu-iotests/082.out | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/082
+++ b/tests/qemu-iotests/082
@@ -XXX,XX +XXX,XX @@ echo === convert: -C and other options ===
 run_qemu_img convert -C -S 4k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -S 8k -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 run_qemu_img convert -C -c -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
+run_qemu_img convert -C --salvage -O $IMGFMT "$TEST_IMG" "$TEST_IMG".target
 
 echo
 echo === amend: Options specified more than once ===
diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/082.out
+++ b/tests/qemu-iotests/082.out
@@ -XXX,XX +XXX,XX @@ qemu-img: Cannot enable copy offloading when -S is used
 Testing: convert -C -c -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
 qemu-img: Cannot enable copy offloading when -c is used
 
+Testing: convert -C --salvage -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
+qemu-img: Cannot use copy offloading in salvaging mode
+
 === amend: Options specified more than once ===
 
 Testing: amend -f foo -f qcow2 -o lazy_refcounts=on TEST_DIR/t.qcow2
-- 
2.21.0

The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:

Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24

for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:

iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)

----------------------------------------------------------------
Block patches:
- The SSH block driver now uses libssh instead of libssh2
- The VMDK block driver gets read-only support for the seSparse
  subformat
- Various fixes

---

v2:
- Squashed Pino's fix for pre-0.8 libssh into the libssh patch

----------------------------------------------------------------
Anton Nefedov (1):
  iotest 134: test cluster-misaligned encrypted write

Klaus Birkelund Jensen (1):
  nvme: do not advertise support for unsupported arbitration mechanism

Max Reitz (1):
  iotests: Fix 205 for concurrent runs

Pino Toscano (1):
  ssh: switch from libssh2 to libssh

Sam Eiderman (3):
  vmdk: Fix comment regarding max l1_size coverage
  vmdk: Reduce the max bound for L1 table size
  vmdk: Add read-only support for seSparse snapshots

Vladimir Sementsov-Ogievskiy (1):
  blockdev: enable non-root nodes for transaction drive-backup source

-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

We forget to enable it for transaction .prepare, while it is already
enabled in do_drive_backup since commit a2d665c1bc362
    "blockdev: loosen restrictions on drive-backup source node"

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
     assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);
     backup = common->action->u.drive_backup.data;
 
-    bs = qmp_get_root_bs(backup->device, errp);
+    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
     if (!bs) {
         return;
     }
-- 
2.21.0

From: Anton Nefedov <anton.nefedov@virtuozzo.com>

COW (even empty/zero) areas require encryption too

Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190516143028.81155-1-anton.nefedov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/134     |  9 +++++++++
 tests/qemu-iotests/134.out | 10 ++++++++++
 2 files changed, 19 insertions(+)

diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/134
+++ b/tests/qemu-iotests/134
@@ -XXX,XX +XXX,XX @@ echo
 echo "== reading whole image =="
 $QEMU_IO --object $SECRET -c "read 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 
+echo
+echo "== rewriting cluster part =="
+$QEMU_IO --object $SECRET -c "write -P 0xb 512 512" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+
+echo
+echo "== verify pattern =="
+$QEMU_IO --object $SECRET -c "read -P 0 0 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+$QEMU_IO --object $SECRET -c "read -P 0xb 512 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+
 echo
 echo "== rewriting whole image =="
 $QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/134.out
+++ b/tests/qemu-iotests/134.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.
 read 134217728/134217728 bytes at offset 0
 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
+== rewriting cluster part ==
+wrote 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify pattern ==
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
 == rewriting whole image ==
 wrote 134217728/134217728 bytes at offset 0
 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")
extended the l1_size check from VMDK4 to VMDK3 but did not update the
default coverage in the moved comment.

The previous vmdk4 calculation:

(512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB

The added vmdk3 calculation:

(512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB

Adding the calculation of vmdk3 to the comment.

In any case, VMware does not offer virtual disks more than 2TB for
vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is
not implemented yet in qemu.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Liran Alon <liran.alon@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com
Reviewed-by: yuchenlin <yuchenlin@synology.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
         return -EFBIG;
     }
     if (l1_size > 512 * 1024 * 1024) {
-        /* Although with big capacity and small l1_entry_sectors, we can get a
+        /*
+         * Although with big capacity and small l1_entry_sectors, we can get a
          * big l1_size, we don't want unbounded value to allocate the table.
-         * Limit it to 512M, which is 16PB for default cluster and L2 table
-         * size */
+         * Limit it to 512M, which is:
+         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
+         *            cluster size: 64KB, L2 table size: 512 entries
+         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
+         *            cluster size: 512B, L2 table size: 4096 entries
+         */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
     }
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

512M of L1 entries is a very loose bound, only 32M are required to store
the maximal supported VMDK file size of 2TB.

Fixed qemu-iotest 59# - now failure occures before on impossible L1
table size.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Liran Alon <liran.alon@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c               | 13 +++++++------
 tests/qemu-iotests/059.out |  2 +-
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
         error_setg(errp, "Invalid granularity, image may be corrupt");
         return -EFBIG;
     }
-    if (l1_size > 512 * 1024 * 1024) {
+    if (l1_size > 32 * 1024 * 1024) {
         /*
          * Although with big capacity and small l1_entry_sectors, we can get a
          * big l1_size, we don't want unbounded value to allocate the table.
-         * Limit it to 512M, which is:
-         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
-         *            cluster size: 64KB, L2 table size: 512 entries
-         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
-         *            cluster size: 512B, L2 table size: 4096 entries
+         * Limit it to 32M, which is enough to store:
+         *     8TB  - for both VMDK3 & VMDK4 with
+         *            minimal cluster size: 512B
+         *            minimal L2 table size: 512 entries
+         *            8 TB is still more than the maximal value supported for
+         *            VMDK3 & VMDK4 which is 2TB.
          */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0x140000000     0x10000         0x50000         TEST_DIR/t-s003.vmdk
 
 === Testing afl image with a very large capacity ===
-qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large
+qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big
 *** done
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in
QEMU).

This format was lacking in the following:

* Grain directory (L1) and grain table (L2) entries were 32-bit,
      allowing access to only 2TB (slightly less) of data.
    * The grain size (default) was 512 bytes - leading to data
      fragmentation and many grain tables.
    * For space reclamation purposes, it was necessary to find all the
      grains which are not pointed to by any grain table - so a reverse
      mapping of "offset of grain in vmdk" to "grain table" must be
      constructed - which takes large amounts of CPU/RAM.

The format specification can be found in VMware's documentation:
https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf

In ESXi 6.5, to support snapshot files larger than 2TB, a new format was
introduced: SESparse (Space Efficient).

This format fixes the above issues:

* All entries are now 64-bit.
    * The grain size (default) is 4KB.
    * Grain directory and grain tables are now located at the beginning
      of the file.
      + seSparse format reserves space for all grain tables.
      + Grain tables can be addressed using an index.
      + Grains are located in the end of the file and can also be
        addressed with an index.
      - seSparse vmdks of large disks (64TB) have huge preallocated
        headers - mainly due to L2 tables, even for empty snapshots.
    * The header contains a reverse mapping ("backmap") of "offset of
      grain in vmdk" to "grain table" and a bitmap ("free bitmap") which
      specifies for each grain - whether it is allocated or not.
      Using these data structures we can implement space reclamation
      efficiently.
    * Due to the fact that the header now maintains two mappings:
        * The regular one (grain directory & grain tables)
        * A reverse one (backmap and free bitmap)
      These data structures can lose consistency upon crash and result
      in a corrupted VMDK.
      Therefore, a journal is also added to the VMDK and is replayed
      when the VMware reopens the file after a crash.

Since ESXi 6.7 - SESparse is the only snapshot format available.

Unfortunately, VMware does not provide documentation regarding the new
seSparse format.

This commit is based on black-box research of the seSparse format.
Various in-guest block operations and their effect on the snapshot file
were tested.

The only VMware provided source of information (regarding the underlying
implementation) was a log file on the ESXi:

/var/log/hostd.log

Whenever an seSparse snapshot is created - the log is being populated
with seSparse records.

Relevant log records are of the form:

[...] Const Header:
[...]  constMagic     = 0xcafebabe
[...]  version        = 2.1
[...]  capacity       = 204800
[...]  grainSize      = 8
[...]  grainTableSize = 64
[...]  flags          = 0
[...] Extents:
[...]  Header         : <1 : 1>
[...]  JournalHdr     : <2 : 2>
[...]  Journal        : <2048 : 2048>
[...]  GrainDirectory : <4096 : 2048>
[...]  GrainTables    : <6144 : 2048>
[...]  FreeBitmap     : <8192 : 2048>
[...]  BackMap        : <10240 : 2048>
[...]  Grain          : <12288 : 204800>
[...] Volatile Header:
[...] volatileMagic     = 0xcafecafe
[...] FreeGTNumber      = 0
[...] nextTxnSeqNumber  = 0
[...] replayJournal     = 0

The sizes that are seen in the log file are in sectors.
Extents are of the following format: <offset : size>

This commit is a strict implementation which enforces:
    * magics
    * version number 2.1
    * grain size of 8 sectors  (4KB)
    * grain table size of 64 sectors
    * zero flags
    * extent locations

Additionally, this commit proivdes only a subset of the functionality
offered by seSparse's format:
    * Read-only
    * No journal replay
    * No space reclamation
    * No unmap support

Hence, journal header, journal, free bitmap and backmap extents are
unused, only the "classic" (L1 -> L2 -> data) grain access is
implemented.

However there are several differences in the grain access itself.
Grain directory (L1):
    * Grain directory entries are indexes (not offsets) to grain
      tables.
    * Valid grain directory entries have their highest nibble set to
      0x1.
    * Since grain tables are always located in the beginning of the
      file - the index can fit into 32 bits - so we can use its low
      part if it's valid.
Grain table (L2):
    * Grain table entries are indexes (not offsets) to grains.
    * If the highest nibble of the entry is:
        0x0:
            The grain in not allocated.
            The rest of the bytes are 0.
        0x1:
            The grain is unmapped - guest sees a zero grain.
            The rest of the bits point to the previously mapped grain,
            see 0x3 case.
        0x2:
            The grain is zero.
        0x3:
            The grain is allocated - to get the index calculate:
            ((entry & 0x0fff000000000000) >> 48) |
            ((entry & 0x0000ffffffffffff) << 12)
    * The difference between 0x1 and 0x2 is that 0x1 is an unallocated
      grain which results from the guest using sg_unmap to unmap the
      grain - but the grain itself still exists in the grain extent - a
      space reclamation procedure should delete it.
      Unmapping a zero grain has no effect (0x2 will not change to 0x1)
      but unmapping an unallocated grain will (0x0 to 0x1) - naturally.

In order to implement seSparse some fields had to be changed to support
both 32-bit and 64-bit entry sizes.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 342 insertions(+), 16 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ typedef struct {
     uint16_t compressAlgorithm;
 } QEMU_PACKED VMDK4Header;
 
+typedef struct VMDKSESparseConstHeader {
+    uint64_t magic;
+    uint64_t version;
+    uint64_t capacity;
+    uint64_t grain_size;
+    uint64_t grain_table_size;
+    uint64_t flags;
+    uint64_t reserved1;
+    uint64_t reserved2;
+    uint64_t reserved3;
+    uint64_t reserved4;
+    uint64_t volatile_header_offset;
+    uint64_t volatile_header_size;
+    uint64_t journal_header_offset;
+    uint64_t journal_header_size;
+    uint64_t journal_offset;
+    uint64_t journal_size;
+    uint64_t grain_dir_offset;
+    uint64_t grain_dir_size;
+    uint64_t grain_tables_offset;
+    uint64_t grain_tables_size;
+    uint64_t free_bitmap_offset;
+    uint64_t free_bitmap_size;
+    uint64_t backmap_offset;
+    uint64_t backmap_size;
+    uint64_t grains_offset;
+    uint64_t grains_size;
+    uint8_t pad[304];
+} QEMU_PACKED VMDKSESparseConstHeader;
+
+typedef struct VMDKSESparseVolatileHeader {
+    uint64_t magic;
+    uint64_t free_gt_number;
+    uint64_t next_txn_seq_number;
+    uint64_t replay_journal;
+    uint8_t pad[480];
+} QEMU_PACKED VMDKSESparseVolatileHeader;
+
 #define L2_CACHE_SIZE 16
 
 typedef struct VmdkExtent {
@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {
     bool compressed;
     bool has_marker;
     bool has_zero_grain;
+    bool sesparse;
+    uint64_t sesparse_l2_tables_offset;
+    uint64_t sesparse_clusters_offset;
+    int32_t entry_size;
     int version;
     int64_t sectors;
     int64_t end_sector;
     int64_t flat_start_offset;
     int64_t l1_table_offset;
     int64_t l1_backup_table_offset;
-    uint32_t *l1_table;
+    void *l1_table;
     uint32_t *l1_backup_table;
     unsigned int l1_size;
     uint32_t l1_entry_sectors;
 
     unsigned int l2_size;
-    uint32_t *l2_cache;
+    void *l2_cache;
     uint32_t l2_cache_offsets[L2_CACHE_SIZE];
     uint32_t l2_cache_counts[L2_CACHE_SIZE];
 
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
          *            minimal L2 table size: 512 entries
          *            8 TB is still more than the maximal value supported for
          *            VMDK3 & VMDK4 which is 2TB.
+         *     64TB - for "ESXi seSparse Extent"
+         *            minimal cluster size: 512B (default is 4KB)
+         *            L2 table size: 4096 entries (const).
+         *            64TB is more than the maximal value supported for
+         *            seSparse VMDKs (which is slightly less than 64TB)
          */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
     extent->l2_size = l2_size;
     extent->cluster_sectors = flat ? sectors : cluster_sectors;
     extent->next_cluster_sector = ROUND_UP(nb_sectors, cluster_sectors);
+    extent->entry_size = sizeof(uint32_t);
 
     if (s->num_extents > 1) {
         extent->end_sector = (*(extent - 1)).end_sector + extent->sectors;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
     int i;
 
     /* read the L1 table */
-    l1_size = extent->l1_size * sizeof(uint32_t);
+    l1_size = extent->l1_size * extent->entry_size;
     extent->l1_table = g_try_malloc(l1_size);
     if (l1_size && extent->l1_table == NULL) {
         return -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
         goto fail_l1;
     }
     for (i = 0; i < extent->l1_size; i++) {
-        le32_to_cpus(&extent->l1_table[i]);
+        if (extent->entry_size == sizeof(uint64_t)) {
+            le64_to_cpus((uint64_t *)extent->l1_table + i);
+        } else {
+            assert(extent->entry_size == sizeof(uint32_t));
+            le32_to_cpus((uint32_t *)extent->l1_table + i);
+        }
     }
 
     if (extent->l1_backup_table_offset) {
+        assert(!extent->sesparse);
         extent->l1_backup_table = g_try_malloc(l1_size);
         if (l1_size && extent->l1_backup_table == NULL) {
             ret = -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
     }
 
     extent->l2_cache =
-        g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE);
+        g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE);
     return 0;
  fail_l1b:
     g_free(extent->l1_backup_table);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
     return ret;
 }
 
+#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe)
+#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe)
+
+/* Strict checks - format not officially documented */
+static int check_se_sparse_const_header(VMDKSESparseConstHeader *header,
+                                        Error **errp)
+{
+    header->magic = le64_to_cpu(header->magic);
+    header->version = le64_to_cpu(header->version);
+    header->grain_size = le64_to_cpu(header->grain_size);
+    header->grain_table_size = le64_to_cpu(header->grain_table_size);
+    header->flags = le64_to_cpu(header->flags);
+    header->reserved1 = le64_to_cpu(header->reserved1);
+    header->reserved2 = le64_to_cpu(header->reserved2);
+    header->reserved3 = le64_to_cpu(header->reserved3);
+    header->reserved4 = le64_to_cpu(header->reserved4);
+
+    header->volatile_header_offset =
+        le64_to_cpu(header->volatile_header_offset);
+    header->volatile_header_size = le64_to_cpu(header->volatile_header_size);
+
+    header->journal_header_offset = le64_to_cpu(header->journal_header_offset);
+    header->journal_header_size = le64_to_cpu(header->journal_header_size);
+
+    header->journal_offset = le64_to_cpu(header->journal_offset);
+    header->journal_size = le64_to_cpu(header->journal_size);
+
+    header->grain_dir_offset = le64_to_cpu(header->grain_dir_offset);
+    header->grain_dir_size = le64_to_cpu(header->grain_dir_size);
+
+    header->grain_tables_offset = le64_to_cpu(header->grain_tables_offset);
+    header->grain_tables_size = le64_to_cpu(header->grain_tables_size);
+
+    header->free_bitmap_offset = le64_to_cpu(header->free_bitmap_offset);
+    header->free_bitmap_size = le64_to_cpu(header->free_bitmap_size);
+
+    header->backmap_offset = le64_to_cpu(header->backmap_offset);
+    header->backmap_size = le64_to_cpu(header->backmap_size);
+
+    header->grains_offset = le64_to_cpu(header->grains_offset);
+    header->grains_size = le64_to_cpu(header->grains_size);
+
+    if (header->magic != SESPARSE_CONST_HEADER_MAGIC) {
+        error_setg(errp, "Bad const header magic: 0x%016" PRIx64,
+                   header->magic);
+        return -EINVAL;
+    }
+
+    if (header->version != 0x0000000200000001) {
+        error_setg(errp, "Unsupported version: 0x%016" PRIx64,
+                   header->version);
+        return -ENOTSUP;
+    }
+
+    if (header->grain_size != 8) {
+        error_setg(errp, "Unsupported grain size: %" PRIu64,
+                   header->grain_size);
+        return -ENOTSUP;
+    }
+
+    if (header->grain_table_size != 64) {
+        error_setg(errp, "Unsupported grain table size: %" PRIu64,
+                   header->grain_table_size);
+        return -ENOTSUP;
+    }
+
+    if (header->flags != 0) {
+        error_setg(errp, "Unsupported flags: 0x%016" PRIx64,
+                   header->flags);
+        return -ENOTSUP;
+    }
+
+    if (header->reserved1 != 0 || header->reserved2 != 0 ||
+        header->reserved3 != 0 || header->reserved4 != 0) {
+        error_setg(errp, "Unsupported reserved bits:"
+                   " 0x%016" PRIx64 " 0x%016" PRIx64
+                   " 0x%016" PRIx64 " 0x%016" PRIx64,
+                   header->reserved1, header->reserved2,
+                   header->reserved3, header->reserved4);
+        return -ENOTSUP;
+    }
+
+    /* check that padding is 0 */
+    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
+        error_setg(errp, "Unsupported non-zero const header padding");
+        return -ENOTSUP;
+    }
+
+    return 0;
+}
+
+static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *header,
+                                           Error **errp)
+{
+    header->magic = le64_to_cpu(header->magic);
+    header->free_gt_number = le64_to_cpu(header->free_gt_number);
+    header->next_txn_seq_number = le64_to_cpu(header->next_txn_seq_number);
+    header->replay_journal = le64_to_cpu(header->replay_journal);
+
+    if (header->magic != SESPARSE_VOLATILE_HEADER_MAGIC) {
+        error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64,
+                   header->magic);
+        return -EINVAL;
+    }
+
+    if (header->replay_journal) {
+        error_setg(errp, "Image is dirty, Replaying journal not supported");
+        return -ENOTSUP;
+    }
+
+    /* check that padding is 0 */
+    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
+        error_setg(errp, "Unsupported non-zero volatile header padding");
+        return -ENOTSUP;
+    }
+
+    return 0;
+}
+
+static int vmdk_open_se_sparse(BlockDriverState *bs,
+                               BdrvChild *file,
+                               int flags, Error **errp)
+{
+    int ret;
+    VMDKSESparseConstHeader const_header;
+    VMDKSESparseVolatileHeader volatile_header;
+    VmdkExtent *extent;
+
+    ret = bdrv_apply_auto_read_only(bs,
+            "No write support for seSparse images available", errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    assert(sizeof(const_header) == SECTOR_SIZE);
+
+    ret = bdrv_pread(file, 0, &const_header, sizeof(const_header));
+    if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
+        error_setg_errno(errp, -ret,
+                         "Could not read const header from file '%s'",
+                         file->bs->filename);
+        return ret;
+    }
+
+    /* check const header */
+    ret = check_se_sparse_const_header(&const_header, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    assert(sizeof(volatile_header) == SECTOR_SIZE);
+
+    ret = bdrv_pread(file,
+                     const_header.volatile_header_offset * SECTOR_SIZE,
+                     &volatile_header, sizeof(volatile_header));
+    if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
+        error_setg_errno(errp, -ret,
+                         "Could not read volatile header from file '%s'",
+                         file->bs->filename);
+        return ret;
+    }
+
+    /* check volatile header */
+    ret = check_se_sparse_volatile_header(&volatile_header, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    ret = vmdk_add_extent(bs, file, false,
+                          const_header.capacity,
+                          const_header.grain_dir_offset * SECTOR_SIZE,
+                          0,
+                          const_header.grain_dir_size *
+                          SECTOR_SIZE / sizeof(uint64_t),
+                          const_header.grain_table_size *
+                          SECTOR_SIZE / sizeof(uint64_t),
+                          const_header.grain_size,
+                          &extent,
+                          errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    extent->sesparse = true;
+    extent->sesparse_l2_tables_offset = const_header.grain_tables_offset;
+    extent->sesparse_clusters_offset = const_header.grains_offset;
+    extent->entry_size = sizeof(uint64_t);
+
+    ret = vmdk_init_tables(bs, extent, errp);
+    if (ret) {
+        /* free extent allocated by vmdk_add_extent */
+        vmdk_free_last_extent(bs);
+    }
+
+    return ret;
+}
+
 static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
                                QDict *options, Error **errp);
 
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          * RW [size in sectors] SPARSE "file-name.vmdk"
          * RW [size in sectors] VMFS "file-name.vmdk"
          * RW [size in sectors] VMFSSPARSE "file-name.vmdk"
+         * RW [size in sectors] SESPARSE "file-name.vmdk"
          */
         flat_offset = -1;
         matches = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
 
         if (sectors <= 0 ||
             (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
-             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) ||
+             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") &&
+             strcmp(type, "SESPARSE")) ||
             (strcmp(access, "RW"))) {
             continue;
         }
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                 return ret;
             }
             extent = &s->extents[s->num_extents - 1];
+        } else if (!strcmp(type, "SESPARSE")) {
+            ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
+            if (ret) {
+                bdrv_unref_child(bs, extent_file);
+                return ret;
+            }
+            extent = &s->extents[s->num_extents - 1];
         } else {
             error_setg(errp, "Unsupported extent type '%s'", type);
             bdrv_unref_child(bs, extent_file);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
     if (strcmp(ct, "monolithicFlat") &&
         strcmp(ct, "vmfs") &&
         strcmp(ct, "vmfsSparse") &&
+        strcmp(ct, "seSparse") &&
         strcmp(ct, "twoGbMaxExtentSparse") &&
         strcmp(ct, "twoGbMaxExtentFlat")) {
         error_setg(errp, "Unsupported image type '%s'", ct);
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
 {
     unsigned int l1_index, l2_offset, l2_index;
     int min_index, i, j;
-    uint32_t min_count, *l2_table;
+    uint32_t min_count;
+    void *l2_table;
     bool zeroed = false;
     int64_t ret;
     int64_t cluster_sector;
+    unsigned int l2_size_bytes = extent->l2_size * extent->entry_size;
 
     if (m_data) {
         m_data->valid = 0;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
     if (l1_index >= extent->l1_size) {
         return VMDK_ERROR;
     }
-    l2_offset = extent->l1_table[l1_index];
+    if (extent->sesparse) {
+        uint64_t l2_offset_u64;
+
+        assert(extent->entry_size == sizeof(uint64_t));
+
+        l2_offset_u64 = ((uint64_t *)extent->l1_table)[l1_index];
+        if (l2_offset_u64 == 0) {
+            l2_offset = 0;
+        } else if ((l2_offset_u64 & 0xffffffff00000000) != 0x1000000000000000) {
+            /*
+             * Top most nibble is 0x1 if grain table is allocated.
+             * strict check - top most 4 bytes must be 0x10000000 since max
+             * supported size is 64TB for disk - so no more than 64TB / 16MB
+             * grain directories which is smaller than uint32,
+             * where 16MB is the only supported default grain table coverage.
+             */
+            return VMDK_ERROR;
+        } else {
+            l2_offset_u64 = l2_offset_u64 & 0x00000000ffffffff;
+            l2_offset_u64 = extent->sesparse_l2_tables_offset +
+                l2_offset_u64 * l2_size_bytes / SECTOR_SIZE;
+            if (l2_offset_u64 > 0x00000000ffffffff) {
+                return VMDK_ERROR;
+            }
+            l2_offset = (unsigned int)(l2_offset_u64);
+        }
+    } else {
+        assert(extent->entry_size == sizeof(uint32_t));
+        l2_offset = ((uint32_t *)extent->l1_table)[l1_index];
+    }
     if (!l2_offset) {
         return VMDK_UNALLOC;
     }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
                     extent->l2_cache_counts[j] >>= 1;
                 }
             }
-            l2_table = extent->l2_cache + (i * extent->l2_size);
+            l2_table = (char *)extent->l2_cache + (i * l2_size_bytes);
             goto found;
         }
     }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
             min_index = i;
         }
     }
-    l2_table = extent->l2_cache + (min_index * extent->l2_size);
+    l2_table = (char *)extent->l2_cache + (min_index * l2_size_bytes);
     BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD);
     if (bdrv_pread(extent->file,
                 (int64_t)l2_offset * 512,
                 l2_table,
-                extent->l2_size * sizeof(uint32_t)
-            ) != extent->l2_size * sizeof(uint32_t)) {
+                l2_size_bytes
+            ) != l2_size_bytes) {
         return VMDK_ERROR;
     }
 
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
     extent->l2_cache_counts[min_index] = 1;
  found:
     l2_index = ((offset >> 9) / extent->cluster_sectors) % extent->l2_size;
-    cluster_sector = le32_to_cpu(l2_table[l2_index]);
 
-    if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
-        zeroed = true;
+    if (extent->sesparse) {
+        cluster_sector = le64_to_cpu(((uint64_t *)l2_table)[l2_index]);
+        switch (cluster_sector & 0xf000000000000000) {
+        case 0x0000000000000000:
+            /* unallocated grain */
+            if (cluster_sector != 0) {
+                return VMDK_ERROR;
+            }
+            break;
+        case 0x1000000000000000:
+            /* scsi-unmapped grain - fallthrough */
+        case 0x2000000000000000:
+            /* zero grain */
+            zeroed = true;
+            break;
+        case 0x3000000000000000:
+            /* allocated grain */
+            cluster_sector = (((cluster_sector & 0x0fff000000000000) >> 48) |
+                              ((cluster_sector & 0x0000ffffffffffff) << 12));
+            cluster_sector = extent->sesparse_clusters_offset +
+                cluster_sector * extent->cluster_sectors;
+            break;
+        default:
+            return VMDK_ERROR;
+        }
+    } else {
+        cluster_sector = le32_to_cpu(((uint32_t *)l2_table)[l2_index]);
+
+        if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
+            zeroed = true;
+        }
     }
 
     if (!cluster_sector || zeroed) {
         if (!allocate) {
             return zeroed ? VMDK_ZEROED : VMDK_UNALLOC;
         }
+        assert(!extent->sesparse);
 
         if (extent->next_cluster_sector >= VMDK_EXTENT_MAX_SECTORS) {
             return VMDK_ERROR;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
             m_data->l1_index = l1_index;
             m_data->l2_index = l2_index;
             m_data->l2_offset = l2_offset;
-            m_data->l2_cache_entry = &l2_table[l2_index];
+            m_data->l2_cache_entry = ((uint32_t *)l2_table) + l2_index;
         }
     }
     *cluster_offset = cluster_sector << BDRV_SECTOR_BITS;
@@ -XXX,XX +XXX,XX @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_t offset,
         if (!extent) {
             return -EIO;
         }
+        if (extent->sesparse) {
+            return -ENOTSUP;
+        }
         offset_in_cluster = vmdk_find_offset_in_cluster(extent, offset);
         n_bytes = MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE
                              - offset_in_cluster);
-- 
2.21.0

From: Pino Toscano <ptoscano@redhat.com>

Rewrite the implementation of the ssh block driver to use libssh instead
of libssh2.  The libssh library has various advantages over libssh2:
- easier API for authentication (for example for using ssh-agent)
- easier API for known_hosts handling
- supports newer types of keys in known_hosts

Use APIs/features available in libssh 0.8 conditionally, to support
older versions (which are not recommended though).

Adjust the iotest 207 according to the different error message, and to
find the default key type for localhost (to properly compare the
fingerprint with).
Contributed-by: Max Reitz <mreitz@redhat.com>

Adjust the various Docker/Travis scripts to use libssh when available
instead of libssh2. The mingw/mxe testing is dropped for now, as there
are no packages for it.

Signed-off-by: Pino Toscano <ptoscano@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20190620200840.17655-1-ptoscano@redhat.com
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 configure                                     |  65 +-
 block/Makefile.objs                           |   6 +-
 block/ssh.c                                   | 652 ++++++++++--------
 .travis.yml                                   |   4 +-
 block/trace-events                            |  14 +-
 docs/qemu-block-drivers.texi                  |   2 +-
 .../dockerfiles/debian-win32-cross.docker     |   1 -
 .../dockerfiles/debian-win64-cross.docker     |   1 -
 tests/docker/dockerfiles/fedora.docker        |   4 +-
 tests/docker/dockerfiles/ubuntu.docker        |   2 +-
 tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
 tests/qemu-iotests/207                        |  54 +-
 tests/qemu-iotests/207.out                    |   2 +-
 13 files changed, 449 insertions(+), 360 deletions(-)

diff --git a/configure b/configure
index XXXXXXX..XXXXXXX 100755
--- a/configure
+++ b/configure
@@ -XXX,XX +XXX,XX @@ auth_pam=""
 vte=""
 virglrenderer=""
 tpm=""
-libssh2=""
+libssh=""
 live_block_migration="yes"
 numa=""
 tcmalloc="no"
@@ -XXX,XX +XXX,XX @@ for opt do
   ;;
   --enable-tpm) tpm="yes"
   ;;
-  --disable-libssh2) libssh2="no"
+  --disable-libssh) libssh="no"
   ;;
-  --enable-libssh2) libssh2="yes"
+  --enable-libssh) libssh="yes"
   ;;
   --disable-live-block-migration) live_block_migration="no"
   ;;
@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
   coroutine-pool  coroutine freelist (better performance)
   glusterfs       GlusterFS backend
   tpm             TPM support
-  libssh2         ssh block device support
+  libssh          ssh block device support
   numa            libnuma support
   libxml2         for Parallels image format
   tcmalloc        tcmalloc support
@@ -XXX,XX +XXX,XX @@ EOF
 fi
 
 ##########################################
-# libssh2 probe
-min_libssh2_version=1.2.8
-if test "$libssh2" != "no" ; then
-  if $pkg_config --atleast-version=$min_libssh2_version libssh2; then
-    libssh2_cflags=$($pkg_config libssh2 --cflags)
-    libssh2_libs=$($pkg_config libssh2 --libs)
-    libssh2=yes
+# libssh probe
+if test "$libssh" != "no" ; then
+  if $pkg_config --exists libssh; then
+    libssh_cflags=$($pkg_config libssh --cflags)
+    libssh_libs=$($pkg_config libssh --libs)
+    libssh=yes
   else
-    if test "$libssh2" = "yes" ; then
-      error_exit "libssh2 >= $min_libssh2_version required for --enable-libssh2"
+    if test "$libssh" = "yes" ; then
+      error_exit "libssh required for --enable-libssh"
     fi
-    libssh2=no
+    libssh=no
   fi
 fi
 
 ##########################################
-# libssh2_sftp_fsync probe
+# Check for libssh 0.8
+# This is done like this instead of using the LIBSSH_VERSION_* and
+# SSH_VERSION_* macros because some distributions in the past shipped
+# snapshots of the future 0.8 from Git, and those snapshots did not
+# have updated version numbers (still referring to 0.7.0).
 
-if test "$libssh2" = "yes"; then
+if test "$libssh" = "yes"; then
   cat > $TMPC <<EOF
-#include <stdio.h>
-#include <libssh2.h>
-#include <libssh2_sftp.h>
-int main(void) {
-    LIBSSH2_SESSION *session;
-    LIBSSH2_SFTP *sftp;
-    LIBSSH2_SFTP_HANDLE *sftp_handle;
-    session = libssh2_session_init ();
-    sftp = libssh2_sftp_init (session);
-    sftp_handle = libssh2_sftp_open (sftp, "/", 0, 0);
-    libssh2_sftp_fsync (sftp_handle);
-    return 0;
-}
+#include <libssh/libssh.h>
+int main(void) { return ssh_get_server_publickey(NULL, NULL); }
 EOF
-  # libssh2_cflags/libssh2_libs defined in previous test.
-  if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then
-    QEMU_CFLAGS="-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS"
+  if compile_prog "$libssh_cflags" "$libssh_libs"; then
+    libssh_cflags="-DHAVE_LIBSSH_0_8 $libssh_cflags"
   fi
 fi
 
@@ -XXX,XX +XXX,XX @@ echo "GlusterFS support $glusterfs"
 echo "gcov              $gcov_tool"
 echo "gcov enabled      $gcov"
 echo "TPM support       $tpm"
-echo "libssh2 support   $libssh2"
+echo "libssh support    $libssh"
 echo "QOM debugging     $qom_cast_debug"
 echo "Live block migration $live_block_migration"
 echo "lzo support       $lzo"
@@ -XXX,XX +XXX,XX @@ if test "$glusterfs_iocb_has_stat" = "yes" ; then
   echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=y" >> $config_host_mak
 fi
 
-if test "$libssh2" = "yes" ; then
-  echo "CONFIG_LIBSSH2=m" >> $config_host_mak
-  echo "LIBSSH2_CFLAGS=$libssh2_cflags" >> $config_host_mak
-  echo "LIBSSH2_LIBS=$libssh2_libs" >> $config_host_mak
+if test "$libssh" = "yes" ; then
+  echo "CONFIG_LIBSSH=m" >> $config_host_mak
+  echo "LIBSSH_CFLAGS=$libssh_cflags" >> $config_host_mak
+  echo "LIBSSH_LIBS=$libssh_libs" >> $config_host_mak
 fi
 
 if test "$live_block_migration" = "yes" ; then
diff --git a/block/Makefile.objs b/block/Makefile.objs
index XXXXXXX..XXXXXXX 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_CURL) += curl.o
 block-obj-$(CONFIG_RBD) += rbd.o
 block-obj-$(CONFIG_GLUSTERFS) += gluster.o
 block-obj-$(CONFIG_VXHS) += vxhs.o
-block-obj-$(CONFIG_LIBSSH2) += ssh.o
+block-obj-$(CONFIG_LIBSSH) += ssh.o
 block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 block-obj-y += backup.o
@@ -XXX,XX +XXX,XX @@ rbd.o-libs         := $(RBD_LIBS)
 gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
 gluster.o-libs     := $(GLUSTERFS_LIBS)
 vxhs.o-libs        := $(VXHS_LIBS)
-ssh.o-cflags       := $(LIBSSH2_CFLAGS)
-ssh.o-libs         := $(LIBSSH2_LIBS)
+ssh.o-cflags       := $(LIBSSH_CFLAGS)
+ssh.o-libs         := $(LIBSSH_LIBS)
 block-obj-dmg-bz2-$(CONFIG_BZIP2) += dmg-bz2.o
 block-obj-$(if $(CONFIG_DMG),m,n) += $(block-obj-dmg-bz2-y)
 dmg-bz2.o-libs     := $(BZIP2_LIBS)
diff --git a/block/ssh.c b/block/ssh.c
index XXXXXXX..XXXXXXX 100644
--- a/block/ssh.c
+++ b/block/ssh.c
@@ -XXX,XX +XXX,XX @@
 
 #include "qemu/osdep.h"
 
-#include <libssh2.h>
-#include <libssh2_sftp.h>
+#include <libssh/libssh.h>
+#include <libssh/sftp.h>
 
 #include "block/block_int.h"
 #include "block/qdict.h"
@@ -XXX,XX +XXX,XX @@
 #include "trace.h"
 
 /*
- * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
- * that this requires that libssh2 was specially compiled with the
- * `./configure --enable-debug' option, so most likely you will have
- * to compile it yourself.  The meaning of <bitmask> is described
- * here: http://www.libssh2.org/libssh2_trace.html
+ * TRACE_LIBSSH=<level> enables tracing in libssh itself.
+ * The meaning of <level> is described here:
+ * http://api.libssh.org/master/group__libssh__log.html
  */
-#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
+#define TRACE_LIBSSH  0 /* see: SSH_LOG_* */
 
 typedef struct BDRVSSHState {
     /* Coroutine. */
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVSSHState {
 
     /* SSH connection. */
     int sock;                         /* socket */
-    LIBSSH2_SESSION *session;         /* ssh session */
-    LIBSSH2_SFTP *sftp;               /* sftp session */
-    LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */
+    ssh_session session;              /* ssh session */
+    sftp_session sftp;                /* sftp session */
+    sftp_file sftp_handle;            /* sftp remote file handle */
 
-    /* See ssh_seek() function below. */
-    int64_t offset;
-    bool offset_op_read;
-
-    /* File attributes at open.  We try to keep the .filesize field
+    /*
+     * File attributes at open.  We try to keep the .size field
      * updated if it changes (eg by writing at the end of the file).
      */
-    LIBSSH2_SFTP_ATTRIBUTES attrs;
+    sftp_attributes attrs;
 
     InetSocketAddress *inet;
 
@@ -XXX,XX +XXX,XX @@ static void ssh_state_init(BDRVSSHState *s)
 {
     memset(s, 0, sizeof *s);
     s->sock = -1;
-    s->offset = -1;
     qemu_co_mutex_init(&s->lock);
 }
 
@@ -XXX,XX +XXX,XX @@ static void ssh_state_free(BDRVSSHState *s)
 {
     g_free(s->user);
 
+    if (s->attrs) {
+        sftp_attributes_free(s->attrs);
+    }
     if (s->sftp_handle) {
-        libssh2_sftp_close(s->sftp_handle);
+        sftp_close(s->sftp_handle);
     }
     if (s->sftp) {
-        libssh2_sftp_shutdown(s->sftp);
+        sftp_free(s->sftp);
     }
     if (s->session) {
-        libssh2_session_disconnect(s->session,
-                                   "from qemu ssh client: "
-                                   "user closed the connection");
-        libssh2_session_free(s->session);
-    }
-    if (s->sock >= 0) {
-        close(s->sock);
+        ssh_disconnect(s->session);
+        ssh_free(s->session); /* This frees s->sock */
     }
 }
 
@@ -XXX,XX +XXX,XX @@ session_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
     va_end(args);
 
     if (s->session) {
-        char *ssh_err;
+        const char *ssh_err;
         int ssh_err_code;
 
-        /* This is not an errno.  See <libssh2.h>. */
-        ssh_err_code = libssh2_session_last_error(s->session,
-                                                  &ssh_err, NULL, 0);
-        error_setg(errp, "%s: %s (libssh2 error code: %d)",
+        /* This is not an errno.  See <libssh/libssh.h>. */
+        ssh_err = ssh_get_error(s->session);
+        ssh_err_code = ssh_get_error_code(s->session);
+        error_setg(errp, "%s: %s (libssh error code: %d)",
                    msg, ssh_err, ssh_err_code);
     } else {
         error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
     va_end(args);
 
     if (s->sftp) {
-        char *ssh_err;
+        const char *ssh_err;
         int ssh_err_code;
-        unsigned long sftp_err_code;
+        int sftp_err_code;
 
-        /* This is not an errno.  See <libssh2.h>. */
-        ssh_err_code = libssh2_session_last_error(s->session,
-                                                  &ssh_err, NULL, 0);
-        /* See <libssh2_sftp.h>. */
-        sftp_err_code = libssh2_sftp_last_error((s)->sftp);
+        /* This is not an errno.  See <libssh/libssh.h>. */
+        ssh_err = ssh_get_error(s->session);
+        ssh_err_code = ssh_get_error_code(s->session);
+        /* See <libssh/sftp.h>. */
+        sftp_err_code = sftp_get_error(s->sftp);
 
         error_setg(errp,
-                   "%s: %s (libssh2 error code: %d, sftp error code: %lu)",
+                   "%s: %s (libssh error code: %d, sftp error code: %d)",
                    msg, ssh_err, ssh_err_code, sftp_err_code);
     } else {
         error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
 
 static void sftp_error_trace(BDRVSSHState *s, const char *op)
 {
-    char *ssh_err;
+    const char *ssh_err;
     int ssh_err_code;
-    unsigned long sftp_err_code;
+    int sftp_err_code;
 
-    /* This is not an errno.  See <libssh2.h>. */
-    ssh_err_code = libssh2_session_last_error(s->session,
-                                              &ssh_err, NULL, 0);
-    /* See <libssh2_sftp.h>. */
-    sftp_err_code = libssh2_sftp_last_error((s)->sftp);
+    /* This is not an errno.  See <libssh/libssh.h>. */
+    ssh_err = ssh_get_error(s->session);
+    ssh_err_code = ssh_get_error_code(s->session);
+    /* See <libssh/sftp.h>. */
+    sftp_err_code = sftp_get_error(s->sftp);
 
     trace_sftp_error(op, ssh_err, ssh_err_code, sftp_err_code);
 }
@@ -XXX,XX +XXX,XX @@ static void ssh_parse_filename(const char *filename, QDict *options,
     parse_uri(filename, options, errp);
 }
 
-static int check_host_key_knownhosts(BDRVSSHState *s,
-                                     const char *host, int port, Error **errp)
+static int check_host_key_knownhosts(BDRVSSHState *s, Error **errp)
 {
-    const char *home;
-    char *knh_file = NULL;
-    LIBSSH2_KNOWNHOSTS *knh = NULL;
-    struct libssh2_knownhost *found;
-    int ret, r;
-    const char *hostkey;
-    size_t len;
-    int type;
-
-    hostkey = libssh2_session_hostkey(s->session, &len, &type);
-    if (!hostkey) {
+    int ret;
+#ifdef HAVE_LIBSSH_0_8
+    enum ssh_known_hosts_e state;
+    int r;
+    ssh_key pubkey;
+    enum ssh_keytypes_e pubkey_type;
+    unsigned char *server_hash = NULL;
+    size_t server_hash_len;
+    char *fingerprint = NULL;
+
+    state = ssh_session_is_known_server(s->session);
+    trace_ssh_server_status(state);
+
+    switch (state) {
+    case SSH_KNOWN_HOSTS_OK:
+        /* OK */
+        trace_ssh_check_host_key_knownhosts();
+        break;
+    case SSH_KNOWN_HOSTS_CHANGED:
         ret = -EINVAL;
-        session_error_setg(errp, s, "failed to read remote host key");
+        r = ssh_get_server_publickey(s->session, &pubkey);
+        if (r == 0) {
+            r = ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256,
+                                       &server_hash, &server_hash_len);
+            pubkey_type = ssh_key_type(pubkey);
+            ssh_key_free(pubkey);
+        }
+        if (r == 0) {
+            fingerprint = ssh_get_fingerprint_hash(SSH_PUBLICKEY_HASH_SHA256,
+                                                   server_hash,
+                                                   server_hash_len);
+            ssh_clean_pubkey_hash(&server_hash);
+        }
+        if (fingerprint) {
+            error_setg(errp,
+                       "host key (%s key with fingerprint %s) does not match "
+                       "the one in known_hosts; this may be a possible attack",
+                       ssh_key_type_to_char(pubkey_type), fingerprint);
+            ssh_string_free_char(fingerprint);
+        } else  {
+            error_setg(errp,
+                       "host key does not match the one in known_hosts; this "
+                       "may be a possible attack");
+        }
         goto out;
-    }
-
-    knh = libssh2_knownhost_init(s->session);
-    if (!knh) {
+    case SSH_KNOWN_HOSTS_OTHER:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                           "failed to initialize known hosts support");
+        error_setg(errp,
+                   "host key for this server not found, another type exists");
+        goto out;
+    case SSH_KNOWN_HOSTS_UNKNOWN:
+        ret = -EINVAL;
+        error_setg(errp, "no host key was found in known_hosts");
+        goto out;
+    case SSH_KNOWN_HOSTS_NOT_FOUND:
+        ret = -ENOENT;
+        error_setg(errp, "known_hosts file not found");
+        goto out;
+    case SSH_KNOWN_HOSTS_ERROR:
+        ret = -EINVAL;
+        error_setg(errp, "error while checking the host");
+        goto out;
+    default:
+        ret = -EINVAL;
+        error_setg(errp, "error while checking for known server (%d)", state);
         goto out;
     }
+#else /* !HAVE_LIBSSH_0_8 */
+    int state;
 
-    home = getenv("HOME");
-    if (home) {
-        knh_file = g_strdup_printf("%s/.ssh/known_hosts", home);
-    } else {
-        knh_file = g_strdup_printf("/root/.ssh/known_hosts");
-    }
-
-    /* Read all known hosts from OpenSSH-style known_hosts file. */
-    libssh2_knownhost_readfile(knh, knh_file, LIBSSH2_KNOWNHOST_FILE_OPENSSH);
+    state = ssh_is_server_known(s->session);
+    trace_ssh_server_status(state);
 
-    r = libssh2_knownhost_checkp(knh, host, port, hostkey, len,
-                                 LIBSSH2_KNOWNHOST_TYPE_PLAIN|
-                                 LIBSSH2_KNOWNHOST_KEYENC_RAW,
-                                 &found);
-    switch (r) {
-    case LIBSSH2_KNOWNHOST_CHECK_MATCH:
+    switch (state) {
+    case SSH_SERVER_KNOWN_OK:
         /* OK */
-        trace_ssh_check_host_key_knownhosts(found->key);
+        trace_ssh_check_host_key_knownhosts();
         break;
-    case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
+    case SSH_SERVER_KNOWN_CHANGED:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                      "host key does not match the one in known_hosts"
-                      " (found key %s)", found->key);
+        error_setg(errp,
+                   "host key does not match the one in known_hosts; this "
+                   "may be a possible attack");
         goto out;
-    case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND:
+    case SSH_SERVER_FOUND_OTHER:
         ret = -EINVAL;
-        session_error_setg(errp, s, "no host key was found in known_hosts");
+        error_setg(errp,
+                   "host key for this server not found, another type exists");
+        goto out;
+    case SSH_SERVER_FILE_NOT_FOUND:
+        ret = -ENOENT;
+        error_setg(errp, "known_hosts file not found");
         goto out;
-    case LIBSSH2_KNOWNHOST_CHECK_FAILURE:
+    case SSH_SERVER_NOT_KNOWN:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                      "failure matching the host key with known_hosts");
+        error_setg(errp, "no host key was found in known_hosts");
+        goto out;
+    case SSH_SERVER_ERROR:
+        ret = -EINVAL;
+        error_setg(errp, "server error");
         goto out;
     default:
         ret = -EINVAL;
-        session_error_setg(errp, s, "unknown error matching the host key"
-                      " with known_hosts (%d)", r);
+        error_setg(errp, "error while checking for known server (%d)", state);
         goto out;
     }
+#endif /* !HAVE_LIBSSH_0_8 */
 
     /* known_hosts checking successful. */
     ret = 0;
 
  out:
-    if (knh != NULL) {
-        libssh2_knownhost_free(knh);
-    }
-    g_free(knh_file);
     return ret;
 }
 
@@ -XXX,XX +XXX,XX @@ static int compare_fingerprint(const unsigned char *fingerprint, size_t len,
 
 static int
 check_host_key_hash(BDRVSSHState *s, const char *hash,
-                    int hash_type, size_t fingerprint_len, Error **errp)
+                    enum ssh_publickey_hash_type type, Error **errp)
 {
-    const char *fingerprint;
-
-    fingerprint = libssh2_hostkey_hash(s->session, hash_type);
-    if (!fingerprint) {
+    int r;
+    ssh_key pubkey;
+    unsigned char *server_hash;
+    size_t server_hash_len;
+
+#ifdef HAVE_LIBSSH_0_8
+    r = ssh_get_server_publickey(s->session, &pubkey);
+#else
+    r = ssh_get_publickey(s->session, &pubkey);
+#endif
+    if (r != SSH_OK) {
         session_error_setg(errp, s, "failed to read remote host key");
         return -EINVAL;
     }
 
-    if(compare_fingerprint((unsigned char *) fingerprint, fingerprint_len,
-                           hash) != 0) {
+    r = ssh_get_publickey_hash(pubkey, type, &server_hash, &server_hash_len);
+    ssh_key_free(pubkey);
+    if (r != 0) {
+        session_error_setg(errp, s,
+                           "failed reading the hash of the server SSH key");
+        return -EINVAL;
+    }
+
+    r = compare_fingerprint(server_hash, server_hash_len, hash);
+    ssh_clean_pubkey_hash(&server_hash);
+    if (r != 0) {
         error_setg(errp, "remote host key does not match host_key_check '%s'",
                    hash);
         return -EPERM;
@@ -XXX,XX +XXX,XX @@ check_host_key_hash(BDRVSSHState *s, const char *hash,
     return 0;
 }
 
-static int check_host_key(BDRVSSHState *s, const char *host, int port,
-                          SshHostKeyCheck *hkc, Error **errp)
+static int check_host_key(BDRVSSHState *s, SshHostKeyCheck *hkc, Error **errp)
 {
     SshHostKeyCheckMode mode;
 
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
     case SSH_HOST_KEY_CHECK_MODE_HASH:
         if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) {
             return check_host_key_hash(s, hkc->u.hash.hash,
-                                       LIBSSH2_HOSTKEY_HASH_MD5, 16, errp);
+                                       SSH_PUBLICKEY_HASH_MD5, errp);
         } else if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_SHA1) {
             return check_host_key_hash(s, hkc->u.hash.hash,
-                                       LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp);
+                                       SSH_PUBLICKEY_HASH_SHA1, errp);
         }
         g_assert_not_reached();
         break;
     case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS:
-        return check_host_key_knownhosts(s, host, port, errp);
+        return check_host_key_knownhosts(s, errp);
     default:
         g_assert_not_reached();
     }
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
     return -EINVAL;
 }
 
-static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
+static int authenticate(BDRVSSHState *s, Error **errp)
 {
     int r, ret;
-    const char *userauthlist;
-    LIBSSH2_AGENT *agent = NULL;
-    struct libssh2_agent_publickey *identity;
-    struct libssh2_agent_publickey *prev_identity = NULL;
+    int method;
 
-    userauthlist = libssh2_userauth_list(s->session, user, strlen(user));
-    if (strstr(userauthlist, "publickey") == NULL) {
+    /* Try to authenticate with the "none" method. */
+    r = ssh_userauth_none(s->session, NULL);
+    if (r == SSH_AUTH_ERROR) {
         ret = -EPERM;
-        error_setg(errp,
-                "remote server does not support \"publickey\" authentication");
+        session_error_setg(errp, s, "failed to authenticate using none "
+                                    "authentication");
         goto out;
-    }
-
-    /* Connect to ssh-agent and try each identity in turn. */
-    agent = libssh2_agent_init(s->session);
-    if (!agent) {
-        ret = -EINVAL;
-        session_error_setg(errp, s, "failed to initialize ssh-agent support");
-        goto out;
-    }
-    if (libssh2_agent_connect(agent)) {
-        ret = -ECONNREFUSED;
-        session_error_setg(errp, s, "failed to connect to ssh-agent");
-        goto out;
-    }
-    if (libssh2_agent_list_identities(agent)) {
-        ret = -EINVAL;
-        session_error_setg(errp, s,
-                           "failed requesting identities from ssh-agent");
+    } else if (r == SSH_AUTH_SUCCESS) {
+        /* Authenticated! */
+        ret = 0;
         goto out;
     }
 
-    for(;;) {
-        r = libssh2_agent_get_identity(agent, &identity, prev_identity);
-        if (r == 1) {           /* end of list */
-            break;
-        }
-        if (r < 0) {
+    method = ssh_userauth_list(s->session, NULL);
+    trace_ssh_auth_methods(method);
+
+    /*
+     * Try to authenticate with publickey, using the ssh-agent
+     * if available.
+     */
+    if (method & SSH_AUTH_METHOD_PUBLICKEY) {
+        r = ssh_userauth_publickey_auto(s->session, NULL, NULL);
+        if (r == SSH_AUTH_ERROR) {
             ret = -EINVAL;
-            session_error_setg(errp, s,
-                               "failed to obtain identity from ssh-agent");
+            session_error_setg(errp, s, "failed to authenticate using "
+                                        "publickey authentication");
             goto out;
-        }
-        r = libssh2_agent_userauth(agent, user, identity);
-        if (r == 0) {
+        } else if (r == SSH_AUTH_SUCCESS) {
             /* Authenticated! */
             ret = 0;
             goto out;
         }
-        /* Failed to authenticate with this identity, try the next one. */
-        prev_identity = identity;
     }
 
     ret = -EPERM;
@@ -XXX,XX +XXX,XX @@ static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
                "and the identities held by your ssh-agent");
 
  out:
-    if (agent != NULL) {
-        /* Note: libssh2 implementation implicitly calls
-         * libssh2_agent_disconnect if necessary.
-         */
-        libssh2_agent_free(agent);
-    }
-
     return ret;
 }
 
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
                           int ssh_flags, int creat_mode, Error **errp)
 {
     int r, ret;
-    long port = 0;
+    unsigned int port = 0;
+    int new_sock = -1;
 
     if (opts->has_user) {
         s->user = g_strdup(opts->user);
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
     s->inet = opts->server;
     opts->server = NULL;
 
-    if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) {
+    if (qemu_strtoui(s->inet->port, NULL, 10, &port) < 0) {
         error_setg(errp, "Use only numeric port value");
         ret = -EINVAL;
         goto err;
     }
 
     /* Open the socket and connect. */
-    s->sock = inet_connect_saddr(s->inet, errp);
-    if (s->sock < 0) {
+    new_sock = inet_connect_saddr(s->inet, errp);
+    if (new_sock < 0) {
         ret = -EIO;
         goto err;
     }
 
+    /*
+     * Try to disable the Nagle algorithm on TCP sockets to reduce latency,
+     * but do not fail if it cannot be disabled.
+     */
+    r = socket_set_nodelay(new_sock);
+    if (r < 0) {
+        warn_report("can't set TCP_NODELAY for the ssh server %s: %s",
+                    s->inet->host, strerror(errno));
+    }
+
     /* Create SSH session. */
-    s->session = libssh2_session_init();
+    s->session = ssh_new();
     if (!s->session) {
         ret = -EINVAL;
-        session_error_setg(errp, s, "failed to initialize libssh2 session");
+        session_error_setg(errp, s, "failed to initialize libssh session");
         goto err;
     }
 
-#if TRACE_LIBSSH2 != 0
-    libssh2_trace(s->session, TRACE_LIBSSH2);
-#endif
+    /*
+     * Make sure we are in blocking mode during the connection and
+     * authentication phases.
+     */
+    ssh_set_blocking(s->session, 1);
 
-    r = libssh2_session_handshake(s->session, s->sock);
-    if (r != 0) {
+    r = ssh_options_set(s->session, SSH_OPTIONS_USER, s->user);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the user in the libssh session");
+        goto err;
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_HOST, s->inet->host);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the host in the libssh session");
+        goto err;
+    }
+
+    if (port > 0) {
+        r = ssh_options_set(s->session, SSH_OPTIONS_PORT, &port);
+        if (r < 0) {
+            ret = -EINVAL;
+            session_error_setg(errp, s,
+                               "failed to set the port in the libssh session");
+            goto err;
+        }
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_COMPRESSION, "none");
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to disable the compression in the libssh "
+                           "session");
+        goto err;
+    }
+
+    /* Read ~/.ssh/config. */
+    r = ssh_options_parse_config(s->session, NULL);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s, "failed to parse ~/.ssh/config");
+        goto err;
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_FD, &new_sock);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the socket in the libssh session");
+        goto err;
+    }
+    /* libssh took ownership of the socket. */
+    s->sock = new_sock;
+    new_sock = -1;
+
+    /* Connect. */
+    r = ssh_connect(s->session);
+    if (r != SSH_OK) {
         ret = -EINVAL;
         session_error_setg(errp, s, "failed to establish SSH session");
         goto err;
     }
 
     /* Check the remote host's key against known_hosts. */
-    ret = check_host_key(s, s->inet->host, port, opts->host_key_check, errp);
+    ret = check_host_key(s, opts->host_key_check, errp);
     if (ret < 0) {
         goto err;
     }
 
     /* Authenticate. */
-    ret = authenticate(s, s->user, errp);
+    ret = authenticate(s, errp);
     if (ret < 0) {
         goto err;
     }
 
     /* Start SFTP. */
-    s->sftp = libssh2_sftp_init(s->session);
+    s->sftp = sftp_new(s->session);
     if (!s->sftp) {
-        session_error_setg(errp, s, "failed to initialize sftp handle");
+        session_error_setg(errp, s, "failed to create sftp handle");
+        ret = -EINVAL;
+        goto err;
+    }
+
+    r = sftp_init(s->sftp);
+    if (r < 0) {
+        sftp_error_setg(errp, s, "failed to initialize sftp handle");
         ret = -EINVAL;
         goto err;
     }
 
     /* Open the remote file. */
     trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
-    s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
-                                       creat_mode);
+    s->sftp_handle = sftp_open(s->sftp, opts->path, ssh_flags, creat_mode);
     if (!s->sftp_handle) {
-        session_error_setg(errp, s, "failed to open remote file '%s'",
-                           opts->path);
+        sftp_error_setg(errp, s, "failed to open remote file '%s'",
+                        opts->path);
         ret = -EINVAL;
         goto err;
     }
 
-    r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs);
-    if (r < 0) {
+    /* Make sure the SFTP file is handled in blocking mode. */
+    sftp_file_set_blocking(s->sftp_handle);
+
+    s->attrs = sftp_fstat(s->sftp_handle);
+    if (!s->attrs) {
         sftp_error_setg(errp, s, "failed to read file attributes");
         return -EINVAL;
     }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
     return 0;
 
  err:
+    if (s->attrs) {
+        sftp_attributes_free(s->attrs);
+    }
+    s->attrs = NULL;
     if (s->sftp_handle) {
-        libssh2_sftp_close(s->sftp_handle);
+        sftp_close(s->sftp_handle);
     }
     s->sftp_handle = NULL;
     if (s->sftp) {
-        libssh2_sftp_shutdown(s->sftp);
+        sftp_free(s->sftp);
     }
     s->sftp = NULL;
     if (s->session) {
-        libssh2_session_disconnect(s->session,
-                                   "from qemu ssh client: "
-                                   "error opening connection");
-        libssh2_session_free(s->session);
+        ssh_disconnect(s->session);
+        ssh_free(s->session);
     }
     s->session = NULL;
+    s->sock = -1;
+    if (new_sock >= 0) {
+        close(new_sock);
+    }
 
     return ret;
 }
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
 
     ssh_state_init(s);
 
-    ssh_flags = LIBSSH2_FXF_READ;
+    ssh_flags = 0;
     if (bdrv_flags & BDRV_O_RDWR) {
-        ssh_flags |= LIBSSH2_FXF_WRITE;
+        ssh_flags |= O_RDWR;
+    } else {
+        ssh_flags |= O_RDONLY;
     }
 
     opts = ssh_parse_options(options, errp);
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
     }
 
     /* Go non-blocking. */
-    libssh2_session_set_blocking(s->session, 0);
+    ssh_set_blocking(s->session, 0);
 
     qapi_free_BlockdevOptionsSsh(opts);
 
     return 0;
 
  err:
-    if (s->sock >= 0) {
-        close(s->sock);
-    }
-    s->sock = -1;
-
     qapi_free_BlockdevOptionsSsh(opts);
 
     return ret;
@@ -XXX,XX +XXX,XX @@ static int ssh_grow_file(BDRVSSHState *s, int64_t offset, Error **errp)
 {
     ssize_t ret;
     char c[1] = { '\0' };
-    int was_blocking = libssh2_session_get_blocking(s->session);
+    int was_blocking = ssh_is_blocking(s->session);
 
     /* offset must be strictly greater than the current size so we do
      * not overwrite anything */
-    assert(offset > 0 && offset > s->attrs.filesize);
+    assert(offset > 0 && offset > s->attrs->size);
 
-    libssh2_session_set_blocking(s->session, 1);
+    ssh_set_blocking(s->session, 1);
 
-    libssh2_sftp_seek64(s->sftp_handle, offset - 1);
-    ret = libssh2_sftp_write(s->sftp_handle, c, 1);
+    sftp_seek64(s->sftp_handle, offset - 1);
+    ret = sftp_write(s->sftp_handle, c, 1);
 
-    libssh2_session_set_blocking(s->session, was_blocking);
+    ssh_set_blocking(s->session, was_blocking);
 
     if (ret < 0) {
         sftp_error_setg(errp, s, "Failed to grow file");
         return -EIO;
     }
 
-    s->attrs.filesize = offset;
+    s->attrs->size = offset;
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ static int ssh_co_create(BlockdevCreateOptions *options, Error **errp)
     ssh_state_init(&s);
 
     ret = connect_to_ssh(&s, opts->location,
-                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
-                         LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
+                         O_RDWR | O_CREAT | O_TRUNC,
                          0644, errp);
     if (ret < 0) {
         goto fail;
@@ -XXX,XX +XXX,XX @@ static int ssh_has_zero_init(BlockDriverState *bs)
     /* Assume false, unless we can positively prove it's true. */
     int has_zero_init = 0;
 
-    if (s->attrs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) {
-        if (s->attrs.permissions & LIBSSH2_SFTP_S_IFREG) {
-            has_zero_init = 1;
-        }
+    if (s->attrs->type == SSH_FILEXFER_TYPE_REGULAR) {
+        has_zero_init = 1;
     }
 
     return has_zero_init;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
         .co = qemu_coroutine_self()
     };
 
-    r = libssh2_session_block_directions(s->session);
+    r = ssh_get_poll_flags(s->session);
 
-    if (r & LIBSSH2_SESSION_BLOCK_INBOUND) {
+    if (r & SSH_READ_PENDING) {
         rd_handler = restart_coroutine;
     }
-    if (r & LIBSSH2_SESSION_BLOCK_OUTBOUND) {
+    if (r & SSH_WRITE_PENDING) {
         wr_handler = restart_coroutine;
     }
 
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
     trace_ssh_co_yield_back(s->sock);
 }
 
-/* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
- * in the remote file.  Notice that it just updates a field in the
- * sftp_handle structure, so there is no network traffic and it cannot
- * fail.
- *
- * However, `libssh2_sftp_seek64' does have a catastrophic effect on
- * performance since it causes the handle to throw away all in-flight
- * reads and buffered readahead data.  Therefore this function tries
- * to be intelligent about when to call the underlying libssh2 function.
- */
-#define SSH_SEEK_WRITE 0
-#define SSH_SEEK_READ  1
-#define SSH_SEEK_FORCE 2
-
-static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
-{
-    bool op_read = (flags & SSH_SEEK_READ) != 0;
-    bool force = (flags & SSH_SEEK_FORCE) != 0;
-
-    if (force || op_read != s->offset_op_read || offset != s->offset) {
-        trace_ssh_seek(offset);
-        libssh2_sftp_seek64(s->sftp_handle, offset);
-        s->offset = offset;
-        s->offset_op_read = op_read;
-    }
-}
-
 static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
                                  int64_t offset, size_t size,
                                  QEMUIOVector *qiov)
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
 
     trace_ssh_read(offset, size);
 
-    ssh_seek(s, offset, SSH_SEEK_READ);
+    trace_ssh_seek(offset);
+    sftp_seek64(s->sftp_handle, offset);
 
     /* This keeps track of the current iovec element ('i'), where we
      * will write to next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
     buf = i->iov_base;
     end_of_vec = i->iov_base + i->iov_len;
 
-    /* libssh2 has a hard-coded limit of 2000 bytes per request,
-     * although it will also do readahead behind our backs.  Therefore
-     * we may have to do repeated reads here until we have read 'size'
-     * bytes.
-     */
     for (got = 0; got < size; ) {
+        size_t request_read_size;
     again:
-        trace_ssh_read_buf(buf, end_of_vec - buf);
-        r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
-        trace_ssh_read_return(r);
+        /*
+         * The size of SFTP packets is limited to 32K bytes, so limit
+         * the amount of data requested to 16K, as libssh currently
+         * does not handle multiple requests on its own.
+         */
+        request_read_size = MIN(end_of_vec - buf, 16384);
+        trace_ssh_read_buf(buf, end_of_vec - buf, request_read_size);
+        r = sftp_read(s->sftp_handle, buf, request_read_size);
+        trace_ssh_read_return(r, sftp_get_error(s->sftp));
 
-        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+        if (r == SSH_AGAIN) {
             co_yield(s, bs);
             goto again;
         }
-        if (r < 0) {
-            sftp_error_trace(s, "read");
-            s->offset = -1;
-            return -EIO;
-        }
-        if (r == 0) {
+        if (r == SSH_EOF || (r == 0 && sftp_get_error(s->sftp) == SSH_FX_EOF)) {
             /* EOF: Short read so pad the buffer with zeroes and return it. */
             qemu_iovec_memset(qiov, got, 0, size - got);
             return 0;
         }
+        if (r <= 0) {
+            sftp_error_trace(s, "read");
+            return -EIO;
+        }
 
         got += r;
         buf += r;
-        s->offset += r;
         if (buf >= end_of_vec && got < size) {
             i++;
             buf = i->iov_base;
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
 
     trace_ssh_write(offset, size);
 
-    ssh_seek(s, offset, SSH_SEEK_WRITE);
+    trace_ssh_seek(offset);
+    sftp_seek64(s->sftp_handle, offset);
 
     /* This keeps track of the current iovec element ('i'), where we
      * will read from next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
     end_of_vec = i->iov_base + i->iov_len;
 
     for (written = 0; written < size; ) {
+        size_t request_write_size;
     again:
-        trace_ssh_write_buf(buf, end_of_vec - buf);
-        r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
-        trace_ssh_write_return(r);
+        /*
+         * Avoid too large data packets, as libssh currently does not
+         * handle multiple requests on its own.
+         */
+        request_write_size = MIN(end_of_vec - buf, 131072);
+        trace_ssh_write_buf(buf, end_of_vec - buf, request_write_size);
+        r = sftp_write(s->sftp_handle, buf, request_write_size);
+        trace_ssh_write_return(r, sftp_get_error(s->sftp));
 
-        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+        if (r == SSH_AGAIN) {
             co_yield(s, bs);
             goto again;
         }
         if (r < 0) {
             sftp_error_trace(s, "write");
-            s->offset = -1;
             return -EIO;
         }
-        /* The libssh2 API is very unclear about this.  A comment in
-         * the code says "nothing was acked, and no EAGAIN was
-         * received!" which apparently means that no data got sent
-         * out, and the underlying channel didn't return any EAGAIN
-         * indication.  I think this is a bug in either libssh2 or
-         * OpenSSH (server-side).  In any case, forcing a seek (to
-         * discard libssh2 internal buffers), and then trying again
-         * works for me.
-         */
-        if (r == 0) {
-            ssh_seek(s, offset + written, SSH_SEEK_WRITE|SSH_SEEK_FORCE);
-            co_yield(s, bs);
-            goto again;
-        }
 
         written += r;
         buf += r;
-        s->offset += r;
         if (buf >= end_of_vec && written < size) {
             i++;
             buf = i->iov_base;
             end_of_vec = i->iov_base + i->iov_len;
         }
 
-        if (offset + written > s->attrs.filesize)
-            s->attrs.filesize = offset + written;
+        if (offset + written > s->attrs->size) {
+            s->attrs->size = offset + written;
+        }
     }
 
     return 0;
@@ -XXX,XX +XXX,XX @@ static void unsafe_flush_warning(BDRVSSHState *s, const char *what)
     }
 }
 
-#ifdef HAS_LIBSSH2_SFTP_FSYNC
+#ifdef HAVE_LIBSSH_0_8
 
 static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
 {
     int r;
 
     trace_ssh_flush();
+
+    if (!sftp_extension_supported(s->sftp, "fsync@openssh.com", "1")) {
+        unsafe_flush_warning(s, "OpenSSH >= 6.3");
+        return 0;
+    }
  again:
-    r = libssh2_sftp_fsync(s->sftp_handle);
-    if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+    r = sftp_fsync(s->sftp_handle);
+    if (r == SSH_AGAIN) {
         co_yield(s, bs);
         goto again;
     }
-    if (r == LIBSSH2_ERROR_SFTP_PROTOCOL &&
-        libssh2_sftp_last_error(s->sftp) == LIBSSH2_FX_OP_UNSUPPORTED) {
-        unsafe_flush_warning(s, "OpenSSH >= 6.3");
-        return 0;
-    }
     if (r < 0) {
         sftp_error_trace(s, "fsync");
         return -EIO;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
     return ret;
 }
 
-#else /* !HAS_LIBSSH2_SFTP_FSYNC */
+#else /* !HAVE_LIBSSH_0_8 */
 
 static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
 {
     BDRVSSHState *s = bs->opaque;
 
-    unsafe_flush_warning(s, "libssh2 >= 1.4.4");
+    unsafe_flush_warning(s, "libssh >= 0.8.0");
     return 0;
 }
 
-#endif /* !HAS_LIBSSH2_SFTP_FSYNC */
+#endif /* !HAVE_LIBSSH_0_8 */
 
 static int64_t ssh_getlength(BlockDriverState *bs)
 {
     BDRVSSHState *s = bs->opaque;
     int64_t length;
 
-    /* Note we cannot make a libssh2 call here. */
-    length = (int64_t) s->attrs.filesize;
+    /* Note we cannot make a libssh call here. */
+    length = (int64_t) s->attrs->size;
     trace_ssh_getlength(length);
 
     return length;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_truncate(BlockDriverState *bs, int64_t offset,
         return -ENOTSUP;
     }
 
-    if (offset < s->attrs.filesize) {
+    if (offset < s->attrs->size) {
         error_setg(errp, "ssh driver does not support shrinking files");
         return -ENOTSUP;
     }
 
-    if (offset == s->attrs.filesize) {
+    if (offset == s->attrs->size) {
         return 0;
     }
 
@@ -XXX,XX +XXX,XX @@ static void bdrv_ssh_init(void)
 {
     int r;
 
-    r = libssh2_init(0);
+    r = ssh_init();
     if (r != 0) {
-        fprintf(stderr, "libssh2 initialization failed, %d\n", r);
+        fprintf(stderr, "libssh initialization failed, %d\n", r);
         exit(EXIT_FAILURE);
     }
 
+#if TRACE_LIBSSH != 0
+    ssh_set_log_level(TRACE_LIBSSH);
+#endif
+
     bdrv_register(&bdrv_ssh);
 }
 
diff --git a/.travis.yml b/.travis.yml
index XXXXXXX..XXXXXXX 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -XXX,XX +XXX,XX @@ addons:
       - libseccomp-dev
       - libspice-protocol-dev
       - libspice-server-dev
-      - libssh2-1-dev
+      - libssh-dev
       - liburcu-dev
       - libusb-1.0-0-dev
       - libvte-2.91-dev
@@ -XXX,XX +XXX,XX @@ matrix:
             - libseccomp-dev
             - libspice-protocol-dev
             - libspice-server-dev
-            - libssh2-1-dev
+            - libssh-dev
             - liburcu-dev
             - libusb-1.0-0-dev
             - libvte-2.91-dev
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ nbd_client_connect_success(const char *export_name) "export '%s'"
 # ssh.c
 ssh_restart_coroutine(void *co) "co=%p"
 ssh_flush(void) "fsync"
-ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
+ssh_check_host_key_knownhosts(void) "host key OK"
 ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
 ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
 ssh_co_yield_back(int sock) "s->sock=%d - back"
 ssh_getlength(int64_t length) "length=%" PRIi64
 ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
 ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
-ssh_read_return(ssize_t ret) "sftp_read returned %zd"
+ssh_read_buf(void *buf, size_t size, size_t actual_size) "sftp_read buf=%p size=%zu (actual size=%zu)"
+ssh_read_return(ssize_t ret, int sftp_err) "sftp_read returned %zd (sftp error=%d)"
 ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
-ssh_write_return(ssize_t ret) "sftp_write returned %zd"
+ssh_write_buf(void *buf, size_t size, size_t actual_size) "sftp_write buf=%p size=%zu (actual size=%zu)"
+ssh_write_return(ssize_t ret, int sftp_err) "sftp_write returned %zd (sftp error=%d)"
 ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
+ssh_auth_methods(int methods) "auth methods=0x%x"
+ssh_server_status(int status) "server status=%d"
 
 # curl.c
 curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
@@ -XXX,XX +XXX,XX @@ sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
 sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
 
 # ssh.c
-sftp_error(const char *op, const char *ssh_err, int ssh_err_code, unsigned long sftp_err_code) "%s failed: %s (libssh2 error code: %d, sftp error code: %lu)"
+sftp_error(const char *op, const char *ssh_err, int ssh_err_code, int sftp_err_code) "%s failed: %s (libssh error code: %d, sftp error code: %d)"
diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi
index XXXXXXX..XXXXXXX 100644
--- a/docs/qemu-block-drivers.texi
+++ b/docs/qemu-block-drivers.texi
@@ -XXX,XX +XXX,XX @@ print a warning when @code{fsync} is not supported:
 
 warning: ssh server @code{ssh.example.com:22} does not support fsync
 
-With sufficiently new versions of libssh2 and OpenSSH, @code{fsync} is
+With sufficiently new versions of libssh and OpenSSH, @code{fsync} is
 supported.
 
 @node disk_images_nvme
diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/debian-win32-cross.docker
+++ b/tests/docker/dockerfiles/debian-win32-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         mxe-$TARGET-w64-mingw32.shared-curl \
         mxe-$TARGET-w64-mingw32.shared-glib \
         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
         mxe-$TARGET-w64-mingw32.shared-libusb1 \
         mxe-$TARGET-w64-mingw32.shared-lzo \
         mxe-$TARGET-w64-mingw32.shared-nettle \
diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/debian-win64-cross.docker
+++ b/tests/docker/dockerfiles/debian-win64-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         mxe-$TARGET-w64-mingw32.shared-curl \
         mxe-$TARGET-w64-mingw32.shared-glib \
         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
         mxe-$TARGET-w64-mingw32.shared-libusb1 \
         mxe-$TARGET-w64-mingw32.shared-lzo \
         mxe-$TARGET-w64-mingw32.shared-nettle \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     libpng-devel \
     librbd-devel \
     libseccomp-devel \
-    libssh2-devel \
+    libssh-devel \
     libubsan \
     libusbx-devel \
     libxml2-devel \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     mingw32-gtk3 \
     mingw32-libjpeg-turbo \
     mingw32-libpng \
-    mingw32-libssh2 \
     mingw32-libtasn1 \
     mingw32-nettle \
     mingw32-pixman \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     mingw64-gtk3 \
     mingw64-libjpeg-turbo \
     mingw64-libpng \
-    mingw64-libssh2 \
     mingw64-libtasn1 \
     mingw64-nettle \
     mingw64-pixman \
diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
-    libssh2-1-dev \
+    libssh-dev \
     libusb-1.0-0-dev \
     libusbredirhost-dev \
     libvdeplug-dev \
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
-    libssh2-1-dev \
+    libssh-dev \
     libusb-1.0-0-dev \
     libusbredirhost-dev \
     libvdeplug-dev \
diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
 
     iotests.img_info_log(remote_path)
 
-    md5_key = subprocess.check_output(
-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-        'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-        shell=True).rstrip().decode('ascii')
+    keys = subprocess.check_output(
+        'ssh-keyscan 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
+        'cut -d" " -f3',
+        shell=True).rstrip().decode('ascii').split('\n')
+
+    # Mappings of base64 representations to digests
+    md5_keys = {}
+    sha1_keys = {}
+
+    for key in keys:
+        md5_keys[key] = subprocess.check_output(
+            'echo %s | base64 -d | md5sum -b | cut -d" " -f1' % key,
+            shell=True).rstrip().decode('ascii')
+
+        sha1_keys[key] = subprocess.check_output(
+            'echo %s | base64 -d | sha1sum -b | cut -d" " -f1' % key,
+            shell=True).rstrip().decode('ascii')
 
     vm.launch()
+
+    # Find correct key first
+    matching_key = None
+    for key in keys:
+        result = vm.qmp('blockdev-add',
+                        driver='ssh', node_name='node0', path=disk_path,
+                        server={
+                             'host': '127.0.0.1',
+                             'port': '22',
+                        }, host_key_check={
+                             'mode': 'hash',
+                             'type': 'md5',
+                             'hash': md5_keys[key],
+                        })
+
+        if 'error' not in result:
+            vm.qmp('blockdev-del', node_name='node0')
+            matching_key = key
+            break
+
+    if matching_key is None:
+        vm.shutdown()
+        iotests.notrun('Did not find a key that fits 127.0.0.1')
+
     blockdev_create(vm, { 'driver': 'ssh',
                           'location': {
                               'path': disk_path,
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                               'host-key-check': {
                                   'mode': 'hash',
                                   'type': 'md5',
-                                  'hash': md5_key,
+                                  'hash': md5_keys[matching_key],
                               }
                           },
                           'size': 8388608 })
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
 
     iotests.img_info_log(remote_path)
 
-    sha1_key = subprocess.check_output(
-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-        'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-        shell=True).rstrip().decode('ascii')
-
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
                           'location': {
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                               'host-key-check': {
                                   'mode': 'hash',
                                   'type': 'sha1',
-                                  'hash': sha1_key,
+                                  'hash': sha1_keys[matching_key],
                               }
                           },
                           'size': 4194304 })
diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/207.out
+++ b/tests/qemu-iotests/207.out
@@ -XXX,XX +XXX,XX @@ virtual size: 4 MiB (4194304 bytes)
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
-Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
+Job failed: failed to open remote file '/this/is/not/an/existing/path': SFTP server: No such file (libssh error code: 1, sftp error code: 2)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-- 
2.21.0