Series comparison

-[PULL 00/19] Block patches
+[Qemu-devel] [PULL v2 0/8] Block patches
-The following changes since commit 96662996eda78c48aadddd4e76d8615c7eb72d80:
+The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:
-  Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20210513a' into staging (2021-05-14 12:03:47 +0100)
+  Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)
 are available in the Git repository at:
-  https://github.com/XanClic/qemu.git tags/pull-block-2021-05-14
+  https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24
-for you to fetch changes up to c61ebf362d0abf288ce266845519d5a550a1d89f:
+for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:
-  write-threshold: deal with includes (2021-05-14 16:14:10 +0200)
+  iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)
 ----------------------------------------------------------------
 Block patches:
-- drop block/io write notifiers
+- The SSH block driver now uses libssh instead of libssh2
-- qemu-iotests enhancements to make debugging easier
+- The VMDK block driver gets read-only support for the seSparse
-- rbd parsing fix
+  subformat
-- HMP qemu-io fix (for iothreads)
+- Various fixes
-- mirror job cancel relaxation (do not cancel in-flight requests when a
-  READY mirror job is canceled with force=false)
+---
-- document qcow2's data_file and data_file_raw features
-- fix iotest 297 for pylint 2.8
+v2:
-- block/copy-on-read refactoring
+- Squashed Pino's fix for pre-0.8 libssh into the libssh patch
 ----------------------------------------------------------------
-Connor Kuehl (3):
+Anton Nefedov (1):
-  iotests/231: Update expected deprecation message
+  iotest 134: test cluster-misaligned encrypted write
   block/rbd: Add an escape-aware strchr helper
   Document qemu-img options data_file and data_file_raw
-Emanuele Giuseppe Esposito (1):
+Klaus Birkelund Jensen (1):
-  qemu-iotests: fix pylint 2.8 consider-using-with error
+  nvme: do not advertise support for unsupported arbitration mechanism
-Paolo Bonzini (5):
+Max Reitz (1):
-  qemu-iotests: do not buffer the test output
+  iotests: Fix 205 for concurrent runs
   qemu-iotests: allow passing unittest.main arguments to the test
     scripts
   qemu-iotests: move command line and environment handling from
     TestRunner to TestEnv
   qemu-iotests: let "check" spawn an arbitrary test command
   qemu-iotests: fix case of SOCK_DIR already in the environment
-Vladimir Sementsov-Ogievskiy (10):
+Pino Toscano (1):
-  monitor: hmp_qemu_io: acquire aio contex, fix crash
+  ssh: switch from libssh2 to libssh
   mirror: stop cancelling in-flight requests on non-force cancel in
     READY
   block/copy-on-read: use bdrv_drop_filter() and drop s->active
   block/write-threshold: don't use write notifiers
   block: drop write notifiers
   test-write-threshold: rewrite test_threshold_(not_)trigger tests
   block/write-threshold: drop extra APIs
   test-write-threshold: drop extra tests
   test-write-threshold: drop extra TestStruct structure
   write-threshold: deal with includes
- docs/tools/qemu-img.rst           |  31 +++++++
+Sam Eiderman (3):
- include/block/block_int.h         |  15 +---
+  vmdk: Fix comment regarding max l1_size coverage
- include/block/write-threshold.h   |  27 ++----
+  vmdk: Reduce the max bound for L1 table size
- include/qemu/job.h                |   2 +-
+  vmdk: Add read-only support for seSparse snapshots
- block.c                           |   1 -
- block/backup.c                    |   2 +-
+Vladimir Sementsov-Ogievskiy (1):
- block/copy-on-read.c              |  33 +------
+  blockdev: enable non-root nodes for transaction drive-backup source
- block/io.c                        |  11 +--
- block/mirror.c                    |   6 +-
+ configure                                     |  65 +-
- block/monitor/block-hmp-cmds.c    |  31 ++++---
+ block/Makefile.objs                           |   6 +-
- block/rbd.c                       |  32 ++++---
+ block/ssh.c                                   | 652 ++++++++++--------
- block/write-threshold.c           |  91 ++++---------------
+ block/vmdk.c                                  | 372 +++++++++-
- job.c                             |   2 +-
+ blockdev.c                                    |   2 +-
- qemu-io-cmds.c                    |   8 +-
+ hw/block/nvme.c                               |   1 -
- qemu-io.c                         |  17 +++-
+ .travis.yml                                   |   4 +-
- tests/unit/test-write-threshold.c |  90 ++-----------------
+ block/trace-events                            |  14 +-
- tests/qemu-iotests/231            |   4 +
+ docs/qemu-block-drivers.texi                  |   2 +-
- tests/qemu-iotests/231.out        |   7 +-
+ .../dockerfiles/debian-win32-cross.docker     |   1 -
- tests/qemu-iotests/240.out        |   8 +-
+ .../dockerfiles/debian-win64-cross.docker     |   1 -
- tests/qemu-iotests/245.out        |   8 +-
+ tests/docker/dockerfiles/fedora.docker        |   4 +-
- tests/qemu-iotests/264            |   2 +-
+ tests/docker/dockerfiles/ubuntu.docker        |   2 +-
- tests/qemu-iotests/295.out        |   6 +-
+ tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
- tests/qemu-iotests/296.out        |   8 +-
+ tests/qemu-iotests/059.out                    |   2 +-
- tests/qemu-iotests/check          |  19 +++-
+ tests/qemu-iotests/134                        |   9 +
- tests/qemu-iotests/iotests.py     | 145 +++++++++++++++++-------------
+ tests/qemu-iotests/134.out                    |  10 +
- tests/qemu-iotests/pylintrc       |   3 +
+ tests/qemu-iotests/205                        |   2 +-
- tests/qemu-iotests/testenv.py     |  22 ++++-
+ tests/qemu-iotests/207                        |  54 +-
- tests/qemu-iotests/testrunner.py  |  37 +++-----
+ tests/qemu-iotests/207.out                    |   2 +-
-files changed, 289 insertions(+), 379 deletions(-)
+files changed, 823 insertions(+), 384 deletions(-)
 --
-.31.1
+.21.0

-[PULL 01/19] iotests/231: Update expected deprecation message
+Deleted patch
-From: Connor Kuehl <ckuehl@redhat.com>
-The deprecation message in the expected output has technically been
-wrong since the wrong version of a patch was applied to it. Because of
-this, the test fails. Correct the expected output so that it passes.
-Signed-off-by: Connor Kuehl <ckuehl@redhat.com>
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
-Message-Id: <20210421212343.85524-2-ckuehl@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/231.out | 4 +---
-file changed, 1 insertion(+), 3 deletions(-)
-diff --git a/tests/qemu-iotests/231.out b/tests/qemu-iotests/231.out
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/231.out
-+++ b/tests/qemu-iotests/231.out
-@@ -XXX,XX +XXX,XX @@
- QA output created by 231
--qemu-img: RBD options encoded in the filename as keyvalue pairs is deprecated.  Future versions may cease to parse these options in the future.
-+qemu-img: warning: RBD options encoded in the filename as keyvalue pairs is deprecated
- unable to get monitor info from DNS SRV with service name: ceph-mon
--no monitors specified to connect to.
- qemu-img: Could not open 'json:{'file.driver':'rbd','file.filename':'rbd:rbd/bogus:conf=BOGUS_CONF'}': error connecting: No such file or directory
- unable to get monitor info from DNS SRV with service name: ceph-mon
--no monitors specified to connect to.
- qemu-img: Could not open 'json:{'file.driver':'rbd','file.pool':'rbd','file.image':'bogus','file.conf':'BOGUS_CONF'}': error connecting: No such file or directory
- *** done
---
-.31.1

-[PULL 19/19] write-threshold: deal with includes
+[Qemu-devel] [PULL v2 1/8] nvme: do not advertise support for unsupported arbitration mechanism
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Klaus Birkelund Jensen <klaus@birkelund.eu>
-"qemu/typedefs.h" is enough for include/block/write-threshold.h header
+The device mistakenly reports that the Weighted Round Robin with Urgent
-with forward declaration of BlockDriverState. Also drop extra includes
+Priority Class arbitration mechanism is supported.
 from block/write-threshold.c and tests/unit/test-write-threshold.c
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+It is not.
-Message-Id: <20210506090621.11848-9-vsementsov@virtuozzo.com>
-Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
+Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
 Message-id: 20190606092530.14206-1-klaus@birkelund.eu
 Acked-by: Maxim Levitsky <mlevitsk@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- include/block/write-threshold.h   | 2 +-
+ hw/block/nvme.c | 1 -
- block/write-threshold.c           | 2 --
+file changed, 1 deletion(-)
  tests/unit/test-write-threshold.c | 1 -
 files changed, 1 insertion(+), 4 deletions(-)
-diff --git a/include/block/write-threshold.h b/include/block/write-threshold.h
+diff --git a/hw/block/nvme.c b/hw/block/nvme.c
 index XXXXXXX..XXXXXXX 100644
---- a/include/block/write-threshold.h
+--- a/hw/block/nvme.c
-+++ b/include/block/write-threshold.h
++++ b/hw/block/nvme.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
- #ifndef BLOCK_WRITE_THRESHOLD_H
+     n->bar.cap = 0;
- #define BLOCK_WRITE_THRESHOLD_H
+     NVME_CAP_SET_MQES(n->bar.cap, 0x7ff);
+     NVME_CAP_SET_CQR(n->bar.cap, 1);
--#include "block/block_int.h"
+-    NVME_CAP_SET_AMS(n->bar.cap, 1);
-+#include "qemu/typedefs.h"
+     NVME_CAP_SET_TO(n->bar.cap, 0xf);
+     NVME_CAP_SET_CSS(n->bar.cap, 1);
- /*
+     NVME_CAP_SET_MPSMAX(n->bar.cap, 4);
   * bdrv_write_threshold_set:
 diff --git a/block/write-threshold.c b/block/write-threshold.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/write-threshold.c
 +++ b/block/write-threshold.c
@@ -XXX,XX +XXX,XX @@
  #include "qemu/osdep.h"
  #include "block/block_int.h"
 -#include "qemu/coroutine.h"
  #include "block/write-threshold.h"
 -#include "qemu/notify.h"
  #include "qapi/error.h"
  #include "qapi/qapi-commands-block-core.h"
  #include "qapi/qapi-events-block-core.h"
 diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/unit/test-write-threshold.c
 +++ b/tests/unit/test-write-threshold.c
@@ -XXX,XX +XXX,XX @@
   */
  #include "qemu/osdep.h"
 -#include "qapi/error.h"
  #include "block/block_int.h"
  #include "block/write-threshold.h"
 --
-.31.1
+.21.0

-[PULL 18/19] test-write-threshold: drop extra TestStruct structure
+[Qemu-devel] [PULL v2 2/8] blockdev: enable non-root nodes for transaction drive-backup source
 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-We don't need this extra logic: it doesn't make code simpler.
+We forget to enable it for transaction .prepare, while it is already
 enabled in do_drive_backup since commit a2d665c1bc362
     "blockdev: loosen restrictions on drive-backup source node"
 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Max Reitz <mreitz@redhat.com>
+Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com
-Message-Id: <20210506090621.11848-8-vsementsov@virtuozzo.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
 Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/unit/test-write-threshold.c | 20 +++-----------------
+ blockdev.c | 2 +-
-file changed, 3 insertions(+), 17 deletions(-)
+file changed, 1 insertion(+), 1 deletion(-)
-diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
+diff --git a/blockdev.c b/blockdev.c
 index XXXXXXX..XXXXXXX 100644
---- a/tests/unit/test-write-threshold.c
+--- a/blockdev.c
-+++ b/tests/unit/test-write-threshold.c
++++ b/blockdev.c
-@@ -XXX,XX +XXX,XX @@ static void test_threshold_trigger(void)
+@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
-     g_assert_cmpuint(bdrv_write_threshold_get(&bs), ==, 0);
+     assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);
- }
+     backup = common->action->u.drive_backup.data;
--typedef struct TestStruct {
+-    bs = qmp_get_root_bs(backup->device, errp);
--    const char *name;
++    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
--    void (*func)(void);
+     if (!bs) {
--} TestStruct;
+         return;
--
+     }
  int main(int argc, char **argv)
  {
 -    size_t i;
 -    TestStruct tests[] = {
 -        { "/write-threshold/not-trigger",
 -          test_threshold_not_trigger },
 -        { "/write-threshold/trigger",
 -          test_threshold_trigger },
 -        { NULL, NULL }
 -    };
 -
      g_test_init(&argc, &argv, NULL);
 -    for (i = 0; tests[i].name != NULL; i++) {
 -        g_test_add_func(tests[i].name, tests[i].func);
 -    }
 +    g_test_add_func("/write-threshold/not-trigger", test_threshold_not_trigger);
 +    g_test_add_func("/write-threshold/trigger", test_threshold_trigger);
 +
      return g_test_run();
  }
 --
-.31.1
+.21.0

-[PULL 17/19] test-write-threshold: drop extra tests
+[Qemu-devel] [PULL v2 3/8] iotest 134: test cluster-misaligned encrypted write
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Anton Nefedov <anton.nefedov@virtuozzo.com>
-Testing set/get of one 64bit variable doesn't seem necessary. We have a
+COW (even empty/zero) areas require encryption too
 lot of such variables. Also remaining tests do test set/get anyway.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
 Reviewed-by: Eric Blake <eblake@redhat.com>
 Reviewed-by: Max Reitz <mreitz@redhat.com>
-Message-Id: <20210506090621.11848-7-vsementsov@virtuozzo.com>
+Reviewed-by: Alberto Garcia <berto@igalia.com>
-Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
+Message-id: 20190516143028.81155-1-anton.nefedov@virtuozzo.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/unit/test-write-threshold.c | 43 -------------------------------
+ tests/qemu-iotests/134     |  9 +++++++++
-file changed, 43 deletions(-)
+ tests/qemu-iotests/134.out | 10 ++++++++++
 files changed, 19 insertions(+)
-diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
+diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/134
 +++ b/tests/qemu-iotests/134
@@ -XXX,XX +XXX,XX @@ echo
  echo "== reading whole image =="
  $QEMU_IO --object $SECRET -c "read 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 +echo
 +echo "== rewriting cluster part =="
 +$QEMU_IO --object $SECRET -c "write -P 0xb 512 512" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 +
 +echo
 +echo "== verify pattern =="
 +$QEMU_IO --object $SECRET -c "read -P 0 0 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 +$QEMU_IO --object $SECRET -c "read -P 0xb 512 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 +
  echo
  echo "== rewriting whole image =="
  $QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/unit/test-write-threshold.c
+--- a/tests/qemu-iotests/134.out
-+++ b/tests/unit/test-write-threshold.c
++++ b/tests/qemu-iotests/134.out
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.
- #include "block/write-threshold.h"
+ read 134217728/134217728 bytes at offset 0
+MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
--static void test_threshold_not_set_on_init(void)
++== rewriting cluster part ==
--{
++wrote 512/512 bytes at offset 512
--    uint64_t res;
++512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
--    BlockDriverState bs;
++
--    memset(&bs, 0, sizeof(bs));
++== verify pattern ==
--
++read 512/512 bytes at offset 0
--    res = bdrv_write_threshold_get(&bs);
++512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
--    g_assert_cmpint(res, ==, 0);
++read 512/512 bytes at offset 512
--}
++512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
--
++
--static void test_threshold_set_get(void)
+ == rewriting whole image ==
--{
+ wrote 134217728/134217728 bytes at offset 0
--    uint64_t threshold = 4 * 1024 * 1024;
+MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 -    uint64_t res;
 -    BlockDriverState bs;
 -    memset(&bs, 0, sizeof(bs));
 -
 -    bdrv_write_threshold_set(&bs, threshold);
 -
 -    res = bdrv_write_threshold_get(&bs);
 -    g_assert_cmpint(res, ==, threshold);
 -}
 -
 -static void test_threshold_multi_set_get(void)
 -{
 -    uint64_t threshold1 = 4 * 1024 * 1024;
 -    uint64_t threshold2 = 15 * 1024 * 1024;
 -    uint64_t res;
 -    BlockDriverState bs;
 -    memset(&bs, 0, sizeof(bs));
 -
 -    bdrv_write_threshold_set(&bs, threshold1);
 -    bdrv_write_threshold_set(&bs, threshold2);
 -    res = bdrv_write_threshold_get(&bs);
 -    g_assert_cmpint(res, ==, threshold2);
 -}
 -
  static void test_threshold_not_trigger(void)
  {
      uint64_t threshold = 4 * 1024 * 1024;
@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
  {
      size_t i;
      TestStruct tests[] = {
 -        { "/write-threshold/not-set-on-init",
 -          test_threshold_not_set_on_init },
 -        { "/write-threshold/set-get",
 -          test_threshold_set_get },
 -        { "/write-threshold/multi-set-get",
 -          test_threshold_multi_set_get },
          { "/write-threshold/not-trigger",
            test_threshold_not_trigger },
          { "/write-threshold/trigger",
 --
-.31.1
+.21.0

-[PULL 16/19] block/write-threshold: drop extra APIs
+[Qemu-devel] [PULL v2 4/8] vmdk: Fix comment regarding max l1_size coverage
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-bdrv_write_threshold_exceeded() is unused.
+Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")
 extended the l1_size check from VMDK4 to VMDK3 but did not update the
 default coverage in the moved comment.
-bdrv_write_threshold_is_set() is used only to double check the value of
+The previous vmdk4 calculation:
 bs->write_threshold_offset in tests. No real sense in it (both tests do
 check real value with help of bdrv_write_threshold_get())
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+    (512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB
 The added vmdk3 calculation:
     (512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB
 Adding the calculation of vmdk3 to the comment.
 In any case, VMware does not offer virtual disks more than 2TB for
 vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is
 not implemented yet in qemu.
 Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
 Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
 Reviewed-by: Liran Alon <liran.alon@oracle.com>
 Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
 Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
 Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com
 Reviewed-by: yuchenlin <yuchenlin@synology.com>
 Reviewed-by: Max Reitz <mreitz@redhat.com>
-Message-Id: <20210506090621.11848-5-vsementsov@virtuozzo.com>
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-[mreitz: Adjusted commit message as per Eric's suggestion]
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- include/block/write-threshold.h   | 24 ------------------------
+ block/vmdk.c | 11 ++++++++---
- block/write-threshold.c           | 19 -------------------
+file changed, 8 insertions(+), 3 deletions(-)
  tests/unit/test-write-threshold.c |  4 ----
 files changed, 47 deletions(-)
-diff --git a/include/block/write-threshold.h b/include/block/write-threshold.h
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/include/block/write-threshold.h
+--- a/block/vmdk.c
-+++ b/include/block/write-threshold.h
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ void bdrv_write_threshold_set(BlockDriverState *bs, uint64_t threshold_bytes);
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-  */
+         return -EFBIG;
- uint64_t bdrv_write_threshold_get(const BlockDriverState *bs);
+     }
+     if (l1_size > 512 * 1024 * 1024) {
--/*
+-        /* Although with big capacity and small l1_entry_sectors, we can get a
-- * bdrv_write_threshold_is_set
++        /*
-- *
++         * Although with big capacity and small l1_entry_sectors, we can get a
-- * Tell if a write threshold is set for a given BDS.
+          * big l1_size, we don't want unbounded value to allocate the table.
-- */
+-         * Limit it to 512M, which is 16PB for default cluster and L2 table
--bool bdrv_write_threshold_is_set(const BlockDriverState *bs);
+-         * size */
--
++         * Limit it to 512M, which is:
--/*
++         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
-- * bdrv_write_threshold_exceeded
++         *            cluster size: 64KB, L2 table size: 512 entries
-- *
++         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
-- * Return the extent of a write request that exceeded the threshold,
++         *            cluster size: 512B, L2 table size: 4096 entries
-- * or zero if the request is below the threshold.
++         */
-- * Return zero also if the threshold was not set.
+         error_setg(errp, "L1 size too big");
-- *
+         return -EFBIG;
-- * NOTE: here we assume the following holds for each request this code
+     }
 - * deals with:
 - *
 - * assert((req->offset + req->bytes) <= UINT64_MAX)
 - *
 - * Please not there is *not* an actual C assert().
 - */
 -uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
 -                                       const BdrvTrackedRequest *req);
 -
  /*
   * bdrv_write_threshold_check_write
   *
 diff --git a/block/write-threshold.c b/block/write-threshold.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/write-threshold.c
 +++ b/block/write-threshold.c
@@ -XXX,XX +XXX,XX @@ uint64_t bdrv_write_threshold_get(const BlockDriverState *bs)
      return bs->write_threshold_offset;
  }
 -bool bdrv_write_threshold_is_set(const BlockDriverState *bs)
 -{
 -    return bs->write_threshold_offset > 0;
 -}
 -
 -uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
 -                                       const BdrvTrackedRequest *req)
 -{
 -    if (bdrv_write_threshold_is_set(bs)) {
 -        if (req->offset > bs->write_threshold_offset) {
 -            return (req->offset - bs->write_threshold_offset) + req->bytes;
 -        }
 -        if ((req->offset + req->bytes) > bs->write_threshold_offset) {
 -            return (req->offset + req->bytes) - bs->write_threshold_offset;
 -        }
 -    }
 -    return 0;
 -}
 -
  void bdrv_write_threshold_set(BlockDriverState *bs, uint64_t threshold_bytes)
  {
      bs->write_threshold_offset = threshold_bytes;
 diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/unit/test-write-threshold.c
 +++ b/tests/unit/test-write-threshold.c
@@ -XXX,XX +XXX,XX @@ static void test_threshold_not_set_on_init(void)
      BlockDriverState bs;
      memset(&bs, 0, sizeof(bs));
 -    g_assert(!bdrv_write_threshold_is_set(&bs));
 -
      res = bdrv_write_threshold_get(&bs);
      g_assert_cmpint(res, ==, 0);
  }
@@ -XXX,XX +XXX,XX @@ static void test_threshold_set_get(void)
      bdrv_write_threshold_set(&bs, threshold);
 -    g_assert(bdrv_write_threshold_is_set(&bs));
 -
      res = bdrv_write_threshold_get(&bs);
      g_assert_cmpint(res, ==, threshold);
  }
 --
-.31.1
+.21.0

-[PULL 02/19] block/rbd: Add an escape-aware strchr helper
+[Qemu-devel] [PULL v2 5/8] vmdk: Reduce the max bound for L1 table size
-From: Connor Kuehl <ckuehl@redhat.com>
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-Sometimes the parser needs to further split a token it has collected
+M of L1 entries is a very loose bound, only 32M are required to store
-from the token input stream. Right now, it does a cursory check to see
+the maximal supported VMDK file size of 2TB.
 if the relevant characters appear in the token to determine if it should
 break it down further.
-However, qemu_rbd_next_tok() will escape characters as it removes tokens
+Fixed qemu-iotest 59# - now failure occures before on impossible L1
-from the token stream and plain strchr() won't. This can make the
+table size.
 initial strchr() check slightly misleading since it implies
 qemu_rbd_next_tok() will find the token and split on it, except the
 reality is that qemu_rbd_next_tok() will pass over it if it is escaped.
-Use a custom strchr to avoid mixing escaped and unescaped string
+Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
-operations. Furthermore, this code is identical to how
+Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
-qemu_rbd_next_tok() seeks its next token, so incorporate this custom
+Reviewed-by: Liran Alon <liran.alon@oracle.com>
-strchr into the body of that function to reduce duplication.
+Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
+Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
-Reported-by: Han Han <hhan@redhat.com>
+Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com
-Fixes: https://bugzilla.redhat.com/1873913
+Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Connor Kuehl <ckuehl@redhat.com>
 Message-Id: <20210421212343.85524-3-ckuehl@redhat.com>
 Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/rbd.c                | 32 +++++++++++++++++++++-----------
+ block/vmdk.c               | 13 +++++++------
- tests/qemu-iotests/231     |  4 ++++
+ tests/qemu-iotests/059.out |  2 +-
- tests/qemu-iotests/231.out |  3 +++
+files changed, 8 insertions(+), 7 deletions(-)
 files changed, 28 insertions(+), 11 deletions(-)
-diff --git a/block/rbd.c b/block/rbd.c
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/rbd.c
+--- a/block/vmdk.c
-+++ b/block/rbd.c
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-                             const char *keypairs, const char *secretid,
+         error_setg(errp, "Invalid granularity, image may be corrupt");
-                             Error **errp);
+         return -EFBIG;
 +static char *qemu_rbd_strchr(char *src, char delim)
 +{
 +    char *p;
 +
 +    for (p = src; *p; ++p) {
 +        if (*p == delim) {
 +            return p;
 +        }
 +        if (*p == '\\' && p[1] != '\0') {
 +            ++p;
 +        }
 +    }
 +
 +    return NULL;
 +}
 +
 +
  static char *qemu_rbd_next_tok(char *src, char delim, char **p)
  {
      char *end;
      *p = NULL;
 -    for (end = src; *end; ++end) {
 -        if (*end == delim) {
 -            break;
 -        }
 -        if (*end == '\\' && end[1] != '\0') {
 -            end++;
 -        }
 -    }
 -    if (*end == delim) {
 +    end = qemu_rbd_strchr(src, delim);
 +    if (end) {
          *p = end + 1;
          *end = '\0';
      }
-@@ -XXX,XX +XXX,XX @@ static void qemu_rbd_parse_filename(const char *filename, QDict *options,
+-    if (l1_size > 512 * 1024 * 1024) {
-     qemu_rbd_unescape(found_str);
++    if (l1_size > 32 * 1024 * 1024) {
-     qdict_put_str(options, "pool", found_str);
+         /*
+          * Although with big capacity and small l1_entry_sectors, we can get a
--    if (strchr(p, '@')) {
+          * big l1_size, we don't want unbounded value to allocate the table.
-+    if (qemu_rbd_strchr(p, '@')) {
+-         * Limit it to 512M, which is:
-         image_name = qemu_rbd_next_tok(p, '@', &p);
+-         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
+-         *            cluster size: 64KB, L2 table size: 512 entries
-         found_str = qemu_rbd_next_tok(p, ':', &p);
+-         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
-@@ -XXX,XX +XXX,XX @@ static void qemu_rbd_parse_filename(const char *filename, QDict *options,
+-         *            cluster size: 512B, L2 table size: 4096 entries
-         image_name = qemu_rbd_next_tok(p, ':', &p);
++         * Limit it to 32M, which is enough to store:
-     }
++         *     8TB  - for both VMDK3 & VMDK4 with
-     /* Check for namespace in the image_name */
++         *            minimal cluster size: 512B
--    if (strchr(image_name, '/')) {
++         *            minimal L2 table size: 512 entries
-+    if (qemu_rbd_strchr(image_name, '/')) {
++         *            8 TB is still more than the maximal value supported for
-         found_str = qemu_rbd_next_tok(image_name, '/', &image_name);
++         *            VMDK3 & VMDK4 which is 2TB.
-         qemu_rbd_unescape(found_str);
+          */
-         qdict_put_str(options, "namespace", found_str);
+         error_setg(errp, "L1 size too big");
-diff --git a/tests/qemu-iotests/231 b/tests/qemu-iotests/231
+         return -EFBIG;
-index XXXXXXX..XXXXXXX 100755
+diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
 --- a/tests/qemu-iotests/231
 +++ b/tests/qemu-iotests/231
@@ -XXX,XX +XXX,XX @@ _filter_conf()
  $QEMU_IMG info "json:{'file.driver':'rbd','file.filename':'rbd:rbd/bogus:conf=${BOGUS_CONF}'}" 2>&1 | _filter_conf
  $QEMU_IMG info "json:{'file.driver':'rbd','file.pool':'rbd','file.image':'bogus','file.conf':'${BOGUS_CONF}'}" 2>&1 | _filter_conf
 +# Regression test: the qemu-img invocation is expected to fail, but it should
 +# not seg fault the parser.
 +$QEMU_IMG create "rbd:rbd/aa\/bb:conf=${BOGUS_CONF}" 1M 2>&1 | _filter_conf
 +
  # success, all done
  echo "*** done"
  rm -f $seq.full
 diff --git a/tests/qemu-iotests/231.out b/tests/qemu-iotests/231.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/231.out
+--- a/tests/qemu-iotests/059.out
-+++ b/tests/qemu-iotests/231.out
++++ b/tests/qemu-iotests/059.out
-@@ -XXX,XX +XXX,XX @@ unable to get monitor info from DNS SRV with service name: ceph-mon
+@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
- qemu-img: Could not open 'json:{'file.driver':'rbd','file.filename':'rbd:rbd/bogus:conf=BOGUS_CONF'}': error connecting: No such file or directory
+x140000000     0x10000         0x50000         TEST_DIR/t-s003.vmdk
- unable to get monitor info from DNS SRV with service name: ceph-mon
- qemu-img: Could not open 'json:{'file.driver':'rbd','file.pool':'rbd','file.image':'bogus','file.conf':'BOGUS_CONF'}': error connecting: No such file or directory
+ === Testing afl image with a very large capacity ===
-+Formatting 'rbd:rbd/aa\/bb:conf=BOGUS_CONF', fmt=raw size=1048576
+-qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large
-+unable to get monitor info from DNS SRV with service name: ceph-mon
++qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big
 +qemu-img: rbd:rbd/aa\/bb:conf=BOGUS_CONF: error connecting: No such file or directory
  *** done
 --
-.31.1
+.21.0

-[PULL 03/19] monitor: hmp_qemu_io: acquire aio contex, fix crash
+[Qemu-devel] [PULL v2 6/8] vmdk: Add read-only support for seSparse snapshots
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-Max reported the following bug:
+Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in
+QEMU).
-$ ./qemu-img create -f raw src.img 1G
-$ ./qemu-img create -f raw dst.img 1G
+This format was lacking in the following:
-$ (echo '
+    * Grain directory (L1) and grain table (L2) entries were 32-bit,
-   {"execute":"qmp_capabilities"}
+      allowing access to only 2TB (slightly less) of data.
-   {"execute":"blockdev-mirror",
+    * The grain size (default) was 512 bytes - leading to data
-    "arguments":{"job-id":"mirror",
+      fragmentation and many grain tables.
-                 "device":"source",
+    * For space reclamation purposes, it was necessary to find all the
-                 "target":"target",
+      grains which are not pointed to by any grain table - so a reverse
-                 "sync":"full",
+      mapping of "offset of grain in vmdk" to "grain table" must be
-                 "filter-node-name":"mirror-top"}}
+      constructed - which takes large amounts of CPU/RAM.
-'; sleep 3; echo '
-   {"execute":"human-monitor-command",
+The format specification can be found in VMware's documentation:
-    "arguments":{"command-line":
+https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf
-                 "qemu-io mirror-top \"write 0 1G\""}}') \
-| x86_64-softmmu/qemu-system-x86_64 \
+In ESXi 6.5, to support snapshot files larger than 2TB, a new format was
-   -qmp stdio \
+introduced: SESparse (Space Efficient).
-   -blockdev file,node-name=source,filename=src.img \
-   -blockdev file,node-name=target,filename=dst.img \
+This format fixes the above issues:
-   -object iothread,id=iothr0 \
-   -device virtio-blk,drive=source,iothread=iothr0
+    * All entries are now 64-bit.
+    * The grain size (default) is 4KB.
-crashes:
+    * Grain directory and grain tables are now located at the beginning
+      of the file.
-raise () at /usr/lib/libc.so.6
+      + seSparse format reserves space for all grain tables.
-abort () at /usr/lib/libc.so.6
+      + Grain tables can be addressed using an index.
-error_exit
+      + Grains are located in the end of the file and can also be
-   (err=<optimized out>,
+        addressed with an index.
-   msg=msg@entry=0x55fbb1634790 <__func__.27> "qemu_mutex_unlock_impl")
+      - seSparse vmdks of large disks (64TB) have huge preallocated
-   at ../util/qemu-thread-posix.c:37
+        headers - mainly due to L2 tables, even for empty snapshots.
-qemu_mutex_unlock_impl
+    * The header contains a reverse mapping ("backmap") of "offset of
-   (mutex=mutex@entry=0x55fbb25ab6e0,
+      grain in vmdk" to "grain table" and a bitmap ("free bitmap") which
-   file=file@entry=0x55fbb1636957 "../util/async.c",
+      specifies for each grain - whether it is allocated or not.
-   line=line@entry=650)
+      Using these data structures we can implement space reclamation
-   at ../util/qemu-thread-posix.c:109
+      efficiently.
-aio_context_release (ctx=ctx@entry=0x55fbb25ab680) at ../util/async.c:650
+    * Due to the fact that the header now maintains two mappings:
-bdrv_do_drained_begin
+        * The regular one (grain directory & grain tables)
-   (bs=bs@entry=0x55fbb3a87000, recursive=recursive@entry=false,
+        * A reverse one (backmap and free bitmap)
-   parent=parent@entry=0x0,
+      These data structures can lose consistency upon crash and result
-   ignore_bds_parents=ignore_bds_parents@entry=false,
+      in a corrupted VMDK.
-   poll=poll@entry=true) at ../block/io.c:441
+      Therefore, a journal is also added to the VMDK and is replayed
-bdrv_do_drained_begin
+      when the VMware reopens the file after a crash.
-   (poll=true, ignore_bds_parents=false, parent=0x0, recursive=false,
-   bs=0x55fbb3a87000) at ../block/io.c:448
+Since ESXi 6.7 - SESparse is the only snapshot format available.
-blk_drain (blk=0x55fbb26c5a00) at ../block/block-backend.c:1718
-blk_unref (blk=0x55fbb26c5a00) at ../block/block-backend.c:498
+Unfortunately, VMware does not provide documentation regarding the new
-blk_unref (blk=0x55fbb26c5a00) at ../block/block-backend.c:491
+seSparse format.
-hmp_qemu_io (mon=0x7fffaf3fc7d0, qdict=<optimized out>)
-   at ../block/monitor/block-hmp-cmds.c:628
+This commit is based on black-box research of the seSparse format.
+Various in-guest block operations and their effect on the snapshot file
-man pthread_mutex_unlock
+were tested.
-...
-    EPERM  The  mutex type is PTHREAD_MUTEX_ERRORCHECK or
+The only VMware provided source of information (regarding the underlying
-    PTHREAD_MUTEX_RECURSIVE, or the mutex is a robust mutex, and the
+implementation) was a log file on the ESXi:
-    current thread does not own the mutex.
+    /var/log/hostd.log
-So, thread doesn't own the mutex. And we have iothread here.
+Whenever an seSparse snapshot is created - the log is being populated
-Next, note that AIO_WAIT_WHILE() documents that ctx must be acquired
+with seSparse records.
-exactly once by caller. But where is it acquired in the call stack?
-Seems nowhere.
+Relevant log records are of the form:
-qemuio_command do acquire aio context.. But we need context acquired
+[...] Const Header:
-around blk_unref() as well and actually around blk_insert_bs() too.
+[...]  constMagic     = 0xcafebabe
+[...]  version        = 2.1
-Let's refactor qemuio_command so that it doesn't acquire aio context
+[...]  capacity       = 204800
-but callers do that instead. This way we can cleanly acquire aio
+[...]  grainSize      = 8
-context in hmp_qemu_io() around all three calls.
+[...]  grainTableSize = 64
+[...]  flags          = 0
-Reported-by: Max Reitz <mreitz@redhat.com>
+[...] Extents:
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+[...]  Header         : <1 : 1>
-Message-Id: <20210423134233.51495-1-vsementsov@virtuozzo.com>
+[...]  JournalHdr     : <2 : 2>
-[mreitz: Fixed comment]
+[...]  Journal        : <2048 : 2048>
 [...]  GrainDirectory : <4096 : 2048>
 [...]  GrainTables    : <6144 : 2048>
 [...]  FreeBitmap     : <8192 : 2048>
 [...]  BackMap        : <10240 : 2048>
 [...]  Grain          : <12288 : 204800>
 [...] Volatile Header:
 [...] volatileMagic     = 0xcafecafe
 [...] FreeGTNumber      = 0
 [...] nextTxnSeqNumber  = 0
 [...] replayJournal     = 0
 The sizes that are seen in the log file are in sectors.
 Extents are of the following format: <offset : size>
 This commit is a strict implementation which enforces:
     * magics
     * version number 2.1
     * grain size of 8 sectors  (4KB)
     * grain table size of 64 sectors
     * zero flags
     * extent locations
 Additionally, this commit proivdes only a subset of the functionality
 offered by seSparse's format:
     * Read-only
     * No journal replay
     * No space reclamation
     * No unmap support
 Hence, journal header, journal, free bitmap and backmap extents are
 unused, only the "classic" (L1 -> L2 -> data) grain access is
 implemented.
 However there are several differences in the grain access itself.
 Grain directory (L1):
     * Grain directory entries are indexes (not offsets) to grain
       tables.
     * Valid grain directory entries have their highest nibble set to
 x1.
     * Since grain tables are always located in the beginning of the
       file - the index can fit into 32 bits - so we can use its low
       part if it's valid.
 Grain table (L2):
     * Grain table entries are indexes (not offsets) to grains.
     * If the highest nibble of the entry is:
 x0:
             The grain in not allocated.
             The rest of the bytes are 0.
 x1:
             The grain is unmapped - guest sees a zero grain.
             The rest of the bits point to the previously mapped grain,
             see 0x3 case.
 x2:
             The grain is zero.
 x3:
             The grain is allocated - to get the index calculate:
             ((entry & 0x0fff000000000000) >> 48) |
             ((entry & 0x0000ffffffffffff) << 12)
     * The difference between 0x1 and 0x2 is that 0x1 is an unallocated
       grain which results from the guest using sg_unmap to unmap the
       grain - but the grain itself still exists in the grain extent - a
       space reclamation procedure should delete it.
       Unmapping a zero grain has no effect (0x2 will not change to 0x1)
       but unmapping an unallocated grain will (0x0 to 0x1) - naturally.
 In order to implement seSparse some fields had to be changed to support
 both 32-bit and 64-bit entry sizes.
 Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
 Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
 Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
 Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
 Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/monitor/block-hmp-cmds.c | 31 +++++++++++++++++++++----------
+ block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---
- qemu-io-cmds.c                 |  8 ++++----
+file changed, 342 insertions(+), 16 deletions(-)
- qemu-io.c                      | 17 +++++++++++++++--
-files changed, 40 insertions(+), 16 deletions(-)
+diff --git a/block/vmdk.c b/block/vmdk.c
 diff --git a/block/monitor/block-hmp-cmds.c b/block/monitor/block-hmp-cmds.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/monitor/block-hmp-cmds.c
+--- a/block/vmdk.c
-+++ b/block/monitor/block-hmp-cmds.c
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ void hmp_eject(Monitor *mon, const QDict *qdict)
+@@ -XXX,XX +XXX,XX @@ typedef struct {
+     uint16_t compressAlgorithm;
- void hmp_qemu_io(Monitor *mon, const QDict *qdict)
+ } QEMU_PACKED VMDK4Header;
- {
--    BlockBackend *blk;
++typedef struct VMDKSESparseConstHeader {
-+    BlockBackend *blk = NULL;
++    uint64_t magic;
-+    BlockDriverState *bs = NULL;
++    uint64_t version;
-     BlockBackend *local_blk = NULL;
++    uint64_t capacity;
-+    AioContext *ctx = NULL;
++    uint64_t grain_size;
-     bool qdev = qdict_get_try_bool(qdict, "qdev", false);
++    uint64_t grain_table_size;
-     const char *device = qdict_get_str(qdict, "device");
++    uint64_t flags;
-     const char *command = qdict_get_str(qdict, "command");
++    uint64_t reserved1;
-@@ -XXX,XX +XXX,XX @@ void hmp_qemu_io(Monitor *mon, const QDict *qdict)
++    uint64_t reserved2;
-     } else {
++    uint64_t reserved3;
-         blk = blk_by_name(device);
++    uint64_t reserved4;
-         if (!blk) {
++    uint64_t volatile_header_offset;
--            BlockDriverState *bs = bdrv_lookup_bs(NULL, device, &err);
++    uint64_t volatile_header_size;
--            if (bs) {
++    uint64_t journal_header_offset;
--                blk = local_blk = blk_new(bdrv_get_aio_context(bs),
++    uint64_t journal_header_size;
--                                          0, BLK_PERM_ALL);
++    uint64_t journal_offset;
--                ret = blk_insert_bs(blk, bs, &err);
++    uint64_t journal_size;
--                if (ret < 0) {
++    uint64_t grain_dir_offset;
--                    goto fail;
++    uint64_t grain_dir_size;
--                }
++    uint64_t grain_tables_offset;
--            } else {
++    uint64_t grain_tables_size;
-+            bs = bdrv_lookup_bs(NULL, device, &err);
++    uint64_t free_bitmap_offset;
-+            if (!bs) {
++    uint64_t free_bitmap_size;
-                 goto fail;
++    uint64_t backmap_offset;
-             }
++    uint64_t backmap_size;
-         }
++    uint64_t grains_offset;
-     }
++    uint64_t grains_size;
++    uint8_t pad[304];
-+    ctx = blk ? blk_get_aio_context(blk) : bdrv_get_aio_context(bs);
++} QEMU_PACKED VMDKSESparseConstHeader;
-+    aio_context_acquire(ctx);
++
-+
++typedef struct VMDKSESparseVolatileHeader {
-+    if (bs) {
++    uint64_t magic;
-+        blk = local_blk = blk_new(bdrv_get_aio_context(bs), 0, BLK_PERM_ALL);
++    uint64_t free_gt_number;
-+        ret = blk_insert_bs(blk, bs, &err);
++    uint64_t next_txn_seq_number;
-+        if (ret < 0) {
++    uint64_t replay_journal;
-+            goto fail;
++    uint8_t pad[480];
 +} QEMU_PACKED VMDKSESparseVolatileHeader;
 +
  #define L2_CACHE_SIZE 16
  typedef struct VmdkExtent {
@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {
      bool compressed;
      bool has_marker;
      bool has_zero_grain;
 +    bool sesparse;
 +    uint64_t sesparse_l2_tables_offset;
 +    uint64_t sesparse_clusters_offset;
 +    int32_t entry_size;
      int version;
      int64_t sectors;
      int64_t end_sector;
      int64_t flat_start_offset;
      int64_t l1_table_offset;
      int64_t l1_backup_table_offset;
 -    uint32_t *l1_table;
 +    void *l1_table;
      uint32_t *l1_backup_table;
      unsigned int l1_size;
      uint32_t l1_entry_sectors;
      unsigned int l2_size;
 -    uint32_t *l2_cache;
 +    void *l2_cache;
      uint32_t l2_cache_offsets[L2_CACHE_SIZE];
      uint32_t l2_cache_counts[L2_CACHE_SIZE];
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
           *            minimal L2 table size: 512 entries
           *            8 TB is still more than the maximal value supported for
           *            VMDK3 & VMDK4 which is 2TB.
 +         *     64TB - for "ESXi seSparse Extent"
 +         *            minimal cluster size: 512B (default is 4KB)
 +         *            L2 table size: 4096 entries (const).
 +         *            64TB is more than the maximal value supported for
 +         *            seSparse VMDKs (which is slightly less than 64TB)
           */
          error_setg(errp, "L1 size too big");
          return -EFBIG;
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
      extent->l2_size = l2_size;
      extent->cluster_sectors = flat ? sectors : cluster_sectors;
      extent->next_cluster_sector = ROUND_UP(nb_sectors, cluster_sectors);
 +    extent->entry_size = sizeof(uint32_t);
      if (s->num_extents > 1) {
          extent->end_sector = (*(extent - 1)).end_sector + extent->sectors;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
      int i;
      /* read the L1 table */
 -    l1_size = extent->l1_size * sizeof(uint32_t);
 +    l1_size = extent->l1_size * extent->entry_size;
      extent->l1_table = g_try_malloc(l1_size);
      if (l1_size && extent->l1_table == NULL) {
          return -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
          goto fail_l1;
      }
      for (i = 0; i < extent->l1_size; i++) {
 -        le32_to_cpus(&extent->l1_table[i]);
 +        if (extent->entry_size == sizeof(uint64_t)) {
 +            le64_to_cpus((uint64_t *)extent->l1_table + i);
 +        } else {
 +            assert(extent->entry_size == sizeof(uint32_t));
 +            le32_to_cpus((uint32_t *)extent->l1_table + i);
 +        }
-+    }
+     }
-+
-     /*
+     if (extent->l1_backup_table_offset) {
-      * Notably absent: Proper permission management. This is sad, but it seems
++        assert(!extent->sesparse);
-      * almost impossible to achieve without changing the semantics and thereby
+         extent->l1_backup_table = g_try_malloc(l1_size);
-@@ -XXX,XX +XXX,XX @@ void hmp_qemu_io(Monitor *mon, const QDict *qdict)
+         if (l1_size && extent->l1_backup_table == NULL) {
+             ret = -ENOMEM;
- fail:
+@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
-     blk_unref(local_blk);
+     }
-+
-+    if (ctx) {
+     extent->l2_cache =
-+        aio_context_release(ctx);
+-        g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE);
-+    }
++        g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE);
-+
+     return 0;
-     hmp_handle_error(mon, err);
+  fail_l1b:
      g_free(extent->l1_backup_table);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
      return ret;
  }
-diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
++#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe)
-index XXXXXXX..XXXXXXX 100644
++#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe)
---- a/qemu-io-cmds.c
++
-+++ b/qemu-io-cmds.c
++/* Strict checks - format not officially documented */
-@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t help_cmd = {
++static int check_se_sparse_const_header(VMDKSESparseConstHeader *header,
-     .oneline    = "help for one or all commands",
++                                        Error **errp)
- };
++{
++    header->magic = le64_to_cpu(header->magic);
-+/*
++    header->version = le64_to_cpu(header->version);
-+ * Called with aio context of blk acquired. Or with qemu_get_aio_context()
++    header->grain_size = le64_to_cpu(header->grain_size);
-+ * context acquired if blk is NULL.
++    header->grain_table_size = le64_to_cpu(header->grain_table_size);
-+ */
++    header->flags = le64_to_cpu(header->flags);
- int qemuio_command(BlockBackend *blk, const char *cmd)
++    header->reserved1 = le64_to_cpu(header->reserved1);
- {
++    header->reserved2 = le64_to_cpu(header->reserved2);
--    AioContext *ctx;
++    header->reserved3 = le64_to_cpu(header->reserved3);
-     char *input;
++    header->reserved4 = le64_to_cpu(header->reserved4);
-     const cmdinfo_t *ct;
++
-     char **v;
++    header->volatile_header_offset =
-@@ -XXX,XX +XXX,XX @@ int qemuio_command(BlockBackend *blk, const char *cmd)
++        le64_to_cpu(header->volatile_header_offset);
-     if (c) {
++    header->volatile_header_size = le64_to_cpu(header->volatile_header_size);
-         ct = find_command(v[0]);
++
-         if (ct) {
++    header->journal_header_offset = le64_to_cpu(header->journal_header_offset);
--            ctx = blk ? blk_get_aio_context(blk) : qemu_get_aio_context();
++    header->journal_header_size = le64_to_cpu(header->journal_header_size);
--            aio_context_acquire(ctx);
++
-             ret = command(blk, ct, c, v);
++    header->journal_offset = le64_to_cpu(header->journal_offset);
--            aio_context_release(ctx);
++    header->journal_size = le64_to_cpu(header->journal_size);
-         } else {
++
-             fprintf(stderr, "command \"%s\" not found\n", v[0]);
++    header->grain_dir_offset = le64_to_cpu(header->grain_dir_offset);
-             ret = -EINVAL;
++    header->grain_dir_size = le64_to_cpu(header->grain_dir_size);
-diff --git a/qemu-io.c b/qemu-io.c
++
-index XXXXXXX..XXXXXXX 100644
++    header->grain_tables_offset = le64_to_cpu(header->grain_tables_offset);
---- a/qemu-io.c
++    header->grain_tables_size = le64_to_cpu(header->grain_tables_size);
-+++ b/qemu-io.c
++
-@@ -XXX,XX +XXX,XX @@ static void prep_fetchline(void *opaque)
++    header->free_bitmap_offset = le64_to_cpu(header->free_bitmap_offset);
-     *fetchable= 1;
++    header->free_bitmap_size = le64_to_cpu(header->free_bitmap_size);
- }
++
++    header->backmap_offset = le64_to_cpu(header->backmap_offset);
-+static int do_qemuio_command(const char *cmd)
++    header->backmap_size = le64_to_cpu(header->backmap_size);
 +
 +    header->grains_offset = le64_to_cpu(header->grains_offset);
 +    header->grains_size = le64_to_cpu(header->grains_size);
 +
 +    if (header->magic != SESPARSE_CONST_HEADER_MAGIC) {
 +        error_setg(errp, "Bad const header magic: 0x%016" PRIx64,
 +                   header->magic);
 +        return -EINVAL;
 +    }
 +
 +    if (header->version != 0x0000000200000001) {
 +        error_setg(errp, "Unsupported version: 0x%016" PRIx64,
 +                   header->version);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->grain_size != 8) {
 +        error_setg(errp, "Unsupported grain size: %" PRIu64,
 +                   header->grain_size);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->grain_table_size != 64) {
 +        error_setg(errp, "Unsupported grain table size: %" PRIu64,
 +                   header->grain_table_size);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->flags != 0) {
 +        error_setg(errp, "Unsupported flags: 0x%016" PRIx64,
 +                   header->flags);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->reserved1 != 0 || header->reserved2 != 0 ||
 +        header->reserved3 != 0 || header->reserved4 != 0) {
 +        error_setg(errp, "Unsupported reserved bits:"
 +                   " 0x%016" PRIx64 " 0x%016" PRIx64
 +                   " 0x%016" PRIx64 " 0x%016" PRIx64,
 +                   header->reserved1, header->reserved2,
 +                   header->reserved3, header->reserved4);
 +        return -ENOTSUP;
 +    }
 +
 +    /* check that padding is 0 */
 +    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
 +        error_setg(errp, "Unsupported non-zero const header padding");
 +        return -ENOTSUP;
 +    }
 +
 +    return 0;
 +}
 +
 +static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *header,
 +                                           Error **errp)
 +{
 +    header->magic = le64_to_cpu(header->magic);
 +    header->free_gt_number = le64_to_cpu(header->free_gt_number);
 +    header->next_txn_seq_number = le64_to_cpu(header->next_txn_seq_number);
 +    header->replay_journal = le64_to_cpu(header->replay_journal);
 +
 +    if (header->magic != SESPARSE_VOLATILE_HEADER_MAGIC) {
 +        error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64,
 +                   header->magic);
 +        return -EINVAL;
 +    }
 +
 +    if (header->replay_journal) {
 +        error_setg(errp, "Image is dirty, Replaying journal not supported");
 +        return -ENOTSUP;
 +    }
 +
 +    /* check that padding is 0 */
 +    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
 +        error_setg(errp, "Unsupported non-zero volatile header padding");
 +        return -ENOTSUP;
 +    }
 +
 +    return 0;
 +}
 +
 +static int vmdk_open_se_sparse(BlockDriverState *bs,
 +                               BdrvChild *file,
 +                               int flags, Error **errp)
 +{
 +    int ret;
-+    AioContext *ctx =
++    VMDKSESparseConstHeader const_header;
-+        qemuio_blk ? blk_get_aio_context(qemuio_blk) : qemu_get_aio_context();
++    VMDKSESparseVolatileHeader volatile_header;
-+
++    VmdkExtent *extent;
-+    aio_context_acquire(ctx);
++
-+    ret = qemuio_command(qemuio_blk, cmd);
++    ret = bdrv_apply_auto_read_only(bs,
-+    aio_context_release(ctx);
++            "No write support for seSparse images available", errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    assert(sizeof(const_header) == SECTOR_SIZE);
 +
 +    ret = bdrv_pread(file, 0, &const_header, sizeof(const_header));
 +    if (ret < 0) {
 +        bdrv_refresh_filename(file->bs);
 +        error_setg_errno(errp, -ret,
 +                         "Could not read const header from file '%s'",
 +                         file->bs->filename);
 +        return ret;
 +    }
 +
 +    /* check const header */
 +    ret = check_se_sparse_const_header(&const_header, errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    assert(sizeof(volatile_header) == SECTOR_SIZE);
 +
 +    ret = bdrv_pread(file,
 +                     const_header.volatile_header_offset * SECTOR_SIZE,
 +                     &volatile_header, sizeof(volatile_header));
 +    if (ret < 0) {
 +        bdrv_refresh_filename(file->bs);
 +        error_setg_errno(errp, -ret,
 +                         "Could not read volatile header from file '%s'",
 +                         file->bs->filename);
 +        return ret;
 +    }
 +
 +    /* check volatile header */
 +    ret = check_se_sparse_volatile_header(&volatile_header, errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    ret = vmdk_add_extent(bs, file, false,
 +                          const_header.capacity,
 +                          const_header.grain_dir_offset * SECTOR_SIZE,
 +                          0,
 +                          const_header.grain_dir_size *
 +                          SECTOR_SIZE / sizeof(uint64_t),
 +                          const_header.grain_table_size *
 +                          SECTOR_SIZE / sizeof(uint64_t),
 +                          const_header.grain_size,
 +                          &extent,
 +                          errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    extent->sesparse = true;
 +    extent->sesparse_l2_tables_offset = const_header.grain_tables_offset;
 +    extent->sesparse_clusters_offset = const_header.grains_offset;
 +    extent->entry_size = sizeof(uint64_t);
 +
 +    ret = vmdk_init_tables(bs, extent, errp);
 +    if (ret) {
 +        /* free extent allocated by vmdk_add_extent */
 +        vmdk_free_last_extent(bs);
 +    }
 +
 +    return ret;
 +}
 +
- static int command_loop(void)
+ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
                                 QDict *options, Error **errp);
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
           * RW [size in sectors] SPARSE "file-name.vmdk"
           * RW [size in sectors] VMFS "file-name.vmdk"
           * RW [size in sectors] VMFSSPARSE "file-name.vmdk"
 +         * RW [size in sectors] SESPARSE "file-name.vmdk"
           */
          flat_offset = -1;
          matches = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          if (sectors <= 0 ||
              (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
 -             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) ||
 +             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") &&
 +             strcmp(type, "SESPARSE")) ||
              (strcmp(access, "RW"))) {
              continue;
          }
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                  return ret;
              }
              extent = &s->extents[s->num_extents - 1];
 +        } else if (!strcmp(type, "SESPARSE")) {
 +            ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
 +            if (ret) {
 +                bdrv_unref_child(bs, extent_file);
 +                return ret;
 +            }
 +            extent = &s->extents[s->num_extents - 1];
          } else {
              error_setg(errp, "Unsupported extent type '%s'", type);
              bdrv_unref_child(bs, extent_file);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
      if (strcmp(ct, "monolithicFlat") &&
          strcmp(ct, "vmfs") &&
          strcmp(ct, "vmfsSparse") &&
 +        strcmp(ct, "seSparse") &&
          strcmp(ct, "twoGbMaxExtentSparse") &&
          strcmp(ct, "twoGbMaxExtentFlat")) {
          error_setg(errp, "Unsupported image type '%s'", ct);
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
  {
-     int i, fetchable = 0, prompted = 0;
+     unsigned int l1_index, l2_offset, l2_index;
-@@ -XXX,XX +XXX,XX @@ static int command_loop(void)
+     int min_index, i, j;
-     char *input;
+-    uint32_t min_count, *l2_table;
++    uint32_t min_count;
-     for (i = 0; !quit_qemu_io && i < ncmdline; i++) {
++    void *l2_table;
--        ret = qemuio_command(qemuio_blk, cmdline[i]);
+     bool zeroed = false;
-+        ret = do_qemuio_command(cmdline[i]);
+     int64_t ret;
-         if (ret < 0) {
+     int64_t cluster_sector;
-             last_error = ret;
++    unsigned int l2_size_bytes = extent->l2_size * extent->entry_size;
      if (m_data) {
          m_data->valid = 0;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
      if (l1_index >= extent->l1_size) {
          return VMDK_ERROR;
      }
 -    l2_offset = extent->l1_table[l1_index];
 +    if (extent->sesparse) {
 +        uint64_t l2_offset_u64;
 +
 +        assert(extent->entry_size == sizeof(uint64_t));
 +
 +        l2_offset_u64 = ((uint64_t *)extent->l1_table)[l1_index];
 +        if (l2_offset_u64 == 0) {
 +            l2_offset = 0;
 +        } else if ((l2_offset_u64 & 0xffffffff00000000) != 0x1000000000000000) {
 +            /*
 +             * Top most nibble is 0x1 if grain table is allocated.
 +             * strict check - top most 4 bytes must be 0x10000000 since max
 +             * supported size is 64TB for disk - so no more than 64TB / 16MB
 +             * grain directories which is smaller than uint32,
 +             * where 16MB is the only supported default grain table coverage.
 +             */
 +            return VMDK_ERROR;
 +        } else {
 +            l2_offset_u64 = l2_offset_u64 & 0x00000000ffffffff;
 +            l2_offset_u64 = extent->sesparse_l2_tables_offset +
 +                l2_offset_u64 * l2_size_bytes / SECTOR_SIZE;
 +            if (l2_offset_u64 > 0x00000000ffffffff) {
 +                return VMDK_ERROR;
 +            }
 +            l2_offset = (unsigned int)(l2_offset_u64);
 +        }
 +    } else {
 +        assert(extent->entry_size == sizeof(uint32_t));
 +        l2_offset = ((uint32_t *)extent->l1_table)[l1_index];
 +    }
      if (!l2_offset) {
          return VMDK_UNALLOC;
      }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
                      extent->l2_cache_counts[j] >>= 1;
                  }
              }
 -            l2_table = extent->l2_cache + (i * extent->l2_size);
 +            l2_table = (char *)extent->l2_cache + (i * l2_size_bytes);
              goto found;
          }
-@@ -XXX,XX +XXX,XX @@ static int command_loop(void)
+     }
-         if (input == NULL) {
+@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
-             break;
+             min_index = i;
          }
--        ret = qemuio_command(qemuio_blk, input);
+     }
-+        ret = do_qemuio_command(input);
+-    l2_table = extent->l2_cache + (min_index * extent->l2_size);
-         g_free(input);
++    l2_table = (char *)extent->l2_cache + (min_index * l2_size_bytes);
+     BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD);
-         if (ret < 0) {
+     if (bdrv_pread(extent->file,
                  (int64_t)l2_offset * 512,
                  l2_table,
 -                extent->l2_size * sizeof(uint32_t)
 -            ) != extent->l2_size * sizeof(uint32_t)) {
 +                l2_size_bytes
 +            ) != l2_size_bytes) {
          return VMDK_ERROR;
      }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
      extent->l2_cache_counts[min_index] = 1;
   found:
      l2_index = ((offset >> 9) / extent->cluster_sectors) % extent->l2_size;
 -    cluster_sector = le32_to_cpu(l2_table[l2_index]);
 -    if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
 -        zeroed = true;
 +    if (extent->sesparse) {
 +        cluster_sector = le64_to_cpu(((uint64_t *)l2_table)[l2_index]);
 +        switch (cluster_sector & 0xf000000000000000) {
 +        case 0x0000000000000000:
 +            /* unallocated grain */
 +            if (cluster_sector != 0) {
 +                return VMDK_ERROR;
 +            }
 +            break;
 +        case 0x1000000000000000:
 +            /* scsi-unmapped grain - fallthrough */
 +        case 0x2000000000000000:
 +            /* zero grain */
 +            zeroed = true;
 +            break;
 +        case 0x3000000000000000:
 +            /* allocated grain */
 +            cluster_sector = (((cluster_sector & 0x0fff000000000000) >> 48) |
 +                              ((cluster_sector & 0x0000ffffffffffff) << 12));
 +            cluster_sector = extent->sesparse_clusters_offset +
 +                cluster_sector * extent->cluster_sectors;
 +            break;
 +        default:
 +            return VMDK_ERROR;
 +        }
 +    } else {
 +        cluster_sector = le32_to_cpu(((uint32_t *)l2_table)[l2_index]);
 +
 +        if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
 +            zeroed = true;
 +        }
      }
      if (!cluster_sector || zeroed) {
          if (!allocate) {
              return zeroed ? VMDK_ZEROED : VMDK_UNALLOC;
          }
 +        assert(!extent->sesparse);
          if (extent->next_cluster_sector >= VMDK_EXTENT_MAX_SECTORS) {
              return VMDK_ERROR;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
              m_data->l1_index = l1_index;
              m_data->l2_index = l2_index;
              m_data->l2_offset = l2_offset;
 -            m_data->l2_cache_entry = &l2_table[l2_index];
 +            m_data->l2_cache_entry = ((uint32_t *)l2_table) + l2_index;
          }
      }
      *cluster_offset = cluster_sector << BDRV_SECTOR_BITS;
@@ -XXX,XX +XXX,XX @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_t offset,
          if (!extent) {
              return -EIO;
          }
 +        if (extent->sesparse) {
 +            return -ENOTSUP;
 +        }
          offset_in_cluster = vmdk_find_offset_in_cluster(extent, offset);
          n_bytes = MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE
                               - offset_in_cluster);
 --
-.31.1
+.21.0

-[PULL 04/19] mirror: stop cancelling in-flight requests on non-force cancel in READY
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-If mirror is READY than cancel operation is not discarding the whole
-result of the operation, but instead it's a documented way get a
-point-in-time snapshot of source disk.
-So, we should not cancel any requests if mirror is READ and
-force=false. Let's fix that case.
-Note, that bug that we have before this commit is not critical, as the
-only .bdrv_cancel_in_flight implementation is nbd_cancel_in_flight()
-and it cancels only requests waiting for reconnection, so it should be
-rare case.
-Fixes: 521ff8b779b11c394dbdc43f02e158dd99df308a
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-Id: <20210421075858.40197-1-vsementsov@virtuozzo.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- include/block/block_int.h | 2 +-
- include/qemu/job.h        | 2 +-
- block/backup.c            | 2 +-
- block/mirror.c            | 6 ++++--
- job.c                     | 2 +-
- tests/qemu-iotests/264    | 2 +-
-files changed, 9 insertions(+), 7 deletions(-)
-diff --git a/include/block/block_int.h b/include/block/block_int.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block_int.h
-+++ b/include/block/block_int.h
-@@ -XXX,XX +XXX,XX @@ struct BlockDriver {
-      * of in-flight requests, so don't waste the time if possible.
-      *
-      * One example usage is to avoid waiting for an nbd target node reconnect
--     * timeout during job-cancel.
-+     * timeout during job-cancel with force=true.
-      */
-     void (*bdrv_cancel_in_flight)(BlockDriverState *bs);
-diff --git a/include/qemu/job.h b/include/qemu/job.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/qemu/job.h
-+++ b/include/qemu/job.h
-@@ -XXX,XX +XXX,XX @@ struct JobDriver {
-     /**
-      * If the callback is not NULL, it will be invoked in job_cancel_async
-      */
--    void (*cancel)(Job *job);
-+    void (*cancel)(Job *job, bool force);
-     /** Called when the job is freed */
-diff --git a/block/backup.c b/block/backup.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/backup.c
-+++ b/block/backup.c
-@@ -XXX,XX +XXX,XX @@ static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed)
-     }
- }
--static void backup_cancel(Job *job)
-+static void backup_cancel(Job *job, bool force)
- {
-     BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
-diff --git a/block/mirror.c b/block/mirror.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/mirror.c
-+++ b/block/mirror.c
-@@ -XXX,XX +XXX,XX @@ static bool mirror_drained_poll(BlockJob *job)
-     return !!s->in_flight;
- }
--static void mirror_cancel(Job *job)
-+static void mirror_cancel(Job *job, bool force)
- {
-     MirrorBlockJob *s = container_of(job, MirrorBlockJob, common.job);
-     BlockDriverState *target = blk_bs(s->target);
--    bdrv_cancel_in_flight(target);
-+    if (force || !job_is_ready(job)) {
-+        bdrv_cancel_in_flight(target);
-+    }
- }
- static const BlockJobDriver mirror_job_driver = {
-diff --git a/job.c b/job.c
-index XXXXXXX..XXXXXXX 100644
---- a/job.c
-+++ b/job.c
-@@ -XXX,XX +XXX,XX @@ static int job_finalize_single(Job *job)
- static void job_cancel_async(Job *job, bool force)
- {
-     if (job->driver->cancel) {
--        job->driver->cancel(job);
-+        job->driver->cancel(job, force);
-     }
-     if (job->user_paused) {
-         /* Do not call job_enter here, the caller will handle it.  */
-diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/264
-+++ b/tests/qemu-iotests/264
-@@ -XXX,XX +XXX,XX @@ class TestNbdReconnect(iotests.QMPTestCase):
-             self.assert_qmp(result, 'return', {})
-     def cancel_job(self):
--        result = self.vm.qmp('block-job-cancel', device='drive0')
-+        result = self.vm.qmp('block-job-cancel', device='drive0', force=True)
-         self.assert_qmp(result, 'return', {})
-         start_t = time.time()
---
-.31.1

-[PULL 05/19] qemu-iotests: do not buffer the test output
+Deleted patch
-From: Paolo Bonzini <pbonzini@redhat.com>
-Instead of buffering the test output into a StringIO, patch it on
-the fly by wrapping sys.stdout's write method.  This can be
-done unconditionally, even if using -d, which makes execute_unittest
-a bit simpler.
-Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
-Message-Id: <20210323181928.311862-2-pbonzini@redhat.com>
-Message-Id: <20210503110110.476887-2-pbonzini@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/240.out    |  8 ++--
- tests/qemu-iotests/245.out    |  8 ++--
- tests/qemu-iotests/295.out    |  6 +--
- tests/qemu-iotests/296.out    |  8 ++--
- tests/qemu-iotests/iotests.py | 70 ++++++++++++++++++++---------------
-files changed, 56 insertions(+), 44 deletions(-)
-diff --git a/tests/qemu-iotests/240.out b/tests/qemu-iotests/240.out
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/240.out
-+++ b/tests/qemu-iotests/240.out
-@@ -XXX,XX +XXX,XX @@
- {"return": {}}
- {"execute": "blockdev-del", "arguments": {"node-name": "hd0"}}
- {"return": {}}
--==Attach two SCSI disks using the same block device and the same iothread==
-+.==Attach two SCSI disks using the same block device and the same iothread==
- {"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "hd0", "read-only": true, "read-zeroes": true}}
- {"return": {}}
- {"execute": "object-add", "arguments": {"id": "iothread0", "qom-type": "iothread"}}
-@@ -XXX,XX +XXX,XX @@
- {"return": {}}
- {"execute": "blockdev-del", "arguments": {"node-name": "hd0"}}
- {"return": {}}
--==Attach two SCSI disks using the same block device but different iothreads==
-+.==Attach two SCSI disks using the same block device but different iothreads==
- {"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "hd0", "read-only": true, "read-zeroes": true}}
- {"return": {}}
- {"execute": "object-add", "arguments": {"id": "iothread0", "qom-type": "iothread"}}
-@@ -XXX,XX +XXX,XX @@
- {"return": {}}
- {"execute": "blockdev-del", "arguments": {"node-name": "hd0"}}
- {"return": {}}
--==Attach a SCSI disks using the same block device as a NBD server==
-+.==Attach a SCSI disks using the same block device as a NBD server==
- {"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "hd0", "read-only": true, "read-zeroes": true}}
- {"return": {}}
- {"execute": "nbd-server-start", "arguments": {"addr": {"data": {"path": "SOCK_DIR/PID-nbd.sock"}, "type": "unix"}}}
-@@ -XXX,XX +XXX,XX @@
- {"return": {}}
- {"execute": "device_add", "arguments": {"drive": "hd0", "driver": "scsi-hd", "id": "scsi-hd0"}}
- {"return": {}}
--....
-+.
- ----------------------------------------------------------------------
- Ran 4 tests
-diff --git a/tests/qemu-iotests/245.out b/tests/qemu-iotests/245.out
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/245.out
-+++ b/tests/qemu-iotests/245.out
-@@ -XXX,XX +XXX,XX @@
--{"execute": "job-finalize", "arguments": {"id": "commit0"}}
-+..{"execute": "job-finalize", "arguments": {"id": "commit0"}}
- {"return": {}}
- {"data": {"id": "commit0", "type": "commit"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
- {"data": {"device": "commit0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
--{"execute": "job-finalize", "arguments": {"id": "stream0"}}
-+...{"execute": "job-finalize", "arguments": {"id": "stream0"}}
- {"return": {}}
- {"data": {"id": "stream0", "type": "stream"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
- {"data": {"device": "stream0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "stream"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
--{"execute": "job-finalize", "arguments": {"id": "stream0"}}
-+.{"execute": "job-finalize", "arguments": {"id": "stream0"}}
- {"return": {}}
- {"data": {"id": "stream0", "type": "stream"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
- {"data": {"device": "stream0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "stream"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
--.....................
-+...............
- ----------------------------------------------------------------------
- Ran 21 tests
-diff --git a/tests/qemu-iotests/295.out b/tests/qemu-iotests/295.out
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/295.out
-+++ b/tests/qemu-iotests/295.out
-@@ -XXX,XX +XXX,XX @@
- {"return": {}}
- {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
- {"return": {}}
--{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
-+.{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
- {"return": {}}
- {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
- {"return": {}}
-@@ -XXX,XX +XXX,XX @@ Job failed: Invalid password, cannot unlock any keyslot
- {"return": {}}
- {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
- {"return": {}}
--{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
-+.{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
- {"return": {}}
- {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
- {"return": {}}
-@@ -XXX,XX +XXX,XX @@ Job failed: All the active keyslots match the (old) password that was given and
- {"return": {}}
- {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
- {"return": {}}
--...
-+.
- ----------------------------------------------------------------------
- Ran 3 tests
-diff --git a/tests/qemu-iotests/296.out b/tests/qemu-iotests/296.out
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/296.out
-+++ b/tests/qemu-iotests/296.out
-@@ -XXX,XX +XXX,XX @@ Job failed: Failed to get shared "consistent read" lock
- qemu-img: Failed to get shared "consistent read" lock
- Is another process using the image [TEST_DIR/test.img]?
--Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
-+.Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
- Job failed: Block node is read-only
- {"execute": "job-dismiss", "arguments": {"id": "job0"}}
-@@ -XXX,XX +XXX,XX @@ Job failed: Failed to get shared "consistent read" lock
- {"return": {}}
- {"execute": "job-dismiss", "arguments": {"id": "job0"}}
- {"return": {}}
--Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
-+.Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
- {"return": {}}
- {"error": {"class": "GenericError", "desc": "Failed to get \"write\" lock"}}
--Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
-+.Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
- {"return": {}}
- {"return": {}}
--....
-+.
- ----------------------------------------------------------------------
- Ran 4 tests
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@
- import bz2
- from collections import OrderedDict
- import faulthandler
--import io
- import json
- import logging
- import os
-@@ -XXX,XX +XXX,XX @@
- import sys
- import time
- from typing import (Any, Callable, Dict, Iterable,
--                    List, Optional, Sequence, Tuple, TypeVar)
-+                    List, Optional, Sequence, TextIO, Tuple, Type, TypeVar)
- import unittest
- from contextlib import contextmanager
-@@ -XXX,XX +XXX,XX @@ def func_wrapper(*args, **kwargs):
-             return func(*args, **kwargs)
-     return func_wrapper
-+# We need to filter out the time taken from the output so that
-+# qemu-iotest can reliably diff the results against master output,
-+# and hide skipped tests from the reference output.
-+
-+class ReproducibleTestResult(unittest.TextTestResult):
-+    def addSkip(self, test, reason):
-+        # Same as TextTestResult, but print dot instead of "s"
-+        unittest.TestResult.addSkip(self, test, reason)
-+        if self.showAll:
-+            self.stream.writeln("skipped {0!r}".format(reason))
-+        elif self.dots:
-+            self.stream.write(".")
-+            self.stream.flush()
-+
-+class ReproducibleStreamWrapper:
-+    def __init__(self, stream: TextIO):
-+        self.stream = stream
-+
-+    def __getattr__(self, attr):
-+        if attr in ('stream', '__getstate__'):
-+            raise AttributeError(attr)
-+        return getattr(self.stream, attr)
-+
-+    def write(self, arg=None):
-+        arg = re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', arg)
-+        arg = re.sub(r' \(skipped=\d+\)', r'', arg)
-+        self.stream.write(arg)
-+
-+class ReproducibleTestRunner(unittest.TextTestRunner):
-+    def __init__(self, stream: Optional[TextIO] = None,
-+             resultclass: Type[unittest.TestResult] = ReproducibleTestResult,
-+             **kwargs: Any) -> None:
-+        rstream = ReproducibleStreamWrapper(stream or sys.stdout)
-+        super().__init__(stream=rstream,           # type: ignore
-+                         descriptions=True,
-+                         resultclass=resultclass,
-+                         **kwargs)
-+
- def execute_unittest(debug=False):
-     """Executes unittests within the calling module."""
-     verbosity = 2 if debug else 1
--
--    if debug:
--        output = sys.stdout
--    else:
--        # We need to filter out the time taken from the output so that
--        # qemu-iotest can reliably diff the results against master output.
--        output = io.StringIO()
--
--    runner = unittest.TextTestRunner(stream=output, descriptions=True,
--                                     verbosity=verbosity)
--    try:
--        # unittest.main() will use sys.exit(); so expect a SystemExit
--        # exception
--        unittest.main(testRunner=runner)
--    finally:
--        # We need to filter out the time taken from the output so that
--        # qemu-iotest can reliably diff the results against master output.
--        if not debug:
--            out = output.getvalue()
--            out = re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', out)
--
--            # Hide skipped tests from the reference output
--            out = re.sub(r'OK \(skipped=\d+\)', 'OK', out)
--            out_first_line, out_rest = out.split('\n', 1)
--            out = out_first_line.replace('s', '.') + '\n' + out_rest
--
--            sys.stderr.write(out)
-+    runner = ReproducibleTestRunner(verbosity=verbosity)
-+    unittest.main(testRunner=runner)
- def execute_setup_common(supported_fmts: Sequence[str] = (),
-                          supported_platforms: Sequence[str] = (),
---
-.31.1

-[PULL 06/19] qemu-iotests: allow passing unittest.main arguments to the test scripts
+Deleted patch
-From: Paolo Bonzini <pbonzini@redhat.com>
-Python test scripts that use unittest consist of multiple tests.
-unittest.main allows selecting which tests to run, but currently this
-is not possible because the iotests wrapper ignores sys.argv.
-unittest.main command line options also allow the user to pick the
-desired options for verbosity, failfast mode, etc.  While "-d" is
-currently translated to "-v", it also enables extra debug output,
-and other options are not available at all.
-These command line options only work if the unittest.main testRunner
-argument is a type, rather than a TestRunner instance.  Therefore, pass
-the class name and "verbosity" argument to unittest.main, and adjust for
-the different default warnings between TextTestRunner and unittest.main.
-Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
-Message-Id: <20210323181928.311862-3-pbonzini@redhat.com>
-Message-Id: <20210503110110.476887-3-pbonzini@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/iotests.py | 14 +++++++++-----
-file changed, 9 insertions(+), 5 deletions(-)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ def __init__(self, stream: Optional[TextIO] = None,
-                          resultclass=resultclass,
-                          **kwargs)
--def execute_unittest(debug=False):
-+def execute_unittest(argv: List[str], debug: bool = False) -> None:
-     """Executes unittests within the calling module."""
--    verbosity = 2 if debug else 1
--    runner = ReproducibleTestRunner(verbosity=verbosity)
--    unittest.main(testRunner=runner)
-+    # Some tests have warnings, especially ResourceWarnings for unclosed
-+    # files and sockets.  Ignore them for now to ensure reproducibility of
-+    # the test output.
-+    unittest.main(argv=argv,
-+                  testRunner=ReproducibleTestRunner,
-+                  verbosity=2 if debug else 1,
-+                  warnings=None if sys.warnoptions else 'ignore')
- def execute_setup_common(supported_fmts: Sequence[str] = (),
-                          supported_platforms: Sequence[str] = (),
-@@ -XXX,XX +XXX,XX @@ def execute_test(*args, test_function=None, **kwargs):
-     debug = execute_setup_common(*args, **kwargs)
-     if not test_function:
--        execute_unittest(debug)
-+        execute_unittest(sys.argv, debug)
-     else:
-         test_function()
---
-.31.1

-[PULL 07/19] qemu-iotests: move command line and environment handling from TestRunner to TestEnv
+Deleted patch
-From: Paolo Bonzini <pbonzini@redhat.com>
-In the next patch, "check" will learn how to execute a test script without
-going through TestRunner.  To enable this, keep only the text output
-and subprocess handling in the TestRunner; move into TestEnv the logic
-to prepare for running a subprocess.
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
-Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
-Message-Id: <20210323181928.311862-4-pbonzini@redhat.com>
-Message-Id: <20210503110110.476887-4-pbonzini@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/testenv.py    | 17 ++++++++++++++++-
- tests/qemu-iotests/testrunner.py | 14 +-------------
-files changed, 17 insertions(+), 14 deletions(-)
-diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testenv.py
-+++ b/tests/qemu-iotests/testenv.py
-@@ -XXX,XX +XXX,XX @@
- import random
- import subprocess
- import glob
--from typing import Dict, Any, Optional, ContextManager
-+from typing import List, Dict, Any, Optional, ContextManager
- def isxfile(path: str) -> bool:
-@@ -XXX,XX +XXX,XX @@ class TestEnv(ContextManager['TestEnv']):
-                      'CACHEMODE_IS_DEFAULT', 'IMGFMT_GENERIC', 'IMGOPTSSYNTAX',
-                      'IMGKEYSECRET', 'QEMU_DEFAULT_MACHINE', 'MALLOC_PERTURB_']
-+    def prepare_subprocess(self, args: List[str]) -> Dict[str, str]:
-+        if self.debug:
-+            args.append('-d')
-+
-+        with open(args[0], encoding="utf-8") as f:
-+            try:
-+                if f.readline().rstrip() == '#!/usr/bin/env python3':
-+                    args.insert(0, self.python)
-+            except UnicodeDecodeError:  # binary test? for future.
-+                pass
-+
-+        os_env = os.environ.copy()
-+        os_env.update(self.get_env())
-+        return os_env
-+
-     def get_env(self) -> Dict[str, str]:
-         env = {}
-         for v in self.env_variables:
-diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testrunner.py
-+++ b/tests/qemu-iotests/testrunner.py
-@@ -XXX,XX +XXX,XX @@ class TestRunner(ContextManager['TestRunner']):
-     def __init__(self, env: TestEnv, makecheck: bool = False,
-                  color: str = 'auto') -> None:
-         self.env = env
--        self.test_run_env = self.env.get_env()
-         self.makecheck = makecheck
-         self.last_elapsed = LastElapsedTime('.last-elapsed-cache', env)
-@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str) -> TestResult:
-             silent_unlink(p)
-         args = [str(f_test.resolve())]
--        if self.env.debug:
--            args.append('-d')
--
--        with f_test.open(encoding="utf-8") as f:
--            try:
--                if f.readline().rstrip() == '#!/usr/bin/env python3':
--                    args.insert(0, self.env.python)
--            except UnicodeDecodeError:  # binary test? for future.
--                pass
--
--        env = os.environ.copy()
--        env.update(self.test_run_env)
-+        env = self.env.prepare_subprocess(args)
-         t0 = time.time()
-         with f_bad.open('w', encoding="utf-8") as f:
---
-.31.1

-[PULL 08/19] qemu-iotests: let "check" spawn an arbitrary test command
+Deleted patch
-From: Paolo Bonzini <pbonzini@redhat.com>
-Right now there is no easy way for "check" to print a reproducer command.
-Because such a reproducer command line would be huge, we can instead teach
-check to start a command of our choice.  This can be for example a Python
-unit test with arguments to only run a specific subtest.
-Move the trailing empty line to print_env(), since it always looks better
-and one caller was not adding it.
-Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
-Message-Id: <20210323181928.311862-5-pbonzini@redhat.com>
-Message-Id: <20210503110110.476887-5-pbonzini@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/check         | 19 ++++++++++++++++++-
- tests/qemu-iotests/testenv.py    |  3 ++-
- tests/qemu-iotests/testrunner.py |  1 -
-files changed, 20 insertions(+), 3 deletions(-)
-diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/check
-+++ b/tests/qemu-iotests/check
-@@ -XXX,XX +XXX,XX @@
- import os
- import sys
- import argparse
-+import shutil
-+from pathlib import Path
-+
- from findtests import TestFinder
- from testenv import TestEnv
- from testrunner import TestRunner
-@@ -XXX,XX +XXX,XX @@ def make_argparser() -> argparse.ArgumentParser:
-                        'rerun failed ./check command, starting from the '
-                        'middle of the process.')
-     g_sel.add_argument('tests', metavar='TEST_FILES', nargs='*',
--                       help='tests to run')
-+                       help='tests to run, or "--" followed by a command')
-     return p
-@@ -XXX,XX +XXX,XX @@ if __name__ == '__main__':
-                   imgopts=args.imgopts, misalign=args.misalign,
-                   debug=args.debug, valgrind=args.valgrind)
-+    if len(sys.argv) > 1 and sys.argv[-len(args.tests)-1] == '--':
-+        if not args.tests:
-+            sys.exit("missing command after '--'")
-+        cmd = args.tests
-+        env.print_env()
-+        exec_pathstr = shutil.which(cmd[0])
-+        if exec_pathstr is None:
-+            sys.exit('command not found: ' + cmd[0])
-+        exec_path = Path(exec_pathstr).resolve()
-+        cmd[0] = str(exec_path)
-+        full_env = env.prepare_subprocess(cmd)
-+        os.chdir(exec_path.parent)
-+        os.execve(cmd[0], cmd, full_env)
-+
-     testfinder = TestFinder(test_dir=env.source_iotests)
-     groups = args.groups.split(',') if args.groups else None
-diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testenv.py
-+++ b/tests/qemu-iotests/testenv.py
-@@ -XXX,XX +XXX,XX @@ def print_env(self) -> None:
- PLATFORM      -- {platform}
- TEST_DIR      -- {TEST_DIR}
- SOCK_DIR      -- {SOCK_DIR}
--SOCKET_SCM_HELPER -- {SOCKET_SCM_HELPER}"""
-+SOCKET_SCM_HELPER -- {SOCKET_SCM_HELPER}
-+"""
-         args = collections.defaultdict(str, self.get_env())
-diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testrunner.py
-+++ b/tests/qemu-iotests/testrunner.py
-@@ -XXX,XX +XXX,XX @@ def run_tests(self, tests: List[str]) -> bool:
-         if not self.makecheck:
-             self.env.print_env()
--            print()
-         test_field_width = max(len(os.path.basename(t)) for t in tests) + 2
---
-.31.1

-[PULL 09/19] qemu-iotests: fix case of SOCK_DIR already in the environment
+Deleted patch
-From: Paolo Bonzini <pbonzini@redhat.com>
-Due to a typo, in this case the SOCK_DIR was not being created.
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
-Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
-Message-Id: <20210323181928.311862-6-pbonzini@redhat.com>
-Message-Id: <20210503110110.476887-6-pbonzini@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/qemu-iotests/testenv.py | 2 +-
-file changed, 1 insertion(+), 1 deletion(-)
-diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testenv.py
-+++ b/tests/qemu-iotests/testenv.py
-@@ -XXX,XX +XXX,XX @@ def init_directories(self) -> None:
-         try:
-             self.sock_dir = os.environ['SOCK_DIR']
-             self.tmp_sock_dir = False
--            Path(self.test_dir).mkdir(parents=True, exist_ok=True)
-+            Path(self.sock_dir).mkdir(parents=True, exist_ok=True)
-         except KeyError:
-             self.sock_dir = tempfile.mkdtemp()
-             self.tmp_sock_dir = True
---
-.31.1

-[PULL 10/19] Document qemu-img options data_file and data_file_raw
+Deleted patch
-From: Connor Kuehl <ckuehl@redhat.com>
-The contents of this patch were initially developed and posted by Han
-Han[1], however, it appears the original patch was not applied. Since
-then, the relevant documentation has been moved and adapted to a new
-format.
-I've taken most of the original wording and tweaked it according to
-some of the feedback from the original patch submission. I've also
-adapted it to restructured text, which is the format the documentation
-currently uses.
-[1] https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg01253.html
-Fixes: https://bugzilla.redhat.com/1763105
-Signed-off-by: Han Han <hhan@redhat.com>
-Suggested-by: Max Reitz <mreitz@redhat.com>
-[ Max: provided description of data_file_raw behavior ]
-Signed-off-by: Connor Kuehl <ckuehl@redhat.com>
-Message-Id: <20210505195512.391128-1-ckuehl@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- docs/tools/qemu-img.rst | 31 +++++++++++++++++++++++++++++++
-file changed, 31 insertions(+)
-diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
-index XXXXXXX..XXXXXXX 100644
---- a/docs/tools/qemu-img.rst
-+++ b/docs/tools/qemu-img.rst
-@@ -XXX,XX +XXX,XX @@ Supported image file formats:
-     issue ``lsattr filename`` to check if the NOCOW flag is set or not
-     (Capital 'C' is NOCOW flag).
-+  ``data_file``
-+    Filename where all guest data will be stored. If this option is used,
-+    the qcow2 file will only contain the image's metadata.
-+
-+    Note: Data loss will occur if the given filename already exists when
-+    using this option with ``qemu-img create`` since ``qemu-img`` will create
-+    the data file anew, overwriting the file's original contents. To simply
-+    update the reference to point to the given pre-existing file, use
-+    ``qemu-img amend``.
-+
-+  ``data_file_raw``
-+    If this option is set to ``on``, QEMU will always keep the external data
-+    file consistent as a standalone read-only raw image.
-+
-+    It does this by forwarding all write accesses to the qcow2 file through to
-+    the raw data file, including their offsets. Therefore, data that is visible
-+    on the qcow2 node (i.e., to the guest) at some offset is visible at the same
-+    offset in the raw data file. This results in a read-only raw image. Writes
-+    that bypass the qcow2 metadata may corrupt the qcow2 metadata because the
-+    out-of-band writes may result in the metadata falling out of sync with the
-+    raw image.
-+
-+    If this option is ``off``, QEMU will use the data file to store data in an
-+    arbitrary manner. The file’s content will not make sense without the
-+    accompanying qcow2 metadata. Where data is written will have no relation to
-+    its offset as seen by the guest, and some writes (specifically zero writes)
-+    may not be forwarded to the data file at all, but will only be handled by
-+    modifying qcow2 metadata.
-+
-+    This option can only be enabled if ``data_file`` is set.
-+
- ``Other``
-   QEMU also supports various other image file formats for
---
-.31.1

-[PULL 11/19] block/copy-on-read: use bdrv_drop_filter() and drop s->active
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Now, after huge update of block graph permission update algorithm, we
-don't need this workaround with active state of the filter. Drop it and
-use new smart bdrv_drop_filter() function.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-Id: <20210506194143.394141-1-vsementsov@virtuozzo.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- block/copy-on-read.c | 33 +--------------------------------
-file changed, 1 insertion(+), 32 deletions(-)
-diff --git a/block/copy-on-read.c b/block/copy-on-read.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/copy-on-read.c
-+++ b/block/copy-on-read.c
-@@ -XXX,XX +XXX,XX @@
- typedef struct BDRVStateCOR {
--    bool active;
-     BlockDriverState *bottom_bs;
-     bool chain_frozen;
- } BDRVStateCOR;
-@@ -XXX,XX +XXX,XX @@ static int cor_open(BlockDriverState *bs, QDict *options, int flags,
-          */
-         bdrv_ref(bottom_bs);
-     }
--    state->active = true;
-     state->bottom_bs = bottom_bs;
-     /*
-@@ -XXX,XX +XXX,XX @@ static void cor_child_perm(BlockDriverState *bs, BdrvChild *c,
-                            uint64_t perm, uint64_t shared,
-                            uint64_t *nperm, uint64_t *nshared)
- {
--    BDRVStateCOR *s = bs->opaque;
--
--    if (!s->active) {
--        /*
--         * While the filter is being removed
--         */
--        *nperm = 0;
--        *nshared = BLK_PERM_ALL;
--        return;
--    }
--
-     *nperm = perm & PERM_PASSTHROUGH;
-     *nshared = (shared & PERM_PASSTHROUGH) | PERM_UNCHANGED;
-@@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_copy_on_read = {
- void bdrv_cor_filter_drop(BlockDriverState *cor_filter_bs)
- {
--    BdrvChild *child;
--    BlockDriverState *bs;
-     BDRVStateCOR *s = cor_filter_bs->opaque;
--    child = bdrv_filter_child(cor_filter_bs);
--    if (!child) {
--        return;
--    }
--    bs = child->bs;
--
--    /* Retain the BDS until we complete the graph change. */
--    bdrv_ref(bs);
--    /* Hold a guest back from writing while permissions are being reset. */
--    bdrv_drained_begin(bs);
--    /* Drop permissions before the graph change. */
--    s->active = false;
-     /* unfreeze, as otherwise bdrv_replace_node() will fail */
-     if (s->chain_frozen) {
-         s->chain_frozen = false;
-         bdrv_unfreeze_backing_chain(cor_filter_bs, s->bottom_bs);
-     }
--    bdrv_child_refresh_perms(cor_filter_bs, child, &error_abort);
--    bdrv_replace_node(cor_filter_bs, bs, &error_abort);
--
--    bdrv_drained_end(bs);
--    bdrv_unref(bs);
-+    bdrv_drop_filter(cor_filter_bs, &error_abort);
-     bdrv_unref(cor_filter_bs);
- }
---
-.31.1

-[PULL 13/19] block/write-threshold: don't use write notifiers
+[Qemu-devel] [PULL v2 7/8] ssh: switch from libssh2 to libssh
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Pino Toscano <ptoscano@redhat.com>
-write-notifiers are used only for write-threshold. New code for such
+Rewrite the implementation of the ssh block driver to use libssh instead
-purpose should create filters.
+of libssh2.  The libssh library has various advantages over libssh2:
 - easier API for authentication (for example for using ssh-agent)
 - easier API for known_hosts handling
 - supports newer types of keys in known_hosts
-Let's better special-case write-threshold and drop write notifiers at
+Use APIs/features available in libssh 0.8 conditionally, to support
-all. (Actually, write-threshold is special-cased anyway, as the only
+older versions (which are not recommended though).
 user of write-notifiers)
-So, create a new direct interface for bdrv_co_write_req_prepare() and
+Adjust the iotest 207 according to the different error message, and to
-drop all write-notifier related logic from write-threshold.c.
+find the default key type for localhost (to properly compare the
 fingerprint with).
 Contributed-by: Max Reitz <mreitz@redhat.com>
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Adjust the various Docker/Travis scripts to use libssh when available
-Reviewed-by: Max Reitz <mreitz@redhat.com>
+instead of libssh2. The mingw/mxe testing is dropped for now, as there
-Message-Id: <20210506090621.11848-2-vsementsov@virtuozzo.com>
+are no packages for it.
-Reviewed-by: Eric Blake <eblake@redhat.com>
-Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
+Signed-off-by: Pino Toscano <ptoscano@redhat.com>
-[mreitz: Adjusted comment as per Eric's suggestion]
+Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
 Acked-by: Alex Bennée <alex.bennee@linaro.org>
 Message-id: 20190620200840.17655-1-ptoscano@redhat.com
 Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
 Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- include/block/block_int.h       |  1 -
+ configure                                     |  65 +-
- include/block/write-threshold.h |  9 +++++
+ block/Makefile.objs                           |   6 +-
- block/io.c                      |  5 ++-
+ block/ssh.c                                   | 652 ++++++++++--------
- block/write-threshold.c         | 70 +++++++--------------------------
+ .travis.yml                                   |   4 +-
-files changed, 27 insertions(+), 58 deletions(-)
+ block/trace-events                            |  14 +-
  docs/qemu-block-drivers.texi                  |   2 +-
  .../dockerfiles/debian-win32-cross.docker     |   1 -
  .../dockerfiles/debian-win64-cross.docker     |   1 -
  tests/docker/dockerfiles/fedora.docker        |   4 +-
  tests/docker/dockerfiles/ubuntu.docker        |   2 +-
  tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
  tests/qemu-iotests/207                        |  54 +-
  tests/qemu-iotests/207.out                    |   2 +-
 files changed, 449 insertions(+), 360 deletions(-)
-diff --git a/include/block/block_int.h b/include/block/block_int.h
+diff --git a/configure b/configure
 index XXXXXXX..XXXXXXX 100755
 --- a/configure
 +++ b/configure
@@ -XXX,XX +XXX,XX @@ auth_pam=""
  vte=""
  virglrenderer=""
  tpm=""
 -libssh2=""
 +libssh=""
  live_block_migration="yes"
  numa=""
  tcmalloc="no"
@@ -XXX,XX +XXX,XX @@ for opt do
    ;;
    --enable-tpm) tpm="yes"
    ;;
 -  --disable-libssh2) libssh2="no"
 +  --disable-libssh) libssh="no"
    ;;
 -  --enable-libssh2) libssh2="yes"
 +  --enable-libssh) libssh="yes"
    ;;
    --disable-live-block-migration) live_block_migration="no"
    ;;
@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
    coroutine-pool  coroutine freelist (better performance)
    glusterfs       GlusterFS backend
    tpm             TPM support
 -  libssh2         ssh block device support
 +  libssh          ssh block device support
    numa            libnuma support
    libxml2         for Parallels image format
    tcmalloc        tcmalloc support
@@ -XXX,XX +XXX,XX @@ EOF
  fi
  ##########################################
 -# libssh2 probe
 -min_libssh2_version=1.2.8
 -if test "$libssh2" != "no" ; then
 -  if $pkg_config --atleast-version=$min_libssh2_version libssh2; then
 -    libssh2_cflags=$($pkg_config libssh2 --cflags)
 -    libssh2_libs=$($pkg_config libssh2 --libs)
 -    libssh2=yes
 +# libssh probe
 +if test "$libssh" != "no" ; then
 +  if $pkg_config --exists libssh; then
 +    libssh_cflags=$($pkg_config libssh --cflags)
 +    libssh_libs=$($pkg_config libssh --libs)
 +    libssh=yes
    else
 -    if test "$libssh2" = "yes" ; then
 -      error_exit "libssh2 >= $min_libssh2_version required for --enable-libssh2"
 +    if test "$libssh" = "yes" ; then
 +      error_exit "libssh required for --enable-libssh"
      fi
 -    libssh2=no
 +    libssh=no
    fi
  fi
  ##########################################
 -# libssh2_sftp_fsync probe
 +# Check for libssh 0.8
 +# This is done like this instead of using the LIBSSH_VERSION_* and
 +# SSH_VERSION_* macros because some distributions in the past shipped
 +# snapshots of the future 0.8 from Git, and those snapshots did not
 +# have updated version numbers (still referring to 0.7.0).
 -if test "$libssh2" = "yes"; then
 +if test "$libssh" = "yes"; then
    cat > $TMPC <<EOF
 -#include <stdio.h>
 -#include <libssh2.h>
 -#include <libssh2_sftp.h>
 -int main(void) {
 -    LIBSSH2_SESSION *session;
 -    LIBSSH2_SFTP *sftp;
 -    LIBSSH2_SFTP_HANDLE *sftp_handle;
 -    session = libssh2_session_init ();
 -    sftp = libssh2_sftp_init (session);
 -    sftp_handle = libssh2_sftp_open (sftp, "/", 0, 0);
 -    libssh2_sftp_fsync (sftp_handle);
 -    return 0;
 -}
 +#include <libssh/libssh.h>
 +int main(void) { return ssh_get_server_publickey(NULL, NULL); }
  EOF
 -  # libssh2_cflags/libssh2_libs defined in previous test.
 -  if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then
 -    QEMU_CFLAGS="-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS"
 +  if compile_prog "$libssh_cflags" "$libssh_libs"; then
 +    libssh_cflags="-DHAVE_LIBSSH_0_8 $libssh_cflags"
    fi
  fi
@@ -XXX,XX +XXX,XX @@ echo "GlusterFS support $glusterfs"
  echo "gcov              $gcov_tool"
  echo "gcov enabled      $gcov"
  echo "TPM support       $tpm"
 -echo "libssh2 support   $libssh2"
 +echo "libssh support    $libssh"
  echo "QOM debugging     $qom_cast_debug"
  echo "Live block migration $live_block_migration"
  echo "lzo support       $lzo"
@@ -XXX,XX +XXX,XX @@ if test "$glusterfs_iocb_has_stat" = "yes" ; then
    echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=y" >> $config_host_mak
  fi
 -if test "$libssh2" = "yes" ; then
 -  echo "CONFIG_LIBSSH2=m" >> $config_host_mak
 -  echo "LIBSSH2_CFLAGS=$libssh2_cflags" >> $config_host_mak
 -  echo "LIBSSH2_LIBS=$libssh2_libs" >> $config_host_mak
 +if test "$libssh" = "yes" ; then
 +  echo "CONFIG_LIBSSH=m" >> $config_host_mak
 +  echo "LIBSSH_CFLAGS=$libssh_cflags" >> $config_host_mak
 +  echo "LIBSSH_LIBS=$libssh_libs" >> $config_host_mak
  fi
  if test "$live_block_migration" = "yes" ; then
 diff --git a/block/Makefile.objs b/block/Makefile.objs
 index XXXXXXX..XXXXXXX 100644
---- a/include/block/block_int.h
+--- a/block/Makefile.objs
-+++ b/include/block/block_int.h
++++ b/block/Makefile.objs
-@@ -XXX,XX +XXX,XX @@ struct BlockDriverState {
+@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_CURL) += curl.o
+ block-obj-$(CONFIG_RBD) += rbd.o
-     /* threshold limit for writes, in bytes. "High water mark". */
+ block-obj-$(CONFIG_GLUSTERFS) += gluster.o
-     uint64_t write_threshold_offset;
+ block-obj-$(CONFIG_VXHS) += vxhs.o
--    NotifierWithReturn write_threshold_notifier;
+-block-obj-$(CONFIG_LIBSSH2) += ssh.o
++block-obj-$(CONFIG_LIBSSH) += ssh.o
-     /* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
+ block-obj-y += accounting.o dirty-bitmap.o
-      * Reading from the list can be done with either the BQL or the
+ block-obj-y += write-threshold.o
-diff --git a/include/block/write-threshold.h b/include/block/write-threshold.h
+ block-obj-y += backup.o
@@ -XXX,XX +XXX,XX @@ rbd.o-libs         := $(RBD_LIBS)
  gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
  gluster.o-libs     := $(GLUSTERFS_LIBS)
  vxhs.o-libs        := $(VXHS_LIBS)
 -ssh.o-cflags       := $(LIBSSH2_CFLAGS)
 -ssh.o-libs         := $(LIBSSH2_LIBS)
 +ssh.o-cflags       := $(LIBSSH_CFLAGS)
 +ssh.o-libs         := $(LIBSSH_LIBS)
  block-obj-dmg-bz2-$(CONFIG_BZIP2) += dmg-bz2.o
  block-obj-$(if $(CONFIG_DMG),m,n) += $(block-obj-dmg-bz2-y)
  dmg-bz2.o-libs     := $(BZIP2_LIBS)
 diff --git a/block/ssh.c b/block/ssh.c
 index XXXXXXX..XXXXXXX 100644
---- a/include/block/write-threshold.h
+--- a/block/ssh.c
-+++ b/include/block/write-threshold.h
++++ b/block/ssh.c
@@ -XXX,XX +XXX,XX @@ bool bdrv_write_threshold_is_set(const BlockDriverState *bs);
  uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
                                         const BdrvTrackedRequest *req);
 +/*
 + * bdrv_write_threshold_check_write
 + *
 + * Check whether the specified request exceeds the write threshold.
 + * If so, send a corresponding event and disable write threshold checking.
 + */
 +void bdrv_write_threshold_check_write(BlockDriverState *bs, int64_t offset,
 +                                      int64_t bytes);
 +
  #endif
 diff --git a/block/io.c b/block/io.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/io.c
 +++ b/block/io.c
 @@ -XXX,XX +XXX,XX @@
- #include "block/blockjob_int.h"
  #include "qemu/osdep.h"
 -#include <libssh2.h>
 -#include <libssh2_sftp.h>
 +#include <libssh/libssh.h>
 +#include <libssh/sftp.h>
  #include "block/block_int.h"
- #include "block/coroutines.h"
+ #include "block/qdict.h"
-+#include "block/write-threshold.h"
+@@ -XXX,XX +XXX,XX @@
- #include "qemu/cutils.h"
+ #include "trace.h"
- #include "qapi/error.h"
- #include "qemu/error-report.h"
+ /*
-@@ -XXX,XX +XXX,XX @@ bdrv_co_write_req_prepare(BdrvChild *child, int64_t offset, int64_t bytes,
+- * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
-         } else {
+- * that this requires that libssh2 was specially compiled with the
-             assert(child->perm & BLK_PERM_WRITE);
+- * `./configure --enable-debug' option, so most likely you will have
 - * to compile it yourself.  The meaning of <bitmask> is described
 - * here: http://www.libssh2.org/libssh2_trace.html
 + * TRACE_LIBSSH=<level> enables tracing in libssh itself.
 + * The meaning of <level> is described here:
 + * http://api.libssh.org/master/group__libssh__log.html
   */
 -#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
 +#define TRACE_LIBSSH  0 /* see: SSH_LOG_* */
  typedef struct BDRVSSHState {
      /* Coroutine. */
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVSSHState {
      /* SSH connection. */
      int sock;                         /* socket */
 -    LIBSSH2_SESSION *session;         /* ssh session */
 -    LIBSSH2_SFTP *sftp;               /* sftp session */
 -    LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */
 +    ssh_session session;              /* ssh session */
 +    sftp_session sftp;                /* sftp session */
 +    sftp_file sftp_handle;            /* sftp remote file handle */
 -    /* See ssh_seek() function below. */
 -    int64_t offset;
 -    bool offset_op_read;
 -
 -    /* File attributes at open.  We try to keep the .filesize field
 +    /*
 +     * File attributes at open.  We try to keep the .size field
       * updated if it changes (eg by writing at the end of the file).
       */
 -    LIBSSH2_SFTP_ATTRIBUTES attrs;
 +    sftp_attributes attrs;
      InetSocketAddress *inet;
@@ -XXX,XX +XXX,XX @@ static void ssh_state_init(BDRVSSHState *s)
  {
      memset(s, 0, sizeof *s);
      s->sock = -1;
 -    s->offset = -1;
      qemu_co_mutex_init(&s->lock);
  }
@@ -XXX,XX +XXX,XX @@ static void ssh_state_free(BDRVSSHState *s)
  {
      g_free(s->user);
 +    if (s->attrs) {
 +        sftp_attributes_free(s->attrs);
 +    }
      if (s->sftp_handle) {
 -        libssh2_sftp_close(s->sftp_handle);
 +        sftp_close(s->sftp_handle);
      }
      if (s->sftp) {
 -        libssh2_sftp_shutdown(s->sftp);
 +        sftp_free(s->sftp);
      }
      if (s->session) {
 -        libssh2_session_disconnect(s->session,
 -                                   "from qemu ssh client: "
 -                                   "user closed the connection");
 -        libssh2_session_free(s->session);
 -    }
 -    if (s->sock >= 0) {
 -        close(s->sock);
 +        ssh_disconnect(s->session);
 +        ssh_free(s->session); /* This frees s->sock */
      }
  }
@@ -XXX,XX +XXX,XX @@ session_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
      va_end(args);
      if (s->session) {
 -        char *ssh_err;
 +        const char *ssh_err;
          int ssh_err_code;
 -        /* This is not an errno.  See <libssh2.h>. */
 -        ssh_err_code = libssh2_session_last_error(s->session,
 -                                                  &ssh_err, NULL, 0);
 -        error_setg(errp, "%s: %s (libssh2 error code: %d)",
 +        /* This is not an errno.  See <libssh/libssh.h>. */
 +        ssh_err = ssh_get_error(s->session);
 +        ssh_err_code = ssh_get_error_code(s->session);
 +        error_setg(errp, "%s: %s (libssh error code: %d)",
                     msg, ssh_err, ssh_err_code);
      } else {
          error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
      va_end(args);
      if (s->sftp) {
 -        char *ssh_err;
 +        const char *ssh_err;
          int ssh_err_code;
 -        unsigned long sftp_err_code;
 +        int sftp_err_code;
 -        /* This is not an errno.  See <libssh2.h>. */
 -        ssh_err_code = libssh2_session_last_error(s->session,
 -                                                  &ssh_err, NULL, 0);
 -        /* See <libssh2_sftp.h>. */
 -        sftp_err_code = libssh2_sftp_last_error((s)->sftp);
 +        /* This is not an errno.  See <libssh/libssh.h>. */
 +        ssh_err = ssh_get_error(s->session);
 +        ssh_err_code = ssh_get_error_code(s->session);
 +        /* See <libssh/sftp.h>. */
 +        sftp_err_code = sftp_get_error(s->sftp);
          error_setg(errp,
 -                   "%s: %s (libssh2 error code: %d, sftp error code: %lu)",
 +                   "%s: %s (libssh error code: %d, sftp error code: %d)",
                     msg, ssh_err, ssh_err_code, sftp_err_code);
      } else {
          error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
  static void sftp_error_trace(BDRVSSHState *s, const char *op)
  {
 -    char *ssh_err;
 +    const char *ssh_err;
      int ssh_err_code;
 -    unsigned long sftp_err_code;
 +    int sftp_err_code;
 -    /* This is not an errno.  See <libssh2.h>. */
 -    ssh_err_code = libssh2_session_last_error(s->session,
 -                                              &ssh_err, NULL, 0);
 -    /* See <libssh2_sftp.h>. */
 -    sftp_err_code = libssh2_sftp_last_error((s)->sftp);
 +    /* This is not an errno.  See <libssh/libssh.h>. */
 +    ssh_err = ssh_get_error(s->session);
 +    ssh_err_code = ssh_get_error_code(s->session);
 +    /* See <libssh/sftp.h>. */
 +    sftp_err_code = sftp_get_error(s->sftp);
      trace_sftp_error(op, ssh_err, ssh_err_code, sftp_err_code);
  }
@@ -XXX,XX +XXX,XX @@ static void ssh_parse_filename(const char *filename, QDict *options,
      parse_uri(filename, options, errp);
  }
 -static int check_host_key_knownhosts(BDRVSSHState *s,
 -                                     const char *host, int port, Error **errp)
 +static int check_host_key_knownhosts(BDRVSSHState *s, Error **errp)
  {
 -    const char *home;
 -    char *knh_file = NULL;
 -    LIBSSH2_KNOWNHOSTS *knh = NULL;
 -    struct libssh2_knownhost *found;
 -    int ret, r;
 -    const char *hostkey;
 -    size_t len;
 -    int type;
 -
 -    hostkey = libssh2_session_hostkey(s->session, &len, &type);
 -    if (!hostkey) {
 +    int ret;
 +#ifdef HAVE_LIBSSH_0_8
 +    enum ssh_known_hosts_e state;
 +    int r;
 +    ssh_key pubkey;
 +    enum ssh_keytypes_e pubkey_type;
 +    unsigned char *server_hash = NULL;
 +    size_t server_hash_len;
 +    char *fingerprint = NULL;
 +
 +    state = ssh_session_is_known_server(s->session);
 +    trace_ssh_server_status(state);
 +
 +    switch (state) {
 +    case SSH_KNOWN_HOSTS_OK:
 +        /* OK */
 +        trace_ssh_check_host_key_knownhosts();
 +        break;
 +    case SSH_KNOWN_HOSTS_CHANGED:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to read remote host key");
 +        r = ssh_get_server_publickey(s->session, &pubkey);
 +        if (r == 0) {
 +            r = ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256,
 +                                       &server_hash, &server_hash_len);
 +            pubkey_type = ssh_key_type(pubkey);
 +            ssh_key_free(pubkey);
 +        }
 +        if (r == 0) {
 +            fingerprint = ssh_get_fingerprint_hash(SSH_PUBLICKEY_HASH_SHA256,
 +                                                   server_hash,
 +                                                   server_hash_len);
 +            ssh_clean_pubkey_hash(&server_hash);
 +        }
 +        if (fingerprint) {
 +            error_setg(errp,
 +                       "host key (%s key with fingerprint %s) does not match "
 +                       "the one in known_hosts; this may be a possible attack",
 +                       ssh_key_type_to_char(pubkey_type), fingerprint);
 +            ssh_string_free_char(fingerprint);
 +        } else  {
 +            error_setg(errp,
 +                       "host key does not match the one in known_hosts; this "
 +                       "may be a possible attack");
 +        }
          goto out;
 -    }
 -
 -    knh = libssh2_knownhost_init(s->session);
 -    if (!knh) {
 +    case SSH_KNOWN_HOSTS_OTHER:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                           "failed to initialize known hosts support");
 +        error_setg(errp,
 +                   "host key for this server not found, another type exists");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_UNKNOWN:
 +        ret = -EINVAL;
 +        error_setg(errp, "no host key was found in known_hosts");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_NOT_FOUND:
 +        ret = -ENOENT;
 +        error_setg(errp, "known_hosts file not found");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_ERROR:
 +        ret = -EINVAL;
 +        error_setg(errp, "error while checking the host");
 +        goto out;
 +    default:
 +        ret = -EINVAL;
 +        error_setg(errp, "error while checking for known server (%d)", state);
          goto out;
      }
 +#else /* !HAVE_LIBSSH_0_8 */
 +    int state;
 -    home = getenv("HOME");
 -    if (home) {
 -        knh_file = g_strdup_printf("%s/.ssh/known_hosts", home);
 -    } else {
 -        knh_file = g_strdup_printf("/root/.ssh/known_hosts");
 -    }
 -
 -    /* Read all known hosts from OpenSSH-style known_hosts file. */
 -    libssh2_knownhost_readfile(knh, knh_file, LIBSSH2_KNOWNHOST_FILE_OPENSSH);
 +    state = ssh_is_server_known(s->session);
 +    trace_ssh_server_status(state);
 -    r = libssh2_knownhost_checkp(knh, host, port, hostkey, len,
 -                                 LIBSSH2_KNOWNHOST_TYPE_PLAIN|
 -                                 LIBSSH2_KNOWNHOST_KEYENC_RAW,
 -                                 &found);
 -    switch (r) {
 -    case LIBSSH2_KNOWNHOST_CHECK_MATCH:
 +    switch (state) {
 +    case SSH_SERVER_KNOWN_OK:
          /* OK */
 -        trace_ssh_check_host_key_knownhosts(found->key);
 +        trace_ssh_check_host_key_knownhosts();
          break;
 -    case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
 +    case SSH_SERVER_KNOWN_CHANGED:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                      "host key does not match the one in known_hosts"
 -                      " (found key %s)", found->key);
 +        error_setg(errp,
 +                   "host key does not match the one in known_hosts; this "
 +                   "may be a possible attack");
          goto out;
 -    case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND:
 +    case SSH_SERVER_FOUND_OTHER:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "no host key was found in known_hosts");
 +        error_setg(errp,
 +                   "host key for this server not found, another type exists");
 +        goto out;
 +    case SSH_SERVER_FILE_NOT_FOUND:
 +        ret = -ENOENT;
 +        error_setg(errp, "known_hosts file not found");
          goto out;
 -    case LIBSSH2_KNOWNHOST_CHECK_FAILURE:
 +    case SSH_SERVER_NOT_KNOWN:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                      "failure matching the host key with known_hosts");
 +        error_setg(errp, "no host key was found in known_hosts");
 +        goto out;
 +    case SSH_SERVER_ERROR:
 +        ret = -EINVAL;
 +        error_setg(errp, "server error");
          goto out;
      default:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "unknown error matching the host key"
 -                      " with known_hosts (%d)", r);
 +        error_setg(errp, "error while checking for known server (%d)", state);
          goto out;
      }
 +#endif /* !HAVE_LIBSSH_0_8 */
      /* known_hosts checking successful. */
      ret = 0;
   out:
 -    if (knh != NULL) {
 -        libssh2_knownhost_free(knh);
 -    }
 -    g_free(knh_file);
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int compare_fingerprint(const unsigned char *fingerprint, size_t len,
  static int
  check_host_key_hash(BDRVSSHState *s, const char *hash,
 -                    int hash_type, size_t fingerprint_len, Error **errp)
 +                    enum ssh_publickey_hash_type type, Error **errp)
  {
 -    const char *fingerprint;
 -
 -    fingerprint = libssh2_hostkey_hash(s->session, hash_type);
 -    if (!fingerprint) {
 +    int r;
 +    ssh_key pubkey;
 +    unsigned char *server_hash;
 +    size_t server_hash_len;
 +
 +#ifdef HAVE_LIBSSH_0_8
 +    r = ssh_get_server_publickey(s->session, &pubkey);
 +#else
 +    r = ssh_get_publickey(s->session, &pubkey);
 +#endif
 +    if (r != SSH_OK) {
          session_error_setg(errp, s, "failed to read remote host key");
          return -EINVAL;
      }
 -    if(compare_fingerprint((unsigned char *) fingerprint, fingerprint_len,
 -                           hash) != 0) {
 +    r = ssh_get_publickey_hash(pubkey, type, &server_hash, &server_hash_len);
 +    ssh_key_free(pubkey);
 +    if (r != 0) {
 +        session_error_setg(errp, s,
 +                           "failed reading the hash of the server SSH key");
 +        return -EINVAL;
 +    }
 +
 +    r = compare_fingerprint(server_hash, server_hash_len, hash);
 +    ssh_clean_pubkey_hash(&server_hash);
 +    if (r != 0) {
          error_setg(errp, "remote host key does not match host_key_check '%s'",
                     hash);
          return -EPERM;
@@ -XXX,XX +XXX,XX @@ check_host_key_hash(BDRVSSHState *s, const char *hash,
      return 0;
  }
 -static int check_host_key(BDRVSSHState *s, const char *host, int port,
 -                          SshHostKeyCheck *hkc, Error **errp)
 +static int check_host_key(BDRVSSHState *s, SshHostKeyCheck *hkc, Error **errp)
  {
      SshHostKeyCheckMode mode;
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
      case SSH_HOST_KEY_CHECK_MODE_HASH:
          if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) {
              return check_host_key_hash(s, hkc->u.hash.hash,
 -                                       LIBSSH2_HOSTKEY_HASH_MD5, 16, errp);
 +                                       SSH_PUBLICKEY_HASH_MD5, errp);
          } else if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_SHA1) {
              return check_host_key_hash(s, hkc->u.hash.hash,
 -                                       LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp);
 +                                       SSH_PUBLICKEY_HASH_SHA1, errp);
          }
--        return notifier_with_return_list_notify(&bs->before_write_notifiers,
+         g_assert_not_reached();
--                                                req);
+         break;
-+        bdrv_write_threshold_check_write(bs, offset, bytes);
+     case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS:
-+        return 0;
+-        return check_host_key_knownhosts(s, host, port, errp);
-     case BDRV_TRACKED_TRUNCATE:
++        return check_host_key_knownhosts(s, errp);
-         assert(child->perm & BLK_PERM_RESIZE);
+     default:
-         return 0;
+         g_assert_not_reached();
-diff --git a/block/write-threshold.c b/block/write-threshold.c
+     }
-index XXXXXXX..XXXXXXX 100644
+@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
---- a/block/write-threshold.c
+     return -EINVAL;
 +++ b/block/write-threshold.c
@@ -XXX,XX +XXX,XX @@ bool bdrv_write_threshold_is_set(const BlockDriverState *bs)
      return bs->write_threshold_offset > 0;
  }
--static void write_threshold_disable(BlockDriverState *bs)
+-static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
 +static int authenticate(BDRVSSHState *s, Error **errp)
  {
      int r, ret;
 -    const char *userauthlist;
 -    LIBSSH2_AGENT *agent = NULL;
 -    struct libssh2_agent_publickey *identity;
 -    struct libssh2_agent_publickey *prev_identity = NULL;
 +    int method;
 -    userauthlist = libssh2_userauth_list(s->session, user, strlen(user));
 -    if (strstr(userauthlist, "publickey") == NULL) {
 +    /* Try to authenticate with the "none" method. */
 +    r = ssh_userauth_none(s->session, NULL);
 +    if (r == SSH_AUTH_ERROR) {
          ret = -EPERM;
 -        error_setg(errp,
 -                "remote server does not support \"publickey\" authentication");
 +        session_error_setg(errp, s, "failed to authenticate using none "
 +                                    "authentication");
          goto out;
 -    }
 -
 -    /* Connect to ssh-agent and try each identity in turn. */
 -    agent = libssh2_agent_init(s->session);
 -    if (!agent) {
 -        ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to initialize ssh-agent support");
 -        goto out;
 -    }
 -    if (libssh2_agent_connect(agent)) {
 -        ret = -ECONNREFUSED;
 -        session_error_setg(errp, s, "failed to connect to ssh-agent");
 -        goto out;
 -    }
 -    if (libssh2_agent_list_identities(agent)) {
 -        ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                           "failed requesting identities from ssh-agent");
 +    } else if (r == SSH_AUTH_SUCCESS) {
 +        /* Authenticated! */
 +        ret = 0;
          goto out;
      }
 -    for(;;) {
 -        r = libssh2_agent_get_identity(agent, &identity, prev_identity);
 -        if (r == 1) {           /* end of list */
 -            break;
 -        }
 -        if (r < 0) {
 +    method = ssh_userauth_list(s->session, NULL);
 +    trace_ssh_auth_methods(method);
 +
 +    /*
 +     * Try to authenticate with publickey, using the ssh-agent
 +     * if available.
 +     */
 +    if (method & SSH_AUTH_METHOD_PUBLICKEY) {
 +        r = ssh_userauth_publickey_auto(s->session, NULL, NULL);
 +        if (r == SSH_AUTH_ERROR) {
              ret = -EINVAL;
 -            session_error_setg(errp, s,
 -                               "failed to obtain identity from ssh-agent");
 +            session_error_setg(errp, s, "failed to authenticate using "
 +                                        "publickey authentication");
              goto out;
 -        }
 -        r = libssh2_agent_userauth(agent, user, identity);
 -        if (r == 0) {
 +        } else if (r == SSH_AUTH_SUCCESS) {
              /* Authenticated! */
              ret = 0;
              goto out;
          }
 -        /* Failed to authenticate with this identity, try the next one. */
 -        prev_identity = identity;
      }
      ret = -EPERM;
@@ -XXX,XX +XXX,XX @@ static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
                 "and the identities held by your ssh-agent");
   out:
 -    if (agent != NULL) {
 -        /* Note: libssh2 implementation implicitly calls
 -         * libssh2_agent_disconnect if necessary.
 -         */
 -        libssh2_agent_free(agent);
 -    }
 -
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
                            int ssh_flags, int creat_mode, Error **errp)
  {
      int r, ret;
 -    long port = 0;
 +    unsigned int port = 0;
 +    int new_sock = -1;
      if (opts->has_user) {
          s->user = g_strdup(opts->user);
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
      s->inet = opts->server;
      opts->server = NULL;
 -    if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) {
 +    if (qemu_strtoui(s->inet->port, NULL, 10, &port) < 0) {
          error_setg(errp, "Use only numeric port value");
          ret = -EINVAL;
          goto err;
      }
      /* Open the socket and connect. */
 -    s->sock = inet_connect_saddr(s->inet, errp);
 -    if (s->sock < 0) {
 +    new_sock = inet_connect_saddr(s->inet, errp);
 +    if (new_sock < 0) {
          ret = -EIO;
          goto err;
      }
 +    /*
 +     * Try to disable the Nagle algorithm on TCP sockets to reduce latency,
 +     * but do not fail if it cannot be disabled.
 +     */
 +    r = socket_set_nodelay(new_sock);
 +    if (r < 0) {
 +        warn_report("can't set TCP_NODELAY for the ssh server %s: %s",
 +                    s->inet->host, strerror(errno));
 +    }
 +
      /* Create SSH session. */
 -    s->session = libssh2_session_init();
 +    s->session = ssh_new();
      if (!s->session) {
          ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to initialize libssh2 session");
 +        session_error_setg(errp, s, "failed to initialize libssh session");
          goto err;
      }
 -#if TRACE_LIBSSH2 != 0
 -    libssh2_trace(s->session, TRACE_LIBSSH2);
 -#endif
 +    /*
 +     * Make sure we are in blocking mode during the connection and
 +     * authentication phases.
 +     */
 +    ssh_set_blocking(s->session, 1);
 -    r = libssh2_session_handshake(s->session, s->sock);
 -    if (r != 0) {
 +    r = ssh_options_set(s->session, SSH_OPTIONS_USER, s->user);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the user in the libssh session");
 +        goto err;
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_HOST, s->inet->host);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the host in the libssh session");
 +        goto err;
 +    }
 +
 +    if (port > 0) {
 +        r = ssh_options_set(s->session, SSH_OPTIONS_PORT, &port);
 +        if (r < 0) {
 +            ret = -EINVAL;
 +            session_error_setg(errp, s,
 +                               "failed to set the port in the libssh session");
 +            goto err;
 +        }
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_COMPRESSION, "none");
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to disable the compression in the libssh "
 +                           "session");
 +        goto err;
 +    }
 +
 +    /* Read ~/.ssh/config. */
 +    r = ssh_options_parse_config(s->session, NULL);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s, "failed to parse ~/.ssh/config");
 +        goto err;
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_FD, &new_sock);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the socket in the libssh session");
 +        goto err;
 +    }
 +    /* libssh took ownership of the socket. */
 +    s->sock = new_sock;
 +    new_sock = -1;
 +
 +    /* Connect. */
 +    r = ssh_connect(s->session);
 +    if (r != SSH_OK) {
          ret = -EINVAL;
          session_error_setg(errp, s, "failed to establish SSH session");
          goto err;
      }
      /* Check the remote host's key against known_hosts. */
 -    ret = check_host_key(s, s->inet->host, port, opts->host_key_check, errp);
 +    ret = check_host_key(s, opts->host_key_check, errp);
      if (ret < 0) {
          goto err;
      }
      /* Authenticate. */
 -    ret = authenticate(s, s->user, errp);
 +    ret = authenticate(s, errp);
      if (ret < 0) {
          goto err;
      }
      /* Start SFTP. */
 -    s->sftp = libssh2_sftp_init(s->session);
 +    s->sftp = sftp_new(s->session);
      if (!s->sftp) {
 -        session_error_setg(errp, s, "failed to initialize sftp handle");
 +        session_error_setg(errp, s, "failed to create sftp handle");
 +        ret = -EINVAL;
 +        goto err;
 +    }
 +
 +    r = sftp_init(s->sftp);
 +    if (r < 0) {
 +        sftp_error_setg(errp, s, "failed to initialize sftp handle");
          ret = -EINVAL;
          goto err;
      }
      /* Open the remote file. */
      trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
 -    s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
 -                                       creat_mode);
 +    s->sftp_handle = sftp_open(s->sftp, opts->path, ssh_flags, creat_mode);
      if (!s->sftp_handle) {
 -        session_error_setg(errp, s, "failed to open remote file '%s'",
 -                           opts->path);
 +        sftp_error_setg(errp, s, "failed to open remote file '%s'",
 +                        opts->path);
          ret = -EINVAL;
          goto err;
      }
 -    r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs);
 -    if (r < 0) {
 +    /* Make sure the SFTP file is handled in blocking mode. */
 +    sftp_file_set_blocking(s->sftp_handle);
 +
 +    s->attrs = sftp_fstat(s->sftp_handle);
 +    if (!s->attrs) {
          sftp_error_setg(errp, s, "failed to read file attributes");
          return -EINVAL;
      }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
      return 0;
   err:
 +    if (s->attrs) {
 +        sftp_attributes_free(s->attrs);
 +    }
 +    s->attrs = NULL;
      if (s->sftp_handle) {
 -        libssh2_sftp_close(s->sftp_handle);
 +        sftp_close(s->sftp_handle);
      }
      s->sftp_handle = NULL;
      if (s->sftp) {
 -        libssh2_sftp_shutdown(s->sftp);
 +        sftp_free(s->sftp);
      }
      s->sftp = NULL;
      if (s->session) {
 -        libssh2_session_disconnect(s->session,
 -                                   "from qemu ssh client: "
 -                                   "error opening connection");
 -        libssh2_session_free(s->session);
 +        ssh_disconnect(s->session);
 +        ssh_free(s->session);
      }
      s->session = NULL;
 +    s->sock = -1;
 +    if (new_sock >= 0) {
 +        close(new_sock);
 +    }
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
      ssh_state_init(s);
 -    ssh_flags = LIBSSH2_FXF_READ;
 +    ssh_flags = 0;
      if (bdrv_flags & BDRV_O_RDWR) {
 -        ssh_flags |= LIBSSH2_FXF_WRITE;
 +        ssh_flags |= O_RDWR;
 +    } else {
 +        ssh_flags |= O_RDONLY;
      }
      opts = ssh_parse_options(options, errp);
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
      }
      /* Go non-blocking. */
 -    libssh2_session_set_blocking(s->session, 0);
 +    ssh_set_blocking(s->session, 0);
      qapi_free_BlockdevOptionsSsh(opts);
      return 0;
   err:
 -    if (s->sock >= 0) {
 -        close(s->sock);
 -    }
 -    s->sock = -1;
 -
      qapi_free_BlockdevOptionsSsh(opts);
      return ret;
@@ -XXX,XX +XXX,XX @@ static int ssh_grow_file(BDRVSSHState *s, int64_t offset, Error **errp)
  {
      ssize_t ret;
      char c[1] = { '\0' };
 -    int was_blocking = libssh2_session_get_blocking(s->session);
 +    int was_blocking = ssh_is_blocking(s->session);
      /* offset must be strictly greater than the current size so we do
       * not overwrite anything */
 -    assert(offset > 0 && offset > s->attrs.filesize);
 +    assert(offset > 0 && offset > s->attrs->size);
 -    libssh2_session_set_blocking(s->session, 1);
 +    ssh_set_blocking(s->session, 1);
 -    libssh2_sftp_seek64(s->sftp_handle, offset - 1);
 -    ret = libssh2_sftp_write(s->sftp_handle, c, 1);
 +    sftp_seek64(s->sftp_handle, offset - 1);
 +    ret = sftp_write(s->sftp_handle, c, 1);
 -    libssh2_session_set_blocking(s->session, was_blocking);
 +    ssh_set_blocking(s->session, was_blocking);
      if (ret < 0) {
          sftp_error_setg(errp, s, "Failed to grow file");
          return -EIO;
      }
 -    s->attrs.filesize = offset;
 +    s->attrs->size = offset;
      return 0;
  }
@@ -XXX,XX +XXX,XX @@ static int ssh_co_create(BlockdevCreateOptions *options, Error **errp)
      ssh_state_init(&s);
      ret = connect_to_ssh(&s, opts->location,
 -                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
 -                         LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
 +                         O_RDWR | O_CREAT | O_TRUNC,
 , errp);
      if (ret < 0) {
          goto fail;
@@ -XXX,XX +XXX,XX @@ static int ssh_has_zero_init(BlockDriverState *bs)
      /* Assume false, unless we can positively prove it's true. */
      int has_zero_init = 0;
 -    if (s->attrs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) {
 -        if (s->attrs.permissions & LIBSSH2_SFTP_S_IFREG) {
 -            has_zero_init = 1;
 -        }
 +    if (s->attrs->type == SSH_FILEXFER_TYPE_REGULAR) {
 +        has_zero_init = 1;
      }
      return has_zero_init;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
          .co = qemu_coroutine_self()
      };
 -    r = libssh2_session_block_directions(s->session);
 +    r = ssh_get_poll_flags(s->session);
 -    if (r & LIBSSH2_SESSION_BLOCK_INBOUND) {
 +    if (r & SSH_READ_PENDING) {
          rd_handler = restart_coroutine;
      }
 -    if (r & LIBSSH2_SESSION_BLOCK_OUTBOUND) {
 +    if (r & SSH_WRITE_PENDING) {
          wr_handler = restart_coroutine;
      }
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
      trace_ssh_co_yield_back(s->sock);
  }
 -/* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
 - * in the remote file.  Notice that it just updates a field in the
 - * sftp_handle structure, so there is no network traffic and it cannot
 - * fail.
 - *
 - * However, `libssh2_sftp_seek64' does have a catastrophic effect on
 - * performance since it causes the handle to throw away all in-flight
 - * reads and buffered readahead data.  Therefore this function tries
 - * to be intelligent about when to call the underlying libssh2 function.
 - */
 -#define SSH_SEEK_WRITE 0
 -#define SSH_SEEK_READ  1
 -#define SSH_SEEK_FORCE 2
 -
 -static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
 -{
--    if (bdrv_write_threshold_is_set(bs)) {
+-    bool op_read = (flags & SSH_SEEK_READ) != 0;
--        notifier_with_return_remove(&bs->write_threshold_notifier);
+-    bool force = (flags & SSH_SEEK_FORCE) != 0;
--        bs->write_threshold_offset = 0;
+-
 -    if (force || op_read != s->offset_op_read || offset != s->offset) {
 -        trace_ssh_seek(offset);
 -        libssh2_sftp_seek64(s->sftp_handle, offset);
 -        s->offset = offset;
 -        s->offset_op_read = op_read;
 -    }
 -}
 -
- uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
+ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
-                                        const BdrvTrackedRequest *req)
+                                  int64_t offset, size_t size,
                                   QEMUIOVector *qiov)
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
      trace_ssh_read(offset, size);
 -    ssh_seek(s, offset, SSH_SEEK_READ);
 +    trace_ssh_seek(offset);
 +    sftp_seek64(s->sftp_handle, offset);
      /* This keeps track of the current iovec element ('i'), where we
       * will write to next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
      buf = i->iov_base;
      end_of_vec = i->iov_base + i->iov_len;
 -    /* libssh2 has a hard-coded limit of 2000 bytes per request,
 -     * although it will also do readahead behind our backs.  Therefore
 -     * we may have to do repeated reads here until we have read 'size'
 -     * bytes.
 -     */
      for (got = 0; got < size; ) {
 +        size_t request_read_size;
      again:
 -        trace_ssh_read_buf(buf, end_of_vec - buf);
 -        r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
 -        trace_ssh_read_return(r);
 +        /*
 +         * The size of SFTP packets is limited to 32K bytes, so limit
 +         * the amount of data requested to 16K, as libssh currently
 +         * does not handle multiple requests on its own.
 +         */
 +        request_read_size = MIN(end_of_vec - buf, 16384);
 +        trace_ssh_read_buf(buf, end_of_vec - buf, request_read_size);
 +        r = sftp_read(s->sftp_handle, buf, request_read_size);
 +        trace_ssh_read_return(r, sftp_get_error(s->sftp));
 -        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +        if (r == SSH_AGAIN) {
              co_yield(s, bs);
              goto again;
          }
 -        if (r < 0) {
 -            sftp_error_trace(s, "read");
 -            s->offset = -1;
 -            return -EIO;
 -        }
 -        if (r == 0) {
 +        if (r == SSH_EOF || (r == 0 && sftp_get_error(s->sftp) == SSH_FX_EOF)) {
              /* EOF: Short read so pad the buffer with zeroes and return it. */
              qemu_iovec_memset(qiov, got, 0, size - got);
              return 0;
          }
 +        if (r <= 0) {
 +            sftp_error_trace(s, "read");
 +            return -EIO;
 +        }
          got += r;
          buf += r;
 -        s->offset += r;
          if (buf >= end_of_vec && got < size) {
              i++;
              buf = i->iov_base;
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
      trace_ssh_write(offset, size);
 -    ssh_seek(s, offset, SSH_SEEK_WRITE);
 +    trace_ssh_seek(offset);
 +    sftp_seek64(s->sftp_handle, offset);
      /* This keeps track of the current iovec element ('i'), where we
       * will read from next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
      end_of_vec = i->iov_base + i->iov_len;
      for (written = 0; written < size; ) {
 +        size_t request_write_size;
      again:
 -        trace_ssh_write_buf(buf, end_of_vec - buf);
 -        r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
 -        trace_ssh_write_return(r);
 +        /*
 +         * Avoid too large data packets, as libssh currently does not
 +         * handle multiple requests on its own.
 +         */
 +        request_write_size = MIN(end_of_vec - buf, 131072);
 +        trace_ssh_write_buf(buf, end_of_vec - buf, request_write_size);
 +        r = sftp_write(s->sftp_handle, buf, request_write_size);
 +        trace_ssh_write_return(r, sftp_get_error(s->sftp));
 -        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +        if (r == SSH_AGAIN) {
              co_yield(s, bs);
              goto again;
          }
          if (r < 0) {
              sftp_error_trace(s, "write");
 -            s->offset = -1;
              return -EIO;
          }
 -        /* The libssh2 API is very unclear about this.  A comment in
 -         * the code says "nothing was acked, and no EAGAIN was
 -         * received!" which apparently means that no data got sent
 -         * out, and the underlying channel didn't return any EAGAIN
 -         * indication.  I think this is a bug in either libssh2 or
 -         * OpenSSH (server-side).  In any case, forcing a seek (to
 -         * discard libssh2 internal buffers), and then trying again
 -         * works for me.
 -         */
 -        if (r == 0) {
 -            ssh_seek(s, offset + written, SSH_SEEK_WRITE|SSH_SEEK_FORCE);
 -            co_yield(s, bs);
 -            goto again;
 -        }
          written += r;
          buf += r;
 -        s->offset += r;
          if (buf >= end_of_vec && written < size) {
              i++;
              buf = i->iov_base;
              end_of_vec = i->iov_base + i->iov_len;
          }
 -        if (offset + written > s->attrs.filesize)
 -            s->attrs.filesize = offset + written;
 +        if (offset + written > s->attrs->size) {
 +            s->attrs->size = offset + written;
 +        }
      }
      return 0;
@@ -XXX,XX +XXX,XX @@ static void unsafe_flush_warning(BDRVSSHState *s, const char *what)
      }
  }
 -#ifdef HAS_LIBSSH2_SFTP_FSYNC
 +#ifdef HAVE_LIBSSH_0_8
  static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
  {
-@@ -XXX,XX +XXX,XX @@ uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
+     int r;
      trace_ssh_flush();
 +
 +    if (!sftp_extension_supported(s->sftp, "fsync@openssh.com", "1")) {
 +        unsafe_flush_warning(s, "OpenSSH >= 6.3");
 +        return 0;
 +    }
   again:
 -    r = libssh2_sftp_fsync(s->sftp_handle);
 -    if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +    r = sftp_fsync(s->sftp_handle);
 +    if (r == SSH_AGAIN) {
          co_yield(s, bs);
          goto again;
      }
 -    if (r == LIBSSH2_ERROR_SFTP_PROTOCOL &&
 -        libssh2_sftp_last_error(s->sftp) == LIBSSH2_FX_OP_UNSUPPORTED) {
 -        unsafe_flush_warning(s, "OpenSSH >= 6.3");
 -        return 0;
 -    }
      if (r < 0) {
          sftp_error_trace(s, "fsync");
          return -EIO;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
      return ret;
  }
 -#else /* !HAS_LIBSSH2_SFTP_FSYNC */
 +#else /* !HAVE_LIBSSH_0_8 */
  static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
  {
      BDRVSSHState *s = bs->opaque;
 -    unsafe_flush_warning(s, "libssh2 >= 1.4.4");
 +    unsafe_flush_warning(s, "libssh >= 0.8.0");
      return 0;
  }
--static int coroutine_fn before_write_notify(NotifierWithReturn *notifier,
+-#endif /* !HAS_LIBSSH2_SFTP_FSYNC */
--                                            void *opaque)
++#endif /* !HAVE_LIBSSH_0_8 */
--{
--    BdrvTrackedRequest *req = opaque;
+ static int64_t ssh_getlength(BlockDriverState *bs)
--    BlockDriverState *bs = req->bs;
+ {
--    uint64_t amount = 0;
+     BDRVSSHState *s = bs->opaque;
      int64_t length;
 -    /* Note we cannot make a libssh2 call here. */
 -    length = (int64_t) s->attrs.filesize;
 +    /* Note we cannot make a libssh call here. */
 +    length = (int64_t) s->attrs->size;
      trace_ssh_getlength(length);
      return length;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_truncate(BlockDriverState *bs, int64_t offset,
          return -ENOTSUP;
      }
 -    if (offset < s->attrs.filesize) {
 +    if (offset < s->attrs->size) {
          error_setg(errp, "ssh driver does not support shrinking files");
          return -ENOTSUP;
      }
 -    if (offset == s->attrs.filesize) {
 +    if (offset == s->attrs->size) {
          return 0;
      }
@@ -XXX,XX +XXX,XX @@ static void bdrv_ssh_init(void)
  {
      int r;
 -    r = libssh2_init(0);
 +    r = ssh_init();
      if (r != 0) {
 -        fprintf(stderr, "libssh2 initialization failed, %d\n", r);
 +        fprintf(stderr, "libssh initialization failed, %d\n", r);
          exit(EXIT_FAILURE);
      }
 +#if TRACE_LIBSSH != 0
 +    ssh_set_log_level(TRACE_LIBSSH);
 +#endif
 +
      bdrv_register(&bdrv_ssh);
  }
 diff --git a/.travis.yml b/.travis.yml
 index XXXXXXX..XXXXXXX 100644
 --- a/.travis.yml
 +++ b/.travis.yml
@@ -XXX,XX +XXX,XX @@ addons:
        - libseccomp-dev
        - libspice-protocol-dev
        - libspice-server-dev
 -      - libssh2-1-dev
 +      - libssh-dev
        - liburcu-dev
        - libusb-1.0-0-dev
        - libvte-2.91-dev
@@ -XXX,XX +XXX,XX @@ matrix:
              - libseccomp-dev
              - libspice-protocol-dev
              - libspice-server-dev
 -            - libssh2-1-dev
 +            - libssh-dev
              - liburcu-dev
              - libusb-1.0-0-dev
              - libvte-2.91-dev
 diff --git a/block/trace-events b/block/trace-events
 index XXXXXXX..XXXXXXX 100644
 --- a/block/trace-events
 +++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ nbd_client_connect_success(const char *export_name) "export '%s'"
  # ssh.c
  ssh_restart_coroutine(void *co) "co=%p"
  ssh_flush(void) "fsync"
 -ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
 +ssh_check_host_key_knownhosts(void) "host key OK"
  ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
  ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
  ssh_co_yield_back(int sock) "s->sock=%d - back"
  ssh_getlength(int64_t length) "length=%" PRIi64
  ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
  ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
 -ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
 -ssh_read_return(ssize_t ret) "sftp_read returned %zd"
 +ssh_read_buf(void *buf, size_t size, size_t actual_size) "sftp_read buf=%p size=%zu (actual size=%zu)"
 +ssh_read_return(ssize_t ret, int sftp_err) "sftp_read returned %zd (sftp error=%d)"
  ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
 -ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
 -ssh_write_return(ssize_t ret) "sftp_write returned %zd"
 +ssh_write_buf(void *buf, size_t size, size_t actual_size) "sftp_write buf=%p size=%zu (actual size=%zu)"
 +ssh_write_return(ssize_t ret, int sftp_err) "sftp_write returned %zd (sftp error=%d)"
  ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
 +ssh_auth_methods(int methods) "auth methods=0x%x"
 +ssh_server_status(int status) "server status=%d"
  # curl.c
  curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
@@ -XXX,XX +XXX,XX @@ sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
  sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
  # ssh.c
 -sftp_error(const char *op, const char *ssh_err, int ssh_err_code, unsigned long sftp_err_code) "%s failed: %s (libssh2 error code: %d, sftp error code: %lu)"
 +sftp_error(const char *op, const char *ssh_err, int ssh_err_code, int sftp_err_code) "%s failed: %s (libssh error code: %d, sftp error code: %d)"
 diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi
 index XXXXXXX..XXXXXXX 100644
 --- a/docs/qemu-block-drivers.texi
 +++ b/docs/qemu-block-drivers.texi
@@ -XXX,XX +XXX,XX @@ print a warning when @code{fsync} is not supported:
  warning: ssh server @code{ssh.example.com:22} does not support fsync
 -With sufficiently new versions of libssh2 and OpenSSH, @code{fsync} is
 +With sufficiently new versions of libssh and OpenSSH, @code{fsync} is
  supported.
  @node disk_images_nvme
 diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/debian-win32-cross.docker
 +++ b/tests/docker/dockerfiles/debian-win32-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
          mxe-$TARGET-w64-mingw32.shared-curl \
          mxe-$TARGET-w64-mingw32.shared-glib \
          mxe-$TARGET-w64-mingw32.shared-libgcrypt \
 -        mxe-$TARGET-w64-mingw32.shared-libssh2 \
          mxe-$TARGET-w64-mingw32.shared-libusb1 \
          mxe-$TARGET-w64-mingw32.shared-lzo \
          mxe-$TARGET-w64-mingw32.shared-nettle \
 diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/debian-win64-cross.docker
 +++ b/tests/docker/dockerfiles/debian-win64-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
          mxe-$TARGET-w64-mingw32.shared-curl \
          mxe-$TARGET-w64-mingw32.shared-glib \
          mxe-$TARGET-w64-mingw32.shared-libgcrypt \
 -        mxe-$TARGET-w64-mingw32.shared-libssh2 \
          mxe-$TARGET-w64-mingw32.shared-libusb1 \
          mxe-$TARGET-w64-mingw32.shared-lzo \
          mxe-$TARGET-w64-mingw32.shared-nettle \
 diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/fedora.docker
 +++ b/tests/docker/dockerfiles/fedora.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
      libpng-devel \
      librbd-devel \
      libseccomp-devel \
 -    libssh2-devel \
 +    libssh-devel \
      libubsan \
      libusbx-devel \
      libxml2-devel \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
      mingw32-gtk3 \
      mingw32-libjpeg-turbo \
      mingw32-libpng \
 -    mingw32-libssh2 \
      mingw32-libtasn1 \
      mingw32-nettle \
      mingw32-pixman \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
      mingw64-gtk3 \
      mingw64-libjpeg-turbo \
      mingw64-libpng \
 -    mingw64-libssh2 \
      mingw64-libtasn1 \
      mingw64-nettle \
      mingw64-pixman \
 diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/ubuntu.docker
 +++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
      libsnappy-dev \
      libspice-protocol-dev \
      libspice-server-dev \
 -    libssh2-1-dev \
 +    libssh-dev \
      libusb-1.0-0-dev \
      libusbredirhost-dev \
      libvdeplug-dev \
 diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/docker/dockerfiles/ubuntu1804.docker
 +++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
      libsnappy-dev \
      libspice-protocol-dev \
      libspice-server-dev \
 -    libssh2-1-dev \
 +    libssh-dev \
      libusb-1.0-0-dev \
      libusbredirhost-dev \
      libvdeplug-dev \
 diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/207
 +++ b/tests/qemu-iotests/207
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
      iotests.img_info_log(remote_path)
 -    md5_key = subprocess.check_output(
 -        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
 -        'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
 -        shell=True).rstrip().decode('ascii')
 +    keys = subprocess.check_output(
 +        'ssh-keyscan 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
 +        'cut -d" " -f3',
 +        shell=True).rstrip().decode('ascii').split('\n')
 +
 +    # Mappings of base64 representations to digests
 +    md5_keys = {}
 +    sha1_keys = {}
 +
 +    for key in keys:
 +        md5_keys[key] = subprocess.check_output(
 +            'echo %s | base64 -d | md5sum -b | cut -d" " -f1' % key,
 +            shell=True).rstrip().decode('ascii')
 +
 +        sha1_keys[key] = subprocess.check_output(
 +            'echo %s | base64 -d | sha1sum -b | cut -d" " -f1' % key,
 +            shell=True).rstrip().decode('ascii')
      vm.launch()
 +
 +    # Find correct key first
 +    matching_key = None
 +    for key in keys:
 +        result = vm.qmp('blockdev-add',
 +                        driver='ssh', node_name='node0', path=disk_path,
 +                        server={
 +                             'host': '127.0.0.1',
 +                             'port': '22',
 +                        }, host_key_check={
 +                             'mode': 'hash',
 +                             'type': 'md5',
 +                             'hash': md5_keys[key],
 +                        })
 +
 +        if 'error' not in result:
 +            vm.qmp('blockdev-del', node_name='node0')
 +            matching_key = key
 +            break
 +
 +    if matching_key is None:
 +        vm.shutdown()
 +        iotests.notrun('Did not find a key that fits 127.0.0.1')
 +
      blockdev_create(vm, { 'driver': 'ssh',
                            'location': {
                                'path': disk_path,
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                                'host-key-check': {
                                    'mode': 'hash',
                                    'type': 'md5',
 -                                  'hash': md5_key,
 +                                  'hash': md5_keys[matching_key],
                                }
                            },
                            'size': 8388608 })
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
      iotests.img_info_log(remote_path)
 -    sha1_key = subprocess.check_output(
 -        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
 -        'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
 -        shell=True).rstrip().decode('ascii')
 -
--    amount = bdrv_write_threshold_exceeded(bs, req);
+     vm.launch()
--    if (amount > 0) {
+     blockdev_create(vm, { 'driver': 'ssh',
--        qapi_event_send_block_write_threshold(
+                           'location': {
--            bs->node_name,
+@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
--            amount,
+                               'host-key-check': {
--            bs->write_threshold_offset);
+                                   'mode': 'hash',
--
+                                   'type': 'sha1',
--        /* autodisable to avoid flooding the monitor */
+-                                  'hash': sha1_key,
--        write_threshold_disable(bs);
++                                  'hash': sha1_keys[matching_key],
--    }
+                               }
--
+                           },
--    return 0; /* should always let other notifiers run */
+                           'size': 4194304 })
--}
+diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
--
+index XXXXXXX..XXXXXXX 100644
--static void write_threshold_register_notifier(BlockDriverState *bs)
+--- a/tests/qemu-iotests/207.out
--{
++++ b/tests/qemu-iotests/207.out
--    bs->write_threshold_notifier.notify = before_write_notify;
+@@ -XXX,XX +XXX,XX @@ virtual size: 4 MiB (4194304 bytes)
--    bdrv_add_before_write_notifier(bs, &bs->write_threshold_notifier);
--}
+ {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
--
+ {"return": {}}
--static void write_threshold_update(BlockDriverState *bs,
+-Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
--                                   int64_t threshold_bytes)
++Job failed: failed to open remote file '/this/is/not/an/existing/path': SFTP server: No such file (libssh error code: 1, sftp error code: 2)
--{
+ {"execute": "job-dismiss", "arguments": {"id": "job0"}}
--    bs->write_threshold_offset = threshold_bytes;
+ {"return": {}}
--}
 -
  void bdrv_write_threshold_set(BlockDriverState *bs, uint64_t threshold_bytes)
  {
 -    if (bdrv_write_threshold_is_set(bs)) {
 -        if (threshold_bytes > 0) {
 -            write_threshold_update(bs, threshold_bytes);
 -        } else {
 -            write_threshold_disable(bs);
 -        }
 -    } else {
 -        if (threshold_bytes > 0) {
 -            /* avoid multiple registration */
 -            write_threshold_register_notifier(bs);
 -            write_threshold_update(bs, threshold_bytes);
 -        }
 -        /* discard bogus disable request */
 -    }
 +    bs->write_threshold_offset = threshold_bytes;
  }
  void qmp_block_set_write_threshold(const char *node_name,
@@ -XXX,XX +XXX,XX @@ void qmp_block_set_write_threshold(const char *node_name,
      aio_context_release(aio_context);
  }
 +
 +void bdrv_write_threshold_check_write(BlockDriverState *bs, int64_t offset,
 +                                      int64_t bytes)
 +{
 +    int64_t end = offset + bytes;
 +    uint64_t wtr = bs->write_threshold_offset;
 +
 +    if (wtr > 0 && end > wtr) {
 +        qapi_event_send_block_write_threshold(bs->node_name, end - wtr, wtr);
 +
 +        /* autodisable to avoid flooding the monitor */
 +        bdrv_write_threshold_set(bs, 0);
 +    }
 +}
 --
-.31.1
+.21.0

-[PULL 12/19] qemu-iotests: fix pylint 2.8 consider-using-with error
+[Qemu-devel] [PULL v2 8/8] iotests: Fix 205 for concurrent runs
-From: Emanuele Giuseppe Esposito <eesposit@redhat.com>
+Tests should place their files into the test directory.  This includes
 Unix sockets.  205 currently fails to do so, which prevents it from
 being run concurrently.
-pylint 2.8 introduces consider-using-with error, suggesting
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-to use the 'with' block statement when possible.
+Message-id: 20190618210238.9524-1-mreitz@redhat.com
+Reviewed-by: Eric Blake <eblake@redhat.com>
 Modify all subprocess.Popen call to use the 'with' statement,
 except the one in __init__ of QemuIoInteractive class, since
 it is assigned to a class field and used in other methods.
 Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
 Message-Id: <20210510190449.65948-1-eesposit@redhat.com>
 [mreitz: Disable bad-option-value warning in the iotests' pylintrc, so
          that disabling consider-using-with in QemuIoInteractive will
          not produce a warning in pre-2.8 pylint versions]
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/iotests.py    | 65 ++++++++++++++++----------------
+ tests/qemu-iotests/205 | 2 +-
- tests/qemu-iotests/pylintrc      |  3 ++
+file changed, 1 insertion(+), 1 deletion(-)
  tests/qemu-iotests/testrunner.py | 22 +++++------
 files changed, 47 insertions(+), 43 deletions(-)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
+diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
-index XXXXXXX..XXXXXXX 100644
+index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/iotests.py
+--- a/tests/qemu-iotests/205
-+++ b/tests/qemu-iotests/iotests.py
++++ b/tests/qemu-iotests/205
-@@ -XXX,XX +XXX,XX @@ def qemu_tool_pipe_and_status(tool: str, args: Sequence[str],
+@@ -XXX,XX +XXX,XX @@ import iotests
-     Run a tool and return both its output and its exit code
+ import time
-     """
+ from iotests import qemu_img_create, qemu_io, filter_qemu_io, QemuIoInteractive
-     stderr = subprocess.STDOUT if connect_stderr else None
--    subp = subprocess.Popen(args,
+-nbd_sock = 'nbd_sock'
--                            stdout=subprocess.PIPE,
++nbd_sock = os.path.join(iotests.test_dir, 'nbd_sock')
--                            stderr=stderr,
+ nbd_uri = 'nbd+unix:///exp?socket=' + nbd_sock
--                            universal_newlines=True)
+ disk = os.path.join(iotests.test_dir, 'disk')
 -    output = subp.communicate()[0]
 -    if subp.returncode < 0:
 -        cmd = ' '.join(args)
 -        sys.stderr.write(f'{tool} received signal {-subp.returncode}: {cmd}\n')
 -    return (output, subp.returncode)
 +    with subprocess.Popen(args, stdout=subprocess.PIPE,
 +                          stderr=stderr, universal_newlines=True) as subp:
 +        output = subp.communicate()[0]
 +        if subp.returncode < 0:
 +            cmd = ' '.join(args)
 +            sys.stderr.write(f'{tool} received signal \
 +                               {-subp.returncode}: {cmd}\n')
 +        return (output, subp.returncode)
  def qemu_img_pipe_and_status(*args: str) -> Tuple[str, int]:
      """
@@ -XXX,XX +XXX,XX @@ def qemu_io_silent_check(*args):
  class QemuIoInteractive:
      def __init__(self, *args):
          self.args = qemu_io_args_no_fmt + list(args)
 +        # We need to keep the Popen objext around, and not
 +        # close it immediately. Therefore, disable the pylint check:
 +        # pylint: disable=consider-using-with
          self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE,
                                     stdout=subprocess.PIPE,
                                     stderr=subprocess.STDOUT,
@@ -XXX,XX +XXX,XX @@ def qemu_nbd_popen(*args):
      cmd.extend(args)
      log('Start NBD server')
 -    p = subprocess.Popen(cmd)
 -    try:
 -        while not os.path.exists(pid_file):
 -            if p.poll() is not None:
 -                raise RuntimeError(
 -                    "qemu-nbd terminated with exit code {}: {}"
 -                    .format(p.returncode, ' '.join(cmd)))
 -
 -            time.sleep(0.01)
 -        yield
 -    finally:
 -        if os.path.exists(pid_file):
 -            os.remove(pid_file)
 -        log('Kill NBD server')
 -        p.kill()
 -        p.wait()
 +    with subprocess.Popen(cmd) as p:
 +        try:
 +            while not os.path.exists(pid_file):
 +                if p.poll() is not None:
 +                    raise RuntimeError(
 +                        "qemu-nbd terminated with exit code {}: {}"
 +                        .format(p.returncode, ' '.join(cmd)))
 +
 +                time.sleep(0.01)
 +            yield
 +        finally:
 +            if os.path.exists(pid_file):
 +                os.remove(pid_file)
 +            log('Kill NBD server')
 +            p.kill()
 +            p.wait()
  def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
      '''Return True if two image files are identical'''
@@ -XXX,XX +XXX,XX @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
  def create_image(name, size):
      '''Create a fully-allocated raw image with sector markers'''
 -    file = open(name, 'wb')
 -    i = 0
 -    while i < size:
 -        sector = struct.pack('>l504xl', i // 512, i // 512)
 -        file.write(sector)
 -        i = i + 512
 -    file.close()
 +    with open(name, 'wb') as file:
 +        i = 0
 +        while i < size:
 +            sector = struct.pack('>l504xl', i // 512, i // 512)
 +            file.write(sector)
 +            i = i + 512
  def image_size(img):
      '''Return image's virtual size'''
 diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/pylintrc
 +++ b/tests/qemu-iotests/pylintrc
@@ -XXX,XX +XXX,XX @@ disable=invalid-name,
          too-many-public-methods,
          # pylint warns about Optional[] etc. as unsubscriptable in 3.9
          unsubscriptable-object,
 +        # Sometimes we need to disable a newly introduced pylint warning.
 +        # Doing so should not produce a warning in older versions of pylint.
 +        bad-option-value,
          # These are temporary, and should be removed:
          missing-docstring,
          too-many-return-statements,
 diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/testrunner.py
 +++ b/tests/qemu-iotests/testrunner.py
@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str) -> TestResult:
          t0 = time.time()
          with f_bad.open('w', encoding="utf-8") as f:
 -            proc = subprocess.Popen(args, cwd=str(f_test.parent), env=env,
 -                                    stdout=f, stderr=subprocess.STDOUT)
 -            try:
 -                proc.wait()
 -            except KeyboardInterrupt:
 -                proc.terminate()
 -                proc.wait()
 -                return TestResult(status='not run',
 -                                  description='Interrupted by user',
 -                                  interrupted=True)
 -            ret = proc.returncode
 +            with subprocess.Popen(args, cwd=str(f_test.parent), env=env,
 +                                  stdout=f, stderr=subprocess.STDOUT) as proc:
 +                try:
 +                    proc.wait()
 +                except KeyboardInterrupt:
 +                    proc.terminate()
 +                    proc.wait()
 +                    return TestResult(status='not run',
 +                                      description='Interrupted by user',
 +                                      interrupted=True)
 +                ret = proc.returncode
          elapsed = round(time.time() - t0, 1)
 --
-.31.1
+.21.0

-[PULL 14/19] block: drop write notifiers
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-They are unused now.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Message-Id: <20210506090621.11848-3-vsementsov@virtuozzo.com>
-Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- include/block/block_int.h | 12 ------------
- block.c                   |  1 -
- block/io.c                |  6 ------
-files changed, 19 deletions(-)
-diff --git a/include/block/block_int.h b/include/block/block_int.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block_int.h
-+++ b/include/block/block_int.h
-@@ -XXX,XX +XXX,XX @@ struct BlockDriverState {
-      */
-     int64_t total_sectors;
--    /* Callback before write request is processed */
--    NotifierWithReturnList before_write_notifiers;
--
-     /* threshold limit for writes, in bytes. "High water mark". */
-     uint64_t write_threshold_offset;
-@@ -XXX,XX +XXX,XX @@ void bdrv_parse_filename_strip_prefix(const char *filename, const char *prefix,
- bool bdrv_backing_overridden(BlockDriverState *bs);
--/**
-- * bdrv_add_before_write_notifier:
-- *
-- * Register a callback that is invoked before write requests are processed but
-- * after any throttling or waiting for overlapping requests.
-- */
--void bdrv_add_before_write_notifier(BlockDriverState *bs,
--                                    NotifierWithReturn *notifier);
--
- /**
-  * bdrv_add_aio_context_notifier:
-  *
-diff --git a/block.c b/block.c
-index XXXXXXX..XXXXXXX 100644
---- a/block.c
-+++ b/block.c
-@@ -XXX,XX +XXX,XX @@ BlockDriverState *bdrv_new(void)
-     for (i = 0; i < BLOCK_OP_TYPE_MAX; i++) {
-         QLIST_INIT(&bs->op_blockers[i]);
-     }
--    notifier_with_return_list_init(&bs->before_write_notifiers);
-     qemu_co_mutex_init(&bs->reqs_lock);
-     qemu_mutex_init(&bs->dirty_bitmap_mutex);
-     bs->refcnt = 1;
-diff --git a/block/io.c b/block/io.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/io.c
-+++ b/block/io.c
-@@ -XXX,XX +XXX,XX @@ bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov)
-     return true;
- }
--void bdrv_add_before_write_notifier(BlockDriverState *bs,
--                                    NotifierWithReturn *notifier)
--{
--    notifier_with_return_list_add(&bs->before_write_notifiers, notifier);
--}
--
- void bdrv_io_plug(BlockDriverState *bs)
- {
-     BdrvChild *child;
---
-.31.1

-[PULL 15/19] test-write-threshold: rewrite test_threshold_(not_)trigger tests
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-These tests use bdrv_write_threshold_exceeded() API, which is used only
-for test (since pre-previous commit). Better is testing real API, which
-is used in block.c as well.
-So, let's call bdrv_write_threshold_check_write(), and check is
-bs->write_threshold_offset cleared or not (it's cleared iff threshold
-triggered).
-Also we get rid of BdrvTrackedRequest use here. Note, that paranoiac
-bdrv_check_request() calls were added in 8b1170012b1 to protect
-BdrvTrackedRequest. Drop them now.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Max Reitz <mreitz@redhat.com>
-Message-Id: <20210506090621.11848-4-vsementsov@virtuozzo.com>
-Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
-Signed-off-by: Max Reitz <mreitz@redhat.com>
----
- tests/unit/test-write-threshold.c | 22 ++++------------------
-file changed, 4 insertions(+), 18 deletions(-)
-diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
-index XXXXXXX..XXXXXXX 100644
---- a/tests/unit/test-write-threshold.c
-+++ b/tests/unit/test-write-threshold.c
-@@ -XXX,XX +XXX,XX @@ static void test_threshold_multi_set_get(void)
- static void test_threshold_not_trigger(void)
- {
--    uint64_t amount = 0;
-     uint64_t threshold = 4 * 1024 * 1024;
-     BlockDriverState bs;
--    BdrvTrackedRequest req;
-     memset(&bs, 0, sizeof(bs));
--    memset(&req, 0, sizeof(req));
--    req.offset = 1024;
--    req.bytes = 1024;
--
--    bdrv_check_request(req.offset, req.bytes, &error_abort);
-     bdrv_write_threshold_set(&bs, threshold);
--    amount = bdrv_write_threshold_exceeded(&bs, &req);
--    g_assert_cmpuint(amount, ==, 0);
-+    bdrv_write_threshold_check_write(&bs, 1024, 1024);
-+    g_assert_cmpuint(bdrv_write_threshold_get(&bs), ==, threshold);
- }
- static void test_threshold_trigger(void)
- {
--    uint64_t amount = 0;
-     uint64_t threshold = 4 * 1024 * 1024;
-     BlockDriverState bs;
--    BdrvTrackedRequest req;
-     memset(&bs, 0, sizeof(bs));
--    memset(&req, 0, sizeof(req));
--    req.offset = (4 * 1024 * 1024) - 1024;
--    req.bytes = 2 * 1024;
--
--    bdrv_check_request(req.offset, req.bytes, &error_abort);
-     bdrv_write_threshold_set(&bs, threshold);
--    amount = bdrv_write_threshold_exceeded(&bs, &req);
--    g_assert_cmpuint(amount, >=, 1024);
-+    bdrv_write_threshold_check_write(&bs, threshold - 1024, 2 * 1024);
-+    g_assert_cmpuint(bdrv_write_threshold_get(&bs), ==, 0);
- }
- typedef struct TestStruct {
---
-.31.1

The following changes since commit 96662996eda78c48aadddd4e76d8615c7eb72d80:

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20210513a' into staging (2021-05-14 12:03:47 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2021-05-14

for you to fetch changes up to c61ebf362d0abf288ce266845519d5a550a1d89f:

write-threshold: deal with includes (2021-05-14 16:14:10 +0200)

----------------------------------------------------------------
Block patches:
- drop block/io write notifiers
- qemu-iotests enhancements to make debugging easier
- rbd parsing fix
- HMP qemu-io fix (for iothreads)
- mirror job cancel relaxation (do not cancel in-flight requests when a
  READY mirror job is canceled with force=false)
- document qcow2's data_file and data_file_raw features
- fix iotest 297 for pylint 2.8
- block/copy-on-read refactoring

----------------------------------------------------------------
Connor Kuehl (3):
  iotests/231: Update expected deprecation message
  block/rbd: Add an escape-aware strchr helper
  Document qemu-img options data_file and data_file_raw

Emanuele Giuseppe Esposito (1):
  qemu-iotests: fix pylint 2.8 consider-using-with error

Paolo Bonzini (5):
  qemu-iotests: do not buffer the test output
  qemu-iotests: allow passing unittest.main arguments to the test
    scripts
  qemu-iotests: move command line and environment handling from
    TestRunner to TestEnv
  qemu-iotests: let "check" spawn an arbitrary test command
  qemu-iotests: fix case of SOCK_DIR already in the environment

Vladimir Sementsov-Ogievskiy (10):
  monitor: hmp_qemu_io: acquire aio contex, fix crash
  mirror: stop cancelling in-flight requests on non-force cancel in
    READY
  block/copy-on-read: use bdrv_drop_filter() and drop s->active
  block/write-threshold: don't use write notifiers
  block: drop write notifiers
  test-write-threshold: rewrite test_threshold_(not_)trigger tests
  block/write-threshold: drop extra APIs
  test-write-threshold: drop extra tests
  test-write-threshold: drop extra TestStruct structure
  write-threshold: deal with includes

-- 
2.31.1

From: Connor Kuehl <ckuehl@redhat.com>

The deprecation message in the expected output has technically been
wrong since the wrong version of a patch was applied to it. Because of
this, the test fails. Correct the expected output so that it passes.

Signed-off-by: Connor Kuehl <ckuehl@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Message-Id: <20210421212343.85524-2-ckuehl@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/231.out | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/231.out b/tests/qemu-iotests/231.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/231.out
+++ b/tests/qemu-iotests/231.out
@@ -XXX,XX +XXX,XX @@
 QA output created by 231
-qemu-img: RBD options encoded in the filename as keyvalue pairs is deprecated.  Future versions may cease to parse these options in the future.
+qemu-img: warning: RBD options encoded in the filename as keyvalue pairs is deprecated
 unable to get monitor info from DNS SRV with service name: ceph-mon
-no monitors specified to connect to.
 qemu-img: Could not open 'json:{'file.driver':'rbd','file.filename':'rbd:rbd/bogus:conf=BOGUS_CONF'}': error connecting: No such file or directory
 unable to get monitor info from DNS SRV with service name: ceph-mon
-no monitors specified to connect to.
 qemu-img: Could not open 'json:{'file.driver':'rbd','file.pool':'rbd','file.image':'bogus','file.conf':'BOGUS_CONF'}': error connecting: No such file or directory
 *** done
-- 
2.31.1

From: Connor Kuehl <ckuehl@redhat.com>

Sometimes the parser needs to further split a token it has collected
from the token input stream. Right now, it does a cursory check to see
if the relevant characters appear in the token to determine if it should
break it down further.

However, qemu_rbd_next_tok() will escape characters as it removes tokens
from the token stream and plain strchr() won't. This can make the
initial strchr() check slightly misleading since it implies
qemu_rbd_next_tok() will find the token and split on it, except the
reality is that qemu_rbd_next_tok() will pass over it if it is escaped.

Use a custom strchr to avoid mixing escaped and unescaped string
operations. Furthermore, this code is identical to how
qemu_rbd_next_tok() seeks its next token, so incorporate this custom
strchr into the body of that function to reduce duplication.

Reported-by: Han Han <hhan@redhat.com>
Fixes: https://bugzilla.redhat.com/1873913
Signed-off-by: Connor Kuehl <ckuehl@redhat.com>
Message-Id: <20210421212343.85524-3-ckuehl@redhat.com>
Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/rbd.c                | 32 +++++++++++++++++++++-----------
 tests/qemu-iotests/231     |  4 ++++
 tests/qemu-iotests/231.out |  3 +++
 3 files changed, 28 insertions(+), 11 deletions(-)

diff --git a/block/rbd.c b/block/rbd.c
index XXXXXXX..XXXXXXX 100644
--- a/block/rbd.c
+++ b/block/rbd.c
@@ -XXX,XX +XXX,XX @@ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
                             const char *keypairs, const char *secretid,
                             Error **errp);
 
+static char *qemu_rbd_strchr(char *src, char delim)
+{
+    char *p;
+
+    for (p = src; *p; ++p) {
+        if (*p == delim) {
+            return p;
+        }
+        if (*p == '\\' && p[1] != '\0') {
+            ++p;
+        }
+    }
+
+    return NULL;
+}
+
+
 static char *qemu_rbd_next_tok(char *src, char delim, char **p)
 {
     char *end;
 
     *p = NULL;
 
-    for (end = src; *end; ++end) {
-        if (*end == delim) {
-            break;
-        }
-        if (*end == '\\' && end[1] != '\0') {
-            end++;
-        }
-    }
-    if (*end == delim) {
+    end = qemu_rbd_strchr(src, delim);
+    if (end) {
         *p = end + 1;
         *end = '\0';
     }
@@ -XXX,XX +XXX,XX @@ static void qemu_rbd_parse_filename(const char *filename, QDict *options,
     qemu_rbd_unescape(found_str);
     qdict_put_str(options, "pool", found_str);
 
-    if (strchr(p, '@')) {
+    if (qemu_rbd_strchr(p, '@')) {
         image_name = qemu_rbd_next_tok(p, '@', &p);
 
         found_str = qemu_rbd_next_tok(p, ':', &p);
@@ -XXX,XX +XXX,XX @@ static void qemu_rbd_parse_filename(const char *filename, QDict *options,
         image_name = qemu_rbd_next_tok(p, ':', &p);
     }
     /* Check for namespace in the image_name */
-    if (strchr(image_name, '/')) {
+    if (qemu_rbd_strchr(image_name, '/')) {
         found_str = qemu_rbd_next_tok(image_name, '/', &image_name);
         qemu_rbd_unescape(found_str);
         qdict_put_str(options, "namespace", found_str);
diff --git a/tests/qemu-iotests/231 b/tests/qemu-iotests/231
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/231
+++ b/tests/qemu-iotests/231
@@ -XXX,XX +XXX,XX @@ _filter_conf()
 $QEMU_IMG info "json:{'file.driver':'rbd','file.filename':'rbd:rbd/bogus:conf=${BOGUS_CONF}'}" 2>&1 | _filter_conf
 $QEMU_IMG info "json:{'file.driver':'rbd','file.pool':'rbd','file.image':'bogus','file.conf':'${BOGUS_CONF}'}" 2>&1 | _filter_conf
 
+# Regression test: the qemu-img invocation is expected to fail, but it should
+# not seg fault the parser.
+$QEMU_IMG create "rbd:rbd/aa\/bb:conf=${BOGUS_CONF}" 1M 2>&1 | _filter_conf
+
 # success, all done
 echo "*** done"
 rm -f $seq.full
diff --git a/tests/qemu-iotests/231.out b/tests/qemu-iotests/231.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/231.out
+++ b/tests/qemu-iotests/231.out
@@ -XXX,XX +XXX,XX @@ unable to get monitor info from DNS SRV with service name: ceph-mon
 qemu-img: Could not open 'json:{'file.driver':'rbd','file.filename':'rbd:rbd/bogus:conf=BOGUS_CONF'}': error connecting: No such file or directory
 unable to get monitor info from DNS SRV with service name: ceph-mon
 qemu-img: Could not open 'json:{'file.driver':'rbd','file.pool':'rbd','file.image':'bogus','file.conf':'BOGUS_CONF'}': error connecting: No such file or directory
+Formatting 'rbd:rbd/aa\/bb:conf=BOGUS_CONF', fmt=raw size=1048576
+unable to get monitor info from DNS SRV with service name: ceph-mon
+qemu-img: rbd:rbd/aa\/bb:conf=BOGUS_CONF: error connecting: No such file or directory
 *** done
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Max reported the following bug:

$ ./qemu-img create -f raw src.img 1G
$ ./qemu-img create -f raw dst.img 1G

$ (echo '
   {"execute":"qmp_capabilities"}
   {"execute":"blockdev-mirror",
    "arguments":{"job-id":"mirror",
                 "device":"source",
                 "target":"target",
                 "sync":"full",
                 "filter-node-name":"mirror-top"}}
'; sleep 3; echo '
   {"execute":"human-monitor-command",
    "arguments":{"command-line":
                 "qemu-io mirror-top \"write 0 1G\""}}') \
| x86_64-softmmu/qemu-system-x86_64 \
   -qmp stdio \
   -blockdev file,node-name=source,filename=src.img \
   -blockdev file,node-name=target,filename=dst.img \
   -object iothread,id=iothr0 \
   -device virtio-blk,drive=source,iothread=iothr0

crashes:

0  raise () at /usr/lib/libc.so.6
1  abort () at /usr/lib/libc.so.6
2  error_exit
   (err=<optimized out>,
   msg=msg@entry=0x55fbb1634790 <__func__.27> "qemu_mutex_unlock_impl")
   at ../util/qemu-thread-posix.c:37
3  qemu_mutex_unlock_impl
   (mutex=mutex@entry=0x55fbb25ab6e0,
   file=file@entry=0x55fbb1636957 "../util/async.c",
   line=line@entry=650)
   at ../util/qemu-thread-posix.c:109
4  aio_context_release (ctx=ctx@entry=0x55fbb25ab680) at ../util/async.c:650
5  bdrv_do_drained_begin
   (bs=bs@entry=0x55fbb3a87000, recursive=recursive@entry=false,
   parent=parent@entry=0x0,
   ignore_bds_parents=ignore_bds_parents@entry=false,
   poll=poll@entry=true) at ../block/io.c:441
6  bdrv_do_drained_begin
   (poll=true, ignore_bds_parents=false, parent=0x0, recursive=false,
   bs=0x55fbb3a87000) at ../block/io.c:448
7  blk_drain (blk=0x55fbb26c5a00) at ../block/block-backend.c:1718
8  blk_unref (blk=0x55fbb26c5a00) at ../block/block-backend.c:498
9  blk_unref (blk=0x55fbb26c5a00) at ../block/block-backend.c:491
10 hmp_qemu_io (mon=0x7fffaf3fc7d0, qdict=<optimized out>)
   at ../block/monitor/block-hmp-cmds.c:628

man pthread_mutex_unlock
...
    EPERM  The  mutex type is PTHREAD_MUTEX_ERRORCHECK or
    PTHREAD_MUTEX_RECURSIVE, or the mutex is a robust mutex, and the
    current thread does not own the mutex.

So, thread doesn't own the mutex. And we have iothread here.

Next, note that AIO_WAIT_WHILE() documents that ctx must be acquired
exactly once by caller. But where is it acquired in the call stack?
Seems nowhere.

qemuio_command do acquire aio context.. But we need context acquired
around blk_unref() as well and actually around blk_insert_bs() too.

Let's refactor qemuio_command so that it doesn't acquire aio context
but callers do that instead. This way we can cleanly acquire aio
context in hmp_qemu_io() around all three calls.

Reported-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210423134233.51495-1-vsementsov@virtuozzo.com>
[mreitz: Fixed comment]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/monitor/block-hmp-cmds.c | 31 +++++++++++++++++++++----------
 qemu-io-cmds.c                 |  8 ++++----
 qemu-io.c                      | 17 +++++++++++++++--
 3 files changed, 40 insertions(+), 16 deletions(-)

diff --git a/block/monitor/block-hmp-cmds.c b/block/monitor/block-hmp-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/block/monitor/block-hmp-cmds.c
+++ b/block/monitor/block-hmp-cmds.c
@@ -XXX,XX +XXX,XX @@ void hmp_eject(Monitor *mon, const QDict *qdict)
 
 void hmp_qemu_io(Monitor *mon, const QDict *qdict)
 {
-    BlockBackend *blk;
+    BlockBackend *blk = NULL;
+    BlockDriverState *bs = NULL;
     BlockBackend *local_blk = NULL;
+    AioContext *ctx = NULL;
     bool qdev = qdict_get_try_bool(qdict, "qdev", false);
     const char *device = qdict_get_str(qdict, "device");
     const char *command = qdict_get_str(qdict, "command");
@@ -XXX,XX +XXX,XX @@ void hmp_qemu_io(Monitor *mon, const QDict *qdict)
     } else {
         blk = blk_by_name(device);
         if (!blk) {
-            BlockDriverState *bs = bdrv_lookup_bs(NULL, device, &err);
-            if (bs) {
-                blk = local_blk = blk_new(bdrv_get_aio_context(bs),
-                                          0, BLK_PERM_ALL);
-                ret = blk_insert_bs(blk, bs, &err);
-                if (ret < 0) {
-                    goto fail;
-                }
-            } else {
+            bs = bdrv_lookup_bs(NULL, device, &err);
+            if (!bs) {
                 goto fail;
             }
         }
     }
 
+    ctx = blk ? blk_get_aio_context(blk) : bdrv_get_aio_context(bs);
+    aio_context_acquire(ctx);
+
+    if (bs) {
+        blk = local_blk = blk_new(bdrv_get_aio_context(bs), 0, BLK_PERM_ALL);
+        ret = blk_insert_bs(blk, bs, &err);
+        if (ret < 0) {
+            goto fail;
+        }
+    }
+
     /*
      * Notably absent: Proper permission management. This is sad, but it seems
      * almost impossible to achieve without changing the semantics and thereby
@@ -XXX,XX +XXX,XX @@ void hmp_qemu_io(Monitor *mon, const QDict *qdict)
 
 fail:
     blk_unref(local_blk);
+
+    if (ctx) {
+        aio_context_release(ctx);
+    }
+
     hmp_handle_error(mon, err);
 }
 
diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -XXX,XX +XXX,XX @@ static const cmdinfo_t help_cmd = {
     .oneline    = "help for one or all commands",
 };
 
+/*
+ * Called with aio context of blk acquired. Or with qemu_get_aio_context()
+ * context acquired if blk is NULL.
+ */
 int qemuio_command(BlockBackend *blk, const char *cmd)
 {
-    AioContext *ctx;
     char *input;
     const cmdinfo_t *ct;
     char **v;
@@ -XXX,XX +XXX,XX @@ int qemuio_command(BlockBackend *blk, const char *cmd)
     if (c) {
         ct = find_command(v[0]);
         if (ct) {
-            ctx = blk ? blk_get_aio_context(blk) : qemu_get_aio_context();
-            aio_context_acquire(ctx);
             ret = command(blk, ct, c, v);
-            aio_context_release(ctx);
         } else {
             fprintf(stderr, "command \"%s\" not found\n", v[0]);
             ret = -EINVAL;
diff --git a/qemu-io.c b/qemu-io.c
index XXXXXXX..XXXXXXX 100644
--- a/qemu-io.c
+++ b/qemu-io.c
@@ -XXX,XX +XXX,XX @@ static void prep_fetchline(void *opaque)
     *fetchable= 1;
 }
 
+static int do_qemuio_command(const char *cmd)
+{
+    int ret;
+    AioContext *ctx =
+        qemuio_blk ? blk_get_aio_context(qemuio_blk) : qemu_get_aio_context();
+
+    aio_context_acquire(ctx);
+    ret = qemuio_command(qemuio_blk, cmd);
+    aio_context_release(ctx);
+
+    return ret;
+}
+
 static int command_loop(void)
 {
     int i, fetchable = 0, prompted = 0;
@@ -XXX,XX +XXX,XX @@ static int command_loop(void)
     char *input;
 
     for (i = 0; !quit_qemu_io && i < ncmdline; i++) {
-        ret = qemuio_command(qemuio_blk, cmdline[i]);
+        ret = do_qemuio_command(cmdline[i]);
         if (ret < 0) {
             last_error = ret;
         }
@@ -XXX,XX +XXX,XX @@ static int command_loop(void)
         if (input == NULL) {
             break;
         }
-        ret = qemuio_command(qemuio_blk, input);
+        ret = do_qemuio_command(input);
         g_free(input);
 
         if (ret < 0) {
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

If mirror is READY than cancel operation is not discarding the whole
result of the operation, but instead it's a documented way get a
point-in-time snapshot of source disk.

So, we should not cancel any requests if mirror is READ and
force=false. Let's fix that case.

Note, that bug that we have before this commit is not critical, as the
only .bdrv_cancel_in_flight implementation is nbd_cancel_in_flight()
and it cancels only requests waiting for reconnection, so it should be
rare case.

Fixes: 521ff8b779b11c394dbdc43f02e158dd99df308a
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210421075858.40197-1-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block_int.h | 2 +-
 include/qemu/job.h        | 2 +-
 block/backup.c            | 2 +-
 block/mirror.c            | 6 ++++--
 job.c                     | 2 +-
 tests/qemu-iotests/264    | 2 +-
 6 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -XXX,XX +XXX,XX @@ struct BlockDriver {
      * of in-flight requests, so don't waste the time if possible.
      *
      * One example usage is to avoid waiting for an nbd target node reconnect
-     * timeout during job-cancel.
+     * timeout during job-cancel with force=true.
      */
     void (*bdrv_cancel_in_flight)(BlockDriverState *bs);
 
diff --git a/include/qemu/job.h b/include/qemu/job.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/job.h
+++ b/include/qemu/job.h
@@ -XXX,XX +XXX,XX @@ struct JobDriver {
     /**
      * If the callback is not NULL, it will be invoked in job_cancel_async
      */
-    void (*cancel)(Job *job);
+    void (*cancel)(Job *job, bool force);
 
 
     /** Called when the job is freed */
diff --git a/block/backup.c b/block/backup.c
index XXXXXXX..XXXXXXX 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -XXX,XX +XXX,XX @@ static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed)
     }
 }
 
-static void backup_cancel(Job *job)
+static void backup_cancel(Job *job, bool force)
 {
     BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
 
diff --git a/block/mirror.c b/block/mirror.c
index XXXXXXX..XXXXXXX 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -XXX,XX +XXX,XX @@ static bool mirror_drained_poll(BlockJob *job)
     return !!s->in_flight;
 }
 
-static void mirror_cancel(Job *job)
+static void mirror_cancel(Job *job, bool force)
 {
     MirrorBlockJob *s = container_of(job, MirrorBlockJob, common.job);
     BlockDriverState *target = blk_bs(s->target);
 
-    bdrv_cancel_in_flight(target);
+    if (force || !job_is_ready(job)) {
+        bdrv_cancel_in_flight(target);
+    }
 }
 
 static const BlockJobDriver mirror_job_driver = {
diff --git a/job.c b/job.c
index XXXXXXX..XXXXXXX 100644
--- a/job.c
+++ b/job.c
@@ -XXX,XX +XXX,XX @@ static int job_finalize_single(Job *job)
 static void job_cancel_async(Job *job, bool force)
 {
     if (job->driver->cancel) {
-        job->driver->cancel(job);
+        job->driver->cancel(job, force);
     }
     if (job->user_paused) {
         /* Do not call job_enter here, the caller will handle it.  */
diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/264
+++ b/tests/qemu-iotests/264
@@ -XXX,XX +XXX,XX @@ class TestNbdReconnect(iotests.QMPTestCase):
             self.assert_qmp(result, 'return', {})
 
     def cancel_job(self):
-        result = self.vm.qmp('block-job-cancel', device='drive0')
+        result = self.vm.qmp('block-job-cancel', device='drive0', force=True)
         self.assert_qmp(result, 'return', {})
 
         start_t = time.time()
-- 
2.31.1

From: Paolo Bonzini <pbonzini@redhat.com>

Instead of buffering the test output into a StringIO, patch it on
the fly by wrapping sys.stdout's write method.  This can be
done unconditionally, even if using -d, which makes execute_unittest
a bit simpler.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Message-Id: <20210323181928.311862-2-pbonzini@redhat.com>
Message-Id: <20210503110110.476887-2-pbonzini@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/240.out    |  8 ++--
 tests/qemu-iotests/245.out    |  8 ++--
 tests/qemu-iotests/295.out    |  6 +--
 tests/qemu-iotests/296.out    |  8 ++--
 tests/qemu-iotests/iotests.py | 70 ++++++++++++++++++++---------------
 5 files changed, 56 insertions(+), 44 deletions(-)

diff --git a/tests/qemu-iotests/240.out b/tests/qemu-iotests/240.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/240.out
+++ b/tests/qemu-iotests/240.out
@@ -XXX,XX +XXX,XX @@
 {"return": {}}
 {"execute": "blockdev-del", "arguments": {"node-name": "hd0"}}
 {"return": {}}
-==Attach two SCSI disks using the same block device and the same iothread==
+.==Attach two SCSI disks using the same block device and the same iothread==
 {"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "hd0", "read-only": true, "read-zeroes": true}}
 {"return": {}}
 {"execute": "object-add", "arguments": {"id": "iothread0", "qom-type": "iothread"}}
@@ -XXX,XX +XXX,XX @@
 {"return": {}}
 {"execute": "blockdev-del", "arguments": {"node-name": "hd0"}}
 {"return": {}}
-==Attach two SCSI disks using the same block device but different iothreads==
+.==Attach two SCSI disks using the same block device but different iothreads==
 {"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "hd0", "read-only": true, "read-zeroes": true}}
 {"return": {}}
 {"execute": "object-add", "arguments": {"id": "iothread0", "qom-type": "iothread"}}
@@ -XXX,XX +XXX,XX @@
 {"return": {}}
 {"execute": "blockdev-del", "arguments": {"node-name": "hd0"}}
 {"return": {}}
-==Attach a SCSI disks using the same block device as a NBD server==
+.==Attach a SCSI disks using the same block device as a NBD server==
 {"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "hd0", "read-only": true, "read-zeroes": true}}
 {"return": {}}
 {"execute": "nbd-server-start", "arguments": {"addr": {"data": {"path": "SOCK_DIR/PID-nbd.sock"}, "type": "unix"}}}
@@ -XXX,XX +XXX,XX @@
 {"return": {}}
 {"execute": "device_add", "arguments": {"drive": "hd0", "driver": "scsi-hd", "id": "scsi-hd0"}}
 {"return": {}}
-....
+.
 ----------------------------------------------------------------------
 Ran 4 tests
 
diff --git a/tests/qemu-iotests/245.out b/tests/qemu-iotests/245.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/245.out
+++ b/tests/qemu-iotests/245.out
@@ -XXX,XX +XXX,XX @@
-{"execute": "job-finalize", "arguments": {"id": "commit0"}}
+..{"execute": "job-finalize", "arguments": {"id": "commit0"}}
 {"return": {}}
 {"data": {"id": "commit0", "type": "commit"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {"data": {"device": "commit0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"execute": "job-finalize", "arguments": {"id": "stream0"}}
+...{"execute": "job-finalize", "arguments": {"id": "stream0"}}
 {"return": {}}
 {"data": {"id": "stream0", "type": "stream"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {"data": {"device": "stream0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "stream"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"execute": "job-finalize", "arguments": {"id": "stream0"}}
+.{"execute": "job-finalize", "arguments": {"id": "stream0"}}
 {"return": {}}
 {"data": {"id": "stream0", "type": "stream"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {"data": {"device": "stream0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "stream"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-.....................
+...............
 ----------------------------------------------------------------------
 Ran 21 tests
 
diff --git a/tests/qemu-iotests/295.out b/tests/qemu-iotests/295.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/295.out
+++ b/tests/qemu-iotests/295.out
@@ -XXX,XX +XXX,XX @@
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
 {"return": {}}
-{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
+.{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
 {"return": {}}
@@ -XXX,XX +XXX,XX @@ Job failed: Invalid password, cannot unlock any keyslot
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
 {"return": {}}
-{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
+.{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
 {"return": {}}
@@ -XXX,XX +XXX,XX @@ Job failed: All the active keyslots match the (old) password that was given and
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
 {"return": {}}
-...
+.
 ----------------------------------------------------------------------
 Ran 3 tests
 
diff --git a/tests/qemu-iotests/296.out b/tests/qemu-iotests/296.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/296.out
+++ b/tests/qemu-iotests/296.out
@@ -XXX,XX +XXX,XX @@ Job failed: Failed to get shared "consistent read" lock
 qemu-img: Failed to get shared "consistent read" lock
 Is another process using the image [TEST_DIR/test.img]?
 
-Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
+.Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
 
 Job failed: Block node is read-only
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
@@ -XXX,XX +XXX,XX @@ Job failed: Failed to get shared "consistent read" lock
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
-Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
+.Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
 
 {"return": {}}
 {"error": {"class": "GenericError", "desc": "Failed to get \"write\" lock"}}
-Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
+.Formatting 'TEST_DIR/test.img', fmt=luks size=1048576 key-secret=keysec0 iter-time=10
 
 {"return": {}}
 {"return": {}}
-....
+.
 ----------------------------------------------------------------------
 Ran 4 tests
 
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@
 import bz2
 from collections import OrderedDict
 import faulthandler
-import io
 import json
 import logging
 import os
@@ -XXX,XX +XXX,XX @@
 import sys
 import time
 from typing import (Any, Callable, Dict, Iterable,
-                    List, Optional, Sequence, Tuple, TypeVar)
+                    List, Optional, Sequence, TextIO, Tuple, Type, TypeVar)
 import unittest
 
 from contextlib import contextmanager
@@ -XXX,XX +XXX,XX @@ def func_wrapper(*args, **kwargs):
             return func(*args, **kwargs)
     return func_wrapper
 
+# We need to filter out the time taken from the output so that
+# qemu-iotest can reliably diff the results against master output,
+# and hide skipped tests from the reference output.
+
+class ReproducibleTestResult(unittest.TextTestResult):
+    def addSkip(self, test, reason):
+        # Same as TextTestResult, but print dot instead of "s"
+        unittest.TestResult.addSkip(self, test, reason)
+        if self.showAll:
+            self.stream.writeln("skipped {0!r}".format(reason))
+        elif self.dots:
+            self.stream.write(".")
+            self.stream.flush()
+
+class ReproducibleStreamWrapper:
+    def __init__(self, stream: TextIO):
+        self.stream = stream
+
+    def __getattr__(self, attr):
+        if attr in ('stream', '__getstate__'):
+            raise AttributeError(attr)
+        return getattr(self.stream, attr)
+
+    def write(self, arg=None):
+        arg = re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', arg)
+        arg = re.sub(r' $skipped=\d+$', r'', arg)
+        self.stream.write(arg)
+
+class ReproducibleTestRunner(unittest.TextTestRunner):
+    def __init__(self, stream: Optional[TextIO] = None,
+             resultclass: Type[unittest.TestResult] = ReproducibleTestResult,
+             **kwargs: Any) -> None:
+        rstream = ReproducibleStreamWrapper(stream or sys.stdout)
+        super().__init__(stream=rstream,           # type: ignore
+                         descriptions=True,
+                         resultclass=resultclass,
+                         **kwargs)
+
 def execute_unittest(debug=False):
     """Executes unittests within the calling module."""
 
     verbosity = 2 if debug else 1
-
-    if debug:
-        output = sys.stdout
-    else:
-        # We need to filter out the time taken from the output so that
-        # qemu-iotest can reliably diff the results against master output.
-        output = io.StringIO()
-
-    runner = unittest.TextTestRunner(stream=output, descriptions=True,
-                                     verbosity=verbosity)
-    try:
-        # unittest.main() will use sys.exit(); so expect a SystemExit
-        # exception
-        unittest.main(testRunner=runner)
-    finally:
-        # We need to filter out the time taken from the output so that
-        # qemu-iotest can reliably diff the results against master output.
-        if not debug:
-            out = output.getvalue()
-            out = re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', out)
-
-            # Hide skipped tests from the reference output
-            out = re.sub(r'OK $skipped=\d+$', 'OK', out)
-            out_first_line, out_rest = out.split('\n', 1)
-            out = out_first_line.replace('s', '.') + '\n' + out_rest
-
-            sys.stderr.write(out)
+    runner = ReproducibleTestRunner(verbosity=verbosity)
+    unittest.main(testRunner=runner)
 
 def execute_setup_common(supported_fmts: Sequence[str] = (),
                          supported_platforms: Sequence[str] = (),
-- 
2.31.1

From: Paolo Bonzini <pbonzini@redhat.com>

Python test scripts that use unittest consist of multiple tests.
unittest.main allows selecting which tests to run, but currently this
is not possible because the iotests wrapper ignores sys.argv.

unittest.main command line options also allow the user to pick the
desired options for verbosity, failfast mode, etc.  While "-d" is
currently translated to "-v", it also enables extra debug output,
and other options are not available at all.

These command line options only work if the unittest.main testRunner
argument is a type, rather than a TestRunner instance.  Therefore, pass
the class name and "verbosity" argument to unittest.main, and adjust for
the different default warnings between TextTestRunner and unittest.main.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Message-Id: <20210323181928.311862-3-pbonzini@redhat.com>
Message-Id: <20210503110110.476887-3-pbonzini@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def __init__(self, stream: Optional[TextIO] = None,
                          resultclass=resultclass,
                          **kwargs)
 
-def execute_unittest(debug=False):
+def execute_unittest(argv: List[str], debug: bool = False) -> None:
     """Executes unittests within the calling module."""
 
-    verbosity = 2 if debug else 1
-    runner = ReproducibleTestRunner(verbosity=verbosity)
-    unittest.main(testRunner=runner)
+    # Some tests have warnings, especially ResourceWarnings for unclosed
+    # files and sockets.  Ignore them for now to ensure reproducibility of
+    # the test output.
+    unittest.main(argv=argv,
+                  testRunner=ReproducibleTestRunner,
+                  verbosity=2 if debug else 1,
+                  warnings=None if sys.warnoptions else 'ignore')
 
 def execute_setup_common(supported_fmts: Sequence[str] = (),
                          supported_platforms: Sequence[str] = (),
@@ -XXX,XX +XXX,XX @@ def execute_test(*args, test_function=None, **kwargs):
 
     debug = execute_setup_common(*args, **kwargs)
     if not test_function:
-        execute_unittest(debug)
+        execute_unittest(sys.argv, debug)
     else:
         test_function()
 
-- 
2.31.1

From: Paolo Bonzini <pbonzini@redhat.com>

In the next patch, "check" will learn how to execute a test script without
going through TestRunner.  To enable this, keep only the text output
and subprocess handling in the TestRunner; move into TestEnv the logic
to prepare for running a subprocess.

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Message-Id: <20210323181928.311862-4-pbonzini@redhat.com>
Message-Id: <20210503110110.476887-4-pbonzini@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/testenv.py    | 17 ++++++++++++++++-
 tests/qemu-iotests/testrunner.py | 14 +-------------
 2 files changed, 17 insertions(+), 14 deletions(-)

diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testenv.py
+++ b/tests/qemu-iotests/testenv.py
@@ -XXX,XX +XXX,XX @@
 import random
 import subprocess
 import glob
-from typing import Dict, Any, Optional, ContextManager
+from typing import List, Dict, Any, Optional, ContextManager
 
 
 def isxfile(path: str) -> bool:
@@ -XXX,XX +XXX,XX @@ class TestEnv(ContextManager['TestEnv']):
                      'CACHEMODE_IS_DEFAULT', 'IMGFMT_GENERIC', 'IMGOPTSSYNTAX',
                      'IMGKEYSECRET', 'QEMU_DEFAULT_MACHINE', 'MALLOC_PERTURB_']
 
+    def prepare_subprocess(self, args: List[str]) -> Dict[str, str]:
+        if self.debug:
+            args.append('-d')
+
+        with open(args[0], encoding="utf-8") as f:
+            try:
+                if f.readline().rstrip() == '#!/usr/bin/env python3':
+                    args.insert(0, self.python)
+            except UnicodeDecodeError:  # binary test? for future.
+                pass
+
+        os_env = os.environ.copy()
+        os_env.update(self.get_env())
+        return os_env
+
     def get_env(self) -> Dict[str, str]:
         env = {}
         for v in self.env_variables:
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -XXX,XX +XXX,XX @@ class TestRunner(ContextManager['TestRunner']):
     def __init__(self, env: TestEnv, makecheck: bool = False,
                  color: str = 'auto') -> None:
         self.env = env
-        self.test_run_env = self.env.get_env()
         self.makecheck = makecheck
         self.last_elapsed = LastElapsedTime('.last-elapsed-cache', env)
 
@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str) -> TestResult:
             silent_unlink(p)
 
         args = [str(f_test.resolve())]
-        if self.env.debug:
-            args.append('-d')
-
-        with f_test.open(encoding="utf-8") as f:
-            try:
-                if f.readline().rstrip() == '#!/usr/bin/env python3':
-                    args.insert(0, self.env.python)
-            except UnicodeDecodeError:  # binary test? for future.
-                pass
-
-        env = os.environ.copy()
-        env.update(self.test_run_env)
+        env = self.env.prepare_subprocess(args)
 
         t0 = time.time()
         with f_bad.open('w', encoding="utf-8") as f:
-- 
2.31.1

From: Paolo Bonzini <pbonzini@redhat.com>

Right now there is no easy way for "check" to print a reproducer command.
Because such a reproducer command line would be huge, we can instead teach
check to start a command of our choice.  This can be for example a Python
unit test with arguments to only run a specific subtest.

Move the trailing empty line to print_env(), since it always looks better
and one caller was not adding it.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Message-Id: <20210323181928.311862-5-pbonzini@redhat.com>
Message-Id: <20210503110110.476887-5-pbonzini@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/check         | 19 ++++++++++++++++++-
 tests/qemu-iotests/testenv.py    |  3 ++-
 tests/qemu-iotests/testrunner.py |  1 -
 3 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -XXX,XX +XXX,XX @@
 import os
 import sys
 import argparse
+import shutil
+from pathlib import Path
+
 from findtests import TestFinder
 from testenv import TestEnv
 from testrunner import TestRunner
@@ -XXX,XX +XXX,XX @@ def make_argparser() -> argparse.ArgumentParser:
                        'rerun failed ./check command, starting from the '
                        'middle of the process.')
     g_sel.add_argument('tests', metavar='TEST_FILES', nargs='*',
-                       help='tests to run')
+                       help='tests to run, or "--" followed by a command')
 
     return p
 
@@ -XXX,XX +XXX,XX @@ if __name__ == '__main__':
                   imgopts=args.imgopts, misalign=args.misalign,
                   debug=args.debug, valgrind=args.valgrind)
 
+    if len(sys.argv) > 1 and sys.argv[-len(args.tests)-1] == '--':
+        if not args.tests:
+            sys.exit("missing command after '--'")
+        cmd = args.tests
+        env.print_env()
+        exec_pathstr = shutil.which(cmd[0])
+        if exec_pathstr is None:
+            sys.exit('command not found: ' + cmd[0])
+        exec_path = Path(exec_pathstr).resolve()
+        cmd[0] = str(exec_path)
+        full_env = env.prepare_subprocess(cmd)
+        os.chdir(exec_path.parent)
+        os.execve(cmd[0], cmd, full_env)
+
     testfinder = TestFinder(test_dir=env.source_iotests)
 
     groups = args.groups.split(',') if args.groups else None
diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testenv.py
+++ b/tests/qemu-iotests/testenv.py
@@ -XXX,XX +XXX,XX @@ def print_env(self) -> None:
 PLATFORM      -- {platform}
 TEST_DIR      -- {TEST_DIR}
 SOCK_DIR      -- {SOCK_DIR}
-SOCKET_SCM_HELPER -- {SOCKET_SCM_HELPER}"""
+SOCKET_SCM_HELPER -- {SOCKET_SCM_HELPER}
+"""
 
         args = collections.defaultdict(str, self.get_env())
 
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -XXX,XX +XXX,XX @@ def run_tests(self, tests: List[str]) -> bool:
 
         if not self.makecheck:
             self.env.print_env()
-            print()
 
         test_field_width = max(len(os.path.basename(t)) for t in tests) + 2
 
-- 
2.31.1

From: Paolo Bonzini <pbonzini@redhat.com>

Due to a typo, in this case the SOCK_DIR was not being created.

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Tested-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Message-Id: <20210323181928.311862-6-pbonzini@redhat.com>
Message-Id: <20210503110110.476887-6-pbonzini@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/testenv.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testenv.py
+++ b/tests/qemu-iotests/testenv.py
@@ -XXX,XX +XXX,XX @@ def init_directories(self) -> None:
         try:
             self.sock_dir = os.environ['SOCK_DIR']
             self.tmp_sock_dir = False
-            Path(self.test_dir).mkdir(parents=True, exist_ok=True)
+            Path(self.sock_dir).mkdir(parents=True, exist_ok=True)
         except KeyError:
             self.sock_dir = tempfile.mkdtemp()
             self.tmp_sock_dir = True
-- 
2.31.1

From: Connor Kuehl <ckuehl@redhat.com>

The contents of this patch were initially developed and posted by Han
Han[1], however, it appears the original patch was not applied. Since
then, the relevant documentation has been moved and adapted to a new
format.

I've taken most of the original wording and tweaked it according to
some of the feedback from the original patch submission. I've also
adapted it to restructured text, which is the format the documentation
currently uses.

[1] https://lists.nongnu.org/archive/html/qemu-block/2019-10/msg01253.html

Fixes: https://bugzilla.redhat.com/1763105
Signed-off-by: Han Han <hhan@redhat.com>
Suggested-by: Max Reitz <mreitz@redhat.com>
[ Max: provided description of data_file_raw behavior ]
Signed-off-by: Connor Kuehl <ckuehl@redhat.com>
Message-Id: <20210505195512.391128-1-ckuehl@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 docs/tools/qemu-img.rst | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst
index XXXXXXX..XXXXXXX 100644
--- a/docs/tools/qemu-img.rst
+++ b/docs/tools/qemu-img.rst
@@ -XXX,XX +XXX,XX @@ Supported image file formats:
     issue ``lsattr filename`` to check if the NOCOW flag is set or not
     (Capital 'C' is NOCOW flag).
 
+  ``data_file``
+    Filename where all guest data will be stored. If this option is used,
+    the qcow2 file will only contain the image's metadata.
+
+    Note: Data loss will occur if the given filename already exists when
+    using this option with ``qemu-img create`` since ``qemu-img`` will create
+    the data file anew, overwriting the file's original contents. To simply
+    update the reference to point to the given pre-existing file, use
+    ``qemu-img amend``.
+
+  ``data_file_raw``
+    If this option is set to ``on``, QEMU will always keep the external data
+    file consistent as a standalone read-only raw image.
+
+    It does this by forwarding all write accesses to the qcow2 file through to
+    the raw data file, including their offsets. Therefore, data that is visible
+    on the qcow2 node (i.e., to the guest) at some offset is visible at the same
+    offset in the raw data file. This results in a read-only raw image. Writes
+    that bypass the qcow2 metadata may corrupt the qcow2 metadata because the
+    out-of-band writes may result in the metadata falling out of sync with the
+    raw image.
+
+    If this option is ``off``, QEMU will use the data file to store data in an
+    arbitrary manner. The file’s content will not make sense without the
+    accompanying qcow2 metadata. Where data is written will have no relation to
+    its offset as seen by the guest, and some writes (specifically zero writes)
+    may not be forwarded to the data file at all, but will only be handled by
+    modifying qcow2 metadata.
+
+    This option can only be enabled if ``data_file`` is set.
+
 ``Other``
 
   QEMU also supports various other image file formats for
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Now, after huge update of block graph permission update algorithm, we
don't need this workaround with active state of the filter. Drop it and
use new smart bdrv_drop_filter() function.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210506194143.394141-1-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/copy-on-read.c | 33 +--------------------------------
 1 file changed, 1 insertion(+), 32 deletions(-)

diff --git a/block/copy-on-read.c b/block/copy-on-read.c
index XXXXXXX..XXXXXXX 100644
--- a/block/copy-on-read.c
+++ b/block/copy-on-read.c
@@ -XXX,XX +XXX,XX @@
 
 
 typedef struct BDRVStateCOR {
-    bool active;
     BlockDriverState *bottom_bs;
     bool chain_frozen;
 } BDRVStateCOR;
@@ -XXX,XX +XXX,XX @@ static int cor_open(BlockDriverState *bs, QDict *options, int flags,
          */
         bdrv_ref(bottom_bs);
     }
-    state->active = true;
     state->bottom_bs = bottom_bs;
 
     /*
@@ -XXX,XX +XXX,XX @@ static void cor_child_perm(BlockDriverState *bs, BdrvChild *c,
                            uint64_t perm, uint64_t shared,
                            uint64_t *nperm, uint64_t *nshared)
 {
-    BDRVStateCOR *s = bs->opaque;
-
-    if (!s->active) {
-        /*
-         * While the filter is being removed
-         */
-        *nperm = 0;
-        *nshared = BLK_PERM_ALL;
-        return;
-    }
-
     *nperm = perm & PERM_PASSTHROUGH;
     *nshared = (shared & PERM_PASSTHROUGH) | PERM_UNCHANGED;
 
@@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_copy_on_read = {
 
 void bdrv_cor_filter_drop(BlockDriverState *cor_filter_bs)
 {
-    BdrvChild *child;
-    BlockDriverState *bs;
     BDRVStateCOR *s = cor_filter_bs->opaque;
 
-    child = bdrv_filter_child(cor_filter_bs);
-    if (!child) {
-        return;
-    }
-    bs = child->bs;
-
-    /* Retain the BDS until we complete the graph change. */
-    bdrv_ref(bs);
-    /* Hold a guest back from writing while permissions are being reset. */
-    bdrv_drained_begin(bs);
-    /* Drop permissions before the graph change. */
-    s->active = false;
     /* unfreeze, as otherwise bdrv_replace_node() will fail */
     if (s->chain_frozen) {
         s->chain_frozen = false;
         bdrv_unfreeze_backing_chain(cor_filter_bs, s->bottom_bs);
     }
-    bdrv_child_refresh_perms(cor_filter_bs, child, &error_abort);
-    bdrv_replace_node(cor_filter_bs, bs, &error_abort);
-
-    bdrv_drained_end(bs);
-    bdrv_unref(bs);
+    bdrv_drop_filter(cor_filter_bs, &error_abort);
     bdrv_unref(cor_filter_bs);
 }
 
-- 
2.31.1

From: Emanuele Giuseppe Esposito <eesposit@redhat.com>

pylint 2.8 introduces consider-using-with error, suggesting
to use the 'with' block statement when possible.

Modify all subprocess.Popen call to use the 'with' statement,
except the one in __init__ of QemuIoInteractive class, since
it is assigned to a class field and used in other methods.

Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Message-Id: <20210510190449.65948-1-eesposit@redhat.com>
[mreitz: Disable bad-option-value warning in the iotests' pylintrc, so
         that disabling consider-using-with in QemuIoInteractive will
         not produce a warning in pre-2.8 pylint versions]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py    | 65 ++++++++++++++++----------------
 tests/qemu-iotests/pylintrc      |  3 ++
 tests/qemu-iotests/testrunner.py | 22 +++++------
 3 files changed, 47 insertions(+), 43 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@ def qemu_tool_pipe_and_status(tool: str, args: Sequence[str],
     Run a tool and return both its output and its exit code
     """
     stderr = subprocess.STDOUT if connect_stderr else None
-    subp = subprocess.Popen(args,
-                            stdout=subprocess.PIPE,
-                            stderr=stderr,
-                            universal_newlines=True)
-    output = subp.communicate()[0]
-    if subp.returncode < 0:
-        cmd = ' '.join(args)
-        sys.stderr.write(f'{tool} received signal {-subp.returncode}: {cmd}\n')
-    return (output, subp.returncode)
+    with subprocess.Popen(args, stdout=subprocess.PIPE,
+                          stderr=stderr, universal_newlines=True) as subp:
+        output = subp.communicate()[0]
+        if subp.returncode < 0:
+            cmd = ' '.join(args)
+            sys.stderr.write(f'{tool} received signal \
+                               {-subp.returncode}: {cmd}\n')
+        return (output, subp.returncode)
 
 def qemu_img_pipe_and_status(*args: str) -> Tuple[str, int]:
     """
@@ -XXX,XX +XXX,XX @@ def qemu_io_silent_check(*args):
 class QemuIoInteractive:
     def __init__(self, *args):
         self.args = qemu_io_args_no_fmt + list(args)
+        # We need to keep the Popen objext around, and not
+        # close it immediately. Therefore, disable the pylint check:
+        # pylint: disable=consider-using-with
         self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE,
                                    stdout=subprocess.PIPE,
                                    stderr=subprocess.STDOUT,
@@ -XXX,XX +XXX,XX @@ def qemu_nbd_popen(*args):
     cmd.extend(args)
 
     log('Start NBD server')
-    p = subprocess.Popen(cmd)
-    try:
-        while not os.path.exists(pid_file):
-            if p.poll() is not None:
-                raise RuntimeError(
-                    "qemu-nbd terminated with exit code {}: {}"
-                    .format(p.returncode, ' '.join(cmd)))
-
-            time.sleep(0.01)
-        yield
-    finally:
-        if os.path.exists(pid_file):
-            os.remove(pid_file)
-        log('Kill NBD server')
-        p.kill()
-        p.wait()
+    with subprocess.Popen(cmd) as p:
+        try:
+            while not os.path.exists(pid_file):
+                if p.poll() is not None:
+                    raise RuntimeError(
+                        "qemu-nbd terminated with exit code {}: {}"
+                        .format(p.returncode, ' '.join(cmd)))
+
+                time.sleep(0.01)
+            yield
+        finally:
+            if os.path.exists(pid_file):
+                os.remove(pid_file)
+            log('Kill NBD server')
+            p.kill()
+            p.wait()
 
 def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
     '''Return True if two image files are identical'''
@@ -XXX,XX +XXX,XX @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
 
 def create_image(name, size):
     '''Create a fully-allocated raw image with sector markers'''
-    file = open(name, 'wb')
-    i = 0
-    while i < size:
-        sector = struct.pack('>l504xl', i // 512, i // 512)
-        file.write(sector)
-        i = i + 512
-    file.close()
+    with open(name, 'wb') as file:
+        i = 0
+        while i < size:
+            sector = struct.pack('>l504xl', i // 512, i // 512)
+            file.write(sector)
+            i = i + 512
 
 def image_size(img):
     '''Return image's virtual size'''
diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/pylintrc
+++ b/tests/qemu-iotests/pylintrc
@@ -XXX,XX +XXX,XX @@ disable=invalid-name,
         too-many-public-methods,
         # pylint warns about Optional[] etc. as unsubscriptable in 3.9
         unsubscriptable-object,
+        # Sometimes we need to disable a newly introduced pylint warning.
+        # Doing so should not produce a warning in older versions of pylint.
+        bad-option-value,
         # These are temporary, and should be removed:
         missing-docstring,
         too-many-return-statements,
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str) -> TestResult:
 
         t0 = time.time()
         with f_bad.open('w', encoding="utf-8") as f:
-            proc = subprocess.Popen(args, cwd=str(f_test.parent), env=env,
-                                    stdout=f, stderr=subprocess.STDOUT)
-            try:
-                proc.wait()
-            except KeyboardInterrupt:
-                proc.terminate()
-                proc.wait()
-                return TestResult(status='not run',
-                                  description='Interrupted by user',
-                                  interrupted=True)
-            ret = proc.returncode
+            with subprocess.Popen(args, cwd=str(f_test.parent), env=env,
+                                  stdout=f, stderr=subprocess.STDOUT) as proc:
+                try:
+                    proc.wait()
+                except KeyboardInterrupt:
+                    proc.terminate()
+                    proc.wait()
+                    return TestResult(status='not run',
+                                      description='Interrupted by user',
+                                      interrupted=True)
+                ret = proc.returncode
 
         elapsed = round(time.time() - t0, 1)
 
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

write-notifiers are used only for write-threshold. New code for such
purpose should create filters.

Let's better special-case write-threshold and drop write notifiers at
all. (Actually, write-threshold is special-cased anyway, as the only
user of write-notifiers)

So, create a new direct interface for bdrv_co_write_req_prepare() and
drop all write-notifier related logic from write-threshold.c.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210506090621.11848-2-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[mreitz: Adjusted comment as per Eric's suggestion]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block_int.h       |  1 -
 include/block/write-threshold.h |  9 +++++
 block/io.c                      |  5 ++-
 block/write-threshold.c         | 70 +++++++--------------------------
 4 files changed, 27 insertions(+), 58 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -XXX,XX +XXX,XX @@ struct BlockDriverState {
 
     /* threshold limit for writes, in bytes. "High water mark". */
     uint64_t write_threshold_offset;
-    NotifierWithReturn write_threshold_notifier;
 
     /* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
      * Reading from the list can be done with either the BQL or the
diff --git a/include/block/write-threshold.h b/include/block/write-threshold.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/write-threshold.h
+++ b/include/block/write-threshold.h
@@ -XXX,XX +XXX,XX @@ bool bdrv_write_threshold_is_set(const BlockDriverState *bs);
 uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
                                        const BdrvTrackedRequest *req);
 
+/*
+ * bdrv_write_threshold_check_write
+ *
+ * Check whether the specified request exceeds the write threshold.
+ * If so, send a corresponding event and disable write threshold checking.
+ */
+void bdrv_write_threshold_check_write(BlockDriverState *bs, int64_t offset,
+                                      int64_t bytes);
+
 #endif
diff --git a/block/io.c b/block/io.c
index XXXXXXX..XXXXXXX 100644
--- a/block/io.c
+++ b/block/io.c
@@ -XXX,XX +XXX,XX @@
 #include "block/blockjob_int.h"
 #include "block/block_int.h"
 #include "block/coroutines.h"
+#include "block/write-threshold.h"
 #include "qemu/cutils.h"
 #include "qapi/error.h"
 #include "qemu/error-report.h"
@@ -XXX,XX +XXX,XX @@ bdrv_co_write_req_prepare(BdrvChild *child, int64_t offset, int64_t bytes,
         } else {
             assert(child->perm & BLK_PERM_WRITE);
         }
-        return notifier_with_return_list_notify(&bs->before_write_notifiers,
-                                                req);
+        bdrv_write_threshold_check_write(bs, offset, bytes);
+        return 0;
     case BDRV_TRACKED_TRUNCATE:
         assert(child->perm & BLK_PERM_RESIZE);
         return 0;
diff --git a/block/write-threshold.c b/block/write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/block/write-threshold.c
+++ b/block/write-threshold.c
@@ -XXX,XX +XXX,XX @@ bool bdrv_write_threshold_is_set(const BlockDriverState *bs)
     return bs->write_threshold_offset > 0;
 }
 
-static void write_threshold_disable(BlockDriverState *bs)
-{
-    if (bdrv_write_threshold_is_set(bs)) {
-        notifier_with_return_remove(&bs->write_threshold_notifier);
-        bs->write_threshold_offset = 0;
-    }
-}
-
 uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
                                        const BdrvTrackedRequest *req)
 {
@@ -XXX,XX +XXX,XX @@ uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
     return 0;
 }
 
-static int coroutine_fn before_write_notify(NotifierWithReturn *notifier,
-                                            void *opaque)
-{
-    BdrvTrackedRequest *req = opaque;
-    BlockDriverState *bs = req->bs;
-    uint64_t amount = 0;
-
-    amount = bdrv_write_threshold_exceeded(bs, req);
-    if (amount > 0) {
-        qapi_event_send_block_write_threshold(
-            bs->node_name,
-            amount,
-            bs->write_threshold_offset);
-
-        /* autodisable to avoid flooding the monitor */
-        write_threshold_disable(bs);
-    }
-
-    return 0; /* should always let other notifiers run */
-}
-
-static void write_threshold_register_notifier(BlockDriverState *bs)
-{
-    bs->write_threshold_notifier.notify = before_write_notify;
-    bdrv_add_before_write_notifier(bs, &bs->write_threshold_notifier);
-}
-
-static void write_threshold_update(BlockDriverState *bs,
-                                   int64_t threshold_bytes)
-{
-    bs->write_threshold_offset = threshold_bytes;
-}
-
 void bdrv_write_threshold_set(BlockDriverState *bs, uint64_t threshold_bytes)
 {
-    if (bdrv_write_threshold_is_set(bs)) {
-        if (threshold_bytes > 0) {
-            write_threshold_update(bs, threshold_bytes);
-        } else {
-            write_threshold_disable(bs);
-        }
-    } else {
-        if (threshold_bytes > 0) {
-            /* avoid multiple registration */
-            write_threshold_register_notifier(bs);
-            write_threshold_update(bs, threshold_bytes);
-        }
-        /* discard bogus disable request */
-    }
+    bs->write_threshold_offset = threshold_bytes;
 }
 
 void qmp_block_set_write_threshold(const char *node_name,
@@ -XXX,XX +XXX,XX @@ void qmp_block_set_write_threshold(const char *node_name,
 
     aio_context_release(aio_context);
 }
+
+void bdrv_write_threshold_check_write(BlockDriverState *bs, int64_t offset,
+                                      int64_t bytes)
+{
+    int64_t end = offset + bytes;
+    uint64_t wtr = bs->write_threshold_offset;
+
+    if (wtr > 0 && end > wtr) {
+        qapi_event_send_block_write_threshold(bs->node_name, end - wtr, wtr);
+
+        /* autodisable to avoid flooding the monitor */
+        bdrv_write_threshold_set(bs, 0);
+    }
+}
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

They are unused now.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210506090621.11848-3-vsementsov@virtuozzo.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/block_int.h | 12 ------------
 block.c                   |  1 -
 block/io.c                |  6 ------
 3 files changed, 19 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -XXX,XX +XXX,XX @@ struct BlockDriverState {
      */
     int64_t total_sectors;
 
-    /* Callback before write request is processed */
-    NotifierWithReturnList before_write_notifiers;
-
     /* threshold limit for writes, in bytes. "High water mark". */
     uint64_t write_threshold_offset;
 
@@ -XXX,XX +XXX,XX @@ void bdrv_parse_filename_strip_prefix(const char *filename, const char *prefix,
 bool bdrv_backing_overridden(BlockDriverState *bs);
 
 
-/**
- * bdrv_add_before_write_notifier:
- *
- * Register a callback that is invoked before write requests are processed but
- * after any throttling or waiting for overlapping requests.
- */
-void bdrv_add_before_write_notifier(BlockDriverState *bs,
-                                    NotifierWithReturn *notifier);
-
 /**
  * bdrv_add_aio_context_notifier:
  *
diff --git a/block.c b/block.c
index XXXXXXX..XXXXXXX 100644
--- a/block.c
+++ b/block.c
@@ -XXX,XX +XXX,XX @@ BlockDriverState *bdrv_new(void)
     for (i = 0; i < BLOCK_OP_TYPE_MAX; i++) {
         QLIST_INIT(&bs->op_blockers[i]);
     }
-    notifier_with_return_list_init(&bs->before_write_notifiers);
     qemu_co_mutex_init(&bs->reqs_lock);
     qemu_mutex_init(&bs->dirty_bitmap_mutex);
     bs->refcnt = 1;
diff --git a/block/io.c b/block/io.c
index XXXXXXX..XXXXXXX 100644
--- a/block/io.c
+++ b/block/io.c
@@ -XXX,XX +XXX,XX @@ bool bdrv_qiov_is_aligned(BlockDriverState *bs, QEMUIOVector *qiov)
     return true;
 }
 
-void bdrv_add_before_write_notifier(BlockDriverState *bs,
-                                    NotifierWithReturn *notifier)
-{
-    notifier_with_return_list_add(&bs->before_write_notifiers, notifier);
-}
-
 void bdrv_io_plug(BlockDriverState *bs)
 {
     BdrvChild *child;
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

These tests use bdrv_write_threshold_exceeded() API, which is used only
for test (since pre-previous commit). Better is testing real API, which
is used in block.c as well.

So, let's call bdrv_write_threshold_check_write(), and check is
bs->write_threshold_offset cleared or not (it's cleared iff threshold
triggered).

Also we get rid of BdrvTrackedRequest use here. Note, that paranoiac
bdrv_check_request() calls were added in 8b1170012b1 to protect
BdrvTrackedRequest. Drop them now.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210506090621.11848-4-vsementsov@virtuozzo.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/unit/test-write-threshold.c | 22 ++++------------------
 1 file changed, 4 insertions(+), 18 deletions(-)

diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/unit/test-write-threshold.c
+++ b/tests/unit/test-write-threshold.c
@@ -XXX,XX +XXX,XX @@ static void test_threshold_multi_set_get(void)
 
 static void test_threshold_not_trigger(void)
 {
-    uint64_t amount = 0;
     uint64_t threshold = 4 * 1024 * 1024;
     BlockDriverState bs;
-    BdrvTrackedRequest req;
 
     memset(&bs, 0, sizeof(bs));
-    memset(&req, 0, sizeof(req));
-    req.offset = 1024;
-    req.bytes = 1024;
-
-    bdrv_check_request(req.offset, req.bytes, &error_abort);
 
     bdrv_write_threshold_set(&bs, threshold);
-    amount = bdrv_write_threshold_exceeded(&bs, &req);
-    g_assert_cmpuint(amount, ==, 0);
+    bdrv_write_threshold_check_write(&bs, 1024, 1024);
+    g_assert_cmpuint(bdrv_write_threshold_get(&bs), ==, threshold);
 }
 
 
 static void test_threshold_trigger(void)
 {
-    uint64_t amount = 0;
     uint64_t threshold = 4 * 1024 * 1024;
     BlockDriverState bs;
-    BdrvTrackedRequest req;
 
     memset(&bs, 0, sizeof(bs));
-    memset(&req, 0, sizeof(req));
-    req.offset = (4 * 1024 * 1024) - 1024;
-    req.bytes = 2 * 1024;
-
-    bdrv_check_request(req.offset, req.bytes, &error_abort);
 
     bdrv_write_threshold_set(&bs, threshold);
-    amount = bdrv_write_threshold_exceeded(&bs, &req);
-    g_assert_cmpuint(amount, >=, 1024);
+    bdrv_write_threshold_check_write(&bs, threshold - 1024, 2 * 1024);
+    g_assert_cmpuint(bdrv_write_threshold_get(&bs), ==, 0);
 }
 
 typedef struct TestStruct {
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

bdrv_write_threshold_exceeded() is unused.

bdrv_write_threshold_is_set() is used only to double check the value of
bs->write_threshold_offset in tests. No real sense in it (both tests do
check real value with help of bdrv_write_threshold_get())

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210506090621.11848-5-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[mreitz: Adjusted commit message as per Eric's suggestion]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/write-threshold.h   | 24 ------------------------
 block/write-threshold.c           | 19 -------------------
 tests/unit/test-write-threshold.c |  4 ----
 3 files changed, 47 deletions(-)

diff --git a/include/block/write-threshold.h b/include/block/write-threshold.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/write-threshold.h
+++ b/include/block/write-threshold.h
@@ -XXX,XX +XXX,XX @@ void bdrv_write_threshold_set(BlockDriverState *bs, uint64_t threshold_bytes);
  */
 uint64_t bdrv_write_threshold_get(const BlockDriverState *bs);
 
-/*
- * bdrv_write_threshold_is_set
- *
- * Tell if a write threshold is set for a given BDS.
- */
-bool bdrv_write_threshold_is_set(const BlockDriverState *bs);
-
-/*
- * bdrv_write_threshold_exceeded
- *
- * Return the extent of a write request that exceeded the threshold,
- * or zero if the request is below the threshold.
- * Return zero also if the threshold was not set.
- *
- * NOTE: here we assume the following holds for each request this code
- * deals with:
- *
- * assert((req->offset + req->bytes) <= UINT64_MAX)
- *
- * Please not there is *not* an actual C assert().
- */
-uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
-                                       const BdrvTrackedRequest *req);
-
 /*
  * bdrv_write_threshold_check_write
  *
diff --git a/block/write-threshold.c b/block/write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/block/write-threshold.c
+++ b/block/write-threshold.c
@@ -XXX,XX +XXX,XX @@ uint64_t bdrv_write_threshold_get(const BlockDriverState *bs)
     return bs->write_threshold_offset;
 }
 
-bool bdrv_write_threshold_is_set(const BlockDriverState *bs)
-{
-    return bs->write_threshold_offset > 0;
-}
-
-uint64_t bdrv_write_threshold_exceeded(const BlockDriverState *bs,
-                                       const BdrvTrackedRequest *req)
-{
-    if (bdrv_write_threshold_is_set(bs)) {
-        if (req->offset > bs->write_threshold_offset) {
-            return (req->offset - bs->write_threshold_offset) + req->bytes;
-        }
-        if ((req->offset + req->bytes) > bs->write_threshold_offset) {
-            return (req->offset + req->bytes) - bs->write_threshold_offset;
-        }
-    }
-    return 0;
-}
-
 void bdrv_write_threshold_set(BlockDriverState *bs, uint64_t threshold_bytes)
 {
     bs->write_threshold_offset = threshold_bytes;
diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/unit/test-write-threshold.c
+++ b/tests/unit/test-write-threshold.c
@@ -XXX,XX +XXX,XX @@ static void test_threshold_not_set_on_init(void)
     BlockDriverState bs;
     memset(&bs, 0, sizeof(bs));
 
-    g_assert(!bdrv_write_threshold_is_set(&bs));
-
     res = bdrv_write_threshold_get(&bs);
     g_assert_cmpint(res, ==, 0);
 }
@@ -XXX,XX +XXX,XX @@ static void test_threshold_set_get(void)
 
     bdrv_write_threshold_set(&bs, threshold);
 
-    g_assert(bdrv_write_threshold_is_set(&bs));
-
     res = bdrv_write_threshold_get(&bs);
     g_assert_cmpint(res, ==, threshold);
 }
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Testing set/get of one 64bit variable doesn't seem necessary. We have a
lot of such variables. Also remaining tests do test set/get anyway.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210506090621.11848-7-vsementsov@virtuozzo.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/unit/test-write-threshold.c | 43 -------------------------------
 1 file changed, 43 deletions(-)

diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/unit/test-write-threshold.c
+++ b/tests/unit/test-write-threshold.c
@@ -XXX,XX +XXX,XX @@
 #include "block/write-threshold.h"
 
 
-static void test_threshold_not_set_on_init(void)
-{
-    uint64_t res;
-    BlockDriverState bs;
-    memset(&bs, 0, sizeof(bs));
-
-    res = bdrv_write_threshold_get(&bs);
-    g_assert_cmpint(res, ==, 0);
-}
-
-static void test_threshold_set_get(void)
-{
-    uint64_t threshold = 4 * 1024 * 1024;
-    uint64_t res;
-    BlockDriverState bs;
-    memset(&bs, 0, sizeof(bs));
-
-    bdrv_write_threshold_set(&bs, threshold);
-
-    res = bdrv_write_threshold_get(&bs);
-    g_assert_cmpint(res, ==, threshold);
-}
-
-static void test_threshold_multi_set_get(void)
-{
-    uint64_t threshold1 = 4 * 1024 * 1024;
-    uint64_t threshold2 = 15 * 1024 * 1024;
-    uint64_t res;
-    BlockDriverState bs;
-    memset(&bs, 0, sizeof(bs));
-
-    bdrv_write_threshold_set(&bs, threshold1);
-    bdrv_write_threshold_set(&bs, threshold2);
-    res = bdrv_write_threshold_get(&bs);
-    g_assert_cmpint(res, ==, threshold2);
-}
-
 static void test_threshold_not_trigger(void)
 {
     uint64_t threshold = 4 * 1024 * 1024;
@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
 {
     size_t i;
     TestStruct tests[] = {
-        { "/write-threshold/not-set-on-init",
-          test_threshold_not_set_on_init },
-        { "/write-threshold/set-get",
-          test_threshold_set_get },
-        { "/write-threshold/multi-set-get",
-          test_threshold_multi_set_get },
         { "/write-threshold/not-trigger",
           test_threshold_not_trigger },
         { "/write-threshold/trigger",
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

We don't need this extra logic: it doesn't make code simpler.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210506090621.11848-8-vsementsov@virtuozzo.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/unit/test-write-threshold.c | 20 +++-----------------
 1 file changed, 3 insertions(+), 17 deletions(-)

diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/unit/test-write-threshold.c
+++ b/tests/unit/test-write-threshold.c
@@ -XXX,XX +XXX,XX @@ static void test_threshold_trigger(void)
     g_assert_cmpuint(bdrv_write_threshold_get(&bs), ==, 0);
 }
 
-typedef struct TestStruct {
-    const char *name;
-    void (*func)(void);
-} TestStruct;
-
 
 int main(int argc, char **argv)
 {
-    size_t i;
-    TestStruct tests[] = {
-        { "/write-threshold/not-trigger",
-          test_threshold_not_trigger },
-        { "/write-threshold/trigger",
-          test_threshold_trigger },
-        { NULL, NULL }
-    };
-
     g_test_init(&argc, &argv, NULL);
-    for (i = 0; tests[i].name != NULL; i++) {
-        g_test_add_func(tests[i].name, tests[i].func);
-    }
+    g_test_add_func("/write-threshold/not-trigger", test_threshold_not_trigger);
+    g_test_add_func("/write-threshold/trigger", test_threshold_trigger);
+
     return g_test_run();
 }
-- 
2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

"qemu/typedefs.h" is enough for include/block/write-threshold.h header
with forward declaration of BlockDriverState. Also drop extra includes
from block/write-threshold.c and tests/unit/test-write-threshold.c

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210506090621.11848-9-vsementsov@virtuozzo.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 include/block/write-threshold.h   | 2 +-
 block/write-threshold.c           | 2 --
 tests/unit/test-write-threshold.c | 1 -
 3 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/include/block/write-threshold.h b/include/block/write-threshold.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/write-threshold.h
+++ b/include/block/write-threshold.h
@@ -XXX,XX +XXX,XX @@
 #ifndef BLOCK_WRITE_THRESHOLD_H
 #define BLOCK_WRITE_THRESHOLD_H
 
-#include "block/block_int.h"
+#include "qemu/typedefs.h"
 
 /*
  * bdrv_write_threshold_set:
diff --git a/block/write-threshold.c b/block/write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/block/write-threshold.c
+++ b/block/write-threshold.c
@@ -XXX,XX +XXX,XX @@
 
 #include "qemu/osdep.h"
 #include "block/block_int.h"
-#include "qemu/coroutine.h"
 #include "block/write-threshold.h"
-#include "qemu/notify.h"
 #include "qapi/error.h"
 #include "qapi/qapi-commands-block-core.h"
 #include "qapi/qapi-events-block-core.h"
diff --git a/tests/unit/test-write-threshold.c b/tests/unit/test-write-threshold.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/unit/test-write-threshold.c
+++ b/tests/unit/test-write-threshold.c
@@ -XXX,XX +XXX,XX @@
  */
 
 #include "qemu/osdep.h"
-#include "qapi/error.h"
 #include "block/block_int.h"
 #include "block/write-threshold.h"
 
-- 
2.31.1

The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:

Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24

for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:

iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)

----------------------------------------------------------------
Block patches:
- The SSH block driver now uses libssh instead of libssh2
- The VMDK block driver gets read-only support for the seSparse
  subformat
- Various fixes

---

v2:
- Squashed Pino's fix for pre-0.8 libssh into the libssh patch

----------------------------------------------------------------
Anton Nefedov (1):
  iotest 134: test cluster-misaligned encrypted write

Klaus Birkelund Jensen (1):
  nvme: do not advertise support for unsupported arbitration mechanism

Max Reitz (1):
  iotests: Fix 205 for concurrent runs

Pino Toscano (1):
  ssh: switch from libssh2 to libssh

Sam Eiderman (3):
  vmdk: Fix comment regarding max l1_size coverage
  vmdk: Reduce the max bound for L1 table size
  vmdk: Add read-only support for seSparse snapshots

Vladimir Sementsov-Ogievskiy (1):
  blockdev: enable non-root nodes for transaction drive-backup source

-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

We forget to enable it for transaction .prepare, while it is already
enabled in do_drive_backup since commit a2d665c1bc362
    "blockdev: loosen restrictions on drive-backup source node"

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
     assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);
     backup = common->action->u.drive_backup.data;
 
-    bs = qmp_get_root_bs(backup->device, errp);
+    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
     if (!bs) {
         return;
     }
-- 
2.21.0

From: Anton Nefedov <anton.nefedov@virtuozzo.com>

COW (even empty/zero) areas require encryption too

Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190516143028.81155-1-anton.nefedov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/134     |  9 +++++++++
 tests/qemu-iotests/134.out | 10 ++++++++++
 2 files changed, 19 insertions(+)

diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/134
+++ b/tests/qemu-iotests/134
@@ -XXX,XX +XXX,XX @@ echo
 echo "== reading whole image =="
 $QEMU_IO --object $SECRET -c "read 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 
+echo
+echo "== rewriting cluster part =="
+$QEMU_IO --object $SECRET -c "write -P 0xb 512 512" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+
+echo
+echo "== verify pattern =="
+$QEMU_IO --object $SECRET -c "read -P 0 0 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+$QEMU_IO --object $SECRET -c "read -P 0xb 512 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+
 echo
 echo "== rewriting whole image =="
 $QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/134.out
+++ b/tests/qemu-iotests/134.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.
 read 134217728/134217728 bytes at offset 0
 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
+== rewriting cluster part ==
+wrote 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify pattern ==
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
 == rewriting whole image ==
 wrote 134217728/134217728 bytes at offset 0
 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")
extended the l1_size check from VMDK4 to VMDK3 but did not update the
default coverage in the moved comment.

The previous vmdk4 calculation:

(512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB

The added vmdk3 calculation:

(512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB

Adding the calculation of vmdk3 to the comment.

In any case, VMware does not offer virtual disks more than 2TB for
vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is
not implemented yet in qemu.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Liran Alon <liran.alon@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com
Reviewed-by: yuchenlin <yuchenlin@synology.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
         return -EFBIG;
     }
     if (l1_size > 512 * 1024 * 1024) {
-        /* Although with big capacity and small l1_entry_sectors, we can get a
+        /*
+         * Although with big capacity and small l1_entry_sectors, we can get a
          * big l1_size, we don't want unbounded value to allocate the table.
-         * Limit it to 512M, which is 16PB for default cluster and L2 table
-         * size */
+         * Limit it to 512M, which is:
+         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
+         *            cluster size: 64KB, L2 table size: 512 entries
+         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
+         *            cluster size: 512B, L2 table size: 4096 entries
+         */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
     }
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

512M of L1 entries is a very loose bound, only 32M are required to store
the maximal supported VMDK file size of 2TB.

Fixed qemu-iotest 59# - now failure occures before on impossible L1
table size.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Liran Alon <liran.alon@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c               | 13 +++++++------
 tests/qemu-iotests/059.out |  2 +-
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
         error_setg(errp, "Invalid granularity, image may be corrupt");
         return -EFBIG;
     }
-    if (l1_size > 512 * 1024 * 1024) {
+    if (l1_size > 32 * 1024 * 1024) {
         /*
          * Although with big capacity and small l1_entry_sectors, we can get a
          * big l1_size, we don't want unbounded value to allocate the table.
-         * Limit it to 512M, which is:
-         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
-         *            cluster size: 64KB, L2 table size: 512 entries
-         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
-         *            cluster size: 512B, L2 table size: 4096 entries
+         * Limit it to 32M, which is enough to store:
+         *     8TB  - for both VMDK3 & VMDK4 with
+         *            minimal cluster size: 512B
+         *            minimal L2 table size: 512 entries
+         *            8 TB is still more than the maximal value supported for
+         *            VMDK3 & VMDK4 which is 2TB.
          */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0x140000000     0x10000         0x50000         TEST_DIR/t-s003.vmdk
 
 === Testing afl image with a very large capacity ===
-qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large
+qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big
 *** done
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in
QEMU).

This format was lacking in the following:

* Grain directory (L1) and grain table (L2) entries were 32-bit,
      allowing access to only 2TB (slightly less) of data.
    * The grain size (default) was 512 bytes - leading to data
      fragmentation and many grain tables.
    * For space reclamation purposes, it was necessary to find all the
      grains which are not pointed to by any grain table - so a reverse
      mapping of "offset of grain in vmdk" to "grain table" must be
      constructed - which takes large amounts of CPU/RAM.

The format specification can be found in VMware's documentation:
https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf

In ESXi 6.5, to support snapshot files larger than 2TB, a new format was
introduced: SESparse (Space Efficient).

This format fixes the above issues:

* All entries are now 64-bit.
    * The grain size (default) is 4KB.
    * Grain directory and grain tables are now located at the beginning
      of the file.
      + seSparse format reserves space for all grain tables.
      + Grain tables can be addressed using an index.
      + Grains are located in the end of the file and can also be
        addressed with an index.
      - seSparse vmdks of large disks (64TB) have huge preallocated
        headers - mainly due to L2 tables, even for empty snapshots.
    * The header contains a reverse mapping ("backmap") of "offset of
      grain in vmdk" to "grain table" and a bitmap ("free bitmap") which
      specifies for each grain - whether it is allocated or not.
      Using these data structures we can implement space reclamation
      efficiently.
    * Due to the fact that the header now maintains two mappings:
        * The regular one (grain directory & grain tables)
        * A reverse one (backmap and free bitmap)
      These data structures can lose consistency upon crash and result
      in a corrupted VMDK.
      Therefore, a journal is also added to the VMDK and is replayed
      when the VMware reopens the file after a crash.

Since ESXi 6.7 - SESparse is the only snapshot format available.

Unfortunately, VMware does not provide documentation regarding the new
seSparse format.

This commit is based on black-box research of the seSparse format.
Various in-guest block operations and their effect on the snapshot file
were tested.

The only VMware provided source of information (regarding the underlying
implementation) was a log file on the ESXi:

/var/log/hostd.log

Whenever an seSparse snapshot is created - the log is being populated
with seSparse records.

Relevant log records are of the form:

[...] Const Header:
[...]  constMagic     = 0xcafebabe
[...]  version        = 2.1
[...]  capacity       = 204800
[...]  grainSize      = 8
[...]  grainTableSize = 64
[...]  flags          = 0
[...] Extents:
[...]  Header         : <1 : 1>
[...]  JournalHdr     : <2 : 2>
[...]  Journal        : <2048 : 2048>
[...]  GrainDirectory : <4096 : 2048>
[...]  GrainTables    : <6144 : 2048>
[...]  FreeBitmap     : <8192 : 2048>
[...]  BackMap        : <10240 : 2048>
[...]  Grain          : <12288 : 204800>
[...] Volatile Header:
[...] volatileMagic     = 0xcafecafe
[...] FreeGTNumber      = 0
[...] nextTxnSeqNumber  = 0
[...] replayJournal     = 0

The sizes that are seen in the log file are in sectors.
Extents are of the following format: <offset : size>

This commit is a strict implementation which enforces:
    * magics
    * version number 2.1
    * grain size of 8 sectors  (4KB)
    * grain table size of 64 sectors
    * zero flags
    * extent locations

Additionally, this commit proivdes only a subset of the functionality
offered by seSparse's format:
    * Read-only
    * No journal replay
    * No space reclamation
    * No unmap support

Hence, journal header, journal, free bitmap and backmap extents are
unused, only the "classic" (L1 -> L2 -> data) grain access is
implemented.

However there are several differences in the grain access itself.
Grain directory (L1):
    * Grain directory entries are indexes (not offsets) to grain
      tables.
    * Valid grain directory entries have their highest nibble set to
      0x1.
    * Since grain tables are always located in the beginning of the
      file - the index can fit into 32 bits - so we can use its low
      part if it's valid.
Grain table (L2):
    * Grain table entries are indexes (not offsets) to grains.
    * If the highest nibble of the entry is:
        0x0:
            The grain in not allocated.
            The rest of the bytes are 0.
        0x1:
            The grain is unmapped - guest sees a zero grain.
            The rest of the bits point to the previously mapped grain,
            see 0x3 case.
        0x2:
            The grain is zero.
        0x3:
            The grain is allocated - to get the index calculate:
            ((entry & 0x0fff000000000000) >> 48) |
            ((entry & 0x0000ffffffffffff) << 12)
    * The difference between 0x1 and 0x2 is that 0x1 is an unallocated
      grain which results from the guest using sg_unmap to unmap the
      grain - but the grain itself still exists in the grain extent - a
      space reclamation procedure should delete it.
      Unmapping a zero grain has no effect (0x2 will not change to 0x1)
      but unmapping an unallocated grain will (0x0 to 0x1) - naturally.

In order to implement seSparse some fields had to be changed to support
both 32-bit and 64-bit entry sizes.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 342 insertions(+), 16 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ typedef struct {
     uint16_t compressAlgorithm;
 } QEMU_PACKED VMDK4Header;
 
+typedef struct VMDKSESparseConstHeader {
+    uint64_t magic;
+    uint64_t version;
+    uint64_t capacity;
+    uint64_t grain_size;
+    uint64_t grain_table_size;
+    uint64_t flags;
+    uint64_t reserved1;
+    uint64_t reserved2;
+    uint64_t reserved3;
+    uint64_t reserved4;
+    uint64_t volatile_header_offset;
+    uint64_t volatile_header_size;
+    uint64_t journal_header_offset;
+    uint64_t journal_header_size;
+    uint64_t journal_offset;
+    uint64_t journal_size;
+    uint64_t grain_dir_offset;
+    uint64_t grain_dir_size;
+    uint64_t grain_tables_offset;
+    uint64_t grain_tables_size;
+    uint64_t free_bitmap_offset;
+    uint64_t free_bitmap_size;
+    uint64_t backmap_offset;
+    uint64_t backmap_size;
+    uint64_t grains_offset;
+    uint64_t grains_size;
+    uint8_t pad[304];
+} QEMU_PACKED VMDKSESparseConstHeader;
+
+typedef struct VMDKSESparseVolatileHeader {
+    uint64_t magic;
+    uint64_t free_gt_number;
+    uint64_t next_txn_seq_number;
+    uint64_t replay_journal;
+    uint8_t pad[480];
+} QEMU_PACKED VMDKSESparseVolatileHeader;
+
 #define L2_CACHE_SIZE 16
 
 typedef struct VmdkExtent {
@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {
     bool compressed;
     bool has_marker;
     bool has_zero_grain;
+    bool sesparse;
+    uint64_t sesparse_l2_tables_offset;
+    uint64_t sesparse_clusters_offset;
+    int32_t entry_size;
     int version;
     int64_t sectors;
     int64_t end_sector;
     int64_t flat_start_offset;
     int64_t l1_table_offset;
     int64_t l1_backup_table_offset;
-    uint32_t *l1_table;
+    void *l1_table;
     uint32_t *l1_backup_table;
     unsigned int l1_size;
     uint32_t l1_entry_sectors;
 
     unsigned int l2_size;
-    uint32_t *l2_cache;
+    void *l2_cache;
     uint32_t l2_cache_offsets[L2_CACHE_SIZE];
     uint32_t l2_cache_counts[L2_CACHE_SIZE];
 
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
          *            minimal L2 table size: 512 entries
          *            8 TB is still more than the maximal value supported for
          *            VMDK3 & VMDK4 which is 2TB.
+         *     64TB - for "ESXi seSparse Extent"
+         *            minimal cluster size: 512B (default is 4KB)
+         *            L2 table size: 4096 entries (const).
+         *            64TB is more than the maximal value supported for
+         *            seSparse VMDKs (which is slightly less than 64TB)
          */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
     extent->l2_size = l2_size;
     extent->cluster_sectors = flat ? sectors : cluster_sectors;
     extent->next_cluster_sector = ROUND_UP(nb_sectors, cluster_sectors);
+    extent->entry_size = sizeof(uint32_t);
 
     if (s->num_extents > 1) {
         extent->end_sector = (*(extent - 1)).end_sector + extent->sectors;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
     int i;
 
     /* read the L1 table */
-    l1_size = extent->l1_size * sizeof(uint32_t);
+    l1_size = extent->l1_size * extent->entry_size;
     extent->l1_table = g_try_malloc(l1_size);
     if (l1_size && extent->l1_table == NULL) {
         return -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
         goto fail_l1;
     }
     for (i = 0; i < extent->l1_size; i++) {
-        le32_to_cpus(&extent->l1_table[i]);
+        if (extent->entry_size == sizeof(uint64_t)) {
+            le64_to_cpus((uint64_t *)extent->l1_table + i);
+        } else {
+            assert(extent->entry_size == sizeof(uint32_t));
+            le32_to_cpus((uint32_t *)extent->l1_table + i);
+        }
     }
 
     if (extent->l1_backup_table_offset) {
+        assert(!extent->sesparse);
         extent->l1_backup_table = g_try_malloc(l1_size);
         if (l1_size && extent->l1_backup_table == NULL) {
             ret = -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
     }
 
     extent->l2_cache =
-        g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE);
+        g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE);
     return 0;
  fail_l1b:
     g_free(extent->l1_backup_table);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
     return ret;
 }
 
+#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe)
+#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe)
+
+/* Strict checks - format not officially documented */
+static int check_se_sparse_const_header(VMDKSESparseConstHeader *header,
+                                        Error **errp)
+{
+    header->magic = le64_to_cpu(header->magic);
+    header->version = le64_to_cpu(header->version);
+    header->grain_size = le64_to_cpu(header->grain_size);
+    header->grain_table_size = le64_to_cpu(header->grain_table_size);
+    header->flags = le64_to_cpu(header->flags);
+    header->reserved1 = le64_to_cpu(header->reserved1);
+    header->reserved2 = le64_to_cpu(header->reserved2);
+    header->reserved3 = le64_to_cpu(header->reserved3);
+    header->reserved4 = le64_to_cpu(header->reserved4);
+
+    header->volatile_header_offset =
+        le64_to_cpu(header->volatile_header_offset);
+    header->volatile_header_size = le64_to_cpu(header->volatile_header_size);
+
+    header->journal_header_offset = le64_to_cpu(header->journal_header_offset);
+    header->journal_header_size = le64_to_cpu(header->journal_header_size);
+
+    header->journal_offset = le64_to_cpu(header->journal_offset);
+    header->journal_size = le64_to_cpu(header->journal_size);
+
+    header->grain_dir_offset = le64_to_cpu(header->grain_dir_offset);
+    header->grain_dir_size = le64_to_cpu(header->grain_dir_size);
+
+    header->grain_tables_offset = le64_to_cpu(header->grain_tables_offset);
+    header->grain_tables_size = le64_to_cpu(header->grain_tables_size);
+
+    header->free_bitmap_offset = le64_to_cpu(header->free_bitmap_offset);
+    header->free_bitmap_size = le64_to_cpu(header->free_bitmap_size);
+
+    header->backmap_offset = le64_to_cpu(header->backmap_offset);
+    header->backmap_size = le64_to_cpu(header->backmap_size);
+
+    header->grains_offset = le64_to_cpu(header->grains_offset);
+    header->grains_size = le64_to_cpu(header->grains_size);
+
+    if (header->magic != SESPARSE_CONST_HEADER_MAGIC) {
+        error_setg(errp, "Bad const header magic: 0x%016" PRIx64,
+                   header->magic);
+        return -EINVAL;
+    }
+
+    if (header->version != 0x0000000200000001) {
+        error_setg(errp, "Unsupported version: 0x%016" PRIx64,
+                   header->version);
+        return -ENOTSUP;
+    }
+
+    if (header->grain_size != 8) {
+        error_setg(errp, "Unsupported grain size: %" PRIu64,
+                   header->grain_size);
+        return -ENOTSUP;
+    }
+
+    if (header->grain_table_size != 64) {
+        error_setg(errp, "Unsupported grain table size: %" PRIu64,
+                   header->grain_table_size);
+        return -ENOTSUP;
+    }
+
+    if (header->flags != 0) {
+        error_setg(errp, "Unsupported flags: 0x%016" PRIx64,
+                   header->flags);
+        return -ENOTSUP;
+    }
+
+    if (header->reserved1 != 0 || header->reserved2 != 0 ||
+        header->reserved3 != 0 || header->reserved4 != 0) {
+        error_setg(errp, "Unsupported reserved bits:"
+                   " 0x%016" PRIx64 " 0x%016" PRIx64
+                   " 0x%016" PRIx64 " 0x%016" PRIx64,
+                   header->reserved1, header->reserved2,
+                   header->reserved3, header->reserved4);
+        return -ENOTSUP;
+    }
+
+    /* check that padding is 0 */
+    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
+        error_setg(errp, "Unsupported non-zero const header padding");
+        return -ENOTSUP;
+    }
+
+    return 0;
+}
+
+static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *header,
+                                           Error **errp)
+{
+    header->magic = le64_to_cpu(header->magic);
+    header->free_gt_number = le64_to_cpu(header->free_gt_number);
+    header->next_txn_seq_number = le64_to_cpu(header->next_txn_seq_number);
+    header->replay_journal = le64_to_cpu(header->replay_journal);
+
+    if (header->magic != SESPARSE_VOLATILE_HEADER_MAGIC) {
+        error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64,
+                   header->magic);
+        return -EINVAL;
+    }
+
+    if (header->replay_journal) {
+        error_setg(errp, "Image is dirty, Replaying journal not supported");
+        return -ENOTSUP;
+    }
+
+    /* check that padding is 0 */
+    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
+        error_setg(errp, "Unsupported non-zero volatile header padding");
+        return -ENOTSUP;
+    }
+
+    return 0;
+}
+
+static int vmdk_open_se_sparse(BlockDriverState *bs,
+                               BdrvChild *file,
+                               int flags, Error **errp)
+{
+    int ret;
+    VMDKSESparseConstHeader const_header;
+    VMDKSESparseVolatileHeader volatile_header;
+    VmdkExtent *extent;
+
+    ret = bdrv_apply_auto_read_only(bs,
+            "No write support for seSparse images available", errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    assert(sizeof(const_header) == SECTOR_SIZE);
+
+    ret = bdrv_pread(file, 0, &const_header, sizeof(const_header));
+    if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
+        error_setg_errno(errp, -ret,
+                         "Could not read const header from file '%s'",
+                         file->bs->filename);
+        return ret;
+    }
+
+    /* check const header */
+    ret = check_se_sparse_const_header(&const_header, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    assert(sizeof(volatile_header) == SECTOR_SIZE);
+
+    ret = bdrv_pread(file,
+                     const_header.volatile_header_offset * SECTOR_SIZE,
+                     &volatile_header, sizeof(volatile_header));
+    if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
+        error_setg_errno(errp, -ret,
+                         "Could not read volatile header from file '%s'",
+                         file->bs->filename);
+        return ret;
+    }
+
+    /* check volatile header */
+    ret = check_se_sparse_volatile_header(&volatile_header, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    ret = vmdk_add_extent(bs, file, false,
+                          const_header.capacity,
+                          const_header.grain_dir_offset * SECTOR_SIZE,
+                          0,
+                          const_header.grain_dir_size *
+                          SECTOR_SIZE / sizeof(uint64_t),
+                          const_header.grain_table_size *
+                          SECTOR_SIZE / sizeof(uint64_t),
+                          const_header.grain_size,
+                          &extent,
+                          errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    extent->sesparse = true;
+    extent->sesparse_l2_tables_offset = const_header.grain_tables_offset;
+    extent->sesparse_clusters_offset = const_header.grains_offset;
+    extent->entry_size = sizeof(uint64_t);
+
+    ret = vmdk_init_tables(bs, extent, errp);
+    if (ret) {
+        /* free extent allocated by vmdk_add_extent */
+        vmdk_free_last_extent(bs);
+    }
+
+    return ret;
+}
+
 static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
                                QDict *options, Error **errp);
 
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          * RW [size in sectors] SPARSE "file-name.vmdk"
          * RW [size in sectors] VMFS "file-name.vmdk"
          * RW [size in sectors] VMFSSPARSE "file-name.vmdk"
+         * RW [size in sectors] SESPARSE "file-name.vmdk"
          */
         flat_offset = -1;
         matches = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
 
         if (sectors <= 0 ||
             (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
-             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) ||
+             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") &&
+             strcmp(type, "SESPARSE")) ||
             (strcmp(access, "RW"))) {
             continue;
         }
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                 return ret;
             }
             extent = &s->extents[s->num_extents - 1];
+        } else if (!strcmp(type, "SESPARSE")) {
+            ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
+            if (ret) {
+                bdrv_unref_child(bs, extent_file);
+                return ret;
+            }
+            extent = &s->extents[s->num_extents - 1];
         } else {
             error_setg(errp, "Unsupported extent type '%s'", type);
             bdrv_unref_child(bs, extent_file);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
     if (strcmp(ct, "monolithicFlat") &&
         strcmp(ct, "vmfs") &&
         strcmp(ct, "vmfsSparse") &&
+        strcmp(ct, "seSparse") &&
         strcmp(ct, "twoGbMaxExtentSparse") &&
         strcmp(ct, "twoGbMaxExtentFlat")) {
         error_setg(errp, "Unsupported image type '%s'", ct);
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
 {
     unsigned int l1_index, l2_offset, l2_index;
     int min_index, i, j;
-    uint32_t min_count, *l2_table;
+    uint32_t min_count;
+    void *l2_table;
     bool zeroed = false;
     int64_t ret;
     int64_t cluster_sector;
+    unsigned int l2_size_bytes = extent->l2_size * extent->entry_size;
 
     if (m_data) {
         m_data->valid = 0;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
     if (l1_index >= extent->l1_size) {
         return VMDK_ERROR;
     }
-    l2_offset = extent->l1_table[l1_index];
+    if (extent->sesparse) {
+        uint64_t l2_offset_u64;
+
+        assert(extent->entry_size == sizeof(uint64_t));
+
+        l2_offset_u64 = ((uint64_t *)extent->l1_table)[l1_index];
+        if (l2_offset_u64 == 0) {
+            l2_offset = 0;
+        } else if ((l2_offset_u64 & 0xffffffff00000000) != 0x1000000000000000) {
+            /*
+             * Top most nibble is 0x1 if grain table is allocated.
+             * strict check - top most 4 bytes must be 0x10000000 since max
+             * supported size is 64TB for disk - so no more than 64TB / 16MB
+             * grain directories which is smaller than uint32,
+             * where 16MB is the only supported default grain table coverage.
+             */
+            return VMDK_ERROR;
+        } else {
+            l2_offset_u64 = l2_offset_u64 & 0x00000000ffffffff;
+            l2_offset_u64 = extent->sesparse_l2_tables_offset +
+                l2_offset_u64 * l2_size_bytes / SECTOR_SIZE;
+            if (l2_offset_u64 > 0x00000000ffffffff) {
+                return VMDK_ERROR;
+            }
+            l2_offset = (unsigned int)(l2_offset_u64);
+        }
+    } else {
+        assert(extent->entry_size == sizeof(uint32_t));
+        l2_offset = ((uint32_t *)extent->l1_table)[l1_index];
+    }
     if (!l2_offset) {
         return VMDK_UNALLOC;
     }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
                     extent->l2_cache_counts[j] >>= 1;
                 }
             }
-            l2_table = extent->l2_cache + (i * extent->l2_size);
+            l2_table = (char *)extent->l2_cache + (i * l2_size_bytes);
             goto found;
         }
     }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
             min_index = i;
         }
     }
-    l2_table = extent->l2_cache + (min_index * extent->l2_size);
+    l2_table = (char *)extent->l2_cache + (min_index * l2_size_bytes);
     BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD);
     if (bdrv_pread(extent->file,
                 (int64_t)l2_offset * 512,
                 l2_table,
-                extent->l2_size * sizeof(uint32_t)
-            ) != extent->l2_size * sizeof(uint32_t)) {
+                l2_size_bytes
+            ) != l2_size_bytes) {
         return VMDK_ERROR;
     }
 
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
     extent->l2_cache_counts[min_index] = 1;
  found:
     l2_index = ((offset >> 9) / extent->cluster_sectors) % extent->l2_size;
-    cluster_sector = le32_to_cpu(l2_table[l2_index]);
 
-    if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
-        zeroed = true;
+    if (extent->sesparse) {
+        cluster_sector = le64_to_cpu(((uint64_t *)l2_table)[l2_index]);
+        switch (cluster_sector & 0xf000000000000000) {
+        case 0x0000000000000000:
+            /* unallocated grain */
+            if (cluster_sector != 0) {
+                return VMDK_ERROR;
+            }
+            break;
+        case 0x1000000000000000:
+            /* scsi-unmapped grain - fallthrough */
+        case 0x2000000000000000:
+            /* zero grain */
+            zeroed = true;
+            break;
+        case 0x3000000000000000:
+            /* allocated grain */
+            cluster_sector = (((cluster_sector & 0x0fff000000000000) >> 48) |
+                              ((cluster_sector & 0x0000ffffffffffff) << 12));
+            cluster_sector = extent->sesparse_clusters_offset +
+                cluster_sector * extent->cluster_sectors;
+            break;
+        default:
+            return VMDK_ERROR;
+        }
+    } else {
+        cluster_sector = le32_to_cpu(((uint32_t *)l2_table)[l2_index]);
+
+        if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
+            zeroed = true;
+        }
     }
 
     if (!cluster_sector || zeroed) {
         if (!allocate) {
             return zeroed ? VMDK_ZEROED : VMDK_UNALLOC;
         }
+        assert(!extent->sesparse);
 
         if (extent->next_cluster_sector >= VMDK_EXTENT_MAX_SECTORS) {
             return VMDK_ERROR;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
             m_data->l1_index = l1_index;
             m_data->l2_index = l2_index;
             m_data->l2_offset = l2_offset;
-            m_data->l2_cache_entry = &l2_table[l2_index];
+            m_data->l2_cache_entry = ((uint32_t *)l2_table) + l2_index;
         }
     }
     *cluster_offset = cluster_sector << BDRV_SECTOR_BITS;
@@ -XXX,XX +XXX,XX @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_t offset,
         if (!extent) {
             return -EIO;
         }
+        if (extent->sesparse) {
+            return -ENOTSUP;
+        }
         offset_in_cluster = vmdk_find_offset_in_cluster(extent, offset);
         n_bytes = MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE
                              - offset_in_cluster);
-- 
2.21.0

From: Pino Toscano <ptoscano@redhat.com>

Rewrite the implementation of the ssh block driver to use libssh instead
of libssh2.  The libssh library has various advantages over libssh2:
- easier API for authentication (for example for using ssh-agent)
- easier API for known_hosts handling
- supports newer types of keys in known_hosts

Use APIs/features available in libssh 0.8 conditionally, to support
older versions (which are not recommended though).

Adjust the iotest 207 according to the different error message, and to
find the default key type for localhost (to properly compare the
fingerprint with).
Contributed-by: Max Reitz <mreitz@redhat.com>

Adjust the various Docker/Travis scripts to use libssh when available
instead of libssh2. The mingw/mxe testing is dropped for now, as there
are no packages for it.

Signed-off-by: Pino Toscano <ptoscano@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20190620200840.17655-1-ptoscano@redhat.com
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 configure                                     |  65 +-
 block/Makefile.objs                           |   6 +-
 block/ssh.c                                   | 652 ++++++++++--------
 .travis.yml                                   |   4 +-
 block/trace-events                            |  14 +-
 docs/qemu-block-drivers.texi                  |   2 +-
 .../dockerfiles/debian-win32-cross.docker     |   1 -
 .../dockerfiles/debian-win64-cross.docker     |   1 -
 tests/docker/dockerfiles/fedora.docker        |   4 +-
 tests/docker/dockerfiles/ubuntu.docker        |   2 +-
 tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
 tests/qemu-iotests/207                        |  54 +-
 tests/qemu-iotests/207.out                    |   2 +-
 13 files changed, 449 insertions(+), 360 deletions(-)

diff --git a/configure b/configure
index XXXXXXX..XXXXXXX 100755
--- a/configure
+++ b/configure
@@ -XXX,XX +XXX,XX @@ auth_pam=""
 vte=""
 virglrenderer=""
 tpm=""
-libssh2=""
+libssh=""
 live_block_migration="yes"
 numa=""
 tcmalloc="no"
@@ -XXX,XX +XXX,XX @@ for opt do
   ;;
   --enable-tpm) tpm="yes"
   ;;
-  --disable-libssh2) libssh2="no"
+  --disable-libssh) libssh="no"
   ;;
-  --enable-libssh2) libssh2="yes"
+  --enable-libssh) libssh="yes"
   ;;
   --disable-live-block-migration) live_block_migration="no"
   ;;
@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
   coroutine-pool  coroutine freelist (better performance)
   glusterfs       GlusterFS backend
   tpm             TPM support
-  libssh2         ssh block device support
+  libssh          ssh block device support
   numa            libnuma support
   libxml2         for Parallels image format
   tcmalloc        tcmalloc support
@@ -XXX,XX +XXX,XX @@ EOF
 fi
 
 ##########################################
-# libssh2 probe
-min_libssh2_version=1.2.8
-if test "$libssh2" != "no" ; then
-  if $pkg_config --atleast-version=$min_libssh2_version libssh2; then
-    libssh2_cflags=$($pkg_config libssh2 --cflags)
-    libssh2_libs=$($pkg_config libssh2 --libs)
-    libssh2=yes
+# libssh probe
+if test "$libssh" != "no" ; then
+  if $pkg_config --exists libssh; then
+    libssh_cflags=$($pkg_config libssh --cflags)
+    libssh_libs=$($pkg_config libssh --libs)
+    libssh=yes
   else
-    if test "$libssh2" = "yes" ; then
-      error_exit "libssh2 >= $min_libssh2_version required for --enable-libssh2"
+    if test "$libssh" = "yes" ; then
+      error_exit "libssh required for --enable-libssh"
     fi
-    libssh2=no
+    libssh=no
   fi
 fi
 
 ##########################################
-# libssh2_sftp_fsync probe
+# Check for libssh 0.8
+# This is done like this instead of using the LIBSSH_VERSION_* and
+# SSH_VERSION_* macros because some distributions in the past shipped
+# snapshots of the future 0.8 from Git, and those snapshots did not
+# have updated version numbers (still referring to 0.7.0).
 
-if test "$libssh2" = "yes"; then
+if test "$libssh" = "yes"; then
   cat > $TMPC <<EOF
-#include <stdio.h>
-#include <libssh2.h>
-#include <libssh2_sftp.h>
-int main(void) {
-    LIBSSH2_SESSION *session;
-    LIBSSH2_SFTP *sftp;
-    LIBSSH2_SFTP_HANDLE *sftp_handle;
-    session = libssh2_session_init ();
-    sftp = libssh2_sftp_init (session);
-    sftp_handle = libssh2_sftp_open (sftp, "/", 0, 0);
-    libssh2_sftp_fsync (sftp_handle);
-    return 0;
-}
+#include <libssh/libssh.h>
+int main(void) { return ssh_get_server_publickey(NULL, NULL); }
 EOF
-  # libssh2_cflags/libssh2_libs defined in previous test.
-  if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then
-    QEMU_CFLAGS="-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS"
+  if compile_prog "$libssh_cflags" "$libssh_libs"; then
+    libssh_cflags="-DHAVE_LIBSSH_0_8 $libssh_cflags"
   fi
 fi
 
@@ -XXX,XX +XXX,XX @@ echo "GlusterFS support $glusterfs"
 echo "gcov              $gcov_tool"
 echo "gcov enabled      $gcov"
 echo "TPM support       $tpm"
-echo "libssh2 support   $libssh2"
+echo "libssh support    $libssh"
 echo "QOM debugging     $qom_cast_debug"
 echo "Live block migration $live_block_migration"
 echo "lzo support       $lzo"
@@ -XXX,XX +XXX,XX @@ if test "$glusterfs_iocb_has_stat" = "yes" ; then
   echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=y" >> $config_host_mak
 fi
 
-if test "$libssh2" = "yes" ; then
-  echo "CONFIG_LIBSSH2=m" >> $config_host_mak
-  echo "LIBSSH2_CFLAGS=$libssh2_cflags" >> $config_host_mak
-  echo "LIBSSH2_LIBS=$libssh2_libs" >> $config_host_mak
+if test "$libssh" = "yes" ; then
+  echo "CONFIG_LIBSSH=m" >> $config_host_mak
+  echo "LIBSSH_CFLAGS=$libssh_cflags" >> $config_host_mak
+  echo "LIBSSH_LIBS=$libssh_libs" >> $config_host_mak
 fi
 
 if test "$live_block_migration" = "yes" ; then
diff --git a/block/Makefile.objs b/block/Makefile.objs
index XXXXXXX..XXXXXXX 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_CURL) += curl.o
 block-obj-$(CONFIG_RBD) += rbd.o
 block-obj-$(CONFIG_GLUSTERFS) += gluster.o
 block-obj-$(CONFIG_VXHS) += vxhs.o
-block-obj-$(CONFIG_LIBSSH2) += ssh.o
+block-obj-$(CONFIG_LIBSSH) += ssh.o
 block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 block-obj-y += backup.o
@@ -XXX,XX +XXX,XX @@ rbd.o-libs         := $(RBD_LIBS)
 gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
 gluster.o-libs     := $(GLUSTERFS_LIBS)
 vxhs.o-libs        := $(VXHS_LIBS)
-ssh.o-cflags       := $(LIBSSH2_CFLAGS)
-ssh.o-libs         := $(LIBSSH2_LIBS)
+ssh.o-cflags       := $(LIBSSH_CFLAGS)
+ssh.o-libs         := $(LIBSSH_LIBS)
 block-obj-dmg-bz2-$(CONFIG_BZIP2) += dmg-bz2.o
 block-obj-$(if $(CONFIG_DMG),m,n) += $(block-obj-dmg-bz2-y)
 dmg-bz2.o-libs     := $(BZIP2_LIBS)
diff --git a/block/ssh.c b/block/ssh.c
index XXXXXXX..XXXXXXX 100644
--- a/block/ssh.c
+++ b/block/ssh.c
@@ -XXX,XX +XXX,XX @@
 
 #include "qemu/osdep.h"
 
-#include <libssh2.h>
-#include <libssh2_sftp.h>
+#include <libssh/libssh.h>
+#include <libssh/sftp.h>
 
 #include "block/block_int.h"
 #include "block/qdict.h"
@@ -XXX,XX +XXX,XX @@
 #include "trace.h"
 
 /*
- * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
- * that this requires that libssh2 was specially compiled with the
- * `./configure --enable-debug' option, so most likely you will have
- * to compile it yourself.  The meaning of <bitmask> is described
- * here: http://www.libssh2.org/libssh2_trace.html
+ * TRACE_LIBSSH=<level> enables tracing in libssh itself.
+ * The meaning of <level> is described here:
+ * http://api.libssh.org/master/group__libssh__log.html
  */
-#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
+#define TRACE_LIBSSH  0 /* see: SSH_LOG_* */
 
 typedef struct BDRVSSHState {
     /* Coroutine. */
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVSSHState {
 
     /* SSH connection. */
     int sock;                         /* socket */
-    LIBSSH2_SESSION *session;         /* ssh session */
-    LIBSSH2_SFTP *sftp;               /* sftp session */
-    LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */
+    ssh_session session;              /* ssh session */
+    sftp_session sftp;                /* sftp session */
+    sftp_file sftp_handle;            /* sftp remote file handle */
 
-    /* See ssh_seek() function below. */
-    int64_t offset;
-    bool offset_op_read;
-
-    /* File attributes at open.  We try to keep the .filesize field
+    /*
+     * File attributes at open.  We try to keep the .size field
      * updated if it changes (eg by writing at the end of the file).
      */
-    LIBSSH2_SFTP_ATTRIBUTES attrs;
+    sftp_attributes attrs;
 
     InetSocketAddress *inet;
 
@@ -XXX,XX +XXX,XX @@ static void ssh_state_init(BDRVSSHState *s)
 {
     memset(s, 0, sizeof *s);
     s->sock = -1;
-    s->offset = -1;
     qemu_co_mutex_init(&s->lock);
 }
 
@@ -XXX,XX +XXX,XX @@ static void ssh_state_free(BDRVSSHState *s)
 {
     g_free(s->user);
 
+    if (s->attrs) {
+        sftp_attributes_free(s->attrs);
+    }
     if (s->sftp_handle) {
-        libssh2_sftp_close(s->sftp_handle);
+        sftp_close(s->sftp_handle);
     }
     if (s->sftp) {
-        libssh2_sftp_shutdown(s->sftp);
+        sftp_free(s->sftp);
     }
     if (s->session) {
-        libssh2_session_disconnect(s->session,
-                                   "from qemu ssh client: "
-                                   "user closed the connection");
-        libssh2_session_free(s->session);
-    }
-    if (s->sock >= 0) {
-        close(s->sock);
+        ssh_disconnect(s->session);
+        ssh_free(s->session); /* This frees s->sock */
     }
 }
 
@@ -XXX,XX +XXX,XX @@ session_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
     va_end(args);
 
     if (s->session) {
-        char *ssh_err;
+        const char *ssh_err;
         int ssh_err_code;
 
-        /* This is not an errno.  See <libssh2.h>. */
-        ssh_err_code = libssh2_session_last_error(s->session,
-                                                  &ssh_err, NULL, 0);
-        error_setg(errp, "%s: %s (libssh2 error code: %d)",
+        /* This is not an errno.  See <libssh/libssh.h>. */
+        ssh_err = ssh_get_error(s->session);
+        ssh_err_code = ssh_get_error_code(s->session);
+        error_setg(errp, "%s: %s (libssh error code: %d)",
                    msg, ssh_err, ssh_err_code);
     } else {
         error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
     va_end(args);
 
     if (s->sftp) {
-        char *ssh_err;
+        const char *ssh_err;
         int ssh_err_code;
-        unsigned long sftp_err_code;
+        int sftp_err_code;
 
-        /* This is not an errno.  See <libssh2.h>. */
-        ssh_err_code = libssh2_session_last_error(s->session,
-                                                  &ssh_err, NULL, 0);
-        /* See <libssh2_sftp.h>. */
-        sftp_err_code = libssh2_sftp_last_error((s)->sftp);
+        /* This is not an errno.  See <libssh/libssh.h>. */
+        ssh_err = ssh_get_error(s->session);
+        ssh_err_code = ssh_get_error_code(s->session);
+        /* See <libssh/sftp.h>. */
+        sftp_err_code = sftp_get_error(s->sftp);
 
         error_setg(errp,
-                   "%s: %s (libssh2 error code: %d, sftp error code: %lu)",
+                   "%s: %s (libssh error code: %d, sftp error code: %d)",
                    msg, ssh_err, ssh_err_code, sftp_err_code);
     } else {
         error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
 
 static void sftp_error_trace(BDRVSSHState *s, const char *op)
 {
-    char *ssh_err;
+    const char *ssh_err;
     int ssh_err_code;
-    unsigned long sftp_err_code;
+    int sftp_err_code;
 
-    /* This is not an errno.  See <libssh2.h>. */
-    ssh_err_code = libssh2_session_last_error(s->session,
-                                              &ssh_err, NULL, 0);
-    /* See <libssh2_sftp.h>. */
-    sftp_err_code = libssh2_sftp_last_error((s)->sftp);
+    /* This is not an errno.  See <libssh/libssh.h>. */
+    ssh_err = ssh_get_error(s->session);
+    ssh_err_code = ssh_get_error_code(s->session);
+    /* See <libssh/sftp.h>. */
+    sftp_err_code = sftp_get_error(s->sftp);
 
     trace_sftp_error(op, ssh_err, ssh_err_code, sftp_err_code);
 }
@@ -XXX,XX +XXX,XX @@ static void ssh_parse_filename(const char *filename, QDict *options,
     parse_uri(filename, options, errp);
 }
 
-static int check_host_key_knownhosts(BDRVSSHState *s,
-                                     const char *host, int port, Error **errp)
+static int check_host_key_knownhosts(BDRVSSHState *s, Error **errp)
 {
-    const char *home;
-    char *knh_file = NULL;
-    LIBSSH2_KNOWNHOSTS *knh = NULL;
-    struct libssh2_knownhost *found;
-    int ret, r;
-    const char *hostkey;
-    size_t len;
-    int type;
-
-    hostkey = libssh2_session_hostkey(s->session, &len, &type);
-    if (!hostkey) {
+    int ret;
+#ifdef HAVE_LIBSSH_0_8
+    enum ssh_known_hosts_e state;
+    int r;
+    ssh_key pubkey;
+    enum ssh_keytypes_e pubkey_type;
+    unsigned char *server_hash = NULL;
+    size_t server_hash_len;
+    char *fingerprint = NULL;
+
+    state = ssh_session_is_known_server(s->session);
+    trace_ssh_server_status(state);
+
+    switch (state) {
+    case SSH_KNOWN_HOSTS_OK:
+        /* OK */
+        trace_ssh_check_host_key_knownhosts();
+        break;
+    case SSH_KNOWN_HOSTS_CHANGED:
         ret = -EINVAL;
-        session_error_setg(errp, s, "failed to read remote host key");
+        r = ssh_get_server_publickey(s->session, &pubkey);
+        if (r == 0) {
+            r = ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256,
+                                       &server_hash, &server_hash_len);
+            pubkey_type = ssh_key_type(pubkey);
+            ssh_key_free(pubkey);
+        }
+        if (r == 0) {
+            fingerprint = ssh_get_fingerprint_hash(SSH_PUBLICKEY_HASH_SHA256,
+                                                   server_hash,
+                                                   server_hash_len);
+            ssh_clean_pubkey_hash(&server_hash);
+        }
+        if (fingerprint) {
+            error_setg(errp,
+                       "host key (%s key with fingerprint %s) does not match "
+                       "the one in known_hosts; this may be a possible attack",
+                       ssh_key_type_to_char(pubkey_type), fingerprint);
+            ssh_string_free_char(fingerprint);
+        } else  {
+            error_setg(errp,
+                       "host key does not match the one in known_hosts; this "
+                       "may be a possible attack");
+        }
         goto out;
-    }
-
-    knh = libssh2_knownhost_init(s->session);
-    if (!knh) {
+    case SSH_KNOWN_HOSTS_OTHER:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                           "failed to initialize known hosts support");
+        error_setg(errp,
+                   "host key for this server not found, another type exists");
+        goto out;
+    case SSH_KNOWN_HOSTS_UNKNOWN:
+        ret = -EINVAL;
+        error_setg(errp, "no host key was found in known_hosts");
+        goto out;
+    case SSH_KNOWN_HOSTS_NOT_FOUND:
+        ret = -ENOENT;
+        error_setg(errp, "known_hosts file not found");
+        goto out;
+    case SSH_KNOWN_HOSTS_ERROR:
+        ret = -EINVAL;
+        error_setg(errp, "error while checking the host");
+        goto out;
+    default:
+        ret = -EINVAL;
+        error_setg(errp, "error while checking for known server (%d)", state);
         goto out;
     }
+#else /* !HAVE_LIBSSH_0_8 */
+    int state;
 
-    home = getenv("HOME");
-    if (home) {
-        knh_file = g_strdup_printf("%s/.ssh/known_hosts", home);
-    } else {
-        knh_file = g_strdup_printf("/root/.ssh/known_hosts");
-    }
-
-    /* Read all known hosts from OpenSSH-style known_hosts file. */
-    libssh2_knownhost_readfile(knh, knh_file, LIBSSH2_KNOWNHOST_FILE_OPENSSH);
+    state = ssh_is_server_known(s->session);
+    trace_ssh_server_status(state);
 
-    r = libssh2_knownhost_checkp(knh, host, port, hostkey, len,
-                                 LIBSSH2_KNOWNHOST_TYPE_PLAIN|
-                                 LIBSSH2_KNOWNHOST_KEYENC_RAW,
-                                 &found);
-    switch (r) {
-    case LIBSSH2_KNOWNHOST_CHECK_MATCH:
+    switch (state) {
+    case SSH_SERVER_KNOWN_OK:
         /* OK */
-        trace_ssh_check_host_key_knownhosts(found->key);
+        trace_ssh_check_host_key_knownhosts();
         break;
-    case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
+    case SSH_SERVER_KNOWN_CHANGED:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                      "host key does not match the one in known_hosts"
-                      " (found key %s)", found->key);
+        error_setg(errp,
+                   "host key does not match the one in known_hosts; this "
+                   "may be a possible attack");
         goto out;
-    case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND:
+    case SSH_SERVER_FOUND_OTHER:
         ret = -EINVAL;
-        session_error_setg(errp, s, "no host key was found in known_hosts");
+        error_setg(errp,
+                   "host key for this server not found, another type exists");
+        goto out;
+    case SSH_SERVER_FILE_NOT_FOUND:
+        ret = -ENOENT;
+        error_setg(errp, "known_hosts file not found");
         goto out;
-    case LIBSSH2_KNOWNHOST_CHECK_FAILURE:
+    case SSH_SERVER_NOT_KNOWN:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                      "failure matching the host key with known_hosts");
+        error_setg(errp, "no host key was found in known_hosts");
+        goto out;
+    case SSH_SERVER_ERROR:
+        ret = -EINVAL;
+        error_setg(errp, "server error");
         goto out;
     default:
         ret = -EINVAL;
-        session_error_setg(errp, s, "unknown error matching the host key"
-                      " with known_hosts (%d)", r);
+        error_setg(errp, "error while checking for known server (%d)", state);
         goto out;
     }
+#endif /* !HAVE_LIBSSH_0_8 */
 
     /* known_hosts checking successful. */
     ret = 0;
 
  out:
-    if (knh != NULL) {
-        libssh2_knownhost_free(knh);
-    }
-    g_free(knh_file);
     return ret;
 }
 
@@ -XXX,XX +XXX,XX @@ static int compare_fingerprint(const unsigned char *fingerprint, size_t len,
 
 static int
 check_host_key_hash(BDRVSSHState *s, const char *hash,
-                    int hash_type, size_t fingerprint_len, Error **errp)
+                    enum ssh_publickey_hash_type type, Error **errp)
 {
-    const char *fingerprint;
-
-    fingerprint = libssh2_hostkey_hash(s->session, hash_type);
-    if (!fingerprint) {
+    int r;
+    ssh_key pubkey;
+    unsigned char *server_hash;
+    size_t server_hash_len;
+
+#ifdef HAVE_LIBSSH_0_8
+    r = ssh_get_server_publickey(s->session, &pubkey);
+#else
+    r = ssh_get_publickey(s->session, &pubkey);
+#endif
+    if (r != SSH_OK) {
         session_error_setg(errp, s, "failed to read remote host key");
         return -EINVAL;
     }
 
-    if(compare_fingerprint((unsigned char *) fingerprint, fingerprint_len,
-                           hash) != 0) {
+    r = ssh_get_publickey_hash(pubkey, type, &server_hash, &server_hash_len);
+    ssh_key_free(pubkey);
+    if (r != 0) {
+        session_error_setg(errp, s,
+                           "failed reading the hash of the server SSH key");
+        return -EINVAL;
+    }
+
+    r = compare_fingerprint(server_hash, server_hash_len, hash);
+    ssh_clean_pubkey_hash(&server_hash);
+    if (r != 0) {
         error_setg(errp, "remote host key does not match host_key_check '%s'",
                    hash);
         return -EPERM;
@@ -XXX,XX +XXX,XX @@ check_host_key_hash(BDRVSSHState *s, const char *hash,
     return 0;
 }
 
-static int check_host_key(BDRVSSHState *s, const char *host, int port,
-                          SshHostKeyCheck *hkc, Error **errp)
+static int check_host_key(BDRVSSHState *s, SshHostKeyCheck *hkc, Error **errp)
 {
     SshHostKeyCheckMode mode;
 
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
     case SSH_HOST_KEY_CHECK_MODE_HASH:
         if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) {
             return check_host_key_hash(s, hkc->u.hash.hash,
-                                       LIBSSH2_HOSTKEY_HASH_MD5, 16, errp);
+                                       SSH_PUBLICKEY_HASH_MD5, errp);
         } else if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_SHA1) {
             return check_host_key_hash(s, hkc->u.hash.hash,
-                                       LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp);
+                                       SSH_PUBLICKEY_HASH_SHA1, errp);
         }
         g_assert_not_reached();
         break;
     case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS:
-        return check_host_key_knownhosts(s, host, port, errp);
+        return check_host_key_knownhosts(s, errp);
     default:
         g_assert_not_reached();
     }
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
     return -EINVAL;
 }
 
-static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
+static int authenticate(BDRVSSHState *s, Error **errp)
 {
     int r, ret;
-    const char *userauthlist;
-    LIBSSH2_AGENT *agent = NULL;
-    struct libssh2_agent_publickey *identity;
-    struct libssh2_agent_publickey *prev_identity = NULL;
+    int method;
 
-    userauthlist = libssh2_userauth_list(s->session, user, strlen(user));
-    if (strstr(userauthlist, "publickey") == NULL) {
+    /* Try to authenticate with the "none" method. */
+    r = ssh_userauth_none(s->session, NULL);
+    if (r == SSH_AUTH_ERROR) {
         ret = -EPERM;
-        error_setg(errp,
-                "remote server does not support \"publickey\" authentication");
+        session_error_setg(errp, s, "failed to authenticate using none "
+                                    "authentication");
         goto out;
-    }
-
-    /* Connect to ssh-agent and try each identity in turn. */
-    agent = libssh2_agent_init(s->session);
-    if (!agent) {
-        ret = -EINVAL;
-        session_error_setg(errp, s, "failed to initialize ssh-agent support");
-        goto out;
-    }
-    if (libssh2_agent_connect(agent)) {
-        ret = -ECONNREFUSED;
-        session_error_setg(errp, s, "failed to connect to ssh-agent");
-        goto out;
-    }
-    if (libssh2_agent_list_identities(agent)) {
-        ret = -EINVAL;
-        session_error_setg(errp, s,
-                           "failed requesting identities from ssh-agent");
+    } else if (r == SSH_AUTH_SUCCESS) {
+        /* Authenticated! */
+        ret = 0;
         goto out;
     }
 
-    for(;;) {
-        r = libssh2_agent_get_identity(agent, &identity, prev_identity);
-        if (r == 1) {           /* end of list */
-            break;
-        }
-        if (r < 0) {
+    method = ssh_userauth_list(s->session, NULL);
+    trace_ssh_auth_methods(method);
+
+    /*
+     * Try to authenticate with publickey, using the ssh-agent
+     * if available.
+     */
+    if (method & SSH_AUTH_METHOD_PUBLICKEY) {
+        r = ssh_userauth_publickey_auto(s->session, NULL, NULL);
+        if (r == SSH_AUTH_ERROR) {
             ret = -EINVAL;
-            session_error_setg(errp, s,
-                               "failed to obtain identity from ssh-agent");
+            session_error_setg(errp, s, "failed to authenticate using "
+                                        "publickey authentication");
             goto out;
-        }
-        r = libssh2_agent_userauth(agent, user, identity);
-        if (r == 0) {
+        } else if (r == SSH_AUTH_SUCCESS) {
             /* Authenticated! */
             ret = 0;
             goto out;
         }
-        /* Failed to authenticate with this identity, try the next one. */
-        prev_identity = identity;
     }
 
     ret = -EPERM;
@@ -XXX,XX +XXX,XX @@ static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
                "and the identities held by your ssh-agent");
 
  out:
-    if (agent != NULL) {
-        /* Note: libssh2 implementation implicitly calls
-         * libssh2_agent_disconnect if necessary.
-         */
-        libssh2_agent_free(agent);
-    }
-
     return ret;
 }
 
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
                           int ssh_flags, int creat_mode, Error **errp)
 {
     int r, ret;
-    long port = 0;
+    unsigned int port = 0;
+    int new_sock = -1;
 
     if (opts->has_user) {
         s->user = g_strdup(opts->user);
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
     s->inet = opts->server;
     opts->server = NULL;
 
-    if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) {
+    if (qemu_strtoui(s->inet->port, NULL, 10, &port) < 0) {
         error_setg(errp, "Use only numeric port value");
         ret = -EINVAL;
         goto err;
     }
 
     /* Open the socket and connect. */
-    s->sock = inet_connect_saddr(s->inet, errp);
-    if (s->sock < 0) {
+    new_sock = inet_connect_saddr(s->inet, errp);
+    if (new_sock < 0) {
         ret = -EIO;
         goto err;
     }
 
+    /*
+     * Try to disable the Nagle algorithm on TCP sockets to reduce latency,
+     * but do not fail if it cannot be disabled.
+     */
+    r = socket_set_nodelay(new_sock);
+    if (r < 0) {
+        warn_report("can't set TCP_NODELAY for the ssh server %s: %s",
+                    s->inet->host, strerror(errno));
+    }
+
     /* Create SSH session. */
-    s->session = libssh2_session_init();
+    s->session = ssh_new();
     if (!s->session) {
         ret = -EINVAL;
-        session_error_setg(errp, s, "failed to initialize libssh2 session");
+        session_error_setg(errp, s, "failed to initialize libssh session");
         goto err;
     }
 
-#if TRACE_LIBSSH2 != 0
-    libssh2_trace(s->session, TRACE_LIBSSH2);
-#endif
+    /*
+     * Make sure we are in blocking mode during the connection and
+     * authentication phases.
+     */
+    ssh_set_blocking(s->session, 1);
 
-    r = libssh2_session_handshake(s->session, s->sock);
-    if (r != 0) {
+    r = ssh_options_set(s->session, SSH_OPTIONS_USER, s->user);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the user in the libssh session");
+        goto err;
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_HOST, s->inet->host);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the host in the libssh session");
+        goto err;
+    }
+
+    if (port > 0) {
+        r = ssh_options_set(s->session, SSH_OPTIONS_PORT, &port);
+        if (r < 0) {
+            ret = -EINVAL;
+            session_error_setg(errp, s,
+                               "failed to set the port in the libssh session");
+            goto err;
+        }
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_COMPRESSION, "none");
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to disable the compression in the libssh "
+                           "session");
+        goto err;
+    }
+
+    /* Read ~/.ssh/config. */
+    r = ssh_options_parse_config(s->session, NULL);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s, "failed to parse ~/.ssh/config");
+        goto err;
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_FD, &new_sock);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the socket in the libssh session");
+        goto err;
+    }
+    /* libssh took ownership of the socket. */
+    s->sock = new_sock;
+    new_sock = -1;
+
+    /* Connect. */
+    r = ssh_connect(s->session);
+    if (r != SSH_OK) {
         ret = -EINVAL;
         session_error_setg(errp, s, "failed to establish SSH session");
         goto err;
     }
 
     /* Check the remote host's key against known_hosts. */
-    ret = check_host_key(s, s->inet->host, port, opts->host_key_check, errp);
+    ret = check_host_key(s, opts->host_key_check, errp);
     if (ret < 0) {
         goto err;
     }
 
     /* Authenticate. */
-    ret = authenticate(s, s->user, errp);
+    ret = authenticate(s, errp);
     if (ret < 0) {
         goto err;
     }
 
     /* Start SFTP. */
-    s->sftp = libssh2_sftp_init(s->session);
+    s->sftp = sftp_new(s->session);
     if (!s->sftp) {
-        session_error_setg(errp, s, "failed to initialize sftp handle");
+        session_error_setg(errp, s, "failed to create sftp handle");
+        ret = -EINVAL;
+        goto err;
+    }
+
+    r = sftp_init(s->sftp);
+    if (r < 0) {
+        sftp_error_setg(errp, s, "failed to initialize sftp handle");
         ret = -EINVAL;
         goto err;
     }
 
     /* Open the remote file. */
     trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
-    s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
-                                       creat_mode);
+    s->sftp_handle = sftp_open(s->sftp, opts->path, ssh_flags, creat_mode);
     if (!s->sftp_handle) {
-        session_error_setg(errp, s, "failed to open remote file '%s'",
-                           opts->path);
+        sftp_error_setg(errp, s, "failed to open remote file '%s'",
+                        opts->path);
         ret = -EINVAL;
         goto err;
     }
 
-    r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs);
-    if (r < 0) {
+    /* Make sure the SFTP file is handled in blocking mode. */
+    sftp_file_set_blocking(s->sftp_handle);
+
+    s->attrs = sftp_fstat(s->sftp_handle);
+    if (!s->attrs) {
         sftp_error_setg(errp, s, "failed to read file attributes");
         return -EINVAL;
     }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
     return 0;
 
  err:
+    if (s->attrs) {
+        sftp_attributes_free(s->attrs);
+    }
+    s->attrs = NULL;
     if (s->sftp_handle) {
-        libssh2_sftp_close(s->sftp_handle);
+        sftp_close(s->sftp_handle);
     }
     s->sftp_handle = NULL;
     if (s->sftp) {
-        libssh2_sftp_shutdown(s->sftp);
+        sftp_free(s->sftp);
     }
     s->sftp = NULL;
     if (s->session) {
-        libssh2_session_disconnect(s->session,
-                                   "from qemu ssh client: "
-                                   "error opening connection");
-        libssh2_session_free(s->session);
+        ssh_disconnect(s->session);
+        ssh_free(s->session);
     }
     s->session = NULL;
+    s->sock = -1;
+    if (new_sock >= 0) {
+        close(new_sock);
+    }
 
     return ret;
 }
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
 
     ssh_state_init(s);
 
-    ssh_flags = LIBSSH2_FXF_READ;
+    ssh_flags = 0;
     if (bdrv_flags & BDRV_O_RDWR) {
-        ssh_flags |= LIBSSH2_FXF_WRITE;
+        ssh_flags |= O_RDWR;
+    } else {
+        ssh_flags |= O_RDONLY;
     }
 
     opts = ssh_parse_options(options, errp);
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
     }
 
     /* Go non-blocking. */
-    libssh2_session_set_blocking(s->session, 0);
+    ssh_set_blocking(s->session, 0);
 
     qapi_free_BlockdevOptionsSsh(opts);
 
     return 0;
 
  err:
-    if (s->sock >= 0) {
-        close(s->sock);
-    }
-    s->sock = -1;
-
     qapi_free_BlockdevOptionsSsh(opts);
 
     return ret;
@@ -XXX,XX +XXX,XX @@ static int ssh_grow_file(BDRVSSHState *s, int64_t offset, Error **errp)
 {
     ssize_t ret;
     char c[1] = { '\0' };
-    int was_blocking = libssh2_session_get_blocking(s->session);
+    int was_blocking = ssh_is_blocking(s->session);
 
     /* offset must be strictly greater than the current size so we do
      * not overwrite anything */
-    assert(offset > 0 && offset > s->attrs.filesize);
+    assert(offset > 0 && offset > s->attrs->size);
 
-    libssh2_session_set_blocking(s->session, 1);
+    ssh_set_blocking(s->session, 1);
 
-    libssh2_sftp_seek64(s->sftp_handle, offset - 1);
-    ret = libssh2_sftp_write(s->sftp_handle, c, 1);
+    sftp_seek64(s->sftp_handle, offset - 1);
+    ret = sftp_write(s->sftp_handle, c, 1);
 
-    libssh2_session_set_blocking(s->session, was_blocking);
+    ssh_set_blocking(s->session, was_blocking);
 
     if (ret < 0) {
         sftp_error_setg(errp, s, "Failed to grow file");
         return -EIO;
     }
 
-    s->attrs.filesize = offset;
+    s->attrs->size = offset;
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ static int ssh_co_create(BlockdevCreateOptions *options, Error **errp)
     ssh_state_init(&s);
 
     ret = connect_to_ssh(&s, opts->location,
-                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
-                         LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
+                         O_RDWR | O_CREAT | O_TRUNC,
                          0644, errp);
     if (ret < 0) {
         goto fail;
@@ -XXX,XX +XXX,XX @@ static int ssh_has_zero_init(BlockDriverState *bs)
     /* Assume false, unless we can positively prove it's true. */
     int has_zero_init = 0;
 
-    if (s->attrs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) {
-        if (s->attrs.permissions & LIBSSH2_SFTP_S_IFREG) {
-            has_zero_init = 1;
-        }
+    if (s->attrs->type == SSH_FILEXFER_TYPE_REGULAR) {
+        has_zero_init = 1;
     }
 
     return has_zero_init;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
         .co = qemu_coroutine_self()
     };
 
-    r = libssh2_session_block_directions(s->session);
+    r = ssh_get_poll_flags(s->session);
 
-    if (r & LIBSSH2_SESSION_BLOCK_INBOUND) {
+    if (r & SSH_READ_PENDING) {
         rd_handler = restart_coroutine;
     }
-    if (r & LIBSSH2_SESSION_BLOCK_OUTBOUND) {
+    if (r & SSH_WRITE_PENDING) {
         wr_handler = restart_coroutine;
     }
 
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
     trace_ssh_co_yield_back(s->sock);
 }
 
-/* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
- * in the remote file.  Notice that it just updates a field in the
- * sftp_handle structure, so there is no network traffic and it cannot
- * fail.
- *
- * However, `libssh2_sftp_seek64' does have a catastrophic effect on
- * performance since it causes the handle to throw away all in-flight
- * reads and buffered readahead data.  Therefore this function tries
- * to be intelligent about when to call the underlying libssh2 function.
- */
-#define SSH_SEEK_WRITE 0
-#define SSH_SEEK_READ  1
-#define SSH_SEEK_FORCE 2
-
-static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
-{
-    bool op_read = (flags & SSH_SEEK_READ) != 0;
-    bool force = (flags & SSH_SEEK_FORCE) != 0;
-
-    if (force || op_read != s->offset_op_read || offset != s->offset) {
-        trace_ssh_seek(offset);
-        libssh2_sftp_seek64(s->sftp_handle, offset);
-        s->offset = offset;
-        s->offset_op_read = op_read;
-    }
-}
-
 static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
                                  int64_t offset, size_t size,
                                  QEMUIOVector *qiov)
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
 
     trace_ssh_read(offset, size);
 
-    ssh_seek(s, offset, SSH_SEEK_READ);
+    trace_ssh_seek(offset);
+    sftp_seek64(s->sftp_handle, offset);
 
     /* This keeps track of the current iovec element ('i'), where we
      * will write to next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
     buf = i->iov_base;
     end_of_vec = i->iov_base + i->iov_len;
 
-    /* libssh2 has a hard-coded limit of 2000 bytes per request,
-     * although it will also do readahead behind our backs.  Therefore
-     * we may have to do repeated reads here until we have read 'size'
-     * bytes.
-     */
     for (got = 0; got < size; ) {
+        size_t request_read_size;
     again:
-        trace_ssh_read_buf(buf, end_of_vec - buf);
-        r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
-        trace_ssh_read_return(r);
+        /*
+         * The size of SFTP packets is limited to 32K bytes, so limit
+         * the amount of data requested to 16K, as libssh currently
+         * does not handle multiple requests on its own.
+         */
+        request_read_size = MIN(end_of_vec - buf, 16384);
+        trace_ssh_read_buf(buf, end_of_vec - buf, request_read_size);
+        r = sftp_read(s->sftp_handle, buf, request_read_size);
+        trace_ssh_read_return(r, sftp_get_error(s->sftp));
 
-        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+        if (r == SSH_AGAIN) {
             co_yield(s, bs);
             goto again;
         }
-        if (r < 0) {
-            sftp_error_trace(s, "read");
-            s->offset = -1;
-            return -EIO;
-        }
-        if (r == 0) {
+        if (r == SSH_EOF || (r == 0 && sftp_get_error(s->sftp) == SSH_FX_EOF)) {
             /* EOF: Short read so pad the buffer with zeroes and return it. */
             qemu_iovec_memset(qiov, got, 0, size - got);
             return 0;
         }
+        if (r <= 0) {
+            sftp_error_trace(s, "read");
+            return -EIO;
+        }
 
         got += r;
         buf += r;
-        s->offset += r;
         if (buf >= end_of_vec && got < size) {
             i++;
             buf = i->iov_base;
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
 
     trace_ssh_write(offset, size);
 
-    ssh_seek(s, offset, SSH_SEEK_WRITE);
+    trace_ssh_seek(offset);
+    sftp_seek64(s->sftp_handle, offset);
 
     /* This keeps track of the current iovec element ('i'), where we
      * will read from next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
     end_of_vec = i->iov_base + i->iov_len;
 
     for (written = 0; written < size; ) {
+        size_t request_write_size;
     again:
-        trace_ssh_write_buf(buf, end_of_vec - buf);
-        r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
-        trace_ssh_write_return(r);
+        /*
+         * Avoid too large data packets, as libssh currently does not
+         * handle multiple requests on its own.
+         */
+        request_write_size = MIN(end_of_vec - buf, 131072);
+        trace_ssh_write_buf(buf, end_of_vec - buf, request_write_size);
+        r = sftp_write(s->sftp_handle, buf, request_write_size);
+        trace_ssh_write_return(r, sftp_get_error(s->sftp));
 
-        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+        if (r == SSH_AGAIN) {
             co_yield(s, bs);
             goto again;
         }
         if (r < 0) {
             sftp_error_trace(s, "write");
-            s->offset = -1;
             return -EIO;
         }
-        /* The libssh2 API is very unclear about this.  A comment in
-         * the code says "nothing was acked, and no EAGAIN was
-         * received!" which apparently means that no data got sent
-         * out, and the underlying channel didn't return any EAGAIN
-         * indication.  I think this is a bug in either libssh2 or
-         * OpenSSH (server-side).  In any case, forcing a seek (to
-         * discard libssh2 internal buffers), and then trying again
-         * works for me.
-         */
-        if (r == 0) {
-            ssh_seek(s, offset + written, SSH_SEEK_WRITE|SSH_SEEK_FORCE);
-            co_yield(s, bs);
-            goto again;
-        }
 
         written += r;
         buf += r;
-        s->offset += r;
         if (buf >= end_of_vec && written < size) {
             i++;
             buf = i->iov_base;
             end_of_vec = i->iov_base + i->iov_len;
         }
 
-        if (offset + written > s->attrs.filesize)
-            s->attrs.filesize = offset + written;
+        if (offset + written > s->attrs->size) {
+            s->attrs->size = offset + written;
+        }
     }
 
     return 0;
@@ -XXX,XX +XXX,XX @@ static void unsafe_flush_warning(BDRVSSHState *s, const char *what)
     }
 }
 
-#ifdef HAS_LIBSSH2_SFTP_FSYNC
+#ifdef HAVE_LIBSSH_0_8
 
 static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
 {
     int r;
 
     trace_ssh_flush();
+
+    if (!sftp_extension_supported(s->sftp, "fsync@openssh.com", "1")) {
+        unsafe_flush_warning(s, "OpenSSH >= 6.3");
+        return 0;
+    }
  again:
-    r = libssh2_sftp_fsync(s->sftp_handle);
-    if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+    r = sftp_fsync(s->sftp_handle);
+    if (r == SSH_AGAIN) {
         co_yield(s, bs);
         goto again;
     }
-    if (r == LIBSSH2_ERROR_SFTP_PROTOCOL &&
-        libssh2_sftp_last_error(s->sftp) == LIBSSH2_FX_OP_UNSUPPORTED) {
-        unsafe_flush_warning(s, "OpenSSH >= 6.3");
-        return 0;
-    }
     if (r < 0) {
         sftp_error_trace(s, "fsync");
         return -EIO;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
     return ret;
 }
 
-#else /* !HAS_LIBSSH2_SFTP_FSYNC */
+#else /* !HAVE_LIBSSH_0_8 */
 
 static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
 {
     BDRVSSHState *s = bs->opaque;
 
-    unsafe_flush_warning(s, "libssh2 >= 1.4.4");
+    unsafe_flush_warning(s, "libssh >= 0.8.0");
     return 0;
 }
 
-#endif /* !HAS_LIBSSH2_SFTP_FSYNC */
+#endif /* !HAVE_LIBSSH_0_8 */
 
 static int64_t ssh_getlength(BlockDriverState *bs)
 {
     BDRVSSHState *s = bs->opaque;
     int64_t length;
 
-    /* Note we cannot make a libssh2 call here. */
-    length = (int64_t) s->attrs.filesize;
+    /* Note we cannot make a libssh call here. */
+    length = (int64_t) s->attrs->size;
     trace_ssh_getlength(length);
 
     return length;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_truncate(BlockDriverState *bs, int64_t offset,
         return -ENOTSUP;
     }
 
-    if (offset < s->attrs.filesize) {
+    if (offset < s->attrs->size) {
         error_setg(errp, "ssh driver does not support shrinking files");
         return -ENOTSUP;
     }
 
-    if (offset == s->attrs.filesize) {
+    if (offset == s->attrs->size) {
         return 0;
     }
 
@@ -XXX,XX +XXX,XX @@ static void bdrv_ssh_init(void)
 {
     int r;
 
-    r = libssh2_init(0);
+    r = ssh_init();
     if (r != 0) {
-        fprintf(stderr, "libssh2 initialization failed, %d\n", r);
+        fprintf(stderr, "libssh initialization failed, %d\n", r);
         exit(EXIT_FAILURE);
     }
 
+#if TRACE_LIBSSH != 0
+    ssh_set_log_level(TRACE_LIBSSH);
+#endif
+
     bdrv_register(&bdrv_ssh);
 }
 
diff --git a/.travis.yml b/.travis.yml
index XXXXXXX..XXXXXXX 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -XXX,XX +XXX,XX @@ addons:
       - libseccomp-dev
       - libspice-protocol-dev
       - libspice-server-dev
-      - libssh2-1-dev
+      - libssh-dev
       - liburcu-dev
       - libusb-1.0-0-dev
       - libvte-2.91-dev
@@ -XXX,XX +XXX,XX @@ matrix:
             - libseccomp-dev
             - libspice-protocol-dev
             - libspice-server-dev
-            - libssh2-1-dev
+            - libssh-dev
             - liburcu-dev
             - libusb-1.0-0-dev
             - libvte-2.91-dev
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ nbd_client_connect_success(const char *export_name) "export '%s'"
 # ssh.c
 ssh_restart_coroutine(void *co) "co=%p"
 ssh_flush(void) "fsync"
-ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
+ssh_check_host_key_knownhosts(void) "host key OK"
 ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
 ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
 ssh_co_yield_back(int sock) "s->sock=%d - back"
 ssh_getlength(int64_t length) "length=%" PRIi64
 ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
 ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
-ssh_read_return(ssize_t ret) "sftp_read returned %zd"
+ssh_read_buf(void *buf, size_t size, size_t actual_size) "sftp_read buf=%p size=%zu (actual size=%zu)"
+ssh_read_return(ssize_t ret, int sftp_err) "sftp_read returned %zd (sftp error=%d)"
 ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
-ssh_write_return(ssize_t ret) "sftp_write returned %zd"
+ssh_write_buf(void *buf, size_t size, size_t actual_size) "sftp_write buf=%p size=%zu (actual size=%zu)"
+ssh_write_return(ssize_t ret, int sftp_err) "sftp_write returned %zd (sftp error=%d)"
 ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
+ssh_auth_methods(int methods) "auth methods=0x%x"
+ssh_server_status(int status) "server status=%d"
 
 # curl.c
 curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
@@ -XXX,XX +XXX,XX @@ sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
 sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
 
 # ssh.c
-sftp_error(const char *op, const char *ssh_err, int ssh_err_code, unsigned long sftp_err_code) "%s failed: %s (libssh2 error code: %d, sftp error code: %lu)"
+sftp_error(const char *op, const char *ssh_err, int ssh_err_code, int sftp_err_code) "%s failed: %s (libssh error code: %d, sftp error code: %d)"
diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi
index XXXXXXX..XXXXXXX 100644
--- a/docs/qemu-block-drivers.texi
+++ b/docs/qemu-block-drivers.texi
@@ -XXX,XX +XXX,XX @@ print a warning when @code{fsync} is not supported:
 
 warning: ssh server @code{ssh.example.com:22} does not support fsync
 
-With sufficiently new versions of libssh2 and OpenSSH, @code{fsync} is
+With sufficiently new versions of libssh and OpenSSH, @code{fsync} is
 supported.
 
 @node disk_images_nvme
diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/debian-win32-cross.docker
+++ b/tests/docker/dockerfiles/debian-win32-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         mxe-$TARGET-w64-mingw32.shared-curl \
         mxe-$TARGET-w64-mingw32.shared-glib \
         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
         mxe-$TARGET-w64-mingw32.shared-libusb1 \
         mxe-$TARGET-w64-mingw32.shared-lzo \
         mxe-$TARGET-w64-mingw32.shared-nettle \
diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/debian-win64-cross.docker
+++ b/tests/docker/dockerfiles/debian-win64-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         mxe-$TARGET-w64-mingw32.shared-curl \
         mxe-$TARGET-w64-mingw32.shared-glib \
         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
         mxe-$TARGET-w64-mingw32.shared-libusb1 \
         mxe-$TARGET-w64-mingw32.shared-lzo \
         mxe-$TARGET-w64-mingw32.shared-nettle \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     libpng-devel \
     librbd-devel \
     libseccomp-devel \
-    libssh2-devel \
+    libssh-devel \
     libubsan \
     libusbx-devel \
     libxml2-devel \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     mingw32-gtk3 \
     mingw32-libjpeg-turbo \
     mingw32-libpng \
-    mingw32-libssh2 \
     mingw32-libtasn1 \
     mingw32-nettle \
     mingw32-pixman \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     mingw64-gtk3 \
     mingw64-libjpeg-turbo \
     mingw64-libpng \
-    mingw64-libssh2 \
     mingw64-libtasn1 \
     mingw64-nettle \
     mingw64-pixman \
diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
-    libssh2-1-dev \
+    libssh-dev \
     libusb-1.0-0-dev \
     libusbredirhost-dev \
     libvdeplug-dev \
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
-    libssh2-1-dev \
+    libssh-dev \
     libusb-1.0-0-dev \
     libusbredirhost-dev \
     libvdeplug-dev \
diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
 
     iotests.img_info_log(remote_path)
 
-    md5_key = subprocess.check_output(
-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-        'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-        shell=True).rstrip().decode('ascii')
+    keys = subprocess.check_output(
+        'ssh-keyscan 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
+        'cut -d" " -f3',
+        shell=True).rstrip().decode('ascii').split('\n')
+
+    # Mappings of base64 representations to digests
+    md5_keys = {}
+    sha1_keys = {}
+
+    for key in keys:
+        md5_keys[key] = subprocess.check_output(
+            'echo %s | base64 -d | md5sum -b | cut -d" " -f1' % key,
+            shell=True).rstrip().decode('ascii')
+
+        sha1_keys[key] = subprocess.check_output(
+            'echo %s | base64 -d | sha1sum -b | cut -d" " -f1' % key,
+            shell=True).rstrip().decode('ascii')
 
     vm.launch()
+
+    # Find correct key first
+    matching_key = None
+    for key in keys:
+        result = vm.qmp('blockdev-add',
+                        driver='ssh', node_name='node0', path=disk_path,
+                        server={
+                             'host': '127.0.0.1',
+                             'port': '22',
+                        }, host_key_check={
+                             'mode': 'hash',
+                             'type': 'md5',
+                             'hash': md5_keys[key],
+                        })
+
+        if 'error' not in result:
+            vm.qmp('blockdev-del', node_name='node0')
+            matching_key = key
+            break
+
+    if matching_key is None:
+        vm.shutdown()
+        iotests.notrun('Did not find a key that fits 127.0.0.1')
+
     blockdev_create(vm, { 'driver': 'ssh',
                           'location': {
                               'path': disk_path,
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                               'host-key-check': {
                                   'mode': 'hash',
                                   'type': 'md5',
-                                  'hash': md5_key,
+                                  'hash': md5_keys[matching_key],
                               }
                           },
                           'size': 8388608 })
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
 
     iotests.img_info_log(remote_path)
 
-    sha1_key = subprocess.check_output(
-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-        'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-        shell=True).rstrip().decode('ascii')
-
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
                           'location': {
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                               'host-key-check': {
                                   'mode': 'hash',
                                   'type': 'sha1',
-                                  'hash': sha1_key,
+                                  'hash': sha1_keys[matching_key],
                               }
                           },
                           'size': 4194304 })
diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/207.out
+++ b/tests/qemu-iotests/207.out
@@ -XXX,XX +XXX,XX @@ virtual size: 4 MiB (4194304 bytes)
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
-Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
+Job failed: failed to open remote file '/this/is/not/an/existing/path': SFTP server: No such file (libssh error code: 1, sftp error code: 2)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-- 
2.21.0