Series comparison

-[PULL 00/23] Block patches
+[Qemu-devel] [PULL v2 0/8] Block patches
-The following changes since commit 9d662a6b22a0838a85c5432385f35db2488a33a5:
+The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:
-  Merge remote-tracking branch 'remotes/legoater/tags/pull-ppc-20220305' into staging (2022-03-05 18:03:15 +0000)
+  Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)
 are available in the Git repository at:
-  https://gitlab.com/hreitz/qemu.git tags/pull-block-2022-03-07
+  https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24
-for you to fetch changes up to 743da0b401cdc3ee94bc519975e339a3cdbe0ad1:
+for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:
-  iotests/image-fleecing: test push backup with fleecing (2022-03-07 09:33:31 +0100)
+  iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)
 ----------------------------------------------------------------
-Block patches for 7.0-rc0:
+Block patches:
-- New fleecing backup scheme
+- The SSH block driver now uses libssh instead of libssh2
-- iotest fixes
+- The VMDK block driver gets read-only support for the seSparse
-- Fixes for the curl block driver
+  subformat
-- Fix for the preallocate block driver
+- Various fixes
-- IDE fix for zero-length TRIM requests
 ---
 v2:
 - Squashed Pino's fix for pre-0.8 libssh into the libssh patch
 ----------------------------------------------------------------
-Hanna Reitz (2):
+Anton Nefedov (1):
-  ide: Increment BB in-flight counter for TRIM BH
+  iotest 134: test cluster-misaligned encrypted write
   iotests: Write test output to TEST_DIR
-Peter Maydell (2):
+Klaus Birkelund Jensen (1):
-  block/curl.c: Set error message string if curl_init_state() fails
+  nvme: do not advertise support for unsupported arbitration mechanism
   block/curl.c: Check error return from curl_easy_setopt()
-Thomas Huth (2):
+Max Reitz (1):
-  tests/qemu-iotests/040: Skip TestCommitWithFilters without 'throttle'
+  iotests: Fix 205 for concurrent runs
   tests/qemu-iotests/testrunner: Quote "case not run" lines in TAP mode
-Vladimir Sementsov-Ogievskiy (17):
+Pino Toscano (1):
-  block: fix preallocate filter: don't do unaligned preallocate requests
+  ssh: switch from libssh2 to libssh
   block/block-copy: move copy_bitmap initialization to
     block_copy_state_new()
   block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value
   block/block-copy: block_copy_state_new(): add bitmap parameter
   block/copy-before-write: add bitmap open parameter
   block/block-copy: add block_copy_reset()
   block: intoduce reqlist
   block/reqlist: reqlist_find_conflict(): use ranges_overlap()
   block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()
   block/reqlist: add reqlist_wait_all()
   block/io: introduce block driver snapshot-access API
   block: introduce snapshot-access block driver
   block: copy-before-write: realize snapshot-access API
   iotests/image-fleecing: add test-case for fleecing format node
   iotests.py: add qemu_io_pipe_and_status()
   iotests/image-fleecing: add test case with bitmap
   iotests/image-fleecing: test push backup with fleecing
- qapi/block-core.json                        |  14 +-
+Sam Eiderman (3):
- include/block/block-common.h                |   3 +-
+  vmdk: Fix comment regarding max l1_size coverage
- include/block/block-copy.h                  |   2 +
+  vmdk: Reduce the max bound for L1 table size
- include/block/block_int-common.h            |  24 ++
+  vmdk: Add read-only support for seSparse snapshots
- include/block/block_int-io.h                |   9 +
- include/block/dirty-bitmap.h                |   4 +-
+Vladimir Sementsov-Ogievskiy (1):
- include/block/reqlist.h                     |  75 ++++++
+  blockdev: enable non-root nodes for transaction drive-backup source
- include/qemu/hbitmap.h                      |  12 +
- block/block-copy.c                          | 150 +++++------
+ configure                                     |  65 +-
- block/copy-before-write.c                   | 265 +++++++++++++++++++-
+ block/Makefile.objs                           |   6 +-
- block/curl.c                                |  92 ++++---
+ block/ssh.c                                   | 652 ++++++++++--------
- block/dirty-bitmap.c                        |  15 +-
+ block/vmdk.c                                  | 372 +++++++++-
- block/io.c                                  |  76 ++++++
+ blockdev.c                                    |   2 +-
- block/monitor/bitmap-qmp-cmds.c             |   5 +-
+ hw/block/nvme.c                               |   1 -
- block/preallocate.c                         |  15 +-
+ .travis.yml                                   |   4 +-
- block/reqlist.c                             |  85 +++++++
+ block/trace-events                            |  14 +-
- block/snapshot-access.c                     | 132 ++++++++++
+ docs/qemu-block-drivers.texi                  |   2 +-
- hw/ide/core.c                               |   7 +
+ .../dockerfiles/debian-win32-cross.docker     |   1 -
- util/hbitmap.c                              |  33 +++
+ .../dockerfiles/debian-win64-cross.docker     |   1 -
- MAINTAINERS                                 |   5 +-
+ tests/docker/dockerfiles/fedora.docker        |   4 +-
- block/meson.build                           |   2 +
+ tests/docker/dockerfiles/ubuntu.docker        |   2 +-
- tests/qemu-iotests/040                      |   1 +
+ tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
- tests/qemu-iotests/257.out                  | 224 +++++++++++++++++
+ tests/qemu-iotests/059.out                    |   2 +-
- tests/qemu-iotests/common.rc                |   6 +-
+ tests/qemu-iotests/134                        |   9 +
- tests/qemu-iotests/iotests.py               |   8 +-
+ tests/qemu-iotests/134.out                    |  10 +
- tests/qemu-iotests/testenv.py               |   5 +-
+ tests/qemu-iotests/205                        |   2 +-
- tests/qemu-iotests/testrunner.py            |  19 +-
+ tests/qemu-iotests/207                        |  54 +-
- tests/qemu-iotests/tests/image-fleecing     | 185 +++++++++++---
+ tests/qemu-iotests/207.out                    |   2 +-
- tests/qemu-iotests/tests/image-fleecing.out | 221 +++++++++++++++-
+files changed, 823 insertions(+), 384 deletions(-)
 files changed, 1499 insertions(+), 195 deletions(-)
  create mode 100644 include/block/reqlist.h
  create mode 100644 block/reqlist.c
  create mode 100644 block/snapshot-access.c
 --
-.34.1
+.21.0

-[PULL 23/23] iotests/image-fleecing: test push backup with fleecing
+[Qemu-devel] [PULL v2 1/8] nvme: do not advertise support for unsupported arbitration mechanism
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Klaus Birkelund Jensen <klaus@birkelund.eu>
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+The device mistakenly reports that the Weighted Round Robin with Urgent
-Message-Id: <20220303194349.2304213-17-vsementsov@virtuozzo.com>
+Priority Class arbitration mechanism is supported.
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 It is not.
 Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
 Message-id: 20190606092530.14206-1-klaus@birkelund.eu
 Acked-by: Maxim Levitsky <mlevitsk@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/tests/image-fleecing     | 125 +++++++++++++++-----
+ hw/block/nvme.c | 1 -
- tests/qemu-iotests/tests/image-fleecing.out |  63 ++++++++++
+file changed, 1 deletion(-)
 files changed, 156 insertions(+), 32 deletions(-)
-diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
+diff --git a/hw/block/nvme.c b/hw/block/nvme.c
 index XXXXXXX..XXXXXXX 100755
 --- a/tests/qemu-iotests/tests/image-fleecing
 +++ b/tests/qemu-iotests/tests/image-fleecing
@@ -XXX,XX +XXX,XX @@ remainder = [('0xd5', '0x108000',  '32k'), # Right-end of partial-left [1]
               ('0xdc', '32M',       '32k'), # Left-end of partial-right [2]
               ('0xcd', '0x3ff0000', '64k')] # patterns[3]
 -def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
 -            fleece_img_path, nbd_sock_path, vm,
 +def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
 +            fleece_img_path, nbd_sock_path=None,
 +            target_img_path=None,
              bitmap=False):
 +    push_backup = target_img_path is not None
 +    assert (nbd_sock_path is not None) != push_backup
 +    if push_backup:
 +        assert use_cbw
 +
      log('--- Setting up images ---')
      log('')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
      else:
          assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
 +    if push_backup:
 +        assert qemu_img('create', '-f', 'qcow2', target_img_path, '64M') == 0
 +
      for p in patterns:
          qemu_io('-f', iotests.imgfmt,
                  '-c', 'write -P%s %s %s' % p, base_img_path)
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
      export_node = 'fl-access' if use_snapshot_access_filter else tmp_node
 -    log('')
 -    log('--- Setting up NBD Export ---')
 -    log('')
 +    if push_backup:
 +        log('')
 +        log('--- Starting actual backup ---')
 +        log('')
 -    nbd_uri = 'nbd+unix:///%s?socket=%s' % (export_node, nbd_sock_path)
 -    log(vm.qmp('nbd-server-start',
 -               {'addr': {'type': 'unix',
 -                         'data': {'path': nbd_sock_path}}}))
 +        log(vm.qmp('blockdev-add', **{
 +            'driver': iotests.imgfmt,
 +            'node-name': 'target',
 +            'file': {
 +                'driver': 'file',
 +                'filename': target_img_path
 +            }
 +        }))
 +        log(vm.qmp('blockdev-backup', device=export_node,
 +                   sync='full', target='target',
 +                   job_id='push-backup', speed=1))
 +    else:
 +        log('')
 +        log('--- Setting up NBD Export ---')
 +        log('')
 -    log(vm.qmp('nbd-server-add', device=export_node))
 +        nbd_uri = 'nbd+unix:///%s?socket=%s' % (export_node, nbd_sock_path)
 +        log(vm.qmp('nbd-server-start',
 +                   {'addr': { 'type': 'unix',
 +                              'data': { 'path': nbd_sock_path } } }))
 -    log('')
 -    log('--- Sanity Check ---')
 -    log('')
 +        log(vm.qmp('nbd-server-add', device=export_node))
 -    for p in patterns + zeroes:
 -        cmd = 'read -P%s %s %s' % p
 -        log(cmd)
 -        out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
 -                                           nbd_uri)
 -        if ret != 0:
 -            print(out)
 +        log('')
 +        log('--- Sanity Check ---')
 +        log('')
 +
 +        for p in patterns + zeroes:
 +            cmd = 'read -P%s %s %s' % p
 +            log(cmd)
 +            out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
 +                                               nbd_uri)
 +            if ret != 0:
 +                print(out)
      log('')
      log('--- Testing COW ---')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
          log(cmd)
          log(vm.hmp_qemu_io(qom_path, cmd, qdev=True))
 +    if push_backup:
 +        # Check that previous operations were done during backup, not after
 +        # If backup is already finished, it's possible that it was finished
 +        # even before hmp qemu_io write, and we didn't actually test
 +        # copy-before-write operation. This should not happen, as we use
 +        # speed=1. But worth checking.
 +        result = vm.qmp('query-block-jobs')
 +        assert len(result['return']) == 1
 +
 +        result = vm.qmp('block-job-set-speed', device='push-backup', speed=0)
 +        assert result == {'return': {}}
 +
 +        log(vm.event_wait(name='BLOCK_JOB_COMPLETED',
 +                          match={'data': {'device': 'push-backup'}}),
 +            filters=[iotests.filter_qmp_event])
 +        log(vm.qmp('blockdev-del', node_name='target'))
 +
      log('')
      log('--- Verifying Data ---')
      log('')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
      for p in patterns + zeroes:
          cmd = 'read -P%s %s %s' % p
          log(cmd)
 -        out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
 -                                           nbd_uri)
 +        args = ['-r', '-c', cmd]
 +        if push_backup:
 +            args += [target_img_path]
 +        else:
 +            args += ['-f', 'raw', nbd_uri]
 +        out, ret = qemu_io_pipe_and_status(*args)
          if ret != 0:
              print(out)
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
      log('--- Cleanup ---')
      log('')
 -    log(vm.qmp('nbd-server-stop'))
 +    if not push_backup:
 +        log(vm.qmp('nbd-server-stop'))
      if use_cbw:
          if use_snapshot_access_filter:
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
      log('Done')
 -def test(use_cbw, use_snapshot_access_filter, bitmap=False):
 +def test(use_cbw, use_snapshot_access_filter,
 +         nbd_sock_path=None, target_img_path=None, bitmap=False):
      with iotests.FilePath('base.img') as base_img_path, \
           iotests.FilePath('fleece.img') as fleece_img_path, \
 -         iotests.FilePath('nbd.sock',
 -                          base_dir=iotests.sock_dir) as nbd_sock_path, \
           iotests.VM() as vm:
 -        do_test(use_cbw, use_snapshot_access_filter, base_img_path,
 -                fleece_img_path, nbd_sock_path, vm, bitmap=bitmap)
 +        do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
 +                fleece_img_path, nbd_sock_path, target_img_path,
 +                bitmap=bitmap)
 +
 +def test_pull(use_cbw, use_snapshot_access_filter, bitmap=False):
 +    with iotests.FilePath('nbd.sock',
 +                          base_dir=iotests.sock_dir) as nbd_sock_path:
 +        test(use_cbw, use_snapshot_access_filter, nbd_sock_path, None,
 +             bitmap=bitmap)
 +
 +def test_push():
 +    with iotests.FilePath('target.img') as target_img_path:
 +        test(True, True, None, target_img_path)
  log('=== Test backup(sync=none) based fleecing ===\n')
 -test(False, False)
 +test_pull(False, False)
  log('=== Test cbw-filter based fleecing ===\n')
 -test(True, False)
 +test_pull(True, False)
  log('=== Test fleecing-format based fleecing ===\n')
 -test(True, True)
 +test_pull(True, True)
  log('=== Test fleecing-format based fleecing with bitmap ===\n')
 -test(True, True, bitmap=True)
 +test_pull(True, True, bitmap=True)
 +
 +log('=== Test push backup with fleecing ===\n')
 +test_push()
 diff --git a/tests/qemu-iotests/tests/image-fleecing.out b/tests/qemu-iotests/tests/image-fleecing.out
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/tests/image-fleecing.out
+--- a/hw/block/nvme.c
-+++ b/tests/qemu-iotests/tests/image-fleecing.out
++++ b/hw/block/nvme.c
-@@ -XXX,XX +XXX,XX @@ read -P0xdc 32M 32k
+@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
- read -P0xcd 0x3ff0000 64k
+     n->bar.cap = 0;
+     NVME_CAP_SET_MQES(n->bar.cap, 0x7ff);
- Done
+     NVME_CAP_SET_CQR(n->bar.cap, 1);
-+=== Test push backup with fleecing ===
+-    NVME_CAP_SET_AMS(n->bar.cap, 1);
-+
+     NVME_CAP_SET_TO(n->bar.cap, 0xf);
-+--- Setting up images ---
+     NVME_CAP_SET_CSS(n->bar.cap, 1);
-+
+     NVME_CAP_SET_MPSMAX(n->bar.cap, 4);
 +Done
 +
 +--- Launching VM ---
 +
 +Done
 +
 +--- Setting up Fleecing Graph ---
 +
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +
 +--- Starting actual backup ---
 +
 +{"return": {}}
 +{"return": {}}
 +
 +--- Testing COW ---
 +
 +write -P0xab 0 64k
 +{"return": ""}
 +write -P0xad 0x00f8000 64k
 +{"return": ""}
 +write -P0x1d 0x2008000 64k
 +{"return": ""}
 +write -P0xea 0x3fe0000 64k
 +{"return": ""}
 +{"data": {"device": "push-backup", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 +{"return": {}}
 +
 +--- Verifying Data ---
 +
 +read -P0x5d 0 64k
 +read -P0xd5 1M 64k
 +read -P0xdc 32M 64k
 +read -P0xcd 0x3ff0000 64k
 +read -P0 0x00f8000 32k
 +read -P0 0x2010000 32k
 +read -P0 0x3fe0000 64k
 +
 +--- Cleanup ---
 +
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +
 +--- Confirming writes ---
 +
 +read -P0xab 0 64k
 +read -P0xad 0x00f8000 64k
 +read -P0x1d 0x2008000 64k
 +read -P0xea 0x3fe0000 64k
 +read -P0xd5 0x108000 32k
 +read -P0xdc 32M 32k
 +read -P0xcd 0x3ff0000 64k
 +
 +Done
 --
-.34.1
+.21.0

-[PULL 13/23] block: intoduce reqlist
+[Qemu-devel] [PULL v2 2/8] blockdev: enable non-root nodes for transaction drive-backup source
 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Split intersecting-requests functionality out of block-copy to be
+We forget to enable it for transaction .prepare, while it is already
-reused in copy-before-write filter.
+enabled in do_drive_backup since commit a2d665c1bc362
+    "blockdev: loosen restrictions on drive-backup source node"
 Note: while being here, fix tiny typo in MAINTAINERS.
 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
+Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com
-Message-Id: <20220303194349.2304213-7-vsementsov@virtuozzo.com>
+Reviewed-by: John Snow <jsnow@redhat.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
+Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- include/block/reqlist.h |  67 +++++++++++++++++++++++
+ blockdev.c | 2 +-
- block/block-copy.c      | 116 +++++++++++++---------------------------
+file changed, 1 insertion(+), 1 deletion(-)
  block/reqlist.c         |  76 ++++++++++++++++++++++++++
  MAINTAINERS             |   4 +-
  block/meson.build       |   1 +
 files changed, 184 insertions(+), 80 deletions(-)
  create mode 100644 include/block/reqlist.h
  create mode 100644 block/reqlist.c
-diff --git a/include/block/reqlist.h b/include/block/reqlist.h
+diff --git a/blockdev.c b/blockdev.c
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/include/block/reqlist.h
@@ -XXX,XX +XXX,XX @@
 +/*
 + * reqlist API
 + *
 + * Copyright (C) 2013 Proxmox Server Solutions
 + * Copyright (c) 2021 Virtuozzo International GmbH.
 + *
 + * Authors:
 + *  Dietmar Maurer (dietmar@proxmox.com)
 + *  Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
 + *
 + * This work is licensed under the terms of the GNU GPL, version 2 or later.
 + * See the COPYING file in the top-level directory.
 + */
 +
 +#ifndef REQLIST_H
 +#define REQLIST_H
 +
 +#include "qemu/coroutine.h"
 +
 +/*
 + * The API is not thread-safe and shouldn't be. The struct is public to be part
 + * of other structures and protected by third-party locks, see
 + * block/block-copy.c for example.
 + */
 +
 +typedef struct BlockReq {
 +    int64_t offset;
 +    int64_t bytes;
 +
 +    CoQueue wait_queue; /* coroutines blocked on this req */
 +    QLIST_ENTRY(BlockReq) list;
 +} BlockReq;
 +
 +typedef QLIST_HEAD(, BlockReq) BlockReqList;
 +
 +/*
 + * Initialize new request and add it to the list. Caller must be sure that
 + * there are no conflicting requests in the list.
 + */
 +void reqlist_init_req(BlockReqList *reqs, BlockReq *req, int64_t offset,
 +                      int64_t bytes);
 +/* Search for request in the list intersecting with @offset/@bytes area. */
 +BlockReq *reqlist_find_conflict(BlockReqList *reqs, int64_t offset,
 +                                int64_t bytes);
 +
 +/*
 + * If there are no intersecting requests return false. Otherwise, wait for the
 + * first found intersecting request to finish and return true.
 + *
 + * @lock is passed to qemu_co_queue_wait()
 + * False return value proves that lock was released at no point.
 + */
 +bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
 +                                   int64_t bytes, CoMutex *lock);
 +
 +/*
 + * Shrink request and wake all waiting coroutines (maybe some of them are not
 + * intersecting with shrunk request).
 + */
 +void coroutine_fn reqlist_shrink_req(BlockReq *req, int64_t new_bytes);
 +
 +/*
 + * Remove request and wake all waiting coroutines. Do not release any memory.
 + */
 +void coroutine_fn reqlist_remove_req(BlockReq *req);
 +
 +#endif /* REQLIST_H */
 diff --git a/block/block-copy.c b/block/block-copy.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/block-copy.c
+--- a/blockdev.c
-+++ b/block/block-copy.c
++++ b/blockdev.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
- #include "trace.h"
+     assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);
- #include "qapi/error.h"
+     backup = common->action->u.drive_backup.data;
- #include "block/block-copy.h"
-+#include "block/reqlist.h"
+-    bs = qmp_get_root_bs(backup->device, errp);
- #include "sysemu/block-backend.h"
++    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
- #include "qemu/units.h"
+     if (!bs) {
  #include "qemu/coroutine.h"
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyTask {
       */
      BlockCopyState *s;
      BlockCopyCallState *call_state;
 -    int64_t offset;
      /*
       * @method can also be set again in the while loop of
       * block_copy_dirty_clusters(), but it is never accessed concurrently
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyTask {
      BlockCopyMethod method;
      /*
 -     * Fields whose state changes throughout the execution
 -     * Protected by lock in BlockCopyState.
 +     * Generally, req is protected by lock in BlockCopyState, Still req.offset
 +     * is only set on task creation, so may be read concurrently after creation.
 +     * req.bytes is changed at most once, and need only protecting the case of
 +     * parallel read while updating @bytes value in block_copy_task_shrink().
       */
 -    CoQueue wait_queue; /* coroutines blocked on this task */
 -    /*
 -     * Only protect the case of parallel read while updating @bytes
 -     * value in block_copy_task_shrink().
 -     */
 -    int64_t bytes;
 -    QLIST_ENTRY(BlockCopyTask) list;
 +    BlockReq req;
  } BlockCopyTask;
  static int64_t task_end(BlockCopyTask *task)
  {
 -    return task->offset + task->bytes;
 +    return task->req.offset + task->req.bytes;
  }
  typedef struct BlockCopyState {
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState {
      CoMutex lock;
      int64_t in_flight_bytes;
      BlockCopyMethod method;
 -    QLIST_HEAD(, BlockCopyTask) tasks; /* All tasks from all block-copy calls */
 +    BlockReqList reqs;
      QLIST_HEAD(, BlockCopyCallState) calls;
      /*
       * skip_unallocated:
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState {
      RateLimit rate_limit;
  } BlockCopyState;
 -/* Called with lock held */
 -static BlockCopyTask *find_conflicting_task(BlockCopyState *s,
 -                                            int64_t offset, int64_t bytes)
 -{
 -    BlockCopyTask *t;
 -
 -    QLIST_FOREACH(t, &s->tasks, list) {
 -        if (offset + bytes > t->offset && offset < t->offset + t->bytes) {
 -            return t;
 -        }
 -    }
 -
 -    return NULL;
 -}
 -
 -/*
 - * If there are no intersecting tasks return false. Otherwise, wait for the
 - * first found intersecting tasks to finish and return true.
 - *
 - * Called with lock held. May temporary release the lock.
 - * Return value of 0 proves that lock was NOT released.
 - */
 -static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset,
 -                                             int64_t bytes)
 -{
 -    BlockCopyTask *task = find_conflicting_task(s, offset, bytes);
 -
 -    if (!task) {
 -        return false;
 -    }
 -
 -    qemu_co_queue_wait(&task->wait_queue, &s->lock);
 -
 -    return true;
 -}
 -
  /* Called with lock held */
  static int64_t block_copy_chunk_size(BlockCopyState *s)
  {
@@ -XXX,XX +XXX,XX @@ block_copy_task_create(BlockCopyState *s, BlockCopyCallState *call_state,
      bytes = QEMU_ALIGN_UP(bytes, s->cluster_size);
      /* region is dirty, so no existent tasks possible in it */
 -    assert(!find_conflicting_task(s, offset, bytes));
 +    assert(!reqlist_find_conflict(&s->reqs, offset, bytes));
      bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes);
      s->in_flight_bytes += bytes;
@@ -XXX,XX +XXX,XX @@ block_copy_task_create(BlockCopyState *s, BlockCopyCallState *call_state,
          .task.func = block_copy_task_entry,
          .s = s,
          .call_state = call_state,
 -        .offset = offset,
 -        .bytes = bytes,
          .method = s->method,
      };
 -    qemu_co_queue_init(&task->wait_queue);
 -    QLIST_INSERT_HEAD(&s->tasks, task, list);
 +    reqlist_init_req(&s->reqs, &task->req, offset, bytes);
      return task;
  }
@@ -XXX,XX +XXX,XX @@ static void coroutine_fn block_copy_task_shrink(BlockCopyTask *task,
                                                  int64_t new_bytes)
  {
      QEMU_LOCK_GUARD(&task->s->lock);
 -    if (new_bytes == task->bytes) {
 +    if (new_bytes == task->req.bytes) {
          return;
      }
--    assert(new_bytes > 0 && new_bytes < task->bytes);
-+    assert(new_bytes > 0 && new_bytes < task->req.bytes);
--    task->s->in_flight_bytes -= task->bytes - new_bytes;
-+    task->s->in_flight_bytes -= task->req.bytes - new_bytes;
-     bdrv_set_dirty_bitmap(task->s->copy_bitmap,
--                          task->offset + new_bytes, task->bytes - new_bytes);
-+                          task->req.offset + new_bytes,
-+                          task->req.bytes - new_bytes);
--    task->bytes = new_bytes;
--    qemu_co_queue_restart_all(&task->wait_queue);
-+    reqlist_shrink_req(&task->req, new_bytes);
- }
- static void coroutine_fn block_copy_task_end(BlockCopyTask *task, int ret)
- {
-     QEMU_LOCK_GUARD(&task->s->lock);
--    task->s->in_flight_bytes -= task->bytes;
-+    task->s->in_flight_bytes -= task->req.bytes;
-     if (ret < 0) {
--        bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->offset, task->bytes);
-+        bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->req.offset,
-+                              task->req.bytes);
-     }
--    QLIST_REMOVE(task, list);
-     if (task->s->progress) {
-         progress_set_remaining(task->s->progress,
-                                bdrv_get_dirty_count(task->s->copy_bitmap) +
-                                task->s->in_flight_bytes);
-     }
--    qemu_co_queue_restart_all(&task->wait_queue);
-+    reqlist_remove_req(&task->req);
- }
- void block_copy_state_free(BlockCopyState *s)
-@@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
-     ratelimit_init(&s->rate_limit);
-     qemu_co_mutex_init(&s->lock);
--    QLIST_INIT(&s->tasks);
-+    QLIST_INIT(&s->reqs);
-     QLIST_INIT(&s->calls);
-     return s;
-@@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_run(AioTaskPool *pool,
-     aio_task_pool_wait_slot(pool);
-     if (aio_task_pool_status(pool) < 0) {
--        co_put_to_shres(task->s->mem, task->bytes);
-+        co_put_to_shres(task->s->mem, task->req.bytes);
-         block_copy_task_end(task, -ECANCELED);
-         g_free(task);
-         return -ECANCELED;
-@@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task)
-     BlockCopyMethod method = t->method;
-     int ret;
--    ret = block_copy_do_copy(s, t->offset, t->bytes, &method, &error_is_read);
-+    ret = block_copy_do_copy(s, t->req.offset, t->req.bytes, &method,
-+                             &error_is_read);
-     WITH_QEMU_LOCK_GUARD(&s->lock) {
-         if (s->method == t->method) {
-@@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task)
-                 t->call_state->error_is_read = error_is_read;
-             }
-         } else if (s->progress) {
--            progress_work_done(s->progress, t->bytes);
-+            progress_work_done(s->progress, t->req.bytes);
-         }
-     }
--    co_put_to_shres(s->mem, t->bytes);
-+    co_put_to_shres(s->mem, t->req.bytes);
-     block_copy_task_end(t, ret);
-     return ret;
-@@ -XXX,XX +XXX,XX @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)
-             trace_block_copy_skip_range(s, offset, bytes);
-             break;
-         }
--        if (task->offset > offset) {
--            trace_block_copy_skip_range(s, offset, task->offset - offset);
-+        if (task->req.offset > offset) {
-+            trace_block_copy_skip_range(s, offset, task->req.offset - offset);
-         }
-         found_dirty = true;
--        ret = block_copy_block_status(s, task->offset, task->bytes,
-+        ret = block_copy_block_status(s, task->req.offset, task->req.bytes,
-                                       &status_bytes);
-         assert(ret >= 0); /* never fail */
--        if (status_bytes < task->bytes) {
-+        if (status_bytes < task->req.bytes) {
-             block_copy_task_shrink(task, status_bytes);
-         }
-         if (qatomic_read(&s->skip_unallocated) &&
-             !(ret & BDRV_BLOCK_ALLOCATED)) {
-             block_copy_task_end(task, 0);
--            trace_block_copy_skip_range(s, task->offset, task->bytes);
-+            trace_block_copy_skip_range(s, task->req.offset, task->req.bytes);
-             offset = task_end(task);
-             bytes = end - offset;
-             g_free(task);
-@@ -XXX,XX +XXX,XX @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)
-             }
-         }
--        ratelimit_calculate_delay(&s->rate_limit, task->bytes);
-+        ratelimit_calculate_delay(&s->rate_limit, task->req.bytes);
--        trace_block_copy_process(s, task->offset);
-+        trace_block_copy_process(s, task->req.offset);
--        co_get_from_shres(s->mem, task->bytes);
-+        co_get_from_shres(s->mem, task->req.bytes);
-         offset = task_end(task);
-         bytes = end - offset;
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)
-                  * Check that there is no task we still need to
-                  * wait to complete
-                  */
--                ret = block_copy_wait_one(s, call_state->offset,
--                                          call_state->bytes);
-+                ret = reqlist_wait_one(&s->reqs, call_state->offset,
-+                                       call_state->bytes, &s->lock);
-                 if (ret == 0) {
-                     /*
-                      * No pending tasks, but check again the bitmap in this
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)
-                      * between this and the critical section in
-                      * block_copy_dirty_clusters().
-                      *
--                     * block_copy_wait_one return value 0 also means that it
-+                     * reqlist_wait_one return value 0 also means that it
-                      * didn't release the lock. So, we are still in the same
-                      * critical section, not interrupted by any concurrent
-                      * access to state.
-diff --git a/block/reqlist.c b/block/reqlist.c
-new file mode 100644
-index XXXXXXX..XXXXXXX
---- /dev/null
-+++ b/block/reqlist.c
-@@ -XXX,XX +XXX,XX @@
-+/*
-+ * reqlist API
-+ *
-+ * Copyright (C) 2013 Proxmox Server Solutions
-+ * Copyright (c) 2021 Virtuozzo International GmbH.
-+ *
-+ * Authors:
-+ *  Dietmar Maurer (dietmar@proxmox.com)
-+ *  Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-+ *
-+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
-+ * See the COPYING file in the top-level directory.
-+ */
-+
-+#include "qemu/osdep.h"
-+
-+#include "block/reqlist.h"
-+
-+void reqlist_init_req(BlockReqList *reqs, BlockReq *req, int64_t offset,
-+                      int64_t bytes)
-+{
-+    assert(!reqlist_find_conflict(reqs, offset, bytes));
-+
-+    *req = (BlockReq) {
-+        .offset = offset,
-+        .bytes = bytes,
-+    };
-+    qemu_co_queue_init(&req->wait_queue);
-+    QLIST_INSERT_HEAD(reqs, req, list);
-+}
-+
-+BlockReq *reqlist_find_conflict(BlockReqList *reqs, int64_t offset,
-+                                int64_t bytes)
-+{
-+    BlockReq *r;
-+
-+    QLIST_FOREACH(r, reqs, list) {
-+        if (offset + bytes > r->offset && offset < r->offset + r->bytes) {
-+            return r;
-+        }
-+    }
-+
-+    return NULL;
-+}
-+
-+bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
-+                                   int64_t bytes, CoMutex *lock)
-+{
-+    BlockReq *r = reqlist_find_conflict(reqs, offset, bytes);
-+
-+    if (!r) {
-+        return false;
-+    }
-+
-+    qemu_co_queue_wait(&r->wait_queue, lock);
-+
-+    return true;
-+}
-+
-+void coroutine_fn reqlist_shrink_req(BlockReq *req, int64_t new_bytes)
-+{
-+    if (new_bytes == req->bytes) {
-+        return;
-+    }
-+
-+    assert(new_bytes > 0 && new_bytes < req->bytes);
-+
-+    req->bytes = new_bytes;
-+    qemu_co_queue_restart_all(&req->wait_queue);
-+}
-+
-+void coroutine_fn reqlist_remove_req(BlockReq *req)
-+{
-+    QLIST_REMOVE(req, list);
-+    qemu_co_queue_restart_all(&req->wait_queue);
-+}
-diff --git a/MAINTAINERS b/MAINTAINERS
-index XXXXXXX..XXXXXXX 100644
---- a/MAINTAINERS
-+++ b/MAINTAINERS
-@@ -XXX,XX +XXX,XX @@ F: block/stream.c
- F: block/mirror.c
- F: qapi/job.json
- F: block/block-copy.c
--F: include/block/block-copy.c
-+F: include/block/block-copy.h
-+F: block/reqlist.c
-+F: include/block/reqlist.h
- F: block/copy-before-write.h
- F: block/copy-before-write.c
- F: include/block/aio_task.h
-diff --git a/block/meson.build b/block/meson.build
-index XXXXXXX..XXXXXXX 100644
---- a/block/meson.build
-+++ b/block/meson.build
-@@ -XXX,XX +XXX,XX @@ block_ss.add(files(
-   'qcow2.c',
-   'quorum.c',
-   'raw-format.c',
-+  'reqlist.c',
-   'snapshot.c',
-   'throttle-groups.c',
-   'throttle.c',
 --
-.34.1
+.21.0

-[PULL 22/23] iotests/image-fleecing: add test case with bitmap
+[Qemu-devel] [PULL v2 3/8] iotest 134: test cluster-misaligned encrypted write
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Anton Nefedov <anton.nefedov@virtuozzo.com>
-Note that reads zero areas (not dirty in the bitmap) fails, that's
+COW (even empty/zero) areas require encryption too
 correct.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
-Message-Id: <20220303194349.2304213-16-vsementsov@virtuozzo.com>
+Reviewed-by: Eric Blake <eblake@redhat.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
+Reviewed-by: Max Reitz <mreitz@redhat.com>
 Reviewed-by: Alberto Garcia <berto@igalia.com>
 Message-id: 20190516143028.81155-1-anton.nefedov@virtuozzo.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- tests/qemu-iotests/tests/image-fleecing     | 38 +++++++---
+ tests/qemu-iotests/134     |  9 +++++++++
- tests/qemu-iotests/tests/image-fleecing.out | 84 +++++++++++++++++++++
+ tests/qemu-iotests/134.out | 10 ++++++++++
-files changed, 113 insertions(+), 9 deletions(-)
+files changed, 19 insertions(+)
-diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
+diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
 index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/tests/image-fleecing
+--- a/tests/qemu-iotests/134
-+++ b/tests/qemu-iotests/tests/image-fleecing
++++ b/tests/qemu-iotests/134
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ echo
- # Creator/Owner: John Snow <jsnow@redhat.com>
+ echo "== reading whole image =="
+ $QEMU_IO --object $SECRET -c "read 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
- import iotests
--from iotests import log, qemu_img, qemu_io, qemu_io_silent
++echo
-+from iotests import log, qemu_img, qemu_io, qemu_io_silent, \
++echo "== rewriting cluster part =="
-+    qemu_io_pipe_and_status
++$QEMU_IO --object $SECRET -c "write -P 0xb 512 512" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
  iotests.script_initialize(
 -    supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk', 'vhdx', 'raw'],
 +    supported_fmts=['qcow2'],
      supported_platforms=['linux'],
      required_fmts=['copy-before-write'],
 +    unsupported_imgopts=['compat']
  )
  patterns = [('0x5d', '0',         '64k'),
@@ -XXX,XX +XXX,XX @@ remainder = [('0xd5', '0x108000',  '32k'), # Right-end of partial-left [1]
               ('0xcd', '0x3ff0000', '64k')] # patterns[3]
  def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
 -            fleece_img_path, nbd_sock_path, vm):
 +            fleece_img_path, nbd_sock_path, vm,
 +            bitmap=False):
      log('--- Setting up images ---')
      log('')
      assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
 +    if bitmap:
 +        assert qemu_img('bitmap', '--add', base_img_path, 'bitmap0') == 0
 +
-     if use_snapshot_access_filter:
++echo
-         assert use_cbw
++echo "== verify pattern =="
-         assert qemu_img('create', '-f', 'raw', fleece_img_path, '64M') == 0
++$QEMU_IO --object $SECRET -c "read -P 0 0 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
++$QEMU_IO --object $SECRET -c "read -P 0xb 512 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
      # Establish CBW from source to fleecing node
      if use_cbw:
 -        log(vm.qmp('blockdev-add', {
 +        fl_cbw = {
              'driver': 'copy-before-write',
              'node-name': 'fl-cbw',
              'file': src_node,
              'target': tmp_node
 -        }))
 +        }
 +
-+        if bitmap:
+ echo
-+            fl_cbw['bitmap'] = {'node': src_node, 'name': 'bitmap0'}
+ echo "== rewriting whole image =="
  $QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/134.out
 +++ b/tests/qemu-iotests/134.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.
  read 134217728/134217728 bytes at offset 0
 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +== rewriting cluster part ==
 +wrote 512/512 bytes at offset 512
 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 +
-+        log(vm.qmp('blockdev-add', fl_cbw))
++== verify pattern ==
++read 512/512 bytes at offset 0
-         log(vm.qmp('qom-set', path=qom_path, property='drive', value='fl-cbw'))
++512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
++read 512/512 bytes at offset 512
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
++512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
      for p in patterns + zeroes:
          cmd = 'read -P%s %s %s' % p
          log(cmd)
 -        assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0
 +        out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
 +                                           nbd_uri)
 +        if ret != 0:
 +            print(out)
      log('')
      log('--- Testing COW ---')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
      for p in patterns + zeroes:
          cmd = 'read -P%s %s %s' % p
          log(cmd)
 -        assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0
 +        out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
 +                                           nbd_uri)
 +        if ret != 0:
 +            print(out)
      log('')
      log('--- Cleanup ---')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
      log('Done')
 -def test(use_cbw, use_snapshot_access_filter):
 +def test(use_cbw, use_snapshot_access_filter, bitmap=False):
      with iotests.FilePath('base.img') as base_img_path, \
           iotests.FilePath('fleece.img') as fleece_img_path, \
           iotests.FilePath('nbd.sock',
                            base_dir=iotests.sock_dir) as nbd_sock_path, \
           iotests.VM() as vm:
          do_test(use_cbw, use_snapshot_access_filter, base_img_path,
 -                fleece_img_path, nbd_sock_path, vm)
 +                fleece_img_path, nbd_sock_path, vm, bitmap=bitmap)
  log('=== Test backup(sync=none) based fleecing ===\n')
@@ -XXX,XX +XXX,XX @@ test(True, False)
  log('=== Test fleecing-format based fleecing ===\n')
  test(True, True)
 +
-+log('=== Test fleecing-format based fleecing with bitmap ===\n')
+ == rewriting whole image ==
-+test(True, True, bitmap=True)
+ wrote 134217728/134217728 bytes at offset 0
-diff --git a/tests/qemu-iotests/tests/image-fleecing.out b/tests/qemu-iotests/tests/image-fleecing.out
+MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qemu-iotests/tests/image-fleecing.out
 +++ b/tests/qemu-iotests/tests/image-fleecing.out
@@ -XXX,XX +XXX,XX @@ read -P0 0x00f8000 32k
  read -P0 0x2010000 32k
  read -P0 0x3fe0000 64k
 +--- Cleanup ---
 +
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +
 +--- Confirming writes ---
 +
 +read -P0xab 0 64k
 +read -P0xad 0x00f8000 64k
 +read -P0x1d 0x2008000 64k
 +read -P0xea 0x3fe0000 64k
 +read -P0xd5 0x108000 32k
 +read -P0xdc 32M 32k
 +read -P0xcd 0x3ff0000 64k
 +
 +Done
 +=== Test fleecing-format based fleecing with bitmap ===
 +
 +--- Setting up images ---
 +
 +Done
 +
 +--- Launching VM ---
 +
 +Done
 +
 +--- Setting up Fleecing Graph ---
 +
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +{"return": {}}
 +
 +--- Setting up NBD Export ---
 +
 +{"return": {}}
 +{"return": {}}
 +
 +--- Sanity Check ---
 +
 +read -P0x5d 0 64k
 +read -P0xd5 1M 64k
 +read -P0xdc 32M 64k
 +read -P0xcd 0x3ff0000 64k
 +read -P0 0x00f8000 32k
 +read failed: Invalid argument
 +
 +read -P0 0x2010000 32k
 +read failed: Invalid argument
 +
 +read -P0 0x3fe0000 64k
 +read failed: Invalid argument
 +
 +
 +--- Testing COW ---
 +
 +write -P0xab 0 64k
 +{"return": ""}
 +write -P0xad 0x00f8000 64k
 +{"return": ""}
 +write -P0x1d 0x2008000 64k
 +{"return": ""}
 +write -P0xea 0x3fe0000 64k
 +{"return": ""}
 +
 +--- Verifying Data ---
 +
 +read -P0x5d 0 64k
 +read -P0xd5 1M 64k
 +read -P0xdc 32M 64k
 +read -P0xcd 0x3ff0000 64k
 +read -P0 0x00f8000 32k
 +read failed: Invalid argument
 +
 +read -P0 0x2010000 32k
 +read failed: Invalid argument
 +
 +read -P0 0x3fe0000 64k
 +read failed: Invalid argument
 +
 +
  --- Cleanup ---
  {"return": {}}
 --
-.34.1
+.21.0

-[PULL 14/23] block/reqlist: reqlist_find_conflict(): use ranges_overlap()
+[Qemu-devel] [PULL v2 4/8] vmdk: Fix comment regarding max l1_size coverage
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-Let's reuse convenient helper.
+Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")
 extended the l1_size check from VMDK4 to VMDK3 but did not update the
 default coverage in the moved comment.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+The previous vmdk4 calculation:
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-8-vsementsov@virtuozzo.com>
+    (512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 The added vmdk3 calculation:
     (512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB
 Adding the calculation of vmdk3 to the comment.
 In any case, VMware does not offer virtual disks more than 2TB for
 vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is
 not implemented yet in qemu.
 Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
 Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
 Reviewed-by: Liran Alon <liran.alon@oracle.com>
 Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
 Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
 Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com
 Reviewed-by: yuchenlin <yuchenlin@synology.com>
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- block/reqlist.c | 3 ++-
+ block/vmdk.c | 11 ++++++++---
-file changed, 2 insertions(+), 1 deletion(-)
+file changed, 8 insertions(+), 3 deletions(-)
-diff --git a/block/reqlist.c b/block/reqlist.c
+diff --git a/block/vmdk.c b/block/vmdk.c
 index XXXXXXX..XXXXXXX 100644
---- a/block/reqlist.c
+--- a/block/vmdk.c
-+++ b/block/reqlist.c
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-  */
+         return -EFBIG;
+     }
- #include "qemu/osdep.h"
+     if (l1_size > 512 * 1024 * 1024) {
-+#include "qemu/range.h"
+-        /* Although with big capacity and small l1_entry_sectors, we can get a
++        /*
- #include "block/reqlist.h"
++         * Although with big capacity and small l1_entry_sectors, we can get a
+          * big l1_size, we don't want unbounded value to allocate the table.
-@@ -XXX,XX +XXX,XX @@ BlockReq *reqlist_find_conflict(BlockReqList *reqs, int64_t offset,
+-         * Limit it to 512M, which is 16PB for default cluster and L2 table
-     BlockReq *r;
+-         * size */
++         * Limit it to 512M, which is:
-     QLIST_FOREACH(r, reqs, list) {
++         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
--        if (offset + bytes > r->offset && offset < r->offset + r->bytes) {
++         *            cluster size: 64KB, L2 table size: 512 entries
-+        if (ranges_overlap(offset, bytes, r->offset, r->bytes)) {
++         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
-             return r;
++         *            cluster size: 512B, L2 table size: 4096 entries
-         }
++         */
          error_setg(errp, "L1 size too big");
          return -EFBIG;
      }
 --
-.34.1
+.21.0

-[PULL 01/23] ide: Increment BB in-flight counter for TRIM BH
+[Qemu-devel] [PULL v2 5/8] vmdk: Reduce the max bound for L1 table size
-When we still have an AIOCB registered for DMA operations, we try to
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
 settle the respective operation by draining the BlockBackend associated
 with the IDE device.
-However, this assumes that every DMA operation is associated with an
+M of L1 entries is a very loose bound, only 32M are required to store
-increment of the BlockBackend’s in-flight counter (e.g. through some
+the maximal supported VMDK file size of 2TB.
 ongoing I/O operation), so that draining the BB until its in-flight
 counter reaches 0 will settle all DMA operations.  That is not the case:
 For TRIM, the guest can issue a zero-length operation that will not
 result in any I/O operation forwarded to the BlockBackend, and also not
 increment the in-flight counter in any other way.  In such a case,
 blk_drain() will be a no-op if no other operations are in flight.
-It is clear that if blk_drain() is a no-op, the value of
+Fixed qemu-iotest 59# - now failure occures before on impossible L1
-s->bus->dma->aiocb will not change between checking it in the `if`
+table size.
 condition and asserting that it is NULL after blk_drain().
-The particular problem is that ide_issue_trim() creates a BH
+Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
-(ide_trim_bh_cb()) to settle the TRIM request: iocb->common.cb() is
+Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
-ide_dma_cb(), which will either create a new request, or find the
+Reviewed-by: Liran Alon <liran.alon@oracle.com>
-transfer to be done and call ide_set_inactive(), which clears
+Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
-s->bus->dma->aiocb.  Therefore, the blk_drain() must wait for
+Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
-ide_trim_bh_cb() to run, which currently it will not always do.
+Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com
 Reviewed-by: Max Reitz <mreitz@redhat.com>
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
  block/vmdk.c               | 13 +++++++------
  tests/qemu-iotests/059.out |  2 +-
 files changed, 8 insertions(+), 7 deletions(-)
-To fix this issue, we increment the BlockBackend's in-flight counter
+diff --git a/block/vmdk.c b/block/vmdk.c
 when the TRIM operation begins (in ide_issue_trim(), when the
 ide_trim_bh_cb() BH is created) and decrement it when ide_trim_bh_cb()
 is done.
 Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2029980
 Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
 Signed-off-by: Hanna Reitz <hreitz@redhat.com>
 Message-Id: <20220120142259.120189-1-hreitz@redhat.com>
 Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
 Reviewed-by: John Snow <jsnow@redhat.com>
 Tested-by: John Snow <jsnow@redhat.com>
 ---
  hw/ide/core.c | 7 +++++++
 file changed, 7 insertions(+)
 diff --git a/hw/ide/core.c b/hw/ide/core.c
 index XXXXXXX..XXXXXXX 100644
---- a/hw/ide/core.c
+--- a/block/vmdk.c
-+++ b/hw/ide/core.c
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@ static const AIOCBInfo trim_aiocb_info = {
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
- static void ide_trim_bh_cb(void *opaque)
+         error_setg(errp, "Invalid granularity, image may be corrupt");
- {
+         return -EFBIG;
-     TrimAIOCB *iocb = opaque;
+     }
-+    BlockBackend *blk = iocb->s->blk;
+-    if (l1_size > 512 * 1024 * 1024) {
++    if (l1_size > 32 * 1024 * 1024) {
-     iocb->common.cb(iocb->common.opaque, iocb->ret);
+         /*
+          * Although with big capacity and small l1_entry_sectors, we can get a
-     qemu_bh_delete(iocb->bh);
+          * big l1_size, we don't want unbounded value to allocate the table.
-     iocb->bh = NULL;
+-         * Limit it to 512M, which is:
-     qemu_aio_unref(iocb);
+-         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
-+
+-         *            cluster size: 64KB, L2 table size: 512 entries
-+    /* Paired with an increment in ide_issue_trim() */
+-         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
-+    blk_dec_in_flight(blk);
+-         *            cluster size: 512B, L2 table size: 4096 entries
- }
++         * Limit it to 32M, which is enough to store:
++         *     8TB  - for both VMDK3 & VMDK4 with
- static void ide_issue_trim_cb(void *opaque, int ret)
++         *            minimal cluster size: 512B
-@@ -XXX,XX +XXX,XX @@ BlockAIOCB *ide_issue_trim(
++         *            minimal L2 table size: 512 entries
-     IDEState *s = opaque;
++         *            8 TB is still more than the maximal value supported for
-     TrimAIOCB *iocb;
++         *            VMDK3 & VMDK4 which is 2TB.
+          */
-+    /* Paired with a decrement in ide_trim_bh_cb() */
+         error_setg(errp, "L1 size too big");
-+    blk_inc_in_flight(s->blk);
+         return -EFBIG;
-+
+diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
-     iocb = blk_aio_get(&trim_aiocb_info, s->blk, cb, cb_opaque);
+index XXXXXXX..XXXXXXX 100644
-     iocb->s = s;
+--- a/tests/qemu-iotests/059.out
-     iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
++++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 x140000000     0x10000         0x50000         TEST_DIR/t-s003.vmdk
  === Testing afl image with a very large capacity ===
 -qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large
 +qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big
  *** done
 --
-.34.1
+.21.0

-[PULL 18/23] block: introduce snapshot-access block driver
+[Qemu-devel] [PULL v2 6/8] vmdk: Add read-only support for seSparse snapshots
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Sam Eiderman <shmuel.eiderman@oracle.com>
-The new block driver simply utilizes snapshot-access API of underlying
+Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in
-block node.
+QEMU).
-In further patches we want to use it like this:
+This format was lacking in the following:
-[guest]                   [NBD export]
+    * Grain directory (L1) and grain table (L2) entries were 32-bit,
-   |                            |
+      allowing access to only 2TB (slightly less) of data.
-   | root                       | root
+    * The grain size (default) was 512 bytes - leading to data
-   v                 file       v
+      fragmentation and many grain tables.
-[copy-before-write]<------[snapshot-access]
+    * For space reclamation purposes, it was necessary to find all the
-   |           |
+      grains which are not pointed to by any grain table - so a reverse
-   | file      | target
+      mapping of "offset of grain in vmdk" to "grain table" must be
-   v           v
+      constructed - which takes large amounts of CPU/RAM.
-[active-disk] [temp.img]
+The format specification can be found in VMware's documentation:
-This way, NBD client will be able to read snapshotted state of active
+https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf
-disk, when active disk is continued to be written by guest. This is
-known as "fleecing", and currently uses another scheme based on qcow2
+In ESXi 6.5, to support snapshot files larger than 2TB, a new format was
-temporary image which backing file is active-disk. New scheme comes
+introduced: SESparse (Space Efficient).
-with benefits - see next commit.
+This format fixes the above issues:
-The other possible application is exporting internal snapshots of
-qcow2, like this:
+    * All entries are now 64-bit.
+    * The grain size (default) is 4KB.
-[guest]          [NBD export]
+    * Grain directory and grain tables are now located at the beginning
-   |                  |
+      of the file.
-   | root             | root
+      + seSparse format reserves space for all grain tables.
-   v       file       v
+      + Grain tables can be addressed using an index.
-[qcow2]<---------[snapshot-access]
+      + Grains are located in the end of the file and can also be
+        addressed with an index.
-For this, we'll need to implement snapshot-access API handlers in
+      - seSparse vmdks of large disks (64TB) have huge preallocated
-qcow2 driver, and improve snapshot-access block driver (and API) to
+        headers - mainly due to L2 tables, even for empty snapshots.
-make it possible to select snapshot by name. Another thing to improve
+    * The header contains a reverse mapping ("backmap") of "offset of
-is size of snapshot. Now for simplicity we just use size of bs->file,
+      grain in vmdk" to "grain table" and a bitmap ("free bitmap") which
-which is OK for backup, but for qcow2 snapshots export we'll need to
+      specifies for each grain - whether it is allocated or not.
-imporve snapshot-access API to get size of snapshot.
+      Using these data structures we can implement space reclamation
+      efficiently.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+    * Due to the fact that the header now maintains two mappings:
-Message-Id: <20220303194349.2304213-12-vsementsov@virtuozzo.com>
+        * The regular one (grain directory & grain tables)
-[hreitz: Rebased on block GS/IO split]
+        * A reverse one (backmap and free bitmap)
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
+      These data structures can lose consistency upon crash and result
       in a corrupted VMDK.
       Therefore, a journal is also added to the VMDK and is replayed
       when the VMware reopens the file after a crash.
 Since ESXi 6.7 - SESparse is the only snapshot format available.
 Unfortunately, VMware does not provide documentation regarding the new
 seSparse format.
 This commit is based on black-box research of the seSparse format.
 Various in-guest block operations and their effect on the snapshot file
 were tested.
 The only VMware provided source of information (regarding the underlying
 implementation) was a log file on the ESXi:
     /var/log/hostd.log
 Whenever an seSparse snapshot is created - the log is being populated
 with seSparse records.
 Relevant log records are of the form:
 [...] Const Header:
 [...]  constMagic     = 0xcafebabe
 [...]  version        = 2.1
 [...]  capacity       = 204800
 [...]  grainSize      = 8
 [...]  grainTableSize = 64
 [...]  flags          = 0
 [...] Extents:
 [...]  Header         : <1 : 1>
 [...]  JournalHdr     : <2 : 2>
 [...]  Journal        : <2048 : 2048>
 [...]  GrainDirectory : <4096 : 2048>
 [...]  GrainTables    : <6144 : 2048>
 [...]  FreeBitmap     : <8192 : 2048>
 [...]  BackMap        : <10240 : 2048>
 [...]  Grain          : <12288 : 204800>
 [...] Volatile Header:
 [...] volatileMagic     = 0xcafecafe
 [...] FreeGTNumber      = 0
 [...] nextTxnSeqNumber  = 0
 [...] replayJournal     = 0
 The sizes that are seen in the log file are in sectors.
 Extents are of the following format: <offset : size>
 This commit is a strict implementation which enforces:
     * magics
     * version number 2.1
     * grain size of 8 sectors  (4KB)
     * grain table size of 64 sectors
     * zero flags
     * extent locations
 Additionally, this commit proivdes only a subset of the functionality
 offered by seSparse's format:
     * Read-only
     * No journal replay
     * No space reclamation
     * No unmap support
 Hence, journal header, journal, free bitmap and backmap extents are
 unused, only the "classic" (L1 -> L2 -> data) grain access is
 implemented.
 However there are several differences in the grain access itself.
 Grain directory (L1):
     * Grain directory entries are indexes (not offsets) to grain
       tables.
     * Valid grain directory entries have their highest nibble set to
 x1.
     * Since grain tables are always located in the beginning of the
       file - the index can fit into 32 bits - so we can use its low
       part if it's valid.
 Grain table (L2):
     * Grain table entries are indexes (not offsets) to grains.
     * If the highest nibble of the entry is:
 x0:
             The grain in not allocated.
             The rest of the bytes are 0.
 x1:
             The grain is unmapped - guest sees a zero grain.
             The rest of the bits point to the previously mapped grain,
             see 0x3 case.
 x2:
             The grain is zero.
 x3:
             The grain is allocated - to get the index calculate:
             ((entry & 0x0fff000000000000) >> 48) |
             ((entry & 0x0000ffffffffffff) << 12)
     * The difference between 0x1 and 0x2 is that 0x1 is an unallocated
       grain which results from the guest using sg_unmap to unmap the
       grain - but the grain itself still exists in the grain extent - a
       space reclamation procedure should delete it.
       Unmapping a zero grain has no effect (0x2 will not change to 0x1)
       but unmapping an unallocated grain will (0x0 to 0x1) - naturally.
 In order to implement seSparse some fields had to be changed to support
 both 32-bit and 64-bit entry sizes.
 Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
 Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
 Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
 Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
 Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com
 Signed-off-by: Max Reitz <mreitz@redhat.com>
 ---
- qapi/block-core.json             |   4 +-
+ block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---
- include/block/block_int-common.h |   6 ++
+file changed, 342 insertions(+), 16 deletions(-)
- block/snapshot-access.c          | 132 +++++++++++++++++++++++++++++++
- MAINTAINERS                      |   1 +
+diff --git a/block/vmdk.c b/block/vmdk.c
  block/meson.build                |   1 +
 files changed, 143 insertions(+), 1 deletion(-)
  create mode 100644 block/snapshot-access.c
 diff --git a/qapi/block-core.json b/qapi/block-core.json
 index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
+--- a/block/vmdk.c
-+++ b/qapi/block-core.json
++++ b/block/vmdk.c
-@@ -XXX,XX +XXX,XX @@
+@@ -XXX,XX +XXX,XX @@ typedef struct {
- # @blkreplay: Since 4.2
+     uint16_t compressAlgorithm;
- # @compress: Since 5.0
+ } QEMU_PACKED VMDK4Header;
- # @copy-before-write: Since 6.2
-+# @snapshot-access: Since 7.0
++typedef struct VMDKSESparseConstHeader {
- #
++    uint64_t magic;
- # Since: 2.9
++    uint64_t version;
- ##
++    uint64_t capacity;
- { 'enum': 'BlockdevDriver',
++    uint64_t grain_size;
-   'data': [ 'blkdebug', 'blklogwrites', 'blkreplay', 'blkverify', 'bochs',
++    uint64_t grain_table_size;
-             'cloop', 'compress', 'copy-before-write', 'copy-on-read', 'dmg',
++    uint64_t flags;
--            'file', 'ftp', 'ftps', 'gluster',
++    uint64_t reserved1;
-+            'file', 'snapshot-access', 'ftp', 'ftps', 'gluster',
++    uint64_t reserved2;
-             {'name': 'host_cdrom', 'if': 'HAVE_HOST_BLOCK_DEVICE' },
++    uint64_t reserved3;
-             {'name': 'host_device', 'if': 'HAVE_HOST_BLOCK_DEVICE' },
++    uint64_t reserved4;
-             'http', 'https', 'iscsi',
++    uint64_t volatile_header_offset;
-@@ -XXX,XX +XXX,XX @@
++    uint64_t volatile_header_size;
-       'rbd':        'BlockdevOptionsRbd',
++    uint64_t journal_header_offset;
-       'replication': { 'type': 'BlockdevOptionsReplication',
++    uint64_t journal_header_size;
-                        'if': 'CONFIG_REPLICATION' },
++    uint64_t journal_offset;
-+      'snapshot-access': 'BlockdevOptionsGenericFormat',
++    uint64_t journal_size;
-       'ssh':        'BlockdevOptionsSsh',
++    uint64_t grain_dir_offset;
-       'throttle':   'BlockdevOptionsThrottle',
++    uint64_t grain_dir_size;
-       'vdi':        'BlockdevOptionsGenericFormat',
++    uint64_t grain_tables_offset;
-diff --git a/include/block/block_int-common.h b/include/block/block_int-common.h
++    uint64_t grain_tables_size;
-index XXXXXXX..XXXXXXX 100644
++    uint64_t free_bitmap_offset;
---- a/include/block/block_int-common.h
++    uint64_t free_bitmap_size;
-+++ b/include/block/block_int-common.h
++    uint64_t backmap_offset;
-@@ -XXX,XX +XXX,XX @@ struct BlockDriver {
++    uint64_t backmap_size;
-      * in generic block-layer: no serializing, no alignment, no tracked
++    uint64_t grains_offset;
-      * requests. So, block-driver that realizes these APIs is fully responsible
++    uint64_t grains_size;
-      * for synchronization between snapshot-access API and normal IO requests.
++    uint8_t pad[304];
-+     *
++} QEMU_PACKED VMDKSESparseConstHeader;
-+     * TODO: To be able to support qcow2's internal snapshots, this API will
++
-+     * need to be extended to:
++typedef struct VMDKSESparseVolatileHeader {
-+     * - be able to select a specific snapshot
++    uint64_t magic;
-+     * - receive the snapshot's actual length (which may differ from bs's
++    uint64_t free_gt_number;
-+     *   length)
++    uint64_t next_txn_seq_number;
-      */
++    uint64_t replay_journal;
-     int coroutine_fn (*bdrv_co_preadv_snapshot)(BlockDriverState *bs,
++    uint8_t pad[480];
-         int64_t offset, int64_t bytes, QEMUIOVector *qiov, size_t qiov_offset);
++} QEMU_PACKED VMDKSESparseVolatileHeader;
-diff --git a/block/snapshot-access.c b/block/snapshot-access.c
++
-new file mode 100644
+ #define L2_CACHE_SIZE 16
-index XXXXXXX..XXXXXXX
---- /dev/null
+ typedef struct VmdkExtent {
-+++ b/block/snapshot-access.c
+@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {
-@@ -XXX,XX +XXX,XX @@
+     bool compressed;
-+/*
+     bool has_marker;
-+ * snapshot_access block driver
+     bool has_zero_grain;
-+ *
++    bool sesparse;
-+ * Copyright (c) 2022 Virtuozzo International GmbH.
++    uint64_t sesparse_l2_tables_offset;
-+ *
++    uint64_t sesparse_clusters_offset;
-+ * Author:
++    int32_t entry_size;
-+ *  Sementsov-Ogievskiy Vladimir <vsementsov@virtuozzo.com>
+     int version;
-+ *
+     int64_t sectors;
-+ * This program is free software; you can redistribute it and/or modify
+     int64_t end_sector;
-+ * it under the terms of the GNU General Public License as published by
+     int64_t flat_start_offset;
-+ * the Free Software Foundation; either version 2 of the License, or
+     int64_t l1_table_offset;
-+ * (at your option) any later version.
+     int64_t l1_backup_table_offset;
-+ *
+-    uint32_t *l1_table;
-+ * This program is distributed in the hope that it will be useful,
++    void *l1_table;
-+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+     uint32_t *l1_backup_table;
-+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+     unsigned int l1_size;
-+ * GNU General Public License for more details.
+     uint32_t l1_entry_sectors;
-+ *
-+ * You should have received a copy of the GNU General Public License
+     unsigned int l2_size;
-+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+-    uint32_t *l2_cache;
-+ */
++    void *l2_cache;
-+
+     uint32_t l2_cache_offsets[L2_CACHE_SIZE];
-+#include "qemu/osdep.h"
+     uint32_t l2_cache_counts[L2_CACHE_SIZE];
-+
-+#include "sysemu/block-backend.h"
+@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
-+#include "qemu/cutils.h"
+          *            minimal L2 table size: 512 entries
-+#include "block/block_int.h"
+          *            8 TB is still more than the maximal value supported for
-+
+          *            VMDK3 & VMDK4 which is 2TB.
-+static coroutine_fn int
++         *     64TB - for "ESXi seSparse Extent"
-+snapshot_access_co_preadv_part(BlockDriverState *bs,
++         *            minimal cluster size: 512B (default is 4KB)
-+                               int64_t offset, int64_t bytes,
++         *            L2 table size: 4096 entries (const).
-+                               QEMUIOVector *qiov, size_t qiov_offset,
++         *            64TB is more than the maximal value supported for
-+                               BdrvRequestFlags flags)
++         *            seSparse VMDKs (which is slightly less than 64TB)
           */
          error_setg(errp, "L1 size too big");
          return -EFBIG;
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
      extent->l2_size = l2_size;
      extent->cluster_sectors = flat ? sectors : cluster_sectors;
      extent->next_cluster_sector = ROUND_UP(nb_sectors, cluster_sectors);
 +    extent->entry_size = sizeof(uint32_t);
      if (s->num_extents > 1) {
          extent->end_sector = (*(extent - 1)).end_sector + extent->sectors;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
      int i;
      /* read the L1 table */
 -    l1_size = extent->l1_size * sizeof(uint32_t);
 +    l1_size = extent->l1_size * extent->entry_size;
      extent->l1_table = g_try_malloc(l1_size);
      if (l1_size && extent->l1_table == NULL) {
          return -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
          goto fail_l1;
      }
      for (i = 0; i < extent->l1_size; i++) {
 -        le32_to_cpus(&extent->l1_table[i]);
 +        if (extent->entry_size == sizeof(uint64_t)) {
 +            le64_to_cpus((uint64_t *)extent->l1_table + i);
 +        } else {
 +            assert(extent->entry_size == sizeof(uint32_t));
 +            le32_to_cpus((uint32_t *)extent->l1_table + i);
 +        }
      }
      if (extent->l1_backup_table_offset) {
 +        assert(!extent->sesparse);
          extent->l1_backup_table = g_try_malloc(l1_size);
          if (l1_size && extent->l1_backup_table == NULL) {
              ret = -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
      }
      extent->l2_cache =
 -        g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE);
 +        g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE);
      return 0;
   fail_l1b:
      g_free(extent->l1_backup_table);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
      return ret;
  }
 +#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe)
 +#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe)
 +
 +/* Strict checks - format not officially documented */
 +static int check_se_sparse_const_header(VMDKSESparseConstHeader *header,
 +                                        Error **errp)
 +{
-+    if (flags) {
++    header->magic = le64_to_cpu(header->magic);
-+        return -ENOTSUP;
++    header->version = le64_to_cpu(header->version);
-+    }
++    header->grain_size = le64_to_cpu(header->grain_size);
-+
++    header->grain_table_size = le64_to_cpu(header->grain_table_size);
-+    return bdrv_co_preadv_snapshot(bs->file, offset, bytes, qiov, qiov_offset);
++    header->flags = le64_to_cpu(header->flags);
-+}
++    header->reserved1 = le64_to_cpu(header->reserved1);
-+
++    header->reserved2 = le64_to_cpu(header->reserved2);
-+static int coroutine_fn
++    header->reserved3 = le64_to_cpu(header->reserved3);
-+snapshot_access_co_block_status(BlockDriverState *bs,
++    header->reserved4 = le64_to_cpu(header->reserved4);
-+                                bool want_zero, int64_t offset,
++
-+                                int64_t bytes, int64_t *pnum,
++    header->volatile_header_offset =
-+                                int64_t *map, BlockDriverState **file)
++        le64_to_cpu(header->volatile_header_offset);
-+{
++    header->volatile_header_size = le64_to_cpu(header->volatile_header_size);
-+    return bdrv_co_snapshot_block_status(bs->file->bs, want_zero, offset,
++
-+                                         bytes, pnum, map, file);
++    header->journal_header_offset = le64_to_cpu(header->journal_header_offset);
-+}
++    header->journal_header_size = le64_to_cpu(header->journal_header_size);
 +
-+static int coroutine_fn snapshot_access_co_pdiscard(BlockDriverState *bs,
++    header->journal_offset = le64_to_cpu(header->journal_offset);
-+                                             int64_t offset, int64_t bytes)
++    header->journal_size = le64_to_cpu(header->journal_size);
-+{
++
-+    return bdrv_co_pdiscard_snapshot(bs->file->bs, offset, bytes);
++    header->grain_dir_offset = le64_to_cpu(header->grain_dir_offset);
-+}
++    header->grain_dir_size = le64_to_cpu(header->grain_dir_size);
 +
-+static int coroutine_fn
++    header->grain_tables_offset = le64_to_cpu(header->grain_tables_offset);
-+snapshot_access_co_pwrite_zeroes(BlockDriverState *bs,
++    header->grain_tables_size = le64_to_cpu(header->grain_tables_size);
-+                                 int64_t offset, int64_t bytes,
++
-+                                 BdrvRequestFlags flags)
++    header->free_bitmap_offset = le64_to_cpu(header->free_bitmap_offset);
-+{
++    header->free_bitmap_size = le64_to_cpu(header->free_bitmap_size);
-+    return -ENOTSUP;
++
-+}
++    header->backmap_offset = le64_to_cpu(header->backmap_offset);
-+
++    header->backmap_size = le64_to_cpu(header->backmap_size);
-+static coroutine_fn int
++
-+snapshot_access_co_pwritev_part(BlockDriverState *bs,
++    header->grains_offset = le64_to_cpu(header->grains_offset);
-+                                int64_t offset, int64_t bytes,
++    header->grains_size = le64_to_cpu(header->grains_size);
-+                                QEMUIOVector *qiov, size_t qiov_offset,
++
-+                                BdrvRequestFlags flags)
++    if (header->magic != SESPARSE_CONST_HEADER_MAGIC) {
-+{
++        error_setg(errp, "Bad const header magic: 0x%016" PRIx64,
-+    return -ENOTSUP;
++                   header->magic);
 +}
 +
 +
 +static void snapshot_access_refresh_filename(BlockDriverState *bs)
 +{
 +    pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
 +            bs->file->bs->filename);
 +}
 +
 +static int snapshot_access_open(BlockDriverState *bs, QDict *options, int flags,
 +                                Error **errp)
 +{
 +    bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
 +                               BDRV_CHILD_DATA | BDRV_CHILD_PRIMARY,
 +                               false, errp);
 +    if (!bs->file) {
 +        return -EINVAL;
 +    }
 +
-+    bs->total_sectors = bs->file->bs->total_sectors;
++    if (header->version != 0x0000000200000001) {
 +        error_setg(errp, "Unsupported version: 0x%016" PRIx64,
 +                   header->version);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->grain_size != 8) {
 +        error_setg(errp, "Unsupported grain size: %" PRIu64,
 +                   header->grain_size);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->grain_table_size != 64) {
 +        error_setg(errp, "Unsupported grain table size: %" PRIu64,
 +                   header->grain_table_size);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->flags != 0) {
 +        error_setg(errp, "Unsupported flags: 0x%016" PRIx64,
 +                   header->flags);
 +        return -ENOTSUP;
 +    }
 +
 +    if (header->reserved1 != 0 || header->reserved2 != 0 ||
 +        header->reserved3 != 0 || header->reserved4 != 0) {
 +        error_setg(errp, "Unsupported reserved bits:"
 +                   " 0x%016" PRIx64 " 0x%016" PRIx64
 +                   " 0x%016" PRIx64 " 0x%016" PRIx64,
 +                   header->reserved1, header->reserved2,
 +                   header->reserved3, header->reserved4);
 +        return -ENOTSUP;
 +    }
 +
 +    /* check that padding is 0 */
 +    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
 +        error_setg(errp, "Unsupported non-zero const header padding");
 +        return -ENOTSUP;
 +    }
 +
 +    return 0;
 +}
 +
-+static void snapshot_access_child_perm(BlockDriverState *bs, BdrvChild *c,
++static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *header,
-+                                BdrvChildRole role,
++                                           Error **errp)
 +                                BlockReopenQueue *reopen_queue,
 +                                uint64_t perm, uint64_t shared,
 +                                uint64_t *nperm, uint64_t *nshared)
 +{
-+    /*
++    header->magic = le64_to_cpu(header->magic);
-+     * Currently, we don't need any permissions. If bs->file provides
++    header->free_gt_number = le64_to_cpu(header->free_gt_number);
-+     * snapshot-access API, we can use it.
++    header->next_txn_seq_number = le64_to_cpu(header->next_txn_seq_number);
-+     */
++    header->replay_journal = le64_to_cpu(header->replay_journal);
-+    *nperm = 0;
++
-+    *nshared = BLK_PERM_ALL;
++    if (header->magic != SESPARSE_VOLATILE_HEADER_MAGIC) {
 +        error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64,
 +                   header->magic);
 +        return -EINVAL;
 +    }
 +
 +    if (header->replay_journal) {
 +        error_setg(errp, "Image is dirty, Replaying journal not supported");
 +        return -ENOTSUP;
 +    }
 +
 +    /* check that padding is 0 */
 +    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
 +        error_setg(errp, "Unsupported non-zero volatile header padding");
 +        return -ENOTSUP;
 +    }
 +
 +    return 0;
 +}
 +
-+BlockDriver bdrv_snapshot_access_drv = {
++static int vmdk_open_se_sparse(BlockDriverState *bs,
-+    .format_name = "snapshot-access",
++                               BdrvChild *file,
-+
++                               int flags, Error **errp)
 +    .bdrv_open                  = snapshot_access_open,
 +
 +    .bdrv_co_preadv_part        = snapshot_access_co_preadv_part,
 +    .bdrv_co_pwritev_part       = snapshot_access_co_pwritev_part,
 +    .bdrv_co_pwrite_zeroes      = snapshot_access_co_pwrite_zeroes,
 +    .bdrv_co_pdiscard           = snapshot_access_co_pdiscard,
 +    .bdrv_co_block_status       = snapshot_access_co_block_status,
 +
 +    .bdrv_refresh_filename      = snapshot_access_refresh_filename,
 +
 +    .bdrv_child_perm            = snapshot_access_child_perm,
 +};
 +
 +static void snapshot_access_init(void)
 +{
-+    bdrv_register(&bdrv_snapshot_access_drv);
++    int ret;
 +    VMDKSESparseConstHeader const_header;
 +    VMDKSESparseVolatileHeader volatile_header;
 +    VmdkExtent *extent;
 +
 +    ret = bdrv_apply_auto_read_only(bs,
 +            "No write support for seSparse images available", errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    assert(sizeof(const_header) == SECTOR_SIZE);
 +
 +    ret = bdrv_pread(file, 0, &const_header, sizeof(const_header));
 +    if (ret < 0) {
 +        bdrv_refresh_filename(file->bs);
 +        error_setg_errno(errp, -ret,
 +                         "Could not read const header from file '%s'",
 +                         file->bs->filename);
 +        return ret;
 +    }
 +
 +    /* check const header */
 +    ret = check_se_sparse_const_header(&const_header, errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    assert(sizeof(volatile_header) == SECTOR_SIZE);
 +
 +    ret = bdrv_pread(file,
 +                     const_header.volatile_header_offset * SECTOR_SIZE,
 +                     &volatile_header, sizeof(volatile_header));
 +    if (ret < 0) {
 +        bdrv_refresh_filename(file->bs);
 +        error_setg_errno(errp, -ret,
 +                         "Could not read volatile header from file '%s'",
 +                         file->bs->filename);
 +        return ret;
 +    }
 +
 +    /* check volatile header */
 +    ret = check_se_sparse_volatile_header(&volatile_header, errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    ret = vmdk_add_extent(bs, file, false,
 +                          const_header.capacity,
 +                          const_header.grain_dir_offset * SECTOR_SIZE,
 +                          0,
 +                          const_header.grain_dir_size *
 +                          SECTOR_SIZE / sizeof(uint64_t),
 +                          const_header.grain_table_size *
 +                          SECTOR_SIZE / sizeof(uint64_t),
 +                          const_header.grain_size,
 +                          &extent,
 +                          errp);
 +    if (ret < 0) {
 +        return ret;
 +    }
 +
 +    extent->sesparse = true;
 +    extent->sesparse_l2_tables_offset = const_header.grain_tables_offset;
 +    extent->sesparse_clusters_offset = const_header.grains_offset;
 +    extent->entry_size = sizeof(uint64_t);
 +
 +    ret = vmdk_init_tables(bs, extent, errp);
 +    if (ret) {
 +        /* free extent allocated by vmdk_add_extent */
 +        vmdk_free_last_extent(bs);
 +    }
 +
 +    return ret;
 +}
 +
-+block_init(snapshot_access_init);
+ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
-diff --git a/MAINTAINERS b/MAINTAINERS
+                                QDict *options, Error **errp);
-index XXXXXXX..XXXXXXX 100644
---- a/MAINTAINERS
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-+++ b/MAINTAINERS
+          * RW [size in sectors] SPARSE "file-name.vmdk"
-@@ -XXX,XX +XXX,XX @@ F: block/reqlist.c
+          * RW [size in sectors] VMFS "file-name.vmdk"
- F: include/block/reqlist.h
+          * RW [size in sectors] VMFSSPARSE "file-name.vmdk"
- F: block/copy-before-write.h
++         * RW [size in sectors] SESPARSE "file-name.vmdk"
- F: block/copy-before-write.c
+          */
-+F: block/snapshot-access.c
+         flat_offset = -1;
- F: include/block/aio_task.h
+         matches = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,
- F: block/aio_task.c
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
- F: util/qemu-co-shared-resource.c
-diff --git a/block/meson.build b/block/meson.build
+         if (sectors <= 0 ||
-index XXXXXXX..XXXXXXX 100644
+             (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
---- a/block/meson.build
+-             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) ||
-+++ b/block/meson.build
++             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") &&
-@@ -XXX,XX +XXX,XX @@ block_ss.add(files(
++             strcmp(type, "SESPARSE")) ||
-   'raw-format.c',
+             (strcmp(access, "RW"))) {
-   'reqlist.c',
+             continue;
-   'snapshot.c',
+         }
-+  'snapshot-access.c',
+@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
-   'throttle-groups.c',
+                 return ret;
-   'throttle.c',
+             }
-   'vhdx-endian.c',
+             extent = &s->extents[s->num_extents - 1];
 +        } else if (!strcmp(type, "SESPARSE")) {
 +            ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
 +            if (ret) {
 +                bdrv_unref_child(bs, extent_file);
 +                return ret;
 +            }
 +            extent = &s->extents[s->num_extents - 1];
          } else {
              error_setg(errp, "Unsupported extent type '%s'", type);
              bdrv_unref_child(bs, extent_file);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
      if (strcmp(ct, "monolithicFlat") &&
          strcmp(ct, "vmfs") &&
          strcmp(ct, "vmfsSparse") &&
 +        strcmp(ct, "seSparse") &&
          strcmp(ct, "twoGbMaxExtentSparse") &&
          strcmp(ct, "twoGbMaxExtentFlat")) {
          error_setg(errp, "Unsupported image type '%s'", ct);
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
  {
      unsigned int l1_index, l2_offset, l2_index;
      int min_index, i, j;
 -    uint32_t min_count, *l2_table;
 +    uint32_t min_count;
 +    void *l2_table;
      bool zeroed = false;
      int64_t ret;
      int64_t cluster_sector;
 +    unsigned int l2_size_bytes = extent->l2_size * extent->entry_size;
      if (m_data) {
          m_data->valid = 0;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
      if (l1_index >= extent->l1_size) {
          return VMDK_ERROR;
      }
 -    l2_offset = extent->l1_table[l1_index];
 +    if (extent->sesparse) {
 +        uint64_t l2_offset_u64;
 +
 +        assert(extent->entry_size == sizeof(uint64_t));
 +
 +        l2_offset_u64 = ((uint64_t *)extent->l1_table)[l1_index];
 +        if (l2_offset_u64 == 0) {
 +            l2_offset = 0;
 +        } else if ((l2_offset_u64 & 0xffffffff00000000) != 0x1000000000000000) {
 +            /*
 +             * Top most nibble is 0x1 if grain table is allocated.
 +             * strict check - top most 4 bytes must be 0x10000000 since max
 +             * supported size is 64TB for disk - so no more than 64TB / 16MB
 +             * grain directories which is smaller than uint32,
 +             * where 16MB is the only supported default grain table coverage.
 +             */
 +            return VMDK_ERROR;
 +        } else {
 +            l2_offset_u64 = l2_offset_u64 & 0x00000000ffffffff;
 +            l2_offset_u64 = extent->sesparse_l2_tables_offset +
 +                l2_offset_u64 * l2_size_bytes / SECTOR_SIZE;
 +            if (l2_offset_u64 > 0x00000000ffffffff) {
 +                return VMDK_ERROR;
 +            }
 +            l2_offset = (unsigned int)(l2_offset_u64);
 +        }
 +    } else {
 +        assert(extent->entry_size == sizeof(uint32_t));
 +        l2_offset = ((uint32_t *)extent->l1_table)[l1_index];
 +    }
      if (!l2_offset) {
          return VMDK_UNALLOC;
      }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
                      extent->l2_cache_counts[j] >>= 1;
                  }
              }
 -            l2_table = extent->l2_cache + (i * extent->l2_size);
 +            l2_table = (char *)extent->l2_cache + (i * l2_size_bytes);
              goto found;
          }
      }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
              min_index = i;
          }
      }
 -    l2_table = extent->l2_cache + (min_index * extent->l2_size);
 +    l2_table = (char *)extent->l2_cache + (min_index * l2_size_bytes);
      BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD);
      if (bdrv_pread(extent->file,
                  (int64_t)l2_offset * 512,
                  l2_table,
 -                extent->l2_size * sizeof(uint32_t)
 -            ) != extent->l2_size * sizeof(uint32_t)) {
 +                l2_size_bytes
 +            ) != l2_size_bytes) {
          return VMDK_ERROR;
      }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
      extent->l2_cache_counts[min_index] = 1;
   found:
      l2_index = ((offset >> 9) / extent->cluster_sectors) % extent->l2_size;
 -    cluster_sector = le32_to_cpu(l2_table[l2_index]);
 -    if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
 -        zeroed = true;
 +    if (extent->sesparse) {
 +        cluster_sector = le64_to_cpu(((uint64_t *)l2_table)[l2_index]);
 +        switch (cluster_sector & 0xf000000000000000) {
 +        case 0x0000000000000000:
 +            /* unallocated grain */
 +            if (cluster_sector != 0) {
 +                return VMDK_ERROR;
 +            }
 +            break;
 +        case 0x1000000000000000:
 +            /* scsi-unmapped grain - fallthrough */
 +        case 0x2000000000000000:
 +            /* zero grain */
 +            zeroed = true;
 +            break;
 +        case 0x3000000000000000:
 +            /* allocated grain */
 +            cluster_sector = (((cluster_sector & 0x0fff000000000000) >> 48) |
 +                              ((cluster_sector & 0x0000ffffffffffff) << 12));
 +            cluster_sector = extent->sesparse_clusters_offset +
 +                cluster_sector * extent->cluster_sectors;
 +            break;
 +        default:
 +            return VMDK_ERROR;
 +        }
 +    } else {
 +        cluster_sector = le32_to_cpu(((uint32_t *)l2_table)[l2_index]);
 +
 +        if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
 +            zeroed = true;
 +        }
      }
      if (!cluster_sector || zeroed) {
          if (!allocate) {
              return zeroed ? VMDK_ZEROED : VMDK_UNALLOC;
          }
 +        assert(!extent->sesparse);
          if (extent->next_cluster_sector >= VMDK_EXTENT_MAX_SECTORS) {
              return VMDK_ERROR;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
              m_data->l1_index = l1_index;
              m_data->l2_index = l2_index;
              m_data->l2_offset = l2_offset;
 -            m_data->l2_cache_entry = &l2_table[l2_index];
 +            m_data->l2_cache_entry = ((uint32_t *)l2_table) + l2_index;
          }
      }
      *cluster_offset = cluster_sector << BDRV_SECTOR_BITS;
@@ -XXX,XX +XXX,XX @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_t offset,
          if (!extent) {
              return -EIO;
          }
 +        if (extent->sesparse) {
 +            return -ENOTSUP;
 +        }
          offset_in_cluster = vmdk_find_offset_in_cluster(extent, offset);
          n_bytes = MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE
                               - offset_in_cluster);
 --
-.34.1
+.21.0

-[PULL 19/23] block: copy-before-write: realize snapshot-access API
+[Qemu-devel] [PULL v2 7/8] ssh: switch from libssh2 to libssh
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+From: Pino Toscano <ptoscano@redhat.com>
-Current scheme of image fleecing looks like this:
+Rewrite the implementation of the ssh block driver to use libssh instead
 of libssh2.  The libssh library has various advantages over libssh2:
 - easier API for authentication (for example for using ssh-agent)
 - easier API for known_hosts handling
 - supports newer types of keys in known_hosts
-[guest]                    [NBD export]
+Use APIs/features available in libssh 0.8 conditionally, to support
-  |                              |
+older versions (which are not recommended though).
   |root                          | root
   v                              v
 [copy-before-write] -----> [temp.qcow2]
   |                 target  |
   |file                     |backing
   v                         |
 [active disk] <-------------+
- - On guest writes copy-before-write filter copies old data from active
+Adjust the iotest 207 according to the different error message, and to
-   disk to temp.qcow2. So fleecing client (NBD export) when reads
+find the default key type for localhost (to properly compare the
-   changed regions from temp.qcow2 image and unchanged from active disk
+fingerprint with).
-   through backing link.
+Contributed-by: Max Reitz <mreitz@redhat.com>
-This patch makes possible new image fleecing scheme:
+Adjust the various Docker/Travis scripts to use libssh when available
 instead of libssh2. The mingw/mxe testing is dropped for now, as there
 are no packages for it.
-[guest]                   [NBD export]
+Signed-off-by: Pino Toscano <ptoscano@redhat.com>
-   |                            |
+Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
-   | root                       | root
+Acked-by: Alex Bennée <alex.bennee@linaro.org>
-   v                 file       v
+Message-id: 20190620200840.17655-1-ptoscano@redhat.com
-[copy-before-write]<------[snapshot-access]
+Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
-   |           |
+Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com
-   | file      | target
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-   v           v
+---
-[active-disk] [temp.img]
+ configure                                     |  65 +-
  block/Makefile.objs                           |   6 +-
  block/ssh.c                                   | 652 ++++++++++--------
  .travis.yml                                   |   4 +-
  block/trace-events                            |  14 +-
  docs/qemu-block-drivers.texi                  |   2 +-
  .../dockerfiles/debian-win32-cross.docker     |   1 -
  .../dockerfiles/debian-win64-cross.docker     |   1 -
  tests/docker/dockerfiles/fedora.docker        |   4 +-
  tests/docker/dockerfiles/ubuntu.docker        |   2 +-
  tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
  tests/qemu-iotests/207                        |  54 +-
  tests/qemu-iotests/207.out                    |   2 +-
 files changed, 449 insertions(+), 360 deletions(-)
- - copy-before-write does CBW operations and also provides
+diff --git a/configure b/configure
-   snapshot-access API. The API may be accessed through
+index XXXXXXX..XXXXXXX 100755
-   snapshot-access driver.
+--- a/configure
++++ b/configure
-Benefits of new scheme:
+@@ -XXX,XX +XXX,XX @@ auth_pam=""
+ vte=""
-. Access control: if remote client try to read data that not covered
+ virglrenderer=""
-   by original dirty bitmap used on copy-before-write open, client gets
+ tpm=""
-   -EACCES.
+-libssh2=""
++libssh=""
-. Discard support: if remote client do DISCARD, this additionally to
+ live_block_migration="yes"
-   discarding data in temp.img informs block-copy process to not copy
+ numa=""
-   these clusters. Next read from discarded area will return -EACCES.
+ tcmalloc="no"
-   This is significant thing: when fleecing user reads data that was
+@@ -XXX,XX +XXX,XX @@ for opt do
-   not yet copied to temp.img, we can avoid copying it on further guest
+   ;;
-   write.
+   --enable-tpm) tpm="yes"
+   ;;
-. Synchronisation between client reads and block-copy write is more
+-  --disable-libssh2) libssh2="no"
-   efficient. In old scheme we just rely on BDRV_REQ_SERIALISING flag
++  --disable-libssh) libssh="no"
-   used for writes to temp.qcow2. New scheme is less blocking:
+   ;;
-     - fleecing reads are never blocked: if data region is untouched or
+-  --enable-libssh2) libssh2="yes"
-       in-flight, we just read from active-disk, otherwise we read from
++  --enable-libssh) libssh="yes"
-       temp.img
+   ;;
-     - writes to temp.img are not blocked by fleecing reads
+   --disable-live-block-migration) live_block_migration="no"
-     - still, guest writes of-course are blocked by in-flight fleecing
+   ;;
-       reads, that currently read from active-disk - it's the minimum
+@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
-       necessary blocking
+   coroutine-pool  coroutine freelist (better performance)
+   glusterfs       GlusterFS backend
-. Temporary image may be of any format, as we don't rely on backing
+   tpm             TPM support
-   feature.
+-  libssh2         ssh block device support
++  libssh          ssh block device support
-. Permission relation are simplified. With old scheme we have to share
+   numa            libnuma support
-   write permission on target child of copy-before-write, otherwise
+   libxml2         for Parallels image format
-   backing link conflicts with copy-before-write file child write
+   tcmalloc        tcmalloc support
-   permissions. With new scheme we don't have backing link, and
+@@ -XXX,XX +XXX,XX @@ EOF
-   copy-before-write node may have unshared access to temporary node.
+ fi
-   (Not realized in this commit, will be in future).
+ ##########################################
-. Having control on fleecing reads we'll be able to implement
+-# libssh2 probe
-   alternative behavior on failed copy-before-write operations.
+-min_libssh2_version=1.2.8
-   Currently we just break guest request (that's a historical behavior
+-if test "$libssh2" != "no" ; then
-   of backup). But in some scenarios it's a bad behavior: better
+-  if $pkg_config --atleast-version=$min_libssh2_version libssh2; then
-   is to drop the backup as failed but don't break guest request.
+-    libssh2_cflags=$($pkg_config libssh2 --cflags)
-   With new scheme we can simply unset some bits in a bitmap on CBW
+-    libssh2_libs=$($pkg_config libssh2 --libs)
-   failure and further fleecing reads will -EACCES, or something like
+-    libssh2=yes
-   this. (Not implemented in this commit, will be in future)
++# libssh probe
-   Additional application for this is implementing timeout for CBW
++if test "$libssh" != "no" ; then
-   operations.
++  if $pkg_config --exists libssh; then
++    libssh_cflags=$($pkg_config libssh --cflags)
-Iotest 257 output is updated, as two more bitmaps now live in
++    libssh_libs=$($pkg_config libssh --libs)
-copy-before-write filter.
++    libssh=yes
+   else
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+-    if test "$libssh2" = "yes" ; then
-Message-Id: <20220303194349.2304213-13-vsementsov@virtuozzo.com>
+-      error_exit "libssh2 >= $min_libssh2_version required for --enable-libssh2"
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
++    if test "$libssh" = "yes" ; then
----
++      error_exit "libssh required for --enable-libssh"
- block/copy-before-write.c  | 212 ++++++++++++++++++++++++++++++++++-
+     fi
- tests/qemu-iotests/257.out | 224 +++++++++++++++++++++++++++++++++++++
+-    libssh2=no
-files changed, 435 insertions(+), 1 deletion(-)
++    libssh=no
+   fi
-diff --git a/block/copy-before-write.c b/block/copy-before-write.c
+ fi
  ##########################################
 -# libssh2_sftp_fsync probe
 +# Check for libssh 0.8
 +# This is done like this instead of using the LIBSSH_VERSION_* and
 +# SSH_VERSION_* macros because some distributions in the past shipped
 +# snapshots of the future 0.8 from Git, and those snapshots did not
 +# have updated version numbers (still referring to 0.7.0).
 -if test "$libssh2" = "yes"; then
 +if test "$libssh" = "yes"; then
    cat > $TMPC <<EOF
 -#include <stdio.h>
 -#include <libssh2.h>
 -#include <libssh2_sftp.h>
 -int main(void) {
 -    LIBSSH2_SESSION *session;
 -    LIBSSH2_SFTP *sftp;
 -    LIBSSH2_SFTP_HANDLE *sftp_handle;
 -    session = libssh2_session_init ();
 -    sftp = libssh2_sftp_init (session);
 -    sftp_handle = libssh2_sftp_open (sftp, "/", 0, 0);
 -    libssh2_sftp_fsync (sftp_handle);
 -    return 0;
 -}
 +#include <libssh/libssh.h>
 +int main(void) { return ssh_get_server_publickey(NULL, NULL); }
  EOF
 -  # libssh2_cflags/libssh2_libs defined in previous test.
 -  if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then
 -    QEMU_CFLAGS="-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS"
 +  if compile_prog "$libssh_cflags" "$libssh_libs"; then
 +    libssh_cflags="-DHAVE_LIBSSH_0_8 $libssh_cflags"
    fi
  fi
@@ -XXX,XX +XXX,XX @@ echo "GlusterFS support $glusterfs"
  echo "gcov              $gcov_tool"
  echo "gcov enabled      $gcov"
  echo "TPM support       $tpm"
 -echo "libssh2 support   $libssh2"
 +echo "libssh support    $libssh"
  echo "QOM debugging     $qom_cast_debug"
  echo "Live block migration $live_block_migration"
  echo "lzo support       $lzo"
@@ -XXX,XX +XXX,XX @@ if test "$glusterfs_iocb_has_stat" = "yes" ; then
    echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=y" >> $config_host_mak
  fi
 -if test "$libssh2" = "yes" ; then
 -  echo "CONFIG_LIBSSH2=m" >> $config_host_mak
 -  echo "LIBSSH2_CFLAGS=$libssh2_cflags" >> $config_host_mak
 -  echo "LIBSSH2_LIBS=$libssh2_libs" >> $config_host_mak
 +if test "$libssh" = "yes" ; then
 +  echo "CONFIG_LIBSSH=m" >> $config_host_mak
 +  echo "LIBSSH_CFLAGS=$libssh_cflags" >> $config_host_mak
 +  echo "LIBSSH_LIBS=$libssh_libs" >> $config_host_mak
  fi
  if test "$live_block_migration" = "yes" ; then
 diff --git a/block/Makefile.objs b/block/Makefile.objs
 index XXXXXXX..XXXXXXX 100644
---- a/block/copy-before-write.c
+--- a/block/Makefile.objs
-+++ b/block/copy-before-write.c
++++ b/block/Makefile.objs
@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_CURL) += curl.o
  block-obj-$(CONFIG_RBD) += rbd.o
  block-obj-$(CONFIG_GLUSTERFS) += gluster.o
  block-obj-$(CONFIG_VXHS) += vxhs.o
 -block-obj-$(CONFIG_LIBSSH2) += ssh.o
 +block-obj-$(CONFIG_LIBSSH) += ssh.o
  block-obj-y += accounting.o dirty-bitmap.o
  block-obj-y += write-threshold.o
  block-obj-y += backup.o
@@ -XXX,XX +XXX,XX @@ rbd.o-libs         := $(RBD_LIBS)
  gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
  gluster.o-libs     := $(GLUSTERFS_LIBS)
  vxhs.o-libs        := $(VXHS_LIBS)
 -ssh.o-cflags       := $(LIBSSH2_CFLAGS)
 -ssh.o-libs         := $(LIBSSH2_LIBS)
 +ssh.o-cflags       := $(LIBSSH_CFLAGS)
 +ssh.o-libs         := $(LIBSSH_LIBS)
  block-obj-dmg-bz2-$(CONFIG_BZIP2) += dmg-bz2.o
  block-obj-$(if $(CONFIG_DMG),m,n) += $(block-obj-dmg-bz2-y)
  dmg-bz2.o-libs     := $(BZIP2_LIBS)
 diff --git a/block/ssh.c b/block/ssh.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/ssh.c
 +++ b/block/ssh.c
 @@ -XXX,XX +XXX,XX @@
- #include "block/block-copy.h"
+ #include "qemu/osdep.h"
- #include "block/copy-before-write.h"
-+#include "block/reqlist.h"
+-#include <libssh2.h>
+-#include <libssh2_sftp.h>
- #include "qapi/qapi-visit-block-core.h"
++#include <libssh/libssh.h>
++#include <libssh/sftp.h>
- typedef struct BDRVCopyBeforeWriteState {
-     BlockCopyState *bcs;
+ #include "block/block_int.h"
-     BdrvChild *target;
+ #include "block/qdict.h"
-+
+@@ -XXX,XX +XXX,XX @@
  #include "trace.h"
  /*
 - * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
 - * that this requires that libssh2 was specially compiled with the
 - * `./configure --enable-debug' option, so most likely you will have
 - * to compile it yourself.  The meaning of <bitmask> is described
 - * here: http://www.libssh2.org/libssh2_trace.html
 + * TRACE_LIBSSH=<level> enables tracing in libssh itself.
 + * The meaning of <level> is described here:
 + * http://api.libssh.org/master/group__libssh__log.html
   */
 -#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
 +#define TRACE_LIBSSH  0 /* see: SSH_LOG_* */
  typedef struct BDRVSSHState {
      /* Coroutine. */
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVSSHState {
      /* SSH connection. */
      int sock;                         /* socket */
 -    LIBSSH2_SESSION *session;         /* ssh session */
 -    LIBSSH2_SFTP *sftp;               /* sftp session */
 -    LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */
 +    ssh_session session;              /* ssh session */
 +    sftp_session sftp;                /* sftp session */
 +    sftp_file sftp_handle;            /* sftp remote file handle */
 -    /* See ssh_seek() function below. */
 -    int64_t offset;
 -    bool offset_op_read;
 -
 -    /* File attributes at open.  We try to keep the .filesize field
 +    /*
-+     * @lock: protects access to @access_bitmap, @done_bitmap and
++     * File attributes at open.  We try to keep the .size field
-+     * @frozen_read_reqs
+      * updated if it changes (eg by writing at the end of the file).
-+     */
+      */
-+    CoMutex lock;
+-    LIBSSH2_SFTP_ATTRIBUTES attrs;
-+
++    sftp_attributes attrs;
-+    /*
-+     * @access_bitmap: represents areas allowed for reading by fleecing user.
+     InetSocketAddress *inet;
-+     * Reading from non-dirty areas leads to -EACCES.
-+     */
+@@ -XXX,XX +XXX,XX @@ static void ssh_state_init(BDRVSSHState *s)
-+    BdrvDirtyBitmap *access_bitmap;
+ {
-+
+     memset(s, 0, sizeof *s);
-+    /*
+     s->sock = -1;
-+     * @done_bitmap: represents areas that was successfully copied to @target by
+-    s->offset = -1;
-+     * copy-before-write operations.
+     qemu_co_mutex_init(&s->lock);
 +     */
 +    BdrvDirtyBitmap *done_bitmap;
 +
 +    /*
 +     * @frozen_read_reqs: current read requests for fleecing user in bs->file
 +     * node. These areas must not be rewritten by guest.
 +     */
 +    BlockReqList frozen_read_reqs;
  } BDRVCopyBeforeWriteState;
  static coroutine_fn int cbw_co_preadv(
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int cbw_co_preadv(
      return bdrv_co_preadv(bs->file, offset, bytes, qiov, flags);
  }
-+/*
+@@ -XXX,XX +XXX,XX @@ static void ssh_state_free(BDRVSSHState *s)
 + * Do copy-before-write operation.
 + *
 + * On failure guest request must be failed too.
 + *
 + * On success, we also wait for all in-flight fleecing read requests in source
 + * node, and it's guaranteed that after cbw_do_copy_before_write() successful
 + * return there are no such requests and they will never appear.
 + */
  static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
          uint64_t offset, uint64_t bytes, BdrvRequestFlags flags)
  {
-     BDRVCopyBeforeWriteState *s = bs->opaque;
+     g_free(s->user);
 +    if (s->attrs) {
 +        sftp_attributes_free(s->attrs);
 +    }
      if (s->sftp_handle) {
 -        libssh2_sftp_close(s->sftp_handle);
 +        sftp_close(s->sftp_handle);
      }
      if (s->sftp) {
 -        libssh2_sftp_shutdown(s->sftp);
 +        sftp_free(s->sftp);
      }
      if (s->session) {
 -        libssh2_session_disconnect(s->session,
 -                                   "from qemu ssh client: "
 -                                   "user closed the connection");
 -        libssh2_session_free(s->session);
 -    }
 -    if (s->sock >= 0) {
 -        close(s->sock);
 +        ssh_disconnect(s->session);
 +        ssh_free(s->session); /* This frees s->sock */
      }
  }
@@ -XXX,XX +XXX,XX @@ session_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
      va_end(args);
      if (s->session) {
 -        char *ssh_err;
 +        const char *ssh_err;
          int ssh_err_code;
 -        /* This is not an errno.  See <libssh2.h>. */
 -        ssh_err_code = libssh2_session_last_error(s->session,
 -                                                  &ssh_err, NULL, 0);
 -        error_setg(errp, "%s: %s (libssh2 error code: %d)",
 +        /* This is not an errno.  See <libssh/libssh.h>. */
 +        ssh_err = ssh_get_error(s->session);
 +        ssh_err_code = ssh_get_error_code(s->session);
 +        error_setg(errp, "%s: %s (libssh error code: %d)",
                     msg, ssh_err, ssh_err_code);
      } else {
          error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
      va_end(args);
      if (s->sftp) {
 -        char *ssh_err;
 +        const char *ssh_err;
          int ssh_err_code;
 -        unsigned long sftp_err_code;
 +        int sftp_err_code;
 -        /* This is not an errno.  See <libssh2.h>. */
 -        ssh_err_code = libssh2_session_last_error(s->session,
 -                                                  &ssh_err, NULL, 0);
 -        /* See <libssh2_sftp.h>. */
 -        sftp_err_code = libssh2_sftp_last_error((s)->sftp);
 +        /* This is not an errno.  See <libssh/libssh.h>. */
 +        ssh_err = ssh_get_error(s->session);
 +        ssh_err_code = ssh_get_error_code(s->session);
 +        /* See <libssh/sftp.h>. */
 +        sftp_err_code = sftp_get_error(s->sftp);
          error_setg(errp,
 -                   "%s: %s (libssh2 error code: %d, sftp error code: %lu)",
 +                   "%s: %s (libssh error code: %d, sftp error code: %d)",
                     msg, ssh_err, ssh_err_code, sftp_err_code);
      } else {
          error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
  static void sftp_error_trace(BDRVSSHState *s, const char *op)
  {
 -    char *ssh_err;
 +    const char *ssh_err;
      int ssh_err_code;
 -    unsigned long sftp_err_code;
 +    int sftp_err_code;
 -    /* This is not an errno.  See <libssh2.h>. */
 -    ssh_err_code = libssh2_session_last_error(s->session,
 -                                              &ssh_err, NULL, 0);
 -    /* See <libssh2_sftp.h>. */
 -    sftp_err_code = libssh2_sftp_last_error((s)->sftp);
 +    /* This is not an errno.  See <libssh/libssh.h>. */
 +    ssh_err = ssh_get_error(s->session);
 +    ssh_err_code = ssh_get_error_code(s->session);
 +    /* See <libssh/sftp.h>. */
 +    sftp_err_code = sftp_get_error(s->sftp);
      trace_sftp_error(op, ssh_err, ssh_err_code, sftp_err_code);
  }
@@ -XXX,XX +XXX,XX @@ static void ssh_parse_filename(const char *filename, QDict *options,
      parse_uri(filename, options, errp);
  }
 -static int check_host_key_knownhosts(BDRVSSHState *s,
 -                                     const char *host, int port, Error **errp)
 +static int check_host_key_knownhosts(BDRVSSHState *s, Error **errp)
  {
 -    const char *home;
 -    char *knh_file = NULL;
 -    LIBSSH2_KNOWNHOSTS *knh = NULL;
 -    struct libssh2_knownhost *found;
 -    int ret, r;
 -    const char *hostkey;
 -    size_t len;
 -    int type;
 -
 -    hostkey = libssh2_session_hostkey(s->session, &len, &type);
 -    if (!hostkey) {
 +    int ret;
-     uint64_t off, end;
++#ifdef HAVE_LIBSSH_0_8
-     int64_t cluster_size = block_copy_cluster_size(s->bcs);
++    enum ssh_known_hosts_e state;
++    int r;
-@@ -XXX,XX +XXX,XX @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
++    ssh_key pubkey;
-     off = QEMU_ALIGN_DOWN(offset, cluster_size);
++    enum ssh_keytypes_e pubkey_type;
-     end = QEMU_ALIGN_UP(offset + bytes, cluster_size);
++    unsigned char *server_hash = NULL;
++    size_t server_hash_len;
--    return block_copy(s->bcs, off, end - off, true);
++    char *fingerprint = NULL;
-+    ret = block_copy(s->bcs, off, end - off, true);
++
-+    if (ret < 0) {
++    state = ssh_session_is_known_server(s->session);
-+        return ret;
++    trace_ssh_server_status(state);
-+    }
++
-+
++    switch (state) {
-+    WITH_QEMU_LOCK_GUARD(&s->lock) {
++    case SSH_KNOWN_HOSTS_OK:
-+        bdrv_set_dirty_bitmap(s->done_bitmap, off, end - off);
++        /* OK */
-+        reqlist_wait_all(&s->frozen_read_reqs, off, end - off, &s->lock);
++        trace_ssh_check_host_key_knownhosts();
-+    }
++        break;
-+
++    case SSH_KNOWN_HOSTS_CHANGED:
-+    return 0;
+         ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to read remote host key");
 +        r = ssh_get_server_publickey(s->session, &pubkey);
 +        if (r == 0) {
 +            r = ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256,
 +                                       &server_hash, &server_hash_len);
 +            pubkey_type = ssh_key_type(pubkey);
 +            ssh_key_free(pubkey);
 +        }
 +        if (r == 0) {
 +            fingerprint = ssh_get_fingerprint_hash(SSH_PUBLICKEY_HASH_SHA256,
 +                                                   server_hash,
 +                                                   server_hash_len);
 +            ssh_clean_pubkey_hash(&server_hash);
 +        }
 +        if (fingerprint) {
 +            error_setg(errp,
 +                       "host key (%s key with fingerprint %s) does not match "
 +                       "the one in known_hosts; this may be a possible attack",
 +                       ssh_key_type_to_char(pubkey_type), fingerprint);
 +            ssh_string_free_char(fingerprint);
 +        } else  {
 +            error_setg(errp,
 +                       "host key does not match the one in known_hosts; this "
 +                       "may be a possible attack");
 +        }
          goto out;
 -    }
 -
 -    knh = libssh2_knownhost_init(s->session);
 -    if (!knh) {
 +    case SSH_KNOWN_HOSTS_OTHER:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                           "failed to initialize known hosts support");
 +        error_setg(errp,
 +                   "host key for this server not found, another type exists");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_UNKNOWN:
 +        ret = -EINVAL;
 +        error_setg(errp, "no host key was found in known_hosts");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_NOT_FOUND:
 +        ret = -ENOENT;
 +        error_setg(errp, "known_hosts file not found");
 +        goto out;
 +    case SSH_KNOWN_HOSTS_ERROR:
 +        ret = -EINVAL;
 +        error_setg(errp, "error while checking the host");
 +        goto out;
 +    default:
 +        ret = -EINVAL;
 +        error_setg(errp, "error while checking for known server (%d)", state);
          goto out;
      }
 +#else /* !HAVE_LIBSSH_0_8 */
 +    int state;
 -    home = getenv("HOME");
 -    if (home) {
 -        knh_file = g_strdup_printf("%s/.ssh/known_hosts", home);
 -    } else {
 -        knh_file = g_strdup_printf("/root/.ssh/known_hosts");
 -    }
 -
 -    /* Read all known hosts from OpenSSH-style known_hosts file. */
 -    libssh2_knownhost_readfile(knh, knh_file, LIBSSH2_KNOWNHOST_FILE_OPENSSH);
 +    state = ssh_is_server_known(s->session);
 +    trace_ssh_server_status(state);
 -    r = libssh2_knownhost_checkp(knh, host, port, hostkey, len,
 -                                 LIBSSH2_KNOWNHOST_TYPE_PLAIN|
 -                                 LIBSSH2_KNOWNHOST_KEYENC_RAW,
 -                                 &found);
 -    switch (r) {
 -    case LIBSSH2_KNOWNHOST_CHECK_MATCH:
 +    switch (state) {
 +    case SSH_SERVER_KNOWN_OK:
          /* OK */
 -        trace_ssh_check_host_key_knownhosts(found->key);
 +        trace_ssh_check_host_key_knownhosts();
          break;
 -    case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
 +    case SSH_SERVER_KNOWN_CHANGED:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                      "host key does not match the one in known_hosts"
 -                      " (found key %s)", found->key);
 +        error_setg(errp,
 +                   "host key does not match the one in known_hosts; this "
 +                   "may be a possible attack");
          goto out;
 -    case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND:
 +    case SSH_SERVER_FOUND_OTHER:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "no host key was found in known_hosts");
 +        error_setg(errp,
 +                   "host key for this server not found, another type exists");
 +        goto out;
 +    case SSH_SERVER_FILE_NOT_FOUND:
 +        ret = -ENOENT;
 +        error_setg(errp, "known_hosts file not found");
          goto out;
 -    case LIBSSH2_KNOWNHOST_CHECK_FAILURE:
 +    case SSH_SERVER_NOT_KNOWN:
          ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                      "failure matching the host key with known_hosts");
 +        error_setg(errp, "no host key was found in known_hosts");
 +        goto out;
 +    case SSH_SERVER_ERROR:
 +        ret = -EINVAL;
 +        error_setg(errp, "server error");
          goto out;
      default:
          ret = -EINVAL;
 -        session_error_setg(errp, s, "unknown error matching the host key"
 -                      " with known_hosts (%d)", r);
 +        error_setg(errp, "error while checking for known server (%d)", state);
          goto out;
      }
 +#endif /* !HAVE_LIBSSH_0_8 */
      /* known_hosts checking successful. */
      ret = 0;
   out:
 -    if (knh != NULL) {
 -        libssh2_knownhost_free(knh);
 -    }
 -    g_free(knh_file);
      return ret;
  }
- static int coroutine_fn cbw_co_pdiscard(BlockDriverState *bs,
+@@ -XXX,XX +XXX,XX @@ static int compare_fingerprint(const unsigned char *fingerprint, size_t len,
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn cbw_co_flush(BlockDriverState *bs)
-     return bdrv_co_flush(bs->file->bs);
+ static int
- }
+ check_host_key_hash(BDRVSSHState *s, const char *hash,
+-                    int hash_type, size_t fingerprint_len, Error **errp)
-+/*
++                    enum ssh_publickey_hash_type type, Error **errp)
 + * If @offset not accessible - return NULL.
 + *
 + * Otherwise, set @pnum to some bytes that accessible from @file (@file is set
 + * to bs->file or to s->target). Return newly allocated BlockReq object that
 + * should be than passed to cbw_snapshot_read_unlock().
 + *
 + * It's guaranteed that guest writes will not interact in the region until
 + * cbw_snapshot_read_unlock() called.
 + */
 +static BlockReq *cbw_snapshot_read_lock(BlockDriverState *bs,
 +                                        int64_t offset, int64_t bytes,
 +                                        int64_t *pnum, BdrvChild **file)
 +{
 +    BDRVCopyBeforeWriteState *s = bs->opaque;
 +    BlockReq *req = g_new(BlockReq, 1);
 +    bool done;
 +
 +    QEMU_LOCK_GUARD(&s->lock);
 +
 +    if (bdrv_dirty_bitmap_next_zero(s->access_bitmap, offset, bytes) != -1) {
 +        g_free(req);
 +        return NULL;
 +    }
 +
 +    done = bdrv_dirty_bitmap_status(s->done_bitmap, offset, bytes, pnum);
 +    if (done) {
 +        /*
 +         * Special invalid BlockReq, that is handled in
 +         * cbw_snapshot_read_unlock(). We don't need to lock something to read
 +         * from s->target.
 +         */
 +        *req = (BlockReq) {.offset = -1, .bytes = -1};
 +        *file = s->target;
 +    } else {
 +        reqlist_init_req(&s->frozen_read_reqs, req, offset, bytes);
 +        *file = bs->file;
 +    }
 +
 +    return req;
 +}
 +
 +static void cbw_snapshot_read_unlock(BlockDriverState *bs, BlockReq *req)
 +{
 +    BDRVCopyBeforeWriteState *s = bs->opaque;
 +
 +    if (req->offset == -1 && req->bytes == -1) {
 +        g_free(req);
 +        return;
 +    }
 +
 +    QEMU_LOCK_GUARD(&s->lock);
 +
 +    reqlist_remove_req(req);
 +    g_free(req);
 +}
 +
 +static coroutine_fn int
 +cbw_co_preadv_snapshot(BlockDriverState *bs, int64_t offset, int64_t bytes,
 +                       QEMUIOVector *qiov, size_t qiov_offset)
 +{
 +    BlockReq *req;
 +    BdrvChild *file;
 +    int ret;
 +
 +    /* TODO: upgrade to async loop using AioTask */
 +    while (bytes) {
 +        int64_t cur_bytes;
 +
 +        req = cbw_snapshot_read_lock(bs, offset, bytes, &cur_bytes, &file);
 +        if (!req) {
 +            return -EACCES;
 +        }
 +
 +        ret = bdrv_co_preadv_part(file, offset, cur_bytes,
 +                                  qiov, qiov_offset, 0);
 +        cbw_snapshot_read_unlock(bs, req);
 +        if (ret < 0) {
 +            return ret;
 +        }
 +
 +        bytes -= cur_bytes;
 +        offset += cur_bytes;
 +        qiov_offset += cur_bytes;
 +    }
 +
 +    return 0;
 +}
 +
 +static int coroutine_fn
 +cbw_co_snapshot_block_status(BlockDriverState *bs,
 +                             bool want_zero, int64_t offset, int64_t bytes,
 +                             int64_t *pnum, int64_t *map,
 +                             BlockDriverState **file)
 +{
 +    BDRVCopyBeforeWriteState *s = bs->opaque;
 +    BlockReq *req;
 +    int ret;
 +    int64_t cur_bytes;
 +    BdrvChild *child;
 +
 +    req = cbw_snapshot_read_lock(bs, offset, bytes, &cur_bytes, &child);
 +    if (!req) {
 +        return -EACCES;
 +    }
 +
 +    ret = bdrv_block_status(child->bs, offset, cur_bytes, pnum, map, file);
 +    if (child == s->target) {
 +        /*
 +         * We refer to s->target only for areas that we've written to it.
 +         * And we can not report unallocated blocks in s->target: this will
 +         * break generic block-status-above logic, that will go to
 +         * copy-before-write filtered child in this case.
 +         */
 +        assert(ret & BDRV_BLOCK_ALLOCATED);
 +    }
 +
 +    cbw_snapshot_read_unlock(bs, req);
 +
 +    return ret;
 +}
 +
 +static int coroutine_fn cbw_co_pdiscard_snapshot(BlockDriverState *bs,
 +                                                 int64_t offset, int64_t bytes)
 +{
 +    BDRVCopyBeforeWriteState *s = bs->opaque;
 +
 +    WITH_QEMU_LOCK_GUARD(&s->lock) {
 +        bdrv_reset_dirty_bitmap(s->access_bitmap, offset, bytes);
 +    }
 +
 +    block_copy_reset(s->bcs, offset, bytes);
 +
 +    return bdrv_co_pdiscard(s->target, offset, bytes);
 +}
 +
  static void cbw_refresh_filename(BlockDriverState *bs)
  {
-     pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
+-    const char *fingerprint;
-@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
+-
- {
+-    fingerprint = libssh2_hostkey_hash(s->session, hash_type);
-     BDRVCopyBeforeWriteState *s = bs->opaque;
+-    if (!fingerprint) {
-     BdrvDirtyBitmap *bitmap = NULL;
++    int r;
-+    int64_t cluster_size;
++    ssh_key pubkey;
++    unsigned char *server_hash;
-     bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
++    size_t server_hash_len;
-                                BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
++
-@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
++#ifdef HAVE_LIBSSH_0_8
 +    r = ssh_get_server_publickey(s->session, &pubkey);
 +#else
 +    r = ssh_get_publickey(s->session, &pubkey);
 +#endif
 +    if (r != SSH_OK) {
          session_error_setg(errp, s, "failed to read remote host key");
          return -EINVAL;
      }
-+    cluster_size = block_copy_cluster_size(s->bcs);
+-    if(compare_fingerprint((unsigned char *) fingerprint, fingerprint_len,
-+
+-                           hash) != 0) {
-+    s->done_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp);
++    r = ssh_get_publickey_hash(pubkey, type, &server_hash, &server_hash_len);
-+    if (!s->done_bitmap) {
++    ssh_key_free(pubkey);
 +    if (r != 0) {
 +        session_error_setg(errp, s,
 +                           "failed reading the hash of the server SSH key");
 +        return -EINVAL;
 +    }
-+    bdrv_disable_dirty_bitmap(s->done_bitmap);
++
-+
++    r = compare_fingerprint(server_hash, server_hash_len, hash);
-+    /* s->access_bitmap starts equal to bcs bitmap */
++    ssh_clean_pubkey_hash(&server_hash);
-+    s->access_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp);
++    if (r != 0) {
-+    if (!s->access_bitmap) {
+         error_setg(errp, "remote host key does not match host_key_check '%s'",
-+        return -EINVAL;
+                    hash);
-+    }
+         return -EPERM;
-+    bdrv_disable_dirty_bitmap(s->access_bitmap);
+@@ -XXX,XX +XXX,XX @@ check_host_key_hash(BDRVSSHState *s, const char *hash,
 +    bdrv_dirty_bitmap_merge_internal(s->access_bitmap,
 +                                     block_copy_dirty_bitmap(s->bcs), NULL,
 +                                     true);
 +
 +    qemu_co_mutex_init(&s->lock);
 +    QLIST_INIT(&s->frozen_read_reqs);
 +
      return 0;
  }
-@@ -XXX,XX +XXX,XX @@ static void cbw_close(BlockDriverState *bs)
+-static int check_host_key(BDRVSSHState *s, const char *host, int port,
 -                          SshHostKeyCheck *hkc, Error **errp)
 +static int check_host_key(BDRVSSHState *s, SshHostKeyCheck *hkc, Error **errp)
  {
-     BDRVCopyBeforeWriteState *s = bs->opaque;
+     SshHostKeyCheckMode mode;
-+    bdrv_release_dirty_bitmap(s->access_bitmap);
+@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
-+    bdrv_release_dirty_bitmap(s->done_bitmap);
+     case SSH_HOST_KEY_CHECK_MODE_HASH:
-+
+         if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) {
-     block_copy_state_free(s->bcs);
+             return check_host_key_hash(s, hkc->u.hash.hash,
-     s->bcs = NULL;
+-                                       LIBSSH2_HOSTKEY_HASH_MD5, 16, errp);
 +                                       SSH_PUBLICKEY_HASH_MD5, errp);
          } else if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_SHA1) {
              return check_host_key_hash(s, hkc->u.hash.hash,
 -                                       LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp);
 +                                       SSH_PUBLICKEY_HASH_SHA1, errp);
          }
          g_assert_not_reached();
          break;
      case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS:
 -        return check_host_key_knownhosts(s, host, port, errp);
 +        return check_host_key_knownhosts(s, errp);
      default:
          g_assert_not_reached();
      }
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
      return -EINVAL;
  }
-@@ -XXX,XX +XXX,XX @@ BlockDriver bdrv_cbw_filter = {
-     .bdrv_co_pdiscard           = cbw_co_pdiscard,
+-static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
-     .bdrv_co_flush              = cbw_co_flush,
++static int authenticate(BDRVSSHState *s, Error **errp)
+ {
-+    .bdrv_co_preadv_snapshot       = cbw_co_preadv_snapshot,
+     int r, ret;
-+    .bdrv_co_pdiscard_snapshot     = cbw_co_pdiscard_snapshot,
+-    const char *userauthlist;
-+    .bdrv_co_snapshot_block_status = cbw_co_snapshot_block_status,
+-    LIBSSH2_AGENT *agent = NULL;
-+
+-    struct libssh2_agent_publickey *identity;
-     .bdrv_refresh_filename      = cbw_refresh_filename,
+-    struct libssh2_agent_publickey *prev_identity = NULL;
++    int method;
-     .bdrv_child_perm            = cbw_child_perm,
-diff --git a/tests/qemu-iotests/257.out b/tests/qemu-iotests/257.out
+-    userauthlist = libssh2_userauth_list(s->session, user, strlen(user));
 -    if (strstr(userauthlist, "publickey") == NULL) {
 +    /* Try to authenticate with the "none" method. */
 +    r = ssh_userauth_none(s->session, NULL);
 +    if (r == SSH_AUTH_ERROR) {
          ret = -EPERM;
 -        error_setg(errp,
 -                "remote server does not support \"publickey\" authentication");
 +        session_error_setg(errp, s, "failed to authenticate using none "
 +                                    "authentication");
          goto out;
 -    }
 -
 -    /* Connect to ssh-agent and try each identity in turn. */
 -    agent = libssh2_agent_init(s->session);
 -    if (!agent) {
 -        ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to initialize ssh-agent support");
 -        goto out;
 -    }
 -    if (libssh2_agent_connect(agent)) {
 -        ret = -ECONNREFUSED;
 -        session_error_setg(errp, s, "failed to connect to ssh-agent");
 -        goto out;
 -    }
 -    if (libssh2_agent_list_identities(agent)) {
 -        ret = -EINVAL;
 -        session_error_setg(errp, s,
 -                           "failed requesting identities from ssh-agent");
 +    } else if (r == SSH_AUTH_SUCCESS) {
 +        /* Authenticated! */
 +        ret = 0;
          goto out;
      }
 -    for(;;) {
 -        r = libssh2_agent_get_identity(agent, &identity, prev_identity);
 -        if (r == 1) {           /* end of list */
 -            break;
 -        }
 -        if (r < 0) {
 +    method = ssh_userauth_list(s->session, NULL);
 +    trace_ssh_auth_methods(method);
 +
 +    /*
 +     * Try to authenticate with publickey, using the ssh-agent
 +     * if available.
 +     */
 +    if (method & SSH_AUTH_METHOD_PUBLICKEY) {
 +        r = ssh_userauth_publickey_auto(s->session, NULL, NULL);
 +        if (r == SSH_AUTH_ERROR) {
              ret = -EINVAL;
 -            session_error_setg(errp, s,
 -                               "failed to obtain identity from ssh-agent");
 +            session_error_setg(errp, s, "failed to authenticate using "
 +                                        "publickey authentication");
              goto out;
 -        }
 -        r = libssh2_agent_userauth(agent, user, identity);
 -        if (r == 0) {
 +        } else if (r == SSH_AUTH_SUCCESS) {
              /* Authenticated! */
              ret = 0;
              goto out;
          }
 -        /* Failed to authenticate with this identity, try the next one. */
 -        prev_identity = identity;
      }
      ret = -EPERM;
@@ -XXX,XX +XXX,XX @@ static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
                 "and the identities held by your ssh-agent");
   out:
 -    if (agent != NULL) {
 -        /* Note: libssh2 implementation implicitly calls
 -         * libssh2_agent_disconnect if necessary.
 -         */
 -        libssh2_agent_free(agent);
 -    }
 -
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
                            int ssh_flags, int creat_mode, Error **errp)
  {
      int r, ret;
 -    long port = 0;
 +    unsigned int port = 0;
 +    int new_sock = -1;
      if (opts->has_user) {
          s->user = g_strdup(opts->user);
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
      s->inet = opts->server;
      opts->server = NULL;
 -    if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) {
 +    if (qemu_strtoui(s->inet->port, NULL, 10, &port) < 0) {
          error_setg(errp, "Use only numeric port value");
          ret = -EINVAL;
          goto err;
      }
      /* Open the socket and connect. */
 -    s->sock = inet_connect_saddr(s->inet, errp);
 -    if (s->sock < 0) {
 +    new_sock = inet_connect_saddr(s->inet, errp);
 +    if (new_sock < 0) {
          ret = -EIO;
          goto err;
      }
 +    /*
 +     * Try to disable the Nagle algorithm on TCP sockets to reduce latency,
 +     * but do not fail if it cannot be disabled.
 +     */
 +    r = socket_set_nodelay(new_sock);
 +    if (r < 0) {
 +        warn_report("can't set TCP_NODELAY for the ssh server %s: %s",
 +                    s->inet->host, strerror(errno));
 +    }
 +
      /* Create SSH session. */
 -    s->session = libssh2_session_init();
 +    s->session = ssh_new();
      if (!s->session) {
          ret = -EINVAL;
 -        session_error_setg(errp, s, "failed to initialize libssh2 session");
 +        session_error_setg(errp, s, "failed to initialize libssh session");
          goto err;
      }
 -#if TRACE_LIBSSH2 != 0
 -    libssh2_trace(s->session, TRACE_LIBSSH2);
 -#endif
 +    /*
 +     * Make sure we are in blocking mode during the connection and
 +     * authentication phases.
 +     */
 +    ssh_set_blocking(s->session, 1);
 -    r = libssh2_session_handshake(s->session, s->sock);
 -    if (r != 0) {
 +    r = ssh_options_set(s->session, SSH_OPTIONS_USER, s->user);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the user in the libssh session");
 +        goto err;
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_HOST, s->inet->host);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the host in the libssh session");
 +        goto err;
 +    }
 +
 +    if (port > 0) {
 +        r = ssh_options_set(s->session, SSH_OPTIONS_PORT, &port);
 +        if (r < 0) {
 +            ret = -EINVAL;
 +            session_error_setg(errp, s,
 +                               "failed to set the port in the libssh session");
 +            goto err;
 +        }
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_COMPRESSION, "none");
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to disable the compression in the libssh "
 +                           "session");
 +        goto err;
 +    }
 +
 +    /* Read ~/.ssh/config. */
 +    r = ssh_options_parse_config(s->session, NULL);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s, "failed to parse ~/.ssh/config");
 +        goto err;
 +    }
 +
 +    r = ssh_options_set(s->session, SSH_OPTIONS_FD, &new_sock);
 +    if (r < 0) {
 +        ret = -EINVAL;
 +        session_error_setg(errp, s,
 +                           "failed to set the socket in the libssh session");
 +        goto err;
 +    }
 +    /* libssh took ownership of the socket. */
 +    s->sock = new_sock;
 +    new_sock = -1;
 +
 +    /* Connect. */
 +    r = ssh_connect(s->session);
 +    if (r != SSH_OK) {
          ret = -EINVAL;
          session_error_setg(errp, s, "failed to establish SSH session");
          goto err;
      }
      /* Check the remote host's key against known_hosts. */
 -    ret = check_host_key(s, s->inet->host, port, opts->host_key_check, errp);
 +    ret = check_host_key(s, opts->host_key_check, errp);
      if (ret < 0) {
          goto err;
      }
      /* Authenticate. */
 -    ret = authenticate(s, s->user, errp);
 +    ret = authenticate(s, errp);
      if (ret < 0) {
          goto err;
      }
      /* Start SFTP. */
 -    s->sftp = libssh2_sftp_init(s->session);
 +    s->sftp = sftp_new(s->session);
      if (!s->sftp) {
 -        session_error_setg(errp, s, "failed to initialize sftp handle");
 +        session_error_setg(errp, s, "failed to create sftp handle");
 +        ret = -EINVAL;
 +        goto err;
 +    }
 +
 +    r = sftp_init(s->sftp);
 +    if (r < 0) {
 +        sftp_error_setg(errp, s, "failed to initialize sftp handle");
          ret = -EINVAL;
          goto err;
      }
      /* Open the remote file. */
      trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
 -    s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
 -                                       creat_mode);
 +    s->sftp_handle = sftp_open(s->sftp, opts->path, ssh_flags, creat_mode);
      if (!s->sftp_handle) {
 -        session_error_setg(errp, s, "failed to open remote file '%s'",
 -                           opts->path);
 +        sftp_error_setg(errp, s, "failed to open remote file '%s'",
 +                        opts->path);
          ret = -EINVAL;
          goto err;
      }
 -    r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs);
 -    if (r < 0) {
 +    /* Make sure the SFTP file is handled in blocking mode. */
 +    sftp_file_set_blocking(s->sftp_handle);
 +
 +    s->attrs = sftp_fstat(s->sftp_handle);
 +    if (!s->attrs) {
          sftp_error_setg(errp, s, "failed to read file attributes");
          return -EINVAL;
      }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
      return 0;
   err:
 +    if (s->attrs) {
 +        sftp_attributes_free(s->attrs);
 +    }
 +    s->attrs = NULL;
      if (s->sftp_handle) {
 -        libssh2_sftp_close(s->sftp_handle);
 +        sftp_close(s->sftp_handle);
      }
      s->sftp_handle = NULL;
      if (s->sftp) {
 -        libssh2_sftp_shutdown(s->sftp);
 +        sftp_free(s->sftp);
      }
      s->sftp = NULL;
      if (s->session) {
 -        libssh2_session_disconnect(s->session,
 -                                   "from qemu ssh client: "
 -                                   "error opening connection");
 -        libssh2_session_free(s->session);
 +        ssh_disconnect(s->session);
 +        ssh_free(s->session);
      }
      s->session = NULL;
 +    s->sock = -1;
 +    if (new_sock >= 0) {
 +        close(new_sock);
 +    }
      return ret;
  }
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
      ssh_state_init(s);
 -    ssh_flags = LIBSSH2_FXF_READ;
 +    ssh_flags = 0;
      if (bdrv_flags & BDRV_O_RDWR) {
 -        ssh_flags |= LIBSSH2_FXF_WRITE;
 +        ssh_flags |= O_RDWR;
 +    } else {
 +        ssh_flags |= O_RDONLY;
      }
      opts = ssh_parse_options(options, errp);
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
      }
      /* Go non-blocking. */
 -    libssh2_session_set_blocking(s->session, 0);
 +    ssh_set_blocking(s->session, 0);
      qapi_free_BlockdevOptionsSsh(opts);
      return 0;
   err:
 -    if (s->sock >= 0) {
 -        close(s->sock);
 -    }
 -    s->sock = -1;
 -
      qapi_free_BlockdevOptionsSsh(opts);
      return ret;
@@ -XXX,XX +XXX,XX @@ static int ssh_grow_file(BDRVSSHState *s, int64_t offset, Error **errp)
  {
      ssize_t ret;
      char c[1] = { '\0' };
 -    int was_blocking = libssh2_session_get_blocking(s->session);
 +    int was_blocking = ssh_is_blocking(s->session);
      /* offset must be strictly greater than the current size so we do
       * not overwrite anything */
 -    assert(offset > 0 && offset > s->attrs.filesize);
 +    assert(offset > 0 && offset > s->attrs->size);
 -    libssh2_session_set_blocking(s->session, 1);
 +    ssh_set_blocking(s->session, 1);
 -    libssh2_sftp_seek64(s->sftp_handle, offset - 1);
 -    ret = libssh2_sftp_write(s->sftp_handle, c, 1);
 +    sftp_seek64(s->sftp_handle, offset - 1);
 +    ret = sftp_write(s->sftp_handle, c, 1);
 -    libssh2_session_set_blocking(s->session, was_blocking);
 +    ssh_set_blocking(s->session, was_blocking);
      if (ret < 0) {
          sftp_error_setg(errp, s, "Failed to grow file");
          return -EIO;
      }
 -    s->attrs.filesize = offset;
 +    s->attrs->size = offset;
      return 0;
  }
@@ -XXX,XX +XXX,XX @@ static int ssh_co_create(BlockdevCreateOptions *options, Error **errp)
      ssh_state_init(&s);
      ret = connect_to_ssh(&s, opts->location,
 -                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
 -                         LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
 +                         O_RDWR | O_CREAT | O_TRUNC,
 , errp);
      if (ret < 0) {
          goto fail;
@@ -XXX,XX +XXX,XX @@ static int ssh_has_zero_init(BlockDriverState *bs)
      /* Assume false, unless we can positively prove it's true. */
      int has_zero_init = 0;
 -    if (s->attrs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) {
 -        if (s->attrs.permissions & LIBSSH2_SFTP_S_IFREG) {
 -            has_zero_init = 1;
 -        }
 +    if (s->attrs->type == SSH_FILEXFER_TYPE_REGULAR) {
 +        has_zero_init = 1;
      }
      return has_zero_init;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
          .co = qemu_coroutine_self()
      };
 -    r = libssh2_session_block_directions(s->session);
 +    r = ssh_get_poll_flags(s->session);
 -    if (r & LIBSSH2_SESSION_BLOCK_INBOUND) {
 +    if (r & SSH_READ_PENDING) {
          rd_handler = restart_coroutine;
      }
 -    if (r & LIBSSH2_SESSION_BLOCK_OUTBOUND) {
 +    if (r & SSH_WRITE_PENDING) {
          wr_handler = restart_coroutine;
      }
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
      trace_ssh_co_yield_back(s->sock);
  }
 -/* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
 - * in the remote file.  Notice that it just updates a field in the
 - * sftp_handle structure, so there is no network traffic and it cannot
 - * fail.
 - *
 - * However, `libssh2_sftp_seek64' does have a catastrophic effect on
 - * performance since it causes the handle to throw away all in-flight
 - * reads and buffered readahead data.  Therefore this function tries
 - * to be intelligent about when to call the underlying libssh2 function.
 - */
 -#define SSH_SEEK_WRITE 0
 -#define SSH_SEEK_READ  1
 -#define SSH_SEEK_FORCE 2
 -
 -static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
 -{
 -    bool op_read = (flags & SSH_SEEK_READ) != 0;
 -    bool force = (flags & SSH_SEEK_FORCE) != 0;
 -
 -    if (force || op_read != s->offset_op_read || offset != s->offset) {
 -        trace_ssh_seek(offset);
 -        libssh2_sftp_seek64(s->sftp_handle, offset);
 -        s->offset = offset;
 -        s->offset_op_read = op_read;
 -    }
 -}
 -
  static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
                                   int64_t offset, size_t size,
                                   QEMUIOVector *qiov)
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
      trace_ssh_read(offset, size);
 -    ssh_seek(s, offset, SSH_SEEK_READ);
 +    trace_ssh_seek(offset);
 +    sftp_seek64(s->sftp_handle, offset);
      /* This keeps track of the current iovec element ('i'), where we
       * will write to next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
      buf = i->iov_base;
      end_of_vec = i->iov_base + i->iov_len;
 -    /* libssh2 has a hard-coded limit of 2000 bytes per request,
 -     * although it will also do readahead behind our backs.  Therefore
 -     * we may have to do repeated reads here until we have read 'size'
 -     * bytes.
 -     */
      for (got = 0; got < size; ) {
 +        size_t request_read_size;
      again:
 -        trace_ssh_read_buf(buf, end_of_vec - buf);
 -        r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
 -        trace_ssh_read_return(r);
 +        /*
 +         * The size of SFTP packets is limited to 32K bytes, so limit
 +         * the amount of data requested to 16K, as libssh currently
 +         * does not handle multiple requests on its own.
 +         */
 +        request_read_size = MIN(end_of_vec - buf, 16384);
 +        trace_ssh_read_buf(buf, end_of_vec - buf, request_read_size);
 +        r = sftp_read(s->sftp_handle, buf, request_read_size);
 +        trace_ssh_read_return(r, sftp_get_error(s->sftp));
 -        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +        if (r == SSH_AGAIN) {
              co_yield(s, bs);
              goto again;
          }
 -        if (r < 0) {
 -            sftp_error_trace(s, "read");
 -            s->offset = -1;
 -            return -EIO;
 -        }
 -        if (r == 0) {
 +        if (r == SSH_EOF || (r == 0 && sftp_get_error(s->sftp) == SSH_FX_EOF)) {
              /* EOF: Short read so pad the buffer with zeroes and return it. */
              qemu_iovec_memset(qiov, got, 0, size - got);
              return 0;
          }
 +        if (r <= 0) {
 +            sftp_error_trace(s, "read");
 +            return -EIO;
 +        }
          got += r;
          buf += r;
 -        s->offset += r;
          if (buf >= end_of_vec && got < size) {
              i++;
              buf = i->iov_base;
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
      trace_ssh_write(offset, size);
 -    ssh_seek(s, offset, SSH_SEEK_WRITE);
 +    trace_ssh_seek(offset);
 +    sftp_seek64(s->sftp_handle, offset);
      /* This keeps track of the current iovec element ('i'), where we
       * will read from next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
      end_of_vec = i->iov_base + i->iov_len;
      for (written = 0; written < size; ) {
 +        size_t request_write_size;
      again:
 -        trace_ssh_write_buf(buf, end_of_vec - buf);
 -        r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
 -        trace_ssh_write_return(r);
 +        /*
 +         * Avoid too large data packets, as libssh currently does not
 +         * handle multiple requests on its own.
 +         */
 +        request_write_size = MIN(end_of_vec - buf, 131072);
 +        trace_ssh_write_buf(buf, end_of_vec - buf, request_write_size);
 +        r = sftp_write(s->sftp_handle, buf, request_write_size);
 +        trace_ssh_write_return(r, sftp_get_error(s->sftp));
 -        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +        if (r == SSH_AGAIN) {
              co_yield(s, bs);
              goto again;
          }
          if (r < 0) {
              sftp_error_trace(s, "write");
 -            s->offset = -1;
              return -EIO;
          }
 -        /* The libssh2 API is very unclear about this.  A comment in
 -         * the code says "nothing was acked, and no EAGAIN was
 -         * received!" which apparently means that no data got sent
 -         * out, and the underlying channel didn't return any EAGAIN
 -         * indication.  I think this is a bug in either libssh2 or
 -         * OpenSSH (server-side).  In any case, forcing a seek (to
 -         * discard libssh2 internal buffers), and then trying again
 -         * works for me.
 -         */
 -        if (r == 0) {
 -            ssh_seek(s, offset + written, SSH_SEEK_WRITE|SSH_SEEK_FORCE);
 -            co_yield(s, bs);
 -            goto again;
 -        }
          written += r;
          buf += r;
 -        s->offset += r;
          if (buf >= end_of_vec && written < size) {
              i++;
              buf = i->iov_base;
              end_of_vec = i->iov_base + i->iov_len;
          }
 -        if (offset + written > s->attrs.filesize)
 -            s->attrs.filesize = offset + written;
 +        if (offset + written > s->attrs->size) {
 +            s->attrs->size = offset + written;
 +        }
      }
      return 0;
@@ -XXX,XX +XXX,XX @@ static void unsafe_flush_warning(BDRVSSHState *s, const char *what)
      }
  }
 -#ifdef HAS_LIBSSH2_SFTP_FSYNC
 +#ifdef HAVE_LIBSSH_0_8
  static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
  {
      int r;
      trace_ssh_flush();
 +
 +    if (!sftp_extension_supported(s->sftp, "fsync@openssh.com", "1")) {
 +        unsafe_flush_warning(s, "OpenSSH >= 6.3");
 +        return 0;
 +    }
   again:
 -    r = libssh2_sftp_fsync(s->sftp_handle);
 -    if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
 +    r = sftp_fsync(s->sftp_handle);
 +    if (r == SSH_AGAIN) {
          co_yield(s, bs);
          goto again;
      }
 -    if (r == LIBSSH2_ERROR_SFTP_PROTOCOL &&
 -        libssh2_sftp_last_error(s->sftp) == LIBSSH2_FX_OP_UNSUPPORTED) {
 -        unsafe_flush_warning(s, "OpenSSH >= 6.3");
 -        return 0;
 -    }
      if (r < 0) {
          sftp_error_trace(s, "fsync");
          return -EIO;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
      return ret;
  }
 -#else /* !HAS_LIBSSH2_SFTP_FSYNC */
 +#else /* !HAVE_LIBSSH_0_8 */
  static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
  {
      BDRVSSHState *s = bs->opaque;
 -    unsafe_flush_warning(s, "libssh2 >= 1.4.4");
 +    unsafe_flush_warning(s, "libssh >= 0.8.0");
      return 0;
  }
 -#endif /* !HAS_LIBSSH2_SFTP_FSYNC */
 +#endif /* !HAVE_LIBSSH_0_8 */
  static int64_t ssh_getlength(BlockDriverState *bs)
  {
      BDRVSSHState *s = bs->opaque;
      int64_t length;
 -    /* Note we cannot make a libssh2 call here. */
 -    length = (int64_t) s->attrs.filesize;
 +    /* Note we cannot make a libssh call here. */
 +    length = (int64_t) s->attrs->size;
      trace_ssh_getlength(length);
      return length;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_truncate(BlockDriverState *bs, int64_t offset,
          return -ENOTSUP;
      }
 -    if (offset < s->attrs.filesize) {
 +    if (offset < s->attrs->size) {
          error_setg(errp, "ssh driver does not support shrinking files");
          return -ENOTSUP;
      }
 -    if (offset == s->attrs.filesize) {
 +    if (offset == s->attrs->size) {
          return 0;
      }
@@ -XXX,XX +XXX,XX @@ static void bdrv_ssh_init(void)
  {
      int r;
 -    r = libssh2_init(0);
 +    r = ssh_init();
      if (r != 0) {
 -        fprintf(stderr, "libssh2 initialization failed, %d\n", r);
 +        fprintf(stderr, "libssh initialization failed, %d\n", r);
          exit(EXIT_FAILURE);
      }
 +#if TRACE_LIBSSH != 0
 +    ssh_set_log_level(TRACE_LIBSSH);
 +#endif
 +
      bdrv_register(&bdrv_ssh);
  }
 diff --git a/.travis.yml b/.travis.yml
 index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/257.out
+--- a/.travis.yml
-+++ b/tests/qemu-iotests/257.out
++++ b/.travis.yml
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+@@ -XXX,XX +XXX,XX @@ addons:
- {"return": ""}
+       - libseccomp-dev
- {
+       - libspice-protocol-dev
-   "bitmaps": {
+       - libspice-server-dev
-+    "backup-top": [
+-      - libssh2-1-dev
-+      {
++      - libssh-dev
-+        "busy": false,
+       - liburcu-dev
-+        "count": 67108864,
+       - libusb-1.0-0-dev
-+        "granularity": 65536,
+       - libvte-2.91-dev
-+        "persistent": false,
+@@ -XXX,XX +XXX,XX @@ matrix:
-+        "recording": false
+             - libseccomp-dev
-+      },
+             - libspice-protocol-dev
-+      {
+             - libspice-server-dev
-+        "busy": false,
+-            - libssh2-1-dev
-+        "count": 458752,
++            - libssh-dev
-+        "granularity": 65536,
+             - liburcu-dev
-+        "persistent": false,
+             - libusb-1.0-0-dev
-+        "recording": false
+             - libvte-2.91-dev
-+      }
+diff --git a/block/trace-events b/block/trace-events
-+    ],
+index XXXXXXX..XXXXXXX 100644
-     "drive0": [
+--- a/block/trace-events
-       {
++++ b/block/trace-events
-         "busy": false,
+@@ -XXX,XX +XXX,XX @@ nbd_client_connect_success(const char *export_name) "export '%s'"
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+ # ssh.c
- {"return": ""}
+ ssh_restart_coroutine(void *co) "co=%p"
- {
+ ssh_flush(void) "fsync"
-   "bitmaps": {
+-ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
-+    "backup-top": [
++ssh_check_host_key_knownhosts(void) "host key OK"
-+      {
+ ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
-+        "busy": false,
+ ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
-+        "count": 67108864,
+ ssh_co_yield_back(int sock) "s->sock=%d - back"
-+        "granularity": 65536,
+ ssh_getlength(int64_t length) "length=%" PRIi64
-+        "persistent": false,
+ ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
-+        "recording": false
+ ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-+      },
+-ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
-+      {
+-ssh_read_return(ssize_t ret) "sftp_read returned %zd"
-+        "busy": false,
++ssh_read_buf(void *buf, size_t size, size_t actual_size) "sftp_read buf=%p size=%zu (actual size=%zu)"
-+        "count": 458752,
++ssh_read_return(ssize_t ret, int sftp_err) "sftp_read returned %zd (sftp error=%d)"
-+        "granularity": 65536,
+ ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-+        "persistent": false,
+-ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
-+        "recording": false
+-ssh_write_return(ssize_t ret) "sftp_write returned %zd"
-+      }
++ssh_write_buf(void *buf, size_t size, size_t actual_size) "sftp_write buf=%p size=%zu (actual size=%zu)"
-+    ],
++ssh_write_return(ssize_t ret, int sftp_err) "sftp_write returned %zd (sftp error=%d)"
-     "drive0": [
+ ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
-       {
++ssh_auth_methods(int methods) "auth methods=0x%x"
-         "busy": false,
++ssh_server_status(int status) "server status=%d"
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
- {"return": ""}
+ # curl.c
- {
+ curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
-   "bitmaps": {
+@@ -XXX,XX +XXX,XX @@ sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
-+    "backup-top": [
+ sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
-+      {
-+        "busy": false,
+ # ssh.c
-+        "count": 67108864,
+-sftp_error(const char *op, const char *ssh_err, int ssh_err_code, unsigned long sftp_err_code) "%s failed: %s (libssh2 error code: %d, sftp error code: %lu)"
-+        "granularity": 65536,
++sftp_error(const char *op, const char *ssh_err, int ssh_err_code, int sftp_err_code) "%s failed: %s (libssh error code: %d, sftp error code: %d)"
-+        "persistent": false,
+diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi
-+        "recording": false
+index XXXXXXX..XXXXXXX 100644
-+      },
+--- a/docs/qemu-block-drivers.texi
-+      {
++++ b/docs/qemu-block-drivers.texi
-+        "busy": false,
+@@ -XXX,XX +XXX,XX @@ print a warning when @code{fsync} is not supported:
-+        "count": 458752,
-+        "granularity": 65536,
+ warning: ssh server @code{ssh.example.com:22} does not support fsync
-+        "persistent": false,
-+        "recording": false
+-With sufficiently new versions of libssh2 and OpenSSH, @code{fsync} is
-+      }
++With sufficiently new versions of libssh and OpenSSH, @code{fsync} is
-+    ],
+ supported.
-     "drive0": [
-       {
+ @node disk_images_nvme
-         "busy": false,
+diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+index XXXXXXX..XXXXXXX 100644
- {"return": ""}
+--- a/tests/docker/dockerfiles/debian-win32-cross.docker
- {
++++ b/tests/docker/dockerfiles/debian-win32-cross.docker
-   "bitmaps": {
+@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
-+    "backup-top": [
+         mxe-$TARGET-w64-mingw32.shared-curl \
-+      {
+         mxe-$TARGET-w64-mingw32.shared-glib \
-+        "busy": false,
+         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-+        "count": 67108864,
+-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
-+        "granularity": 65536,
+         mxe-$TARGET-w64-mingw32.shared-libusb1 \
-+        "persistent": false,
+         mxe-$TARGET-w64-mingw32.shared-lzo \
-+        "recording": false
+         mxe-$TARGET-w64-mingw32.shared-nettle \
-+      },
+diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
-+      {
+index XXXXXXX..XXXXXXX 100644
-+        "busy": false,
+--- a/tests/docker/dockerfiles/debian-win64-cross.docker
-+        "count": 458752,
++++ b/tests/docker/dockerfiles/debian-win64-cross.docker
-+        "granularity": 65536,
+@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
-+        "persistent": false,
+         mxe-$TARGET-w64-mingw32.shared-curl \
-+        "recording": false
+         mxe-$TARGET-w64-mingw32.shared-glib \
-+      }
+         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-+    ],
+-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
-     "drive0": [
+         mxe-$TARGET-w64-mingw32.shared-libusb1 \
-       {
+         mxe-$TARGET-w64-mingw32.shared-lzo \
-         "busy": false,
+         mxe-$TARGET-w64-mingw32.shared-nettle \
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
- {"return": ""}
+index XXXXXXX..XXXXXXX 100644
- {
+--- a/tests/docker/dockerfiles/fedora.docker
-   "bitmaps": {
++++ b/tests/docker/dockerfiles/fedora.docker
-+    "backup-top": [
+@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
-+      {
+     libpng-devel \
-+        "busy": false,
+     librbd-devel \
-+        "count": 67108864,
+     libseccomp-devel \
-+        "granularity": 65536,
+-    libssh2-devel \
-+        "persistent": false,
++    libssh-devel \
-+        "recording": false
+     libubsan \
-+      },
+     libusbx-devel \
-+      {
+     libxml2-devel \
-+        "busy": false,
+@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
-+        "count": 458752,
+     mingw32-gtk3 \
-+        "granularity": 65536,
+     mingw32-libjpeg-turbo \
-+        "persistent": false,
+     mingw32-libpng \
-+        "recording": false
+-    mingw32-libssh2 \
-+      }
+     mingw32-libtasn1 \
-+    ],
+     mingw32-nettle \
-     "drive0": [
+     mingw32-pixman \
-       {
+@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
-         "busy": false,
+     mingw64-gtk3 \
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+     mingw64-libjpeg-turbo \
- {"return": ""}
+     mingw64-libpng \
- {
+-    mingw64-libssh2 \
-   "bitmaps": {
+     mingw64-libtasn1 \
-+    "backup-top": [
+     mingw64-nettle \
-+      {
+     mingw64-pixman \
-+        "busy": false,
+diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
-+        "count": 67108864,
+index XXXXXXX..XXXXXXX 100644
-+        "granularity": 65536,
+--- a/tests/docker/dockerfiles/ubuntu.docker
-+        "persistent": false,
++++ b/tests/docker/dockerfiles/ubuntu.docker
-+        "recording": false
+@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
-+      },
+     libsnappy-dev \
-+      {
+     libspice-protocol-dev \
-+        "busy": false,
+     libspice-server-dev \
-+        "count": 458752,
+-    libssh2-1-dev \
-+        "granularity": 65536,
++    libssh-dev \
-+        "persistent": false,
+     libusb-1.0-0-dev \
-+        "recording": false
+     libusbredirhost-dev \
-+      }
+     libvdeplug-dev \
-+    ],
+diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
-     "drive0": [
+index XXXXXXX..XXXXXXX 100644
-       {
+--- a/tests/docker/dockerfiles/ubuntu1804.docker
-         "busy": false,
++++ b/tests/docker/dockerfiles/ubuntu1804.docker
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
- {"return": ""}
+     libsnappy-dev \
- {
+     libspice-protocol-dev \
-   "bitmaps": {
+     libspice-server-dev \
-+    "backup-top": [
+-    libssh2-1-dev \
-+      {
++    libssh-dev \
-+        "busy": false,
+     libusb-1.0-0-dev \
-+        "count": 67108864,
+     libusbredirhost-dev \
-+        "granularity": 65536,
+     libvdeplug-dev \
-+        "persistent": false,
+diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
-+        "recording": false
+index XXXXXXX..XXXXXXX 100755
-+      },
+--- a/tests/qemu-iotests/207
-+      {
++++ b/tests/qemu-iotests/207
-+        "busy": false,
+@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
-+        "count": 458752,
-+        "granularity": 65536,
+     iotests.img_info_log(remote_path)
-+        "persistent": false,
-+        "recording": false
+-    md5_key = subprocess.check_output(
-+      }
+-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-+    ],
+-        'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-     "drive0": [
+-        shell=True).rstrip().decode('ascii')
-       {
++    keys = subprocess.check_output(
-         "busy": false,
++        'ssh-keyscan 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
++        'cut -d" " -f3',
- {"return": ""}
++        shell=True).rstrip().decode('ascii').split('\n')
- {
++
-   "bitmaps": {
++    # Mappings of base64 representations to digests
-+    "backup-top": [
++    md5_keys = {}
-+      {
++    sha1_keys = {}
-+        "busy": false,
++
-+        "count": 67108864,
++    for key in keys:
-+        "granularity": 65536,
++        md5_keys[key] = subprocess.check_output(
-+        "persistent": false,
++            'echo %s | base64 -d | md5sum -b | cut -d" " -f1' % key,
-+        "recording": false
++            shell=True).rstrip().decode('ascii')
-+      },
++
-+      {
++        sha1_keys[key] = subprocess.check_output(
-+        "busy": false,
++            'echo %s | base64 -d | sha1sum -b | cut -d" " -f1' % key,
-+        "count": 458752,
++            shell=True).rstrip().decode('ascii')
-+        "granularity": 65536,
-+        "persistent": false,
+     vm.launch()
-+        "recording": false
++
-+      }
++    # Find correct key first
-+    ],
++    matching_key = None
-     "drive0": [
++    for key in keys:
-       {
++        result = vm.qmp('blockdev-add',
-         "busy": false,
++                        driver='ssh', node_name='node0', path=disk_path,
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
++                        server={
- {"return": ""}
++                             'host': '127.0.0.1',
- {
++                             'port': '22',
-   "bitmaps": {
++                        }, host_key_check={
-+    "backup-top": [
++                             'mode': 'hash',
-+      {
++                             'type': 'md5',
-+        "busy": false,
++                             'hash': md5_keys[key],
-+        "count": 67108864,
++                        })
-+        "granularity": 65536,
++
-+        "persistent": false,
++        if 'error' not in result:
-+        "recording": false
++            vm.qmp('blockdev-del', node_name='node0')
-+      },
++            matching_key = key
-+      {
++            break
-+        "busy": false,
++
-+        "count": 458752,
++    if matching_key is None:
-+        "granularity": 65536,
++        vm.shutdown()
-+        "persistent": false,
++        iotests.notrun('Did not find a key that fits 127.0.0.1')
-+        "recording": false
++
-+      }
+     blockdev_create(vm, { 'driver': 'ssh',
-+    ],
+                           'location': {
-     "drive0": [
+                               'path': disk_path,
-       {
+@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
-         "busy": false,
+                               'host-key-check': {
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+                                   'mode': 'hash',
- {"return": ""}
+                                   'type': 'md5',
- {
+-                                  'hash': md5_key,
-   "bitmaps": {
++                                  'hash': md5_keys[matching_key],
-+    "backup-top": [
+                               }
-+      {
+                           },
-+        "busy": false,
+                           'size': 8388608 })
-+        "count": 67108864,
+@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
-+        "granularity": 65536,
-+        "persistent": false,
+     iotests.img_info_log(remote_path)
-+        "recording": false
-+      },
+-    sha1_key = subprocess.check_output(
-+      {
+-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-+        "busy": false,
+-        'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-+        "count": 458752,
+-        shell=True).rstrip().decode('ascii')
-+        "granularity": 65536,
+-
-+        "persistent": false,
+     vm.launch()
-+        "recording": false
+     blockdev_create(vm, { 'driver': 'ssh',
-+      }
+                           'location': {
-+    ],
+@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
-     "drive0": [
+                               'host-key-check': {
-       {
+                                   'mode': 'hash',
-         "busy": false,
+                                   'type': 'sha1',
-@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
+-                                  'hash': sha1_key,
- {"return": ""}
++                                  'hash': sha1_keys[matching_key],
- {
+                               }
-   "bitmaps": {
+                           },
-+    "backup-top": [
+                           'size': 4194304 })
-+      {
+diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
-+        "busy": false,
+index XXXXXXX..XXXXXXX 100644
-+        "count": 67108864,
+--- a/tests/qemu-iotests/207.out
-+        "granularity": 65536,
++++ b/tests/qemu-iotests/207.out
-+        "persistent": false,
+@@ -XXX,XX +XXX,XX @@ virtual size: 4 MiB (4194304 bytes)
-+        "recording": false
-+      },
+ {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
-+      {
+ {"return": {}}
-+        "busy": false,
+-Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
-+        "count": 458752,
++Job failed: failed to open remote file '/this/is/not/an/existing/path': SFTP server: No such file (libssh error code: 1, sftp error code: 2)
-+        "granularity": 65536,
+ {"execute": "job-dismiss", "arguments": {"id": "job0"}}
-+        "persistent": false,
+ {"return": {}}
-+        "recording": false
 +      }
 +    ],
      "drive0": [
        {
          "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
  {"return": ""}
  {
    "bitmaps": {
 +    "backup-top": [
 +      {
 +        "busy": false,
 +        "count": 67108864,
 +        "granularity": 65536,
 +        "persistent": false,
 +        "recording": false
 +      },
 +      {
 +        "busy": false,
 +        "count": 458752,
 +        "granularity": 65536,
 +        "persistent": false,
 +        "recording": false
 +      }
 +    ],
      "drive0": [
        {
          "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
  {"return": ""}
  {
    "bitmaps": {
 +    "backup-top": [
 +      {
 +        "busy": false,
 +        "count": 67108864,
 +        "granularity": 65536,
 +        "persistent": false,
 +        "recording": false
 +      },
 +      {
 +        "busy": false,
 +        "count": 458752,
 +        "granularity": 65536,
 +        "persistent": false,
 +        "recording": false
 +      }
 +    ],
      "drive0": [
        {
          "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
  {"return": ""}
  {
    "bitmaps": {
 +    "backup-top": [
 +      {
 +        "busy": false,
 +        "count": 67108864,
 +        "granularity": 65536,
 +        "persistent": false,
 +        "recording": false
 +      },
 +      {
 +        "busy": false,
 +        "count": 458752,
 +        "granularity": 65536,
 +        "persistent": false,
 +        "recording": false
 +      }
 +    ],
      "drive0": [
        {
          "busy": false,
 --
-.34.1
+.21.0

-[PULL 02/23] block/curl.c: Set error message string if curl_init_state() fails
+[Qemu-devel] [PULL v2 8/8] iotests: Fix 205 for concurrent runs
-From: Peter Maydell <peter.maydell@linaro.org>
+Tests should place their files into the test directory.  This includes
 Unix sockets.  205 currently fails to do so, which prevents it from
 being run concurrently.
-In curl_open(), the 'out' label assumes that the state->errmsg string
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-has been set (either by curl_easy_perform() or by manually copying a
+Message-id: 20190618210238.9524-1-mreitz@redhat.com
-string into it); however if curl_init_state() fails we will jump to
+Reviewed-by: Eric Blake <eblake@redhat.com>
-that label without setting the string.  Add the missing error string
+Signed-off-by: Max Reitz <mreitz@redhat.com>
-setup.
+---
  tests/qemu-iotests/205 | 2 +-
 file changed, 1 insertion(+), 1 deletion(-)
-(We can't be specific about the cause of failure: the documentation
+diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
-of curl_easy_init() just says "If this function returns NULL,
+index XXXXXXX..XXXXXXX 100755
-something went wrong".)
+--- a/tests/qemu-iotests/205
++++ b/tests/qemu-iotests/205
-Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
+@@ -XXX,XX +XXX,XX @@ import iotests
-Message-Id: <20220222152341.850419-2-peter.maydell@linaro.org>
+ import time
-Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
+ from iotests import qemu_img_create, qemu_io, filter_qemu_io, QemuIoInteractive
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
+-nbd_sock = 'nbd_sock'
----
++nbd_sock = os.path.join(iotests.test_dir, 'nbd_sock')
- block/curl.c | 2 ++
+ nbd_uri = 'nbd+unix:///exp?socket=' + nbd_sock
-file changed, 2 insertions(+)
+ disk = os.path.join(iotests.test_dir, 'disk')
 diff --git a/block/curl.c b/block/curl.c
 index XXXXXXX..XXXXXXX 100644
 --- a/block/curl.c
 +++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
      // Get file size
      if (curl_init_state(s, state) < 0) {
 +        pstrcpy(state->errmsg, CURL_ERROR_SIZE,
 +                "curl library initialization failed.");
          goto out;
      }
 --
-.34.1
+.21.0

-[PULL 03/23] block/curl.c: Check error return from curl_easy_setopt()
+Deleted patch
-From: Peter Maydell <peter.maydell@linaro.org>
-Coverity points out that we aren't checking the return value
-from curl_easy_setopt() for any of the calls to it we make
-in block/curl.c.
-Some of these options are documented as always succeeding (e.g.
-CURLOPT_VERBOSE) but others have documented failure cases (e.g.
-CURLOPT_URL).  For consistency we check every call, even the ones
-that theoretically cannot fail.
-Fixes: Coverity CID 1459336, 1459482, 1460331
-Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
-Message-Id: <20220222152341.850419-3-peter.maydell@linaro.org>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- block/curl.c | 90 +++++++++++++++++++++++++++++++++-------------------
-file changed, 57 insertions(+), 33 deletions(-)
-diff --git a/block/curl.c b/block/curl.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/curl.c
-+++ b/block/curl.c
-@@ -XXX,XX +XXX,XX @@ static int curl_init_state(BDRVCURLState *s, CURLState *state)
-         if (!state->curl) {
-             return -EIO;
-         }
--        curl_easy_setopt(state->curl, CURLOPT_URL, s->url);
--        curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYPEER,
--                         (long) s->sslverify);
--        curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYHOST,
--                         s->sslverify ? 2L : 0L);
-+        if (curl_easy_setopt(state->curl, CURLOPT_URL, s->url) ||
-+            curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYPEER,
-+                             (long) s->sslverify) ||
-+            curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYHOST,
-+                             s->sslverify ? 2L : 0L)) {
-+            goto err;
-+        }
-         if (s->cookie) {
--            curl_easy_setopt(state->curl, CURLOPT_COOKIE, s->cookie);
-+            if (curl_easy_setopt(state->curl, CURLOPT_COOKIE, s->cookie)) {
-+                goto err;
-+            }
-+        }
-+        if (curl_easy_setopt(state->curl, CURLOPT_TIMEOUT, (long)s->timeout) ||
-+            curl_easy_setopt(state->curl, CURLOPT_WRITEFUNCTION,
-+                             (void *)curl_read_cb) ||
-+            curl_easy_setopt(state->curl, CURLOPT_WRITEDATA, (void *)state) ||
-+            curl_easy_setopt(state->curl, CURLOPT_PRIVATE, (void *)state) ||
-+            curl_easy_setopt(state->curl, CURLOPT_AUTOREFERER, 1) ||
-+            curl_easy_setopt(state->curl, CURLOPT_FOLLOWLOCATION, 1) ||
-+            curl_easy_setopt(state->curl, CURLOPT_NOSIGNAL, 1) ||
-+            curl_easy_setopt(state->curl, CURLOPT_ERRORBUFFER, state->errmsg) ||
-+            curl_easy_setopt(state->curl, CURLOPT_FAILONERROR, 1)) {
-+            goto err;
-         }
--        curl_easy_setopt(state->curl, CURLOPT_TIMEOUT, (long)s->timeout);
--        curl_easy_setopt(state->curl, CURLOPT_WRITEFUNCTION,
--                         (void *)curl_read_cb);
--        curl_easy_setopt(state->curl, CURLOPT_WRITEDATA, (void *)state);
--        curl_easy_setopt(state->curl, CURLOPT_PRIVATE, (void *)state);
--        curl_easy_setopt(state->curl, CURLOPT_AUTOREFERER, 1);
--        curl_easy_setopt(state->curl, CURLOPT_FOLLOWLOCATION, 1);
--        curl_easy_setopt(state->curl, CURLOPT_NOSIGNAL, 1);
--        curl_easy_setopt(state->curl, CURLOPT_ERRORBUFFER, state->errmsg);
--        curl_easy_setopt(state->curl, CURLOPT_FAILONERROR, 1);
--
-         if (s->username) {
--            curl_easy_setopt(state->curl, CURLOPT_USERNAME, s->username);
-+            if (curl_easy_setopt(state->curl, CURLOPT_USERNAME, s->username)) {
-+                goto err;
-+            }
-         }
-         if (s->password) {
--            curl_easy_setopt(state->curl, CURLOPT_PASSWORD, s->password);
-+            if (curl_easy_setopt(state->curl, CURLOPT_PASSWORD, s->password)) {
-+                goto err;
-+            }
-         }
-         if (s->proxyusername) {
--            curl_easy_setopt(state->curl,
--                             CURLOPT_PROXYUSERNAME, s->proxyusername);
-+            if (curl_easy_setopt(state->curl,
-+                                 CURLOPT_PROXYUSERNAME, s->proxyusername)) {
-+                goto err;
-+            }
-         }
-         if (s->proxypassword) {
--            curl_easy_setopt(state->curl,
--                             CURLOPT_PROXYPASSWORD, s->proxypassword);
-+            if (curl_easy_setopt(state->curl,
-+                                 CURLOPT_PROXYPASSWORD, s->proxypassword)) {
-+                goto err;
-+            }
-         }
-         /* Restrict supported protocols to avoid security issues in the more
-@@ -XXX,XX +XXX,XX @@ static int curl_init_state(BDRVCURLState *s, CURLState *state)
-          * Restricting protocols is only supported from 7.19.4 upwards.
-          */
- #if LIBCURL_VERSION_NUM >= 0x071304
--        curl_easy_setopt(state->curl, CURLOPT_PROTOCOLS, PROTOCOLS);
--        curl_easy_setopt(state->curl, CURLOPT_REDIR_PROTOCOLS, PROTOCOLS);
-+        if (curl_easy_setopt(state->curl, CURLOPT_PROTOCOLS, PROTOCOLS) ||
-+            curl_easy_setopt(state->curl, CURLOPT_REDIR_PROTOCOLS, PROTOCOLS)) {
-+            goto err;
-+        }
- #endif
- #ifdef DEBUG_VERBOSE
--        curl_easy_setopt(state->curl, CURLOPT_VERBOSE, 1);
-+        if (curl_easy_setopt(state->curl, CURLOPT_VERBOSE, 1)) {
-+            goto err;
-+        }
- #endif
-     }
-     state->s = s;
-     return 0;
-+
-+err:
-+    curl_easy_cleanup(state->curl);
-+    state->curl = NULL;
-+    return -EIO;
- }
- /* Called with s->mutex held.  */
-@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
-     }
-     s->accept_range = false;
--    curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1);
--    curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION,
--                     curl_header_cb);
--    curl_easy_setopt(state->curl, CURLOPT_HEADERDATA, s);
-+    if (curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1) ||
-+        curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION, curl_header_cb) ||
-+        curl_easy_setopt(state->curl, CURLOPT_HEADERDATA, s)) {
-+        pstrcpy(state->errmsg, CURL_ERROR_SIZE,
-+                "curl library initialization failed.");
-+        goto out;
-+    }
-     if (curl_easy_perform(state->curl))
-         goto out;
-     if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &d)) {
-@@ -XXX,XX +XXX,XX @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
-     snprintf(state->range, 127, "%" PRIu64 "-%" PRIu64, start, end);
-     trace_curl_setup_preadv(acb->bytes, start, state->range);
--    curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range);
--
--    if (curl_multi_add_handle(s->multi, state->curl) != CURLM_OK) {
-+    if (curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range) ||
-+        curl_multi_add_handle(s->multi, state->curl) != CURLM_OK) {
-         state->acb[0] = NULL;
-         acb->ret = -EIO;
---
-.34.1

-[PULL 04/23] block: fix preallocate filter: don't do unaligned preallocate requests
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-There is a bug in handling BDRV_REQ_NO_WAIT flag: we still may wait in
-wait_serialising_requests() if request is unaligned. And this is
-possible for the only user of this flag (preallocate filter) if
-underlying file is unaligned to its request_alignment on start.
-So, we have to fix preallocate filter to do only aligned preallocate
-requests.
-Next, we should fix generic block/io.c somehow. Keeping in mind that
-preallocate is the only user of BDRV_REQ_NO_WAIT and that we have to
-fix its behavior now, it seems more safe to just assert that we never
-use BDRV_REQ_NO_WAIT with unaligned requests and add corresponding
-comment. Let's do so.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Denis V. Lunev <den@openvz.org>
-Message-Id: <20220215121609.38570-1-vsementsov@virtuozzo.com>
-[hreitz: Rebased on block GS/IO split]
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- include/block/block-common.h |  3 ++-
- block/io.c                   |  4 ++++
- block/preallocate.c          | 15 ++++++++++++---
-files changed, 18 insertions(+), 4 deletions(-)
-diff --git a/include/block/block-common.h b/include/block/block-common.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block-common.h
-+++ b/include/block/block-common.h
-@@ -XXX,XX +XXX,XX @@ typedef enum {
-     /*
-      * If we need to wait for other requests, just fail immediately. Used
--     * only together with BDRV_REQ_SERIALISING.
-+     * only together with BDRV_REQ_SERIALISING. Used only with requests aligned
-+     * to request_alignment (corresponding assertions are in block/io.c).
-      */
-     BDRV_REQ_NO_WAIT = 0x400,
-diff --git a/block/io.c b/block/io.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/io.c
-+++ b/block/io.c
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_do_zero_pwritev(BdrvChild *child,
-     padding = bdrv_init_padding(bs, offset, bytes, &pad);
-     if (padding) {
-+        assert(!(flags & BDRV_REQ_NO_WAIT));
-         bdrv_make_request_serialising(req, align);
-         bdrv_padding_rmw_read(child, req, &pad, true);
-@@ -XXX,XX +XXX,XX @@ int coroutine_fn bdrv_co_pwritev_part(BdrvChild *child,
-          * serialize the request to prevent interactions of the
-          * widened region with other transactions.
-          */
-+        assert(!(flags & BDRV_REQ_NO_WAIT));
-         bdrv_make_request_serialising(&req, align);
-         bdrv_padding_rmw_read(child, &req, &pad, false);
-     }
-@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_copy_range_internal(
-     /* TODO We can support BDRV_REQ_NO_FALLBACK here */
-     assert(!(read_flags & BDRV_REQ_NO_FALLBACK));
-     assert(!(write_flags & BDRV_REQ_NO_FALLBACK));
-+    assert(!(read_flags & BDRV_REQ_NO_WAIT));
-+    assert(!(write_flags & BDRV_REQ_NO_WAIT));
-     if (!dst || !dst->bs || !bdrv_is_inserted(dst->bs)) {
-         return -ENOMEDIUM;
-diff --git a/block/preallocate.c b/block/preallocate.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/preallocate.c
-+++ b/block/preallocate.c
-@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn handle_write(BlockDriverState *bs, int64_t offset,
-     int64_t end = offset + bytes;
-     int64_t prealloc_start, prealloc_end;
-     int ret;
-+    uint32_t file_align = bs->file->bs->bl.request_alignment;
-+    uint32_t prealloc_align = MAX(s->opts.prealloc_align, file_align);
-+
-+    assert(QEMU_IS_ALIGNED(prealloc_align, file_align));
-     if (!has_prealloc_perms(bs)) {
-         /* We don't have state neither should try to recover it */
-@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn handle_write(BlockDriverState *bs, int64_t offset,
-     /* Now we want new preallocation, as request writes beyond s->file_end. */
--    prealloc_start = want_merge_zero ? MIN(offset, s->file_end) : s->file_end;
--    prealloc_end = QEMU_ALIGN_UP(end + s->opts.prealloc_size,
--                                 s->opts.prealloc_align);
-+    prealloc_start = QEMU_ALIGN_UP(
-+            want_merge_zero ? MIN(offset, s->file_end) : s->file_end,
-+            file_align);
-+    prealloc_end = QEMU_ALIGN_UP(
-+            MAX(prealloc_start, end) + s->opts.prealloc_size,
-+            prealloc_align);
-+
-+    want_merge_zero = want_merge_zero && (prealloc_start <= offset);
-     ret = bdrv_co_pwrite_zeroes(
-             bs->file, prealloc_start, prealloc_end - prealloc_start,
---
-.34.1

-[PULL 05/23] tests/qemu-iotests/040: Skip TestCommitWithFilters without 'throttle'
+Deleted patch
-From: Thomas Huth <thuth@redhat.com>
-iotest 040 already has some checks for the availability of the 'throttle'
-driver, but some new code has been added in the course of time that
-depends on 'throttle' but does not check for its availability. Add
-a check to the TestCommitWithFilters class so that this iotest now
-also passes again if 'throttle' has not been enabled in the QEMU
-binaries.
-Signed-off-by: Thomas Huth <thuth@redhat.com>
-Message-Id: <20220223123127.3206042-1-thuth@redhat.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- tests/qemu-iotests/040 | 1 +
-file changed, 1 insertion(+)
-diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/040
-+++ b/tests/qemu-iotests/040
-@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
-                              pattern_file)
-             self.assertFalse('Pattern verification failed' in result)
-+    @iotests.skip_if_unsupported(['throttle'])
-     def setUp(self):
-         qemu_img('create', '-f', iotests.imgfmt, self.img0, '64M')
-         qemu_img('create', '-f', iotests.imgfmt, self.img1, '64M')
---
-.34.1

-[PULL 06/23] tests/qemu-iotests/testrunner: Quote "case not run" lines in TAP mode
+Deleted patch
-From: Thomas Huth <thuth@redhat.com>
-In TAP mode, the stdout is reserved for the TAP protocol, so we
-have to make sure to mark other lines with a comment '#' character
-at the beginning to avoid that the TAP parser at the other end
-gets confused.
-To test this condition, run "configure" for example with:
- --block-drv-rw-whitelist=copy-before-write,qcow2,raw,file,host_device,blkdebug,null-co,copy-on-read
-so that iotest 041 will report that some tests are not run due to
-the missing "quorum" driver. Without this change, "make check-block"
-fails since the meson tap parser gets confused by these messages.
-Signed-off-by: Thomas Huth <thuth@redhat.com>
-Message-Id: <20220223124353.3273898-1-thuth@redhat.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- tests/qemu-iotests/testrunner.py | 5 ++++-
-file changed, 4 insertions(+), 1 deletion(-)
-diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testrunner.py
-+++ b/tests/qemu-iotests/testrunner.py
-@@ -XXX,XX +XXX,XX @@ def run_test(self, test: str,
-                                  description=res.description)
-         if res.casenotrun:
--            print(res.casenotrun)
-+            if self.tap:
-+                print('#' + res.casenotrun.replace('\n', '\n#'))
-+            else:
-+                print(res.casenotrun)
-         return res
---
-.34.1

-[PULL 07/23] iotests: Write test output to TEST_DIR
+Deleted patch
-Drop the use of OUTPUT_DIR (test/qemu-iotests under the build
-directory), and instead write test output files (.out.bad, .notrun, and
-.casenotrun) to TEST_DIR.
-With this, the same test can be run concurrently without the separate
-instances interfering, because they will need separate TEST_DIRs anyway.
-Running the same test separately is useful when running the iotests with
-various format/protocol combinations in parallel, or when you just want
-to aggressively exercise a single test (e.g. when it fails only
-sporadically).
-Putting this output into TEST_DIR means that it will stick around for
-inspection after the test run is done (though running the same test in
-the same TEST_DIR will overwrite it, just as it used to be); but given
-that TEST_DIR is a scratch directory, it should be clear that users can
-delete all of its content at any point.  (And if TEST_DIR is on tmpfs,
-it will just disappear on shutdown.)  Contrarily, alternative approaches
-that would put these output files into OUTPUT_DIR with some prefix to
-differentiate between separate test runs might easily lead to cluttering
-OUTPUT_DIR.
-(This change means OUTPUT_DIR is no longer written to by the iotests, so
-we can drop its usage altogether.)
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220221172909.762858-1-hreitz@redhat.com>
-[hreitz: Simplified `Path(os.path.join(x, y))` to `Path(x, y)`, as
-         suggested by Vladimir; and rebased on 9086c7639822b6
-         ("tests/qemu-iotests: Rework the checks and spots using GNU
-         sed")]
-Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
----
- tests/qemu-iotests/common.rc     |  6 +++---
- tests/qemu-iotests/iotests.py    |  5 ++---
- tests/qemu-iotests/testenv.py    |  5 +----
- tests/qemu-iotests/testrunner.py | 14 ++++++++------
-files changed, 14 insertions(+), 16 deletions(-)
-diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/common.rc
-+++ b/tests/qemu-iotests/common.rc
-@@ -XXX,XX +XXX,XX @@
- # bail out, setting up .notrun file
- _notrun()
- {
--    echo "$*" >"$OUTPUT_DIR/$seq.notrun"
-+    echo "$*" >"$TEST_DIR/$seq.notrun"
-     echo "$seq not run: $*"
-     status=0
-     exit
-@@ -XXX,XX +XXX,XX @@ _img_info()
- #
- _casenotrun()
- {
--    echo "    [case not run] $*" >>"$OUTPUT_DIR/$seq.casenotrun"
-+    echo "    [case not run] $*" >>"$TEST_DIR/$seq.casenotrun"
- }
- # just plain bail out
- #
- _fail()
- {
--    echo "$*" | tee -a "$OUTPUT_DIR/$seq.full"
-+    echo "$*" | tee -a "$TEST_DIR/$seq.full"
-     echo "(see $seq.full for details)"
-     status=1
-     exit 1
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@
- imgfmt = os.environ.get('IMGFMT', 'raw')
- imgproto = os.environ.get('IMGPROTO', 'file')
--output_dir = os.environ.get('OUTPUT_DIR', '.')
- try:
-     test_dir = os.environ['TEST_DIR']
-@@ -XXX,XX +XXX,XX @@ def notrun(reason):
-     # Each test in qemu-iotests has a number ("seq")
-     seq = os.path.basename(sys.argv[0])
--    with open('%s/%s.notrun' % (output_dir, seq), 'w', encoding='utf-8') \
-+    with open('%s/%s.notrun' % (test_dir, seq), 'w', encoding='utf-8') \
-             as outfile:
-         outfile.write(reason + '\n')
-     logger.warning("%s not run: %s", seq, reason)
-@@ -XXX,XX +XXX,XX @@ def case_notrun(reason):
-     # Each test in qemu-iotests has a number ("seq")
-     seq = os.path.basename(sys.argv[0])
--    with open('%s/%s.casenotrun' % (output_dir, seq), 'a', encoding='utf-8') \
-+    with open('%s/%s.casenotrun' % (test_dir, seq), 'a', encoding='utf-8') \
-             as outfile:
-         outfile.write('    [case not run] ' + reason + '\n')
-diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testenv.py
-+++ b/tests/qemu-iotests/testenv.py
-@@ -XXX,XX +XXX,XX @@ class TestEnv(ContextManager['TestEnv']):
-     # pylint: disable=too-many-instance-attributes
-     env_variables = ['PYTHONPATH', 'TEST_DIR', 'SOCK_DIR', 'SAMPLE_IMG_DIR',
--                     'OUTPUT_DIR', 'PYTHON', 'QEMU_PROG', 'QEMU_IMG_PROG',
-+                     'PYTHON', 'QEMU_PROG', 'QEMU_IMG_PROG',
-                      'QEMU_IO_PROG', 'QEMU_NBD_PROG', 'QSD_PROG',
-                      'QEMU_OPTIONS', 'QEMU_IMG_OPTIONS',
-                      'QEMU_IO_OPTIONS', 'QEMU_IO_OPTIONS_NO_FMT',
-@@ -XXX,XX +XXX,XX @@ def init_directories(self) -> None:
-              TEST_DIR
-              SOCK_DIR
-              SAMPLE_IMG_DIR
--             OUTPUT_DIR
-         """
-         # Path where qemu goodies live in this source tree.
-@@ -XXX,XX +XXX,XX @@ def init_directories(self) -> None:
-                                         os.path.join(self.source_iotests,
-                                                      'sample_images'))
--        self.output_dir = os.getcwd()  # OUTPUT_DIR
--
-     def init_binaries(self) -> None:
-         """Init binary path variables:
-              PYTHON (for bash tests)
-diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/testrunner.py
-+++ b/tests/qemu-iotests/testrunner.py
-@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
-         """
-         f_test = Path(test)
--        f_bad = Path(f_test.name + '.out.bad')
--        f_notrun = Path(f_test.name + '.notrun')
--        f_casenotrun = Path(f_test.name + '.casenotrun')
-         f_reference = Path(self.find_reference(test))
-         if not f_test.exists():
-@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
-                               description='No qualified output '
-                                           f'(expected {f_reference})')
--        for p in (f_bad, f_notrun, f_casenotrun):
--            silent_unlink(p)
--
-         args = [str(f_test.resolve())]
-         env = self.env.prepare_subprocess(args)
-         if mp:
-@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
-                 env[d] = os.path.join(env[d], f_test.name)
-                 Path(env[d]).mkdir(parents=True, exist_ok=True)
-+        test_dir = env['TEST_DIR']
-+        f_bad = Path(test_dir, f_test.name + '.out.bad')
-+        f_notrun = Path(test_dir, f_test.name + '.notrun')
-+        f_casenotrun = Path(test_dir, f_test.name + '.casenotrun')
-+
-+        for p in (f_notrun, f_casenotrun):
-+            silent_unlink(p)
-+
-         t0 = time.time()
-         with f_bad.open('w', encoding="utf-8") as f:
-             with subprocess.Popen(args, cwd=str(f_test.parent), env=env,
---
-.34.1

-[PULL 08/23] block/block-copy: move copy_bitmap initialization to block_copy_state_new()
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-We are going to complicate bitmap initialization in the further
-commit. And in future, backup job will be able to work without filter
-(when source is immutable), so we'll need same bitmap initialization in
-copy-before-write filter and in backup job. So, it's reasonable to do
-it in block-copy.
-Note that for now cbw_open() is the only caller of
-block_copy_state_new().
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-2-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- block/block-copy.c        | 1 +
- block/copy-before-write.c | 4 ----
-files changed, 1 insertion(+), 4 deletions(-)
-diff --git a/block/block-copy.c b/block/block-copy.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/block-copy.c
-+++ b/block/block-copy.c
-@@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
-         return NULL;
-     }
-     bdrv_disable_dirty_bitmap(copy_bitmap);
-+    bdrv_set_dirty_bitmap(copy_bitmap, 0, bdrv_dirty_bitmap_size(copy_bitmap));
-     /*
-      * If source is in backing chain of target assume that target is going to be
-diff --git a/block/copy-before-write.c b/block/copy-before-write.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/copy-before-write.c
-+++ b/block/copy-before-write.c
-@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
-                     Error **errp)
- {
-     BDRVCopyBeforeWriteState *s = bs->opaque;
--    BdrvDirtyBitmap *copy_bitmap;
-     bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
-                                BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
-@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
-         return -EINVAL;
-     }
--    copy_bitmap = block_copy_dirty_bitmap(s->bcs);
--    bdrv_set_dirty_bitmap(copy_bitmap, 0, bdrv_dirty_bitmap_size(copy_bitmap));
--
-     return 0;
- }
---
-.34.1

-[PULL 09/23] block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-That simplifies handling failure in existing code and in further new
-usage of bdrv_merge_dirty_bitmap().
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-3-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- include/block/dirty-bitmap.h    | 2 +-
- block/dirty-bitmap.c            | 9 +++++++--
- block/monitor/bitmap-qmp-cmds.c | 5 +----
-files changed, 9 insertions(+), 7 deletions(-)
-diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/dirty-bitmap.h
-+++ b/include/block/dirty-bitmap.h
-@@ -XXX,XX +XXX,XX @@ void bdrv_dirty_bitmap_set_persistence(BdrvDirtyBitmap *bitmap,
-                                        bool persistent);
- void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap);
- void bdrv_dirty_bitmap_set_busy(BdrvDirtyBitmap *bitmap, bool busy);
--void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
-+bool bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
-                              HBitmap **backup, Error **errp);
- void bdrv_dirty_bitmap_skip_store(BdrvDirtyBitmap *bitmap, bool skip);
- bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset);
-diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/dirty-bitmap.c
-+++ b/block/dirty-bitmap.c
-@@ -XXX,XX +XXX,XX @@ bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
-  * Ensures permissions on bitmaps are reasonable; use for public API.
-  *
-  * @backup: If provided, make a copy of dest here prior to merge.
-+ *
-+ * Returns true on success, false on failure. In case of failure bitmaps are
-+ * untouched.
-  */
--void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
-+bool bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
-                              HBitmap **backup, Error **errp)
- {
--    bool ret;
-+    bool ret = false;
-     bdrv_dirty_bitmaps_lock(dest->bs);
-     if (src->bs != dest->bs) {
-@@ -XXX,XX +XXX,XX @@ out:
-     if (src->bs != dest->bs) {
-         bdrv_dirty_bitmaps_unlock(src->bs);
-     }
-+
-+    return ret;
- }
- /**
-diff --git a/block/monitor/bitmap-qmp-cmds.c b/block/monitor/bitmap-qmp-cmds.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/monitor/bitmap-qmp-cmds.c
-+++ b/block/monitor/bitmap-qmp-cmds.c
-@@ -XXX,XX +XXX,XX @@ BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
-     BlockDriverState *bs;
-     BdrvDirtyBitmap *dst, *src, *anon;
-     BlockDirtyBitmapMergeSourceList *lst;
--    Error *local_err = NULL;
-     GLOBAL_STATE_CODE();
-@@ -XXX,XX +XXX,XX @@ BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
-             abort();
-         }
--        bdrv_merge_dirty_bitmap(anon, src, NULL, &local_err);
--        if (local_err) {
--            error_propagate(errp, local_err);
-+        if (!bdrv_merge_dirty_bitmap(anon, src, NULL, errp)) {
-             dst = NULL;
-             goto out;
-         }
---
-.34.1

-[PULL 10/23] block/block-copy: block_copy_state_new(): add bitmap parameter
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-This will be used in the following commit to bring "incremental" mode
-to copy-before-write filter.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-4-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- include/block/block-copy.h |  1 +
- block/block-copy.c         | 14 +++++++++++++-
- block/copy-before-write.c  |  2 +-
-files changed, 15 insertions(+), 2 deletions(-)
-diff --git a/include/block/block-copy.h b/include/block/block-copy.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block-copy.h
-+++ b/include/block/block-copy.h
-@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState BlockCopyState;
- typedef struct BlockCopyCallState BlockCopyCallState;
- BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
-+                                     const BdrvDirtyBitmap *bitmap,
-                                      Error **errp);
- /* Function should be called prior any actual copy request */
-diff --git a/block/block-copy.c b/block/block-copy.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/block-copy.c
-+++ b/block/block-copy.c
-@@ -XXX,XX +XXX,XX @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target,
- }
- BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
-+                                     const BdrvDirtyBitmap *bitmap,
-                                      Error **errp)
- {
-+    ERRP_GUARD();
-     BlockCopyState *s;
-     int64_t cluster_size;
-     BdrvDirtyBitmap *copy_bitmap;
-@@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
-         return NULL;
-     }
-     bdrv_disable_dirty_bitmap(copy_bitmap);
--    bdrv_set_dirty_bitmap(copy_bitmap, 0, bdrv_dirty_bitmap_size(copy_bitmap));
-+    if (bitmap) {
-+        if (!bdrv_merge_dirty_bitmap(copy_bitmap, bitmap, NULL, errp)) {
-+            error_prepend(errp, "Failed to merge bitmap '%s' to internal "
-+                          "copy-bitmap: ", bdrv_dirty_bitmap_name(bitmap));
-+            bdrv_release_dirty_bitmap(copy_bitmap);
-+            return NULL;
-+        }
-+    } else {
-+        bdrv_set_dirty_bitmap(copy_bitmap, 0,
-+                              bdrv_dirty_bitmap_size(copy_bitmap));
-+    }
-     /*
-      * If source is in backing chain of target assume that target is going to be
-diff --git a/block/copy-before-write.c b/block/copy-before-write.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/copy-before-write.c
-+++ b/block/copy-before-write.c
-@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
-             ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) &
-              bs->file->bs->supported_zero_flags);
--    s->bcs = block_copy_state_new(bs->file, s->target, errp);
-+    s->bcs = block_copy_state_new(bs->file, s->target, NULL, errp);
-     if (!s->bcs) {
-         error_prepend(errp, "Cannot create block-copy-state: ");
-         return -EINVAL;
---
-.34.1

-[PULL 11/23] block/copy-before-write: add bitmap open parameter
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-This brings "incremental" mode to copy-before-write filter: user can
-specify bitmap so that filter will copy only "dirty" areas.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-Id: <20220303194349.2304213-5-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- qapi/block-core.json      | 10 +++++++-
- block/copy-before-write.c | 51 ++++++++++++++++++++++++++++++++++++++-
-files changed, 59 insertions(+), 2 deletions(-)
-diff --git a/qapi/block-core.json b/qapi/block-core.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/block-core.json
-+++ b/qapi/block-core.json
-@@ -XXX,XX +XXX,XX @@
- #
- # @target: The target for copy-before-write operations.
- #
-+# @bitmap: If specified, copy-before-write filter will do
-+#          copy-before-write operations only for dirty regions of the
-+#          bitmap. Bitmap size must be equal to length of file and
-+#          target child of the filter. Note also, that bitmap is used
-+#          only to initialize internal bitmap of the process, so further
-+#          modifications (or removing) of specified bitmap doesn't
-+#          influence the filter. (Since 7.0)
-+#
- # Since: 6.2
- ##
- { 'struct': 'BlockdevOptionsCbw',
-   'base': 'BlockdevOptionsGenericFormat',
--  'data': { 'target': 'BlockdevRef' } }
-+  'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap' } }
- ##
- # @BlockdevOptions:
-diff --git a/block/copy-before-write.c b/block/copy-before-write.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/copy-before-write.c
-+++ b/block/copy-before-write.c
-@@ -XXX,XX +XXX,XX @@
- #include "block/copy-before-write.h"
-+#include "qapi/qapi-visit-block-core.h"
-+
- typedef struct BDRVCopyBeforeWriteState {
-     BlockCopyState *bcs;
-     BdrvChild *target;
-@@ -XXX,XX +XXX,XX @@ static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c,
-     }
- }
-+static bool cbw_parse_bitmap_option(QDict *options, BdrvDirtyBitmap **bitmap,
-+                                    Error **errp)
-+{
-+    QDict *bitmap_qdict = NULL;
-+    BlockDirtyBitmap *bmp_param = NULL;
-+    Visitor *v = NULL;
-+    bool ret = false;
-+
-+    *bitmap = NULL;
-+
-+    qdict_extract_subqdict(options, &bitmap_qdict, "bitmap.");
-+    if (!qdict_size(bitmap_qdict)) {
-+        ret = true;
-+        goto out;
-+    }
-+
-+    v = qobject_input_visitor_new_flat_confused(bitmap_qdict, errp);
-+    if (!v) {
-+        goto out;
-+    }
-+
-+    visit_type_BlockDirtyBitmap(v, NULL, &bmp_param, errp);
-+    if (!bmp_param) {
-+        goto out;
-+    }
-+
-+    *bitmap = block_dirty_bitmap_lookup(bmp_param->node, bmp_param->name, NULL,
-+                                        errp);
-+    if (!*bitmap) {
-+        goto out;
-+    }
-+
-+    ret = true;
-+
-+out:
-+    qapi_free_BlockDirtyBitmap(bmp_param);
-+    visit_free(v);
-+    qobject_unref(bitmap_qdict);
-+
-+    return ret;
-+}
-+
- static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
-                     Error **errp)
- {
-     BDRVCopyBeforeWriteState *s = bs->opaque;
-+    BdrvDirtyBitmap *bitmap = NULL;
-     bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
-                                BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
-@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
-         return -EINVAL;
-     }
-+    if (!cbw_parse_bitmap_option(options, &bitmap, errp)) {
-+        return -EINVAL;
-+    }
-+
-     bs->total_sectors = bs->file->bs->total_sectors;
-     bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
-             (BDRV_REQ_FUA & bs->file->bs->supported_write_flags);
-@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
-             ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) &
-              bs->file->bs->supported_zero_flags);
--    s->bcs = block_copy_state_new(bs->file, s->target, NULL, errp);
-+    s->bcs = block_copy_state_new(bs->file, s->target, bitmap, errp);
-     if (!s->bcs) {
-         error_prepend(errp, "Cannot create block-copy-state: ");
-         return -EINVAL;
---
-.34.1

-[PULL 12/23] block/block-copy: add block_copy_reset()
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Split block_copy_reset() out of block_copy_reset_unallocated() to be
-used separately later.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-6-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- include/block/block-copy.h |  1 +
- block/block-copy.c         | 21 +++++++++++++--------
-files changed, 14 insertions(+), 8 deletions(-)
-diff --git a/include/block/block-copy.h b/include/block/block-copy.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block-copy.h
-+++ b/include/block/block-copy.h
-@@ -XXX,XX +XXX,XX @@ void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm);
- void block_copy_state_free(BlockCopyState *s);
-+void block_copy_reset(BlockCopyState *s, int64_t offset, int64_t bytes);
- int64_t block_copy_reset_unallocated(BlockCopyState *s,
-                                      int64_t offset, int64_t *count);
-diff --git a/block/block-copy.c b/block/block-copy.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/block-copy.c
-+++ b/block/block-copy.c
-@@ -XXX,XX +XXX,XX @@ static int block_copy_is_cluster_allocated(BlockCopyState *s, int64_t offset,
-     }
- }
-+void block_copy_reset(BlockCopyState *s, int64_t offset, int64_t bytes)
-+{
-+    QEMU_LOCK_GUARD(&s->lock);
-+
-+    bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes);
-+    if (s->progress) {
-+        progress_set_remaining(s->progress,
-+                               bdrv_get_dirty_count(s->copy_bitmap) +
-+                               s->in_flight_bytes);
-+    }
-+}
-+
- /*
-  * Reset bits in copy_bitmap starting at offset if they represent unallocated
-  * data in the image. May reset subsequent contiguous bits.
-@@ -XXX,XX +XXX,XX @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,
-     bytes = clusters * s->cluster_size;
-     if (!ret) {
--        qemu_co_mutex_lock(&s->lock);
--        bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes);
--        if (s->progress) {
--            progress_set_remaining(s->progress,
--                                   bdrv_get_dirty_count(s->copy_bitmap) +
--                                   s->in_flight_bytes);
--        }
--        qemu_co_mutex_unlock(&s->lock);
-+        block_copy_reset(s, offset, bytes);
-     }
-     *count = bytes;
---
-.34.1

-[PULL 15/23] block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Add a convenient function similar with bdrv_block_status() to get
-status of dirty bitmap.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-9-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- include/block/dirty-bitmap.h |  2 ++
- include/qemu/hbitmap.h       | 12 ++++++++++++
- block/dirty-bitmap.c         |  6 ++++++
- util/hbitmap.c               | 33 +++++++++++++++++++++++++++++++++
-files changed, 53 insertions(+)
-diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/dirty-bitmap.h
-+++ b/include/block/dirty-bitmap.h
-@@ -XXX,XX +XXX,XX @@ int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, int64_t offset,
- bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
-         int64_t start, int64_t end, int64_t max_dirty_count,
-         int64_t *dirty_start, int64_t *dirty_count);
-+bool bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap, int64_t offset,
-+                              int64_t bytes, int64_t *count);
- BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
-                                                   Error **errp);
-diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/qemu/hbitmap.h
-+++ b/include/qemu/hbitmap.h
-@@ -XXX,XX +XXX,XX @@ bool hbitmap_next_dirty_area(const HBitmap *hb, int64_t start, int64_t end,
-                              int64_t max_dirty_count,
-                              int64_t *dirty_start, int64_t *dirty_count);
-+/*
-+ * bdrv_dirty_bitmap_status:
-+ * @hb: The HBitmap to operate on
-+ * @start: The bit to start from
-+ * @count: Number of bits to proceed
-+ * @pnum: Out-parameter. How many bits has same value starting from @start
-+ *
-+ * Returns true if bitmap is dirty at @start, false otherwise.
-+ */
-+bool hbitmap_status(const HBitmap *hb, int64_t start, int64_t count,
-+                    int64_t *pnum);
-+
- /**
-  * hbitmap_iter_next:
-  * @hbi: HBitmapIter to operate on.
-diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/dirty-bitmap.c
-+++ b/block/dirty-bitmap.c
-@@ -XXX,XX +XXX,XX @@ bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
-                                    dirty_start, dirty_count);
- }
-+bool bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap, int64_t offset,
-+                              int64_t bytes, int64_t *count)
-+{
-+    return hbitmap_status(bitmap->bitmap, offset, bytes, count);
-+}
-+
- /**
-  * bdrv_merge_dirty_bitmap: merge src into dest.
-  * Ensures permissions on bitmaps are reasonable; use for public API.
-diff --git a/util/hbitmap.c b/util/hbitmap.c
-index XXXXXXX..XXXXXXX 100644
---- a/util/hbitmap.c
-+++ b/util/hbitmap.c
-@@ -XXX,XX +XXX,XX @@ bool hbitmap_next_dirty_area(const HBitmap *hb, int64_t start, int64_t end,
-     return true;
- }
-+bool hbitmap_status(const HBitmap *hb, int64_t start, int64_t count,
-+                    int64_t *pnum)
-+{
-+    int64_t next_dirty, next_zero;
-+
-+    assert(start >= 0);
-+    assert(count > 0);
-+    assert(start + count <= hb->orig_size);
-+
-+    next_dirty = hbitmap_next_dirty(hb, start, count);
-+    if (next_dirty == -1) {
-+        *pnum = count;
-+        return false;
-+    }
-+
-+    if (next_dirty > start) {
-+        *pnum = next_dirty - start;
-+        return false;
-+    }
-+
-+    assert(next_dirty == start);
-+
-+    next_zero = hbitmap_next_zero(hb, start, count);
-+    if (next_zero == -1) {
-+        *pnum = count;
-+        return true;
-+    }
-+
-+    assert(next_zero > start);
-+    *pnum = next_zero - start;
-+    return false;
-+}
-+
- bool hbitmap_empty(const HBitmap *hb)
- {
-     return hb->count == 0;
---
-.34.1

-[PULL 16/23] block/reqlist: add reqlist_wait_all()
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Add function to wait for all intersecting requests.
-To be used in the further commit.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Nikita Lapshin <nikita.lapshin@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-10-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- include/block/reqlist.h | 8 ++++++++
- block/reqlist.c         | 8 ++++++++
-files changed, 16 insertions(+)
-diff --git a/include/block/reqlist.h b/include/block/reqlist.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/reqlist.h
-+++ b/include/block/reqlist.h
-@@ -XXX,XX +XXX,XX @@ BlockReq *reqlist_find_conflict(BlockReqList *reqs, int64_t offset,
- bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
-                                    int64_t bytes, CoMutex *lock);
-+/*
-+ * Wait for all intersecting requests. It just calls reqlist_wait_one() in a
-+ * loop, caller is responsible to stop producing new requests in this region
-+ * in parallel, otherwise reqlist_wait_all() may never return.
-+ */
-+void coroutine_fn reqlist_wait_all(BlockReqList *reqs, int64_t offset,
-+                                   int64_t bytes, CoMutex *lock);
-+
- /*
-  * Shrink request and wake all waiting coroutines (maybe some of them are not
-  * intersecting with shrunk request).
-diff --git a/block/reqlist.c b/block/reqlist.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/reqlist.c
-+++ b/block/reqlist.c
-@@ -XXX,XX +XXX,XX @@ bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
-     return true;
- }
-+void coroutine_fn reqlist_wait_all(BlockReqList *reqs, int64_t offset,
-+                                   int64_t bytes, CoMutex *lock)
-+{
-+    while (reqlist_wait_one(reqs, offset, bytes, lock)) {
-+        /* continue */
-+    }
-+}
-+
- void coroutine_fn reqlist_shrink_req(BlockReq *req, int64_t new_bytes)
- {
-     if (new_bytes == req->bytes) {
---
-.34.1

-[PULL 17/23] block/io: introduce block driver snapshot-access API
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Add new block driver handlers and corresponding generic wrappers.
-It will be used to allow copy-before-write filter to provide
-reach fleecing interface in further commit.
-In future this approach may be used to allow reading qcow2 internal
-snapshots, for example to export them through NBD.
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-11-vsementsov@virtuozzo.com>
-[hreitz: Rebased on block GS/IO split]
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- include/block/block_int-common.h | 18 ++++++++
- include/block/block_int-io.h     |  9 ++++
- block/io.c                       | 72 ++++++++++++++++++++++++++++++++
-files changed, 99 insertions(+)
-diff --git a/include/block/block_int-common.h b/include/block/block_int-common.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block_int-common.h
-+++ b/include/block/block_int-common.h
-@@ -XXX,XX +XXX,XX @@ struct BlockDriver {
-         bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum,
-         int64_t *map, BlockDriverState **file);
-+    /*
-+     * Snapshot-access API.
-+     *
-+     * Block-driver may provide snapshot-access API: special functions to access
-+     * some internal "snapshot". The functions are similar with normal
-+     * read/block_status/discard handler, but don't have any specific handling
-+     * in generic block-layer: no serializing, no alignment, no tracked
-+     * requests. So, block-driver that realizes these APIs is fully responsible
-+     * for synchronization between snapshot-access API and normal IO requests.
-+     */
-+    int coroutine_fn (*bdrv_co_preadv_snapshot)(BlockDriverState *bs,
-+        int64_t offset, int64_t bytes, QEMUIOVector *qiov, size_t qiov_offset);
-+    int coroutine_fn (*bdrv_co_snapshot_block_status)(BlockDriverState *bs,
-+        bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum,
-+        int64_t *map, BlockDriverState **file);
-+    int coroutine_fn (*bdrv_co_pdiscard_snapshot)(BlockDriverState *bs,
-+        int64_t offset, int64_t bytes);
-+
-     /*
-      * Invalidate any cached meta-data.
-      */
-diff --git a/include/block/block_int-io.h b/include/block/block_int-io.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/block/block_int-io.h
-+++ b/include/block/block_int-io.h
-@@ -XXX,XX +XXX,XX @@
-  * the I/O API.
-  */
-+int coroutine_fn bdrv_co_preadv_snapshot(BdrvChild *child,
-+    int64_t offset, int64_t bytes, QEMUIOVector *qiov, size_t qiov_offset);
-+int coroutine_fn bdrv_co_snapshot_block_status(BlockDriverState *bs,
-+    bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum,
-+    int64_t *map, BlockDriverState **file);
-+int coroutine_fn bdrv_co_pdiscard_snapshot(BlockDriverState *bs,
-+    int64_t offset, int64_t bytes);
-+
-+
- int coroutine_fn bdrv_co_preadv(BdrvChild *child,
-     int64_t offset, int64_t bytes, QEMUIOVector *qiov,
-     BdrvRequestFlags flags);
-diff --git a/block/io.c b/block/io.c
-index XXXXXXX..XXXXXXX 100644
---- a/block/io.c
-+++ b/block/io.c
-@@ -XXX,XX +XXX,XX @@ void bdrv_cancel_in_flight(BlockDriverState *bs)
-         bs->drv->bdrv_cancel_in_flight(bs);
-     }
- }
-+
-+int coroutine_fn
-+bdrv_co_preadv_snapshot(BdrvChild *child, int64_t offset, int64_t bytes,
-+                        QEMUIOVector *qiov, size_t qiov_offset)
-+{
-+    BlockDriverState *bs = child->bs;
-+    BlockDriver *drv = bs->drv;
-+    int ret;
-+    IO_CODE();
-+
-+    if (!drv) {
-+        return -ENOMEDIUM;
-+    }
-+
-+    if (!drv->bdrv_co_preadv_snapshot) {
-+        return -ENOTSUP;
-+    }
-+
-+    bdrv_inc_in_flight(bs);
-+    ret = drv->bdrv_co_preadv_snapshot(bs, offset, bytes, qiov, qiov_offset);
-+    bdrv_dec_in_flight(bs);
-+
-+    return ret;
-+}
-+
-+int coroutine_fn
-+bdrv_co_snapshot_block_status(BlockDriverState *bs,
-+                              bool want_zero, int64_t offset, int64_t bytes,
-+                              int64_t *pnum, int64_t *map,
-+                              BlockDriverState **file)
-+{
-+    BlockDriver *drv = bs->drv;
-+    int ret;
-+    IO_CODE();
-+
-+    if (!drv) {
-+        return -ENOMEDIUM;
-+    }
-+
-+    if (!drv->bdrv_co_snapshot_block_status) {
-+        return -ENOTSUP;
-+    }
-+
-+    bdrv_inc_in_flight(bs);
-+    ret = drv->bdrv_co_snapshot_block_status(bs, want_zero, offset, bytes,
-+                                             pnum, map, file);
-+    bdrv_dec_in_flight(bs);
-+
-+    return ret;
-+}
-+
-+int coroutine_fn
-+bdrv_co_pdiscard_snapshot(BlockDriverState *bs, int64_t offset, int64_t bytes)
-+{
-+    BlockDriver *drv = bs->drv;
-+    int ret;
-+    IO_CODE();
-+
-+    if (!drv) {
-+        return -ENOMEDIUM;
-+    }
-+
-+    if (!drv->bdrv_co_pdiscard_snapshot) {
-+        return -ENOTSUP;
-+    }
-+
-+    bdrv_inc_in_flight(bs);
-+    ret = drv->bdrv_co_pdiscard_snapshot(bs, offset, bytes);
-+    bdrv_dec_in_flight(bs);
-+
-+    return ret;
-+}
---
-.34.1

-[PULL 20/23] iotests/image-fleecing: add test-case for fleecing format node
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Reviewed-by: Hanna Reitz <hreitz@redhat.com>
-Message-Id: <20220303194349.2304213-14-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- tests/qemu-iotests/tests/image-fleecing     | 64 +++++++++++++-----
- tests/qemu-iotests/tests/image-fleecing.out | 74 ++++++++++++++++++++-
-files changed, 119 insertions(+), 19 deletions(-)
-diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
-index XXXXXXX..XXXXXXX 100755
---- a/tests/qemu-iotests/tests/image-fleecing
-+++ b/tests/qemu-iotests/tests/image-fleecing
-@@ -XXX,XX +XXX,XX @@ remainder = [('0xd5', '0x108000',  '32k'), # Right-end of partial-left [1]
-              ('0xdc', '32M',       '32k'), # Left-end of partial-right [2]
-              ('0xcd', '0x3ff0000', '64k')] # patterns[3]
--def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
-+def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
-+            fleece_img_path, nbd_sock_path, vm):
-     log('--- Setting up images ---')
-     log('')
-     assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
--    assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
-+    if use_snapshot_access_filter:
-+        assert use_cbw
-+        assert qemu_img('create', '-f', 'raw', fleece_img_path, '64M') == 0
-+    else:
-+        assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
-     for p in patterns:
-         qemu_io('-f', iotests.imgfmt,
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
-     log('')
--    # create tmp_node backed by src_node
--    log(vm.qmp('blockdev-add', {
--        'driver': 'qcow2',
--        'node-name': tmp_node,
--        'file': {
-+    if use_snapshot_access_filter:
-+        log(vm.qmp('blockdev-add', {
-+            'node-name': tmp_node,
-             'driver': 'file',
-             'filename': fleece_img_path,
--        },
--        'backing': src_node,
--    }))
-+        }))
-+    else:
-+        # create tmp_node backed by src_node
-+        log(vm.qmp('blockdev-add', {
-+            'driver': 'qcow2',
-+            'node-name': tmp_node,
-+            'file': {
-+                'driver': 'file',
-+                'filename': fleece_img_path,
-+            },
-+            'backing': src_node,
-+        }))
-     # Establish CBW from source to fleecing node
-     if use_cbw:
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
-         }))
-         log(vm.qmp('qom-set', path=qom_path, property='drive', value='fl-cbw'))
-+
-+        if use_snapshot_access_filter:
-+            log(vm.qmp('blockdev-add', {
-+                'driver': 'snapshot-access',
-+                'node-name': 'fl-access',
-+                'file': 'fl-cbw',
-+            }))
-     else:
-         log(vm.qmp('blockdev-backup',
-                    job_id='fleecing',
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
-                    target=tmp_node,
-                    sync='none'))
-+    export_node = 'fl-access' if use_snapshot_access_filter else tmp_node
-+
-     log('')
-     log('--- Setting up NBD Export ---')
-     log('')
--    nbd_uri = 'nbd+unix:///%s?socket=%s' % (tmp_node, nbd_sock_path)
-+    nbd_uri = 'nbd+unix:///%s?socket=%s' % (export_node, nbd_sock_path)
-     log(vm.qmp('nbd-server-start',
-                {'addr': {'type': 'unix',
-                          'data': {'path': nbd_sock_path}}}))
--    log(vm.qmp('nbd-server-add', device=tmp_node))
-+    log(vm.qmp('nbd-server-add', device=export_node))
-     log('')
-     log('--- Sanity Check ---')
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
-     log('--- Cleanup ---')
-     log('')
-+    log(vm.qmp('nbd-server-stop'))
-+
-     if use_cbw:
-+        if use_snapshot_access_filter:
-+            log(vm.qmp('blockdev-del', node_name='fl-access'))
-         log(vm.qmp('qom-set', path=qom_path, property='drive', value=src_node))
-         log(vm.qmp('blockdev-del', node_name='fl-cbw'))
-     else:
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
-         assert e is not None
-         log(e, filters=[iotests.filter_qmp_event])
--    log(vm.qmp('nbd-server-stop'))
-     log(vm.qmp('blockdev-del', node_name=tmp_node))
-     vm.shutdown()
-@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
-     log('Done')
--def test(use_cbw):
-+def test(use_cbw, use_snapshot_access_filter):
-     with iotests.FilePath('base.img') as base_img_path, \
-          iotests.FilePath('fleece.img') as fleece_img_path, \
-          iotests.FilePath('nbd.sock',
-                           base_dir=iotests.sock_dir) as nbd_sock_path, \
-          iotests.VM() as vm:
--        do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm)
-+        do_test(use_cbw, use_snapshot_access_filter, base_img_path,
-+                fleece_img_path, nbd_sock_path, vm)
- log('=== Test backup(sync=none) based fleecing ===\n')
--test(False)
-+test(False, False)
-+
-+log('=== Test cbw-filter based fleecing ===\n')
-+test(True, False)
--log('=== Test filter based fleecing ===\n')
--test(True)
-+log('=== Test fleecing-format based fleecing ===\n')
-+test(True, True)
-diff --git a/tests/qemu-iotests/tests/image-fleecing.out b/tests/qemu-iotests/tests/image-fleecing.out
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/tests/image-fleecing.out
-+++ b/tests/qemu-iotests/tests/image-fleecing.out
-@@ -XXX,XX +XXX,XX @@ read -P0 0x3fe0000 64k
- --- Cleanup ---
-+{"return": {}}
- {"return": {}}
- {"data": {"device": "fleecing", "len": 67108864, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
- {"return": {}}
-+
-+--- Confirming writes ---
-+
-+read -P0xab 0 64k
-+read -P0xad 0x00f8000 64k
-+read -P0x1d 0x2008000 64k
-+read -P0xea 0x3fe0000 64k
-+read -P0xd5 0x108000 32k
-+read -P0xdc 32M 32k
-+read -P0xcd 0x3ff0000 64k
-+
-+Done
-+=== Test cbw-filter based fleecing ===
-+
-+--- Setting up images ---
-+
-+Done
-+
-+--- Launching VM ---
-+
-+Done
-+
-+--- Setting up Fleecing Graph ---
-+
-+{"return": {}}
-+{"return": {}}
-+{"return": {}}
-+
-+--- Setting up NBD Export ---
-+
-+{"return": {}}
-+{"return": {}}
-+
-+--- Sanity Check ---
-+
-+read -P0x5d 0 64k
-+read -P0xd5 1M 64k
-+read -P0xdc 32M 64k
-+read -P0xcd 0x3ff0000 64k
-+read -P0 0x00f8000 32k
-+read -P0 0x2010000 32k
-+read -P0 0x3fe0000 64k
-+
-+--- Testing COW ---
-+
-+write -P0xab 0 64k
-+{"return": ""}
-+write -P0xad 0x00f8000 64k
-+{"return": ""}
-+write -P0x1d 0x2008000 64k
-+{"return": ""}
-+write -P0xea 0x3fe0000 64k
-+{"return": ""}
-+
-+--- Verifying Data ---
-+
-+read -P0x5d 0 64k
-+read -P0xd5 1M 64k
-+read -P0xdc 32M 64k
-+read -P0xcd 0x3ff0000 64k
-+read -P0 0x00f8000 32k
-+read -P0 0x2010000 32k
-+read -P0 0x3fe0000 64k
-+
-+--- Cleanup ---
-+
-+{"return": {}}
-+{"return": {}}
-+{"return": {}}
- {"return": {}}
- --- Confirming writes ---
-@@ -XXX,XX +XXX,XX @@ read -P0xdc 32M 32k
- read -P0xcd 0x3ff0000 64k
- Done
--=== Test filter based fleecing ===
-+=== Test fleecing-format based fleecing ===
- --- Setting up images ---
-@@ -XXX,XX +XXX,XX @@ Done
- {"return": {}}
- {"return": {}}
- {"return": {}}
-+{"return": {}}
- --- Setting up NBD Export ---
-@@ -XXX,XX +XXX,XX @@ read -P0 0x3fe0000 64k
- {"return": {}}
- {"return": {}}
- {"return": {}}
-+{"return": {}}
- --- Confirming writes ---
---
-.34.1

-[PULL 21/23] iotests.py: add qemu_io_pipe_and_status()
+Deleted patch
-From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Add helper that returns both status and output, to be used in the
-following commit
-Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
-Message-Id: <20220303194349.2304213-15-vsementsov@virtuozzo.com>
-Signed-off-by: Hanna Reitz <hreitz@redhat.com>
----
- tests/qemu-iotests/iotests.py | 3 +++
-file changed, 3 insertions(+)
-diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qemu-iotests/iotests.py
-+++ b/tests/qemu-iotests/iotests.py
-@@ -XXX,XX +XXX,XX @@ def qemu_io(*args):
-     '''Run qemu-io and return the stdout data'''
-     return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))[0]
-+def qemu_io_pipe_and_status(*args):
-+    return qemu_tool_pipe_and_status('qemu-io', qemu_io_wrap_args(args))
-+
- def qemu_io_log(*args):
-     result = qemu_io(*args)
-     log(result, filters=[filter_testfiles, filter_qemu_io])
---
-.34.1

The following changes since commit 9d662a6b22a0838a85c5432385f35db2488a33a5:

Merge remote-tracking branch 'remotes/legoater/tags/pull-ppc-20220305' into staging (2022-03-05 18:03:15 +0000)

are available in the Git repository at:

https://gitlab.com/hreitz/qemu.git tags/pull-block-2022-03-07

for you to fetch changes up to 743da0b401cdc3ee94bc519975e339a3cdbe0ad1:

iotests/image-fleecing: test push backup with fleecing (2022-03-07 09:33:31 +0100)

----------------------------------------------------------------
Block patches for 7.0-rc0:
- New fleecing backup scheme
- iotest fixes
- Fixes for the curl block driver
- Fix for the preallocate block driver
- IDE fix for zero-length TRIM requests

----------------------------------------------------------------
Hanna Reitz (2):
  ide: Increment BB in-flight counter for TRIM BH
  iotests: Write test output to TEST_DIR

Peter Maydell (2):
  block/curl.c: Set error message string if curl_init_state() fails
  block/curl.c: Check error return from curl_easy_setopt()

Thomas Huth (2):
  tests/qemu-iotests/040: Skip TestCommitWithFilters without 'throttle'
  tests/qemu-iotests/testrunner: Quote "case not run" lines in TAP mode

Vladimir Sementsov-Ogievskiy (17):
  block: fix preallocate filter: don't do unaligned preallocate requests
  block/block-copy: move copy_bitmap initialization to
    block_copy_state_new()
  block/dirty-bitmap: bdrv_merge_dirty_bitmap(): add return value
  block/block-copy: block_copy_state_new(): add bitmap parameter
  block/copy-before-write: add bitmap open parameter
  block/block-copy: add block_copy_reset()
  block: intoduce reqlist
  block/reqlist: reqlist_find_conflict(): use ranges_overlap()
  block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()
  block/reqlist: add reqlist_wait_all()
  block/io: introduce block driver snapshot-access API
  block: introduce snapshot-access block driver
  block: copy-before-write: realize snapshot-access API
  iotests/image-fleecing: add test-case for fleecing format node
  iotests.py: add qemu_io_pipe_and_status()
  iotests/image-fleecing: add test case with bitmap
  iotests/image-fleecing: test push backup with fleecing

-- 
2.34.1

When we still have an AIOCB registered for DMA operations, we try to
settle the respective operation by draining the BlockBackend associated
with the IDE device.

However, this assumes that every DMA operation is associated with an
increment of the BlockBackend’s in-flight counter (e.g. through some
ongoing I/O operation), so that draining the BB until its in-flight
counter reaches 0 will settle all DMA operations.  That is not the case:
For TRIM, the guest can issue a zero-length operation that will not
result in any I/O operation forwarded to the BlockBackend, and also not
increment the in-flight counter in any other way.  In such a case,
blk_drain() will be a no-op if no other operations are in flight.

It is clear that if blk_drain() is a no-op, the value of
s->bus->dma->aiocb will not change between checking it in the `if`
condition and asserting that it is NULL after blk_drain().

The particular problem is that ide_issue_trim() creates a BH
(ide_trim_bh_cb()) to settle the TRIM request: iocb->common.cb() is
ide_dma_cb(), which will either create a new request, or find the
transfer to be done and call ide_set_inactive(), which clears
s->bus->dma->aiocb.  Therefore, the blk_drain() must wait for
ide_trim_bh_cb() to run, which currently it will not always do.

To fix this issue, we increment the BlockBackend's in-flight counter
when the TRIM operation begins (in ide_issue_trim(), when the
ide_trim_bh_cb() BH is created) and decrement it when ide_trim_bh_cb()
is done.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2029980
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220120142259.120189-1-hreitz@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Tested-by: John Snow <jsnow@redhat.com>
---
 hw/ide/core.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/hw/ide/core.c b/hw/ide/core.c
index XXXXXXX..XXXXXXX 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -XXX,XX +XXX,XX @@ static const AIOCBInfo trim_aiocb_info = {
 static void ide_trim_bh_cb(void *opaque)
 {
     TrimAIOCB *iocb = opaque;
+    BlockBackend *blk = iocb->s->blk;
 
     iocb->common.cb(iocb->common.opaque, iocb->ret);
 
     qemu_bh_delete(iocb->bh);
     iocb->bh = NULL;
     qemu_aio_unref(iocb);
+
+    /* Paired with an increment in ide_issue_trim() */
+    blk_dec_in_flight(blk);
 }
 
 static void ide_issue_trim_cb(void *opaque, int ret)
@@ -XXX,XX +XXX,XX @@ BlockAIOCB *ide_issue_trim(
     IDEState *s = opaque;
     TrimAIOCB *iocb;
 
+    /* Paired with a decrement in ide_trim_bh_cb() */
+    blk_inc_in_flight(s->blk);
+
     iocb = blk_aio_get(&trim_aiocb_info, s->blk, cb, cb_opaque);
     iocb->s = s;
     iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
-- 
2.34.1

From: Peter Maydell <peter.maydell@linaro.org>

In curl_open(), the 'out' label assumes that the state->errmsg string
has been set (either by curl_easy_perform() or by manually copying a
string into it); however if curl_init_state() fails we will jump to
that label without setting the string.  Add the missing error string
setup.

(We can't be specific about the cause of failure: the documentation
of curl_easy_init() just says "If this function returns NULL,
something went wrong".)

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20220222152341.850419-2-peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 block/curl.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
     // Get file size
 
     if (curl_init_state(s, state) < 0) {
+        pstrcpy(state->errmsg, CURL_ERROR_SIZE,
+                "curl library initialization failed.");
         goto out;
     }
 
-- 
2.34.1

From: Peter Maydell <peter.maydell@linaro.org>

Coverity points out that we aren't checking the return value
from curl_easy_setopt() for any of the calls to it we make
in block/curl.c.

Some of these options are documented as always succeeding (e.g.
CURLOPT_VERBOSE) but others have documented failure cases (e.g.
CURLOPT_URL).  For consistency we check every call, even the ones
that theoretically cannot fail.

Fixes: Coverity CID 1459336, 1459482, 1460331
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20220222152341.850419-3-peter.maydell@linaro.org>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 block/curl.c | 90 +++++++++++++++++++++++++++++++++-------------------
 1 file changed, 57 insertions(+), 33 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index XXXXXXX..XXXXXXX 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -XXX,XX +XXX,XX @@ static int curl_init_state(BDRVCURLState *s, CURLState *state)
         if (!state->curl) {
             return -EIO;
         }
-        curl_easy_setopt(state->curl, CURLOPT_URL, s->url);
-        curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYPEER,
-                         (long) s->sslverify);
-        curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYHOST,
-                         s->sslverify ? 2L : 0L);
+        if (curl_easy_setopt(state->curl, CURLOPT_URL, s->url) ||
+            curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYPEER,
+                             (long) s->sslverify) ||
+            curl_easy_setopt(state->curl, CURLOPT_SSL_VERIFYHOST,
+                             s->sslverify ? 2L : 0L)) {
+            goto err;
+        }
         if (s->cookie) {
-            curl_easy_setopt(state->curl, CURLOPT_COOKIE, s->cookie);
+            if (curl_easy_setopt(state->curl, CURLOPT_COOKIE, s->cookie)) {
+                goto err;
+            }
+        }
+        if (curl_easy_setopt(state->curl, CURLOPT_TIMEOUT, (long)s->timeout) ||
+            curl_easy_setopt(state->curl, CURLOPT_WRITEFUNCTION,
+                             (void *)curl_read_cb) ||
+            curl_easy_setopt(state->curl, CURLOPT_WRITEDATA, (void *)state) ||
+            curl_easy_setopt(state->curl, CURLOPT_PRIVATE, (void *)state) ||
+            curl_easy_setopt(state->curl, CURLOPT_AUTOREFERER, 1) ||
+            curl_easy_setopt(state->curl, CURLOPT_FOLLOWLOCATION, 1) ||
+            curl_easy_setopt(state->curl, CURLOPT_NOSIGNAL, 1) ||
+            curl_easy_setopt(state->curl, CURLOPT_ERRORBUFFER, state->errmsg) ||
+            curl_easy_setopt(state->curl, CURLOPT_FAILONERROR, 1)) {
+            goto err;
         }
-        curl_easy_setopt(state->curl, CURLOPT_TIMEOUT, (long)s->timeout);
-        curl_easy_setopt(state->curl, CURLOPT_WRITEFUNCTION,
-                         (void *)curl_read_cb);
-        curl_easy_setopt(state->curl, CURLOPT_WRITEDATA, (void *)state);
-        curl_easy_setopt(state->curl, CURLOPT_PRIVATE, (void *)state);
-        curl_easy_setopt(state->curl, CURLOPT_AUTOREFERER, 1);
-        curl_easy_setopt(state->curl, CURLOPT_FOLLOWLOCATION, 1);
-        curl_easy_setopt(state->curl, CURLOPT_NOSIGNAL, 1);
-        curl_easy_setopt(state->curl, CURLOPT_ERRORBUFFER, state->errmsg);
-        curl_easy_setopt(state->curl, CURLOPT_FAILONERROR, 1);
-
         if (s->username) {
-            curl_easy_setopt(state->curl, CURLOPT_USERNAME, s->username);
+            if (curl_easy_setopt(state->curl, CURLOPT_USERNAME, s->username)) {
+                goto err;
+            }
         }
         if (s->password) {
-            curl_easy_setopt(state->curl, CURLOPT_PASSWORD, s->password);
+            if (curl_easy_setopt(state->curl, CURLOPT_PASSWORD, s->password)) {
+                goto err;
+            }
         }
         if (s->proxyusername) {
-            curl_easy_setopt(state->curl,
-                             CURLOPT_PROXYUSERNAME, s->proxyusername);
+            if (curl_easy_setopt(state->curl,
+                                 CURLOPT_PROXYUSERNAME, s->proxyusername)) {
+                goto err;
+            }
         }
         if (s->proxypassword) {
-            curl_easy_setopt(state->curl,
-                             CURLOPT_PROXYPASSWORD, s->proxypassword);
+            if (curl_easy_setopt(state->curl,
+                                 CURLOPT_PROXYPASSWORD, s->proxypassword)) {
+                goto err;
+            }
         }
 
         /* Restrict supported protocols to avoid security issues in the more
@@ -XXX,XX +XXX,XX @@ static int curl_init_state(BDRVCURLState *s, CURLState *state)
          * Restricting protocols is only supported from 7.19.4 upwards.
          */
 #if LIBCURL_VERSION_NUM >= 0x071304
-        curl_easy_setopt(state->curl, CURLOPT_PROTOCOLS, PROTOCOLS);
-        curl_easy_setopt(state->curl, CURLOPT_REDIR_PROTOCOLS, PROTOCOLS);
+        if (curl_easy_setopt(state->curl, CURLOPT_PROTOCOLS, PROTOCOLS) ||
+            curl_easy_setopt(state->curl, CURLOPT_REDIR_PROTOCOLS, PROTOCOLS)) {
+            goto err;
+        }
 #endif
 
 #ifdef DEBUG_VERBOSE
-        curl_easy_setopt(state->curl, CURLOPT_VERBOSE, 1);
+        if (curl_easy_setopt(state->curl, CURLOPT_VERBOSE, 1)) {
+            goto err;
+        }
 #endif
     }
 
     state->s = s;
 
     return 0;
+
+err:
+    curl_easy_cleanup(state->curl);
+    state->curl = NULL;
+    return -EIO;
 }
 
 /* Called with s->mutex held.  */
@@ -XXX,XX +XXX,XX @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     s->accept_range = false;
-    curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1);
-    curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION,
-                     curl_header_cb);
-    curl_easy_setopt(state->curl, CURLOPT_HEADERDATA, s);
+    if (curl_easy_setopt(state->curl, CURLOPT_NOBODY, 1) ||
+        curl_easy_setopt(state->curl, CURLOPT_HEADERFUNCTION, curl_header_cb) ||
+        curl_easy_setopt(state->curl, CURLOPT_HEADERDATA, s)) {
+        pstrcpy(state->errmsg, CURL_ERROR_SIZE,
+                "curl library initialization failed.");
+        goto out;
+    }
     if (curl_easy_perform(state->curl))
         goto out;
     if (curl_easy_getinfo(state->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &d)) {
@@ -XXX,XX +XXX,XX @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
 
     snprintf(state->range, 127, "%" PRIu64 "-%" PRIu64, start, end);
     trace_curl_setup_preadv(acb->bytes, start, state->range);
-    curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range);
-
-    if (curl_multi_add_handle(s->multi, state->curl) != CURLM_OK) {
+    if (curl_easy_setopt(state->curl, CURLOPT_RANGE, state->range) ||
+        curl_multi_add_handle(s->multi, state->curl) != CURLM_OK) {
         state->acb[0] = NULL;
         acb->ret = -EIO;
 
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

There is a bug in handling BDRV_REQ_NO_WAIT flag: we still may wait in
wait_serialising_requests() if request is unaligned. And this is
possible for the only user of this flag (preallocate filter) if
underlying file is unaligned to its request_alignment on start.

So, we have to fix preallocate filter to do only aligned preallocate
requests.

Next, we should fix generic block/io.c somehow. Keeping in mind that
preallocate is the only user of BDRV_REQ_NO_WAIT and that we have to
fix its behavior now, it seems more safe to just assert that we never
use BDRV_REQ_NO_WAIT with unaligned requests and add corresponding
comment. Let's do so.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Message-Id: <20220215121609.38570-1-vsementsov@virtuozzo.com>
[hreitz: Rebased on block GS/IO split]
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/block-common.h |  3 ++-
 block/io.c                   |  4 ++++
 block/preallocate.c          | 15 ++++++++++++---
 3 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/include/block/block-common.h b/include/block/block-common.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block-common.h
+++ b/include/block/block-common.h
@@ -XXX,XX +XXX,XX @@ typedef enum {
 
     /*
      * If we need to wait for other requests, just fail immediately. Used
-     * only together with BDRV_REQ_SERIALISING.
+     * only together with BDRV_REQ_SERIALISING. Used only with requests aligned
+     * to request_alignment (corresponding assertions are in block/io.c).
      */
     BDRV_REQ_NO_WAIT = 0x400,
 
diff --git a/block/io.c b/block/io.c
index XXXXXXX..XXXXXXX 100644
--- a/block/io.c
+++ b/block/io.c
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_do_zero_pwritev(BdrvChild *child,
 
     padding = bdrv_init_padding(bs, offset, bytes, &pad);
     if (padding) {
+        assert(!(flags & BDRV_REQ_NO_WAIT));
         bdrv_make_request_serialising(req, align);
 
         bdrv_padding_rmw_read(child, req, &pad, true);
@@ -XXX,XX +XXX,XX @@ int coroutine_fn bdrv_co_pwritev_part(BdrvChild *child,
          * serialize the request to prevent interactions of the
          * widened region with other transactions.
          */
+        assert(!(flags & BDRV_REQ_NO_WAIT));
         bdrv_make_request_serialising(&req, align);
         bdrv_padding_rmw_read(child, &req, &pad, false);
     }
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_copy_range_internal(
     /* TODO We can support BDRV_REQ_NO_FALLBACK here */
     assert(!(read_flags & BDRV_REQ_NO_FALLBACK));
     assert(!(write_flags & BDRV_REQ_NO_FALLBACK));
+    assert(!(read_flags & BDRV_REQ_NO_WAIT));
+    assert(!(write_flags & BDRV_REQ_NO_WAIT));
 
     if (!dst || !dst->bs || !bdrv_is_inserted(dst->bs)) {
         return -ENOMEDIUM;
diff --git a/block/preallocate.c b/block/preallocate.c
index XXXXXXX..XXXXXXX 100644
--- a/block/preallocate.c
+++ b/block/preallocate.c
@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn handle_write(BlockDriverState *bs, int64_t offset,
     int64_t end = offset + bytes;
     int64_t prealloc_start, prealloc_end;
     int ret;
+    uint32_t file_align = bs->file->bs->bl.request_alignment;
+    uint32_t prealloc_align = MAX(s->opts.prealloc_align, file_align);
+
+    assert(QEMU_IS_ALIGNED(prealloc_align, file_align));
 
     if (!has_prealloc_perms(bs)) {
         /* We don't have state neither should try to recover it */
@@ -XXX,XX +XXX,XX @@ static bool coroutine_fn handle_write(BlockDriverState *bs, int64_t offset,
 
     /* Now we want new preallocation, as request writes beyond s->file_end. */
 
-    prealloc_start = want_merge_zero ? MIN(offset, s->file_end) : s->file_end;
-    prealloc_end = QEMU_ALIGN_UP(end + s->opts.prealloc_size,
-                                 s->opts.prealloc_align);
+    prealloc_start = QEMU_ALIGN_UP(
+            want_merge_zero ? MIN(offset, s->file_end) : s->file_end,
+            file_align);
+    prealloc_end = QEMU_ALIGN_UP(
+            MAX(prealloc_start, end) + s->opts.prealloc_size,
+            prealloc_align);
+
+    want_merge_zero = want_merge_zero && (prealloc_start <= offset);
 
     ret = bdrv_co_pwrite_zeroes(
             bs->file, prealloc_start, prealloc_end - prealloc_start,
-- 
2.34.1

From: Thomas Huth <thuth@redhat.com>

iotest 040 already has some checks for the availability of the 'throttle'
driver, but some new code has been added in the course of time that
depends on 'throttle' but does not check for its availability. Add
a check to the TestCommitWithFilters class so that this iotest now
also passes again if 'throttle' has not been enabled in the QEMU
binaries.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220223123127.3206042-1-thuth@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/040 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -XXX,XX +XXX,XX @@ class TestCommitWithFilters(iotests.QMPTestCase):
                              pattern_file)
             self.assertFalse('Pattern verification failed' in result)
 
+    @iotests.skip_if_unsupported(['throttle'])
     def setUp(self):
         qemu_img('create', '-f', iotests.imgfmt, self.img0, '64M')
         qemu_img('create', '-f', iotests.imgfmt, self.img1, '64M')
-- 
2.34.1

From: Thomas Huth <thuth@redhat.com>

In TAP mode, the stdout is reserved for the TAP protocol, so we
have to make sure to mark other lines with a comment '#' character
at the beginning to avoid that the TAP parser at the other end
gets confused.

To test this condition, run "configure" for example with:

--block-drv-rw-whitelist=copy-before-write,qcow2,raw,file,host_device,blkdebug,null-co,copy-on-read

so that iotest 041 will report that some tests are not run due to
the missing "quorum" driver. Without this change, "make check-block"
fails since the meson tap parser gets confused by these messages.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220223124353.3273898-1-thuth@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/testrunner.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -XXX,XX +XXX,XX @@ def run_test(self, test: str,
                                  description=res.description)
 
         if res.casenotrun:
-            print(res.casenotrun)
+            if self.tap:
+                print('#' + res.casenotrun.replace('\n', '\n#'))
+            else:
+                print(res.casenotrun)
 
         return res
 
-- 
2.34.1

Drop the use of OUTPUT_DIR (test/qemu-iotests under the build
directory), and instead write test output files (.out.bad, .notrun, and
.casenotrun) to TEST_DIR.

With this, the same test can be run concurrently without the separate
instances interfering, because they will need separate TEST_DIRs anyway.
Running the same test separately is useful when running the iotests with
various format/protocol combinations in parallel, or when you just want
to aggressively exercise a single test (e.g. when it fails only
sporadically).

Putting this output into TEST_DIR means that it will stick around for
inspection after the test run is done (though running the same test in
the same TEST_DIR will overwrite it, just as it used to be); but given
that TEST_DIR is a scratch directory, it should be clear that users can
delete all of its content at any point.  (And if TEST_DIR is on tmpfs,
it will just disappear on shutdown.)  Contrarily, alternative approaches
that would put these output files into OUTPUT_DIR with some prefix to
differentiate between separate test runs might easily lead to cluttering
OUTPUT_DIR.

(This change means OUTPUT_DIR is no longer written to by the iotests, so
we can drop its usage altogether.)

Signed-off-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220221172909.762858-1-hreitz@redhat.com>
[hreitz: Simplified `Path(os.path.join(x, y))` to `Path(x, y)`, as
         suggested by Vladimir; and rebased on 9086c7639822b6
         ("tests/qemu-iotests: Rework the checks and spots using GNU
         sed")]
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/common.rc     |  6 +++---
 tests/qemu-iotests/iotests.py    |  5 ++---
 tests/qemu-iotests/testenv.py    |  5 +----
 tests/qemu-iotests/testrunner.py | 14 ++++++++------
 4 files changed, 14 insertions(+), 16 deletions(-)

diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -XXX,XX +XXX,XX @@
 # bail out, setting up .notrun file
 _notrun()
 {
-    echo "$*" >"$OUTPUT_DIR/$seq.notrun"
+    echo "$*" >"$TEST_DIR/$seq.notrun"
     echo "$seq not run: $*"
     status=0
     exit
@@ -XXX,XX +XXX,XX @@ _img_info()
 #
 _casenotrun()
 {
-    echo "    [case not run] $*" >>"$OUTPUT_DIR/$seq.casenotrun"
+    echo "    [case not run] $*" >>"$TEST_DIR/$seq.casenotrun"
 }
 
 # just plain bail out
 #
 _fail()
 {
-    echo "$*" | tee -a "$OUTPUT_DIR/$seq.full"
+    echo "$*" | tee -a "$TEST_DIR/$seq.full"
     echo "(see $seq.full for details)"
     status=1
     exit 1
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -XXX,XX +XXX,XX @@
 
 imgfmt = os.environ.get('IMGFMT', 'raw')
 imgproto = os.environ.get('IMGPROTO', 'file')
-output_dir = os.environ.get('OUTPUT_DIR', '.')
 
 try:
     test_dir = os.environ['TEST_DIR']
@@ -XXX,XX +XXX,XX @@ def notrun(reason):
     # Each test in qemu-iotests has a number ("seq")
     seq = os.path.basename(sys.argv[0])
 
-    with open('%s/%s.notrun' % (output_dir, seq), 'w', encoding='utf-8') \
+    with open('%s/%s.notrun' % (test_dir, seq), 'w', encoding='utf-8') \
             as outfile:
         outfile.write(reason + '\n')
     logger.warning("%s not run: %s", seq, reason)
@@ -XXX,XX +XXX,XX @@ def case_notrun(reason):
     # Each test in qemu-iotests has a number ("seq")
     seq = os.path.basename(sys.argv[0])
 
-    with open('%s/%s.casenotrun' % (output_dir, seq), 'a', encoding='utf-8') \
+    with open('%s/%s.casenotrun' % (test_dir, seq), 'a', encoding='utf-8') \
             as outfile:
         outfile.write('    [case not run] ' + reason + '\n')
 
diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testenv.py
+++ b/tests/qemu-iotests/testenv.py
@@ -XXX,XX +XXX,XX @@ class TestEnv(ContextManager['TestEnv']):
     # pylint: disable=too-many-instance-attributes
 
     env_variables = ['PYTHONPATH', 'TEST_DIR', 'SOCK_DIR', 'SAMPLE_IMG_DIR',
-                     'OUTPUT_DIR', 'PYTHON', 'QEMU_PROG', 'QEMU_IMG_PROG',
+                     'PYTHON', 'QEMU_PROG', 'QEMU_IMG_PROG',
                      'QEMU_IO_PROG', 'QEMU_NBD_PROG', 'QSD_PROG',
                      'QEMU_OPTIONS', 'QEMU_IMG_OPTIONS',
                      'QEMU_IO_OPTIONS', 'QEMU_IO_OPTIONS_NO_FMT',
@@ -XXX,XX +XXX,XX @@ def init_directories(self) -> None:
              TEST_DIR
              SOCK_DIR
              SAMPLE_IMG_DIR
-             OUTPUT_DIR
         """
 
         # Path where qemu goodies live in this source tree.
@@ -XXX,XX +XXX,XX @@ def init_directories(self) -> None:
                                         os.path.join(self.source_iotests,
                                                      'sample_images'))
 
-        self.output_dir = os.getcwd()  # OUTPUT_DIR
-
     def init_binaries(self) -> None:
         """Init binary path variables:
              PYTHON (for bash tests)
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
         """
 
         f_test = Path(test)
-        f_bad = Path(f_test.name + '.out.bad')
-        f_notrun = Path(f_test.name + '.notrun')
-        f_casenotrun = Path(f_test.name + '.casenotrun')
         f_reference = Path(self.find_reference(test))
 
         if not f_test.exists():
@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
                               description='No qualified output '
                                           f'(expected {f_reference})')
 
-        for p in (f_bad, f_notrun, f_casenotrun):
-            silent_unlink(p)
-
         args = [str(f_test.resolve())]
         env = self.env.prepare_subprocess(args)
         if mp:
@@ -XXX,XX +XXX,XX @@ def do_run_test(self, test: str, mp: bool) -> TestResult:
                 env[d] = os.path.join(env[d], f_test.name)
                 Path(env[d]).mkdir(parents=True, exist_ok=True)
 
+        test_dir = env['TEST_DIR']
+        f_bad = Path(test_dir, f_test.name + '.out.bad')
+        f_notrun = Path(test_dir, f_test.name + '.notrun')
+        f_casenotrun = Path(test_dir, f_test.name + '.casenotrun')
+
+        for p in (f_notrun, f_casenotrun):
+            silent_unlink(p)
+
         t0 = time.time()
         with f_bad.open('w', encoding="utf-8") as f:
             with subprocess.Popen(args, cwd=str(f_test.parent), env=env,
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

We are going to complicate bitmap initialization in the further
commit. And in future, backup job will be able to work without filter
(when source is immutable), so we'll need same bitmap initialization in
copy-before-write filter and in backup job. So, it's reasonable to do
it in block-copy.

Note that for now cbw_open() is the only caller of
block_copy_state_new().

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-2-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 block/block-copy.c        | 1 +
 block/copy-before-write.c | 4 ----
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/block/block-copy.c b/block/block-copy.c
index XXXXXXX..XXXXXXX 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
         return NULL;
     }
     bdrv_disable_dirty_bitmap(copy_bitmap);
+    bdrv_set_dirty_bitmap(copy_bitmap, 0, bdrv_dirty_bitmap_size(copy_bitmap));
 
     /*
      * If source is in backing chain of target assume that target is going to be
diff --git a/block/copy-before-write.c b/block/copy-before-write.c
index XXXXXXX..XXXXXXX 100644
--- a/block/copy-before-write.c
+++ b/block/copy-before-write.c
@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
                     Error **errp)
 {
     BDRVCopyBeforeWriteState *s = bs->opaque;
-    BdrvDirtyBitmap *copy_bitmap;
 
     bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
                                BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
-    copy_bitmap = block_copy_dirty_bitmap(s->bcs);
-    bdrv_set_dirty_bitmap(copy_bitmap, 0, bdrv_dirty_bitmap_size(copy_bitmap));
-
     return 0;
 }
 
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

That simplifies handling failure in existing code and in further new
usage of bdrv_merge_dirty_bitmap().

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-3-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/dirty-bitmap.h    | 2 +-
 block/dirty-bitmap.c            | 9 +++++++--
 block/monitor/bitmap-qmp-cmds.c | 5 +----
 3 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -XXX,XX +XXX,XX @@ void bdrv_dirty_bitmap_set_persistence(BdrvDirtyBitmap *bitmap,
                                        bool persistent);
 void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap);
 void bdrv_dirty_bitmap_set_busy(BdrvDirtyBitmap *bitmap, bool busy);
-void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
+bool bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
                              HBitmap **backup, Error **errp);
 void bdrv_dirty_bitmap_skip_store(BdrvDirtyBitmap *bitmap, bool skip);
 bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index XXXXXXX..XXXXXXX 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -XXX,XX +XXX,XX @@ bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
  * Ensures permissions on bitmaps are reasonable; use for public API.
  *
  * @backup: If provided, make a copy of dest here prior to merge.
+ *
+ * Returns true on success, false on failure. In case of failure bitmaps are
+ * untouched.
  */
-void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
+bool bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
                              HBitmap **backup, Error **errp)
 {
-    bool ret;
+    bool ret = false;
 
     bdrv_dirty_bitmaps_lock(dest->bs);
     if (src->bs != dest->bs) {
@@ -XXX,XX +XXX,XX @@ out:
     if (src->bs != dest->bs) {
         bdrv_dirty_bitmaps_unlock(src->bs);
     }
+
+    return ret;
 }
 
 /**
diff --git a/block/monitor/bitmap-qmp-cmds.c b/block/monitor/bitmap-qmp-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/block/monitor/bitmap-qmp-cmds.c
+++ b/block/monitor/bitmap-qmp-cmds.c
@@ -XXX,XX +XXX,XX @@ BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
     BlockDriverState *bs;
     BdrvDirtyBitmap *dst, *src, *anon;
     BlockDirtyBitmapMergeSourceList *lst;
-    Error *local_err = NULL;
 
     GLOBAL_STATE_CODE();
 
@@ -XXX,XX +XXX,XX @@ BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
             abort();
         }
 
-        bdrv_merge_dirty_bitmap(anon, src, NULL, &local_err);
-        if (local_err) {
-            error_propagate(errp, local_err);
+        if (!bdrv_merge_dirty_bitmap(anon, src, NULL, errp)) {
             dst = NULL;
             goto out;
         }
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

This will be used in the following commit to bring "incremental" mode
to copy-before-write filter.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-4-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/block-copy.h |  1 +
 block/block-copy.c         | 14 +++++++++++++-
 block/copy-before-write.c  |  2 +-
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState BlockCopyState;
 typedef struct BlockCopyCallState BlockCopyCallState;
 
 BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
+                                     const BdrvDirtyBitmap *bitmap,
                                      Error **errp);
 
 /* Function should be called prior any actual copy request */
diff --git a/block/block-copy.c b/block/block-copy.c
index XXXXXXX..XXXXXXX 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -XXX,XX +XXX,XX @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target,
 }
 
 BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
+                                     const BdrvDirtyBitmap *bitmap,
                                      Error **errp)
 {
+    ERRP_GUARD();
     BlockCopyState *s;
     int64_t cluster_size;
     BdrvDirtyBitmap *copy_bitmap;
@@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
         return NULL;
     }
     bdrv_disable_dirty_bitmap(copy_bitmap);
-    bdrv_set_dirty_bitmap(copy_bitmap, 0, bdrv_dirty_bitmap_size(copy_bitmap));
+    if (bitmap) {
+        if (!bdrv_merge_dirty_bitmap(copy_bitmap, bitmap, NULL, errp)) {
+            error_prepend(errp, "Failed to merge bitmap '%s' to internal "
+                          "copy-bitmap: ", bdrv_dirty_bitmap_name(bitmap));
+            bdrv_release_dirty_bitmap(copy_bitmap);
+            return NULL;
+        }
+    } else {
+        bdrv_set_dirty_bitmap(copy_bitmap, 0,
+                              bdrv_dirty_bitmap_size(copy_bitmap));
+    }
 
     /*
      * If source is in backing chain of target assume that target is going to be
diff --git a/block/copy-before-write.c b/block/copy-before-write.c
index XXXXXXX..XXXXXXX 100644
--- a/block/copy-before-write.c
+++ b/block/copy-before-write.c
@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
             ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) &
              bs->file->bs->supported_zero_flags);
 
-    s->bcs = block_copy_state_new(bs->file, s->target, errp);
+    s->bcs = block_copy_state_new(bs->file, s->target, NULL, errp);
     if (!s->bcs) {
         error_prepend(errp, "Cannot create block-copy-state: ");
         return -EINVAL;
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

This brings "incremental" mode to copy-before-write filter: user can
specify bitmap so that filter will copy only "dirty" areas.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20220303194349.2304213-5-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 qapi/block-core.json      | 10 +++++++-
 block/copy-before-write.c | 51 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 59 insertions(+), 2 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 #
 # @target: The target for copy-before-write operations.
 #
+# @bitmap: If specified, copy-before-write filter will do
+#          copy-before-write operations only for dirty regions of the
+#          bitmap. Bitmap size must be equal to length of file and
+#          target child of the filter. Note also, that bitmap is used
+#          only to initialize internal bitmap of the process, so further
+#          modifications (or removing) of specified bitmap doesn't
+#          influence the filter. (Since 7.0)
+#
 # Since: 6.2
 ##
 { 'struct': 'BlockdevOptionsCbw',
   'base': 'BlockdevOptionsGenericFormat',
-  'data': { 'target': 'BlockdevRef' } }
+  'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap' } }
 
 ##
 # @BlockdevOptions:
diff --git a/block/copy-before-write.c b/block/copy-before-write.c
index XXXXXXX..XXXXXXX 100644
--- a/block/copy-before-write.c
+++ b/block/copy-before-write.c
@@ -XXX,XX +XXX,XX @@
 
 #include "block/copy-before-write.h"
 
+#include "qapi/qapi-visit-block-core.h"
+
 typedef struct BDRVCopyBeforeWriteState {
     BlockCopyState *bcs;
     BdrvChild *target;
@@ -XXX,XX +XXX,XX @@ static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c,
     }
 }
 
+static bool cbw_parse_bitmap_option(QDict *options, BdrvDirtyBitmap **bitmap,
+                                    Error **errp)
+{
+    QDict *bitmap_qdict = NULL;
+    BlockDirtyBitmap *bmp_param = NULL;
+    Visitor *v = NULL;
+    bool ret = false;
+
+    *bitmap = NULL;
+
+    qdict_extract_subqdict(options, &bitmap_qdict, "bitmap.");
+    if (!qdict_size(bitmap_qdict)) {
+        ret = true;
+        goto out;
+    }
+
+    v = qobject_input_visitor_new_flat_confused(bitmap_qdict, errp);
+    if (!v) {
+        goto out;
+    }
+
+    visit_type_BlockDirtyBitmap(v, NULL, &bmp_param, errp);
+    if (!bmp_param) {
+        goto out;
+    }
+
+    *bitmap = block_dirty_bitmap_lookup(bmp_param->node, bmp_param->name, NULL,
+                                        errp);
+    if (!*bitmap) {
+        goto out;
+    }
+
+    ret = true;
+
+out:
+    qapi_free_BlockDirtyBitmap(bmp_param);
+    visit_free(v);
+    qobject_unref(bitmap_qdict);
+
+    return ret;
+}
+
 static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
                     Error **errp)
 {
     BDRVCopyBeforeWriteState *s = bs->opaque;
+    BdrvDirtyBitmap *bitmap = NULL;
 
     bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
                                BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
+    if (!cbw_parse_bitmap_option(options, &bitmap, errp)) {
+        return -EINVAL;
+    }
+
     bs->total_sectors = bs->file->bs->total_sectors;
     bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
             (BDRV_REQ_FUA & bs->file->bs->supported_write_flags);
@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
             ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) &
              bs->file->bs->supported_zero_flags);
 
-    s->bcs = block_copy_state_new(bs->file, s->target, NULL, errp);
+    s->bcs = block_copy_state_new(bs->file, s->target, bitmap, errp);
     if (!s->bcs) {
         error_prepend(errp, "Cannot create block-copy-state: ");
         return -EINVAL;
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Split block_copy_reset() out of block_copy_reset_unallocated() to be
used separately later.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-6-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/block-copy.h |  1 +
 block/block-copy.c         | 21 +++++++++++++--------
 2 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/include/block/block-copy.h b/include/block/block-copy.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block-copy.h
+++ b/include/block/block-copy.h
@@ -XXX,XX +XXX,XX @@ void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm);
 
 void block_copy_state_free(BlockCopyState *s);
 
+void block_copy_reset(BlockCopyState *s, int64_t offset, int64_t bytes);
 int64_t block_copy_reset_unallocated(BlockCopyState *s,
                                      int64_t offset, int64_t *count);
 
diff --git a/block/block-copy.c b/block/block-copy.c
index XXXXXXX..XXXXXXX 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -XXX,XX +XXX,XX @@ static int block_copy_is_cluster_allocated(BlockCopyState *s, int64_t offset,
     }
 }
 
+void block_copy_reset(BlockCopyState *s, int64_t offset, int64_t bytes)
+{
+    QEMU_LOCK_GUARD(&s->lock);
+
+    bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes);
+    if (s->progress) {
+        progress_set_remaining(s->progress,
+                               bdrv_get_dirty_count(s->copy_bitmap) +
+                               s->in_flight_bytes);
+    }
+}
+
 /*
  * Reset bits in copy_bitmap starting at offset if they represent unallocated
  * data in the image. May reset subsequent contiguous bits.
@@ -XXX,XX +XXX,XX @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,
     bytes = clusters * s->cluster_size;
 
     if (!ret) {
-        qemu_co_mutex_lock(&s->lock);
-        bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes);
-        if (s->progress) {
-            progress_set_remaining(s->progress,
-                                   bdrv_get_dirty_count(s->copy_bitmap) +
-                                   s->in_flight_bytes);
-        }
-        qemu_co_mutex_unlock(&s->lock);
+        block_copy_reset(s, offset, bytes);
     }
 
     *count = bytes;
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Split intersecting-requests functionality out of block-copy to be
reused in copy-before-write filter.

Note: while being here, fix tiny typo in MAINTAINERS.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-7-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/reqlist.h |  67 +++++++++++++++++++++++
 block/block-copy.c      | 116 +++++++++++++---------------------------
 block/reqlist.c         |  76 ++++++++++++++++++++++++++
 MAINTAINERS             |   4 +-
 block/meson.build       |   1 +
 5 files changed, 184 insertions(+), 80 deletions(-)
 create mode 100644 include/block/reqlist.h
 create mode 100644 block/reqlist.c

diff --git a/include/block/reqlist.h b/include/block/reqlist.h
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/include/block/reqlist.h
@@ -XXX,XX +XXX,XX @@
+/*
+ * reqlist API
+ *
+ * Copyright (C) 2013 Proxmox Server Solutions
+ * Copyright (c) 2021 Virtuozzo International GmbH.
+ *
+ * Authors:
+ *  Dietmar Maurer (dietmar@proxmox.com)
+ *  Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef REQLIST_H
+#define REQLIST_H
+
+#include "qemu/coroutine.h"
+
+/*
+ * The API is not thread-safe and shouldn't be. The struct is public to be part
+ * of other structures and protected by third-party locks, see
+ * block/block-copy.c for example.
+ */
+
+typedef struct BlockReq {
+    int64_t offset;
+    int64_t bytes;
+
+    CoQueue wait_queue; /* coroutines blocked on this req */
+    QLIST_ENTRY(BlockReq) list;
+} BlockReq;
+
+typedef QLIST_HEAD(, BlockReq) BlockReqList;
+
+/*
+ * Initialize new request and add it to the list. Caller must be sure that
+ * there are no conflicting requests in the list.
+ */
+void reqlist_init_req(BlockReqList *reqs, BlockReq *req, int64_t offset,
+                      int64_t bytes);
+/* Search for request in the list intersecting with @offset/@bytes area. */
+BlockReq *reqlist_find_conflict(BlockReqList *reqs, int64_t offset,
+                                int64_t bytes);
+
+/*
+ * If there are no intersecting requests return false. Otherwise, wait for the
+ * first found intersecting request to finish and return true.
+ *
+ * @lock is passed to qemu_co_queue_wait()
+ * False return value proves that lock was released at no point.
+ */
+bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
+                                   int64_t bytes, CoMutex *lock);
+
+/*
+ * Shrink request and wake all waiting coroutines (maybe some of them are not
+ * intersecting with shrunk request).
+ */
+void coroutine_fn reqlist_shrink_req(BlockReq *req, int64_t new_bytes);
+
+/*
+ * Remove request and wake all waiting coroutines. Do not release any memory.
+ */
+void coroutine_fn reqlist_remove_req(BlockReq *req);
+
+#endif /* REQLIST_H */
diff --git a/block/block-copy.c b/block/block-copy.c
index XXXXXXX..XXXXXXX 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -XXX,XX +XXX,XX @@
 #include "trace.h"
 #include "qapi/error.h"
 #include "block/block-copy.h"
+#include "block/reqlist.h"
 #include "sysemu/block-backend.h"
 #include "qemu/units.h"
 #include "qemu/coroutine.h"
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyTask {
      */
     BlockCopyState *s;
     BlockCopyCallState *call_state;
-    int64_t offset;
     /*
      * @method can also be set again in the while loop of
      * block_copy_dirty_clusters(), but it is never accessed concurrently
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyTask {
     BlockCopyMethod method;
 
     /*
-     * Fields whose state changes throughout the execution
-     * Protected by lock in BlockCopyState.
+     * Generally, req is protected by lock in BlockCopyState, Still req.offset
+     * is only set on task creation, so may be read concurrently after creation.
+     * req.bytes is changed at most once, and need only protecting the case of
+     * parallel read while updating @bytes value in block_copy_task_shrink().
      */
-    CoQueue wait_queue; /* coroutines blocked on this task */
-    /*
-     * Only protect the case of parallel read while updating @bytes
-     * value in block_copy_task_shrink().
-     */
-    int64_t bytes;
-    QLIST_ENTRY(BlockCopyTask) list;
+    BlockReq req;
 } BlockCopyTask;
 
 static int64_t task_end(BlockCopyTask *task)
 {
-    return task->offset + task->bytes;
+    return task->req.offset + task->req.bytes;
 }
 
 typedef struct BlockCopyState {
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState {
     CoMutex lock;
     int64_t in_flight_bytes;
     BlockCopyMethod method;
-    QLIST_HEAD(, BlockCopyTask) tasks; /* All tasks from all block-copy calls */
+    BlockReqList reqs;
     QLIST_HEAD(, BlockCopyCallState) calls;
     /*
      * skip_unallocated:
@@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState {
     RateLimit rate_limit;
 } BlockCopyState;
 
-/* Called with lock held */
-static BlockCopyTask *find_conflicting_task(BlockCopyState *s,
-                                            int64_t offset, int64_t bytes)
-{
-    BlockCopyTask *t;
-
-    QLIST_FOREACH(t, &s->tasks, list) {
-        if (offset + bytes > t->offset && offset < t->offset + t->bytes) {
-            return t;
-        }
-    }
-
-    return NULL;
-}
-
-/*
- * If there are no intersecting tasks return false. Otherwise, wait for the
- * first found intersecting tasks to finish and return true.
- *
- * Called with lock held. May temporary release the lock.
- * Return value of 0 proves that lock was NOT released.
- */
-static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset,
-                                             int64_t bytes)
-{
-    BlockCopyTask *task = find_conflicting_task(s, offset, bytes);
-
-    if (!task) {
-        return false;
-    }
-
-    qemu_co_queue_wait(&task->wait_queue, &s->lock);
-
-    return true;
-}
-
 /* Called with lock held */
 static int64_t block_copy_chunk_size(BlockCopyState *s)
 {
@@ -XXX,XX +XXX,XX @@ block_copy_task_create(BlockCopyState *s, BlockCopyCallState *call_state,
     bytes = QEMU_ALIGN_UP(bytes, s->cluster_size);
 
     /* region is dirty, so no existent tasks possible in it */
-    assert(!find_conflicting_task(s, offset, bytes));
+    assert(!reqlist_find_conflict(&s->reqs, offset, bytes));
 
     bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes);
     s->in_flight_bytes += bytes;
@@ -XXX,XX +XXX,XX @@ block_copy_task_create(BlockCopyState *s, BlockCopyCallState *call_state,
         .task.func = block_copy_task_entry,
         .s = s,
         .call_state = call_state,
-        .offset = offset,
-        .bytes = bytes,
         .method = s->method,
     };
-    qemu_co_queue_init(&task->wait_queue);
-    QLIST_INSERT_HEAD(&s->tasks, task, list);
+    reqlist_init_req(&s->reqs, &task->req, offset, bytes);
 
     return task;
 }
@@ -XXX,XX +XXX,XX @@ static void coroutine_fn block_copy_task_shrink(BlockCopyTask *task,
                                                 int64_t new_bytes)
 {
     QEMU_LOCK_GUARD(&task->s->lock);
-    if (new_bytes == task->bytes) {
+    if (new_bytes == task->req.bytes) {
         return;
     }
 
-    assert(new_bytes > 0 && new_bytes < task->bytes);
+    assert(new_bytes > 0 && new_bytes < task->req.bytes);
 
-    task->s->in_flight_bytes -= task->bytes - new_bytes;
+    task->s->in_flight_bytes -= task->req.bytes - new_bytes;
     bdrv_set_dirty_bitmap(task->s->copy_bitmap,
-                          task->offset + new_bytes, task->bytes - new_bytes);
+                          task->req.offset + new_bytes,
+                          task->req.bytes - new_bytes);
 
-    task->bytes = new_bytes;
-    qemu_co_queue_restart_all(&task->wait_queue);
+    reqlist_shrink_req(&task->req, new_bytes);
 }
 
 static void coroutine_fn block_copy_task_end(BlockCopyTask *task, int ret)
 {
     QEMU_LOCK_GUARD(&task->s->lock);
-    task->s->in_flight_bytes -= task->bytes;
+    task->s->in_flight_bytes -= task->req.bytes;
     if (ret < 0) {
-        bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->offset, task->bytes);
+        bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->req.offset,
+                              task->req.bytes);
     }
-    QLIST_REMOVE(task, list);
     if (task->s->progress) {
         progress_set_remaining(task->s->progress,
                                bdrv_get_dirty_count(task->s->copy_bitmap) +
                                task->s->in_flight_bytes);
     }
-    qemu_co_queue_restart_all(&task->wait_queue);
+    reqlist_remove_req(&task->req);
 }
 
 void block_copy_state_free(BlockCopyState *s)
@@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,
 
     ratelimit_init(&s->rate_limit);
     qemu_co_mutex_init(&s->lock);
-    QLIST_INIT(&s->tasks);
+    QLIST_INIT(&s->reqs);
     QLIST_INIT(&s->calls);
 
     return s;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_run(AioTaskPool *pool,
 
     aio_task_pool_wait_slot(pool);
     if (aio_task_pool_status(pool) < 0) {
-        co_put_to_shres(task->s->mem, task->bytes);
+        co_put_to_shres(task->s->mem, task->req.bytes);
         block_copy_task_end(task, -ECANCELED);
         g_free(task);
         return -ECANCELED;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task)
     BlockCopyMethod method = t->method;
     int ret;
 
-    ret = block_copy_do_copy(s, t->offset, t->bytes, &method, &error_is_read);
+    ret = block_copy_do_copy(s, t->req.offset, t->req.bytes, &method,
+                             &error_is_read);
 
     WITH_QEMU_LOCK_GUARD(&s->lock) {
         if (s->method == t->method) {
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task)
                 t->call_state->error_is_read = error_is_read;
             }
         } else if (s->progress) {
-            progress_work_done(s->progress, t->bytes);
+            progress_work_done(s->progress, t->req.bytes);
         }
     }
-    co_put_to_shres(s->mem, t->bytes);
+    co_put_to_shres(s->mem, t->req.bytes);
     block_copy_task_end(t, ret);
 
     return ret;
@@ -XXX,XX +XXX,XX @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)
             trace_block_copy_skip_range(s, offset, bytes);
             break;
         }
-        if (task->offset > offset) {
-            trace_block_copy_skip_range(s, offset, task->offset - offset);
+        if (task->req.offset > offset) {
+            trace_block_copy_skip_range(s, offset, task->req.offset - offset);
         }
 
         found_dirty = true;
 
-        ret = block_copy_block_status(s, task->offset, task->bytes,
+        ret = block_copy_block_status(s, task->req.offset, task->req.bytes,
                                       &status_bytes);
         assert(ret >= 0); /* never fail */
-        if (status_bytes < task->bytes) {
+        if (status_bytes < task->req.bytes) {
             block_copy_task_shrink(task, status_bytes);
         }
         if (qatomic_read(&s->skip_unallocated) &&
             !(ret & BDRV_BLOCK_ALLOCATED)) {
             block_copy_task_end(task, 0);
-            trace_block_copy_skip_range(s, task->offset, task->bytes);
+            trace_block_copy_skip_range(s, task->req.offset, task->req.bytes);
             offset = task_end(task);
             bytes = end - offset;
             g_free(task);
@@ -XXX,XX +XXX,XX @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)
             }
         }
 
-        ratelimit_calculate_delay(&s->rate_limit, task->bytes);
+        ratelimit_calculate_delay(&s->rate_limit, task->req.bytes);
 
-        trace_block_copy_process(s, task->offset);
+        trace_block_copy_process(s, task->req.offset);
 
-        co_get_from_shres(s->mem, task->bytes);
+        co_get_from_shres(s->mem, task->req.bytes);
 
         offset = task_end(task);
         bytes = end - offset;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)
                  * Check that there is no task we still need to
                  * wait to complete
                  */
-                ret = block_copy_wait_one(s, call_state->offset,
-                                          call_state->bytes);
+                ret = reqlist_wait_one(&s->reqs, call_state->offset,
+                                       call_state->bytes, &s->lock);
                 if (ret == 0) {
                     /*
                      * No pending tasks, but check again the bitmap in this
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)
                      * between this and the critical section in
                      * block_copy_dirty_clusters().
                      *
-                     * block_copy_wait_one return value 0 also means that it
+                     * reqlist_wait_one return value 0 also means that it
                      * didn't release the lock. So, we are still in the same
                      * critical section, not interrupted by any concurrent
                      * access to state.
diff --git a/block/reqlist.c b/block/reqlist.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/block/reqlist.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * reqlist API
+ *
+ * Copyright (C) 2013 Proxmox Server Solutions
+ * Copyright (c) 2021 Virtuozzo International GmbH.
+ *
+ * Authors:
+ *  Dietmar Maurer (dietmar@proxmox.com)
+ *  Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+
+#include "block/reqlist.h"
+
+void reqlist_init_req(BlockReqList *reqs, BlockReq *req, int64_t offset,
+                      int64_t bytes)
+{
+    assert(!reqlist_find_conflict(reqs, offset, bytes));
+
+    *req = (BlockReq) {
+        .offset = offset,
+        .bytes = bytes,
+    };
+    qemu_co_queue_init(&req->wait_queue);
+    QLIST_INSERT_HEAD(reqs, req, list);
+}
+
+BlockReq *reqlist_find_conflict(BlockReqList *reqs, int64_t offset,
+                                int64_t bytes)
+{
+    BlockReq *r;
+
+    QLIST_FOREACH(r, reqs, list) {
+        if (offset + bytes > r->offset && offset < r->offset + r->bytes) {
+            return r;
+        }
+    }
+
+    return NULL;
+}
+
+bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
+                                   int64_t bytes, CoMutex *lock)
+{
+    BlockReq *r = reqlist_find_conflict(reqs, offset, bytes);
+
+    if (!r) {
+        return false;
+    }
+
+    qemu_co_queue_wait(&r->wait_queue, lock);
+
+    return true;
+}
+
+void coroutine_fn reqlist_shrink_req(BlockReq *req, int64_t new_bytes)
+{
+    if (new_bytes == req->bytes) {
+        return;
+    }
+
+    assert(new_bytes > 0 && new_bytes < req->bytes);
+
+    req->bytes = new_bytes;
+    qemu_co_queue_restart_all(&req->wait_queue);
+}
+
+void coroutine_fn reqlist_remove_req(BlockReq *req)
+{
+    QLIST_REMOVE(req, list);
+    qemu_co_queue_restart_all(&req->wait_queue);
+}
diff --git a/MAINTAINERS b/MAINTAINERS
index XXXXXXX..XXXXXXX 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: block/stream.c
 F: block/mirror.c
 F: qapi/job.json
 F: block/block-copy.c
-F: include/block/block-copy.c
+F: include/block/block-copy.h
+F: block/reqlist.c
+F: include/block/reqlist.h
 F: block/copy-before-write.h
 F: block/copy-before-write.c
 F: include/block/aio_task.h
diff --git a/block/meson.build b/block/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/block/meson.build
+++ b/block/meson.build
@@ -XXX,XX +XXX,XX @@ block_ss.add(files(
   'qcow2.c',
   'quorum.c',
   'raw-format.c',
+  'reqlist.c',
   'snapshot.c',
   'throttle-groups.c',
   'throttle.c',
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Add a convenient function similar with bdrv_block_status() to get
status of dirty bitmap.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-9-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/dirty-bitmap.h |  2 ++
 include/qemu/hbitmap.h       | 12 ++++++++++++
 block/dirty-bitmap.c         |  6 ++++++
 util/hbitmap.c               | 33 +++++++++++++++++++++++++++++++++
 4 files changed, 53 insertions(+)

diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -XXX,XX +XXX,XX @@ int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, int64_t offset,
 bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
         int64_t start, int64_t end, int64_t max_dirty_count,
         int64_t *dirty_start, int64_t *dirty_count);
+bool bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap, int64_t offset,
+                              int64_t bytes, int64_t *count);
 BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
                                                   Error **errp);
 
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -XXX,XX +XXX,XX @@ bool hbitmap_next_dirty_area(const HBitmap *hb, int64_t start, int64_t end,
                              int64_t max_dirty_count,
                              int64_t *dirty_start, int64_t *dirty_count);
 
+/*
+ * bdrv_dirty_bitmap_status:
+ * @hb: The HBitmap to operate on
+ * @start: The bit to start from
+ * @count: Number of bits to proceed
+ * @pnum: Out-parameter. How many bits has same value starting from @start
+ *
+ * Returns true if bitmap is dirty at @start, false otherwise.
+ */
+bool hbitmap_status(const HBitmap *hb, int64_t start, int64_t count,
+                    int64_t *pnum);
+
 /**
  * hbitmap_iter_next:
  * @hbi: HBitmapIter to operate on.
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index XXXXXXX..XXXXXXX 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -XXX,XX +XXX,XX @@ bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
                                    dirty_start, dirty_count);
 }
 
+bool bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap, int64_t offset,
+                              int64_t bytes, int64_t *count)
+{
+    return hbitmap_status(bitmap->bitmap, offset, bytes, count);
+}
+
 /**
  * bdrv_merge_dirty_bitmap: merge src into dest.
  * Ensures permissions on bitmaps are reasonable; use for public API.
diff --git a/util/hbitmap.c b/util/hbitmap.c
index XXXXXXX..XXXXXXX 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -XXX,XX +XXX,XX @@ bool hbitmap_next_dirty_area(const HBitmap *hb, int64_t start, int64_t end,
     return true;
 }
 
+bool hbitmap_status(const HBitmap *hb, int64_t start, int64_t count,
+                    int64_t *pnum)
+{
+    int64_t next_dirty, next_zero;
+
+    assert(start >= 0);
+    assert(count > 0);
+    assert(start + count <= hb->orig_size);
+
+    next_dirty = hbitmap_next_dirty(hb, start, count);
+    if (next_dirty == -1) {
+        *pnum = count;
+        return false;
+    }
+
+    if (next_dirty > start) {
+        *pnum = next_dirty - start;
+        return false;
+    }
+
+    assert(next_dirty == start);
+
+    next_zero = hbitmap_next_zero(hb, start, count);
+    if (next_zero == -1) {
+        *pnum = count;
+        return true;
+    }
+
+    assert(next_zero > start);
+    *pnum = next_zero - start;
+    return false;
+}
+
 bool hbitmap_empty(const HBitmap *hb)
 {
     return hb->count == 0;
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Add function to wait for all intersecting requests.
To be used in the further commit.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Nikita Lapshin <nikita.lapshin@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-10-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/reqlist.h | 8 ++++++++
 block/reqlist.c         | 8 ++++++++
 2 files changed, 16 insertions(+)

diff --git a/include/block/reqlist.h b/include/block/reqlist.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/reqlist.h
+++ b/include/block/reqlist.h
@@ -XXX,XX +XXX,XX @@ BlockReq *reqlist_find_conflict(BlockReqList *reqs, int64_t offset,
 bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
                                    int64_t bytes, CoMutex *lock);
 
+/*
+ * Wait for all intersecting requests. It just calls reqlist_wait_one() in a
+ * loop, caller is responsible to stop producing new requests in this region
+ * in parallel, otherwise reqlist_wait_all() may never return.
+ */
+void coroutine_fn reqlist_wait_all(BlockReqList *reqs, int64_t offset,
+                                   int64_t bytes, CoMutex *lock);
+
 /*
  * Shrink request and wake all waiting coroutines (maybe some of them are not
  * intersecting with shrunk request).
diff --git a/block/reqlist.c b/block/reqlist.c
index XXXXXXX..XXXXXXX 100644
--- a/block/reqlist.c
+++ b/block/reqlist.c
@@ -XXX,XX +XXX,XX @@ bool coroutine_fn reqlist_wait_one(BlockReqList *reqs, int64_t offset,
     return true;
 }
 
+void coroutine_fn reqlist_wait_all(BlockReqList *reqs, int64_t offset,
+                                   int64_t bytes, CoMutex *lock)
+{
+    while (reqlist_wait_one(reqs, offset, bytes, lock)) {
+        /* continue */
+    }
+}
+
 void coroutine_fn reqlist_shrink_req(BlockReq *req, int64_t new_bytes)
 {
     if (new_bytes == req->bytes) {
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Add new block driver handlers and corresponding generic wrappers.
It will be used to allow copy-before-write filter to provide
reach fleecing interface in further commit.

In future this approach may be used to allow reading qcow2 internal
snapshots, for example to export them through NBD.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-11-vsementsov@virtuozzo.com>
[hreitz: Rebased on block GS/IO split]
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 include/block/block_int-common.h | 18 ++++++++
 include/block/block_int-io.h     |  9 ++++
 block/io.c                       | 72 ++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)

diff --git a/include/block/block_int-common.h b/include/block/block_int-common.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block_int-common.h
+++ b/include/block/block_int-common.h
@@ -XXX,XX +XXX,XX @@ struct BlockDriver {
         bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum,
         int64_t *map, BlockDriverState **file);
 
+    /*
+     * Snapshot-access API.
+     *
+     * Block-driver may provide snapshot-access API: special functions to access
+     * some internal "snapshot". The functions are similar with normal
+     * read/block_status/discard handler, but don't have any specific handling
+     * in generic block-layer: no serializing, no alignment, no tracked
+     * requests. So, block-driver that realizes these APIs is fully responsible
+     * for synchronization between snapshot-access API and normal IO requests.
+     */
+    int coroutine_fn (*bdrv_co_preadv_snapshot)(BlockDriverState *bs,
+        int64_t offset, int64_t bytes, QEMUIOVector *qiov, size_t qiov_offset);
+    int coroutine_fn (*bdrv_co_snapshot_block_status)(BlockDriverState *bs,
+        bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum,
+        int64_t *map, BlockDriverState **file);
+    int coroutine_fn (*bdrv_co_pdiscard_snapshot)(BlockDriverState *bs,
+        int64_t offset, int64_t bytes);
+
     /*
      * Invalidate any cached meta-data.
      */
diff --git a/include/block/block_int-io.h b/include/block/block_int-io.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block_int-io.h
+++ b/include/block/block_int-io.h
@@ -XXX,XX +XXX,XX @@
  * the I/O API.
  */
 
+int coroutine_fn bdrv_co_preadv_snapshot(BdrvChild *child,
+    int64_t offset, int64_t bytes, QEMUIOVector *qiov, size_t qiov_offset);
+int coroutine_fn bdrv_co_snapshot_block_status(BlockDriverState *bs,
+    bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum,
+    int64_t *map, BlockDriverState **file);
+int coroutine_fn bdrv_co_pdiscard_snapshot(BlockDriverState *bs,
+    int64_t offset, int64_t bytes);
+
+
 int coroutine_fn bdrv_co_preadv(BdrvChild *child,
     int64_t offset, int64_t bytes, QEMUIOVector *qiov,
     BdrvRequestFlags flags);
diff --git a/block/io.c b/block/io.c
index XXXXXXX..XXXXXXX 100644
--- a/block/io.c
+++ b/block/io.c
@@ -XXX,XX +XXX,XX @@ void bdrv_cancel_in_flight(BlockDriverState *bs)
         bs->drv->bdrv_cancel_in_flight(bs);
     }
 }
+
+int coroutine_fn
+bdrv_co_preadv_snapshot(BdrvChild *child, int64_t offset, int64_t bytes,
+                        QEMUIOVector *qiov, size_t qiov_offset)
+{
+    BlockDriverState *bs = child->bs;
+    BlockDriver *drv = bs->drv;
+    int ret;
+    IO_CODE();
+
+    if (!drv) {
+        return -ENOMEDIUM;
+    }
+
+    if (!drv->bdrv_co_preadv_snapshot) {
+        return -ENOTSUP;
+    }
+
+    bdrv_inc_in_flight(bs);
+    ret = drv->bdrv_co_preadv_snapshot(bs, offset, bytes, qiov, qiov_offset);
+    bdrv_dec_in_flight(bs);
+
+    return ret;
+}
+
+int coroutine_fn
+bdrv_co_snapshot_block_status(BlockDriverState *bs,
+                              bool want_zero, int64_t offset, int64_t bytes,
+                              int64_t *pnum, int64_t *map,
+                              BlockDriverState **file)
+{
+    BlockDriver *drv = bs->drv;
+    int ret;
+    IO_CODE();
+
+    if (!drv) {
+        return -ENOMEDIUM;
+    }
+
+    if (!drv->bdrv_co_snapshot_block_status) {
+        return -ENOTSUP;
+    }
+
+    bdrv_inc_in_flight(bs);
+    ret = drv->bdrv_co_snapshot_block_status(bs, want_zero, offset, bytes,
+                                             pnum, map, file);
+    bdrv_dec_in_flight(bs);
+
+    return ret;
+}
+
+int coroutine_fn
+bdrv_co_pdiscard_snapshot(BlockDriverState *bs, int64_t offset, int64_t bytes)
+{
+    BlockDriver *drv = bs->drv;
+    int ret;
+    IO_CODE();
+
+    if (!drv) {
+        return -ENOMEDIUM;
+    }
+
+    if (!drv->bdrv_co_pdiscard_snapshot) {
+        return -ENOTSUP;
+    }
+
+    bdrv_inc_in_flight(bs);
+    ret = drv->bdrv_co_pdiscard_snapshot(bs, offset, bytes);
+    bdrv_dec_in_flight(bs);
+
+    return ret;
+}
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

The new block driver simply utilizes snapshot-access API of underlying
block node.

In further patches we want to use it like this:

This way, NBD client will be able to read snapshotted state of active
disk, when active disk is continued to be written by guest. This is
known as "fleecing", and currently uses another scheme based on qcow2
temporary image which backing file is active-disk. New scheme comes
with benefits - see next commit.

The other possible application is exporting internal snapshots of
qcow2, like this:

[guest]          [NBD export]
   |                  |
   | root             | root
   v       file       v
[qcow2]<---------[snapshot-access]

For this, we'll need to implement snapshot-access API handlers in
qcow2 driver, and improve snapshot-access block driver (and API) to
make it possible to select snapshot by name. Another thing to improve
is size of snapshot. Now for simplicity we just use size of bs->file,
which is OK for backup, but for qcow2 snapshots export we'll need to
imporve snapshot-access API to get size of snapshot.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20220303194349.2304213-12-vsementsov@virtuozzo.com>
[hreitz: Rebased on block GS/IO split]
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 qapi/block-core.json             |   4 +-
 include/block/block_int-common.h |   6 ++
 block/snapshot-access.c          | 132 +++++++++++++++++++++++++++++++
 MAINTAINERS                      |   1 +
 block/meson.build                |   1 +
 5 files changed, 143 insertions(+), 1 deletion(-)
 create mode 100644 block/snapshot-access.c

diff --git a/qapi/block-core.json b/qapi/block-core.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -XXX,XX +XXX,XX @@
 # @blkreplay: Since 4.2
 # @compress: Since 5.0
 # @copy-before-write: Since 6.2
+# @snapshot-access: Since 7.0
 #
 # Since: 2.9
 ##
 { 'enum': 'BlockdevDriver',
   'data': [ 'blkdebug', 'blklogwrites', 'blkreplay', 'blkverify', 'bochs',
             'cloop', 'compress', 'copy-before-write', 'copy-on-read', 'dmg',
-            'file', 'ftp', 'ftps', 'gluster',
+            'file', 'snapshot-access', 'ftp', 'ftps', 'gluster',
             {'name': 'host_cdrom', 'if': 'HAVE_HOST_BLOCK_DEVICE' },
             {'name': 'host_device', 'if': 'HAVE_HOST_BLOCK_DEVICE' },
             'http', 'https', 'iscsi',
@@ -XXX,XX +XXX,XX @@
       'rbd':        'BlockdevOptionsRbd',
       'replication': { 'type': 'BlockdevOptionsReplication',
                        'if': 'CONFIG_REPLICATION' },
+      'snapshot-access': 'BlockdevOptionsGenericFormat',
       'ssh':        'BlockdevOptionsSsh',
       'throttle':   'BlockdevOptionsThrottle',
       'vdi':        'BlockdevOptionsGenericFormat',
diff --git a/include/block/block_int-common.h b/include/block/block_int-common.h
index XXXXXXX..XXXXXXX 100644
--- a/include/block/block_int-common.h
+++ b/include/block/block_int-common.h
@@ -XXX,XX +XXX,XX @@ struct BlockDriver {
      * in generic block-layer: no serializing, no alignment, no tracked
      * requests. So, block-driver that realizes these APIs is fully responsible
      * for synchronization between snapshot-access API and normal IO requests.
+     *
+     * TODO: To be able to support qcow2's internal snapshots, this API will
+     * need to be extended to:
+     * - be able to select a specific snapshot
+     * - receive the snapshot's actual length (which may differ from bs's
+     *   length)
      */
     int coroutine_fn (*bdrv_co_preadv_snapshot)(BlockDriverState *bs,
         int64_t offset, int64_t bytes, QEMUIOVector *qiov, size_t qiov_offset);
diff --git a/block/snapshot-access.c b/block/snapshot-access.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/block/snapshot-access.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * snapshot_access block driver
+ *
+ * Copyright (c) 2022 Virtuozzo International GmbH.
+ *
+ * Author:
+ *  Sementsov-Ogievskiy Vladimir <vsementsov@virtuozzo.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+
+#include "sysemu/block-backend.h"
+#include "qemu/cutils.h"
+#include "block/block_int.h"
+
+static coroutine_fn int
+snapshot_access_co_preadv_part(BlockDriverState *bs,
+                               int64_t offset, int64_t bytes,
+                               QEMUIOVector *qiov, size_t qiov_offset,
+                               BdrvRequestFlags flags)
+{
+    if (flags) {
+        return -ENOTSUP;
+    }
+
+    return bdrv_co_preadv_snapshot(bs->file, offset, bytes, qiov, qiov_offset);
+}
+
+static int coroutine_fn
+snapshot_access_co_block_status(BlockDriverState *bs,
+                                bool want_zero, int64_t offset,
+                                int64_t bytes, int64_t *pnum,
+                                int64_t *map, BlockDriverState **file)
+{
+    return bdrv_co_snapshot_block_status(bs->file->bs, want_zero, offset,
+                                         bytes, pnum, map, file);
+}
+
+static int coroutine_fn snapshot_access_co_pdiscard(BlockDriverState *bs,
+                                             int64_t offset, int64_t bytes)
+{
+    return bdrv_co_pdiscard_snapshot(bs->file->bs, offset, bytes);
+}
+
+static int coroutine_fn
+snapshot_access_co_pwrite_zeroes(BlockDriverState *bs,
+                                 int64_t offset, int64_t bytes,
+                                 BdrvRequestFlags flags)
+{
+    return -ENOTSUP;
+}
+
+static coroutine_fn int
+snapshot_access_co_pwritev_part(BlockDriverState *bs,
+                                int64_t offset, int64_t bytes,
+                                QEMUIOVector *qiov, size_t qiov_offset,
+                                BdrvRequestFlags flags)
+{
+    return -ENOTSUP;
+}
+
+
+static void snapshot_access_refresh_filename(BlockDriverState *bs)
+{
+    pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
+            bs->file->bs->filename);
+}
+
+static int snapshot_access_open(BlockDriverState *bs, QDict *options, int flags,
+                                Error **errp)
+{
+    bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
+                               BDRV_CHILD_DATA | BDRV_CHILD_PRIMARY,
+                               false, errp);
+    if (!bs->file) {
+        return -EINVAL;
+    }
+
+    bs->total_sectors = bs->file->bs->total_sectors;
+
+    return 0;
+}
+
+static void snapshot_access_child_perm(BlockDriverState *bs, BdrvChild *c,
+                                BdrvChildRole role,
+                                BlockReopenQueue *reopen_queue,
+                                uint64_t perm, uint64_t shared,
+                                uint64_t *nperm, uint64_t *nshared)
+{
+    /*
+     * Currently, we don't need any permissions. If bs->file provides
+     * snapshot-access API, we can use it.
+     */
+    *nperm = 0;
+    *nshared = BLK_PERM_ALL;
+}
+
+BlockDriver bdrv_snapshot_access_drv = {
+    .format_name = "snapshot-access",
+
+    .bdrv_open                  = snapshot_access_open,
+
+    .bdrv_co_preadv_part        = snapshot_access_co_preadv_part,
+    .bdrv_co_pwritev_part       = snapshot_access_co_pwritev_part,
+    .bdrv_co_pwrite_zeroes      = snapshot_access_co_pwrite_zeroes,
+    .bdrv_co_pdiscard           = snapshot_access_co_pdiscard,
+    .bdrv_co_block_status       = snapshot_access_co_block_status,
+
+    .bdrv_refresh_filename      = snapshot_access_refresh_filename,
+
+    .bdrv_child_perm            = snapshot_access_child_perm,
+};
+
+static void snapshot_access_init(void)
+{
+    bdrv_register(&bdrv_snapshot_access_drv);
+}
+
+block_init(snapshot_access_init);
diff --git a/MAINTAINERS b/MAINTAINERS
index XXXXXXX..XXXXXXX 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -XXX,XX +XXX,XX @@ F: block/reqlist.c
 F: include/block/reqlist.h
 F: block/copy-before-write.h
 F: block/copy-before-write.c
+F: block/snapshot-access.c
 F: include/block/aio_task.h
 F: block/aio_task.c
 F: util/qemu-co-shared-resource.c
diff --git a/block/meson.build b/block/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/block/meson.build
+++ b/block/meson.build
@@ -XXX,XX +XXX,XX @@ block_ss.add(files(
   'raw-format.c',
   'reqlist.c',
   'snapshot.c',
+  'snapshot-access.c',
   'throttle-groups.c',
   'throttle.c',
   'vhdx-endian.c',
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Current scheme of image fleecing looks like this:

- On guest writes copy-before-write filter copies old data from active
   disk to temp.qcow2. So fleecing client (NBD export) when reads
   changed regions from temp.qcow2 image and unchanged from active disk
   through backing link.

This patch makes possible new image fleecing scheme:

- copy-before-write does CBW operations and also provides
   snapshot-access API. The API may be accessed through
   snapshot-access driver.

Benefits of new scheme:

1. Access control: if remote client try to read data that not covered
   by original dirty bitmap used on copy-before-write open, client gets
   -EACCES.

2. Discard support: if remote client do DISCARD, this additionally to
   discarding data in temp.img informs block-copy process to not copy
   these clusters. Next read from discarded area will return -EACCES.
   This is significant thing: when fleecing user reads data that was
   not yet copied to temp.img, we can avoid copying it on further guest
   write.

3. Synchronisation between client reads and block-copy write is more
   efficient. In old scheme we just rely on BDRV_REQ_SERIALISING flag
   used for writes to temp.qcow2. New scheme is less blocking:
     - fleecing reads are never blocked: if data region is untouched or
       in-flight, we just read from active-disk, otherwise we read from
       temp.img
     - writes to temp.img are not blocked by fleecing reads
     - still, guest writes of-course are blocked by in-flight fleecing
       reads, that currently read from active-disk - it's the minimum
       necessary blocking

4. Temporary image may be of any format, as we don't rely on backing
   feature.

5. Permission relation are simplified. With old scheme we have to share
   write permission on target child of copy-before-write, otherwise
   backing link conflicts with copy-before-write file child write
   permissions. With new scheme we don't have backing link, and
   copy-before-write node may have unshared access to temporary node.
   (Not realized in this commit, will be in future).

6. Having control on fleecing reads we'll be able to implement
   alternative behavior on failed copy-before-write operations.
   Currently we just break guest request (that's a historical behavior
   of backup). But in some scenarios it's a bad behavior: better
   is to drop the backup as failed but don't break guest request.
   With new scheme we can simply unset some bits in a bitmap on CBW
   failure and further fleecing reads will -EACCES, or something like
   this. (Not implemented in this commit, will be in future)
   Additional application for this is implementing timeout for CBW
   operations.

Iotest 257 output is updated, as two more bitmaps now live in
copy-before-write filter.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20220303194349.2304213-13-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 block/copy-before-write.c  | 212 ++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/257.out | 224 +++++++++++++++++++++++++++++++++++++
 2 files changed, 435 insertions(+), 1 deletion(-)

diff --git a/block/copy-before-write.c b/block/copy-before-write.c
index XXXXXXX..XXXXXXX 100644
--- a/block/copy-before-write.c
+++ b/block/copy-before-write.c
@@ -XXX,XX +XXX,XX @@
 #include "block/block-copy.h"
 
 #include "block/copy-before-write.h"
+#include "block/reqlist.h"
 
 #include "qapi/qapi-visit-block-core.h"
 
 typedef struct BDRVCopyBeforeWriteState {
     BlockCopyState *bcs;
     BdrvChild *target;
+
+    /*
+     * @lock: protects access to @access_bitmap, @done_bitmap and
+     * @frozen_read_reqs
+     */
+    CoMutex lock;
+
+    /*
+     * @access_bitmap: represents areas allowed for reading by fleecing user.
+     * Reading from non-dirty areas leads to -EACCES.
+     */
+    BdrvDirtyBitmap *access_bitmap;
+
+    /*
+     * @done_bitmap: represents areas that was successfully copied to @target by
+     * copy-before-write operations.
+     */
+    BdrvDirtyBitmap *done_bitmap;
+
+    /*
+     * @frozen_read_reqs: current read requests for fleecing user in bs->file
+     * node. These areas must not be rewritten by guest.
+     */
+    BlockReqList frozen_read_reqs;
 } BDRVCopyBeforeWriteState;
 
 static coroutine_fn int cbw_co_preadv(
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int cbw_co_preadv(
     return bdrv_co_preadv(bs->file, offset, bytes, qiov, flags);
 }
 
+/*
+ * Do copy-before-write operation.
+ *
+ * On failure guest request must be failed too.
+ *
+ * On success, we also wait for all in-flight fleecing read requests in source
+ * node, and it's guaranteed that after cbw_do_copy_before_write() successful
+ * return there are no such requests and they will never appear.
+ */
 static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
         uint64_t offset, uint64_t bytes, BdrvRequestFlags flags)
 {
     BDRVCopyBeforeWriteState *s = bs->opaque;
+    int ret;
     uint64_t off, end;
     int64_t cluster_size = block_copy_cluster_size(s->bcs);
 
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
     off = QEMU_ALIGN_DOWN(offset, cluster_size);
     end = QEMU_ALIGN_UP(offset + bytes, cluster_size);
 
-    return block_copy(s->bcs, off, end - off, true);
+    ret = block_copy(s->bcs, off, end - off, true);
+    if (ret < 0) {
+        return ret;
+    }
+
+    WITH_QEMU_LOCK_GUARD(&s->lock) {
+        bdrv_set_dirty_bitmap(s->done_bitmap, off, end - off);
+        reqlist_wait_all(&s->frozen_read_reqs, off, end - off, &s->lock);
+    }
+
+    return 0;
 }
 
 static int coroutine_fn cbw_co_pdiscard(BlockDriverState *bs,
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn cbw_co_flush(BlockDriverState *bs)
     return bdrv_co_flush(bs->file->bs);
 }
 
+/*
+ * If @offset not accessible - return NULL.
+ *
+ * Otherwise, set @pnum to some bytes that accessible from @file (@file is set
+ * to bs->file or to s->target). Return newly allocated BlockReq object that
+ * should be than passed to cbw_snapshot_read_unlock().
+ *
+ * It's guaranteed that guest writes will not interact in the region until
+ * cbw_snapshot_read_unlock() called.
+ */
+static BlockReq *cbw_snapshot_read_lock(BlockDriverState *bs,
+                                        int64_t offset, int64_t bytes,
+                                        int64_t *pnum, BdrvChild **file)
+{
+    BDRVCopyBeforeWriteState *s = bs->opaque;
+    BlockReq *req = g_new(BlockReq, 1);
+    bool done;
+
+    QEMU_LOCK_GUARD(&s->lock);
+
+    if (bdrv_dirty_bitmap_next_zero(s->access_bitmap, offset, bytes) != -1) {
+        g_free(req);
+        return NULL;
+    }
+
+    done = bdrv_dirty_bitmap_status(s->done_bitmap, offset, bytes, pnum);
+    if (done) {
+        /*
+         * Special invalid BlockReq, that is handled in
+         * cbw_snapshot_read_unlock(). We don't need to lock something to read
+         * from s->target.
+         */
+        *req = (BlockReq) {.offset = -1, .bytes = -1};
+        *file = s->target;
+    } else {
+        reqlist_init_req(&s->frozen_read_reqs, req, offset, bytes);
+        *file = bs->file;
+    }
+
+    return req;
+}
+
+static void cbw_snapshot_read_unlock(BlockDriverState *bs, BlockReq *req)
+{
+    BDRVCopyBeforeWriteState *s = bs->opaque;
+
+    if (req->offset == -1 && req->bytes == -1) {
+        g_free(req);
+        return;
+    }
+
+    QEMU_LOCK_GUARD(&s->lock);
+
+    reqlist_remove_req(req);
+    g_free(req);
+}
+
+static coroutine_fn int
+cbw_co_preadv_snapshot(BlockDriverState *bs, int64_t offset, int64_t bytes,
+                       QEMUIOVector *qiov, size_t qiov_offset)
+{
+    BlockReq *req;
+    BdrvChild *file;
+    int ret;
+
+    /* TODO: upgrade to async loop using AioTask */
+    while (bytes) {
+        int64_t cur_bytes;
+
+        req = cbw_snapshot_read_lock(bs, offset, bytes, &cur_bytes, &file);
+        if (!req) {
+            return -EACCES;
+        }
+
+        ret = bdrv_co_preadv_part(file, offset, cur_bytes,
+                                  qiov, qiov_offset, 0);
+        cbw_snapshot_read_unlock(bs, req);
+        if (ret < 0) {
+            return ret;
+        }
+
+        bytes -= cur_bytes;
+        offset += cur_bytes;
+        qiov_offset += cur_bytes;
+    }
+
+    return 0;
+}
+
+static int coroutine_fn
+cbw_co_snapshot_block_status(BlockDriverState *bs,
+                             bool want_zero, int64_t offset, int64_t bytes,
+                             int64_t *pnum, int64_t *map,
+                             BlockDriverState **file)
+{
+    BDRVCopyBeforeWriteState *s = bs->opaque;
+    BlockReq *req;
+    int ret;
+    int64_t cur_bytes;
+    BdrvChild *child;
+
+    req = cbw_snapshot_read_lock(bs, offset, bytes, &cur_bytes, &child);
+    if (!req) {
+        return -EACCES;
+    }
+
+    ret = bdrv_block_status(child->bs, offset, cur_bytes, pnum, map, file);
+    if (child == s->target) {
+        /*
+         * We refer to s->target only for areas that we've written to it.
+         * And we can not report unallocated blocks in s->target: this will
+         * break generic block-status-above logic, that will go to
+         * copy-before-write filtered child in this case.
+         */
+        assert(ret & BDRV_BLOCK_ALLOCATED);
+    }
+
+    cbw_snapshot_read_unlock(bs, req);
+
+    return ret;
+}
+
+static int coroutine_fn cbw_co_pdiscard_snapshot(BlockDriverState *bs,
+                                                 int64_t offset, int64_t bytes)
+{
+    BDRVCopyBeforeWriteState *s = bs->opaque;
+
+    WITH_QEMU_LOCK_GUARD(&s->lock) {
+        bdrv_reset_dirty_bitmap(s->access_bitmap, offset, bytes);
+    }
+
+    block_copy_reset(s->bcs, offset, bytes);
+
+    return bdrv_co_pdiscard(s->target, offset, bytes);
+}
+
 static void cbw_refresh_filename(BlockDriverState *bs)
 {
     pstrcpy(bs->exact_filename, sizeof(bs->exact_filename),
@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
 {
     BDRVCopyBeforeWriteState *s = bs->opaque;
     BdrvDirtyBitmap *bitmap = NULL;
+    int64_t cluster_size;
 
     bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
                                BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
@@ -XXX,XX +XXX,XX @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
+    cluster_size = block_copy_cluster_size(s->bcs);
+
+    s->done_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp);
+    if (!s->done_bitmap) {
+        return -EINVAL;
+    }
+    bdrv_disable_dirty_bitmap(s->done_bitmap);
+
+    /* s->access_bitmap starts equal to bcs bitmap */
+    s->access_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp);
+    if (!s->access_bitmap) {
+        return -EINVAL;
+    }
+    bdrv_disable_dirty_bitmap(s->access_bitmap);
+    bdrv_dirty_bitmap_merge_internal(s->access_bitmap,
+                                     block_copy_dirty_bitmap(s->bcs), NULL,
+                                     true);
+
+    qemu_co_mutex_init(&s->lock);
+    QLIST_INIT(&s->frozen_read_reqs);
+
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ static void cbw_close(BlockDriverState *bs)
 {
     BDRVCopyBeforeWriteState *s = bs->opaque;
 
+    bdrv_release_dirty_bitmap(s->access_bitmap);
+    bdrv_release_dirty_bitmap(s->done_bitmap);
+
     block_copy_state_free(s->bcs);
     s->bcs = NULL;
 }
@@ -XXX,XX +XXX,XX @@ BlockDriver bdrv_cbw_filter = {
     .bdrv_co_pdiscard           = cbw_co_pdiscard,
     .bdrv_co_flush              = cbw_co_flush,
 
+    .bdrv_co_preadv_snapshot       = cbw_co_preadv_snapshot,
+    .bdrv_co_pdiscard_snapshot     = cbw_co_pdiscard_snapshot,
+    .bdrv_co_snapshot_block_status = cbw_co_snapshot_block_status,
+
     .bdrv_refresh_filename      = cbw_refresh_filename,
 
     .bdrv_child_perm            = cbw_child_perm,
diff --git a/tests/qemu-iotests/257.out b/tests/qemu-iotests/257.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/257.out
+++ b/tests/qemu-iotests/257.out
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
@@ -XXX,XX +XXX,XX @@ write -P0x67 0x3fe0000 0x20000
 {"return": ""}
 {
   "bitmaps": {
+    "backup-top": [
+      {
+        "busy": false,
+        "count": 67108864,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      },
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": false
+      }
+    ],
     "drive0": [
       {
         "busy": false,
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220303194349.2304213-14-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/tests/image-fleecing     | 64 +++++++++++++-----
 tests/qemu-iotests/tests/image-fleecing.out | 74 ++++++++++++++++++++-
 2 files changed, 119 insertions(+), 19 deletions(-)

diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/image-fleecing
+++ b/tests/qemu-iotests/tests/image-fleecing
@@ -XXX,XX +XXX,XX @@ remainder = [('0xd5', '0x108000',  '32k'), # Right-end of partial-left [1]
              ('0xdc', '32M',       '32k'), # Left-end of partial-right [2]
              ('0xcd', '0x3ff0000', '64k')] # patterns[3]
 
-def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
+def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
+            fleece_img_path, nbd_sock_path, vm):
     log('--- Setting up images ---')
     log('')
 
     assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
-    assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
+    if use_snapshot_access_filter:
+        assert use_cbw
+        assert qemu_img('create', '-f', 'raw', fleece_img_path, '64M') == 0
+    else:
+        assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
 
     for p in patterns:
         qemu_io('-f', iotests.imgfmt,
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
     log('')
 
 
-    # create tmp_node backed by src_node
-    log(vm.qmp('blockdev-add', {
-        'driver': 'qcow2',
-        'node-name': tmp_node,
-        'file': {
+    if use_snapshot_access_filter:
+        log(vm.qmp('blockdev-add', {
+            'node-name': tmp_node,
             'driver': 'file',
             'filename': fleece_img_path,
-        },
-        'backing': src_node,
-    }))
+        }))
+    else:
+        # create tmp_node backed by src_node
+        log(vm.qmp('blockdev-add', {
+            'driver': 'qcow2',
+            'node-name': tmp_node,
+            'file': {
+                'driver': 'file',
+                'filename': fleece_img_path,
+            },
+            'backing': src_node,
+        }))
 
     # Establish CBW from source to fleecing node
     if use_cbw:
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
         }))
 
         log(vm.qmp('qom-set', path=qom_path, property='drive', value='fl-cbw'))
+
+        if use_snapshot_access_filter:
+            log(vm.qmp('blockdev-add', {
+                'driver': 'snapshot-access',
+                'node-name': 'fl-access',
+                'file': 'fl-cbw',
+            }))
     else:
         log(vm.qmp('blockdev-backup',
                    job_id='fleecing',
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
                    target=tmp_node,
                    sync='none'))
 
+    export_node = 'fl-access' if use_snapshot_access_filter else tmp_node
+
     log('')
     log('--- Setting up NBD Export ---')
     log('')
 
-    nbd_uri = 'nbd+unix:///%s?socket=%s' % (tmp_node, nbd_sock_path)
+    nbd_uri = 'nbd+unix:///%s?socket=%s' % (export_node, nbd_sock_path)
     log(vm.qmp('nbd-server-start',
                {'addr': {'type': 'unix',
                          'data': {'path': nbd_sock_path}}}))
 
-    log(vm.qmp('nbd-server-add', device=tmp_node))
+    log(vm.qmp('nbd-server-add', device=export_node))
 
     log('')
     log('--- Sanity Check ---')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
     log('--- Cleanup ---')
     log('')
 
+    log(vm.qmp('nbd-server-stop'))
+
     if use_cbw:
+        if use_snapshot_access_filter:
+            log(vm.qmp('blockdev-del', node_name='fl-access'))
         log(vm.qmp('qom-set', path=qom_path, property='drive', value=src_node))
         log(vm.qmp('blockdev-del', node_name='fl-cbw'))
     else:
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
         assert e is not None
         log(e, filters=[iotests.filter_qmp_event])
 
-    log(vm.qmp('nbd-server-stop'))
     log(vm.qmp('blockdev-del', node_name=tmp_node))
     vm.shutdown()
 
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
     log('Done')
 
 
-def test(use_cbw):
+def test(use_cbw, use_snapshot_access_filter):
     with iotests.FilePath('base.img') as base_img_path, \
          iotests.FilePath('fleece.img') as fleece_img_path, \
          iotests.FilePath('nbd.sock',
                           base_dir=iotests.sock_dir) as nbd_sock_path, \
          iotests.VM() as vm:
-        do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm)
+        do_test(use_cbw, use_snapshot_access_filter, base_img_path,
+                fleece_img_path, nbd_sock_path, vm)
 
 
 log('=== Test backup(sync=none) based fleecing ===\n')
-test(False)
+test(False, False)
+
+log('=== Test cbw-filter based fleecing ===\n')
+test(True, False)
 
-log('=== Test filter based fleecing ===\n')
-test(True)
+log('=== Test fleecing-format based fleecing ===\n')
+test(True, True)
diff --git a/tests/qemu-iotests/tests/image-fleecing.out b/tests/qemu-iotests/tests/image-fleecing.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/tests/image-fleecing.out
+++ b/tests/qemu-iotests/tests/image-fleecing.out
@@ -XXX,XX +XXX,XX @@ read -P0 0x3fe0000 64k
 
 --- Cleanup ---
 
+{"return": {}}
 {"return": {}}
 {"data": {"device": "fleecing", "len": 67108864, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {"return": {}}
+
+--- Confirming writes ---
+
+read -P0xab 0 64k
+read -P0xad 0x00f8000 64k
+read -P0x1d 0x2008000 64k
+read -P0xea 0x3fe0000 64k
+read -P0xd5 0x108000 32k
+read -P0xdc 32M 32k
+read -P0xcd 0x3ff0000 64k
+
+Done
+=== Test cbw-filter based fleecing ===
+
+--- Setting up images ---
+
+Done
+
+--- Launching VM ---
+
+Done
+
+--- Setting up Fleecing Graph ---
+
+{"return": {}}
+{"return": {}}
+{"return": {}}
+
+--- Setting up NBD Export ---
+
+{"return": {}}
+{"return": {}}
+
+--- Sanity Check ---
+
+read -P0x5d 0 64k
+read -P0xd5 1M 64k
+read -P0xdc 32M 64k
+read -P0xcd 0x3ff0000 64k
+read -P0 0x00f8000 32k
+read -P0 0x2010000 32k
+read -P0 0x3fe0000 64k
+
+--- Testing COW ---
+
+write -P0xab 0 64k
+{"return": ""}
+write -P0xad 0x00f8000 64k
+{"return": ""}
+write -P0x1d 0x2008000 64k
+{"return": ""}
+write -P0xea 0x3fe0000 64k
+{"return": ""}
+
+--- Verifying Data ---
+
+read -P0x5d 0 64k
+read -P0xd5 1M 64k
+read -P0xdc 32M 64k
+read -P0xcd 0x3ff0000 64k
+read -P0 0x00f8000 32k
+read -P0 0x2010000 32k
+read -P0 0x3fe0000 64k
+
+--- Cleanup ---
+
+{"return": {}}
+{"return": {}}
+{"return": {}}
 {"return": {}}
 
 --- Confirming writes ---
@@ -XXX,XX +XXX,XX @@ read -P0xdc 32M 32k
 read -P0xcd 0x3ff0000 64k
 
 Done
-=== Test filter based fleecing ===
+=== Test fleecing-format based fleecing ===
 
 --- Setting up images ---
 
@@ -XXX,XX +XXX,XX @@ Done
 {"return": {}}
 {"return": {}}
 {"return": {}}
+{"return": {}}
 
 --- Setting up NBD Export ---
 
@@ -XXX,XX +XXX,XX @@ read -P0 0x3fe0000 64k
 {"return": {}}
 {"return": {}}
 {"return": {}}
+{"return": {}}
 
 --- Confirming writes ---
 
-- 
2.34.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Note that reads zero areas (not dirty in the bitmap) fails, that's
correct.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20220303194349.2304213-16-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/tests/image-fleecing     | 38 +++++++---
 tests/qemu-iotests/tests/image-fleecing.out | 84 +++++++++++++++++++++
 2 files changed, 113 insertions(+), 9 deletions(-)

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20220303194349.2304213-17-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
---
 tests/qemu-iotests/tests/image-fleecing     | 125 +++++++++++++++-----
 tests/qemu-iotests/tests/image-fleecing.out |  63 ++++++++++
 2 files changed, 156 insertions(+), 32 deletions(-)

diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/tests/image-fleecing
+++ b/tests/qemu-iotests/tests/image-fleecing
@@ -XXX,XX +XXX,XX @@ remainder = [('0xd5', '0x108000',  '32k'), # Right-end of partial-left [1]
              ('0xdc', '32M',       '32k'), # Left-end of partial-right [2]
              ('0xcd', '0x3ff0000', '64k')] # patterns[3]
 
-def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
-            fleece_img_path, nbd_sock_path, vm,
+def do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
+            fleece_img_path, nbd_sock_path=None,
+            target_img_path=None,
             bitmap=False):
+    push_backup = target_img_path is not None
+    assert (nbd_sock_path is not None) != push_backup
+    if push_backup:
+        assert use_cbw
+
     log('--- Setting up images ---')
     log('')
 
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
     else:
         assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
 
+    if push_backup:
+        assert qemu_img('create', '-f', 'qcow2', target_img_path, '64M') == 0
+
     for p in patterns:
         qemu_io('-f', iotests.imgfmt,
                 '-c', 'write -P%s %s %s' % p, base_img_path)
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
 
     export_node = 'fl-access' if use_snapshot_access_filter else tmp_node
 
-    log('')
-    log('--- Setting up NBD Export ---')
-    log('')
+    if push_backup:
+        log('')
+        log('--- Starting actual backup ---')
+        log('')
 
-    nbd_uri = 'nbd+unix:///%s?socket=%s' % (export_node, nbd_sock_path)
-    log(vm.qmp('nbd-server-start',
-               {'addr': {'type': 'unix',
-                         'data': {'path': nbd_sock_path}}}))
+        log(vm.qmp('blockdev-add', **{
+            'driver': iotests.imgfmt,
+            'node-name': 'target',
+            'file': {
+                'driver': 'file',
+                'filename': target_img_path
+            }
+        }))
+        log(vm.qmp('blockdev-backup', device=export_node,
+                   sync='full', target='target',
+                   job_id='push-backup', speed=1))
+    else:
+        log('')
+        log('--- Setting up NBD Export ---')
+        log('')
 
-    log(vm.qmp('nbd-server-add', device=export_node))
+        nbd_uri = 'nbd+unix:///%s?socket=%s' % (export_node, nbd_sock_path)
+        log(vm.qmp('nbd-server-start',
+                   {'addr': { 'type': 'unix',
+                              'data': { 'path': nbd_sock_path } } }))
 
-    log('')
-    log('--- Sanity Check ---')
-    log('')
+        log(vm.qmp('nbd-server-add', device=export_node))
 
-    for p in patterns + zeroes:
-        cmd = 'read -P%s %s %s' % p
-        log(cmd)
-        out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
-                                           nbd_uri)
-        if ret != 0:
-            print(out)
+        log('')
+        log('--- Sanity Check ---')
+        log('')
+
+        for p in patterns + zeroes:
+            cmd = 'read -P%s %s %s' % p
+            log(cmd)
+            out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
+                                               nbd_uri)
+            if ret != 0:
+                print(out)
 
     log('')
     log('--- Testing COW ---')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
         log(cmd)
         log(vm.hmp_qemu_io(qom_path, cmd, qdev=True))
 
+    if push_backup:
+        # Check that previous operations were done during backup, not after
+        # If backup is already finished, it's possible that it was finished
+        # even before hmp qemu_io write, and we didn't actually test
+        # copy-before-write operation. This should not happen, as we use
+        # speed=1. But worth checking.
+        result = vm.qmp('query-block-jobs')
+        assert len(result['return']) == 1
+
+        result = vm.qmp('block-job-set-speed', device='push-backup', speed=0)
+        assert result == {'return': {}}
+
+        log(vm.event_wait(name='BLOCK_JOB_COMPLETED',
+                          match={'data': {'device': 'push-backup'}}),
+            filters=[iotests.filter_qmp_event])
+        log(vm.qmp('blockdev-del', node_name='target'))
+
     log('')
     log('--- Verifying Data ---')
     log('')
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
     for p in patterns + zeroes:
         cmd = 'read -P%s %s %s' % p
         log(cmd)
-        out, ret = qemu_io_pipe_and_status('-r', '-f', 'raw', '-c', cmd,
-                                           nbd_uri)
+        args = ['-r', '-c', cmd]
+        if push_backup:
+            args += [target_img_path]
+        else:
+            args += ['-f', 'raw', nbd_uri]
+        out, ret = qemu_io_pipe_and_status(*args)
         if ret != 0:
             print(out)
 
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
     log('--- Cleanup ---')
     log('')
 
-    log(vm.qmp('nbd-server-stop'))
+    if not push_backup:
+        log(vm.qmp('nbd-server-stop'))
 
     if use_cbw:
         if use_snapshot_access_filter:
@@ -XXX,XX +XXX,XX @@ def do_test(use_cbw, use_snapshot_access_filter, base_img_path,
     log('Done')
 
 
-def test(use_cbw, use_snapshot_access_filter, bitmap=False):
+def test(use_cbw, use_snapshot_access_filter,
+         nbd_sock_path=None, target_img_path=None, bitmap=False):
     with iotests.FilePath('base.img') as base_img_path, \
          iotests.FilePath('fleece.img') as fleece_img_path, \
-         iotests.FilePath('nbd.sock',
-                          base_dir=iotests.sock_dir) as nbd_sock_path, \
          iotests.VM() as vm:
-        do_test(use_cbw, use_snapshot_access_filter, base_img_path,
-                fleece_img_path, nbd_sock_path, vm, bitmap=bitmap)
+        do_test(vm, use_cbw, use_snapshot_access_filter, base_img_path,
+                fleece_img_path, nbd_sock_path, target_img_path,
+                bitmap=bitmap)
+
+def test_pull(use_cbw, use_snapshot_access_filter, bitmap=False):
+    with iotests.FilePath('nbd.sock',
+                          base_dir=iotests.sock_dir) as nbd_sock_path:
+        test(use_cbw, use_snapshot_access_filter, nbd_sock_path, None,
+             bitmap=bitmap)
+
+def test_push():
+    with iotests.FilePath('target.img') as target_img_path:
+        test(True, True, None, target_img_path)
 
 
 log('=== Test backup(sync=none) based fleecing ===\n')
-test(False, False)
+test_pull(False, False)
 
 log('=== Test cbw-filter based fleecing ===\n')
-test(True, False)
+test_pull(True, False)
 
 log('=== Test fleecing-format based fleecing ===\n')
-test(True, True)
+test_pull(True, True)
 
 log('=== Test fleecing-format based fleecing with bitmap ===\n')
-test(True, True, bitmap=True)
+test_pull(True, True, bitmap=True)
+
+log('=== Test push backup with fleecing ===\n')
+test_push()
diff --git a/tests/qemu-iotests/tests/image-fleecing.out b/tests/qemu-iotests/tests/image-fleecing.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/tests/image-fleecing.out
+++ b/tests/qemu-iotests/tests/image-fleecing.out
@@ -XXX,XX +XXX,XX @@ read -P0xdc 32M 32k
 read -P0xcd 0x3ff0000 64k
 
 Done
+=== Test push backup with fleecing ===
+
+--- Setting up images ---
+
+Done
+
+--- Launching VM ---
+
+Done
+
+--- Setting up Fleecing Graph ---
+
+{"return": {}}
+{"return": {}}
+{"return": {}}
+{"return": {}}
+
+--- Starting actual backup ---
+
+{"return": {}}
+{"return": {}}
+
+--- Testing COW ---
+
+write -P0xab 0 64k
+{"return": ""}
+write -P0xad 0x00f8000 64k
+{"return": ""}
+write -P0x1d 0x2008000 64k
+{"return": ""}
+write -P0xea 0x3fe0000 64k
+{"return": ""}
+{"data": {"device": "push-backup", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": {}}
+
+--- Verifying Data ---
+
+read -P0x5d 0 64k
+read -P0xd5 1M 64k
+read -P0xdc 32M 64k
+read -P0xcd 0x3ff0000 64k
+read -P0 0x00f8000 32k
+read -P0 0x2010000 32k
+read -P0 0x3fe0000 64k
+
+--- Cleanup ---
+
+{"return": {}}
+{"return": {}}
+{"return": {}}
+{"return": {}}
+
+--- Confirming writes ---
+
+read -P0xab 0 64k
+read -P0xad 0x00f8000 64k
+read -P0x1d 0x2008000 64k
+read -P0xea 0x3fe0000 64k
+read -P0xd5 0x108000 32k
+read -P0xdc 32M 32k
+read -P0xcd 0x3ff0000 64k
+
+Done
-- 
2.34.1

The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:

Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24

for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:

iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)

----------------------------------------------------------------
Block patches:
- The SSH block driver now uses libssh instead of libssh2
- The VMDK block driver gets read-only support for the seSparse
  subformat
- Various fixes

---

v2:
- Squashed Pino's fix for pre-0.8 libssh into the libssh patch

----------------------------------------------------------------
Anton Nefedov (1):
  iotest 134: test cluster-misaligned encrypted write

Klaus Birkelund Jensen (1):
  nvme: do not advertise support for unsupported arbitration mechanism

Max Reitz (1):
  iotests: Fix 205 for concurrent runs

Pino Toscano (1):
  ssh: switch from libssh2 to libssh

Sam Eiderman (3):
  vmdk: Fix comment regarding max l1_size coverage
  vmdk: Reduce the max bound for L1 table size
  vmdk: Add read-only support for seSparse snapshots

Vladimir Sementsov-Ogievskiy (1):
  blockdev: enable non-root nodes for transaction drive-backup source

-- 
2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

We forget to enable it for transaction .prepare, while it is already
enabled in do_drive_backup since commit a2d665c1bc362
    "blockdev: loosen restrictions on drive-backup source node"

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 blockdev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/blockdev.c b/blockdev.c
index XXXXXXX..XXXXXXX 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
     assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);
     backup = common->action->u.drive_backup.data;
 
-    bs = qmp_get_root_bs(backup->device, errp);
+    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
     if (!bs) {
         return;
     }
-- 
2.21.0

From: Anton Nefedov <anton.nefedov@virtuozzo.com>

COW (even empty/zero) areas require encryption too

Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Message-id: 20190516143028.81155-1-anton.nefedov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/134     |  9 +++++++++
 tests/qemu-iotests/134.out | 10 ++++++++++
 2 files changed, 19 insertions(+)

diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/134
+++ b/tests/qemu-iotests/134
@@ -XXX,XX +XXX,XX @@ echo
 echo "== reading whole image =="
 $QEMU_IO --object $SECRET -c "read 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
 
+echo
+echo "== rewriting cluster part =="
+$QEMU_IO --object $SECRET -c "write -P 0xb 512 512" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+
+echo
+echo "== verify pattern =="
+$QEMU_IO --object $SECRET -c "read -P 0 0 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+$QEMU_IO --object $SECRET -c "read -P 0xb 512 512"  --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
+
 echo
 echo "== rewriting whole image =="
 $QEMU_IO --object $SECRET -c "write -P 0xa 0 $size" --image-opts $IMGSPEC | _filter_qemu_io | _filter_testdir
diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/134.out
+++ b/tests/qemu-iotests/134.out
@@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.
 read 134217728/134217728 bytes at offset 0
 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
+== rewriting cluster part ==
+wrote 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== verify pattern ==
+read 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 512/512 bytes at offset 512
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
 == rewriting whole image ==
 wrote 134217728/134217728 bytes at offset 0
 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")
extended the l1_size check from VMDK4 to VMDK3 but did not update the
default coverage in the moved comment.

The previous vmdk4 calculation:

(512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB

The added vmdk3 calculation:

(512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB

Adding the calculation of vmdk3 to the comment.

In any case, VMware does not offer virtual disks more than 2TB for
vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is
not implemented yet in qemu.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Liran Alon <liran.alon@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com
Reviewed-by: yuchenlin <yuchenlin@synology.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
         return -EFBIG;
     }
     if (l1_size > 512 * 1024 * 1024) {
-        /* Although with big capacity and small l1_entry_sectors, we can get a
+        /*
+         * Although with big capacity and small l1_entry_sectors, we can get a
          * big l1_size, we don't want unbounded value to allocate the table.
-         * Limit it to 512M, which is 16PB for default cluster and L2 table
-         * size */
+         * Limit it to 512M, which is:
+         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
+         *            cluster size: 64KB, L2 table size: 512 entries
+         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
+         *            cluster size: 512B, L2 table size: 4096 entries
+         */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
     }
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

512M of L1 entries is a very loose bound, only 32M are required to store
the maximal supported VMDK file size of 2TB.

Fixed qemu-iotest 59# - now failure occures before on impossible L1
table size.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Liran Alon <liran.alon@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c               | 13 +++++++------
 tests/qemu-iotests/059.out |  2 +-
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
         error_setg(errp, "Invalid granularity, image may be corrupt");
         return -EFBIG;
     }
-    if (l1_size > 512 * 1024 * 1024) {
+    if (l1_size > 32 * 1024 * 1024) {
         /*
          * Although with big capacity and small l1_entry_sectors, we can get a
          * big l1_size, we don't want unbounded value to allocate the table.
-         * Limit it to 512M, which is:
-         *     16PB - for default "Hosted Sparse Extent" (VMDK4)
-         *            cluster size: 64KB, L2 table size: 512 entries
-         *     1PB  - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)
-         *            cluster size: 512B, L2 table size: 4096 entries
+         * Limit it to 32M, which is enough to store:
+         *     8TB  - for both VMDK3 & VMDK4 with
+         *            minimal cluster size: 512B
+         *            minimal L2 table size: 512 entries
+         *            8 TB is still more than the maximal value supported for
+         *            VMDK3 & VMDK4 which is 2TB.
          */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -XXX,XX +XXX,XX @@ Offset          Length          Mapped to       File
 0x140000000     0x10000         0x50000         TEST_DIR/t-s003.vmdk
 
 === Testing afl image with a very large capacity ===
-qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large
+qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big
 *** done
-- 
2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in
QEMU).

This format was lacking in the following:

* Grain directory (L1) and grain table (L2) entries were 32-bit,
      allowing access to only 2TB (slightly less) of data.
    * The grain size (default) was 512 bytes - leading to data
      fragmentation and many grain tables.
    * For space reclamation purposes, it was necessary to find all the
      grains which are not pointed to by any grain table - so a reverse
      mapping of "offset of grain in vmdk" to "grain table" must be
      constructed - which takes large amounts of CPU/RAM.

The format specification can be found in VMware's documentation:
https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf

In ESXi 6.5, to support snapshot files larger than 2TB, a new format was
introduced: SESparse (Space Efficient).

This format fixes the above issues:

* All entries are now 64-bit.
    * The grain size (default) is 4KB.
    * Grain directory and grain tables are now located at the beginning
      of the file.
      + seSparse format reserves space for all grain tables.
      + Grain tables can be addressed using an index.
      + Grains are located in the end of the file and can also be
        addressed with an index.
      - seSparse vmdks of large disks (64TB) have huge preallocated
        headers - mainly due to L2 tables, even for empty snapshots.
    * The header contains a reverse mapping ("backmap") of "offset of
      grain in vmdk" to "grain table" and a bitmap ("free bitmap") which
      specifies for each grain - whether it is allocated or not.
      Using these data structures we can implement space reclamation
      efficiently.
    * Due to the fact that the header now maintains two mappings:
        * The regular one (grain directory & grain tables)
        * A reverse one (backmap and free bitmap)
      These data structures can lose consistency upon crash and result
      in a corrupted VMDK.
      Therefore, a journal is also added to the VMDK and is replayed
      when the VMware reopens the file after a crash.

Since ESXi 6.7 - SESparse is the only snapshot format available.

Unfortunately, VMware does not provide documentation regarding the new
seSparse format.

This commit is based on black-box research of the seSparse format.
Various in-guest block operations and their effect on the snapshot file
were tested.

The only VMware provided source of information (regarding the underlying
implementation) was a log file on the ESXi:

/var/log/hostd.log

Whenever an seSparse snapshot is created - the log is being populated
with seSparse records.

Relevant log records are of the form:

[...] Const Header:
[...]  constMagic     = 0xcafebabe
[...]  version        = 2.1
[...]  capacity       = 204800
[...]  grainSize      = 8
[...]  grainTableSize = 64
[...]  flags          = 0
[...] Extents:
[...]  Header         : <1 : 1>
[...]  JournalHdr     : <2 : 2>
[...]  Journal        : <2048 : 2048>
[...]  GrainDirectory : <4096 : 2048>
[...]  GrainTables    : <6144 : 2048>
[...]  FreeBitmap     : <8192 : 2048>
[...]  BackMap        : <10240 : 2048>
[...]  Grain          : <12288 : 204800>
[...] Volatile Header:
[...] volatileMagic     = 0xcafecafe
[...] FreeGTNumber      = 0
[...] nextTxnSeqNumber  = 0
[...] replayJournal     = 0

The sizes that are seen in the log file are in sectors.
Extents are of the following format: <offset : size>

This commit is a strict implementation which enforces:
    * magics
    * version number 2.1
    * grain size of 8 sectors  (4KB)
    * grain table size of 64 sectors
    * zero flags
    * extent locations

Additionally, this commit proivdes only a subset of the functionality
offered by seSparse's format:
    * Read-only
    * No journal replay
    * No space reclamation
    * No unmap support

Hence, journal header, journal, free bitmap and backmap extents are
unused, only the "classic" (L1 -> L2 -> data) grain access is
implemented.

However there are several differences in the grain access itself.
Grain directory (L1):
    * Grain directory entries are indexes (not offsets) to grain
      tables.
    * Valid grain directory entries have their highest nibble set to
      0x1.
    * Since grain tables are always located in the beginning of the
      file - the index can fit into 32 bits - so we can use its low
      part if it's valid.
Grain table (L2):
    * Grain table entries are indexes (not offsets) to grains.
    * If the highest nibble of the entry is:
        0x0:
            The grain in not allocated.
            The rest of the bytes are 0.
        0x1:
            The grain is unmapped - guest sees a zero grain.
            The rest of the bits point to the previously mapped grain,
            see 0x3 case.
        0x2:
            The grain is zero.
        0x3:
            The grain is allocated - to get the index calculate:
            ((entry & 0x0fff000000000000) >> 48) |
            ((entry & 0x0000ffffffffffff) << 12)
    * The difference between 0x1 and 0x2 is that 0x1 is an unallocated
      grain which results from the guest using sg_unmap to unmap the
      grain - but the grain itself still exists in the grain extent - a
      space reclamation procedure should delete it.
      Unmapping a zero grain has no effect (0x2 will not change to 0x1)
      but unmapping an unallocated grain will (0x0 to 0x1) - naturally.

In order to implement seSparse some fields had to be changed to support
both 32-bit and 64-bit entry sizes.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>
Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>
Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>
Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>
Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 342 insertions(+), 16 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index XXXXXXX..XXXXXXX 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -XXX,XX +XXX,XX @@ typedef struct {
     uint16_t compressAlgorithm;
 } QEMU_PACKED VMDK4Header;
 
+typedef struct VMDKSESparseConstHeader {
+    uint64_t magic;
+    uint64_t version;
+    uint64_t capacity;
+    uint64_t grain_size;
+    uint64_t grain_table_size;
+    uint64_t flags;
+    uint64_t reserved1;
+    uint64_t reserved2;
+    uint64_t reserved3;
+    uint64_t reserved4;
+    uint64_t volatile_header_offset;
+    uint64_t volatile_header_size;
+    uint64_t journal_header_offset;
+    uint64_t journal_header_size;
+    uint64_t journal_offset;
+    uint64_t journal_size;
+    uint64_t grain_dir_offset;
+    uint64_t grain_dir_size;
+    uint64_t grain_tables_offset;
+    uint64_t grain_tables_size;
+    uint64_t free_bitmap_offset;
+    uint64_t free_bitmap_size;
+    uint64_t backmap_offset;
+    uint64_t backmap_size;
+    uint64_t grains_offset;
+    uint64_t grains_size;
+    uint8_t pad[304];
+} QEMU_PACKED VMDKSESparseConstHeader;
+
+typedef struct VMDKSESparseVolatileHeader {
+    uint64_t magic;
+    uint64_t free_gt_number;
+    uint64_t next_txn_seq_number;
+    uint64_t replay_journal;
+    uint8_t pad[480];
+} QEMU_PACKED VMDKSESparseVolatileHeader;
+
 #define L2_CACHE_SIZE 16
 
 typedef struct VmdkExtent {
@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {
     bool compressed;
     bool has_marker;
     bool has_zero_grain;
+    bool sesparse;
+    uint64_t sesparse_l2_tables_offset;
+    uint64_t sesparse_clusters_offset;
+    int32_t entry_size;
     int version;
     int64_t sectors;
     int64_t end_sector;
     int64_t flat_start_offset;
     int64_t l1_table_offset;
     int64_t l1_backup_table_offset;
-    uint32_t *l1_table;
+    void *l1_table;
     uint32_t *l1_backup_table;
     unsigned int l1_size;
     uint32_t l1_entry_sectors;
 
     unsigned int l2_size;
-    uint32_t *l2_cache;
+    void *l2_cache;
     uint32_t l2_cache_offsets[L2_CACHE_SIZE];
     uint32_t l2_cache_counts[L2_CACHE_SIZE];
 
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
          *            minimal L2 table size: 512 entries
          *            8 TB is still more than the maximal value supported for
          *            VMDK3 & VMDK4 which is 2TB.
+         *     64TB - for "ESXi seSparse Extent"
+         *            minimal cluster size: 512B (default is 4KB)
+         *            L2 table size: 4096 entries (const).
+         *            64TB is more than the maximal value supported for
+         *            seSparse VMDKs (which is slightly less than 64TB)
          */
         error_setg(errp, "L1 size too big");
         return -EFBIG;
@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,
     extent->l2_size = l2_size;
     extent->cluster_sectors = flat ? sectors : cluster_sectors;
     extent->next_cluster_sector = ROUND_UP(nb_sectors, cluster_sectors);
+    extent->entry_size = sizeof(uint32_t);
 
     if (s->num_extents > 1) {
         extent->end_sector = (*(extent - 1)).end_sector + extent->sectors;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
     int i;
 
     /* read the L1 table */
-    l1_size = extent->l1_size * sizeof(uint32_t);
+    l1_size = extent->l1_size * extent->entry_size;
     extent->l1_table = g_try_malloc(l1_size);
     if (l1_size && extent->l1_table == NULL) {
         return -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
         goto fail_l1;
     }
     for (i = 0; i < extent->l1_size; i++) {
-        le32_to_cpus(&extent->l1_table[i]);
+        if (extent->entry_size == sizeof(uint64_t)) {
+            le64_to_cpus((uint64_t *)extent->l1_table + i);
+        } else {
+            assert(extent->entry_size == sizeof(uint32_t));
+            le32_to_cpus((uint32_t *)extent->l1_table + i);
+        }
     }
 
     if (extent->l1_backup_table_offset) {
+        assert(!extent->sesparse);
         extent->l1_backup_table = g_try_malloc(l1_size);
         if (l1_size && extent->l1_backup_table == NULL) {
             ret = -ENOMEM;
@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,
     }
 
     extent->l2_cache =
-        g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE);
+        g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE);
     return 0;
  fail_l1b:
     g_free(extent->l1_backup_table);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,
     return ret;
 }
 
+#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe)
+#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe)
+
+/* Strict checks - format not officially documented */
+static int check_se_sparse_const_header(VMDKSESparseConstHeader *header,
+                                        Error **errp)
+{
+    header->magic = le64_to_cpu(header->magic);
+    header->version = le64_to_cpu(header->version);
+    header->grain_size = le64_to_cpu(header->grain_size);
+    header->grain_table_size = le64_to_cpu(header->grain_table_size);
+    header->flags = le64_to_cpu(header->flags);
+    header->reserved1 = le64_to_cpu(header->reserved1);
+    header->reserved2 = le64_to_cpu(header->reserved2);
+    header->reserved3 = le64_to_cpu(header->reserved3);
+    header->reserved4 = le64_to_cpu(header->reserved4);
+
+    header->volatile_header_offset =
+        le64_to_cpu(header->volatile_header_offset);
+    header->volatile_header_size = le64_to_cpu(header->volatile_header_size);
+
+    header->journal_header_offset = le64_to_cpu(header->journal_header_offset);
+    header->journal_header_size = le64_to_cpu(header->journal_header_size);
+
+    header->journal_offset = le64_to_cpu(header->journal_offset);
+    header->journal_size = le64_to_cpu(header->journal_size);
+
+    header->grain_dir_offset = le64_to_cpu(header->grain_dir_offset);
+    header->grain_dir_size = le64_to_cpu(header->grain_dir_size);
+
+    header->grain_tables_offset = le64_to_cpu(header->grain_tables_offset);
+    header->grain_tables_size = le64_to_cpu(header->grain_tables_size);
+
+    header->free_bitmap_offset = le64_to_cpu(header->free_bitmap_offset);
+    header->free_bitmap_size = le64_to_cpu(header->free_bitmap_size);
+
+    header->backmap_offset = le64_to_cpu(header->backmap_offset);
+    header->backmap_size = le64_to_cpu(header->backmap_size);
+
+    header->grains_offset = le64_to_cpu(header->grains_offset);
+    header->grains_size = le64_to_cpu(header->grains_size);
+
+    if (header->magic != SESPARSE_CONST_HEADER_MAGIC) {
+        error_setg(errp, "Bad const header magic: 0x%016" PRIx64,
+                   header->magic);
+        return -EINVAL;
+    }
+
+    if (header->version != 0x0000000200000001) {
+        error_setg(errp, "Unsupported version: 0x%016" PRIx64,
+                   header->version);
+        return -ENOTSUP;
+    }
+
+    if (header->grain_size != 8) {
+        error_setg(errp, "Unsupported grain size: %" PRIu64,
+                   header->grain_size);
+        return -ENOTSUP;
+    }
+
+    if (header->grain_table_size != 64) {
+        error_setg(errp, "Unsupported grain table size: %" PRIu64,
+                   header->grain_table_size);
+        return -ENOTSUP;
+    }
+
+    if (header->flags != 0) {
+        error_setg(errp, "Unsupported flags: 0x%016" PRIx64,
+                   header->flags);
+        return -ENOTSUP;
+    }
+
+    if (header->reserved1 != 0 || header->reserved2 != 0 ||
+        header->reserved3 != 0 || header->reserved4 != 0) {
+        error_setg(errp, "Unsupported reserved bits:"
+                   " 0x%016" PRIx64 " 0x%016" PRIx64
+                   " 0x%016" PRIx64 " 0x%016" PRIx64,
+                   header->reserved1, header->reserved2,
+                   header->reserved3, header->reserved4);
+        return -ENOTSUP;
+    }
+
+    /* check that padding is 0 */
+    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
+        error_setg(errp, "Unsupported non-zero const header padding");
+        return -ENOTSUP;
+    }
+
+    return 0;
+}
+
+static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *header,
+                                           Error **errp)
+{
+    header->magic = le64_to_cpu(header->magic);
+    header->free_gt_number = le64_to_cpu(header->free_gt_number);
+    header->next_txn_seq_number = le64_to_cpu(header->next_txn_seq_number);
+    header->replay_journal = le64_to_cpu(header->replay_journal);
+
+    if (header->magic != SESPARSE_VOLATILE_HEADER_MAGIC) {
+        error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64,
+                   header->magic);
+        return -EINVAL;
+    }
+
+    if (header->replay_journal) {
+        error_setg(errp, "Image is dirty, Replaying journal not supported");
+        return -ENOTSUP;
+    }
+
+    /* check that padding is 0 */
+    if (!buffer_is_zero(header->pad, sizeof(header->pad))) {
+        error_setg(errp, "Unsupported non-zero volatile header padding");
+        return -ENOTSUP;
+    }
+
+    return 0;
+}
+
+static int vmdk_open_se_sparse(BlockDriverState *bs,
+                               BdrvChild *file,
+                               int flags, Error **errp)
+{
+    int ret;
+    VMDKSESparseConstHeader const_header;
+    VMDKSESparseVolatileHeader volatile_header;
+    VmdkExtent *extent;
+
+    ret = bdrv_apply_auto_read_only(bs,
+            "No write support for seSparse images available", errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    assert(sizeof(const_header) == SECTOR_SIZE);
+
+    ret = bdrv_pread(file, 0, &const_header, sizeof(const_header));
+    if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
+        error_setg_errno(errp, -ret,
+                         "Could not read const header from file '%s'",
+                         file->bs->filename);
+        return ret;
+    }
+
+    /* check const header */
+    ret = check_se_sparse_const_header(&const_header, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    assert(sizeof(volatile_header) == SECTOR_SIZE);
+
+    ret = bdrv_pread(file,
+                     const_header.volatile_header_offset * SECTOR_SIZE,
+                     &volatile_header, sizeof(volatile_header));
+    if (ret < 0) {
+        bdrv_refresh_filename(file->bs);
+        error_setg_errno(errp, -ret,
+                         "Could not read volatile header from file '%s'",
+                         file->bs->filename);
+        return ret;
+    }
+
+    /* check volatile header */
+    ret = check_se_sparse_volatile_header(&volatile_header, errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    ret = vmdk_add_extent(bs, file, false,
+                          const_header.capacity,
+                          const_header.grain_dir_offset * SECTOR_SIZE,
+                          0,
+                          const_header.grain_dir_size *
+                          SECTOR_SIZE / sizeof(uint64_t),
+                          const_header.grain_table_size *
+                          SECTOR_SIZE / sizeof(uint64_t),
+                          const_header.grain_size,
+                          &extent,
+                          errp);
+    if (ret < 0) {
+        return ret;
+    }
+
+    extent->sesparse = true;
+    extent->sesparse_l2_tables_offset = const_header.grain_tables_offset;
+    extent->sesparse_clusters_offset = const_header.grains_offset;
+    extent->entry_size = sizeof(uint64_t);
+
+    ret = vmdk_init_tables(bs, extent, errp);
+    if (ret) {
+        /* free extent allocated by vmdk_add_extent */
+        vmdk_free_last_extent(bs);
+    }
+
+    return ret;
+}
+
 static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
                                QDict *options, Error **errp);
 
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
          * RW [size in sectors] SPARSE "file-name.vmdk"
          * RW [size in sectors] VMFS "file-name.vmdk"
          * RW [size in sectors] VMFSSPARSE "file-name.vmdk"
+         * RW [size in sectors] SESPARSE "file-name.vmdk"
          */
         flat_offset = -1;
         matches = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
 
         if (sectors <= 0 ||
             (strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&
-             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) ||
+             strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") &&
+             strcmp(type, "SESPARSE")) ||
             (strcmp(access, "RW"))) {
             continue;
         }
@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,
                 return ret;
             }
             extent = &s->extents[s->num_extents - 1];
+        } else if (!strcmp(type, "SESPARSE")) {
+            ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);
+            if (ret) {
+                bdrv_unref_child(bs, extent_file);
+                return ret;
+            }
+            extent = &s->extents[s->num_extents - 1];
         } else {
             error_setg(errp, "Unsupported extent type '%s'", type);
             bdrv_unref_child(bs, extent_file);
@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,
     if (strcmp(ct, "monolithicFlat") &&
         strcmp(ct, "vmfs") &&
         strcmp(ct, "vmfsSparse") &&
+        strcmp(ct, "seSparse") &&
         strcmp(ct, "twoGbMaxExtentSparse") &&
         strcmp(ct, "twoGbMaxExtentFlat")) {
         error_setg(errp, "Unsupported image type '%s'", ct);
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
 {
     unsigned int l1_index, l2_offset, l2_index;
     int min_index, i, j;
-    uint32_t min_count, *l2_table;
+    uint32_t min_count;
+    void *l2_table;
     bool zeroed = false;
     int64_t ret;
     int64_t cluster_sector;
+    unsigned int l2_size_bytes = extent->l2_size * extent->entry_size;
 
     if (m_data) {
         m_data->valid = 0;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
     if (l1_index >= extent->l1_size) {
         return VMDK_ERROR;
     }
-    l2_offset = extent->l1_table[l1_index];
+    if (extent->sesparse) {
+        uint64_t l2_offset_u64;
+
+        assert(extent->entry_size == sizeof(uint64_t));
+
+        l2_offset_u64 = ((uint64_t *)extent->l1_table)[l1_index];
+        if (l2_offset_u64 == 0) {
+            l2_offset = 0;
+        } else if ((l2_offset_u64 & 0xffffffff00000000) != 0x1000000000000000) {
+            /*
+             * Top most nibble is 0x1 if grain table is allocated.
+             * strict check - top most 4 bytes must be 0x10000000 since max
+             * supported size is 64TB for disk - so no more than 64TB / 16MB
+             * grain directories which is smaller than uint32,
+             * where 16MB is the only supported default grain table coverage.
+             */
+            return VMDK_ERROR;
+        } else {
+            l2_offset_u64 = l2_offset_u64 & 0x00000000ffffffff;
+            l2_offset_u64 = extent->sesparse_l2_tables_offset +
+                l2_offset_u64 * l2_size_bytes / SECTOR_SIZE;
+            if (l2_offset_u64 > 0x00000000ffffffff) {
+                return VMDK_ERROR;
+            }
+            l2_offset = (unsigned int)(l2_offset_u64);
+        }
+    } else {
+        assert(extent->entry_size == sizeof(uint32_t));
+        l2_offset = ((uint32_t *)extent->l1_table)[l1_index];
+    }
     if (!l2_offset) {
         return VMDK_UNALLOC;
     }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
                     extent->l2_cache_counts[j] >>= 1;
                 }
             }
-            l2_table = extent->l2_cache + (i * extent->l2_size);
+            l2_table = (char *)extent->l2_cache + (i * l2_size_bytes);
             goto found;
         }
     }
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
             min_index = i;
         }
     }
-    l2_table = extent->l2_cache + (min_index * extent->l2_size);
+    l2_table = (char *)extent->l2_cache + (min_index * l2_size_bytes);
     BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD);
     if (bdrv_pread(extent->file,
                 (int64_t)l2_offset * 512,
                 l2_table,
-                extent->l2_size * sizeof(uint32_t)
-            ) != extent->l2_size * sizeof(uint32_t)) {
+                l2_size_bytes
+            ) != l2_size_bytes) {
         return VMDK_ERROR;
     }
 
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
     extent->l2_cache_counts[min_index] = 1;
  found:
     l2_index = ((offset >> 9) / extent->cluster_sectors) % extent->l2_size;
-    cluster_sector = le32_to_cpu(l2_table[l2_index]);
 
-    if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
-        zeroed = true;
+    if (extent->sesparse) {
+        cluster_sector = le64_to_cpu(((uint64_t *)l2_table)[l2_index]);
+        switch (cluster_sector & 0xf000000000000000) {
+        case 0x0000000000000000:
+            /* unallocated grain */
+            if (cluster_sector != 0) {
+                return VMDK_ERROR;
+            }
+            break;
+        case 0x1000000000000000:
+            /* scsi-unmapped grain - fallthrough */
+        case 0x2000000000000000:
+            /* zero grain */
+            zeroed = true;
+            break;
+        case 0x3000000000000000:
+            /* allocated grain */
+            cluster_sector = (((cluster_sector & 0x0fff000000000000) >> 48) |
+                              ((cluster_sector & 0x0000ffffffffffff) << 12));
+            cluster_sector = extent->sesparse_clusters_offset +
+                cluster_sector * extent->cluster_sectors;
+            break;
+        default:
+            return VMDK_ERROR;
+        }
+    } else {
+        cluster_sector = le32_to_cpu(((uint32_t *)l2_table)[l2_index]);
+
+        if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {
+            zeroed = true;
+        }
     }
 
     if (!cluster_sector || zeroed) {
         if (!allocate) {
             return zeroed ? VMDK_ZEROED : VMDK_UNALLOC;
         }
+        assert(!extent->sesparse);
 
         if (extent->next_cluster_sector >= VMDK_EXTENT_MAX_SECTORS) {
             return VMDK_ERROR;
@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,
             m_data->l1_index = l1_index;
             m_data->l2_index = l2_index;
             m_data->l2_offset = l2_offset;
-            m_data->l2_cache_entry = &l2_table[l2_index];
+            m_data->l2_cache_entry = ((uint32_t *)l2_table) + l2_index;
         }
     }
     *cluster_offset = cluster_sector << BDRV_SECTOR_BITS;
@@ -XXX,XX +XXX,XX @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_t offset,
         if (!extent) {
             return -EIO;
         }
+        if (extent->sesparse) {
+            return -ENOTSUP;
+        }
         offset_in_cluster = vmdk_find_offset_in_cluster(extent, offset);
         n_bytes = MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE
                              - offset_in_cluster);
-- 
2.21.0

From: Pino Toscano <ptoscano@redhat.com>

Rewrite the implementation of the ssh block driver to use libssh instead
of libssh2.  The libssh library has various advantages over libssh2:
- easier API for authentication (for example for using ssh-agent)
- easier API for known_hosts handling
- supports newer types of keys in known_hosts

Use APIs/features available in libssh 0.8 conditionally, to support
older versions (which are not recommended though).

Adjust the iotest 207 according to the different error message, and to
find the default key type for localhost (to properly compare the
fingerprint with).
Contributed-by: Max Reitz <mreitz@redhat.com>

Adjust the various Docker/Travis scripts to use libssh when available
instead of libssh2. The mingw/mxe testing is dropped for now, as there
are no packages for it.

Signed-off-by: Pino Toscano <ptoscano@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20190620200840.17655-1-ptoscano@redhat.com
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 configure                                     |  65 +-
 block/Makefile.objs                           |   6 +-
 block/ssh.c                                   | 652 ++++++++++--------
 .travis.yml                                   |   4 +-
 block/trace-events                            |  14 +-
 docs/qemu-block-drivers.texi                  |   2 +-
 .../dockerfiles/debian-win32-cross.docker     |   1 -
 .../dockerfiles/debian-win64-cross.docker     |   1 -
 tests/docker/dockerfiles/fedora.docker        |   4 +-
 tests/docker/dockerfiles/ubuntu.docker        |   2 +-
 tests/docker/dockerfiles/ubuntu1804.docker    |   2 +-
 tests/qemu-iotests/207                        |  54 +-
 tests/qemu-iotests/207.out                    |   2 +-
 13 files changed, 449 insertions(+), 360 deletions(-)

diff --git a/configure b/configure
index XXXXXXX..XXXXXXX 100755
--- a/configure
+++ b/configure
@@ -XXX,XX +XXX,XX @@ auth_pam=""
 vte=""
 virglrenderer=""
 tpm=""
-libssh2=""
+libssh=""
 live_block_migration="yes"
 numa=""
 tcmalloc="no"
@@ -XXX,XX +XXX,XX @@ for opt do
   ;;
   --enable-tpm) tpm="yes"
   ;;
-  --disable-libssh2) libssh2="no"
+  --disable-libssh) libssh="no"
   ;;
-  --enable-libssh2) libssh2="yes"
+  --enable-libssh) libssh="yes"
   ;;
   --disable-live-block-migration) live_block_migration="no"
   ;;
@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:
   coroutine-pool  coroutine freelist (better performance)
   glusterfs       GlusterFS backend
   tpm             TPM support
-  libssh2         ssh block device support
+  libssh          ssh block device support
   numa            libnuma support
   libxml2         for Parallels image format
   tcmalloc        tcmalloc support
@@ -XXX,XX +XXX,XX @@ EOF
 fi
 
 ##########################################
-# libssh2 probe
-min_libssh2_version=1.2.8
-if test "$libssh2" != "no" ; then
-  if $pkg_config --atleast-version=$min_libssh2_version libssh2; then
-    libssh2_cflags=$($pkg_config libssh2 --cflags)
-    libssh2_libs=$($pkg_config libssh2 --libs)
-    libssh2=yes
+# libssh probe
+if test "$libssh" != "no" ; then
+  if $pkg_config --exists libssh; then
+    libssh_cflags=$($pkg_config libssh --cflags)
+    libssh_libs=$($pkg_config libssh --libs)
+    libssh=yes
   else
-    if test "$libssh2" = "yes" ; then
-      error_exit "libssh2 >= $min_libssh2_version required for --enable-libssh2"
+    if test "$libssh" = "yes" ; then
+      error_exit "libssh required for --enable-libssh"
     fi
-    libssh2=no
+    libssh=no
   fi
 fi
 
 ##########################################
-# libssh2_sftp_fsync probe
+# Check for libssh 0.8
+# This is done like this instead of using the LIBSSH_VERSION_* and
+# SSH_VERSION_* macros because some distributions in the past shipped
+# snapshots of the future 0.8 from Git, and those snapshots did not
+# have updated version numbers (still referring to 0.7.0).
 
-if test "$libssh2" = "yes"; then
+if test "$libssh" = "yes"; then
   cat > $TMPC <<EOF
-#include <stdio.h>
-#include <libssh2.h>
-#include <libssh2_sftp.h>
-int main(void) {
-    LIBSSH2_SESSION *session;
-    LIBSSH2_SFTP *sftp;
-    LIBSSH2_SFTP_HANDLE *sftp_handle;
-    session = libssh2_session_init ();
-    sftp = libssh2_sftp_init (session);
-    sftp_handle = libssh2_sftp_open (sftp, "/", 0, 0);
-    libssh2_sftp_fsync (sftp_handle);
-    return 0;
-}
+#include <libssh/libssh.h>
+int main(void) { return ssh_get_server_publickey(NULL, NULL); }
 EOF
-  # libssh2_cflags/libssh2_libs defined in previous test.
-  if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then
-    QEMU_CFLAGS="-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS"
+  if compile_prog "$libssh_cflags" "$libssh_libs"; then
+    libssh_cflags="-DHAVE_LIBSSH_0_8 $libssh_cflags"
   fi
 fi
 
@@ -XXX,XX +XXX,XX @@ echo "GlusterFS support $glusterfs"
 echo "gcov              $gcov_tool"
 echo "gcov enabled      $gcov"
 echo "TPM support       $tpm"
-echo "libssh2 support   $libssh2"
+echo "libssh support    $libssh"
 echo "QOM debugging     $qom_cast_debug"
 echo "Live block migration $live_block_migration"
 echo "lzo support       $lzo"
@@ -XXX,XX +XXX,XX @@ if test "$glusterfs_iocb_has_stat" = "yes" ; then
   echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=y" >> $config_host_mak
 fi
 
-if test "$libssh2" = "yes" ; then
-  echo "CONFIG_LIBSSH2=m" >> $config_host_mak
-  echo "LIBSSH2_CFLAGS=$libssh2_cflags" >> $config_host_mak
-  echo "LIBSSH2_LIBS=$libssh2_libs" >> $config_host_mak
+if test "$libssh" = "yes" ; then
+  echo "CONFIG_LIBSSH=m" >> $config_host_mak
+  echo "LIBSSH_CFLAGS=$libssh_cflags" >> $config_host_mak
+  echo "LIBSSH_LIBS=$libssh_libs" >> $config_host_mak
 fi
 
 if test "$live_block_migration" = "yes" ; then
diff --git a/block/Makefile.objs b/block/Makefile.objs
index XXXXXXX..XXXXXXX 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_CURL) += curl.o
 block-obj-$(CONFIG_RBD) += rbd.o
 block-obj-$(CONFIG_GLUSTERFS) += gluster.o
 block-obj-$(CONFIG_VXHS) += vxhs.o
-block-obj-$(CONFIG_LIBSSH2) += ssh.o
+block-obj-$(CONFIG_LIBSSH) += ssh.o
 block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 block-obj-y += backup.o
@@ -XXX,XX +XXX,XX @@ rbd.o-libs         := $(RBD_LIBS)
 gluster.o-cflags   := $(GLUSTERFS_CFLAGS)
 gluster.o-libs     := $(GLUSTERFS_LIBS)
 vxhs.o-libs        := $(VXHS_LIBS)
-ssh.o-cflags       := $(LIBSSH2_CFLAGS)
-ssh.o-libs         := $(LIBSSH2_LIBS)
+ssh.o-cflags       := $(LIBSSH_CFLAGS)
+ssh.o-libs         := $(LIBSSH_LIBS)
 block-obj-dmg-bz2-$(CONFIG_BZIP2) += dmg-bz2.o
 block-obj-$(if $(CONFIG_DMG),m,n) += $(block-obj-dmg-bz2-y)
 dmg-bz2.o-libs     := $(BZIP2_LIBS)
diff --git a/block/ssh.c b/block/ssh.c
index XXXXXXX..XXXXXXX 100644
--- a/block/ssh.c
+++ b/block/ssh.c
@@ -XXX,XX +XXX,XX @@
 
 #include "qemu/osdep.h"
 
-#include <libssh2.h>
-#include <libssh2_sftp.h>
+#include <libssh/libssh.h>
+#include <libssh/sftp.h>
 
 #include "block/block_int.h"
 #include "block/qdict.h"
@@ -XXX,XX +XXX,XX @@
 #include "trace.h"
 
 /*
- * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself.  Note
- * that this requires that libssh2 was specially compiled with the
- * `./configure --enable-debug' option, so most likely you will have
- * to compile it yourself.  The meaning of <bitmask> is described
- * here: http://www.libssh2.org/libssh2_trace.html
+ * TRACE_LIBSSH=<level> enables tracing in libssh itself.
+ * The meaning of <level> is described here:
+ * http://api.libssh.org/master/group__libssh__log.html
  */
-#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */
+#define TRACE_LIBSSH  0 /* see: SSH_LOG_* */
 
 typedef struct BDRVSSHState {
     /* Coroutine. */
@@ -XXX,XX +XXX,XX @@ typedef struct BDRVSSHState {
 
     /* SSH connection. */
     int sock;                         /* socket */
-    LIBSSH2_SESSION *session;         /* ssh session */
-    LIBSSH2_SFTP *sftp;               /* sftp session */
-    LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */
+    ssh_session session;              /* ssh session */
+    sftp_session sftp;                /* sftp session */
+    sftp_file sftp_handle;            /* sftp remote file handle */
 
-    /* See ssh_seek() function below. */
-    int64_t offset;
-    bool offset_op_read;
-
-    /* File attributes at open.  We try to keep the .filesize field
+    /*
+     * File attributes at open.  We try to keep the .size field
      * updated if it changes (eg by writing at the end of the file).
      */
-    LIBSSH2_SFTP_ATTRIBUTES attrs;
+    sftp_attributes attrs;
 
     InetSocketAddress *inet;
 
@@ -XXX,XX +XXX,XX @@ static void ssh_state_init(BDRVSSHState *s)
 {
     memset(s, 0, sizeof *s);
     s->sock = -1;
-    s->offset = -1;
     qemu_co_mutex_init(&s->lock);
 }
 
@@ -XXX,XX +XXX,XX @@ static void ssh_state_free(BDRVSSHState *s)
 {
     g_free(s->user);
 
+    if (s->attrs) {
+        sftp_attributes_free(s->attrs);
+    }
     if (s->sftp_handle) {
-        libssh2_sftp_close(s->sftp_handle);
+        sftp_close(s->sftp_handle);
     }
     if (s->sftp) {
-        libssh2_sftp_shutdown(s->sftp);
+        sftp_free(s->sftp);
     }
     if (s->session) {
-        libssh2_session_disconnect(s->session,
-                                   "from qemu ssh client: "
-                                   "user closed the connection");
-        libssh2_session_free(s->session);
-    }
-    if (s->sock >= 0) {
-        close(s->sock);
+        ssh_disconnect(s->session);
+        ssh_free(s->session); /* This frees s->sock */
     }
 }
 
@@ -XXX,XX +XXX,XX @@ session_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
     va_end(args);
 
     if (s->session) {
-        char *ssh_err;
+        const char *ssh_err;
         int ssh_err_code;
 
-        /* This is not an errno.  See <libssh2.h>. */
-        ssh_err_code = libssh2_session_last_error(s->session,
-                                                  &ssh_err, NULL, 0);
-        error_setg(errp, "%s: %s (libssh2 error code: %d)",
+        /* This is not an errno.  See <libssh/libssh.h>. */
+        ssh_err = ssh_get_error(s->session);
+        ssh_err_code = ssh_get_error_code(s->session);
+        error_setg(errp, "%s: %s (libssh error code: %d)",
                    msg, ssh_err, ssh_err_code);
     } else {
         error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
     va_end(args);
 
     if (s->sftp) {
-        char *ssh_err;
+        const char *ssh_err;
         int ssh_err_code;
-        unsigned long sftp_err_code;
+        int sftp_err_code;
 
-        /* This is not an errno.  See <libssh2.h>. */
-        ssh_err_code = libssh2_session_last_error(s->session,
-                                                  &ssh_err, NULL, 0);
-        /* See <libssh2_sftp.h>. */
-        sftp_err_code = libssh2_sftp_last_error((s)->sftp);
+        /* This is not an errno.  See <libssh/libssh.h>. */
+        ssh_err = ssh_get_error(s->session);
+        ssh_err_code = ssh_get_error_code(s->session);
+        /* See <libssh/sftp.h>. */
+        sftp_err_code = sftp_get_error(s->sftp);
 
         error_setg(errp,
-                   "%s: %s (libssh2 error code: %d, sftp error code: %lu)",
+                   "%s: %s (libssh error code: %d, sftp error code: %d)",
                    msg, ssh_err, ssh_err_code, sftp_err_code);
     } else {
         error_setg(errp, "%s", msg);
@@ -XXX,XX +XXX,XX @@ sftp_error_setg(Error **errp, BDRVSSHState *s, const char *fs, ...)
 
 static void sftp_error_trace(BDRVSSHState *s, const char *op)
 {
-    char *ssh_err;
+    const char *ssh_err;
     int ssh_err_code;
-    unsigned long sftp_err_code;
+    int sftp_err_code;
 
-    /* This is not an errno.  See <libssh2.h>. */
-    ssh_err_code = libssh2_session_last_error(s->session,
-                                              &ssh_err, NULL, 0);
-    /* See <libssh2_sftp.h>. */
-    sftp_err_code = libssh2_sftp_last_error((s)->sftp);
+    /* This is not an errno.  See <libssh/libssh.h>. */
+    ssh_err = ssh_get_error(s->session);
+    ssh_err_code = ssh_get_error_code(s->session);
+    /* See <libssh/sftp.h>. */
+    sftp_err_code = sftp_get_error(s->sftp);
 
     trace_sftp_error(op, ssh_err, ssh_err_code, sftp_err_code);
 }
@@ -XXX,XX +XXX,XX @@ static void ssh_parse_filename(const char *filename, QDict *options,
     parse_uri(filename, options, errp);
 }
 
-static int check_host_key_knownhosts(BDRVSSHState *s,
-                                     const char *host, int port, Error **errp)
+static int check_host_key_knownhosts(BDRVSSHState *s, Error **errp)
 {
-    const char *home;
-    char *knh_file = NULL;
-    LIBSSH2_KNOWNHOSTS *knh = NULL;
-    struct libssh2_knownhost *found;
-    int ret, r;
-    const char *hostkey;
-    size_t len;
-    int type;
-
-    hostkey = libssh2_session_hostkey(s->session, &len, &type);
-    if (!hostkey) {
+    int ret;
+#ifdef HAVE_LIBSSH_0_8
+    enum ssh_known_hosts_e state;
+    int r;
+    ssh_key pubkey;
+    enum ssh_keytypes_e pubkey_type;
+    unsigned char *server_hash = NULL;
+    size_t server_hash_len;
+    char *fingerprint = NULL;
+
+    state = ssh_session_is_known_server(s->session);
+    trace_ssh_server_status(state);
+
+    switch (state) {
+    case SSH_KNOWN_HOSTS_OK:
+        /* OK */
+        trace_ssh_check_host_key_knownhosts();
+        break;
+    case SSH_KNOWN_HOSTS_CHANGED:
         ret = -EINVAL;
-        session_error_setg(errp, s, "failed to read remote host key");
+        r = ssh_get_server_publickey(s->session, &pubkey);
+        if (r == 0) {
+            r = ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256,
+                                       &server_hash, &server_hash_len);
+            pubkey_type = ssh_key_type(pubkey);
+            ssh_key_free(pubkey);
+        }
+        if (r == 0) {
+            fingerprint = ssh_get_fingerprint_hash(SSH_PUBLICKEY_HASH_SHA256,
+                                                   server_hash,
+                                                   server_hash_len);
+            ssh_clean_pubkey_hash(&server_hash);
+        }
+        if (fingerprint) {
+            error_setg(errp,
+                       "host key (%s key with fingerprint %s) does not match "
+                       "the one in known_hosts; this may be a possible attack",
+                       ssh_key_type_to_char(pubkey_type), fingerprint);
+            ssh_string_free_char(fingerprint);
+        } else  {
+            error_setg(errp,
+                       "host key does not match the one in known_hosts; this "
+                       "may be a possible attack");
+        }
         goto out;
-    }
-
-    knh = libssh2_knownhost_init(s->session);
-    if (!knh) {
+    case SSH_KNOWN_HOSTS_OTHER:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                           "failed to initialize known hosts support");
+        error_setg(errp,
+                   "host key for this server not found, another type exists");
+        goto out;
+    case SSH_KNOWN_HOSTS_UNKNOWN:
+        ret = -EINVAL;
+        error_setg(errp, "no host key was found in known_hosts");
+        goto out;
+    case SSH_KNOWN_HOSTS_NOT_FOUND:
+        ret = -ENOENT;
+        error_setg(errp, "known_hosts file not found");
+        goto out;
+    case SSH_KNOWN_HOSTS_ERROR:
+        ret = -EINVAL;
+        error_setg(errp, "error while checking the host");
+        goto out;
+    default:
+        ret = -EINVAL;
+        error_setg(errp, "error while checking for known server (%d)", state);
         goto out;
     }
+#else /* !HAVE_LIBSSH_0_8 */
+    int state;
 
-    home = getenv("HOME");
-    if (home) {
-        knh_file = g_strdup_printf("%s/.ssh/known_hosts", home);
-    } else {
-        knh_file = g_strdup_printf("/root/.ssh/known_hosts");
-    }
-
-    /* Read all known hosts from OpenSSH-style known_hosts file. */
-    libssh2_knownhost_readfile(knh, knh_file, LIBSSH2_KNOWNHOST_FILE_OPENSSH);
+    state = ssh_is_server_known(s->session);
+    trace_ssh_server_status(state);
 
-    r = libssh2_knownhost_checkp(knh, host, port, hostkey, len,
-                                 LIBSSH2_KNOWNHOST_TYPE_PLAIN|
-                                 LIBSSH2_KNOWNHOST_KEYENC_RAW,
-                                 &found);
-    switch (r) {
-    case LIBSSH2_KNOWNHOST_CHECK_MATCH:
+    switch (state) {
+    case SSH_SERVER_KNOWN_OK:
         /* OK */
-        trace_ssh_check_host_key_knownhosts(found->key);
+        trace_ssh_check_host_key_knownhosts();
         break;
-    case LIBSSH2_KNOWNHOST_CHECK_MISMATCH:
+    case SSH_SERVER_KNOWN_CHANGED:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                      "host key does not match the one in known_hosts"
-                      " (found key %s)", found->key);
+        error_setg(errp,
+                   "host key does not match the one in known_hosts; this "
+                   "may be a possible attack");
         goto out;
-    case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND:
+    case SSH_SERVER_FOUND_OTHER:
         ret = -EINVAL;
-        session_error_setg(errp, s, "no host key was found in known_hosts");
+        error_setg(errp,
+                   "host key for this server not found, another type exists");
+        goto out;
+    case SSH_SERVER_FILE_NOT_FOUND:
+        ret = -ENOENT;
+        error_setg(errp, "known_hosts file not found");
         goto out;
-    case LIBSSH2_KNOWNHOST_CHECK_FAILURE:
+    case SSH_SERVER_NOT_KNOWN:
         ret = -EINVAL;
-        session_error_setg(errp, s,
-                      "failure matching the host key with known_hosts");
+        error_setg(errp, "no host key was found in known_hosts");
+        goto out;
+    case SSH_SERVER_ERROR:
+        ret = -EINVAL;
+        error_setg(errp, "server error");
         goto out;
     default:
         ret = -EINVAL;
-        session_error_setg(errp, s, "unknown error matching the host key"
-                      " with known_hosts (%d)", r);
+        error_setg(errp, "error while checking for known server (%d)", state);
         goto out;
     }
+#endif /* !HAVE_LIBSSH_0_8 */
 
     /* known_hosts checking successful. */
     ret = 0;
 
  out:
-    if (knh != NULL) {
-        libssh2_knownhost_free(knh);
-    }
-    g_free(knh_file);
     return ret;
 }
 
@@ -XXX,XX +XXX,XX @@ static int compare_fingerprint(const unsigned char *fingerprint, size_t len,
 
 static int
 check_host_key_hash(BDRVSSHState *s, const char *hash,
-                    int hash_type, size_t fingerprint_len, Error **errp)
+                    enum ssh_publickey_hash_type type, Error **errp)
 {
-    const char *fingerprint;
-
-    fingerprint = libssh2_hostkey_hash(s->session, hash_type);
-    if (!fingerprint) {
+    int r;
+    ssh_key pubkey;
+    unsigned char *server_hash;
+    size_t server_hash_len;
+
+#ifdef HAVE_LIBSSH_0_8
+    r = ssh_get_server_publickey(s->session, &pubkey);
+#else
+    r = ssh_get_publickey(s->session, &pubkey);
+#endif
+    if (r != SSH_OK) {
         session_error_setg(errp, s, "failed to read remote host key");
         return -EINVAL;
     }
 
-    if(compare_fingerprint((unsigned char *) fingerprint, fingerprint_len,
-                           hash) != 0) {
+    r = ssh_get_publickey_hash(pubkey, type, &server_hash, &server_hash_len);
+    ssh_key_free(pubkey);
+    if (r != 0) {
+        session_error_setg(errp, s,
+                           "failed reading the hash of the server SSH key");
+        return -EINVAL;
+    }
+
+    r = compare_fingerprint(server_hash, server_hash_len, hash);
+    ssh_clean_pubkey_hash(&server_hash);
+    if (r != 0) {
         error_setg(errp, "remote host key does not match host_key_check '%s'",
                    hash);
         return -EPERM;
@@ -XXX,XX +XXX,XX @@ check_host_key_hash(BDRVSSHState *s, const char *hash,
     return 0;
 }
 
-static int check_host_key(BDRVSSHState *s, const char *host, int port,
-                          SshHostKeyCheck *hkc, Error **errp)
+static int check_host_key(BDRVSSHState *s, SshHostKeyCheck *hkc, Error **errp)
 {
     SshHostKeyCheckMode mode;
 
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
     case SSH_HOST_KEY_CHECK_MODE_HASH:
         if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) {
             return check_host_key_hash(s, hkc->u.hash.hash,
-                                       LIBSSH2_HOSTKEY_HASH_MD5, 16, errp);
+                                       SSH_PUBLICKEY_HASH_MD5, errp);
         } else if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_SHA1) {
             return check_host_key_hash(s, hkc->u.hash.hash,
-                                       LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp);
+                                       SSH_PUBLICKEY_HASH_SHA1, errp);
         }
         g_assert_not_reached();
         break;
     case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS:
-        return check_host_key_knownhosts(s, host, port, errp);
+        return check_host_key_knownhosts(s, errp);
     default:
         g_assert_not_reached();
     }
@@ -XXX,XX +XXX,XX @@ static int check_host_key(BDRVSSHState *s, const char *host, int port,
     return -EINVAL;
 }
 
-static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
+static int authenticate(BDRVSSHState *s, Error **errp)
 {
     int r, ret;
-    const char *userauthlist;
-    LIBSSH2_AGENT *agent = NULL;
-    struct libssh2_agent_publickey *identity;
-    struct libssh2_agent_publickey *prev_identity = NULL;
+    int method;
 
-    userauthlist = libssh2_userauth_list(s->session, user, strlen(user));
-    if (strstr(userauthlist, "publickey") == NULL) {
+    /* Try to authenticate with the "none" method. */
+    r = ssh_userauth_none(s->session, NULL);
+    if (r == SSH_AUTH_ERROR) {
         ret = -EPERM;
-        error_setg(errp,
-                "remote server does not support \"publickey\" authentication");
+        session_error_setg(errp, s, "failed to authenticate using none "
+                                    "authentication");
         goto out;
-    }
-
-    /* Connect to ssh-agent and try each identity in turn. */
-    agent = libssh2_agent_init(s->session);
-    if (!agent) {
-        ret = -EINVAL;
-        session_error_setg(errp, s, "failed to initialize ssh-agent support");
-        goto out;
-    }
-    if (libssh2_agent_connect(agent)) {
-        ret = -ECONNREFUSED;
-        session_error_setg(errp, s, "failed to connect to ssh-agent");
-        goto out;
-    }
-    if (libssh2_agent_list_identities(agent)) {
-        ret = -EINVAL;
-        session_error_setg(errp, s,
-                           "failed requesting identities from ssh-agent");
+    } else if (r == SSH_AUTH_SUCCESS) {
+        /* Authenticated! */
+        ret = 0;
         goto out;
     }
 
-    for(;;) {
-        r = libssh2_agent_get_identity(agent, &identity, prev_identity);
-        if (r == 1) {           /* end of list */
-            break;
-        }
-        if (r < 0) {
+    method = ssh_userauth_list(s->session, NULL);
+    trace_ssh_auth_methods(method);
+
+    /*
+     * Try to authenticate with publickey, using the ssh-agent
+     * if available.
+     */
+    if (method & SSH_AUTH_METHOD_PUBLICKEY) {
+        r = ssh_userauth_publickey_auto(s->session, NULL, NULL);
+        if (r == SSH_AUTH_ERROR) {
             ret = -EINVAL;
-            session_error_setg(errp, s,
-                               "failed to obtain identity from ssh-agent");
+            session_error_setg(errp, s, "failed to authenticate using "
+                                        "publickey authentication");
             goto out;
-        }
-        r = libssh2_agent_userauth(agent, user, identity);
-        if (r == 0) {
+        } else if (r == SSH_AUTH_SUCCESS) {
             /* Authenticated! */
             ret = 0;
             goto out;
         }
-        /* Failed to authenticate with this identity, try the next one. */
-        prev_identity = identity;
     }
 
     ret = -EPERM;
@@ -XXX,XX +XXX,XX @@ static int authenticate(BDRVSSHState *s, const char *user, Error **errp)
                "and the identities held by your ssh-agent");
 
  out:
-    if (agent != NULL) {
-        /* Note: libssh2 implementation implicitly calls
-         * libssh2_agent_disconnect if necessary.
-         */
-        libssh2_agent_free(agent);
-    }
-
     return ret;
 }
 
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
                           int ssh_flags, int creat_mode, Error **errp)
 {
     int r, ret;
-    long port = 0;
+    unsigned int port = 0;
+    int new_sock = -1;
 
     if (opts->has_user) {
         s->user = g_strdup(opts->user);
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
     s->inet = opts->server;
     opts->server = NULL;
 
-    if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) {
+    if (qemu_strtoui(s->inet->port, NULL, 10, &port) < 0) {
         error_setg(errp, "Use only numeric port value");
         ret = -EINVAL;
         goto err;
     }
 
     /* Open the socket and connect. */
-    s->sock = inet_connect_saddr(s->inet, errp);
-    if (s->sock < 0) {
+    new_sock = inet_connect_saddr(s->inet, errp);
+    if (new_sock < 0) {
         ret = -EIO;
         goto err;
     }
 
+    /*
+     * Try to disable the Nagle algorithm on TCP sockets to reduce latency,
+     * but do not fail if it cannot be disabled.
+     */
+    r = socket_set_nodelay(new_sock);
+    if (r < 0) {
+        warn_report("can't set TCP_NODELAY for the ssh server %s: %s",
+                    s->inet->host, strerror(errno));
+    }
+
     /* Create SSH session. */
-    s->session = libssh2_session_init();
+    s->session = ssh_new();
     if (!s->session) {
         ret = -EINVAL;
-        session_error_setg(errp, s, "failed to initialize libssh2 session");
+        session_error_setg(errp, s, "failed to initialize libssh session");
         goto err;
     }
 
-#if TRACE_LIBSSH2 != 0
-    libssh2_trace(s->session, TRACE_LIBSSH2);
-#endif
+    /*
+     * Make sure we are in blocking mode during the connection and
+     * authentication phases.
+     */
+    ssh_set_blocking(s->session, 1);
 
-    r = libssh2_session_handshake(s->session, s->sock);
-    if (r != 0) {
+    r = ssh_options_set(s->session, SSH_OPTIONS_USER, s->user);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the user in the libssh session");
+        goto err;
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_HOST, s->inet->host);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the host in the libssh session");
+        goto err;
+    }
+
+    if (port > 0) {
+        r = ssh_options_set(s->session, SSH_OPTIONS_PORT, &port);
+        if (r < 0) {
+            ret = -EINVAL;
+            session_error_setg(errp, s,
+                               "failed to set the port in the libssh session");
+            goto err;
+        }
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_COMPRESSION, "none");
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to disable the compression in the libssh "
+                           "session");
+        goto err;
+    }
+
+    /* Read ~/.ssh/config. */
+    r = ssh_options_parse_config(s->session, NULL);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s, "failed to parse ~/.ssh/config");
+        goto err;
+    }
+
+    r = ssh_options_set(s->session, SSH_OPTIONS_FD, &new_sock);
+    if (r < 0) {
+        ret = -EINVAL;
+        session_error_setg(errp, s,
+                           "failed to set the socket in the libssh session");
+        goto err;
+    }
+    /* libssh took ownership of the socket. */
+    s->sock = new_sock;
+    new_sock = -1;
+
+    /* Connect. */
+    r = ssh_connect(s->session);
+    if (r != SSH_OK) {
         ret = -EINVAL;
         session_error_setg(errp, s, "failed to establish SSH session");
         goto err;
     }
 
     /* Check the remote host's key against known_hosts. */
-    ret = check_host_key(s, s->inet->host, port, opts->host_key_check, errp);
+    ret = check_host_key(s, opts->host_key_check, errp);
     if (ret < 0) {
         goto err;
     }
 
     /* Authenticate. */
-    ret = authenticate(s, s->user, errp);
+    ret = authenticate(s, errp);
     if (ret < 0) {
         goto err;
     }
 
     /* Start SFTP. */
-    s->sftp = libssh2_sftp_init(s->session);
+    s->sftp = sftp_new(s->session);
     if (!s->sftp) {
-        session_error_setg(errp, s, "failed to initialize sftp handle");
+        session_error_setg(errp, s, "failed to create sftp handle");
+        ret = -EINVAL;
+        goto err;
+    }
+
+    r = sftp_init(s->sftp);
+    if (r < 0) {
+        sftp_error_setg(errp, s, "failed to initialize sftp handle");
         ret = -EINVAL;
         goto err;
     }
 
     /* Open the remote file. */
     trace_ssh_connect_to_ssh(opts->path, ssh_flags, creat_mode);
-    s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags,
-                                       creat_mode);
+    s->sftp_handle = sftp_open(s->sftp, opts->path, ssh_flags, creat_mode);
     if (!s->sftp_handle) {
-        session_error_setg(errp, s, "failed to open remote file '%s'",
-                           opts->path);
+        sftp_error_setg(errp, s, "failed to open remote file '%s'",
+                        opts->path);
         ret = -EINVAL;
         goto err;
     }
 
-    r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs);
-    if (r < 0) {
+    /* Make sure the SFTP file is handled in blocking mode. */
+    sftp_file_set_blocking(s->sftp_handle);
+
+    s->attrs = sftp_fstat(s->sftp_handle);
+    if (!s->attrs) {
         sftp_error_setg(errp, s, "failed to read file attributes");
         return -EINVAL;
     }
@@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts,
     return 0;
 
  err:
+    if (s->attrs) {
+        sftp_attributes_free(s->attrs);
+    }
+    s->attrs = NULL;
     if (s->sftp_handle) {
-        libssh2_sftp_close(s->sftp_handle);
+        sftp_close(s->sftp_handle);
     }
     s->sftp_handle = NULL;
     if (s->sftp) {
-        libssh2_sftp_shutdown(s->sftp);
+        sftp_free(s->sftp);
     }
     s->sftp = NULL;
     if (s->session) {
-        libssh2_session_disconnect(s->session,
-                                   "from qemu ssh client: "
-                                   "error opening connection");
-        libssh2_session_free(s->session);
+        ssh_disconnect(s->session);
+        ssh_free(s->session);
     }
     s->session = NULL;
+    s->sock = -1;
+    if (new_sock >= 0) {
+        close(new_sock);
+    }
 
     return ret;
 }
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
 
     ssh_state_init(s);
 
-    ssh_flags = LIBSSH2_FXF_READ;
+    ssh_flags = 0;
     if (bdrv_flags & BDRV_O_RDWR) {
-        ssh_flags |= LIBSSH2_FXF_WRITE;
+        ssh_flags |= O_RDWR;
+    } else {
+        ssh_flags |= O_RDONLY;
     }
 
     opts = ssh_parse_options(options, errp);
@@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags,
     }
 
     /* Go non-blocking. */
-    libssh2_session_set_blocking(s->session, 0);
+    ssh_set_blocking(s->session, 0);
 
     qapi_free_BlockdevOptionsSsh(opts);
 
     return 0;
 
  err:
-    if (s->sock >= 0) {
-        close(s->sock);
-    }
-    s->sock = -1;
-
     qapi_free_BlockdevOptionsSsh(opts);
 
     return ret;
@@ -XXX,XX +XXX,XX @@ static int ssh_grow_file(BDRVSSHState *s, int64_t offset, Error **errp)
 {
     ssize_t ret;
     char c[1] = { '\0' };
-    int was_blocking = libssh2_session_get_blocking(s->session);
+    int was_blocking = ssh_is_blocking(s->session);
 
     /* offset must be strictly greater than the current size so we do
      * not overwrite anything */
-    assert(offset > 0 && offset > s->attrs.filesize);
+    assert(offset > 0 && offset > s->attrs->size);
 
-    libssh2_session_set_blocking(s->session, 1);
+    ssh_set_blocking(s->session, 1);
 
-    libssh2_sftp_seek64(s->sftp_handle, offset - 1);
-    ret = libssh2_sftp_write(s->sftp_handle, c, 1);
+    sftp_seek64(s->sftp_handle, offset - 1);
+    ret = sftp_write(s->sftp_handle, c, 1);
 
-    libssh2_session_set_blocking(s->session, was_blocking);
+    ssh_set_blocking(s->session, was_blocking);
 
     if (ret < 0) {
         sftp_error_setg(errp, s, "Failed to grow file");
         return -EIO;
     }
 
-    s->attrs.filesize = offset;
+    s->attrs->size = offset;
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ static int ssh_co_create(BlockdevCreateOptions *options, Error **errp)
     ssh_state_init(&s);
 
     ret = connect_to_ssh(&s, opts->location,
-                         LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE|
-                         LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC,
+                         O_RDWR | O_CREAT | O_TRUNC,
                          0644, errp);
     if (ret < 0) {
         goto fail;
@@ -XXX,XX +XXX,XX @@ static int ssh_has_zero_init(BlockDriverState *bs)
     /* Assume false, unless we can positively prove it's true. */
     int has_zero_init = 0;
 
-    if (s->attrs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) {
-        if (s->attrs.permissions & LIBSSH2_SFTP_S_IFREG) {
-            has_zero_init = 1;
-        }
+    if (s->attrs->type == SSH_FILEXFER_TYPE_REGULAR) {
+        has_zero_init = 1;
     }
 
     return has_zero_init;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
         .co = qemu_coroutine_self()
     };
 
-    r = libssh2_session_block_directions(s->session);
+    r = ssh_get_poll_flags(s->session);
 
-    if (r & LIBSSH2_SESSION_BLOCK_INBOUND) {
+    if (r & SSH_READ_PENDING) {
         rd_handler = restart_coroutine;
     }
-    if (r & LIBSSH2_SESSION_BLOCK_OUTBOUND) {
+    if (r & SSH_WRITE_PENDING) {
         wr_handler = restart_coroutine;
     }
 
@@ -XXX,XX +XXX,XX @@ static coroutine_fn void co_yield(BDRVSSHState *s, BlockDriverState *bs)
     trace_ssh_co_yield_back(s->sock);
 }
 
-/* SFTP has a function `libssh2_sftp_seek64' which seeks to a position
- * in the remote file.  Notice that it just updates a field in the
- * sftp_handle structure, so there is no network traffic and it cannot
- * fail.
- *
- * However, `libssh2_sftp_seek64' does have a catastrophic effect on
- * performance since it causes the handle to throw away all in-flight
- * reads and buffered readahead data.  Therefore this function tries
- * to be intelligent about when to call the underlying libssh2 function.
- */
-#define SSH_SEEK_WRITE 0
-#define SSH_SEEK_READ  1
-#define SSH_SEEK_FORCE 2
-
-static void ssh_seek(BDRVSSHState *s, int64_t offset, int flags)
-{
-    bool op_read = (flags & SSH_SEEK_READ) != 0;
-    bool force = (flags & SSH_SEEK_FORCE) != 0;
-
-    if (force || op_read != s->offset_op_read || offset != s->offset) {
-        trace_ssh_seek(offset);
-        libssh2_sftp_seek64(s->sftp_handle, offset);
-        s->offset = offset;
-        s->offset_op_read = op_read;
-    }
-}
-
 static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
                                  int64_t offset, size_t size,
                                  QEMUIOVector *qiov)
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
 
     trace_ssh_read(offset, size);
 
-    ssh_seek(s, offset, SSH_SEEK_READ);
+    trace_ssh_seek(offset);
+    sftp_seek64(s->sftp_handle, offset);
 
     /* This keeps track of the current iovec element ('i'), where we
      * will write to next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_read(BDRVSSHState *s, BlockDriverState *bs,
     buf = i->iov_base;
     end_of_vec = i->iov_base + i->iov_len;
 
-    /* libssh2 has a hard-coded limit of 2000 bytes per request,
-     * although it will also do readahead behind our backs.  Therefore
-     * we may have to do repeated reads here until we have read 'size'
-     * bytes.
-     */
     for (got = 0; got < size; ) {
+        size_t request_read_size;
     again:
-        trace_ssh_read_buf(buf, end_of_vec - buf);
-        r = libssh2_sftp_read(s->sftp_handle, buf, end_of_vec - buf);
-        trace_ssh_read_return(r);
+        /*
+         * The size of SFTP packets is limited to 32K bytes, so limit
+         * the amount of data requested to 16K, as libssh currently
+         * does not handle multiple requests on its own.
+         */
+        request_read_size = MIN(end_of_vec - buf, 16384);
+        trace_ssh_read_buf(buf, end_of_vec - buf, request_read_size);
+        r = sftp_read(s->sftp_handle, buf, request_read_size);
+        trace_ssh_read_return(r, sftp_get_error(s->sftp));
 
-        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+        if (r == SSH_AGAIN) {
             co_yield(s, bs);
             goto again;
         }
-        if (r < 0) {
-            sftp_error_trace(s, "read");
-            s->offset = -1;
-            return -EIO;
-        }
-        if (r == 0) {
+        if (r == SSH_EOF || (r == 0 && sftp_get_error(s->sftp) == SSH_FX_EOF)) {
             /* EOF: Short read so pad the buffer with zeroes and return it. */
             qemu_iovec_memset(qiov, got, 0, size - got);
             return 0;
         }
+        if (r <= 0) {
+            sftp_error_trace(s, "read");
+            return -EIO;
+        }
 
         got += r;
         buf += r;
-        s->offset += r;
         if (buf >= end_of_vec && got < size) {
             i++;
             buf = i->iov_base;
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
 
     trace_ssh_write(offset, size);
 
-    ssh_seek(s, offset, SSH_SEEK_WRITE);
+    trace_ssh_seek(offset);
+    sftp_seek64(s->sftp_handle, offset);
 
     /* This keeps track of the current iovec element ('i'), where we
      * will read from next ('buf'), and the end of the current iovec
@@ -XXX,XX +XXX,XX @@ static int ssh_write(BDRVSSHState *s, BlockDriverState *bs,
     end_of_vec = i->iov_base + i->iov_len;
 
     for (written = 0; written < size; ) {
+        size_t request_write_size;
     again:
-        trace_ssh_write_buf(buf, end_of_vec - buf);
-        r = libssh2_sftp_write(s->sftp_handle, buf, end_of_vec - buf);
-        trace_ssh_write_return(r);
+        /*
+         * Avoid too large data packets, as libssh currently does not
+         * handle multiple requests on its own.
+         */
+        request_write_size = MIN(end_of_vec - buf, 131072);
+        trace_ssh_write_buf(buf, end_of_vec - buf, request_write_size);
+        r = sftp_write(s->sftp_handle, buf, request_write_size);
+        trace_ssh_write_return(r, sftp_get_error(s->sftp));
 
-        if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+        if (r == SSH_AGAIN) {
             co_yield(s, bs);
             goto again;
         }
         if (r < 0) {
             sftp_error_trace(s, "write");
-            s->offset = -1;
             return -EIO;
         }
-        /* The libssh2 API is very unclear about this.  A comment in
-         * the code says "nothing was acked, and no EAGAIN was
-         * received!" which apparently means that no data got sent
-         * out, and the underlying channel didn't return any EAGAIN
-         * indication.  I think this is a bug in either libssh2 or
-         * OpenSSH (server-side).  In any case, forcing a seek (to
-         * discard libssh2 internal buffers), and then trying again
-         * works for me.
-         */
-        if (r == 0) {
-            ssh_seek(s, offset + written, SSH_SEEK_WRITE|SSH_SEEK_FORCE);
-            co_yield(s, bs);
-            goto again;
-        }
 
         written += r;
         buf += r;
-        s->offset += r;
         if (buf >= end_of_vec && written < size) {
             i++;
             buf = i->iov_base;
             end_of_vec = i->iov_base + i->iov_len;
         }
 
-        if (offset + written > s->attrs.filesize)
-            s->attrs.filesize = offset + written;
+        if (offset + written > s->attrs->size) {
+            s->attrs->size = offset + written;
+        }
     }
 
     return 0;
@@ -XXX,XX +XXX,XX @@ static void unsafe_flush_warning(BDRVSSHState *s, const char *what)
     }
 }
 
-#ifdef HAS_LIBSSH2_SFTP_FSYNC
+#ifdef HAVE_LIBSSH_0_8
 
 static coroutine_fn int ssh_flush(BDRVSSHState *s, BlockDriverState *bs)
 {
     int r;
 
     trace_ssh_flush();
+
+    if (!sftp_extension_supported(s->sftp, "fsync@openssh.com", "1")) {
+        unsafe_flush_warning(s, "OpenSSH >= 6.3");
+        return 0;
+    }
  again:
-    r = libssh2_sftp_fsync(s->sftp_handle);
-    if (r == LIBSSH2_ERROR_EAGAIN || r == LIBSSH2_ERROR_TIMEOUT) {
+    r = sftp_fsync(s->sftp_handle);
+    if (r == SSH_AGAIN) {
         co_yield(s, bs);
         goto again;
     }
-    if (r == LIBSSH2_ERROR_SFTP_PROTOCOL &&
-        libssh2_sftp_last_error(s->sftp) == LIBSSH2_FX_OP_UNSUPPORTED) {
-        unsafe_flush_warning(s, "OpenSSH >= 6.3");
-        return 0;
-    }
     if (r < 0) {
         sftp_error_trace(s, "fsync");
         return -EIO;
@@ -XXX,XX +XXX,XX @@ static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
     return ret;
 }
 
-#else /* !HAS_LIBSSH2_SFTP_FSYNC */
+#else /* !HAVE_LIBSSH_0_8 */
 
 static coroutine_fn int ssh_co_flush(BlockDriverState *bs)
 {
     BDRVSSHState *s = bs->opaque;
 
-    unsafe_flush_warning(s, "libssh2 >= 1.4.4");
+    unsafe_flush_warning(s, "libssh >= 0.8.0");
     return 0;
 }
 
-#endif /* !HAS_LIBSSH2_SFTP_FSYNC */
+#endif /* !HAVE_LIBSSH_0_8 */
 
 static int64_t ssh_getlength(BlockDriverState *bs)
 {
     BDRVSSHState *s = bs->opaque;
     int64_t length;
 
-    /* Note we cannot make a libssh2 call here. */
-    length = (int64_t) s->attrs.filesize;
+    /* Note we cannot make a libssh call here. */
+    length = (int64_t) s->attrs->size;
     trace_ssh_getlength(length);
 
     return length;
@@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_truncate(BlockDriverState *bs, int64_t offset,
         return -ENOTSUP;
     }
 
-    if (offset < s->attrs.filesize) {
+    if (offset < s->attrs->size) {
         error_setg(errp, "ssh driver does not support shrinking files");
         return -ENOTSUP;
     }
 
-    if (offset == s->attrs.filesize) {
+    if (offset == s->attrs->size) {
         return 0;
     }
 
@@ -XXX,XX +XXX,XX @@ static void bdrv_ssh_init(void)
 {
     int r;
 
-    r = libssh2_init(0);
+    r = ssh_init();
     if (r != 0) {
-        fprintf(stderr, "libssh2 initialization failed, %d\n", r);
+        fprintf(stderr, "libssh initialization failed, %d\n", r);
         exit(EXIT_FAILURE);
     }
 
+#if TRACE_LIBSSH != 0
+    ssh_set_log_level(TRACE_LIBSSH);
+#endif
+
     bdrv_register(&bdrv_ssh);
 }
 
diff --git a/.travis.yml b/.travis.yml
index XXXXXXX..XXXXXXX 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -XXX,XX +XXX,XX @@ addons:
       - libseccomp-dev
       - libspice-protocol-dev
       - libspice-server-dev
-      - libssh2-1-dev
+      - libssh-dev
       - liburcu-dev
       - libusb-1.0-0-dev
       - libvte-2.91-dev
@@ -XXX,XX +XXX,XX @@ matrix:
             - libseccomp-dev
             - libspice-protocol-dev
             - libspice-server-dev
-            - libssh2-1-dev
+            - libssh-dev
             - liburcu-dev
             - libusb-1.0-0-dev
             - libvte-2.91-dev
diff --git a/block/trace-events b/block/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/block/trace-events
+++ b/block/trace-events
@@ -XXX,XX +XXX,XX @@ nbd_client_connect_success(const char *export_name) "export '%s'"
 # ssh.c
 ssh_restart_coroutine(void *co) "co=%p"
 ssh_flush(void) "fsync"
-ssh_check_host_key_knownhosts(const char *key) "host key OK: %s"
+ssh_check_host_key_knownhosts(void) "host key OK"
 ssh_connect_to_ssh(char *path, int flags, int mode) "opening file %s flags=0x%x creat_mode=0%o"
 ssh_co_yield(int sock, void *rd_handler, void *wr_handler) "s->sock=%d rd_handler=%p wr_handler=%p"
 ssh_co_yield_back(int sock) "s->sock=%d - back"
 ssh_getlength(int64_t length) "length=%" PRIi64
 ssh_co_create_opts(uint64_t size) "total_size=%" PRIu64
 ssh_read(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-ssh_read_buf(void *buf, size_t size) "sftp_read buf=%p size=%zu"
-ssh_read_return(ssize_t ret) "sftp_read returned %zd"
+ssh_read_buf(void *buf, size_t size, size_t actual_size) "sftp_read buf=%p size=%zu (actual size=%zu)"
+ssh_read_return(ssize_t ret, int sftp_err) "sftp_read returned %zd (sftp error=%d)"
 ssh_write(int64_t offset, size_t size) "offset=%" PRIi64 " size=%zu"
-ssh_write_buf(void *buf, size_t size) "sftp_write buf=%p size=%zu"
-ssh_write_return(ssize_t ret) "sftp_write returned %zd"
+ssh_write_buf(void *buf, size_t size, size_t actual_size) "sftp_write buf=%p size=%zu (actual size=%zu)"
+ssh_write_return(ssize_t ret, int sftp_err) "sftp_write returned %zd (sftp error=%d)"
 ssh_seek(int64_t offset) "seeking to offset=%" PRIi64
+ssh_auth_methods(int methods) "auth methods=0x%x"
+ssh_server_status(int status) "server status=%d"
 
 # curl.c
 curl_timer_cb(long timeout_ms) "timer callback timeout_ms %ld"
@@ -XXX,XX +XXX,XX @@ sheepdog_snapshot_create(const char *sn_name, const char *id) "%s %s"
 sheepdog_snapshot_create_inode(const char *name, uint32_t snap, uint32_t vdi) "s->inode: name %s snap_id 0x%" PRIx32 " vdi 0x%" PRIx32
 
 # ssh.c
-sftp_error(const char *op, const char *ssh_err, int ssh_err_code, unsigned long sftp_err_code) "%s failed: %s (libssh2 error code: %d, sftp error code: %lu)"
+sftp_error(const char *op, const char *ssh_err, int ssh_err_code, int sftp_err_code) "%s failed: %s (libssh error code: %d, sftp error code: %d)"
diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi
index XXXXXXX..XXXXXXX 100644
--- a/docs/qemu-block-drivers.texi
+++ b/docs/qemu-block-drivers.texi
@@ -XXX,XX +XXX,XX @@ print a warning when @code{fsync} is not supported:
 
 warning: ssh server @code{ssh.example.com:22} does not support fsync
 
-With sufficiently new versions of libssh2 and OpenSSH, @code{fsync} is
+With sufficiently new versions of libssh and OpenSSH, @code{fsync} is
 supported.
 
 @node disk_images_nvme
diff --git a/tests/docker/dockerfiles/debian-win32-cross.docker b/tests/docker/dockerfiles/debian-win32-cross.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/debian-win32-cross.docker
+++ b/tests/docker/dockerfiles/debian-win32-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         mxe-$TARGET-w64-mingw32.shared-curl \
         mxe-$TARGET-w64-mingw32.shared-glib \
         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
         mxe-$TARGET-w64-mingw32.shared-libusb1 \
         mxe-$TARGET-w64-mingw32.shared-lzo \
         mxe-$TARGET-w64-mingw32.shared-nettle \
diff --git a/tests/docker/dockerfiles/debian-win64-cross.docker b/tests/docker/dockerfiles/debian-win64-cross.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/debian-win64-cross.docker
+++ b/tests/docker/dockerfiles/debian-win64-cross.docker
@@ -XXX,XX +XXX,XX @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         mxe-$TARGET-w64-mingw32.shared-curl \
         mxe-$TARGET-w64-mingw32.shared-glib \
         mxe-$TARGET-w64-mingw32.shared-libgcrypt \
-        mxe-$TARGET-w64-mingw32.shared-libssh2 \
         mxe-$TARGET-w64-mingw32.shared-libusb1 \
         mxe-$TARGET-w64-mingw32.shared-lzo \
         mxe-$TARGET-w64-mingw32.shared-nettle \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     libpng-devel \
     librbd-devel \
     libseccomp-devel \
-    libssh2-devel \
+    libssh-devel \
     libubsan \
     libusbx-devel \
     libxml2-devel \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     mingw32-gtk3 \
     mingw32-libjpeg-turbo \
     mingw32-libpng \
-    mingw32-libssh2 \
     mingw32-libtasn1 \
     mingw32-nettle \
     mingw32-pixman \
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES \
     mingw64-gtk3 \
     mingw64-libjpeg-turbo \
     mingw64-libpng \
-    mingw64-libssh2 \
     mingw64-libtasn1 \
     mingw64-nettle \
     mingw64-pixman \
diff --git a/tests/docker/dockerfiles/ubuntu.docker b/tests/docker/dockerfiles/ubuntu.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/ubuntu.docker
+++ b/tests/docker/dockerfiles/ubuntu.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
-    libssh2-1-dev \
+    libssh-dev \
     libusb-1.0-0-dev \
     libusbredirhost-dev \
     libvdeplug-dev \
diff --git a/tests/docker/dockerfiles/ubuntu1804.docker b/tests/docker/dockerfiles/ubuntu1804.docker
index XXXXXXX..XXXXXXX 100644
--- a/tests/docker/dockerfiles/ubuntu1804.docker
+++ b/tests/docker/dockerfiles/ubuntu1804.docker
@@ -XXX,XX +XXX,XX @@ ENV PACKAGES flex bison \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
-    libssh2-1-dev \
+    libssh-dev \
     libusb-1.0-0-dev \
     libusbredirhost-dev \
     libvdeplug-dev \
diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index XXXXXXX..XXXXXXX 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
 
     iotests.img_info_log(remote_path)
 
-    md5_key = subprocess.check_output(
-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-        'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-        shell=True).rstrip().decode('ascii')
+    keys = subprocess.check_output(
+        'ssh-keyscan 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
+        'cut -d" " -f3',
+        shell=True).rstrip().decode('ascii').split('\n')
+
+    # Mappings of base64 representations to digests
+    md5_keys = {}
+    sha1_keys = {}
+
+    for key in keys:
+        md5_keys[key] = subprocess.check_output(
+            'echo %s | base64 -d | md5sum -b | cut -d" " -f1' % key,
+            shell=True).rstrip().decode('ascii')
+
+        sha1_keys[key] = subprocess.check_output(
+            'echo %s | base64 -d | sha1sum -b | cut -d" " -f1' % key,
+            shell=True).rstrip().decode('ascii')
 
     vm.launch()
+
+    # Find correct key first
+    matching_key = None
+    for key in keys:
+        result = vm.qmp('blockdev-add',
+                        driver='ssh', node_name='node0', path=disk_path,
+                        server={
+                             'host': '127.0.0.1',
+                             'port': '22',
+                        }, host_key_check={
+                             'mode': 'hash',
+                             'type': 'md5',
+                             'hash': md5_keys[key],
+                        })
+
+        if 'error' not in result:
+            vm.qmp('blockdev-del', node_name='node0')
+            matching_key = key
+            break
+
+    if matching_key is None:
+        vm.shutdown()
+        iotests.notrun('Did not find a key that fits 127.0.0.1')
+
     blockdev_create(vm, { 'driver': 'ssh',
                           'location': {
                               'path': disk_path,
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                               'host-key-check': {
                                   'mode': 'hash',
                                   'type': 'md5',
-                                  'hash': md5_key,
+                                  'hash': md5_keys[matching_key],
                               }
                           },
                           'size': 8388608 })
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
 
     iotests.img_info_log(remote_path)
 
-    sha1_key = subprocess.check_output(
-        'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
-        'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-        shell=True).rstrip().decode('ascii')
-
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
                           'location': {
@@ -XXX,XX +XXX,XX @@ with iotests.FilePath('t.img') as disk_path, \
                               'host-key-check': {
                                   'mode': 'hash',
                                   'type': 'sha1',
-                                  'hash': sha1_key,
+                                  'hash': sha1_keys[matching_key],
                               }
                           },
                           'size': 4194304 })
diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
index XXXXXXX..XXXXXXX 100644
--- a/tests/qemu-iotests/207.out
+++ b/tests/qemu-iotests/207.out
@@ -XXX,XX +XXX,XX @@ virtual size: 4 MiB (4194304 bytes)
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
 {"return": {}}
-Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
+Job failed: failed to open remote file '/this/is/not/an/existing/path': SFTP server: No such file (libssh error code: 1, sftp error code: 2)
 {"execute": "job-dismiss", "arguments": {"id": "job0"}}
 {"return": {}}
 
-- 
2.21.0