The following changes since commit 9a7beaad3dbba982f7a461d676b55a5c3851d312:

Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-20210304' into staging (2021-03-05 10:47:46 +0000)

for you to fetch changes up to 67bedc3aed5c455b629c2cb5f523b536c46adff9:

docs: qsd: Explain --export nbd,name=... default (2021-03-05 17:09:46 +0100)

- qemu-storage-daemon: add --pidfile option

- qemu-storage-daemon: CLI error messages include the option name now

- vhost-user-blk export: Misc fixes, added test cases

- docs: Improvements for qemu-storage-daemon documentation

- parallels: load bitmap extension

- backup-top: Don't crash on post-finalize accesses

- iotests improvements

Alberto Garcia (1):

iotests: Drop deprecated 'props' from object-add

Coiby Xu (1):

test: new qTest case to test the vhost-user-blk-server

Eric Blake (1):

iotests: Fix up python style in 300

Kevin Wolf (1):

docs: qsd: Explain --export nbd,name=... default

Max Reitz (3):

backup: Remove nodes from job in .clean()

backup-top: Refuse I/O in inactive state

iotests/283: Check that finalize drops backup-top

Paolo Bonzini (2):

storage-daemon: report unexpected arguments on the fly

storage-daemon: include current command line option in the errors

Stefan Hajnoczi (14):

qemu-storage-daemon: add --pidfile option

docs: show how to spawn qemu-storage-daemon with fd passing

docs: replace insecure /tmp examples in qsd docs

vhost-user-blk: fix blkcfg->num_queues endianness

libqtest: add qtest_socket_server()

libqtest: add qtest_kill_qemu()

libqtest: add qtest_remove_abrt_handler()

tests/qtest: add multi-queue test case to vhost-user-blk-test

block/export: fix blk_size double byteswap

block/export: use VIRTIO_BLK_SECTOR_BITS

block/export: fix vhost-user-blk export sector number calculation

block/export: port virtio-blk discard/write zeroes input validation

vhost-user-blk-test: test discard/write zeroes invalid inputs

block/export: port virtio-blk read/write range check

Stefano Garzarella (1):

blockjob: report a better error message

Vladimir Sementsov-Ogievskiy (7):

qcow2-bitmap: make bytes_covered_by_bitmap_cluster() public

parallels.txt: fix bitmap L1 table description

block/parallels: BDRVParallelsState: add cluster_size field

parallels: support bitmap extension for read-only mode

iotests.py: add unarchive_sample_image() helper

iotests: add parallels-read-bitmap test

MAINTAINERS: update parallels block driver

docs/interop/parallels.txt | 28 +-

docs/tools/qemu-storage-daemon.rst | 68 +-

block/parallels.h | 7 +-

include/block/dirty-bitmap.h | 2 +

tests/qtest/libqos/libqtest.h | 37 +

tests/qtest/libqos/vhost-user-blk.h | 48 +

block/backup-top.c | 10 +

block/backup.c | 1 +

block/dirty-bitmap.c | 13 +

block/export/vhost-user-blk-server.c | 150 +++-

block/parallels-ext.c | 300 +++++++

block/parallels.c | 26 +-

block/qcow2-bitmap.c | 16 +-

blockjob.c | 10 +-

hw/block/vhost-user-blk.c | 7 +-

storage-daemon/qemu-storage-daemon.c | 56 +-

tests/qtest/libqos/vhost-user-blk.c | 130 +++

tests/qtest/libqtest.c | 82 +-

tests/qtest/vhost-user-blk-test.c | 983 +++++++++++++++++++++

tests/qemu-iotests/iotests.py | 10 +

MAINTAINERS | 5 +

block/meson.build | 3 +-

tests/qemu-iotests/087 | 8 +-

tests/qemu-iotests/184 | 18 +-

tests/qemu-iotests/218 | 2 +-

tests/qemu-iotests/235 | 2 +-

tests/qemu-iotests/245 | 4 +-

tests/qemu-iotests/258 | 6 +-

tests/qemu-iotests/258.out | 4 +-

tests/qemu-iotests/283 | 53 ++

tests/qemu-iotests/283.out | 15 +

tests/qemu-iotests/295 | 2 +-

tests/qemu-iotests/296 | 2 +-

tests/qemu-iotests/300 | 10 +-

.../sample_images/parallels-with-bitmap.bz2 | Bin 0 -> 203 bytes

.../sample_images/parallels-with-bitmap.sh | 51 ++

tests/qemu-iotests/tests/parallels-read-bitmap | 55 ++

tests/qemu-iotests/tests/parallels-read-bitmap.out | 6 +

tests/qtest/libqos/meson.build | 1 +

tests/qtest/meson.build | 4 +

40 files changed, 2098 insertions(+), 137 deletions(-)

create mode 100644 tests/qtest/libqos/vhost-user-blk.h

create mode 100644 block/parallels-ext.c

create mode 100644 tests/qtest/libqos/vhost-user-blk.c

create mode 100644 tests/qtest/vhost-user-blk-test.c

create mode 100644 tests/qemu-iotests/sample_images/parallels-with-bitmap.bz2

create mode 100755 tests/qemu-iotests/sample_images/parallels-with-bitmap.sh

create mode 100755 tests/qemu-iotests/tests/parallels-read-bitmap

create mode 100644 tests/qemu-iotests/tests/parallels-read-bitmap.out

From: Alberto Garcia <berto@igalia.com>

Signed-off-by: Alberto Garcia <berto@igalia.com>

Message-Id: <20210222115737.2993-1-berto@igalia.com>

Signed-off-by: Kevin Wolf <kwolf@redhat.com>

---

tests/qemu-iotests/087 | 8 ++------

tests/qemu-iotests/184 | 18 ++++++------------

tests/qemu-iotests/218 | 2 +-

tests/qemu-iotests/235 | 2 +-

tests/qemu-iotests/245 | 4 ++--

tests/qemu-iotests/258 | 6 +++---

tests/qemu-iotests/258.out | 4 ++--

tests/qemu-iotests/295 | 2 +-

tests/qemu-iotests/296 | 2 +-

9 files changed, 19 insertions(+), 29 deletions(-)

diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/087

+++ b/tests/qemu-iotests/087

@@ -XXX,XX +XXX,XX @@ run_qemu <<EOF

"arguments": {

"qom-type": "secret",

"id": "sec0",

- "props": {

- "data": "123456"

- }

+ "data": "123456"

}

}

{ "execute": "blockdev-add",

@@ -XXX,XX +XXX,XX @@ run_qemu <<EOF

"arguments": {

"qom-type": "secret",

"id": "sec0",

- "props": {

- "data": "123456"

- }

+ "data": "123456"

}

}

{ "execute": "blockdev-add",

diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/184

+++ b/tests/qemu-iotests/184

@@ -XXX,XX +XXX,XX @@ run_qemu <<EOF

"arguments": {

"qom-type": "throttle-group",

"id": "group0",

- "props": {

- "limits" : {

- "iops-total": 1000

- }

+ "limits" : {

+ "iops-total": 1000

}

}

}

@@ -XXX,XX +XXX,XX @@ run_qemu <<EOF

"arguments": {

"qom-type": "throttle-group",

"id": "group0",

- "props" : {

- "limits": {

- "iops-total": 1000

- }

+ "limits": {

+ "iops-total": 1000

}

}

}

@@ -XXX,XX +XXX,XX @@ run_qemu <<EOF

"arguments": {

"qom-type": "throttle-group",

"id": "group0",

- "props" : {

- "limits": {

- "iops-total": 1000

- }

+ "limits": {

+ "iops-total": 1000

}

}

}

diff --git a/tests/qemu-iotests/218 b/tests/qemu-iotests/218

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/218

+++ b/tests/qemu-iotests/218

@@ -XXX,XX +XXX,XX @@ with iotests.VM() as vm, \

vm.launch()

ret = vm.qmp('object-add', qom_type='throttle-group', id='tg',

- props={'x-bps-read': 4096})

+ limits={'bps-read': 4096})

assert ret['return'] == {}

ret = vm.qmp('blockdev-add',

diff --git a/tests/qemu-iotests/235 b/tests/qemu-iotests/235

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/235

+++ b/tests/qemu-iotests/235

@@ -XXX,XX +XXX,XX @@ vm.add_args('-drive', 'id=src,file=' + disk)

vm.launch()

log(vm.qmp('object-add', qom_type='throttle-group', id='tg0',

- props={ 'x-bps-total': size }))

+ limits={'bps-total': size}))

log(vm.qmp('blockdev-add',

**{ 'node-name': 'target',

diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/245

+++ b/tests/qemu-iotests/245

@@ -XXX,XX +XXX,XX @@ class TestBlockdevReopen(iotests.QMPTestCase):

###### throttle ######

######################

opts = { 'qom-type': 'throttle-group', 'id': 'group0',

- 'props': { 'limits': { 'iops-total': 1000 } } }

+ 'limits': { 'iops-total': 1000 } }

result = self.vm.qmp('object-add', conv_keys = False, **opts)

self.assert_qmp(result, 'return', {})

opts = { 'qom-type': 'throttle-group', 'id': 'group1',

- 'props': { 'limits': { 'iops-total': 2000 } } }

+ 'limits': { 'iops-total': 2000 } }

result = self.vm.qmp('object-add', conv_keys = False, **opts)

self.assert_qmp(result, 'return', {})

diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/258

+++ b/tests/qemu-iotests/258

@@ -XXX,XX +XXX,XX @@ def test_concurrent_finish(write_to_stream_node):

vm.qmp_log('object-add',

qom_type='throttle-group',

id='tg',

- props={

- 'x-iops-write': 1,

- 'x-iops-write-max': 1

+ limits={

+ 'iops-write': 1,

+ 'iops-write-max': 1

})

vm.qmp_log('blockdev-add',

diff --git a/tests/qemu-iotests/258.out b/tests/qemu-iotests/258.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/258.out

+++ b/tests/qemu-iotests/258.out

@@ -XXX,XX +XXX,XX @@ Running tests:

=== Commit and stream finish concurrently (letting stream write) ===

-{"execute": "object-add", "arguments": {"id": "tg", "props": {"x-iops-write": 1, "x-iops-write-max": 1}, "qom-type": "throttle-group"}}

+{"execute": "object-add", "arguments": {"id": "tg", "limits": {"iops-write": 1, "iops-write-max": 1}, "qom-type": "throttle-group"}}

{"return": {}}

{"execute": "blockdev-add", "arguments": {"backing": {"backing": {"backing": {"backing": {"driver": "raw", "file": {"driver": "file", "filename": "TEST_DIR/PID-node0.img"}, "node-name": "node0"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-node1.img"}, "node-name": "node1"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-node2.img"}, "node-name": "node2"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-node3.img"}, "node-name": "node3"}, "driver": "IMGFMT", "file": {"driver": "throttle", "file": {"driver": "file", "filename": "TEST_DIR/PID-node4.img"}, "throttle-group": "tg"}, "node-name": "node4"}}

{"return": {}}

@@ -XXX,XX +XXX,XX @@ Running tests:

=== Commit and stream finish concurrently (letting commit write) ===

-{"execute": "object-add", "arguments": {"id": "tg", "props": {"x-iops-write": 1, "x-iops-write-max": 1}, "qom-type": "throttle-group"}}

+{"execute": "object-add", "arguments": {"id": "tg", "limits": {"iops-write": 1, "iops-write-max": 1}, "qom-type": "throttle-group"}}

{"return": {}}

{"execute": "blockdev-add", "arguments": {"backing": {"backing": {"backing": {"backing": {"driver": "raw", "file": {"driver": "throttle", "file": {"driver": "file", "filename": "TEST_DIR/PID-node0.img"}, "throttle-group": "tg"}, "node-name": "node0"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-node1.img"}, "node-name": "node1"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-node2.img"}, "node-name": "node2"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-node3.img"}, "node-name": "node3"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-node4.img"}, "node-name": "node4"}}

{"return": {}}

diff --git a/tests/qemu-iotests/295 b/tests/qemu-iotests/295

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/295

+++ b/tests/qemu-iotests/295

@@ -XXX,XX +XXX,XX @@ class Secret:

def to_qmp_object(self):

return { "qom_type" : "secret", "id": self.id(),

- "props": { "data": self.secret() } }

+ "data": self.secret() }

################################################################################

class EncryptionSetupTestCase(iotests.QMPTestCase):

diff --git a/tests/qemu-iotests/296 b/tests/qemu-iotests/296

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/296

+++ b/tests/qemu-iotests/296

@@ -XXX,XX +XXX,XX @@ class Secret:

def to_qmp_object(self):

return { "qom_type" : "secret", "id": self.id(),

- "props": { "data": self.secret() } }

+ "data": self.secret() }

################################################################################

--

2.29.2

From: Max Reitz <mreitz@redhat.com>

The block job holds a reference to the backup-top node (because it is

passed as the main job BDS to block_job_create()). Therefore,

bdrv_backup_top_drop() cannot delete the backup-top node (replacing it

by its child does not affect the job parent, because that has

.stay_at_node set). That is a problem, because all of its I/O functions

assume the BlockCopyState (s->bcs) to be valid and that it has a

filtered child; but after bdrv_backup_top_drop(), neither of those

things are true.

It does not make sense to add new parents to backup-top after

backup_clean(), so we should detach it from the job before

bdrv_backup_top_drop(). Because there is no function to do that for a

single node, just detach all of the job's nodes -- the job does not do

anything past backup_clean() anyway.

Signed-off-by: Max Reitz <mreitz@redhat.com>

Message-Id: <20210219153348.41861-2-mreitz@redhat.com>

block/backup.c | 1 +

1 file changed, 1 insertion(+)

@@ -XXX,XX +XXX,XX @@ static void backup_abort(Job *job)

static void backup_clean(Job *job)

BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);

+ block_job_remove_all_bdrv(&s->common);

bdrv_backup_top_drop(s->backup_top);

}

2.29.2

When the backup-top node transitions from active to inactive in

bdrv_backup_top_drop(), the BlockCopyState is freed and the filtered

child is removed, so the node effectively becomes unusable.

However, noone told its I/O functions this, so they will happily

continue accessing bs->backing and s->bcs. Prevent that by aborting

early when s->active is false.

(After the preceding patch, the node should be gone after

bdrv_backup_top_drop(), so this should largely be a theoretical problem.

But still, better to be safe than sorry, and also I think it just makes

sense to check s->active in the I/O functions.)

Message-Id: <20210219153348.41861-3-mreitz@redhat.com>

block/backup-top.c | 10 ++++++++++

1 file changed, 10 insertions(+)

diff --git a/block/backup-top.c b/block/backup-top.c

--- a/block/backup-top.c

+++ b/block/backup-top.c

@@ -XXX,XX +XXX,XX @@ static coroutine_fn int backup_top_co_preadv(

BlockDriverState *bs, uint64_t offset, uint64_t bytes,

QEMUIOVector *qiov, int flags)

{

+ BDRVBackupTopState *s = bs->opaque;

+ if (!s->active) {

+ return -EIO;

return bdrv_co_preadv(bs->backing, offset, bytes, qiov, flags);

@@ -XXX,XX +XXX,XX @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset,

BDRVBackupTopState *s = bs->opaque;

uint64_t off, end;

+ if (!s->active) {

+ return -EIO;

+ }

if (flags & BDRV_REQ_WRITE_UNCHANGED) {

return 0;

}

2.29.2

Without any of HEAD^ or HEAD^^ applied, qemu will most likely crash on

the qemu-io invocation, for a variety of immediate reasons. The

underlying problem is generally a use-after-free access into

backup-top's BlockCopyState.

With only HEAD^ applied, qemu-io will run into an EIO (which is not

capture by the output, but you can see that the qemu-io invocation will

be accepted (i.e., qemu-io will run) in contrast to the reference

output, where the node name cannot be found), and qemu will then crash

in query-named-block-nodes: bdrv_get_allocated_file_size() detects

backup-top to be a filter and passes the request through to its child.

However, after bdrv_backup_top_drop(), that child is NULL, so the

recursive call crashes.

With HEAD^^ applied, this test should pass.

Message-Id: <20210219153348.41861-4-mreitz@redhat.com>

tests/qemu-iotests/283 | 53 ++++++++++++++++++++++++++++++++++++++

tests/qemu-iotests/283.out | 15 +++++++++++

2 files changed, 68 insertions(+)

diff --git a/tests/qemu-iotests/283 b/tests/qemu-iotests/283

--- a/tests/qemu-iotests/283

+++ b/tests/qemu-iotests/283

@@ -XXX,XX +XXX,XX @@ vm.qmp_log('blockdev-add', **{

vm.qmp_log('blockdev-backup', sync='full', device='source', target='target')

vm.shutdown()

+

+print('\n=== backup-top should be gone after job-finalize ===\n')

+

+# Check that the backup-top node is gone after job-finalize.

+#

+# During finalization, the node becomes inactive and can no longer

+# function. If it is still present, new parents might be attached, and

+# there would be no meaningful way to handle their I/O requests.

+

+vm = iotests.VM()

+vm.launch()

+

+vm.qmp_log('blockdev-add', **{

+ 'node-name': 'source',

+ 'driver': 'null-co',

+})

+

+vm.qmp_log('blockdev-add', **{

+ 'node-name': 'target',

+ 'driver': 'null-co',

+})

+

+vm.qmp_log('blockdev-backup',

+ job_id='backup',

+ device='source',

+ target='target',

+ sync='full',

+ filter_node_name='backup-filter',

+ auto_finalize=False,

+ auto_dismiss=False)

+

+vm.event_wait('BLOCK_JOB_PENDING', 5.0)

+

+# The backup-top filter should still be present prior to finalization

+assert vm.node_info('backup-filter') is not None

+

+vm.qmp_log('job-finalize', id='backup')

+vm.event_wait('BLOCK_JOB_COMPLETED', 5.0)

+

+# The filter should be gone now. Check that by trying to access it

+# with qemu-io (which will most likely crash qemu if it is still

+# there.).

+vm.qmp_log('human-monitor-command',

+ command_line='qemu-io backup-filter "write 0 1M"')

+

+# (Also, do an explicit check.)

+assert vm.node_info('backup-filter') is None

+

+vm.qmp_log('job-dismiss', id='backup')

+vm.event_wait('JOB_STATUS_CHANGE', 5.0, {'data': {'status': 'null'}})

+

+vm.shutdown()

diff --git a/tests/qemu-iotests/283.out b/tests/qemu-iotests/283.out

--- a/tests/qemu-iotests/283.out

+++ b/tests/qemu-iotests/283.out

@@ -XXX,XX +XXX,XX @@

{"return": {}}

{"execute": "blockdev-backup", "arguments": {"device": "source", "sync": "full", "target": "target"}}

{"error": {"class": "GenericError", "desc": "Cannot set permissions for backup-top filter: Conflicts with use by other as 'image', which uses 'write' on base"}}

+

+=== backup-top should be gone after job-finalize ===

+

+{"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "source"}}

+{"return": {}}

+{"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "target"}}

+{"return": {}}

+{"execute": "blockdev-backup", "arguments": {"auto-dismiss": false, "auto-finalize": false, "device": "source", "filter-node-name": "backup-filter", "job-id": "backup", "sync": "full", "target": "target"}}

+{"return": {}}

+{"execute": "job-finalize", "arguments": {"id": "backup"}}

+{"return": {}}

+{"execute": "human-monitor-command", "arguments": {"command-line": "qemu-io backup-filter \"write 0 1M\""}}

+{"return": "Error: Cannot find device= nor node_name=backup-filter\r\n"}

+{"execute": "job-dismiss", "arguments": {"id": "backup"}}

+{"return": {}}

2.29.2

Break some long lines, and relax our type hints to be more generic to

any JSON, in order to more easily permit the additional JSON depth now

possible in migration parameters. Detected by iotest 297.

Fixes: ca4bfec41d56

(qemu-iotests: 300: Add test case for modifying persistence of bitmap)

Reported-by: Kevin Wolf <kwolf@redhat.com>

Message-Id: <20210215220518.1745469-1-eblake@redhat.com>

Reviewed-by: John Snow <jsnow@redhat.com>

tests/qemu-iotests/300 | 10 ++++++----

1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/300

+++ b/tests/qemu-iotests/300