The following changes since commit eb6490f544388dd24c0d054a96dd304bc7284450:

Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20200703' into staging (2020-07-04 16:08:41 +0100)

https://github.com/XanClic/qemu.git tags/pull-block-2020-07-06

for you to fetch changes up to 365fed5111b06d31c1632af63c7528dfe49d62a2:

qed: Simplify backing reads (2020-07-06 10:34:14 +0200)

Block patches for 5.1:

- LUKS keyslot amendment

(+ patches to make the iotests pass on non-Linux systems, and to keep

the tests passing for qcow v1, and to skip LUKS tests (including

qcow2 LUKS) when the built qemu does not support it)

- Refactoring in the block layer: Drop the basically unnecessary

unallocated_blocks_are_zero field from BlockDriverInfo

- Fix qcow2 preallocation when the image size is not a multiple of the

cluster size

- Fix in block-copy code

Alberto Garcia (1):

qcow2: Fix preallocation on images with unaligned sizes

Eric Blake (1):

qed: Simplify backing reads

Max Reitz (5):

iotests: Make _filter_img_create more active

iotests/common.rc: Add _require_working_luks

iotests.py: Add qemu_img_pipe_and_status()

iotests.py: Add (verify|has)_working_luks()

iotests: Check whether luks works

Maxim Levitsky (14):

iotests: filter few more luks specific create options

qcrypto/core: add generic infrastructure for crypto options amendment

qcrypto/luks: implement encryption key management

block/amend: add 'force' option

block/amend: separate amend and create options for qemu-img

block/amend: refactor qcow2 amend options

block/crypto: rename two functions

block/crypto: implement the encryption key management

block/qcow2: extend qemu-img amend interface with crypto options

iotests: qemu-img tests for luks key management

block/core: add generic infrastructure for x-blockdev-amend qmp

command

block/crypto: implement blockdev-amend

block/qcow2: implement blockdev-amend

iotests: add tests for blockdev-amend

Vladimir Sementsov-Ogievskiy (10):

block/block-copy: block_copy_dirty_clusters: fix failure check

qemu-img: convert: don't use unallocated_blocks_are_zero

block: inline bdrv_unallocated_blocks_are_zero()

block/vdi: return ZERO block-status when appropriate

block/vpc: return ZERO block-status when appropriate

block/crypto: drop unallocated_blocks_are_zero

block/iscsi: drop unallocated_blocks_are_zero

block/file-posix: drop unallocated_blocks_are_zero

block/vhdx: drop unallocated_blocks_are_zero

block: drop unallocated_blocks_are_zero

docs/tools/qemu-img.rst | 5 +-

qapi/block-core.json | 68 +++++

qapi/crypto.json | 73 +++++-

qapi/job.json | 4 +-

block/crypto.h | 37 +++

block/qed.h | 1 -

crypto/blockpriv.h | 8 +

include/block/block.h | 7 +-

include/block/block_int.h | 36 ++-

include/crypto/block.h | 22 ++

block.c | 19 +-

block/amend.c | 113 +++++++++

block/block-copy.c | 4 +-

block/crypto.c | 207 +++++++++++++--

block/file-posix.c | 3 -

block/io.c | 8 +-

block/iscsi.c | 1 -

block/qcow2.c | 350 +++++++++++++++-----------

block/qed.c | 65 +----

block/vdi.c | 3 +-

block/vhdx.c | 3 -

block/vpc.c | 3 +-

crypto/block-luks.c | 416 ++++++++++++++++++++++++++++++-

crypto/block.c | 29 +++

qemu-img.c | 48 ++--

block/Makefile.objs | 2 +-

qemu-img-cmds.hx | 4 +-

tests/qemu-iotests/049.out | 102 ++++----

tests/qemu-iotests/061.out | 12 +-

tests/qemu-iotests/082.out | 185 ++++----------

tests/qemu-iotests/085.out | 38 +--

tests/qemu-iotests/087 | 1 +

tests/qemu-iotests/087.out | 6 +-

tests/qemu-iotests/112.out | 2 +-

tests/qemu-iotests/125 | 24 ++

tests/qemu-iotests/125.out | 9 +

tests/qemu-iotests/134.out | 2 +-

tests/qemu-iotests/141 | 2 +-

tests/qemu-iotests/144.out | 4 +-

tests/qemu-iotests/146 | 60 +++--

tests/qemu-iotests/146.out | 405 ++++++++++++++++++++++++++++--

tests/qemu-iotests/153 | 9 +-

tests/qemu-iotests/158.out | 4 +-

tests/qemu-iotests/178 | 1 +

tests/qemu-iotests/182.out | 2 +-

tests/qemu-iotests/185.out | 8 +-

tests/qemu-iotests/188 | 1 +

tests/qemu-iotests/188.out | 2 +-

tests/qemu-iotests/189 | 1 +

tests/qemu-iotests/189.out | 4 +-

tests/qemu-iotests/198 | 1 +

tests/qemu-iotests/198.out | 4 +-

tests/qemu-iotests/206 | 1 +

tests/qemu-iotests/255.out | 8 +-

tests/qemu-iotests/263 | 1 +

tests/qemu-iotests/263.out | 4 +-

tests/qemu-iotests/274.out | 46 ++--

tests/qemu-iotests/280.out | 2 +-

tests/qemu-iotests/284 | 1 +

tests/qemu-iotests/284.out | 6 +-

tests/qemu-iotests/293 | 208 ++++++++++++++++

tests/qemu-iotests/293.out | 99 ++++++++

tests/qemu-iotests/294 | 90 +++++++

tests/qemu-iotests/294.out | 30 +++

tests/qemu-iotests/295 | 280 +++++++++++++++++++++

tests/qemu-iotests/295.out | 40 +++

tests/qemu-iotests/296 | 234 +++++++++++++++++

tests/qemu-iotests/296.out | 33 +++

tests/qemu-iotests/common.filter | 106 ++++++--

tests/qemu-iotests/common.rc | 30 +++

tests/qemu-iotests/group | 4 +

tests/qemu-iotests/iotests.py | 84 +++++--

72 files changed, 3103 insertions(+), 632 deletions(-)

create mode 100644 block/amend.c

create mode 100755 tests/qemu-iotests/293

create mode 100644 tests/qemu-iotests/293.out

create mode 100755 tests/qemu-iotests/294

create mode 100644 tests/qemu-iotests/294.out

create mode 100755 tests/qemu-iotests/295

create mode 100644 tests/qemu-iotests/295.out

create mode 100755 tests/qemu-iotests/296

create mode 100644 tests/qemu-iotests/296.out

2.26.2

From: Alberto Garcia <berto@igalia.com>

When resizing an image with qcow2_co_truncate() using the falloc or

full preallocation modes the code assumes that both the old and new

sizes are cluster-aligned.

There are two problems with this:

1) The calculation of how many clusters are involved does not always

get the right result.

Example: creating a 60KB image and resizing it (with

preallocation=full) to 80KB won't allocate the second cluster.

2) No copy-on-write is performed, so in the previous example if

there is a backing file then the first 60KB of the first cluster

won't be filled with data from the backing file.

This patch fixes both issues.

Signed-off-by: Alberto Garcia <berto@igalia.com>

Message-Id: <20200617140036.20311-1-berto@igalia.com>

Reviewed-by: Eric Blake <eblake@redhat.com>

block/qcow2.c | 17 ++++++++++++++---

tests/qemu-iotests/125 | 24 ++++++++++++++++++++++++

tests/qemu-iotests/125.out | 9 +++++++++

3 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c

index XXXXXXX..XXXXXXX 100644

--- a/block/qcow2.c

+++ b/block/qcow2.c

@@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,

old_file_size = ROUND_UP(old_file_size, s->cluster_size);

}

- nb_new_data_clusters = DIV_ROUND_UP(offset - old_length,

- s->cluster_size);

+ nb_new_data_clusters = (ROUND_UP(offset, s->cluster_size) -

+ start_of_cluster(s, old_length)) >> s->cluster_bits;

/* This is an overestimation; we will not actually allocate space for

* these in the file but just make sure the new refcount structures are

@@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,

int64_t nb_clusters = MIN(

nb_new_data_clusters,

s->l2_slice_size - offset_to_l2_slice_index(s, guest_offset));

- QCowL2Meta allocation = {

+ unsigned cow_start_length = offset_into_cluster(s, guest_offset);

+ QCowL2Meta allocation;

+ guest_offset = start_of_cluster(s, guest_offset);

+ allocation = (QCowL2Meta) {

.offset = guest_offset,

.alloc_offset = host_offset,

.nb_clusters = nb_clusters,

+ .cow_start = {

+ .offset = 0,

+ .nb_bytes = cow_start_length,

+ },

+ .cow_end = {

+ .offset = nb_clusters << s->cluster_bits,

+ .nb_bytes = 0,

+ },

};

qemu_co_queue_init(&allocation.dependent_requests);

diff --git a/tests/qemu-iotests/125 b/tests/qemu-iotests/125

--- a/tests/qemu-iotests/125

+++ b/tests/qemu-iotests/125

@@ -XXX,XX +XXX,XX @@ for GROWTH_SIZE in 16 48 80; do

done

+# Test image resizing using preallocation and unaligned offsets

+$QEMU_IMG create -f raw "$TEST_IMG.base" 128k | _filter_img_create

+$QEMU_IO -c 'write -q -P 1 0 128k' -f raw "$TEST_IMG.base"

+for orig_size in 31k 33k; do

+ echo "--- Resizing image from $orig_size to 96k ---"

+ _make_test_img -F raw -b "$TEST_IMG.base" -o cluster_size=64k "$orig_size"

+ $QEMU_IMG resize -f "$IMGFMT" --preallocation=full "$TEST_IMG" 96k

+ # The first part of the image should contain data from the backing file

+ $QEMU_IO -c "read -q -P 1 0 ${orig_size}" "$TEST_IMG"

+ # The resized part of the image should contain zeroes

+ $QEMU_IO -c "read -q -P 0 ${orig_size} 63k" "$TEST_IMG"

+ # If the image does not have an external data file we can also verify its

+ # actual size. The resized image should have 7 clusters:

+ # header, L1 table, L2 table, refcount table, refcount block, 2 data clusters

+ if ! _get_data_file "$TEST_IMG" > /dev/null; then

+ expected_file_length=$((65536 * 7))

+ file_length=$(stat -c '%s' "$TEST_IMG_FILE")

+ if [ "$file_length" != "$expected_file_length" ]; then

+ echo "ERROR: file length $file_length (expected $expected_file_length)"

+ fi

+ fi

echo '*** done'

diff --git a/tests/qemu-iotests/125.out b/tests/qemu-iotests/125.out

--- a/tests/qemu-iotests/125.out

+++ b/tests/qemu-iotests/125.out

@@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0

wrote 81920/81920 bytes at offset 2048000

80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=raw size=131072

+--- Resizing image from 31k to 96k ---

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=31744 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=raw

+Image resized.

+--- Resizing image from 33k to 96k ---

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33792 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=raw

+Image resized.

*** done

2.26.2

Right now, _filter_img_create just filters out everything that looks

format-dependent, and applies some filename filters. That means that we

have to add another filter line every time some format gets a new

creation option. This can be avoided by instead discarding everything

and just keeping what we know is format-independent (format, size,

backing file, encryption information[1], preallocation) or just

interesting to have in the reference output (external data file path).

Furthermore, we probably want to sort these options. Format drivers are

not required to define them in any specific order, so the output is

effectively random (although this has never bothered us until now). We

need a specific order for our reference outputs, though. Unfortunately,

just using a plain "sort" would change a lot of existing reference

outputs, so we have to pre-filter the option keys to keep our existing

order (fmt, size, backing*, data, encryption info, preallocation).

Finally, this makes it difficult for _filter_img_create to automagically

work for QMP output. Thus, this patch adds a separate

_filter_img_create_for_qmp function that echos every line verbatim that

does not start with "Formatting", and pipes those "Formatting" lines to

_filter_img_create.

[1] Actually, the only thing that is really important is whether

encryption is enabled or not. A patch by Maxim thus removes all

other "encrypt.*" options from the output:

https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg00339.html

But that patch needs to come later so we can get away with changing

as few reference outputs in this patch here as possible.

Signed-off-by: Max Reitz <mreitz@redhat.com>

Message-Id: <20200625125548.870061-2-mreitz@redhat.com>

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

---

tests/qemu-iotests/112.out | 2 +-

tests/qemu-iotests/141 | 2 +-

tests/qemu-iotests/153 | 9 ++-

tests/qemu-iotests/common.filter | 109 ++++++++++++++++++++++++-------

4 files changed, 91 insertions(+), 31 deletions(-)

diff --git a/tests/qemu-iotests/112.out b/tests/qemu-iotests/112.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/112.out

+++ b/tests/qemu-iotests/112.out

@@ -XXX,XX +XXX,XX @@ QA output created by 112

qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits

Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864

qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 refcount_bits=-1

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864

qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits

Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864

qemu-img: TEST_DIR/t.IMGFMT: Refcount width must be a power of two and may not exceed 64 bits

diff --git a/tests/qemu-iotests/141 b/tests/qemu-iotests/141

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/141

+++ b/tests/qemu-iotests/141

@@ -XXX,XX +XXX,XX @@ test_blockjob()

_send_qemu_cmd $QEMU_HANDLE \

"$1" \

"$2" \

- | _filter_img_create | _filter_qmp_empty_return

+ | _filter_img_create_in_qmp | _filter_qmp_empty_return

# We want this to return an error because the block job is still running

_send_qemu_cmd $QEMU_HANDLE \

diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/153

+++ b/tests/qemu-iotests/153

@@ -XXX,XX +XXX,XX @@ done

echo

echo "== Creating ${TEST_IMG}.[abc] ==" | _filter_testdir

-(

- $QEMU_IMG create -f qcow2 "${TEST_IMG}.a" -b "${TEST_IMG}"

- $QEMU_IMG create -f qcow2 "${TEST_IMG}.b" -b "${TEST_IMG}"

- $QEMU_IMG create -f qcow2 "${TEST_IMG}.c" -b "${TEST_IMG}.b"

-) | _filter_img_create

+$QEMU_IMG create -f qcow2 "${TEST_IMG}.a" -b "${TEST_IMG}" | _filter_img_create

+$QEMU_IMG create -f qcow2 "${TEST_IMG}.b" -b "${TEST_IMG}" | _filter_img_create

+$QEMU_IMG create -f qcow2 "${TEST_IMG}.c" -b "${TEST_IMG}.b" \

+ | _filter_img_create

echo

echo "== Two devices sharing the same file in backing chain =="

diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/common.filter

+++ b/tests/qemu-iotests/common.filter

@@ -XXX,XX +XXX,XX @@ _filter_actual_image_size()

# replace driver-specific options in the "Formatting..." line

_filter_img_create()

{

- data_file_filter=()

- if data_file=$(_get_data_file "$TEST_IMG"); then

- data_file_filter=(-e "s# data_file=$data_file##")

+ # Split the line into the pre-options part ($filename_part, which

+ # precedes ", fmt=") and the options part ($options, which starts

+ # with "fmt=")

+ # (And just echo everything before the first "^Formatting")

+ readarray formatting_line < <($SED -e 's/, fmt=/\n/')

+

+ filename_part=''

+ options=''

+ lines=${#formatting_line[@]}

+ for ((i = 0; i < $lines; i++)); do

+ line=${formatting_line[i]}

+ unset formatting_line[i]

+

+ filename_part="$filename_part$line"

+

+ if echo "$line" | grep -q '^Formatting'; then

+ next_i=$((i + 1))

+ if [ -n "${formatting_line[next_i]}" ]; then

+ options="fmt=${formatting_line[@]}"

+ fi

+ break

+ fi

+ done

+

+ # Set grep_data_file to '\|data_file' to keep it; make it empty

+ # to drop it.

+ # We want to drop it if it is part of the global $IMGOPTS, and we

+ # want to keep it otherwise (if the test specifically wants to

+ # test data files).

+ grep_data_file=(-e data_file)

+ if _get_data_file "$TEST_IMG" > /dev/null; then

+ grep_data_file=()

fi

- $SED "${data_file_filter[@]}" \

+ filename_filters=(

-e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \

-e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \

-e "s#$TEST_DIR#TEST_DIR#g" \

-e "s#$SOCK_DIR#SOCK_DIR#g" \

-e "s#$IMGFMT#IMGFMT#g" \

-e 's#nbd:127.0.0.1:[0-9]\\+#TEST_DIR/t.IMGFMT#g' \

- -e 's#nbd+unix:///\??socket=SOCK_DIR/nbd#TEST_DIR/t.IMGFMT#g' \

- -e "s# encryption=off##g" \

- -e "s# cluster_size=[0-9]\\+##g" \

- -e "s# table_size=[0-9]\\+##g" \

- -e "s# compat=[^ ]*##g" \

- -e "s# compat6=\\(on\\|off\\)##g" \

- -e "s# static=\\(on\\|off\\)##g" \

- -e "s# zeroed_grain=\\(on\\|off\\)##g" \

- -e "s# subformat=[^ ]*##g" \

- -e "s# adapter_type=[^ ]*##g" \

- -e "s# hwversion=[^ ]*##g" \

- -e "s# lazy_refcounts=\\(on\\|off\\)##g" \

- -e "s# block_size=[0-9]\\+##g" \

- -e "s# block_state_zero=\\(on\\|off\\)##g" \

- -e "s# log_size=[0-9]\\+##g" \

- -e "s# refcount_bits=[0-9]\\+##g" \

- -e "s# key-secret=[a-zA-Z0-9]\\+##g" \

- -e "s# iter-time=[0-9]\\+##g" \

- -e "s# force_size=\\(on\\|off\\)##g" \

- -e "s# compression_type=[a-zA-Z0-9]\\+##g"

+ -e 's#nbd+unix:///\??socket=SOCK_DIR/nbd#TEST_DIR/t.IMGFMT#g'

+ )

+

+ filename_part=$(echo "$filename_part" | $SED "${filename_filters[@]}")

+

+ # Break the option line before each option (preserving pre-existing

+ # line breaks by replacing them by \0 and restoring them at the end),

+ # then filter out the options we want to keep and sort them according

+ # to some order that all block drivers used at the time of writing

+ # this function.

+ options=$(

+ echo "$options" \

+ | tr '\n' '\0' \

+ | $SED -e 's/ \([a-z0-9_.-]*\)=/\n\1=/g' \

+ | grep -a -e '^fmt' -e '^size' -e '^backing' -e '^preallocation' \

+ -e '^encrypt' "${grep_data_file[@]}" \

+ | $SED "${filename_filters[@]}" \

+ -e 's/^\(fmt\)/0-\1/' \

+ -e 's/^\(size\)/1-\1/' \

+ -e 's/^\(backing\)/2-\1/' \

+ -e 's/^\(data_file\)/3-\1/' \

+ -e 's/^\(encryption\)/4-\1/' \

+ -e 's/^\(encrypt\.format\)/5-\1/' \

+ -e 's/^\(encrypt\.key-secret\)/6-\1/' \

+ -e 's/^\(encrypt\.iter-time\)/7-\1/' \

+ -e 's/^\(preallocation\)/8-\1/' \

+ | sort \

+ | $SED -e 's/^[0-9]-//' \

+ | tr '\n\0' ' \n' \

+ | $SED -e 's/^ *$//' -e 's/ *$//'

+ )

+

+ if [ -n "$options" ]; then

+ echo "$filename_part, $options"

+ elif [ -n "$filename_part" ]; then

+ echo "$filename_part"

+ fi

+}

+

+# Filter the "Formatting..." line in QMP output (leaving the QMP output

+# untouched)

+# (In contrast to _filter_img_create(), this function does not support

+# multi-line Formatting output)

+_filter_img_create_in_qmp()

+{

+ while read -r line; do

+ if echo "$line" | grep -q '^Formatting'; then

+ echo "$line" | _filter_img_create

+ else

+ echo "$line"

+ fi

+ done

}

_filter_img_create_size()

--

2.26.2

From: Maxim Levitsky <mlevitsk@redhat.com>

This allows more tests to be able to have same output on both qcow2 luks encrypted images

and raw luks images

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

Message-Id: <20200625125548.870061-3-mreitz@redhat.com>

---

tests/qemu-iotests/087.out | 6 +++---

tests/qemu-iotests/134.out | 2 +-

tests/qemu-iotests/158.out | 4 ++--

tests/qemu-iotests/188.out | 2 +-

tests/qemu-iotests/189.out | 4 ++--

tests/qemu-iotests/198.out | 4 ++--

tests/qemu-iotests/263.out | 4 ++--

tests/qemu-iotests/284.out | 6 +++---

tests/qemu-iotests/common.filter | 5 +----

9 files changed, 17 insertions(+), 20 deletions(-)

diff --git a/tests/qemu-iotests/087.out b/tests/qemu-iotests/087.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/087.out

+++ b/tests/qemu-iotests/087.out

@@ -XXX,XX +XXX,XX @@ QMP_VERSION

=== Encrypted image QCow ===

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.key-secret=sec0

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on

Testing:

QMP_VERSION

{"return": {}}

@@ -XXX,XX +XXX,XX @@ QMP_VERSION

=== Encrypted image LUKS ===

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encrypt.format=luks encrypt.key-secret=sec0

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728

Testing:

QMP_VERSION

{"return": {}}

@@ -XXX,XX +XXX,XX @@ QMP_VERSION

=== Missing driver ===

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.key-secret=sec0

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on

Testing: -S

QMP_VERSION

{"return": {}}

diff --git a/tests/qemu-iotests/134.out b/tests/qemu-iotests/134.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/134.out

+++ b/tests/qemu-iotests/134.out

@@ -XXX,XX +XXX,XX @@

QA output created by 134

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on encrypt.key-secret=sec0

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encryption=on

== reading whole image ==

read 134217728/134217728 bytes at offset 0

diff --git a/tests/qemu-iotests/158.out b/tests/qemu-iotests/158.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/158.out

+++ b/tests/qemu-iotests/158.out

@@ -XXX,XX +XXX,XX @@

QA output created by 158

== create base ==

-Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728 encryption=on encrypt.key-secret=sec0

+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728 encryption=on

== writing whole image ==

wrote 134217728/134217728 bytes at offset 0

@@ -XXX,XX +XXX,XX @@ wrote 134217728/134217728 bytes at offset 0

read 134217728/134217728 bytes at offset 0

128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

== create overlay ==

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base encryption=on encrypt.key-secret=sec0

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base encryption=on

== writing part of a cluster ==

wrote 1024/1024 bytes at offset 0

diff --git a/tests/qemu-iotests/188.out b/tests/qemu-iotests/188.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/188.out

+++ b/tests/qemu-iotests/188.out

@@ -XXX,XX +XXX,XX @@

QA output created by 188

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216

== reading whole image ==

read 16777216/16777216 bytes at offset 0

diff --git a/tests/qemu-iotests/189.out b/tests/qemu-iotests/189.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/189.out

+++ b/tests/qemu-iotests/189.out

@@ -XXX,XX +XXX,XX @@

QA output created by 189

== create base ==

-Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=16777216 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=16777216

== writing whole image ==

wrote 16777216/16777216 bytes at offset 0

@@ -XXX,XX +XXX,XX @@ wrote 16777216/16777216 bytes at offset 0

read 16777216/16777216 bytes at offset 0

16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

== create overlay ==

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 backing_file=TEST_DIR/t.IMGFMT.base encrypt.format=luks encrypt.key-secret=sec1 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 backing_file=TEST_DIR/t.IMGFMT.base

== writing part of a cluster ==

wrote 1024/1024 bytes at offset 0

diff --git a/tests/qemu-iotests/198.out b/tests/qemu-iotests/198.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/198.out

+++ b/tests/qemu-iotests/198.out

@@ -XXX,XX +XXX,XX @@

QA output created by 198

== create base ==

-Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=16777216 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=16777216

== writing whole image base ==

wrote 16777216/16777216 bytes at offset 0

16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

== create overlay ==

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 backing_file=TEST_DIR/t.IMGFMT.base encrypt.format=luks encrypt.key-secret=sec1 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 backing_file=TEST_DIR/t.IMGFMT.base

== writing whole image layer ==

wrote 16777216/16777216 bytes at offset 0

diff --git a/tests/qemu-iotests/263.out b/tests/qemu-iotests/263.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/263.out

+++ b/tests/qemu-iotests/263.out

@@ -XXX,XX +XXX,XX @@ QA output created by 263

testing LUKS qcow2 encryption

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576

== reading the whole image ==

read 1048576/1048576 bytes at offset 0

1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

@@ -XXX,XX +XXX,XX @@ read 982528/982528 bytes at offset 66048

testing legacy AES qcow2 encryption

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=aes encrypt.key-secret=sec0

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576

== reading the whole image ==

read 1048576/1048576 bytes at offset 0

1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

diff --git a/tests/qemu-iotests/284.out b/tests/qemu-iotests/284.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/284.out

+++ b/tests/qemu-iotests/284.out

@@ -XXX,XX +XXX,XX @@ QA output created by 284

testing LUKS qcow2 encryption

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576

== cluster size 512

== checking image refcounts ==

@@ -XXX,XX +XXX,XX @@ wrote 1/1 bytes at offset 512

== rechecking image refcounts ==

No errors were found on the image.

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576

== cluster size 2048

== checking image refcounts ==

@@ -XXX,XX +XXX,XX @@ wrote 1/1 bytes at offset 2048

== rechecking image refcounts ==

No errors were found on the image.

-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10

+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576

== cluster size 32768

== checking image refcounts ==

diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/common.filter

+++ b/tests/qemu-iotests/common.filter

@@ -XXX,XX +XXX,XX @@ _filter_img_create()

| tr '\n' '\0' \

| $SED -e 's/ \([a-z0-9_.-]*\)=/\n\1=/g' \

| grep -a -e '^fmt' -e '^size' -e '^backing' -e '^preallocation' \

- -e '^encrypt' "${grep_data_file[@]}" \

+ -e '^encryption' "${grep_data_file[@]}" \

| $SED "${filename_filters[@]}" \

-e 's/^\(fmt\)/0-\1/' \

-e 's/^\(size\)/1-\1/' \

-e 's/^\(backing\)/2-\1/' \

-e 's/^\(data_file\)/3-\1/' \

-e 's/^\(encryption\)/4-\1/' \

- -e 's/^\(encrypt\.format\)/5-\1/' \

- -e 's/^\(encrypt\.key-secret\)/6-\1/' \

- -e 's/^\(encrypt\.iter-time\)/7-\1/' \

-e 's/^\(preallocation\)/8-\1/' \

| sort \

| $SED -e 's/^[0-9]-//' \

--

2.26.2

This function will be used by the next patch, which intends to check

both the exit code and qemu-img's output.

Message-Id: <20200625125548.870061-5-mreitz@redhat.com>

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

[mreitz: Rebased on 49438972b8c2e]

tests/qemu-iotests/iotests.py | 40 +++++++++++++++++------------------

1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/iotests.py

+++ b/tests/qemu-iotests/iotests.py

@@ -XXX,XX +XXX,XX @@ import struct

import subprocess

import sys

from typing import (Any, Callable, Dict, Iterable,

- List, Optional, Sequence, TypeVar)

+ List, Optional, Sequence, Tuple, TypeVar)

import unittest

# pylint: disable=import-error, wrong-import-position

@@ -XXX,XX +XXX,XX @@ luks_default_secret_object = 'secret,id=keysec0,data=' + \

luks_default_key_secret_opt = 'key-secret=keysec0'

-def qemu_img(*args):

- '''Run qemu-img and return the exit code'''

- devnull = open('/dev/null', 'r+')

- exitcode = subprocess.call(qemu_img_args + list(args),

- stdin=devnull, stdout=devnull)

- if exitcode < 0:

+def qemu_img_pipe_and_status(*args: str) -> Tuple[str, int]:

+ """

+ Run qemu-img and return both its output and its exit code

+ """

+ subp = subprocess.Popen(qemu_img_args + list(args),

+ stdout=subprocess.PIPE,

+ stderr=subprocess.STDOUT,

+ universal_newlines=True)

+ output = subp.communicate()[0]

+ if subp.returncode < 0:

sys.stderr.write('qemu-img received signal %i: %s\n'

- % (-exitcode, ' '.join(qemu_img_args + list(args))))

- return exitcode

+ % (-subp.returncode,

+ ' '.join(qemu_img_args + list(args))))

+ return (output, subp.returncode)

+

+def qemu_img(*args: str) -> int:

+ '''Run qemu-img and return the exit code'''

+ return qemu_img_pipe_and_status(*args)[1]

def ordered_qmp(qmsg, conv_keys=True):

# Dictionaries are not ordered prior to 3.6, therefore:

@@ -XXX,XX +XXX,XX @@ def qemu_img_verbose(*args):

% (-exitcode, ' '.join(qemu_img_args + list(args))))

return exitcode

-def qemu_img_pipe(*args):

+def qemu_img_pipe(*args: str) -> str:

'''Run qemu-img and return its output'''

- subp = subprocess.Popen(qemu_img_args + list(args),

- stdout=subprocess.PIPE,

- stderr=subprocess.STDOUT,

- universal_newlines=True)

- output = subp.communicate()[0]

- if subp.returncode < 0:

- sys.stderr.write('qemu-img received signal %i: %s\n'

- % (-subp.returncode,

- ' '.join(qemu_img_args + list(args))))

- return output

+ return qemu_img_pipe_and_status(*args)[0]

def qemu_img_log(*args):

result = qemu_img_pipe(*args)

2.26.2

Whenever running an iotest for the luks format, we should check whether

luks actually really works.

Tests that try to create luks-encrypted qcow2 images should do the same.

Signed-off-by: Max Reitz <mreitz@redhat.com>

Message-Id: <20200625125548.870061-7-mreitz@redhat.com>

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

---

tests/qemu-iotests/087 | 1 +

tests/qemu-iotests/178 | 1 +

tests/qemu-iotests/188 | 1 +

tests/qemu-iotests/189 | 1 +

tests/qemu-iotests/198 | 1 +

tests/qemu-iotests/206 | 1 +

tests/qemu-iotests/263 | 1 +

tests/qemu-iotests/284 | 1 +

tests/qemu-iotests/common.rc | 3 +++

tests/qemu-iotests/iotests.py | 5 +++++

10 files changed, 16 insertions(+)

diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087

--- a/tests/qemu-iotests/087

+++ b/tests/qemu-iotests/087

_supported_os Linux

+_require_working_luks

diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178

--- a/tests/qemu-iotests/178

+++ b/tests/qemu-iotests/178

_supported_fmt raw qcow2

+_require_working_luks

echo "== Input validation =="

echo

diff --git a/tests/qemu-iotests/188 b/tests/qemu-iotests/188

--- a/tests/qemu-iotests/188

+++ b/tests/qemu-iotests/188

_supported_fmt qcow2

_supported_proto generic

+_require_working_luks

size=16M

diff --git a/tests/qemu-iotests/189 b/tests/qemu-iotests/189

--- a/tests/qemu-iotests/189

+++ b/tests/qemu-iotests/189

_supported_fmt qcow2

_supported_proto generic

_supported_os Linux

+_require_working_luks

size=16M

diff --git a/tests/qemu-iotests/198 b/tests/qemu-iotests/198

--- a/tests/qemu-iotests/198

+++ b/tests/qemu-iotests/198

_supported_fmt qcow2

_supported_proto generic

_supported_os Linux

+_require_working_luks

size=16M

diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206

--- a/tests/qemu-iotests/206

+++ b/tests/qemu-iotests/206

@@ -XXX,XX +XXX,XX @@ import iotests

from iotests import imgfmt

iotests.script_initialize(supported_fmts=['qcow2'])

+iotests.verify_working_luks()

with iotests.FilePath('t.qcow2') as disk_path, \

iotests.FilePath('t.qcow2.base') as backing_path, \

diff --git a/tests/qemu-iotests/263 b/tests/qemu-iotests/263

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/263

+++ b/tests/qemu-iotests/263

@@ -XXX,XX +XXX,XX @@ trap "_cleanup; exit \$status" 0 1 2 3 15

_supported_fmt qcow2

_supported_proto generic

+_require_working_luks

size=1M

diff --git a/tests/qemu-iotests/284 b/tests/qemu-iotests/284

--- a/tests/qemu-iotests/284

+++ b/tests/qemu-iotests/284

_supported_proto generic

_supported_os Linux

+_require_working_luks

size=1M

@@ -XXX,XX +XXX,XX @@ _supported_fmt()

# setting IMGFMT_GENERIC to false.

for f; do

if [ "$f" = "$IMGFMT" -o "$f" = "generic" -a "$IMGFMT_GENERIC" = "true" ]; then

+ if [ "$IMGFMT" = "luks" ]; then

+ _require_working_luks

+ fi

return

fi

done

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/iotests.py

+++ b/tests/qemu-iotests/iotests.py

@@ -XXX,XX +XXX,XX @@ def _verify_image_format(supported_fmts: Sequence[str] = (),

# similar to

# _supported_fmt generic

# for bash tests

+ if imgfmt == 'luks':

+ verify_working_luks()

return

not_sup = supported_fmts and (imgfmt not in supported_fmts)

if not_sup or (imgfmt in unsupported_fmts):

notrun('not suitable for this image format: %s' % imgfmt)

+ if imgfmt == 'luks':

+ verify_working_luks()

def _verify_protocol(supported: Sequence[str] = (),

unsupported: Sequence[str] = ()) -> None:

assert not (supported and unsupported)

2.26.2

From: Maxim Levitsky <mlevitsk@redhat.com>

This will be used first to implement luks keyslot management.

block_crypto_amend_opts_init will be used to convert

qemu-img cmdline to QCryptoBlockAmendOptions

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Message-Id: <20200608094030.670121-2-mlevitsk@redhat.com>

qapi/crypto.json | 16 ++++++++++++++++

block/crypto.h | 3 +++

crypto/blockpriv.h | 8 ++++++++

include/crypto/block.h | 22 ++++++++++++++++++++++

block/crypto.c | 17 +++++++++++++++++

crypto/block.c | 29 +++++++++++++++++++++++++++++

6 files changed, 95 insertions(+)

diff --git a/qapi/crypto.json b/qapi/crypto.json

--- a/qapi/crypto.json

+++ b/qapi/crypto.json

@@ -XXX,XX +XXX,XX @@

'base': 'QCryptoBlockInfoBase',

'discriminator': 'format',

'data': { 'luks': 'QCryptoBlockInfoLUKS' } }

+

+

+

+##

+# @QCryptoBlockAmendOptions:

+#

+# The options that are available for all encryption formats

+# when amending encryption settings

+#

+# Since: 5.1

+##

+{ 'union': 'QCryptoBlockAmendOptions',

+ 'base': 'QCryptoBlockOptionsBase',

+ 'discriminator': 'format',

+ 'data': {

+ } }

diff --git a/block/crypto.h b/block/crypto.h

index XXXXXXX..XXXXXXX 100644

--- a/block/crypto.h

+++ b/block/crypto.h

@@ -XXX,XX +XXX,XX @@

QCryptoBlockCreateOptions *

block_crypto_create_opts_init(QDict *opts, Error **errp);

+QCryptoBlockAmendOptions *

+block_crypto_amend_opts_init(QDict *opts, Error **errp);

+

QCryptoBlockOpenOptions *

block_crypto_open_opts_init(QDict *opts, Error **errp);

diff --git a/crypto/blockpriv.h b/crypto/blockpriv.h

index XXXXXXX..XXXXXXX 100644

--- a/crypto/blockpriv.h

+++ b/crypto/blockpriv.h

@@ -XXX,XX +XXX,XX @@ struct QCryptoBlockDriver {