[PATCH 00/19] block: LUKS encryption slot management + iotest tweaks

Max Reitz posted 19 patches 3 years, 9 months ago
Test FreeBSD passed
Test docker-quick@centos7 passed
Test checkpatch failed
Test docker-mingw@fedora passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200625125548.870061-1-mreitz@redhat.com
Maintainers: Kevin Wolf <kwolf@redhat.com>, Max Reitz <mreitz@redhat.com>
docs/tools/qemu-img.rst          |   5 +-
qapi/block-core.json             |  68 +++++
qapi/crypto.json                 |  73 +++++-
qapi/job.json                    |   4 +-
block/crypto.h                   |  37 +++
crypto/blockpriv.h               |   8 +
include/block/block.h            |   1 +
include/block/block_int.h        |  24 +-
include/crypto/block.h           |  22 ++
block.c                          |   4 +-
block/amend.c                    | 113 +++++++++
block/crypto.c                   | 206 +++++++++++++--
block/qcow2.c                    | 332 +++++++++++++-----------
crypto/block-luks.c              | 416 ++++++++++++++++++++++++++++++-
crypto/block.c                   |  29 +++
qemu-img.c                       |  44 +++-
block/Makefile.objs              |   2 +-
qemu-img-cmds.hx                 |   4 +-
tests/qemu-iotests/049.out       | 102 ++++----
tests/qemu-iotests/061.out       |  12 +-
tests/qemu-iotests/082.out       | 185 ++++----------
tests/qemu-iotests/085.out       |  38 +--
tests/qemu-iotests/087           |   1 +
tests/qemu-iotests/087.out       |   6 +-
tests/qemu-iotests/112.out       |   2 +-
tests/qemu-iotests/134.out       |   2 +-
tests/qemu-iotests/141           |   2 +-
tests/qemu-iotests/144.out       |   4 +-
tests/qemu-iotests/153           |   9 +-
tests/qemu-iotests/158.out       |   4 +-
tests/qemu-iotests/178           |   1 +
tests/qemu-iotests/182.out       |   2 +-
tests/qemu-iotests/185.out       |   8 +-
tests/qemu-iotests/188           |   1 +
tests/qemu-iotests/188.out       |   2 +-
tests/qemu-iotests/189           |   1 +
tests/qemu-iotests/189.out       |   4 +-
tests/qemu-iotests/198           |   1 +
tests/qemu-iotests/198.out       |   4 +-
tests/qemu-iotests/206           |   1 +
tests/qemu-iotests/255.out       |   8 +-
tests/qemu-iotests/263           |   1 +
tests/qemu-iotests/263.out       |   4 +-
tests/qemu-iotests/274.out       |  46 ++--
tests/qemu-iotests/280.out       |   2 +-
tests/qemu-iotests/284           |   1 +
tests/qemu-iotests/284.out       |   6 +-
tests/qemu-iotests/293           | 208 ++++++++++++++++
tests/qemu-iotests/293.out       |  99 ++++++++
tests/qemu-iotests/294           |  90 +++++++
tests/qemu-iotests/294.out       |  30 +++
tests/qemu-iotests/295           | 280 +++++++++++++++++++++
tests/qemu-iotests/295.out       |  40 +++
tests/qemu-iotests/296           | 234 +++++++++++++++++
tests/qemu-iotests/296.out       |  33 +++
tests/qemu-iotests/common.filter | 106 ++++++--
tests/qemu-iotests/common.rc     |  30 +++
tests/qemu-iotests/group         |   4 +
tests/qemu-iotests/iotests.py    |  78 ++++--
59 files changed, 2591 insertions(+), 493 deletions(-)
create mode 100644 block/amend.c
create mode 100755 tests/qemu-iotests/293
create mode 100644 tests/qemu-iotests/293.out
create mode 100755 tests/qemu-iotests/294
create mode 100644 tests/qemu-iotests/294.out
create mode 100755 tests/qemu-iotests/295
create mode 100644 tests/qemu-iotests/295.out
create mode 100755 tests/qemu-iotests/296
create mode 100644 tests/qemu-iotests/296.out
[PATCH 00/19] block: LUKS encryption slot management + iotest tweaks
Posted by Max Reitz 3 years, 9 months ago
Hi,

This series collects all patches from Maxim’s series “LUKS: encryption
slot management using amend interface”, plus iotest patches that seem
useful to have before it.  (And then I modified some of Maxim’s patches
to take advantage of those patches.)

Maxim’s series was here:
https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg00330.html

I already posted patches 1 and 2 here:
https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg00954.html

Change from the latter series:
- Patch 1:
  - Use $SED instead of plain “sed” in one place I missed
  - Older bash versions do not support readarray’s -d switch, so we have
    to separate the Formatting line by a \n instead of \0.
    Unfortunately, that means we cannot just split by \0 and be done
    with it, but we have to ensure we put everything before the line
    starting by “Formatting” into $filename_part, and we do that with a
    loop that iterates over the split array until it finds the
    “Formatting” line.
  - Use multiple -e for grep instead of '(...|...|...)'
- Patch 2: Rebase fallout du to grep now using multiple -e parameters


git-backport-diff against Maxim’s series:

001/19:[down] 'iotests: Make _filter_img_create more active'
002/19:[0053] [FC] 'iotests: filter few more luks specific create options'
003/19:[down] 'iotests/common.rc: Add _require_working_luks'
004/19:[down] 'iotests.py: Add qemu_img_pipe_and_status()'
005/19:[down] 'iotests.py: Add (verify|has)_working_luks()'
006/19:[down] 'iotests: Check whether luks works'
007/19:[----] [--] 'qcrypto/core: add generic infrastructure for crypto options amendment'
008/19:[----] [--] 'qcrypto/luks: implement encryption key management'
009/19:[----] [--] 'block/amend: add 'force' option'
010/19:[----] [--] 'block/amend: separate amend and create options for qemu-img'
011/19:[0268] [FC] 'block/amend: refactor qcow2 amend options'
012/19:[----] [--] 'block/crypto: rename two functions'
013/19:[----] [--] 'block/crypto: implement the encryption key management'
014/19:[----] [--] 'block/qcow2: extend qemu-img amend interface with crypto options'
015/19:[0007] [FC] 'iotests: qemu-img tests for luks key management'
016/19:[----] [--] 'block/core: add generic infrastructure for x-blockdev-amend qmp command'
017/19:[----] [--] 'block/crypto: implement blockdev-amend'
018/19:[----] [--] 'block/qcow2: implement blockdev-amend'
019/19:[0005] [FC] 'iotests: add tests for blockdev-amend'


git-backport-diff against “iotests: Make _filter_img_create more
active”:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/19:[0035] [FC] 'iotests: Make _filter_img_create more active'
002/19:[0004] [FC] 'iotests: filter few more luks specific create options'
003/19:[down] 'iotests/common.rc: Add _require_working_luks'
004/19:[down] 'iotests.py: Add qemu_img_pipe_and_status()'
005/19:[down] 'iotests.py: Add (verify|has)_working_luks()'
006/19:[down] 'iotests: Check whether luks works'
007/19:[down] 'qcrypto/core: add generic infrastructure for crypto options amendment'
008/19:[down] 'qcrypto/luks: implement encryption key management'
009/19:[down] 'block/amend: add 'force' option'
010/19:[down] 'block/amend: separate amend and create options for qemu-img'
011/19:[down] 'block/amend: refactor qcow2 amend options'
012/19:[down] 'block/crypto: rename two functions'
013/19:[down] 'block/crypto: implement the encryption key management'
014/19:[down] 'block/qcow2: extend qemu-img amend interface with crypto options'
015/19:[down] 'iotests: qemu-img tests for luks key management'
016/19:[down] 'block/core: add generic infrastructure for x-blockdev-amend qmp command'
017/19:[down] 'block/crypto: implement blockdev-amend'
018/19:[down] 'block/qcow2: implement blockdev-amend'
019/19:[down] 'iotests: add tests for blockdev-amend'


Max Reitz (5):
  iotests: Make _filter_img_create more active
  iotests/common.rc: Add _require_working_luks
  iotests.py: Add qemu_img_pipe_and_status()
  iotests.py: Add (verify|has)_working_luks()
  iotests: Check whether luks works

Maxim Levitsky (14):
  iotests: filter few more luks specific create options
  qcrypto/core: add generic infrastructure for crypto options amendment
  qcrypto/luks: implement encryption key management
  block/amend: add 'force' option
  block/amend: separate amend and create options for qemu-img
  block/amend: refactor qcow2 amend options
  block/crypto: rename two functions
  block/crypto: implement the encryption key management
  block/qcow2: extend qemu-img amend interface with crypto options
  iotests: qemu-img tests for luks key management
  block/core: add generic infrastructure for x-blockdev-amend qmp
    command
  block/crypto: implement blockdev-amend
  block/qcow2: implement blockdev-amend
  iotests: add tests for blockdev-amend

 docs/tools/qemu-img.rst          |   5 +-
 qapi/block-core.json             |  68 +++++
 qapi/crypto.json                 |  73 +++++-
 qapi/job.json                    |   4 +-
 block/crypto.h                   |  37 +++
 crypto/blockpriv.h               |   8 +
 include/block/block.h            |   1 +
 include/block/block_int.h        |  24 +-
 include/crypto/block.h           |  22 ++
 block.c                          |   4 +-
 block/amend.c                    | 113 +++++++++
 block/crypto.c                   | 206 +++++++++++++--
 block/qcow2.c                    | 332 +++++++++++++-----------
 crypto/block-luks.c              | 416 ++++++++++++++++++++++++++++++-
 crypto/block.c                   |  29 +++
 qemu-img.c                       |  44 +++-
 block/Makefile.objs              |   2 +-
 qemu-img-cmds.hx                 |   4 +-
 tests/qemu-iotests/049.out       | 102 ++++----
 tests/qemu-iotests/061.out       |  12 +-
 tests/qemu-iotests/082.out       | 185 ++++----------
 tests/qemu-iotests/085.out       |  38 +--
 tests/qemu-iotests/087           |   1 +
 tests/qemu-iotests/087.out       |   6 +-
 tests/qemu-iotests/112.out       |   2 +-
 tests/qemu-iotests/134.out       |   2 +-
 tests/qemu-iotests/141           |   2 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/153           |   9 +-
 tests/qemu-iotests/158.out       |   4 +-
 tests/qemu-iotests/178           |   1 +
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/185.out       |   8 +-
 tests/qemu-iotests/188           |   1 +
 tests/qemu-iotests/188.out       |   2 +-
 tests/qemu-iotests/189           |   1 +
 tests/qemu-iotests/189.out       |   4 +-
 tests/qemu-iotests/198           |   1 +
 tests/qemu-iotests/198.out       |   4 +-
 tests/qemu-iotests/206           |   1 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/263           |   1 +
 tests/qemu-iotests/263.out       |   4 +-
 tests/qemu-iotests/274.out       |  46 ++--
 tests/qemu-iotests/280.out       |   2 +-
 tests/qemu-iotests/284           |   1 +
 tests/qemu-iotests/284.out       |   6 +-
 tests/qemu-iotests/293           | 208 ++++++++++++++++
 tests/qemu-iotests/293.out       |  99 ++++++++
 tests/qemu-iotests/294           |  90 +++++++
 tests/qemu-iotests/294.out       |  30 +++
 tests/qemu-iotests/295           | 280 +++++++++++++++++++++
 tests/qemu-iotests/295.out       |  40 +++
 tests/qemu-iotests/296           | 234 +++++++++++++++++
 tests/qemu-iotests/296.out       |  33 +++
 tests/qemu-iotests/common.filter | 106 ++++++--
 tests/qemu-iotests/common.rc     |  30 +++
 tests/qemu-iotests/group         |   4 +
 tests/qemu-iotests/iotests.py    |  78 ++++--
 59 files changed, 2591 insertions(+), 493 deletions(-)
 create mode 100644 block/amend.c
 create mode 100755 tests/qemu-iotests/293
 create mode 100644 tests/qemu-iotests/293.out
 create mode 100755 tests/qemu-iotests/294
 create mode 100644 tests/qemu-iotests/294.out
 create mode 100755 tests/qemu-iotests/295
 create mode 100644 tests/qemu-iotests/295.out
 create mode 100755 tests/qemu-iotests/296
 create mode 100644 tests/qemu-iotests/296.out

-- 
2.26.2


Re: [PATCH 00/19] block: LUKS encryption slot management + iotest tweaks
Posted by no-reply@patchew.org 3 years, 9 months ago
Patchew URL: https://patchew.org/QEMU/20200625125548.870061-1-mreitz@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [PATCH 00/19] block: LUKS encryption slot management + iotest tweaks
Type: series
Message-id: 20200625125548.870061-1-mreitz@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

From https://github.com/patchew-project/qemu
   27c77b1..0250c59  master     -> master
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/20200625125548.870061-1-mreitz@redhat.com -> patchew/20200625125548.870061-1-mreitz@redhat.com
Switched to a new branch 'test'
e117214 iotests: add tests for blockdev-amend
cae47ee block/qcow2: implement blockdev-amend
88311ff block/crypto: implement blockdev-amend
a0a3f15 block/core: add generic infrastructure for x-blockdev-amend qmp command
1384b4f iotests: qemu-img tests for luks key management
49c50ee block/qcow2: extend qemu-img amend interface with crypto options
f83b5c1 block/crypto: implement the encryption key management
ff40d29 block/crypto: rename two functions
67b2028 block/amend: refactor qcow2 amend options
90caec5 block/amend: separate amend and create options for qemu-img
2df2e01 block/amend: add 'force' option
340bac8 qcrypto/luks: implement encryption key management
9d2d1ed qcrypto/core: add generic infrastructure for crypto options amendment
235478b iotests: Check whether luks works
5dd17f0 iotests.py: Add (verify|has)_working_luks()
5d918f7 iotests.py: Add qemu_img_pipe_and_status()
4330ee1 iotests/common.rc: Add _require_working_luks
c365035 iotests: filter few more luks specific create options
daa65e4 iotests: Make _filter_img_create more active

=== OUTPUT BEGIN ===
1/19 Checking commit daa65e49e845 (iotests: Make _filter_img_create more active)
2/19 Checking commit c3650352b6fd (iotests: filter few more luks specific create options)
3/19 Checking commit 4330ee13d808 (iotests/common.rc: Add _require_working_luks)
4/19 Checking commit 5d918f763241 (iotests.py: Add qemu_img_pipe_and_status())
5/19 Checking commit 5dd17f035980 (iotests.py: Add (verify|has)_working_luks())
6/19 Checking commit 235478b9f3d3 (iotests: Check whether luks works)
7/19 Checking commit 9d2d1ed28ae9 (qcrypto/core: add generic infrastructure for crypto options amendment)
8/19 Checking commit 340bac8a40ad (qcrypto/luks: implement encryption key management)
9/19 Checking commit 2df2e01ef83b (block/amend: add 'force' option)
10/19 Checking commit 90caec5acfcd (block/amend: separate amend and create options for qemu-img)
ERROR: Macros with multiple statements should be enclosed in a do - while loop
#34: FILE: block/qcow2.c:5652:
+#define QCOW_COMMON_OPTIONS                                         \
+    {                                                               \
+        .name = BLOCK_OPT_SIZE,                                     \
+        .type = QEMU_OPT_SIZE,                                      \
+        .help = "Virtual disk size"                                 \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_COMPAT_LEVEL,                             \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Compatibility level (v2 [0.10] or v3 [1.1])"       \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_BACKING_FILE,                             \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "File name of a base image"                         \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_BACKING_FMT,                              \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Image format of the base image"                    \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_DATA_FILE,                                \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "File name of an external data file"                \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_DATA_FILE_RAW,                            \
+        .type = QEMU_OPT_BOOL,                                      \
+        .help = "The external data file must stay valid "           \
+                "as a raw image"                                    \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_ENCRYPT,                                  \
+        .type = QEMU_OPT_BOOL,                                      \
+        .help = "Encrypt the image with format 'aes'. (Deprecated " \
+                "in favor of " BLOCK_OPT_ENCRYPT_FORMAT "=aes)",    \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_ENCRYPT_FORMAT,                           \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Encrypt the image, format choices: 'aes', 'luks'", \
+    },                                                              \
+    BLOCK_CRYPTO_OPT_DEF_KEY_SECRET("encrypt.",                     \
+        "ID of secret providing qcow AES key or LUKS passphrase"),  \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_CIPHER_ALG("encrypt."),               \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_CIPHER_MODE("encrypt."),              \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_IVGEN_ALG("encrypt."),                \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_IVGEN_HASH_ALG("encrypt."),           \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_HASH_ALG("encrypt."),                 \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_ITER_TIME("encrypt."),                \
+    {                                                               \
+        .name = BLOCK_OPT_CLUSTER_SIZE,                             \
+        .type = QEMU_OPT_SIZE,                                      \
+        .help = "qcow2 cluster size",                               \
+        .def_value_str = stringify(DEFAULT_CLUSTER_SIZE)            \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_PREALLOC,                                 \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Preallocation mode (allowed values: off, "         \
+                "metadata, falloc, full)"                           \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_LAZY_REFCOUNTS,                           \
+        .type = QEMU_OPT_BOOL,                                      \
+        .help = "Postpone refcount updates",                        \
+        .def_value_str = "off"                                      \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_REFCOUNT_BITS,                            \
+        .type = QEMU_OPT_NUMBER,                                    \
+        .help = "Width of a reference count entry in bits",         \
+        .def_value_str = "16"                                       \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_COMPRESSION_TYPE,                         \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Compression method used for image cluster "        \
+                "compression",                                      \
+        .def_value_str = "zlib"                                     \
+    }

total: 1 errors, 0 warnings, 244 lines checked

Patch 10/19 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

11/19 Checking commit 67b20281fba1 (block/amend: refactor qcow2 amend options)
12/19 Checking commit ff40d29caf37 (block/crypto: rename two functions)
13/19 Checking commit f83b5c15d7e2 (block/crypto: implement the encryption key management)
14/19 Checking commit 49c50ee64bcd (block/qcow2: extend qemu-img amend interface with crypto options)
15/19 Checking commit 1384b4f4917d (iotests: qemu-img tests for luks key management)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#18: 
new file mode 100755

total: 0 errors, 1 warnings, 433 lines checked

Patch 15/19 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
16/19 Checking commit a0a3f156abe8 (block/core: add generic infrastructure for x-blockdev-amend qmp command)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#33: 
new file mode 100644

total: 0 errors, 1 warnings, 221 lines checked

Patch 16/19 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
17/19 Checking commit 88311ff56332 (block/crypto: implement blockdev-amend)
18/19 Checking commit cae47ee034f7 (block/qcow2: implement blockdev-amend)
19/19 Checking commit e11721462ab2 (iotests: add tests for blockdev-amend)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#18: 
new file mode 100755

total: 0 errors, 1 warnings, 593 lines checked

Patch 19/19 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200625125548.870061-1-mreitz@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH 00/19] block: LUKS encryption slot management + iotest tweaks
Posted by Max Reitz 3 years, 9 months ago
On 25.06.20 14:55, Max Reitz wrote:
> Hi,
> 
> This series collects all patches from Maxim’s series “LUKS: encryption
> slot management using amend interface”, plus iotest patches that seem
> useful to have before it.  (And then I modified some of Maxim’s patches
> to take advantage of those patches.)
> 
> Maxim’s series was here:
> https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg00330.html
> 
> I already posted patches 1 and 2 here:
> https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg00954.html
> 
> Change from the latter series:
> - Patch 1:
>   - Use $SED instead of plain “sed” in one place I missed
>   - Older bash versions do not support readarray’s -d switch, so we have
>     to separate the Formatting line by a \n instead of \0.
>     Unfortunately, that means we cannot just split by \0 and be done
>     with it, but we have to ensure we put everything before the line
>     starting by “Formatting” into $filename_part, and we do that with a
>     loop that iterates over the split array until it finds the
>     “Formatting” line.
>   - Use multiple -e for grep instead of '(...|...|...)'
> - Patch 2: Rebase fallout du to grep now using multiple -e parameters

Thanks for reviewing, Maxim; (re-)applied to my block branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block