The following changes since commit 3fbd3405d2b0604ea530fc7a1828f19da1e95ff9:

Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2019-08-17' into staging (2019-08-19 14:14:09 +0100)

https://github.com/XanClic/qemu.git tags/pull-block-2019-08-19

for you to fetch changes up to fa27c478102a6b5d1c6b02c005607ad9404b915f:

doc: Preallocation does not require writing zeroes (2019-08-19 17:13:26 +0200)

Block patches:

- preallocation=falloc/full support for LUKS

- Various minor fixes

Maxim Levitsky (1):

LUKS: support preallocation

qapi/block-core.json | 15 +++++---

include/block/block.h | 1 +

include/block/block_int.h | 9 +++++

block.c | 21 +++++++++++

block/crypto.c | 30 ++++++++++++++--

block/file-posix.c | 1 +

block/file-win32.c | 1 +

block/gluster.c | 4 +++

block/mirror.c | 11 ++++--

block/nfs.c | 1 +

block/parallels.c | 2 +-

block/qcow2.c | 30 +++++++++++++++-

block/qed.c | 1 +

block/raw-format.c | 6 ++++

block/rbd.c | 1 +

block/sheepdog.c | 1 +

block/ssh.c | 1 +

block/vdi.c | 16 +++++++--

block/vhdx.c | 28 +++++++++++++--

block/vmdk.c | 3 ++

block/vpc.c | 2 +-

blockdev.c | 16 +++++++--

qemu-img.c | 11 ++++--

tests/test-block-iothread.c | 2 +-

docs/qemu-block-drivers.texi | 4 +--

qemu-img.texi | 4 +--

tests/qemu-iotests/041 | 62 +++++++++++++++++++++++++++++---

tests/qemu-iotests/041.out | 4 +--

tests/qemu-iotests/122 | 17 +++++++++

tests/qemu-iotests/122.out | 8 +++++

tests/qemu-iotests/141 | 9 +++--

tests/qemu-iotests/141.out | 5 ---

tests/qemu-iotests/188 | 20 ++++++++++-

tests/qemu-iotests/188.out | 4 +++

tests/qemu-iotests/common.filter | 5 +++

35 files changed, 313 insertions(+), 43 deletions(-)

2.21.0

When preallocating an encrypted qcow2 image, it just lets the protocol

driver write data and then does not mark the clusters as zero.

Therefore, reading this image will yield effectively random data.

As such, we have not fulfilled the promise of always writing zeroes when

preallocating an image in a while. It seems that nobody has really

cared, so change the documentation to conform to qemu's actual behavior.

Signed-off-by: Max Reitz <mreitz@redhat.com>

Message-id: 20190711132935.13070-1-mreitz@redhat.com

Reviewed-by: Eric Blake <eblake@redhat.com>

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

qapi/block-core.json | 9 +++++----

docs/qemu-block-drivers.texi | 4 ++--

qemu-img.texi | 4 ++--

3 files changed, 9 insertions(+), 8 deletions(-)

# @off: no preallocation

# @metadata: preallocate only for metadata

# @falloc: like @full preallocation but allocate disk space by

-# posix_fallocate() rather than writing zeros.

-# @full: preallocate all data by writing zeros to device to ensure disk

-# space is really available. @full preallocation also sets up

-# metadata correctly.

+# posix_fallocate() rather than writing data.

+# @full: preallocate all data by writing it to the device to ensure

+# disk space is really available. This data may or may not be

+# zero, depending on the image format and storage.

+# @full preallocation also sets up metadata correctly.

# Since: 2.2

diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi

--- a/docs/qemu-block-drivers.texi

+++ b/docs/qemu-block-drivers.texi

@@ -XXX,XX +XXX,XX @@ Supported options:

@item preallocation

Preallocation mode (allowed values: @code{off}, @code{falloc}, @code{full}).

@code{falloc} mode preallocates space for image by calling posix_fallocate().

-@code{full} mode preallocates space for image by writing zeros to underlying

-storage.

+@code{full} mode preallocates space for image by writing data to underlying

+storage. This data may or may not be zero, depending on the storage location.

@end table

@item qcow2

diff --git a/qemu-img.texi b/qemu-img.texi

index XXXXXXX..XXXXXXX 100644

--- a/qemu-img.texi

+++ b/qemu-img.texi

@@ -XXX,XX +XXX,XX @@ Supported options:

@item preallocation

Preallocation mode (allowed values: @code{off}, @code{falloc}, @code{full}).

@code{falloc} mode preallocates space for image by calling posix_fallocate().

-@code{full} mode preallocates space for image by writing zeros to underlying

-storage.

+@code{full} mode preallocates space for image by writing data to underlying

+storage. This data may or may not be zero, depending on the storage location.

@end table

@item qcow2

2.21.0

bdrv_has_zero_init() only has meaning for newly created images or image

areas. If the mirror job itself did not create the image, it cannot

rely on bdrv_has_zero_init()'s result to carry any meaning.

This is the case for drive-mirror with mode=existing and always for

blockdev-mirror.

Note that we only have to zero-initialize the target with sync=full,

because other modes actually do not promise that the target will contain

the same data as the source after the job -- sync=top only promises to

copy anything allocated in the top layer, and sync=none will only copy

new I/O. (Which is how mirror has always handled it.)

Signed-off-by: Max Reitz <mreitz@redhat.com>

Message-id: 20190724171239.8764-3-mreitz@redhat.com

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

---

include/block/block_int.h | 2 ++

block/mirror.c | 11 ++++++++---

blockdev.c | 16 +++++++++++++---

tests/test-block-iothread.c | 2 +-

4 files changed, 24 insertions(+), 7 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h

--- a/include/block/block_int.h

+++ b/include/block/block_int.h

@@ -XXX,XX +XXX,XX @@ BlockJob *commit_active_start(const char *job_id, BlockDriverState *bs,

* @buf_size: The amount of data that can be in flight at one time.

* @mode: Whether to collapse all images in the chain to the target.

* @backing_mode: How to establish the target's backing chain after completion.

+ * @zero_target: Whether the target should be explicitly zero-initialized

* @on_source_error: The action to take upon error reading from the source.

* @on_target_error: The action to take upon error writing to the target.

* @unmap: Whether to unmap target where source sectors only contain zeroes.

@@ -XXX,XX +XXX,XX @@ void mirror_start(const char *job_id, BlockDriverState *bs,

int creation_flags, int64_t speed,

uint32_t granularity, int64_t buf_size,

MirrorSyncMode mode, BlockMirrorBackingMode backing_mode,

+ bool zero_target,

BlockdevOnError on_source_error,

BlockdevOnError on_target_error,

bool unmap, const char *filter_node_name,

diff --git a/block/mirror.c b/block/mirror.c

--- a/block/mirror.c

+++ b/block/mirror.c

@@ -XXX,XX +XXX,XX @@ typedef struct MirrorBlockJob {

Error *replace_blocker;

bool is_none_mode;

BlockMirrorBackingMode backing_mode;

+ /* Whether the target image requires explicit zero-initialization */

+ bool zero_target;

MirrorCopyMode copy_mode;

BlockdevOnError on_source_error, on_target_error;

bool synced;

@@ -XXX,XX +XXX,XX @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)

int ret;

int64_t count;

- if (base == NULL && !bdrv_has_zero_init(target_bs)) {

+ if (s->zero_target) {

if (!bdrv_can_write_zeroes_with_unmap(target_bs)) {

bdrv_set_dirty_bitmap(s->dirty_bitmap, 0, s->bdev_length);

return 0;

@@ -XXX,XX +XXX,XX @@ static BlockJob *mirror_start_job(

const char *replaces, int64_t speed,

uint32_t granularity, int64_t buf_size,

BlockMirrorBackingMode backing_mode,

+ bool zero_target,

BlockdevOnError on_source_error,

BlockdevOnError on_target_error,

bool unmap,

@@ -XXX,XX +XXX,XX @@ static BlockJob *mirror_start_job(

s->on_target_error = on_target_error;

s->is_none_mode = is_none_mode;

s->backing_mode = backing_mode;

+ s->zero_target = zero_target;

s->copy_mode = copy_mode;

s->base = base;

s->granularity = granularity;

@@ -XXX,XX +XXX,XX @@ void mirror_start(const char *job_id, BlockDriverState *bs,

int creation_flags, int64_t speed,

uint32_t granularity, int64_t buf_size,

MirrorSyncMode mode, BlockMirrorBackingMode backing_mode,

+ bool zero_target,

BlockdevOnError on_source_error,

BlockdevOnError on_target_error,

bool unmap, const char *filter_node_name,

@@ -XXX,XX +XXX,XX @@ void mirror_start(const char *job_id, BlockDriverState *bs,

is_none_mode = mode == MIRROR_SYNC_MODE_NONE;

base = mode == MIRROR_SYNC_MODE_TOP ? backing_bs(bs) : NULL;

mirror_start_job(job_id, bs, creation_flags, target, replaces,

- speed, granularity, buf_size, backing_mode,

+ speed, granularity, buf_size, backing_mode, zero_target,

on_source_error, on_target_error, unmap, NULL, NULL,

&mirror_job_driver, is_none_mode, base, false,

filter_node_name, true, copy_mode, errp);

@@ -XXX,XX +XXX,XX @@ BlockJob *commit_active_start(const char *job_id, BlockDriverState *bs,

ret = mirror_start_job(

job_id, bs, creation_flags, base, NULL, speed, 0, 0,

- MIRROR_LEAVE_BACKING_CHAIN,

+ MIRROR_LEAVE_BACKING_CHAIN, false,

on_error, on_error, true, cb, opaque,

&commit_active_job_driver, false, base, auto_complete,

filter_node_name, false, MIRROR_COPY_MODE_BACKGROUND,

diff --git a/blockdev.c b/blockdev.c

--- a/blockdev.c

+++ b/blockdev.c

@@ -XXX,XX +XXX,XX @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,

bool has_replaces, const char *replaces,

enum MirrorSyncMode sync,

BlockMirrorBackingMode backing_mode,

+ bool zero_target,

bool has_speed, int64_t speed,

bool has_granularity, uint32_t granularity,

bool has_buf_size, int64_t buf_size,

@@ -XXX,XX +XXX,XX @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,

*/

mirror_start(job_id, bs, target,

has_replaces ? replaces : NULL, job_flags,

- speed, granularity, buf_size, sync, backing_mode,

+ speed, granularity, buf_size, sync, backing_mode, zero_target,

on_source_error, on_target_error, unmap, filter_node_name,

copy_mode, errp);

@@ -XXX,XX +XXX,XX @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)

int flags;

int64_t size;

const char *format = arg->format;

+ bool zero_target;

int ret;

bs = qmp_get_root_bs(arg->device, errp);

@@ -XXX,XX +XXX,XX @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)

+ zero_target = (arg->sync == MIRROR_SYNC_MODE_FULL &&

+ (arg->mode == NEW_IMAGE_MODE_EXISTING ||

+ !bdrv_has_zero_init(target_bs)));

+

ret = bdrv_try_set_aio_context(target_bs, aio_context, errp);

if (ret < 0) {

bdrv_unref(target_bs);

@@ -XXX,XX +XXX,XX @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)

blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,

arg->has_replaces, arg->replaces, arg->sync,

- backing_mode, arg->has_speed, arg->speed,

+ backing_mode, zero_target,

+ arg->has_speed, arg->speed,

arg->has_granularity, arg->granularity,

arg->has_buf_size, arg->buf_size,

arg->has_on_source_error, arg->on_source_error,

@@ -XXX,XX +XXX,XX @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,

AioContext *aio_context;

BlockMirrorBackingMode backing_mode = MIRROR_LEAVE_BACKING_CHAIN;

Error *local_err = NULL;

+ bool zero_target;

int ret;

bs = qmp_get_root_bs(device, errp);

@@ -XXX,XX +XXX,XX @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,

+ zero_target = (sync == MIRROR_SYNC_MODE_FULL);

+

aio_context = bdrv_get_aio_context(bs);

aio_context_acquire(aio_context);

@@ -XXX,XX +XXX,XX @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,

blockdev_mirror_common(has_job_id ? job_id : NULL, bs, target_bs,

has_replaces, replaces, sync, backing_mode,

- has_speed, speed,

+ zero_target, has_speed, speed,

has_granularity, granularity,

has_buf_size, buf_size,

has_on_source_error, on_source_error,

diff --git a/tests/test-block-iothread.c b/tests/test-block-iothread.c

--- a/tests/test-block-iothread.c

+++ b/tests/test-block-iothread.c

@@ -XXX,XX +XXX,XX @@ static void test_propagate_mirror(void)

/* Start a mirror job */

mirror_start("job0", src, target, NULL, JOB_DEFAULT, 0, 0, 0,

- MIRROR_SYNC_MODE_NONE, MIRROR_OPEN_BACKING_CHAIN,

+ MIRROR_SYNC_MODE_NONE, MIRROR_OPEN_BACKING_CHAIN, false,

BLOCKDEV_ON_ERROR_REPORT, BLOCKDEV_ON_ERROR_REPORT,

false, "filter_node", MIRROR_COPY_MODE_BACKGROUND,

&error_abort);

2.21.0

If a qcow2 file is preallocated, it can no longer guarantee that it

initially appears as filled with zeroes.

So implement .bdrv_has_zero_init() by checking whether the file is

preallocated; if so, forward the call to the underlying storage node,

except for when it is encrypted: Encrypted preallocated images always

return effectively random data, so .bdrv_has_zero_init() must always

return 0 for them.

.bdrv_has_zero_init_truncate() can remain bdrv_has_zero_init_1(),

because it presupposes PREALLOC_MODE_OFF.

Reported-by: Stefano Garzarella <sgarzare@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

Message-id: 20190724171239.8764-7-mreitz@redhat.com

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

---

block/qcow2.c | 29 ++++++++++++++++++++++++++++-

1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/block/qcow2.c b/block/qcow2.c

--- a/block/qcow2.c

+++ b/block/qcow2.c

@@ -XXX,XX +XXX,XX @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,

return spec_info;