[PULL 00/22] implement discard operation for Parallels images

Denis V. Lunev posted 22 patches 7 months, 1 week ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20230920092108.258898-1-den@openvz.org
Maintainers: Stefan Hajnoczi <stefanha@redhat.com>, "Denis V. Lunev" <den@openvz.org>, Kevin Wolf <kwolf@redhat.com>, Hanna Reitz <hreitz@redhat.com>
There is a newer version of this series
block/parallels.c                             | 389 ++++++++++++++++++++------
block/parallels.h                             |   3 +
tests/qemu-iotests/131                        |  52 ++++
tests/qemu-iotests/131.out                    |  60 ++++
tests/qemu-iotests/tests/parallels-checks     |  76 ++++-
tests/qemu-iotests/tests/parallels-checks.out |  65 ++++-
6 files changed, 544 insertions(+), 101 deletions(-)
[PULL 00/22] implement discard operation for Parallels images
Posted by Denis V. Lunev 7 months, 1 week ago
The following changes since commit 4907644841e3200aea6475c0f72d3d987e9f3d93:

  Merge tag 'mem-2023-09-19' of https://github.com/davidhildenbrand/qemu into staging (2023-09-19 13:22:19 -0400)

are available in the Git repository at:

  https://src.openvz.org/scm/~den/qemu.git tags/pull-parallels-2023-09-20

for you to fetch changes up to ead1064587ba6534aa2c3da6383713a009dafcb1:

  tests: extend test 131 to cover availability of the write-zeroes (2023-09-20 10:14:15 +0200)

----------------------------------------------------------------
Parallels format driver:
* regular calculation of cluster used bitmap of the image file
* cluster allocation on the base of that bitmap (effectively allocation of
  new clusters could be done inside the image if that offset space is unused)
* support of DISCARD and WRITE_ZEROES operations
* image check bugfixes
* unit tests fixes
* unit tests covering new functionality

----------------------------------------------------------------
Denis V. Lunev (22):
      parallels: fix formatting in bdrv_parallels initialization
      parallels: mark driver as supporting CBT
      parallels: fix memory leak in parallels_open()
      parallels: invent parallels_opts_prealloc() helper to parse prealloc opts
      parallels: return earler in fail_format branch in parallels_open()
      parallels: return earlier from parallels_open() function on error
      parallels: refactor path when we need to re-check image in parallels_open
      parallels: create mark_used() helper which sets bit in used bitmap
      tests: ensure that image validation will not cure the corruption
      parallels: fix broken parallels_check_data_off()
      parallels: add test which will validate data_off fixes through repair
      parallels: collect bitmap of used clusters at open
      tests: fix broken deduplication check in parallels format test
      tests: test self-cure of parallels image with duplicated clusters
      parallels: accept multiple clusters in mark_used()
      parallels: update used bitmap in allocate_cluster
      parallels: naive implementation of allocate_clusters with used bitmap
      parallels: improve readability of allocate_clusters
      parallels: naive implementation of parallels_co_pdiscard
      tests: extend test 131 to cover availability of the discard operation
      parallels: naive implementation of parallels_co_pwrite_zeroes
      tests: extend test 131 to cover availability of the write-zeroes

 block/parallels.c                             | 389 ++++++++++++++++++++------
 block/parallels.h                             |   3 +
 tests/qemu-iotests/131                        |  52 ++++
 tests/qemu-iotests/131.out                    |  60 ++++
 tests/qemu-iotests/tests/parallels-checks     |  76 ++++-
 tests/qemu-iotests/tests/parallels-checks.out |  65 ++++-
 6 files changed, 544 insertions(+), 101 deletions(-)

-- 
2.34.1
Re: [PULL 00/22] implement discard operation for Parallels images
Posted by Stefan Hajnoczi 7 months, 1 week ago
On Wed, 20 Sept 2023 at 05:22, Denis V. Lunev <den@openvz.org> wrote:
>
> The following changes since commit 4907644841e3200aea6475c0f72d3d987e9f3d93:
>
>   Merge tag 'mem-2023-09-19' of https://github.com/davidhildenbrand/qemu into staging (2023-09-19 13:22:19 -0400)
>
> are available in the Git repository at:
>
>   https://src.openvz.org/scm/~den/qemu.git tags/pull-parallels-2023-09-20

Hi Denis,
Please take a look at the following CI failure. I have dropped this
series for now.

clang -m64 -mcx16 -Ilibblock.fa.p -I. -I.. -Iqapi -Itrace -Iui
-Iui/shader -Iblock -I/usr/include/p11-kit-1 -I/usr/include/uuid
-I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include
-I/usr/include/sysprof-4 -flto -fcolor-diagnostics -Wall -Winvalid-pch
-Werror -std=gnu11 -O2 -g -fstack-protector-strong
-fsanitize=safe-stack -Wundef -Wwrite-strings -Wmissing-prototypes
-Wstrict-prototypes -Wredundant-decls -Wold-style-definition
-Wtype-limits -Wformat-security -Wformat-y2k -Winit-self
-Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels
-Wexpansion-to-defined -Wmissing-format-attribute
-Wno-initializer-overrides -Wno-missing-include-dirs
-Wno-shift-negative-value -Wno-string-plus-int
-Wno-typedef-redefinition -Wno-tautological-type-limit-compare
-Wno-psabi -Wno-gnu-variable-sized-type-not-at-end -Wthread-safety
-isystem /builds/qemu-project/qemu/linux-headers -isystem
linux-headers -iquote . -iquote /builds/qemu-project/qemu -iquote
/builds/qemu-project/qemu/include -iquote
/builds/qemu-project/qemu/host/include/x86_64 -iquote
/builds/qemu-project/qemu/host/include/generic -iquote
/builds/qemu-project/qemu/tcg/i386 -pthread -D_GNU_SOURCE
-D_LARGEFILE_SOURCE -fno-strict-aliasing -fno-common -fwrapv
-fsanitize=cfi-icall -fsanitize-cfi-icall-generalize-pointers
-fno-sanitize-trap=cfi-icall -fPIE -D_FILE_OFFSET_BITS=64
-D__USE_FILE_OFFSET64 -D__USE_LARGEFILE64 -DUSE_POSIX_ACLS=1 -MD -MQ
libblock.fa.p/block_parallels.c.o -MF
libblock.fa.p/block_parallels.c.o.d -o
libblock.fa.p/block_parallels.c.o -c ../block/parallels.c
../block/parallels.c:210:21: error: calling function
'bdrv_co_getlength' requires holding mutex 'graph_lock'
[-Werror,-Wthread-safety-analysis]
payload_bytes = bdrv_co_getlength(bs->file->bs);
^
../block/parallels.c:572:15: error: calling function
'bdrv_co_pdiscard' requires holding mutex 'graph_lock'
[-Werror,-Wthread-safety-analysis]
ret = bdrv_co_pdiscard(bs->file, host_off, s->cluster_size);
^
2 errors generated.

https://gitlab.com/qemu-project/qemu/-/jobs/5131277794

Stefan

>
> for you to fetch changes up to ead1064587ba6534aa2c3da6383713a009dafcb1:
>
>   tests: extend test 131 to cover availability of the write-zeroes (2023-09-20 10:14:15 +0200)
>
> ----------------------------------------------------------------
> Parallels format driver:
> * regular calculation of cluster used bitmap of the image file
> * cluster allocation on the base of that bitmap (effectively allocation of
>   new clusters could be done inside the image if that offset space is unused)
> * support of DISCARD and WRITE_ZEROES operations
> * image check bugfixes
> * unit tests fixes
> * unit tests covering new functionality
>
> ----------------------------------------------------------------
> Denis V. Lunev (22):
>       parallels: fix formatting in bdrv_parallels initialization
>       parallels: mark driver as supporting CBT
>       parallels: fix memory leak in parallels_open()
>       parallels: invent parallels_opts_prealloc() helper to parse prealloc opts
>       parallels: return earler in fail_format branch in parallels_open()
>       parallels: return earlier from parallels_open() function on error
>       parallels: refactor path when we need to re-check image in parallels_open
>       parallels: create mark_used() helper which sets bit in used bitmap
>       tests: ensure that image validation will not cure the corruption
>       parallels: fix broken parallels_check_data_off()
>       parallels: add test which will validate data_off fixes through repair
>       parallels: collect bitmap of used clusters at open
>       tests: fix broken deduplication check in parallels format test
>       tests: test self-cure of parallels image with duplicated clusters
>       parallels: accept multiple clusters in mark_used()
>       parallels: update used bitmap in allocate_cluster
>       parallels: naive implementation of allocate_clusters with used bitmap
>       parallels: improve readability of allocate_clusters
>       parallels: naive implementation of parallels_co_pdiscard
>       tests: extend test 131 to cover availability of the discard operation
>       parallels: naive implementation of parallels_co_pwrite_zeroes
>       tests: extend test 131 to cover availability of the write-zeroes
>
>  block/parallels.c                             | 389 ++++++++++++++++++++------
>  block/parallels.h                             |   3 +
>  tests/qemu-iotests/131                        |  52 ++++
>  tests/qemu-iotests/131.out                    |  60 ++++
>  tests/qemu-iotests/tests/parallels-checks     |  76 ++++-
>  tests/qemu-iotests/tests/parallels-checks.out |  65 ++++-
>  6 files changed, 544 insertions(+), 101 deletions(-)
>
> --
> 2.34.1
>
>
Re: [PULL 00/22] implement discard operation for Parallels images
Posted by Denis V. Lunev 7 months, 1 week ago
On 9/20/23 19:55, Stefan Hajnoczi wrote:
> On Wed, 20 Sept 2023 at 05:22, Denis V. Lunev <den@openvz.org> wrote:
>> The following changes since commit 4907644841e3200aea6475c0f72d3d987e9f3d93:
>>
>>    Merge tag 'mem-2023-09-19' of https://github.com/davidhildenbrand/qemu into staging (2023-09-19 13:22:19 -0400)
>>
>> are available in the Git repository at:
>>
>>    https://src.openvz.org/scm/~den/qemu.git tags/pull-parallels-2023-09-20
> Hi Denis,
> Please take a look at the following CI failure. I have dropped this
> series for now.
>
> clang -m64 -mcx16 -Ilibblock.fa.p -I. -I.. -Iqapi -Itrace -Iui
> -Iui/shader -Iblock -I/usr/include/p11-kit-1 -I/usr/include/uuid
> -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include
> -I/usr/include/sysprof-4 -flto -fcolor-diagnostics -Wall -Winvalid-pch
> -Werror -std=gnu11 -O2 -g -fstack-protector-strong
> -fsanitize=safe-stack -Wundef -Wwrite-strings -Wmissing-prototypes
> -Wstrict-prototypes -Wredundant-decls -Wold-style-definition
> -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self
> -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels
> -Wexpansion-to-defined -Wmissing-format-attribute
> -Wno-initializer-overrides -Wno-missing-include-dirs
> -Wno-shift-negative-value -Wno-string-plus-int
> -Wno-typedef-redefinition -Wno-tautological-type-limit-compare
> -Wno-psabi -Wno-gnu-variable-sized-type-not-at-end -Wthread-safety
> -isystem /builds/qemu-project/qemu/linux-headers -isystem
> linux-headers -iquote . -iquote /builds/qemu-project/qemu -iquote
> /builds/qemu-project/qemu/include -iquote
> /builds/qemu-project/qemu/host/include/x86_64 -iquote
> /builds/qemu-project/qemu/host/include/generic -iquote
> /builds/qemu-project/qemu/tcg/i386 -pthread -D_GNU_SOURCE
> -D_LARGEFILE_SOURCE -fno-strict-aliasing -fno-common -fwrapv
> -fsanitize=cfi-icall -fsanitize-cfi-icall-generalize-pointers
> -fno-sanitize-trap=cfi-icall -fPIE -D_FILE_OFFSET_BITS=64
> -D__USE_FILE_OFFSET64 -D__USE_LARGEFILE64 -DUSE_POSIX_ACLS=1 -MD -MQ
> libblock.fa.p/block_parallels.c.o -MF
> libblock.fa.p/block_parallels.c.o.d -o
> libblock.fa.p/block_parallels.c.o -c ../block/parallels.c
> ../block/parallels.c:210:21: error: calling function
> 'bdrv_co_getlength' requires holding mutex 'graph_lock'
> [-Werror,-Wthread-safety-analysis]
> payload_bytes = bdrv_co_getlength(bs->file->bs);
> ^
> ../block/parallels.c:572:15: error: calling function
> 'bdrv_co_pdiscard' requires holding mutex 'graph_lock'
> [-Werror,-Wthread-safety-analysis]
> ret = bdrv_co_pdiscard(bs->file, host_off, s->cluster_size);
> ^
> 2 errors generated.
>
> https://gitlab.com/qemu-project/qemu/-/jobs/5131277794
>
> Stefan
>
>

It seems that GCC and CLANG environments are different
nowadays. I have had a smell of that but have not have
a proof. Will try to understand.

Den