Hi all!

These series are here to address the following problem:
block-status-above functions may consider space after EOF of
intermediate backing files as unallocated, which is wrong, as these
backing files are the reason of producing zeroes, we never go further by
backing chain after a short backing file. So, if such short-backing file
is _inside_ requested sub-chain of the backing chain, we should never
report space after its EOF as unallocated.

See patches 01,04,05 for details.

Note, that this series leaves for another day the general problem
around block-status: misuse of BDRV_BLOCK_ALLOCATED as is-fs-allocated
vs go-to-backing.
Audit for this problem is done here:
"backing chain & block status & filters"
https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg04706.html
And I'm going to prepare series to address this problem.

Also, get_block_status func have same disease, but remains unfixed here:
I want to make separate series for it.

v7:
- add Alberto's r-bs in all patchs
- rebase to new backing-chain handling, bdrv_filter_or_cow_bs is used instead of backing_bs


Based on series "[PATCH v9 0/7] coroutines: generate wrapper code" or
in other words:
Based-on: <20200924185414.28642-1-vsementsov@virtuozzo.com>

Vladimir Sementsov-Ogievskiy (5):
  block/io: fix bdrv_co_block_status_above
  block/io: bdrv_common_block_status_above: support include_base
  block/io: bdrv_common_block_status_above: support bs == base
  block/io: fix bdrv_is_allocated_above
  iotests: add commit top->base cases to 274

 block/coroutines.h         |   2 +
 block/io.c                 | 132 +++++++++++++++++++++----------------
 block/qcow2.c              |  16 ++++-
 tests/qemu-iotests/274     |  20 ++++++
 tests/qemu-iotests/274.out |  68 +++++++++++++++++++
 5 files changed, 179 insertions(+), 59 deletions(-)

-- 
2.21.3

On Thu, Sep 24, 2020 at 10:39:58PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> These series are here to address the following problem:
> block-status-above functions may consider space after EOF of
> intermediate backing files as unallocated, which is wrong, as these
> backing files are the reason of producing zeroes, we never go further by
> backing chain after a short backing file. So, if such short-backing file
> is _inside_ requested sub-chain of the backing chain, we should never
> report space after its EOF as unallocated.
> 
> See patches 01,04,05 for details.
> 
> Note, that this series leaves for another day the general problem
> around block-status: misuse of BDRV_BLOCK_ALLOCATED as is-fs-allocated
> vs go-to-backing.
> Audit for this problem is done here:
> "backing chain & block status & filters"
> https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg04706.html
> And I'm going to prepare series to address this problem.
> 
> Also, get_block_status func have same disease, but remains unfixed here:
> I want to make separate series for it.
> 
> v7:
> - add Alberto's r-bs in all patchs
> - rebase to new backing-chain handling, bdrv_filter_or_cow_bs is used instead of backing_bs
> 
> 
> Based on series "[PATCH v9 0/7] coroutines: generate wrapper code" or
> in other words:
> Based-on: <20200924185414.28642-1-vsementsov@virtuozzo.com>
> 
> Vladimir Sementsov-Ogievskiy (5):
>   block/io: fix bdrv_co_block_status_above
>   block/io: bdrv_common_block_status_above: support include_base
>   block/io: bdrv_common_block_status_above: support bs == base
>   block/io: fix bdrv_is_allocated_above
>   iotests: add commit top->base cases to 274
> 
>  block/coroutines.h         |   2 +
>  block/io.c                 | 132 +++++++++++++++++++++----------------
>  block/qcow2.c              |  16 ++++-
>  tests/qemu-iotests/274     |  20 ++++++
>  tests/qemu-iotests/274.out |  68 +++++++++++++++++++
>  5 files changed, 179 insertions(+), 59 deletions(-)
> 
> -- 
> 2.21.3
> 

Thanks, applied to my block tree with Eric's spelling fixes:
https://github.com/stefanha/qemu/commits/block

Stefan

Patchew URL: https://patchew.org/QEMU/20200924194003.22080-1-vsementsov@virtuozzo.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

C linker for the host machine: cc ld.bfd 2.27-43
Host machine cpu family: x86_64
Host machine cpu: x86_64
../src/meson.build:10: WARNING: Module unstable-keyval has no backwards or forwards compatibility and might not exist in future releases.
Program sh found: YES
Program python3 found: YES (/usr/bin/python3)
Configuring ninjatool using configuration
---
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11406: ordinal not in range(128)
Generating 'libqemu-aarch64-softmmu.fa.p/decode-neon-shared.c.inc'.
make: *** [block/block-gen.c.stamp] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 709, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--rm', '--label', 'com.qemu.instance.uuid=528b329e049d459c994676e3ba6dc69a', '-u', '1003', '--security-opt', 'seccomp=unconfined', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-e72243g9/src/docker-src.2020-09-24-17.42.11.20907:/var/tmp/qemu:z,ro', 'qemu/centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=528b329e049d459c994676e3ba6dc69a
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-e72243g9/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    3m12.373s
user    0m16.084s


The full log is available at
http://patchew.org/logs/20200924194003.22080-1-vsementsov@virtuozzo.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

It's all because underlying "[PATCH v9 0/7] coroutines: generate wrapper code" series, I've answered in "[PATCH v9 0/7] coroutines: generate wrapper code" thread.

25.09.2020 00:45, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20200924194003.22080-1-vsementsov@virtuozzo.com/
> 
> 
> 
> Hi,
> 
> This series failed the docker-quick@centos7 build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> make docker-image-centos7 V=1 NETWORK=1
> time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
> === TEST SCRIPT END ===
> 
> C linker for the host machine: cc ld.bfd 2.27-43
> Host machine cpu family: x86_64
> Host machine cpu: x86_64
> ../src/meson.build:10: WARNING: Module unstable-keyval has no backwards or forwards compatibility and might not exist in future releases.
> Program sh found: YES
> Program python3 found: YES (/usr/bin/python3)
> Configuring ninjatool using configuration
> ---
>      return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11406: ordinal not in range(128)
> Generating 'libqemu-aarch64-softmmu.fa.p/decode-neon-shared.c.inc'.
> make: *** [block/block-gen.c.stamp] Error 1
> make: *** Waiting for unfinished jobs....
> Traceback (most recent call last):
>    File "./tests/docker/docker.py", line 709, in <module>
> ---
>      raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--rm', '--label', 'com.qemu.instance.uuid=528b329e049d459c994676e3ba6dc69a', '-u', '1003', '--security-opt', 'seccomp=unconfined', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-e72243g9/src/docker-src.2020-09-24-17.42.11.20907:/var/tmp/qemu:z,ro', 'qemu/centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
> filter=--filter=label=com.qemu.instance.uuid=528b329e049d459c994676e3ba6dc69a
> make[1]: *** [docker-run] Error 1
> make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-e72243g9/src'
> make: *** [docker-run-test-quick@centos7] Error 2
> 
> real    3m12.373s
> user    0m16.084s
> 
> 
> The full log is available at
> http://patchew.org/logs/20200924194003.22080-1-vsementsov@virtuozzo.com/testing.docker-quick@centos7/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
> 


-- 
Best regards,
Vladimir