[Qemu-devel] [PATCH v5 00/42] block: Deal with filters

Max Reitz posted 42 patches 13 weeks ago
Failed in applying to current master (apply log)
qapi/block-core.json          |   4 +
include/block/block.h         |   2 +
include/block/block_int.h     | 109 ++++---
block.c                       | 523 +++++++++++++++++++++++++++++-----
block/backup.c                |   9 +-
block/blkdebug.c              |   7 +-
block/blklogwrites.c          |   1 -
block/block-backend.c         |  16 +-
block/commit.c                | 100 +++++--
block/copy-on-read.c          |  13 +-
block/io.c                    | 115 ++++----
block/mirror.c                | 113 ++++++--
block/qapi.c                  |  42 +--
block/qcow2.c                 |   9 +
block/snapshot.c              |  74 +++--
block/stream.c                |  23 +-
block/throttle.c              |  11 +-
blockdev.c                    | 139 +++++++--
nbd/server.c                  |   6 +-
qemu-img.c                    |  36 +--
tests/qemu-iotests/020        |  36 +++
tests/qemu-iotests/020.out    |  10 +
tests/qemu-iotests/040        | 238 ++++++++++++++++
tests/qemu-iotests/040.out    |   4 +-
tests/qemu-iotests/041        | 270 +++++++++++++++++-
tests/qemu-iotests/041.out    |   4 +-
tests/qemu-iotests/184.out    |   7 +-
tests/qemu-iotests/191.out    |   1 -
tests/qemu-iotests/204.out    |   1 +
tests/qemu-iotests/228        |   6 +-
tests/qemu-iotests/228.out    |   6 +-
tests/qemu-iotests/245        |   4 +-
tests/qemu-iotests/iotests.py |  10 +-
33 files changed, 1610 insertions(+), 339 deletions(-)

[Qemu-devel] [PATCH v5 00/42] block: Deal with filters

Posted by Max Reitz 13 weeks ago
Hi,

When we introduced filters, we did it a bit casually.  Sure, we talked a
lot about them before, but that was mostly discussion about where
implicit filters should be added to the graph (note that we currently
only have two implicit filters, those being mirror and commit).  But in
the end, we really just designated some drivers filters (Quorum,
blkdebug, etc.) and added some specifically (throttle, COR), without
really looking through the block layer to see where issues might occur.

It turns out vast areas of the block layer just don’t know about filters
and cannot really handle them.  Many cases will work in practice, in
others, well, too bad, you cannot use some feature because some part
deep inside the block layer looks at your filters and thinks they are
format nodes.

This is one reason why this series is needed.  Over time (since v1), a
second reason has made its way in:

bs->file is not necessarily the place where a node’s data is stored.
qcow2 now has external data files, and currently there is no way for the
general block layer to know that the data is not stored in bs->file.
Right now, I do not think that has any real consequences (all functions
that need access to the actual data storage file should only do so as a
fallback if the driver does not provide some functionality, but qcow2
should provide it all), but it still shows that we need some way to let
the general block layer know about such data files.  (Also, I will need
this for v1 of my “Inquire images’ rotational info” series.)

I won’t go on and on about this series now, I think the patches pretty
much speak for themselves now.  If the cover letter gets too long,
nobody reads it anyway (see previous versions).


*** This series depends on some others. ***

Dependencies:
- [PATCH 0/4] block: Keep track of parent quiescing
- [PATCH 0/2] vl: Drain before (block) job cancel when quitting
- [PATCH v2 0/2] blockdev: Overlays are not snapshots

Based-on: <20190605161118.14544-1-mreitz@redhat.com>
Based-on: <20190612220839.1374-1-mreitz@redhat.com>
Based-on: <20190603202236.1342-1-mreitz@redhat.com>


v5:
- Split the huge patches 2 and 3 from the previous version into many
  smaller patches to maintain the potential reviewers’ sanity [Vladimir]

- Added support for compressed writes to the COR and throttle filter
  drivers to demonstrate how that looks, because the backup job needs to
  deal with filters that have such support

- Added differentiation between bdrv_storage_child(),
  bdrv_primary_child(), and bdrv_metadata_child()

- A whole lot of things Vladimir has noted

- Made the block jobs really work with filters.  In case of commit and
  stream, this now means that filters go away if they are between top
  and base.  I think that’s OK because it’s the user’s choice to include
  filters or not.  (They can move the filters around if they prefer a
  different result.)
  - This changes the “Add filter commit test cases” from checking that
    most things do not work to checking that they do

- Added the “blockdev: Fix active commit choice” patch because it turned
  out this became necessary after I allowed committing through and with
  filters.


Max Reitz (42):
  block: Mark commit and mirror as filter drivers
  copy-on-read: Support compressed writes
  throttle: Support compressed writes
  block: Add child access functions
  block: Add chain helper functions
  qcow2: Implement .bdrv_storage_child()
  block: *filtered_cow_child() for *has_zero_init()
  block: bdrv_set_backing_hd() is about bs->backing
  block: Include filters when freezing backing chain
  block: Use CAF in bdrv_is_encrypted()
  block: Add bdrv_supports_compressed_writes()
  block: Use bdrv_filtered_rw* where obvious
  block: Use CAFs in block status functions
  block: Use CAFs when working with backing chains
  block: Re-evaluate backing file handling in reopen
  block: Use child access functions when flushing
  block: Use CAFs in bdrv_refresh_limits()
  block: Use CAFs in bdrv_refresh_filename()
  block: Use CAF in bdrv_co_rw_vmstate()
  block/snapshot: Fall back to storage child
  block: Use CAFs for debug breakpoints
  block: Use CAFs in bdrv_get_allocated_file_size()
  blockdev: Use CAF in external_snapshot_prepare()
  block: Use child access functions for QAPI queries
  mirror: Deal with filters
  backup: Deal with filters
  commit: Deal with filters
  stream: Deal with filters
  nbd: Use CAF when looking for dirty bitmap
  qemu-img: Use child access functions
  block: Drop backing_bs()
  block: Make bdrv_get_cumulative_perm() public
  blockdev: Fix active commit choice
  block: Inline bdrv_co_block_status_from_*()
  block: Fix check_to_replace_node()
  iotests: Add tests for mirror @replaces loops
  block: Leave BDS.backing_file constant
  iotests: Let complete_and_wait() work with commit
  iotests: Add filter commit test cases
  iotests: Add filter mirror test cases
  iotests: Add test for commit in sub directory
  iotests: Test committing to overridden backing

 qapi/block-core.json          |   4 +
 include/block/block.h         |   2 +
 include/block/block_int.h     | 109 ++++---
 block.c                       | 523 +++++++++++++++++++++++++++++-----
 block/backup.c                |   9 +-
 block/blkdebug.c              |   7 +-
 block/blklogwrites.c          |   1 -
 block/block-backend.c         |  16 +-
 block/commit.c                | 100 +++++--
 block/copy-on-read.c          |  13 +-
 block/io.c                    | 115 ++++----
 block/mirror.c                | 113 ++++++--
 block/qapi.c                  |  42 +--
 block/qcow2.c                 |   9 +
 block/snapshot.c              |  74 +++--
 block/stream.c                |  23 +-
 block/throttle.c              |  11 +-
 blockdev.c                    | 139 +++++++--
 nbd/server.c                  |   6 +-
 qemu-img.c                    |  36 +--
 tests/qemu-iotests/020        |  36 +++
 tests/qemu-iotests/020.out    |  10 +
 tests/qemu-iotests/040        | 238 ++++++++++++++++
 tests/qemu-iotests/040.out    |   4 +-
 tests/qemu-iotests/041        | 270 +++++++++++++++++-
 tests/qemu-iotests/041.out    |   4 +-
 tests/qemu-iotests/184.out    |   7 +-
 tests/qemu-iotests/191.out    |   1 -
 tests/qemu-iotests/204.out    |   1 +
 tests/qemu-iotests/228        |   6 +-
 tests/qemu-iotests/228.out    |   6 +-
 tests/qemu-iotests/245        |   4 +-
 tests/qemu-iotests/iotests.py |  10 +-
 33 files changed, 1610 insertions(+), 339 deletions(-)

-- 
2.21.0


Re: [Qemu-devel] [PATCH v5 00/42] block: Deal with filters

Posted by Vladimir Sementsov-Ogievskiy 13 weeks ago
13.06.2019 1:09, Max Reitz wrote:
> Hi,
> 
> When we introduced filters, we did it a bit casually.  Sure, we talked a
> lot about them before, but that was mostly discussion about where
> implicit filters should be added to the graph (note that we currently
> only have two implicit filters, those being mirror and commit).  But in
> the end, we really just designated some drivers filters (Quorum,
> blkdebug, etc.) and added some specifically (throttle, COR), without
> really looking through the block layer to see where issues might occur.
> 
> It turns out vast areas of the block layer just don’t know about filters
> and cannot really handle them.  Many cases will work in practice, in
> others, well, too bad, you cannot use some feature because some part
> deep inside the block layer looks at your filters and thinks they are
> format nodes.
> 
> This is one reason why this series is needed.  Over time (since v1), a
> second reason has made its way in:
> 
> bs->file is not necessarily the place where a node’s data is stored.
> qcow2 now has external data files, and currently there is no way for the
> general block layer to know that the data is not stored in bs->file.
> Right now, I do not think that has any real consequences (all functions
> that need access to the actual data storage file should only do so as a
> fallback if the driver does not provide some functionality, but qcow2
> should provide it all), but it still shows that we need some way to let
> the general block layer know about such data files.  (Also, I will need
> this for v1 of my “Inquire images’ rotational info” series.)
> 
> I won’t go on and on about this series now, I think the patches pretty
> much speak for themselves now.  If the cover letter gets too long,
> nobody reads it anyway (see previous versions).
> 
> 
> *** This series depends on some others. ***
> 
> Dependencies:
> - [PATCH 0/4] block: Keep track of parent quiescing
> - [PATCH 0/2] vl: Drain before (block) job cancel when quitting
> - [PATCH v2 0/2] blockdev: Overlays are not snapshots
> 
> Based-on: <20190605161118.14544-1-mreitz@redhat.com>
> Based-on: <20190612220839.1374-1-mreitz@redhat.com>
> Based-on: <20190603202236.1342-1-mreitz@redhat.com>

Could you please export a branch?

> 
> 
> v5:
> - Split the huge patches 2 and 3 from the previous version into many
>    smaller patches to maintain the potential reviewers’ sanity [Vladimir]

Thank you! In spite of frightening amount of patches, reviewing became a lot
simpler.

> 
> - Added support for compressed writes to the COR and throttle filter
>    drivers to demonstrate how that looks, because the backup job needs to
>    deal with filters that have such support
> 
> - Added differentiation between bdrv_storage_child(),
>    bdrv_primary_child(), and bdrv_metadata_child()
> 
> - A whole lot of things Vladimir has noted
> 
> - Made the block jobs really work with filters.  In case of commit and
>    stream, this now means that filters go away if they are between top
>    and base.  I think that’s OK because it’s the user’s choice to include
>    filters or not.  (They can move the filters around if they prefer a
>    different result.)
>    - This changes the “Add filter commit test cases” from checking that
>      most things do not work to checking that they do
> 
> - Added the “blockdev: Fix active commit choice” patch because it turned
>    out this became necessary after I allowed committing through and with
>    filters.
> 
> 
> Max Reitz (42):
>    block: Mark commit and mirror as filter drivers
>    copy-on-read: Support compressed writes
>    throttle: Support compressed writes
>    block: Add child access functions
>    block: Add chain helper functions
>    qcow2: Implement .bdrv_storage_child()
>    block: *filtered_cow_child() for *has_zero_init()
>    block: bdrv_set_backing_hd() is about bs->backing
>    block: Include filters when freezing backing chain
>    block: Use CAF in bdrv_is_encrypted()
>    block: Add bdrv_supports_compressed_writes()
>    block: Use bdrv_filtered_rw* where obvious
>    block: Use CAFs in block status functions
>    block: Use CAFs when working with backing chains
>    block: Re-evaluate backing file handling in reopen
>    block: Use child access functions when flushing
>    block: Use CAFs in bdrv_refresh_limits()
>    block: Use CAFs in bdrv_refresh_filename()
>    block: Use CAF in bdrv_co_rw_vmstate()
>    block/snapshot: Fall back to storage child
>    block: Use CAFs for debug breakpoints
>    block: Use CAFs in bdrv_get_allocated_file_size()
>    blockdev: Use CAF in external_snapshot_prepare()
>    block: Use child access functions for QAPI queries
>    mirror: Deal with filters
>    backup: Deal with filters
>    commit: Deal with filters
>    stream: Deal with filters
>    nbd: Use CAF when looking for dirty bitmap
>    qemu-img: Use child access functions
>    block: Drop backing_bs()
>    block: Make bdrv_get_cumulative_perm() public
>    blockdev: Fix active commit choice
>    block: Inline bdrv_co_block_status_from_*()
>    block: Fix check_to_replace_node()
>    iotests: Add tests for mirror @replaces loops
>    block: Leave BDS.backing_file constant
>    iotests: Let complete_and_wait() work with commit
>    iotests: Add filter commit test cases
>    iotests: Add filter mirror test cases
>    iotests: Add test for commit in sub directory
>    iotests: Test committing to overridden backing
> 
>   qapi/block-core.json          |   4 +
>   include/block/block.h         |   2 +
>   include/block/block_int.h     | 109 ++++---
>   block.c                       | 523 +++++++++++++++++++++++++++++-----
>   block/backup.c                |   9 +-
>   block/blkdebug.c              |   7 +-
>   block/blklogwrites.c          |   1 -
>   block/block-backend.c         |  16 +-
>   block/commit.c                | 100 +++++--
>   block/copy-on-read.c          |  13 +-
>   block/io.c                    | 115 ++++----
>   block/mirror.c                | 113 ++++++--
>   block/qapi.c                  |  42 +--
>   block/qcow2.c                 |   9 +
>   block/snapshot.c              |  74 +++--
>   block/stream.c                |  23 +-
>   block/throttle.c              |  11 +-
>   blockdev.c                    | 139 +++++++--
>   nbd/server.c                  |   6 +-
>   qemu-img.c                    |  36 +--
>   tests/qemu-iotests/020        |  36 +++
>   tests/qemu-iotests/020.out    |  10 +
>   tests/qemu-iotests/040        | 238 ++++++++++++++++
>   tests/qemu-iotests/040.out    |   4 +-
>   tests/qemu-iotests/041        | 270 +++++++++++++++++-
>   tests/qemu-iotests/041.out    |   4 +-
>   tests/qemu-iotests/184.out    |   7 +-
>   tests/qemu-iotests/191.out    |   1 -
>   tests/qemu-iotests/204.out    |   1 +
>   tests/qemu-iotests/228        |   6 +-
>   tests/qemu-iotests/228.out    |   6 +-
>   tests/qemu-iotests/245        |   4 +-
>   tests/qemu-iotests/iotests.py |  10 +-
>   33 files changed, 1610 insertions(+), 339 deletions(-)
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v5 00/42] block: Deal with filters

Posted by Max Reitz 13 weeks ago
On 13.06.19 17:28, Vladimir Sementsov-Ogievskiy wrote:
> 13.06.2019 1:09, Max Reitz wrote:
>> Hi,
>>
>> When we introduced filters, we did it a bit casually.  Sure, we talked a
>> lot about them before, but that was mostly discussion about where
>> implicit filters should be added to the graph (note that we currently
>> only have two implicit filters, those being mirror and commit).  But in
>> the end, we really just designated some drivers filters (Quorum,
>> blkdebug, etc.) and added some specifically (throttle, COR), without
>> really looking through the block layer to see where issues might occur.
>>
>> It turns out vast areas of the block layer just don’t know about filters
>> and cannot really handle them.  Many cases will work in practice, in
>> others, well, too bad, you cannot use some feature because some part
>> deep inside the block layer looks at your filters and thinks they are
>> format nodes.
>>
>> This is one reason why this series is needed.  Over time (since v1), a
>> second reason has made its way in:
>>
>> bs->file is not necessarily the place where a node’s data is stored.
>> qcow2 now has external data files, and currently there is no way for the
>> general block layer to know that the data is not stored in bs->file.
>> Right now, I do not think that has any real consequences (all functions
>> that need access to the actual data storage file should only do so as a
>> fallback if the driver does not provide some functionality, but qcow2
>> should provide it all), but it still shows that we need some way to let
>> the general block layer know about such data files.  (Also, I will need
>> this for v1 of my “Inquire images’ rotational info” series.)
>>
>> I won’t go on and on about this series now, I think the patches pretty
>> much speak for themselves now.  If the cover letter gets too long,
>> nobody reads it anyway (see previous versions).
>>
>>
>> *** This series depends on some others. ***
>>
>> Dependencies:
>> - [PATCH 0/4] block: Keep track of parent quiescing
>> - [PATCH 0/2] vl: Drain before (block) job cancel when quitting
>> - [PATCH v2 0/2] blockdev: Overlays are not snapshots
>>
>> Based-on: <20190605161118.14544-1-mreitz@redhat.com>
>> Based-on: <20190612220839.1374-1-mreitz@redhat.com>
>> Based-on: <20190603202236.1342-1-mreitz@redhat.com>
> 
> Could you please export a branch?

Sure:

https://git.xanclic.moe/XanClic/qemu child-access-functions-v5
Or:
https://github.com/XanClic/qemu child-access-functions-v5


(And the base branch is:

https://git.xanclic.moe/XanClic/qemu child-access-functions-base
https://github.com/XanClic/qemu child-access-functions-base
)

>> v5:
>> - Split the huge patches 2 and 3 from the previous version into many
>>    smaller patches to maintain the potential reviewers’ sanity [Vladimir]
> 
> Thank you! In spite of frightening amount of patches, reviewing became a lot
> simpler.

I had hoped making it exactly 42 patches would make it a bit more welcoming.

Again, thanks a lot for reviewing!

Max