[PATCH v10 0/9] discard blockstats

Anton Nefedov posted 9 patches 4 years, 7 months ago
Test docker-clang@ubuntu passed
Test docker-mingw@fedora passed
Test docker-quick@centos7 passed
Test checkpatch passed
Test asan passed
Test FreeBSD passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190923121737.83281-1-anton.nefedov@virtuozzo.com
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, Kevin Wolf <kwolf@redhat.com>, John Snow <jsnow@redhat.com>, Markus Armbruster <armbru@redhat.com>, Max Reitz <mreitz@redhat.com>, Fam Zheng <fam@euphon.net>, Eric Blake <eblake@redhat.com>
qapi/block-core.json       | 81 ++++++++++++++++++++++++++++++++------
include/block/accounting.h |  2 +
include/block/block.h      |  1 +
include/block/block_int.h  |  1 +
block.c                    |  9 +++++
block/accounting.c         |  6 +++
block/file-posix.c         | 54 ++++++++++++++++++++++++-
block/qapi.c               | 11 ++++++
hw/ide/core.c              | 12 ++++++
hw/scsi/scsi-disk.c        | 34 ++++++++++------
tests/qemu-iotests/227.out | 18 +++++++++
11 files changed, 206 insertions(+), 23 deletions(-)
[PATCH v10 0/9] discard blockstats
Posted by Anton Nefedov 4 years, 7 months ago
v10:
  - patch 3 commit message updated

v9: https://lists.nongnu.org/archive/html/qemu-devel/2019-09/msg01190.html

----

qmp query-blockstats provides stats info for write/read/flush ops.

Patches 1-7 implement the similar for discard (unmap) command for scsi
and ide disks.
Discard stat "unmap_ops / unmap_bytes" is supposed to account the ops that
have completed without an error.

However, discard operation is advisory. Specifically,
 - common block layer ignores ENOTSUP error code.
   That might be returned if the block driver does not support discard,
   or discard has been configured to be ignored.
 - format drivers such as qcow2 may ignore discard if they were configured
   to ignore that, or if the corresponding area is already marked unused
   (unallocated / zero clusters).

And what is actually useful is the number of bytes actually discarded
down on the host filesystem.
To achieve that, driver-specific statistics has been added to blockstats
(patch 9).
With patch 8, file-posix driver accounts discard operations on its level too.

query-blockstat result:

(note the difference between blockdevice unmap and file discard stats. qcow2
sends fewer ops down to the file as the clusters are actually unallocated
on qcow2 level)

    {
      "device": "drive-scsi0-0-0-0",
      "node-name": "#block159",
      "stats": {
>       "invalid_unmap_operations": 0,
>       "failed_unmap_operations": 0,
        "wr_highest_offset": 13411688448,
        "rd_total_time_ns": 2859566315,
        "rd_bytes": 103182336,
        "rd_merged": 0,
        "flush_operations": 19,
        "invalid_wr_operations": 0,
        "flush_total_time_ns": 23111608,
        "failed_rd_operations": 0,
        "failed_flush_operations": 0,
        "invalid_flush_operations": 0,
        "timed_stats": [
          
        ],
        "wr_merged": 0,
        "wr_bytes": 1702912,
>       "unmap_bytes": 11954954240,
>       "unmap_operations": 865,
        "idle_time_ns": 2669508623,
        "account_invalid": true,
>       "unmap_total_time_ns": 19698002,
        "wr_operations": 143,
        "failed_wr_operations": 0,
        "rd_operations": 4816,
        "account_failed": true,
>       "unmap_merged": 0,
        "wr_total_time_ns": 1262686124,
        "invalid_rd_operations": 0
      },
      "parent": {
>       "driver-specific": {
>         "discard-nb-failed": 0,
>         "discard-bytes-ok": 720896,
>         "driver": "file",
>         "discard-nb-ok": 8
>       },
        "node-name": "#block009",
        "stats": {
        [..]
        }
      }
    },
    {
      "device": "floppy0",

Anton Nefedov (9):
  qapi: group BlockDeviceStats fields
  qapi: add unmap to BlockDeviceStats
  block: add empty account cookie type
  ide: account UNMAP (TRIM) operations
  scsi: store unmap offset and nb_sectors in request struct
  scsi: move unmap error checking to the complete callback
  scsi: account unmap operations
  file-posix: account discard operations
  qapi: query-blockstat: add driver specific file-posix stats

 qapi/block-core.json       | 81 ++++++++++++++++++++++++++++++++------
 include/block/accounting.h |  2 +
 include/block/block.h      |  1 +
 include/block/block_int.h  |  1 +
 block.c                    |  9 +++++
 block/accounting.c         |  6 +++
 block/file-posix.c         | 54 ++++++++++++++++++++++++-
 block/qapi.c               | 11 ++++++
 hw/ide/core.c              | 12 ++++++
 hw/scsi/scsi-disk.c        | 34 ++++++++++------
 tests/qemu-iotests/227.out | 18 +++++++++
 11 files changed, 206 insertions(+), 23 deletions(-)

-- 
2.17.1


Re: [PATCH v10 0/9] discard blockstats
Posted by Max Reitz 4 years, 7 months ago
On 23.09.19 14:17, Anton Nefedov wrote:
> v10:
>   - patch 3 commit message updated
> 
> v9: https://lists.nongnu.org/archive/html/qemu-devel/2019-09/msg01190.html
> 
> ----
> 
> qmp query-blockstats provides stats info for write/read/flush ops.
> 
> Patches 1-7 implement the similar for discard (unmap) command for scsi
> and ide disks.
> Discard stat "unmap_ops / unmap_bytes" is supposed to account the ops that
> have completed without an error.
> 
> However, discard operation is advisory. Specifically,
>  - common block layer ignores ENOTSUP error code.
>    That might be returned if the block driver does not support discard,
>    or discard has been configured to be ignored.
>  - format drivers such as qcow2 may ignore discard if they were configured
>    to ignore that, or if the corresponding area is already marked unused
>    (unallocated / zero clusters).
> 
> And what is actually useful is the number of bytes actually discarded
> down on the host filesystem.
> To achieve that, driver-specific statistics has been added to blockstats
> (patch 9).
> With patch 8, file-posix driver accounts discard operations on its level too.
> 
> query-blockstat result:
> 
> (note the difference between blockdevice unmap and file discard stats. qcow2
> sends fewer ops down to the file as the clusters are actually unallocated
> on qcow2 level)
> 
>     {
>       "device": "drive-scsi0-0-0-0",
>       "node-name": "#block159",
>       "stats": {
>>       "invalid_unmap_operations": 0,
>>       "failed_unmap_operations": 0,
>         "wr_highest_offset": 13411688448,
>         "rd_total_time_ns": 2859566315,
>         "rd_bytes": 103182336,
>         "rd_merged": 0,
>         "flush_operations": 19,
>         "invalid_wr_operations": 0,
>         "flush_total_time_ns": 23111608,
>         "failed_rd_operations": 0,
>         "failed_flush_operations": 0,
>         "invalid_flush_operations": 0,
>         "timed_stats": [
>           
>         ],
>         "wr_merged": 0,
>         "wr_bytes": 1702912,
>>       "unmap_bytes": 11954954240,
>>       "unmap_operations": 865,
>         "idle_time_ns": 2669508623,
>         "account_invalid": true,
>>       "unmap_total_time_ns": 19698002,
>         "wr_operations": 143,
>         "failed_wr_operations": 0,
>         "rd_operations": 4816,
>         "account_failed": true,
>>       "unmap_merged": 0,
>         "wr_total_time_ns": 1262686124,
>         "invalid_rd_operations": 0
>       },
>       "parent": {
>>       "driver-specific": {
>>         "discard-nb-failed": 0,
>>         "discard-bytes-ok": 720896,
>>         "driver": "file",
>>         "discard-nb-ok": 8
>>       },
>         "node-name": "#block009",
>         "stats": {
>         [..]
>         }
>       }
>     },
>     {
>       "device": "floppy0",
> 
> Anton Nefedov (9):
>   qapi: group BlockDeviceStats fields
>   qapi: add unmap to BlockDeviceStats
>   block: add empty account cookie type
>   ide: account UNMAP (TRIM) operations
>   scsi: store unmap offset and nb_sectors in request struct
>   scsi: move unmap error checking to the complete callback
>   scsi: account unmap operations
>   file-posix: account discard operations
>   qapi: query-blockstat: add driver specific file-posix stats
> 
>  qapi/block-core.json       | 81 ++++++++++++++++++++++++++++++++------
>  include/block/accounting.h |  2 +
>  include/block/block.h      |  1 +
>  include/block/block_int.h  |  1 +
>  block.c                    |  9 +++++
>  block/accounting.c         |  6 +++
>  block/file-posix.c         | 54 ++++++++++++++++++++++++-
>  block/qapi.c               | 11 ++++++
>  hw/ide/core.c              | 12 ++++++
>  hw/scsi/scsi-disk.c        | 34 ++++++++++------
>  tests/qemu-iotests/227.out | 18 +++++++++
>  11 files changed, 206 insertions(+), 23 deletions(-)

Thanks, applied to my block branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block

Max