From: Chun Feng Wu <danielwuwy@163.com>
Hi,
I am thinking to leverage "throttle block filter" in QEMU to support more flexible I/O limits(e.g. tiered I/O groups), one sample provided by QEMU doc is:
https://github.com/qemu/qemu/blob/master/docs/throttle.txt
"For example, let's say that we have three different drives and we want to set I/O limits for
each one of them and an additional set of limits for the combined I/O of all three drives."
The implementation idea is to
- Define throttle groups(limit) in domain
- Define throttle filter to reference throttle group within disk
- Within domain disk, throttle filters references multiple throttle groups to form filter chain to apply multiple limits in QEMU like above sample
- Add new virsh cmds for throttle group management:
domthrottlegroupset Add or update a throttling group.
domthrottlegroupdel Delete a throttling group.
domthrottlegroupinfo Get a throttling group.
domthrottlegrouplist list all domain throttlegroups
- Update "attach-disk" to add one more option "--throttle-groups" to apply throttle filters e.g. "virsh attach-disk $VM_ID ${DISK_PATH}/vm1_disk_2.qcow2 vdd --driver qemu --subdriver qcow2 --targetbus virtio --throttle-groups limit2,limit012"
- I chose above semantics as I felt they're appropriate, if there are better ones please kindly suggest.
Note, this implementation requires flag "QEMU_CAPS_OBJECT_JSON".
From QMP perspective, the sample flow works this way:
- Throttle group creation:
virsh qemu-monitor-command 1 '{"execute":"object-add", "arguments":{"qom-type":"throttle-group","id":"limit0","limits":{"iops-total":200,"iops-read":0,"iops-total-max":200,"iops-total-max-length":1}}}'
virsh qemu-monitor-command 1 '{"execute":"object-add", "arguments":{"qom-type":"throttle-group","id":"limit1","limits":{"iops-total":250,"iops-read":0,"iops-total-max":250,"iops-total-max-length":1}}}'
virsh qemu-monitor-command 1 '{"execute":"object-add", "arguments":{"qom-type":"throttle-group","id":"limit2","limits":{"iops-total":300,"iops-read":0,"iops-total-max":300,"iops-total-max-length":1}}}'
virsh qemu-monitor-command 1 '{"execute":"object-add", "arguments":{"qom-type":"throttle-group","id":"limit012","limits":{"iops-total":400,"iops-read":0,"iops-total-max":400,"iops-total-max-length":1}}}'
- Chain up filters during attaching disk to apply two filters(limit0 and limit012):
virsh qemu-monitor-command 1 '{"execute":"blockdev-add", "arguments": {"driver":"file","filename":"/virt/disks/vm1_disk_1.qcow2","node-name":"test-3-storage","auto-read-only":true,"discard":"unmap"}}'
virsh qemu-monitor-command 1 '{"execute":"blockdev-add", "arguments":{"node-name":"test-4-format","read-only":false,"driver":"qcow2","file":"test-3-storage","backing":null}}'
virsh qemu-monitor-command 1 '{"execute":"blockdev-add", "arguments":{"driver":"throttle","node-name":"libvirt-5-filter","throttle-group": "limit0","file":"test-4-format"}}'
virsh qemu-monitor-command 1 '{"execute":"blockdev-add", "arguments": {"driver":"throttle","node-name":"libvirt-6-filter","throttle-group":"limit012","file":"libvirt-5-filter"}}'
virsh qemu-monitor-command 1 '{"execute": "device_add", "arguments": {"driver":"virtio-blk-pci","scsi":false,"bus":"pci.0","addr":"0x5","drive":"libvirt-6-filter","id":"virtio-disk1"}}'
This patchset includes:
- schema: Add new domain elements to support multiple throttle groups
- schema: Add new domain elements to support multiple throttle filters
- config: Introduce ThrottleGroup and corresponding XML parsing
- config: Introduce ThrottleFilter and corresponding XML parsing
- qemu: monitor: Add support for ThrottleGroup operations
- tests: Test qemuMonitorJSONGetThrottleGroup and qemuMonitorJSONUpdateThrottleGroup
- remote: New APIs for ThrottleGroup lifecycle management
- qemu: Refactor qemuDomainSetBlockIoTune to extract common methods
- qemu: Implement qemu driver for throttle API
- qemu: helper: throttle filter nodename and preparation processing
- qemu: block: Support block disk along with throttle filters
- config: validate: Verify iotune, throttle group and filter
- qemuxmlconftest: Add 'throttlefilter' tests
- test_driver: Test throttle group lifecycle APIs
- virsh: Refactor iotune options for re-use
- virsh: Add support for throttle group operations
- virsh: Add option "throttle-groups" to "attach_disk"
v4 changes:
- Format commit msgs
- Update version to '10.7.0'
- Update formatdomain.rst to remove nodename chain sample which exists in QEMU, and add description that order within <throttlefilters> doesn't mater
- delete "virDomainThrottleGroupFind", which is duplicate to "virDomainThrottleGroupByName"
- Remove "id" within _virDomainThrottleFilterDef, which is not required
- Avoid export for methods not used anywhere else
- Add manpage for new virsh cmds and options
- Update error code&msg
- Consolidate validation logic into one commit
- Format throttle filter nodename into disk/privateData/nodenames/nodename(type='throttle-filter') and parse it reversely
- Add tests for statusxml(disk/privateData/nodenames/nodename) test regarding throttle filter nodename parse and format
- Add more complicated test cases and negative case for throttle filter xml
- Add a 'throttle-' prefix so that we clearly separate the throttle group objects into their own QEMU namespace.
- Update virDomainGetThrottleGroup and virDomainDelThrottleGroup to remove TYPED_PARAM_STRING check
- Update remote_protocol.x "REMOTE_PROC_DOMAIN_DEL_THROTTLE_GROUP" to set "@acl: domain:write" rather than "read"
- Split commit of "qemu: Implement qemu driver for throttle API" into two commits
- Check QEMU_CAPS_OBJECT_JSON when vm is virDomainObjIsActive(vm), and update error msg
- Address leak in qemuBuildThrottleFiltersAttachPrepareBlockdev
- Prefix new cmd with "dom", e.g. domthrottlegroupset
- Refactor completer and "domthrottlegrouplist" to share the same method "virshGetThrottleGroupNames"
- Update qemuBlockThrottleFilterAttachPrepareBlockdev to remove "data->filterAttached = true;"
- Support CopyOnRead by re-org chain: device -> throttle -> copyOnRead Layer-> image chain
Any comments/suggestions will be appriciated!
Chun Feng Wu (17):
schema: Add new domain elements to support multiple throttle groups
schema: Add new domain elements to support multiple throttle filters
config: Introduce ThrottleGroup and corresponding XML parsing
config: Introduce ThrottleFilter and corresponding XML parsing
qemu: monitor: Add support for ThrottleGroup operations
tests: Test qemuMonitorJSONGetThrottleGroup and
qemuMonitorJSONUpdateThrottleGroup
remote: New APIs for ThrottleGroup lifecycle management
qemu: Refactor qemuDomainSetBlockIoTune to extract common methods
qemu: Implement qemu driver for throttle API
qemu: helper: throttle filter nodename and preparation processing
qemu: block: Support block disk along with throttle filters
config: validate: Verify iotune, throttle group and filter
qemuxmlconftest: Add 'throttlefilter' tests
test_driver: Test throttle group lifecycle APIs
virsh: Refactor iotune options for re-use
virsh: Add support for throttle group operations
virsh: Add option "throttle-groups" to "attach_disk"
docs/formatdomain.rst | 47 ++
docs/manpages/virsh.rst | 135 +++-
include/libvirt/libvirt-domain.h | 21 +
src/conf/domain_conf.c | 398 ++++++++++
src/conf/domain_conf.h | 45 ++
src/conf/domain_validate.c | 119 ++-
src/conf/schemas/domaincommon.rng | 293 ++++----
src/conf/virconftypes.h | 4 +
src/driver-hypervisor.h | 22 +
src/libvirt-domain.c | 174 +++++
src/libvirt_private.syms | 8 +
src/libvirt_public.syms | 7 +
src/qemu/qemu_block.c | 136 ++++
src/qemu/qemu_block.h | 49 ++
src/qemu/qemu_command.c | 180 +++++
src/qemu/qemu_command.h | 6 +
src/qemu/qemu_domain.c | 73 +-
src/qemu/qemu_driver.c | 619 +++++++++++++---
src/qemu/qemu_hotplug.c | 33 +
src/qemu/qemu_monitor.c | 34 +
src/qemu/qemu_monitor.h | 14 +
src/qemu/qemu_monitor_json.c | 134 ++++
src/qemu/qemu_monitor_json.h | 14 +
src/remote/remote_daemon_dispatch.c | 44 ++
src/remote/remote_driver.c | 40 ++
src/remote/remote_protocol.x | 48 +-
src/remote_protocol-structs | 28 +
src/test/test_driver.c | 452 ++++++++----
tests/qemumonitorjsontest.c | 86 +++
.../throttlefilter-in.xml | 392 ++++++++++
.../throttlefilter-out.xml | 392 ++++++++++
tests/qemuxmlactivetest.c | 1 +
.../throttlefilter-invalid.x86_64-latest.err | 1 +
.../throttlefilter-invalid.xml | 89 +++
.../throttlefilter.x86_64-latest.args | 55 ++
.../throttlefilter.x86_64-latest.xml | 105 +++
tests/qemuxmlconfdata/throttlefilter.xml | 95 +++
tests/qemuxmlconftest.c | 2 +
tools/virsh-completer-domain.c | 82 +++
tools/virsh-completer-domain.h | 16 +
tools/virsh-domain.c | 680 ++++++++++++++----
41 files changed, 4648 insertions(+), 525 deletions(-)
create mode 100644 tests/qemustatusxml2xmldata/throttlefilter-in.xml
create mode 100644 tests/qemustatusxml2xmldata/throttlefilter-out.xml
create mode 100644 tests/qemuxmlconfdata/throttlefilter-invalid.x86_64-latest.err
create mode 100644 tests/qemuxmlconfdata/throttlefilter-invalid.xml
create mode 100644 tests/qemuxmlconfdata/throttlefilter.x86_64-latest.args
create mode 100644 tests/qemuxmlconfdata/throttlefilter.x86_64-latest.xml
create mode 100644 tests/qemuxmlconfdata/throttlefilter.xml
--
2.43.0