[v11] support concurrent sync io for bfq on a specail occasion

[patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Yu Kuai 1 year, 7 months ago

From: Yu Kuai <yukuai3@huawei.com>

Changes in v11:
 - keep the comments in bfq_weights_tree_remove() and move it to the
 caller where bfqq can be freed.
 - add two followed up cleanup patches.

Changes in v10:
 - Add reviewed-tag for patch 2

Changes in v9:
 - also update how many bfqqs have pending_reqs bfq_bfqq_move().
 - fix one language in patch 4
 - Add reviewed-tag for patch 1,3,4

Changes in v8:
 - Instead of using whether bfqq is busy, using whether bfqq has pending
 requests. As Paolo pointed out the former way is problematic.

Changes in v7:
 - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
 also retest this patchset on v5.18 to make sure functionality is
 correct.
 - move the updating of 'bfqd->busy_queues' into new apis

Changes in v6:
 - add reviewed-by tag for patch 1

Changes in v5:
 - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
 - fix wrong definition in patch 1
 - fix spelling mistake in patch 2: leaset -> least
 - update comments in patch 3
 - add reviewed-by tag in patch 2,3

Changes in v4:
 - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
   suggested by Jan Kara.
 - remove unused 'in_groups_with_pending_reqs',

Changes in v3:
 - remove the cleanup patch that is irrelevant now(I'll post it
   separately).
 - instead of hacking wr queues and using weights tree insertion/removal,
   using bfq_add/del_bfqq_busy() to count the number of groups
   (suggested by Jan Kara).

Changes in v2:
 - Use a different approch to count root group, which is much simple.

Currently, bfq can't handle sync io concurrently as long as they
are not issued from root group. This is because
'bfqd->num_groups_with_pending_reqs > 0' is always true in
bfq_asymmetric_scenario().

The way that bfqg is counted into 'num_groups_with_pending_reqs':

Before this patchset:
 1) root group will never be counted.
 2) Count if bfqg or it's child bfqgs have pending requests.
 3) Don't count if bfqg and it's child bfqgs complete all the requests.

After this patchset:
 1) root group is counted.
 2) Count if bfqg has pending requests.
 3) Don't count if bfqg complete all the requests.

With the above changes, concurrent sync io can be supported if only
one group is activated.

fio test script(startdelay is used to avoid queue merging):
[global]
filename=/dev/sda
allow_mounted_write=0
ioengine=psync
direct=1
ioscheduler=bfq
offset_increment=10g
group_reporting
rw=randwrite
bs=4k

[test1]
numjobs=1

[test2]
startdelay=1
numjobs=1

[test3]
startdelay=2
numjobs=1

[test4]
startdelay=3
numjobs=1

[test5]
startdelay=4
numjobs=1

[test6]
startdelay=5
numjobs=1

[test7]
startdelay=6
numjobs=1

[test8]
startdelay=7
numjobs=1

test result:
running fio on root cgroup
v5.18:	   112 Mib/s
v5.18-patched: 112 Mib/s

running fio on non-root cgroup
v5.18:	   51.2 Mib/s
v5.18-patched: 112 Mib/s

Note that I also test null_blk with "irqmode=2
completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
that service guarantees are still preserved.

Previous versions:
RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/


Yu Kuai (6):
  block, bfq: support to track if bfqq has pending requests
  block, bfq: record how many queues have pending requests
  block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
  block, bfq: do not idle if only one group is activated
  block, bfq: cleanup bfq_weights_tree add/remove apis
  block, bfq: cleanup __bfq_weights_tree_remove()

 block/bfq-cgroup.c  | 10 +++++++
 block/bfq-iosched.c | 71 +++++++--------------------------------------
 block/bfq-iosched.h | 30 +++++++++----------
 block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
 4 files changed, 76 insertions(+), 104 deletions(-)

-- 
2.31.1

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Yu Kuai 1 year, 6 months ago

Hi, Jens

在 2022/09/16 15:19, Yu Kuai 写道:
> From: Yu Kuai <yukuai3@huawei.com>
>
> 
> Currently, bfq can't handle sync io concurrently as long as they
> are not issued from root group. This is because
> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
> bfq_asymmetric_scenario().
> 
> The way that bfqg is counted into 'num_groups_with_pending_reqs':
> 
> Before this patchset:
>   1) root group will never be counted.
>   2) Count if bfqg or it's child bfqgs have pending requests.
>   3) Don't count if bfqg and it's child bfqgs complete all the requests.
> 
> After this patchset:
>   1) root group is counted.
>   2) Count if bfqg has pending requests.
>   3) Don't count if bfqg complete all the requests.
> 
> With the above changes, concurrent sync io can be supported if only
> one group is activated.

Can you apply this patchset?

Thanks,
Kuai

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Paolo Valente 1 year, 7 months ago


> Il giorno 16 set 2022, alle ore 09:19, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> 
> From: Yu Kuai <yukuai3@huawei.com>
> 
> Changes in v11:
> - keep the comments in bfq_weights_tree_remove() and move it to the
> caller where bfqq can be freed.
> - add two followed up cleanup patches.
> 
> Changes in v10:
> - Add reviewed-tag for patch 2
> 
> Changes in v9:
> - also update how many bfqqs have pending_reqs bfq_bfqq_move().
> - fix one language in patch 4
> - Add reviewed-tag for patch 1,3,4
> 
> Changes in v8:
> - Instead of using whether bfqq is busy, using whether bfqq has pending
> requests. As Paolo pointed out the former way is problematic.
> 
> Changes in v7:
> - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
> also retest this patchset on v5.18 to make sure functionality is
> correct.
> - move the updating of 'bfqd->busy_queues' into new apis
> 
> Changes in v6:
> - add reviewed-by tag for patch 1
> 
> Changes in v5:
> - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
> - fix wrong definition in patch 1
> - fix spelling mistake in patch 2: leaset -> least
> - update comments in patch 3
> - add reviewed-by tag in patch 2,3
> 
> Changes in v4:
> - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
>   suggested by Jan Kara.
> - remove unused 'in_groups_with_pending_reqs',
> 
> Changes in v3:
> - remove the cleanup patch that is irrelevant now(I'll post it
>   separately).
> - instead of hacking wr queues and using weights tree insertion/removal,
>   using bfq_add/del_bfqq_busy() to count the number of groups
>   (suggested by Jan Kara).
> 
> Changes in v2:
> - Use a different approch to count root group, which is much simple.
> 
> Currently, bfq can't handle sync io concurrently as long as they
> are not issued from root group. This is because
> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
> bfq_asymmetric_scenario().
> 
> The way that bfqg is counted into 'num_groups_with_pending_reqs':
> 
> Before this patchset:
> 1) root group will never be counted.
> 2) Count if bfqg or it's child bfqgs have pending requests.
> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
> 
> After this patchset:
> 1) root group is counted.
> 2) Count if bfqg has pending requests.
> 3) Don't count if bfqg complete all the requests.
> 
> With the above changes, concurrent sync io can be supported if only
> one group is activated.
> 
> fio test script(startdelay is used to avoid queue merging):
> [global]
> filename=/dev/sda
> allow_mounted_write=0
> ioengine=psync
> direct=1
> ioscheduler=bfq
> offset_increment=10g
> group_reporting
> rw=randwrite
> bs=4k
> 
> [test1]
> numjobs=1
> 
> [test2]
> startdelay=1
> numjobs=1
> 
> [test3]
> startdelay=2
> numjobs=1
> 
> [test4]
> startdelay=3
> numjobs=1
> 
> [test5]
> startdelay=4
> numjobs=1
> 
> [test6]
> startdelay=5
> numjobs=1
> 
> [test7]
> startdelay=6
> numjobs=1
> 
> [test8]
> startdelay=7
> numjobs=1
> 
> test result:
> running fio on root cgroup
> v5.18:	   112 Mib/s
> v5.18-patched: 112 Mib/s
> 
> running fio on non-root cgroup
> v5.18:	   51.2 Mib/s
> v5.18-patched: 112 Mib/s
> 
> Note that I also test null_blk with "irqmode=2
> completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
> that service guarantees are still preserved.
> 

Your patches seem ok to me now (thanks for you contribution and, above all, for your patience). I have only a high-level concern: what do you mean when you say that service guarantees are still preserved? What test did you run exactly? This point is very important to me. I'd like to see some convincing test with differentiated weights. In case you don't have other tools for executing such tests quickly, you may want to use the bandwidth-latency test in my simple S benchmark suite (for which I'm willing to help).

Thanks,
Paolo

> Previous versions:
> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/
> 
> 
> Yu Kuai (6):
>  block, bfq: support to track if bfqq has pending requests
>  block, bfq: record how many queues have pending requests
>  block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>  block, bfq: do not idle if only one group is activated
>  block, bfq: cleanup bfq_weights_tree add/remove apis
>  block, bfq: cleanup __bfq_weights_tree_remove()
> 
> block/bfq-cgroup.c  | 10 +++++++
> block/bfq-iosched.c | 71 +++++++--------------------------------------
> block/bfq-iosched.h | 30 +++++++++----------
> block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
> 4 files changed, 76 insertions(+), 104 deletions(-)
> 
> -- 
> 2.31.1
>

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Yu Kuai 1 year, 6 months ago

Hi, paolo

在 2022/09/28 0:38, Paolo Valente 写道:
> 
> 
>> Il giorno 16 set 2022, alle ore 09:19, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>
>> From: Yu Kuai <yukuai3@huawei.com>
>>
>> Changes in v11:
>> - keep the comments in bfq_weights_tree_remove() and move it to the
>> caller where bfqq can be freed.
>> - add two followed up cleanup patches.
>>
>> Changes in v10:
>> - Add reviewed-tag for patch 2
>>
>> Changes in v9:
>> - also update how many bfqqs have pending_reqs bfq_bfqq_move().
>> - fix one language in patch 4
>> - Add reviewed-tag for patch 1,3,4
>>
>> Changes in v8:
>> - Instead of using whether bfqq is busy, using whether bfqq has pending
>> requests. As Paolo pointed out the former way is problematic.
>>
>> Changes in v7:
>> - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
>> also retest this patchset on v5.18 to make sure functionality is
>> correct.
>> - move the updating of 'bfqd->busy_queues' into new apis
>>
>> Changes in v6:
>> - add reviewed-by tag for patch 1
>>
>> Changes in v5:
>> - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
>> - fix wrong definition in patch 1
>> - fix spelling mistake in patch 2: leaset -> least
>> - update comments in patch 3
>> - add reviewed-by tag in patch 2,3
>>
>> Changes in v4:
>> - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
>>    suggested by Jan Kara.
>> - remove unused 'in_groups_with_pending_reqs',
>>
>> Changes in v3:
>> - remove the cleanup patch that is irrelevant now(I'll post it
>>    separately).
>> - instead of hacking wr queues and using weights tree insertion/removal,
>>    using bfq_add/del_bfqq_busy() to count the number of groups
>>    (suggested by Jan Kara).
>>
>> Changes in v2:
>> - Use a different approch to count root group, which is much simple.
>>
>> Currently, bfq can't handle sync io concurrently as long as they
>> are not issued from root group. This is because
>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>> bfq_asymmetric_scenario().
>>
>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>
>> Before this patchset:
>> 1) root group will never be counted.
>> 2) Count if bfqg or it's child bfqgs have pending requests.
>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>
>> After this patchset:
>> 1) root group is counted.
>> 2) Count if bfqg has pending requests.
>> 3) Don't count if bfqg complete all the requests.
>>
>> With the above changes, concurrent sync io can be supported if only
>> one group is activated.
>>
>> fio test script(startdelay is used to avoid queue merging):
>> [global]
>> filename=/dev/sda
>> allow_mounted_write=0
>> ioengine=psync
>> direct=1
>> ioscheduler=bfq
>> offset_increment=10g
>> group_reporting
>> rw=randwrite
>> bs=4k
>>
>> [test1]
>> numjobs=1
>>
>> [test2]
>> startdelay=1
>> numjobs=1
>>
>> [test3]
>> startdelay=2
>> numjobs=1
>>
>> [test4]
>> startdelay=3
>> numjobs=1
>>
>> [test5]
>> startdelay=4
>> numjobs=1
>>
>> [test6]
>> startdelay=5
>> numjobs=1
>>
>> [test7]
>> startdelay=6
>> numjobs=1
>>
>> [test8]
>> startdelay=7
>> numjobs=1
>>
>> test result:
>> running fio on root cgroup
>> v5.18:	   112 Mib/s
>> v5.18-patched: 112 Mib/s
>>
>> running fio on non-root cgroup
>> v5.18:	   51.2 Mib/s
>> v5.18-patched: 112 Mib/s
>>
>> Note that I also test null_blk with "irqmode=2
>> completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
>> that service guarantees are still preserved.
>>
> 
> Your patches seem ok to me now (thanks for you contribution and, above all, for your patience). I have only a high-level concern: what do you mean when you say that service guarantees are still preserved? What test did you run exactly? This point is very important to me. I'd like to see some convincing test with differentiated weights. In case you don't have other tools for executing such tests quickly, you may want to use the bandwidth-latency test in my simple S benchmark suite (for which I'm willing to help).

Is there any test that you wish me to try?

By the way, I think for the case that multiple groups are activaced, (
specifically num_groups_with_pendind_rqs > 1), io path in bfq is the
same with or without this patchset.

Thanks,
Kuai
> 
> Thanks,
> Paolo
> 
>> Previous versions:
>> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
>> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
>> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
>> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
>> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
>> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
>> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
>> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/
>>
>>
>> Yu Kuai (6):
>>   block, bfq: support to track if bfqq has pending requests
>>   block, bfq: record how many queues have pending requests
>>   block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>>   block, bfq: do not idle if only one group is activated
>>   block, bfq: cleanup bfq_weights_tree add/remove apis
>>   block, bfq: cleanup __bfq_weights_tree_remove()
>>
>> block/bfq-cgroup.c  | 10 +++++++
>> block/bfq-iosched.c | 71 +++++++--------------------------------------
>> block/bfq-iosched.h | 30 +++++++++----------
>> block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
>> 4 files changed, 76 insertions(+), 104 deletions(-)
>>
>> -- 
>> 2.31.1
>>
> 
> .
>

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Paolo Valente 1 year, 6 months ago


> Il giorno 11 ott 2022, alle ore 10:11, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> 
> Hi, paolo
> 
> 在 2022/09/28 0:38, Paolo Valente 写道:
>>> Il giorno 16 set 2022, alle ore 09:19, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>> 
>>> From: Yu Kuai <yukuai3@huawei.com>
>>> 
>>> Changes in v11:
>>> - keep the comments in bfq_weights_tree_remove() and move it to the
>>> caller where bfqq can be freed.
>>> - add two followed up cleanup patches.
>>> 
>>> Changes in v10:
>>> - Add reviewed-tag for patch 2
>>> 
>>> Changes in v9:
>>> - also update how many bfqqs have pending_reqs bfq_bfqq_move().
>>> - fix one language in patch 4
>>> - Add reviewed-tag for patch 1,3,4
>>> 
>>> Changes in v8:
>>> - Instead of using whether bfqq is busy, using whether bfqq has pending
>>> requests. As Paolo pointed out the former way is problematic.
>>> 
>>> Changes in v7:
>>> - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
>>> also retest this patchset on v5.18 to make sure functionality is
>>> correct.
>>> - move the updating of 'bfqd->busy_queues' into new apis
>>> 
>>> Changes in v6:
>>> - add reviewed-by tag for patch 1
>>> 
>>> Changes in v5:
>>> - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
>>> - fix wrong definition in patch 1
>>> - fix spelling mistake in patch 2: leaset -> least
>>> - update comments in patch 3
>>> - add reviewed-by tag in patch 2,3
>>> 
>>> Changes in v4:
>>> - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
>>>   suggested by Jan Kara.
>>> - remove unused 'in_groups_with_pending_reqs',
>>> 
>>> Changes in v3:
>>> - remove the cleanup patch that is irrelevant now(I'll post it
>>>   separately).
>>> - instead of hacking wr queues and using weights tree insertion/removal,
>>>   using bfq_add/del_bfqq_busy() to count the number of groups
>>>   (suggested by Jan Kara).
>>> 
>>> Changes in v2:
>>> - Use a different approch to count root group, which is much simple.
>>> 
>>> Currently, bfq can't handle sync io concurrently as long as they
>>> are not issued from root group. This is because
>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>> bfq_asymmetric_scenario().
>>> 
>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>> 
>>> Before this patchset:
>>> 1) root group will never be counted.
>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>> 
>>> After this patchset:
>>> 1) root group is counted.
>>> 2) Count if bfqg has pending requests.
>>> 3) Don't count if bfqg complete all the requests.
>>> 
>>> With the above changes, concurrent sync io can be supported if only
>>> one group is activated.
>>> 
>>> fio test script(startdelay is used to avoid queue merging):
>>> [global]
>>> filename=/dev/sda
>>> allow_mounted_write=0
>>> ioengine=psync
>>> direct=1
>>> ioscheduler=bfq
>>> offset_increment=10g
>>> group_reporting
>>> rw=randwrite
>>> bs=4k
>>> 
>>> [test1]
>>> numjobs=1
>>> 
>>> [test2]
>>> startdelay=1
>>> numjobs=1
>>> 
>>> [test3]
>>> startdelay=2
>>> numjobs=1
>>> 
>>> [test4]
>>> startdelay=3
>>> numjobs=1
>>> 
>>> [test5]
>>> startdelay=4
>>> numjobs=1
>>> 
>>> [test6]
>>> startdelay=5
>>> numjobs=1
>>> 
>>> [test7]
>>> startdelay=6
>>> numjobs=1
>>> 
>>> [test8]
>>> startdelay=7
>>> numjobs=1
>>> 
>>> test result:
>>> running fio on root cgroup
>>> v5.18:	   112 Mib/s
>>> v5.18-patched: 112 Mib/s
>>> 
>>> running fio on non-root cgroup
>>> v5.18:	   51.2 Mib/s
>>> v5.18-patched: 112 Mib/s
>>> 
>>> Note that I also test null_blk with "irqmode=2
>>> completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
>>> that service guarantees are still preserved.
>>> 
>> Your patches seem ok to me now (thanks for you contribution and, above all, for your patience). I have only a high-level concern: what do you mean when you say that service guarantees are still preserved? What test did you run exactly? This point is very important to me. I'd like to see some convincing test with differentiated weights. In case you don't have other tools for executing such tests quickly, you may want to use the bandwidth-latency test in my simple S benchmark suite (for which I'm willing to help).
> 
> Is there any test that you wish me to try?
> 
> By the way, I think for the case that multiple groups are activaced, (
> specifically num_groups_with_pendind_rqs > 1), io path in bfq is the
> same with or without this patchset.
> 

The tests cases you mentioned are ok for me (whatever tool or personal
code you use to run them).  Just show me your results with and without
your patchset applied.

Thanks,
Paolo

> Thanks,
> Kuai
>> Thanks,
>> Paolo
>>> Previous versions:
>>> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
>>> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
>>> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
>>> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
>>> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
>>> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
>>> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
>>> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/
>>> 
>>> 
>>> Yu Kuai (6):
>>>  block, bfq: support to track if bfqq has pending requests
>>>  block, bfq: record how many queues have pending requests
>>>  block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>>>  block, bfq: do not idle if only one group is activated
>>>  block, bfq: cleanup bfq_weights_tree add/remove apis
>>>  block, bfq: cleanup __bfq_weights_tree_remove()
>>> 
>>> block/bfq-cgroup.c  | 10 +++++++
>>> block/bfq-iosched.c | 71 +++++++--------------------------------------
>>> block/bfq-iosched.h | 30 +++++++++----------
>>> block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
>>> 4 files changed, 76 insertions(+), 104 deletions(-)
>>> 
>>> -- 
>>> 2.31.1
>>> 
>> .

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Yu Kuai 1 year, 6 months ago

Hi, Paolo

在 2022/10/11 16:21, Paolo Valente 写道:
> 
> 
>> Il giorno 11 ott 2022, alle ore 10:11, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>
>> Hi, paolo
>>
>> 在 2022/09/28 0:38, Paolo Valente 写道:
>>>> Il giorno 16 set 2022, alle ore 09:19, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>>>
>>>> From: Yu Kuai <yukuai3@huawei.com>
>>>>
>>>> Changes in v11:
>>>> - keep the comments in bfq_weights_tree_remove() and move it to the
>>>> caller where bfqq can be freed.
>>>> - add two followed up cleanup patches.
>>>>
>>>> Changes in v10:
>>>> - Add reviewed-tag for patch 2
>>>>
>>>> Changes in v9:
>>>> - also update how many bfqqs have pending_reqs bfq_bfqq_move().
>>>> - fix one language in patch 4
>>>> - Add reviewed-tag for patch 1,3,4
>>>>
>>>> Changes in v8:
>>>> - Instead of using whether bfqq is busy, using whether bfqq has pending
>>>> requests. As Paolo pointed out the former way is problematic.
>>>>
>>>> Changes in v7:
>>>> - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
>>>> also retest this patchset on v5.18 to make sure functionality is
>>>> correct.
>>>> - move the updating of 'bfqd->busy_queues' into new apis
>>>>
>>>> Changes in v6:
>>>> - add reviewed-by tag for patch 1
>>>>
>>>> Changes in v5:
>>>> - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
>>>> - fix wrong definition in patch 1
>>>> - fix spelling mistake in patch 2: leaset -> least
>>>> - update comments in patch 3
>>>> - add reviewed-by tag in patch 2,3
>>>>
>>>> Changes in v4:
>>>> - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
>>>>    suggested by Jan Kara.
>>>> - remove unused 'in_groups_with_pending_reqs',
>>>>
>>>> Changes in v3:
>>>> - remove the cleanup patch that is irrelevant now(I'll post it
>>>>    separately).
>>>> - instead of hacking wr queues and using weights tree insertion/removal,
>>>>    using bfq_add/del_bfqq_busy() to count the number of groups
>>>>    (suggested by Jan Kara).
>>>>
>>>> Changes in v2:
>>>> - Use a different approch to count root group, which is much simple.
>>>>
>>>> Currently, bfq can't handle sync io concurrently as long as they
>>>> are not issued from root group. This is because
>>>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>>>> bfq_asymmetric_scenario().
>>>>
>>>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>>>
>>>> Before this patchset:
>>>> 1) root group will never be counted.
>>>> 2) Count if bfqg or it's child bfqgs have pending requests.
>>>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>>>
>>>> After this patchset:
>>>> 1) root group is counted.
>>>> 2) Count if bfqg has pending requests.
>>>> 3) Don't count if bfqg complete all the requests.
>>>>
>>>> With the above changes, concurrent sync io can be supported if only
>>>> one group is activated.
>>>>
>>>> fio test script(startdelay is used to avoid queue merging):
>>>> [global]
>>>> filename=/dev/sda
>>>> allow_mounted_write=0
>>>> ioengine=psync
>>>> direct=1
>>>> ioscheduler=bfq
>>>> offset_increment=10g
>>>> group_reporting
>>>> rw=randwrite
>>>> bs=4k
>>>>
>>>> [test1]
>>>> numjobs=1
>>>>
>>>> [test2]
>>>> startdelay=1
>>>> numjobs=1
>>>>
>>>> [test3]
>>>> startdelay=2
>>>> numjobs=1
>>>>
>>>> [test4]
>>>> startdelay=3
>>>> numjobs=1
>>>>
>>>> [test5]
>>>> startdelay=4
>>>> numjobs=1
>>>>
>>>> [test6]
>>>> startdelay=5
>>>> numjobs=1
>>>>
>>>> [test7]
>>>> startdelay=6
>>>> numjobs=1
>>>>
>>>> [test8]
>>>> startdelay=7
>>>> numjobs=1
>>>>
>>>> test result:
>>>> running fio on root cgroup
>>>> v5.18:	   112 Mib/s
>>>> v5.18-patched: 112 Mib/s
>>>>
>>>> running fio on non-root cgroup
>>>> v5.18:	   51.2 Mib/s
>>>> v5.18-patched: 112 Mib/s
>>>>
>>>> Note that I also test null_blk with "irqmode=2
>>>> completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
>>>> that service guarantees are still preserved.
>>>>
>>> Your patches seem ok to me now (thanks for you contribution and, above all, for your patience). I have only a high-level concern: what do you mean when you say that service guarantees are still preserved? What test did you run exactly? This point is very important to me. I'd like to see some convincing test with differentiated weights. In case you don't have other tools for executing such tests quickly, you may want to use the bandwidth-latency test in my simple S benchmark suite (for which I'm willing to help).
>>
>> Is there any test that you wish me to try?
>>
>> By the way, I think for the case that multiple groups are activaced, (
>> specifically num_groups_with_pendind_rqs > 1), io path in bfq is the
>> same with or without this patchset.

I just ran the test for one time, result is a liiter inconsistent, do
you think it's in the normal fluctuation range?

test script:
fio -filename=/dev/nullb0 -ioengine=libaio -ioscheduler=bfq -jumjobs=1 
-iodepth=64 -direct=1 -bs=4k -rw=randread -runtime=60 -name=test

without this patchset:
|                 |      |      |      |      |      |
| --------------- | ---- | ---- | ---- | ---- | ---- |
| cg1 weight      | 10   | 20   | 30   | 40   | 50   |
| cg2 weight      | 90   | 80   | 70   | 60   | 50   |
| cg1 bw MiB/s    | 25.8 | 51.0 | 80.1 | 90.5 | 138  |
| cg2 bw MiB/s    | 193  | 179  | 162  | 127  | 136  |
| cg2 bw / cg1 bw | 7.48 | 3.51 | 2.02 | 1.40 | 0.98 |

with this patchset
|                 |      |      |      |      |      |
| --------------- | ---- | ---- | ---- | ---- | ---- |
| cg1 weight      | 10   | 20   | 30   | 40   | 50   |
| cg2 weight      | 90   | 80   | 70   | 60   | 50   |
| cg1 bw MiB/s    | 21.5 | 43.9 | 62.7 | 87.4 | 136  |
| cg2 bw MiB/s    | 195  | 185  | 173  | 138  | 141  |
| cg2 bw / cg1 bw | 9.07 | 4.21 | 2.75 | 1.57 | 0.96 |
>>
> 
> The tests cases you mentioned are ok for me (whatever tool or personal
> code you use to run them).  Just show me your results with and without
> your patchset applied.
> 
> Thanks,
> Paolo
> 
>> Thanks,
>> Kuai
>>> Thanks,
>>> Paolo
>>>> Previous versions:
>>>> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
>>>> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
>>>> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
>>>> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
>>>> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
>>>> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
>>>> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
>>>> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/
>>>>
>>>>
>>>> Yu Kuai (6):
>>>>   block, bfq: support to track if bfqq has pending requests
>>>>   block, bfq: record how many queues have pending requests
>>>>   block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>>>>   block, bfq: do not idle if only one group is activated
>>>>   block, bfq: cleanup bfq_weights_tree add/remove apis
>>>>   block, bfq: cleanup __bfq_weights_tree_remove()
>>>>
>>>> block/bfq-cgroup.c  | 10 +++++++
>>>> block/bfq-iosched.c | 71 +++++++--------------------------------------
>>>> block/bfq-iosched.h | 30 +++++++++----------
>>>> block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
>>>> 4 files changed, 76 insertions(+), 104 deletions(-)
>>>>
>>>> -- 
>>>> 2.31.1
>>>>
>>> .
> 
> .
>

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Yu Kuai 1 year, 6 months ago

Hi, Paolo

在 2022/10/11 17:36, Yu Kuai 写道:
>>>> Your patches seem ok to me now (thanks for you contribution and, 
>>>> above all, for your patience). I have only a high-level concern: 
>>>> what do you mean when you say that service guarantees are still 
>>>> preserved? What test did you run exactly? This point is very 
>>>> important to me. I'd like to see some convincing test with 
>>>> differentiated weights. In case you don't have other tools for 
>>>> executing such tests quickly, you may want to use the 
>>>> bandwidth-latency test in my simple S benchmark suite (for which I'm 
>>>> willing to help).
>>>
>>> Is there any test that you wish me to try?
>>>
>>> By the way, I think for the case that multiple groups are activaced, (
>>> specifically num_groups_with_pendind_rqs > 1), io path in bfq is the
>>> same with or without this patchset.
> 
> I just ran the test for one time, result is a liiter inconsistent, do
> you think it's in the normal fluctuation range?

I rerun the manually test for 5 times, here is the average result:

without this patchset / with this patchset:

| --------------- | ------------- | ------------ | -------------- | 
------------- | -------------- |
| cg1 weight      | 10            | 20           | 30             | 40 
          | 50             |
| cg2 weight      | 90            | 80           | 70             | 60 
          | 50             |
| cg1 bw MiB/s    | 21.4 / 21.74  | 42.72 / 46.6 | 63.82 / 61.52  | 
94.74 / 90.92 | 140 / 138.2    |
| cg2 bw MiB/s    | 197.2 / 197.4 | 182 / 181.2  | 171.2 / 173.44 | 162 
/ 156.8   | 138.6 / 137.04 |
| cg2 bw / cg1 bw | 9.22 / 9.08   | 4.26 / 3.89  | 2.68 / 2.82    | 1.71 
/ 1.72   | 0.99 / 0.99    |

> 
> test script:
> fio -filename=/dev/nullb0 -ioengine=libaio -ioscheduler=bfq -jumjobs=1 
> -iodepth=64 -direct=1 -bs=4k -rw=randread -runtime=60 -name=test
> 
> without this patchset:
> |                 |      |      |      |      |      |
> | --------------- | ---- | ---- | ---- | ---- | ---- |
> | cg1 weight      | 10   | 20   | 30   | 40   | 50   |
> | cg2 weight      | 90   | 80   | 70   | 60   | 50   |
> | cg1 bw MiB/s    | 25.8 | 51.0 | 80.1 | 90.5 | 138  |
> | cg2 bw MiB/s    | 193  | 179  | 162  | 127  | 136  |
> | cg2 bw / cg1 bw | 7.48 | 3.51 | 2.02 | 1.40 | 0.98 |
> 
> with this patchset
> |                 |      |      |      |      |      |
> | --------------- | ---- | ---- | ---- | ---- | ---- |
> | cg1 weight      | 10   | 20   | 30   | 40   | 50   |
> | cg2 weight      | 90   | 80   | 70   | 60   | 50   |
> | cg1 bw MiB/s    | 21.5 | 43.9 | 62.7 | 87.4 | 136  |
> | cg2 bw MiB/s    | 195  | 185  | 173  | 138  | 141  |
> | cg2 bw / cg1 bw | 9.07 | 4.21 | 2.75 | 1.57 | 0.96 |
>>>
>>
>> The tests cases you mentioned are ok for me (whatever tool or personal
>> code you use to run them).  Just show me your results with and without
>> your patchset applied.
>>
>> Thanks,
>> Paolo
>>
>>> Thanks,
>>> Kuai
>>>> Thanks,
>>>> Paolo
>>>>> Previous versions:
>>>>> RFC: 
>>>>> https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/ 
>>>>>
>>>>> v1: 
>>>>> https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/ 
>>>>>
>>>>> v2: 
>>>>> https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/ 
>>>>>
>>>>> v3: 
>>>>> https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
>>>>> v4: 
>>>>> https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/ 
>>>>>
>>>>> v5: 
>>>>> https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/ 
>>>>>
>>>>> v6: 
>>>>> https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/ 
>>>>>
>>>>> v7: 
>>>>> https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/ 
>>>>>
>>>>>
>>>>>
>>>>> Yu Kuai (6):
>>>>>   block, bfq: support to track if bfqq has pending requests
>>>>>   block, bfq: record how many queues have pending requests
>>>>>   block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>>>>>   block, bfq: do not idle if only one group is activated
>>>>>   block, bfq: cleanup bfq_weights_tree add/remove apis
>>>>>   block, bfq: cleanup __bfq_weights_tree_remove()
>>>>>
>>>>> block/bfq-cgroup.c  | 10 +++++++
>>>>> block/bfq-iosched.c | 71 +++++++--------------------------------------
>>>>> block/bfq-iosched.h | 30 +++++++++----------
>>>>> block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
>>>>> 4 files changed, 76 insertions(+), 104 deletions(-)
>>>>>
>>>>> -- 
>>>>> 2.31.1
>>>>>
>>>> .
>>
>> .
>>
> 
> .
>

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Paolo VALENTE 1 year, 6 months ago


> Il giorno 18 ott 2022, alle ore 06:00, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
> 
> Hi, Paolo
> 
> 在 2022/10/11 17:36, Yu Kuai 写道:
>>>>> Your patches seem ok to me now (thanks for you contribution and, above all, for your patience). I have only a high-level concern: what do you mean when you say that service guarantees are still preserved? What test did you run exactly? This point is very important to me. I'd like to see some convincing test with differentiated weights. In case you don't have other tools for executing such tests quickly, you may want to use the bandwidth-latency test in my simple S benchmark suite (for which I'm willing to help).
>>>> 
>>>> Is there any test that you wish me to try?
>>>> 
>>>> By the way, I think for the case that multiple groups are activaced, (
>>>> specifically num_groups_with_pendind_rqs > 1), io path in bfq is the
>>>> same with or without this patchset.
>> I just ran the test for one time, result is a liiter inconsistent, do
>> you think it's in the normal fluctuation range?
> 
> I rerun the manually test for 5 times, here is the average result:
> 
> without this patchset / with this patchset:
> 
> | --------------- | ------------- | ------------ | -------------- | ------------- | -------------- |
> | cg1 weight      | 10            | 20           | 30             | 40          | 50             |
> | cg2 weight      | 90            | 80           | 70             | 60          | 50             |
> | cg1 bw MiB/s    | 21.4 / 21.74  | 42.72 / 46.6 | 63.82 / 61.52  | 94.74 / 90.92 | 140 / 138.2    |
> | cg2 bw MiB/s    | 197.2 / 197.4 | 182 / 181.2  | 171.2 / 173.44 | 162 / 156.8   | 138.6 / 137.04 |
> | cg2 bw / cg1 bw | 9.22 / 9.08   | 4.26 / 3.89  | 2.68 / 2.82    | 1.71 / 1.72   | 0.99 / 0.99    |

Great!  Results are (statistically) the same, with and without your
patchset.  For me your patches are ok.  Thank you very much for this
contribution, and sorry again for my delay.

Acked-by: Paolo Valente <paolo.valente@linaro.org>

Thanks,
Paolo

> 
>> test script:
>> fio -filename=/dev/nullb0 -ioengine=libaio -ioscheduler=bfq -jumjobs=1 -iodepth=64 -direct=1 -bs=4k -rw=randread -runtime=60 -name=test
>> without this patchset:
>> |                 |      |      |      |      |      |
>> | --------------- | ---- | ---- | ---- | ---- | ---- |
>> | cg1 weight      | 10   | 20   | 30   | 40   | 50   |
>> | cg2 weight      | 90   | 80   | 70   | 60   | 50   |
>> | cg1 bw MiB/s    | 25.8 | 51.0 | 80.1 | 90.5 | 138  |
>> | cg2 bw MiB/s    | 193  | 179  | 162  | 127  | 136  |
>> | cg2 bw / cg1 bw | 7.48 | 3.51 | 2.02 | 1.40 | 0.98 |
>> with this patchset
>> |                 |      |      |      |      |      |
>> | --------------- | ---- | ---- | ---- | ---- | ---- |
>> | cg1 weight      | 10   | 20   | 30   | 40   | 50   |
>> | cg2 weight      | 90   | 80   | 70   | 60   | 50   |
>> | cg1 bw MiB/s    | 21.5 | 43.9 | 62.7 | 87.4 | 136  |
>> | cg2 bw MiB/s    | 195  | 185  | 173  | 138  | 141  |
>> | cg2 bw / cg1 bw | 9.07 | 4.21 | 2.75 | 1.57 | 0.96 |
>>>> 
>>> 
>>> The tests cases you mentioned are ok for me (whatever tool or personal
>>> code you use to run them).  Just show me your results with and without
>>> your patchset applied.
>>> 
>>> Thanks,
>>> Paolo
>>> 
>>>> Thanks,
>>>> Kuai
>>>>> Thanks,
>>>>> Paolo
>>>>>> Previous versions:
>>>>>> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/ 
>>>>>> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/ 
>>>>>> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/ 
>>>>>> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
>>>>>> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/ 
>>>>>> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/ 
>>>>>> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/ 
>>>>>> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/ 
>>>>>> 
>>>>>> 
>>>>>> Yu Kuai (6):
>>>>>>   block, bfq: support to track if bfqq has pending requests
>>>>>>   block, bfq: record how many queues have pending requests
>>>>>>   block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>>>>>>   block, bfq: do not idle if only one group is activated
>>>>>>   block, bfq: cleanup bfq_weights_tree add/remove apis
>>>>>>   block, bfq: cleanup __bfq_weights_tree_remove()
>>>>>> 
>>>>>> block/bfq-cgroup.c  | 10 +++++++
>>>>>> block/bfq-iosched.c | 71 +++++++--------------------------------------
>>>>>> block/bfq-iosched.h | 30 +++++++++----------
>>>>>> block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
>>>>>> 4 files changed, 76 insertions(+), 104 deletions(-)
>>>>>> 
>>>>>> -- 
>>>>>> 2.31.1
>>>>>> 
>>>>> .
>>> 
>>> .
>>> 
>> .
>

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Yu Kuai 1 year, 6 months ago

在 2022/10/25 14:34, Paolo VALENTE 写道:

>>
>> I rerun the manually test for 5 times, here is the average result:
>>
>> without this patchset / with this patchset:
>>
>> | --------------- | ------------- | ------------ | -------------- | ------------- | -------------- |
>> | cg1 weight      | 10            | 20           | 30             | 40          | 50             |
>> | cg2 weight      | 90            | 80           | 70             | 60          | 50             |
>> | cg1 bw MiB/s    | 21.4 / 21.74  | 42.72 / 46.6 | 63.82 / 61.52  | 94.74 / 90.92 | 140 / 138.2    |
>> | cg2 bw MiB/s    | 197.2 / 197.4 | 182 / 181.2  | 171.2 / 173.44 | 162 / 156.8   | 138.6 / 137.04 |
>> | cg2 bw / cg1 bw | 9.22 / 9.08   | 4.26 / 3.89  | 2.68 / 2.82    | 1.71 / 1.72   | 0.99 / 0.99    |
> 
> Great!  Results are (statistically) the same, with and without your
> patchset.  For me your patches are ok.  Thank you very much for this
> contribution, and sorry again for my delay.
> 
> Acked-by: Paolo Valente <paolo.valente@linaro.org>

Thanks for the review, it's nice to get this done finally!
> 
> Thanks,
> Paolo

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Yu Kuai 1 year, 7 months ago

Hi, Paolo

在 2022/09/28 0:38, Paolo Valente 写道:
> 
> 
>> Il giorno 16 set 2022, alle ore 09:19, Yu Kuai <yukuai1@huaweicloud.com> ha scritto:
>>
>> From: Yu Kuai <yukuai3@huawei.com>
>>
>> Changes in v11:
>> - keep the comments in bfq_weights_tree_remove() and move it to the
>> caller where bfqq can be freed.
>> - add two followed up cleanup patches.
>>
>> Changes in v10:
>> - Add reviewed-tag for patch 2
>>
>> Changes in v9:
>> - also update how many bfqqs have pending_reqs bfq_bfqq_move().
>> - fix one language in patch 4
>> - Add reviewed-tag for patch 1,3,4
>>
>> Changes in v8:
>> - Instead of using whether bfqq is busy, using whether bfqq has pending
>> requests. As Paolo pointed out the former way is problematic.
>>
>> Changes in v7:
>> - fix mismatch bfq_inc/del_busy_queues() and bfqq_add/del_bfqq_busy(),
>> also retest this patchset on v5.18 to make sure functionality is
>> correct.
>> - move the updating of 'bfqd->busy_queues' into new apis
>>
>> Changes in v6:
>> - add reviewed-by tag for patch 1
>>
>> Changes in v5:
>> - rename bfq_add_busy_queues() to bfq_inc_busy_queues() in patch 1
>> - fix wrong definition in patch 1
>> - fix spelling mistake in patch 2: leaset -> least
>> - update comments in patch 3
>> - add reviewed-by tag in patch 2,3
>>
>> Changes in v4:
>> - split bfq_update_busy_queues() to bfq_add/dec_busy_queues(),
>>    suggested by Jan Kara.
>> - remove unused 'in_groups_with_pending_reqs',
>>
>> Changes in v3:
>> - remove the cleanup patch that is irrelevant now(I'll post it
>>    separately).
>> - instead of hacking wr queues and using weights tree insertion/removal,
>>    using bfq_add/del_bfqq_busy() to count the number of groups
>>    (suggested by Jan Kara).
>>
>> Changes in v2:
>> - Use a different approch to count root group, which is much simple.
>>
>> Currently, bfq can't handle sync io concurrently as long as they
>> are not issued from root group. This is because
>> 'bfqd->num_groups_with_pending_reqs > 0' is always true in
>> bfq_asymmetric_scenario().
>>
>> The way that bfqg is counted into 'num_groups_with_pending_reqs':
>>
>> Before this patchset:
>> 1) root group will never be counted.
>> 2) Count if bfqg or it's child bfqgs have pending requests.
>> 3) Don't count if bfqg and it's child bfqgs complete all the requests.
>>
>> After this patchset:
>> 1) root group is counted.
>> 2) Count if bfqg has pending requests.
>> 3) Don't count if bfqg complete all the requests.
>>
>> With the above changes, concurrent sync io can be supported if only
>> one group is activated.
>>
>> fio test script(startdelay is used to avoid queue merging):
>> [global]
>> filename=/dev/sda
>> allow_mounted_write=0
>> ioengine=psync
>> direct=1
>> ioscheduler=bfq
>> offset_increment=10g
>> group_reporting
>> rw=randwrite
>> bs=4k
>>
>> [test1]
>> numjobs=1
>>
>> [test2]
>> startdelay=1
>> numjobs=1
>>
>> [test3]
>> startdelay=2
>> numjobs=1
>>
>> [test4]
>> startdelay=3
>> numjobs=1
>>
>> [test5]
>> startdelay=4
>> numjobs=1
>>
>> [test6]
>> startdelay=5
>> numjobs=1
>>
>> [test7]
>> startdelay=6
>> numjobs=1
>>
>> [test8]
>> startdelay=7
>> numjobs=1
>>
>> test result:
>> running fio on root cgroup
>> v5.18:	   112 Mib/s
>> v5.18-patched: 112 Mib/s
>>
>> running fio on non-root cgroup
>> v5.18:	   51.2 Mib/s
>> v5.18-patched: 112 Mib/s
>>
>> Note that I also test null_blk with "irqmode=2
>> completion_nsec=100000000(100ms) hw_queue_depth=1", and tests show
>> that service guarantees are still preserved.
>>
> 
> Your patches seem ok to me now (thanks for you contribution and, above all, for your patience). I have only a high-level concern: what do you mean when you say that service guarantees are still preserved? What test did you run exactly? This point is very important to me. I'd like to see some convincing test with differentiated weights. In case you don't have other tools for executing such tests quickly, you may want to use the bandwidth-latency test in my simple S benchmark suite (for which I'm willing to help).

I'm runnnig some tests manually, just issuing same io to two cgroups,
and changing weights manually, specifically (1:10, 2:8, ..., 5:5),
then observe bandwidth from two cgroups.

Of course I'm glad to try your benchmark suite.

Thanks,
Kuai
> 
> Thanks,
> Paolo
> 
>> Previous versions:
>> RFC: https://lore.kernel.org/all/20211127101132.486806-1-yukuai3@huawei.com/
>> v1: https://lore.kernel.org/all/20220305091205.4188398-1-yukuai3@huawei.com/
>> v2: https://lore.kernel.org/all/20220416093753.3054696-1-yukuai3@huawei.com/
>> v3: https://lore.kernel.org/all/20220427124722.48465-1-yukuai3@huawei.com/
>> v4: https://lore.kernel.org/all/20220428111907.3635820-1-yukuai3@huawei.com/
>> v5: https://lore.kernel.org/all/20220428120837.3737765-1-yukuai3@huawei.com/
>> v6: https://lore.kernel.org/all/20220523131818.2798712-1-yukuai3@huawei.com/
>> v7: https://lore.kernel.org/all/20220528095020.186970-1-yukuai3@huawei.com/
>>
>>
>> Yu Kuai (6):
>>   block, bfq: support to track if bfqq has pending requests
>>   block, bfq: record how many queues have pending requests
>>   block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
>>   block, bfq: do not idle if only one group is activated
>>   block, bfq: cleanup bfq_weights_tree add/remove apis
>>   block, bfq: cleanup __bfq_weights_tree_remove()
>>
>> block/bfq-cgroup.c  | 10 +++++++
>> block/bfq-iosched.c | 71 +++++++--------------------------------------
>> block/bfq-iosched.h | 30 +++++++++----------
>> block/bfq-wf2q.c    | 69 ++++++++++++++++++++++++++-----------------
>> 4 files changed, 76 insertions(+), 104 deletions(-)
>>
>> -- 
>> 2.31.1
>>
> 
> .
>

Re: [patch v11 0/6] support concurrent sync io for bfq on a specail occasion

Posted by Jens Axboe 1 year, 6 months ago

On Fri, 16 Sep 2022 15:19:36 +0800, Yu Kuai wrote:
> From: Yu Kuai <yukuai3@huawei.com>
> 
> Changes in v11:
>  - keep the comments in bfq_weights_tree_remove() and move it to the
>  caller where bfqq can be freed.
>  - add two followed up cleanup patches.
> 
> [...]

Applied, thanks!

[1/6] block, bfq: support to track if bfqq has pending requests
      commit: 3d89bd12d352e20f4f7c8f11a0f1a712b95a5295
[2/6] block, bfq: record how many queues have pending requests
      commit: 60a6e10c537a7459dd53882186bd16fff257fb03
[3/6] block, bfq: refactor the counting of 'num_groups_with_pending_reqs'
      commit: 71f8ca77cb8764d46f656b725999e8b8b1aec215
[4/6] block, bfq: do not idle if only one group is activated
      commit: eed3ecc991c90a4a0ce32ea2b35378dc351f012b
[5/6] block, bfq: cleanup bfq_weights_tree add/remove apis
      commit: afdba14612622ec75896e5646950b3562a9aadd3
[6/6] block, bfq: cleanup __bfq_weights_tree_remove()
      commit: eb5bca73655cb6aa3bb608253e1e47283240c933

Best regards,
-- 
Jens Axboe