The disk I/O throttling options have been listed for a long time but
never explained on the QEMU man page.

Suggested-by: Nini Gu <ngu@redhat.com>
Cc: Alberto Garcia <berto@igalia.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 qemu-options.hx | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/qemu-options.hx b/qemu-options.hx
index 5633d39..b2254cc 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -587,6 +587,31 @@ file sectors into the image file.
 conversion of plain zero writes by the OS to driver specific optimized
 zero write commands. You may even choose "unmap" if @var{discard} is set
 to "unmap" to allow a zero write to be converted to an UNMAP operation.
+@item bps=@var{b},bps_rd=@var{r},bps_wr=@var{w}
+Specify bandwidth throttling limits in bytes per second, either for all request
+types or for reads or writes only.  Values must be larger than the maximum
+request size to avoid timeouts or hangs in the guest.  At minimum use 2 MB/s
+for disks.
+@item bps_max=@var{bm},bps_rd_max=@var{rm},bps_wr_max=@var{wm}
+Specify bursts in bytes per second, either for all request types or for reads
+or writes only.  Bursts allow the guest I/O to spike above the limit
+temporarily.  The default burst value is 1/10th of the limit.
+@item iops=@var{i},iops_rd=@var{r},iops_wr=@var{w}
+Specify request rate limits in requests per second, either for all request
+types or for reads or writes only.
+@item bps_max=@var{bm},bps_rd_max=@var{rm},bps_wr_max=@var{wm}
+Specify bursts in requests per second, either for all request types or for reads
+or writes only.  Bursts allow the guest I/O to spike above the limit
+temporarily.  The default burst value is 1/10th of the limit.
+@item iops_size=@var{is}
+Let every @var{is} bytes of a request count as a new request for iops
+throttling purposes.  Use this option to prevent guests from circumventing iops
+limits by sending fewer but larger requests.
+@item group=@var{g}
+Join a throttling quota group with given name @var{g}.  All drives that are
+members of the same group are accounted for together.  Use this option to
+prevent guests from circumventing throttling limits by using many small disks
+instead of a single larger disk.
 @end table
 
 By default, the @option{cache=writeback} mode is used. It will report data
-- 
2.9.3

On 02/20/2017 10:52 AM, Stefan Hajnoczi wrote:
> The disk I/O throttling options have been listed for a long time but
> never explained on the QEMU man page.
> 
> Suggested-by: Nini Gu <ngu@redhat.com>
> Cc: Alberto Garcia <berto@igalia.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  qemu-options.hx | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 5633d39..b2254cc 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -587,6 +587,31 @@ file sectors into the image file.
>  conversion of plain zero writes by the OS to driver specific optimized
>  zero write commands. You may even choose "unmap" if @var{discard} is set
>  to "unmap" to allow a zero write to be converted to an UNMAP operation.
> +@item bps=@var{b},bps_rd=@var{r},bps_wr=@var{w}
> +Specify bandwidth throttling limits in bytes per second, either for all request
> +types or for reads or writes only.  Values must be larger than the maximum
> +request size to avoid timeouts or hangs in the guest.  At minimum use 2 MB/s

s/At minimum/At a minimum,/

With that tweak,
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

On Mon, Feb 20, 2017 at 01:40:21PM -0600, Eric Blake wrote:
> On 02/20/2017 10:52 AM, Stefan Hajnoczi wrote:
> > The disk I/O throttling options have been listed for a long time but
> > never explained on the QEMU man page.
> > 
> > Suggested-by: Nini Gu <ngu@redhat.com>
> > Cc: Alberto Garcia <berto@igalia.com>
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> >  qemu-options.hx | 25 +++++++++++++++++++++++++
> >  1 file changed, 25 insertions(+)
> > 
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index 5633d39..b2254cc 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -587,6 +587,31 @@ file sectors into the image file.
> >  conversion of plain zero writes by the OS to driver specific optimized
> >  zero write commands. You may even choose "unmap" if @var{discard} is set
> >  to "unmap" to allow a zero write to be converted to an UNMAP operation.
> > +@item bps=@var{b},bps_rd=@var{r},bps_wr=@var{w}
> > +Specify bandwidth throttling limits in bytes per second, either for all request
> > +types or for reads or writes only.  Values must be larger than the maximum
> > +request size to avoid timeouts or hangs in the guest.  At minimum use 2 MB/s
> 
> s/At minimum/At a minimum,/
> 
> With that tweak,
> Reviewed-by: Eric Blake <eblake@redhat.com>

Will fix when merging.

Stefan

On Mon 20 Feb 2017 05:52:04 PM CET, Stefan Hajnoczi wrote:
> The disk I/O throttling options have been listed for a long time but
> never explained on the QEMU man page.

> +@item bps=@var{b},bps_rd=@var{r},bps_wr=@var{w}
> +Specify bandwidth throttling limits in bytes per second, either for all request
> +types or for reads or writes only.

Perhaps for the manual page it makes sense to use them, but bps, bps_rd,
etc. are the legacy names for those options. They're internally renamed
to throttling.bps-total, throttling.bps-read, ...

> Values must be larger than the maximum
> +request size to avoid timeouts or hangs in the guest.  At minimum use 2 MB/s
> +for disks.

Is this so? throttle_compute_wait() does not use the request size at
all. The size is used to do the accounting afterwards. In other words:
requests are not throttled if they exceed the limit, they are throttled
after the limit has been exceeded by previous requests.

A low limit will certainly slow down the guest and can cause
timeouts, but I don't know if being larger or smaller than the maximum
request size is what makes the difference.

> +@item bps_max=@var{bm},bps_rd_max=@var{rm},bps_wr_max=@var{wm}
> +Specify bursts in bytes per second, either for all request types or for reads
> +or writes only.  Bursts allow the guest I/O to spike above the limit
> +temporarily.  The default burst value is 1/10th of the limit.

"The default burst value is 1/10th of the limit" is an implementation
detail that the user doesn't need to know about. What it means is that a
bps limit of 10 MB/s is implemented internally as 1MB per 100ms.

I would leave that out, it doesn't even make sense that the burst limit
is lower than the normal limit, we actually forbid that (aaa1e77ffae52).

> +@item iops=@var{i},iops_rd=@var{r},iops_wr=@var{w}
> +Specify request rate limits in requests per second, either for all request
> +types or for reads or writes only.

> +@item bps_max=@var{bm},bps_rd_max=@var{rm},bps_wr_max=@var{wm}

You meant iops_max, iops_rd_max and iops_wr_max here?

Berto

On Tue, Feb 21, 2017 at 11:58:00AM +0100, Alberto Garcia wrote:
> On Mon 20 Feb 2017 05:52:04 PM CET, Stefan Hajnoczi wrote:
> > The disk I/O throttling options have been listed for a long time but
> > never explained on the QEMU man page.
> 
> > +@item bps=@var{b},bps_rd=@var{r},bps_wr=@var{w}
> > +Specify bandwidth throttling limits in bytes per second, either for all request
> > +types or for reads or writes only.
> 
> Perhaps for the manual page it makes sense to use them, but bps, bps_rd,
> etc. are the legacy names for those options. They're internally renamed
> to throttling.bps-total, throttling.bps-read, ...

I used the parameter names already listed in the -drive documentation.
We need to keep the legacy names but I will add another patch that
documents the new names and encourages using them.

> > Values must be larger than the maximum
> > +request size to avoid timeouts or hangs in the guest.  At minimum use 2 MB/s
> > +for disks.
> 
> Is this so? throttle_compute_wait() does not use the request size at
> all. The size is used to do the accounting afterwards. In other words:
> requests are not throttled if they exceed the limit, they are throttled
> after the limit has been exceeded by previous requests.
> 
> A low limit will certainly slow down the guest and can cause
> timeouts, but I don't know if being larger or smaller than the maximum
> request size is what makes the difference.

You are right, I'm still have the behavior of the old throttling
implementation in mind.

I'd still like to recommend a sane minimum because anything around the
size of a single request leads to timeouts and hangs inside the guest.

How about:

  Small values can lead to timeouts or hangs inside the guest.  A safe
  minimum for disks is 2 MB/s.

> > +@item bps_max=@var{bm},bps_rd_max=@var{rm},bps_wr_max=@var{wm}
> > +Specify bursts in bytes per second, either for all request types or for reads
> > +or writes only.  Bursts allow the guest I/O to spike above the limit
> > +temporarily.  The default burst value is 1/10th of the limit.
> 
> "The default burst value is 1/10th of the limit" is an implementation
> detail that the user doesn't need to know about. What it means is that a
> bps limit of 10 MB/s is implemented internally as 1MB per 100ms.
>
> I would leave that out, it doesn't even make sense that the burst limit
> is lower than the normal limit, we actually forbid that (aaa1e77ffae52).

Unfortunately the bps_max = avg / 10 default value is visible via
query-block.  A bug was recently reported about this:
https://bugzilla.redhat.com/show_bug.cgi?id=1414630

Would you prefer a private field in the throttling code so this detail
is hidden from users?

> > +@item iops=@var{i},iops_rd=@var{r},iops_wr=@var{w}
> > +Specify request rate limits in requests per second, either for all request
> > +types or for reads or writes only.
> 
> > +@item bps_max=@var{bm},bps_rd_max=@var{rm},bps_wr_max=@var{wm}
> 
> You meant iops_max, iops_rd_max and iops_wr_max here?

Yes, thanks!  Will fix in v2.

Stefan

On Wed 22 Feb 2017 12:14:33 PM CET, Stefan Hajnoczi wrote:
>> > Values must be larger than the maximum
>> > +request size to avoid timeouts or hangs in the guest.  At minimum use 2 MB/s
>> > +for disks.
>> 
>> Is this so? throttle_compute_wait() does not use the request size at
>> all. The size is used to do the accounting afterwards. In other
>> words: requests are not throttled if they exceed the limit, they are
>> throttled after the limit has been exceeded by previous requests.
>> 
>> A low limit will certainly slow down the guest and can cause
>> timeouts, but I don't know if being larger or smaller than the maximum
>> request size is what makes the difference.
>
> You are right, I'm still have the behavior of the old throttling
> implementation in mind.
>
> I'd still like to recommend a sane minimum because anything around the
> size of a single request leads to timeouts and hangs inside the guest.

Yes, that sounds sensible.

> How about:
>
>   Small values can lead to timeouts or hangs inside the guest.  A safe
>   minimum for disks is 2 MB/s.

Looks good to me.

>> > +@item bps_max=@var{bm},bps_rd_max=@var{rm},bps_wr_max=@var{wm}
>> > +Specify bursts in bytes per second, either for all request types or for reads
>> > +or writes only.  Bursts allow the guest I/O to spike above the limit
>> > +temporarily.  The default burst value is 1/10th of the limit.
>> 
>> "The default burst value is 1/10th of the limit" is an implementation
>> detail that the user doesn't need to know about. What it means is that a
>> bps limit of 10 MB/s is implemented internally as 1MB per 100ms.
>>
>> I would leave that out, it doesn't even make sense that the burst limit
>> is lower than the normal limit, we actually forbid that (aaa1e77ffae52).
>
> Unfortunately the bps_max = avg / 10 default value is visible via
> query-block.  A bug was recently reported about this:
> https://bugzilla.redhat.com/show_bug.cgi?id=1414630
>
> Would you prefer a private field in the throttling code so this detail
> is hidden from users?

Oh, I see. I would rather hide it from the user, either in throttle.c or
even in bdrv_block_device_info().

Berto