Bandaid to "fix" bulk stats API on big hosts

[libvirt] [RFC PATCH 1/2] rpc: Bump maximum message size to 32M

Posted by Peter Krempa 8 years, 8 months ago

While most of the APIs are okay with 16M messages, the bulk stats API
can run into the limit in big configurations. Before we devise a new
plan for this, bump this limit slightly to accomodate some more configs.
---
 src/rpc/virnetprotocol.x | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/rpc/virnetprotocol.x b/src/rpc/virnetprotocol.x
index ee9899059..901c67159 100644
--- a/src/rpc/virnetprotocol.x
+++ b/src/rpc/virnetprotocol.x
@@ -40,13 +40,13 @@ const VIR_NET_MESSAGE_INITIAL = 65536;
 const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262120;

 /* Maximum total message size (serialised). */
-const VIR_NET_MESSAGE_MAX = 16777216;
+const VIR_NET_MESSAGE_MAX = 33554432;

 /* Size of struct virNetMessageHeader (serialised)*/
 const VIR_NET_MESSAGE_HEADER_MAX = 24;

 /* Size of message payload */
-const VIR_NET_MESSAGE_PAYLOAD_MAX = 16777192;
+const VIR_NET_MESSAGE_PAYLOAD_MAX = 33554408;

 /* Size of message length field. Not counted in VIR_NET_MESSAGE_MAX
  * and VIR_NET_MESSAGE_INITIAL.
-- 
2.12.2

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [RFC PATCH 1/2] rpc: Bump maximum message size to 32M

Posted by Martin Kletzander 8 years, 8 months ago

On Mon, May 22, 2017 at 06:00:12PM +0200, Peter Krempa wrote:
>While most of the APIs are okay with 16M messages, the bulk stats API
>can run into the limit in big configurations. Before we devise a new
>plan for this, bump this limit slightly to accomodate some more configs.
>---
> src/rpc/virnetprotocol.x | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
>diff --git a/src/rpc/virnetprotocol.x b/src/rpc/virnetprotocol.x
>index ee9899059..901c67159 100644
>--- a/src/rpc/virnetprotocol.x
>+++ b/src/rpc/virnetprotocol.x
>@@ -40,13 +40,13 @@ const VIR_NET_MESSAGE_INITIAL = 65536;
> const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262120;
>
> /* Maximum total message size (serialised). */
>-const VIR_NET_MESSAGE_MAX = 16777216;
>+const VIR_NET_MESSAGE_MAX = 33554432;
>

ACK.

Although if you changed it to something nicer to read so that it doesn't
take someone too long to figure out it's actually just 32 MiB, that
would be even better.  Whatever suits you, be it (1 << 25) or (32 * 2 <<
20) or (32 * 1024 * 1024) or anything else.  I know other numbers could
be changed as well, but this one is most updated, currently.

> /* Size of struct virNetMessageHeader (serialised)*/
> const VIR_NET_MESSAGE_HEADER_MAX = 24;
>
> /* Size of message payload */
>-const VIR_NET_MESSAGE_PAYLOAD_MAX = 16777192;
>+const VIR_NET_MESSAGE_PAYLOAD_MAX = 33554408;
>

and this could be:

const VIR_NET_MESSAGE_PAYLOAD_MAX = VIR_NET_MESSAGE_MAX - VIR_NET_MESSAGE_HEADER_MAX;

or is none of that supported by the parser?  In that case, just leave it
like that.

> /* Size of message length field. Not counted in VIR_NET_MESSAGE_MAX
>  * and VIR_NET_MESSAGE_INITIAL.
>--
>2.12.2
>
>--
>libvir-list mailing list
>libvir-list@redhat.com
>https://www.redhat.com/mailman/listinfo/libvir-list
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [RFC PATCH 1/2] rpc: Bump maximum message size to 32M

Posted by Peter Krempa 8 years, 8 months ago

On Tue, May 23, 2017 at 09:38:01 +0200, Martin Kletzander wrote:
> On Mon, May 22, 2017 at 06:00:12PM +0200, Peter Krempa wrote:
> > While most of the APIs are okay with 16M messages, the bulk stats API
> > can run into the limit in big configurations. Before we devise a new
> > plan for this, bump this limit slightly to accomodate some more configs.
> > ---
> > src/rpc/virnetprotocol.x | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/src/rpc/virnetprotocol.x b/src/rpc/virnetprotocol.x
> > index ee9899059..901c67159 100644
> > --- a/src/rpc/virnetprotocol.x
> > +++ b/src/rpc/virnetprotocol.x
> > @@ -40,13 +40,13 @@ const VIR_NET_MESSAGE_INITIAL = 65536;
> > const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262120;
> > 
> > /* Maximum total message size (serialised). */
> > -const VIR_NET_MESSAGE_MAX = 16777216;
> > +const VIR_NET_MESSAGE_MAX = 33554432;
> > 
> 
> ACK.
> 
> Although if you changed it to something nicer to read so that it doesn't
> take someone too long to figure out it's actually just 32 MiB, that
> would be even better.  Whatever suits you, be it (1 << 25) or (32 * 2 <<
> 20) or (32 * 1024 * 1024) or anything else.  I know other numbers could
> be changed as well, but this one is most updated, currently.


I thought about that, but rpcgen is not clever enough:

  GEN      rpc/virnetprotocol.h
const VIR_NET_MESSAGE_MAX = 32 * 1024 * 1024;
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rpc/virnetprotocol.x, line 43: expected ';'
cannot shutdown /usr/bin/rpcgen:  at ./rpc/genprotocol.pl line 139.


> > /* Size of struct virNetMessageHeader (serialised)*/
> > const VIR_NET_MESSAGE_HEADER_MAX = 24;
> > 
> > /* Size of message payload */
> > -const VIR_NET_MESSAGE_PAYLOAD_MAX = 16777192;
> > +const VIR_NET_MESSAGE_PAYLOAD_MAX = 33554408;
> > 
> 
> and this could be:
> 
> const VIR_NET_MESSAGE_PAYLOAD_MAX = VIR_NET_MESSAGE_MAX - VIR_NET_MESSAGE_HEADER_MAX;

And this does not work in the same manner.

> 
> or is none of that supported by the parser?  In that case, just leave it
> like that.

Welp ...
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list