For very large framebuffers, it is theoretically possible for the result
of 'vs->throttle_output_offset * VNC_THROTTLE_OUTPUT_LIMIT_SCALE' to
exceed the size of a 32-bit int. For this to happen in practice, the
video RAM would have to be set to a large enough value, which is not
likely today. None the less we can be paranoid against future growth by
using division instead of multiplication when checking the limits.

Reported-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 ui/vnc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index 93731accb6..e14e524764 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -1572,8 +1572,8 @@ void vnc_write(VncState *vs, const void *data, size_t len)
      * handshake, or from the job thread's VncState clone
      */
     if (vs->throttle_output_offset != 0 &&
-        vs->output.offset > (vs->throttle_output_offset *
-                             VNC_THROTTLE_OUTPUT_LIMIT_SCALE)) {
+        (vs->output.offset / VNC_THROTTLE_OUTPUT_LIMIT_SCALE) >
+        vs->throttle_output_offset) {
         trace_vnc_client_output_limit(vs, vs->ioc, vs->output.offset,
                                       vs->throttle_output_offset);
         vnc_disconnect_start(vs);
-- 
2.14.3

On 02/05/18 12:49, Daniel P. Berrangé wrote:
> For very large framebuffers, it is theoretically possible for the result
> of 'vs->throttle_output_offset * VNC_THROTTLE_OUTPUT_LIMIT_SCALE' to
> exceed the size of a 32-bit int. For this to happen in practice, the
> video RAM would have to be set to a large enough value, which is not
> likely today. None the less we can be paranoid against future growth by
> using division instead of multiplication when checking the limits.
> 
> Reported-by: Laszlo Ersek <lersek@redhat.com>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  ui/vnc.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/ui/vnc.c b/ui/vnc.c
> index 93731accb6..e14e524764 100644
> --- a/ui/vnc.c
> +++ b/ui/vnc.c
> @@ -1572,8 +1572,8 @@ void vnc_write(VncState *vs, const void *data, size_t len)
>       * handshake, or from the job thread's VncState clone
>       */
>      if (vs->throttle_output_offset != 0 &&
> -        vs->output.offset > (vs->throttle_output_offset *
> -                             VNC_THROTTLE_OUTPUT_LIMIT_SCALE)) {
> +        (vs->output.offset / VNC_THROTTLE_OUTPUT_LIMIT_SCALE) >
> +        vs->throttle_output_offset) {
>          trace_vnc_client_output_limit(vs, vs->ioc, vs->output.offset,
>                                        vs->throttle_output_offset);
>          vnc_disconnect_start(vs);
> 

Reviewed-by: Laszlo Ersek <lersek@redhat.com>

On 02/05/2018 08:49 AM, Daniel P. Berrangé wrote:
> For very large framebuffers, it is theoretically possible for the result
> of 'vs->throttle_output_offset * VNC_THROTTLE_OUTPUT_LIMIT_SCALE' to
> exceed the size of a 32-bit int. For this to happen in practice, the
> video RAM would have to be set to a large enough value, which is not
> likely today. None the less we can be paranoid against future growth by
> using division instead of multiplication when checking the limits.
> 
> Reported-by: Laszlo Ersek <lersek@redhat.com>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

> ---
>  ui/vnc.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/ui/vnc.c b/ui/vnc.c
> index 93731accb6..e14e524764 100644
> --- a/ui/vnc.c
> +++ b/ui/vnc.c
> @@ -1572,8 +1572,8 @@ void vnc_write(VncState *vs, const void *data, size_t len)
>       * handshake, or from the job thread's VncState clone
>       */
>      if (vs->throttle_output_offset != 0 &&
> -        vs->output.offset > (vs->throttle_output_offset *
> -                             VNC_THROTTLE_OUTPUT_LIMIT_SCALE)) {
> +        (vs->output.offset / VNC_THROTTLE_OUTPUT_LIMIT_SCALE) >
> +        vs->throttle_output_offset) {
>          trace_vnc_client_output_limit(vs, vs->ioc, vs->output.offset,
>                                        vs->throttle_output_offset);
>          vnc_disconnect_start(vs);
>