[v1] virtio-gpu: OUT_OF_MEMORY if failing to create udmabuf

[PATCH] virtio-gpu: OUT_OF_MEMORY if failing to create udmabuf

Posted by Dongwon Kim 2 years, 7 months ago

Respond with VIRTIO_GPU_RESP_ERR_OUT_OF_MEMORY if it fails to create
an udmabuf for the blob resource.

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: Vivek Kasireddy <vivek.kasireddy@intel.com>
Signed-off-by: Dongwon Kim <dongwon.kim@intel.com>
---
 hw/display/virtio-gpu.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/hw/display/virtio-gpu.c b/hw/display/virtio-gpu.c
index 66cddd94d9..efe66ca7a3 100644
--- a/hw/display/virtio-gpu.c
+++ b/hw/display/virtio-gpu.c
@@ -635,9 +635,11 @@ static void virtio_gpu_do_set_scanout(VirtIOGPU *g,
             if (!virtio_gpu_update_dmabuf(g, scanout_id, res, fb, r)) {
                 virtio_gpu_update_scanout(g, scanout_id, res, r);
                 return;
+            } else {
+                *error = VIRTIO_GPU_RESP_ERR_OUT_OF_MEMORY;
+                return;
             }
         }
-
         data = res->blob;
     } else {
         data = (uint8_t *)pixman_image_get_data(res->image);
-- 
2.34.1

Re: [PATCH] virtio-gpu: OUT_OF_MEMORY if failing to create udmabuf

Posted by Marc-André Lureau 2 years, 7 months ago

Hi

On Wed, Jun 21, 2023 at 11:40 PM Dongwon Kim <dongwon.kim@intel.com> wrote:

> Respond with VIRTIO_GPU_RESP_ERR_OUT_OF_MEMORY if it fails to create
> an udmabuf for the blob resource.
>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
> Cc: Vivek Kasireddy <vivek.kasireddy@intel.com>
> Signed-off-by: Dongwon Kim <dongwon.kim@intel.com>
> ---
>  hw/display/virtio-gpu.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/hw/display/virtio-gpu.c b/hw/display/virtio-gpu.c
> index 66cddd94d9..efe66ca7a3 100644
> --- a/hw/display/virtio-gpu.c
> +++ b/hw/display/virtio-gpu.c
> @@ -635,9 +635,11 @@ static void virtio_gpu_do_set_scanout(VirtIOGPU *g,
>              if (!virtio_gpu_update_dmabuf(g, scanout_id, res, fb, r)) {
>                  virtio_gpu_update_scanout(g, scanout_id, res, r);
>                  return;
> +            } else {
> +                *error = VIRTIO_GPU_RESP_ERR_OUT_OF_MEMORY;
> +                return;
>              }
>          }
> -
>

unrelated style change


>          data = res->blob;
>      } else {
>          data = (uint8_t *)pixman_image_get_data(res->image);
> --
> 2.34.1
>
>
>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>


-- 
Marc-André Lureau

[PATCH v2] virtio-gpu: OUT_OF_MEMORY if failing to create udmabuf

Posted by Dongwon Kim 2 years, 7 months ago

Respond with VIRTIO_GPU_RESP_ERR_OUT_OF_MEMORY if it fails to create
an udmabuf for the blob resource.

v2: consolidated return statments and removed an unnecessary style change

Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: Vivek Kasireddy <vivek.kasireddy@intel.com>
Signed-off-by: Dongwon Kim <dongwon.kim@intel.com>
---
 hw/display/virtio-gpu.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/hw/display/virtio-gpu.c b/hw/display/virtio-gpu.c
index 66cddd94d9..ce019941b8 100644
--- a/hw/display/virtio-gpu.c
+++ b/hw/display/virtio-gpu.c
@@ -634,8 +634,10 @@ static void virtio_gpu_do_set_scanout(VirtIOGPU *g,
         if (console_has_gl(scanout->con)) {
             if (!virtio_gpu_update_dmabuf(g, scanout_id, res, fb, r)) {
                 virtio_gpu_update_scanout(g, scanout_id, res, r);
-                return;
+            } else {
+                *error = VIRTIO_GPU_RESP_ERR_OUT_OF_MEMORY;
             }
+            return;
         }
 
         data = res->blob;
-- 
2.34.1