qga-vss: Exit with non-zero code when register fail

[PATCH] qga-vss: Exit with non-zero code when register fail

Posted by Kostiantyn Kostiuk 6 months ago

QGA installer uses rundll32 to run the DLLCOMRegister function
from qga-vss.dll and perform VSS provider registration.
rundll32 ignores the return value of the function and always
exits with a zero exit code. This causes a situation where
the installer does not know the status of VSS provider registration.

This commit forces to change exit code when the VSS provider
registration fails.

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/rundll32

Signed-off-by: Kostiantyn Kostiuk <kkostiuk@redhat.com>
---
 qga/vss-win32/install.cpp | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/qga/vss-win32/install.cpp b/qga/vss-win32/install.cpp
index 5cea5bcf74..6ee2f44a10 100644
--- a/qga/vss-win32/install.cpp
+++ b/qga/vss-win32/install.cpp
@@ -385,7 +385,10 @@ out:
 STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int);
 STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int)
 {
-    COMRegister();
+    HRESULT hr = COMRegister();
+    if (FAILED(hr)) {
+        exit(hr);
+    }
 }
 
 STDAPI_(void) CALLBACK DLLCOMUnregister(HWND, HINSTANCE, LPSTR, int);
-- 
2.48.1

Re: [PATCH] qga-vss: Exit with non-zero code when register fail

Posted by Yan Vugenfirer 6 months ago

On Fri, Jun 20, 2025 at 12:33 PM Kostiantyn Kostiuk <kkostiuk@redhat.com> wrote:
>
> QGA installer uses rundll32 to run the DLLCOMRegister function
> from qga-vss.dll and perform VSS provider registration.
> rundll32 ignores the return value of the function and always
> exits with a zero exit code. This causes a situation where
> the installer does not know the status of VSS provider registration.
>
> This commit forces to change exit code when the VSS provider
> registration fails.
>
> https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/rundll32
>
> Signed-off-by: Kostiantyn Kostiuk <kkostiuk@redhat.com>
> ---
>  qga/vss-win32/install.cpp | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/qga/vss-win32/install.cpp b/qga/vss-win32/install.cpp
> index 5cea5bcf74..6ee2f44a10 100644
> --- a/qga/vss-win32/install.cpp
> +++ b/qga/vss-win32/install.cpp
> @@ -385,7 +385,10 @@ out:
>  STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int);
>  STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int)
>  {
> -    COMRegister();
> +    HRESULT hr = COMRegister();
> +    if (FAILED(hr)) {
> +        exit(hr);
> +    }
>  }
>
>  STDAPI_(void) CALLBACK DLLCOMUnregister(HWND, HINSTANCE, LPSTR, int);
> --
> 2.48.1
>
>

Reviewed-by: Yan Vugenfirer <yvugenfi@redhat.com>

Re: [PATCH] qga-vss: Exit with non-zero code when register fail

Posted by Dehan Meng 5 months, 3 weeks ago

Tested-by: Dehan Meng <*demeng@redhat.com <demeng@redhat.com>*>

On Fri, Jun 20, 2025 at 8:31 PM Yan Vugenfirer <yvugenfi@redhat.com> wrote:

> On Fri, Jun 20, 2025 at 12:33 PM Kostiantyn Kostiuk <kkostiuk@redhat.com>
> wrote:
> >
> > QGA installer uses rundll32 to run the DLLCOMRegister function
> > from qga-vss.dll and perform VSS provider registration.
> > rundll32 ignores the return value of the function and always
> > exits with a zero exit code. This causes a situation where
> > the installer does not know the status of VSS provider registration.
> >
> > This commit forces to change exit code when the VSS provider
> > registration fails.
> >
> >
> https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/rundll32
> >
> > Signed-off-by: Kostiantyn Kostiuk <kkostiuk@redhat.com>
> > ---
> >  qga/vss-win32/install.cpp | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/qga/vss-win32/install.cpp b/qga/vss-win32/install.cpp
> > index 5cea5bcf74..6ee2f44a10 100644
> > --- a/qga/vss-win32/install.cpp
> > +++ b/qga/vss-win32/install.cpp
> > @@ -385,7 +385,10 @@ out:
> >  STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int);
> >  STDAPI_(void) CALLBACK DLLCOMRegister(HWND, HINSTANCE, LPSTR, int)
> >  {
> > -    COMRegister();
> > +    HRESULT hr = COMRegister();
> > +    if (FAILED(hr)) {
> > +        exit(hr);
> > +    }
> >  }
> >
> >  STDAPI_(void) CALLBACK DLLCOMUnregister(HWND, HINSTANCE, LPSTR, int);
> > --
> > 2.48.1
> >
> >
>
> Reviewed-by: Yan Vugenfirer <yvugenfi@redhat.com>
>
>
>