[v6] tests/qtest: Enable running qtest on Windows

[PATCH v6 11/11] tests/qtest: Enable qtest build on Windows

Posted by Bin Meng 3 years, 3 months ago

Now that we have fixed various test case issues as seen when running
on Windows, let's enable the qtest build on Windows.

Signed-off-by: Bin Meng <bin.meng@windriver.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>

---

Changes in v5:
- Drop patches that are already merged

Changes in v3:
- Drop the host test

Changes in v2:
- new patch: "tests/qtest: Enable qtest build on Windows"

 tests/qtest/meson.build | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index c07a5b1a5f..f0ebb5fac6 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -1,9 +1,3 @@
-# All QTests for now are POSIX-only, but the dependencies are
-# really in libqtest, not in the testcases themselves.
-if not config_host.has_key('CONFIG_POSIX')
-  subdir_done()
-endif
-
 slow_qtests = {
   'ahci-test' : 60,
   'bios-tables-test' : 120,
-- 
2.25.1

Re: [PATCH v6 11/11] tests/qtest: Enable qtest build on Windows

Posted by Marc-André Lureau 3 years, 2 months ago

Hi Bin

On Fri, Oct 28, 2022 at 9:06 AM Bin Meng <bin.meng@windriver.com> wrote:
>
> Now that we have fixed various test case issues as seen when running
> on Windows, let's enable the qtest build on Windows.
>
> Signed-off-by: Bin Meng <bin.meng@windriver.com>
> Reviewed-by: Thomas Huth <thuth@redhat.com>

We haven't solved the CI timing out or eating all the CPU time, right?

Can we simply exclude it from CI for now, ie add to this patch

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 093276ddbc..ba9045ec38 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -62,7 +62,7 @@ msys2-64bit:
   - .\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
       --enable-capstone'
   - .\msys64\usr\bin\bash -lc 'make'
-  - .\msys64\usr\bin\bash -lc 'make check || { cat
build/meson-logs/testlog.txt; exit 1; } ;'
+  - .\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
qtest" || { cat build/meson-logs/testlog.txt; exit 1; } ;'

 msys2-32bit:
   extends: .shared_msys2_builder
@@ -96,4 +96,4 @@ msys2-32bit:
   - cd output
   - ..\msys64\usr\bin\bash -lc "../configure --target-list=ppc64-softmmu"
   - ..\msys64\usr\bin\bash -lc 'make'
-  - ..\msys64\usr\bin\bash -lc 'make check || { cat
meson-logs/testlog.txt; exit 1; } ;'
+  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
qtest" || { cat meson-logs/testlog.txt; exit 1; } ;'


Could you resubmit your missing win test patches and check if gitlab is happy?

thanks

>
> ---
>
> Changes in v5:
> - Drop patches that are already merged
>
> Changes in v3:
> - Drop the host test
>
> Changes in v2:
> - new patch: "tests/qtest: Enable qtest build on Windows"
>
>  tests/qtest/meson.build | 6 ------
>  1 file changed, 6 deletions(-)
>
> diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
> index c07a5b1a5f..f0ebb5fac6 100644
> --- a/tests/qtest/meson.build
> +++ b/tests/qtest/meson.build
> @@ -1,9 +1,3 @@
> -# All QTests for now are POSIX-only, but the dependencies are
> -# really in libqtest, not in the testcases themselves.
> -if not config_host.has_key('CONFIG_POSIX')
> -  subdir_done()
> -endif
> -
>  slow_qtests = {
>    'ahci-test' : 60,
>    'bios-tables-test' : 120,
> --
> 2.25.1
>
>


-- 
Marc-André Lureau

Re: [PATCH v6 11/11] tests/qtest: Enable qtest build on Windows

Posted by Bin Meng 3 years, 2 months ago

Hi Marc-André,

On Wed, Nov 23, 2022 at 10:14 PM Marc-André Lureau
<marcandre.lureau@gmail.com> wrote:
>
> Hi Bin
>
> On Fri, Oct 28, 2022 at 9:06 AM Bin Meng <bin.meng@windriver.com> wrote:
> >
> > Now that we have fixed various test case issues as seen when running
> > on Windows, let's enable the qtest build on Windows.
> >
> > Signed-off-by: Bin Meng <bin.meng@windriver.com>
> > Reviewed-by: Thomas Huth <thuth@redhat.com>
>
> We haven't solved the CI timing out or eating all the CPU time, right?

Correct

>
> Can we simply exclude it from CI for now, ie add to this patch

Thanks for your suggestion!

>
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 093276ddbc..ba9045ec38 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -62,7 +62,7 @@ msys2-64bit:
>    - .\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
>        --enable-capstone'
>    - .\msys64\usr\bin\bash -lc 'make'
> -  - .\msys64\usr\bin\bash -lc 'make check || { cat
> build/meson-logs/testlog.txt; exit 1; } ;'
> +  - .\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> qtest" || { cat build/meson-logs/testlog.txt; exit 1; } ;'

The double quote should be escaped.

>
>  msys2-32bit:
>    extends: .shared_msys2_builder
> @@ -96,4 +96,4 @@ msys2-32bit:
>    - cd output
>    - ..\msys64\usr\bin\bash -lc "../configure --target-list=ppc64-softmmu"
>    - ..\msys64\usr\bin\bash -lc 'make'
> -  - ..\msys64\usr\bin\bash -lc 'make check || { cat
> meson-logs/testlog.txt; exit 1; } ;'
> +  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> qtest" || { cat meson-logs/testlog.txt; exit 1; } ;'

I don't think we need to touch 32-bit.

> Could you resubmit your missing win test patches and check if gitlab is happy?
>

Sure, will do soon.

Regards,
Bin

Re: [PATCH v6 11/11] tests/qtest: Enable qtest build on Windows

Posted by Thomas Huth 3 years, 2 months ago

On 23/11/2022 15.13, Marc-André Lureau wrote:
> Hi Bin
> 
> On Fri, Oct 28, 2022 at 9:06 AM Bin Meng <bin.meng@windriver.com> wrote:
>>
>> Now that we have fixed various test case issues as seen when running
>> on Windows, let's enable the qtest build on Windows.
>>
>> Signed-off-by: Bin Meng <bin.meng@windriver.com>
>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 
> We haven't solved the CI timing out or eating all the CPU time, right?
> 
> Can we simply exclude it from CI for now, ie add to this patch
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 093276ddbc..ba9045ec38 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -62,7 +62,7 @@ msys2-64bit:
>     - .\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
>         --enable-capstone'
>     - .\msys64\usr\bin\bash -lc 'make'
> -  - .\msys64\usr\bin\bash -lc 'make check || { cat
> build/meson-logs/testlog.txt; exit 1; } ;'
> +  - .\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> qtest" || { cat build/meson-logs/testlog.txt; exit 1; } ;'
> 
>   msys2-32bit:
>     extends: .shared_msys2_builder
> @@ -96,4 +96,4 @@ msys2-32bit:
>     - cd output
>     - ..\msys64\usr\bin\bash -lc "../configure --target-list=ppc64-softmmu"
>     - ..\msys64\usr\bin\bash -lc 'make'
> -  - ..\msys64\usr\bin\bash -lc 'make check || { cat
> meson-logs/testlog.txt; exit 1; } ;'
> +  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> qtest" || { cat meson-logs/testlog.txt; exit 1; } ;'

I think it's only the 64-bit job that is really problematic, so we could 
still run the qtests in the 32-bit job?

Alternatively, what about switching the 64-bit to another target that does 
not have so many qtests enabled? Some mips-softmmu or riscv-softmmu maybe? 
... we still check x86_64-softmmu in the .cirrus.yml builds, so this is 
hopefully not such a big loss...

  Thomas

Re: [PATCH v6 11/11] tests/qtest: Enable qtest build on Windows

Posted by Marc-André Lureau 3 years, 2 months ago

Hi Thomas

On Wed, Nov 23, 2022 at 6:19 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 23/11/2022 15.13, Marc-André Lureau wrote:
> > Hi Bin
> >
> > On Fri, Oct 28, 2022 at 9:06 AM Bin Meng <bin.meng@windriver.com> wrote:
> >>
> >> Now that we have fixed various test case issues as seen when running
> >> on Windows, let's enable the qtest build on Windows.
> >>
> >> Signed-off-by: Bin Meng <bin.meng@windriver.com>
> >> Reviewed-by: Thomas Huth <thuth@redhat.com>
> >
> > We haven't solved the CI timing out or eating all the CPU time, right?
> >
> > Can we simply exclude it from CI for now, ie add to this patch
> >
> > diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> > index 093276ddbc..ba9045ec38 100644
> > --- a/.gitlab-ci.d/windows.yml
> > +++ b/.gitlab-ci.d/windows.yml
> > @@ -62,7 +62,7 @@ msys2-64bit:
> >     - .\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
> >         --enable-capstone'
> >     - .\msys64\usr\bin\bash -lc 'make'
> > -  - .\msys64\usr\bin\bash -lc 'make check || { cat
> > build/meson-logs/testlog.txt; exit 1; } ;'
> > +  - .\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> > qtest" || { cat build/meson-logs/testlog.txt; exit 1; } ;'
> >
> >   msys2-32bit:
> >     extends: .shared_msys2_builder
> > @@ -96,4 +96,4 @@ msys2-32bit:
> >     - cd output
> >     - ..\msys64\usr\bin\bash -lc "../configure --target-list=ppc64-softmmu"
> >     - ..\msys64\usr\bin\bash -lc 'make'
> > -  - ..\msys64\usr\bin\bash -lc 'make check || { cat
> > meson-logs/testlog.txt; exit 1; } ;'
> > +  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> > qtest" || { cat meson-logs/testlog.txt; exit 1; } ;'
>
> I think it's only the 64-bit job that is really problematic, so we could
> still run the qtests in the 32-bit job?
>
> Alternatively, what about switching the 64-bit to another target that does
> not have so many qtests enabled? Some mips-softmmu or riscv-softmmu maybe?
> ... we still check x86_64-softmmu in the .cirrus.yml builds, so this is
> hopefully not such a big loss...
>

The change I propose above is to simply skip the qtests on msys CI
builds. They are not running today on !POSIX.


-- 
Marc-André Lureau

Re: [PATCH v6 11/11] tests/qtest: Enable qtest build on Windows

Posted by Thomas Huth 3 years, 2 months ago

On 24/11/2022 12.17, Marc-André Lureau wrote:
> Hi Thomas
> 
> On Wed, Nov 23, 2022 at 6:19 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 23/11/2022 15.13, Marc-André Lureau wrote:
>>> Hi Bin
>>>
>>> On Fri, Oct 28, 2022 at 9:06 AM Bin Meng <bin.meng@windriver.com> wrote:
>>>>
>>>> Now that we have fixed various test case issues as seen when running
>>>> on Windows, let's enable the qtest build on Windows.
>>>>
>>>> Signed-off-by: Bin Meng <bin.meng@windriver.com>
>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>>
>>> We haven't solved the CI timing out or eating all the CPU time, right?
>>>
>>> Can we simply exclude it from CI for now, ie add to this patch
>>>
>>> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
>>> index 093276ddbc..ba9045ec38 100644
>>> --- a/.gitlab-ci.d/windows.yml
>>> +++ b/.gitlab-ci.d/windows.yml
>>> @@ -62,7 +62,7 @@ msys2-64bit:
>>>      - .\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
>>>          --enable-capstone'
>>>      - .\msys64\usr\bin\bash -lc 'make'
>>> -  - .\msys64\usr\bin\bash -lc 'make check || { cat
>>> build/meson-logs/testlog.txt; exit 1; } ;'
>>> +  - .\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
>>> qtest" || { cat build/meson-logs/testlog.txt; exit 1; } ;'
>>>
>>>    msys2-32bit:
>>>      extends: .shared_msys2_builder
>>> @@ -96,4 +96,4 @@ msys2-32bit:
>>>      - cd output
>>>      - ..\msys64\usr\bin\bash -lc "../configure --target-list=ppc64-softmmu"
>>>      - ..\msys64\usr\bin\bash -lc 'make'
>>> -  - ..\msys64\usr\bin\bash -lc 'make check || { cat
>>> meson-logs/testlog.txt; exit 1; } ;'
>>> +  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
>>> qtest" || { cat meson-logs/testlog.txt; exit 1; } ;'
>>
>> I think it's only the 64-bit job that is really problematic, so we could
>> still run the qtests in the 32-bit job?
>>
>> Alternatively, what about switching the 64-bit to another target that does
>> not have so many qtests enabled? Some mips-softmmu or riscv-softmmu maybe?
>> ... we still check x86_64-softmmu in the .cirrus.yml builds, so this is
>> hopefully not such a big loss...
>>
> 
> The change I propose above is to simply skip the qtests on msys CI
> builds. They are not running today on !POSIX.

Sure ... question is what would be more valuable in the gitlab-CI ... only 
compile-testing x86_64-softmmu on msys2 (since that also compile-tests the 
HAX and WHPX stuff), but without qtests, or also run a (limited) set of 
qtests with another smaller softmmu target?
I don't mind either way, I just wanted to suggest an alternative to consider.

  Thomas

Re: [PATCH v6 11/11] tests/qtest: Enable qtest build on Windows

Posted by Marc-André Lureau 3 years, 2 months ago

Hi

On Thu, Nov 24, 2022 at 3:49 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 24/11/2022 12.17, Marc-André Lureau wrote:
> > Hi Thomas
> >
> > On Wed, Nov 23, 2022 at 6:19 PM Thomas Huth <thuth@redhat.com> wrote:
> >>
> >> On 23/11/2022 15.13, Marc-André Lureau wrote:
> >>> Hi Bin
> >>>
> >>> On Fri, Oct 28, 2022 at 9:06 AM Bin Meng <bin.meng@windriver.com> wrote:
> >>>>
> >>>> Now that we have fixed various test case issues as seen when running
> >>>> on Windows, let's enable the qtest build on Windows.
> >>>>
> >>>> Signed-off-by: Bin Meng <bin.meng@windriver.com>
> >>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> >>>
> >>> We haven't solved the CI timing out or eating all the CPU time, right?
> >>>
> >>> Can we simply exclude it from CI for now, ie add to this patch
> >>>
> >>> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> >>> index 093276ddbc..ba9045ec38 100644
> >>> --- a/.gitlab-ci.d/windows.yml
> >>> +++ b/.gitlab-ci.d/windows.yml
> >>> @@ -62,7 +62,7 @@ msys2-64bit:
> >>>      - .\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
> >>>          --enable-capstone'
> >>>      - .\msys64\usr\bin\bash -lc 'make'
> >>> -  - .\msys64\usr\bin\bash -lc 'make check || { cat
> >>> build/meson-logs/testlog.txt; exit 1; } ;'
> >>> +  - .\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> >>> qtest" || { cat build/meson-logs/testlog.txt; exit 1; } ;'
> >>>
> >>>    msys2-32bit:
> >>>      extends: .shared_msys2_builder
> >>> @@ -96,4 +96,4 @@ msys2-32bit:
> >>>      - cd output
> >>>      - ..\msys64\usr\bin\bash -lc "../configure --target-list=ppc64-softmmu"
> >>>      - ..\msys64\usr\bin\bash -lc 'make'
> >>> -  - ..\msys64\usr\bin\bash -lc 'make check || { cat
> >>> meson-logs/testlog.txt; exit 1; } ;'
> >>> +  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS="--no-suite
> >>> qtest" || { cat meson-logs/testlog.txt; exit 1; } ;'
> >>
> >> I think it's only the 64-bit job that is really problematic, so we could
> >> still run the qtests in the 32-bit job?
> >>
> >> Alternatively, what about switching the 64-bit to another target that does
> >> not have so many qtests enabled? Some mips-softmmu or riscv-softmmu maybe?
> >> ... we still check x86_64-softmmu in the .cirrus.yml builds, so this is
> >> hopefully not such a big loss...
> >>
> >
> > The change I propose above is to simply skip the qtests on msys CI
> > builds. They are not running today on !POSIX.
>
> Sure ... question is what would be more valuable in the gitlab-CI ... only
> compile-testing x86_64-softmmu on msys2 (since that also compile-tests the
> HAX and WHPX stuff), but without qtests, or also run a (limited) set of
> qtests with another smaller softmmu target?
> I don't mind either way, I just wanted to suggest an alternative to consider.
>

Yes, we can do that on top though. I would want qtest to run on my
local msys build first, hence Bin's patch :)


-- 
Marc-André Lureau