[PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements

Thomas Huth posted 24 patches 3 years, 7 months ago
Test docker-quick@centos7 failed
Test docker-mingw@fedora failed
Test checkpatch failed
Test FreeBSD failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200916114731.102080-1-thuth@redhat.com
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, Fam Zheng <fam@euphon.net>, Max Reitz <mreitz@redhat.com>, Kevin Wolf <kwolf@redhat.com>, Stefan Weil <sw@weilnetz.de>, Laurent Vivier <lvivier@redhat.com>, Alexander Bulekov <alxndr@bu.edu>, Thomas Huth <thuth@redhat.com>, Bandan Das <bsd@redhat.com>, Stefan Hajnoczi <stefanha@redhat.com>
.cirrus.yml                    |  98 ++++++++++++++++++--------
.gitignore                     | 152 -----------------------------------------
include/qemu/osdep.h           |   2 +-
include/qemu/rcu.h             |   1 +
meson.build                    |   6 --
qga/commands-win32.c           |   5 +-
tests/meson.build              |  10 ++-
tests/qapi-schema/meson.build  |   2 +-
tests/qemu-iotests/check       |   2 +-
tests/qtest/fuzz/fuzz.c        |   5 +-
tests/qtest/qmp-cmd-test.c     |  10 +--
tests/socket-helpers.c         |   3 +-
tests/test-io-channel-file.c   |  12 +++-
tests/test-io-channel-socket.c |   2 +
tests/test-logging.c           |   5 +-
tests/test-qdev-global-props.c |   9 ++-
tests/test-replication.c       |  22 ++++--
tests/test-util-sockets.c      |   6 +-
tests/test-vmstate.c           |   3 +-
util/aio-win32.c               |  12 +++-
util/rcu.c                     |  55 +++++++++++++++
21 files changed, 207 insertions(+), 215 deletions(-)
[PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Thomas Huth 3 years, 7 months ago
 Hi Peter,

the following changes since commit de39a045bd8d2b49e4f3d07976622c29d58e0bac:

  Merge remote-tracking branch 'remotes/kraxel/tags/vga-20200915-pull-request' into staging (2020-09-15 14:25:05 +0100)

are available in the Git repository at:

  https://gitlab.com/huth/qemu.git tags/pull-request-2020-09-16

for you to fetch changes up to 114daec31d64600e5a070abcdb8de2cf74d01e94:

  cirrus: Building freebsd in a single shot (2020-09-16 12:15:07 +0200)

----------------------------------------------------------------
* Fix "readlink -f" problem in iotests on macOS (to fix the Cirrus-CI tests)
* Some minor qtest improvements
* Fix the unit tests to work on MSYS2, too
* Enable building and testing on MSYS2 in the Cirrus-CI
* Build FreeBSD with one task again in the Cirrus-CI
----------------------------------------------------------------

Alexander Bulekov (1):
      fuzz: Change the way we write qtest log to stderr

Max Reitz (1):
      iotests: Drop readlink -f

Maxim Levitsky (1):
      rcu: Implement drain_call_rcu

Thomas Huth (4):
      tests/qtest/qmp-cmd-test: Use inclusive language
      tests/socket-helpers: Treat EAI_NONAME as EADDRNOTAVAIL
      Simplify the .gitignore file
      qga/commands-win32: Fix problem with redundant protype declaration

Yonggang Luo (17):
      tests: Convert g_free to g_autofree macro in test-logging.c
      rcu: fixes test-logging.c by call drain_call_rcu before rmdir_full
      meson: Disable test-char on msys2/mingw for fixing tests stuck
      meson: disable crypto tests are empty under win32
      tests: Fixes test-replication.c on msys2/mingw.
      tests: test-replication disable /replication/secondary/* on msys2/mingw.
      osdep: file locking functions are not available on Win32
      meson: Use -b to ignore CR vs. CR-LF issues on Windows
      meson: remove empty else and duplicated gio deps
      vmstate: Fixes test-vmstate.c on msys2/mingw
      tests: Fixes test-io-channel-socket.c tests under msys2/mingw
      tests: fixes aio-win32 about aio_remove_fd_handler, get it consistence with aio-posix.c
      tests: Fixes test-io-channel-file by mask only owner file state mask bits
      tests: fix test-util-sockets.c
      tests: Fixes test-qdev-global-props.c
      ci: Enable msys2 ci in cirrus
      cirrus: Building freebsd in a single shot

 .cirrus.yml                    |  98 ++++++++++++++++++--------
 .gitignore                     | 152 -----------------------------------------
 include/qemu/osdep.h           |   2 +-
 include/qemu/rcu.h             |   1 +
 meson.build                    |   6 --
 qga/commands-win32.c           |   5 +-
 tests/meson.build              |  10 ++-
 tests/qapi-schema/meson.build  |   2 +-
 tests/qemu-iotests/check       |   2 +-
 tests/qtest/fuzz/fuzz.c        |   5 +-
 tests/qtest/qmp-cmd-test.c     |  10 +--
 tests/socket-helpers.c         |   3 +-
 tests/test-io-channel-file.c   |  12 +++-
 tests/test-io-channel-socket.c |   2 +
 tests/test-logging.c           |   5 +-
 tests/test-qdev-global-props.c |   9 ++-
 tests/test-replication.c       |  22 ++++--
 tests/test-util-sockets.c      |   6 +-
 tests/test-vmstate.c           |   3 +-
 util/aio-win32.c               |  12 +++-
 util/rcu.c                     |  55 +++++++++++++++
 21 files changed, 207 insertions(+), 215 deletions(-)


Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Peter Maydell 3 years, 7 months ago
On Wed, 16 Sep 2020 at 12:47, Thomas Huth <thuth@redhat.com> wrote:
> ----------------------------------------------------------------
> * Fix "readlink -f" problem in iotests on macOS (to fix the Cirrus-CI tests)
> * Some minor qtest improvements
> * Fix the unit tests to work on MSYS2, too
> * Enable building and testing on MSYS2 in the Cirrus-CI
> * Build FreeBSD with one task again in the Cirrus-CI
> ----------------------------------------------------------------

After merging this one I tried installing GNU sed on my local
box so it could run the iotests, but they fail like this
because they assume readarray, which doesn't exist in
OSX bash:

--- /Users/pm215/src/qemu-for-merges/tests/qemu-iotests/001.out
2015-09-29 16:54:34.000000000 +0100
+++ /Users/pm215/src/qemu-for-merges/build/all/tests/qemu-iotests/001.out.bad
  2020-09-17 15:03:22.000000000 +0100
@@ -1,5 +1,6 @@
 QA output created by 001
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
+./common.filter: line 142: readarray: command not found
+, fmt=

thanks
-- PMM

Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Thomas Huth 3 years, 7 months ago
On 17/09/2020 17.14, Peter Maydell wrote:
> On Wed, 16 Sep 2020 at 12:47, Thomas Huth <thuth@redhat.com> wrote:
>> ----------------------------------------------------------------
>> * Fix "readlink -f" problem in iotests on macOS (to fix the Cirrus-CI tests)
>> * Some minor qtest improvements
>> * Fix the unit tests to work on MSYS2, too
>> * Enable building and testing on MSYS2 in the Cirrus-CI
>> * Build FreeBSD with one task again in the Cirrus-CI
>> ----------------------------------------------------------------
> 
> After merging this one I tried installing GNU sed on my local
> box so it could run the iotests, but they fail like this
> because they assume readarray, which doesn't exist in
> OSX bash
Ok, sounds like I should respin my patch here:

https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg04476.html

... and could you install a newer version of bash via homebrew on your
system?

 Thomas


Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Peter Maydell 3 years, 7 months ago
On Thu, 17 Sep 2020 at 16:29, Thomas Huth <thuth@redhat.com> wrote:
> Ok, sounds like I should respin my patch here:
>
> https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg04476.html
>
> ... and could you install a newer version of bash via homebrew on your
> system?

I could in theory, but I don't really want to replace the system bash,
because this machine is not solely there to run the QEMU tests.

thanks
-- PMM

Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Thomas Huth 3 years, 7 months ago
On 17/09/2020 17.30, Peter Maydell wrote:
> On Thu, 17 Sep 2020 at 16:29, Thomas Huth <thuth@redhat.com> wrote:
>> Ok, sounds like I should respin my patch here:
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg04476.html
>>
>> ... and could you install a newer version of bash via homebrew on your
>> system?
> 
> I could in theory, but I don't really want to replace the system bash,
> because this machine is not solely there to run the QEMU tests.

Would it maybe be possible to install it in a different location, and
only use it for your merge tests?

 Thomas


Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by 罗勇刚 (Yonggang Luo) 3 years, 7 months ago
On Thu, Sep 17, 2020 at 11:29 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 17/09/2020 17.14, Peter Maydell wrote:
> > On Wed, 16 Sep 2020 at 12:47, Thomas Huth <thuth@redhat.com> wrote:
> >> ----------------------------------------------------------------
> >> * Fix "readlink -f" problem in iotests on macOS (to fix the Cirrus-CI
tests)
> >> * Some minor qtest improvements
> >> * Fix the unit tests to work on MSYS2, too
> >> * Enable building and testing on MSYS2 in the Cirrus-CI
> >> * Build FreeBSD with one task again in the Cirrus-CI
> >> ----------------------------------------------------------------
> >
> > After merging this one I tried installing GNU sed on my local
> > box so it could run the iotests, but they fail like this
> > because they assume readarray, which doesn't exist in
> > OSX bash
> Ok, sounds like I should respin my patch here:
>
> https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg04476.html
>
> ... and could you install a newer version of bash via homebrew on your
> system?
What's the reason to exclude these list on win32? That's didn't optimize
the time much, and indeed they can be
compiled.
--target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"

>
>  Thomas
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo
Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Thomas Huth 3 years, 7 months ago
On 17/09/2020 18.33, 罗勇刚(Yonggang Luo) wrote:
> 
> 
> On Thu, Sep 17, 2020 at 11:29 PM Thomas Huth <thuth@redhat.com
> <mailto:thuth@redhat.com>> wrote:
>>
>> On 17/09/2020 17.14, Peter Maydell wrote:
>> > On Wed, 16 Sep 2020 at 12:47, Thomas Huth <thuth@redhat.com
> <mailto:thuth@redhat.com>> wrote:
>> >> ----------------------------------------------------------------
>> >> * Fix "readlink -f" problem in iotests on macOS (to fix the
> Cirrus-CI tests)
>> >> * Some minor qtest improvements
>> >> * Fix the unit tests to work on MSYS2, too
>> >> * Enable building and testing on MSYS2 in the Cirrus-CI
>> >> * Build FreeBSD with one task again in the Cirrus-CI
>> >> ----------------------------------------------------------------
>> >
>> > After merging this one I tried installing GNU sed on my local
>> > box so it could run the iotests, but they fail like this
>> > because they assume readarray, which doesn't exist in
>> > OSX bash
>> Ok, sounds like I should respin my patch here:
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg04476.html
>>
>> ... and could you install a newer version of bash via homebrew on your
>> system?
> What's the reason to exclude these list on win32? That's didn't optimize
> the time much, and indeed they can be
> compiled.
> --target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"

The total run time of the job is already 50 minutes, so I was afraid
that it would hit the 1h time limit otherwise.
The targets that are excluded are just subsets of other targets, e.g.
i386-softmmu is a subset of x86_64-softmmu, so we don't lose much test
coverage by excluding them.

 Thomas


Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Peter Maydell 3 years, 7 months ago
On Wed, 16 Sep 2020 at 12:47, Thomas Huth <thuth@redhat.com> wrote:
>
>  Hi Peter,
>
> the following changes since commit de39a045bd8d2b49e4f3d07976622c29d58e0bac:
>
>   Merge remote-tracking branch 'remotes/kraxel/tags/vga-20200915-pull-request' into staging (2020-09-15 14:25:05 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/huth/qemu.git tags/pull-request-2020-09-16
>
> for you to fetch changes up to 114daec31d64600e5a070abcdb8de2cf74d01e94:
>
>   cirrus: Building freebsd in a single shot (2020-09-16 12:15:07 +0200)
>
> ----------------------------------------------------------------
> * Fix "readlink -f" problem in iotests on macOS (to fix the Cirrus-CI tests)
> * Some minor qtest improvements
> * Fix the unit tests to work on MSYS2, too
> * Enable building and testing on MSYS2 in the Cirrus-CI
> * Build FreeBSD with one task again in the Cirrus-CI
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.2
for any user-visible changes.

-- PMM

Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by 罗勇刚 (Yonggang Luo) 3 years, 7 months ago
On Thu, Sep 17, 2020 at 9:46 PM Peter Maydell <peter.maydell@linaro.org>
wrote:
>
> On Wed, 16 Sep 2020 at 12:47, Thomas Huth <thuth@redhat.com> wrote:
> >
> >  Hi Peter,
> >
> > the following changes since commit
de39a045bd8d2b49e4f3d07976622c29d58e0bac:
> >
> >   Merge remote-tracking branch
'remotes/kraxel/tags/vga-20200915-pull-request' into staging (2020-09-15
14:25:05 +0100)
> >
> > are available in the Git repository at:
> >
> >   https://gitlab.com/huth/qemu.git tags/pull-request-2020-09-16
> >
> > for you to fetch changes up to 114daec31d64600e5a070abcdb8de2cf74d01e94:
> >
> >   cirrus: Building freebsd in a single shot (2020-09-16 12:15:07 +0200)
> >
> > ----------------------------------------------------------------
> > * Fix "readlink -f" problem in iotests on macOS (to fix the Cirrus-CI
tests)
> > * Some minor qtest improvements
> > * Fix the unit tests to work on MSYS2, too
> > * Enable building and testing on MSYS2 in the Cirrus-CI
> > * Build FreeBSD with one task again in the Cirrus-CI
> > ----------------------------------------------------------------
>
>
> Applied, thanks.
>
> Please update the changelog at https://wiki.qemu.org/ChangeLog/5.2
> for any user-visible changes.
>
> -- PMM

cirrus are too slow:
```

pm215 Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2…
…
5e0a8fd2 hours ago

Git stats

 80,622 commits
--

after two hours still not finished.

Github have far more hardware resources. except doesn't have freebsd.
When I using github with wind32, it's running around 1hour with full
build(all target build).
That's because when trigger github ci, it's will allocating resouce
instantly

         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo
Re: [PULL 00/24] qtests, unit tests and Cirrus-CI fixes / improvements
Posted by Thomas Huth 3 years, 7 months ago
On 17/09/2020 16.40, 罗勇刚(Yonggang Luo) wrote:
> 
> 
> On Thu, Sep 17, 2020 at 9:46 PM Peter Maydell <peter.maydell@linaro.org
> <mailto:peter.maydell@linaro.org>> wrote:
>>
>> On Wed, 16 Sep 2020 at 12:47, Thomas Huth <thuth@redhat.com
> <mailto:thuth@redhat.com>> wrote:
>> >
>> >  Hi Peter,
>> >
>> > the following changes since commit
> de39a045bd8d2b49e4f3d07976622c29d58e0bac:
>> >
>> >   Merge remote-tracking branch
> 'remotes/kraxel/tags/vga-20200915-pull-request' into staging (2020-09-15
> 14:25:05 +0100)
>> >
>> > are available in the Git repository at:
>> >
>> >   https://gitlab.com/huth/qemu.git tags/pull-request-2020-09-16
>> >
>> > for you to fetch changes up to 114daec31d64600e5a070abcdb8de2cf74d01e94:
>> >
>> >   cirrus: Building freebsd in a single shot (2020-09-16 12:15:07 +0200)
>> >
>> > ----------------------------------------------------------------
>> > * Fix "readlink -f" problem in iotests on macOS (to fix the
> Cirrus-CI tests)
>> > * Some minor qtest improvements
>> > * Fix the unit tests to work on MSYS2, too
>> > * Enable building and testing on MSYS2 in the Cirrus-CI
>> > * Build FreeBSD with one task again in the Cirrus-CI
>> > ----------------------------------------------------------------
>>
>>
>> Applied, thanks.
>>
>> Please update the changelog at https://wiki.qemu.org/ChangeLog/5.2
>> for any user-visible changes.
>>
>> -- PMM
> 
> cirrus are too slow:
> ```
> 
> pm215 Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2…
> …
> 5e0a8fd2 hours ago
> 
> Git stats
> 
>  80,622 commits
> --
> 
> after two hours still not finished.

It now finished successfully:

 https://cirrus-ci.com/task/6178816496238592

Sometimes you just have to be a little bit patient :-)

 Thomas