This is an alternative and/or complementary to Thomas' proposal
to use CLang with msys:

  https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html

First of all, the current msys installer we're using is over 12
months out of date. Thus after running the install, pacman then
replaces most of what we've just installed with new downloaded
content. Using the most update installer cuts 3+1/2 minutes off
the msys install time - 7 minutes becomes 3+1/2.

Secondly, QEMU defaults to compiling with -O2 and this is more
computationally expensive for GCC. Switching to -O0 drops the
build time from 60 minutes down to 45 minutes.

Thirdly, including debug symbols also has an overhead, and turning
that off reduces time still further down to 38 minutes.

IOW, between all three changes, we can cut approx 25-26 minutes
off the job execution time, bringing it nicely within the job
timeout.

The actually phase of installing the mingw deps still accounts
for about 10 minutes and has not been optimized.

Possibly the same trick of -O0 and skipping -g would also help
the clang alternative Thomas' proposed. If so, that could be
enough to let us enable more features / targets during the
msys build.

Daniel P. Berrangé (8):
  gitlab: remove duplication between msys jobs
  gitlab: print timestamps during windows msys jobs
  gitlab: always use updated msys installer
  gitlab: drop $CI_PROJECT_DIR from cache path
  gitlab: always populate cache for windows msys jobs
  configure: support passthrough of -Dxxx args to meson
  gitlab: disable optimization and debug symbols in msys build
  gitlab: disable FF_SCRIPT_SECTIONS on msys jobs

 .gitlab-ci.d/windows.yml | 174 +++++++++++++++++++--------------------
 configure                |   5 ++
 2 files changed, 92 insertions(+), 87 deletions(-)

-- 
2.41.0

Daniel P. Berrangé <berrange@redhat.com> writes:

> This is an alternative and/or complementary to Thomas' proposal
> to use CLang with msys:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
>
> First of all, the current msys installer we're using is over 12
> months out of date. Thus after running the install, pacman then
> replaces most of what we've just installed with new downloaded
> content. Using the most update installer cuts 3+1/2 minutes off
> the msys install time - 7 minutes becomes 3+1/2.
>
> Secondly, QEMU defaults to compiling with -O2 and this is more
> computationally expensive for GCC. Switching to -O0 drops the
> build time from 60 minutes down to 45 minutes.

From the fine manual[*]: "The effectiveness of some warnings depends on
optimizations also being enabled.  For example '-Wsuggest-final-types'
is more effective with link-time optimization and some instances of
other warnings may not be issued at all unless optimization is enabled.
While optimization in general improves the efficacy of control and data
flow sensitive warnings, in some cases it may also cause false
positives."  Do we care?

[...]


[*] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Warning-Options.html

On Tue, Aug 01, 2023 at 03:53:22PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > This is an alternative and/or complementary to Thomas' proposal
> > to use CLang with msys:
> >
> >   https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
> >
> > First of all, the current msys installer we're using is over 12
> > months out of date. Thus after running the install, pacman then
> > replaces most of what we've just installed with new downloaded
> > content. Using the most update installer cuts 3+1/2 minutes off
> > the msys install time - 7 minutes becomes 3+1/2.
> >
> > Secondly, QEMU defaults to compiling with -O2 and this is more
> > computationally expensive for GCC. Switching to -O0 drops the
> > build time from 60 minutes down to 45 minutes.
> 
> From the fine manual[*]: "The effectiveness of some warnings depends on
> optimizations also being enabled.  For example '-Wsuggest-final-types'
> is more effective with link-time optimization and some instances of
> other warnings may not be issued at all unless optimization is enabled.
> While optimization in general improves the efficacy of control and data
> flow sensitive warnings, in some cases it may also cause false
> positives."  Do we care?

In general, yes, we do care.

In this specific case though, we're battling to figure out the lesser
of multiple evils.

Right now we configure with:

  --target-list=x86_64-softmmu --without-default-devices 

and so with optimization enabled, we'll get good warning coverage of
a small amount of code, except we don't because people started
ignoring the msys jobs as they timeout too frequently.

If we can use Thomas' clang switch or my -O0 patches, we can get
within the timeout, so people can trust the job once again. If we
can do both ideas and cut the time even more, then we can enable
more features (perhaps drop --without-default-devices).

So the warnings might not be quite as good, but we'll have the
warnings across a larger amount of code.

Alot of the warnings from the Linux/macOS builds will also apply
in the Windows builds. I think on balance I'd probably prefer us
to build a larger amount of code for Windows. This is in context
of free shared runners at least.

As a more drastic option, we might need to consider using the
Azure credits for Windows  runners too. If we could have bigger
VMs for Windows CI, we can build more and have better warnings
at the same time. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 01/08/2023 16.35, Daniel P. Berrangé wrote:
> On Tue, Aug 01, 2023 at 03:53:22PM +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>>> This is an alternative and/or complementary to Thomas' proposal
>>> to use CLang with msys:
>>>
>>>    https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
>>>
>>> First of all, the current msys installer we're using is over 12
>>> months out of date. Thus after running the install, pacman then
>>> replaces most of what we've just installed with new downloaded
>>> content. Using the most update installer cuts 3+1/2 minutes off
>>> the msys install time - 7 minutes becomes 3+1/2.
>>>
>>> Secondly, QEMU defaults to compiling with -O2 and this is more
>>> computationally expensive for GCC. Switching to -O0 drops the
>>> build time from 60 minutes down to 45 minutes.
>>
>>  From the fine manual[*]: "The effectiveness of some warnings depends on
>> optimizations also being enabled.  For example '-Wsuggest-final-types'
>> is more effective with link-time optimization and some instances of
>> other warnings may not be issued at all unless optimization is enabled.
>> While optimization in general improves the efficacy of control and data
>> flow sensitive warnings, in some cases it may also cause false
>> positives."  Do we care?
> 
> In general, yes, we do care.
> 
> In this specific case though, we're battling to figure out the lesser
> of multiple evils.

I agree. Additionally, we also test compiling for Windows with the MinGW 
cross compiler suite in a Fedora container, and we still use the default 
optimization there, so we should have that covered.

  Thomas

On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> This is an alternative and/or complementary to Thomas' proposal
> to use CLang with msys:
> 
>    https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
> 
> First of all, the current msys installer we're using is over 12
> months out of date. Thus after running the install, pacman then
> replaces most of what we've just installed with new downloaded
> content. Using the most update installer cuts 3+1/2 minutes off
> the msys install time - 7 minutes becomes 3+1/2.
> 
> Secondly, QEMU defaults to compiling with -O2 and this is more
> computationally expensive for GCC. Switching to -O0 drops the
> build time from 60 minutes down to 45 minutes.
> 
> Thirdly, including debug symbols also has an overhead, and turning
> that off reduces time still further down to 38 minutes.
> 
> IOW, between all three changes, we can cut approx 25-26 minutes
> off the job execution time, bringing it nicely within the job
> timeout.
> 
> The actually phase of installing the mingw deps still accounts
> for about 10 minutes and has not been optimized.
> 
> Possibly the same trick of -O0 and skipping -g would also help
> the clang alternative Thomas' proposed. If so, that could be
> enough to let us enable more features / targets during the
> msys build.

I really like the idea! And I guess my idea with Clang needs some more work 
'til it is acceptable, so let's go with your idea for now to fix the timeout 
problem in the CI ... we can still optimize later with Clang in case we 
found a good solution for that ms_struct problem...

  Thomas