automation/gitlab-ci/build.yaml | 1 + automation/scripts/build | 7 ++----- automation/scripts/qemu-deps-check.py | 19 ------------------- 3 files changed, 3 insertions(+), 24 deletions(-) delete mode 100755 automation/scripts/qemu-deps-check.py
We spend an awful lot of CI time building QEMU, even though most changes don't
touch the subset of tools/libs/ used by QEMU. Some numbers taken at a time
when CI was otherwise quiet:
With Without
Alpine: 13m38s 6m04s
Debian 12: 10m05s 8m10s
OpenSUSE Tumbleweed: 11m40s 7m54s
Ubuntu 24.04: 14m56s 8m06s
which is a >50% improvement in wallclock time in some cases.
The only build we have that needs QEMU is alpine-3.18-gcc-debug. This is the
build deployed and used by the QubesOS ADL-* and Zen3p-* jobs.
Xilinx-x86_64 deploys it too, but is PVH-only and doesn't use QEMU.
QEMU is also built by CirrusCI for FreeBSD (fully Clang/LLVM toolchain).
This should help quite a lot with Gitlab CI capacity.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
I did consider putting in a build of QEMU on arm64 for good measure, but I
can't find any build job which currently does; none of the arm64 containers
have ninja by the looks of things.
Run with this patch in place:
https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/1486822209
---
automation/gitlab-ci/build.yaml | 1 +
automation/scripts/build | 7 ++-----
automation/scripts/qemu-deps-check.py | 19 -------------------
3 files changed, 3 insertions(+), 24 deletions(-)
delete mode 100755 automation/scripts/qemu-deps-check.py
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index c668736bdc2f..c83e0bdbe119 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -339,6 +339,7 @@ alpine-3.18-gcc-debug:
extends: .gcc-x86-64-build-debug
variables:
CONTAINER: alpine:3.18
+ QEMU: y
debian-bookworm-gcc-debug:
extends: .gcc-x86-64-build-debug
diff --git a/automation/scripts/build b/automation/scripts/build
index 34416297a4b7..926cf5b46d17 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -91,11 +91,8 @@ else
cfgargs+=("--with-extra-qemuu-configure-args=\"--disable-werror\"")
fi
- # Qemu requires Python 3.8 or later, and ninja
- # and Clang 10 or later
- if ! type python3 || ! python3 automation/scripts/qemu-deps-check.py \
- || [[ "$cc_is_clang" == y && "$cc_ver" -lt 0x0a0000 ]] \
- || ! type ninja; then
+ # QEMU is only for those who ask
+ if [[ "$QEMU" != "y" ]]; then
cfgargs+=("--with-system-qemu=/bin/false")
fi
diff --git a/automation/scripts/qemu-deps-check.py b/automation/scripts/qemu-deps-check.py
deleted file mode 100755
index f6188afb3e2a..000000000000
--- a/automation/scripts/qemu-deps-check.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-import sys
-
-if sys.version_info < (3, 8):
- print("Python %d.%d.%d too old" %
- (sys.version_info.major,
- sys.version_info.minor,
- sys.version_info.micro))
- exit(1)
-
-try:
- import tomllib
-except ImportError:
- try:
- import tomli
- except ImportError:
- print("No tomli")
- exit(1)
--
2.39.5
On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > index c668736bdc2f..c83e0bdbe119 100644 > --- a/automation/gitlab-ci/build.yaml > +++ b/automation/gitlab-ci/build.yaml > @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: > extends: .gcc-x86-64-build-debug > variables: > CONTAINER: alpine:3.18 > + QEMU: y Could you use a different name for the variable? This is exposed as an environment variable, it could easily be used in a build system already, like used to store a path to a QEMU to use. We don't really have a name space for CI variable, but maybe BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other usages. Otherwise patch looks fine. Cheers, -- Anthony Perard | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
On 08/10/2024 5:27 pm, Anthony PERARD wrote: > On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: >> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml >> index c668736bdc2f..c83e0bdbe119 100644 >> --- a/automation/gitlab-ci/build.yaml >> +++ b/automation/gitlab-ci/build.yaml >> @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: >> extends: .gcc-x86-64-build-debug >> variables: >> CONTAINER: alpine:3.18 >> + QEMU: y > Could you use a different name for the variable? This is exposed as an > environment variable, it could easily be used in a build system already, > like used to store a path to a QEMU to use. > > We don't really have a name space for CI variable, but maybe > BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other > usages. > > Otherwise patch looks fine. A can switch to BUILD_QEMU, but I don't plan to repost just for that. ~Andrew
On Tue, 8 Oct 2024, Andrew Cooper wrote: > On 08/10/2024 5:27 pm, Anthony PERARD wrote: > > On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: > >> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > >> index c668736bdc2f..c83e0bdbe119 100644 > >> --- a/automation/gitlab-ci/build.yaml > >> +++ b/automation/gitlab-ci/build.yaml > >> @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: > >> extends: .gcc-x86-64-build-debug > >> variables: > >> CONTAINER: alpine:3.18 > >> + QEMU: y > > Could you use a different name for the variable? This is exposed as an > > environment variable, it could easily be used in a build system already, > > like used to store a path to a QEMU to use. > > > > We don't really have a name space for CI variable, but maybe > > BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other > > usages. > > > > Otherwise patch looks fine. > > A can switch to BUILD_QEMU, but I don't plan to repost just for that. The patch is fine. BUILD_QEMU is a bit better. BUILD_QEMU_XEN or BUILD_QEMU_DM would be even better to clarify we are building a QEMU version to serve as device model. It is a pity that typically we call qemu-dm the ancient QEMU fork because "qemu device model" is more meaningful than "qemu xen". Anyway, pick your preference and feel free to change the variable name on commit. Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
On 08/10/2024 7:54 pm, Stefano Stabellini wrote: > On Tue, 8 Oct 2024, Andrew Cooper wrote: >> On 08/10/2024 5:27 pm, Anthony PERARD wrote: >>> On Tue, Oct 08, 2024 at 04:50:23PM +0100, Andrew Cooper wrote: >>>> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml >>>> index c668736bdc2f..c83e0bdbe119 100644 >>>> --- a/automation/gitlab-ci/build.yaml >>>> +++ b/automation/gitlab-ci/build.yaml >>>> @@ -339,6 +339,7 @@ alpine-3.18-gcc-debug: >>>> extends: .gcc-x86-64-build-debug >>>> variables: >>>> CONTAINER: alpine:3.18 >>>> + QEMU: y >>> Could you use a different name for the variable? This is exposed as an >>> environment variable, it could easily be used in a build system already, >>> like used to store a path to a QEMU to use. >>> >>> We don't really have a name space for CI variable, but maybe >>> BUILD_QEMU or BUILD_QEMU_XEN would be less likely to clash with other >>> usages. >>> >>> Otherwise patch looks fine. >> A can switch to BUILD_QEMU, but I don't plan to repost just for that. > The patch is fine. > > BUILD_QEMU is a bit better. BUILD_QEMU_XEN or BUILD_QEMU_DM would be > even better to clarify we are building a QEMU version to serve as device > model. It is a pity that typically we call qemu-dm the ancient QEMU > fork because "qemu device model" is more meaningful than "qemu xen". > > Anyway, pick your preference and feel free to change the variable name > on commit. > > Reviewed-by: Stefano Stabellini <sstabellini@kernel.org> Thanks. I think I'll go with BUILD_QEMU_XEN because that what the rest of Xen's build system calls the thing being discussed here. ~Andrew
© 2016 - 2024 Red Hat, Inc.