This series has started as a tooling for SVM AVIC series to allow
running AVIC tests [1] on a real target. The series depends on
the SVM AVIC series, see [2].
The series performs the overhaul of the existing XTF tooling to allow
XTF test suite orchestration on both QEMU and real hardware targets.
It also fixes up the XTF error code reporting and accounting of the
tests, e.g.:
```
Total : 2
CRASH : 0
ERROR : 0
SKIP : 0
FAIL : 0
UNKNOWN : 0
EOF : 0
PASS : 2
TIMEOUT : 0
```
QEMU XTF jobs updates:
- x86 XTF jobs now batch 75 XTFs each at once (~7m for the entire run)
XTF batching eliminates few existing jobs which simplifies the
existing test pipeline.
- Arm XTF job run 2 available tests at once (<1m)
Patch 1 is a housekeeping to avoid committing test artifacts after
local QEMU XTF runs. Turns out to be very convenient to run QEMU XTFs
locally.
Patches 2 and 3 is a non-functional cleanup meant to unify the CI jobs
definitions.
Patch 4 adds per-arch per-compiler XTF builds. This way the XTF test
jobs can skip re-building the XTF world and pull from the artifactory
instead.
Patch 5 updates console.exp to recognize known XTF-reported test
statuses for building fine-grain test reports.
Patch 5 adds binutils to docker containers used by XTF jobs. This is
needed for patch 6.
Patch 6 is a mega-overhaul of the existing qemu-xtf.sh. I kept it bash
for simplicity, but it makes sense to make another pass and switch
(finally) to pytest (see [3] and [4]). I deliberately decided not to
split it into smaller chunks and I __hope__ that would be OK for this
code to be reviewed in such a form, since that is tooling and there's a
public evidence of it being functional. But - sorry in advance, that is
a big change. The example output is here:
https://gitlab.com/xen-project/people/dmukhin/xen/-/jobs/17749986186
Patches 7-12 - enable qubes runners to be used for XTFs. All XTF jobs
runnable on a real hardware are made disabled by default. They can be
selected via CI variables or manually triggered (but still needs
elevated CI privileges). Here I want to thank Marek for providing Zen5
runner and the initial set of Zen5 jobs.
Patch 13 is meant to provide a way to execute the new interrupt-latency
test on AMD x86 runners. Shall not be merged.
[1] XTF branch: https://gitlab.com/xen-project/people/dmukhin/xtf/-/tree/x86_64-ipi-latency?ref_type=heads
[2] Xen AVIC + CI infra branch: https://gitlab.com/xen-project/people/dmukhin/xen/-/tree/svm-avic-v3?ref_type=heads
[3] https://lore.kernel.org/xen-devel/20260922100900.29129-1-anthony@xenproject.org/
[4] https://lore.kernel.org/xen-devel/1793715358.8631fc262581453bbf619ec5b2062170.1a1a1c44b87000403d@vates.tech/
[5] CI: https://gitlab.com/xen-project/people/dmukhin/xen/-/pipelines/3028917985
Denis Mukhin (12):
gitignore: exclude QEMU testing artifacts
CI: unify hypervisor console parametrization
CI: unify naming for hypervisor extra arguments
CI: enable XTF build jobs
CI: enable XTF status codes
CI: enable binutils for alpine and debian containers
CI: rework to allow hardware runners
CI: enable on-demand XTF on qubes runners
CI: enable on-demand IPI latency test on AMD Zen2/Zen3 runners
CI: x86: attempt to fixup flaky tests
CI: add on-demand XTF test jobs on AMD Zen5
CI: do not merge: re-wire XTF repos to custom branches
Marek Marczykowski-Górecki (1):
CI: add test jobs on AMD Zen5
.gitignore | 5 +
.../build/alpine/3.24-arm64v8.dockerfile | 3 +
.../build/alpine/3.24-x86_64.dockerfile | 3 +
automation/build/debian/12-arm64v8.dockerfile | 3 +
automation/build/debian/13-x86_64.dockerfile | 3 +
automation/gitlab-ci/build.yaml | 56 +++
automation/gitlab-ci/test.yaml | 289 ++++++++++---
automation/scripts/.gitignore | 1 +
automation/scripts/console.exp | 25 +-
.../scripts/include/configs/xtf-arm64-config | 2 -
.../scripts/include/configs/xtf-x86_64-config | 0
.../include/configs/xtf-x86_64-efi-config | 0
.../scripts/include/manifests/qemu-x86_64 | 75 ++++
automation/scripts/include/xtf-runner | 138 ------
.../include/{xtf-arm64 => xtf/qemu-arm64} | 13 +-
.../include/{xtf-x86_64 => xtf/qemu-x86_64} | 13 +-
.../{xtf-x86_64-efi => xtf/qemu-x86_64-efi} | 13 +-
automation/scripts/include/xtf/qubes-hw1 | 1 +
automation/scripts/include/xtf/qubes-hw11 | 1 +
automation/scripts/include/xtf/qubes-hw2 | 1 +
automation/scripts/include/xtf/qubes-hw21 | 1 +
automation/scripts/include/xtf/qubes-hw22 | 1 +
automation/scripts/include/xtf/qubes-hw3 | 1 +
automation/scripts/include/xtf/qubes-x86_64 | 54 +++
automation/scripts/qemu-xtf.sh | 28 --
automation/scripts/qubes-x86_64.sh | 4 +-
.../scripts/xilinx-smoke-dom0-x86_64.sh | 2 +-
automation/scripts/xtf | 395 ++++++++++++++++++
28 files changed, 864 insertions(+), 267 deletions(-)
delete mode 100644 automation/scripts/include/configs/xtf-arm64-config
delete mode 100644 automation/scripts/include/configs/xtf-x86_64-config
delete mode 100644 automation/scripts/include/configs/xtf-x86_64-efi-config
create mode 100644 automation/scripts/include/manifests/qemu-x86_64
delete mode 100644 automation/scripts/include/xtf-runner
rename automation/scripts/include/{xtf-arm64 => xtf/qemu-arm64} (79%)
rename automation/scripts/include/{xtf-x86_64 => xtf/qemu-x86_64} (53%)
rename automation/scripts/include/{xtf-x86_64-efi => xtf/qemu-x86_64-efi} (75%)
create mode 120000 automation/scripts/include/xtf/qubes-hw1
create mode 120000 automation/scripts/include/xtf/qubes-hw11
create mode 120000 automation/scripts/include/xtf/qubes-hw2
create mode 120000 automation/scripts/include/xtf/qubes-hw21
create mode 120000 automation/scripts/include/xtf/qubes-hw22
create mode 120000 automation/scripts/include/xtf/qubes-hw3
create mode 100644 automation/scripts/include/xtf/qubes-x86_64
delete mode 100755 automation/scripts/qemu-xtf.sh
create mode 100755 automation/scripts/xtf
--
2.55.0