Xen is being made safety certifiable to IEC 61508 SIL 3 and ISO 26262 ASIL
D, with Arm and x86 as the current targets [1]. Evidence at those levels is
requirements based testing plus structural coverage, produced automatically
and repeatably in CI.
QTB (QEMU Test Bench) [2] is the framework AMD wrote for it as part of the
Xen safety initiative. It drives a live QEMU instance over qtest, QMP and
GDB, so a test has full access to the machine while emulation runs: read
and write the consoles, inspect registers and memory, inject interrupts and
faults, all from Python and all reproducible in a pipeline. QTB is not
upstream in QEMU yet, but is planned to be.
RISC-V is not in the certification scope today. It is also the youngest
port, so it has close to no test infrastructure to undo, which makes it the
cheapest place to adopt QTB. Starting the riscv64 tests on the framework
now means the port grows its tests in the shape certification asks for as
the port itself grows, instead of a pile of expect scripts to convert later
if riscv64 ever becomes a certification target. It also puts a second
architecture on QTB, which is useful to the framework itself before it is
proposed to QEMU upstream.
Concretely, the riscv64 CI coverage today is one expect script,
automation/scripts/qemu-smoke-riscv64.sh, which boots Xen alone under QEMU
and greps a single string out of one serial console. The machine it boots
is hardcoded, so a second configuration means a second script, and a test
with a DomU in it means growing guest handling from scratch.
This series replaces that script with a data-driven framework using QTB. A
machine is a YAML entry (pcpus, scheduler, interrupt controller, boot
arguments), a test type is a small Python class saying what to do with a
booted machine, and a test binds a machine to that type's expectations.
Adding a configuration to CI is then a config change and a job stanza.
Patch 1 goes to test-artifacts [3] and must land first, since patches 2-6
run inside the container it adds. Patches 2-6 go to xen.git.
qemu-smoke-riscv64-gcc also pulls the riscv64 QEMU and OpenSBI binaries
from the qemu-9.0.0-riscv64 job in test-artifacts, which is not in
test-artifacts master yet and has to land there too.
test-artifacts [3]:
- Add a QTB container to run the Xen riscv64 tests
debian:13-qtb-riscv64, carrying qemu.qtb from AMD's QEMU fork and the
dtc/fdt dependencies the framework needs at run time.
xen.git:
- automation/qtb: add jinja2 device trees for riscv64 smoke tests
The host tree varies with hart count, MMU type and device set, so it is
rendered per machine from a template rather than shipping one static
.dts per configuration.
- automation/qtb: add Python QTB framework with the console-test type
The base layer every test type builds on (machine catalog lookup, host
DT generation, QEMU invocation, per-console log capture) plus the first
test type, which asserts every string listed in console-test.yaml is
printed on the expected console.
- automation/qtb: add unit tests for the QTB framework
pytest coverage of the QEMU-agnostic parts, for developers only.
- automation/qtb: add QTB framework README
How to add a machine, add a test type and run one locally.
- CI: run the riscv64 smoke test via QTB framework console-test
Repoints qemu-smoke-riscv64-gcc at the framework and drops
qemu-smoke-riscv64.sh, which then has no caller left.
Testing: QEMU only, as the existing riscv64 CI does.
qemu-smoke-riscv64-gcc runs the same check as before.
The catalog ships a single Xen-only machine, since Xen cannot boot a DomU
on RISC-V yet. DomU machines plus an irq-test type using QTest interrupt
injection follow once DomU support lands.
[1] https://elisa.tech/blog/2026/07/22/the-final-phase-of-xen-safety-solving-coverage-and-residual-gaps-stefano-stabellini-amd/
[2] https://gitlab.com/xen-project/people/amd/qemu/-/tree/safety
[3] https://gitlab.com/xen-project/hardware/test-artifacts
CI test:
https://gitlab.com/xen-project/people/baptleduc/xen/-/pipelines/2740836946
Baptiste Le Duc (5):
automation/qtb: add jinja2 device trees for riscv64 smoke tests
automation/qtb: add Python QTB framework with the console-test type
automation/qtb: add unit tests for the QTB framework
automation/qtb: add QTB framework README
CI: run the riscv64 smoke test via QTB framework console-test
.gitlab-ci.yml | 3 +
automation/gitlab-ci/test.yaml | 26 ++-
automation/scripts/qemu-smoke-riscv64.sh | 19 --
automation/scripts/qemu_smoke_riscv64.py | 122 ++++++++++
automation/scripts/qtb/__init__.py | 2 +
automation/scripts/qtb/riscv/README.md | 182 +++++++++++++++
automation/scripts/qtb/riscv/__init__.py | 9 +
automation/scripts/qtb/riscv/config.py | 125 ++++++++++
automation/scripts/qtb/riscv/config.yaml | 19 ++
.../qtb/riscv/console_test/__init__.py | 4 +
.../qtb/riscv/console_test/console-test.yaml | 18 ++
.../qtb/riscv/console_test/console_test.py | 145 ++++++++++++
automation/scripts/qtb/riscv/dt.py | 57 +++++
.../scripts/qtb/riscv/dts/qemu-host.dts.j2 | 160 +++++++++++++
automation/scripts/qtb/riscv/machine.py | 56 +++++
automation/scripts/qtb/riscv/paths.py | 51 ++++
automation/scripts/qtb/riscv/qtb_test.py | 53 +++++
automation/scripts/qtb/riscv/unit/__init__.py | 2 +
automation/scripts/qtb/riscv/unit/conftest.py | 42 ++++
.../scripts/qtb/riscv/unit/test_config.py | 128 +++++++++++
.../qtb/riscv/unit/test_console_test.py | 217 ++++++++++++++++++
automation/scripts/qtb/riscv/unit/test_dt.py | 99 ++++++++
.../scripts/qtb/riscv/unit/test_machine.py | 58 +++++
.../scripts/qtb/riscv/unit/test_temp_dir.py | 42 ++++
.../scripts/qtb/riscv/unit/test_xen_dt.py | 46 ++++
automation/scripts/qtb/riscv/xen_dt.py | 58 +++++
26 files changed, 1718 insertions(+), 25 deletions(-)
delete mode 100755 automation/scripts/qemu-smoke-riscv64.sh
create mode 100755 automation/scripts/qemu_smoke_riscv64.py
create mode 100644 automation/scripts/qtb/__init__.py
create mode 100644 automation/scripts/qtb/riscv/README.md
create mode 100644 automation/scripts/qtb/riscv/__init__.py
create mode 100644 automation/scripts/qtb/riscv/config.py
create mode 100644 automation/scripts/qtb/riscv/config.yaml
create mode 100644 automation/scripts/qtb/riscv/console_test/__init__.py
create mode 100644 automation/scripts/qtb/riscv/console_test/console-test.yaml
create mode 100644 automation/scripts/qtb/riscv/console_test/console_test.py
create mode 100644 automation/scripts/qtb/riscv/dt.py
create mode 100644 automation/scripts/qtb/riscv/dts/qemu-host.dts.j2
create mode 100644 automation/scripts/qtb/riscv/machine.py
create mode 100644 automation/scripts/qtb/riscv/paths.py
create mode 100644 automation/scripts/qtb/riscv/qtb_test.py
create mode 100644 automation/scripts/qtb/riscv/unit/__init__.py
create mode 100644 automation/scripts/qtb/riscv/unit/conftest.py
create mode 100644 automation/scripts/qtb/riscv/unit/test_config.py
create mode 100644 automation/scripts/qtb/riscv/unit/test_console_test.py
create mode 100644 automation/scripts/qtb/riscv/unit/test_dt.py
create mode 100644 automation/scripts/qtb/riscv/unit/test_machine.py
create mode 100644 automation/scripts/qtb/riscv/unit/test_temp_dir.py
create mode 100644 automation/scripts/qtb/riscv/unit/test_xen_dt.py
create mode 100644 automation/scripts/qtb/riscv/xen_dt.py
--
2.55.0
--
Baptiste Le Duc | Vates Hypervisor & Kernel Engineer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech