Add a way to build minimal Xen + Linux dom0 images, and their corresponding
cross-compilation toolchains with Buildroot [1]. This effectively provides
a consistent cross-architecture development environment.

Buildroot is a project that makes it easy to build embedded system images,
including Linux, Xen or bootloaders. Leverage that build system to build
minimal images for use in the CI environment of Xen sub-projects, namely
Unikraft.

This approach is based on a Buildroot External tree [2] which currently only
provides 2 target defconfigs (arm64 and x86_64, both running on qemu).
It uses a recent development version of Buildroot that integrates changes
to build Xen on x86_64 [3], on its latest official version 4.22.0 [4].
It also includes the latest kernel (linux-7.2.7 at the time of writing).

As a byproduct, aside xen+kernel+rootfs, we also get a cross-compilation
toolchain (that runs on x86_64 hosts). This makes it easy to cross-compile
domUs for arm64, and it alleviates dynamic libraries mismatches when
adding userspace programs to the dom0 Linux image. As an example, one such
problem: Alpine linux is based on musl while your regular development
computer is probably based on glibc, which causes issues if you are
recompiling `xl` and try to use it in the Alpine image.

[1] https://buildroot.org/
[2] https://buildroot.org/downloads/manual/manual.html#outside-br-custom
[3] https://gitlab.com/buildroot.org/buildroot/-/commit/0bf8b2ecee6b1b41746e8863b57c2c5ba3a623b1
[4] https://gitlab.com/buildroot.org/buildroot/-/commit/a962be2eea146e06661712159698bc7cbf3305f2

This work is derived from this original series:
https://gitlab.com/xen-project/fusa/unikraft/-/merge_requests/1

Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>
---
 .gitlab-ci.yml                                | 20 ++++++++++
 br2-xendevel-external/.gitignore              |  2 +
 br2-xendevel-external/Config.in               |  9 +++++
 .../board/xendevel/common/rootfs-prune.sh     | 21 ++++++++++
 .../xendevel/x86_64/linux-xen-dom0.fragment   | 40 +++++++++++++++++++
 .../configs/xendevel_dom0_arm64_defconfig     | 34 ++++++++++++++++
 .../configs/xendevel_dom0_x86_64_defconfig    | 38 ++++++++++++++++++
 br2-xendevel-external/external.desc           |  2 +
 br2-xendevel-external/external.mk             |  9 +++++
 scripts/build-buildroot.sh                    | 35 ++++++++++++++++
 10 files changed, 210 insertions(+)
 create mode 100644 br2-xendevel-external/.gitignore
 create mode 100644 br2-xendevel-external/Config.in
 create mode 100755 br2-xendevel-external/board/xendevel/common/rootfs-prune.sh
 create mode 100644 br2-xendevel-external/board/xendevel/x86_64/linux-xen-dom0.fragment
 create mode 100644 br2-xendevel-external/configs/xendevel_dom0_arm64_defconfig
 create mode 100644 br2-xendevel-external/configs/xendevel_dom0_x86_64_defconfig
 create mode 100644 br2-xendevel-external/external.desc
 create mode 100644 br2-xendevel-external/external.mk
 create mode 100755 scripts/build-buildroot.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d04822f..8db3a0b 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -81,3 +81,23 @@ linux-6.6.56-x86_64:
 microcode-x86:
   extends: .x86_64-artifacts
   script: ./scripts/x86-microcode.sh
+
+
+#
+# Buildroot based artifacts
+#
+buildroot:
+  extends: .artifacts
+  image:
+    name: registry.gitlab.com/buildroot.org/buildroot/base:20250218.2110
+  parallel:
+    matrix:
+      - VARIANT: [arm64, x86_64]
+  cache:
+    key: buildroot-dl-${VARIANT}
+    paths:
+      - br2-xendevel-external/dl/
+  script:
+    - ./scripts/build-buildroot.sh "${VARIANT}"
+    - mkdir -p binaries
+    - cp br2-xendevel-external/xen-${VARIANT}/images/* binaries/
diff --git a/br2-xendevel-external/.gitignore b/br2-xendevel-external/.gitignore
new file mode 100644
index 0000000..424351b
--- /dev/null
+++ b/br2-xendevel-external/.gitignore
@@ -0,0 +1,2 @@
+xen-arm64
+xen-x86_64
diff --git a/br2-xendevel-external/Config.in b/br2-xendevel-external/Config.in
new file mode 100644
index 0000000..3c8e444
--- /dev/null
+++ b/br2-xendevel-external/Config.in
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# Kconfig entry point for the XENDEVEL external tree. Entries appear under
+# "External options -> Xen dom0 + Unikraft domU development environment" in
+# `make menuconfig`.
+#
+# No XENDEVEL packages are defined yet; add
+#   source "$BR2_EXTERNAL_XENDEVEL_PATH/package/<name>/Config.in"
+# lines here as they are introduced.
diff --git a/br2-xendevel-external/board/xendevel/common/rootfs-prune.sh b/br2-xendevel-external/board/xendevel/common/rootfs-prune.sh
new file mode 100755
index 0000000..7e3a661
--- /dev/null
+++ b/br2-xendevel-external/board/xendevel/common/rootfs-prune.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# Buildroot post-build script: shrink the dom0 target filesystem so the cpio
+# initramfs unpacks inside the dom0 memory window. $1 is the target directory.
+set -e
+T="$1"
+
+# qemu-xen BIOS/ROM/keymap blobs (~283MB) — unused by PV/PVH guests.
+rm -rf "$T/usr/share/qemu-xen"
+
+# Docs / man / info — not needed at runtime.
+rm -rf "$T/usr/share/doc" "$T/usr/share/man" "$T/usr/share/info"
+
+# The arch default kernel config is monolithic enough to boot dom0 without
+# loadable modules for the smoke test. Keep the xen-related modules if present,
+# drop the rest to shrink the initramfs.
+if [ -d "$T/lib/modules" ]; then
+	find "$T/lib/modules" -type f -name '*.ko' \
+		! -path '*xen*' -delete 2>/dev/null || true
+fi
diff --git a/br2-xendevel-external/board/xendevel/x86_64/linux-xen-dom0.fragment b/br2-xendevel-external/board/xendevel/x86_64/linux-xen-dom0.fragment
new file mode 100644
index 0000000..a999333
--- /dev/null
+++ b/br2-xendevel-external/board/xendevel/x86_64/linux-xen-dom0.fragment
@@ -0,0 +1,40 @@
+# Linux kernel config fragment: x86_64 Xen dom0.
+#
+# The stock x86_64 kernel defconfig does NOT enable Xen (CONFIG_XEN is not set),
+# so the resulting bzImage carries no Xen PV ELF notes / PVH entry and Xen
+# refuses to build dom0 ("Will only load images built for the generic loader or
+# Linux images ... or with PHYS32_ENTRY set" / "Could not construct d0"). Turn on
+# Xen PV+PVH dom0, the hvc console, and the backend/toolstack devices (evtchn,
+# gntdev, gntalloc, xenfs, blk/net back) so the xl toolstack can drive the domU.
+CONFIG_HYPERVISOR_GUEST=y
+CONFIG_PARAVIRT=y
+CONFIG_XEN=y
+CONFIG_XEN_PV=y
+CONFIG_XEN_PV_SMP=y
+CONFIG_XEN_DOM0=y
+CONFIG_XEN_PV_DOM0=y
+CONFIG_XEN_PVH=y
+CONFIG_XEN_PVHVM=y
+CONFIG_XEN_SAVE_RESTORE=y
+CONFIG_XEN_BACKEND=y
+CONFIG_XENFS=y
+CONFIG_XEN_COMPAT_XENFS=y
+CONFIG_XEN_SYS_HYPERVISOR=y
+CONFIG_XEN_XENBUS_FRONTEND=y
+CONFIG_XEN_GNTDEV=y
+CONFIG_XEN_GRANT_DEV_ALLOC=y
+CONFIG_XEN_DEV_EVTCHN=y
+CONFIG_XEN_PRIVCMD=y
+CONFIG_XEN_BLKDEV_FRONTEND=y
+CONFIG_XEN_BLKDEV_BACKEND=y
+CONFIG_XEN_NETDEV_FRONTEND=y
+CONFIG_XEN_NETDEV_BACKEND=y
+CONFIG_HVC_XEN=y
+CONFIG_HVC_XEN_FRONTEND=y
+CONFIG_TTY=y
+# Monolithic kernel: build everything in, no loadable modules. This drops the
+# whole udev/libkmod module-loading path at boot (no .ko files to insert), so
+# the rootfs can also drop kmod + eudev. Every driver this QEMU dom0 needs
+# (ata_piix, e1000, virtio, 9p, xen blk/net back) is already =y in the x86_64
+# arch default config, so nothing required is lost.
+# CONFIG_MODULES is not set
diff --git a/br2-xendevel-external/configs/xendevel_dom0_arm64_defconfig b/br2-xendevel-external/configs/xendevel_dom0_arm64_defconfig
new file mode 100644
index 0000000..cb802a8
--- /dev/null
+++ b/br2-xendevel-external/configs/xendevel_dom0_arm64_defconfig
@@ -0,0 +1,34 @@
+# aarch64 Xen dom0: hypervisor + xl toolstack + Linux
+
+BR2_aarch64=y
+BR2_cortex_a72=y
+
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_AARCH64_GLIBC_STABLE=y
+
+BR2_TARGET_GENERIC_HOSTNAME="xen-dom0"
+BR2_TARGET_GENERIC_ISSUE="Xen dom0 (br2-xendevel-external)"
+BR2_TARGET_GENERIC_GETTY_PORT="console"
+
+# devtmpfs mounted by init + eudev, so /dev/null and the Xen
+# gnttab/evtchn/privcmd device nodes exist at runtime.
+BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV=y
+
+BR2_LINUX_KERNEL=y
+BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG=y
+BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL=y
+
+BR2_PACKAGE_XEN=y
+BR2_PACKAGE_XEN_HYPERVISOR=y
+BR2_PACKAGE_XEN_TOOLS=y
+
+BR2_PACKAGE_KMOD=y
+BR2_PACKAGE_UTIL_LINUX_BINARIES=y
+
+BR2_TARGET_ROOTFS_CPIO=y
+BR2_TARGET_ROOTFS_CPIO_GZIP=y
+# BR2_TARGET_ROOTFS_TAR is not set
+BR2_ROOTFS_POST_BUILD_SCRIPT="$(BR2_EXTERNAL_XENDEVEL_PATH)/board/xendevel/common/rootfs-prune.sh"
+
+BR2_PACKAGE_HOST_ENVIRONMENT_SETUP=y
diff --git a/br2-xendevel-external/configs/xendevel_dom0_x86_64_defconfig b/br2-xendevel-external/configs/xendevel_dom0_x86_64_defconfig
new file mode 100644
index 0000000..7e0d894
--- /dev/null
+++ b/br2-xendevel-external/configs/xendevel_dom0_x86_64_defconfig
@@ -0,0 +1,38 @@
+# x86_64 Xen dom0: hypervisor + xl toolstack + Linux
+
+BR2_x86_64=y
+
+BR2_TOOLCHAIN_EXTERNAL=y
+BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_X86_64_GLIBC_STABLE=y
+
+BR2_TARGET_GENERIC_HOSTNAME="xen-dom0"
+BR2_TARGET_GENERIC_ISSUE="Xen dom0 (br2-xendevel-external)"
+BR2_TARGET_GENERIC_GETTY_PORT="console"
+
+# The monolithic kernel has CONFIG_DEVTMPFS_MOUNT=y, so it creates every dev
+# (including the Xen control nodes /dev/xen/{evtchn,gntdev,privcmd,...} that
+# xenstored/xenconsoled need) at driver registration. That removes the need for
+# a userspace device manager and the libkmod/udev boot noise entirely.
+BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_DEVTMPFS=y
+
+BR2_LINUX_KERNEL=y
+BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG=y
+BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="$(BR2_EXTERNAL_XENDEVEL_PATH)/board/xendevel/x86_64/linux-xen-dom0.fragment"
+BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL=y
+BR2_LINUX_KERNEL_NEEDS_HOST_LIBELF=y
+
+BR2_PACKAGE_XEN=y
+BR2_PACKAGE_XEN_HYPERVISOR=y
+BR2_PACKAGE_XEN_TOOLS=y
+
+# No kmod: the kernel is monolithic (CONFIG_MODULES=n), so there are no modules
+# to insert and nothing links against libkmod. Dropping it (and eudev, above)
+# removes the only consumer that was failing on the missing libkmod.so.2.
+
+BR2_TARGET_ROOTFS_CPIO=y
+BR2_TARGET_ROOTFS_CPIO_GZIP=y
+# BR2_TARGET_ROOTFS_TAR is not set
+BR2_ROOTFS_POST_BUILD_SCRIPT="$(BR2_EXTERNAL_XENDEVEL_PATH)/board/xendevel/common/rootfs-prune.sh"
+
+BR2_PACKAGE_HOST_ENVIRONMENT_SETUP=y
diff --git a/br2-xendevel-external/external.desc b/br2-xendevel-external/external.desc
new file mode 100644
index 0000000..232eb0e
--- /dev/null
+++ b/br2-xendevel-external/external.desc
@@ -0,0 +1,2 @@
+name: XENDEVEL
+desc: Xen development environment
diff --git a/br2-xendevel-external/external.mk b/br2-xendevel-external/external.mk
new file mode 100644
index 0000000..37cd1d8
--- /dev/null
+++ b/br2-xendevel-external/external.mk
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# Buildroot external tree makefile fragment.
+#
+# Every package/*/*.mk under this tree is included by Buildroot. There are no
+# XENDEVEL-specific packages yet: the tree currently provides the dom0
+# defconfigs and board support files only.
+
+include $(sort $(wildcard $(BR2_EXTERNAL_XENDEVEL_PATH)/package/*/*.mk))
diff --git a/scripts/build-buildroot.sh b/scripts/build-buildroot.sh
new file mode 100755
index 0000000..73ab123
--- /dev/null
+++ b/scripts/build-buildroot.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -e -o pipefail
+
+BR2_EXT_DIR=$(realpath $(dirname $0)/../br2-xendevel-external)
+
+BR_REPO=https://gitlab.com/buildroot.org/buildroot
+# First Buildroot version (2026.08-345-ga962be2eea) with Xen 4.22 on x86_64
+BR_COMMIT=a962be2eea146e06661712159698bc7cbf3305f2
+BR_DIR=${BR2_EXT_DIR}/buildroot
+export BR2_DL_DIR=${BR2_EXT_DIR}/dl
+
+# Not using `git clone --revision` as it requires git >= 2.49
+if [ "$(git -C ${BR_DIR} rev-parse HEAD 2>/dev/null)" != "${BR_COMMIT}" ]; then
+	git init -q ${BR_DIR}
+	git -C ${BR_DIR} fetch --depth=1 ${BR_REPO} ${BR_COMMIT}
+	git -C ${BR_DIR} checkout -q FETCH_HEAD
+fi
+
+do_build() {	
+	build_dir=${BR2_EXT_DIR}/xen-${1}
+	make BR2_EXTERNAL=${BR2_EXT_DIR} O=${build_dir} -C ${BR_DIR} xendevel_dom0_${1}_defconfig
+	make -C ${build_dir} source all sdk | grep '>>>'
+}
+
+for arg in $*; do
+	case "$arg" in
+		arm64|x86_64)
+			do_build $arg
+			;;
+		*)
+			echo "Unknown target $arg" >&2
+			exit 1
+	esac
+done
-- 
2.55.0

On 01/10/2026 12:52 pm, Titouan Christophe wrote:
> Add a way to build minimal Xen + Linux dom0 images, and their corresponding
> cross-compilation toolchains with Buildroot [1]. This effectively provides
> a consistent cross-architecture development environment.
>
> Buildroot is a project that makes it easy to build embedded system images,
> including Linux, Xen or bootloaders. Leverage that build system to build
> minimal images for use in the CI environment of Xen sub-projects, namely
> Unikraft.
>
> This approach is based on a Buildroot External tree [2] which currently only
> provides 2 target defconfigs (arm64 and x86_64, both running on qemu).
> It uses a recent development version of Buildroot that integrates changes
> to build Xen on x86_64 [3], on its latest official version 4.22.0 [4].
> It also includes the latest kernel (linux-7.2.7 at the time of writing).
>
> As a byproduct, aside xen+kernel+rootfs, we also get a cross-compilation
> toolchain (that runs on x86_64 hosts). This makes it easy to cross-compile
> domUs for arm64, and it alleviates dynamic libraries mismatches when
> adding userspace programs to the dom0 Linux image. As an example, one such
> problem: Alpine linux is based on musl while your regular development
> computer is probably based on glibc, which causes issues if you are
> recompiling `xl` and try to use it in the Alpine image.
>
> [1] https://buildroot.org/
> [2] https://buildroot.org/downloads/manual/manual.html#outside-br-custom
> [3] https://gitlab.com/buildroot.org/buildroot/-/commit/0bf8b2ecee6b1b41746e8863b57c2c5ba3a623b1
> [4] https://gitlab.com/buildroot.org/buildroot/-/commit/a962be2eea146e06661712159698bc7cbf3305f2
>
> This work is derived from this original series:
> https://gitlab.com/xen-project/fusa/unikraft/-/merge_requests/1
>
> Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>

Thankyou for your submission.

How are you intending to use this image?  There's no consumer shown, so
it's hard to judge the patch.

One thing I see is that it intentionally embeds a fixed kernel and
(commented but not actually added yet) Xen, and I'm not sure that's a
route we want to go down.

As for compiling the binary somewhere and copying it to a different
environment, we have build containers for that, and I don't see how
using buildroot changes the problem.

~Andrew

On 1/10/26 14:32, Andrew Cooper wrote:
> On 01/10/2026 12:52 pm, Titouan Christophe wrote:
>> Add a way to build minimal Xen + Linux dom0 images, and their corresponding
>> cross-compilation toolchains with Buildroot [1]. This effectively provides
>> a consistent cross-architecture development environment.
>>
>> Buildroot is a project that makes it easy to build embedded system images,
>> including Linux, Xen or bootloaders. Leverage that build system to build
>> minimal images for use in the CI environment of Xen sub-projects, namely
>> Unikraft.
>>
>> This approach is based on a Buildroot External tree [2] which currently only
>> provides 2 target defconfigs (arm64 and x86_64, both running on qemu).
>> It uses a recent development version of Buildroot that integrates changes
>> to build Xen on x86_64 [3], on its latest official version 4.22.0 [4].
>> It also includes the latest kernel (linux-7.2.7 at the time of writing).
>>
>> As a byproduct, aside xen+kernel+rootfs, we also get a cross-compilation
>> toolchain (that runs on x86_64 hosts). This makes it easy to cross-compile
>> domUs for arm64, and it alleviates dynamic libraries mismatches when
>> adding userspace programs to the dom0 Linux image. As an example, one such
>> problem: Alpine linux is based on musl while your regular development
>> computer is probably based on glibc, which causes issues if you are
>> recompiling `xl` and try to use it in the Alpine image.
>>
>> [1] https://buildroot.org/
>> [2] https://buildroot.org/downloads/manual/manual.html#outside-br-custom
>> [3] https://gitlab.com/buildroot.org/buildroot/-/commit/0bf8b2ecee6b1b41746e8863b57c2c5ba3a623b1
>> [4] https://gitlab.com/buildroot.org/buildroot/-/commit/a962be2eea146e06661712159698bc7cbf3305f2
>>
>> This work is derived from this original series:
>> https://gitlab.com/xen-project/fusa/unikraft/-/merge_requests/1
>>
>> Signed-off-by: Titouan Christophe <titouan.christophe@mind.be>
> Thankyou for your submission.
Hi Andrew, and thank you for having looked so quickly into this patch.

>
> How are you intending to use this image?  There's no consumer shown, so
> it's hard to judge the patch.

The goal is to obtain a `binaries` directory containing xen, linux and 
its rootfs; as well as the corresponding cross-compilation toolchain 
(either additional linux applications, or "bare" domUs to run with Xen).

Concretely, my next step would be to use that in Unikraft's CI in the 
following manner:
1. Download the latest binaries from test-artifacts (that should provide 
the latest Xen and a suitable linux)
2. Cross-compile a Unikraft application
3. Launch it on Xen, and assert expected output (similarly to the smoke 
tests in 
https://gitlab.com/xen-project/xen/-/tree/staging/automation/scripts)

Therefore, Xen serves as a base platform for the above, and it feels 
like a stretch to have to recompile it from scratch in Unikraft's CI. 
Perhaps there are other projects that could benefit from this ?

As far as I understand, all Xen tests are currently run on their native 
architecture (ie. arm64 runs on arm64 hosts etc...), and I had some 
trouble adding programs to the alpine images due to dynamic libraries 
mismatches (musl vs. glibc for example).

> One thing I see is that it intentionally embeds a fixed kernel and
> (commented but not actually added yet) Xen, and I'm not sure that's a
> route we want to go down.

Nit: Xen IS actually included, not sure to which commented section you 
are referring ?

>
> As for compiling the binary somewhere and copying it to a different
> environment, we have build containers for that, and I don't see how
> using buildroot changes the problem.

As you might guess, I'm quite new to the Xen Project :) Are these "build 
containers" the ones obtained in the test-artifacts repository ? If not, 
where are they coming from, and how are they created ?

>
> ~Andrew

Kind regards,
Titouan