hw/ppc/spapr_hcall.c | 14 ++++++ linux-headers/asm-powerpc/kvm.h | 19 +++++++ linux-headers/linux/kvm.h | 3 ++ target/ppc/kvm.c | 87 +++++++++++++++++++++++++++++++++ target/ppc/kvm_ppc.h | 7 +++ 5 files changed, 130 insertions(+)
On POWER systems, newer processor generations can operate in compatibility
modes corresponding to earlier generations (e.g., a Power11 system running
in Power10 compatibility mode). In such cases, the effective CPU level
exposed to guests differs from the physical processor generation.
This creates issues for nested virtualization. When booting a nested KVM
guest, QEMU may derive the CPU model from the raw hardware PVR and attempt
to configure the guest accordingly. However, the host is constrained by the
compatibility level negotiated with the hypervisor, and requests exceeding
that level are rejected by KVM, leading to guest boot failures such as:
KVM-NESTEDv2: couldn't set guest wide elements
This series addresses the issue by preventing fallback to raw mode when the
host itself is booted in a compatibility mode, and by querying the effective
CPU compatibility modes supported by the host via KVM. The kernel interface
uses copy_struct_from/to_user() for forward and backward ABI compatibility.
With these changes, QEMU ensures that nested guests are configured with CPU
models consistent with the host compatibility mode, allowing them to boot
correctly.
Patch summary:
[1/3] [DO_NOT_MERGE] linux-headers: Add uapi header changes
[2/3] target/ppc/kvm: Add support for querying host compatibility mode
[3/3] target/ppc/kvm: Use host compatibility mode for nested guests
Changes in v5:
- Patch 1: Updated KVM_PPC_GET_COMPAT_CAPS ioctl number from 0xe4 to
0xb8 to match the corresponding Linux kernel v6 series change; the
0xe0-0xe3 range is reserved for KVM_CREATE_DEVICE fd ioctls
Testing (with kernel v6 patches):
KVM APIv1 Testing
=================
On P10 PowerNV machine (L0)
---------------------------
- P10 L1 KVM guest -> works
- P10 nested L2 KVM guest -> works
- P9 compat nested L2 KVM guest -> works
- P9 compat L1 KVM guest -> works
- P9 nested L2 KVM guest -> works
On Powernv11 TCG Guest (L0)
---------------------------
- P11 PowerNV TCG L0 guest -> works
- P11 L1 KVM guest -> works
- P11 L2 KVM guest -> works
- P10 compat L1 KVM guest -> works
- P10 L2 KVM guest -> works
- P9 compat L1 KVM guest -> works
- P9 L2 KVM guest -> works
KVM APIv2 Testing
=================
On P11 PowerVM LPAR (L1)
------------------------
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
- Without QEMU patches but Linux patches
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
- Without Linux patches but QEMU patches
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
On P11 LPAR in P10 compat (L1)
------------------------------
- P10 (host compat) L2 KVM guest -> works
- Without QEMU patch but Linux patches
- P10 guest fails to boot as expected (error: kvm run failed Invalid argument)
- Without Linux patch but QEMU patches
- P10 guest fails to boot as expected (KVM: unknown exit, hardware reason ffffffffffffffea)
On P10 PowerVM LPAR (L1)
------------------------
- P10 L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
TCG pSeries Guest
=================
- P11 (default) pSeries guest boots fine
ABI Extensibility Testing (struct size 32, extra member)
=========================================================
- Newer struct on QEMU, older kernel -> works (kernel returns -E2BIG,
QEMU retries with correct size)
- New struct on Linux kernel, older QEMU -> works (kernel zero-pads
trailing fields, QEMU gets correct data)
Note: Patch 1 is marked DO_NOT_MERGE as it contains linux-headers updates
that will be synced separately once the corresponding kernel patches are
merged.
The corresponding Linux kernel patches (v6) are being posted concurrently.
v4: https://lore.kernel.org/all/20260701052341.62289-1-amachhiw@linux.ibm.com/
v3: https://lore.kernel.org/all/20260616113915.25589-1-amachhiw@linux.ibm.com/
v2: https://lore.kernel.org/all/20260502140021.69712-1-amachhiw@linux.ibm.com/
v1: https://lore.kernel.org/all/20260430061333.37905-1-amachhiw@linux.ibm.com/
Previous kernel patch versions:
v6: https://lore.kernel.org/all/20260804180705.59160-1-amachhiw@linux.ibm.com/
v5: https://lore.kernel.org/all/20260701051409.51820-1-amachhiw@linux.ibm.com/
v4: https://lore.kernel.org/all/20260616123314.82721-1-amachhiw@linux.ibm.com/
v3: https://lore.kernel.org/all/20260522152744.55251-1-amachhiw@linux.ibm.com/
v2: https://lore.kernel.org/all/20260513100755.83195-1-amachhiw@linux.ibm.com/
v1: https://lore.kernel.org/all/20260430054906.94401-1-amachhiw@linux.ibm.com/
Amit Machhiwal (3):
[DO_NOT_MERGE] linux-headers: Add uapi header changes
target/ppc/kvm: Add support for querying host compatibility mode
target/ppc/kvm: Use host compatibility mode for nested guests
hw/ppc/spapr_hcall.c | 14 ++++++
linux-headers/asm-powerpc/kvm.h | 19 +++++++
linux-headers/linux/kvm.h | 3 ++
target/ppc/kvm.c | 87 +++++++++++++++++++++++++++++++++
target/ppc/kvm_ppc.h | 7 +++
5 files changed, 130 insertions(+)
base-commit: b428fe036233cbd15d37e3c027ab6ca4d3661a80
--
2.50.1 (Apple Git-155)
On 04/08/26 11:59 PM, Amit Machhiwal wrote: > On POWER systems, newer processor generations can operate in compatibility > modes corresponding to earlier generations (e.g., a Power11 system running > in Power10 compatibility mode). In such cases, the effective CPU level > exposed to guests differs from the physical processor generation. > > This creates issues for nested virtualization. When booting a nested KVM > guest, QEMU may derive the CPU model from the raw hardware PVR and attempt > to configure the guest accordingly. However, the host is constrained by the > compatibility level negotiated with the hypervisor, and requests exceeding > that level are rejected by KVM, leading to guest boot failures such as: > > KVM-NESTEDv2: couldn't set guest wide elements > > This series addresses the issue by preventing fallback to raw mode when the > host itself is booted in a compatibility mode, and by querying the effective > CPU compatibility modes supported by the host via KVM. The kernel interface > uses copy_struct_from/to_user() for forward and backward ABI compatibility. > With these changes, QEMU ensures that nested guests are configured with CPU > models consistent with the host compatibility mode, allowing them to boot > correctly. > > Patch summary: > [1/3] [DO_NOT_MERGE] linux-headers: Add uapi header changes > [2/3] target/ppc/kvm: Add support for querying host compatibility mode > [3/3] target/ppc/kvm: Use host compatibility mode for nested guests > > Changes in v5: > - Patch 1: Updated KVM_PPC_GET_COMPAT_CAPS ioctl number from 0xe4 to > 0xb8 to match the corresponding Linux kernel v6 series change; the > 0xe0-0xe3 range is reserved for KVM_CREATE_DEVICE fd ioctls > > Testing (with kernel v6 patches): > > KVM APIv1 Testing > ================= > On P10 PowerNV machine (L0) > --------------------------- > - P10 L1 KVM guest -> works > - P10 nested L2 KVM guest -> works > - P9 compat nested L2 KVM guest -> works > - P9 compat L1 KVM guest -> works > - P9 nested L2 KVM guest -> works > > On Powernv11 TCG Guest (L0) > --------------------------- > - P11 PowerNV TCG L0 guest -> works > - P11 L1 KVM guest -> works > - P11 L2 KVM guest -> works > - P10 compat L1 KVM guest -> works > - P10 L2 KVM guest -> works > - P9 compat L1 KVM guest -> works > - P9 L2 KVM guest -> works > > KVM APIv2 Testing > ================= > On P11 PowerVM LPAR (L1) > ------------------------ > - P11 L2 KVM guest -> works > - P10 compat L2 KVM guest -> works > - P9 compat L2 KVM guest fails to boot as expected > - Without QEMU patches but Linux patches > - P11 L2 KVM guest -> works > - P10 compat L2 KVM guest -> works > - P9 compat L2 KVM guest fails to boot as expected > - Without Linux patches but QEMU patches > - P11 L2 KVM guest -> works > - P10 compat L2 KVM guest -> works > > On P11 LPAR in P10 compat (L1) > ------------------------------ > - P10 (host compat) L2 KVM guest -> works > - Without QEMU patch but Linux patches > - P10 guest fails to boot as expected (error: kvm run failed Invalid argument) > - Without Linux patch but QEMU patches > - P10 guest fails to boot as expected (KVM: unknown exit, hardware reason ffffffffffffffea) > > On P10 PowerVM LPAR (L1) > ------------------------ > - P10 L2 KVM guest -> works > - P9 compat L2 KVM guest fails to boot as expected > > TCG pSeries Guest > ================= > - P11 (default) pSeries guest boots fine > > ABI Extensibility Testing (struct size 32, extra member) > ========================================================= > - Newer struct on QEMU, older kernel -> works (kernel returns -E2BIG, > QEMU retries with correct size) > - New struct on Linux kernel, older QEMU -> works (kernel zero-pads > trailing fields, QEMU gets correct data) > > Note: Patch 1 is marked DO_NOT_MERGE as it contains linux-headers updates > that will be synced separately once the corresponding kernel patches are > merged. > > The corresponding Linux kernel patches (v6) are being posted concurrently. > > v4: https://lore.kernel.org/all/20260701052341.62289-1-amachhiw@linux.ibm.com/ > v3: https://lore.kernel.org/all/20260616113915.25589-1-amachhiw@linux.ibm.com/ > v2: https://lore.kernel.org/all/20260502140021.69712-1-amachhiw@linux.ibm.com/ > v1: https://lore.kernel.org/all/20260430061333.37905-1-amachhiw@linux.ibm.com/ > > Previous kernel patch versions: > v6: https://lore.kernel.org/all/20260804180705.59160-1-amachhiw@linux.ibm.com/ > v5: https://lore.kernel.org/all/20260701051409.51820-1-amachhiw@linux.ibm.com/ > v4: https://lore.kernel.org/all/20260616123314.82721-1-amachhiw@linux.ibm.com/ > v3: https://lore.kernel.org/all/20260522152744.55251-1-amachhiw@linux.ibm.com/ > v2: https://lore.kernel.org/all/20260513100755.83195-1-amachhiw@linux.ibm.com/ > v1: https://lore.kernel.org/all/20260430054906.94401-1-amachhiw@linux.ibm.com/ > > Amit Machhiwal (3): > [DO_NOT_MERGE] linux-headers: Add uapi header changes > target/ppc/kvm: Add support for querying host compatibility mode > target/ppc/kvm: Use host compatibility mode for nested guests > > hw/ppc/spapr_hcall.c | 14 ++++++ > linux-headers/asm-powerpc/kvm.h | 19 +++++++ > linux-headers/linux/kvm.h | 3 ++ > target/ppc/kvm.c | 87 +++++++++++++++++++++++++++++++++ > target/ppc/kvm_ppc.h | 7 +++ > 5 files changed, 130 insertions(+) > > > base-commit: b428fe036233cbd15d37e3c027ab6ca4d3661a80 Hi Amit, I have tested this patch and it works as expected. Here is my analysis : I booted a host with Power10 compat mode and tried following scenarios - lscpu on host : Architecture: ppc64le Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Model name: POWER10 (architected), altivec supported Before applying the patch : When I am trying to bringup the guest on a compat mode host it was bringing up a Power11 guest and was failing as [ 1411.578944] [ T2928] KVM-NESTEDv2: couldn't set guest wide elements [ 1411.578963] [ T2928] vcpu 000000000b9c4155 (0): [ 1411.578968] [ T2928] pc = 000000007daf9790 msr = 8000000000103000 trap = ffffffea [ 1411.578973] [ T2928] r 0 = 8000000000003000 r16 = 0000000000000000 [ 1411.578978] [ T2928] r 1 = 000000007e581e20 r17 = 0000000000000000 [ 1411.578982] [ T2928] r 2 = 000000007db26c00 r18 = 0000000000000000 [ 1411.578985] [ T2928] r 3 = 0000000000000000 r19 = 0000000000000000 [ 1411.578989] [ T2928] r 4 = 0000000002e30c80 r20 = 0000000000000000 [ 1411.578993] [ T2928] r 5 = 000000007df80000 r21 = 0000000000000000 [ 1411.578996] [ T2928] r 6 = 0000000000200000 r22 = 00000000018c5fd6 [ 1411.579000] [ T2928] r 7 = 000000007df80000 r23 = 000000007db21cc0 [ 1411.579003] [ T2928] r 8 = 000000007db6e5d8 r24 = 000000007db66000 [ 1411.579006] [ T2928] r 9 = 000000007e6655d8 r25 = 000000007e665508 [ 1411.579010] [ T2928] r10 = 000000007db6e5d0 r26 = 00000000018c5fd6 [ 1411.579013] [ T2928] r11 = 0000000000003000 r27 = 0000000000000003 [ 1411.579017] [ T2928] r12 = 8000000000000001 r28 = 000000007db6e5e0 [ 1411.579020] [ T2928] r13 = 0000000000000000 r29 = 000000007db224b0 [ 1411.579024] [ T2928] r14 = 0000000000000000 r30 = 000000007daf274c [ 1411.579028] [ T2928] r15 = 0000000000000000 r31 = 000000007db76000 [ 1411.579033] [ T2928] ctr = 000000007daf1b44 lr = 000000007daf1b7c [ 1411.579037] [ T2928] srr0 = 000000007daf9790 srr1 = 8000000000102000 [ 1411.579041] [ T2928] sprg0 = 0000000000000000 sprg1 = 000000000000ff10 [ 1411.579045] [ T2928] sprg2 = 0000000000000000 sprg3 = 0000000000000000 [ 1411.579049] [ T2928] cr = 20000402 xer = 0000000020040000 dsisr = 00000000 [ 1411.579054] [ T2928] dar = 0000000000000000 [ 1411.579057] [ T2928] fault dar = 0000000000000000 dsisr = 00000000 [ 1411.579061] [ T2928] SLB (0 entries): [ 1411.579064] [ T2928] lpcr = 0040000000020400 sdr1 = 0000000000000000 last_inst = ffffffffffffffff [ 1411.579069] [ T2928] trap=0xffffffea | pc=0x7daf9790 | msr=0x8000000000103000 After applying this patch along with the kernel built with it's dependent patch (https://lore.kernel.org/all/20260804180705.59160-1-amachhiw@linux.ibm.com/): I am able to bringup a guest and it got boot up with Power10 by default: lscpu on guest - ltcbonn53-vm2:~ # lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Model name: POWER10 (architected), altivec supported Model: 2.0 (pvr 0082 0200) Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Please feel free to add my tested-by: Tested-by: Anushree Mathur <anushree.mathur@linux.ibm.com> Thank you, Anushree Mathur
© 2016 - 2026 Red Hat, Inc.