:p
atchew
Login
On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation. This creates issues for nested virtualization. When booting a nested KVM guest, QEMU may derive the CPU model from the raw hardware PVR and attempt to configure the guest accordingly. However, the host is constrained by the compatibility level negotiated with the hypervisor, and requests exceeding that level are rejected by KVM, leading to guest boot failures such as: KVM-NESTEDv2: couldn't set guest wide elements This series addresses the issue by preventing fallback to raw mode when the host itself is booted in a compatibility mode, and by querying the effective CPU compatibility modes supported by the host via KVM. The implementation includes size field validation for ABI compatibility between QEMU and the kernel. With these changes, QEMU ensures that nested guests are configured with CPU models consistent with the host compatibility mode, allowing them to boot correctly. Patch summary: [1/3] [DO_NOT_MERGE] linux-headers: Add uapi header changes [2/3] target/ppc/kvm: Add support for querying host compatibility mode [3/3] target/ppc/kvm: Use host compatibility mode for nested guests Changes in v3 (based on review from Vaibhav): - Patch 1: Moved compatibility mode check from do_client_architecture_support() to cas_check_pvr(). Instead of error handling when KVM rejects compat mode, now prevents raw mode fallback when host is booted in compatibility mode. This ensures guests cannot exceed the host's compatibility level. - Patch 2: Added size field to kvm_ppc_compat_caps structure for ABI versioning. Changed flags field from __u32 to __u64. Added capability bit definitions (KVM_PPC_COMPAT_CAP_POWER9/10/11) and KVM_PPC_COMPAT_BITMASK. - Patch 3: Added size field initialization and validation in kvmppc_get_compat_caps() to ensure ABI compatibility between QEMU and kernel. Changed from H_GUEST_CAP_* constants to KVM_PPC_COMPAT_CAP_* constants. Added capability masking with KVM_PPC_COMPAT_BITMASK. - Dropped Tested-by from Anushree due to code changes in v3. Changes in v2: - Patch 3: Guard compatibility mode code with #if defined(TARGET_PPC64) to fix compilation for ppc32 targets. The POWER9/10/11 PVR constants are only defined for 64-bit builds, and compatibility modes are only relevant for 64-bit systems. Testing (with kernel v4 patches): KVM APIv1 Testing ================= On P10 PowerNV machine (L0) --------------------------- - P10 L1 KVM guest -> works - P10 nested L2 KVM guest -> works - P9 compat nested L2 KVM guest -> works - P9 compat L1 KVM guest -> works - P9 nested L2 KVM guest -> works On Powernv11 TCG Guest (L0) --------------------------- - P11 L1 KVM guest -> works - P11 L2 KVM guest -> works - P10 compat L1 KVM guest -> works - P10 L2 KVM guest -> works - P9 compat L1 KVM guest -> works - P9 L2 KVM guest -> works KVM APIv2 Testing ================= On P11 PowerVM LPAR (L1) ------------------------ - P11 L2 KVM guest -> works - P10 compat L2 KVM guest -> works On P11 LPAR in P10 compat (L1) ------------------------------ - P10 (host compat) L2 KVM guest -> works On P10 PowerVM LPAR (L1) ------------------------ - P10 L2 KVM guest -> works CI test results: https://gitlab.com/amachhiw/qemu/-/pipelines/2604638350 Note: Patch 1 is marked DO_NOT_MERGE as it contains linux-headers updates that will be synced separately once the corresponding kernel patches are merged. v2: https://lore.kernel.org/all/20260502140021.69712-1-amachhiw@linux.ibm.com/ v1: https://lore.kernel.org/all/20260430061333.37905-1-amachhiw@linux.ibm.com/ The corresponding Linux kernel patches (v4) are being posted concurrently. Previous kernel patch versions: v3: https://lore.kernel.org/all/20260522152744.55251-1-amachhiw@linux.ibm.com/ v2: https://lore.kernel.org/all/20260513100755.83215-1-amachhiw@linux.ibm.com/ v1: https://lore.kernel.org/all/20260430054906.94431-1-amachhiw@linux.ibm.com/ Amit Machhiwal (3): linux-headers: Add uapi header changes target/ppc/kvm: Add support for querying host compatibility mode target/ppc/kvm: Use host compatibility mode for nested guests hw/ppc/spapr_hcall.c | 14 +++++++ linux-headers/asm-powerpc/kvm.h | 18 ++++++++ linux-headers/linux/kvm.h | 3 ++ target/ppc/kvm.c | 74 +++++++++++++++++++++++++++++++++ target/ppc/kvm_ppc.h | 16 +++++++ 5 files changed, 125 insertions(+) base-commit: 2f28d34ea0aead9830478cd1d3d0dd9d9191d82e -- 2.50.1 (Apple Git-155)
This is a temporary patch intended for review and testing purposes only. It syncs the QEMU linux-headers with the kernel changes that introduce the KVM_PPC_GET_COMPAT_CAPS ioctl for querying host CPU compatibility capabilities. This includes the kvm_ppc_compat_caps structure with a size field for ABI versioning, the KVM_CAP_PPC_COMPAT_CAPS capability definition, the KVM_PPC_GET_COMPAT_CAPS ioctl definition, and capability bit definitions for POWER9, POWER10, and POWER11 compatibility modes. The actual header sync will be done via scripts/update-linux-headers.sh once the kernel changes [1] are merged upstream. [1] https://lore.kernel.org/all/20260430054906.94431-1-amachhiw@linux.ibm.com/ Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com> --- linux-headers/asm-powerpc/kvm.h | 18 ++++++++++++++++++ linux-headers/linux/kvm.h | 3 +++ 2 files changed, 21 insertions(+) diff --git a/linux-headers/asm-powerpc/kvm.h b/linux-headers/asm-powerpc/kvm.h index XXXXXXX..XXXXXXX 100644 --- a/linux-headers/asm-powerpc/kvm.h +++ b/linux-headers/asm-powerpc/kvm.h @@ -XXX,XX +XXX,XX @@ struct kvm_ppc_cpu_char { __u64 behaviour_mask; /* valid bits in behaviour */ }; +/* For KVM_PPC_GET_COMPAT_CAPS */ +struct kvm_ppc_compat_caps { + __u64 flags; /* Reserved for future use */ + __u64 size; /* Size of this structure */ + __u64 compat_capabilities; /* Capabilities supported by the host */ +}; +/* + * Capability bits for compat_capabilities field in kvm_ppc_compat_caps. + * These bits indicate which processor compatibility modes are supported. + */ +#define KVM_PPC_COMPAT_CAP_POWER9 (1ULL << 62) +#define KVM_PPC_COMPAT_CAP_POWER10 (1ULL << 61) +#define KVM_PPC_COMPAT_CAP_POWER11 (1ULL << 60) +#define KVM_PPC_COMPAT_BITMASK (KVM_PPC_COMPAT_CAP_POWER9 | \ + KVM_PPC_COMPAT_CAP_POWER10 | \ + KVM_PPC_COMPAT_CAP_POWER11) + + /* * Values for character and character_mask. * These are identical to the values used by H_GET_CPU_CHARACTERISTICS. diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h index XXXXXXX..XXXXXXX 100644 --- a/linux-headers/linux/kvm.h +++ b/linux-headers/linux/kvm.h @@ -XXX,XX +XXX,XX @@ struct kvm_enable_cap { #define KVM_CAP_S390_USER_OPEREXEC 246 #define KVM_CAP_S390_KEYOP 247 #define KVM_CAP_S390_VSIE_ESAMODE 248 +#define KVM_CAP_PPC_COMPAT_CAPS 249 struct kvm_irq_routing_irqchip { __u32 irqchip; @@ -XXX,XX +XXX,XX @@ struct kvm_s390_keyop { /* Available with KVM_CAP_COUNTER_OFFSET */ #define KVM_ARM_SET_COUNTER_OFFSET _IOW(KVMIO, 0xb5, struct kvm_arm_counter_offset) #define KVM_ARM_GET_REG_WRITABLE_MASKS _IOR(KVMIO, 0xb6, struct reg_mask_range) +/* Available with KVM_CAP_PPC_COMPAT_CAPS */ +#define KVM_PPC_GET_COMPAT_CAPS _IOR(KVMIO, 0xe4, struct kvm_ppc_compat_caps) /* ioctl for vm fd */ #define KVM_CREATE_DEVICE _IOWR(KVMIO, 0xe0, struct kvm_create_device) -- 2.50.1 (Apple Git-155)
Add infrastructure to query the host CPU compatibility mode via the KVM_PPC_GET_COMPAT_CAPS ioctl. This allows QEMU to determine if the host is running in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). The kvmppc_get_compat_caps() function queries KVM for host compatibility capabilities with size field validation for ABI compatibility between QEMU and the kernel. The kvm_ppc_host_compat_pvr() function derives the effective PVR based on the compatibility mode reported by KVM. Additionally, cas_check_pvr() in hw/ppc/spapr_hcall.c is updated to prevent fallback to raw mode when the host is running in compatibility mode. This ensures that nested guests cannot exceed the host's compatibility level. If the capability is not supported or the query fails, the functions return 0, allowing fallback to existing behavior. Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com> --- hw/ppc/spapr_hcall.c | 14 ++++++++++ target/ppc/kvm.c | 65 ++++++++++++++++++++++++++++++++++++++++++++ target/ppc/kvm_ppc.h | 16 +++++++++++ 3 files changed, 95 insertions(+) diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index XXXXXXX..XXXXXXX 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -XXX,XX +XXX,XX @@ static uint32_t cas_check_pvr(PowerPCCPU *cpu, uint32_t max_compat, { bool explicit_match = false; /* Matched the CPU's real PVR */ uint32_t best_compat = 0; + uint32_t compat_host_pvr = 0; int i; /* @@ -XXX,XX +XXX,XX @@ static uint32_t cas_check_pvr(PowerPCCPU *cpu, uint32_t max_compat, } } + if (explicit_match) { + compat_host_pvr = kvm_ppc_host_compat_pvr(); + /* + * If the host is booted in a compatibility mode, do not try booting in + * the raw mode as it may allow KVM guests to boot with a higher CPU + * version compared to what host was booted with; which should not be + * allowed. + */ + if (compat_host_pvr) { + explicit_match = false; + } + } + *raw_mode_supported = explicit_match; /* Parsing finished */ diff --git a/target/ppc/kvm.c b/target/ppc/kvm.c index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/kvm.c +++ b/target/ppc/kvm.c @@ -XXX,XX +XXX,XX @@ bool kvmppc_supports_ail_3(void) return cap_ail_mode_3; } +#if defined(TARGET_PPC64) +static target_ulong kvmppc_get_compat_caps(void) +{ + struct kvm_ppc_compat_caps host_compat; + target_ulong host_caps; + int ret; + + if (!kvm_check_extension(kvm_state, KVM_CAP_PPC_COMPAT_CAPS)) { + return 0; + } + + /* Initialize the structure with size field for forward compatibility */ + memset(&host_compat, 0, sizeof(host_compat)); + host_compat.size = sizeof(host_compat); + + ret = kvm_vm_ioctl(kvm_state, KVM_PPC_GET_COMPAT_CAPS, &host_compat); + if (ret < 0) { + fprintf(stderr, "KVM: failed to get host capabilities\n"); + return 0; + } + + /* + * Validate the returned size matches our structure size. + * The kernel validates that userspace provides sufficient size before + * the ioctl, and returns its own structure size. A mismatch indicates + * a version incompatibility between QEMU and the kernel. + */ + if (host_compat.size != sizeof(host_compat)) { + fprintf(stderr, "KVM: host_compat size mismatch (expected %zu, got %lu)\n", + sizeof(host_compat), host_compat.size); + return 0; + } + + host_caps = host_compat.compat_capabilities; + return host_caps; +} + +uint32_t kvm_ppc_host_compat_pvr(void) +{ + uint32_t compat_host_pvr = 0; + int cap_idx = 0; + target_ulong host_caps = kvmppc_get_compat_caps(); + + host_caps = host_caps & KVM_PPC_COMPAT_BITMASK; + if (host_caps) { + cap_idx = 63 - __builtin_ctzll(host_caps); + switch (cap_idx) { + case KVM_PPC_COMPAT_CAP_P9_IDX: + compat_host_pvr = CPU_POWERPC_POWER9_DD22; + break; + case KVM_PPC_COMPAT_CAP_P10_IDX: + compat_host_pvr = CPU_POWERPC_POWER10_DD20; + break; + case KVM_PPC_COMPAT_CAP_P11_IDX: + compat_host_pvr = CPU_POWERPC_POWER11_DD20; + break; + default: + break; + } + } + + return compat_host_pvr; +} +#endif /* TARGET_PPC64 */ + PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void) { uint32_t host_pvr = mfpvr(); diff --git a/target/ppc/kvm_ppc.h b/target/ppc/kvm_ppc.h index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/kvm_ppc.h +++ b/target/ppc/kvm_ppc.h @@ -XXX,XX +XXX,XX @@ bool kvmppc_supports_ail_3(void); int kvmppc_enable_hwrng(void); int kvmppc_put_books_sregs(PowerPCCPU *cpu); PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void); + +/* + * Bit position indices for KVM_PPC_COMPAT_CAP_* capabilities. + * These represent the result of (63 - bit_number) for each capability bit. + * Used for identifying which compatibility mode is supported by the host. + */ +#define KVM_PPC_COMPAT_CAP_P9_IDX 1 /* 63 - 62 */ +#define KVM_PPC_COMPAT_CAP_P10_IDX 2 /* 63 - 61 */ +#define KVM_PPC_COMPAT_CAP_P11_IDX 3 /* 63 - 60 */ + +uint32_t kvm_ppc_host_compat_pvr(void); void kvmppc_check_papr_resize_hpt(Error **errp); int kvmppc_resize_hpt_prepare(PowerPCCPU *cpu, target_ulong flags, int shift); int kvmppc_resize_hpt_commit(PowerPCCPU *cpu, target_ulong flags, int shift); @@ -XXX,XX +XXX,XX @@ static inline PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void) return NULL; } +static inline uint32_t kvm_ppc_host_compat_pvr(void) +{ + return 0; +} + static inline void kvmppc_check_papr_resize_hpt(Error **errp) { } -- 2.50.1 (Apple Git-155)
On POWER systems, the host CPU may run in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). When running nested KVM guests, QEMU currently derives the host CPU type using mfpvr(), which reflects the physical processor version. This can result in a mismatch between the CPU model used by QEMU and the compatibility mode enforced by the host, leading to guest boot failures such as "KVM-NESTEDv2: couldn't set guest wide elements". Update kvm_ppc_get_host_cpu_class() to check if the host is running in a compatibility mode using kvm_ppc_host_compat_pvr(). When available, use the compatibility PVR instead of the raw hardware PVR when selecting the CPU model. This ensures that QEMU selects a CPU model consistent with the host compatibility mode, allowing nested guests to boot correctly. Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com> --- target/ppc/kvm.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/target/ppc/kvm.c b/target/ppc/kvm.c index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/kvm.c +++ b/target/ppc/kvm.c @@ -XXX,XX +XXX,XX @@ PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void) uint32_t host_pvr = mfpvr(); PowerPCCPUClass *pvr_pcc; +#if defined(TARGET_PPC64) + uint32_t compat_host_pvr; + + compat_host_pvr = kvm_ppc_host_compat_pvr(); + if (compat_host_pvr) { + host_pvr = compat_host_pvr; + } +#endif /* TARGET_PPC64 */ + pvr_pcc = ppc_cpu_class_by_pvr(host_pvr); if (pvr_pcc == NULL) { pvr_pcc = ppc_cpu_class_by_pvr_mask(host_pvr); -- 2.50.1 (Apple Git-155)
On POWER systems, newer processor generations can operate in compatibility modes corresponding to earlier generations (e.g., a Power11 system running in Power10 compatibility mode). In such cases, the effective CPU level exposed to guests differs from the physical processor generation. This creates issues for nested virtualization. When booting a nested KVM guest, QEMU may derive the CPU model from the raw hardware PVR and attempt to configure the guest accordingly. However, the host is constrained by the compatibility level negotiated with the hypervisor, and requests exceeding that level are rejected by KVM, leading to guest boot failures such as: KVM-NESTEDv2: couldn't set guest wide elements This series addresses the issue by preventing fallback to raw mode when the host itself is booted in a compatibility mode, and by querying the effective CPU compatibility modes supported by the host via KVM. The kernel interface uses copy_struct_from/to_user() for forward and backward ABI compatibility. With these changes, QEMU ensures that nested guests are configured with CPU models consistent with the host compatibility mode, allowing them to boot correctly. Patch summary: [1/3] [DO_NOT_MERGE] linux-headers: Add uapi header changes [2/3] target/ppc/kvm: Add support for querying host compatibility mode [3/3] target/ppc/kvm: Use host compatibility mode for nested guests Changes in v5: - Patch 1: Updated KVM_PPC_GET_COMPAT_CAPS ioctl number from 0xe4 to 0xb8 to match the corresponding Linux kernel v6 series change; the 0xe0-0xe3 range is reserved for KVM_CREATE_DEVICE fd ioctls Testing (with kernel v6 patches): KVM APIv1 Testing ================= On P10 PowerNV machine (L0) --------------------------- - P10 L1 KVM guest -> works - P10 nested L2 KVM guest -> works - P9 compat nested L2 KVM guest -> works - P9 compat L1 KVM guest -> works - P9 nested L2 KVM guest -> works On Powernv11 TCG Guest (L0) --------------------------- - P11 PowerNV TCG L0 guest -> works - P11 L1 KVM guest -> works - P11 L2 KVM guest -> works - P10 compat L1 KVM guest -> works - P10 L2 KVM guest -> works - P9 compat L1 KVM guest -> works - P9 L2 KVM guest -> works KVM APIv2 Testing ================= On P11 PowerVM LPAR (L1) ------------------------ - P11 L2 KVM guest -> works - P10 compat L2 KVM guest -> works - P9 compat L2 KVM guest fails to boot as expected - Without QEMU patches but Linux patches - P11 L2 KVM guest -> works - P10 compat L2 KVM guest -> works - P9 compat L2 KVM guest fails to boot as expected - Without Linux patches but QEMU patches - P11 L2 KVM guest -> works - P10 compat L2 KVM guest -> works On P11 LPAR in P10 compat (L1) ------------------------------ - P10 (host compat) L2 KVM guest -> works - Without QEMU patch but Linux patches - P10 guest fails to boot as expected (error: kvm run failed Invalid argument) - Without Linux patch but QEMU patches - P10 guest fails to boot as expected (KVM: unknown exit, hardware reason ffffffffffffffea) On P10 PowerVM LPAR (L1) ------------------------ - P10 L2 KVM guest -> works - P9 compat L2 KVM guest fails to boot as expected TCG pSeries Guest ================= - P11 (default) pSeries guest boots fine ABI Extensibility Testing (struct size 32, extra member) ========================================================= - Newer struct on QEMU, older kernel -> works (kernel returns -E2BIG, QEMU retries with correct size) - New struct on Linux kernel, older QEMU -> works (kernel zero-pads trailing fields, QEMU gets correct data) Note: Patch 1 is marked DO_NOT_MERGE as it contains linux-headers updates that will be synced separately once the corresponding kernel patches are merged. The corresponding Linux kernel patches (v6) are being posted concurrently. v4: https://lore.kernel.org/all/20260701052341.62289-1-amachhiw@linux.ibm.com/ v3: https://lore.kernel.org/all/20260616113915.25589-1-amachhiw@linux.ibm.com/ v2: https://lore.kernel.org/all/20260502140021.69712-1-amachhiw@linux.ibm.com/ v1: https://lore.kernel.org/all/20260430061333.37905-1-amachhiw@linux.ibm.com/ Previous kernel patch versions: v6: https://lore.kernel.org/all/20260804180705.59160-1-amachhiw@linux.ibm.com/ v5: https://lore.kernel.org/all/20260701051409.51820-1-amachhiw@linux.ibm.com/ v4: https://lore.kernel.org/all/20260616123314.82721-1-amachhiw@linux.ibm.com/ v3: https://lore.kernel.org/all/20260522152744.55251-1-amachhiw@linux.ibm.com/ v2: https://lore.kernel.org/all/20260513100755.83195-1-amachhiw@linux.ibm.com/ v1: https://lore.kernel.org/all/20260430054906.94401-1-amachhiw@linux.ibm.com/ Amit Machhiwal (3): [DO_NOT_MERGE] linux-headers: Add uapi header changes target/ppc/kvm: Add support for querying host compatibility mode target/ppc/kvm: Use host compatibility mode for nested guests hw/ppc/spapr_hcall.c | 14 ++++++ linux-headers/asm-powerpc/kvm.h | 19 +++++++ linux-headers/linux/kvm.h | 3 ++ target/ppc/kvm.c | 87 +++++++++++++++++++++++++++++++++ target/ppc/kvm_ppc.h | 7 +++ 5 files changed, 130 insertions(+) base-commit: b428fe036233cbd15d37e3c027ab6ca4d3661a80 -- 2.50.1 (Apple Git-155)
This is a temporary patch intended for review and testing purposes only. It syncs the QEMU linux-headers with the kernel v6 changes that introduce the KVM_PPC_GET_COMPAT_CAPS ioctl for querying host CPU compatibility capabilities. The struct kvm_ppc_compat_caps places 'size' as the first field as required by copy_struct_from/to_user() versioning, with KVM_PPC_COMPAT_CAPS_SIZE_VER0 (24) defined as the frozen version-floor constant. The capability number is KVM_CAP_PPC_COMPAT_CAPS (250) and the ioctl is defined as _IO so the ioctl number remains stable if the struct grows in future versions. Capability bit definitions for POWER9, POWER10, and POWER11 compatibility modes are also included. The actual header sync will be done via scripts/update-linux-headers.sh once the kernel changes [1] are merged upstream. [1] https://lore.kernel.org/all/20260804180705.59160-1-amachhiw@linux.ibm.com/ Tested-by: Gautam Menghani <gautam@linux.ibm.com> Reviewed-by: Gautam Menghani <gautam@linux.ibm.com> Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com> --- Changes in this version: - Updated KVM_PPC_GET_COMPAT_CAPS ioctl number from 0xe4 to 0xb8 to match the corresponding Linux kernel v6 series change; the 0xe0-0xe3 range is reserved for KVM_CREATE_DEVICE fd ioctls linux-headers/asm-powerpc/kvm.h | 19 +++++++++++++++++++ linux-headers/linux/kvm.h | 3 +++ 2 files changed, 22 insertions(+) diff --git a/linux-headers/asm-powerpc/kvm.h b/linux-headers/asm-powerpc/kvm.h index XXXXXXX..XXXXXXX 100644 --- a/linux-headers/asm-powerpc/kvm.h +++ b/linux-headers/asm-powerpc/kvm.h @@ -XXX,XX +XXX,XX @@ struct kvm_ppc_cpu_char { __u64 behaviour_mask; /* valid bits in behaviour */ }; +/* For KVM_PPC_GET_COMPAT_CAPS */ +struct kvm_ppc_compat_caps { + __u64 size; /* Size of this structure */ + __u64 flags; /* Reserved for future use */ + __u64 compat_capabilities; /* Capabilities supported by the host */ +}; +#define KVM_PPC_COMPAT_CAPS_SIZE_VER0 24 /* sizeof first published struct */ + +/* + * Capability bits for compat_capabilities field in kvm_ppc_compat_caps. + * These bits indicate which processor compatibility modes are supported. + */ +#define KVM_PPC_COMPAT_CAP_POWER9 (1ULL << 62) +#define KVM_PPC_COMPAT_CAP_POWER10 (1ULL << 61) +#define KVM_PPC_COMPAT_CAP_POWER11 (1ULL << 60) +#define KVM_PPC_COMPAT_BITMASK (KVM_PPC_COMPAT_CAP_POWER9 | \ + KVM_PPC_COMPAT_CAP_POWER10 | \ + KVM_PPC_COMPAT_CAP_POWER11) + /* * Values for character and character_mask. * These are identical to the values used by H_GET_CPU_CHARACTERISTICS. diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h index XXXXXXX..XXXXXXX 100644 --- a/linux-headers/linux/kvm.h +++ b/linux-headers/linux/kvm.h @@ -XXX,XX +XXX,XX @@ struct kvm_enable_cap { #define KVM_CAP_S390_KEYOP 247 #define KVM_CAP_S390_VSIE_ESAMODE 248 #define KVM_CAP_S390_HPAGE_2G 249 +#define KVM_CAP_PPC_COMPAT_CAPS 250 struct kvm_irq_routing_irqchip { __u32 irqchip; @@ -XXX,XX +XXX,XX @@ struct kvm_s390_keyop { /* Available with KVM_CAP_COUNTER_OFFSET */ #define KVM_ARM_SET_COUNTER_OFFSET _IOW(KVMIO, 0xb5, struct kvm_arm_counter_offset) #define KVM_ARM_GET_REG_WRITABLE_MASKS _IOR(KVMIO, 0xb6, struct reg_mask_range) +/* Available with KVM_CAP_PPC_COMPAT_CAPS */ +#define KVM_PPC_GET_COMPAT_CAPS _IO(KVMIO, 0xb8) /* ioctl for vm fd */ #define KVM_CREATE_DEVICE _IOWR(KVMIO, 0xe0, struct kvm_create_device) -- 2.50.1 (Apple Git-155)
Add infrastructure to query the host CPU compatibility mode via the KVM_PPC_GET_COMPAT_CAPS ioctl. This allows QEMU to determine if the host is running in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). The kvmppc_get_compat_caps() function issues the ioctl and returns the compat_capabilities bitmap. The kvm_ppc_host_compat_pvr() function derives the effective PVR from the bitmap using ctz64() to find the lowest set bit (highest supported compat level in IBM MSB-0 numbering). The struct kvm_ppc_compat_caps places 'size' first and userspace sets it to sizeof(struct kvm_ppc_compat_caps) before calling the ioctl. The kernel uses copy_struct_from/to_user() to handle forward and backward ABI compatibility: an older userspace with a smaller struct gets trailing fields zero-padded. When newer userspace passes a larger struct to an older kernel (usize > ksize), the kernel unconditionally returns -E2BIG and writes its own ksize back into host_compat.size. QEMU detects this, validates the returned size against KVM_PPC_COMPAT_CAPS_SIZE_VER0, and retries with that size. Additionally, cas_check_pvr() in hw/ppc/spapr_hcall.c is updated to prevent fallback to raw mode when the host is running in compatibility mode. This ensures that nested guests cannot exceed the host's compatibility level. The call is guarded with kvm_enabled() since kvm_ppc_host_compat_pvr() invokes kvm_vm_ioctl() which dereferences kvm_state; without the guard, a TCG guest on a CONFIG_KVM=y binary would segfault. If the capability is not supported or the query fails, the functions return 0, allowing fallback to existing behavior. Tested-by: Gautam Menghani <gautam@linux.ibm.com> Reviewed-by: Gautam Menghani <gautam@linux.ibm.com> Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com> --- No changes in this version. hw/ppc/spapr_hcall.c | 14 +++++++++ target/ppc/kvm.c | 75 ++++++++++++++++++++++++++++++++++++++++++++ target/ppc/kvm_ppc.h | 7 +++++ 3 files changed, 96 insertions(+) diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index XXXXXXX..XXXXXXX 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -XXX,XX +XXX,XX @@ static uint32_t cas_check_pvr(PowerPCCPU *cpu, uint32_t max_compat, { bool explicit_match = false; /* Matched the CPU's real PVR */ uint32_t best_compat = 0; + uint32_t compat_host_pvr = 0; int i; /* @@ -XXX,XX +XXX,XX @@ static uint32_t cas_check_pvr(PowerPCCPU *cpu, uint32_t max_compat, } } + if (explicit_match && kvm_enabled()) { + compat_host_pvr = kvm_ppc_host_compat_pvr(); + /* + * If the host is booted in a compatibility mode, do not try booting in + * the raw mode as it may allow KVM guests to boot with a higher CPU + * version compared to what host was booted with; which should not be + * allowed. + */ + if (compat_host_pvr) { + explicit_match = false; + } + } + *raw_mode_supported = explicit_match; /* Parsing finished */ diff --git a/target/ppc/kvm.c b/target/ppc/kvm.c index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/kvm.c +++ b/target/ppc/kvm.c @@ -XXX,XX +XXX,XX @@ bool kvmppc_supports_ail_3(void) return cap_ail_mode_3; } +#if defined(TARGET_PPC64) +static target_ulong kvmppc_get_compat_caps(void) +{ + struct kvm_ppc_compat_caps host_compat; + int ret; + + if (!kvm_check_extension(kvm_state, KVM_CAP_PPC_COMPAT_CAPS)) { + return 0; + } + + /* + * Set size to sizeof(struct kvm_ppc_compat_caps) so the kernel applies + * copy_struct_from/to_user() versioning. size must be >= VER0. + */ + memset(&host_compat, 0, sizeof(host_compat)); + host_compat.size = sizeof(host_compat); + + ret = kvm_vm_ioctl(kvm_state, KVM_PPC_GET_COMPAT_CAPS, &host_compat); + if (ret == -E2BIG && host_compat.size >= KVM_PPC_COMPAT_CAPS_SIZE_VER0) { + /* + * Kernel is older and knows only a smaller struct version. It + * wrote back its ksize into host_compat.size. Retry with that + * size so the kernel accepts the call. + * + * When a VER1 struct is introduced, add a check here: + * if (host_compat.size >= KVM_PPC_COMPAT_CAPS_SIZE_VER1) { ... } + */ + uint64_t ksize = host_compat.size; + memset(&host_compat, 0, sizeof(host_compat)); + host_compat.size = ksize; + ret = kvm_vm_ioctl(kvm_state, KVM_PPC_GET_COMPAT_CAPS, &host_compat); + } + + if (ret < 0) { + error_report("KVM: failed to get host CPU compat capabilities: %s", + strerror(-ret)); + return 0; + } + + return host_compat.compat_capabilities & KVM_PPC_COMPAT_BITMASK; +} + +/* + * Return the effective host PVR based on the CPU compatibility mode + * reported by KVM. Returns 0 if no compat mode is active or the + * capability is not supported, in which case the caller falls back + * to the raw hardware PVR. + */ +uint32_t kvm_ppc_host_compat_pvr(void) +{ + uint32_t compat_host_pvr = 0; + uint64_t cap_idx = 0; + target_ulong host_caps = kvmppc_get_compat_caps(); + + if (host_caps) { + cap_idx = 1ULL << ctz64(host_caps); + switch (cap_idx) { + case KVM_PPC_COMPAT_CAP_POWER9: + compat_host_pvr = CPU_POWERPC_POWER9_DD22; + break; + case KVM_PPC_COMPAT_CAP_POWER10: + compat_host_pvr = CPU_POWERPC_POWER10_DD20; + break; + case KVM_PPC_COMPAT_CAP_POWER11: + compat_host_pvr = CPU_POWERPC_POWER11_DD20; + break; + default: + break; + } + } + + return compat_host_pvr; +} +#endif /* TARGET_PPC64 */ + PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void) { uint32_t host_pvr = mfpvr(); diff --git a/target/ppc/kvm_ppc.h b/target/ppc/kvm_ppc.h index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/kvm_ppc.h +++ b/target/ppc/kvm_ppc.h @@ -XXX,XX +XXX,XX @@ bool kvmppc_supports_ail_3(void); int kvmppc_enable_hwrng(void); int kvmppc_put_books_sregs(PowerPCCPU *cpu); PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void); + +uint32_t kvm_ppc_host_compat_pvr(void); void kvmppc_check_papr_resize_hpt(Error **errp); int kvmppc_resize_hpt_prepare(PowerPCCPU *cpu, target_ulong flags, int shift); int kvmppc_resize_hpt_commit(PowerPCCPU *cpu, target_ulong flags, int shift); @@ -XXX,XX +XXX,XX @@ static inline PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void) return NULL; } +static inline uint32_t kvm_ppc_host_compat_pvr(void) +{ + return 0; +} + static inline void kvmppc_check_papr_resize_hpt(Error **errp) { } -- 2.50.1 (Apple Git-155)
On POWER systems, the host CPU may run in a compatibility mode (e.g., a Power11 processor operating in Power10 compatibility mode). When running nested KVM guests, QEMU currently derives the host CPU type using mfpvr(), which reflects the physical processor version. This can result in a mismatch between the CPU model used by QEMU and the compatibility mode enforced by the host, leading to guest boot failures such as "KVM-NESTEDv2: couldn't set guest wide elements". Update kvm_ppc_get_host_cpu_class() to check if the host is running in a compatibility mode using kvm_ppc_host_compat_pvr(). When available, use the compatibility PVR instead of the raw hardware PVR when selecting the CPU model. This ensures that QEMU selects a CPU model consistent with the host compatibility mode, allowing nested guests to boot correctly. The guard uses #if defined(TARGET_PPC64) to prevent build breakage on ppc32 targets where the POWER9/10/11 PVR constants are not defined. Tested-by: Gautam Menghani <gautam@linux.ibm.com> Reviewed-by: Gautam Menghani <gautam@linux.ibm.com> Signed-off-by: Amit Machhiwal <amachhiw@linux.ibm.com> --- No changes in this version. target/ppc/kvm.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/target/ppc/kvm.c b/target/ppc/kvm.c index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/kvm.c +++ b/target/ppc/kvm.c @@ -XXX,XX +XXX,XX @@ PowerPCCPUClass *kvm_ppc_get_host_cpu_class(void) uint32_t host_pvr = mfpvr(); PowerPCCPUClass *pvr_pcc; +#if defined(TARGET_PPC64) +#ifndef CONFIG_KVM +#error "CONFIG_KVM is not enabled" +#endif + uint32_t compat_host_pvr; + + compat_host_pvr = kvm_ppc_host_compat_pvr(); + if (compat_host_pvr) { + host_pvr = compat_host_pvr; + } +#endif /* TARGET_PPC64 */ + pvr_pcc = ppc_cpu_class_by_pvr(host_pvr); if (pvr_pcc == NULL) { pvr_pcc = ppc_cpu_class_by_pvr_mask(host_pvr); -- 2.50.1 (Apple Git-155)