From nobody Sun Sep 28 17:06:23 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1757598283; cv=none; d=zohomail.com; s=zohoarc; b=byOOZ9wjqoFj5I12wbr74rBNkWFo1Mb5M8Kz/UZQSVh+RBBqdxs3Fv23A3tkaX1LkRZSJdLHcfOT+j+5fwqVvJV6gJ72+bsMrp28Y/41Gmlf88NkELg/RlveoJEW+SmWKQ3nn3UWjXEuiLo6eN8YXvzEfsDLCqNiyTpHuFvqp1o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757598283; h=Content-Type:Content-Transfer-Encoding:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=iJE52WELL8ju9OR0d+WZwTwi5R18JbMSBCtMSDbor+8=; b=cfmjEyQEz5THLWcYftXQUasxhwICm9bkCvsm5CNpCfUUbbgvRdF2/DdNOW5O9wYXYsISN3MDpAHDV7qdb79i3qkq/fmHeXn+Rob7go3ca1C3Y9EzmvGT3v86lQGm/tV7Ay4D6DPU/hILZN5kUw3WtcNqpDm9YSBanR5Nar1UiTI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757598283242474.1055006367284; Thu, 11 Sep 2025 06:44:43 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwhaZ-0003aL-8Q; Thu, 11 Sep 2025 09:43:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwhaV-0003Yy-Ef for qemu-devel@nongnu.org; Thu, 11 Sep 2025 09:43:47 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwhaR-0003V6-PX for qemu-devel@nongnu.org; Thu, 11 Sep 2025 09:43:46 -0400 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-385-maa4L9mdOLudpuotnqpvuA-1; Thu, 11 Sep 2025 09:43:37 -0400 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A1683180057E; Thu, 11 Sep 2025 13:43:35 +0000 (UTC) Received: from laptop.redhat.com (unknown [10.45.224.190]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 174AB1944CEB; Thu, 11 Sep 2025 13:43:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1757598221; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iJE52WELL8ju9OR0d+WZwTwi5R18JbMSBCtMSDbor+8=; b=YS1qntHK30Yot7E+h2eRTi8Ie4KvqwhCPMVQze01/TwKPo4qlVIi4s1m6kUl77ca4mXvOq UfQM6eTDWX3HXQiCQ3div2ZHhXezwKZ2DGClPO4TO0kWwl4oPRrQ0kE+Bo4RwzDHIhD/IS iVZUejIk2aoxgrtUTRU0L4N53pglKEs= X-MC-Unique: maa4L9mdOLudpuotnqpvuA-1 X-Mimecast-MFC-AGG-ID: maa4L9mdOLudpuotnqpvuA_1757598216 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, qemu-devel@nongnu.org, qemu-arm@nongnu.org, peter.maydell@linaro.org, cohuck@redhat.com, maz@kernel.org, oliver.upton@linux.dev, sebott@redhat.com, gshan@redhat.com Subject: [RFC 1/3] target/arm/cpu: Add new CPU property for KVM regs to hide Date: Thu, 11 Sep 2025 15:40:27 +0200 Message-ID: <20250911134324.3702720-2-eric.auger@redhat.com> In-Reply-To: <20250911134324.3702720-1-eric.auger@redhat.com> References: <20250911134324.3702720-1-eric.auger@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1757598285684116600 New kernels sometimes expose new registers in an unconditionnal manner. This situation breaks backward migration as qemu notices there are more registers to store on guest than supported in the destination kerenl. This leads to a "failed to load cpu:cpreg_vmstate_array_len" error. A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register in v6.16 by commit C0000e58c74e (=E2=80=9CKVM: arm64: Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2=E2=80=9D). Trying to do backward migration from a host kernel which features the commit to a destination host that doesn't fail. Currently QEMU is not using that feature so ignoring this latter is not a problem. An easy way to fix the migration issue is to teach qemu we don't care about that register and we can simply ignore it, including its state migration. This patch introduces a CPU property, under the form of an array of reg indices which indicates which registers can be ignored. The goal then is to set this property in machine type compats such as: static GlobalProperty arm_virt_kernel_compat_10_1[] =3D { /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */ { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" }, } Signed-off-by: Eric Auger Reviewed-by: Sebastian Ott --- target/arm/cpu.h | 4 ++++ target/arm/kvm.c | 36 ++++++++++++++++++++++++++++++++++-- target/arm/trace-events | 2 ++ 3 files changed, 40 insertions(+), 2 deletions(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index c15d79a106..121b4372b2 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1031,6 +1031,10 @@ struct ArchCPU { /* KVM steal time */ OnOffAuto kvm_steal_time; =20 + /* KVM registers that must be ignored/hidden */ + uint64_t *kvm_hidden_regs; + uint32_t nr_kvm_hidden_regs; + /* Uniprocessor system with MP extensions */ bool mp_is_up; =20 diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 6672344855..67675781f4 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -39,6 +39,8 @@ #include "qemu/log.h" #include "hw/acpi/acpi.h" #include "hw/acpi/ghes.h" +#include "hw/qdev-properties.h" +#include "hw/qdev-properties-system.h" #include "target/arm/gtimer.h" #include "migration/blocker.h" =20 @@ -483,6 +485,10 @@ static void kvm_steal_time_set(Object *obj, bool value= , Error **errp) ARM_CPU(obj)->kvm_steal_time =3D value ? ON_OFF_AUTO_ON : ON_OFF_AUTO_= OFF; } =20 +static const Property arm_cpu_kvm_compat_hidden_regs_property =3D + DEFINE_PROP_ARRAY("kvm-hidden-regs", ARMCPU, + nr_kvm_hidden_regs, kvm_hidden_regs, qdev_prop_uint6= 4, uint64_t); + /* KVM VCPU properties should be prefixed with "kvm-". */ void kvm_arm_add_vcpu_properties(ARMCPU *cpu) { @@ -504,6 +510,8 @@ void kvm_arm_add_vcpu_properties(ARMCPU *cpu) kvm_steal_time_set); object_property_set_description(obj, "kvm-steal-time", "Set off to disable KVM steal time."); + + qdev_property_add_static(DEVICE(obj), &arm_cpu_kvm_compat_hidden_regs_= property); } =20 bool kvm_arm_pmu_supported(void) @@ -764,6 +772,26 @@ static bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t = regidx) } } =20 +/** + * kvm_vcpu_compat_hidden_reg: + * @cpu: ARMCPU + * @regidx: index of the register to check + * + * Depending on the CPU compat returns true if @regidx must be + * ignored during sync & migration + */ +static inline bool +kvm_vcpu_compat_hidden_reg(ARMCPU *cpu, uint64_t regidx) +{ + for (int i =3D 0; i < cpu->nr_kvm_hidden_regs; i++) { + if (cpu->kvm_hidden_regs[i] =3D=3D regidx) { + trace_kvm_vcpu_compat_hidden_reg(regidx); + return true; + } + } + return false; +} + /** * kvm_arm_init_cpreg_list: * @cpu: ARMCPU @@ -798,7 +826,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) qsort(&rlp->reg, rlp->n, sizeof(rlp->reg[0]), compare_u64); =20 for (i =3D 0, arraylen =3D 0; i < rlp->n; i++) { - if (!kvm_arm_reg_syncs_via_cpreg_list(rlp->reg[i])) { + if (!kvm_arm_reg_syncs_via_cpreg_list(rlp->reg[i]) || + kvm_vcpu_compat_hidden_reg(cpu, rlp->reg[i])) { continue; } switch (rlp->reg[i] & KVM_REG_SIZE_MASK) { @@ -814,6 +843,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) arraylen++; } =20 + trace_kvm_arm_init_cpreg_list_arraylen(arraylen); + cpu->cpreg_indexes =3D g_renew(uint64_t, cpu->cpreg_indexes, arraylen); cpu->cpreg_values =3D g_renew(uint64_t, cpu->cpreg_values, arraylen); cpu->cpreg_vmstate_indexes =3D g_renew(uint64_t, cpu->cpreg_vmstate_in= dexes, @@ -825,7 +856,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) =20 for (i =3D 0, arraylen =3D 0; i < rlp->n; i++) { uint64_t regidx =3D rlp->reg[i]; - if (!kvm_arm_reg_syncs_via_cpreg_list(regidx)) { + if (!kvm_arm_reg_syncs_via_cpreg_list(regidx) || + kvm_vcpu_compat_hidden_reg(cpu, regidx)) { continue; } cpu->cpreg_indexes[arraylen] =3D regidx; diff --git a/target/arm/trace-events b/target/arm/trace-events index 4438dce7be..1b4ab0c683 100644 --- a/target/arm/trace-events +++ b/target/arm/trace-events @@ -13,3 +13,5 @@ arm_gt_update_irq(int timer, int irqstate) "gt_update_irq= : timer %d irqstate %d" =20 # kvm.c kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova =3D 0x%"PRI= x64" is translated into 0x%"PRIx64 +kvm_arm_init_cpreg_list_arraylen(uint32_t arraylen) "arraylen=3D%d" +kvm_vcpu_compat_hidden_reg(uint64_t regidx) "0x%"PRIx64" is hidden" --=20 2.49.0 From nobody Sun Sep 28 17:06:23 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1757598333; cv=none; d=zohomail.com; s=zohoarc; b=lu/UamuLADsJvdStkmbAGVCgVKMabgymqQj6mwdXakYgHwp2BVe3p42tc4sHBvjRIXJ5vRfUn5Fz8GdyTtoHvxOwEwSnc3hcsfyy7WixT+kZ+LolvMV3r+WrAgn9uIf4+R2v+xiD9uoSl7TyPpfb0c12S6dOq3wGIlWHblfbmX0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757598333; h=Content-Type:Content-Transfer-Encoding:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=09+9glxCNez7qUz7UASruAF+e/F/Ri1H0Rleipm38aA=; b=UkU2z/n8cUUt0WgYs+OIVgZ79aM5NtvP1RqXYZxzDRUe/KprRsRWXm0DHM/4RYzLO2Se2v8lj4GYWMHvkPxHsJMyv5T0jeGkkMt8+1q7ZFI9O1cVw8SN7HbJnbNMsdfFkiqiTzAIebb3ihIhvRK4T1NMojLfdYuUxT2SjM5KXyo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757598333585526.2064259332061; Thu, 11 Sep 2025 06:45:33 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwhaa-0003b8-6F; Thu, 11 Sep 2025 09:43:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwhaW-0003Z5-RR for qemu-devel@nongnu.org; Thu, 11 Sep 2025 09:43:48 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwhaS-0003VU-Bm for qemu-devel@nongnu.org; Thu, 11 Sep 2025 09:43:47 -0400 Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-103-PeP8B2FRPgiOmznj6r2tjw-1; Thu, 11 Sep 2025 09:43:41 -0400 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B473C1953947; Thu, 11 Sep 2025 13:43:39 +0000 (UTC) Received: from laptop.redhat.com (unknown [10.45.224.190]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2C24B1944CEB; Thu, 11 Sep 2025 13:43:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1757598223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=09+9glxCNez7qUz7UASruAF+e/F/Ri1H0Rleipm38aA=; b=fTgU4zPgp2/KP8sUNtuH2+4/TFlCKo2a8h1ar8pJX031rDlT7Yacq77YbhNmkWBYiu341b B23+IOLXBYLJm+lZYquFiW1S0wlROcrg5dC9jG88Zp5FQAa2ng9PMJijmmCS1OohfSEa7Z 3IsgWgizGY1f2W9urvUVw20nivC14NA= X-MC-Unique: PeP8B2FRPgiOmznj6r2tjw-1 X-Mimecast-MFC-AGG-ID: PeP8B2FRPgiOmznj6r2tjw_1757598219 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, qemu-devel@nongnu.org, qemu-arm@nongnu.org, peter.maydell@linaro.org, cohuck@redhat.com, maz@kernel.org, oliver.upton@linux.dev, sebott@redhat.com, gshan@redhat.com Subject: [RFC 2/3] target/arm/kvm: Add new CPU property for KVM regs to enforce Date: Thu, 11 Sep 2025 15:40:28 +0200 Message-ID: <20250911134324.3702720-3-eric.auger@redhat.com> In-Reply-To: <20250911134324.3702720-1-eric.auger@redhat.com> References: <20250911134324.3702720-1-eric.auger@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1757598336445116600 Newer kernels sometimes revokes exposure of KVM regs to userspace. This can happen if some registers were unconditionnally exposed, by mistake, in previous versions and then conditionnally exposed. Recently this happened with 3 registers: TCR2_EL1, PIRE0_EL1, PIR_EL1. The associated kernel commit were: 0fcb4eea5345=C2=A0 KVM: arm64: Hide TCR2_EL1 from userspace when disabled f= or guests a68cddbe47ef=C2=A0 KVM: arm64: Hide S1PIE registers from userspace when dis= abled for guests Those commits were actual fixes but the cons is that is breaks forward miration from old to new host kernels as the number of KVM regs exposed to userspace decreases and this leads to "failed to load cpu:cpreg_vmstate_= array_len" when migrating the CPU state. This patchs adds a new CPU property, under the form of an array of reg indices that teach (destination) QEMU that some registers must exist. If they don't (because KVM does not expose them anymore), qemu fakes them b= ut does not care of their state. This is meant to be applied as a cmahine type compat like: static GlobalProperty arm_virt_kernel_compat_10_1[] =3D { { TYPE_ARM_CPU, "kvm-enforced-regs", /* TCR_EL1, PIRE0_EL1, PIR_EL1 */ "0x603000000013c103, 0x603000000013c512, 0x603000000013c513" }, }; In case we migrate from a source where the KVM regs are not exposed and where regs are faked to a destination where KVM expose them, this means that anyway FEAT_TCR2 and S1PIE are not supported by the guest so it should not matter that dumb values are written on dest. But obviously introducing fake registers is a last resort solution. Signed-off-by: Eric Auger --- target/arm/cpu.h | 11 +++++++ target/arm/kvm.c | 63 +++++++++++++++++++++++++++++++++++++++-- target/arm/trace-events | 4 +++ 3 files changed, 76 insertions(+), 2 deletions(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 121b4372b2..ea9045e5ff 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1035,6 +1035,17 @@ struct ArchCPU { uint64_t *kvm_hidden_regs; uint32_t nr_kvm_hidden_regs; =20 + /* + * KVM registers whose presence must be enforced + * Either they must be exposed to user space by KVM or + * they must be faked for migration sake + */ + uint64_t *kvm_enforced_regs; + uint32_t nr_kvm_enforced_regs; + + /* registers among those to be enforced that are faked */ + uint64_t *kvm_fake_regs; + /* Uniprocessor system with MP extensions */ bool mp_is_up; =20 diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 67675781f4..f8c36ffa2f 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -489,6 +489,11 @@ static const Property arm_cpu_kvm_compat_hidden_regs_p= roperty =3D DEFINE_PROP_ARRAY("kvm-hidden-regs", ARMCPU, nr_kvm_hidden_regs, kvm_hidden_regs, qdev_prop_uint6= 4, uint64_t); =20 +static const Property arm_cpu_kvm_compat_enforced_regs_property =3D + DEFINE_PROP_ARRAY("kvm-enforced-regs", ARMCPU, + nr_kvm_enforced_regs, kvm_enforced_regs, + qdev_prop_uint64, uint64_t); + /* KVM VCPU properties should be prefixed with "kvm-". */ void kvm_arm_add_vcpu_properties(ARMCPU *cpu) { @@ -512,6 +517,7 @@ void kvm_arm_add_vcpu_properties(ARMCPU *cpu) "Set off to disable KVM steal time."); =20 qdev_property_add_static(DEVICE(obj), &arm_cpu_kvm_compat_hidden_regs_= property); + qdev_property_add_static(DEVICE(obj), &arm_cpu_kvm_compat_enforced_reg= s_property); } =20 bool kvm_arm_pmu_supported(void) @@ -772,6 +778,27 @@ static bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t = regidx) } } =20 +/** + * kvm_vcpu_compat_fake_reg: + * @cpu: ARMCPU + * @regidx: index of the register to check + * + * Depending on the CPU compat returns true if @regidx is a + * fake register that does not need any sync, false otherwise + */ +static inline bool +kvm_vcpu_compat_fake_reg(ARMCPU *cpu, uint64_t regidx) +{ + for (int i =3D 0; i < cpu->nr_kvm_enforced_regs; i++) { + if (cpu->kvm_fake_regs[i] && + cpu->kvm_enforced_regs[i] =3D=3D regidx) { + trace_kvm_vcpu_compat_fake_reg(regidx); + return true; + } + } + return false; +} + /** * kvm_vcpu_compat_hidden_reg: * @cpu: ARMCPU @@ -806,7 +833,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) { struct kvm_reg_list rl; struct kvm_reg_list *rlp; - int i, ret, arraylen; + int i, ret, arraylen, rln; + int nr_fake_regs =3D 0; CPUState *cs =3D CPU(cpu); =20 rl.n =3D 0; @@ -814,12 +842,34 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) if (ret !=3D -E2BIG) { return ret; } - rlp =3D g_malloc(sizeof(struct kvm_reg_list) + rl.n * sizeof(uint64_t)= ); + rln =3D rl.n + cpu->nr_kvm_enforced_regs; + rlp =3D g_malloc(sizeof(struct kvm_reg_list) + rln * sizeof(uint64_t)); rlp->n =3D rl.n; ret =3D kvm_vcpu_ioctl(cs, KVM_GET_REG_LIST, rlp); if (ret) { goto out; } + + trace_kvm_arm_init_cpreg_list(rlp->n, cpu->nr_kvm_enforced_regs); + + cpu->kvm_fake_regs =3D g_new0(uint64_t, cpu->nr_kvm_enforced_regs); + + for (int j =3D 0; j < cpu->nr_kvm_enforced_regs; j++) { + uint64_t v64; + int res; + + res =3D kvm_get_one_reg(cs, cpu->kvm_enforced_regs[j], &v64); + + if (res !=3D -ENOENT) { + trace_kvm_arm_init_cpreg_exposed(cpu->kvm_enforced_regs[j], v6= 4, res); + continue; + } + rlp->reg[j + rl.n] =3D cpu->kvm_enforced_regs[j]; + cpu->kvm_fake_regs[j] =3D true; + nr_fake_regs++; + } + rlp->n =3D rl.n + nr_fake_regs; + /* Sort the list we get back from the kernel, since cpreg_tuples * must be in strictly ascending order. */ @@ -912,6 +962,10 @@ bool write_kvmstate_to_list(ARMCPU *cpu) uint32_t v32; int ret; =20 + if (kvm_vcpu_compat_fake_reg(cpu, regidx)) { + continue; + } + switch (regidx & KVM_REG_SIZE_MASK) { case KVM_REG_SIZE_U32: ret =3D kvm_get_one_reg(cs, regidx, &v32); @@ -947,6 +1001,10 @@ bool write_list_to_kvmstate(ARMCPU *cpu, int level) continue; } =20 + if (kvm_vcpu_compat_fake_reg(cpu, regidx)) { + continue; + } + switch (regidx & KVM_REG_SIZE_MASK) { case KVM_REG_SIZE_U32: v32 =3D cpu->cpreg_values[i]; @@ -1654,6 +1712,7 @@ static void kvm_arch_set_eager_split_size(Object *obj= , Visitor *v, s->kvm_eager_split_size =3D value; } =20 + void kvm_arch_accel_class_init(ObjectClass *oc) { object_class_property_add(oc, "eager-split-size", "size", diff --git a/target/arm/trace-events b/target/arm/trace-events index 1b4ab0c683..57d2b6afd4 100644 --- a/target/arm/trace-events +++ b/target/arm/trace-events @@ -15,3 +15,7 @@ arm_gt_update_irq(int timer, int irqstate) "gt_update_irq= : timer %d irqstate %d" kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova =3D 0x%"PRI= x64" is translated into 0x%"PRIx64 kvm_arm_init_cpreg_list_arraylen(uint32_t arraylen) "arraylen=3D%d" kvm_vcpu_compat_hidden_reg(uint64_t regidx) "0x%"PRIx64" is hidden" +kvm_arm_init_cpreg_list(uint32_t kvm_regs, uint32_t fake_regs) "%d regs ex= posed by KVM, %d enforced regs" +kvm_vcpu_compat_fake_reg(uint64_t regidx) "0x%"PRIx64" is fake" +kvm_arm_init_cpreg_exposed(uint64_t regidx, uint64_t val, int ret) "enforc= ed reg 0x%"PRIx64" is already exposed by KVM: value=3D0x%"PRIx64 " ret=3D%d= : nothing to do" + --=20 2.49.0 From nobody Sun Sep 28 17:06:23 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1757598289; cv=none; d=zohomail.com; s=zohoarc; b=cpw5nLTcybsDYRL5dn+ASGtfxsBz/cvlsBQlkRKeNxVHWLP+CPMQW77Xsa3n03TLgZ18md+oGZtuRJ4AdWPAYkcL5glmx8K/wrwzShH7OfCrTSuxFhoWIQyBFpnV2luSkgQigN8SvNkS5q1X4igKJf8POvM51wugpFGCTwbThQw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757598289; h=Content-Transfer-Encoding:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=SJe/KYJAhOxAiHSyQRTinz6TLWYZ74JDhD11G1hMJxA=; b=FymPiB0lLoTjWIaqQvqDEb6dWxyOwFzuRo20+X4yAf+MdjhW30dvwmMnS3tmQ6xNcMKYtgnhi1cUvPk7lpNpyvuomeGEnLtQdZH9R6PqfKdt2MMzu9bMhJrif2Y6v6bEOcmZj2HKLwsb5JtVSBzGUCp2Mmc6HJAHrQrbVCxxjyg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757598289702906.6991523434896; Thu, 11 Sep 2025 06:44:49 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwhad-0003cH-TN; Thu, 11 Sep 2025 09:43:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwhab-0003bV-TC for qemu-devel@nongnu.org; Thu, 11 Sep 2025 09:43:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwhaY-0003W6-Uu for qemu-devel@nongnu.org; Thu, 11 Sep 2025 09:43:53 -0400 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-37-MKsxTTnOORC3vBHB8W1MbA-1; Thu, 11 Sep 2025 09:43:45 -0400 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A78A0195609F; Thu, 11 Sep 2025 13:43:43 +0000 (UTC) Received: from laptop.redhat.com (unknown [10.45.224.190]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3E3D51944CCA; Thu, 11 Sep 2025 13:43:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1757598228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SJe/KYJAhOxAiHSyQRTinz6TLWYZ74JDhD11G1hMJxA=; b=NrF/mMA21tAphVLelyrzZNX8muSiLkjsQ6bEy84VWktLIAGiUn6Z5KC7xXCvlEfrKkClit BUImcMJZCXwJhMem0UhimohOhIaw+Ae224CJs+Ls8Vr3+xUdbY4cVeLcjVUcScJTAXGjNO Xk2vJNgdK7LaRzY4DevYjhVTZXsEgbQ= X-MC-Unique: MKsxTTnOORC3vBHB8W1MbA-1 X-Mimecast-MFC-AGG-ID: MKsxTTnOORC3vBHB8W1MbA_1757598223 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, qemu-devel@nongnu.org, qemu-arm@nongnu.org, peter.maydell@linaro.org, cohuck@redhat.com, maz@kernel.org, oliver.upton@linux.dev, sebott@redhat.com, gshan@redhat.com Subject: [RFC 3/3] hw/arm/virt: [DO NOT UPSTREAM] Enforce compatibility with older kernels Date: Thu, 11 Sep 2025 15:40:29 +0200 Message-ID: <20250911134324.3702720-4-eric.auger@redhat.com> In-Reply-To: <20250911134324.3702720-1-eric.auger@redhat.com> References: <20250911134324.3702720-1-eric.auger@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1757598293702116600 Content-Type: text/plain; charset="utf-8" This is an example on how to use the new CPU options. This catters to distributions who want machines to be migratable (forward and backward) accross different host kernel versions in case KVM registers exposed to qemu vary accross kernels. This patch is not meant to be upstreamed as it is really kernel dependent. The goal is to illustrate how this would be used. In this example, For 10_1 machines types and older we apply the following host kernel related compats: 1) Make sure the KVM_REG_ARM_VENDOR_HYP_BMAP_2 exposed from v6.15 onwards is ignored/hidden. 2) Make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are always seen by qemu although not exposed by KVM. They were unconditionnally exposed before v6.13 while from v6.13 they are only exposed if supported by the guest. This will allow 10_1 machines types and older machines to migrate forward and backward from old downstream kernels that do not feature those changes to newer kernels (>=3D v6.15). Signed-off-by: Eric Auger --- hw/arm/virt.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 6f01746e74..1cceddcd2a 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -96,6 +96,23 @@ static GlobalProperty arm_virt_compat[] =3D { }; static const size_t arm_virt_compat_len =3D G_N_ELEMENTS(arm_virt_compat); =20 +/* + * if a 10_1 machine type or older is used: + * 1) make sure TCR_EL1, PIRE0_EL1, PIR_EL1 are enforced, even if they are= not + * exposed by the kernel + * 2) hide KVM_REG_ARM_VENDOR_HYP_BMAP_2 + */ +static GlobalProperty arm_virt_kernel_compat_10_1[] =3D { + /* KVM_REG_ARM_VENDOR_HYP_BMAP_2 */ + { TYPE_ARM_CPU, "kvm-hidden-regs", "0x6030000000160003" }, + /* TCR_EL1, PIRE0_EL1, PIR_EL1 */ + { TYPE_ARM_CPU, "kvm-enforced-regs", + "0x603000000013c103, 0x603000000013c512, 0x603000000013c513" }, +}; +static const size_t arm_virt_kernel_compat_10_1_len =3D + G_N_ELEMENTS(arm_virt_kernel_compat_10_1); + + /* * This cannot be called from the virt_machine_class_init() because * TYPE_VIRT_MACHINE is abstract and mc->compat_props g_ptr_array_new() @@ -3462,6 +3479,8 @@ static void virt_machine_10_1_options(MachineClass *m= c) { virt_machine_10_2_options(mc); compat_props_add(mc->compat_props, hw_compat_10_1, hw_compat_10_1_len); + compat_props_add(mc->compat_props, + arm_virt_kernel_compat_10_1, arm_virt_kernel_compat_1= 0_1_len); } DEFINE_VIRT_MACHINE(10, 1) =20 --=20 2.49.0