From nobody Fri Nov 14 18:24:39 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1760623411; cv=none; d=zohomail.com; s=zohoarc; b=HrtR/nFGtc2JHSk24hTjrGtuE1aXs+VZMVBepljSLp2TX+wIafjWi0Jsed+LgVaMgqyvxFN5jwtz3Vcj2QBvTd15bg1/GgwigNtXnaNsV1u1twtiUKkURuYYtuw4KodNo66KRtPZk9NFQSe+sFsa7k5bmy3D+tY6bbsHCNkM7qg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1760623411; h=Content-Type:Content-Transfer-Encoding:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=Qg/bYKOUnPg3adpJtACOEERgoJTmDD5Mw35BYXgsPzs=; b=nCv0RasQdu0tyLNHLDAaYc+T1qQnqzqDX/mXpsvDBLyibUWLPZ7iMYM6hw0Q6Ywmdp/BpHkfOyTuSYMg6LHzL+XARnfLGND1KKBm8xUPRDOUZVtnFkWns8t6AHSxe21BNBOTG6+nMADaN5yCSQq9vi9m/1sHAG+6yGrjN5BJ3uc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1760623411797475.7375942174964; Thu, 16 Oct 2025 07:03:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1v9OXk-0002e8-8l; Thu, 16 Oct 2025 10:01:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1v9OXf-0002cR-HE for qemu-devel@nongnu.org; Thu, 16 Oct 2025 10:01:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1v9OXZ-0006t5-1w for qemu-devel@nongnu.org; Thu, 16 Oct 2025 10:01:18 -0400 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-212-ZZ3CANBKPGOljR_zqKUwIQ-1; Thu, 16 Oct 2025 10:00:58 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2009F1801370; Thu, 16 Oct 2025 14:00:57 +0000 (UTC) Received: from laptop.redhat.com (unknown [10.44.32.238]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DCC94300019F; Thu, 16 Oct 2025 14:00:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760623267; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Qg/bYKOUnPg3adpJtACOEERgoJTmDD5Mw35BYXgsPzs=; b=QRCaeJwqJnX3AnnbfexiqlgFD5+FRHEAlbRqGQHKGLU+YKX8TkrdSIr3lT27rlUUi7CTzI 44Wv7a/Lw99zBedSUlfACbtWzbXL3BuBruGftJ7qApHyHsnpGEEx2v0jLGdcbd7kIj8Wia f+oddKy8j1+NpTGd5TmstHIpsu1LHmU= X-MC-Unique: ZZ3CANBKPGOljR_zqKUwIQ-1 X-Mimecast-MFC-AGG-ID: ZZ3CANBKPGOljR_zqKUwIQ_1760623257 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, qemu-devel@nongnu.org, qemu-arm@nongnu.org, peter.maydell@linaro.org, cohuck@redhat.com, maz@kernel.org, oliver.upton@linux.dev, sebott@redhat.com, gshan@redhat.com, ddutile@redhat.com, peterx@redhat.com, philmd@linaro.org, pbonzini@redhat.com Subject: [RESEND PATCH 2/7] target/arm/kvm: Introduce the concept of hidden KVM regs Date: Thu, 16 Oct 2025 15:59:47 +0200 Message-ID: <20251016140039.250111-3-eric.auger@redhat.com> In-Reply-To: <20251016140039.250111-1-eric.auger@redhat.com> References: <20251016140039.250111-1-eric.auger@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, T_SPF_TEMPERROR=0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1760623412704158500 More recent kernels sometimes expose new registers in an unconditionnal manner. This situation breaks backward migration as qemu notices there are more registers in the input stream than supported on the destination host. This leads to a "failed to load cpu:cpreg_vmstate_array_len" error. A good example is the introduction of KVM_REG_ARM_VENDOR_HYP_BMAP_2 pseudo FW register in v6.16 by commit C0000e58c74e (=E2=80=9CKVM: arm64: Introduce KVM_REG_ARM_VENDOR_HYP_BMAP_2=E2=80=9D). Trying to do backward migration from a host kernel that features the commit to a destination host that doesn't, fail with above error. Currently QEMU is not using that feature so ignoring this latter is not a problem. An easy way to fix the migration issue is to teach qemu we don't care about that register and we can simply ignore it when syncing its state during migration. This patch introduces an array of such hidden registers. Soon it will be settable through an array property. If hidden, the register is moved out of the array of cpreg which is built in kvm_arm_init_cpreg_list(). That way their state won't be synced. Signed-off-by: Eric Auger --- v1 -> v2: - Move the property in a separate patch - improve the commit msg - change the trace point to just print info in kvm_arm_init_cpreg_list() - improve comment in cpu.h (Connie) --- target/arm/cpu.h | 11 +++++++++++ target/arm/kvm.c | 31 ++++++++++++++++++++++++++++++- target/arm/trace-events | 2 ++ 3 files changed, 43 insertions(+), 1 deletion(-) diff --git a/target/arm/cpu.h b/target/arm/cpu.h index a7ed3f34f8..75cf7ee506 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1041,6 +1041,17 @@ struct ArchCPU { /* KVM steal time */ OnOffAuto kvm_steal_time; =20 + /* + * KVM registers that must be ignored/hidden. While they may be + * exposed by KVM to userspace, they are not intended to be used + * by qemu and more importantly we don't want them to be migrated + * to another host which wouldn't expose them. This would break + * the migration. This may be useful to allow backward migration to + * older kernels with less features. + */ + uint64_t *kvm_hidden_regs; + uint32_t nr_kvm_hidden_regs; + /* Uniprocessor system with MP extensions */ bool mp_is_up; =20 diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 58c6075a9e..60fb743423 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -754,6 +754,25 @@ static bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t = regidx) } } =20 +/** + * kvm_vcpu_compat_hidden_reg: + * @cpu: ARMCPU + * @regidx: index of the register to check + * + * Depending on the CPU compat returns true if @regidx must be + * ignored during sync & migration + */ +static inline bool +kvm_vcpu_compat_hidden_reg(ARMCPU *cpu, uint64_t regidx) +{ + for (int i =3D 0; i < cpu->nr_kvm_hidden_regs; i++) { + if (cpu->kvm_hidden_regs[i] =3D=3D regidx) { + return true; + } + } + return false; +} + /** * kvm_arm_init_cpreg_list: * @cpu: ARMCPU @@ -788,7 +807,10 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) qsort(&rlp->reg, rlp->n, sizeof(rlp->reg[0]), compare_u64); =20 for (i =3D 0, arraylen =3D 0; i < rlp->n; i++) { - if (!kvm_arm_reg_syncs_via_cpreg_list(rlp->reg[i])) { + uint64_t regidx =3D rlp->reg[i]; + + if (!kvm_arm_reg_syncs_via_cpreg_list(regidx) || + kvm_vcpu_compat_hidden_reg(cpu, regidx)) { continue; } switch (rlp->reg[i] & KVM_REG_SIZE_MASK) { @@ -804,6 +826,8 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) arraylen++; } =20 + trace_kvm_arm_init_cpreg_list_arraylen(arraylen); + cpu->cpreg_indexes =3D g_renew(uint64_t, cpu->cpreg_indexes, arraylen); cpu->cpreg_values =3D g_renew(uint64_t, cpu->cpreg_values, arraylen); cpu->cpreg_vmstate_indexes =3D g_renew(uint64_t, cpu->cpreg_vmstate_in= dexes, @@ -815,9 +839,14 @@ static int kvm_arm_init_cpreg_list(ARMCPU *cpu) =20 for (i =3D 0, arraylen =3D 0; i < rlp->n; i++) { uint64_t regidx =3D rlp->reg[i]; + if (!kvm_arm_reg_syncs_via_cpreg_list(regidx)) { continue; } + if (kvm_vcpu_compat_hidden_reg(cpu, regidx)) { + trace_kvm_arm_init_cpreg_list_skip_hidden_reg(rlp->reg[i]); + continue; + } cpu->cpreg_indexes[arraylen] =3D regidx; arraylen++; } diff --git a/target/arm/trace-events b/target/arm/trace-events index c497df8fe7..31386cc1f2 100644 --- a/target/arm/trace-events +++ b/target/arm/trace-events @@ -14,6 +14,8 @@ arm_gt_update_irq(int timer, int irqstate) "gt_update_irq= : timer %d irqstate %d" # kvm.c kvm_arm_fixup_msi_route(uint64_t iova, uint64_t gpa) "MSI iova =3D 0x%"PRI= x64" is translated into 0x%"PRIx64 kvm_arm_cpu_post_load_missing_reg(char *name) "Missing register in input s= tream: %s" +kvm_arm_init_cpreg_list_arraylen(uint32_t arraylen) "arraylen=3D%d" +kvm_arm_init_cpreg_list_skip_hidden_reg(uint64_t regidx) "hidden 0x%"PRIx6= 4" is skipped" =20 # cpu.c arm_cpu_reset(uint64_t mp_aff) "cpu %" PRIu64 --=20 2.49.0