From nobody Sat Jun 20 20:54:04 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF846396599 for ; Thu, 9 Apr 2026 22:42:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774572; cv=none; b=ZqrSZ6Ee0SNVOJ+FWMecQCibsKiyyjmO/eRhvQoHCrFfmlz1GZS6nkduVYLlHV2xda3h0wksQOeDO/HkAC9Hr3abYJVNZQEkQtmWPmvlF6Xd3NGKqImkNIS/Rtrp2esyEgEUekSFm5kjFzHf4iAOLmyG0xoXvqLC2sVfWBtKR/o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774572; c=relaxed/simple; bh=9PY6IN9ye/ndMvnLulYGBqgEuAbDBhoAft7hTQkaxoY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j2W722/4P0wWtH/Z42yQWWUEbXXrbwh+k2ffLrmBbVkg1NAoA6bl3AAkAR1te2dFubma6G6SuK7mO3ea0C/4gNe5n4jyWsTe5Rr/HfTi9eSes59C1T2j+pSrFBdNKz8WFJR138OZcg1rhuBFpJhovIXF9vz5XeeVHGKA5DMeP6M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=eK2pm1dr; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="eK2pm1dr" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-82c8768a704so690689b3a.3 for ; Thu, 09 Apr 2026 15:42:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1775774569; x=1776379369; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=BA9IxKBBNNHN4chge8KN4r7euKndhCGWijVvRNxopgU=; b=eK2pm1dr+d6VDiWZl/t6frS1Ac3tgAvh7MsQv0dcXg9+Hnot5djXuzK9ik3om2Ttir qoBPbVV9fFa+2RtQpBqk/7ujPTrupN5UATBLQJHL/oM2y535Ir9x7ejvfkumTS9tUvV3 wH3jCMudFdRqOoyDqEnxnqmG4A3LFIO/5L2eRFlqp5dyGI5qlFXT6Efu3TqB18ERetwH CMpn7p+71+mP0q5HtEBMHVMj+SMLyd4MgeKojb7RE1esI1C7DMkcj8hcZ0Ljcki3ObEu CZjSQKGaWE4GiGOHIaIe/5YuV1WXQL/8hN7rlVNTr9qc0JM9VTyCwSIq1rhrYdkwVky0 meBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775774569; x=1776379369; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=BA9IxKBBNNHN4chge8KN4r7euKndhCGWijVvRNxopgU=; b=mi9QNBFd9F1DlOW4jVsiwp1SRp5crhy9B/Xfy+syuMFv5AyGxCFoY8m+0wX8wFnFIY 1WtENb1cpsRhTEfAyBgUNujIIS7kGB7GcZ1lxnFnHW+BITABP2IePUYybQgG1etTe1tB wWBprb7buOlAICXvfn4+mE5YwgVn8HOXMoJBXyGaFqbjwnEsJT2cWGC7h59hxuX0snqu kR1yWCsjIasO50u+pTEh1Hvm3/wuSTv159VSgzY66SO4IQrQ2RDDXLkpTXjgChsz2n3e kebymYxLheBLlm5n3JiDoMeNyiM9YwVQP3YljafvWlM5FeBysKCMr7HvGg69tIq5ZORd E58Q== X-Forwarded-Encrypted: i=1; AJvYcCWyUUl6UArd26wlr4gQhQl9Jp9aaRgdhZGqiL7x//MuVqYXXxKBFRcM3r7KBzsXmSpWTnSyHCSpdzwMjC8=@vger.kernel.org X-Gm-Message-State: AOJu0YxB4c6IVcz/7AYwvOlLJThbjRRsd/s11/11JzpiFI0PZzBvc1nI 2oTTeLhlTmvwCj+zSerr4M34mmGsfLQ3yfmnDvpdbxdKLq5ALNO3FgCA97Mb6fGtd5DX/jTey2+ W2KdkXA== X-Received: from pfblp15.prod.google.com ([2002:a05:6a00:3d4f:b0:829:8aa0:dc3c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2190:b0:82a:1529:2b4f with SMTP id d2e1a72fcca58-82f0c3256e2mr804385b3a.44.1775774569094; Thu, 09 Apr 2026 15:42:49 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 9 Apr 2026 15:42:30 -0700 In-Reply-To: <20260409224236.2021562-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260409224236.2021562-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.1213.gd9a14994de-goog Message-ID: <20260409224236.2021562-2-seanjc@google.com> Subject: [PATCH v2 1/6] KVM: x86: Add dedicated storage for guest RIP From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Chang S . Bae" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add kvm_vcpu_arch.rip to track guest RIP instead of including it in the generic regs[] array. Decoupling RIP from regs[] will allow using a *completely* arbitrary index for RIP, as opposed to the mostly-arbitrary index that is currently used. That in turn will allow using indices 16-31 to track R16-R31 that are coming with APX. Note, although RIP can used for addressing, it does NOT have an architecturally defined index, and so can't be reached via flows like get_vmx_mem_address() where KVM "blindly" reads a general purpose register given the SIB information reported by hardware. For RIP-relative addressing, hardware reports the full "offset" in vmcs.EXIT_QUALIFICATION. Note #2, keep the available/dirty tracking as RSP is context switched through the VMCS, i.e. needs to be cached for VMX. Opportunistically rename NR_VCPU_REGS to NR_VCPU_GENERAL_PURPOSE_REGS to better capture what it tracks, and so that KVM can slot in R16-R13 without running into weirdness where KVM's definition of "EXREG" doesn't line up with APX's definition of "extended reg". No functional change intended. Cc: Chang S. Bae Signed-off-by: Sean Christopherson Reviewed-by: Chang S. Bae Reviewed-by: Kai Huang Tested-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 10 ++++++---- arch/x86/kvm/kvm_cache_regs.h | 12 ++++++++---- arch/x86/kvm/svm/sev.c | 2 +- arch/x86/kvm/svm/svm.c | 6 +++--- arch/x86/kvm/vmx/vmx.c | 8 ++++---- arch/x86/kvm/vmx/vmx.h | 2 +- 6 files changed, 23 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index c470e40a00aa..68a11325e8bc 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -191,10 +191,11 @@ enum kvm_reg { VCPU_REGS_R14 =3D __VCPU_REGS_R14, VCPU_REGS_R15 =3D __VCPU_REGS_R15, #endif - VCPU_REGS_RIP, - NR_VCPU_REGS, + NR_VCPU_GENERAL_PURPOSE_REGS, =20 - VCPU_EXREG_PDPTR =3D NR_VCPU_REGS, + VCPU_REG_RIP =3D NR_VCPU_GENERAL_PURPOSE_REGS, + + VCPU_EXREG_PDPTR, VCPU_EXREG_CR0, /* * Alias AMD's ERAPS (not a real register) to CR3 so that common code @@ -799,7 +800,8 @@ struct kvm_vcpu_arch { * rip and regs accesses must go through * kvm_{register,rip}_{read,write} functions. */ - unsigned long regs[NR_VCPU_REGS]; + unsigned long regs[NR_VCPU_GENERAL_PURPOSE_REGS]; + unsigned long rip; u32 regs_avail; u32 regs_dirty; =20 diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index 8ddb01191d6f..9b7df9de0e87 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -112,7 +112,7 @@ static __always_inline bool kvm_register_test_and_mark_= available(struct kvm_vcpu */ static inline unsigned long kvm_register_read_raw(struct kvm_vcpu *vcpu, i= nt reg) { - if (WARN_ON_ONCE((unsigned int)reg >=3D NR_VCPU_REGS)) + if (WARN_ON_ONCE((unsigned int)reg >=3D NR_VCPU_GENERAL_PURPOSE_REGS)) return 0; =20 if (!kvm_register_is_available(vcpu, reg)) @@ -124,7 +124,7 @@ static inline unsigned long kvm_register_read_raw(struc= t kvm_vcpu *vcpu, int reg static inline void kvm_register_write_raw(struct kvm_vcpu *vcpu, int reg, unsigned long val) { - if (WARN_ON_ONCE((unsigned int)reg >=3D NR_VCPU_REGS)) + if (WARN_ON_ONCE((unsigned int)reg >=3D NR_VCPU_GENERAL_PURPOSE_REGS)) return; =20 vcpu->arch.regs[reg] =3D val; @@ -133,12 +133,16 @@ static inline void kvm_register_write_raw(struct kvm_= vcpu *vcpu, int reg, =20 static inline unsigned long kvm_rip_read(struct kvm_vcpu *vcpu) { - return kvm_register_read_raw(vcpu, VCPU_REGS_RIP); + if (!kvm_register_is_available(vcpu, VCPU_REG_RIP)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_RIP); + + return vcpu->arch.rip; } =20 static inline void kvm_rip_write(struct kvm_vcpu *vcpu, unsigned long val) { - kvm_register_write_raw(vcpu, VCPU_REGS_RIP, val); + vcpu->arch.rip =3D val; + kvm_register_mark_dirty(vcpu, VCPU_REG_RIP); } =20 static inline unsigned long kvm_rsp_read(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 75d0c03d69bc..2010b157e288 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -967,7 +967,7 @@ static int sev_es_sync_vmsa(struct vcpu_svm *svm) save->r14 =3D svm->vcpu.arch.regs[VCPU_REGS_R14]; save->r15 =3D svm->vcpu.arch.regs[VCPU_REGS_R15]; #endif - save->rip =3D svm->vcpu.arch.regs[VCPU_REGS_RIP]; + save->rip =3D svm->vcpu.arch.rip; =20 /* Sync some non-GPR registers before encrypting */ save->xcr0 =3D svm->vcpu.arch.xcr0; diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index e7fdd7a9c280..85edaee27b03 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4420,7 +4420,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) =20 svm->vmcb->save.rax =3D vcpu->arch.regs[VCPU_REGS_RAX]; svm->vmcb->save.rsp =3D vcpu->arch.regs[VCPU_REGS_RSP]; - svm->vmcb->save.rip =3D vcpu->arch.regs[VCPU_REGS_RIP]; + svm->vmcb->save.rip =3D vcpu->arch.rip; =20 /* * Disable singlestep if we're injecting an interrupt/exception. @@ -4506,7 +4506,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) vcpu->arch.cr2 =3D svm->vmcb->save.cr2; vcpu->arch.regs[VCPU_REGS_RAX] =3D svm->vmcb->save.rax; vcpu->arch.regs[VCPU_REGS_RSP] =3D svm->vmcb->save.rsp; - vcpu->arch.regs[VCPU_REGS_RIP] =3D svm->vmcb->save.rip; + vcpu->arch.rip =3D svm->vmcb->save.rip; } vcpu->arch.regs_dirty =3D 0; =20 @@ -4946,7 +4946,7 @@ static int svm_enter_smm(struct kvm_vcpu *vcpu, union= kvm_smram *smram) =20 svm->vmcb->save.rax =3D vcpu->arch.regs[VCPU_REGS_RAX]; svm->vmcb->save.rsp =3D vcpu->arch.regs[VCPU_REGS_RSP]; - svm->vmcb->save.rip =3D vcpu->arch.regs[VCPU_REGS_RIP]; + svm->vmcb->save.rip =3D vcpu->arch.rip; =20 nested_svm_simple_vmexit(svm, SVM_EXIT_SW); =20 diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a29896a9ef14..577b0c6286ad 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -2604,8 +2604,8 @@ void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_re= g reg) case VCPU_REGS_RSP: vcpu->arch.regs[VCPU_REGS_RSP] =3D vmcs_readl(GUEST_RSP); break; - case VCPU_REGS_RIP: - vcpu->arch.regs[VCPU_REGS_RIP] =3D vmcs_readl(GUEST_RIP); + case VCPU_REG_RIP: + vcpu->arch.rip =3D vmcs_readl(GUEST_RIP); break; case VCPU_EXREG_PDPTR: if (enable_ept) @@ -7536,8 +7536,8 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 ru= n_flags) =20 if (kvm_register_is_dirty(vcpu, VCPU_REGS_RSP)) vmcs_writel(GUEST_RSP, vcpu->arch.regs[VCPU_REGS_RSP]); - if (kvm_register_is_dirty(vcpu, VCPU_REGS_RIP)) - vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]); + if (kvm_register_is_dirty(vcpu, VCPU_REG_RIP)) + vmcs_writel(GUEST_RIP, vcpu->arch.rip); vcpu->arch.regs_dirty =3D 0; =20 if (run_flags & KVM_RUN_LOAD_GUEST_DR6) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index db84e8001da5..d0cc5f6c6879 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -620,7 +620,7 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC_C= ONTROL, 64) * cache on demand. Other registers not listed here are synced to * the cache immediately after VM-Exit. */ -#define VMX_REGS_LAZY_LOAD_SET ((1 << VCPU_REGS_RIP) | \ +#define VMX_REGS_LAZY_LOAD_SET ((1 << VCPU_REG_RIP) | \ (1 << VCPU_REGS_RSP) | \ (1 << VCPU_EXREG_RFLAGS) | \ (1 << VCPU_EXREG_PDPTR) | \ --=20 2.53.0.1213.gd9a14994de-goog From nobody Sat Jun 20 20:54:04 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2345395D9F for ; Thu, 9 Apr 2026 22:42:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774573; cv=none; b=DVTexpJS3wGfdtja0KdaJJHlxGsHlteTcFOP2J7Dn8aYfFKApBufBpVMO6kRuY38U95IQLYniE07kSskD92mTVHAPBUQb/nFRJD9edZYiUpIm/9x3+mityieJo5If05J6P12fSi9JJsjVT3LeuqcqsbmgfJIBhX2sY3eKppvwBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774573; c=relaxed/simple; bh=BWqoYQtBhm8kqHPCG//Ltf+xQapfCYWKp4l+Z2Ugs0o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qrAoAlHYv3JSrsWj3Gyoab7I/LxzqGQQ+HRzdigczaBzFS15zQud0V0Z+0wHUJ2PT95M9r5gB+5r7Rk/6QsY0wlwrU83EJ0LhGSHsL9ne19QgdmhlbjKff2WeyGZliMxk7FrDlc0mlvP1dYd5LEIb+ll0mNvjk8MS84wV6TVRf4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=khzQmL7X; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="khzQmL7X" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2b24cd2e2b3so15751345ad.0 for ; Thu, 09 Apr 2026 15:42:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1775774571; x=1776379371; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:from:to:cc:subject:date :message-id:reply-to; bh=ISba7WJRVs+RImaJEsr8hL9vwMMdBHQV2nut6xn7NB4=; b=khzQmL7X37N7UtDeuv1GaOo3TkdptB8hlJacwmuwNnGPECBrWxEfsJ6l32l8wiLMpL xekrA1PZJUut0K49bLLMMkOipnYc4otiSc5fv9sFypLcOs/BQvwn5pSho+uQ/Z74a3m2 Mdoi384bROGBistFumN7DPEVfSFA/msg1ANxp9DYRDC5wg9Gq4XtEr3luB/uEYC/VSFN kUtC2eCvAOSj4l7C80xCXMvYjMXUvcdwrKVWdzwk7Jzvdf/zGiLCPaHzX3fzCysoX9g/ 3x/yru2tWTei6HEbwq9/1i/oPvh6/DqUDjgQ0d1FjJGzKkXojMERoNwmJXy0p3C2c33T alNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775774571; x=1776379371; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=ISba7WJRVs+RImaJEsr8hL9vwMMdBHQV2nut6xn7NB4=; b=hxUfBXyFDjWn3+d7LDN/RrU3vBFqXgVECh7gWM8cNxbSyzFKXY8lt72ca8wjGyHJp3 t5n3vUXX70/8RrYOahDYodfGXvkLRg29aCjeN8xTlIeECCHfdLPDq2sC9BuZ+bD8l49S tiGPhrh+SISB0zGLh2bbjvCjvbd8lWEbAZf6t2OB9iCKI+wkByt+jS8R2PX5U9E6l6k/ /y4ZzVeMVt2+WGKiN8rt+e4xRSxhbpcYcAAuP/pzI7JkaehMZ4z4seb07mh2mZLfcbCf UioNfltdousuhxel8B+w9YRfPQ5VcUN4QBorOHRL7nblyocUKwUdwVwgemfo7T5oICco 0duQ== X-Forwarded-Encrypted: i=1; AJvYcCXaF5IjCXfqndyo4xT5G0DGzS0m0lMWQjMsSZ7DMJ/8AlokpvNzlIBh9DuJ0eBkUuS21yPBajQ0eelJYw8=@vger.kernel.org X-Gm-Message-State: AOJu0YzcoDsfLMpmX1IYOSEKfYYr4UeRbhgoHaYMsyhW/i1ZD/mTO/os zlPOLklZy1+FTCb+/5Zt34kZxEYnxAvvXnBJDOKu5es5lEMzaH4LGBFLJJB5OX9NIFD3Np8yK08 /VbYe5A== X-Received: from plgv18.prod.google.com ([2002:a17:902:e8d2:b0:2b0:aa9a:2c78]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:7616:b0:2ae:4a4e:1e25 with SMTP id d9443c01a7336-2b2d5a348f0mr4593565ad.25.1775774570949; Thu, 09 Apr 2026 15:42:50 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 9 Apr 2026 15:42:31 -0700 In-Reply-To: <20260409224236.2021562-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260409224236.2021562-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.1213.gd9a14994de-goog Message-ID: <20260409224236.2021562-3-seanjc@google.com> Subject: [PATCH v2 2/6] KVM: x86: Drop the "EX" part of "EXREG" to avoid collision with APX From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Chang S . Bae" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that NR_VCPU_REGS is no longer a thing, and now that now that RIP is effectively an EXREG, drop the "EX" is for extended (or maybe extra?") prefix from non-GPR registers to avoid a collision with APX (Advanced Performance Extensions), which adds: 16 additional general-purpose registers (GPRs) R16=E2=80=93R31, also refe= rred to as Extended GPRs (EGPRs) in this document; I.e. KVM's version of "extended" won't match with APX's definition. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Tested-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 18 +++++++-------- arch/x86/kvm/kvm_cache_regs.h | 16 ++++++------- arch/x86/kvm/svm/svm.c | 6 ++--- arch/x86/kvm/svm/svm.h | 2 +- arch/x86/kvm/vmx/nested.c | 6 ++--- arch/x86/kvm/vmx/tdx.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 40 ++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.h | 20 ++++++++--------- arch/x86/kvm/x86.c | 16 ++++++------- 9 files changed, 64 insertions(+), 64 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 68a11325e8bc..b1eae1e7b04f 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -195,8 +195,8 @@ enum kvm_reg { =20 VCPU_REG_RIP =3D NR_VCPU_GENERAL_PURPOSE_REGS, =20 - VCPU_EXREG_PDPTR, - VCPU_EXREG_CR0, + VCPU_REG_PDPTR, + VCPU_REG_CR0, /* * Alias AMD's ERAPS (not a real register) to CR3 so that common code * can trigger emulation of the RAP (Return Address Predictor) with @@ -204,13 +204,13 @@ enum kvm_reg { * is cleared on writes to CR3, i.e. marking CR3 dirty will naturally * mark ERAPS dirty as well. */ - VCPU_EXREG_CR3, - VCPU_EXREG_ERAPS =3D VCPU_EXREG_CR3, - VCPU_EXREG_CR4, - VCPU_EXREG_RFLAGS, - VCPU_EXREG_SEGMENTS, - VCPU_EXREG_EXIT_INFO_1, - VCPU_EXREG_EXIT_INFO_2, + VCPU_REG_CR3, + VCPU_REG_ERAPS =3D VCPU_REG_CR3, + VCPU_REG_CR4, + VCPU_REG_RFLAGS, + VCPU_REG_SEGMENTS, + VCPU_REG_EXIT_INFO_1, + VCPU_REG_EXIT_INFO_2, }; =20 enum { diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index 9b7df9de0e87..ac1f9867a234 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -159,8 +159,8 @@ static inline u64 kvm_pdptr_read(struct kvm_vcpu *vcpu,= int index) { might_sleep(); /* on svm */ =20 - if (!kvm_register_is_available(vcpu, VCPU_EXREG_PDPTR)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_PDPTR); + if (!kvm_register_is_available(vcpu, VCPU_REG_PDPTR)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_PDPTR); =20 return vcpu->arch.walk_mmu->pdptrs[index]; } @@ -174,8 +174,8 @@ static inline ulong kvm_read_cr0_bits(struct kvm_vcpu *= vcpu, ulong mask) { ulong tmask =3D mask & KVM_POSSIBLE_CR0_GUEST_BITS; if ((tmask & vcpu->arch.cr0_guest_owned_bits) && - !kvm_register_is_available(vcpu, VCPU_EXREG_CR0)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_CR0); + !kvm_register_is_available(vcpu, VCPU_REG_CR0)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_CR0); return vcpu->arch.cr0 & mask; } =20 @@ -196,8 +196,8 @@ static inline ulong kvm_read_cr4_bits(struct kvm_vcpu *= vcpu, ulong mask) { ulong tmask =3D mask & KVM_POSSIBLE_CR4_GUEST_BITS; if ((tmask & vcpu->arch.cr4_guest_owned_bits) && - !kvm_register_is_available(vcpu, VCPU_EXREG_CR4)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_CR4); + !kvm_register_is_available(vcpu, VCPU_REG_CR4)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_CR4); return vcpu->arch.cr4 & mask; } =20 @@ -211,8 +211,8 @@ static __always_inline bool kvm_is_cr4_bit_set(struct k= vm_vcpu *vcpu, =20 static inline ulong kvm_read_cr3(struct kvm_vcpu *vcpu) { - if (!kvm_register_is_available(vcpu, VCPU_EXREG_CR3)) - kvm_x86_call(cache_reg)(vcpu, VCPU_EXREG_CR3); + if (!kvm_register_is_available(vcpu, VCPU_REG_CR3)) + kvm_x86_call(cache_reg)(vcpu, VCPU_REG_CR3); return vcpu->arch.cr3; } =20 diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 85edaee27b03..ee5749d8b3e8 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1517,7 +1517,7 @@ static void svm_cache_reg(struct kvm_vcpu *vcpu, enum= kvm_reg reg) kvm_register_mark_available(vcpu, reg); =20 switch (reg) { - case VCPU_EXREG_PDPTR: + case VCPU_REG_PDPTR: /* * When !npt_enabled, mmu->pdptrs[] is already available since * it is always updated per SDM when moving to CRs. @@ -4179,7 +4179,7 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, = gva_t gva) =20 static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu) { - kvm_register_mark_dirty(vcpu, VCPU_EXREG_ERAPS); + kvm_register_mark_dirty(vcpu, VCPU_REG_ERAPS); =20 svm_flush_tlb_asid(vcpu); } @@ -4457,7 +4457,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) svm->vmcb->save.cr2 =3D vcpu->arch.cr2; =20 if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) && - kvm_register_is_dirty(vcpu, VCPU_EXREG_ERAPS)) + kvm_register_is_dirty(vcpu, VCPU_REG_ERAPS)) svm->vmcb->control.erap_ctl |=3D ERAP_CONTROL_CLEAR_RAP; =20 svm_fixup_nested_rips(vcpu); diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index fd0652b32c81..677d268ae9c7 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -474,7 +474,7 @@ static inline bool svm_is_vmrun_failure(u64 exit_code) * KVM_REQ_LOAD_MMU_PGD is always requested when the cached vcpu->arch.cr3 * is changed. svm_load_mmu_pgd() then syncs the new CR3 value into the V= MCB. */ -#define SVM_REGS_LAZY_LOAD_SET (1 << VCPU_EXREG_PDPTR) +#define SVM_REGS_LAZY_LOAD_SET (1 << VCPU_REG_PDPTR) =20 static inline void __vmcb_set_intercept(unsigned long *intercepts, u32 bit) { diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 3fe88f29be7a..22b1f06a9d40 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -1189,7 +1189,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu,= unsigned long cr3, } =20 vcpu->arch.cr3 =3D cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); =20 /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ kvm_init_mmu(vcpu); @@ -4972,7 +4972,7 @@ static void nested_vmx_restore_host_state(struct kvm_= vcpu *vcpu) =20 nested_ept_uninit_mmu_context(vcpu); vcpu->arch.cr3 =3D vmcs_readl(GUEST_CR3); - kvm_register_mark_available(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_available(vcpu, VCPU_REG_CR3); =20 /* * Use ept_save_pdptrs(vcpu) to load the MMU's cached PDPTRs @@ -5074,7 +5074,7 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 v= m_exit_reason, kvm_service_local_tlb_flush_requests(vcpu); =20 /* - * VCPU_EXREG_PDPTR will be clobbered in arch/x86/kvm/vmx/vmx.h between + * VCPU_REG_PDPTR will be clobbered in arch/x86/kvm/vmx/vmx.h between * now and the new vmentry. Ensure that the VMCS02 PDPTR fields are * up-to-date before switching to L1. */ diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 1e47c194af53..c23ec4ac8bc8 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1013,8 +1013,8 @@ static fastpath_t tdx_exit_handlers_fastpath(struct k= vm_vcpu *vcpu) return EXIT_FASTPATH_NONE; } =20 -#define TDX_REGS_AVAIL_SET (BIT_ULL(VCPU_EXREG_EXIT_INFO_1) | \ - BIT_ULL(VCPU_EXREG_EXIT_INFO_2) | \ +#define TDX_REGS_AVAIL_SET (BIT_ULL(VCPU_REG_EXIT_INFO_1) | \ + BIT_ULL(VCPU_REG_EXIT_INFO_2) | \ BIT_ULL(VCPU_REGS_RAX) | \ BIT_ULL(VCPU_REGS_RBX) | \ BIT_ULL(VCPU_REGS_RCX) | \ diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 577b0c6286ad..aa1c26018439 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -843,8 +843,8 @@ static bool vmx_segment_cache_test_set(struct vcpu_vmx = *vmx, unsigned seg, bool ret; u32 mask =3D 1 << (seg * SEG_FIELD_NR + field); =20 - if (!kvm_register_is_available(&vmx->vcpu, VCPU_EXREG_SEGMENTS)) { - kvm_register_mark_available(&vmx->vcpu, VCPU_EXREG_SEGMENTS); + if (!kvm_register_is_available(&vmx->vcpu, VCPU_REG_SEGMENTS)) { + kvm_register_mark_available(&vmx->vcpu, VCPU_REG_SEGMENTS); vmx->segment_cache.bitmask =3D 0; } ret =3D vmx->segment_cache.bitmask & mask; @@ -1609,8 +1609,8 @@ unsigned long vmx_get_rflags(struct kvm_vcpu *vcpu) struct vcpu_vmx *vmx =3D to_vmx(vcpu); unsigned long rflags, save_rflags; =20 - if (!kvm_register_is_available(vcpu, VCPU_EXREG_RFLAGS)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_RFLAGS); + if (!kvm_register_is_available(vcpu, VCPU_REG_RFLAGS)) { + kvm_register_mark_available(vcpu, VCPU_REG_RFLAGS); rflags =3D vmcs_readl(GUEST_RFLAGS); if (vmx->rmode.vm86_active) { rflags &=3D RMODE_GUEST_OWNED_EFLAGS_BITS; @@ -1633,7 +1633,7 @@ void vmx_set_rflags(struct kvm_vcpu *vcpu, unsigned l= ong rflags) * if L1 runs L2 as a restricted guest. */ if (is_unrestricted_guest(vcpu)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_RFLAGS); + kvm_register_mark_available(vcpu, VCPU_REG_RFLAGS); vmx->rflags =3D rflags; vmcs_writel(GUEST_RFLAGS, rflags); return; @@ -2607,17 +2607,17 @@ void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_= reg reg) case VCPU_REG_RIP: vcpu->arch.rip =3D vmcs_readl(GUEST_RIP); break; - case VCPU_EXREG_PDPTR: + case VCPU_REG_PDPTR: if (enable_ept) ept_save_pdptrs(vcpu); break; - case VCPU_EXREG_CR0: + case VCPU_REG_CR0: guest_owned_bits =3D vcpu->arch.cr0_guest_owned_bits; =20 vcpu->arch.cr0 &=3D ~guest_owned_bits; vcpu->arch.cr0 |=3D vmcs_readl(GUEST_CR0) & guest_owned_bits; break; - case VCPU_EXREG_CR3: + case VCPU_REG_CR3: /* * When intercepting CR3 loads, e.g. for shadowing paging, KVM's * CR3 is loaded into hardware, not the guest's CR3. @@ -2625,7 +2625,7 @@ void vmx_cache_reg(struct kvm_vcpu *vcpu, enum kvm_re= g reg) if (!(exec_controls_get(to_vmx(vcpu)) & CPU_BASED_CR3_LOAD_EXITING)) vcpu->arch.cr3 =3D vmcs_readl(GUEST_CR3); break; - case VCPU_EXREG_CR4: + case VCPU_REG_CR4: guest_owned_bits =3D vcpu->arch.cr4_guest_owned_bits; =20 vcpu->arch.cr4 &=3D ~guest_owned_bits; @@ -3350,7 +3350,7 @@ void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu =3D vcpu->arch.walk_mmu; =20 - if (!kvm_register_is_dirty(vcpu, VCPU_EXREG_PDPTR)) + if (!kvm_register_is_dirty(vcpu, VCPU_REG_PDPTR)) return; =20 if (is_pae_paging(vcpu)) { @@ -3373,7 +3373,7 @@ void ept_save_pdptrs(struct kvm_vcpu *vcpu) mmu->pdptrs[2] =3D vmcs_read64(GUEST_PDPTR2); mmu->pdptrs[3] =3D vmcs_read64(GUEST_PDPTR3); =20 - kvm_register_mark_available(vcpu, VCPU_EXREG_PDPTR); + kvm_register_mark_available(vcpu, VCPU_REG_PDPTR); } =20 #define CR3_EXITING_BITS (CPU_BASED_CR3_LOAD_EXITING | \ @@ -3416,7 +3416,7 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long= cr0) vmcs_writel(CR0_READ_SHADOW, cr0); vmcs_writel(GUEST_CR0, hw_cr0); vcpu->arch.cr0 =3D cr0; - kvm_register_mark_available(vcpu, VCPU_EXREG_CR0); + kvm_register_mark_available(vcpu, VCPU_REG_CR0); =20 #ifdef CONFIG_X86_64 if (vcpu->arch.efer & EFER_LME) { @@ -3434,8 +3434,8 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long= cr0) * (correctly) stop reading vmcs.GUEST_CR3 because it thinks * KVM's CR3 is installed. */ - if (!kvm_register_is_available(vcpu, VCPU_EXREG_CR3)) - vmx_cache_reg(vcpu, VCPU_EXREG_CR3); + if (!kvm_register_is_available(vcpu, VCPU_REG_CR3)) + vmx_cache_reg(vcpu, VCPU_REG_CR3); =20 /* * When running with EPT but not unrestricted guest, KVM must @@ -3472,7 +3472,7 @@ void vmx_set_cr0(struct kvm_vcpu *vcpu, unsigned long= cr0) * GUEST_CR3 is still vmx->ept_identity_map_addr if EPT + !URG. */ if (!(old_cr0_pg & X86_CR0_PG) && (cr0 & X86_CR0_PG)) - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); } =20 /* depends on vcpu->arch.cr0 to be set to a new value */ @@ -3501,7 +3501,7 @@ void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t ro= ot_hpa, int root_level) =20 if (!enable_unrestricted_guest && !is_paging(vcpu)) guest_cr3 =3D to_kvm_vmx(kvm)->ept_identity_map_addr; - else if (kvm_register_is_dirty(vcpu, VCPU_EXREG_CR3)) + else if (kvm_register_is_dirty(vcpu, VCPU_REG_CR3)) guest_cr3 =3D vcpu->arch.cr3; else /* vmcs.GUEST_CR3 is already up-to-date. */ update_guest_cr3 =3D false; @@ -3561,7 +3561,7 @@ void vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long= cr4) } =20 vcpu->arch.cr4 =3D cr4; - kvm_register_mark_available(vcpu, VCPU_EXREG_CR4); + kvm_register_mark_available(vcpu, VCPU_REG_CR4); =20 if (!enable_unrestricted_guest) { if (enable_ept) { @@ -5021,7 +5021,7 @@ void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_= event) vmcs_write32(GUEST_IDTR_LIMIT, 0xffff); =20 vmx_segment_cache_clear(vmx); - kvm_register_mark_available(vcpu, VCPU_EXREG_SEGMENTS); + kvm_register_mark_available(vcpu, VCPU_REG_SEGMENTS); =20 vmcs_write32(GUEST_ACTIVITY_STATE, GUEST_ACTIVITY_ACTIVE); vmcs_write32(GUEST_INTERRUPTIBILITY_INFO, 0); @@ -7514,9 +7514,9 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 ru= n_flags) =20 vmx->vt.exit_reason.full =3D EXIT_REASON_INVALID_STATE; vmx->vt.exit_reason.failed_vmentry =3D 1; - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1); + kvm_register_mark_available(vcpu, VCPU_REG_EXIT_INFO_1); vmx->vt.exit_qualification =3D ENTRY_FAIL_DEFAULT; - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2); + kvm_register_mark_available(vcpu, VCPU_REG_EXIT_INFO_2); vmx->vt.exit_intr_info =3D 0; return EXIT_FASTPATH_NONE; } diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index d0cc5f6c6879..9fb76ea48caf 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -317,7 +317,7 @@ static __always_inline unsigned long vmx_get_exit_qual(= struct kvm_vcpu *vcpu) { struct vcpu_vt *vt =3D to_vt(vcpu); =20 - if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1) && + if (!kvm_register_test_and_mark_available(vcpu, VCPU_REG_EXIT_INFO_1) && !WARN_ON_ONCE(is_td_vcpu(vcpu))) vt->exit_qualification =3D vmcs_readl(EXIT_QUALIFICATION); =20 @@ -328,7 +328,7 @@ static __always_inline u32 vmx_get_intr_info(struct kvm= _vcpu *vcpu) { struct vcpu_vt *vt =3D to_vt(vcpu); =20 - if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2) && + if (!kvm_register_test_and_mark_available(vcpu, VCPU_REG_EXIT_INFO_2) && !WARN_ON_ONCE(is_td_vcpu(vcpu))) vt->exit_intr_info =3D vmcs_read32(VM_EXIT_INTR_INFO); =20 @@ -622,14 +622,14 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC= _CONTROL, 64) */ #define VMX_REGS_LAZY_LOAD_SET ((1 << VCPU_REG_RIP) | \ (1 << VCPU_REGS_RSP) | \ - (1 << VCPU_EXREG_RFLAGS) | \ - (1 << VCPU_EXREG_PDPTR) | \ - (1 << VCPU_EXREG_SEGMENTS) | \ - (1 << VCPU_EXREG_CR0) | \ - (1 << VCPU_EXREG_CR3) | \ - (1 << VCPU_EXREG_CR4) | \ - (1 << VCPU_EXREG_EXIT_INFO_1) | \ - (1 << VCPU_EXREG_EXIT_INFO_2)) + (1 << VCPU_REG_RFLAGS) | \ + (1 << VCPU_REG_PDPTR) | \ + (1 << VCPU_REG_SEGMENTS) | \ + (1 << VCPU_REG_CR0) | \ + (1 << VCPU_REG_CR3) | \ + (1 << VCPU_REG_CR4) | \ + (1 << VCPU_REG_EXIT_INFO_1) | \ + (1 << VCPU_REG_EXIT_INFO_2)) =20 static inline unsigned long vmx_l1_guest_owned_cr0_bits(void) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 0a1b63c63d1a..ac05cc289b56 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1090,14 +1090,14 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned lon= g cr3) } =20 /* - * Marking VCPU_EXREG_PDPTR dirty doesn't work for !tdp_enabled. + * Marking VCPU_REG_PDPTR dirty doesn't work for !tdp_enabled. * Shadow page roots need to be reconstructed instead. */ if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); =20 memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)); - kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); vcpu->arch.pdptrs_from_userspace =3D false; =20 @@ -1478,7 +1478,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long = cr3) kvm_mmu_new_pgd(vcpu, cr3); =20 vcpu->arch.cr3 =3D cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); /* Do not call post_set_cr3, we do not get here for confidential guests. = */ =20 handle_tlb_flush: @@ -12473,7 +12473,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu= , struct kvm_sregs *sregs, vcpu->arch.cr2 =3D sregs->cr2; *mmu_reset_needed |=3D kvm_read_cr3(vcpu) !=3D sregs->cr3; vcpu->arch.cr3 =3D sregs->cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); =20 kvm_set_cr8(vcpu, sregs->cr8); @@ -12566,7 +12566,7 @@ static int __set_sregs2(struct kvm_vcpu *vcpu, stru= ct kvm_sregs2 *sregs2) for (i =3D 0; i < 4 ; i++) kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); =20 - kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); mmu_reset_needed =3D 1; vcpu->arch.pdptrs_from_userspace =3D true; } @@ -13111,7 +13111,7 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool ini= t_event) kvm_rip_write(vcpu, 0xfff0); =20 vcpu->arch.cr3 =3D 0; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); =20 /* * CR0.CD/NW are set on RESET, preserved on INIT. Note, some versions @@ -14323,7 +14323,7 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsig= ned long type, gva_t gva) * the RAP (Return Address Predicator). */ if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS)) - kvm_register_is_dirty(vcpu, VCPU_EXREG_ERAPS); + kvm_register_is_dirty(vcpu, VCPU_REG_ERAPS); =20 kvm_invalidate_pcid(vcpu, operand.pcid); return kvm_skip_emulated_instruction(vcpu); @@ -14339,7 +14339,7 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsig= ned long type, gva_t gva) fallthrough; case INVPCID_TYPE_ALL_INCL_GLOBAL: /* - * Don't bother marking VCPU_EXREG_ERAPS dirty, SVM will take + * Don't bother marking VCPU_REG_ERAPS dirty, SVM will take * care of doing so when emulating the full guest TLB flush * (the RAP is cleared on all implicit TLB flushes). */ --=20 2.53.0.1213.gd9a14994de-goog From nobody Sat Jun 20 20:54:04 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EB62397E80 for ; Thu, 9 Apr 2026 22:42:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774575; cv=none; b=XCGXKeLIA6rVI8my1ZqVmbeCmXwmLdlg7eEq9Q7o/HpjlQiGcpRxcaGHFEAquTyb6EFOtp4JItxMT8iw8qMvfZ3DPxYsBQ5fumHwMl1ckZw5UZn5tnOX/1a0Yhffy6kBz4kFcBtdX4G+34F1lzE1P4+NFBRm6Z59FN90L69/o/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774575; c=relaxed/simple; bh=Cfxa2UAE6j7eG3LqQr4Mhci8tHPj0OKBsNtcqmiHvgI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MsNlqcv/6qLl3Xvv1bqOFJPafJWvYs5LmRZhP2FEB6ggZkqx1una3n4MPMYPMilWSfjwti292IFEcf5QKj/MBfIo+O/1enhSDJ5H6VG02GrDVX5TCGo/CfV7gnag7rrkAWBWZhI75BvTKiMNw/fWSa8geNdDJVXxJDrrGNlVdiQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cAJ5+wST; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cAJ5+wST" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c70ea91bfe1so777182a12.1 for ; Thu, 09 Apr 2026 15:42:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1775774573; x=1776379373; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=dy1xR7/tBUVQ93pwLppnnl1roCB8CkMUcZoZrxWj4Ak=; b=cAJ5+wSTxAeEQw2sYFDuWOOjkztukurVzKIvbQQ0gaZVMVf0qSiTpusnxUUwn5b/zj JJTg6ZMSnhZElHat62CxdxGr7QJPQGs68+99uY5UVd+JHSc7XYvTk+6F0J7GIFfNxyTw lkpZLbHZpCIwxiNwdV6FKYmgr26WNd7CKrgo+LJH+G8X1ZqW4lvjk9/XhqmNniIb5I6E qHt/LxRGTJ+XB2V47powtHFuLuHA7USBTfNrEpQZkh8Yknx6WadN3jeIPWR18wjMf+Gb /nDxz+rF9TLjJJ+4PydJh1QlXnkftpLCH76EnibUxlk7bEQqMykugV0NZghchGcCsiu/ xtPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775774573; x=1776379373; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dy1xR7/tBUVQ93pwLppnnl1roCB8CkMUcZoZrxWj4Ak=; b=JuQXtLl1+tBAukJflY3o1/wzREFWj75EKob08VfXIG6s3ffx4FI9mkT3BdDKehJBVd 7bFcZXNxD9FaKaO+DSgcf3/zBYbtzLNL7Jog/b9V4FqzMJPszSe9w1/QDaTKnYIcjyQq a/bSUzNCmJWVq3H0tssZUioFth71bMxzpsdezPn89gHyNfIZgJkMY7wDiNgmMaCdMYQ4 pbtu7kWs8WjTLA0q8Aig5pfgLu5YBaOnvonflQNrdaG94g4qcvB1p4uUhbeJRY9epSto S6l1WHMy42dDQn/tgzQMhErEZgGCkvIA62VlnGbsThubSSTb0GHYpO9wXRQzzXEf4DCy pcwA== X-Forwarded-Encrypted: i=1; AJvYcCW9T2n61nnGhJLzjRrFJvd5Bv5acu6rhK46Rj5YdQmd/y9TqE+DfBV+oH8q/Dfgu/3AnkZAhDRl5McFC2Q=@vger.kernel.org X-Gm-Message-State: AOJu0YxgtfbODq65VdKvk59xD+wr5hGho07Cb/B8KfxfVc8CTJ5V5tTR 5BzM+O9gKyaMXcfOkY9i9zUpR6UJqAr9z2RgVM1JXjnq5cxfa4eHiSPblUMVMGe7nXGFP/go1lv /qwg3XQ== X-Received: from pghv6.prod.google.com ([2002:a63:f206:0:b0:c73:7970:8223]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:9981:b0:39f:43c3:b8b9 with SMTP id adf61e73a8af0-39fe40d7f36mr740699637.57.1775774572752; Thu, 09 Apr 2026 15:42:52 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 9 Apr 2026 15:42:32 -0700 In-Reply-To: <20260409224236.2021562-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260409224236.2021562-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.1213.gd9a14994de-goog Message-ID: <20260409224236.2021562-4-seanjc@google.com> Subject: [PATCH v2 3/6] KVM: nVMX: Do a bitwise-AND of regs_avail when switching active VMCS From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Chang S . Bae" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When switching between vmcs01 and vmcs02, do a bitwise-AND of regs_avail to effectively reset the mask for the new VMCS, purely to be consistent with all other "full" writes of regs_avail. In practice, a straight write versus a bitwise-AND will yield the same result, as kvm_arch_vcpu_create() marks *all* registers available (and dirty), and KVM never marks registers unavailable unless they're lazily loaded. This will allow adding wrapper APIs to set regs_{avail,dirty} without having to add special handling for a nVMX use case that doesn't exist in practice. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Tested-by: Kai Huang --- arch/x86/kvm/vmx/nested.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 22b1f06a9d40..63c4ca8c97d5 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -310,7 +310,7 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, stru= ct loaded_vmcs *vmcs) vmx_sync_vmcs_host_state(vmx, prev); put_cpu(); =20 - vcpu->arch.regs_avail =3D ~VMX_REGS_LAZY_LOAD_SET; + vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; =20 /* * All lazily updated registers will be reloaded from VMCS12 on both --=20 2.53.0.1213.gd9a14994de-goog From nobody Sat Jun 20 20:54:04 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5BF32398915 for ; Thu, 9 Apr 2026 22:42:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774576; cv=none; b=u8X6PwFRmM2fqh7kWcDKqekJabe8QZG+U8+6kAr2QoJ3d+IwKmbLvpnriEhFZYN1WyeA2YbgwgWD6vAPyy7LWFTL5R8wxcIXGbMjh1i14or4lKEq7t1dn1kXipRfjaCqOYnmsgUW8eB9BW3jzDUXiJnnz0AsxreZ6K2xmHElWpE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774576; c=relaxed/simple; bh=Bdr/VKS2Xm+/pYaSrVzYfwfoth1+7klUYzIyzSDdgrQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uc0zPkg4yYwvA1KJ/DVSgiWdCPEpI62hsuxOUtR0K9SNii2gZPBXWjDmU0V2gtFVEeV4VfmB1nqGmK1+0SBqZDTg56nlIRcBTVI0KqDjtIt8gI0Anx+79ZDYA5aSwo/L8oAt86ubUWYzC60gmg3CTROYnf5plX7kIY7SEGzomvE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QcTk9axb; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QcTk9axb" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-82c1e1a6cfbso1108460b3a.0 for ; Thu, 09 Apr 2026 15:42:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1775774575; x=1776379375; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=b6+4lUswVTvjDCVcs/ZLq4/1sWKR4eUi2Zroo4Ie2qI=; b=QcTk9axbmbDBMoY8vQbzv1tx3OAoWXt92fwy1JoabNJkAXcj043OtuxOKCDnPeNPtZ zUVktoJP/7wVb09s6+hbGNXNwy29P3zUQ18e0JtN7TdQTk5UjoaatQY9SHmhBvIZRqRj LIOfqiVNMuei9oxzVa6Q5YiG9+vMI0sqZiHuDU+nTKEtekwFkJfwhyBVvfynGzTtjmA0 tALDblHmrsW8mEnbR0LM4Aqu4V1DSyhXdh5jBpsyf9O+t33yi1dWDDnEMNdlnFSZpDcz C0tLFIBSkCAFzjh6EsbAbUOCzMc8aAVamACHX541Tz1Y8H3+Jy3Wt5US6uxx+CcqFsJd zS9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775774575; x=1776379375; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=b6+4lUswVTvjDCVcs/ZLq4/1sWKR4eUi2Zroo4Ie2qI=; b=lFmNGCDFYmW8jcmdofKWWl1yg99651P7znk/OZMzgqSxCz9WLHDE3jnk8Q0a+nMZqX YDAUMvjIuSLbLLDTuGksuUiB9j4sQeqiYWYQe9FuAQ4fKPB3sz9RwQKgRc6xsT8JnxM6 uD4qkxVhePB4M9/+/aP5C+i9qpeiClL0sPaF799iI6Q2+F8MvRhhDn/eOkWxCghEwwRY WpKZ7z9eEn6TGiYaJU6UyN4QZ/X3kXpOAWTkZtGZSSOpvmIPck2yhQs9oF6/U2gx3djO CMZ3CcYzLSMGXy+Iz76ovROWuNHcJT8SyllOyNy2OUTVN3HFW8OJyiamN02Zu3s51vax Ts7A== X-Forwarded-Encrypted: i=1; AJvYcCW+0dROsc+Ww6FZCcdbutEE6nWKZ3y3DPJW7K/eD7jXQYwgtl/j98GjK22KUEHIx4sj3GMgYxtT+mbQImY=@vger.kernel.org X-Gm-Message-State: AOJu0YwfpmtPrdSy+ide/FnPTZtNEeJJEv+O9MlbY/EVh6gssC9U0aVd wn8O4DDNF+TtUdAt2JevywxWRSpux7/NbAEslRpo7yG1zu32IjZEAJZjnt573bUARLcAD9gB7rC 7n5qpZA== X-Received: from pfwz16.prod.google.com ([2002:a05:6a00:1d90:b0:824:a65c:adf]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:12cc:b0:82c:eafa:8875 with SMTP id d2e1a72fcca58-82f0c26b33bmr964653b3a.2.1775774574526; Thu, 09 Apr 2026 15:42:54 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 9 Apr 2026 15:42:33 -0700 In-Reply-To: <20260409224236.2021562-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260409224236.2021562-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.1213.gd9a14994de-goog Message-ID: <20260409224236.2021562-5-seanjc@google.com> Subject: [PATCH v2 4/6] KVM: x86: Add wrapper APIs to reset dirty/available register masks From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Chang S . Bae" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add wrappers for setting regs_{avail,dirty} in anticipation of turning the fields into proper bitmaps, at which point direct writes won't work so well. Deliberately leave the initialization in kvm_arch_vcpu_create() as-is, because the regs_avail logic in particular is special in that it's the one and only place where KVM marks eagerly synchronized registers as available. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Tested-by: Kai Huang --- arch/x86/kvm/kvm_cache_regs.h | 18 ++++++++++++++++++ arch/x86/kvm/svm/svm.c | 4 ++-- arch/x86/kvm/vmx/nested.c | 4 ++-- arch/x86/kvm/vmx/tdx.c | 2 +- arch/x86/kvm/vmx/vmx.c | 4 ++-- 5 files changed, 25 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index ac1f9867a234..7f71d468178c 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -105,6 +105,24 @@ static __always_inline bool kvm_register_test_and_mark= _available(struct kvm_vcpu return arch___test_and_set_bit(reg, (unsigned long *)&vcpu->arch.regs_ava= il); } =20 +static __always_inline void kvm_clear_available_registers(struct kvm_vcpu = *vcpu, + u32 clear_mask) +{ + /* + * Note the bitwise-AND! In practice, a straight write would also work + * as KVM initializes the mask to all ones and never clears registers + * that are eagerly synchronized. Using a bitwise-AND adds a bit of + * sanity checking as incorrectly marking an eagerly sync'd register + * unavailable will generate a WARN due to an unexpected cache request. + */ + vcpu->arch.regs_avail &=3D ~clear_mask; +} + +static __always_inline void kvm_reset_dirty_registers(struct kvm_vcpu *vcp= u) +{ + vcpu->arch.regs_dirty =3D 0; +} + /* * The "raw" register helpers are only for cases where the full 64 bits of= a * register are read/written irrespective of current vCPU mode. In other = words, diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index ee5749d8b3e8..2b73d2650155 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4508,7 +4508,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) vcpu->arch.regs[VCPU_REGS_RSP] =3D svm->vmcb->save.rsp; vcpu->arch.rip =3D svm->vmcb->save.rip; } - vcpu->arch.regs_dirty =3D 0; + kvm_reset_dirty_registers(vcpu); =20 if (unlikely(svm->vmcb->control.exit_code =3D=3D SVM_EXIT_NMI)) kvm_before_interrupt(vcpu, KVM_HANDLING_NMI); @@ -4554,7 +4554,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_= vcpu *vcpu, u64 run_flags) vcpu->arch.apf.host_apf_flags =3D kvm_read_and_reset_apf_flags(); =20 - vcpu->arch.regs_avail &=3D ~SVM_REGS_LAZY_LOAD_SET; + kvm_clear_available_registers(vcpu, SVM_REGS_LAZY_LOAD_SET); =20 if (!msr_write_intercepted(vcpu, MSR_AMD64_PERF_CNTR_GLOBAL_CTL)) rdmsrq(MSR_AMD64_PERF_CNTR_GLOBAL_CTL, vcpu_to_pmu(vcpu)->global_ctrl); diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 63c4ca8c97d5..c4d2bc080add 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -310,13 +310,13 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, st= ruct loaded_vmcs *vmcs) vmx_sync_vmcs_host_state(vmx, prev); put_cpu(); =20 - vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; + kvm_clear_available_registers(vcpu, VMX_REGS_LAZY_LOAD_SET); =20 /* * All lazily updated registers will be reloaded from VMCS12 on both * vmentry and vmexit. */ - vcpu->arch.regs_dirty =3D 0; + kvm_reset_dirty_registers(vcpu); } =20 static void nested_put_vmcs12_pages(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index c23ec4ac8bc8..c9ab7902151f 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1098,7 +1098,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 ru= n_flags) =20 tdx_load_host_xsave_state(vcpu); =20 - vcpu->arch.regs_avail &=3D TDX_REGS_AVAIL_SET; + kvm_clear_available_registers(vcpu, ~(u32)TDX_REGS_AVAIL_SET); =20 if (unlikely(tdx->vp_enter_ret =3D=3D EXIT_REASON_EPT_MISCONFIG)) return EXIT_FASTPATH_NONE; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index aa1c26018439..61eeafcd70f1 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7472,7 +7472,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vc= pu *vcpu, flags); =20 vcpu->arch.cr2 =3D native_read_cr2(); - vcpu->arch.regs_avail &=3D ~VMX_REGS_LAZY_LOAD_SET; + kvm_clear_available_registers(vcpu, VMX_REGS_LAZY_LOAD_SET); =20 vmx->idt_vectoring_info =3D 0; =20 @@ -7538,7 +7538,7 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 ru= n_flags) vmcs_writel(GUEST_RSP, vcpu->arch.regs[VCPU_REGS_RSP]); if (kvm_register_is_dirty(vcpu, VCPU_REG_RIP)) vmcs_writel(GUEST_RIP, vcpu->arch.rip); - vcpu->arch.regs_dirty =3D 0; + kvm_reset_dirty_registers(vcpu); =20 if (run_flags & KVM_RUN_LOAD_GUEST_DR6) set_debugreg(vcpu->arch.dr6, 6); --=20 2.53.0.1213.gd9a14994de-goog From nobody Sat Jun 20 20:54:04 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68C4D396B8E for ; Thu, 9 Apr 2026 22:42:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774578; cv=none; b=QYReC7LPkdOKGDJp4XNArfLEnyAut5fl+fk72nbO0Pbu7N2pgRbx6NWZNxrX5WrcwxVDbonzh78WzBfr7n1Z2wjN+GjvR+ASGcJyKnypjC9qOovE59karJBox4IBIP57f2nFmFo45JWuAEhvruyNdKcDzOPimYAUGZZTxDoErh0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774578; c=relaxed/simple; bh=60Q/zHtyyGDIBP/ak7485shWUHxtFzoB9QyfDmyCpMc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fm00oV1q+SUayGJj9ohEVLHAxNu+fFWP9kQx0DB0Ypvr4RQ+zRbmeWzex/iwH9ssvSUoDUz9b7ch7sX54XEa3+cTg8L5vTS1Iow1HGgdmWcQ7fAeFPJcRP0bJVHKz3jzL1AEVDx2Z/kh245PrUM8d7xyOyd0tDGViDEomNyLFTo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fiVQyvVz; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fiVQyvVz" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-82c6929bd26so1455193b3a.3 for ; Thu, 09 Apr 2026 15:42:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1775774577; x=1776379377; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=7rpQhjNm3mTGjbc05HVtkw07FHdO4o+YZx1lMSvn+bc=; b=fiVQyvVz2bhZQHA87I5g+R/Xxvnvl3jw+A5EwEIveFUBSbD0Au/julNDq1LlLuPhiF a2oqMSvTLaou8FD++Vcz4uv0N/XOtT/zytdENxtTMJccafSk/c67eOPuty+QAFA5fcQI FDoxSXvmBpRXlmMbkD1YRtRo2peqNFdsuxH56FfwV9oY3EXkX5SEJVW3G5TLY3pIcFK6 5UGdB/CcOoJij2JPsEMr5hEwAOb/cBb6lPk1iYIHjunM0w7KtMmlLJcbLqKZ84zZMQZi TsQoUfY5JK9q6LRRWeJH4P9f8+D5Gn7qDw9wQ3waeA7nRCHHFgMmIgMuk0ysc5/Z1+Kf dtlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775774577; x=1776379377; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7rpQhjNm3mTGjbc05HVtkw07FHdO4o+YZx1lMSvn+bc=; b=NjJh3+Lyl27RFf5aApepceJr6ple6amlxgyfCG11kDGnGiuzaEfE3DAIBUhTWMO/Mx ihjSL5IwaVXOBI9+a2Ha2ACetvIMB4U4jAaieqyFpTVVquNYbDBuQ5NQ7cvL4V1a2vBP eEdCMR7doH0qwHlOeQiWJ3zIdpUXUc3UdNAP7UehfzAVA8UqaL4HzAgyNmGqfIvjRs65 APi4ycX249WdRjuKj+D9iMuj6195zC46ZsDEJWnaAFiq8FNbX+tIBu+sq94jiwloXdav N8e0qBprAu3h7cbuECPcwiHRIZjh0GQA9RkoddSm8KC6+8a0eB7gAkCTVWrX0Gg5FQlW 3Lrw== X-Forwarded-Encrypted: i=1; AJvYcCV9yinrkeRn8AHMkIHcVJ1RjCANFwRNTygjO+z+tvCh2aT59wlnWOi2TeBhTvsctGnDf3LNFRi95DIZUwY=@vger.kernel.org X-Gm-Message-State: AOJu0YwkaFtQu685XEGgtt6H+Kgi30qpeVrMy7PDNRvKTD2MwmAnWQIe tkMv3imbP5iCNE5U9kwDK5pMIqzg5+kYJgt0Jnf+SxX7fF/gm1PNDOFotgCw6yYl5A2m7y0IfZ6 Kf6kluQ== X-Received: from pfbdh4.prod.google.com ([2002:a05:6a00:4784:b0:829:94bc:6c15]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4c98:b0:82c:ec1b:9e15 with SMTP id d2e1a72fcca58-82f0c1e99bbmr915561b3a.1.1775774576585; Thu, 09 Apr 2026 15:42:56 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 9 Apr 2026 15:42:34 -0700 In-Reply-To: <20260409224236.2021562-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260409224236.2021562-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.1213.gd9a14994de-goog Message-ID: <20260409224236.2021562-6-seanjc@google.com> Subject: [PATCH v2 5/6] KVM: x86: Track available/dirty register masks as "unsigned long" values From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Chang S . Bae" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert regs_{avail,dirty} and all related masks to "unsigned long" values as an intermediate step towards declaring the fields as actual bitmaps, and as a step toward support APX, which will push the total number of registers beyond 32 on 64-bit kernels. Opportunistically convert TDX's ULL bitmask to a UL to match everything else (TDX is 64-bit only, so it's a nop in the end). No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Tested-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/kvm_cache_regs.h | 2 +- arch/x86/kvm/svm/svm.h | 2 +- arch/x86/kvm/vmx/tdx.c | 36 ++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.h | 20 +++++++++--------- 5 files changed, 32 insertions(+), 32 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index b1eae1e7b04f..c47eb294c066 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -802,8 +802,8 @@ struct kvm_vcpu_arch { */ unsigned long regs[NR_VCPU_GENERAL_PURPOSE_REGS]; unsigned long rip; - u32 regs_avail; - u32 regs_dirty; + unsigned long regs_avail; + unsigned long regs_dirty; =20 unsigned long cr0; unsigned long cr0_guest_owned_bits; diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index 7f71d468178c..171e6bc2e169 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -106,7 +106,7 @@ static __always_inline bool kvm_register_test_and_mark_= available(struct kvm_vcpu } =20 static __always_inline void kvm_clear_available_registers(struct kvm_vcpu = *vcpu, - u32 clear_mask) + unsigned long clear_mask) { /* * Note the bitwise-AND! In practice, a straight write would also work diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 677d268ae9c7..7b46a3f13de1 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -474,7 +474,7 @@ static inline bool svm_is_vmrun_failure(u64 exit_code) * KVM_REQ_LOAD_MMU_PGD is always requested when the cached vcpu->arch.cr3 * is changed. svm_load_mmu_pgd() then syncs the new CR3 value into the V= MCB. */ -#define SVM_REGS_LAZY_LOAD_SET (1 << VCPU_REG_PDPTR) +#define SVM_REGS_LAZY_LOAD_SET (BIT(VCPU_REG_PDPTR)) =20 static inline void __vmcb_set_intercept(unsigned long *intercepts, u32 bit) { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index c9ab7902151f..85f28363e4cc 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1013,23 +1013,23 @@ static fastpath_t tdx_exit_handlers_fastpath(struct= kvm_vcpu *vcpu) return EXIT_FASTPATH_NONE; } =20 -#define TDX_REGS_AVAIL_SET (BIT_ULL(VCPU_REG_EXIT_INFO_1) | \ - BIT_ULL(VCPU_REG_EXIT_INFO_2) | \ - BIT_ULL(VCPU_REGS_RAX) | \ - BIT_ULL(VCPU_REGS_RBX) | \ - BIT_ULL(VCPU_REGS_RCX) | \ - BIT_ULL(VCPU_REGS_RDX) | \ - BIT_ULL(VCPU_REGS_RBP) | \ - BIT_ULL(VCPU_REGS_RSI) | \ - BIT_ULL(VCPU_REGS_RDI) | \ - BIT_ULL(VCPU_REGS_R8) | \ - BIT_ULL(VCPU_REGS_R9) | \ - BIT_ULL(VCPU_REGS_R10) | \ - BIT_ULL(VCPU_REGS_R11) | \ - BIT_ULL(VCPU_REGS_R12) | \ - BIT_ULL(VCPU_REGS_R13) | \ - BIT_ULL(VCPU_REGS_R14) | \ - BIT_ULL(VCPU_REGS_R15)) +#define TDX_REGS_AVAIL_SET (BIT(VCPU_REG_EXIT_INFO_1) | \ + BIT(VCPU_REG_EXIT_INFO_2) | \ + BIT(VCPU_REGS_RAX) | \ + BIT(VCPU_REGS_RBX) | \ + BIT(VCPU_REGS_RCX) | \ + BIT(VCPU_REGS_RDX) | \ + BIT(VCPU_REGS_RBP) | \ + BIT(VCPU_REGS_RSI) | \ + BIT(VCPU_REGS_RDI) | \ + BIT(VCPU_REGS_R8) | \ + BIT(VCPU_REGS_R9) | \ + BIT(VCPU_REGS_R10) | \ + BIT(VCPU_REGS_R11) | \ + BIT(VCPU_REGS_R12) | \ + BIT(VCPU_REGS_R13) | \ + BIT(VCPU_REGS_R14) | \ + BIT(VCPU_REGS_R15)) =20 static void tdx_load_host_xsave_state(struct kvm_vcpu *vcpu) { @@ -1098,7 +1098,7 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 ru= n_flags) =20 tdx_load_host_xsave_state(vcpu); =20 - kvm_clear_available_registers(vcpu, ~(u32)TDX_REGS_AVAIL_SET); + kvm_clear_available_registers(vcpu, ~TDX_REGS_AVAIL_SET); =20 if (unlikely(tdx->vp_enter_ret =3D=3D EXIT_REASON_EPT_MISCONFIG)) return EXIT_FASTPATH_NONE; diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 9fb76ea48caf..48447fa983f4 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -620,16 +620,16 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC= _CONTROL, 64) * cache on demand. Other registers not listed here are synced to * the cache immediately after VM-Exit. */ -#define VMX_REGS_LAZY_LOAD_SET ((1 << VCPU_REG_RIP) | \ - (1 << VCPU_REGS_RSP) | \ - (1 << VCPU_REG_RFLAGS) | \ - (1 << VCPU_REG_PDPTR) | \ - (1 << VCPU_REG_SEGMENTS) | \ - (1 << VCPU_REG_CR0) | \ - (1 << VCPU_REG_CR3) | \ - (1 << VCPU_REG_CR4) | \ - (1 << VCPU_REG_EXIT_INFO_1) | \ - (1 << VCPU_REG_EXIT_INFO_2)) +#define VMX_REGS_LAZY_LOAD_SET (BIT(VCPU_REGS_RSP) | \ + BIT(VCPU_REG_RIP) | \ + BIT(VCPU_REG_RFLAGS) | \ + BIT(VCPU_REG_PDPTR) | \ + BIT(VCPU_REG_SEGMENTS) | \ + BIT(VCPU_REG_CR0) | \ + BIT(VCPU_REG_CR3) | \ + BIT(VCPU_REG_CR4) | \ + BIT(VCPU_REG_EXIT_INFO_1) | \ + BIT(VCPU_REG_EXIT_INFO_2)) =20 static inline unsigned long vmx_l1_guest_owned_cr0_bits(void) { --=20 2.53.0.1213.gd9a14994de-goog From nobody Sat Jun 20 20:54:04 2026 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDF33399368 for ; Thu, 9 Apr 2026 22:42:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774580; cv=none; b=iqai4nf4V3Vgn0xvV54w9Mal8kI4wPgBmWHRkYj0kKuoKR0BOsqUQSIUnPNBGmcjh58l0YUDzOOZIxyoLyNysZT1ebFAJR+TDONVkrZYks6oFCb0jEgxhxc2M0sPI69fP9x/E7CF26Hz1WBt+ZONfuwXTxbWpcPcJ1TttF87dZw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775774580; c=relaxed/simple; bh=l2tksM2qxMIs0YsaAut7K8eN8vuMtiw3vlyXTQPw/No=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bvflfY3oO/RHF3BBVmqgqxHL6qLyuYy3oBImbmsa1MBWfY2OWPr0QYRjsZ6YHjZ1IUPEhV+w40BLT5h9MfUyLcWwK4ZDlKBR/Mbf2fkdPzOL+BIHvXpJdZlNuWVEUcSCGVw7e96XAyoPToLYczBmGzk68vm1VAjDF7WAJYXR2cA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Pl83S6tT; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Pl83S6tT" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-82ca8323edbso1387108b3a.1 for ; Thu, 09 Apr 2026 15:42:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1775774578; x=1776379378; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=tDN4tVJ6/vh7WLhGggTNs8OwwMerYRn5pnEFOX+nQmM=; b=Pl83S6tTd4UwI4wHXxgfOiJ7X2ue8aHp776IGXd+RzZx9HZ8diYl8iGu2Z/amTSHQg ysT41Kn+UbT0Ba2hHRQs6L5ara5JdHDis7k44tUvVCNWnO0NJC3VgAxTf0J2t474HCfm xZ4VAYTwY8MAl/Ojg1IVHgQEeS9L/m6Xx6oPs+/KcmXSPhNyc9R6s8/ulnxw5YUJVH/B hNUvhy1MOsZEoeUYANPUzk2aHzQ35vfwQ3O2ZVfiIoFq5w2QOfq9S3pj7IOWp620EksS i8v3NOolTUT5B5LwvCuHoHNsrpgJgNC9WCzJRZ8SulrrU9XrAtWOIp07rQp2EFfTfEDh k4AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775774578; x=1776379378; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tDN4tVJ6/vh7WLhGggTNs8OwwMerYRn5pnEFOX+nQmM=; b=CUXSpdAjWCY2/MgX19kSOhxqANFSgZevJ26qu77m6cBUmmyt24ezSZHnZxgJdPn3n/ 2tJRm+aT6YJoP2C8//pcqV6rmGviscKRN5Ij+FP3r4YSNDU8XLJ/Pd8U4Z3SMtRfCPUM 4I718ZthaDGJeHCyz6YyyRAEYggCx9FZ+yR9J0PQzDnXNkG9SKxhxB9Jg8LIdbNG/Wic ZakaH2cQDLTchshRgpS22GPz6JOXd+Ht5sM9IWrwag2gKFu7yjPbXEmoSrzs4nAgQYnJ dKsW0h7395TbMxJ+pQcGmE/eFN5mnnt79OlMH+VCblJZsp9QLTzsBxCkMBz9ANDOI1YB ToOQ== X-Forwarded-Encrypted: i=1; AJvYcCXJv6GzT7IiDPndAOy4Fy2CCdMvr42TuLhnyBUkD/1HkJlP5JWZ5znJVtsk81kWBX+zlmJMwqPsc3b9TgA=@vger.kernel.org X-Gm-Message-State: AOJu0YzEDgS/4iTprNgvf2weRZH8KEb2shM4k+zv32z/0vr5cE8mF6qg sXfIaeOm2lsaxomSLWh240XFlhX/L+0qnWRyVOvSxSW8V8eIBDvgw3wcRZHmqoCECN/2QkGseYm ZRT8JDQ== X-Received: from pfoo1.prod.google.com ([2002:a05:6a00:1a01:b0:82c:9835:bf12]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:12e0:b0:821:7d7e:41cd with SMTP id d2e1a72fcca58-82f0c1ea02fmr979859b3a.10.1775774578196; Thu, 09 Apr 2026 15:42:58 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 9 Apr 2026 15:42:35 -0700 In-Reply-To: <20260409224236.2021562-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260409224236.2021562-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.1213.gd9a14994de-goog Message-ID: <20260409224236.2021562-7-seanjc@google.com> Subject: [PATCH v2 6/6] KVM: x86: Use a proper bitmap for tracking available/dirty registers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Chang S . Bae" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define regs_{avail,dirty} as bitmaps instead of U32s to harden against overflow, and to allow for dynamically sizing the bitmaps when APX comes along, which will add 16 more GPRs (R16-R31) and thus increase the total number of registers beyond 32. Open code writes in the "reset" APIs, as the writes are hot paths and bitmap_write() is complete overkill for what KVM needs. Even better, hardcoding writes to entry '0' in the array is a perfect excuse to assert that the array contains exactly one entry, e.g. to effectively add guard against defining R16-R31 in 32-bit kernels. For all intents and purposes, no functional change intended even though using bitmap_fill() will mean "undefined" registers are no longer marked available and dirty (KVM should never be querying those bits). Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Tested-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 6 ++++-- arch/x86/kvm/kvm_cache_regs.h | 20 ++++++++++++-------- arch/x86/kvm/x86.c | 4 ++-- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index c47eb294c066..ef0c368676c5 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -211,6 +211,8 @@ enum kvm_reg { VCPU_REG_SEGMENTS, VCPU_REG_EXIT_INFO_1, VCPU_REG_EXIT_INFO_2, + + NR_VCPU_TOTAL_REGS, }; =20 enum { @@ -802,8 +804,8 @@ struct kvm_vcpu_arch { */ unsigned long regs[NR_VCPU_GENERAL_PURPOSE_REGS]; unsigned long rip; - unsigned long regs_avail; - unsigned long regs_dirty; + DECLARE_BITMAP(regs_avail, NR_VCPU_TOTAL_REGS); + DECLARE_BITMAP(regs_dirty, NR_VCPU_TOTAL_REGS); =20 unsigned long cr0; unsigned long cr0_guest_owned_bits; diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index 171e6bc2e169..2ae492ad6412 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -67,29 +67,29 @@ static inline bool kvm_register_is_available(struct kvm= _vcpu *vcpu, enum kvm_reg reg) { kvm_assert_register_caching_allowed(vcpu); - return test_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); + return test_bit(reg, vcpu->arch.regs_avail); } =20 static inline bool kvm_register_is_dirty(struct kvm_vcpu *vcpu, enum kvm_reg reg) { kvm_assert_register_caching_allowed(vcpu); - return test_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty); + return test_bit(reg, vcpu->arch.regs_dirty); } =20 static inline void kvm_register_mark_available(struct kvm_vcpu *vcpu, enum kvm_reg reg) { kvm_assert_register_caching_allowed(vcpu); - __set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); + __set_bit(reg, vcpu->arch.regs_avail); } =20 static inline void kvm_register_mark_dirty(struct kvm_vcpu *vcpu, enum kvm_reg reg) { kvm_assert_register_caching_allowed(vcpu); - __set_bit(reg, (unsigned long *)&vcpu->arch.regs_avail); - __set_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty); + __set_bit(reg, vcpu->arch.regs_avail); + __set_bit(reg, vcpu->arch.regs_dirty); } =20 /* @@ -102,12 +102,15 @@ static __always_inline bool kvm_register_test_and_mar= k_available(struct kvm_vcpu enum kvm_reg reg) { kvm_assert_register_caching_allowed(vcpu); - return arch___test_and_set_bit(reg, (unsigned long *)&vcpu->arch.regs_ava= il); + return arch___test_and_set_bit(reg, vcpu->arch.regs_avail); } =20 static __always_inline void kvm_clear_available_registers(struct kvm_vcpu = *vcpu, unsigned long clear_mask) { + BUILD_BUG_ON(sizeof(clear_mask) !=3D sizeof(vcpu->arch.regs_avail[0])); + BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.regs_avail) !=3D 1); + /* * Note the bitwise-AND! In practice, a straight write would also work * as KVM initializes the mask to all ones and never clears registers @@ -115,12 +118,13 @@ static __always_inline void kvm_clear_available_regis= ters(struct kvm_vcpu *vcpu, * sanity checking as incorrectly marking an eagerly sync'd register * unavailable will generate a WARN due to an unexpected cache request. */ - vcpu->arch.regs_avail &=3D ~clear_mask; + vcpu->arch.regs_avail[0] &=3D ~clear_mask; } =20 static __always_inline void kvm_reset_dirty_registers(struct kvm_vcpu *vcp= u) { - vcpu->arch.regs_dirty =3D 0; + BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.regs_dirty) !=3D 1); + vcpu->arch.regs_dirty[0] =3D 0; } =20 /* diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index ac05cc289b56..b8a91feec8e1 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -12836,8 +12836,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) int r; =20 vcpu->arch.last_vmentry_cpu =3D -1; - vcpu->arch.regs_avail =3D ~0; - vcpu->arch.regs_dirty =3D ~0; + bitmap_fill(vcpu->arch.regs_avail, NR_VCPU_TOTAL_REGS); + bitmap_fill(vcpu->arch.regs_dirty, NR_VCPU_TOTAL_REGS); =20 kvm_gpc_init(&vcpu->arch.pv_time, vcpu->kvm); =20 --=20 2.53.0.1213.gd9a14994de-goog