From nobody Wed Oct 30 22:11:10 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E9C3146D7A for ; Thu, 1 Aug 2024 18:34:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537300; cv=none; b=jEG+LY2AVRY1197kYX29suJNUWQaRXeApoEp/cxp9SVmrOcjz5y4moFPmuRy/VCJaaMTLv9O3/nu/86/GQRFpdMA8iA+lgOKP3Vnv/bIdGf2W4SgEFABlZuzFL32Vl4sJTMHGkO2vOiao8CzRKcxu2BwHbWQeBFlM+ZgmW/J5F8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537300; c=relaxed/simple; bh=B4M5zM2LxlbX4e+OKo938Pgq8tauyjOo2Mz9UK1WRwE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=X9plHNFGXjcFFaoL1YaUyffLyU3qzuQdZHvTEeujguU+jmZxWxbP8r0p2+MCNs2FZKPOZR+w0VpieodJYzfkmvP08CvrEbX1sTy38hufzt0p72Q6XAa2PxxIaMYI5sR+JSnMbjgTIolFs8i4qRrx7d4viwTVYAvRWPjwRTuCfIk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xqnkioJg; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xqnkioJg" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e0be2fa8f68so53262276.3 for ; Thu, 01 Aug 2024 11:34:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537298; x=1723142098; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=pYuydgqnu7KvS+vD/kJx/xLlR2Fg91BOlW6zTjcno7Q=; b=xqnkioJgRWzyMTa9C3AyqvcLKXN1NXw1GUktnC8nbyN5UNZOJf49BsVPbZikkX1xXR x+4xK5Q2URGVSmX0ZbAQ70ZwN9eFbPW6jrUJuqA2rhMI2vzLwHxVK/CU8VHU26QyyPmv sB5LDDsFOym5GSofCSESv1Iop7cZzQ1TOMBN38YOTaHqQEClvXoed/8ZO0DEbpk674Xb EYzGWJbuNXZJrepSbzsqm/WUThX5yu3M8+5kAUN93pvcVDOl8Ki8vMwdJnKUc46M73TZ VP5TFcLtBjfYB1JCMysdb/tUVWiJCKM7sAKuL7c9p2o+GDrq/Fgc5vtCKYUnwadbuB7b 0AFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537298; x=1723142098; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pYuydgqnu7KvS+vD/kJx/xLlR2Fg91BOlW6zTjcno7Q=; b=NUmwf6cEfErsl/zCt6EKoszB9RCSPnoZLexsJarAJoeQ5nGP/svTbHmUfjVdYEIZXL 2wbjoPwZmbF9r1o6v+9InXtn4/zncfOyr/jhfFbIoP5vJKpo099L9hGmL3+41t6tsr3W Wen8wxxIZwE797H+f+2GBuVaHjLpuX/t3BbyXhBiuEfwB5KAd3topzDBFLCxu5eQJFdM IiT9/oV991A6Bhifd85oU+rxH7ba7YSkJRe5mOYZfTK2HnQPbMHpMPHdO0fkTI0weEO2 QhEVms6LLWSxQNByLChJZPd3qi9FwTbCrfifCRRBSoHmLwR4YKNdBAj9MLCX81f3kPB0 7Jgw== X-Forwarded-Encrypted: i=1; AJvYcCWgr7zBK7I13mWfDjkaTj5uY7ybTkwJShaaCWXuS8s2ZRSRsWRIG4kRCJN2Nd8+6z4G+Ts2rUVKc+ES5LDuixGH4NSVl5qGxWi9le8A X-Gm-Message-State: AOJu0Yy54DJ879TOSjUV9Jot2Xt25kmACTcS6zfFCZXMEOzZucRJ4xe7 GZZjorXfnP5PjtxcGAFLHb5qjbTf1SaUdQhz42dU3I3MF92bjglRNCPyfxYKO3V58lxHLdCUoJF EbA== X-Google-Smtp-Source: AGHT+IFfRgjDzueDHA1wB4xIJlPq2Sh0SGDxzPYZTvnWrBT2GBLIr7FzlALzApVC+WiYgVOffmnlJ9uzeLA= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:2b02:b0:e03:31ec:8a24 with SMTP id 3f1490d57ef6-e0bde422b24mr29349276.8.1722537298373; Thu, 01 Aug 2024 11:34:58 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:45 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-2-seanjc@google.com> Subject: [RFC PATCH 1/9] KVM: x86/mmu: Add a dedicated flag to track if A/D bits are globally enabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a dedicated flag to track if KVM has enabled A/D bits at the module level, instead of inferring the state based on whether or not the MMU's shadow_accessed_mask is non-zero. This will allow defining and using shadow_accessed_mask even when A/D bits aren't used by hardware. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 6 +++--- arch/x86/kvm/mmu/spte.c | 6 ++++++ arch/x86/kvm/mmu/spte.h | 20 +++++++++----------- arch/x86/kvm/mmu/tdp_mmu.c | 4 ++-- 4 files changed, 20 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 5979eeb916cd..1e24bc4a06db 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3319,7 +3319,7 @@ static bool page_fault_can_be_fast(struct kvm *kvm, s= truct kvm_page_fault *fault * by setting the Writable bit, which can be done out of mmu_lock. */ if (!fault->present) - return !kvm_ad_enabled(); + return !kvm_ad_enabled; =20 /* * Note, instruction fetches and writes are mutually exclusive, ignore @@ -3454,7 +3454,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, str= uct kvm_page_fault *fault) * uses A/D bits for non-nested MMUs. Thus, if A/D bits are * enabled, the SPTE can't be an access-tracked SPTE. */ - if (unlikely(!kvm_ad_enabled()) && is_access_track_spte(spte)) + if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) new_spte =3D restore_acc_track_spte(new_spte); =20 /* @@ -5429,7 +5429,7 @@ kvm_calc_tdp_mmu_root_page_role(struct kvm_vcpu *vcpu, role.efer_nx =3D true; role.smm =3D cpu_role.base.smm; role.guest_mode =3D cpu_role.base.guest_mode; - role.ad_disabled =3D !kvm_ad_enabled(); + role.ad_disabled =3D !kvm_ad_enabled; role.level =3D kvm_mmu_get_tdp_level(vcpu); role.direct =3D true; role.has_4_byte_gpte =3D false; diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 2c5650390d3b..b713a6542eeb 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -24,6 +24,8 @@ static bool __ro_after_init allow_mmio_caching; module_param_named(mmio_caching, enable_mmio_caching, bool, 0444); EXPORT_SYMBOL_GPL(enable_mmio_caching); =20 +bool __read_mostly kvm_ad_enabled; + u64 __read_mostly shadow_host_writable_mask; u64 __read_mostly shadow_mmu_writable_mask; u64 __read_mostly shadow_nx_mask; @@ -435,6 +437,8 @@ EXPORT_SYMBOL_GPL(kvm_mmu_set_me_spte_mask); =20 void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_exec_only) { + kvm_ad_enabled =3D has_ad_bits; + shadow_user_mask =3D VMX_EPT_READABLE_MASK; shadow_accessed_mask =3D has_ad_bits ? VMX_EPT_ACCESS_BIT : 0ull; shadow_dirty_mask =3D has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; @@ -468,6 +472,8 @@ void kvm_mmu_reset_all_pte_masks(void) u8 low_phys_bits; u64 mask; =20 + kvm_ad_enabled =3D true; + /* * If the CPU has 46 or less physical address bits, then set an * appropriate mask to guard against L1TF attacks. Otherwise, it is diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index ef793c459b05..d722b37b7434 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -167,6 +167,15 @@ static_assert(!(SHADOW_NONPRESENT_VALUE & SPTE_MMU_PRE= SENT_MASK)); #define SHADOW_NONPRESENT_VALUE 0ULL #endif =20 + +/* + * True if A/D bits are supported in hardware and are enabled by KVM. When + * enabled, KVM uses A/D bits for all non-nested MMUs. Because L1 can dis= able + * A/D bits in EPTP12, SP and SPTE variants are needed to handle the scena= rio + * where KVM is using A/D bits for L1, but not L2. + */ +extern bool __read_mostly kvm_ad_enabled; + extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; @@ -285,17 +294,6 @@ static inline bool is_ept_ve_possible(u64 spte) (spte & VMX_EPT_RWX_MASK) !=3D VMX_EPT_MISCONFIG_WX_VALUE; } =20 -/* - * Returns true if A/D bits are supported in hardware and are enabled by K= VM. - * When enabled, KVM uses A/D bits for all non-nested MMUs. Because L1 can - * disable A/D bits in EPTP12, SP and SPTE variants are needed to handle t= he - * scenario where KVM is using A/D bits for L1, but not L2. - */ -static inline bool kvm_ad_enabled(void) -{ - return !!shadow_accessed_mask; -} - static inline bool sp_ad_disabled(struct kvm_mmu_page *sp) { return sp->role.ad_disabled; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index dc153cf92a40..2b0fc601d2ce 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1072,7 +1072,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm= _vcpu *vcpu, static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, struct kvm_mmu_page *sp, bool shared) { - u64 spte =3D make_nonleaf_spte(sp->spt, !kvm_ad_enabled()); + u64 spte =3D make_nonleaf_spte(sp->spt, !kvm_ad_enabled); int ret =3D 0; =20 if (shared) { @@ -1488,7 +1488,7 @@ static bool tdp_mmu_need_write_protect(struct kvm_mmu= _page *sp) * from level, so it is valid to key off any shadow page to determine if * write protection is needed for an entire tree. */ - return kvm_mmu_page_ad_need_write_protect(sp) || !kvm_ad_enabled(); + return kvm_mmu_page_ad_need_write_protect(sp) || !kvm_ad_enabled; } =20 static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *ro= ot, --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:10 2024 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E3AE1494C3 for ; Thu, 1 Aug 2024 18:35:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537302; cv=none; b=mPZVX3SHYGJKtLhAwDPl5D/Q/BrAbmhzbJlXEXN1bF0Ww63xkpHq9MhQvncJhNfISgPJAOpAH+pKh8BVYQoWbJKe0IvpGcVAkwgR5lsUQQfVnbAo8Gm3QHmRlqtXQaYshd2fmnNb0dWDNI3HrGVt4/D4hn3l24nX3LxI4RQufhk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537302; c=relaxed/simple; bh=sQ3534Fng9t06Zq2te+F41BHRISWvOlzZzrXklcUmXg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=il5ZAnggOIVgO2lSLMAPNJzXECmKyy6SSqR5HGwkMwMR3xHFY7/6JLvkaiNNazNKvx4TIHGj6tsCwVyhnt+Nvo92mU5Kv+okB4QYUwTWfagQgWiXu6VuenzpDt32+bnoAEIttWCmROzrmwcejjthFzBF1De5TTQIZIzhEErtXIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tvaSzOsE; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tvaSzOsE" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1ff192decb8so55806325ad.0 for ; Thu, 01 Aug 2024 11:35:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537300; x=1723142100; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Ilp3BRqf5qrS1kR3cVkw+ddXUmTszI9WBxmraRZDLeQ=; b=tvaSzOsEsshFYilc3fDBpqCUBHqDrRvBz6odHOx50TtaGq/Za95Wbv5wL4QIyiQtOX iG7AKKyG3M1VhD/2Sa/xeqa2y+V3h6B/mnxuWOuyaRBcNO3Y1k3bE8BAYEg4KoQtnxL8 PMzlp/v0XkprRy3667xUllopOZlvAm4TFjEHxFINhUqu1kVA5ZGeaRv6pGF/SbcBEwXR aeld0XGNIYow9oTDQHsh2RO7G4Cw9TfGgjPIDhotcPQRb9h9UIJ4qRpqBNK2gbZJeqCI AVuMZ27rkbD+oNNf1BYWJUbm38weJaCF+P3S+S0ewYlY5t+VZlgoyGva6L1Fwi1DXVS/ IJ8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537300; x=1723142100; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ilp3BRqf5qrS1kR3cVkw+ddXUmTszI9WBxmraRZDLeQ=; b=PtE48oLndUM0fhfkkWpIIIBwVk8NzQZua2G5Q2THsXgWuRXlDIrm+YeY5lmHwWF6Xk sYYYmz87MhYF3NslS7qbRNdUzOC2L9h1P8gJDOVltL2ewPHJEgcI1awvACjbXeJ73uqa 928IrNv/v4DUCAhxgSZZQpjRJnDuWyvnGWjJ02LIWZfB86qqeQJnCT886id8ro7XnJTh 6m0+51sARVH88ohqEwXMnxJwLpjVIMJgQi77IuNu/p7jUkcOjBYi2TGz3jvX6TTXMoBT klFF560ups6X2h3VNO3MaetkBpYTyMRclhtp+kKiilfVvk2WbABsFdjfgjQItFX+MXRv LDRQ== X-Forwarded-Encrypted: i=1; AJvYcCXajGYTWtcBBwsYH6eJrlBTZ+yOAB4ojh5fz617JxlQV6QwEHSikF86zFHO6IojWY/YSWKCKvh5b2XTGc3WFooek95/5JRIKE9cZsfc X-Gm-Message-State: AOJu0Yydsb9PkCyV/ST+0rdL9okZz05LCbwgYDA6zo8SwZXDgBLOtn9y +WudJqr5H+dcuicP83Gcs8zqNiHJL6wO0Ra5DS7WxUbYCfrX5rr2GaYN8EEusLSsRGvQwltizT4 Jhw== X-Google-Smtp-Source: AGHT+IFVeWW3z3HLbr3aiIANRVn0Pzn9JWDp1pBfkw1NJGvfolLz0JIjag5F+BCKoOgA8ow9J6dxgy2bag4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:1d2:b0:1fc:6faf:671f with SMTP id d9443c01a7336-1ff57262fc9mr25895ad.6.1722537300275; Thu, 01 Aug 2024 11:35:00 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:46 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-3-seanjc@google.com> Subject: [RFC PATCH 2/9] KVM: x86/mmu: Set shadow_accessed_mask for EPT even if A/D bits disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that KVM doesn't use shadow_accessed_mask to detect if hardware A/D bits are enabled, set shadow_accessed_mask for EPT even when A/D bits are disabled in hardware. This will allow using shadow_accessed_mask for software purposes, e.g. to preserve accessed status in a non-present SPTE. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index b713a6542eeb..cae45825617c 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -440,7 +440,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_e= xec_only) kvm_ad_enabled =3D has_ad_bits; =20 shadow_user_mask =3D VMX_EPT_READABLE_MASK; - shadow_accessed_mask =3D has_ad_bits ? VMX_EPT_ACCESS_BIT : 0ull; + shadow_accessed_mask =3D VMX_EPT_ACCESS_BIT; shadow_dirty_mask =3D has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; shadow_nx_mask =3D 0ull; shadow_x_mask =3D VMX_EPT_EXECUTABLE_MASK; --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:11 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20BF314A4F1 for ; Thu, 1 Aug 2024 18:35:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537304; cv=none; b=ebF3lTCsFbJt8xOJAYZEwlbyVNfSlEIrQfh/uba1upHtjaoKlgJSZWKIIsSiRxuWQXCAQRDQeOFEK6BDB55+22nUBFZ0kAMhu7W6dqlo2HF4llr3zJHB1WhhEA4cXVsRGyHan54rFeD9G8uIMY+b2yiwbnrzg2mGL3FAqQyCGXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537304; c=relaxed/simple; bh=Vh7Yjg37p23UJZ6ej/K5Cd4ShyYhS3DKAU1YW8l2Bu8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=StNwAbVcUTqTmsp+/fApkwyTtxHPoDaCMrGwXOQRBuk99l3y+LlkNgQr+j5+7C7Fs5cbj86G2M6tCFkK+ocLQFZY/6e2GLb/wf1I3eBZsJwdF/skJ5ZIjpLllBGEfxJQfcB8GPJOyRR/We686xxT7O+SX+laKFhRHK/hBKYiPjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Z2gB1DkB; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z2gB1DkB" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-669f8778d6bso161990767b3.0 for ; Thu, 01 Aug 2024 11:35:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537302; x=1723142102; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=1YHjewNqSuQw+aKISRkd4hlaIK++6vC5e9hG8f7rkrc=; b=Z2gB1DkBTKg5tFcjlUi+t1manENVuYnoPqTdf6wIhk/QXJLXfQMlQdcPBiCPfxWTRw JR/f1k7qLLxyAmValomylwHEkd7dVs/kYm6g87bObW4+LsM92kL3aYEDTpvfuEMmabq8 c8Jb4zVyjtL1FDTTvQ8uG2198/5pfyfWykaW0MPsmvApdt/AJqUW74j2bSm1fV9UyTzV ztUt52sm5f8womKcA1fOKwXufZm3m7hCcT4Wze8VQf0/KaaWDdWwMUvIR9N5ovW7P7lu zG8NmSmqf33ZnsbkfdQve+aMnTFofH8OuYP9hdde0EKfzeQE+9j31WzI94py3SDlyFuM A8Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537302; x=1723142102; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1YHjewNqSuQw+aKISRkd4hlaIK++6vC5e9hG8f7rkrc=; b=m9xwuMbt8k/9k9BUeg24DneUO6x06sQ2GJyxYqHfBmIj1l3F157WE9VAs1btQ1KnTG 913xezyz/RwVfhOK+RGcFACLQcGXFHpjN5+1lLCGmoeKSl26nEL5tuTSTMHU93skk+SQ EMBGfJhOQ4gtEA3+5viE+HaEGCFLbGTu/IBV00M33WBuseQ2YkxV/HXFCm1+NhEPibn8 3Lx9Rt67dVFiFyYz4nhbtdXWJMFnQzOc2tV4nYO3GFSOlQ2AdbIhO5UpCTDH6Jbnss2z mOq6WoKmHHHfoAzV+S4OWtUye3iGe8nSpk960/BuyCBREJpywxs43/8U4DBX31wFkqFx fQeA== X-Forwarded-Encrypted: i=1; AJvYcCWM907jAt2qFBaV7OXyUosSpwyvz5l2+aOnHdTdXm23oqK+7DtxBhCQYjXl3hpQZxuN/9gCk8M/CuBXCuN8+zeMwZSUGToYHEgd32Ig X-Gm-Message-State: AOJu0YwMRu62ppuEXg3oP4Gl4xJsv6t5jMrA8TT35kaiKIfV2X6PiDDp oH3/weeMjWUzIFMFYsOe3gOf3s3yPmxY4YwAvfJQArm3OfWinsIQK18lsw7F9pLhiulzAunFidV NYw== X-Google-Smtp-Source: AGHT+IEAyNBNNAjkoq0+TR2DU9aO3yJBNbaHwXM7Z2FBl7YPBn99Zn4C9dL3lWrRoy2Zle815Z1OJBKpL3o= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:660c:b0:61b:e73d:bea2 with SMTP id 00721157ae682-68963706dc0mr17197b3.5.1722537302250; Thu, 01 Aug 2024 11:35:02 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:47 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-4-seanjc@google.com> Subject: [RFC PATCH 3/9] KVM: x86/mmu: Set shadow_dirty_mask for EPT even if A/D bits disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Set shadow_dirty_mask to the architectural EPT Dirty bit value even if A/D bits are disabled at the module level, i.e. even if KVM will never enable A/D bits in hardware. Doing so provides consistent behavior for Accessed and Dirty bits, i.e. doesn't leave KVM in a state where it sets shadow_accessed_mask but not shadow_dirty_mask. Functionally, this should be one big nop, as consumption of shadow_dirty_mask is always guarded by a check that hardware A/D bits are enabled. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index cae45825617c..a0ff504f1e7e 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -441,7 +441,7 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits, bool has_e= xec_only) =20 shadow_user_mask =3D VMX_EPT_READABLE_MASK; shadow_accessed_mask =3D VMX_EPT_ACCESS_BIT; - shadow_dirty_mask =3D has_ad_bits ? VMX_EPT_DIRTY_BIT : 0ull; + shadow_dirty_mask =3D VMX_EPT_DIRTY_BIT; shadow_nx_mask =3D 0ull; shadow_x_mask =3D VMX_EPT_EXECUTABLE_MASK; /* VMX_EPT_SUPPRESS_VE_BIT is needed for W or X violation. */ --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:11 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07EDB14AD3D for ; Thu, 1 Aug 2024 18:35:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537306; cv=none; b=scPHFc6+Gi7sYkIaJQPhTfwjFWIstkV1CVBcNQ+4sRuoIG1LjIwPZl+RlP9XXuv1oSjoUFC3vFl44gDXFs0PDWxi8fgs40R+espbzwBtLnKesnx8hcfe+hdYD83FsNSbXqOdm0cpUnB8T4KNoa+nMlaSVhHvIXfgou4FzMnrWn8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537306; c=relaxed/simple; bh=AFPTyvXyadKJHf2QzotKTohG8D5ttOVVpldlFI2PF/8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SggkFADCQHio5qD9QkTmXt+NLkedP+DFIdw7ff61reZFAbsixYRADt1XeyiBgIuSlYlsrXOopRKfMmM8fDp1TKNcq3P4osIA9y+1zXHWjd+JtPKr8gHp/8W9zH9ha6HkW/xc9ACo6pO+UTICLwZloEGCo/JeCIpmFpzNg+srdZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=C7tdv47U; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="C7tdv47U" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e0babce6718so5540723276.3 for ; Thu, 01 Aug 2024 11:35:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537304; x=1723142104; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=3iPJtAYuasIIRGf+GbxPHB6qYRb6caAhSB2/PDr0qIc=; b=C7tdv47URhonZ+WxIGRenpMDCXavAWGZS2cD2LJ1vVBzYKxwrDTJeULd6p4pt26iGD jEnLYrUgL+4qKG7R+ssYR6RwTVbXN9WAHW8n5IFqLrQpeLotqioICzTPrHOzBdXCkMgF 0s45KB1ZjtEezrkU98nMIJ17jXGjGODUQXPX2cYQiN/xI3yCgAOQTIqyUZ/yyJssJOp4 2pUvNcMqjQittJ44Z26466+qHpvtLxrkWTMH27jKiA4E7xswA7XXj0CBNonKLbKjqDZx Id4obGNAHSM73X7GbPTbkZiFulvePz9lfdPc1Qy7VZ2iVhXslYGZOkz82b6wbuuTepwr 9/Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537304; x=1723142104; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3iPJtAYuasIIRGf+GbxPHB6qYRb6caAhSB2/PDr0qIc=; b=uyTkixoWrBANnp7MCJXfrZWDz96+fplnOzISGnPOk65aO7Yyc4gnEsBzVyzMdy6quV O695bdMpc3rxIqIDkbQU866YQwSbFwY2Zd546eGvB8lJenUDWmV6gg639L9PJnyIovoe D5N/fzyeoiMuvrYT+0QRdl8ePJonikRkkJm0+EveOmsXjnpqarn2rpoghliyQdupkce0 xS4svCxsbwIMKa4FpZDggn4QZ+bsT82xwFaKueR7pIaBYVILejicqRxBJPGqBLqDiWuC IG/dMmvuUXI/5vUTOIkag2prer/kpQKFwm72lizjyd54c01FjZqdFNUaP00+acvjU+nx /Fxg== X-Forwarded-Encrypted: i=1; AJvYcCXxwxfBiZ7tXZYQluoC1qjkGBxC2g6f0Fnddnq+P9nThlqe3NEJUUrmtgOsHC1s2r/+GNWjO3N7p6tIaLE=@vger.kernel.org X-Gm-Message-State: AOJu0YxPRup26z7InZrZOBSznA/Lv1aOrTARQzszy9f7g/LqfnahViVP OWvDGNQinRAX/X2pFlgl5QDFBn1rBN4j0hL1if9gK/fxFwVe2nFOfNX21T9eHOdkht5ffaH8VG6 c1A== X-Google-Smtp-Source: AGHT+IEPP91NM9n4/Te5UnC7Xo0l/Vq7qcp/6WQ3dD16sls4NkylR/VCD/51Fsi8RHqYZr1QyafatST0IEY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:2605:b0:e0b:bd79:307b with SMTP id 3f1490d57ef6-e0bde439c87mr43504276.9.1722537303997; Thu, 01 Aug 2024 11:35:03 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:48 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-5-seanjc@google.com> Subject: [RFC PATCH 4/9] KVM: x86/mmu: Use Accessed bit even when _hardware_ A/D bits are disabled From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the Accessed bit in SPTEs even when A/D bits are disabled in hardware, i.e. propagate accessed information to SPTE.Accessed even when KVM is doing manual tracking by making SPTEs not-present. In addition to eliminating a small amount of code in is_accessed_spte(), this also paves the way for preserving Accessed information when a SPTE is zapped in response to a mmu_notifier PROTECTION event, e.g. if a SPTE is zapped because NUMA balancing kicks in. Note, EPT is the only flavor of paging in which A/D bits are conditionally enabled, and the Accessed (and Dirty) bit is software-available when A/D bits are disabled. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 6 ++++-- arch/x86/kvm/mmu/spte.c | 6 +++--- arch/x86/kvm/mmu/spte.h | 11 +---------- 3 files changed, 8 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1e24bc4a06db..c8fc59fcc8e0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3454,8 +3454,10 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, st= ruct kvm_page_fault *fault) * uses A/D bits for non-nested MMUs. Thus, if A/D bits are * enabled, the SPTE can't be an access-tracked SPTE. */ - if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) - new_spte =3D restore_acc_track_spte(new_spte); + if (unlikely(!kvm_ad_enabled) && is_access_track_spte(spte)) { + new_spte =3D restore_acc_track_spte(new_spte) | + shadow_accessed_mask; + } =20 /* * To keep things simple, only SPTEs that are MMU-writable can diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index a0ff504f1e7e..ca1a8116de34 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -181,7 +181,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_pa= ge *sp, =20 spte |=3D shadow_present_mask; if (!prefetch) - spte |=3D spte_shadow_accessed_mask(spte); + spte |=3D shadow_accessed_mask; =20 /* * For simplicity, enforce the NX huge page mitigation even if not @@ -258,7 +258,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_pa= ge *sp, } =20 if (pte_access & ACC_WRITE_MASK) - spte |=3D spte_shadow_dirty_mask(spte); + spte |=3D shadow_accessed_mask; =20 out: if (prefetch) @@ -367,7 +367,7 @@ u64 mark_spte_for_access_track(u64 spte) =20 spte |=3D (spte & SHADOW_ACC_TRACK_SAVED_BITS_MASK) << SHADOW_ACC_TRACK_SAVED_BITS_SHIFT; - spte &=3D ~shadow_acc_track_mask; + spte &=3D ~(shadow_acc_track_mask | shadow_accessed_mask); =20 return spte; } diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index d722b37b7434..ba7ff1dfbeb2 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -316,12 +316,6 @@ static inline bool spte_ad_need_write_protect(u64 spte) return (spte & SPTE_TDP_AD_MASK) !=3D SPTE_TDP_AD_ENABLED; } =20 -static inline u64 spte_shadow_accessed_mask(u64 spte) -{ - KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); - return spte_ad_enabled(spte) ? shadow_accessed_mask : 0; -} - static inline u64 spte_shadow_dirty_mask(u64 spte) { KVM_MMU_WARN_ON(!is_shadow_present_pte(spte)); @@ -355,10 +349,7 @@ static inline kvm_pfn_t spte_to_pfn(u64 pte) =20 static inline bool is_accessed_spte(u64 spte) { - u64 accessed_mask =3D spte_shadow_accessed_mask(spte); - - return accessed_mask ? spte & accessed_mask - : !is_access_track_spte(spte); + return spte & shadow_accessed_mask; } =20 static inline bool is_dirty_spte(u64 spte) --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:11 2024 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 761C514B950 for ; Thu, 1 Aug 2024 18:35:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537307; cv=none; b=h9YFcZsSKSzol0suFTWaw6rN83moOBm3riW6TmKFs0OMuFjn/j8svHAXa9bF+9cPTkAUhtuZtWMyW+iwARa/7nC4eIV0h5+IaS+JQNfE6l20/827N7oSEbdheaTuWmeTw8ep5sPuIYCcHar7NW8DZfBSVlLfSVEpNYaBeKFAxDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537307; c=relaxed/simple; bh=+209Dl5mjZfEzPCjJbS6cYtHb+T8VqI2r8B2gBeStTo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mrWSaFA/OKBeObYVYXbTmR8QFj6twgSfqT0cVgzVPyIc4Zud2X7FUyozZqrhlJUbpGuaDIUj3vBhk3O34VLvHkNYGEPaFE/AfP6ALfqFGeDkV8oreKOjKrvn/QFnuLEKkaqOxiF+bSxw94Gtm68bsyERcPDbWwcLOaS73ODf2sg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yg+DOyGW; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yg+DOyGW" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7a30753fe30so5527924a12.3 for ; Thu, 01 Aug 2024 11:35:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537306; x=1723142106; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=nLllgIeEL1wMurU3ykZwAR8levnh4hSK0+7xKRqL5Is=; b=yg+DOyGW4l4CuPriZwCOwA1CXdeCsL5OHnUHQq4LbhxF81Vjc7HCcYYkvOKz+kvbt2 ABmdS3dqKolKdl9fGuiZ5n/vTP18iMb3WEWlr6jbjMMEMihkBvMPkV1IZNyV0Yk/3qVm DWCXtK1YziHMIcnvFP8g0EgdfDL7jz1GbnNll3EAmTMcACLhmnvSKXZXZmKUHsJqamLZ SfFCqFkhiiyFAFAzLAUYQpwBZnC7l3+k0w3LyYltZTlRehAPBMffsQykie7Nr7azDIG/ WgyN2cMvPQS9V/cV5jhzxCrjtRZ5VvxbD252RIKZDWlqjhMP5pBnPWD8CiipV6q61JMu K5tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537306; x=1723142106; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nLllgIeEL1wMurU3ykZwAR8levnh4hSK0+7xKRqL5Is=; b=eCvDR9NalrC90O9oC6x01ynvPChtKMwaRCbGYC398LhMSJXHmVOcWfDKAFZ0nXlcmm EpIWTUK4CNihKNHo00YS5FeXNUkTyWd8ThTQJY5wAZoybqdoqXm+nnDkzS2KHKLCn5Ob PfyBwztB0c/5d6MZXWXRCBzygky1pR7/ZrUHwrXskl1wfxZLXEe9Bjv/xKvlhmxfO8wi SLjqyEGK/lN1WXSdbk0Uwsz+CTxR0C1OkAG/HKWE0qkVEtQeqUMYKhnswL/yOlMODQZX sSRmmUbdPKGYoqA4xFSz9MQMNLq3wgVUoJ8u+ayFNWP/seoHq8VAI4apCyFfY1UQ/9+3 Tglw== X-Forwarded-Encrypted: i=1; AJvYcCWQXawmHbhlr4uFOfpsMjiH7cGEhx76FIQ4CG7Jfmxf3IBX/+dSCQwDwOXo2ZiFVOcGVnp1/95SlnriFc11UAl+FZNfOLUv9PGdzUrf X-Gm-Message-State: AOJu0YweV7vM+IXSfHYdTEqYQ0wTDdvyencFc+UFvqKSx6mnU6t0THP5 0HsQSM9OrQDFCQ0QAxAo6fZre4cEprkboTp45rXZXGHqO2iATx5lTU5hCw4tmrCq0t6bQtzhWZa WPw== X-Google-Smtp-Source: AGHT+IGJTez0OFwM+Kmo7Ln7XZmkdAZ4Q1iTleNtDtm577nIAAAdrxOHonp4x5mucOt9Mkl1xYEJT8P1AFM= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:3f41:0:b0:7a1:2fb5:3ff7 with SMTP id 41be03b00d2f7-7b7438b1dc8mr1940a12.0.1722537305573; Thu, 01 Aug 2024 11:35:05 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:49 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-6-seanjc@google.com> Subject: [RFC PATCH 5/9] KVM: x86/mmu: Free up A/D bits in FROZEN_SPTE From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove all flavors of A/D bits from FROZEN_SPTE so that KVM can keep A/D bits set in SPTEs that are frozen, without getting false positives. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/spte.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index ba7ff1dfbeb2..d403ecdfcb8e 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -216,15 +216,17 @@ extern u64 __read_mostly shadow_nonpresent_or_rsvd_ma= sk; * should not modify the SPTE. * * Use a semi-arbitrary value that doesn't set RWX bits, i.e. is not-prese= nt on - * both AMD and Intel CPUs, and doesn't set PFN bits, i.e. doesn't create = a L1TF - * vulnerability. + * both AMD and Intel CPUs, doesn't set any A/D bits, and doesn't set PFN = bits, + * i.e. doesn't create a L1TF vulnerability. * * Only used by the TDP MMU. */ -#define FROZEN_SPTE (SHADOW_NONPRESENT_VALUE | 0x5a0ULL) +#define FROZEN_SPTE (SHADOW_NONPRESENT_VALUE | 0x498ULL) =20 /* Removed SPTEs must not be misconstrued as shadow present PTEs. */ static_assert(!(FROZEN_SPTE & SPTE_MMU_PRESENT_MASK)); +static_assert(!(FROZEN_SPTE & (PT_ACCESSED_MASK | VMX_EPT_ACCESS_BIT))); +static_assert(!(FROZEN_SPTE & (PT_DIRTY_MASK | VMX_EPT_DIRTY_BIT))); =20 static inline bool is_frozen_spte(u64 spte) { --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:11 2024 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4928414C596 for ; Thu, 1 Aug 2024 18:35:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537309; cv=none; b=rZEh2HXSi87nFsnR05Ncz3w4ahQWAYcxhLz4WS4/x6yYnnN4lTjaeaDor7GAV/STFJCZ6MOhbFRaiLIcTNdDYoE0uW7NPgI0ME7EWENAv8D6MpvBmBP3TY7u6pjotSCq8iM0IpN25fWtyq9qZVbxCScz7D1uc3pq6dKIhrifG5A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537309; c=relaxed/simple; bh=/XSoRdfaJzNo/ZGf04C230tSmY5GFwai1Y6z9V+ll3s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TbOroBtNPsgZ8kSwXADjqkLE6M6mIUW3icrru3skLkkiXqNX+ukdEkdWy6hIdJPSlZ/h4x9hQFXgLhCmCX1nwCmPwfHgghUQRel4mySgfiIUEOI4p6Fwwt9PrsZUbVVMBLF9cRd7Z7SkkmAChcutOudIJgO3TJhlZ3upQP39ggk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=T6f4Ughv; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="T6f4Ughv" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-7104f37771eso2359045b3a.2 for ; Thu, 01 Aug 2024 11:35:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537308; x=1723142108; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=YgfeHTUFXLVGjVFJpfNzwAaN7GsDEU7AXhuO1w5LlJU=; b=T6f4UghvEEptu57168f+L/zM2OSsfTUEmq1aYaownOtHbbaFnbOQ8PWU0JuKJgce0H 9fQZZawMVXL1956CtTaup8jRe66DfJ2jxln5sAhr19titUwv3w2uBWqXPqqVv6dd5WTu cGEgRY1qU5xXsf66qMwFiRjqe1LhSabKEpXXdGiP8fkoZFZ/AXnxagDFTf506ChfutbB BerF3q22V5B07c89wofMVqoMFo/AZJvDayCRHBYir2P56J1tZ7SoqPkDBHGQY5kJMLAd LCfJVXCXpldMdtKUwjNRwX2jOX+xLfvMyHOearGAJAIiUZPadY5Q6Hq2DGw2t2lH8qkY Il3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537308; x=1723142108; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YgfeHTUFXLVGjVFJpfNzwAaN7GsDEU7AXhuO1w5LlJU=; b=klE+UCHLhe0f02uEL2TL3bRfIWOntPDfw8bK+zG70Ar+aWzA99eCSJUCUARnGPbY+q oEDtLBr6Bw1L/YajSoQUCt6nEr5cXatJy5mpocaOpfmW9o5YeZNGiz7ketxErVHVUBJG /lBmonlb9BjIDEhVV9DAf7HqmT5pE0frubChvm7CKUd+1q5RyRLyvCahkwlptpGjq+jX 7vwjScDbEWIsnip7z+kzy9EdojUOuPPT7cVMrkeBydHcTZiAi9ibkc3GT/12V4wqBaKM u+SJ+gZgBOZBnfZGx33WG2+Wn27h+RTun4PycZHosTKNPihEl73PKRJqbYSq/P6tRlLc DtIA== X-Forwarded-Encrypted: i=1; AJvYcCXoyeFjOSXPYRnUmveFntHCvI0Hr1TlT74vAPREikkMvpMGatKcYJgmREfnn8yz5kqbcA10pu+Y8oSMHjnZVajYJYPvpOiSGg91dn/Q X-Gm-Message-State: AOJu0YyR6PX5V+xtX2f/Vs1QISStcxEDHZY971OlGFGW9hdf62jjl7xx n9cccR33C07ydWqFKSVaMYjkcLz0T/i1YqbIqGLYMT5IiPRg2IRf04qkbxs8T1bdg/l4/vdMGcC qRA== X-Google-Smtp-Source: AGHT+IHTWEhjwHtp09vWEbUgtVyznzcr5+x5rCkfOWJrq49bIvfwRkVTlE43qqkCOx+dDWfaukt6/MOxYM0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:949c:b0:705:ca19:2d08 with SMTP id d2e1a72fcca58-7106d0ca7camr8792b3a.6.1722537307533; Thu, 01 Aug 2024 11:35:07 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:50 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-7-seanjc@google.com> Subject: [RFC PATCH 6/9] KVM: x86/mmu: Process only valid TDP MMU roots when aging a gfn range From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Skip invalid TDP MMU roots when aging a gfn range. There is zero reason to process invalid roots, as they by definition hold stale information. E.g. if a root is invalid because its from a previous memslot generation, in the unlikely event the root has a SPTE for the gfn, then odds are good that the gfn=3D>hva mapping is different, i.e. doesn't map to the hva that is being aged by the primary MMU. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 2b0fc601d2ce..b358642890e1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1202,9 +1202,11 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(s= truct kvm *kvm, =20 /* * Don't support rescheduling, none of the MMU notifiers that funnel - * into this helper allow blocking; it'd be dead, wasteful code. + * into this helper allow blocking; it'd be dead, wasteful code. Note, + * this helper must NOT be used to unmap GFNs, as it processes only + * valid roots! */ - for_each_tdp_mmu_root(kvm, root, range->slot->as_id) { + for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { rcu_read_lock(); =20 tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:11 2024 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBC4B14D2B7 for ; Thu, 1 Aug 2024 18:35:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537311; cv=none; b=MEuanbDQguW2psr7YP/4Ov+NJIXFzwUrOInrqUWZgEE/J9zqoQL7dvR/wnhjY+6yVT3svku8/DXyx0hVlhclJXMVD2G22ZzrRtXJJFE5LnmAmF5P9mEv2izWNIjzSGDleTkzB9VPCEMjTsl2j1NX4Gb9V8d5jze4vylOQmXLuME= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537311; c=relaxed/simple; bh=kcyZHtFtSWTsZyQHZ3oFwxa1IXjn9+p3AjCTE5ktFqM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BYWWxWDYuJpUOVwx5Ha4/xZ3pSmblaypFNqRwwASwQJWc+sb27TA4R8VCrjv4kQxEuTZDMn3DzHCaP+iJAIspHane9JCkGn+z0gfxOCkTxm6QgEyQJvMsBCcUqLl/Axk8gBmP1f+CiWp5CIWGRee7TMCFBIkrvkKhWaf70DGJgg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=R/AJ2u0i; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="R/AJ2u0i" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-70d1df50db2so2447996b3a.0 for ; Thu, 01 Aug 2024 11:35:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537309; x=1723142109; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=MzTNDvjeVqrs7z6ZdAikXXC/sttVLmYnLoCNz/Cy7ro=; b=R/AJ2u0in/q7N5U13dxxbPnEoXpNcBME4V/loLgl8eYiNLb455O8l9sqEB4GPJa+xF 5LOt4K98IAOjlX4JOv8O7uyVVB4mAliIgWKcMFqdjqS5m6rePZd8ZuI4dbTpB/Oyu2iM MXjtLD+C6DZBoCXTMByNLtWV3l+lFUD8tDIQlh0wsU7hNIoWG6yM+590HHzII+QDfkar VR12eVFGBcdFCvtzbiKME+HqEHawIOcyXzTcoFCS+gs0C5GinTiEYUFfjttPUyuduJzI U0vJIVJJXKcoRZbOzayOgcyXS2eVHowuJD0YpfX5jNjvq1LJke0KEFcCn6u5cVlQhr1z sFnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537309; x=1723142109; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MzTNDvjeVqrs7z6ZdAikXXC/sttVLmYnLoCNz/Cy7ro=; b=vkGAYAlY1XiEGAtpXKb4eLTxCgOJ08WhLAzDKXr/kE+hgz9CKLHVgjIEFTlypSuH2r dfWbnE8wslh5lZrwwjJYEh5pSag33KwuDhfKH1hjOicQwZE4VfCrFQVIMup0zozYcRfO sK/vs5h5K46N2Ov+pNuvWdDFf+/kROQV0j4KaJVPg+2ZRLJFf4sW4a3iBsmjs4IHjaTU 6iMXAGAbVJjUo2+Engeo3Oaax1SKl0rpzARUUx3X1ib9SImbkkKG9rQhmL4cDZGueS75 eys3/STPlFkjUzwWrPe9wKm9OUP3RN1wH8n2/eNKBAGEWi0K2U4+Nka9JSP+FhcpvdfU 4DCQ== X-Forwarded-Encrypted: i=1; AJvYcCVA6+LaCyj2yU3D/mXZYdIihR0colCCWQk9pgoddpm/9SBhnms8ob0cNlBZmr/GBxEdxyFH93bSLX5JVwUrL7B/WHFknVV5cImWaCqU X-Gm-Message-State: AOJu0YzKSoeolbEX9qcpxFc8YObYSCpFBh5dfHf6+iwz51Ap2Et4+FiS xQ8hgJPF5l5HIF6eODvkvHWl32/znC6Z0OREa+BCxftRxxdGvJFbSS8SVER6lAtke5G+7aW1zg6 nhw== X-Google-Smtp-Source: AGHT+IHBKWw5BAUfyMToUbh6nUKAFn26IAUUz572bHA0Rz/4Cn41vP3IxvpndgvHdAhxE1s8QOG25zRXuL0= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:6f12:b0:710:4e4c:a4ad with SMTP id d2e1a72fcca58-71065b6f385mr40762b3a.0.1722537309268; Thu, 01 Aug 2024 11:35:09 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:51 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-8-seanjc@google.com> Subject: [RFC PATCH 7/9] KVM: x86/mmu: Stop processing TDP MMU roots for test_age if young SPTE found From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Return immediately if a young SPTE is found when testing, but not updating, SPTEs. The return value is a boolean, i.e. whether there is one young SPTE or fifty is irrelevant (ignoring the fact that it's impossible for there to be fifty SPTEs, as KVM has a hard limit on the number of valid TDP MMU roots). Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 84 ++++++++++++++++++-------------------- 1 file changed, 40 insertions(+), 44 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index b358642890e1..ac3200ce00f9 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1189,35 +1189,6 @@ bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, st= ruct kvm_gfn_range *range, return flush; } =20 -typedef bool (*tdp_handler_t)(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range); - -static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm, - struct kvm_gfn_range *range, - tdp_handler_t handler) -{ - struct kvm_mmu_page *root; - struct tdp_iter iter; - bool ret =3D false; - - /* - * Don't support rescheduling, none of the MMU notifiers that funnel - * into this helper allow blocking; it'd be dead, wasteful code. Note, - * this helper must NOT be used to unmap GFNs, as it processes only - * valid roots! - */ - for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { - rcu_read_lock(); - - tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) - ret |=3D handler(kvm, &iter, range); - - rcu_read_unlock(); - } - - return ret; -} - /* * Mark the SPTEs range of GFNs [start, end) unaccessed and return non-zero * if any of the GFNs in the range have been accessed. @@ -1226,15 +1197,10 @@ static __always_inline bool kvm_tdp_mmu_handle_gfn(= struct kvm *kvm, * from the clear_young() or clear_flush_young() notifier, which uses the * return value to determine if the page has been accessed. */ -static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range) +static void kvm_tdp_mmu_age_spte(struct tdp_iter *iter) { u64 new_spte; =20 - /* If we have a non-accessed entry we don't need to change the pte. */ - if (!is_accessed_spte(iter->old_spte)) - return false; - if (spte_ad_enabled(iter->old_spte)) { iter->old_spte =3D tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1250,23 +1216,53 @@ static bool age_gfn_range(struct kvm *kvm, struct t= dp_iter *iter, =20 trace_kvm_tdp_mmu_spte_changed(iter->as_id, iter->gfn, iter->level, iter->old_spte, new_spte); - return true; +} + +static bool __kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, + bool test_only) +{ + struct kvm_mmu_page *root; + struct tdp_iter iter; + bool ret =3D false; + + /* + * Don't support rescheduling, none of the MMU notifiers that funnel + * into this helper allow blocking; it'd be dead, wasteful code. Note, + * this helper must NOT be used to unmap GFNs, as it processes only + * valid roots! + */ + for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { + rcu_read_lock(); + + tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) { + if (!is_accessed_spte(iter.old_spte)) + continue; + + ret =3D true; + if (test_only) + break; + + kvm_tdp_mmu_age_spte(&iter); + } + + rcu_read_unlock(); + + if (ret && test_only) + break; + } + + return ret; } =20 bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *rang= e) { - return kvm_tdp_mmu_handle_gfn(kvm, range, age_gfn_range); -} - -static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_gfn_range *range) -{ - return is_accessed_spte(iter->old_spte); + return __kvm_tdp_mmu_age_gfn_range(kvm, range, false); } =20 bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn); + return __kvm_tdp_mmu_age_gfn_range(kvm, range, true); } =20 /* --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:11 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E58AF148FE6 for ; Thu, 1 Aug 2024 18:35:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537313; cv=none; b=ZkF+OD2iSN2AAi3CyLyLSy8vSL/GP2uyCOowZxcy5btycd7f9ILgEDNgx7Soe3mDWJEBsd15y6FL7r4ThlYoQ+8lOvdjD6bebuWBFSK1SZFTCAs5L+B6XT/mbmPPFfq61IfkXlXzHgl2TDZaGW0IrTUnw6OdstyDHHbL7/f7zDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537313; c=relaxed/simple; bh=RbIH3IP3u28tDH3NR3/offoYXurj3HvTAqkxRIuQJEQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nUA2jTqUAd4tkNrWzpyP0u/vwvrYZglmqa+vXbvls1UUHXOTcjX9WxHORvpePZfCjQ1judPpwGuZmejQJuF4STQr3MaQVjhT5RYtWZXqc/QQPOBcqqvhq4jPjskmK2zGbOhyiIqOAG+BTsek2iZfkLtQTZuIBQ8c/HEe+WOw24c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=38WhoNgc; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="38WhoNgc" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e03623b24ddso10619637276.1 for ; Thu, 01 Aug 2024 11:35:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537311; x=1723142111; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=reMwPkVdpNL6CI3+i6mf1k9M5Xmr8NrhYIDXv6qn8Wo=; b=38WhoNgcqwS1ix5Duj6UAQjG74zQZN6kXOnA8QAjZxDHPynKrn8mwHwbqhFw0Vgu7s uKCv/OEnLAcKY7on2oUUyUNZMXRVyjCnDnXrxGVKxq5fuKUPLvWc4f/NfvmzcvNVj8O5 UDuuxE33Hzwgj9k9x2D3ma5rdrwRqnakk9OjzA8TchN4u6KC7qVXVA6N0EUxN03LFP/K MK65/QCl5tdOCxJsE7M4AshoS/qgDBOlfqRbV3lyO7YoYn9nAL2aL/cvtTUf7sMvdv3x OA75tTlfPTmC6uLLIFlBPpjWDOZtaK3ShrZIQhGhxDu9mPOrfVmFxe9LSPk+pqF/t+sA OXiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537311; x=1723142111; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=reMwPkVdpNL6CI3+i6mf1k9M5Xmr8NrhYIDXv6qn8Wo=; b=wGM6coc1wHoqztdjhpJjRraCVgKNNuC+kv+j7/atiUSsa5NxvFCTVNyYutaCD7DEjn AJdB4USVq97FzjGo2Is7A4I7ndMLXnZLqtyGrhOa8AL8g4231XyCHKZoqT+VsXrHW1T2 PvxDe13zqCsONeQZnq0HSQ/VipG2XASU2zBy0Uzu8imqE2yGVDhg/PBMJkqZuROBxo5u NSrwbTDBa1bHg5LX3xS/DgCIpfIe4xDzthmXjXKWA6RLwIg09f8TQm6JoB+PlS4scfIt 68YUq2oxWzj6yJoXb6IrSGgjkj3x2d1dq0dnv7//JfFOkEIcE5SM0CHIkjfZHVsheeaE Kxzg== X-Forwarded-Encrypted: i=1; AJvYcCX0qcK/vpQdexJKefViYALhyDT0qWtM7RGjYD4muX1fpuqR8CEikjb7rubgB/67V3UrY9cTtTqgsiHx7M8raeuHy/t9FESOXTq6H23M X-Gm-Message-State: AOJu0YwsOYAs7crBjQnUlgotRH+ODwi+kwQNQ5ICL5wJ0CElkNN41CWq SvR+KUFnvqyZxyVClGHhWyTGbrWN+2G/cs5HslpuA/OV0iA+xAG+BKS4kZELPg5ix4nvx7PpCjx kWA== X-Google-Smtp-Source: AGHT+IF7cOBWTMyYm7b893kA5i2raHJ45P5vW+o+z7f1YVRpVI+wpdWY6KX7vJ+SzwpepoEVO382RoCb96g= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:2483:b0:e03:a0b2:f73 with SMTP id 3f1490d57ef6-e0bde2f3cdbmr14329276.6.1722537310980; Thu, 01 Aug 2024 11:35:10 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:52 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-9-seanjc@google.com> Subject: [RFC PATCH 8/9] KVM: Plumb mmu_notifier invalidation event type into arch code From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Forward the mmu_notifier invalidation event information into the arch handler so that arch code can take different actions based on the invalidation type. E.g. x86 will use the information to preserve Accessed information when zapping SPTEs because of a protection change. Signed-off-by: Sean Christopherson --- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 1 + 2 files changed, 2 insertions(+) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 006668db9911..1fce5cf73b8e 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -252,6 +252,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg { unsigned long attributes; + enum mmu_notifier_event event; }; =20 struct kvm_gfn_range { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index e279140f2425..3aa04e785d32 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -719,6 +719,7 @@ static int kvm_mmu_notifier_invalidate_range_start(stru= ct mmu_notifier *mn, .start =3D range->start, .end =3D range->end, .handler =3D kvm_mmu_unmap_gfn_range, + .arg.event =3D range->event, .on_lock =3D kvm_mmu_invalidate_begin, .flush_on_ret =3D true, .may_block =3D mmu_notifier_range_blockable(range), --=20 2.46.0.rc1.232.g9752f9e123-goog From nobody Wed Oct 30 22:11:11 2024 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5070214D71F for ; Thu, 1 Aug 2024 18:35:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537314; cv=none; b=XstIDss1ucWN6/hskfhyrPcT1rO8LuUsKi442n40aQcLb0hR++tR0wZDDw22sQKq5HqDvZZBtQpmCDVTlpom+SivVY2NzTMHZKTs84fPmE3bPBiJ+sQ0n0vTWs69XPvEuI6fukUnJfYlMmalsbqWj+ouZ41ghcl0NG15PAn3Z/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722537314; c=relaxed/simple; bh=QjtPf6OwO3F3v2qbeeMqBdAhvi3//8swMm8d6nCHE5E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ogIraGDKahWmwlLPRlcfyK8kC7C/msJVzHvg0QSMGyC5Ab+m1KUMiZn2UKVwq1l8zgV70jtBRXsP1XFahcZoVJm4J8NVE7la2ZurFNip5WQYvZrN02u2j2PxYRbQhNnO0j4uiqAWMDwn8qdi0zkOzuFGmWD/Xr7i50Alf7PBiWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=0vlCK4Du; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="0vlCK4Du" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-70d1469f5e1so6333809b3a.3 for ; Thu, 01 Aug 2024 11:35:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722537313; x=1723142113; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=pjpYJOfuFfBZxoqHyzYZOHO/aDFDcIHjHTQpaKTRfzU=; b=0vlCK4DuMb/cGY3MjcBI30JIteqLSMLxIDLShEgeDyBjmBo32FiQfYT0z2cRE01Ent LJFmalieCjcyZV5xjEFgd1LvEjVKr1/EMFw6WZLb/H/1Tdcbuk2WIXjP22m+fLM2wtGo /vnsYno5ZhfdjGu+zESkheRexa1pTHDrkWZsrZzUJ+GlEKO82GldDIDd6CLdbShu7BOZ 6CFRZcOEzQAZjQodiJUI0iNuHyY69vs70AWWmP1l1IM7O7lBur/1OPyFQ93SKdDC+UDu RFCvQEtCoRHBbGSGinbDRLVPgWtHytidmdSAmNXGuvfVbqzIzFujWkErrN5d2GpMkK7r CE4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722537313; x=1723142113; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pjpYJOfuFfBZxoqHyzYZOHO/aDFDcIHjHTQpaKTRfzU=; b=LsF6f4LaVqRDS2ZoJlS/KBrC1sbwjTgnIKs9jQkgnWFjoIbG9/Kz1CKjV3x4VFNy8c TJIDT6WhPKdTe/Eygl3k6yKMBVRD6peKkIP5XYyIBfh9umuIxTYhmxKneRHsN3s7I2wb WD0sXtYHofo+Pr8Z00XxTftX3panHLnMPSQp68nreacg5lOKXOyJEu9ZyxUtFQsw8yAg r2vWrYXL2PQOWI+7l4sdTFGczIyBul3z678nL6jADbYzn3AC/zqjWj0HHNfktUbNkqOw rKPT4Tgqq3BdT83bJdorTtv8C8p9YUoxkeCkCvRZ/nIgnGGjZULksvsknnWOhSLcNs7G h+CQ== X-Forwarded-Encrypted: i=1; AJvYcCXFgWuitMnEaToDQDc/VXHaQMusQdB8pkidjP+g/QLLHdGbA9lJ/dlZCEhyWW+mb88iVQSIyFgjJPNlZ5H7ZKWA1MmkfEM3p3GrkbTw X-Gm-Message-State: AOJu0Yy+XBTNRY9Apskh7UU0k3iClC0M+dlXMdDBTNEaL+VfhqjnDtN2 YNvKsc7g65mgdmQCkoTqpDklOEFITauz1mCyrIWIpxVNRKiVcOv1s9MWIarPBuTdCIV2tYiplfc ihw== X-Google-Smtp-Source: AGHT+IFcO+Vr6uQpb300nP1wpu6DqsAAMIX15/3LQD4+pSQyjhlaJV4bhz777we1jEIWnrv4DvkpVknRK04= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:6f13:b0:70e:9e1b:1a83 with SMTP id d2e1a72fcca58-7106d01e048mr17749b3a.2.1722537312706; Thu, 01 Aug 2024 11:35:12 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 1 Aug 2024 11:34:53 -0700 In-Reply-To: <20240801183453.57199-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240801183453.57199-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.rc2.264.g509ed76dc8-goog Message-ID: <20240801183453.57199-10-seanjc@google.com> Subject: [RFC PATCH 9/9] KVM: x86/mmu: Track SPTE accessed info across mmu_notifier PROT changes From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Preserve Accessed information when zapping SPTEs in response to an mmu_notifier protection change, e.g. if KVM is zapping SPTEs because NUMA balancing kicked in. KVM is not required to fully unmap the SPTE, and the core VMA information isn't changing, i.e. the information is still fresh and useful. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index ac3200ce00f9..780f35a22c05 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -838,7 +838,8 @@ bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu= _page *sp) * operation can cause a soft lockup. */ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct kvm_mmu_page *root, - gfn_t start, gfn_t end, bool can_yield, bool flush) + gfn_t start, gfn_t end, bool can_yield, + bool keep_accessed_bit, bool flush) { struct tdp_iter iter; =20 @@ -849,17 +850,29 @@ static bool tdp_mmu_zap_leafs(struct kvm *kvm, struct= kvm_mmu_page *root, rcu_read_lock(); =20 for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) { + u64 new_spte =3D SHADOW_NONPRESENT_VALUE; + if (can_yield && tdp_mmu_iter_cond_resched(kvm, &iter, flush, false)) { flush =3D false; continue; } =20 + /* + * Note, this will fail to clear non-present, accessed SPTEs, + * but that isn't a functional problem, it can only result in + * a _potential_ false positive in the unlikely scenario that + * the primary MMU zaps an hva, reinstalls a new hva, and ages + * the new hva, all before KVM accesses the hva. + */ if (!is_shadow_present_pte(iter.old_spte) || !is_last_spte(iter.old_spte, iter.level)) continue; =20 - tdp_mmu_iter_set_spte(kvm, &iter, SHADOW_NONPRESENT_VALUE); + if (keep_accessed_bit) + new_spte |=3D iter.old_spte & shadow_accessed_mask; + + tdp_mmu_iter_set_spte(kvm, &iter, new_spte); =20 /* * Zappings SPTEs in invalid roots doesn't require a TLB flush, @@ -889,7 +902,7 @@ bool kvm_tdp_mmu_zap_leafs(struct kvm *kvm, gfn_t start= , gfn_t end, bool flush) =20 lockdep_assert_held_write(&kvm->mmu_lock); for_each_valid_tdp_mmu_root_yield_safe(kvm, root, -1) - flush =3D tdp_mmu_zap_leafs(kvm, root, start, end, true, flush); + flush =3D tdp_mmu_zap_leafs(kvm, root, start, end, true, false, flush); =20 return flush; } @@ -1180,11 +1193,13 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct k= vm_page_fault *fault) bool kvm_tdp_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *ra= nge, bool flush) { + bool keep_a_bit =3D range->arg.event =3D=3D MMU_NOTIFY_PROTECTION_VMA || + range->arg.event =3D=3D MMU_NOTIFY_PROTECTION_PAGE; struct kvm_mmu_page *root; =20 __for_each_tdp_mmu_root_yield_safe(kvm, root, range->slot->as_id, false) flush =3D tdp_mmu_zap_leafs(kvm, root, range->start, range->end, - range->may_block, flush); + range->may_block, keep_a_bit, flush); =20 return flush; } @@ -1201,7 +1216,11 @@ static void kvm_tdp_mmu_age_spte(struct tdp_iter *it= er) { u64 new_spte; =20 - if (spte_ad_enabled(iter->old_spte)) { + if (spte_ad_enabled(iter->old_spte) || + !is_shadow_present_pte(iter->old_spte)) { + KVM_MMU_WARN_ON(!is_shadow_present_pte(iter->old_spte) && + iter->old_spte !=3D (SHADOW_NONPRESENT_VALUE | shadow_accessed_mask)); + iter->old_spte =3D tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, shadow_accessed_mask, @@ -1235,7 +1254,7 @@ static bool __kvm_tdp_mmu_age_gfn_range(struct kvm *k= vm, for_each_valid_tdp_mmu_root(kvm, root, range->slot->as_id) { rcu_read_lock(); =20 - tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) { + tdp_root_for_each_pte(iter, root, range->start, range->end) { if (!is_accessed_spte(iter.old_spte)) continue; =20 --=20 2.46.0.rc1.232.g9752f9e123-goog