From nobody Thu Apr 2 20:10:51 2026 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D672B3A8745 for ; Thu, 26 Mar 2026 18:17:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774549083; cv=none; b=I4I563Q9X3lyuV1y+Oie6thAZzvQZ6JiDrHNQ3pyceFkx9xr+dgva2N7c8YY1oevt+NQDlQ2lmXVM0rkjN/9jmjuEo/IHrP1dXmjHojckFrnECl5XpDDB6SbQkd1Q+ZNXjjkcPxPvW/5bl2F2Ez8afuLgSE0CT6MSO9Ew4iXauE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774549083; c=relaxed/simple; bh=pS1J8W8fO4+3tnDj88CZnxAyKm0FKvsBhMP+n0FPrMc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lyI6ZYiVODcagX7LQGhqO/PFrulZeLl4DptL9XA0/V32XV3DG8YmTluO/Qoekkgz5SvycBoRp6/q1ACVfayyVTsX50PDm4+izna80fgO0SoXh4a5bVQkuYTsRQCfM18YsuqUuFCMmkkiXroeaBbv2J4c1r7OBOPOSF8K19iU2T4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=YhZrWOiY; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=pv76CWUR; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YhZrWOiY"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="pv76CWUR" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1774549077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K8T8CyELIjmZrSOM7uiAWDk5u4n6lA2+sMqy1wRdzGU=; b=YhZrWOiYYno3XYTv/ymgJLJs3hwT5ACbhIag7+1kh4Gn+UJV7HY3PXuRlHVKVJLZuEEAYw bam0pt2ydJNBS4zK0NpbLROJc/rAkG2N3BwDr2h5kvKrQB477j6/7BVekAR+HkwI/3yJCO PGCpIn6Hw0VQfgfWz4VcnsW65S7G7rg= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-EwdQNFTsMR-KnIeYg47Rhw-1; Thu, 26 Mar 2026 14:17:54 -0400 X-MC-Unique: EwdQNFTsMR-KnIeYg47Rhw-1 X-Mimecast-MFC-AGG-ID: EwdQNFTsMR-KnIeYg47Rhw_1774549073 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-43b8f23bdc7so94116f8f.3 for ; Thu, 26 Mar 2026 11:17:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1774549072; x=1775153872; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K8T8CyELIjmZrSOM7uiAWDk5u4n6lA2+sMqy1wRdzGU=; b=pv76CWUR+uVQiJITEeL8bO/xrA/ZgiMrciFb7w8oQiRuZjtQqLgK3VW4oqwyZuepdS enfDCJhdhKiKdJdCsBzaenxKQs7iA3T0T2z/FfMv2E4evMxIutVUh3vOtxjACry8wZT7 MysqHU86lo2VgR0g5YWgzm/Um4oI9L7yHlEJcmipIzcgFlFspodJaQbCCiuV9Y19n1H7 NafDZxct59bQrfRyyk0SBah65N3O4//XmVRrk5wsydeWEnz22f8B9s/ls/DlmAO+fguo 357jxcgzwFwbmxEgTQPztqE9V/17YzysDZkwqurYdgzcu/gXiIbc+7Rp1Qx54fjilhy/ yFnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774549072; x=1775153872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=K8T8CyELIjmZrSOM7uiAWDk5u4n6lA2+sMqy1wRdzGU=; b=FxpDyhS4Hj6cHcTlgDddye/p/R60CSkQ9jgAaiady0lJaq7X/wbRHcRCqykmKfC3pW h2Wm1pzB5XQBIcUjI6IYglFxjXmKxttsEYpevGmm+L/nY0TiOUk3GP2cQnmL16wn+3oK 1Pz3YltLWU4t+zjsYOHQaivjcOC973azz8LDD/P6F3k5w8fOSJsIIUmRLxylmuOqaeD2 pu1Nl5oi58m75KVy+rn4kLq3qNNkKdMuxrQ2kiWGltZfhewA9RJ/zf1zv8DR3lpVbjDW OQYLzl0NXqZhb2uYD7cCGPXcfQIRkJUFe5NzNkdSsmuyXNHf1UBv2wX1XfMhPYLU6cwq NpOg== X-Gm-Message-State: AOJu0Yx5JlC2ijqESH6VTSXozDUBmkQ4w27PAypTwwbSsOfv0r0YTFZX Wx8LrnPmQx4XTVaDFaELLgz+oo40d1VgtHH6f/sqttsot8o6m8i4zgzOBjbd7qjDayfkLgx3NyZ WQ5fW8gZHSqnqYgsDFDI4+/yKofjpJUgHLFwPLaLL6H5JNzhtkK/LXK5tW+kUc0qPR/ZIl+Qrnw HAi/C2mjJocxCZYshrjGOmH0E8f5iGcN+wl8j6+Ury+8sssFwDuQ== X-Gm-Gg: ATEYQzzAlaJwvfc3beF6mKbjArJQYMO9g5tNu65u6WE+995fkMzKPws7La3QJ7rzIoN 36OUFmpZHSDaEs3xjMJcFuaSYj3sAmVXwERKEzjsfo1VUmZ1rvhvQg+zZYd5xGsEOsYVZ6lmK1w 2dtRzVmZRjl6sD/iekkoORpaSCXY7OOV7zJyjcGc71Yi7dBi9q9uUc1NG410pp5TbL2BGdegRK/ ghupNk8ad25VvtNURpu8Lgin1AFfodS4nXxUy4xJcAh65OoFBYhOHi8nQmt0Mi6wfzSTrwjFvoa usfVbS36Z2YP5x2DetDmU3zn30MXsn33e0Royka2ezYeUdlUGLHZDnMZWWm35Qcj9VMJ9vNT9BP t3bXaeqdR1U3kNvutnn0dsvoS2fsgRJ2PTesQQV4h1ad9PwcG7bIApJdGKAh2dlCRy3FfFm2nzn 5zo1h6jw+jpsWsEyb0bqW5nCrN X-Received: by 2002:a05:6000:220b:b0:43b:43ae:8c25 with SMTP id ffacd0b85a97d-43b88a44842mr13657638f8f.52.1774549072132; Thu, 26 Mar 2026 11:17:52 -0700 (PDT) X-Received: by 2002:a05:6000:220b:b0:43b:43ae:8c25 with SMTP id ffacd0b85a97d-43b88a44842mr13657569f8f.52.1774549071543; Thu, 26 Mar 2026 11:17:51 -0700 (PDT) Received: from [192.168.10.48] ([151.49.85.67]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-43b9192e3f9sm10324620f8f.7.2026.03.26.11.17.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Mar 2026 11:17:50 -0700 (PDT) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Jon Kohler , Nikunj A Dadhania , Amit Shah , Sean Christopherson , Marcelo Tosatti Subject: [PATCH 10/24] KVM: x86/mmu: split XS/XU bits for EPT Date: Thu, 26 Mar 2026 19:17:08 +0100 Message-ID: <20260326181723.218115-11-pbonzini@redhat.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260326181723.218115-1-pbonzini@redhat.com> References: <20260326181723.218115-1-pbonzini@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When EPT is in use, replace ACC_USER_MASK with ACC_USER_EXEC_MASK, so that supervisor and user-mode execution can be controlled independently (ACC_USER_MASK would not allow a setting similar to XU=3D0 XS=3D1 W=3D1 R=3D1). Replace shadow_x_mask with shadow_xs_mask/shadow_xu_mask, to allow setting XS and XU bits separately in EPT entries. Note that ACC_USER_EXEC_MASK is already set through ACC_ALL in the kvm_mmu_page roles, but it does not propagate to the XU bit because shadow_xs_mask =3D=3D shadow_xu_mask. On the other hand, access tracking for eptad=3D0 does take it into account when saving/restoring page permissions. Signed-off-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/mmu/mmutrace.h | 6 ++--- arch/x86/kvm/mmu/spte.c | 49 +++++++++++++++++++++++-------------- arch/x86/kvm/mmu/spte.h | 8 +++--- 4 files changed, 40 insertions(+), 25 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index dd5419a1f891..a6ee467ad838 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5472,7 +5472,7 @@ static void reset_shadow_zero_bits_mask(struct kvm_vc= pu *vcpu, static inline bool boot_cpu_is_amd(void) { WARN_ON_ONCE(!tdp_enabled); - return shadow_x_mask =3D=3D 0; + return shadow_xs_mask =3D=3D 0; } =20 /* diff --git a/arch/x86/kvm/mmu/mmutrace.h b/arch/x86/kvm/mmu/mmutrace.h index dcfdfedfc4e9..3429c1413f42 100644 --- a/arch/x86/kvm/mmu/mmutrace.h +++ b/arch/x86/kvm/mmu/mmutrace.h @@ -357,8 +357,8 @@ TRACE_EVENT( __entry->sptep =3D virt_to_phys(sptep); __entry->level =3D level; __entry->r =3D shadow_present_mask || (__entry->spte & PT_PRESENT_MASK); - __entry->x =3D is_executable_pte(__entry->spte); - __entry->u =3D shadow_user_mask ? !!(__entry->spte & shadow_user_mask) := -1; + __entry->x =3D (__entry->spte & (shadow_xs_mask | shadow_nx_mask)) =3D= =3D shadow_xs_mask; + __entry->u =3D !!(__entry->spte & (shadow_xu_mask | shadow_user_mask)); ), =20 TP_printk("gfn %llx spte %llx (%s%s%s%s) level %d at %llx", @@ -366,7 +366,7 @@ TRACE_EVENT( __entry->r ? "r" : "-", __entry->spte & PT_WRITABLE_MASK ? "w" : "-", __entry->x ? "x" : "-", - __entry->u =3D=3D -1 ? "" : (__entry->u ? "u" : "-"), + __entry->u ? "u" : "-", __entry->level, __entry->sptep ) ); diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 7b5f118ae211..fc7eb73476f6 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -29,8 +29,9 @@ bool __read_mostly kvm_ad_enabled; u64 __read_mostly shadow_host_writable_mask; u64 __read_mostly shadow_mmu_writable_mask; u64 __read_mostly shadow_nx_mask; -u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ u64 __read_mostly shadow_user_mask; +u64 __read_mostly shadow_xs_mask; /* mutual exclusive with nx_mask and use= r_mask */ +u64 __read_mostly shadow_xu_mask; /* mutual exclusive with nx_mask and use= r_mask */ u64 __read_mostly shadow_accessed_mask; u64 __read_mostly shadow_dirty_mask; u64 __read_mostly shadow_mmio_value; @@ -216,22 +217,30 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_= page *sp, * when CR0.PG is toggled, but leveraging that to ignore the mitigation * would tie make_spte() further to vCPU/MMU state, and add complexity * just to optimize a mode that is anything but performance critical. + * + * Use ACC_USER_EXEC_MASK here assuming only Intel processors (EPT) + * are affected by the NX huge page erratum. */ - if (level > PG_LEVEL_4K && (pte_access & ACC_EXEC_MASK) && + if (level > PG_LEVEL_4K && + (pte_access & (ACC_EXEC_MASK | ACC_USER_EXEC_MASK)) && is_nx_huge_page_enabled(vcpu->kvm)) { - pte_access &=3D ~ACC_EXEC_MASK; + pte_access &=3D ~(ACC_EXEC_MASK | ACC_USER_EXEC_MASK); } =20 if (pte_access & ACC_READ_MASK) spte |=3D PT_PRESENT_MASK; /* or VMX_EPT_READABLE_MASK */ =20 - if (pte_access & ACC_EXEC_MASK) - spte |=3D shadow_x_mask; - else - spte |=3D shadow_nx_mask; - - if (pte_access & ACC_USER_MASK) - spte |=3D shadow_user_mask; + if (shadow_nx_mask) { + if (!(pte_access & ACC_EXEC_MASK)) + spte |=3D shadow_nx_mask; + if (pte_access & ACC_USER_MASK) + spte |=3D shadow_user_mask; + } else { + if (pte_access & ACC_EXEC_MASK) + spte |=3D shadow_xs_mask; + if (pte_access & ACC_USER_EXEC_MASK) + spte |=3D shadow_xu_mask; + } =20 if (level > PG_LEVEL_4K) spte |=3D PT_PAGE_SIZE_MASK; @@ -318,11 +327,13 @@ static u64 make_spte_executable(u64 spte, u8 access) { u64 set, clear; =20 - if (access & ACC_EXEC_MASK) - set =3D shadow_x_mask; + if (shadow_nx_mask) + set =3D (access & ACC_EXEC_MASK) ? 0 : shadow_nx_mask; else - set =3D shadow_nx_mask; - clear =3D set ^ (shadow_nx_mask | shadow_x_mask); + set =3D + (access & ACC_EXEC_MASK ? shadow_xs_mask : 0) | + (access & ACC_USER_EXEC_MASK ? shadow_xu_mask : 0); + clear =3D set ^ (shadow_nx_mask | shadow_xs_mask | shadow_xu_mask); return modify_spte_protections(spte, set, clear); } =20 @@ -389,7 +400,7 @@ u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled) =20 spte |=3D __pa(child_pt) | shadow_present_mask | PT_WRITABLE_MASK | PT_PRESENT_MASK /* or VMX_EPT_READABLE_MASK */ | - shadow_user_mask | shadow_x_mask | shadow_me_value; + shadow_user_mask | shadow_xs_mask | shadow_xu_mask | shadow_me_value; =20 if (ad_disabled) spte |=3D SPTE_TDP_AD_DISABLED; @@ -497,10 +508,11 @@ void kvm_mmu_set_ept_masks(bool has_ad_bits) shadow_accessed_mask =3D VMX_EPT_ACCESS_BIT; shadow_dirty_mask =3D VMX_EPT_DIRTY_BIT; shadow_nx_mask =3D 0ull; - shadow_x_mask =3D VMX_EPT_EXECUTABLE_MASK; + shadow_xs_mask =3D VMX_EPT_EXECUTABLE_MASK; + shadow_xu_mask =3D VMX_EPT_EXECUTABLE_MASK; shadow_present_mask =3D VMX_EPT_SUPPRESS_VE_BIT; =20 - shadow_acc_track_mask =3D VMX_EPT_RWX_MASK; + shadow_acc_track_mask =3D VMX_EPT_RWX_MASK | VMX_EPT_USER_EXECUTABLE_MASK; shadow_host_writable_mask =3D EPT_SPTE_HOST_WRITABLE; shadow_mmu_writable_mask =3D EPT_SPTE_MMU_WRITABLE; =20 @@ -548,7 +560,8 @@ void kvm_mmu_reset_all_pte_masks(void) shadow_accessed_mask =3D PT_ACCESSED_MASK; shadow_dirty_mask =3D PT_DIRTY_MASK; shadow_nx_mask =3D PT64_NX_MASK; - shadow_x_mask =3D 0; + shadow_xs_mask =3D 0; + shadow_xu_mask =3D 0; shadow_present_mask =3D PT_PRESENT_MASK; =20 shadow_acc_track_mask =3D 0; diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 121bfb2217e8..204f16aaf4e5 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -54,7 +54,8 @@ static_assert(SPTE_TDP_AD_ENABLED =3D=3D 0); =20 #define ACC_READ_MASK PT_PRESENT_MASK #define ACC_WRITE_MASK PT_WRITABLE_MASK -#define ACC_USER_MASK PT_USER_MASK +#define ACC_USER_MASK PT_USER_MASK /* non EPT */ +#define ACC_USER_EXEC_MASK ACC_USER_MASK /* EPT only */ #define ACC_EXEC_MASK 8 #define ACC_ALL (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK |= ACC_READ_MASK) =20 @@ -184,8 +185,9 @@ extern bool __read_mostly kvm_ad_enabled; extern u64 __read_mostly shadow_host_writable_mask; extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; -extern u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ extern u64 __read_mostly shadow_user_mask; +extern u64 __read_mostly shadow_xs_mask; /* mutual exclusive with nx_mask = and user_mask */ +extern u64 __read_mostly shadow_xu_mask; /* mutual exclusive with nx_mask = and user_mask */ extern u64 __read_mostly shadow_accessed_mask; extern u64 __read_mostly shadow_dirty_mask; extern u64 __read_mostly shadow_mmio_value; @@ -363,7 +365,7 @@ static inline bool is_last_spte(u64 pte, int level) =20 static inline bool is_executable_pte(u64 spte) { - return (spte & (shadow_x_mask | shadow_nx_mask)) =3D=3D shadow_x_mask; + return (spte & (shadow_xs_mask | shadow_xu_mask | shadow_nx_mask)) !=3D s= hadow_nx_mask; } =20 static inline kvm_pfn_t spte_to_pfn(u64 pte) --=20 2.53.0