From nobody Wed Dec 17 10:45:03 2025 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A005517A5A3 for ; Fri, 12 Jul 2024 17:01:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720803677; cv=none; b=k+lj2bwjcRxRlAn2SsKQliPRB/B3k3n6i/iyKEtC7qYvJACaqeOHTWKGBLdpybfVormexa05pr1rtsRD8tysfWFAnDDBN4MsWQSxufiyZWlKX7OJ/bfmCpREjOy8yTWPh+zixmiQabDlDT6lJaAsL+lilgNioMwPiKKTOaA2kYQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720803677; c=relaxed/simple; bh=ysS/sHsfGcOywr5ecIDWgWosegBwRS/ro8phsMXIPz4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tiZcKxcmVAjFRWHT52DSjVONReum1VwbSNV9z/xwXzZwXalp981/OY+IpNSs+iycZTr32U93vR50Axd51QulHT7goaViDJHdI1iy+fgYiDemeHXHPjQDL+mx7LYiIuaIulmMQlopOKlP1LIR4XOX9pgjLgPE0VoLNIMMvlGY3yo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pk/WG/fz; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pk/WG/fz" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e035f7b5976so4162815276.0 for ; Fri, 12 Jul 2024 10:01:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803675; x=1721408475; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=pk/WG/fz6s6/h2pPP3hkwdrfCZOpJr0JqEWwtHGKTm6ffeauyv1fxP7yiu80GgzWYH Dlg1qxyrCru9YHA2HRYfnuSATsDsIr6qYPzf7CBXsAdPfHXOpTk++C4Ay5lfcYQqmRXC jhgNUS0pWhh9/BMP8HOJEASo+bQRFHbR81Afjdy77g0YLD8v9KYDAuotCAgqVS1sDvrh QMHIhLfP6BSoRrmy8z141exdP3BZ44P/4WD1Q0Gd6shYlDKdoSvhXE60ZOAlvjlG7cAM T+9fWdTxOQkLL/ak3CsoJZrKFvPTlRoe84UsWBZUJyG2EfHQfGNKRtLwAU0XpY7xhHP7 6kZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803675; x=1721408475; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=FjXFSt4oFYpe63snYc+ErFxvmuySADXbwLVmbC5FTfydz11cBo84GY0RVcOc2XbfTq i3EWylpZAdLwVXTTiVbVzHP17ncLkM42M6VwPmuB2OQh6qAXHS6bjo7QuGqET32QeHla Oq6QHmo+85VJIuQsEr0j3TDG6iPARXmICaQE+zQyZgKPPGO4Cr0fEqNv2NQSpE9yVeuP Hhp/o7CpGTFRCWMwvn4XGKLSQrHYxVY2U7bH4DPQ0X3IKz7jaGWMXU/7LsovHNFZQQEY e6cQm0ZdKdQjDq/aRPvgW+R2JmkhmvmnfKnIwC4svmqnST1BXNelNpCVb8WtRMkZPZVS kWlg== X-Forwarded-Encrypted: i=1; AJvYcCVjTRm+vwikDYcyFGGIEhUvg0FIgpKEnxLP241uaKLZYV4TCS2B7nAIjhyHXrjH0alf3W5at0/rMCb3UyGjYkVH0hLf0onT9QZ2+XYc X-Gm-Message-State: AOJu0YwhNw9keTV9P6fsT/2qh7iOeUGW+0smg1bBpX3oh33VMfxegpVi 6uFp5+IXl3DoVvT7ZHebelvnjiFj6UR4dOuJQvjGwuk0S+jwC4ihP5ly0+kNEktbUw1GLpMjale okPW3mzHzVw== X-Google-Smtp-Source: AGHT+IH090AhnVnRaBqnnakPZ4/V5j0nTOpLdhxmJg4znTZivgjhg0KK45kar3tFbJqN9PhohJA5PmDXL/BkUA== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a25:945:0:b0:e05:7113:920c with SMTP id 3f1490d57ef6-e058a6cfb8dmr11142276.6.1720803674689; Fri, 12 Jul 2024 10:01:14 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:26 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-8-144b319a40d8@google.com> Subject: [PATCH 08/26] mm: asi: Use separate PCIDs for restricted address spaces From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Junaid Shahid Each restricted address space is assigned a separate PCID. Since currently only one ASI instance per-class exists for a given process, the PCID is just derived from the class index. This commit only sets the appropriate PCID when switching CR3, but does not actually use the NOFLUSH bit. That will be done by later patches. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 10 +++++++++- arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/mm/asi.c | 7 ++++--- arch/x86/mm/tlb.c | 44 +++++++++++++++++++++++++++++++++++++= ---- 4 files changed, 56 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index df34a8c0560b..1a19a925300c 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -69,7 +69,14 @@ #define static_asi_enabled() cpu_feature_enabled(X86_FEATURE_ASI) =20 #define ASI_MAX_NUM_ORDER 2 -#define ASI_MAX_NUM (1 << ASI_MAX_NUM_ORDER) +/* + * We include an ASI identifier in the higher bits of PCID to use + * different PCID for restricted ASIs from non-restricted ASIs (see asi_pc= id). + * The ASI identifier we use for this is asi_index + 1, as asi_index + * starts from 0. The -1 below for ASI_MAX_NUM comes from this PCID + * space availability. + */ +#define ASI_MAX_NUM ((1 << ASI_MAX_NUM_ORDER) - 1) =20 struct asi_hooks { /* @@ -101,6 +108,7 @@ struct asi { struct asi_class *class; struct mm_struct *mm; int64_t ref_count; + u16 index; }; =20 DECLARE_PER_CPU_ALIGNED(struct asi *, curr_asi); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflus= h.h index ed847567b25d..3605f6b99da7 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -392,6 +392,9 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, p= md_t newpmd) #define huge_pmd_needs_flush huge_pmd_needs_flush =20 unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam); +unsigned long build_cr3_pcid(pgd_t *pgd, u16 pcid, unsigned long lam, bool= noflush); + +u16 asi_pcid(struct asi *asi, u16 asid); =20 #ifdef CONFIG_ADDRESS_MASKING static inline u64 tlbstate_lam_cr3_mask(void) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 2cd8e93a4415..0ba156f879d3 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -140,6 +140,7 @@ int asi_init(struct mm_struct *mm, int asi_index, struc= t asi **out_asi) =20 asi->class =3D &asi_class[asi_index]; asi->mm =3D mm; + asi->index =3D asi_index; =20 exit_unlock: if (err) @@ -174,6 +175,7 @@ EXPORT_SYMBOL_GPL(asi_destroy); noinstr void __asi_enter(void) { u64 asi_cr3; + u16 pcid; struct asi *target =3D asi_get_target(current); =20 /* @@ -200,9 +202,8 @@ noinstr void __asi_enter(void) */ this_cpu_write(curr_asi, target); =20 - asi_cr3 =3D build_cr3(target->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid), - tlbstate_lam_cr3_mask()); + pcid =3D asi_pcid(target, this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + asi_cr3 =3D build_cr3_pcid(target->pgd, pcid, tlbstate_lam_cr3_mask(), fa= lse); write_cr3(asi_cr3); =20 if (target->class->ops.post_asi_enter) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 9a5afeac9654..34d61b56d33f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -98,7 +98,12 @@ # define PTI_CONSUMED_PCID_BITS 0 #endif =20 -#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS) +#define ASI_CONSUMED_PCID_BITS ASI_MAX_NUM_ORDER +#define ASI_PCID_BITS_SHIFT CR3_AVAIL_PCID_BITS +#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS - \ + ASI_CONSUMED_PCID_BITS) + +static_assert(BIT(CR3_AVAIL_PCID_BITS) > TLB_NR_DYN_ASIDS); =20 /* * ASIDs are zero-based: 0->MAX_AVAIL_ASID are valid. -1 below to account @@ -155,18 +160,23 @@ static inline u16 user_pcid(u16 asid) return ret; } =20 +static inline unsigned long __build_cr3(pgd_t *pgd, u16 pcid, unsigned lon= g lam) +{ + return __sme_pa_nodebug(pgd) | pcid | lam; +} + inline_or_noinstr unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned l= ong lam) { - unsigned long cr3 =3D __sme_pa_nodebug(pgd) | lam; + u16 pcid =3D 0; =20 if (static_cpu_has(X86_FEATURE_PCID)) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); - cr3 |=3D kern_pcid(asid); + pcid =3D kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid !=3D 0); } =20 - return cr3; + return __build_cr3(pgd, pcid, lam); } =20 static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, @@ -181,6 +191,19 @@ static inline unsigned long build_cr3_noflush(pgd_t *p= gd, u16 asid, return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } =20 +inline_or_noinstr unsigned long build_cr3_pcid(pgd_t *pgd, u16 pcid, + unsigned long lam, bool noflush) +{ + u64 noflush_bit =3D 0; + + if (!static_cpu_has(X86_FEATURE_PCID)) + pcid =3D 0; + else if (noflush) + noflush_bit =3D CR3_NOFLUSH; + + return __build_cr3(pgd, pcid, lam) | noflush_bit; +} + /* * We get here when we do something requiring a TLB invalidation * but could not go invalidate all of the contexts. We do the @@ -995,6 +1018,19 @@ static void put_flush_tlb_info(void) #endif } =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +inline_or_noinstr u16 asi_pcid(struct asi *asi, u16 asid) +{ + return kern_pcid(asid) | ((asi->index + 1) << ASI_PCID_BITS_SHIFT); +} + +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +u16 asi_pcid(struct asi *asi, u16 asid) { return kern_pcid(asid); } + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables) --=20 2.45.2.993.g49e7a77208-goog