From nobody Thu Dec 18 07:20:24 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 698BF1F12E7 for ; Thu, 13 Mar 2025 18:11:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889514; cv=none; b=PTRtuJH3m1kunr0+Gqx189uzi0bEaDS2wZT9qWGazx+ZvPtSTE5TFfcArFkt2kET0RoKnlKETlA0JynNYBWXkxpPI9dPqRT+ZCEmZBAvkV66/WrzbLcD8rkbqEo/r9M0NI01xDRC0Hc015sEBIXq94S6YBeiBZ74ZzbYWT4UfXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889514; c=relaxed/simple; bh=R7Z7ffLemwCnPpH5COhJObTeiUx2wzTsUCtrICT8b20=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UxFQpwXHhJzIUnAL3HRnLCsOkIF54eCWHyQkx1X8vMEZTYmsz008KIhz2L36DNGkR8p5Z1W97vIxgsdKAvOlPUduIrYaq7ytFL9p7zmmPjRZq0xHp10/wGJZtlqSFg9AbHvGEfIE85Rhs6u5tJ3U9BUzT5I/Dsj14u+Jx85QRaY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=T4iPb9+3; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="T4iPb9+3" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43d007b2c79so9403545e9.2 for ; Thu, 13 Mar 2025 11:11:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889511; x=1742494311; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=T4iPb9+35pSq3olPvxKQJXiTNWbvZ/oXKHfHCzsKHEAiWr/yIX63F9RmDEpUfnNkvP rNAXLjSzBA+mg3Lq3UnCMwRgYJjy43yynxBNtoPpmHIS2EqxH08lx5gRykfCb4wq/tJQ tosUxhstUa9Tcua4nhCPQCbItn6ef36okHsdSVPK6P3Z+fPWftfj1yd8udv/hf6ZepZC s27xDXihuCdf520FtLoUOSsOYXh3PvWVVosAe/Z0pt1ZkfC7ORrixNuublwqOPCEZFgN YiCIzEostx5ofHkJjyLtsq4dcLSoT3Y6r6QBNiYEs8j2vBglshnhBrOcv4RiiEfT9Nf2 X1kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889511; x=1742494311; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=B1gsB+NBjzKkun6duRp1ZyKyzm680AhV0O9aT1RhocAa8s8nnS6H5kqcjZ7Vg4/626 EJN4kqjIvZ8Tct7xBx9UX7NoYArvpaH9+0uICIAlpZot4IoY00ZSk2T1SaU7yoXR00mj osxc4Jzjqxml+pSvd8cqkIgWdoZ/ADGSChNP97prEpj50H12XCZlyQhfM4uBtArUjK74 ZbAwaXpy4ic1TPm2tVGoQ4CklqHlGxDI2OW1h83CxRvqmx2RqKPk4d5rs87QeRYnzTdm qis6N3IGKX/NSoNvjKLzcciQkTRxi50ztJPYrNbMohuBL9rEYcLAKB0tEDCekRX23UN1 GsMQ== X-Gm-Message-State: AOJu0YynvhC8+hRDBECl++v1fRCMChbMrt9BeAXDiD1omFpG0fsCk6do bif6TcPHjOuaTdNKfobRsCoebdOMLf+hAgJrK1TgrBGG07p2F/Ou07dwem2EaG9a347/9wqXsxr +dk0aMuKCrg== X-Google-Smtp-Source: AGHT+IF292MH6JqAkiIN8jtyln5hVYHbTRuF9S/Wm78jwx44s0FdKYuBZ0MucR5gttxIvSprRDkRwKMLpsnBdw== X-Received: from wmbfj6.prod.google.com ([2002:a05:600c:c86:b0:43c:f256:f4b3]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d12:b0:43c:eec7:eab7 with SMTP id 5b1f17b1804b1-43d01bdd224mr152891025e9.11.1741889510788; Thu, 13 Mar 2025 11:11:50 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:24 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-5-04972e046cea@google.com> Subject: [PATCH RFC HACKS 05/11] Add asi_map() and asi_unmap() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Do not worry too much about the implementation of this. I am trying to implement this more neatly using the x86 PAT logic but haven't managed to get it working in time for the RFC. To enable testing & review, I'm very hastily throwing something together that basically works, based on a simplified version of what was used for the latest RFC [0]. [0] https://lore.kernel.org/linux-mm/20250110-asi-rfc-v2-v2-0-8419288bc805@= google.com/ Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 3 ++ arch/x86/mm/asi.c | 77 ++++++++++++++++++++++++++++++++++++++++++= ++++ include/linux/asi.h | 2 ++ include/linux/vmalloc.h | 4 +++ mm/vmalloc.c | 32 +++++++++++-------- 5 files changed, 105 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index b8f604df6a36508acbc10710f821d5f95e8cdceb..cf8be544de8b108190b765e3eb3= 37089866207a2 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -23,6 +23,9 @@ static __always_inline pgd_t *asi_pgd(struct asi *asi) return asi ? asi->pgd : NULL; } =20 +void asi_map(struct page *page, int numpages); +void asi_unmap(struct page *page, int numpages); + #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ =20 #endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index e5a981a7b3192655cd981633514fbf945b92c9b6..570233224789631352891f47ac2= f0453a7adc06e 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -1,8 +1,85 @@ // SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include + #include +#include =20 static __aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD= ]; =20 struct asi __asi_global_nonsensitive =3D { .pgd =3D asi_global_nonsensitive_pgd, }; + +/* + * Map the given pages into the ASI nonsensitive physmap. The source of the + * mapping is the regular unrestricted page tables. Only supports mapping = at + * pageblock granularity. Does no synchronization. + */ +void asi_map(struct page *page, int numpages) +{ + unsigned long virt; + unsigned long start =3D (size_t)(page_to_virt(page)); + unsigned long end =3D start + PAGE_SIZE * numpages; + unsigned long page_size; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + for (virt =3D start; virt < end; virt =3D ALIGN(virt + 1, page_size)) { + pte_t *pte, *pte_asi; + int level, level_asi; + pgd_t *pgd =3D pgd_offset_pgd(asi_global_nonsensitive_pgd, virt); + + pte_asi =3D lookup_pgtable_in_pgd(pgd, virt, &level_asi); + page_size =3D page_level_size(level_asi); + + pte =3D lookup_address(virt, &level); + if (!pte || pte_none(*pte)) + continue; + + /* + * Physmap should already be setup by PAT code, with no pages + * smaller than 2M. This function should only be called at + * pageblock granularity. Thus it should never be required to + * break up pages here. + */ + if (WARN_ON_ONCE(!pte_asi) || + WARN_ON_ONCE(ALIGN_DOWN(virt, page_size) < virt) || + ALIGN(virt, page_size) > end) + continue; + + /* + * Existing mappings should already match the structure of the + * unrestricted physmap. + */ + if (WARN_ON_ONCE(level !=3D level_asi)) + continue; + + set_pte(pte_asi, *pte); + } +} + +/* + * Unmap pages previously mapped via asi_map(). + * + * Interrupts must be enabled as this does a TLB shootdown. + */ +void asi_unmap(struct page *page, int numpages) +{ + size_t start =3D (size_t)page_to_virt(page); + size_t end =3D start + (PAGE_SIZE * numpages); + pgtbl_mod_mask mask =3D 0; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + vunmap_pgd_range(asi_pgd(ASI_GLOBAL_NONSENSITIVE), start, end, &mask); + + flush_tlb_kernel_range(start, end - 1); +} diff --git a/include/linux/asi.h b/include/linux/asi.h index 2d3049d5fe423e139dcce8f3d68cdffcc0ec0bfe..ee9811f04a417556cf2e930644e= af05f3c9bfee3 100644 --- a/include/linux/asi.h +++ b/include/linux/asi.h @@ -13,6 +13,8 @@ struct asi {}; =20 static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } +static inline void asi_map(struct page *page, int numpages) { } +static inline void asi_unmap(struct page *page, int numpages) { } =20 #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _INCLUDE_ASI_H */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 31e9ffd936e39334ddaff910222d4751c18da5e7..c498ba127b4a511b5a6f10afa2a= ae535509fc153 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -11,6 +11,7 @@ #include /* pgprot_t */ #include #include +#include =20 #include =20 @@ -324,4 +325,7 @@ bool vmalloc_dump_obj(void *object); static inline bool vmalloc_dump_obj(void *object) { return false; } #endif =20 +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long = end, + pgtbl_mod_mask *mask); + #endif /* _LINUX_VMALLOC_H */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 61981ee1c9d2f769d4a06ab542fc84334c1b0cbd..ffeb823398809388c0599f51929= a7f3506ed035f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -427,6 +427,24 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long= addr, unsigned long end, } while (p4d++, addr =3D next, addr !=3D end); } =20 +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long = end, + pgtbl_mod_mask *mask) +{ + unsigned long next; + pgd_t *pgd =3D pgd_offset_pgd(pgd_table, addr); + + BUG_ON(addr >=3D end); + + do { + next =3D pgd_addr_end(addr, end); + if (pgd_bad(*pgd)) + *mask |=3D PGTBL_PGD_MODIFIED; + if (pgd_none_or_clear_bad(pgd)) + continue; + vunmap_p4d_range(pgd, addr, next, mask); + } while (pgd++, addr =3D next, addr !=3D end); +} + /* * vunmap_range_noflush is similar to vunmap_range, but does not * flush caches or TLBs. @@ -441,21 +459,9 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long= addr, unsigned long end, */ void __vunmap_range_noflush(unsigned long start, unsigned long end) { - unsigned long next; - pgd_t *pgd; - unsigned long addr =3D start; pgtbl_mod_mask mask =3D 0; =20 - BUG_ON(addr >=3D end); - pgd =3D pgd_offset_k(addr); - do { - next =3D pgd_addr_end(addr, end); - if (pgd_bad(*pgd)) - mask |=3D PGTBL_PGD_MODIFIED; - if (pgd_none_or_clear_bad(pgd)) - continue; - vunmap_p4d_range(pgd, addr, next, &mask); - } while (pgd++, addr =3D next, addr !=3D end); + vunmap_pgd_range(init_mm.pgd, start, end, &mask); =20 if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); --=20 2.49.0.rc1.451.g8f38331e32-goog