From nobody Sat Apr 4 06:08:31 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 031213EF0CF for ; Fri, 20 Mar 2026 18:24:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774031052; cv=none; b=q+PrPiGQ5bT8nJ9B3pY/qFO1BsXVIT96qWRB3+8smC0oE6P6HkxktkDqNA4rKOpwYP6+keg/I6qfK1RqidK4Y1h5Pre2QjwDdzLlyMWv8LeBt9IAxmSBWykwSh5QJMVjFynDJQzcdTE7pMorjPlDHKcYt9vlqEWOj2oV4xPnO+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774031052; c=relaxed/simple; bh=ouRXGQZQ8TA4aWFvMAgFnyK9HjOWmdsQrWDRneNw6K8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nI/I91dZiuKw+v02PyjKaXJUAo3PxSyg9B5hx44m04azJIq547Z9r1FZDVNRuUEnnO3+I1AIHXYACuNy1vTuS7r2WvQYKmRGdmVY4zPdeiul0qCWL7V5BLiB3BVlSP3TIi8okf7FdIHVfGCqlS+yFBV41VM36CQpTMzS9apUjgM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZA71MwXL; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZA71MwXL" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-48542d5aa9eso9199265e9.0 for ; Fri, 20 Mar 2026 11:24:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774031049; x=1774635849; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SDyJ1t2Ao3Sy5T6GqN9GajMItk/yFlj6HTa9OXsypQ4=; b=ZA71MwXLGMW8smj8WYqaakRcEQfHXwN3NrpLQZIalL/I/jPWfAoHHjWrmNKkSPRoQr 7au7N8OcUwZi1Kc0OIG2Yh2nqHT/Eeo1FrxFH9QTzk2xvGy/v7uXF3CphEeX3NlYCq4J QMF8lq6lPiHna/efJ0+fcvelAzrnyzdw6lKBbtnT2bU0rPLZG9AHZEHDfErwY5pHzmcK pUIKG8OQ6RFx5Pb4HWqfZdujsV/irWF8jlbg/RmXk0MFLV5cBGvfnHT0CN+s9tBFEIRA TUphSDbIlmyFVmrtq/sGTueZz2QqIn3ljtkFDpZiZCo2WVBFAP7DjbgOF2Q8yNzlgPrP ggzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774031049; x=1774635849; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SDyJ1t2Ao3Sy5T6GqN9GajMItk/yFlj6HTa9OXsypQ4=; b=mQk+wef6E2854bvf0Fa0eVlAzeQw6Zm1unOdaeS8de48yPSsHdt9bGYHM+R6TKeuVn yV79QRHYslLhsNJIRIMc3Va0J6BUu5MWrrjFrt4/CKMTEbkFJ8hc3IU/1TtbagOIkGds GueISpLo+wjcWTQd41CeXlezkDDsmLPYEEyVLPNjMf1uQxkXgdlIPt+oEIKkx0cs+Myz uYO+ZQN8wXZ+Hn35F/hMq3gLy2jjLarSa4uiQsKSbfvTg/CD5Q4rYIMIZcBVCkEqyXXQ fr6HQ6xMjIR3XYmkfg2LQjecAt1o38Xsz8APcGkqISvO4BLaquh4TfiZn6leVoP3qdF0 T4CA== X-Forwarded-Encrypted: i=1; AJvYcCWCzg2ag0MGfLYRfiCduwpYqCNDXvUGf5M7QEq7No99Z6wIK9wtTzeFqR/RgVOzSnEk+zikCbJSpiRV08U=@vger.kernel.org X-Gm-Message-State: AOJu0YzSXVd9rUIGonharPo8XkqIubqunLQ9ELCnqoSkLcksBlS9NdGU iByEkLSqvpgpKF1nnCmTg2c8NLUTiTCiBi0TzCZx1Z39jyBO+C9VLNzjByGxKeUaonFPLmSz1K7 LFpYwFM01tFnlZg== X-Received: from wmbhi3.prod.google.com ([2002:a05:600c:5343:b0:47f:941a:613d]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:524e:b0:485:3ae8:2231 with SMTP id 5b1f17b1804b1-486ff01efd8mr58852475e9.30.1774031049299; Fri, 20 Mar 2026 11:24:09 -0700 (PDT) Date: Fri, 20 Mar 2026 18:23:44 +0000 In-Reply-To: <20260320-page_alloc-unmapped-v2-0-28bf1bd54f41@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260320-page_alloc-unmapped-v2-0-28bf1bd54f41@google.com> X-Mailer: b4 0.14.3 Message-ID: <20260320-page_alloc-unmapped-v2-20-28bf1bd54f41@google.com> Subject: [PATCH v2 20/22] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations From: Brendan Jackman To: Borislav Petkov , Dave Hansen , Peter Zijlstra , Andrew Morton , David Hildenbrand , Vlastimil Babka , Wei Xu , Johannes Weiner , Zi Yan , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, rppt@kernel.org, Sumit Garg , derkling@google.com, reijiw@google.com, Will Deacon , rientjes@google.com, "Kalyazin, Nikita" , patrick.roy@linux.dev, "Itazuri, Takahiro" , Andy Lutomirski , David Kaplan , Thomas Gleixner , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The pages being zeroed here are unmapped, so they can't be zeroed via the direct map. Temporarily mapping them in the direct map is not possible because: - In general this requires allocating pagetables, - Unmapping them would require a TLB shootdown, which can't be done in general from the allocator (x86 requires IRQs on). Therefore, use the new mermap mechanism to zero these pages. The main mermap API is expected to fail very often. In order to avoid needing to fail allocations when that happens, instead fallback to the special mermap_get_reserved() variant, which is less efficient. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/pgtable_types.h | 2 + mm/Kconfig | 11 +++++- mm/page_alloc.c | 76 +++++++++++++++++++++++++++++++-= ---- 3 files changed, 78 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pg= table_types.h index 2ec250ba467e2..c3d73bdfff1fa 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -223,6 +223,7 @@ enum page_cache_mode { #define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) #define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G) #define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_NOGLOBAL (__PP|__RW| 0|___A|__NX|___D| 0| 0) #define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G) #define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| _= _NC) #define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G) @@ -245,6 +246,7 @@ enum page_cache_mode { #define __pgprot_mask(x) __pgprot((x) & __default_kernel_pte_mask) =20 #define PAGE_KERNEL __pgprot_mask(__PAGE_KERNEL | _ENC) +#define PAGE_KERNEL_NOGLOBAL __pgprot_mask(__PAGE_KERNEL_NOGLOBAL | _ENC) #define PAGE_KERNEL_NOENC __pgprot_mask(__PAGE_KERNEL | 0) #define PAGE_KERNEL_RO __pgprot_mask(__PAGE_KERNEL_RO | _ENC) #define PAGE_KERNEL_EXEC __pgprot_mask(__PAGE_KERNEL_EXEC | _ENC) diff --git a/mm/Kconfig b/mm/Kconfig index e4cb52149acad..05b2bb841d0e0 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1506,7 +1506,14 @@ config MERMAP_KUNIT_TEST If unsure, say N. =20 config PAGE_ALLOC_UNMAPPED - bool "Support allocating pages that aren't in the direct map" if COMPILE_= TEST - default COMPILE_TEST + bool "Support allocating pages that aren't in the direct map" + depends on MERMAP + +config PAGE_ALLOC_KUNIT_TESTS + tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Builds KUnit tests for the page allocator. =20 endmenu diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 710ee9f46d467..7c91dcbe32576 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -14,6 +14,7 @@ * (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ =20 +#include #include #include #include @@ -1327,15 +1328,72 @@ static inline bool should_skip_kasan_poison(struct = page *page) return page_kasan_tag(page) =3D=3D KASAN_TAG_KERNEL; } =20 -static void kernel_init_pages(struct page *page, int numpages) +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +static inline bool pageblock_unmapped(struct page *page) { - int i; + return freetype_flags(get_pageblock_freetype(page)) & FREETYPE_UNMAPPED; +} =20 - /* s390's use of memset() could override KASAN redzones. */ - kasan_disable_current(); - for (i =3D 0; i < numpages; i++) - clear_highpage_kasan_tagged(page + i); - kasan_enable_current(); +static inline void clear_page_mermap(struct page *page, unsigned int numpa= ges) +{ + void *mermap; + + BUILD_BUG_ON(IS_ENABLED(CONFIG_HIGHMEM)); + + /* Fast path: single mapping (may fail under preemption). */ + mermap =3D mermap_get(page, numpages << PAGE_SHIFT, PAGE_KERNEL_NOGLOBAL); + if (mermap) { + void *buf =3D kasan_reset_tag(mermap_addr(mermap)); + + for (int i =3D 0; i < numpages; i++) + clear_page(buf + (i << PAGE_SHIFT)); + mermap_put(mermap); + return; + } + + /* Slow path, map each page individually (always succeeds). */ + for (int i =3D 0; i < numpages; i++) { + unsigned long flags; + + local_irq_save(flags); + mermap =3D mermap_get_reserved(page + i, PAGE_KERNEL_NOGLOBAL); + clear_page(kasan_reset_tag(mermap_addr(mermap))); + mermap_put(mermap); + local_irq_restore(flags); + } +} +#else +static inline bool pageblock_unmapped(struct page *page) +{ + return false; +} + +static inline void clear_page_mermap(struct page *page, unsigned int numpa= ges) +{ + BUG(); +} +#endif + +static void kernel_init_pages(struct page *page, unsigned int numpages) +{ + int num_blocks =3D DIV_ROUND_UP(numpages, pageblock_nr_pages); + + for (int block =3D 0; block < num_blocks; block++) { + struct page *block_page =3D page + (block << pageblock_order); + bool unmapped =3D pageblock_unmapped(block_page); + + /* s390's use of memset() could override KASAN redzones. */ + kasan_disable_current(); + if (unmapped) { + clear_page_mermap(block_page, numpages); + } else { + for (int i =3D 0; i < min(numpages, pageblock_nr_pages); i++) + clear_highpage_kasan_tagged(block_page + i); + } + kasan_enable_current(); + + numpages -=3D pageblock_nr_pages; + } } =20 #ifdef CONFIG_MEM_ALLOC_PROFILING @@ -5250,8 +5308,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask= , unsigned int order, ac->nodemask =3D nodemask; ac->freetype =3D gfp_freetype(gfp_mask); =20 - /* Not implemented yet. */ - if (freetype_flags(ac->freetype) & FREETYPE_UNMAPPED && gfp_mask & __GFP_= ZERO) + if (freetype_flags(ac->freetype) & FREETYPE_UNMAPPED && + WARN_ON(!mermap_ready())) return false; =20 if (cpusets_enabled()) { --=20 2.51.2