From nobody Mon Feb 9 19:26:35 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A2AB1EEA4B for ; Wed, 29 Jan 2025 22:42:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738190563; cv=none; b=J3UhkfPttva0+0clfKQa9ipcxrp/njKyUsveaQP1VWKXQWe5jxzMvBoWMEqnRVa1oPL+tLUU5QqooyiV5bT0WB3mV7nhWKrXPUsiAOOs8iDLj0xglqds3g1bJt7B5PVOA5vQ0arqHZ3sib8o8xejyNGA85Bswla4inQ4AeGVXMM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738190563; c=relaxed/simple; bh=afzoeEgScdzoLRJeIhQVvNfYS6XRmn7ijiQwW15m+Kw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UM1HdAINFG+DSaQZcM7QRLSXHwWLomhDYKaPsS0uy0hHjnUrKCvqETUgamvl83B/SGNICpmIVKKVnwkJ1iSMNHDiLcSl16DQK0lhGdquEzmYYCfbxj+WgqG3RwRese0ZInqVdSVV8N+5CBOtCd5WJxfkooCF822CZqnSNEd59ek= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--fvdl.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=p9QBqXLv; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--fvdl.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="p9QBqXLv" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2166855029eso3072785ad.0 for ; Wed, 29 Jan 2025 14:42:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738190561; x=1738795361; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=csxUnMb8aBY+LA7oJhb6s5xyn3ur++3PPVPSPZ4uvnE=; b=p9QBqXLv3aQRv7qy+Za+z2eaSzOnFno6cJJKZ8CctimxTXHXTXweM85GThlYN/nmmV /kiflRtmxGrZ7f/ZZ7VZL6xbWoRfkdN/9aU7WFdNhqWLBgc8wQxg3UXRP2ZJJpzBFYMz HR2QnUSg9UnhxFuKUAJLiOxTCMChTv40UBAIteVWeGc+mQ4wK+JQeNSyW/VrTUn1logQ LUwZfXp7KRaLFqbySwH618Ic3CaUZ1Tp/RhscoZ+eiPNC4ta7lvFVGZX8nV1tqxDxCzC V0NR6sT1fr+wWyDNaNmYbbaGpRFXP2EzVxBUsUSZi4EXfVabGHmj4uGUcN8QI/nYDBi3 kzuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738190561; x=1738795361; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=csxUnMb8aBY+LA7oJhb6s5xyn3ur++3PPVPSPZ4uvnE=; b=ErZQEB9eGcdo0QsLhBM/VbwqJb1n85bYU2JdPR2By4R84BVwhvlZVQ8hMs0XTNGfvH uuMr3oFRCGlqX9vtd90rzaVbnFZtzyu51NKwTpZCSL8GWPWgiyORYhQfmp2nSUsJIYMY jcuCIT7Lc8nlsXWfRICNFFi+PvhpOudgYmCjGTJVGpgwJ0D5HKvg6Q4QqETNl/uAlQf8 PeY4NHJc/icT+6q+4lPG+8YSXaXbbsbbfw/TgbBO3v8b6OnfnUUs6+R+EuFSal/XLTo5 Z4Qkebeh7OhJ31qoiLK72DJ5OK4WFEddXZ00jE5/T6wP9YNDsFr77UG2R8SJi6RYwVRt UTIg== X-Forwarded-Encrypted: i=1; AJvYcCX/holTjDRJJtp6g0DS4zdYkSOmCGzLj1DzUCdUYOrRhtBMrLbDxbQDhb80cqVxROhq8H/vQo/7BHpQlNU=@vger.kernel.org X-Gm-Message-State: AOJu0YyJuZoBHA6EeZc1akMMNOX7qvwzNX+cyDYdOXld2uQZZ+W2+kFW ogbMLvWwKP7OOkJ4Wy8K6iHZ9OIA6RcGOkEFHwQUEZbZ6no3HyV6INMstbTNPxorIw/0iA== X-Google-Smtp-Source: AGHT+IE8Pv/vPlQg3Zsgd/miPk81S+cEO0cUOfh7UtusaUeUsumjXw+Yjo59fdOEUHr7Es0q7li1hf72 X-Received: from pfbbe5.prod.google.com ([2002:a05:6a00:1f05:b0:728:e3af:6bb0]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:3a0f:b0:71e:e4f:3e58 with SMTP id d2e1a72fcca58-72fd0c623ccmr6357836b3a.17.1738190560818; Wed, 29 Jan 2025 14:42:40 -0800 (PST) Date: Wed, 29 Jan 2025 22:41:44 +0000 In-Reply-To: <20250129224157.2046079-1-fvdl@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250129224157.2046079-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250129224157.2046079-16-fvdl@google.com> Subject: [PATCH v2 15/28] mm/sparse: add vmemmap_*_hvo functions From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a few functions to enable early HVO: vmemmap_populate_hvo vmemmap_undo_hvo vmemmap_wrprotect_hvo The populate and undo functions are expected to be used in early init, from the sparse_init_nid_early() function. The wrprotect function is to be used, potentially, later. To implement these functions, mostly re-use the existing compound pages vmemmap logic used by DAX. vmemmap_populate_address has its argument changed a bit in this commit: the page structure passed in to be reused in the mapping is replaced by a PFN and a flag. The flag indicates whether an extra ref should be taken on the vmemmap page containing the head page structure. Taking the ref is appropriate to for DAX / ZONE_DEVICE, but not for HugeTLB HVO. The HugeTLB vmemmap optimization maps tail page structure pages read-only. The vmemmap_wrprotect_hvo function that does this is implemented separately, because it cannot be guaranteed that reserved page structures will not be write accessed during memory initialization. Even with CONFIG_DEFERRED_STRUCT_PAGE_INIT, they might still be written to (if they are at the bottom of a zone). So, vmemmap_populate_hvo leaves the tail page structure pages RW initially, and then later during initialization, after memmap init is fully done, vmemmap_wrprotect_hvo must be called to finish the job. Subsequent commits will use these functions for early HugeTLB HVO. Signed-off-by: Frank van der Linden --- include/linux/mm.h | 9 ++- mm/sparse-vmemmap.c | 141 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 135 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index df83653ed6e3..0463c062fd7a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3837,7 +3837,8 @@ p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long= addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap, struct page *reuse); + struct vmem_altmap *altmap, unsigned long ptpfn, + unsigned long flags); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node, @@ -3853,6 +3854,12 @@ int vmemmap_populate_hugepages(unsigned long start, = unsigned long end, int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); +int vmemmap_populate_hvo(unsigned long start, unsigned long end, int node, + unsigned long headsize); +int vmemmap_undo_hvo(unsigned long start, unsigned long end, int node, + unsigned long headsize); +void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int nod= e, + unsigned long headsize); void vmemmap_populate_print_last(void); #ifdef CONFIG_MEMORY_HOTPLUG void vmemmap_free(unsigned long start, unsigned long end, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 8751c46c35e4..bee22ca93654 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -30,6 +30,13 @@ =20 #include #include +#include + +/* + * Flags for vmemmap_populate_range and friends. + */ +/* Get a ref on the head page struct page, for ZONE_DEVICE compound pages = */ +#define VMEMMAP_POPULATE_PAGEREF 0x0001 =20 #include "internal.h" =20 @@ -144,17 +151,18 @@ void __meminit vmemmap_verify(pte_t *pte, int node, =20 pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int= node, struct vmem_altmap *altmap, - struct page *reuse) + unsigned long ptpfn, unsigned long flags) { pte_t *pte =3D pte_offset_kernel(pmd, addr); if (pte_none(ptep_get(pte))) { pte_t entry; void *p; =20 - if (!reuse) { + if (!ptpfn) { p =3D vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); if (!p) return NULL; + ptpfn =3D PHYS_PFN(__pa(p)); } else { /* * When a PTE/PMD entry is freed from the init_mm @@ -165,10 +173,10 @@ pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, un= signed long addr, int node, * and through vmemmap_populate_compound_pages() when * slab is available. */ - get_page(reuse); - p =3D page_to_virt(reuse); + if (flags & VMEMMAP_POPULATE_PAGEREF) + get_page(pfn_to_page(ptpfn)); } - entry =3D pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + entry =3D pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } return pte; @@ -238,7 +246,8 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long ad= dr, int node) =20 static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int = node, struct vmem_altmap *altmap, - struct page *reuse) + unsigned long ptpfn, + unsigned long flags) { pgd_t *pgd; p4d_t *p4d; @@ -258,7 +267,7 @@ static pte_t * __meminit vmemmap_populate_address(unsig= ned long addr, int node, pmd =3D vmemmap_pmd_populate(pud, addr, node); if (!pmd) return NULL; - pte =3D vmemmap_pte_populate(pmd, addr, node, altmap, reuse); + pte =3D vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn, flags); if (!pte) return NULL; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); @@ -269,13 +278,15 @@ static pte_t * __meminit vmemmap_populate_address(uns= igned long addr, int node, static int __meminit vmemmap_populate_range(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap, - struct page *reuse) + unsigned long ptpfn, + unsigned long flags) { unsigned long addr =3D start; pte_t *pte; =20 for (; addr < end; addr +=3D PAGE_SIZE) { - pte =3D vmemmap_populate_address(addr, node, altmap, reuse); + pte =3D vmemmap_populate_address(addr, node, altmap, + ptpfn, flags); if (!pte) return -ENOMEM; } @@ -286,7 +297,107 @@ static int __meminit vmemmap_populate_range(unsigned = long start, int __meminit vmemmap_populate_basepages(unsigned long start, unsigned lon= g end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_range(start, end, node, altmap, NULL); + return vmemmap_populate_range(start, end, node, altmap, 0, 0); +} + +/* + * Undo populate_hvo, and replace it with a normal base page mapping. + * Used in memory init in case a HVO mapping needs to be undone. + * + * This can happen when it is discovered that a memblock allocated + * hugetlb page spans multiple zones, which can only be verified + * after zones have been initialized. + * + * We know that: + * 1) The first @headsize / PAGE_SIZE vmemmap pages were individually + * allocated through memblock, and mapped. + * + * 2) The rest of the vmemmap pages are mirrors of the last head page. + */ +int __meminit vmemmap_undo_hvo(unsigned long addr, unsigned long end, + int node, unsigned long headsize) +{ + unsigned long maddr, pfn; + pte_t *pte; + int headpages; + + /* + * Should only be called early in boot, so nothing will + * be accessing these page structures. + */ + WARN_ON(!early_boot_irqs_disabled); + + headpages =3D headsize >> PAGE_SHIFT; + + /* + * Clear mirrored mappings for tail page structs. + */ + for (maddr =3D addr + headsize; maddr < end; maddr +=3D PAGE_SIZE) { + pte =3D virt_to_kpte(maddr); + pte_clear(&init_mm, maddr, pte); + } + + /* + * Clear and free mappings for head page and first tail page + * structs. + */ + for (maddr =3D addr; headpages-- > 0; maddr +=3D PAGE_SIZE) { + pte =3D virt_to_kpte(maddr); + pfn =3D pte_pfn(ptep_get(pte)); + pte_clear(&init_mm, maddr, pte); + memblock_phys_free(PFN_PHYS(pfn), PAGE_SIZE); + } + + flush_tlb_kernel_range(addr, end); + + return vmemmap_populate(addr, end, node, NULL); +} + +/* + * Write protect the mirrored tail page structs for HVO. This will be + * called from the hugetlb code when gathering and initializing the + * memblock allocated gigantic pages. The write protect can't be + * done earlier, since it can't be guaranteed that the reserved + * page structures will not be written to during initialization, + * even if CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled. + * + * The PTEs are known to exist, and nothing else should be touching + * these pages. The caller is responsible for any TLB flushing. + */ +void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, + int node, unsigned long headsize) +{ + unsigned long maddr; + pte_t *pte; + + for (maddr =3D addr + headsize; maddr < end; maddr +=3D PAGE_SIZE) { + pte =3D virt_to_kpte(maddr); + ptep_set_wrprotect(&init_mm, maddr, pte); + } +} + +/* + * Populate vmemmap pages HVO-style. The first page contains the head + * page and needed tail pages, the other ones are mirrors of the first + * page. + */ +int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, + int node, unsigned long headsize) +{ + pte_t *pte; + unsigned long maddr; + + for (maddr =3D addr; maddr < addr + headsize; maddr +=3D PAGE_SIZE) { + pte =3D vmemmap_populate_address(maddr, node, NULL, 0, 0); + if (!pte) + return -ENOMEM; + } + + /* + * Reuse the last page struct page mapped above for the rest. + */ + return vmemmap_populate_range(maddr, end, node, NULL, + pte_pfn(ptep_get(pte)), 0); } =20 void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, @@ -409,7 +520,8 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, * with just tail struct pages. */ return vmemmap_populate_range(start, end, node, NULL, - pte_page(ptep_get(pte))); + pte_pfn(ptep_get(pte)), + VMEMMAP_POPULATE_PAGEREF); } =20 size =3D min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); @@ -417,13 +529,13 @@ static int __meminit vmemmap_populate_compound_pages(= unsigned long start_pfn, unsigned long next, last =3D addr + size; =20 /* Populate the head page vmemmap page */ - pte =3D vmemmap_populate_address(addr, node, NULL, NULL); + pte =3D vmemmap_populate_address(addr, node, NULL, 0, 0); if (!pte) return -ENOMEM; =20 /* Populate the tail pages vmemmap page */ next =3D addr + PAGE_SIZE; - pte =3D vmemmap_populate_address(next, node, NULL, NULL); + pte =3D vmemmap_populate_address(next, node, NULL, 0, 0); if (!pte) return -ENOMEM; =20 @@ -433,7 +545,8 @@ static int __meminit vmemmap_populate_compound_pages(un= signed long start_pfn, */ next +=3D PAGE_SIZE; rc =3D vmemmap_populate_range(next, last, node, NULL, - pte_page(ptep_get(pte))); + pte_pfn(ptep_get(pte)), + VMEMMAP_POPULATE_PAGEREF); if (rc) return -ENOMEM; } --=20 2.48.1.262.g85cc9f2d1e-goog