From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57FD81A239A for ; Wed, 8 Apr 2026 02:51:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616696; cv=none; b=gSJs3be9LSf0kSFuKkzZSh02OJFhXCzLcrFBA6RMS+WgwFUs+99sMQXkZJ1Xna10PimXt7TFyjOqyN/uUGPpeTScacSEMLg2dNUHEiJ5UKczTGAlaPc65J3GAFl0Q6l6UF2KsAfst8HhmcC5LNnRO4RVWtRf3+Bmu9OdzodmdJk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616696; c=relaxed/simple; bh=4KmW52ojotdS2xyp5e7cHouikLW+lgqmv208rSJWWic=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Lf9B4Y02RiJdC11U+h1c97iG0FafdpFR3q8l+feHNsi+hEdnk0URRYbAk4SIYmJcCAV30p1mR7zEQt4L2LFfiAm84nj5nMltdr7ykdd4Q4cwVET+xZGURSt6dDGIoytAubRZCYpqMmi2VcSpjH5pK9M8wS6zBqlFziP2ulHJDtA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BzW0ige8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BzW0ige8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4A765C19421; Wed, 8 Apr 2026 02:51:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616696; bh=4KmW52ojotdS2xyp5e7cHouikLW+lgqmv208rSJWWic=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BzW0ige8moHKxCsRbPugd62Yndyj4ujFsahw2nRcpTm9uavw9pSg+Cl3QXZMbhbOs CRJ/obpvlt+QtfpvDJqG8erfrQUDxd/jWu23tqqnM6x+1x30CnyupCQeq5j9qD0kxm JCSvjzPRI2QgUgvpr9pYX6THqbL5wDqC0F8YDbumK+wvx3tKpJxUN8VB+WXAk1WGa5 RJuGyF0FkI25BxDdvd7tqG7jgI2YTJ7vwg7NSKi9reKMPaQ8jeRM/iCXcV5rbidTje H40htvvu58zYGzXh1d37lXzt0+aPfGvRwj8LGEX24mIvIpT14Bwm7cRX5l5Wy4tBOb SutlY4vi8RBXQ== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 1/8] arm64/hugetlb: Extend batching of multiple CONT_PTE in a single PTE setup Date: Wed, 8 Apr 2026 10:51:08 +0800 Message-Id: <20260408025115.27368-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For sizes aligned to CONT_PTE_SIZE and smaller than PMD_SIZE, we can batch CONT_PTE settings instead of handling them individually. Signed-off-by: Barry Song (Xiaomi) --- arch/arm64/mm/hugetlbpage.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index a42c05cf5640..bf31c11ebd3b 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -110,6 +110,12 @@ static inline int num_contig_ptes(unsigned long size, = size_t *pgsize) contig_ptes =3D CONT_PTES; break; default: + if (size < CONT_PMD_SIZE && size > 0 && + IS_ALIGNED(size, CONT_PTE_SIZE)) { + contig_ptes =3D size >> PAGE_SHIFT; + *pgsize =3D PAGE_SIZE; + break; + } WARN_ON(!__hugetlb_valid_size(size)); } =20 @@ -359,6 +365,10 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shi= ft, vm_flags_t flags) case CONT_PTE_SIZE: return pte_mkcont(entry); default: + if (pagesize < CONT_PMD_SIZE && pagesize > 0 && + IS_ALIGNED(pagesize, CONT_PTE_SIZE)) + return pte_mkcont(entry); + break; } pr_warn("%s: unrecognized huge page size 0x%lx\n", --=20 2.39.3 (Apple Git-146) From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E2AE202C29 for ; Wed, 8 Apr 2026 02:51:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616699; cv=none; b=Wp3lhm2fYGfHCSOsD9GWuZNTLcdPU6vSGjk2Z8HlGIu35D8t6QpxtDe/7Y+5gb2jKIGZLPymFVhb2Ueffvs2s2sdshP8QyYXyKnIB8RUtkZHES9eW3vAQO8sli4zqGirWedI7Eil0PSvVEpC9ikzlVyT/8z7K49exCVwE2bWsfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616699; c=relaxed/simple; bh=WaU87kHk5fdaxiwAyibksVupa8Ba+++SUKhvjx8Yo4g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EXyFfYvx511ZXRW61ggozRBGUcmcmslWc01DGh5RpbFihtX0UM9dTlrTFipMwaIUhOvzdjqrESN7xg2aEtD4GuBZNdCKDSonDaxrFls7Rirj/lFS0mNhC4hGxpjZTNZjuNhG065pJpvC8LxFW1mLiyNe5VymIkcVPuY+vCMZ/OU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dx7NiL2d; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dx7NiL2d" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 715D1C19424; Wed, 8 Apr 2026 02:51:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616699; bh=WaU87kHk5fdaxiwAyibksVupa8Ba+++SUKhvjx8Yo4g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dx7NiL2d22U6fw3kZORHXgxH7PInpyHrU15js7b2oe2n3ns+9bjxR6fCg5/6GWmbZ X100BzWea1lmMJjJ6MdI4MUdHcBeVF7B5uM/WEvWcCEm8BVy6Pbh1p/3A+RD0RPLyd ibBOiRdc68xmP+7tu0ifGfDelO2NXJ+pGkpj3oWQRlDQvP2UoUKzd/9deSD6GkZpfj CWHUftf54l4oWYzxj53bFKSLczJLFYUKvXew2QSbMul0LTXPMa71Mw+zyAJ6cgsVd6 2i+tgpHiraeRUGyW7OH5pBqjXSAwzzFJJYGcfEpN75VTBYNu2rwUlTiNTjPq/GQsBi vAmBHA+W6YXnA== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 2/8] arm64/vmalloc: Allow arch_vmap_pte_range_map_size to batch multiple CONT_PTE Date: Wed, 8 Apr 2026 10:51:09 +0800 Message-Id: <20260408025115.27368-3-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Allow arch_vmap_pte_range_map_size to batch multiple CONT_PTE hugepages, reducing both PTE setup and TLB flush iterations. Signed-off-by: Barry Song (Xiaomi) --- arch/arm64/include/asm/vmalloc.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmal= loc.h index 4ec1acd3c1b3..9eea06d0f75d 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -23,6 +23,8 @@ static inline unsigned long arch_vmap_pte_range_map_size(= unsigned long addr, unsigned long end, u64 pfn, unsigned int max_page_shift) { + unsigned long size; + /* * If the block is at least CONT_PTE_SIZE in size, and is naturally * aligned in both virtual and physical space, then we can pte-map the @@ -40,7 +42,9 @@ static inline unsigned long arch_vmap_pte_range_map_size(= unsigned long addr, if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE)) return PAGE_SIZE; =20 - return CONT_PTE_SIZE; + size =3D min3(end - addr, 1UL << max_page_shift, PMD_SIZE >> 1); + size =3D 1UL << (fls(size) - 1); + return size; } =20 #define arch_vmap_pte_range_unmap_size arch_vmap_pte_range_unmap_size --=20 2.39.3 (Apple Git-146) From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D7401A239A for ; Wed, 8 Apr 2026 02:51:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616702; cv=none; b=fXEF8STfyPidCDqzafvoAoGQjDQLcvDARP7Bs8afzau3r1sHiBSXqWVAFSdxBlxlJqTydKGrt7OWoA3iJAOGQjlUVz6oANxh69f2msrb7OtG+p1K8VItBtwSDkj7BmVhYzo8eSGbhqq8jAE5li5zToARybXNyf44sBwssdxFLJQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616702; c=relaxed/simple; bh=vLbaLx4OjOo4lPW5G2eyteG1jD4WkqeXjsMrkMJ13sU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SKraoZz95LMgpco636zxcDPjsVK5/K23M8lVZFUro+rgC+wDAvV/dJdyBxMFT6ak0WDsujcLpSUea7cZe78COYb7Pjcyerc1q+MUVYMeyUGw5S6fwjpbdPzNuewgVesHF/iGH5k37Um/FBmD3KGATdw3IbK/wZ0W//Stq7DKc4c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JNKTa01p; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JNKTa01p" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9872CC116C6; Wed, 8 Apr 2026 02:51:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616702; bh=vLbaLx4OjOo4lPW5G2eyteG1jD4WkqeXjsMrkMJ13sU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JNKTa01piFnudi3PShsKsr4fjMQhJQweXnRAGNJInS81yIwQ0/WNgLRLHXzOtXi8T +f+GOdgecQ0+UKlxiWWECMDV6AUIMrOdtUAbRrZ7XLxpMMytszyB7FQGlte8tETF4+ 37ds6U1eurtYjfkcH5ysxGsB+MTLyUhUkScCx83nAspwbJq7IHGUtbRm6ymsgtTQ2W y5twO6kfteIv78kJHPd210N2a3N7djdUf/DP/9MR0beotXu2yYaDv3ZdIHd8CqabmW /DDW1p9LILJ1vaNB0SYFBXpPz+kxV0UbcWu61Wwq3vqyDsoHTLvM/ZVT1a/qKGGsBs p2tOBr9D8GzzQ== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 3/8] mm/vmalloc: Extend vmap_small_pages_range_noflush() to support larger page_shift sizes Date: Wed, 8 Apr 2026 10:51:10 +0800 Message-Id: <20260408025115.27368-4-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vmap_small_pages_range_noflush() provides a clean interface by taking struct page **pages and mapping them via direct PTE iteration. This avoids the page table zigzag seen when using vmap_range_noflush() for page_shift values other than PAGE_SHIFT. Extend it to support larger page_shift values, and add PMD- and contiguous-PTE mappings as well. Signed-off-by: Barry Song (Xiaomi) --- mm/vmalloc.c | 54 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 42 insertions(+), 12 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 57eae99d9909..5bf072297536 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -524,8 +524,9 @@ void vunmap_range(unsigned long addr, unsigned long end) =20 static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { + unsigned int steps =3D 1; int err =3D 0; pte_t *pte; =20 @@ -543,6 +544,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned lo= ng addr, do { struct page *page =3D pages[*nr]; =20 + steps =3D 1; if (WARN_ON(!pte_none(ptep_get(pte)))) { err =3D -EBUSY; break; @@ -556,9 +558,24 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned l= ong addr, break; } =20 +#ifdef CONFIG_HUGETLB_PAGE + if (shift !=3D PAGE_SHIFT) { + unsigned long pfn =3D page_to_pfn(page), size; + + size =3D arch_vmap_pte_range_map_size(addr, end, pfn, shift); + if (size !=3D PAGE_SIZE) { + steps =3D size >> PAGE_SHIFT; + pte_t entry =3D pfn_pte(pfn, prot); + + entry =3D arch_make_huge_pte(entry, ilog2(size), 0); + set_huge_pte_at(&init_mm, addr, pte, entry, size); + continue; + } + } +#endif + set_pte_at(&init_mm, addr, pte, mk_pte(page, prot)); - (*nr)++; - } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); + } while (pte +=3D steps, *nr +=3D steps, addr +=3D PAGE_SIZE * steps, add= r !=3D end); =20 lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; @@ -568,7 +585,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned lo= ng addr, =20 static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pmd_t *pmd; unsigned long next; @@ -578,7 +595,20 @@ static int vmap_pages_pmd_range(pud_t *pud, unsigned l= ong addr, return -ENOMEM; do { next =3D pmd_addr_end(addr, end); - if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) + + if (shift =3D=3D PMD_SHIFT) { + struct page *page =3D pages[*nr]; + phys_addr_t phys_addr =3D page_to_phys(page); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, + shift)) { + *mask |=3D PGTBL_PMD_MODIFIED; + *nr +=3D 1 << (shift - PAGE_SHIFT); + continue; + } + } + + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask, shift)) return -ENOMEM; } while (pmd++, addr =3D next, addr !=3D end); return 0; @@ -586,7 +616,7 @@ static int vmap_pages_pmd_range(pud_t *pud, unsigned lo= ng addr, =20 static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pud_t *pud; unsigned long next; @@ -596,7 +626,7 @@ static int vmap_pages_pud_range(p4d_t *p4d, unsigned lo= ng addr, return -ENOMEM; do { next =3D pud_addr_end(addr, end); - if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask, shift)) return -ENOMEM; } while (pud++, addr =3D next, addr !=3D end); return 0; @@ -604,7 +634,7 @@ static int vmap_pages_pud_range(p4d_t *p4d, unsigned lo= ng addr, =20 static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { p4d_t *p4d; unsigned long next; @@ -614,14 +644,14 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned = long addr, return -ENOMEM; do { next =3D p4d_addr_end(addr, end); - if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask, shift)) return -ENOMEM; } while (p4d++, addr =3D next, addr !=3D end); return 0; } =20 static int vmap_small_pages_range_noflush(unsigned long addr, unsigned lon= g end, - pgprot_t prot, struct page **pages) + pgprot_t prot, struct page **pages, unsigned int shift) { unsigned long start =3D addr; pgd_t *pgd; @@ -636,7 +666,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, next =3D pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |=3D PGTBL_PGD_MODIFIED; - err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask, s= hift); if (err) break; } while (pgd++, addr =3D next, addr !=3D end); @@ -665,7 +695,7 @@ int __vmap_pages_range_noflush(unsigned long addr, unsi= gned long end, =20 if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || page_shift =3D=3D PAGE_SHIFT) - return vmap_small_pages_range_noflush(addr, end, prot, pages); + return vmap_small_pages_range_noflush(addr, end, prot, pages, PAGE_SHIFT= ); =20 for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { int err; --=20 2.39.3 (Apple Git-146) From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA244202C29 for ; Wed, 8 Apr 2026 02:51:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616706; cv=none; b=WaM02p4pm3Bd7vgBgpcwcnQls41mL2tmmb6MjKjdRQ0vhJfmWa0HdsK+DwrvCPGhdFI02KQOJc9dKUQbHGHa0GMF2KTXzKM3DcFxC48JyRGBc4G7Li8Td+wbkrArqplUsCEvqtMK8rV6LzmzvnjWQUIMw6rjEkolinW8wNy2FCg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616706; c=relaxed/simple; bh=HLCqPTjcCmdnanlT8SVpYIjINZxqvkrrFi6ei5+WS7Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DP++t2wcw7x2Ezk616cksz6jI200TR+NPvQzWCm6TX9dgNjgJMt7y5eNapuyt1XyJF5J9kjhkpvrj3tpA+Ja88hLTrUZ70L1DymgUMqFHqrrt1x2Ye0KOsHZChNIGEWRspm+e/tzeX4i/9/N1QlYiHLTvYYuJuP2LkZ6KQ95XA4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=E98cltUu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="E98cltUu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D11D8C19421; Wed, 8 Apr 2026 02:51:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616705; bh=HLCqPTjcCmdnanlT8SVpYIjINZxqvkrrFi6ei5+WS7Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E98cltUu5pz4jFx/09mSwR9dZ3lbqdssVOpqPjSPh9hhfM8JVF4YU+v01kkNe5m0H M0RZ2LNiEuLFM69/fPap3dkuh2UBPmIRHtnr+CBeOiAFjrpGGv0oL3Xhr/FrzVKUT+ 91SFi6HW5MU80MqN6PC15ebGS4uBiGtfv3eSIjPApij0yY3mF0grfx6pFmmBryGNDX T4dg3i6bfttSbwFHZTbPPIXkwMYYxc67fZY9aFx5k7ksURsZp6MEHkTE6CFq65amCg aiuuBvwfHF0QXa0S1PqSec3wkI9cH1pU6AY+Nn90uFLsSYE1BRNiXXttGTcot4Fmbx dkSeaxzsotkdg== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 4/8] mm/vmalloc: Eliminate page table zigzag for huge vmalloc mappings Date: Wed, 8 Apr 2026 10:51:11 +0800 Message-Id: <20260408025115.27368-5-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For vmalloc() allocations with VM_ALLOW_HUGE_VMAP, we no longer need to iterate over pages one by one, which would otherwise lead to zigzag page table mappings. The code is now unified with the PAGE_SHIFT case by simply calling vmap_small_pages_range_noflush(). Signed-off-by: Barry Song (Xiaomi) --- mm/vmalloc.c | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5bf072297536..eba436386929 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -689,27 +689,13 @@ static int vmap_small_pages_range_noflush(unsigned lo= ng addr, unsigned long end, int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { - unsigned int i, nr =3D (end - addr) >> PAGE_SHIFT; - WARN_ON(page_shift < PAGE_SHIFT); =20 - if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || - page_shift =3D=3D PAGE_SHIFT) - return vmap_small_pages_range_noflush(addr, end, prot, pages, PAGE_SHIFT= ); - - for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { - int err; - - err =3D vmap_range_noflush(addr, addr + (1UL << page_shift), - page_to_phys(pages[i]), prot, - page_shift); - if (err) - return err; + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC)) + page_shift =3D PAGE_SHIFT; =20 - addr +=3D 1UL << page_shift; - } - - return 0; + return vmap_small_pages_range_noflush(addr, end, prot, pages, + min(page_shift, PMD_SHIFT)); } =20 int vmap_pages_range_noflush(unsigned long addr, unsigned long end, --=20 2.39.3 (Apple Git-146) From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D54D4202C29 for ; Wed, 8 Apr 2026 02:51:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616708; cv=none; b=QH6vXENKo40ZjYQ+f6VkmBowITIkZyD8o06qlIUnO0kWyLgTJwBeQqpXqet+L7U0drEQgAOhiierpuZPtWiZKCwSU7JqyPo2yTxsjuW9kRNvgRGsf0DvwMSX6j68Di4IpQ7Vpq8MG+nNuP1B9yFvXDQIT+OQrkC/d5GpGsKaE0M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616708; c=relaxed/simple; bh=ADjBqE0C1KfIyVTZP4SAf0I43MDC/OSOfU23K3IWy+c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=ckChqRDyp3DozDkx5UjsEjA0iDf/hccqvAXJ/tt+QUT74Yo07ykRD3FiyYNnUu+h8+Fpjt+vtmZYhJfPq0yxS1w9WqAGhhg823gMOAA/G/fjHKuwO/KlhOC5NjtwxI1mnToBKttq4UGqSHXLvu8E03LBm81jvf1O5N8izp8w3DI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j51XMXfv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j51XMXfv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07F78C2BC9E; Wed, 8 Apr 2026 02:51:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616708; bh=ADjBqE0C1KfIyVTZP4SAf0I43MDC/OSOfU23K3IWy+c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=j51XMXfvi+Uxqq+1/UvQP3n+KERV+0p0XtWw/Uhmkna6BimHQotLAW11I6s5QRSr6 0crdor5UZv60wH48QovXiIHoA36Ujapw2ywR0srgEx4fVQ7Fqbt6OmZxnp6yUB6G6k yyJoygwMMFIBaxuWQf3aKwv45lImLjGwWMafowo0YFHZ2LA3rHAM7XvizQqptwU20H eIFEaq9hRfJZgAwFlcEzyVGomeRJMEllO2maK95kgCj7wAE8eD59Jw21DAkW1vTxgJ CNFnpjcC2b94Zz11qOIM2c7yZz83pxIpgYbRw45w3/fAQZV/2ksIOpJPQ505DPR8tg aTa0X/Be3f8cA== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 5/8] mm/vmalloc: map contiguous pages in batches for vmap() if possible Date: Wed, 8 Apr 2026 10:51:12 +0800 Message-Id: <20260408025115.27368-6-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable In many cases, the pages passed to vmap() may include high-order pages allocated with __GFP_COMP flags. For example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently, vmap() iterates over every page individually=E2=80=94even pages inside a high-order block are handled one by one. This patch detects high-order pages and maps them as a single contiguous block whenever possible. An alternative would be to implement a new API, vmap_sg(), but that change seems to be large in scope. Signed-off-by: Barry Song (Xiaomi) --- mm/vmalloc.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index eba436386929..e8dbfada42bc 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3529,6 +3529,53 @@ void vunmap(const void *addr) } EXPORT_SYMBOL(vunmap); =20 +static inline int get_vmap_batch_order(struct page **pages, + unsigned int max_steps, unsigned int idx) +{ + unsigned int nr_pages; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP) || + ioremap_max_page_shift =3D=3D PAGE_SHIFT) + return 0; + + nr_pages =3D compound_nr(pages[idx]); + if (nr_pages =3D=3D 1 || max_steps < nr_pages) + return 0; + + if (num_pages_contiguous(&pages[idx], nr_pages) =3D=3D nr_pages) + return compound_order(pages[idx]); + return 0; +} + +static int vmap_contig_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) +{ + unsigned int count =3D (end - addr) >> PAGE_SHIFT; + int err; + + err =3D kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + PAGE_SHIFT, GFP_KERNEL); + if (err) + goto out; + + for (unsigned int i =3D 0; i < count; ) { + unsigned int shift =3D PAGE_SHIFT + + get_vmap_batch_order(pages, count - i, i); + + err =3D vmap_range_noflush(addr, addr + (1UL << shift), + page_to_phys(pages[i]), prot, shift); + if (err) + goto out; + + addr +=3D 1UL << shift; + i +=3D 1U << (shift - PAGE_SHIFT); + } + +out: + flush_cache_vmap(addr, end); + return err; +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3572,8 +3619,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; =20 addr =3D (unsigned long)area->addr; - if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { + if (vmap_contig_pages_range(addr, addr + size, pgprot_nx(prot), + pages) < 0) { vunmap(area->addr); return NULL; } --=20 2.39.3 (Apple Git-146) From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 502E61A239A for ; Wed, 8 Apr 2026 02:51:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616712; cv=none; b=hX7f34DRfm9OTXOWvFKykGfUgw2KKgDK/Sg6JKgiRfxdn016Q7xUGpw8UAhcuUeFauvntu29j3HlPe/9WAFZZK+9Hx1eFIkJtoxZiFBIIFSale0HCHjTIwnO8mD3CoFEczbbblw2rmDbViMGb5DGljqjmMYfdLw0PHsf83McElM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616712; c=relaxed/simple; bh=PKtItPJH+nSfbPKiSnTVSIKouqPIOVzmyf2uKcpw7Ds=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AglpuwubSruNozN2fMWqSTvhtWg4F7IgxbF1TicEfdmASFRKh9a4m8DoxPz4pK1bVDfGg74WiYpDYUr7sfage5w81QVB/tnrRBxRXBxhXpVAAdqx9bPshC+euNIkOOMEygTU9QFoFZ2MJoakgI19ehdb4+W1CZEQhuglJ6Q2zNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=K+u/rD5a; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="K+u/rD5a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 31F42C19424; Wed, 8 Apr 2026 02:51:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616711; bh=PKtItPJH+nSfbPKiSnTVSIKouqPIOVzmyf2uKcpw7Ds=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K+u/rD5azpr8es305v/KMUlewCknD5c7RdCe2Nv1fnif5jdekzR5z72uB0XVvbhf8 559XCvpf2ujN+0v055UU2ailQuyIx4O6fsijGq5EdqomDTipii0OPoSxYhGpq83F5W +rOF9PK/XyLmm5YdJdA/KzBtgDWtqgJajiWkLyeFGrWvjf/4uRD9T4CLtVAdP/j1HP 3mbPUWakkrUQrYOcK+XwBqka08Q9f0V3Wa3Y6fCFXD3rvsfAfFgLrhP8ONEC8QPSkO 6tC4+Bad8HvLf3ZLhkkMBega2S2viD1f4xf6bzgx5hAabqvbCniwxYN+CDTpaHsD4V IuL2CUt5ClpWA== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 6/8] mm/vmalloc: align vm_area so vmap() can batch mappings Date: Wed, 8 Apr 2026 10:51:13 +0800 Message-Id: <20260408025115.27368-7-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Try to align the vmap virtual address to PMD_SHIFT or a larger PTE mapping size hinted by the architecture, so contiguous pages can be batch-mapped when setting PMD or PTE entries. Signed-off-by: Barry Song (Xiaomi) --- mm/vmalloc.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e8dbfada42bc..6643ec0288cd 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3576,6 +3576,35 @@ static int vmap_contig_pages_range(unsigned long add= r, unsigned long end, return err; } =20 +static struct vm_struct *get_aligned_vm_area(unsigned long size, unsigned = long flags) +{ + unsigned int shift =3D (size >=3D PMD_SIZE) ? PMD_SHIFT : + arch_vmap_pte_supported_shift(size); + struct vm_struct *vm_area =3D NULL; + + /* + * Try to allocate an aligned vm_area so contiguous pages can be + * mapped in batches. + */ + while (1) { + unsigned long align =3D 1UL << shift; + + vm_area =3D __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + NUMA_NO_NODE, GFP_KERNEL, + __builtin_return_address(0)); + if (vm_area || shift <=3D PAGE_SHIFT) + goto out; + if (shift =3D=3D PMD_SHIFT) + shift =3D arch_vmap_pte_supported_shift(size); + else if (shift > PAGE_SHIFT) + shift =3D PAGE_SHIFT; + } + +out: + return vm_area; +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3614,7 +3643,7 @@ void *vmap(struct page **pages, unsigned int count, return NULL; =20 size =3D (unsigned long)count << PAGE_SHIFT; - area =3D get_vm_area_caller(size, flags, __builtin_return_address(0)); + area =3D get_aligned_vm_area(size, flags); if (!area) return NULL; =20 --=20 2.39.3 (Apple Git-146) From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74A321A239A for ; Wed, 8 Apr 2026 02:51:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616715; cv=none; b=Xb1Mt5N2jMNUG+b6wY2xZt+w+889K+v/YpklKhgSBGfurU6fM3qJ8pCOFInxYD6Hty5+5BOoOlFlhqc2VGbIj0oDy4VA1OC4v+zyhS25mRCuC8z6wyP65ijpyWaLykvsKzPfcf/VC853dZyVbmILbdCwER7sqrh4EVPNxrIqKTI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616715; c=relaxed/simple; bh=zaZ8iI84M3gZDm1hFnDFMcUBYZoPlyljQ9RYRlcP8K4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hBfgasQfFZwzcqi9rVe61atAvwyc13Zk4yvApFoot3qw1pGd5ERMU29hhvvsuKVRlHAVKgZyNkMuqbXVrQiakCpNXy+31neFtJOFI+1MWZsq1PJ5bK8iP82K/Xp7a8LyupHv2K9THkNwqgd/JdL5x8F5m/4iajBjIlweuR40X+M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aKK4jkTJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aKK4jkTJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5CE7EC19421; Wed, 8 Apr 2026 02:51:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616715; bh=zaZ8iI84M3gZDm1hFnDFMcUBYZoPlyljQ9RYRlcP8K4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aKK4jkTJ5pywEdQAoHMXFLD7R7Mhv5Fnty4Vg5GxCHuMhVafbgKxAzllimIrmfxVz k1L/3VDHClK8mu07LOtpbPbjoWCRs94wrnmP0v3vjSgBSk4dKGJxmq6GZ+4RiXf5Dv ezwuHP0DNCVYdCLaWPBR6CI5rzkzA4fXL0f4ql25HHBqmX3Bfk+9nCWJzmRnuaoFET I1fRFIrfVxkyGP5oWkJc0xIXD2ijRTNoV7RyahVXzArx7n+PTOrf7sKnCQxgOUf0dW 1TEHNCYbfGT0eEbzT7ZkcdMF7XDAbWYwl/ASDIGj1w56XtQsqOVKIMnnNRCUmoXLHN A21hs6OW851Rw== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 7/8] mm/vmalloc: Coalesce same page_shift mappings in vmap to avoid pgtable zigzag Date: Wed, 8 Apr 2026 10:51:14 +0800 Message-Id: <20260408025115.27368-8-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For vmap(), detect pages with the same page_shift and map them in batches, avoiding the pgtable zigzag caused by per-page mapping. Signed-off-by: Barry Song (Xiaomi) --- mm/vmalloc.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 6643ec0288cd..3c3b7217693a 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3551,6 +3551,8 @@ static int vmap_contig_pages_range(unsigned long addr= , unsigned long end, pgprot_t prot, struct page **pages) { unsigned int count =3D (end - addr) >> PAGE_SHIFT; + unsigned int prev_shift =3D 0, idx =3D 0; + unsigned long map_addr =3D addr; int err; =20 err =3D kmsan_vmap_pages_range_noflush(addr, end, prot, pages, @@ -3562,15 +3564,29 @@ static int vmap_contig_pages_range(unsigned long ad= dr, unsigned long end, unsigned int shift =3D PAGE_SHIFT + get_vmap_batch_order(pages, count - i, i); =20 - err =3D vmap_range_noflush(addr, addr + (1UL << shift), - page_to_phys(pages[i]), prot, shift); - if (err) - goto out; + if (!i) + prev_shift =3D shift; + + if (shift !=3D prev_shift) { + err =3D vmap_small_pages_range_noflush(map_addr, addr, + prot, pages + idx, + min(prev_shift, PMD_SHIFT)); + if (err) + goto out; + prev_shift =3D shift; + map_addr =3D addr; + idx =3D i; + } =20 addr +=3D 1UL << shift; i +=3D 1U << (shift - PAGE_SHIFT); } =20 + /* Remaining */ + if (map_addr < end) + err =3D vmap_small_pages_range_noflush(map_addr, end, + prot, pages + idx, min(prev_shift, PMD_SHIFT)); + out: flush_cache_vmap(addr, end); return err; --=20 2.39.3 (Apple Git-146) From nobody Mon Jun 15 03:52:26 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91847202C29 for ; Wed, 8 Apr 2026 02:51:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616718; cv=none; b=TaFJh9s9CdSeomArkFHf5GXeUxCMmzWjFUthGYxmbxLdAhJqxAC4Gkp2E5OrYB+7XGIMy2C9qk2xAwX1Npr4aUCQJEA5kMzFG49HeHA/1qTVrtL9UIljFLBjkQjCpNYfu718Zi+LykDU6ckyVtjf4jb9jPsnUZer8ti2I/R44Mk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775616718; c=relaxed/simple; bh=4BDs5dW9zJtaGW/eXyIzqwPGAqrmR/Pere8PJG68bKA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=Cb1ctmHHpC0HaWxItI1XOFhoDtwY9MCl61NfhOsdfIEhA9oPrv3C+XM7tTtwKp59C8JIXsk7Z5d330R8UBevotYBmdcx0O4dExjt+G4qdFm2xTztE90YaJi673C/X7V+wTgSIuTfCifqoKowOvBZ545yb4wh7XLSqEdZ328uu94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MnttrxLf; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MnttrxLf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86EBAC116C6; Wed, 8 Apr 2026 02:51:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775616718; bh=4BDs5dW9zJtaGW/eXyIzqwPGAqrmR/Pere8PJG68bKA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MnttrxLfgWIjceDLBrD7T6mxsHluWlpQuAnVSfvkfssMuGim5D3nMqUWysoDzrF8q Vg/d/PYq+g2PiJwuyJV53u9fAYWH/e9GDDETxM/1qujw7fSUU23VQ6Wt5AIjgy3h3J yA4pB9Y6m/4MjHSU2+Ord5ZwlDmOp41LGv5wDQVrRkX+48iVntSbaEiNZPpV95rzM0 o7dixtMLL6Sb4hxcCk7KqWMKwZ2KVNgYnuzXISDbFUU7y3Q4f6QgzSKahf/5r+Kenq XLy0p+dk34T72eWKhEMyJhjy1poxLFhIex1zBjGY2rYzoSZkAW6IIsHC/aG1QuLpjL F2hb+JBvyQPQQ== From: "Barry Song (Xiaomi)" To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, ryan.roberts@arm.com, ajd@linux.ibm.com, rppt@kernel.org, david@kernel.org, Xueyuan.chen21@gmail.com, "Barry Song (Xiaomi)" Subject: [RFC PATCH 8/8] mm/vmalloc: Stop scanning for compound pages after encountering small pages in vmap Date: Wed, 8 Apr 2026 10:51:15 +0800 Message-Id: <20260408025115.27368-9-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260408025115.27368-1-baohua@kernel.org> References: <20260408025115.27368-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Users typically allocate memory in descending orders, e.g. 8 =E2=86=92 4 =E2=86=92 0. Once an order-0 page is encountered, subsequent pages are likely to also be order-0, so we stop scanning for compound pages at that point. Signed-off-by: Barry Song (Xiaomi) --- mm/vmalloc.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 3c3b7217693a..242f4bc1379c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3577,6 +3577,12 @@ static int vmap_contig_pages_range(unsigned long add= r, unsigned long end, map_addr =3D addr; idx =3D i; } + /* + * Once small pages are encountered, the remaining pages + * are likely small as well + */ + if (shift =3D=3D PAGE_SHIFT) + break; =20 addr +=3D 1UL << shift; i +=3D 1U << (shift - PAGE_SHIFT); --=20 2.39.3 (Apple Git-146)