From nobody Fri Jun 19 03:49:02 2026 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11E803B4EB2 for ; Thu, 18 Jun 2026 08:47:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772461; cv=none; b=QD+BrIAIDYpsJ/sVUyNUcKngcTlioIVKJf8YdzH+62HS7q9SoF724OUmGTkqA7Er1UDfORhge4/JHFRtCdymi8J4M8trDO7UkgNsSka4Czc0hi2FnHDJIY9mppdrI1KykSm5zVzyHIP0S3hk9e31b4MwULJgCL59tuRDilwt2KM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772461; c=relaxed/simple; bh=UShEaVTvh7LAz3TMlHc7gmBLkZbgjZTNeCAtBRyQVMc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jJCb7sk3oMZBp6kkqAQO2YrjFsazEwcaIG8AwX3HwPst+/h6aI5Jn9kUK4ZlfxXVqWD2pl61isyJQbwNPH5tsZyRdRdqmDoJazj54VW00nd/tmwtpNebMUpsc+nHkfuVcCv64kEIjNT6HFcvh4ojDVUjWnlPW3vCJ8O5ZIHXwFY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YZn7ck5o; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YZn7ck5o" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-36d630c0e35so613383a91.3 for ; Thu, 18 Jun 2026 01:47:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772459; x=1782377259; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u2WEu3pL2331FUekzYMzPgYnW/t+6wHKRR7Q73nVowU=; b=YZn7ck5oQFP8T4F3F62deMq8fPObzp63I+VBLbEs8FAIIUXqwXLptQvvlN4qa4slXn zVv3ass5F8/wcbg+V02lzrBo6ljQItFU0zP6H6IVAtDHJqz8Boh6D8ciLZSgJVBtY81P ZP56YyAElFYqdGoH8V8/7nlgVeFKxh241RHjfgE8sF3Q5ILG423azyha87tf35aCgaE+ XWgHxfuDztPExFAijd6HrjEPxPiVU3YY48LXRziCzxnPn/40BgRkLT6ViNxlsgaNleM4 sYvpUJNdXnOFfu6P09pc4CaikKvy7UCr9ghpq4Q2MXGrZ74liIuKMBKMd8IuJ3FodNT9 XDQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772459; x=1782377259; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=u2WEu3pL2331FUekzYMzPgYnW/t+6wHKRR7Q73nVowU=; b=l6Tv1f5GvIOJmW7DWMImpqjju2l7jBZqSRI0WTRFPstKx9hQrNuAbLJ1r8X1JGpeVQ 2gt5pN5mS0ZhAPmcfJ/1A/cJwKzRDp814pLg7U26aQD+sshXuleM2NLqAUB/0d4rKgfY 01yVQjvSCC+sJymwbu6d5rJigwbaA/sgJd7Dq00Ypsm/QH1q9UpLtPrV9hdzgvHYkTtr /sg0qezdHRRQLc5e8Qooys4qybSFO5rNusSS6rM2BHclm0mmSweoV7AdGPmo9My3htB9 Yu5UKqmrIt90l6ZhZxd5Li9Kzi1IPXvJ/Db3LXk0nqoeqWw/TPSBXu/r4e6v4S9UXEYX Do7Q== X-Forwarded-Encrypted: i=1; AFNElJ9MShRv0qrbui/RAekqzQv17NiXkVVlORTwSTOjta5jU4Hs+B69ool6ipadYjt2e7u7c3eWnODVpKy/FHI=@vger.kernel.org X-Gm-Message-State: AOJu0Yyf0wRdatoWDZj9JHmMHvIIBqS8yEglRLNXDlVZAiFaV+jP8SPx 0u5mk+As5N6pd9qLxK5oeIpOQKX9wAMJeET+/mkR+huh3VCiW2a6yJKt X-Gm-Gg: AfdE7cn3p5IBLIYvKIpzLaT1TntDGahSb16LKfucN0OgMk9VG4bIbegar8GMc9zC9dy ZBHx8ypprsqJp2KFQXBH18eoEACsl12o8+SjKKSgks1RzMyysZHMegkJf8aPQ7nqP9ZzsN6geii a7Uzw8BJgWjY4glsQkU7xO0eBy8r9T2kbKNzeK8byzVg1mI1Ly+tXmZiw8I1rH25lX9xlvAERaU OubbnqYjUeHZy/fvzdltYyTteKW+6fg4cYqvCbPrGT3s95Sd32O2Z/cZX1oHwzG1zYcZZJFj79i mMOwjKqZCX0Tva9Na/fvQJfIQXOP4ggHQDv0mVlJDCZDKVhohd8mS1jjW5hcF7LdNjB0Q7W4G+/ kgSZspK3GbJq50vQ9W1cdgMEk2XVt+83UHuu4e0ylAnXwvlf5JTsJ4Cb5vgeCyzWKwBTjGtqYNl lGD321FuwwmGrBggnmaAeVrXYHiJUK6jBx+Q4= X-Received: by 2002:a17:90b:2812:b0:368:341a:a925 with SMTP id 98e67ed59e1d1-37ce46d8c8bmr2750204a91.23.1781772459491; Thu, 18 Jun 2026 01:47:39 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.47.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:47:38 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 1/6] arm64/hugetlb: Extend batching of multiple CONT_PTE in a single PTE setup Date: Thu, 18 Jun 2026 16:47:21 +0800 Message-Id: <20260618084726.1070022-2-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260618084726.1070022-1-jiangwen6@xiaomi.com> References: <20260618084726.1070022-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" For sizes aligned to CONT_PTE_SIZE and smaller than PMD_SIZE, we can handle CONT_PTE_SIZE groups together. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen --- arch/arm64/mm/hugetlbpage.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index a42c05cf56408..c4d8b226126cb 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -110,6 +110,12 @@ static inline int num_contig_ptes(unsigned long size, = size_t *pgsize) contig_ptes =3D CONT_PTES; break; default: + if (size > 0 && size < PMD_SIZE && + IS_ALIGNED(size, CONT_PTE_SIZE)) { + contig_ptes =3D size >> PAGE_SHIFT; + *pgsize =3D PAGE_SIZE; + break; + } WARN_ON(!__hugetlb_valid_size(size)); } =20 @@ -359,6 +365,10 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shi= ft, vm_flags_t flags) case CONT_PTE_SIZE: return pte_mkcont(entry); default: + if (pagesize > 0 && pagesize < PMD_SIZE && + IS_ALIGNED(pagesize, CONT_PTE_SIZE)) + return pte_mkcont(entry); + break; } pr_warn("%s: unrecognized huge page size 0x%lx\n", --=20 2.34.1 From nobody Fri Jun 19 03:49:02 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A4A93B4EA9 for ; Thu, 18 Jun 2026 08:47:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772466; cv=none; b=uS9Brk05xYEedfJU9cL5iWEEbdZsyJcBiyoNJBgOq5SsmTPRXi78CpVwf9gJv0OIpQEPUbURbhrgPZmeiEBCH1gYhZE/yPi6ZBrzMIhEyI2D/UH1uZvEttmdqjbKvdHdzUxkMAuDT9W5QgKpzRrDe10+SecKa6Y/djbdTqjAXek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772466; c=relaxed/simple; bh=GoxWC53WLTl4+q/7rFOagw4b55aIyls6LidDCLsRFN0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DylfVrcXFxR3AddN0j2OxpmP5l+g0w6ukWI1JlxW0wX0lHutgFjj0F1T19q6gZ3ZGLW7ojpx9QOdQTU50liF81FQgNXbfUc5YqK7pE4BwlYS4AMU9as3fX2SgqtaiwPLXoaWkLQ303Gy7HTO9IJTUzjq8MmFX8JlDz0s+e4nyCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ayN1/4Ze; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ayN1/4Ze" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-37ce68a54f8so251583a91.0 for ; Thu, 18 Jun 2026 01:47:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772464; x=1782377264; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8bbcoKjOxWp5rK+koIv3oGOfWy9Ig7vlkgEF+yecgH8=; b=ayN1/4Ze43GL3rzhhVD9bXhzmxlHKsnhlrJVNVtyeggT/KYmi5VIOJMjf4XEIcXmEr xWeWQQBcJ/p75mWOHUjivmdOBK7Ps1mIwtXDYEJkMXEmYXvF149e4F0JvWxOw4ximtB5 Q5PkBkhIu2ANfR6jqbvVnlXgS9UGu5sGuOvXAgMhe4R6fZaZs1owcr3fuG7muVzTTWLq eDru0PYwCEBeID2jjx2kbKdZXbe6DeuAVC5Tp38Gm9bmTPO2e3lRacij5B1cmxve1wtY pD8hCTqZ81SDyAoiAe3RKYxOkiuDsWxoeJIUHTGsyEXoxjrX456ShAvq5AXlfww1IBLF YEWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772464; x=1782377264; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8bbcoKjOxWp5rK+koIv3oGOfWy9Ig7vlkgEF+yecgH8=; b=aE9Eg8jgS1L7NQQwlT62ztdHQx7OwDdm4ErvZaT9ISFgO9VHPnGrWRyOGMqT1ftC44 dyaqOhU+x053Y3pHMKVS+NTFSmSzdQYypilWvoaGXrccKqynqJ9GAXFtCqlhSTnCYZSW RdFPjFiZSKRcBUmn19+jR7h4FMf9l/GOCZYTMsLHIYjGbJzhmb6XFoCkUSgWIZzhxmpJ KBgsXL/nPXjAll/Sb7ZuGkBheOOdQtdtePe3c3JA++IGrnZMOLegAGLy0JSyqHFDLqdF RphtJA/QZWZolHj4XvL5oSQfQ5RR2IiFtyO152FcdiiIBB6OUcmk5MK94zR+zWMdrHne FXBg== X-Forwarded-Encrypted: i=1; AFNElJ9KlsR3Htpm50xAfN/X7mgI6wTEvwjhxhx1rbG/+IO9NVzgYaOp6NSOw0hoR7MMVzizwpCV/ZkFB0Dtq0c=@vger.kernel.org X-Gm-Message-State: AOJu0YwNVYSkP1QiRpUN3/wKtqqaN47MpqRiFl2YsMHZSGF1TSRjLGFH bFxeAsPM4MFAb9o3C9G+TtIZ6X5bSBRZz1G6vIr/cYjS8BQ5qXSy46bC X-Gm-Gg: AfdE7clWNBe37cXNZRQ/LDug0ueS2HlRj/hhgBx3RTYgmrUvJH8Uo6EEQKdaSGaAHbS /xayXsL6R7aL56yt/EBxTPQdHJUZXqVf80i25Pc8krj4zlLsCvs5F3/QbR3gJ6B79MWn5kKDw4w 6xw+WOCsDf3SRt2z3yGF4h4L5aSisM/T2MNZptbOEqaME/fIQcRuJylWEMnKzKm63vK7fSjw93/ N4niBlNik4yc7ex1Hqq5WfWpsG4rPp0XvY6d/Oue2gpQk7AErKMWEw8TYmyuS/MHREj/8lCxA7L omfB2VSh0beKMDmQRAT8peeejTcA+CDRlI+9FcuPPid9rNuMtniRa9QNmIciwSII70KNPJvax0l tkkd40yd6C/yqkVWdaKCpLWSe37XXS/RGFBt2U3H5E3dwIC6c1ojtoxYYdDOO1EX+ScvYE9/EKV KAecL3vequr2LJL9Q+NHV0y2WShrgHMWvIROI= X-Received: by 2002:a17:90b:2b47:b0:366:132:fda6 with SMTP id 98e67ed59e1d1-37c9dbbdb5cmr7450798a91.11.1781772464450; Thu, 18 Jun 2026 01:47:44 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.47.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:47:44 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 2/6] arm64/vmalloc: Allow arch_vmap_pte_range_map_size to batch multiple CONT_PTE Date: Thu, 18 Jun 2026 16:47:22 +0800 Message-Id: <20260618084726.1070022-3-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260618084726.1070022-1-jiangwen6@xiaomi.com> References: <20260618084726.1070022-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" Allow arch_vmap_pte_range_map_size to batch across multiple CONT_PTE blocks, reducing both PTE setup and TLB flush iterations. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen --- arch/arm64/include/asm/vmalloc.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmal= loc.h index 4ec1acd3c1b34..787fd17b48e2c 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -23,6 +23,8 @@ static inline unsigned long arch_vmap_pte_range_map_size(= unsigned long addr, unsigned long end, u64 pfn, unsigned int max_page_shift) { + unsigned long size; + /* * If the block is at least CONT_PTE_SIZE in size, and is naturally * aligned in both virtual and physical space, then we can pte-map the @@ -40,7 +42,9 @@ static inline unsigned long arch_vmap_pte_range_map_size(= unsigned long addr, if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE)) return PAGE_SIZE; =20 - return CONT_PTE_SIZE; + size =3D min3(end - addr, 1UL << max_page_shift, PMD_SIZE >> 1); + size =3D 1UL << __fls(size); + return size; } =20 #define arch_vmap_pte_range_unmap_size arch_vmap_pte_range_unmap_size --=20 2.34.1 From nobody Fri Jun 19 03:49:02 2026 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 381943112C0 for ; Thu, 18 Jun 2026 08:47:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772471; cv=none; b=PEgKFWOnBr/QUIgANTp7dL9DYyZvxuJbhtr0vmlKi77GR8MC+XTHsk5Srh4JtfWUBy+NPWnunRtXc57l8Ki4OcwEMdVR9vZaIZyAK53rUwaI720mrE2DofSu+20vtFi2sBRg4ZfR4HXMEyqzQ9Tdpe0kTZWuSIwBEKFxMHtUGH0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772471; c=relaxed/simple; bh=Y5QumQBQTIiktJS2KTLZyIru55QuFXJ1JnNYsubgOnU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Zp74douGw6mCnaDFeDTxam8qaFNprKr1Ke+7AFdEDIIHh/axAhToN4RZsTEYRlbptN5C8tyPkrgXRc1O0oG+jfm621OIipqfS4ACA+PQguAHfMnBKaj217tGkRNFvGxQ7qE4OPDeTfPJ+HAR6p/1GJzvqSwiQRjZOeYYGJiLIdA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jJa+9Tzf; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jJa+9Tzf" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-37cb36ca63bso339155a91.0 for ; Thu, 18 Jun 2026 01:47:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772470; x=1782377270; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kLRuKvFhInROLg4Q8kC57ftYeHhuWOIlgmE1PN84SUM=; b=jJa+9TzfNN9bXMxsPiAN7pQMBy/PbqUqlcPHeAEoAd1jYe5BvRWpKw3Nn/Puu5lGdX u/IVoV65gOcj4Sl0vpRbklQH8SEgvf3104O5XRtW9wptB9nOKpT5lwAW7aY7QmNHNX9r Vw/ievjFIeQybLltgtxAoGSoQCdmyJly26Lv24u5kCxFdDgqQq0DQids8mTWA8habgoS 9iqws5BgNICjZHZSjwJ27BHino6MUxodjNKUe/IvDta4EvErODweGHjM3EZIbAASKS7w HyzinjcgItjCHwqGwMCLCUGMa+f01k1ZSh0/fqGwF9qmXXXO1TlrXLWTL7b1fZKnQ5JF pjPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772470; x=1782377270; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=kLRuKvFhInROLg4Q8kC57ftYeHhuWOIlgmE1PN84SUM=; b=fj8NqkMXZc6rYZlSzUIK2uxAceays4YgDOxkwahpJIDiEU6ltUvZXJ61CnKs3k42L4 V3lUl3ANZsfGuCTg2CeQu1oMJI7rPA7Ibssx3lCwTtkauK2wJRD34vEuf+RdCrPzooTO tCuDKajKKDUYP3EJ+uSb351emy5llL7U+65nr+OasrFzu9QsnjEnlMgBAvf7+u9atqnJ vs5ZTmU22KARFMCagzZMn4CcsZ4UCUAJBpbtOt6DRMPLDY8J2DpQpCi6+6tVKTG6Qg6w IfXb0yib5uCb6KePGz4V2dritZh/z0Iw1nu+SzHneFSqrO37yt9iN6MAYPIXCtncYV+a WPnA== X-Forwarded-Encrypted: i=1; AFNElJ/NWL7yzQM/+YhlssZj0sv76BRDbA/92/rGtlPD47Yt8ik/Yi9seyWFll0lhoWG6z9YO8S0L+rzXamkqBw=@vger.kernel.org X-Gm-Message-State: AOJu0YykdqWp0fRvZWg+7EPaAIl5rLZJkgIxMy4e2R5Dt2yKMpKvC7tQ izV9bHfx3fzVfbz9UeIU4T240/ETzfSBb4duEM/Nhwl3Mi29pP+/FWiy X-Gm-Gg: AfdE7cmghO9cFxlDJAZzKK180On45cQirDKCkciMMmkUZSFc7n4WN/4jeyRmm5MQKi+ 1LlJvgGJtJ6nPpj2wicXiVHmZ8uHGARYKHxxiB3Dntx/k4T9jIJqHFo8kCOTOIcQIoBCVvt0Wkn fsbejWHHiY9xC/2Ws1lDCgb3+FH6KJqY22Q6lfLGgvjBrX3wVqenXZzizkkbDWkOicZcWZm8oJ+ nyvcQF223SCo6r+CVReBuM9zd/QSG8UXbCqQ1YNm1ip6Ux6huywJqbAE+IL7Ewjpbln1zA0scJT Dke1FMnG4IQrX7ABVdcbhPnbJi6USjOsIW72+j8X3s3ztvHfaf7DUg7Ef7116F+9PqLvy94jLzy Pj0QqFvvf/iv/eORAJPXXpH2u5fgwGtZycMD4AxxPm3WxJ8hbBxlmU1PuQN4sZFoSXbQjOGd1po 74FRuBVjx9lEMaJ57fwsgYRtN12cYYlGTaEaI= X-Received: by 2002:a17:90b:1344:b0:37c:607b:2cd9 with SMTP id 98e67ed59e1d1-37ce43c0ed0mr2690740a91.0.1781772469588; Thu, 18 Jun 2026 01:47:49 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.47.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:47:49 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 3/6] mm/vmalloc: Extract vmap_set_ptes() to consolidate PTE mapping logic Date: Thu, 18 Jun 2026 16:47:23 +0800 Message-Id: <20260618084726.1070022-4-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260618084726.1070022-1-jiangwen6@xiaomi.com> References: <20260618084726.1070022-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the common PTE mapping logic from vmap_pte_range() into a shared helper vmap_set_ptes(). This handles both CONT_PTE and regular PTE mappings in a single function, preparing for the next patch which will extend vmap_pages_pte_range() to also use this helper. The #ifdef CONFIG_HUGETLB_PAGE guard is moved inside vmap_set_ptes(), so callers no longer need to handle the conditional compilation. No functional change. Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen --- mm/vmalloc.c | 44 +++++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 2c2f74a07f396..6660f240d27c9 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -91,6 +91,35 @@ struct vfree_deferred { static DEFINE_PER_CPU(struct vfree_deferred, vfree_deferred); =20 /*** Page table manipulation functions ***/ + +/* + * Set PTE mappings for the given PFN. Try CONT_PTE mappings first when + * supported, otherwise fall back to PAGE_SIZE mappings. + * + * Return: mapping size. + */ +static __always_inline unsigned long vmap_set_ptes(pte_t *pte, + unsigned long addr, unsigned long end, u64 pfn, + pgprot_t prot, unsigned int max_page_shift) +{ +#ifdef CONFIG_HUGETLB_PAGE + if (max_page_shift > PAGE_SHIFT) { + unsigned long size; + + size =3D arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); + if (size !=3D PAGE_SIZE) { + pte_t entry =3D pfn_pte(pfn, prot); + + entry =3D arch_make_huge_pte(entry, ilog2(size), 0); + set_huge_pte_at(&init_mm, addr, pte, entry, size); + return size; + } + } +#endif + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + return PAGE_SIZE; +} + static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long en= d, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, pgtbl_mod_mask *mask) @@ -119,19 +148,8 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long ad= dr, unsigned long end, BUG(); } =20 -#ifdef CONFIG_HUGETLB_PAGE - size =3D arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); - if (size !=3D PAGE_SIZE) { - pte_t entry =3D pfn_pte(pfn, prot); - - entry =3D arch_make_huge_pte(entry, ilog2(size), 0); - set_huge_pte_at(&init_mm, addr, pte, entry, size); - pfn +=3D PFN_DOWN(size); - continue; - } -#endif - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; + size =3D vmap_set_ptes(pte, addr, end, pfn, prot, max_page_shift); + pfn +=3D PFN_DOWN(size); } while (pte +=3D PFN_DOWN(size), addr +=3D size, addr !=3D end); =20 lazy_mmu_mode_disable(); --=20 2.34.1 From nobody Fri Jun 19 03:49:02 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B92E3112C0 for ; Thu, 18 Jun 2026 08:47:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772476; cv=none; b=Sw9Fe93FSitGNqm2OsUPyHp2RbZaD108EGa6vJhHxu5KwMOcx+uIXIDQW9V340RPC8SyGKPCqR9S4nYnVcfsewQwQaP7BecIKorRT63aMthjS2EvX/+m1WZG0LCvFEXU8+zMzdQQ1zvsxpLe7FREDreIS5BI84tAU7aFc+aRnZ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772476; c=relaxed/simple; bh=qmP3KcqXBJinMcHqXsdyBjpqx/kADt66FJXrU5DCy9M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gfQw0CtP8Uwf4OQ6QJRBhY9yTZ8hv/o7+Wox/xeyYXa8GhNeyMbzlLhJ+t1NV17s+OSu747GC6w/eXusPDhkVcqEE1AgEa2osPXbb4vmNSHWLOYYUoLdo4csxtCNVTwUclH7b1tVUYhG32Y5LjDjqLawYlVdCV2/ifX589Qg1vc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Yh7Qh0Ly; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Yh7Qh0Ly" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-36da8439078so544845a91.2 for ; Thu, 18 Jun 2026 01:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772475; x=1782377275; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OfwAKcFuVrniumYvLLu8JFz9uYaRHWvyegwwsVszSZM=; b=Yh7Qh0LyyDLCAgcwEQ1IJoCZQG7LGBVGlhPCIivBrvmrABfnOhg4pfaraNs2a/iacZ rpzVAd3EXKuP+9u2UpgeOtXw/3eJkqOh4NJejlHPOz3mBcmz5lUxuAH7vybtbwbWehAq SKx/S121CJR7dvmt04EpmRa3c6kkF/UApM6glDuXIJegRJcSJneXz+U7Xx6ogqtVIZq0 +C/5WVXOku/F1UEelY6Nc77CEg4VYMq2mbZ57jDJUFhHIziaXdzhlkiDduJ/01apNU01 wBpWhYwJY6jHMFTg6Zg7+f0GXWgAMVNL4CGYgaiicA7jvMmMTeHciky12BvoFp9+hzJZ TauQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772475; x=1782377275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OfwAKcFuVrniumYvLLu8JFz9uYaRHWvyegwwsVszSZM=; b=LO6AiBWyBm+jHvraNrLEFjV4vOC0TRJ1hrLvM/7QyKraR8Oacf9UuCJuGtuGmHQEY5 79BRjG8XLhO7nIMaBQBDWIa+rQThDOGgPJeufOCXhiLiCqzFBwSMDcNu5Ld1LQCBqr3n FpU959GkSWa0iqgz3UghzGNAyga+b/UTV8eN0ebQJqP50IC8Maydfkl6Ej9mpPkSTH5A CuOftVKRt2B9sH9lTuGPZtghH2A3LdZUS8Juy9eI0Et8FB4KDOO3+E6FtyK6UcXynE9c hSMXXkQIZobO0oFHwVe2yd6lnS5c64TPEo357N72MTwQ49JMKIXGVHfaQYa5fGPljPGD BrEw== X-Forwarded-Encrypted: i=1; AFNElJ+ZRm88FgXGFA1qmBUmlCSV+0cSr5pfrNDZ523xQYwmeCA3xOz0ussXEE3191oNcRJKIj6mXX5hjKqaLSA=@vger.kernel.org X-Gm-Message-State: AOJu0YykkXyvwFhCb2ygPA9MqtsNOYoENO4bEPHZHFSuZtVDPnBAKqbI QNeZkv68j93/QUkBmTfUiVC3QwhkKj0vbUVq6VL0t+dHiDARWUdPUL1W X-Gm-Gg: AfdE7cnUNkCAUOHF++D3ZbtBec3RIIeXBLSD1nJVZQHyz+N0fTDeBb2Bnu5/PHkU+/c lE1N+hK+O+d6UWjxbKVApFZr5JCYsYW0MslM4yURGJgZUiQ+6ROAHaAONC1D6zler87YrjbbL0L BUgGM9fHdZGxgUiIpDxJnbKYY4u2WaxfUgoBoYlbDNmh+qq0aiG1q148RZuiAw11HHLYB+P7HLc xueMBo8yb6VW8ty4VSNE42JO63aUZPqQkkoMwTdNDi3qI2SXPrCVO4N+djJd5dTvD7Cxup4u7GN helkdcSP8/34upNiuBQ3je19YHxihSNcRAQxvE0GaCtD+bNm1pySmoP2/GAg2gE9voIp0hYOGt6 T2fIdvE2v55rRFwSBWNOLLaXPL7SyeDeBN3DwAfsHsGFfM/UWP2Lq3oy0T5XQ2Sn9g8mGEbDK88 ORcjP3GOT1l46dKa0a+UPisfO1566KuFuUk5p0l1RnFqKVFQ== X-Received: by 2002:a17:90b:390f:b0:36d:635b:85a0 with SMTP id 98e67ed59e1d1-37c936a1d28mr8015697a91.3.1781772474606; Thu, 18 Jun 2026 01:47:54 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.47.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:47:54 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 4/6] mm/vmalloc: Extend page table walk to support larger page_shift sizes and eliminate page table rewalk Date: Thu, 18 Jun 2026 16:47:24 +0800 Message-Id: <20260618084726.1070022-5-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260618084726.1070022-1-jiangwen6@xiaomi.com> References: <20260618084726.1070022-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" vmap_pages_range_noflush_walk() (formerly vmap_small_pages_range_noflush()) provides a clean interface by taking struct page **pages and mapping them via direct PTE iteration. This avoids the page table rewalk seen when using vmap_range_noflush() for page_shift values other than PAGE_SHIFT. Extend it to support larger page_shift values, and add PMD- and contiguous-PTE mappings as well. Rename it to vmap_pages_range_noflush_walk= () since it now handles more than just small pages. For vmalloc() allocations with VM_ALLOW_HUGE_VMAP, we no longer need to iterate over pages one by one via vmap_range_noflush(), which would otherwise lead to page table rewalk. The code is now unified with the PAGE_SHIFT case by simply calling vmap_pages_range_noflush_walk(). Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen --- mm/vmalloc.c | 81 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 47 insertions(+), 34 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 6660f240d27c9..253e017130e09 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -127,7 +127,8 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long add= r, unsigned long end, pte_t *pte; u64 pfn; struct page *page; - unsigned long size =3D PAGE_SIZE; + unsigned long size; + unsigned int steps; =20 if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; @@ -149,8 +150,8 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long add= r, unsigned long end, } =20 size =3D vmap_set_ptes(pte, addr, end, pfn, prot, max_page_shift); - pfn +=3D PFN_DOWN(size); - } while (pte +=3D PFN_DOWN(size), addr +=3D size, addr !=3D end); + steps =3D PFN_DOWN(size); + } while (pte +=3D steps, pfn +=3D steps, addr +=3D size, addr !=3D end); =20 lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; @@ -542,8 +543,10 @@ void vunmap_range(unsigned long addr, unsigned long en= d) =20 static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { + unsigned long pfn, size; + unsigned int steps; int err =3D 0; pte_t *pte; =20 @@ -574,9 +577,10 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned l= ong addr, break; } =20 - set_pte_at(&init_mm, addr, pte, mk_pte(page, prot)); - (*nr)++; - } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); + pfn =3D page_to_pfn(page); + size =3D vmap_set_ptes(pte, addr, end, pfn, prot, shift); + steps =3D PFN_DOWN(size); + } while (pte +=3D steps, *nr +=3D steps, addr +=3D size, addr !=3D end); =20 lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; @@ -586,7 +590,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned lo= ng addr, =20 static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pmd_t *pmd; unsigned long next; @@ -596,7 +600,27 @@ static int vmap_pages_pmd_range(pud_t *pud, unsigned l= ong addr, return -ENOMEM; do { next =3D pmd_addr_end(addr, end); - if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) + + if (shift >=3D PMD_SHIFT) { + struct page *page =3D pages[*nr]; + phys_addr_t phys_addr; + + if (WARN_ON(!page)) + return -ENOMEM; + if (WARN_ON(!pfn_valid(page_to_pfn(page)))) + return -EINVAL; + + phys_addr =3D page_to_phys(page); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, + shift)) { + *mask |=3D PGTBL_PMD_MODIFIED; + *nr +=3D 1 << (PMD_SHIFT - PAGE_SHIFT); + continue; + } + } + + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask, shift)) return -ENOMEM; } while (pmd++, addr =3D next, addr !=3D end); return 0; @@ -604,7 +628,7 @@ static int vmap_pages_pmd_range(pud_t *pud, unsigned lo= ng addr, =20 static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pud_t *pud; unsigned long next; @@ -614,7 +638,7 @@ static int vmap_pages_pud_range(p4d_t *p4d, unsigned lo= ng addr, return -ENOMEM; do { next =3D pud_addr_end(addr, end); - if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask, shift)) return -ENOMEM; } while (pud++, addr =3D next, addr !=3D end); return 0; @@ -622,7 +646,7 @@ static int vmap_pages_pud_range(p4d_t *p4d, unsigned lo= ng addr, =20 static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { p4d_t *p4d; unsigned long next; @@ -632,14 +656,18 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned = long addr, return -ENOMEM; do { next =3D p4d_addr_end(addr, end); - if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask, shift)) return -ENOMEM; } while (p4d++, addr =3D next, addr !=3D end); return 0; } =20 -static int vmap_small_pages_range_noflush(unsigned long addr, unsigned lon= g end, - pgprot_t prot, struct page **pages) +/* + * It can take an array of pages which are not all contiguous, but it + * may have contiguous chunks, as hinted by @shift. + */ +static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long= end, + pgprot_t prot, struct page **pages, unsigned int shift) { unsigned long start =3D addr; pgd_t *pgd; @@ -654,7 +682,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, next =3D pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |=3D PGTBL_PGD_MODIFIED; - err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask, s= hift); if (err) break; } while (pgd++, addr =3D next, addr !=3D end); @@ -677,27 +705,12 @@ static int vmap_small_pages_range_noflush(unsigned lo= ng addr, unsigned long end, int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { - unsigned int i, nr =3D (end - addr) >> PAGE_SHIFT; - WARN_ON(page_shift < PAGE_SHIFT); =20 - if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || - page_shift =3D=3D PAGE_SHIFT) - return vmap_small_pages_range_noflush(addr, end, prot, pages); + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC)) + page_shift =3D PAGE_SHIFT; =20 - for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { - int err; - - err =3D vmap_range_noflush(addr, addr + (1UL << page_shift), - page_to_phys(pages[i]), prot, - page_shift); - if (err) - return err; - - addr +=3D 1UL << page_shift; - } - - return 0; + return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift); } =20 int vmap_pages_range_noflush(unsigned long addr, unsigned long end, --=20 2.34.1 From nobody Fri Jun 19 03:49:02 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7BE33B4404 for ; Thu, 18 Jun 2026 08:48:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772482; cv=none; b=BiZCnlcgX2/zzEmRKiB2sDBo9ko5WMLAQ5BcSRM+1WHaQzMhdmTLYEQau9TAAZE7vHohv9VtDPBcSacyjN0XEjB2UYY93H6vxBRqPheFw1+zH6iFPRCV0qiiN2tKRd4NZM/C/tHSuRZmv1+rHhmsesEnPvPj8OkdZHGvcMoJsWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772482; c=relaxed/simple; bh=0jA4bWRxCS9RYlViZzqAf2RUbzV0nYU610uVSvrK/KQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=RRjBNmTD0vWcoon/ID4m9oL9GpUCG8IRQHqPpZ4VVwI1ceEZovlnx+eipGEpSB2Kn7UvTjQnodaCq9+WTnOq0xVKaRYvn8nMVgmcR2VdLh7eF2QmVqcoIcxeXeNqSq0YZ5+/eUpPJftG/RT/UP90gWMaG3c0owhPGN5q9QjF6UA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rL/BMpGU; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rL/BMpGU" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-36bcbd7821fso221883a91.2 for ; Thu, 18 Jun 2026 01:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772480; x=1782377280; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OaoSxKyClWE3jnYRtJaf7Jb5FKch1UvGKeE4Hl6seUo=; b=rL/BMpGUW2/XCSN2HMJEfIp4nX9xWx3d/67hiN9fjc266ksze/Cd1XLf5iX5t9Yvjm oSFM8C3N1T8z8mwc8f3SmS97rCsGwSM+yghtthrhd0WCtSJnKefRrW20goaftRkF1ncL 59d7AGCJqbYfaThdWD0Mj9PQLHayo6NSr7sM/c29/pjDPv6P8Xth6t4p5bFIPLb4RtnZ eIla7aneNFVirHct1gejnJzeSPqKw6jGiKEjKok+FMXptbdn1ANjMZcxWIOpfhx7A73d ai4bC/jCQTSkWz1PEw3TXBT/ydIjLyVbnKuTwGVroV/TybpSQW8RiXocXgxVm9fO+JFP 7d7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772480; x=1782377280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OaoSxKyClWE3jnYRtJaf7Jb5FKch1UvGKeE4Hl6seUo=; b=aMP1de90LJ/KR1c4yjiSPIK6nImzHPfU2adbxGXtDjpMH07JM+PM3WhDhUHY/64VXs AYpMTFGI9prHZB9ZRyheHHTAUQZfQThFQhGdwcgt63Lol8BnuL0nHNaPRz6AuJee8/Cb hjkkgiPYEPuJqM+Tz7xySKOPmlymfoRTH2NFrxIp11wPN9596cTuNXJlWRVqxRfEEDLt S1tdIoFhis8/MMvGGWT2AlwJDkU/4+X+Tf628Ayr+NKsQBtb8XpSVCYiYtoJF2RYruwh 5ZmSYUJMyxa/m93XpPgGYJeQ21vrH0HV1YItTD4A148zwtCgPe3CAQJSXH369i/rHO+M 6pSw== X-Forwarded-Encrypted: i=1; AFNElJ9Pvc9vCB9vnBTSttqgIvsqzL3Q5moZ0tLfnEV4r1BPcEtri0gZa+aIYd5yktXk75hqNKXCLTDOv9cLDaY=@vger.kernel.org X-Gm-Message-State: AOJu0Yzpv4Lut+YzdA1PXa+dAsoYbkCZ/9tKcDtYVfFIx6KPu4p4mvR9 2XCXDHGMvc0DlfZkpulsYmYxkeqmNWqJJRorBBUh/9xvGgqWJpUWO6Fz X-Gm-Gg: AfdE7clmiPISLtDUCJVuyNak+a7xWIVnBdMP+VFZoj067p2wP4eNNisYefGVf5r2na4 RzqNexANsZWlP9LzpM2Q3STnqkHnPi19+uDBiI6Bl6pOPJQDZtKyXLp1WQ2iO4PFhL8VzqebdTQ bOqAaeL+GrtEj3d29ouNJG5yX7OYbEYFSoZNunvB5nFsyTPVDXxD+0UuFbEbCUHbSkwlB7Cx9ss ygWBVlkx2piinLWaAsbgy6l9R+XC6r6UILvl7+/lj7nn31uL6/FMUiQsz0fyWRI5lBT1d21XslW gYYG02LK77d6unncwvmBSNSC9cEAtTqdbRaRbvCzj/XMVrKii/U+vl7pciPmsmZBH1Xqll+TgYd RXl3wPl0jYHnPewUHRQWGuk4xOwcs86RCTx7ndJ+FVNCtogIYYlE6hQpxkaDzutuCsHtWW7g+TV WUfXCjNzesdwR/gvAAXOWQZeixQ6lt9zOh448= X-Received: by 2002:a17:90b:1dcd:b0:36d:f28a:c5ee with SMTP id 98e67ed59e1d1-37c9ed08579mr7793228a91.25.1781772479946; Thu, 18 Jun 2026 01:47:59 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.47.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:47:59 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 5/6] mm/vmalloc: map contiguous pages in batches for vmap() if possible Date: Thu, 18 Jun 2026 16:47:25 +0800 Message-Id: <20260618084726.1070022-6-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260618084726.1070022-1-jiangwen6@xiaomi.com> References: <20260618084726.1070022-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: "Barry Song (Xiaomi)" In many cases, the pages passed to vmap() may include high-order pages. For example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently, vmap() iterates over every page individually=E2=80=94even pages inside a high-order block are handled one by one. This patch detects physically contiguous pages (regardless of whether they are compound or non-compound) by scanning with num_pages_contiguous(), and maps them as a single contiguous block whenever possible. The mapping order is determined by taking the minimum of the contiguous page count and the pfn alignment, allowing graceful degradation when pfn alignment is less than the contiguous range. Pages with the same page_shift are coalesced and mapped via vmap_pages_range_noflush_walk() to avoid page table rewalk. As users typically allocate memory in descending orders (e.g. 8 =E2=86=92 4 =E2=86=92 0), once an order-0 page is encountered, we stop sc= anning for contiguous pages since subsequent pages are likely order-0 as well. Signed-off-by: Barry Song (Xiaomi) Co-developed-by: Dev Jain Signed-off-by: Dev Jain Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen --- mm/vmalloc.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 85 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 253e017130e09..fffb885cb2158 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3545,6 +3545,89 @@ void vunmap(const void *addr) } EXPORT_SYMBOL(vunmap); =20 +static inline unsigned int vm_shift(pgprot_t prot, unsigned long size) +{ + if (arch_vmap_pmd_supported(prot) && size >=3D PMD_SIZE) + return PMD_SHIFT; + + return arch_vmap_pte_supported_shift(size); +} + +static inline int get_vmap_batch_order(struct page **pages, + pgprot_t prot, unsigned int max_steps, unsigned int idx) +{ + unsigned int nr_contig; + int order; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) + return 0; + + nr_contig =3D num_pages_contiguous(&pages[idx], max_steps); + if (nr_contig < 2) + return 0; + + order =3D ilog2(nr_contig); + + /* Limit order by pfn alignment */ + order =3D min_t(int, order, __ffs(page_to_pfn(pages[idx]))); + + if (vm_shift(prot, PAGE_SIZE << order) =3D=3D PAGE_SHIFT) + return 0; + + return order; +} + +static int vmap_batched(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) +{ + unsigned int count =3D (end - addr) >> PAGE_SHIFT; + unsigned int prev_shift =3D 0, idx =3D 0; + unsigned long start =3D addr, map_addr =3D addr; + int err; + + err =3D kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + PAGE_SHIFT, GFP_KERNEL); + if (err) + goto out; + + for (unsigned int i =3D 0; i < count; ) { + unsigned int shift =3D PAGE_SHIFT + + get_vmap_batch_order(pages, prot, count - i, i); + + if (!i) + prev_shift =3D shift; + + if (shift !=3D prev_shift) { + err =3D vmap_pages_range_noflush_walk(map_addr, addr, + prot, pages + idx, prev_shift); + if (err) + goto out; + prev_shift =3D shift; + map_addr =3D addr; + idx =3D i; + } + + /* + * Once small pages are encountered, the remaining pages + * are likely small as well. + */ + if (shift =3D=3D PAGE_SHIFT) + break; + + addr +=3D 1UL << shift; + i +=3D 1U << (shift - PAGE_SHIFT); + } + + /* Remaining */ + if (map_addr < end) + err =3D vmap_pages_range_noflush_walk(map_addr, end, + prot, pages + idx, prev_shift); + +out: + flush_cache_vmap(start, end); + return err; +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3588,8 +3671,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; =20 addr =3D (unsigned long)area->addr; - if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { + if (vmap_batched(addr, addr + size, pgprot_nx(prot), + pages) < 0) { vunmap(area->addr); return NULL; } --=20 2.34.1 From nobody Fri Jun 19 03:49:02 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B54B73B47C6 for ; Thu, 18 Jun 2026 08:48:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772486; cv=none; b=CHF4uwxf6zInjYhAnw6br77DypZfDtpSgZQUS/vVsj5nkRRWJyBrpYrglaw0oOICp5XQdTxIcqM93MyjuBpzQa89dv3uIsY4R8FYUCXtD860CzkvTlZBe1NwioabjlV21rPrxK1TPZAlOFeF//yVy1AB5beCKzN7Q3hHwrxte7o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781772486; c=relaxed/simple; bh=CcAdwbgzJvkNtJI8RhR4K2fzod1YtphpK9heSCwfy/g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UzK5/H4BVF6ctPnVcIRHvXRLJ6kT2WcswfNFVjEaVsM3BmVqw1QUkQ3WtHGGiNsib+JV6vKwJRcThs/G+tU6RF6ccl7xazSx+6+CDgkoBexrHlOO1Rh1EsVvg1GUeMcvH3L0JNPRMuPkaQo7hUgc7odTjXnsw5kdrjbnh+edHkA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KIDcQyjv; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KIDcQyjv" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-36baeec21dcso345163a91.3 for ; Thu, 18 Jun 2026 01:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781772485; x=1782377285; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O9Am4qJqS/MfI7jTQw49PX1QkIeiHV3cGmQD98EHd9U=; b=KIDcQyjvIaaCD3C5wil77jXW84YIsJm9xmkTCkQxCAnq+cfDEAKLY/VVbftqXIwEwY MMs4WzEvIz2bJ/k/634lSF3yjZSk+HKF0ucEAh9TIG7GpvEKJNwh0RhmXNrBK3XE3SkK as85PC95XM4f0NXze925w/FSHNTsBgZoF4cphMsX6kPslRnSsnyciIJT6ZLSFmUop/ex 9bWSX5b59UUUIOUm9lWBunHEq/E5+HTeDeFtbMc7tPxjdsz0Np2ydh5K6pXJdUUPUPJa O7O73ZUnLzN9JzcRF/bWms5SneYe+J4H6RtmEsjT3BkFos58g7GfhdiCLxcfd1R7FNUe q+NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781772485; x=1782377285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=O9Am4qJqS/MfI7jTQw49PX1QkIeiHV3cGmQD98EHd9U=; b=T8JzkdQvzXuMPqRsflTrdWjBqRP5eEHZqPi3HSwL7hl2Vs/nPOuRLrtiL03bfviiaO XbchWjC0w0DjFLRyf86Uua2F6Lra7H/h88w5/ad7AhtA8ICTZZUbfdz6x67T/A/WTHmz BcE7noUKlAt7UyPNdzUJQfhh0ehV8MUa3AO2wimBw/rqEUD1+S1KetXfU+Hgocqtn2a4 HyM1i+323OKFS5tYIcZ/YYeQm+SrcqneS1fRVtgDMHJqRab6QBZypVmEl7/1uIWO8Hlo joUgx3iRDKMW6nEYrKWe/eSsLtJJLy4kLOJP1dR7rsNd/U+QxD/Qa6G6+5Z7DmPIrNqx Wa6w== X-Forwarded-Encrypted: i=1; AFNElJ/DP0dmPQHpMYjJ8nrE2FoTdWUeejLmJYH+6qaFOvckUP9rvX01+UICAlAEc1cLEBD64HFgZbFOSomtAsE=@vger.kernel.org X-Gm-Message-State: AOJu0Yw6CR/+xF4UXVSWIFBnqSV2Rgo8O71doO21dnRrdJ+Lzb4C+OcW qcroSMzyv0dDp1aYWrsy8uD8ES5ZzySSO12XHgB1vXFhjwHTi1O34olA X-Gm-Gg: AfdE7clybN8PdqrmO0mVK+D0IueEoXsNEsoC477rvCKyAS6PsfCEnFM58GEcOIwlLBM NflKifn0P9UHcB4qY/7mlK5OgRktF91HKEyw4Z4wq6/4oZkJj1Z6LrV3X2n41Gyu8pUgSo/FsNr ZifwXo0dYItUbhONacl4f5yLdLUNdnxwiHVO3FNwpVSViFXEEUh5MiDIUOg7GyUpd1RRJzwudZd /nenlDigRPh/yj/vyjZsOGAsOiJeOb15eWjk/To8gC8eNHzG0b49dy6A8Q5htce5ha3++6YduE+ 9SREWQ5oqJ+mCZuqVnvAx2vTSdyv8Psn5f+r7J21O6BojMlSwUlEhR0l7f4X402BCgWg30E9W4i sqw/bwMspJPZxwSJQjdQr3V7Se+4eY2AbEbtYXafRnP97RsY/nThbNyuqjVqQM8pBJmvmjVfcgG jQwazNC3PU2jtDd2vTb1vnPkRpwhu//hNihsE= X-Received: by 2002:a17:90b:2fd0:b0:366:3517:1aa2 with SMTP id 98e67ed59e1d1-37c9be73275mr6808218a91.0.1781772485070; Thu, 18 Jun 2026 01:48:05 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-37c521d8e7bsm8628856a91.5.2026.06.18.01.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2026 01:48:04 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org, akpm@linux-foundation.org, urezki@gmail.com Cc: baohua@kernel.org, Xueyuan.chen21@gmail.com, dev.jain@arm.com, rppt@kernel.org, david@kernel.org, ryan.roberts@arm.com, anshuman.khandual@arm.com, ajd@linux.ibm.com, linux-kernel@vger.kernel.org, jiangwen6@xiaomi.com, shanghaoqiang@xiaomi.com Subject: [PATCH v4 6/6] mm/vmalloc: align vm_area so vmap() can batch mappings Date: Thu, 18 Jun 2026 16:47:26 +0800 Message-Id: <20260618084726.1070022-7-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260618084726.1070022-1-jiangwen6@xiaomi.com> References: <20260618084726.1070022-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" Try to align the vmap virtual address to PMD_SHIFT or a larger PTE mapping size hinted by the architecture, so contiguous pages can be batch-mapped when setting PMD or PTE entries. Add __get_vm_area_node_aligned_caller() as a wrapper over __get_vm_area_node() to simplify repeated calls with fixed arguments. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen --- mm/vmalloc.c | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index fffb885cb2158..bc9fa93e2bdc6 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3628,6 +3628,41 @@ static int vmap_batched(unsigned long addr, unsigned= long end, return err; } =20 +static struct vm_struct *__get_vm_area_node_aligned_caller(unsigned long s= ize, + unsigned long align, unsigned long flags, const void *caller) +{ + return __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + NUMA_NO_NODE, GFP_KERNEL, caller); +} + +static struct vm_struct *vmap_get_aligned_vm_area(unsigned long size, + unsigned long flags, const void *caller) +{ + struct vm_struct *vm_area; + unsigned int shift; + + /* Try PMD alignment for large sizes */ + if (size >=3D PMD_SIZE) { + vm_area =3D __get_vm_area_node_aligned_caller(size, PMD_SIZE, + flags, caller); + if (vm_area) + return vm_area; + } + + /* Try CONT_PTE alignment */ + shift =3D arch_vmap_pte_supported_shift(size); + if (shift > PAGE_SHIFT) { + vm_area =3D __get_vm_area_node_aligned_caller(size, 1UL << shift, + flags, caller); + if (vm_area) + return vm_area; + } + + /* Fall back to page alignment */ + return __get_vm_area_node_aligned_caller(size, PAGE_SIZE, flags, caller); +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3666,7 +3701,7 @@ void *vmap(struct page **pages, unsigned int count, return NULL; =20 size =3D (unsigned long)count << PAGE_SHIFT; - area =3D get_vm_area_caller(size, flags, __builtin_return_address(0)); + area =3D vmap_get_aligned_vm_area(size, flags, __builtin_return_address(0= )); if (!area) return NULL; =20 --=20 2.34.1