From nobody Sat Jul 25 17:34:24 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7BB5438474 for ; Wed, 15 Jul 2026 12:08:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117312; cv=none; b=HYNSXAONpjMH4w43EOMKnr2snDAWHDEod8mK8pWxHegipnSjw2hNel6Axpwv9j/pe00hPI9mC1ukTQR9P3HZ4TgUI5eQ7hOcoZYzheCai37jQkLa9nsvtYIO3H4y7Go/blDiZ1fG5MDCn4BjVMTrXKuMg/sWKkQ+GV+BVDYoOxo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117312; c=relaxed/simple; bh=hCZ9ZDKx/Q/LrLRawncxe7Cg9mQ9OSgFt6wLBYBBP80=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QRWUbQXJ+KC6klWGV92VarMCYsvo1qCgsoj8Sx8CHDUFbVim8Nj5oJsZ+uYgrIypGU9UsOwqqJU8iB2kd9SmIT6SySUTtqDzEQSx3+p0TxL0J9dcYrSPwEAMNyCDPmh3ZqWczwZzrSu5dMweybmTsjR5mQdhoN7+dl2E5ioqtDg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Kl+bupaZ; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Kl+bupaZ" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2ceb096e675so44547195ad.0 for ; Wed, 15 Jul 2026 05:08:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117308; x=1784722108; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=etJA2syXRyIgV9yIfYwz42KuGi2MCTWS+UAi0vZeNE4=; b=Kl+bupaZdWcwEAEbfP/IaMfyBwOI5NsgPDTmiLczZZUQ6y/KwUDWtnxU9j9z9R3mSQ M3jQnh0nTeEyqP0mkhJFg2qEfLUvngiSaCQekcz3OmIf5Hgrxja0WQ2UonZ9NPRsPZHw 9guctjgQz+xXbjb9iwqxU1Ih2s/i81tNoTmGqbQmN1WrqjuKaECJdlgSK3oDNDFr6UFN LshZ+veF2YXLmngkU33mSL6lLWmfiNKO6Wr5hpxF1SZb2U3mgs3lhT2mfx2b5cdQe2V/ rjeodZ0+GI7CTj+se9vzmMWqMVROC3hI97ENHZWHRPu33kD7tF6LXIpQ87KHD4Q8l/XC ZJxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117308; x=1784722108; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=etJA2syXRyIgV9yIfYwz42KuGi2MCTWS+UAi0vZeNE4=; b=sldLSZMn1yxndy9+RLivirbUs4YrzyQzJafNcNtdC4oJ/rqAJF6V6uVV98UIGpSeDc VO7wkhE+eCJ3/GAnXW4v17RllmFpp/SLM0vfW+vwYszEl7CrdUNEjRbm5Ybj2wSxs2SO IyEdqyzdYBut8GK5ajdssH5GEib77pofAOqE9FWHFhLauZgmMQGW4po1e76ZpGNMT9WS ktcNV3iY2WciK3osVibHJqDjVhZMYOE8POLXtoK+Bng7IdmOR2cPBNFp7AM/ZcR3dB4z jcpjkxDQmrBudizhUbyevSs368TQKScxPLW/nAwnTFO/JPrSG0NJglGW7XjU6wpBs6fL SW8g== X-Forwarded-Encrypted: i=1; AHgh+RoTkWP7qrbq9pSuZ/bG6ee0o8k/4AY+qY2z9hTRCeba0rXw++mco1eALrl92aeAUscf9f3Iyi3NiV5Yy/A=@vger.kernel.org X-Gm-Message-State: AOJu0Yw8jjovE4sE9KqFavCP6fXca6aBBWq9zPJOv+c6M8QEug4e651F X8I98vGP9SJ8rEIdICyMEjk4l1OfJ7oDkaa55tAhouztlscvmZ0Dw8og X-Gm-Gg: AfdE7cneqviJSKHLDY6zlf4FTa8xO7MJrmHyFCHzNPpLz/j6rD4Pm57hzWxpUWB2vy9 B+tqzQ4hmuUNv2lvuD7GUrwL4szuuFORQMGgS+1+vcH7SbdVek5i+qO7To/+HfJGq+RwNeJpp6A 2vuXKKb5udcgARqFxaRC9/cEmmNbcX5ZAOLx9JCAqSXRkvUEtthZcox6zxbVG+L+D6ZoOq/WX9L AgTR9uYDJpo0tP+WRkUzCVgPg32+juabg/jWdreob2HJjignTD9KnwjBipN4FweWSWBwxcqc+m1 YeF3Uwbl8edEgtkxzk4rL314XhTD34l0EEmEaJo+axOZKByVQhsFoGeijP7vss9QgvuDTKhR/TT QJvp5LlYPASBg+WXbd84kklpGeKjFhpzVHg/bNGtDEWphkbwLSlXA+JiOXLr+9hCgbsFmHKnBvF PBToFdjcssi5/XHwqEQRKTOwUjF1UKM2fyjco= X-Received: by 2002:a17:902:ecce:b0:2c9:cf5d:d9b8 with SMTP id d9443c01a7336-2cf03d24104mr22917075ad.33.1784117308327; Wed, 15 Jul 2026 05:08:28 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:08:27 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 1/7] arm64/hugetlb: Extend batching of multiple CONT_PTE in a single PTE setup Date: Wed, 15 Jul 2026 20:08:07 +0800 Message-Id: <20260715120813.3609949-2-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" For sizes aligned to CONT_PTE_SIZE and smaller than PMD_SIZE, we can handle CONT_PTE_SIZE groups together. These additional sizes are mapping spans used by non-hugetlbfs(vmalloc) mm code, not new HugeTLB hstate sizes. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Dev Jain --- arch/arm64/mm/hugetlbpage.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index a42c05cf56408..7ce159483a354 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -94,6 +94,11 @@ static int find_num_contig(struct mm_struct *mm, unsigne= d long addr, return CONT_PTES; } +/* + * num_contig_ptes(), set_huge_pte_at() and arch_make_huge_pte() can be + * used by non-hugetlbfs(vmalloc) mm code to set multiple huge mappings + * at the PTE level. + */ static inline int num_contig_ptes(unsigned long size, size_t *pgsize) { int contig_ptes =3D 1; @@ -110,6 +115,12 @@ static inline int num_contig_ptes(unsigned long size, = size_t *pgsize) contig_ptes =3D CONT_PTES; break; default: + if (size > 0 && size < PMD_SIZE && + IS_ALIGNED(size, CONT_PTE_SIZE)) { + *pgsize =3D PAGE_SIZE; + contig_ptes =3D size >> PAGE_SHIFT; + break; + } WARN_ON(!__hugetlb_valid_size(size)); } @@ -359,6 +370,10 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shi= ft, vm_flags_t flags) case CONT_PTE_SIZE: return pte_mkcont(entry); default: + if (pagesize > 0 && pagesize < PMD_SIZE && + IS_ALIGNED(pagesize, CONT_PTE_SIZE)) + return pte_mkcont(entry); + break; } pr_warn("%s: unrecognized huge page size 0x%lx\n", -- 2.34.1 From nobody Sat Jul 25 17:34:24 2026 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 939F943785E for ; Wed, 15 Jul 2026 12:08:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117320; cv=none; b=o7chYL8L9KkaND7V8EpF/iPxougbSizRK5CyP935U7aq3exiCdeEYVG9+tLrQQNHyjLnJ6ppODt62QmLklqELBYI3wswiVIeOmN7wKAPs6JPJP4ND/ntebCkKabZDai9LlcFrWJ2qdtMMfHzcUgUKIAr9CZJd4RC2z0HJkywv+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117320; c=relaxed/simple; bh=Ppu0MtSDHQ8A9F8kzPxP1vbPKRBrorbg7rLHm/YR99k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cuPmpc4MoQbwGGghS4LzYthJTYFaStEXeJcflaFsMWilZrn70BlQ5l4rNFR+SIh8junKNlONbO/k3E3scXnWDaWYsoVowf681v4BC/7VWrIsh6tiUS/LhfSLSCfXP1eOy/xzoDI97IPXnC2jdtoQVUviRdCS0luzoMdf7atqrOQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DUufz+N1; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DUufz+N1" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2cc73e322dbso64369865ad.1 for ; Wed, 15 Jul 2026 05:08:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117314; x=1784722114; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=v3KNxe0yLoF7PUMUYd/OqmVO3pDmLAAWwOynTW5KQ9A=; b=DUufz+N1DTIC/55K15Yni0bgNi9V0R+gV0wyT0YLD8g4iwHRxd0HbeNifFR1OBqP0r 0Lb7IEtbyfrJd7CoaGogQ1W8BcprMh7VquJHv+P/B28yQuvLE+T2B87nvqky7YhRK5CO xqRfYgFwUuBQGt4olVKGPkTBCY3fjKLxJthoOfI/b6ZINFLx+oWrXvvGwQr2/cQwYftE 4TIYmiFro6saepSgz1yfwcJw+m/KFfuaMmc7gXT7nLw8wBG+2LfT7orRU3myfKLekzfA fSuyQ0hs9u8SHtSnNFe6CW4UVSdY4x2JWohtOVwU2vHHP0Wb3YrimyZJqSuUtkl+Ai7i tjKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117314; x=1784722114; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=v3KNxe0yLoF7PUMUYd/OqmVO3pDmLAAWwOynTW5KQ9A=; b=AHLiMXFqeS1V9T7SIjoW85VTolN3JUgK6+y4ZviQ/+byYlgFXssaqGB5Dsw24/Uagg yzTpCbZRtE8CBT84pOvkynNb48Ys1vYeDO7AyHyAqNJ6Q8nC+Zt/45h3FfJ2qLHthf3P 1UoJMmlDuXw53RA4YpZxYxzXmPp7qqfydIn7VvWA/bk3dFunpBl5uAYo14JnhKo7hZBs uCTdGrpYmWmv5mRxtffVKcrDOx0vDLdtPIOs0yA4EhMrlSPEpRTbDJzqdSAHHWSzUJ0N +rNlkteU9ConXyzJk/HyMuQ+4lxQ8v/2SymyCN25D6LGS668dAWELhvLwbwjsXaAR5wd 6cDA== X-Forwarded-Encrypted: i=1; AHgh+Ro2gBwm9JjCWZ1egtTpdn8yaw8vdM4YVDVVcx9Wmgch9oIWmS8ChAgSEEXYovfF8u9i9L3maqoQ35L/bJ4=@vger.kernel.org X-Gm-Message-State: AOJu0YxpfQ6cUE6ujaGNmgN9wWGVNPew2JvGGSypi/2Dhd+I1r0U96kq Anr6RW5v7ElJIjJhtsEVlof5cEsn6f0m1HSyzuz411x35t/pP0J98G1K X-Gm-Gg: AfdE7cnFDfq5bFUazvxAxBa1upvK2MCrA5MyLcrRjZ/M96O/xB+AFPHTGx18nC4mOHf QSNXv6H9uxO3EddOCkLK6wwUJW83G3abWh4YHmmG+JmMv81WE2oCe9Tp2eljWeCRusnginM5wo9 HZS4mTeem6mdY3beWnTEumD46CL4SIEhI4rCrgj+82dlDtkuc8FC5ndzASNBzEOC6C1T2vWrXhV X3gN2iEdpyXdMlOaXdThpQP32HErE5Cp1LwRQHAmTaq4uT+l3Uz7sfF41oDaXehnVwdgTlPer8i TD3g4at8FUItN2nK2pMaX1aQVGLLCxVu/FHPZGDoRjEACI8Ys7QJgxgR9IuM7mN+OVUgOmHR7UH hY3ICwWFhOVvZva9g+OBvGFwkqkUqpqFGQiltHabyYCmDDXUTxeowJTRVdkHAAxHVJ7EZz/f7ql wYdrnEemCScfDGNB7pwjZluFwrj2+CDQHeF3c= X-Received: by 2002:a17:902:ced1:b0:2ca:ec6f:aeca with SMTP id d9443c01a7336-2cf03c76231mr23240965ad.8.1784117314292; Wed, 15 Jul 2026 05:08:34 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:08:33 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 2/7] arm64/vmalloc: Allow arch_vmap_pte_range_map_size to batch multiple CONT_PTE Date: Wed, 15 Jul 2026 20:08:08 +0800 Message-Id: <20260715120813.3609949-3-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" Allow arch_vmap_pte_range_map_size to batch across multiple CONT_PTE blocks, reducing both PTE setup and TLB flush iterations. For CONT_PTE_SIZE-aligned ranges, return a power-of-two mapping size that may cover multiple CONT_PTE blocks, capped below PMD_SIZE. These sizes are vmalloc mapping spans, not HugeTLB hstate sizes. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Dev Jain --- arch/arm64/include/asm/vmalloc.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmal= loc.h index 4ec1acd3c1b34..d665f9d687422 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -23,10 +23,14 @@ static inline unsigned long arch_vmap_pte_range_map_siz= e(unsigned long addr, unsigned long end, u64 pfn, unsigned int max_page_shift) { + unsigned long size; + /* * If the block is at least CONT_PTE_SIZE in size, and is naturally * aligned in both virtual and physical space, then we can pte-map the * block using the PTE_CONT bit for more efficient use of the TLB. + * The returned mapping size may cover multiple CONT_PTE_SIZE blocks, + * capped below PMD_SIZE. */ if (max_page_shift < CONT_PTE_SHIFT) return PAGE_SIZE; @@ -40,7 +44,9 @@ static inline unsigned long arch_vmap_pte_range_map_size(= unsigned long addr, if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE)) return PAGE_SIZE; - return CONT_PTE_SIZE; + size =3D min3(end - addr, 1UL << max_page_shift, PMD_SIZE >> 1); + size =3D rounddown_pow_of_two(size); + return size; } #define arch_vmap_pte_range_unmap_size arch_vmap_pte_range_unmap_size -- 2.34.1 From nobody Sat Jul 25 17:34:24 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D09BB46AEC5 for ; Wed, 15 Jul 2026 12:08:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117324; cv=none; b=kSF5jcxckRjMpeiZbUPmv/hi2TZsSLRcTRVjwXJ7KrTLgh4qqOmezsXnAqlBmHO+PDQF8abRdVGnA6pr4y7WFdsBL09cPr8MtWyvtXmDZHtDE+WQ2FAX5UGuyKYCjkySNKqzwoGfuQatT+DZSI0H7z7T6BHe9nVg0V7LCkEpS5U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117324; c=relaxed/simple; bh=Rhfw51yRYnuaGewszcozYbs6KuvmQwaD2hdsXZ+C+iM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iXdsYWLR+sT4AW5s+AFVcr19goeT/vpu6Y/xKaTFFGUd9AKtUIqIO6vaNMKYEdBA5b5AK97Vx0UsU4MRzluKV2+4hOhjDtqVBTcLW1/DftqojYI8LwHItKgObwXLpjKGIzWtMX08hyCnLMAGGPGp9uytL8zY0QrO2iA22GFuwXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=J0JkzgPs; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J0JkzgPs" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2caf228a910so40364625ad.2 for ; Wed, 15 Jul 2026 05:08:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117320; x=1784722120; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=gJXfNsewMtLeFnLJ7bmWocyu9UQcX6Ca0Z8isc9W34U=; b=J0JkzgPsADuCBtp2kiFyiq/TO7XJjtdeROgmqPBSJlSiqycVvB0bqjL7ms81p/Lkw0 ZW0c80OUS0efUeBqAulyGVRCHjhXftUxfGQuqSeGSyLb8lt4XcCM9dGBfe3ESB2EiHNY Yne81aypCbb3yCTmZLV8edoVdGZHD7zgtjQKq3JkpQW7v2CK6jq+6Ux46Re2d03MbTnR 3yOoyOQZPJGjCfvEqQ2JkhDY6JDVBxPC+4RdE8aX3zMp2SlWP+6u3kD7CzwygRLE7T0I WPYxs8K86sgpzJqh82ENGjoD0HhaPB3p06Bbz+i94R65LdahOLw18rpWWGyQLIll2367 Uk/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117320; x=1784722120; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=gJXfNsewMtLeFnLJ7bmWocyu9UQcX6Ca0Z8isc9W34U=; b=C7NUuk6B+Y2VPlcjOU0JRvXalgGpXQcHEjehEUxbPZBMxOlKACGMHUp0WFRzDmb237 JYF3k3tJMN68fPz35g8sB+ttv811sDXwjAb83Ty3MNHfnRG5qG3RNK6MZIv0Xbe/L9fW wyjJQPVaiMpOm/me1GbT3qFFm3K5yG/xKaGi82UrCTChI/cxUCuHwa/ILsg2xzklb7fe T2pJt4vHpw5JbDLx0cozYBfJshqV73mtIuNdAJW63T5rGRTm8DMtAZ1KaHZbsMcE/T/w R23FPW8y8qkMoIFO+xOxGz5gVDXtT57mY5sK5kOSe/wtFbXuDWhDMCf1SY2tr4vyM+j9 vPQA== X-Forwarded-Encrypted: i=1; AHgh+RoNP3UEWrPs6i4nMU6bM9NX+t6wVTd/Xqr7hl4yZ14+TBBPgesDhZKSXNbtMpjIRBElEDQtJgRaEH5xui4=@vger.kernel.org X-Gm-Message-State: AOJu0YwyJHN5BqMbiW2dXCfm2Pt6QdfuIx1Am6n+GAy1MB1af3RffUXk nmaSu1oUm/yA67KlN9AcQDXk8cbN2MxpBFAx8msAkNzW5GSo0VpB5efi X-Gm-Gg: AfdE7ckQu5alOkJ4pwuHAoeCH9VrB/OkMyPDsr+Rr0qP8C94LH92CoyCifcotiHXRsp gzuh5N51cbiIUcsMxDjM+z3dFTB9xzqHfL5BRH+YhmEuH/tNPltrNd5gBO8ZO9VH4MN6Ahg4zJr arlc+jOIqz7HR7jrGpbUFOwQFG/9eN6kTBWpCLh46Ql2wtWXOed98yOxQDwsPUbxQLkqptuc59s CW4a+gIKMJ8HRWAHLYONJL0mTsOKL45BUtDnrJhTXS2z6ReU3CIbnf7qOorlDG+WT2zcFt8xjha 6w4IVKS+91oT8zO+PB2gpqBW+2iesHw0O+kUf+8bQgNhOfJ4X+UDklSPeLEe7U6/ngiZUYBpNZa 6FQwcIFwPcJhFERext4vDGmOZ84Gt1qVSHyBNDp6aiDqZH0+eSe2F0vb6sbx2CvJg0AxQdz7LRW Sc+qJsXI3/Wv46ZdUWPOPix/WnR8B9XRMbHT/ETAgdjRdx6w== X-Received: by 2002:a17:903:468e:b0:2ca:d7a0:1312 with SMTP id d9443c01a7336-2cf03cef70emr23475045ad.25.1784117319865; Wed, 15 Jul 2026 05:08:39 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:08:39 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 3/7] mm/vmalloc: Extract vmap_set_ptes() to consolidate PTE mapping logic Date: Wed, 15 Jul 2026 20:08:09 +0800 Message-Id: <20260715120813.3609949-4-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the common PTE mapping logic from vmap_pte_range() into a shared helper vmap_set_ptes(). This handles both CONT_PTE and regular PTE mappings in a single function, preparing for the next patch which will extend vmap_pages_pte_range() to also use this helper. The #ifdef CONFIG_HUGETLB_PAGE guard is moved inside vmap_set_ptes(), so callers no longer need to handle the conditional compilation. No functional change. Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Dev Jain --- mm/vmalloc.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 2c2f74a07f396..b9b29f7857bb2 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -91,6 +91,33 @@ struct vfree_deferred { static DEFINE_PER_CPU(struct vfree_deferred, vfree_deferred); /*** Page table manipulation functions ***/ + +/* + * Try contiguous mappings at the PTE level for arches which support them,= and if + * requested by the caller. Fall back to PAGE_SIZE mappings otherwise. + * + * Return: mapping size. + */ +static __always_inline unsigned long vmap_set_ptes(pte_t *pte, + unsigned long addr, unsigned long end, u64 pfn, + pgprot_t prot, unsigned int max_page_shift) +{ +#ifdef CONFIG_HUGETLB_PAGE + unsigned long size; + + size =3D arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); + if (size !=3D PAGE_SIZE) { + pte_t entry =3D pfn_pte(pfn, prot); + + entry =3D arch_make_huge_pte(entry, ilog2(size), 0); + set_huge_pte_at(&init_mm, addr, pte, entry, size); + return size; + } +#endif + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + return PAGE_SIZE; +} + static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long en= d, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, pgtbl_mod_mask *mask) @@ -98,7 +125,8 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr= , unsigned long end, pte_t *pte; u64 pfn; struct page *page; - unsigned long size =3D PAGE_SIZE; + unsigned long size; + unsigned int steps; if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; @@ -119,20 +147,9 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long ad= dr, unsigned long end, BUG(); } -#ifdef CONFIG_HUGETLB_PAGE - size =3D arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); - if (size !=3D PAGE_SIZE) { - pte_t entry =3D pfn_pte(pfn, prot); - - entry =3D arch_make_huge_pte(entry, ilog2(size), 0); - set_huge_pte_at(&init_mm, addr, pte, entry, size); - pfn +=3D PFN_DOWN(size); - continue; - } -#endif - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte +=3D PFN_DOWN(size), addr +=3D size, addr !=3D end); + size =3D vmap_set_ptes(pte, addr, end, pfn, prot, max_page_shift); + steps =3D PFN_DOWN(size); + } while (pte +=3D steps, pfn +=3D steps, addr +=3D size, addr !=3D end); lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; -- 2.34.1 From nobody Sat Jul 25 17:34:24 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3DF046AF31 for ; Wed, 15 Jul 2026 12:08:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117330; cv=none; b=iWLD0GTJ3/sO7bgwSL+pmETxoyLRgV4rGuxtaFGgfZYtxtiCm2BpGEcF/gQr6Px2QEjJZxdaLcT6r0FfPLpWaLSw0LcMiiAuwMHIJeVMHq3gxQcqyCZVD8VZkFtfYVLeSagit5r9KYjXmPMwKEaZ3mi6Cnk9NSVnS89jIo99NcY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117330; c=relaxed/simple; bh=zATR21J6ZMyycSmgFjALpDH5VlVkpA2bkQHKq758ZeY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=leFSSzTArr1HEenS45NBu3B6Sr/6g1Vd2RC6pSDFcenGR9IlAWvR+B5eHHNVjOg6QMhdYe+FNJ8CXInLl8GStUAcBBLdlzrfksf8enfMhHqB0ESY6D9/1VsJinAYt7InjpyugzFpZRxTRLHvJQDdDSY5ZVVACtl3lcOuW5uqv5Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=rIEu/n8J; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rIEu/n8J" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2cab973140bso62231555ad.3 for ; Wed, 15 Jul 2026 05:08:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117325; x=1784722125; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Q6yq7ihIfejq4Lg+OKf3wnzi+aWZFQXVEshsSbOMWJY=; b=rIEu/n8JYOWLWwwDyX1Cxs7aBzmunrLxsqihZpXmvxrl8NIW05vdLygOFcgC2By80i LJdtcVy1DWHjlZu/76QLJ3Lh5EAAny+T3V+3pdgetMqGK8K93EAiUBcpaWf0sc3UR6Sq 7PD3WfBFXtiVYqBk+7kjcsOUz7+KZmzPQXGFzVcR5PZ1wJs12p2fJBWKdnQJkPV7TAOy Z1c3Q2lqJFK37dWqJQpZi3I9yZm5l8KSMKM7cNe/G/u76dt/VGWg9aGctmQ6QeAkMHhs 6GhQjwBEGzFpi28exBqOYEvymZZXkYsmJLjyPbyBN5EgzkaKM5/bWk7T5IN9gODxhVRr AGzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117325; x=1784722125; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Q6yq7ihIfejq4Lg+OKf3wnzi+aWZFQXVEshsSbOMWJY=; b=SanG5nzYlLT2L8L3xpC0IzLx77p65nx6OHoxB1NG4CgDYNsyi5ACSPpXudd16olDAF Sx9nwMXG3yM6cBOkGXKNCXxnfiSVhbhaPPmj+sC0i5Brh6/iVgYSLUKWzSkZRP4/KS1v vykvjv3GdTNmtR+p/Tt9nwBb3oVs88eiIl/HD8vZFy//a7CX5jObEwncd9flzZKrv7kl QvZlgtXxh2no8AeLSgBBGwUz6C8zFGGSjfNF86i/b119KieN050eVdxVMk5L2Pq414iv +XlNNjKQFqDLn9Lr3KKoueSGo9vbEYSJXbIPV/va8iHiWFQdEukU0KQHYyRMQTFzMSZi Vzkw== X-Forwarded-Encrypted: i=1; AHgh+RodxFfL+6/RxzZS2ehI/2CJYqsUNxCd1cLJEg3icBSMRO07AicDQEbRVcZZXDLJfaI2N4zTdj6s/BidAEo=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4FsTCw2fKvRM8cchriOnlrZGABAT6jBS4TQ4+wCHXrWUcsipb ETB1xzlxl/Dm0swifVOLLkh5mnYU1viFafaEzE15/BlgIqFJP2vQRRCv X-Gm-Gg: AfdE7cnhg1U64oJhSkCFWmytnerdXZJUUk9cLZF91qS8tejXbHj2yypbLvjRFniwZy2 kMiZFOyuBU/pmfU2aO+CahbDqOAjfFFNxQkFsIy5XXcWoM/98ITjCG2zwZTQDt2U6dGyev9eGU2 C7TjGx8zTr0HX4xpht28/p1CwLnWKeAGo/noMBm5WcxQDPN91O1SGNG1u9mASR6MdxlfsedH991 oxzivVRgRJFzgF0ciuZA6Vfr/OZh4Xwb8tskdO98potz1CkU5vyK3AlGyGKq/3HpjKVS8EfV28c AVpz2pvQDwDA3P7dPMb1qyOlI49AqpqjuS1lj3D/CWD/U75s5RauAW2NKjcs4PmKv3X64ndFHDI NvjIIsx4RGZaaPVebciJ2w4uQALrcrRVmr8zFuWkoyBBRzpUwu/VdnBhcQutevTEFdxDKF52oHZ wUrR7rpMw8QneAJd+3eCFS29XiQlcC286Sw8w= X-Received: by 2002:a17:903:1a45:b0:2c6:90ec:f601 with SMTP id d9443c01a7336-2cf03bd51b0mr23224295ad.8.1784117325413; Wed, 15 Jul 2026 05:08:45 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:08:44 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 4/7] mm/vmalloc: Extend page table walk to support larger page_shift sizes and eliminate page table rewalk Date: Wed, 15 Jul 2026 20:08:10 +0800 Message-Id: <20260715120813.3609949-5-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" vmap_pages_range_noflush_walk() (formerly vmap_small_pages_range_noflush()) provides a clean interface by taking struct page **pages and mapping them via direct PTE iteration. This avoids the page table rewalk seen when using vmap_range_noflush() for page_shift values other than PAGE_SHIFT. Extend it to support larger page_shift values, and add PMD- and contiguous-PTE mappings as well. Rename it to vmap_pages_range_noflush_walk() since it now handles more than just small pages. For vmalloc() allocations with VM_ALLOW_HUGE_VMAP, we no longer need to iterate over pages one by one via vmap_range_noflush(), which would otherwise lead to page table rewalk. The code is now unified with the PAGE_SHIFT case by simply calling vmap_pages_range_noflush_walk(). Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Dev Jain --- mm/vmalloc.c | 86 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 52 insertions(+), 34 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index b9b29f7857bb2..afc695d5973e8 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -541,8 +541,10 @@ void vunmap_range(unsigned long addr, unsigned long en= d) static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { + unsigned long pfn, size; + unsigned int steps; int err =3D 0; pte_t *pte; @@ -573,9 +575,10 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned l= ong addr, break; } - set_pte_at(&init_mm, addr, pte, mk_pte(page, prot)); - (*nr)++; - } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); + pfn =3D page_to_pfn(page); + size =3D vmap_set_ptes(pte, addr, end, pfn, prot, shift); + steps =3D PFN_DOWN(size); + } while (pte +=3D steps, *nr +=3D steps, addr +=3D size, addr !=3D end); lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; @@ -585,60 +588,90 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned = long addr, static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pmd_t *pmd; unsigned long next; + int err; pmd =3D pmd_alloc_track(&init_mm, pud, addr, mask); if (!pmd) return -ENOMEM; do { next =3D pmd_addr_end(addr, end); - if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) - return -ENOMEM; + + if (shift >=3D PMD_SHIFT) { + struct page *page =3D pages[*nr]; + phys_addr_t phys_addr; + + if (WARN_ON(!page)) + return -ENOMEM; + if (WARN_ON(!pfn_valid(page_to_pfn(page)))) + return -EINVAL; + + phys_addr =3D page_to_phys(page); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, + shift)) { + *mask |=3D PGTBL_PMD_MODIFIED; + *nr +=3D 1 << (PMD_SHIFT - PAGE_SHIFT); + continue; + } + } + + err =3D vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask, shi= ft); + if (err) + return err; } while (pmd++, addr =3D next, addr !=3D end); return 0; } static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pud_t *pud; unsigned long next; + int err; pud =3D pud_alloc_track(&init_mm, p4d, addr, mask); if (!pud) return -ENOMEM; do { next =3D pud_addr_end(addr, end); - if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) - return -ENOMEM; + err =3D vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask, shi= ft); + if (err) + return err; } while (pud++, addr =3D next, addr !=3D end); return 0; } static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { p4d_t *p4d; unsigned long next; + int err; p4d =3D p4d_alloc_track(&init_mm, pgd, addr, mask); if (!p4d) return -ENOMEM; do { next =3D p4d_addr_end(addr, end); - if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) - return -ENOMEM; + err =3D vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask, shi= ft); + if (err) + return err; } while (p4d++, addr =3D next, addr !=3D end); return 0; } -static int vmap_small_pages_range_noflush(unsigned long addr, unsigned lon= g end, - pgprot_t prot, struct page **pages) +/* + * It can take an array of pages which are not all contiguous, but it + * may have contiguous chunks, as hinted by @shift. + */ +static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long= end, + pgprot_t prot, struct page **pages, unsigned int shift) { unsigned long start =3D addr; pgd_t *pgd; @@ -653,7 +686,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, next =3D pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |=3D PGTBL_PGD_MODIFIED; - err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask, s= hift); if (err) break; } while (pgd++, addr =3D next, addr !=3D end); @@ -676,27 +709,12 @@ static int vmap_small_pages_range_noflush(unsigned lo= ng addr, unsigned long end, int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { - unsigned int i, nr =3D (end - addr) >> PAGE_SHIFT; - WARN_ON(page_shift < PAGE_SHIFT); - if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || - page_shift =3D=3D PAGE_SHIFT) - return vmap_small_pages_range_noflush(addr, end, prot, pages); + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC)) + page_shift =3D PAGE_SHIFT; - for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { - int err; - - err =3D vmap_range_noflush(addr, addr + (1UL << page_shift), - page_to_phys(pages[i]), prot, - page_shift); - if (err) - return err; - - addr +=3D 1UL << page_shift; - } - - return 0; + return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift); } int vmap_pages_range_noflush(unsigned long addr, unsigned long end, -- 2.34.1 From nobody Sat Jul 25 17:34:24 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30D48434E55 for ; Wed, 15 Jul 2026 12:08:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117336; cv=none; b=b88qejG2gkeI6j5jHG44nXpi2hCyTNEROUpkuhV6OpNyHYlqZKbGT9GNG4SQVjksBH9TAvdCsswUzuWB6txjIdZlhIwOnF7wD12t0Idg1aE4gI0DJYUXopcZE6wJsfgKa/O1+h/yBFUZNQq89pFZ7tTzn8mLKMx/xgQa8mUylcs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117336; c=relaxed/simple; bh=hCQPPrwoDsRjRg0GuJC1eZcI0rSRl6mPEj6BujTcWVM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=do/g/WtgtNVUlvPZcLr6H6nskjdCVpBvoFG3ghHFH/rjOmd3+Kd7PbONkxCeV9osiPxZvkogXOmTqsXWNadHSxTmTSJbY0pM+inGn0ZByM1cnHZLspEwRG3qn10D8YZQXadgkNzfry7SW1fQFlgRbVWyOb8xkKPKiaKB6TaWbRM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GlU4BswG; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GlU4BswG" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2ccf2360620so17611835ad.3 for ; Wed, 15 Jul 2026 05:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117331; x=1784722131; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=E3pHMdsibx37YzyzqXGwFL/dQqT2DUhJgZdMFQuT3hs=; b=GlU4BswGwaV1Uz1zkSoN8r291k6g3c5C2tCEtePShQ/uGlV+OQnL0c8sqhkm5pw/+z FBG7pL6ZBza5QJpt6FkBSjxjKDhJ8P4a7t6LxkH87SHhKYkd66m/eI4bi98UV26H9dme 4b5FsGxjZJDXiTdMJujQYO22tpo83JeP2lF8aQiaLJmTucjA58rBPmnN4pcrOiVSJmv2 AgrzvxwiDAoy3X4QIZqumpRZfeXUEL0sM74Rl2nTlY2Vyo90CjJwsxsOSaB78vThZqou 7Y2SogtmmooGkF0HFMCIVEQtY4qxncSP7jBYGeUEcRi41Ol8UwoYVnQW6YRQaq4fc00H I/VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117331; x=1784722131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=E3pHMdsibx37YzyzqXGwFL/dQqT2DUhJgZdMFQuT3hs=; b=MSTHsxk0glwdqRNHQgIm1fsuG0otNkhh29IVR1cXpjr3RNnY11gWVClnSRp+UMkQjz cyIBgdm17tfnypNGq+Dz7VZRUxGIrcNaeLBKEOrG3Wm/yJ0W/l4tjitcSiI8Ms/cH6Vy 3+QUy5PmTiaaLKBo94pT6hdLoTk8e6LrE3a9K+6qlpQ4f+ilnuwPh0Xh5yL2d1xHrhs+ LQkgXhycDz+clAyH8eEuTYCK33OWbIqIZcSbghIUIxLp7WxI547lrgFX48lWsA8Ztz+o 2UtiitkyLAP5JAPQBTfuwJjWYEJu0fA91CcbL68PPgBKK5/cJfGWAUcXvSDLJYF0nV+s X5ow== X-Forwarded-Encrypted: i=1; AHgh+RpdCyn57tcKh+2zroJzyEKXjth3TR3Mg33P0qBvih44R/GPe+dwLuZQnunej2LghZ7Va183eEX5xGOFeZU=@vger.kernel.org X-Gm-Message-State: AOJu0Yzuu+Q1Y8kfRnXFYweISayD14EZo3fVsLrBc9bM/HPSe07AGkPy +9GpioXUn6uFuTKErhqBggcGbA3Yo6l8ezcGW6BfvDWphYJVWpyVPHXJ X-Gm-Gg: AfdE7ckoombI7GS7KvOOWaCGlHI3N4Qr/WZ5zfRDYkLXVkFEf9yz3Rfnr2iuZR5ebIC 7f87gE6FA4/X4qH5+fdytl8c0Zl4v2qzr8Pbn3aATPQDubGZwuPe8/vHTvQCoiA/6ILk9w4s6Fe aKEjfZIIpMwabOp4W2tqOP3c37KyUSpOGT43igO3YuCOuSkpBNcvkSlrETTvVg2ogLwUy203qYS //Q5ZHK9dT8vigQ9+ItYybmeYyf8cBF9TQk6eDW/3gGiYyC8Urbjit9+AWLmFEZ8FkR0p0h7JBt nMStYyCmljBS7PBtNOduDOLYXhkRvxxhZvnXgd4k2tVvVkygJDhffhDYu4VVAlanOiRLeUIqzyA dSg20yl661fBSaHKQuC1n7M5iZ/nTjaQVouCdNLAFbr8ztVNK24NJtuz09l+j2D8cMZQuLPqKfq prCaP0oQl0XFzSSmRbHcdDKjLzFk9nS0m3nx1ZeoDlt87w7A== X-Received: by 2002:a17:902:d2c6:b0:2ca:cb8a:f26b with SMTP id d9443c01a7336-2ce9e5a5774mr161404365ad.10.1784117331030; Wed, 15 Jul 2026 05:08:51 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:08:50 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 5/7] mm/vmalloc: Extract vm_shift() to consolidate mapping shift selection Date: Wed, 15 Jul 2026 20:08:11 +0800 Message-Id: <20260715120813.3609949-6-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the vmalloc mapping shift selection from __vmalloc_node_range_noprof() into vm_shift(), preparing for reuse by the vmap batching path. No functional change. Suggested-by: Dev Jain Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Dev Jain --- mm/vmalloc.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index afc695d5973e8..a1e025120e9df 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3549,6 +3549,14 @@ void vunmap(const void *addr) } EXPORT_SYMBOL(vunmap); +static inline unsigned int vm_shift(pgprot_t prot, unsigned long size) +{ + if (arch_vmap_pmd_supported(prot) && size >=3D PMD_SIZE) + return PMD_SHIFT; + + return arch_vmap_pte_supported_shift(size); +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -4047,11 +4055,7 @@ void *__vmalloc_node_range_noprof(unsigned long size= , unsigned long align, * supporting them. */ - if (arch_vmap_pmd_supported(prot) && size >=3D PMD_SIZE) - shift =3D PMD_SHIFT; - else - shift =3D arch_vmap_pte_supported_shift(size); - + shift =3D vm_shift(prot, size); align =3D max(original_align, 1UL << shift); } -- 2.34.1 From nobody Sat Jul 25 17:34:24 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C3E8451062 for ; Wed, 15 Jul 2026 12:08:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117342; cv=none; b=DEEwd6qgkLR0E6GtM8/jDK5KWwNZbFdtd3Fz847MYO51u93hI+TjtGU7sf7rC26Xmb1MrvIeQ+MDpDK1P0sqc9NcBq1UT2EQzZ6FHTsUYPASQ0QMZf1b0DN0ofxrI+lm5wvAQqQED/zKKR3xZUPa8KNymumnnb7fbhzgZDTXuRI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117342; c=relaxed/simple; bh=RXjyYv08xorg4GntxMS8ChvKvuX96S5jNDbgjdUo19A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=RyX6/WrYUhmaKmCzziToD+mRkxdHu7r3iXDjxQoPwmqZtj/fMZPzygcvluwHB3VomIQsMBUZqwdha8UH/kmDJUgjaq9HXweBglQZNtG1Q7BI+W6Ml2b8dNoYPcLgi/FoRtF1eHUvqibGznUm86GmvF5dmxVTYVmxzsh5sMnrA48= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=sZmMVW04; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="sZmMVW04" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2cee9b74ee1so19310365ad.3 for ; Wed, 15 Jul 2026 05:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117336; x=1784722136; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=93J8dJ31kBsJOKH3R1rbCLn8YwyTsJsj4AeCzglR+bg=; b=sZmMVW04ZwKc9A0WKkeQcgPaDN03j/I4fBDzP60vn4nh35DjNoi99Zclo3/hbBJ4bV ZnaxHPf6A9is7gcEfrBEr6HSEt0qwFgsBkDcFzwpxLjIMIkeN2/zIWmWGXAX+QG2uX7P PVIu4Yn7S6KhbLw2jBZZM8XRN/oglXmOf43+7ItAiyP8T54bNtZ+qJ35sEDQ/FwANoni JSq35XJXTCAkpQqLzxCq3P6BP8aZH1hpLNra7/FK3qwBv2KNQWLIHPS/WXmfb9GKGBhk 7pG6CI2OeBx8WdchZUEemudDyQqzd//ZLBb8l9h96xot/21M4zMDnK3PhCc2fyuvl1ko 4Fwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117336; x=1784722136; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=93J8dJ31kBsJOKH3R1rbCLn8YwyTsJsj4AeCzglR+bg=; b=khzoeID4J0UNThhKY5z0hoIPQ3wsPS2gytoLtF3drhWpNEEB6a82GgYpqi60RnUSn/ M19EI8JHdvXEgyugtWOAciVzU3QrrQa8pgPtFmoNsdUEyP/psQcoTIRAJ5VTmXb2p6wu q9nkLk51lf0V5jvQtiheaNcajhA/n11ipcFzcrVTbI8qDsoZTTWY9GeJTXljcPhWXl8+ JW24ePVgChMKYHmJbPF8X2Gnwkoy/Jw2uqVHz62M29+i/4vbMPtyFSPT2eeOT7Ubei8s K/bI4n8yuRNd9aPUIg/PZJMUc7f3rHmjFRfSGjJzLjcKO+wTR2cgGvnxD0TfiVEOTxt3 tLxQ== X-Forwarded-Encrypted: i=1; AHgh+RppTyuLdhI06Eds2tvEA6RL+J8lS1eM3J+DQssyJNKaHOFNBMhMu7hPZ0V3bnL+lVivlYgOHeqSqU/n4Lc=@vger.kernel.org X-Gm-Message-State: AOJu0Yx9XhlsY/QnBLzFX5q95bOpL3HGCvCaTtphWqdo7ZBoHEqx5Dv/ PucrNCQyTPQkwA3BfETWUQKFxAdLq05WfeVw1MLgtoSqd/7IB67hlHOM X-Gm-Gg: AfdE7cnQeSnECaDs8gM4gZrNuRZ3mnj9qOZ46ZfHg6ZmyeppHNvEocvQaxjiyLYwX4X oBB0Cs2JJ6CR/5HsNWlbB5PS8IajbbGWcZPTzCpCaemt4XjMRP2Zck/skJYP9OsznNsAwAZwvCV v2jggmfAEIADxXX574Qd0BNooJL/Op7tZ1qmI/7zdJ+jGLxNNu64LMmGa+HTfV1huFDJfdPKhRA bQ8GZBki+y6NWO2MlCYjzdDhQqnvy2LfmxQxqDCKeiflOW6mdmr615JCitjneovVVacCK4jezOh pzL9T6W85vAJ+VvxshcTalJhq41OtARlsry4BQWAQWRy105HJ6fG6WWLY66Trt1d4PGW9d7p1Rv 5WjFMp9VCtj/58VvRv7htOGHb4TLfxPgckDCsEIOhJ+/7bd4a+Jlak/Yy9/V0WcoiehIembHDg7 0DNsSOSYvIyy1V/hDNOFV58DdNgRW/XG9Iy9E= X-Received: by 2002:a17:902:f686:b0:2ca:3df2:919a with SMTP id d9443c01a7336-2cf03dca19fmr22554225ad.33.1784117336405; Wed, 15 Jul 2026 05:08:56 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:08:56 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 6/7] mm/vmalloc: map contiguous pages in batches for vmap() if possible Date: Wed, 15 Jul 2026 20:08:12 +0800 Message-Id: <20260715120813.3609949-7-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: "Barry Song (Xiaomi)" In many cases, the pages passed to vmap() may include high-order pages. For example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently, vmap() iterates over every page individually=E2=80=94even pages inside a high-order block are handled one by one. This patch detects physically contiguous pages (regardless of whether they are compound or non-compound) by scanning with num_pages_contiguous(), and maps them as a single contiguous block whenever possible. The mapping order is determined by taking the minimum of the contiguous page count and the pfn alignment, allowing graceful degradation when pfn alignment is less than the contiguous range. Pages with the same page_shift are coalesced and mapped via vmap_pages_range_noflush_walk() to avoid page table rewalk. As users typically allocate memory in descending orders (e.g. 8 =E2=86=92 4 =E2=86=92 0), once an order-0 page is encountered, we stop sc= anning for contiguous pages since subsequent pages are likely order-0 as well. Signed-off-by: Barry Song (Xiaomi) Co-developed-by: Dev Jain Signed-off-by: Dev Jain Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Dev Jain Suggested-by: David Hildenbrand (Arm) --- mm/vmalloc.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 80 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a1e025120e9df..92f9cd5e9def5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3557,6 +3557,84 @@ static inline unsigned int vm_shift(pgprot_t prot, u= nsigned long size) return arch_vmap_pte_supported_shift(size); } +static inline int get_vmap_batch_order(struct page **pages, + pgprot_t prot, unsigned int max_steps, unsigned int idx) +{ + unsigned long pfn; + unsigned int nr_contig; + int order; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) + return 0; + + nr_contig =3D num_pages_contiguous(&pages[idx], max_steps); + if (nr_contig < 2) + return 0; + + order =3D ilog2(nr_contig); + pfn =3D page_to_pfn(pages[idx]); + + /* Limit order by pfn alignment */ + if (pfn > 0) + order =3D min_t(int, order, __ffs(pfn)); + + if (vm_shift(prot, PAGE_SIZE << order) =3D=3D PAGE_SHIFT) + return 0; + + return order; +} + +static int vmap_pages_range_batched(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) +{ + unsigned int count =3D (end - addr) >> PAGE_SHIFT; + unsigned int prev_shift =3D 0, idx =3D 0; + unsigned long map_addr =3D addr, batch_end =3D addr; + int err; + + err =3D kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + PAGE_SHIFT, GFP_KERNEL); + if (err) + goto out; + + for (unsigned int i =3D 0; i < count; ) { + unsigned int shift =3D PAGE_SHIFT + + get_vmap_batch_order(pages, prot, count - i, i); + + if (!i) + prev_shift =3D shift; + + if (shift !=3D prev_shift) { + err =3D vmap_pages_range_noflush_walk(map_addr, batch_end, + prot, pages + idx, prev_shift); + if (err) + goto out; + prev_shift =3D shift; + map_addr =3D batch_end; + idx =3D i; + } + + /* + * Once small pages are encountered, the remaining pages + * are likely small as well. + */ + if (shift =3D=3D PAGE_SHIFT) + break; + + batch_end +=3D 1UL << shift; + i +=3D 1U << (shift - PAGE_SHIFT); + } + + /* Remaining */ + if (map_addr < end) + err =3D vmap_pages_range_noflush_walk(map_addr, end, + prot, pages + idx, prev_shift); + +out: + flush_cache_vmap(addr, end); + return err; +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3600,8 +3678,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; addr =3D (unsigned long)area->addr; - if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { + if (vmap_pages_range_batched(addr, addr + size, pgprot_nx(prot), + pages) < 0) { vunmap(area->addr); return NULL; } -- 2.34.1 From nobody Sat Jul 25 17:34:24 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70F1B46AF0E for ; Wed, 15 Jul 2026 12:09:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117349; cv=none; b=jBdOTnzA3eCkX7JD0jznO6Br7OQGQuVAcrHlwd2EsNzAIGYeyZzE5g+70jybmFFKhbONQ5LFV3FZMA+x446O62kpFPXIiJi901w5WlMqPvHSfmMlNGrgrJy7WCfzbsIuMNwMG/G1Gltt0XUXmwWCf94pJH481d1QzcglUI9NL+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784117349; c=relaxed/simple; bh=7kE0M9O83eoa5l07cHU/MrTyDkd/3c8aQLotCPs7vfk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=faAz0izS95Ptwuj9IX18TGcou/oZE2TsXvkgJetXW86ivDeo23hdRE4InO7sZCuYOdiKPBGfT1d0Dlj9xocImTsakFDY7AlcUQ4ruey8Mc9577ckBPnczsr0hW5Mv6ziIcxZ4NkH1UFyqCos3BYAVEZ5EfdYv6iI490l6aXTfgQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AZwXhd7A; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AZwXhd7A" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2cc7ef7ec27so61734565ad.1 for ; Wed, 15 Jul 2026 05:09:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784117342; x=1784722142; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=v8qJhOk4bnpu1WPc4oSH+kzjjMIN2seacJoWKksNoa0=; b=AZwXhd7AhUCog1x62ar1338GTC/0gBVRWWzWmzWdVlSU5UJ9uKd/JHkTLV/T/TG0hV w9q4DMqHOW3slFepvKzYjEcqvoTB7djDRJ6G8WfxB35tbcY6s1pwcZUq3R5lGpg+EICu oEpU+e0k+NaEgQfz6f1LvUQr/lADWhCMmu/pxQTx7OWS7P6sBTjcIgea/v+8q/tPBF45 6V2oJG0l0J0QON/QbhkEB+rFfwTEoURQny4y9FlKEYsBAZz2J0P1S3VxA3VkRtknd++c MBD50O04lDH+AtjgJcQDwgECAs9dOJziKrGr9ATmTy67pDEn2Bqa6wlnUi1ApbTVbX6U srXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784117342; x=1784722142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=v8qJhOk4bnpu1WPc4oSH+kzjjMIN2seacJoWKksNoa0=; b=HzqSUllRcUK24tYpQ2/APF6s66xRtvUxMNf/QxV7b/hI3cqs5CRkqm9SGtK/vqv+gq 3jXhVGkixLLvPcjL6fEIxjLZ81/0zmqvqemO3izfSe4od9/izjskwfEftwr0qDvhlfy9 9rfoeuRb2agkN/GDUUHI6p11Ri6nYuGVa93mlvd7t861gfvXnblPjV8JX1g8qF47I4zV lEPyqaQo6HyLmFp/hlA2LsbRK2uR0PSAmjQbMoCVtBMqcMZfjVEABkqUITlCCQjsI/Kh Mf9dTJxCl99U4/IUCEeS4kQjD4woW91pUDkPiMKB2cU5eVcavAjFZegHQYagSkM9bGlb 5viA== X-Forwarded-Encrypted: i=1; AHgh+RoJGeyLHxImtNGqG9Fgwee60C2M0OcZs9wXxXvGuxJYhY/zXrfeJfe7UgRrToC8nJLouvMaHp2nogsHE9Y=@vger.kernel.org X-Gm-Message-State: AOJu0YxTm7ju28qFkMZM0CbgiDYAHxw7/Y6XqqIeL9lYr/24Ggfm23tU x/X4hK0B6WCcXyZ5RlR/tqRKURI0Fkvmkve1WDmWdI42cy2nO8/15Cwk1jbgRw== X-Gm-Gg: AfdE7cmP/blYHY4aNDbs9gBc5gouvIDG5/RD+dNj6ZjXfmrv0ZeLYTWUaE5stlkWurT MVmb7DwgSsDTK2pYfrKwWd343xtfZ1uSHyoasbrxHyNBvGQrJdhJhnCIL63637/m7TJsn7jnrc7 hCjqBeVdYFPgEXecfYL7fgv3vvR5f3XG0jd/MuRMe6/jQRfV7njHfXn1j8azSxZx4xdBzz1h/+g AfnLvwXsb61X63MVJ/9NixKfDf/N9rqQmcBfSXGXrmPJcirTYWrvv+h3j8k4cc6XCLMFziKoanF UlO6BRglOXHvBsUFg+beuJn1r/gwIWFZFJmCwSyfRVZ0D7KebCfu0HCH+JwZyzV7kBkNHk3skRj WmQsdoL5UFeeUhTT6sQBiJNXHxDy0kXtnlPP+a+Fm/B9J/cQWn3bPjK56gN1dVF6+jVBIDN1iYt OV3oOHQdPha1F8XnDm1sNsnQbJaIVP2Co1At0= X-Received: by 2002:a17:902:e5c3:b0:2c9:e6d7:fbb4 with SMTP id d9443c01a7336-2cf03cf6203mr22618325ad.31.1784117341728; Wed, 15 Jul 2026 05:09:01 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf105bc642sm1216325ad.63.2026.07.15.05.08.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 05:09:01 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, david@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, rppt@kernel.org, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, Wen Jiang , Xueyuan Chen , Leo Yan Subject: [PATCH v7 7/7] mm/vmalloc: align vm_area so vmap() can batch mappings Date: Wed, 15 Jul 2026 20:08:13 +0800 Message-Id: <20260715120813.3609949-8-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260715120813.3609949-1-jiangwen6@xiaomi.com> References: <20260715120813.3609949-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" Try to align the vmap virtual address to PMD_SHIFT or a larger PTE mapping size hinted by the architecture, so contiguous pages can be batch-mapped when setting PMD or PTE entries. We do not expect any significant overhead for this. Add __get_vm_area_node_aligned_caller() as a wrapper over __get_vm_area_node() to simplify repeated calls with fixed arguments. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Dev Jain --- mm/vmalloc.c | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 92f9cd5e9def5..7a3aa9e5c885a 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3320,6 +3320,14 @@ struct vm_struct *get_vm_area_caller(unsigned long s= ize, unsigned long flags, NUMA_NO_NODE, GFP_KERNEL, caller); } +static struct vm_struct *__get_vm_area_node_aligned_caller(unsigned long s= ize, + unsigned long align, unsigned long flags, const void *caller) +{ + return __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + NUMA_NO_NODE, GFP_KERNEL, caller); +} + /** * find_vm_area - find a continuous kernel virtual area * @addr: base address @@ -3635,6 +3643,30 @@ static int vmap_pages_range_batched(unsigned long ad= dr, unsigned long end, return err; } +static struct vm_struct *vmap_get_aligned_vm_area(unsigned long size, + unsigned long flags, pgprot_t prot, const void *caller) +{ + struct vm_struct *vm_area; + unsigned int shift; + + if (arch_vmap_pmd_supported(prot) && size >=3D PMD_SIZE) { + vm_area =3D __get_vm_area_node_aligned_caller(size, PMD_SIZE, + flags, caller); + if (vm_area) + return vm_area; + } + + shift =3D arch_vmap_pte_supported_shift(size); + if (shift > PAGE_SHIFT) { + vm_area =3D __get_vm_area_node_aligned_caller(size, 1UL << shift, + flags, caller); + if (vm_area) + return vm_area; + } + + return __get_vm_area_node_aligned_caller(size, PAGE_SIZE, flags, caller); +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3673,7 +3705,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; size =3D (unsigned long)count << PAGE_SHIFT; - area =3D get_vm_area_caller(size, flags, __builtin_return_address(0)); + area =3D vmap_get_aligned_vm_area(size, flags, prot, + __builtin_return_address(0)); if (!area) return NULL; -- 2.34.1