From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06C7043DEBC for ; Wed, 23 Sep 2026 06:28:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144929; cv=none; b=gCyzTV/HIKVv+x2ZuGTY72fBZ9wtfd6PzgMiUmyFgaU1PhuDQpGMuHMmwC1I+YpRzZVa0xbt/5KTEfVKG7/XXCiFMxFRGpJpulEyuevmew2Vh3Q+7euGNtJX8xpKobcFbrYekzrp70njxG17hxZkBoiJpjE+RJeV0pm+OZzrg1g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144929; c=relaxed/simple; bh=TZIuyM87cT1jcfLZCH3Gw/sQ620Bhi+JwxUTofKDHIU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=q6UO8y4zqZjByrHkggqKy4r+bVOOQqhU4hbqQVIMdjl7Een28eeiFqphxesryQrhZu2S0+5yJSvpGQkHuMpZvO0rTLjK0XjsUSKhHv8oOu7/hxaJluVxaE/ie9kqykCJ0qRA6bfSpJW+3KZ3PN2pxGunsrjsNQcFNVKBpP1B5Ew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=n8gJCFBq; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="n8gJCFBq" Received: by mail-pj2-f12.google.com with SMTP id d9443c01a7336-2db22383fe8so2352745ad.2 for ; Tue, 22 Sep 2026 23:28:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144926; x=1790749726; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7/fcB7RWsYoduiGVrXMcsoy9SKXwqPZ/B+Usu82ABAo=; b=n8gJCFBq66UpQqJLP8pvq8e0uwC93oJwq15KLVHFG7K0xKY7K/CQ9q1tJbhw4PF2mA 6ATT0wxN6K0TnqdUnBOMriG2MoEDt4ONoK62u7/A0Re4lW/4OYeq6pVpGyvgUJwZjDWG jTkw/5vhGWzyISIaN0+MtYYqJy6P/PjZdLfClya38G3hFJnIopTPQB3GeSIU1/Y/uQO1 6O4r2v1ewk/wi4IsMXVpNW+/x1ddMYbGt1s9Y0rCVFY9clEr0pNhqBveCgfB8R8UH659 mfsQAm9MZmHy1Vt+4zjohvPCzFc/MeMzdLq48RaHQtmhQWQyEkoj0CjSKGFGp9sfZsfa dLkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144926; x=1790749726; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=7/fcB7RWsYoduiGVrXMcsoy9SKXwqPZ/B+Usu82ABAo=; b=Nb7ga3k4A9zzOOdFgUvEu35NxtgurEmbcBBVgvtKTP5PSSNUI0HHcT5VXxanpeR/KF QmMlW7MpCwung/4BRbUSVTggFqC/dWQoD4tZSfrBTRWzbQJRn1pIgKEFkcM9vPg+eORZ m0sTMd8hjq4rT1/Ldw83L6+MihWR5UZDWtiN0+hD2N4JPwkansJqMd2LXJ3AtpRpdzez PgdqSaW75Audc6762e74qv0qS/HWmdzd9RdO0j4G+A+F9hon+2US+gH7t5fiC44E6EH1 ZItEC+E6rZhHu2fZ6p4tkt+H65GR+pqBrk3Zq+98ttY2Qg7+ZlUPM1Sh+3jgob9Nxi7C S7IQ== X-Forwarded-Encrypted: i=1; AKwUvBxF4Xir6cPhhLBS+E0zTiTiIN9g/L+awBqytgbF+LmKGANsVla634l2Nopkq5uuxdPCgFvcsjvrrSCIjIs=@vger.kernel.org X-Gm-Message-State: AFuF++krLSBELToWuHfXD/YvdbWjIIcO9wyCutgGlcyM5jh2d1olqW0z u4Q7INlNaHi9c4BnBtguPAdRv7Y2288eSMlKW4yxkvA0CLznYn4oTOB8 X-Gm-Gg: AYBFou0WHM62bjVK4qFKOQlhQvQSt5bmKnh061e6jLxukIA93mz8WK+l6LQPrgKZnAF 4Cqy0IVTyzKpVZhazcSRF9/7vbjh9AabVTjtlrbP78A960r0/I042WY+KFR/7ebHGOxc53XH6A8 drZyK2BPTbfxK2qFPizdIRJqJvIJJ3d3Fg1G54nmz+Q9SVh2w2sJ8MsZn0hJwkX/2snHjIa7hV2 ZXyUZGKlA3LHW7mDfpqS8T3wV0qQ5ZfDF8XPHCBJ34bzS+PQEvJi5tmYaHzPcHJ1eYllh1I5Gl9 LarcJpSbYOKC39cd5smfDShbWrJrMIT9v15KlOGMCuUAQQtTSTy9+A9h+k2j1qvD8xogAlYhPdc 8nXJuwLi0J6KWM/RYVh7jqepUevZz7x+yKjs8XXlgypg37X/vNh9vd+9Oluhn6eTaCDJTNlG7JZ k6fyIV2witreKEVEqixK0QrPpa5VnNCqvilHzxKP6o9jJd3HKSzXMunt6176uB35pjPHH+nG+q5 6wCej65mRQ4JpvUdHEtwA== X-Received: by 2002:a17:902:d2c2:b0:2dd:c0ff:e724 with SMTP id d9443c01a7336-2df69de446amr13040735ad.54.1790144926198; Tue, 22 Sep 2026 23:28:46 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.28.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:28:45 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 01/10] arm64/mm: add pte_set_huge() and pte_clear_huge() Date: Wed, 23 Sep 2026 14:28:23 +0800 Message-Id: <20260923062832.479455-2-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vmap installs PTE-level block mappings (PTE_CONT on arm64) by reusing set_huge_pte_at() and huge_ptep_get_and_clear(), which are HugeTLB helpers gated by CONFIG_HUGETLB_PAGE. This makes the feature silently unavailable on CONFIG_HUGETLB_PAGE=3Dn kernels and couples mm/vmalloc.c to HugeTLB internals it does not otherwise need. Add pte_set_huge()/pte_clear_huge() to arm64, joining the existing pmd_set_huge()/pud_set_huge() family in mm/mmu.c. They build on __set_ptes() and __get_and_clear_full_ptes() directly, so they do not depend on CONFIG_HUGETLB_PAGE. There is no caller yet: mm/vmalloc.c is converted later in this series, once the generic fallbacks are in place. Signed-off-by: Wen Jiang Reviewed-by: Barry Song --- arch/arm64/include/asm/pgtable.h | 6 ++++++ arch/arm64/mm/mmu.c | 20 ++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 6000905a2e865..9b762eb68cff0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1978,6 +1978,12 @@ static inline void clear_young_dirty_ptes(struct vm_= area_struct *vma, =20 #endif /* CONFIG_ARM64_CONTPTE */ =20 +#define __HAVE_ARCH_PTE_SET_HUGE +void pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys, + pgprot_t prot, unsigned long size); +#define __HAVE_ARCH_PTE_CLEAR_HUGE +pte_t pte_clear_huge(pte_t *ptep, unsigned long addr, unsigned long size); + #endif /* !__ASSEMBLER__ */ =20 #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 79d90226fd5dc..4a3d084bb7556 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1912,6 +1912,26 @@ int pmd_clear_huge(pmd_t *pmdp) return 1; } =20 +void pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys, + pgprot_t prot, unsigned long size) +{ + unsigned long pfn =3D __phys_to_pfn(phys); + pte_t pte =3D pte_mkcont(pfn_pte(pfn, prot)); + unsigned int nr =3D size >> PAGE_SHIFT; + + VM_WARN_ON(!IS_ALIGNED(size, CONT_PTE_SIZE)); + VM_WARN_ON(!IS_ALIGNED(addr, CONT_PTE_SIZE)); + VM_WARN_ON(!IS_ALIGNED(pfn, CONT_PTES)); + + __set_ptes(&init_mm, addr, ptep, pte, nr); +} + +pte_t pte_clear_huge(pte_t *ptep, unsigned long addr, unsigned long size) +{ + return __get_and_clear_full_ptes(&init_mm, addr, ptep, + size >> PAGE_SHIFT, 0); +} + int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) { pte_t *table; --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f42.google.com (mail-pj2-f42.google.com [74.125.227.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB43E332EC5 for ; Wed, 23 Sep 2026 06:28:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144935; cv=none; b=d3g60vhqQI75TVXwXiazyEV+HfEByDQrqCMGBd9icVtdDK56Lem0jHtaFGzoq7KhEx2NmfIsgGTiFNRla5aIsBkfs89TOQlv68i+ozR7ir5LpA0rkheLNFeQY1W+OFrf163V+tBVYDEp8el2i3OP9mQ1znw+4XNon66u8y+ocnQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144935; c=relaxed/simple; bh=eB5nUweJYyasHZvrL5U2pDBzT4tzqm9PVbMd4iLFcz0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lMNlEi70WEFd6EH9dwBl1JRQN+Uia0MlVugV01nrJMAuX02q44AoloZxcfsAR2RnNiTR2BzF8r03cxBBV5q8yXsvdo8vlbZEnQfK7xG9wwZwJuZtEK3kUB1cXI/JJ+aZMKeht2Xk2Bc1M1afOkgx6ukqvxW/E57g+MGlkKN03Dg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ppKWw3n9; arc=none smtp.client-ip=74.125.227.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ppKWw3n9" Received: by mail-pj2-f42.google.com with SMTP id d9443c01a7336-2df4c9d14b8so1832455ad.0 for ; Tue, 22 Sep 2026 23:28:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144931; x=1790749731; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Qd05crYEDf1zOBsfrx9KiTFMAUGQioHKS/Nf5VbEp9U=; b=ppKWw3n95pJdp2LLRqC+kpaWqsYSL1ZsqmvA5rY3KEOu/M0tigwhbXXPnE1lL7m0jR e1B6zlVs0I/K83vDEfbt0RwlnN4jmmQzBQ9TFwsAJXIyjYns7UwstcpBdOjYTVNOE/sq BFtpCOtyX2bU8qC6PTKwYuLgcx2Krf6D275U1ScfwdqvCmD+GmHsQIszMRyoMMZ4OtXd R1j3jcXhbA9hE1aw6PMRbuqvR6w6cQYf1UbwOn65hxk77qvVGw3gMKh9x/ZS+IxKNP+1 s7/w7gewobMJ0ZKTTB7KjAocNOgf/M9WmU9zgNiyWhvraidWGBJbiHe3Bn+Arcz3gJ3d Fbaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144931; x=1790749731; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Qd05crYEDf1zOBsfrx9KiTFMAUGQioHKS/Nf5VbEp9U=; b=YGuUHDdCD31zKTh0ukJs6LcfNThIry5FCMIaD/mC35/EOhMm4m1c2DlILf2ll1bQq0 fQ0H+y4r1fZ1SWwXyGoeOLfNUjpq/rniyKA1VpUQgVG57+6jR0AA9VFr3UqJhD1MN6Xd Sw0x34XjerfLNwQTNdTRLdv3nGKa0H6FwjSY4tINTZr+23dyMTGoPLdehyYMQeK1Nxh3 W5gathQO1J0Xe3TEXiiuFQGbcUmkLz8ftcJ4yqghwMfxFBUMO64/pKiYLpH/ZKKBRYpO LbMWtqBYn1C40EZaqkzsXYtHNehZpfe06l1wO0KranrPyOiw+/XVpTinpUI0CjxV/3sX MKIw== X-Forwarded-Encrypted: i=1; AKwUvBwBsBP8jljAXShl92Up7TaqpJmQtLXqIJiRgwLAJURkmHlNE8/PJUwHr81L/ozVvEDwBHXYPLDHIuXOv54=@vger.kernel.org X-Gm-Message-State: AFuF++nMWNF55hiW8ujvnaQqJaz0AjzF5RP0QbkW9EZuK1lCgOkchoBL KaMDooaQYoTVw6LocD1cRQ0biafxJaWtehKlhT3ikX7/oOxTyQPobQdscjEqkzxR X-Gm-Gg: AYBFou1s/H8tQBhWAsEzuTwRgII9Saej6mWZ42FHpk/po27Sg6GVD3dcTB0BfOv0v4d rV2p5LnQuXcsAr1dSqxUnMTuPOVmAPADBAfgV/WI/MsVi7By3b5yNsk9GWH39U3YxB2EM2o7oqF ArrEaaQfpf0jr8wWxaAENRY3oA5+u2XbGTS0RnD500+1ewlkBM9BPT3zWH2L10hXiUsx+yPVMSl QyPQz7FKbjAj2xSp7Q+cxSLgj/evhYbOBrXsYzCRqgEr35qy+xXF/e3+NlpDUv/eJJnbKaNnBWR F5+P5SnaHcHXb2EJjjKLSimJKrVdiSmYyn+4vjq9Fs7+9vxZlld7tDLj0Q4JI2s9/7Z7G2N+nDE Fk8dcsGdy2S7uVic8qORz/xRMkt+cq+MiOT6jGohWdnNRCqUOgonrkE6JrO795D24Xuxf39nngs 8QOWxCMvnT476JI5smyj8rlozbY5qCJitWW5rxARy5gLBPNStWk9u9lXfvrdeXqXkj/oDRXNCC3 mSsIM6e4h+5pFn2hqgjKw== X-Received: by 2002:a17:902:d4cf:b0:2cf:b9f9:18bb with SMTP id d9443c01a7336-2df6987811fmr11433845ad.4.1790144931608; Tue, 22 Sep 2026 23:28:51 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.28.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:28:51 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 02/10] powerpc/8xx: add pte_set_huge() Date: Wed, 23 Sep 2026 14:28:24 +0800 Message-Id: <20260923062832.479455-3-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vmap installs PTE-level block mappings (SZ_16K/SZ_512K on powerpc/8xx) by reusing set_huge_pte_at(), a HugeTLB helper gated by CONFIG_HUGETLB_PAGE. This makes the feature silently unavailable on CONFIG_HUGETLB_PAGE=3Dn kernels and couples mm/vmalloc.c to HugeTLB internals it does not otherwise need. Add pte_set_huge() to powerpc/8xx, next to the existing pmd_clear_huge()/pud_clear_huge() in mm/nohash/8xx.c. It builds the huge PTE and writes it into the backing cells directly, without going through set_huge_pte_at(), so it does not depend on CONFIG_HUGETLB_PAGE. No pte_clear_huge() is needed: 8xx does not implement arch_vmap_pte_range_unmap_size(), so the vmap unmap path never takes the PTE-clearing branch. In practice PPC_8xx selects HUGETLBFS unconditionally, so CONFIG_HUGETLB_PAGE=3Dn does not occur there today; this is a decoupling cleanup rather than a new configuration. There is no caller yet: mm/vmalloc.c is converted later in this series, once the generic fallbacks are in place. Signed-off-by: Wen Jiang Reviewed-by: Christophe Leroy (CS GROUP) --- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 4 +++ arch/powerpc/mm/nohash/8xx.c | 29 ++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/arch/powerpc/include/asm/nohash/32/pte-8xx.h b/arch/powerpc/in= clude/asm/nohash/32/pte-8xx.h index e2ea8ba9f8cae..1ee3f8f0de974 100644 --- a/arch/powerpc/include/asm/nohash/32/pte-8xx.h +++ b/arch/powerpc/include/asm/nohash/32/pte-8xx.h @@ -235,6 +235,10 @@ static inline pte_t ptep_get(pte_t *ptep) } #endif /* CONFIG_PPC_16K_PAGES */ =20 +#define __HAVE_ARCH_PTE_SET_HUGE +void pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys, + pgprot_t prot, unsigned long size); + #endif =20 #endif /* __KERNEL__ */ diff --git a/arch/powerpc/mm/nohash/8xx.c b/arch/powerpc/mm/nohash/8xx.c index a9d3f4729eada..2518d2a6bdb41 100644 --- a/arch/powerpc/mm/nohash/8xx.c +++ b/arch/powerpc/mm/nohash/8xx.c @@ -220,3 +220,32 @@ int pmd_clear_huge(pmd_t *pmd) { return 0; } + +/* + * vmap PTE-level block mapping. Sets a present kernel mapping directly, + * so unlike set_huge_pte_at() it needs neither CONFIG_HUGETLB_PAGE nor + * set_pte_filter() (a no-op for non-exec kernel mappings). + */ +void pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys, + pgprot_t prot, unsigned long size) +{ + pmd_t *pmdp =3D pmd_off(&init_mm, addr); + pte_t pte =3D pfn_pte(PHYS_PFN(phys), prot); + pte_basic_t val; + pte_basic_t *entry =3D (pte_basic_t *)ptep; + int num, i; + + pte =3D arch_make_huge_pte(pte, ilog2(size), 0); + val =3D pte_val(pte); + + /* + * Make sure hardware valid bit is not set. We don't do + * tlb flush for this update. + */ + VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep)); + + num =3D number_of_cells_per_pte(pmdp, val, 1); + + for (i =3D 0; i < num; i++, entry++, val +=3D SZ_4K) + *entry =3D val; +} --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A10343D4EF for ; Wed, 23 Sep 2026 06:29:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144944; cv=none; b=qGrzkH6pdIAhkLWy9qCyZTbz2FnLPzI6VfpQYEMQcqyryvEUINWn1iF9gI1v+u/b7t4UCHO5x3JonYtJjdWaZjoCPU3/zIgViVi8sf0CsaOAD0qYGrNkiAJz8zeoBZhoI8DDfFI8hiP1giRPkxDnQu8wMRYW9M5ISRz0fGhZZ+4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144944; c=relaxed/simple; bh=P3n/0wgc3upMk0hBSxFRBopoPCpanSNn6Oaz90YMgOc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UVggjRrlnsZsegKTGeQ4EmHzRUOeUOwaQoCPnfgQPgbBEh1NADMbyyVqCdZMa0CiUaSdC5GDzwTupMiocXuhkE239SeBIZmqDFZzXgZTrOIhC+CzIDUqSdXHgVPm5eV1EvfH94vYdlfj8yGEHODRPR+6yo9vjlnl84AVfu/j2Pk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nJkXotoz; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nJkXotoz" Received: by mail-pj2-f12.google.com with SMTP id d9443c01a7336-2d747f0135fso3966455ad.0 for ; Tue, 22 Sep 2026 23:28:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144937; x=1790749737; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=aFifGOrpSVWAs42j7AiNSWIOQBImLaH7pX+Uvfxk6V4=; b=nJkXotozYLtP5Plot6l+WBktZ0ny5fJ2QSa45lA0DGH6th33kdF17JbDObaPG6VhPx jki88w/D2Mftucc7yBQ5XPfIIGOfpRiXDF0kgI3HrvHbWK9iXbEpvhdpU3wFi6Ff+MQH bDy+M5vTBtor/MQ+iZZpb72Q3qS3l7SJjZQyU8thtWxBBxmDzI94RoPiKIWEXVfdjxMK V2MFSl7Ssr6AYOhhpzNMfe/I06sLQ7rDLjn/6/wErDNDW7CkyWeulWp+VVhwvGhmFBPu i344SuxiSCDdJUXMIWzwFjW73FvdKHyzwTbeX3Nf+bYCpsL5nvZBMDFGGKsxnPZwvYjp RKbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144937; x=1790749737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=aFifGOrpSVWAs42j7AiNSWIOQBImLaH7pX+Uvfxk6V4=; b=mmQf2nWvZ/y5Kc3JScNzE9f+nm3aA3K2WYAZZ1lRGeG7iFtabqujxhKbaOz6DuWjRO n3J9HUMhCFZwek/TUF7Cq26FncidHvDWF0DLw3hC96BALlONsN9I23RbfvIWo1hiARpC WZZ/1ttDrCBhFTSL2VBK4Emms4N+CwPahv8jwxMZTdnqbzSGbHStXbvhQSNrTfYCXPwn MQyRa258GUr6PMfNUPavQAYB/hCYDBYJlxYoMNBzv4yzqBhZBco4m1JdiuGnGwA5VKOL JRvT714xXay3mWbzp0epkgrxyNsglurK35EMWIilAehRaO8PMlFVjCCspDKCxcqUbvSw kaQw== X-Forwarded-Encrypted: i=1; AKwUvBysHVC4bppF+tKtO/DMNwOR0HeGiHnHUa/Hoy68w0v+9iNg+25aIro64bv8TsvfsKxwHfl4s3fJO57PBWw=@vger.kernel.org X-Gm-Message-State: AFuF++muUqvyShnyDDKI5mIEe7a7vekU3YwmGin2NeH8/yqupDbs8kQn o8BGwy0SCFBuZxF1CU1QQWx7Vs5d758e9w6vmRraqbtj35LuNcn4y67K X-Gm-Gg: AYBFou2HBF0VIJa8T7uePpWjJPr3dJPIBBWYLOjn84gPzxQP02wNnde4Yxzj9BPWkKC f+HbzTysZPBoR1eC4vvssrUtlCO1mHmbEMTitUUHXNQ3Tb/IKXqezgvTwh/jp/SgJpQW24jtrx3 92wKzXsxKvPGwdBI6Tf0JWUcWop8+7vFNP+t6rp1Kh+d2Rm3Y0aTSbcQtHTUovJ90WR/o156TFg xLONtCeC9daSnAleDoXYECTYHXZxBssNrTzvFT7SRKy+82secbHsJrFSbjjIzREVoY3RS3jP8Fh HDcmPH5jeVG5NRzi0EayxEshthGpkZor3gJjRwgKEaSu1BJG/wZDbIQs2JxkQPmLZZWSRBwmyg4 Vac3aWpK8GifBViIA8iOBq9/cXSoMNmXUv5HLi8uddVRhTuZV1a3ppSD+rA05jeI3EQKJpAgoO+ TlvL42iEoM3hd7Me6PRryeWU3rAIKvwlrzWjZurhXKq1b+FdUzhMq/fbeVTDJbI3btLMumWX23D YfBjazd0fTPbADbAnnxVnyI5XVk5BgG X-Received: by 2002:a17:903:3904:b0:2dd:c100:9435 with SMTP id d9443c01a7336-2df69e02500mr15989755ad.51.1790144936998; Tue, 22 Sep 2026 23:28:56 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.28.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:28:56 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 03/10] mm/vmalloc: use pte_set_huge()/pte_clear_huge() for PTE-level block mappings Date: Wed, 23 Sep 2026 14:28:25 +0800 Message-Id: <20260923062832.479455-4-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vmap installs PTE-level block mappings by reusing set_huge_pte_at() and huge_ptep_get_and_clear() under #ifdef CONFIG_HUGETLB_PAGE. This makes the feature silently unavailable on CONFIG_HUGETLB_PAGE=3Dn kernels and couples mm/vmalloc.c to HugeTLB internals it does not otherwise need. Now that arm64 and powerpc/8xx provide pte_set_huge() and pte_clear_huge(), add the generic fallbacks next to the existing pmd/pud_set_huge() family and convert vmap_pte_range() and vunmap_pte_range() to the new helpers. The CONFIG_HUGETLB_PAGE guards around the block-mapping paths are dropped, so PTE-level block mappings now also work on CONFIG_HUGETLB_PAGE=3Dn kernels, and mm/vmalloc.c no longer includes . The fallbacks exist only to keep the build working on architectures without PTE-level block mapping support. They are unreachable there: the callers only run when arch_vmap_pte_range_map_size() or arch_vmap_pte_range_unmap_size() return a size other than PAGE_SIZE, which requires an arch implementation. BUILD_BUG() makes that explicit rather than silently doing nothing. Signed-off-by: Wen Jiang Reviewed-by: Barry Song --- include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++ mm/vmalloc.c | 19 ++++++------------- 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8c093c119e5a8..a7fd51fec8a17 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -2204,6 +2204,35 @@ static inline int pmd_free_pte_page(pmd_t *pmd, unsi= gned long addr) } #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ =20 +/* + * PTE-level block mappings for vmap. + * + * pte_set_huge() only has to be implemented by architectures whose + * arch_vmap_pte_range_map_size() can return a size other than PAGE_SIZE. + */ +#ifndef __HAVE_ARCH_PTE_SET_HUGE +static inline void pte_set_huge(pte_t *ptep, unsigned long addr, + phys_addr_t phys, pgprot_t prot, + unsigned long size) +{ + BUILD_BUG(); +} +#endif + +/* + * Likewise, pte_clear_huge() only has to be implemented by architectures + * whose arch_vmap_pte_range_unmap_size() can return a size other than + * PAGE_SIZE. + */ +#ifndef __HAVE_ARCH_PTE_CLEAR_HUGE +static inline pte_t pte_clear_huge(pte_t *ptep, unsigned long addr, + unsigned long size) +{ + BUILD_BUG(); + return __pte(0); +} +#endif + #ifndef __HAVE_ARCH_FLUSH_PMD_TLB_RANGE #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index bea9f76ed7e74..fb757d95bde25 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -38,7 +38,6 @@ #include #include #include -#include #include #include #include @@ -100,7 +99,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr= , unsigned long end, pte_t *pte; u64 pfn; struct page *page; - unsigned long size =3D PAGE_SIZE; + unsigned long size; =20 if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; @@ -121,17 +120,12 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long a= ddr, unsigned long end, BUG(); } =20 -#ifdef CONFIG_HUGETLB_PAGE size =3D arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); if (size !=3D PAGE_SIZE) { - pte_t entry =3D pfn_pte(pfn, prot); - - entry =3D arch_make_huge_pte(entry, ilog2(size), 0); - set_huge_pte_at(&init_mm, addr, pte, entry, size); + pte_set_huge(pte, addr, PFN_PHYS(pfn), prot, size); pfn +=3D PFN_DOWN(size); continue; } -#endif set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); pfn++; } while (pte +=3D PFN_DOWN(size), addr +=3D size, addr !=3D end); @@ -391,25 +385,24 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned lon= g addr, unsigned long end, { pte_t *pte; pte_t ptent; - unsigned long size =3D PAGE_SIZE; + unsigned long size; =20 pte =3D pte_offset_kernel(pmd, addr); lazy_mmu_mode_enable(); =20 do { -#ifdef CONFIG_HUGETLB_PAGE size =3D arch_vmap_pte_range_unmap_size(addr, pte); if (size !=3D PAGE_SIZE) { if (WARN_ON(!IS_ALIGNED(addr, size))) { addr =3D ALIGN_DOWN(addr, size); pte =3D PTR_ALIGN_DOWN(pte, sizeof(*pte) * (size >> PAGE_SHIFT)); } - ptent =3D huge_ptep_get_and_clear(&init_mm, addr, pte, size); + ptent =3D pte_clear_huge(pte, addr, size); if (WARN_ON(end - addr < size)) size =3D end - addr; - } else -#endif + } else { ptent =3D ptep_get_and_clear(&init_mm, addr, pte); + } WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte +=3D (size >> PAGE_SHIFT), addr +=3D size, addr !=3D end); =20 --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91EF6440A34 for ; Wed, 23 Sep 2026 06:29:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144953; cv=none; b=j3ATxsmEB/xNyG+SNEOTQBn6CQ1dqVFhLjVMFGacnX2ax85gvss1OAbvGcAxqiN35mcWoVkg+kdTiOiu7Z6KyJs8n2wDRu2yLdCMqIVJYj5bqmmUKMM1oJZKiTEPjyq5sDY89xqcz0i+xG7bEIAf1Jni2tPwxNw4uElTRBynBUE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144953; c=relaxed/simple; bh=6LwTUabxUc5+IDaDWcTvEwViigIdmQF2vH7XRwOYptg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pW5jzwPpcAcwvauBoKEVwvkuA4lwAT7fzNvEMuLcP4rvGy/GyA9+H+BS/CT22YpHpXDktbVwLI8De4HD49Kc0CyIFqw6s0j6jGRVfmbakPGyxvwXhawYfbKEANxOcZSQYJtV5LlXysygsqVGGDEWZdiliRVh/gEI7QCuv/TCZZo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=P3jfJUZR; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="P3jfJUZR" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2d9004a1ac0so2356325ad.3 for ; Tue, 22 Sep 2026 23:29:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144942; x=1790749742; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=OYZzRE39rM1i8Kn24WfXB8ZLiPdAdycC2kyj5f6F4So=; b=P3jfJUZRlxGyVPLt75SakDjZoqtiwY53Q6ZJfzam5DT1WYkYiWm/cuWasEK7+GkoOM UWa4r/Bh45m74OoXOZgSXRIgdnwaIoW8qIcOHCFPkF0cUhZTtUUA7islBXVOdqvVqbBS AciXTa9t3X0vSLnYzuo4k4BjMb9xWe70vN7ysdJDTv3DupM46mQuzP6ACZk4b00AmqxK 8Y59wXNPyOmrBb0c72DrbjlIutMcNSZpnkEPv+e+76TCyWsMppaq0HGHof62WlR0cZK3 mP3s+3rAQ6PR1RxyeL5Wuvtl/2sNBeJQ5mBf6Pf2fEnqgw7z3dYrvVYhycUQmj6L8sGy luCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144942; x=1790749742; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=OYZzRE39rM1i8Kn24WfXB8ZLiPdAdycC2kyj5f6F4So=; b=yGyBEAcZAlCknLNfI3u9ZSfvaxAJvEX2zZus05uGyADZ2zV5TUG4sMMJMSM71CYK79 G9+efzbZURw1zQUOXD76Fa3Gwia++o8HWoJoH2S5BvnjMQG5iDgmdgZ36Txz10WdE8L1 9BOQpziR+Q4fVEeSeM994rpNUezK+8wazwG+5qyuODN0scBFsEY/i7lybtMGko91T9v3 mjPt89Hh6oE/JtRNbESpByvR+uh6pEwXewbt3ru4EusWegfzJmLIeD1ua9QJc1GWKTDl De8bubp58hGX7J8lSL4pF6B5dUZYfeBtBCG5jjjBoUFSPBv2jYIVkBfRPxZH0RNWbWrv 9QDw== X-Forwarded-Encrypted: i=1; AKwUvByp02nwQOFNW7kkKyhZVLPFOnaRGtFBXwm38tdrXKey4iTdvruVmv6C0hRE2zQNEIrWw/58mialnk8K0aA=@vger.kernel.org X-Gm-Message-State: AFuF++nRxbuyea1DqsgMbBd7L5MkMN06S+rJvQ5OLH0OkbOeTqbekWza ulQpHKCXbuKxVlrZg0v45I4eZ93ZFOBUvZDtFfKOw0zY4R6dz2F8Bqd8 X-Gm-Gg: AYBFou1SSfeeZtCt4Sfdq1kONWbBlZC7+oJN6ush0AiJipCitp9XdVwq1JE7E2Ut/cB MZb3P90EgjOtNJUFdm5hGSP7DeNLVIpJudB6Mh0kM70uZbcDcuMruyxoUeAKkoWrJVEv4P5iH7H 6q5stU3BFjQbKtFOC2w9+aBmeHlzquCuj9oifGKX/P+a3qkrk2Z2Jicq8hrQ6JX9Jt14S8WAFTq bZVALSUxf0UxVo+sr9RhBYOPRE54QBL1eakJ7We9Isq2Vc7L1BVwXKOT7arL4x1kk1GhNqL+LZS G6GbfUV2HYHgfxxI+WduSXV/YhBvsRSbL7wKR5XtMdeMFsfULXmGlTESLtwUUtD5lZAmbtQ1Bvc AqTWqPDndyk1IKierLMRpnAPhLtL5+VFKfNSqsH+I/ewFE0WCMHaDX+plxX9eJcsXgx2K7dh2lX WMQY8E4nepMFJUOGl8ggmlEElS+w20iBZUTMue8oqkkQaccoMJdrpOmmOCX5f9fY/UD8EAcaufJ bD/CDmvJivaNon+KPVvtWg= X-Received: by 2002:a17:903:2350:b0:2db:31d5:1448 with SMTP id d9443c01a7336-2df69d689a1mr14297125ad.22.1790144942400; Tue, 22 Sep 2026 23:29:02 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.28.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:29:01 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 04/10] arm64/hugetlb: drop the init_mm special case in clear_flush() Date: Wed, 23 Sep 2026 14:28:26 +0800 Message-Id: <20260923062832.479455-5-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" clear_flush() calls flush_tlb_kernel_range() when mm is &init_mm. This was added by commit 06fc959fcff7 ("arm64/mm: Support huge pte-mapped pages in vmap") because vmap called set_huge_pte_at(&init_mm, ...). Now that vmap uses pte_set_huge()/pte_clear_huge(), mm/vmalloc.c is no longer a caller of set_huge_pte_at() on arm64. The only remaining callers passing &init_mm are in arch/powerpc, which uses its own implementation. Remove the dead branch. No functional change. Signed-off-by: Wen Jiang Reviewed-by: Barry Song --- arch/arm64/mm/hugetlbpage.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 8e799c1fe0aa6..ce247b06fe03d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -206,10 +206,7 @@ static void clear_flush(struct mm_struct *mm, for (i =3D 0; i < ncontig; i++, addr +=3D pgsize, ptep++) __ptep_get_and_clear_anysz(mm, addr, ptep, pgsize); =20 - if (mm =3D=3D &init_mm) - flush_tlb_kernel_range(saddr, addr); - else - __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, TLBF_NOWALKCACHE); + __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, TLBF_NOWALKCACHE); } =20 void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f26.google.com (mail-pj2-f26.google.com [74.125.227.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBB9E416878 for ; Wed, 23 Sep 2026 06:29:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.154 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144956; cv=none; b=JoXCqkxNzhYnA1oqZ9Mz8V3WzrOy5QHm8J+j1IeO993B3wM7jzQCORPSFHV81Y3EMNgTBylM/GITVbG3eoKow6WnUuL4UOKDlAb7hlaL+wHzjIsy9M+l5xPSrmXZyjOUdfP1LDQeC91blcx3il/QkYWLg9qVGFBPzZyMV33qnko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144956; c=relaxed/simple; bh=ufE2Uz6n3OfWWdak1O+/fDPxZjq1jLayzDJlYGRApYc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=k32afflWDq0IzgNh4wrauWo1Q8gaSZnefWbTVr/HYWeKAsCj8/HmhaO9JSOYc0vLxaRTlULDprf+ZyR6qMKLl6YoYQy9QTB5Uitww18OHKrDb6Ng6hAFZBa+zJJ8qMLkQKRqJ+uiaTy4MrRBgQmUNpss2M60UdxyiRHhyoesfAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nRAg5i5Z; arc=none smtp.client-ip=74.125.227.154 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nRAg5i5Z" Received: by mail-pj2-f26.google.com with SMTP id d9443c01a7336-2d8fb334e72so2535255ad.1 for ; Tue, 22 Sep 2026 23:29:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144948; x=1790749748; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=RRHIBzX1Ph+VWLlGic49OjVgc6epG84F5iB9tVuXpJQ=; b=nRAg5i5ZvRrMND/HgSdR/8NzDjsOOhkKdUUdyVTjGsqeIYmd4Z7UQa6Gaua+ilXU9J qYantHpBx1CdpgjxsjZw6FZOdAv0BJGHM3vJ68SCRLTc5un97DERUJY4FzRKfddOLqsC ZQUrZt+j6h3wtnNL/F4qXRWUphhuOKfuTo8eftNgpGhbXXhl7tZjY5tAXnYGnOnS8VVy HmQHih8r2UUoej82IKbdbk0OJb26lveX9aEqfxn2Mv0UyWZgwDrIDVveirz4/y73tXIj thdO2NfPO5vCvJCZY1Ya01FBFEqX0iKPOtjoSLbbLtjftA9+nGEbMShQ7lKIDynjcV25 W2hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144948; x=1790749748; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=RRHIBzX1Ph+VWLlGic49OjVgc6epG84F5iB9tVuXpJQ=; b=qIKGZBahSBvq+CABU8EX4qLrX/2YwejgSxis1EGGiFgufz0T0aPYqAhDIqFojH8a/5 6bawtbZKabkfOiVuRoIe2iMLMo6H+m6K1DNrvlEoIg/4v/PjezBEOgX1c3G3N8/x+ZWd qeHlgJPp7Ys9spNEiDJZ3nXMZkTOW4TjNTbxDs/AtV7gHO2OznO2AhKULhduGcBVjXhS ckRWUsyj2veV0ty44K7nbXsZ0Fsydktk11FffV66GxF6w03PSDwOJmkMLLUtniohcXe/ vU4+drr5nHsTcuqoiaXVqjFfEKRzCX9sc5IUkK4C/FgCvXeRC0c5aA98ZSK+nr+p5jjs 8bpQ== X-Forwarded-Encrypted: i=1; AKwUvBz3o5x71+2DXdqjchUx888a+CQxal8FIUyegLa8Q0qpqQejJk37w3ybVfztj46boTC8k2Y+vN3BMUJyyZk=@vger.kernel.org X-Gm-Message-State: AFuF++nmmPxQj5osFDGYpF6ZkHTa614qJYSCCKbiHDEiu00Jrn8B6wqY P+fstv+HoDCJwz4vQ0K6RVP+8aoDsrl/i0tVZdt1Dw+bfY9RDUyVUVCq X-Gm-Gg: AYBFou3Xo0FFukcTSt5QIyzDuCva+1HXnXat9s03UvS8fQ9BEJcTcCsYHBmivL5mB7p VpDEDbmApuy574rJVhWnu1c3o2GPPgznpqfGphduShguUtlukkzf4CnJEJDGfmfvb0RnlfuF26y EU2zxq6ceEFGVdmKlia4KWWmfl1pp90jhYwGpdeyMJc4PLLnBNT+HON+jLQJN/lV2Ux2CLQha8i sFAb8Q7ffcI2Wnrvy7eGn4siwZHoj5aDuvtIVubZWCLubLfvl8HFrw+7NjSuOTk5R8D3K5sqBcA NBnJ0tCTRwU3dxPIl2fpYzQ1eecSn5imiH3iYLgMQDz6HLsY5bonlTY0WxoxveOWp9UW9MbISo6 AQJR+OzvZeW3abR45YaVvRzxF3LHjPDM2h3/FP2c0p3hodJZWI2YqLlLDD4NGfrqWuYbW37f7AG Hw0pQH0VzytwauC+wzUtNieLMXYDkq09ZMVarNS5LisgesdSlojAUg2Dm6jZ4FKRMqokDylwQ/Z Kpk582OipzoAx6UcKx4ZA== X-Received: by 2002:a17:902:ce0e:b0:2dd:ad73:5b6c with SMTP id d9443c01a7336-2df69d56537mr15363405ad.28.1790144948204; Tue, 22 Sep 2026 23:29:08 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.29.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:29:07 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 05/10] arm64/vmalloc: allow arch_vmap_pte_range_map_size() to batch multiple CONT_PTE Date: Wed, 23 Sep 2026 14:28:27 +0800 Message-Id: <20260923062832.479455-6-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" Allow arch_vmap_pte_range_map_size to batch across multiple CONT_PTE blocks, reducing both PTE setup and TLB flush iterations. For CONT_PTE_SIZE-aligned ranges, return a mapping size that may cover multiple CONT_PTE blocks, capped below PMD_SIZE. These sizes are vmalloc mapping spans, not HugeTLB hstate sizes. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan --- arch/arm64/include/asm/vmalloc.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmal= loc.h index 4ec1acd3c1b34..4053b1ec1902e 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -23,10 +23,14 @@ static inline unsigned long arch_vmap_pte_range_map_siz= e(unsigned long addr, unsigned long end, u64 pfn, unsigned int max_page_shift) { + unsigned long size; + /* * If the block is at least CONT_PTE_SIZE in size, and is naturally * aligned in both virtual and physical space, then we can pte-map the * block using the PTE_CONT bit for more efficient use of the TLB. + * The returned mapping size may cover multiple CONT_PTE_SIZE blocks, + * capped below PMD_SIZE. */ if (max_page_shift < CONT_PTE_SHIFT) return PAGE_SIZE; @@ -40,7 +44,9 @@ static inline unsigned long arch_vmap_pte_range_map_size(= unsigned long addr, if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE)) return PAGE_SIZE; =20 - return CONT_PTE_SIZE; + size =3D min3(end - addr, 1UL << max_page_shift, PMD_SIZE >> 1); + size =3D round_down(size, CONT_PTE_SIZE); + return size; } =20 #define arch_vmap_pte_range_unmap_size arch_vmap_pte_range_unmap_size --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pz2-f43.google.com (mail-pz2-f43.google.com [74.125.228.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22CED444705 for ; Wed, 23 Sep 2026 06:29:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.228.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144965; cv=none; b=Q0/9/Lnha9DIAL1i81Bai9Kouz8z/jQ+AtWIvbcWHL7g7MEO60UGo/BeqWYvj76Jsy4nkmgg7yhWoLPK4TMHbLa4gnImqmpykfoq5ENvWKYRLmGfSTLStcIEIZh/6eEu6F4zCM4Ecnhs1WJ3zu548lTxt4wTWUxuvBM+ajnkEVo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144965; c=relaxed/simple; bh=p+t6rwYDFXv9MMHDHyLII541DjI3rl25xQ77OuO+OVM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DpqZ5nKyHZx5BwRQu5XNaQX0+UcjCnIAOz4NDfvRut/fPY3tkGSDbdWYRKMorIzjY011FElTwN1PPwifIgVeeLeN4vEodVELR/8sNwNNC3rlJGvAJM+/ndz12WO5/F06KbOxJgZKrElxILbxe7DPo+MZvqHoRJ1FPGsx+ct7hGA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gz4WDphN; arc=none smtp.client-ip=74.125.228.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gz4WDphN" Received: by mail-pz2-f43.google.com with SMTP id 41be03b00d2f7-cc433d52421so316904a12.3 for ; Tue, 22 Sep 2026 23:29:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144955; x=1790749755; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XMhtuKKM9junrQcHfwBMD2RoOkQktkfIp1wjOSUMcpY=; b=gz4WDphNRBWqrFbYo2DhKpq7mcnHd3AwBCirquh9e0CrLjCmmO7fo0NUB7PAUcS4XB 0VfnquxR7E4pn6f+pV/bU75TxgbwrJgUJFGGRRtLGerWTGK22pBqI4zDHt+SeV22ms2E +cLDOfcWQ5WYVEK80/RhzIGbu63NbbC4AWPgTJfyvL9i3qnDHKUxy7zhlmcIYQXd6Izq OZrQCi9VHj7Sp0Xab+yAkD9j1z1Lr7h9O3Ps7PA/xB0K9o6IC+/xrNPNUTOiklGGkL46 TYyRmwc5/LNpKHwZuVT7hioB7/n0ek30Amx4BolpPb6Bsn74UgY1Rs2SIyrdYXE+jlyn 4lZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144955; x=1790749755; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=XMhtuKKM9junrQcHfwBMD2RoOkQktkfIp1wjOSUMcpY=; b=hYgaL83vpcxRZYRUgtgVO68xn3kCFfWe0ECI2h7gqKxreGzJjOCbE1WrRmp6+dKW3R rT/isk4UY9UonltVYFFWP14r5RdbVtIBphKcn687B/cJYHa4sBw972BTKxlwqRU9Hq1N M9hQVEXzn/vbznFGlOxfbzm2wc9QKJiHHmohsSolas47QPz0a5ybueB1exCicZMMXJaR wa5cBK7p088kVqvHE1S+ySGo8b27s0ajAkskn/g+VGQ3iwFVkyF3vxrYz21KQdY6hRSm VZO7y720dALuhPDQGLXero9qtoiS20EQJHr2zPNUTwDgSYh0uVSaofObBLzMPtlMX0AM OuPg== X-Forwarded-Encrypted: i=1; AKwUvBwxAsvew2N1ZdEdrIse+2mT0BnalUUGOBUIN71meKyn+Q71tEF2ZKnfCEK/NbTW0nOk1DNx/iD4ZzDvWT0=@vger.kernel.org X-Gm-Message-State: AFuF++mUWEUGm5vJQbAwNiiq1I9HWXxeuKOuDTly49qdZJbB4v4kk/jY PmWpi/FKUIq9XIXQLUabz7O8Jpa2BNO8B2Xfwgijzl3rYJMAfgn6z1Q2 X-Gm-Gg: AYBFou15tfY+Lm1TPP1mFmBrygPTvrZrvel1+s41inGRN0lhFzjojIQWMfoncyXqQI0 /LiZiB8kuFcjQ8AoXEHQPw6cxDlk2iBs5R74pAQp8JAM8XFWk8xHlUj787OwoSuafqUkbyAMyPO zRwCm924rhUsKMJ2Hsm/WCfMGJ6Gl66NNH20Vtq9S/0M8LD6I/x8WEucCl71ylD4wqRNjMNOn18 LQvZQCq4f4bKrsbgEGDKaM8N3ivFr28GEIezxv87fJRw4iqA8Ac5HOjwRXcSNlQog5sO/5KJeIq VdUam5m+GKQFPvNe9Qij6WljXiU3gSHf8aCbUm4oaPEJ5Yoht6WE1FAfeCtV3sjBfH0l1BwnkLc FHmVlprjIImw1rYMuSbJ4nZmZfdNEOnjW02SVNnVlQUiDV1RUoTW7xxmQaZpzX9YbsYiwDVT0K7 /SdBaPXG7dMz9mwnu3xeBJv2knuzOWYSvhfitwZZNuyGajhvu6+UZdgN2DxDxiprj0Qs7/Q1vc2 p5hV7G+aPtQ+vj6ZohAQA== X-Received: by 2002:a17:902:ccc2:b0:2dd:c100:4249 with SMTP id d9443c01a7336-2df69de978fmr13493295ad.45.1790144955081; Tue, 22 Sep 2026 23:29:15 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.29.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:29:14 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 06/10] mm/vmalloc: extract vmap_set_ptes() to consolidate PTE mapping logic Date: Wed, 23 Sep 2026 14:28:28 +0800 Message-Id: <20260923062832.479455-7-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the common PTE mapping logic from vmap_pte_range() into a shared helper vmap_set_ptes(). This handles both CONT_PTE and regular PTE mappings in a single function, preparing for the next patch which will extend vmap_pages_pte_range() to also use this helper. No functional change. Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan --- mm/vmalloc.c | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index fb757d95bde25..d475e52110e4a 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -92,6 +92,29 @@ struct vfree_deferred { static DEFINE_PER_CPU(struct vfree_deferred, vfree_deferred); =20 /*** Page table manipulation functions ***/ + +/* + * Try contiguous mappings at the PTE level for arches which support them,= and if + * requested by the caller. Fall back to PAGE_SIZE mappings otherwise. + * + * Return: mapping size. + */ +static __always_inline unsigned long vmap_set_ptes(pte_t *ptep, + unsigned long addr, unsigned long end, u64 pfn, + pgprot_t prot, unsigned int max_page_shift) +{ + unsigned long size; + + size =3D arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); + if (size !=3D PAGE_SIZE) { + pte_set_huge(ptep, addr, PFN_PHYS(pfn), prot, size); + return size; + } + + set_pte_at(&init_mm, addr, ptep, pfn_pte(pfn, prot)); + return PAGE_SIZE; +} + static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long en= d, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, pgtbl_mod_mask *mask) @@ -100,6 +123,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long add= r, unsigned long end, u64 pfn; struct page *page; unsigned long size; + unsigned int steps; =20 if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr))) return -EINVAL; @@ -120,15 +144,9 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long ad= dr, unsigned long end, BUG(); } =20 - size =3D arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift); - if (size !=3D PAGE_SIZE) { - pte_set_huge(pte, addr, PFN_PHYS(pfn), prot, size); - pfn +=3D PFN_DOWN(size); - continue; - } - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte +=3D PFN_DOWN(size), addr +=3D size, addr !=3D end); + size =3D vmap_set_ptes(pte, addr, end, pfn, prot, max_page_shift); + steps =3D PFN_DOWN(size); + } while (pte +=3D steps, pfn +=3D steps, addr +=3D size, addr !=3D end); =20 lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f42.google.com (mail-pj2-f42.google.com [74.125.227.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A516B2C325C for ; Wed, 23 Sep 2026 06:29:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144970; cv=none; b=kvYp2pYV6xcaQj/A6TGxoA6qZokXTwF0gbQK6g2ZVPvn4iSX6HgOpyCw2PhDY95/H2ExW5a/VaWlsL/haivKeFMO9m6jI57liVhcMZsBFTJHjDUMChLtGw45ThcBBQQ6K7ZxSWCyXbYUbuszqiQcXdWI2WO3QGOH5jUkXbLal60= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144970; c=relaxed/simple; bh=KW7jFI+Eve2BhzAQ+1W0ZZu7PTqn/iqP5QHSkiquawM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gfVIMk1uOkjO+Od57i+gYfYdg5DZSyZX3Rw//Vxx0Era1jefQjwMysuEvLpbH0jDHFc9cqzC0mzLaOsSdy6Ejzn1QIFXvR4LyfMRk55LOiqcWCMSYvpFjRhnLdRLeBF5Y/FQnjOJZbSoRBHG9XbejjaPbCirlSKHwYxwJJ9cO84= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KA/cpV3z; arc=none smtp.client-ip=74.125.227.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KA/cpV3z" Received: by mail-pj2-f42.google.com with SMTP id d9443c01a7336-2df4aa80a73so3560825ad.3 for ; Tue, 22 Sep 2026 23:29:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144961; x=1790749761; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+a2Rdy8KxrMJDsvufLYSYTpVzrgqa1a95U57Pa83CJI=; b=KA/cpV3zYsXnesL5h7R9fOlF8ktVYNfyXTODqPIJ7SVLfTXwEFJ/K4hSjKvhHXYMbr /PNCSdCk003b7nyX8iE2P1Pu56dbXcPdCI58JKcckSHeCpYOqD0zhKAPMY17FHDU11M5 1WtQ5qGDpiGihi1NP7K8Iw3aGWRK3c2aTGv0e/KAGq68PnzzJ2/7PqErGjWZ3pxqyt1L kESqHFj9zessXjt7vLGOCFIePoWo/GZFsBDsYB+dKoipm8TqYJeJx/Xxv88sYwrbK4Mm fo1WTgPa81HXfJdnYDdzLD8uqGx8fAoUr4/iP06CjphadrQXQbUyhP2PwK7fwcTtzOSY VRtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144961; x=1790749761; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=+a2Rdy8KxrMJDsvufLYSYTpVzrgqa1a95U57Pa83CJI=; b=h1U5rWoxr7QTYcjbaF30aPVvIGS0BkakNiekQxnZDIJr7p2uJdMg3mjqZgHxOYCS2q oMZZXuXAkg5nefExdKoWrwbvV9vyZfUrclYlknpPC4QlZZqemnWK/YnDe6GFM8myPWXh Ce4fAFTHpOkgBBJybC7V53NagW/dCRxtA9nfnb9bVXCflRXlE0JYbiQPqtnG3deOYelH tihVPufOVpaZWIl4+4OOS+eqDDANvnD/0hBhwtytiRQ/1oUZ3PJJOzEROnQyqhsuY6ZB V0y1jyU/FC8/nYwHrYw8MN/EpZTVKFsdEucBeotTAMC+wq5TGCsyzrY/+ebH1D0VU+o1 iHfw== X-Forwarded-Encrypted: i=1; AKwUvBz64cyO07jIuG1eGdOOxgEIlFi8qEwBWrGvfkxPd7mIIDAcjQMXz8kZy8ffXGDEdJqXgY09JdtuzdZ7P2A=@vger.kernel.org X-Gm-Message-State: AFuF++ne/8qEduJU3ljPhG1xHDwCtqDfuMcqB/52DxYGke4TqUaxt27v n4dyjqr84xSt/COrTkRFSzMaWhLwI54MRa6AFH0ZAhqXC8eRtiM3h8+J X-Gm-Gg: AYBFou3K0UO7nt0Ya2cfoziJ6zK8HE0n99/AF8FsBV/q06sIf3Hd5I8rZn5wNr5XVjZ onA9qyfBtPxr57B0FBWVSS22NmvzxLhNHzMnnfnjpAHagl4JsMAOK8NHhZqOxrxWQGTvwuuZ5vR pJ4JT7uYmP5uWEg5f9HueMY+OnmjzzAgk4TrYcUHgqx56QV2BBPtfCayRgIqWWXwMI2hztg4nhE BjLX111ItncWa1lDQyrVp4n7/i+61KujgoIrAgY5UBG+6gL1SuXI6qILVgKJ95Bugmjji3TyhaL lMHdHnbQiymhQyW+jlhJxBm5yU3F3S8NtKybCIaUM9k/SAv323jOngdr8sjfOpqtzBmDUP1n9Wr x80SbpKgTmbLFWdWgPLUMf+/SFSv3fGBTYpYzvW7l57Eq8GEpNrahpm+yIFwg8Bkt4I9/Hg8Fx2 z1Mr1o0xuVmZI3PyiWeRYA5MCtVxh/vjKlun5swXXuTLt4S5/mcLJZaO1RcpylidEuCZDPaJacj pJfXUdpJt2MGTSNqWo+5Q== X-Received: by 2002:a17:903:390d:b0:2d6:df31:5bd0 with SMTP id d9443c01a7336-2df69d22254mr17039015ad.10.1790144961284; Tue, 22 Sep 2026 23:29:21 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.29.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:29:20 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 07/10] mm/vmalloc: extend page table walk to support larger page_shift sizes and eliminate page table rewalk Date: Wed, 23 Sep 2026 14:28:29 +0800 Message-Id: <20260923062832.479455-8-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" vmap_pages_range_noflush_walk() (formerly vmap_small_pages_range_noflush()) provides a clean interface by taking struct page **pages and mapping them via direct PTE iteration. This avoids the page table rewalk seen when using vmap_range_noflush() for page_shift values other than PAGE_SHIFT. Extend it to support larger page_shift values, and add PMD- and contiguous-PTE mappings as well. Rename it to vmap_pages_range_noflush_walk() since it now handles more than just small pages. For vmalloc() allocations with VM_ALLOW_HUGE_VMAP, we no longer need to iterate over pages one by one via vmap_range_noflush(), which would otherwise lead to page table rewalk. The code is now unified with the PAGE_SHIFT case by simply calling vmap_pages_range_noflush_walk(). Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Dev Jain --- mm/vmalloc.c | 84 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 50 insertions(+), 34 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d475e52110e4a..6e0dec73107a7 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -558,8 +558,10 @@ void vunmap_range(unsigned long addr, unsigned long en= d) =20 static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { + unsigned long pfn, size; + unsigned int steps; int err =3D 0; pte_t *pte; =20 @@ -590,9 +592,10 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned l= ong addr, break; } =20 - set_pte_at(&init_mm, addr, pte, mk_pte(page, prot)); - (*nr)++; - } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); + pfn =3D page_to_pfn(page); + size =3D vmap_set_ptes(pte, addr, end, pfn, prot, shift); + steps =3D PFN_DOWN(size); + } while (pte +=3D steps, *nr +=3D steps, addr +=3D size, addr !=3D end); =20 lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; @@ -602,60 +605,88 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned = long addr, =20 static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pmd_t *pmd; unsigned long next; + int err; =20 pmd =3D pmd_alloc_track(&init_mm, pud, addr, mask); if (!pmd) return -ENOMEM; do { next =3D pmd_addr_end(addr, end); - if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) - return -ENOMEM; + + if (shift >=3D PMD_SHIFT) { + struct page *page =3D pages[*nr]; + phys_addr_t phys_addr; + + if (WARN_ON(!page)) + return -ENOMEM; + if (WARN_ON(!pfn_valid(page_to_pfn(page)))) + return -EINVAL; + + phys_addr =3D page_to_phys(page); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, shift)) { + *mask |=3D PGTBL_PMD_MODIFIED; + *nr +=3D 1 << (PMD_SHIFT - PAGE_SHIFT); + continue; + } + } + err =3D vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask, shi= ft); + if (err) + return err; } while (pmd++, addr =3D next, addr !=3D end); return 0; } =20 static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { pud_t *pud; unsigned long next; + int err; =20 pud =3D pud_alloc_track(&init_mm, p4d, addr, mask); if (!pud) return -ENOMEM; do { next =3D pud_addr_end(addr, end); - if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) - return -ENOMEM; + err =3D vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask, shi= ft); + if (err) + return err; } while (pud++, addr =3D next, addr !=3D end); return 0; } =20 static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) + pgtbl_mod_mask *mask, unsigned int shift) { p4d_t *p4d; unsigned long next; + int err; =20 p4d =3D p4d_alloc_track(&init_mm, pgd, addr, mask); if (!p4d) return -ENOMEM; do { next =3D p4d_addr_end(addr, end); - if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) - return -ENOMEM; + err =3D vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask, shi= ft); + if (err) + return err; } while (p4d++, addr =3D next, addr !=3D end); return 0; } =20 -static int vmap_small_pages_range_noflush(unsigned long addr, unsigned lon= g end, - pgprot_t prot, struct page **pages) +/* + * It can take an array of pages which are not all contiguous, but it + * may have contiguous chunks, as hinted by @shift. + */ +static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long= end, + pgprot_t prot, struct page **pages, unsigned int shift) { unsigned long start =3D addr; pgd_t *pgd; @@ -670,7 +701,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, next =3D pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |=3D PGTBL_PGD_MODIFIED; - err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err =3D vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask, s= hift); if (err) break; } while (pgd++, addr =3D next, addr !=3D end); @@ -693,27 +724,12 @@ static int vmap_small_pages_range_noflush(unsigned lo= ng addr, unsigned long end, int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { - unsigned int i, nr =3D (end - addr) >> PAGE_SHIFT; - WARN_ON(page_shift < PAGE_SHIFT); =20 - if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || - page_shift =3D=3D PAGE_SHIFT) - return vmap_small_pages_range_noflush(addr, end, prot, pages); + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC)) + page_shift =3D PAGE_SHIFT; =20 - for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { - int err; - - err =3D vmap_range_noflush(addr, addr + (1UL << page_shift), - page_to_phys(pages[i]), prot, - page_shift); - if (err) - return err; - - addr +=3D 1UL << page_shift; - } - - return 0; + return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift); } =20 int vmap_pages_range_noflush(unsigned long addr, unsigned long end, --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E536445ADA for ; Wed, 23 Sep 2026 06:29:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144978; cv=none; b=ApQ8mv91qZxEc1+WuddxSn3XLDlksP+QSUk0rMsacN5ahyfYNOAMkyEbJLUsxcp1/AtkQf+1wSET6IqMgFjnGUuQbt+D0VDD24/5dXQnRMhpxZ4jS4WI9zX9STo10/7bajw4AnNPMf52b+Pq6YNbnOKxe2mzb7543bj236WvZN4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144978; c=relaxed/simple; bh=Bmj2nhdQ7L8Wg4P/gKwr0Uz/KKXJKocTqJsXYXBeQ2k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ctt7hxijB/VtofrS1+LmJSvo5o0o9oFhbZ2fpXrShhE9BQNl4BOAheaxMmmBYulMmlWq1NLlBS1Sc3hh1WXq/ZcIG/ONNlHyhA7y4XEiIxpJ11/mIHQrS0PcIusxWYUO/ojmO0tf7a1C5fAL0GBLupjOCWMEdLbwXCwamYuj06c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ANmaUx9R; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ANmaUx9R" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2caced6038eso4079005ad.0 for ; Tue, 22 Sep 2026 23:29:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144967; x=1790749767; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BLJDwfik8v5sXhgfZiByNmP7K0w0uZHSBWxfXEf9m1M=; b=ANmaUx9RDSh/uX8dcbiX8Q9zrlR6BBu3Jw9kgrxTgN54IW1HDhYptbmQoL/1sYGzEv vaV6AylNMPgvkuLXTaim4rCzmfyIur1UIQjLZ8GIFh8IvShSqhDpDQIavtw8p6YkeWT+ 3gm3U4q4x5xnrwxUtGOCDruCdvjcEuUaEvWxNDH6gmbyhDa7tMiBbGczSMUBhBawkv/v TLGtb/FTAHPUCgrCZdXJ3salHj6oHlnRZ8kuCOuaSnJBUqcOKosnvwvmAsfaGb08ZJUC T9RgmlSDz0YhIOIH8eZ5dgZXBOYIO+DVERxV8XWNpYvY83rim/5Pi/jMfaTv78pTDOKi U0ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144967; x=1790749767; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BLJDwfik8v5sXhgfZiByNmP7K0w0uZHSBWxfXEf9m1M=; b=Yq6ZuFwQUNB7YLnRtQbkKesN4hmr1zm+VXrB5u155AKTsmsG/ggRx5UUARbDtYgqI9 ZixmLS+XOO6UbMvbf8JY/2o+9T6yxRP2GfKuepLgwFL1zCZsafAcQIQ0T0uiH9pZDyqH ND/h5nY3SU3tJQ0ce1M3Du5TwWaGmDt86M++joss1jqU+5xTKSjlKhaw6RhB0gIjWTLX ZCt/vw4P0AplkpjfNPZpMKfjnnEbkYe+ezceH93CAvIA6TDQV+6fFBcLTOfKnXNESKBA 6Eck1asIMoTlD+ZiAINCJiQHAnHI1NoUlaUs+pG3GXg8lIi10pSz3azgwhfHiFClp+wS dJzw== X-Forwarded-Encrypted: i=1; AKwUvBx7X+pXsqLmWPHgH07O7E8h60whOvDaoeBMlKUAK15EYKIWAv4XbUe9Vu2Ftmw+0klN2W8xRxPBu+62h8I=@vger.kernel.org X-Gm-Message-State: AFuF++nL3CT+9Txw7Lv68kNc3L0khe+dP+eJooxbfaym9L63qUNGewmV WTBws6XSixJ7kBkICSr+k8KI41DGDvAT6Hg+LrPWMx5UCkh8IuiuiOlz X-Gm-Gg: AYBFou25Ux3HCniaW6l/v81qdvkS/lPFQ9OaqWyzVt7qazdRQaUngNGGyqNG0uNSMDY Grp7SVmu+HwThcktQ0gYg0ji96WFgo/yn++sDiscNtR+rF9VWZV8fiIW1sBKdQngVcGMSDPIEav Omv1LLvNpQeZ9kfAut73Ep0XItaXCOi0f0lw+RjxK4R/RfCwpNVKRgA92MVlHTHBenQmlHH5115 OSYMqtGqskX1G0vyPyPYI+NorHBL1udQ4O9qTVC0TZFAuUwXBJelA0ZzbKbAtNipGYC2Jegw2NP CHnemGjPFUGw6wxf1SCF2EmNVSXtoenlGJgRqfBPqcnzwqyhT0cVDywyWvApFq602jG3l/sTvlh GibsDDWp2xUDeBAjaNrbwZmhFxWhxeHad1Xu/7iO0THiD5kCcteBXW2kd8+I3BUtxaRgR9uvpgj pjK4m8SgWyLibUQu1RVN60WNcz3qPyvqiJLUtJsFg3UXuWi6p6Nv+BG6I6aIo0TcDhKSKmjx2bj RsCe5xRBauR29wnK/m/cQ== X-Received: by 2002:a17:903:3c2c:b0:2d7:1cbd:9ceb with SMTP id d9443c01a7336-2df60aee398mr25552315ad.12.1790144966712; Tue, 22 Sep 2026 23:29:26 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:29:26 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 08/10] mm/vmalloc: extract vm_shift() to consolidate mapping shift selection Date: Wed, 23 Sep 2026 14:28:30 +0800 Message-Id: <20260923062832.479455-9-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the vmalloc mapping shift selection from __vmalloc_node_range_noprof() into vm_shift(), preparing for reuse by the vmap batching path. No functional change. Suggested-by: Dev Jain Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan --- mm/vmalloc.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 6e0dec73107a7..a43965016eb4d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3568,6 +3568,14 @@ void vunmap(const void *addr) } EXPORT_SYMBOL(vunmap); =20 +static inline unsigned int vm_shift(pgprot_t prot, unsigned long size) +{ + if (arch_vmap_pmd_supported(prot) && size >=3D PMD_SIZE) + return PMD_SHIFT; + + return arch_vmap_pte_supported_shift(size); +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -4079,11 +4087,7 @@ void *__vmalloc_node_range_noprof(unsigned long size= , unsigned long align, * supporting them. */ =20 - if (arch_vmap_pmd_supported(prot) && size >=3D PMD_SIZE) - shift =3D PMD_SHIFT; - else - shift =3D arch_vmap_pte_supported_shift(size); - + shift =3D vm_shift(prot, size); align =3D max(original_align, 1UL << shift); } =20 --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f13.google.com (mail-pj2-f13.google.com [74.125.227.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8BF04156F9 for ; Wed, 23 Sep 2026 06:29:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144979; cv=none; b=FA+H4uUvEPbXU5RXiUoE4da/4mgg1aPJlbrdgtBoG5MHHDITUTnS4R+gvYqzCkhdfo0JxsjzdlZ5n2/DQmU3H7UtWOV218NDn57qv5JReh7GLp0rJSWCqlPoftubhLu+Y+tw0dFezzGeNLfZIPP7hvIIx/fGCzNRgQ26czeN98A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144979; c=relaxed/simple; bh=j2LDewBnaNSHSszWiLsF8rrjIezPLg/AfJBstLlMPfY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=oRlz3r/ffstVf53zp+/4PPThSY6uJMSNXrgoeFGztPQOGlzQ+D5HrBmLPBVELInNyAAhoPnDpnBaECBQZLQv2Iz934uhuNtFV1g5UoHzXIgZdDYKgprR5QKb97rlWfadKTxHXmIdfrQb9hQJ8/qiMiQBxqQaYP0FKbFU+CuBcek= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Jn8fIPLv; arc=none smtp.client-ip=74.125.227.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jn8fIPLv" Received: by mail-pj2-f13.google.com with SMTP id d9443c01a7336-2d747ed1368so3508725ad.1 for ; Tue, 22 Sep 2026 23:29:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144972; x=1790749772; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=dEaCg+SoOuqpLOGmqy1zgShmdKDFtyrROF+HoSyjUpM=; b=Jn8fIPLvUJNKQOUWYa9dIyD1NuCuK7P8bsLyR0mBQV/Te/H+wFaS2tW6vjmmH/PwYI Eo16vqlWI4qBzGTI8wKoLfQ69TWaTR6+aEiFuKhlU0vyWbCaHFLP/UIZOW5KDMw52UoQ Wx8uNsXzRhW/FM+gbFwarvC6U9WWWGD5GHp5CsJwRzNt88zSIo4/HppMZBMUj7nr+KPr 015ao1YFkmNdRz81EepBqjfL3vIhZDAm19kYLDr/KXgH8lCmvedI3q+IbYlDZu9DOtPR aKV585VttvqEV0qKuIXqndRDEvtSXw/HLg7hpN7sbtGQXjajKmOpp93K8JFZnfLQlJWi 7/+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144972; x=1790749772; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=dEaCg+SoOuqpLOGmqy1zgShmdKDFtyrROF+HoSyjUpM=; b=2Ss1ps9PSk7ZtVPHbqCCQpsM8H6R81YZP6k/K6lZqjDsrVibO93JE/iLKk9b5GqyFM SnJuEWSW+dLwElhhpl70mfli0sJGF9m8sFr/spj2ApWQuItzVSB9hDeBtxctfUngD/Cl aBsavEulcQ5S1QRaDO5MvIEzjzGAu9Tdi1viJoNB6hoXUZjzR9rwvFFgq3BtI5WkFaOj udfN2qjIwxV+APTHOQqpAq4C4tXLIsfg/zCgblTgWQ+GeNj69jzriRLlzI3fgZMRmBEX 7MKJyiCWbiown9LqzjOvj6ud6HJbzuDPv3I8RYgsApOfGCrhhO76mluLC+lz+RRQFw5T PztA== X-Forwarded-Encrypted: i=1; AKwUvBwIZ/ZlvC7dfXs5ybqYE0H7wepbGNIh03y+zTT1aAz9k+M3vrqZZ6ih4PFmpO+GRfheef9FNnDxUASPwS8=@vger.kernel.org X-Gm-Message-State: AFuF++nut7xpOaXLEr3049lo3DNNc9X0ocb/53P4g7cI201Daj8lOO7u pUAMaKPmLy9w+qW9Bk/IR4DuzxyZDl6szEBjgo6+L+ojjmrrWIcuQs7N X-Gm-Gg: AYBFou2ZHVma8gveCjHLj4g/N2fuzV5IkvyVN9r/1ZqhgY26Uz8sBISRXE7foFyQ4Z4 YpDDE6MthYuoxrNZqEomT8ZHfVQGnyOzOlix/5pSuNEhxkrwaBIyJtIZUzkTIBn61ZItQ2DZJ45 4JfoBKM4Ryop8IxTyN3p1cEgGrlFf50vWCCLVy3ZNqy4GYE2zd1QFeSAIyNkgXqgdsH0zsi/0ED 80isJE1gFAH+E7xJtobpDUhmBCefHEAisiEK3GMHxzEMun6qSBCvHz2pc2yEinO8z7Kx0hiWUAw oVRG8dHpW7x6kSw1Uv4nNL8AD8ClfUrLXdmnr8f1sLWseK799tcoVT8XbzgLlfEdqSDb8/pbbFx iEPBjRGp6nRA4G2s27oOGz/ep711cAzhK5ZOEvbSl/7G659sgR8BcUhOS/rAssGdaV4eDOWeOvd RWWlyYNZ3fTAhD4VrwSdL4vURzUNHat/o49fNl1m5x9EVTh1M8m1Y+dM6BqHPuCL1Y2I564FgLa BX/Dyf2jZXV8HrlCSC+4w== X-Received: by 2002:a17:902:db0b:b0:2dd:c100:b2c2 with SMTP id d9443c01a7336-2df69e01d0cmr15666125ad.45.1790144972237; Tue, 22 Sep 2026 23:29:32 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.29.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:29:31 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 09/10] mm/vmalloc: map contiguous pages in batches for vmap() if possible Date: Wed, 23 Sep 2026 14:28:31 +0800 Message-Id: <20260923062832.479455-10-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: "Barry Song (Xiaomi)" In many cases, the pages passed to vmap() may include high-order pages. For example, the systemheap often allocates pages in descending order: order 8, then 4, then 0. Currently, vmap() iterates over every page individually=E2=80=94even pages inside a high-order block are handled one by one. This patch detects physically contiguous pages (regardless of whether they are compound or non-compound) by scanning with num_pages_contiguous(), and maps them as a single contiguous block whenever possible. The mapping order is determined by taking the minimum of the contiguous page count and the pfn alignment, allowing graceful degradation when pfn alignment is less than the contiguous range. Pages with the same page_shift are coalesced and mapped via vmap_pages_range_noflush_walk() to avoid page table rewalk. As users typically allocate memory in descending orders (e.g. 8 =E2=86=92 4 =E2=86=92 0), once an order-0 page is encountered, we stop sc= anning for contiguous pages since subsequent pages are likely order-0 as well. Signed-off-by: Barry Song (Xiaomi) Co-developed-by: Dev Jain Signed-off-by: Dev Jain Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan --- mm/vmalloc.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 80 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a43965016eb4d..37cddb7f45a4f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3576,6 +3576,85 @@ static inline unsigned int vm_shift(pgprot_t prot, u= nsigned long size) return arch_vmap_pte_supported_shift(size); } =20 +static inline int get_vmap_batch_order(struct page **pages, + pgprot_t prot, unsigned int nr_pages) +{ + unsigned long pfn; + unsigned int nr_contig; + int order; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) + return 0; + + /* Limit nr_pages by pfn alignment */ + pfn =3D page_to_pfn(*pages); + if (pfn > 0) + nr_pages =3D min_t(size_t, nr_pages, 1UL << __ffs(pfn)); + + nr_contig =3D num_pages_contiguous(pages, nr_pages); + if (nr_contig < 2) + return 0; + + order =3D ilog2(nr_contig); + + if (vm_shift(prot, PAGE_SIZE << order) =3D=3D PAGE_SHIFT) + return 0; + + return order; +} + +static int vmap_pages_range_batched(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) +{ + const unsigned int nr_pages =3D (end - addr) >> PAGE_SHIFT; + unsigned int prev_shift =3D 0, batch_idx =3D 0; + unsigned long batch_start =3D addr, batch_end =3D addr; + int err; + + err =3D kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + PAGE_SHIFT, GFP_KERNEL); + + if (err) + goto out; + + for (unsigned int i =3D 0; i < nr_pages; ) { + unsigned int shift =3D PAGE_SHIFT + + get_vmap_batch_order(pages + i, prot, nr_pages - i); + + if (!i) + prev_shift =3D shift; + + if (shift !=3D prev_shift) { + err =3D vmap_pages_range_noflush_walk(batch_start, batch_end, + prot, pages + batch_idx, prev_shift); + if (err) + goto out; + prev_shift =3D shift; + batch_start =3D batch_end; + batch_idx =3D i; + } + + /* + * Once we fail to batch pages, we expect to fail batching + * for all remaining pages, so just give up. + */ + if (shift =3D=3D PAGE_SHIFT) + break; + + batch_end +=3D 1UL << shift; + i +=3D 1U << (shift - PAGE_SHIFT); + } + + /* Remaining */ + if (batch_start < end) + err =3D vmap_pages_range_noflush_walk(batch_start, end, prot, + pages + batch_idx, prev_shift); + +out: + flush_cache_vmap(addr, end); + return err; +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3619,8 +3698,7 @@ void *vmap(struct page **pages, unsigned int count, return NULL; =20 addr =3D (unsigned long)area->addr; - if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { + if (vmap_pages_range_batched(addr, addr + size, pgprot_nx(prot), pages) <= 0) { vunmap(area->addr); return NULL; } --=20 2.34.1 From nobody Thu Sep 24 13:39:20 2026 Received: from mail-pj2-f19.google.com (mail-pj2-f19.google.com [74.125.227.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D80764483B9 for ; Wed, 23 Sep 2026 06:29:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144987; cv=none; b=efM5TR7H6J/Pag5fq7ODUtNveiFgzL6RrR4UJUzH+AWJrI2UNplvM6vEYsj7nevLiAmD0T8kOtVsUvl2u4ZwcQKMcaKUGQwFy9OW3Zi9gIMXtRT+FfjjmAB4M1I4Kuthxfm+cDU7mHpR1+WL5tWsD8bjR/tll5i5YwemLqKW1cM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790144987; c=relaxed/simple; bh=bspJBmRxLS4qW3UQTFV0QVGd0Nf0mf04koeOIwHSQFA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PopbwOA9vpz+JYA4UxafmSQ0GGqbeg3bGi+WttLwZOSATZkbFBVkFQ1uiH5FPkysuC79J5eW2O854qgZYvcowItzygNCRRcIy4hswwO13OL8sxbwFawCKpbWUlTzwCVJvrFDuSwDFzbGSRRUl9UuvU5kdavYc5Qyq87lymEYWpQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=G88Z+KwI; arc=none smtp.client-ip=74.125.227.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="G88Z+KwI" Received: by mail-pj2-f19.google.com with SMTP id d9443c01a7336-2db18fe433fso2719975ad.2 for ; Tue, 22 Sep 2026 23:29:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790144978; x=1790749778; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Qz31YaIUZL0HkPkLOInD8YA56WfSRIuJpUh6x60MqwE=; b=G88Z+KwIzih7XULAfpBP2PFtsokNa3be8ORhSfBVtcfGN+5Q6Zz5gX+A+yNsrzv3MP cqzeHKMhkZ2EaPESf8rygH+dxipQu21ZlVKj7wssRFlKlOvlG8vkzGs7JjfviWNJoS8w PuHphWdcAVEdd3hR78lyhcINfP9e9nQsQYax1YurP/x5+eFa0YirKvBulLfaTtFpkMGI Gn/ZKXPMl4KczvaxhW4PcynC20Gp5KqqxpIstyTeUg5zdTl1B1YsAfs09eNPHHE69st6 gMFggSVr+/1diAIss8BIKPiSLjbyiFsH0A2qpVXOaasMHYcbu9AJGOcthfEx4S0ddiI9 ZP2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790144978; x=1790749778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Qz31YaIUZL0HkPkLOInD8YA56WfSRIuJpUh6x60MqwE=; b=QP+4XORPIaoBiuuD4LpYu1DjrrpP801RzRKxTLuLeBrKC+Y4HXfRsfCauZOhqqDPiN tubYIGXQtyHjQh9zE5ZXt5xBBvTiqaZIS2kvYz3xHOB35o478G6jIYgMz6l4viRpKp42 usNq+5sPjzYQ5bD5Tzo1uMD/NccUuTgxt5fYHZ4bWSkFzTLlfgMCALKWIfZGV1ecitP6 hMrwYwyGPTo6ceDrgzaCb/3YMbiw1DQC+QvpTH8CjgOAahbEqgmzWKV2HtUbaO7HQJOz Xq86piWIj6qAcA/cMpiL2zXwHr+0CW1bxYPEzfnuQD6bV7yfrONpig9XKntEX4HrO0Hr e6ig== X-Forwarded-Encrypted: i=1; AKwUvBwPyaUHDl85sfaQO0GmP+WM116iQvGHJ9QqRxpmpt1+fH+3vp1iw5PDWD2M41/KyYkTvwVKvpVCmGAf1QA=@vger.kernel.org X-Gm-Message-State: AFuF++lCV2iVGnz8x7O8r42yBQ3zNznXyBNIpa18J7SkWV1iOpfezzqA kMnq090X9qOBfX9Ob+/qbVKrWLrmTamrDpzeonbhDyjvgXzjNWB4HWiC X-Gm-Gg: AYBFou1KEALzXn1wALiJRpIWY3hgQzR4WCKNBKnl1p/sQXol2QjQ9tBNg00lqfcKOQ1 +3RLZZMvqRChyVgbCbMDpuh9K7rwb6R7zLNrx9ekz/E8bjkRkWk6fvuBbo8XoBQ1P+dudiUzwLi Xi5X/fdKVPVkG2A6H5o+xCixPL5ooiRh2VFsRIrto1ZMH6rfKkPyF9556Fqwlphg/END8TFpKbu zV+fMk2kY8qKYAKIIfxAtWgg8gypqlbUeaFZnBa9h8X4GgiGm10/txnkozJWffs5lhgEMmToAZw Uintn65Lr4tkxZ/Qi9ogl9LrEZ3mGEThY93b+tevc2ZBvVEhNPcsxb8BlJj8yl9naMSueEjKjlp xL3v4FiqvjEfeZFFAkgWPQ/r9z/Z5T3KbP7AZWcKJGE3ybgY9Zbjib4a1v9BoXKYZidcsNlWYG6 WsafoLd9e+XmXYINTN6iRAUN+QoTzHm92RYEtg6/lMNscrqKXxHGQ3ROSgVoKF72XWmSvOJQ7uy B4R3XuTD5C8nOgyYEsf+g== X-Received: by 2002:a17:902:d483:b0:2dd:c053:c20b with SMTP id d9443c01a7336-2df69ddf0cbmr15391165ad.39.1790144977728; Tue, 22 Sep 2026 23:29:37 -0700 (PDT) Received: from mi-OptiPlex-7060.mioffice.cn ([43.224.245.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2df6a516bfdsm5620465ad.5.2026.09.22.23.29.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 23:29:37 -0700 (PDT) From: Wen Jiang X-Google-Original-From: Wen Jiang To: akpm@linux-foundation.org, catalin.marinas@arm.com, linux-mm@kvack.org, urezki@gmail.com, will@kernel.org Cc: Xueyuan.chen21@gmail.com, ajd@linux.ibm.com, anshuman.khandual@arm.com, baohua@kernel.org, chleroy@kernel.org, david@kernel.org, dev.jain@arm.com, jiangwen6@xiaomi.com, leo.yan@arm.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, maddy@linux.ibm.com, mpe@ellerman.id.au, npiggin@gmail.com, rppt@kernel.org, ryan.roberts@arm.com Subject: [PATCH v9 10/10] mm/vmalloc: align vm_area so vmap() can batch mappings Date: Wed, 23 Sep 2026 14:28:32 +0800 Message-Id: <20260923062832.479455-11-jiangwen6@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923062832.479455-1-jiangwen6@xiaomi.com> References: <20260923062832.479455-1-jiangwen6@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Barry Song (Xiaomi)" Try to align the vmap virtual address to PMD_SHIFT or a larger PTE mapping size hinted by the architecture, so contiguous pages can be batch-mapped when setting PMD or PTE entries. We do not expect any significant overhead for this. Add __get_vm_area_node_aligned_caller() as a wrapper over __get_vm_area_node() to simplify repeated calls with fixed arguments. Signed-off-by: Barry Song (Xiaomi) Signed-off-by: Wen Jiang Tested-by: Xueyuan Chen Tested-by: Leo Yan Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Dev Jain --- mm/vmalloc.c | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 37cddb7f45a4f..1a3b191885024 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3328,6 +3328,14 @@ struct vm_struct *get_vm_area_caller(unsigned long s= ize, unsigned long flags, NUMA_NO_NODE, GFP_KERNEL, caller); } =20 +static struct vm_struct *__get_vm_area_node_aligned_caller(unsigned long s= ize, + unsigned long align, unsigned long flags, const void *caller) +{ + return __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + NUMA_NO_NODE, GFP_KERNEL, caller); +} + /** * find_vm_area - find a continuous kernel virtual area * @addr: base address @@ -3655,6 +3663,30 @@ static int vmap_pages_range_batched(unsigned long ad= dr, unsigned long end, return err; } =20 +static struct vm_struct *vmap_get_aligned_vm_area(unsigned long size, + unsigned long flags, pgprot_t prot, const void *caller) +{ + struct vm_struct *vm_area; + unsigned int shift; + + if (arch_vmap_pmd_supported(prot) && size >=3D PMD_SIZE) { + vm_area =3D __get_vm_area_node_aligned_caller(size, PMD_SIZE, + flags, caller); + if (vm_area) + return vm_area; + } + + shift =3D arch_vmap_pte_supported_shift(size); + if (shift > PAGE_SHIFT) { + vm_area =3D __get_vm_area_node_aligned_caller(size, 1UL << shift, + flags, caller); + if (vm_area) + return vm_area; + } + + return __get_vm_area_node_aligned_caller(size, PAGE_SIZE, flags, caller); +} + /** * vmap - map an array of pages into virtually contiguous space * @pages: array of page pointers @@ -3693,7 +3725,8 @@ void *vmap(struct page **pages, unsigned int count, return NULL; =20 size =3D (unsigned long)count << PAGE_SHIFT; - area =3D get_vm_area_caller(size, flags, __builtin_return_address(0)); + area =3D vmap_get_aligned_vm_area(size, flags, prot, + __builtin_return_address(0)); if (!area) return NULL; =20 --=20 2.34.1