From nobody Mon Feb 9 10:34:32 2026 Received: from mx0a-00823401.pphosted.com (mx0a-00823401.pphosted.com [148.163.148.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F8B81DFE1 for ; Tue, 16 Apr 2024 17:25:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.148.104 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713288311; cv=none; b=DIxnThbTdeZa5DMdZgSjG0X0gzLqG7yh0Wy1LeFZ1mR6+4pNIXTxEaLykPq/c15jiXyugETXsmRT6FeoG5c6Rh4IAIP7eO5k0meYyegwg9V4b1BxFcjjusqZvzmr+Ovg6WJ1figvZ6r6PCh+U/68EaJeDbhviN+nHqQvZVH9A3E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713288311; c=relaxed/simple; bh=GxcofnKNVNbbBFF4cyNRNVeMqKEmmntSNw15YNzkPZI=; h=Message-Id:In-Reply-To:References:To:Cc:From:Date:Subject; b=ZEohMGxn3lp7LP0zuTZk4sQMN+8lw/WyHxlDffuc9EGDYeCiB70tRxwBJnO2NATYS87QgdYO4R1IZ38zy7mFx85597+6XFhzJRSjunMUG4oMrp2muIT/Abqk+YCuRo23GSP7zpN4sRefgwf3GJfKgAYnPD/PLtP9CUnjwuEz8m4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com; spf=pass smtp.mailfrom=motorola.com; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b=UXNGIZCS; arc=none smtp.client-ip=148.163.148.104 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=motorola.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b="UXNGIZCS" Received: from pps.filterd (m0355087.ppops.net [127.0.0.1]) by mx0a-00823401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 43GFPN5k006602; Tue, 16 Apr 2024 17:24:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=motorola.com; h= message-id:in-reply-to:references:to:cc:from:date:subject; s= DKIM202306; bh=Ktj2knpZkxEzepUnXbFCAHnzLfkAa26Odp8kjW5qVXw=; b=U XNGIZCS9YmDgBUeEu5cofHA9mvaV6V1cdLYUPyD490DtaRc+ylTIkjPFzxuWV+4T 21f6DuwRlCKVXuIbxpUAP7zIVfKAOnpoOVrxFs50JPTCkWAv031LnE5jmeiZi1aA R4XwQPSs9ieKI8NnJOZWkmkEZnXk98ir5fZoWx004yfu9JRghgvTKQx8l5r46UH2 3Gxac+BT+Y/DdxHClOLeMtkCkpMCot4eJMbl5uT1eqEPYTiWpENBGPURZC3kfUqR JGoVqQIu/+3pYXSYSfFrBnBWrgCOkuJqt8EReXdQEsqrYza/cDg2Dejomjcze8ru uOgM14JDYDWyjwAaePTTg== Received: from ilclpfpp01.lenovo.com ([144.188.128.67]) by mx0a-00823401.pphosted.com (PPS) with ESMTPS id 3xhfb73r8r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 16 Apr 2024 17:24:58 +0000 (GMT) Received: from va32lmmrp01.lenovo.com (va32lmmrp01.mot.com [10.62.177.113]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ilclpfpp01.lenovo.com (Postfix) with ESMTPS id 4VJrWs1z69zfBZq; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Received: from ilclbld243.mot.com (ilclbld243.mot.com [100.64.22.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: mbland) by va32lmmrp01.lenovo.com (Postfix) with ESMTPSA id 4VJrWs0bYbz2VZS4; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Message-Id: <20240416122254.868007168-2-mbland@motorola.com> In-Reply-To: <20240416122254.868007168-1-mbland@motorola.com> References: <20240416122254.868007168-1-mbland@motorola.com> To: linux-mm@kvack.org Cc: Maxwell Bland , linux-kernel@vger.kernel.org, Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes From: Maxwell Bland Date: Tue, 2 Apr 2024 15:15:01 -0500 Subject: [PATCH 1/5] mm: allow arch refinement/skip for vmap alloc X-Proofpoint-GUID: m5q-_kUrWpAiWSiKGKl9KqD-wJ45KyMZ X-Proofpoint-ORIG-GUID: m5q-_kUrWpAiWSiKGKl9KqD-wJ45KyMZ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-16_14,2024-04-16_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 impostorscore=0 mlxscore=0 suspectscore=0 adultscore=0 lowpriorityscore=0 malwarescore=0 clxscore=1015 mlxlogscore=999 phishscore=0 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2404010003 definitions=main-2404160108 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Makes red black tree allocation more flexible on a per-architecture basis by introducing an optional hooks to refine the red-black tree structuring and exposing vmalloc functions for clipping vmap areas, finding vmap areas, and inserting vmap areas. With this patch, the red-black vmap tree can be refined to account for architecture-specific memory management operations, most notably address space layout randomization, as these features conflict with generic management of a single vmalloc_start to vmalloc_end range as given by mm/vmalloc.c. For example, x86 is forced to restrict aslr to 1024 possible locations, which is a very, very small number, and arm64 breaks standard code/data partitioning altogether, which prevents the enforcement of performant immmutability on kernel page tables. Signed-off-by: Maxwell Bland --- include/linux/vmalloc.h | 24 ++++++++++++++++++++++++ mm/vmalloc.c | 16 ++++++++++------ 2 files changed, 34 insertions(+), 6 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 98ea90e90439..3c5ce7ee0bea 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -12,6 +12,7 @@ =20 #include =20 +struct kmem_cache; struct vm_area_struct; /* vma defining user mapping in mm_types.h */ struct notifier_block; /* in notifier.h */ struct iov_iter; /* in uio.h */ @@ -125,6 +126,21 @@ static inline pgprot_t arch_vmap_pgprot_tagged(pgprot_= t prot) } #endif =20 +#ifndef arch_skip_va +static inline bool arch_skip_va(struct vmap_area *va, unsigned long vstart) +{ + return false; +} +#endif + +#ifndef arch_refine_vmap_space +static inline void arch_refine_vmap_space(struct rb_root *root, + struct list_head *head, + struct kmem_cache *cachep) +{ +} +#endif + /* * Highlevel APIs for driver use */ @@ -214,6 +230,14 @@ extern struct vm_struct *__get_vm_area_caller(unsigned= long size, void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); +extern void insert_vmap_area_augment(struct vmap_area *va, struct rb_node = *from, + struct rb_root *root, + struct list_head *head); +extern int va_clip(struct rb_root *root, struct list_head *head, + struct vmap_area *va, unsigned long nva_start_addr, + unsigned long size); +extern struct vmap_area *__find_vmap_area(unsigned long addr, + struct rb_root *root); struct vmap_area *find_vmap_area(unsigned long addr); =20 static inline bool is_vm_area_hugepages(const void *addr) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 68fa001648cc..de4577a3708e 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -989,7 +989,7 @@ unsigned long vmalloc_nr_pages(void) return atomic_long_read(&nr_vmalloc_pages); } =20 -static struct vmap_area *__find_vmap_area(unsigned long addr, struct rb_ro= ot *root) +struct vmap_area *__find_vmap_area(unsigned long addr, struct rb_root *roo= t) { struct rb_node *n =3D root->rb_node; =20 @@ -1322,7 +1322,7 @@ insert_vmap_area(struct vmap_area *va, link_va(va, root, parent, link, head); } =20 -static void +void insert_vmap_area_augment(struct vmap_area *va, struct rb_node *from, struct rb_root *root, struct list_head *head) @@ -1501,7 +1501,7 @@ find_vmap_lowest_match(struct rb_root *root, unsigned= long size, vstart < va->va_start) { node =3D node->rb_left; } else { - if (is_within_this_va(va, size, align, vstart)) + if (!arch_skip_va(va, vstart) && is_within_this_va(va, size, align, vst= art)) return va; =20 /* @@ -1522,7 +1522,8 @@ find_vmap_lowest_match(struct rb_root *root, unsigned= long size, */ while ((node =3D rb_parent(node))) { va =3D rb_entry(node, struct vmap_area, rb_node); - if (is_within_this_va(va, size, align, vstart)) + if (!arch_skip_va(va, vstart) && + is_within_this_va(va, size, align, vstart)) return va; =20 if (get_subtree_max_size(node->rb_right) >=3D length && @@ -1554,7 +1555,7 @@ find_vmap_lowest_linear_match(struct list_head *head,= unsigned long size, struct vmap_area *va; =20 list_for_each_entry(va, head, list) { - if (!is_within_this_va(va, size, align, vstart)) + if (arch_skip_va(va, vstart) || !is_within_this_va(va, size, align, vsta= rt)) continue; =20 return va; @@ -1617,7 +1618,7 @@ classify_va_fit_type(struct vmap_area *va, return type; } =20 -static __always_inline int +__always_inline int va_clip(struct rb_root *root, struct list_head *head, struct vmap_area *va, unsigned long nva_start_addr, unsigned long size) @@ -5129,4 +5130,7 @@ void __init vmalloc_init(void) vmap_node_shrinker->count_objects =3D vmap_node_shrink_count; vmap_node_shrinker->scan_objects =3D vmap_node_shrink_scan; shrinker_register(vmap_node_shrinker); + + arch_refine_vmap_space(&free_vmap_area_root, &free_vmap_area_list, + vmap_area_cachep); } --=20 2.39.2 From nobody Mon Feb 9 10:34:32 2026 Received: from mx0a-00823401.pphosted.com (mx0a-00823401.pphosted.com [148.163.148.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB0ED1F956; Tue, 16 Apr 2024 17:25:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.148.104 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713288338; cv=none; b=lW3mQ1ytAgnN8Xo0f/p9qrjAA6FlJfDoFrj1rlmZUQesXdkoIqv8GhyE+38/w7iUbdCaESQspikaQinF7ikQpRFDrRIeAeDjqtW5wjSSAxz2pnGSA+r2P4rKtB2zHZiA/7n0LGMUhK8qQnnqYD5aQtUxhB8sXB4MRdyNC6r2ZEk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713288338; c=relaxed/simple; bh=5A/wZwvAMjpCBh3rhO1cku9B/yboVsBZLlPIaX5Kru0=; h=Message-Id:In-Reply-To:References:To:Cc:From:Date:Subject; b=DPxzb5Rj0Wm5aAPgHpfib1m4n+LSozss7TCrwJQ2SSEX76bBYkj25ILnj4nf7PFPutgL7b8Z5P4C10muY4xheh4yPgG4y9GA+UIhVIgI92Tj3zoRsVvrGV59J6DJfbA95EsL/xNoR6Dck/oPio4YBW/QZvrkVwrZ8J7fLJyY4h8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com; spf=pass smtp.mailfrom=motorola.com; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b=sw41Pz1c; arc=none smtp.client-ip=148.163.148.104 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=motorola.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b="sw41Pz1c" Received: from pps.filterd (m0355085.ppops.net [127.0.0.1]) by mx0a-00823401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 43GEh0Zk006822; Tue, 16 Apr 2024 17:24:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=motorola.com; h= message-id:in-reply-to:references:to:cc:from:date:subject; s= DKIM202306; bh=WtpkOtfUVklTETlWsQkVzQfkRpaN5YigWXkv8+BPIYU=; b=s w41Pz1cA5VDn6mBbbiYwn9PX+cQjWhh9XdAvz2IOJX7TMz/0Sxn6N2jrLkdOKdYg /whIbQ8ivQ7viekFNKMrGyY9UqFud9GSPI7JYPrckgL+CJ26q4vezQdZVCeNROMb FMPWF5p9rQE7TYFBuFddPpFXAVsUEUhSIP8h2RlIYEzqSzMZ+jp38y+HHs6seeql aYk94e1mJinCaILLf6M4qs6ATWjPC+/2OB/qMzUD2mxOYZjRMCaSWfxu1kwHbkUM FC1R8PRkwUaAcDRVlRTTykk+8VlYvW8Ud/CN7bGBnURWVwBCZ5lLO9btQw/2hxjE qiDJ4vgvm7uZHhIhMdRRg== Received: from va32lpfpp04.lenovo.com ([104.232.228.24]) by mx0a-00823401.pphosted.com (PPS) with ESMTPS id 3xhea8bv1a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 16 Apr 2024 17:24:58 +0000 (GMT) Received: from va32lmmrp01.lenovo.com (va32lmmrp01.mot.com [10.62.177.113]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by va32lpfpp04.lenovo.com (Postfix) with ESMTPS id 4VJrWs0vh7zj9hF; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Received: from ilclbld243.mot.com (ilclbld243.mot.com [100.64.22.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: mbland) by va32lmmrp01.lenovo.com (Postfix) with ESMTPSA id 4VJrWs0fdFz2VZS6; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Message-Id: <20240416122254.868007168-3-mbland@motorola.com> In-Reply-To: <20240416122254.868007168-1-mbland@motorola.com> References: <20240416122254.868007168-1-mbland@motorola.com> To: linux-arm-kernel@lists.infradead.org Cc: Maxwell Bland , linux-kernel@vger.kernel.org, Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Mark Rutland , Ard Biesheuvel , Maxwell Bland , Kees Cook , Sami Tolvanen , Baoquan He , Jonathan Cameron , Greg Kroah-Hartman , Ryo Takakura , James Morse , Christophe Leroy , bpf@vger.kernel.org From: Maxwell Bland Date: Wed, 3 Apr 2024 16:08:15 -0500 Subject: [PATCH 2/5] arm64: mm: code and data partitioning for aslr X-Proofpoint-GUID: oSkwsmd8buYDKwvkDlsObdQxE94z6XCb X-Proofpoint-ORIG-GUID: oSkwsmd8buYDKwvkDlsObdQxE94z6XCb X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-16_14,2024-04-16_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 bulkscore=0 phishscore=0 adultscore=0 spamscore=0 lowpriorityscore=0 clxscore=1015 mlxlogscore=999 suspectscore=0 impostorscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2404010003 definitions=main-2404160108 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Uses hooks in the vmalloc infrastructure to prevent interleaving code and data pages, working to both maintain compatible management assumptions made by non-arch-specific code and make management of these regions more precise and conformant, allowing, for example, the maintenance of PXNTable bits on dynamically allocated memory or the immutability of certain page middle directory and higher level descriptors. Signed-off-by: Maxwell Bland --- arch/arm64/include/asm/module.h | 12 +++++ arch/arm64/include/asm/vmalloc.h | 17 ++++++- arch/arm64/kernel/Makefile | 2 +- arch/arm64/kernel/module.c | 7 ++- arch/arm64/kernel/probes/kprobes.c | 7 +-- arch/arm64/kernel/setup.c | 4 ++ arch/arm64/kernel/vmalloc.c | 71 ++++++++++++++++++++++++++++++ arch/arm64/mm/ptdump.c | 4 +- arch/arm64/net/bpf_jit_comp.c | 8 ++-- 9 files changed, 117 insertions(+), 15 deletions(-) create mode 100644 arch/arm64/kernel/vmalloc.c diff --git a/arch/arm64/include/asm/module.h b/arch/arm64/include/asm/modul= e.h index 79550b22ba19..e50d7a240ad7 100644 --- a/arch/arm64/include/asm/module.h +++ b/arch/arm64/include/asm/module.h @@ -65,4 +65,16 @@ static inline const Elf_Shdr *find_section(const Elf_Ehd= r *hdr, return NULL; } =20 +extern u64 module_direct_base __ro_after_init; +extern u64 module_plt_base __ro_after_init; + +int __init module_init_limits(void); + +#define MODULES_ASLR_START ((module_plt_base) ? module_plt_base : \ + module_direct_base) +#define MODULES_ASLR_END ((module_plt_base) ? module_plt_base + SZ_2G : \ + module_direct_base + SZ_128M) + +void *module_alloc(unsigned long size); + #endif /* __ASM_MODULE_H */ diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmal= loc.h index 38fafffe699f..93f8f1e2b1ce 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -4,6 +4,9 @@ #include #include =20 +struct vmap_area; +struct kmem_cache; + #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP =20 #define arch_vmap_pud_supported arch_vmap_pud_supported @@ -23,7 +26,7 @@ static inline bool arch_vmap_pmd_supported(pgprot_t prot) return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } =20 -#endif +#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ =20 #define arch_vmap_pgprot_tagged arch_vmap_pgprot_tagged static inline pgprot_t arch_vmap_pgprot_tagged(pgprot_t prot) @@ -31,4 +34,16 @@ static inline pgprot_t arch_vmap_pgprot_tagged(pgprot_t = prot) return pgprot_tagged(prot); } =20 +#ifdef CONFIG_RANDOMIZE_BASE + +#define arch_skip_va arch_skip_va +inline bool arch_skip_va(struct vmap_area *va, unsigned long vstart); + +#define arch_refine_vmap_space arch_refine_vmap_space +inline void arch_refine_vmap_space(struct rb_root *root, + struct list_head *head, + struct kmem_cache *cachep); + +#endif /* CONFIG_RANDOMIZE_BASE */ + #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 763824963ed1..4298a2168544 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -56,7 +56,7 @@ obj-$(CONFIG_ACPI) +=3D acpi.o obj-$(CONFIG_ACPI_NUMA) +=3D acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) +=3D acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) +=3D paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) +=3D kaslr.o +obj-$(CONFIG_RANDOMIZE_BASE) +=3D kaslr.o vmalloc.o obj-$(CONFIG_HIBERNATION) +=3D hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) +=3D elfcore.o obj-$(CONFIG_KEXEC_CORE) +=3D machine_kexec.o relocate_kernel.o \ diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 47e0be610bb6..58329b27624d 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -26,8 +26,8 @@ #include #include =20 -static u64 module_direct_base __ro_after_init =3D 0; -static u64 module_plt_base __ro_after_init =3D 0; +u64 module_direct_base __ro_after_init; +u64 module_plt_base __ro_after_init; =20 /* * Choose a random page-aligned base address for a window of 'size' bytes = which @@ -66,7 +66,7 @@ static u64 __init random_bounding_box(u64 size, u64 start= , u64 end) * we may fall back to PLTs where they could have been avoided, but this k= eeps * the logic significantly simpler. */ -static int __init module_init_limits(void) +int __init module_init_limits(void) { u64 kernel_end =3D (u64)_end; u64 kernel_start =3D (u64)_text; @@ -108,7 +108,6 @@ static int __init module_init_limits(void) =20 return 0; } -subsys_initcall(module_init_limits); =20 void *module_alloc(unsigned long size) { diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/= kprobes.c index 327855a11df2..89968f05177f 100644 --- a/arch/arm64/kernel/probes/kprobes.c +++ b/arch/arm64/kernel/probes/kprobes.c @@ -131,9 +131,10 @@ int __kprobes arch_prepare_kprobe(struct kprobe *p) =20 void *alloc_insn_page(void) { - return __vmalloc_node_range(PAGE_SIZE, 1, VMALLOC_START, VMALLOC_END, - GFP_KERNEL, PAGE_KERNEL_ROX, VM_FLUSH_RESET_PERMS, - NUMA_NO_NODE, __builtin_return_address(0)); + return __vmalloc_node_range(PAGE_SIZE, 1, MODULES_ASLR_START, + MODULES_ASLR_END, GFP_KERNEL, PAGE_KERNEL_ROX, + VM_FLUSH_RESET_PERMS, NUMA_NO_NODE, + __builtin_return_address(0)); } =20 /* arm kprobe: install breakpoint in text */ diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 65a052bf741f..908ee0ccc606 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -53,6 +53,7 @@ #include #include #include +#include =20 static int num_standard_resources; static struct resource *standard_resources; @@ -321,6 +322,7 @@ void __init __no_sanitize_address setup_arch(char **cmd= line_p) =20 arm64_memblock_init(); =20 + paging_init(); =20 acpi_table_upgrade(); @@ -366,6 +368,8 @@ void __init __no_sanitize_address setup_arch(char **cmd= line_p) "This indicates a broken bootloader or old kernel\n", boot_args[1], boot_args[2], boot_args[3]); } + + module_init_limits(); } =20 static inline bool cpu_can_disable(unsigned int cpu) diff --git a/arch/arm64/kernel/vmalloc.c b/arch/arm64/kernel/vmalloc.c new file mode 100644 index 000000000000..00a463f3692f --- /dev/null +++ b/arch/arm64/kernel/vmalloc.c @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * AArch64 vmap area management code + * + * Author: Maxwell Bland + */ + +#include +#include + +#include + +/* + * Prevents the allocation of new vmap_areas from dynamic code + * region if the virtual address requested is not explicitly the + * module region. + */ +inline bool arch_skip_va(struct vmap_area *va, unsigned long vstart) +{ + return (vstart !=3D MODULES_ASLR_START && + va->va_start >=3D MODULES_ASLR_START && + va->va_end <=3D MODULES_ASLR_END); +} + +/* + * Splits a vmap area in two and allocates a new area if needed + */ +inline struct vmap_area * +try_split_alloc_vmap_area(struct rb_root *root, + struct list_head *head, + struct kmem_cache *vmap_area_cachep, + unsigned long addr) +{ + struct vmap_area *va; + int ret; + struct vmap_area *lva =3D NULL; + + va =3D __find_vmap_area(addr, root); + if (!va) { + pr_err("%s: could not find vmap\n", __func__); + return NULL; + } + + lva =3D kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT); + if (!lva) { + pr_err("%s: unable to allocate va for range\n", __func__); + return NULL; + } + lva->va_start =3D addr; + lva->va_end =3D va->va_end; + ret =3D va_clip(root, head, va, addr, va->va_end - addr); + if (WARN_ON_ONCE(ret)) { + pr_err("%s: unable to clip code base region\n", __func__); + kmem_cache_free(vmap_area_cachep, lva); + return NULL; + } + insert_vmap_area_augment(lva, NULL, root, head); + return lva; +} + +/* + * Run during vmalloc_init, ensures that there exist explicit rb tree + * node delineations between code and data + */ +inline void arch_refine_vmap_space(struct rb_root *root, + struct list_head *head, + struct kmem_cache *cachep) +{ + try_split_alloc_vmap_area(root, head, cachep, MODULES_ASLR_START); + try_split_alloc_vmap_area(root, head, cachep, MODULES_ASLR_END); +} diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 6986827e0d64..796231a4fd63 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -261,9 +261,7 @@ static void note_page(struct ptdump_state *pt_st, unsig= ned long addr, int level, } pt_dump_seq_printf(st->seq, "%9lu%c %s", delta, *unit, pg_level[st->level].name); - if (st->current_prot && pg_level[st->level].bits) - dump_prot(st, pg_level[st->level].bits, - pg_level[st->level].num); + dump_prot(st, pg_level[st->level].bits, pg_level[st->level].num); pt_dump_seq_puts(st->seq, "\n"); =20 if (addr >=3D st->marker[1].start_address) { diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 122021f9bdfc..6ed6e00b8b4a 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -13,6 +13,8 @@ #include #include #include +#include +#include =20 #include #include @@ -1790,18 +1792,18 @@ void *bpf_arch_text_copy(void *dst, void *src, size= _t len) =20 u64 bpf_jit_alloc_exec_limit(void) { - return VMALLOC_END - VMALLOC_START; + return MODULES_ASLR_END - MODULES_ASLR_START; } =20 void *bpf_jit_alloc_exec(unsigned long size) { /* Memory is intended to be executable, reset the pointer tag. */ - return kasan_reset_tag(vmalloc(size)); + return kasan_reset_tag(module_alloc(size)); } =20 void bpf_jit_free_exec(void *addr) { - return vfree(addr); + return module_memfree(addr); } =20 /* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ --=20 2.39.2 From nobody Mon Feb 9 10:34:32 2026 Received: from mx0a-00823401.pphosted.com (mx0a-00823401.pphosted.com [148.163.148.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 064CE137929; Tue, 16 Apr 2024 19:19:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.148.104 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713295168; cv=none; b=qOgxVU+usNDT6jbLsGkFz/hP5uEwRn4krOFIvRFGUGRTrq0iRe+5KIBrR02TTERBq5q3GjoPSLJxLADCdW8EbalMWYeLi4REEVURbED3f0p37jTz8mr9JthJSl2zXIVQ8qpLKDp2edNhRMQZGuxuKRoQWPeUNi/xIYce5gOdAgI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713295168; c=relaxed/simple; bh=skpJt66dHhajZRxQ+0OwGnjgJvZfMcLix3I7ncdjrrg=; h=Message-Id:In-Reply-To:References:To:Cc:From:Subject:Date; b=YqwEjBC0LCkGgrY7BXvcvCNSXiPrXvdpANZuz8tqlSr3AIpbNw2mW17Iz9dX7HXoroOrf3oOPjevBVxadc72vVDIaeagOFvbMQSuH2CuRtNJjmqja9XybBi6DEd1Y9OTMD0yeVDYqaiRXbcJXWIgYH64gfk5tTHKJb5GgaGzIh4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com; spf=pass smtp.mailfrom=motorola.com; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b=Qla0/1la; arc=none smtp.client-ip=148.163.148.104 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=motorola.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b="Qla0/1la" Received: from pps.filterd (m0355086.ppops.net [127.0.0.1]) by mx0a-00823401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 43GHWPP4029238; Tue, 16 Apr 2024 19:19:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=motorola.com; h= message-id:in-reply-to:references:to:cc:from:subject:date; s= DKIM202306; bh=9vTyU2l3t+ufezl9NM7trdVAZw+NQOYaPFZvexa2H5c=; b=Q la0/1lagQ+CRwBPmfuSMHqN/BmkLBAxYzvtE5AgErO/S0QtIQyXCkBGaZQh/o+e8 U1sruERg1FQY7Gxhx1ZDI55iL21Ck2ejCYnX2DFnh2/85yarjsbufcIcWAcqpsQu QChNWY0l++bLFGpMgyifp69z9tpd43PzV6VWZGrbPF/BxpdxqrmwKyHobmPvQ2Xy qR8+dgxqO2ZJCLu8z0+M5RxIKSr4iBso5Cy+2TZBJgpxFGxsraDqmmcN9xQxQ4XR uH5BYRrkhUDg7Syx5AgOy2XBoSs7JlpQ47UYf+5hTgmf4b/1YkF5y4GxQO+ngpCS sSghLiKpHLFrj734kjoNQ== Received: from ilclpfpp01.lenovo.com ([144.188.128.67]) by mx0a-00823401.pphosted.com (PPS) with ESMTPS id 3xhwtrr5qb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 16 Apr 2024 19:19:23 +0000 (GMT) Received: from va32lmmrp02.lenovo.com (va32lmmrp02.mot.com [10.62.176.191]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ilclpfpp01.lenovo.com (Postfix) with ESMTPS id 4VJv3t1m7yzdDt6; Tue, 16 Apr 2024 19:19:22 +0000 (UTC) Received: from ilclbld243.mot.com (ilclbld243.mot.com [100.64.22.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: mbland) by va32lmmrp02.lenovo.com (Postfix) with ESMTPSA id 4VJv3t09CVz2Z11p; Tue, 16 Apr 2024 19:19:22 +0000 (UTC) Message-Id: <20240416122254.868007168-4-mbland@motorola.com> In-Reply-To: <20240416122254.868007168-1-mbland@motorola.com> References: <20240416122254.868007168-1-mbland@motorola.com> To: linux-mm@kvack.org Cc: Maxwell Bland , Richard Henderson , Ivan Kokshaysky , Matt Turner , Vineet Gupta , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Russell King , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Catalin Marinas , Will Deacon , Guo Ren , Brian Cain , Huacai Chen , WANG Xuerui , Geert Uytterhoeven , Sam Creasey , Michal Simek , Thomas Bogendoerfer , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Andreas Larsson , Richard Weinberger , Anton Ivanov , Johannes Berg , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Chris Zankel , Max Filippov , Andrew Morton , Muchun Song , Dennis Zhou , Tejun Heo , Christoph Lameter , Maxwell Bland , Linus Walleij , David Hildenbrand , Arnd Bergmann , Ard Biesheuvel , Ryan Roberts , Mark Rutland , Nikhil V , Rick Edgecombe , Baolin Wang , Bibo Mao , Tianrui Zhao , Randy Dunlap , Vlastimil Babka , Kent Overstreet , Peter Xu , Jiangfeng Xiao , Alexandre Ghiti , Jisheng Zhang , Conor Dooley , Mason Huo , Sia Jee Heng , Song Shuai , Gerald Schaefer , Qi Zheng , Hugh Dickins , Jason Gunthorpe , Breno Leitao , Josh Poimboeuf , linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, kvm@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, Mark Rutland , Greg Kroah-Hartman , Christoph Hellwig , Christophe Leroy , David Hildenbrand , Conor Dooley , linux-um@lists.infradead.org From: Maxwell Bland Subject: [PATCH 3/5 RESEND] mm: add vaddr param to pmd_populate_kernel Date: Tue, 16 Apr 2024 14:18:17 -0500 X-Proofpoint-ORIG-GUID: 8j___v2Rb6e_FTNDOgqru6h3NJbVVmB0 X-Proofpoint-GUID: 8j___v2Rb6e_FTNDOgqru6h3NJbVVmB0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-16_17,2024-04-16_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 bulkscore=0 suspectscore=0 clxscore=1015 impostorscore=0 malwarescore=0 spamscore=0 lowpriorityscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2404010003 definitions=main-2404160122 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch affords each architecture the ability to condition the population of page middle directory entries on the virtual address being allocated, matching existing PTE infrastructure, easing the necessity of performing a reverse page table walk in cases where the population context is not readily accessible, i.e. dynamic vmalloc calls on arm64. To achieve this goal, it modifies every call and implementation of the pmd_populate_kernel function across architectures, ensuring uniform adoption across all kernel deployments. Signed-off-by: Maxwell Bland --- Hi all, Thank you for taking the time to review this change. This effects many subarchitectures so the maintainers list is large. Apologies in advance if there is a specific maintainer I should have spoken with directly for deployment across subprojects. The reason for such a sweeping change is from=20 lore.kernel.org/all/cf5409c3-254a-459b-8969-429db2ec6439@redhat.com It is my understanding as well that some subarchitectures may have separate "next" or development branches ahead of the main upstream linux. Please let me know if a cherry-pick to that branch is desired and I will do my best to check out and deploy it as possible. arch/alpha/include/asm/pgalloc.h | 5 +++-- arch/arc/include/asm/pgalloc.h | 3 ++- arch/arc/mm/highmem.c | 2 +- arch/arm/include/asm/kfence.h | 2 +- arch/arm/include/asm/pgalloc.h | 3 ++- arch/arm/mm/kasan_init.c | 2 +- arch/arm/mm/mmu.c | 2 +- arch/arm64/include/asm/pgalloc.h | 3 ++- arch/arm64/mm/trans_pgd.c | 2 +- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/include/asm/pgalloc.h | 3 ++- arch/loongarch/mm/init.c | 2 +- arch/loongarch/mm/kasan_init.c | 2 +- arch/m68k/include/asm/mcf_pgalloc.h | 2 +- arch/m68k/include/asm/motorola_pgalloc.h | 3 ++- arch/m68k/include/asm/sun3_pgalloc.h | 3 ++- arch/microblaze/include/asm/pgalloc.h | 2 +- arch/mips/include/asm/pgalloc.h | 2 +- arch/mips/kvm/mmu.c | 2 +- arch/nios2/include/asm/pgalloc.h | 2 +- arch/openrisc/include/asm/pgalloc.h | 2 +- arch/parisc/include/asm/pgalloc.h | 5 +++-- arch/parisc/mm/init.c | 6 +++--- arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 2 +- arch/powerpc/include/asm/nohash/32/pgalloc.h | 2 +- arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +- arch/powerpc/mm/kasan/init_32.c | 4 ++-- arch/powerpc/mm/kasan/init_book3e_64.c | 9 ++++++--- arch/powerpc/mm/kasan/init_book3s_64.c | 7 +++++-- arch/powerpc/mm/nohash/book3e_pgtable.c | 2 +- arch/powerpc/mm/pgtable_32.c | 4 ++-- arch/riscv/include/asm/pgalloc.h | 2 +- arch/riscv/kernel/hibernate.c | 2 +- arch/s390/include/asm/pgalloc.h | 2 +- arch/sh/include/asm/pgalloc.h | 2 +- arch/sh/mm/init.c | 2 +- arch/sparc/include/asm/pgalloc_32.h | 3 ++- arch/sparc/include/asm/pgalloc_64.h | 4 ++-- arch/sparc/mm/init_64.c | 8 ++++---- arch/um/include/asm/pgalloc.h | 4 ++-- arch/x86/include/asm/pgalloc.h | 3 ++- arch/x86/mm/init_64.c | 14 +++++++++++--- arch/x86/mm/ioremap.c | 2 +- arch/x86/mm/kasan_init_64.c | 2 +- arch/xtensa/include/asm/pgalloc.h | 2 +- include/linux/mm.h | 4 ++-- mm/hugetlb_vmemmap.c | 4 ++-- mm/kasan/init.c | 14 +++++++++----- mm/memory.c | 4 ++-- mm/percpu.c | 2 +- mm/pgalloc-track.h | 3 ++- mm/sparse-vmemmap.c | 2 +- 55 files changed, 107 insertions(+), 78 deletions(-) diff --git a/arch/alpha/include/asm/pgalloc.h b/arch/alpha/include/asm/pgal= loc.h index 68be7adbfe58..1d3d86cad3cc 100644 --- a/arch/alpha/include/asm/pgalloc.h +++ b/arch/alpha/include/asm/pgalloc.h @@ -7,7 +7,7 @@ =20 #include =20 -/* =20 +/* * Allocate and free page tables. The xxx_kernel() versions are * used to allocate a kernel page table - this turns on ASN bits * if any. @@ -20,7 +20,8 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t = pte) } =20 static inline void -pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) +pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte, + unsigned long vaddr) { pmd_set(pmd, pte); } diff --git a/arch/arc/include/asm/pgalloc.h b/arch/arc/include/asm/pgalloc.h index 096b8ef58edb..c0ebfa44b204 100644 --- a/arch/arc/include/asm/pgalloc.h +++ b/arch/arc/include/asm/pgalloc.h @@ -34,7 +34,8 @@ #include =20 static inline void -pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) +pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte, + unsigned long vaddr) { /* * The cast to long below is OK in 32-bit PAE40 regime with long long pte diff --git a/arch/arc/mm/highmem.c b/arch/arc/mm/highmem.c index c79912a6b196..2d327cf35722 100644 --- a/arch/arc/mm/highmem.c +++ b/arch/arc/mm/highmem.c @@ -57,7 +57,7 @@ static noinline pte_t * __init alloc_kmap_pgtable(unsigne= d long kvaddr) panic("%s: Failed to allocate %lu bytes align=3D0x%lx\n", __func__, PAGE_SIZE, PAGE_SIZE); =20 - pmd_populate_kernel(&init_mm, pmd_k, pte_k); + pmd_populate_kernel(&init_mm, pmd_k, pte_k, kvaddr); return pte_k; } =20 diff --git a/arch/arm/include/asm/kfence.h b/arch/arm/include/asm/kfence.h index 7980d0f2271f..dd4e4325d354 100644 --- a/arch/arm/include/asm/kfence.h +++ b/arch/arm/include/asm/kfence.h @@ -19,7 +19,7 @@ static inline int split_pmd_page(pmd_t *pmd, unsigned lon= g addr) =20 for (i =3D 0; i < PTRS_PER_PTE; i++) set_pte_ext(pte + i, pfn_pte(pfn + i, PAGE_KERNEL), 0); - pmd_populate_kernel(&init_mm, pmd, pte); + pmd_populate_kernel(&init_mm, pmd, pte, addr); =20 flush_tlb_kernel_range(addr, addr + PMD_SIZE); return 0; diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h index a17f01235c29..0a88346db17e 100644 --- a/arch/arm/include/asm/pgalloc.h +++ b/arch/arm/include/asm/pgalloc.h @@ -122,7 +122,8 @@ static inline void __pmd_populate(pmd_t *pmdp, phys_add= r_t pte, * Ensure that we always set both PMD entries. */ static inline void -pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep) +pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep, + unsigned long vaddr) { /* * The pmd must be loaded with the physical address of the PTE table diff --git a/arch/arm/mm/kasan_init.c b/arch/arm/mm/kasan_init.c index 111d4f703136..9b3af2dce71e 100644 --- a/arch/arm/mm/kasan_init.c +++ b/arch/arm/mm/kasan_init.c @@ -111,7 +111,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsi= gned long addr, __func__, addr); return; } - pmd_populate_kernel(&init_mm, pmdp, p); + pmd_populate_kernel(&init_mm, pmdp, p, addr); flush_pmd_entry(pmdp); } =20 diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c index c24e29c0b9a4..3cfed8dc4a19 100644 --- a/arch/arm/mm/mmu.c +++ b/arch/arm/mm/mmu.c @@ -384,7 +384,7 @@ void __init early_fixmap_init(void) !=3D FIXADDR_TOP >> PMD_SHIFT); =20 pmd =3D fixmap_pmd(FIXADDR_TOP); - pmd_populate_kernel(&init_mm, pmd, bm_pte); + pmd_populate_kernel(&init_mm, pmd, bm_pte, __fix_to_virt(FIXADDR_TOP)); =20 pte_offset_fixmap =3D pte_offset_early_fixmap; } diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index 8ff5f2a2579e..5785272144e8 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -124,7 +124,8 @@ static inline void __pmd_populate(pmd_t *pmdp, phys_add= r_t ptep, * of the mm address space. */ static inline void -pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep) +pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep, + unsigned long vaddr) { VM_BUG_ON(mm && mm !=3D &init_mm); __pmd_populate(pmdp, __pa(ptep), PMD_TYPE_TABLE | PMD_TABLE_UXN); diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 5139a28130c0..f84244d13099 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -69,7 +69,7 @@ static int copy_pte(struct trans_pgd_info *info, pmd_t *d= st_pmdp, dst_ptep =3D trans_alloc(info); if (!dst_ptep) return -ENOMEM; - pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep, addr); dst_ptep =3D pte_offset_kernel(dst_pmdp, start); =20 src_ptep =3D pte_offset_kernel(src_pmdp, start); diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgallo= c.h index 9c84c9012e53..f2c244c58acf 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -11,7 +11,7 @@ #include =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { set_pmd(pmd, __pmd(__pa(pte))); } diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/= pgalloc.h index 55988625e6fb..2be773a5ffeb 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -62,7 +62,7 @@ static inline void pmd_populate(struct mm_struct *mm, pmd= _t *pmd, * kernel map of the active thread who's calling pmd_populate_kernel... */ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { extern spinlock_t kmap_gen_lock; pmd_t *ppmd; diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/= asm/pgalloc.h index 4e2d6b7ca2ee..6384391e69bd 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -13,7 +13,8 @@ #include =20 static inline void pmd_populate_kernel(struct mm_struct *mm, - pmd_t *pmd, pte_t *pte) + pmd_t *pmd, pte_t *pte, + unsigned long vaddr) { set_pmd(pmd, __pmd((unsigned long)pte)); } diff --git a/arch/loongarch/mm/init.c b/arch/loongarch/mm/init.c index 4dd53427f657..b8952899b120 100644 --- a/arch/loongarch/mm/init.c +++ b/arch/loongarch/mm/init.c @@ -200,7 +200,7 @@ pte_t * __init populate_kernel_pte(unsigned long addr) pte =3D memblock_alloc(PAGE_SIZE, PAGE_SIZE); if (!pte) panic("%s: Failed to allocate memory\n", __func__); - pmd_populate_kernel(&init_mm, pmd, pte); + pmd_populate_kernel(&init_mm, pmd, pte, addr); } =20 return pte_offset_kernel(pmd, addr); diff --git a/arch/loongarch/mm/kasan_init.c b/arch/loongarch/mm/kasan_init.c index c608adc99845..51d40ff43aa9 100644 --- a/arch/loongarch/mm/kasan_init.c +++ b/arch/loongarch/mm/kasan_init.c @@ -110,7 +110,7 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsi= gned long addr, int node, __pa_symbol(kasan_early_shadow_pte) : kasan_alloc_zeroed_page(node); if (!early) memcpy(__va(pte_phys), kasan_early_shadow_pte, sizeof(kasan_early_shado= w_pte)); - pmd_populate_kernel(NULL, pmdp, (pte_t *)__va(pte_phys)); + pmd_populate_kernel(NULL, pmdp, (pte_t *)__va(pte_phys), addr); } =20 return pte_offset_kernel(pmdp, addr); diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mc= f_pgalloc.h index 302c5bf67179..989a1aaa8aa1 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -30,7 +30,7 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigne= d long address) =20 #define pmd_populate(mm, pmd, pte) (pmd_val(*pmd) =3D (unsigned long)(pte)) =20 -#define pmd_populate_kernel pmd_populate +#define pmd_populate_kernel(mm, pmd, pte, vaddr) pmd_populate(mm, pmd, pte) =20 static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pgtabl= e, unsigned long address) diff --git a/arch/m68k/include/asm/motorola_pgalloc.h b/arch/m68k/include/a= sm/motorola_pgalloc.h index 74a817d9387f..74aec6965981 100644 --- a/arch/m68k/include/asm/motorola_pgalloc.h +++ b/arch/m68k/include/asm/motorola_pgalloc.h @@ -79,7 +79,8 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) } =20 =20 -static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, p= te_t *pte) +static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, + pte_t *pte, unsigned long vaddr) { pmd_set(pmd, pte); } diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/s= un3_pgalloc.h index 4a137eecb6fe..550283e8bf4d 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -23,7 +23,8 @@ do { \ tlb_remove_page_ptdesc((tlb), page_ptdesc(pte)); \ } while (0) =20 -static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, p= te_t *pte) +static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, + pte_t *pte, unsigned long vaddr) { pmd_val(*pmd) =3D __pa((unsigned long)pte); } diff --git a/arch/microblaze/include/asm/pgalloc.h b/arch/microblaze/includ= e/asm/pgalloc.h index 6c33b05f730f..b3cc2cd8fc50 100644 --- a/arch/microblaze/include/asm/pgalloc.h +++ b/arch/microblaze/include/asm/pgalloc.h @@ -35,7 +35,7 @@ extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); #define pmd_populate(mm, pmd, pte) \ (pmd_val(*(pmd)) =3D (unsigned long)page_address(pte)) =20 -#define pmd_populate_kernel(mm, pmd, pte) \ +#define pmd_populate_kernel(mm, pmd, pte, vaddr) \ (pmd_val(*(pmd)) =3D (unsigned long) (pte)) =20 #endif /* _ASM_MICROBLAZE_PGALLOC_H */ diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgallo= c.h index f4440edcd8fe..fb71c8776a04 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -19,7 +19,7 @@ #include =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { set_pmd(pmd, __pmd((unsigned long)pte)); } diff --git a/arch/mips/kvm/mmu.c b/arch/mips/kvm/mmu.c index 467ee6b95ae1..47f48929a124 100644 --- a/arch/mips/kvm/mmu.c +++ b/arch/mips/kvm/mmu.c @@ -133,7 +133,7 @@ static pte_t *kvm_mips_walk_pgd(pgd_t *pgd, struct kvm_= mmu_memory_cache *cache, return NULL; new_pte =3D kvm_mmu_memory_cache_alloc(cache); clear_page(new_pte); - pmd_populate_kernel(NULL, pmd, new_pte); + pmd_populate_kernel(NULL, pmd, new_pte, addr); } return pte_offset_kernel(pmd, addr); } diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgal= loc.h index ce6bb8e74271..ea99d36a6fdd 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -15,7 +15,7 @@ #include =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { set_pmd(pmd, __pmd((unsigned long)pte)); } diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/as= m/pgalloc.h index c6a73772a546..304cf8955bec 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -25,7 +25,7 @@ =20 extern int mem_init_done; =20 -#define pmd_populate_kernel(mm, pmd, pte) \ +#define pmd_populate_kernel(mm, pmd, pte, vaddr) \ set_pmd(pmd, __pmd(_KERNPG_TABLE + __pa(pte))) =20 static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pg= alloc.h index e3e142b1c5c5..cba92c90a62a 100644 --- a/arch/parisc/include/asm/pgalloc.h +++ b/arch/parisc/include/asm/pgalloc.h @@ -61,13 +61,14 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t= *pmd) #endif =20 static inline void -pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) +pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte, unsigned= long vaddr) { set_pmd(pmd, __pmd((PxD_FLAG_PRESENT | PxD_FLAG_VALID) + (__u32)(__pa((unsigned long)pte) >> PxD_VALUE_SHIFT))); } =20 #define pmd_populate(mm, pmd, pte_page) \ - pmd_populate_kernel(mm, pmd, page_address(pte_page)) + pmd_populate_kernel(mm, pmd, page_address(pte_page), \ + (unsigned long)page_to_virt(pte_page)) =20 #endif diff --git a/arch/parisc/mm/init.c b/arch/parisc/mm/init.c index f876af56e13f..1cf3aae67023 100644 --- a/arch/parisc/mm/init.c +++ b/arch/parisc/mm/init.c @@ -390,7 +390,7 @@ static void __ref map_pages(unsigned long start_vaddr, pg_table =3D memblock_alloc(PAGE_SIZE, PAGE_SIZE); if (!pg_table) panic("page table allocation failed\n"); - pmd_populate_kernel(NULL, pmd, pg_table); + pmd_populate_kernel(NULL, pmd, pg_table, vaddr); } =20 pg_table =3D pte_offset_kernel(pmd, vaddr); @@ -481,7 +481,7 @@ void free_initmem(void) /* finally dump all the instructions which were cached, since the * pages are no-longer executable */ flush_icache_range(init_begin, init_end); -=09 + free_initmem_default(POISON_FREE_INITMEM); =20 /* set up a new led state on systems shipped LED State panel */ @@ -694,7 +694,7 @@ static void __init fixmap_init(void) if (!pte) panic("fixmap: pte allocation failed.\n"); =20 - pmd_populate_kernel(&init_mm, pmd, pte); + pmd_populate_kernel(&init_mm, pmd, pte, addr); =20 addr +=3D PAGE_SIZE; } while (addr < end); diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/in= clude/asm/book3s/32/pgalloc.h index dc5c039eb28e..b85105158686 100644 --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h @@ -26,7 +26,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *= pgd) /* #define pgd_populate(mm, pmd, pte) BUG() */ =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { *pmdp =3D __pmd(__pa(pte) | _PMD_PRESENT); } diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/in= clude/asm/book3s/64/pgalloc.h index dd2cff53a111..061c4be60166 100644 --- a/arch/powerpc/include/asm/book3s/64/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h @@ -156,7 +156,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tl= b, pmd_t *pmd, } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { *pmd =3D __pmd(__pgtable_ptr_val(pte) | PMD_VAL_BITS); } diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/in= clude/asm/nohash/32/pgalloc.h index 11eac371e7e0..2788ce005b95 100644 --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h @@ -15,7 +15,7 @@ /* #define pgd_populate(mm, pmd, pte) BUG() */ =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { if (IS_ENABLED(CONFIG_BOOKE)) *pmdp =3D __pmd((unsigned long)pte | _PMD_PRESENT); diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/in= clude/asm/nohash/64/pgalloc.h index e50b211becb3..d069443b4014 100644 --- a/arch/powerpc/include/asm/nohash/64/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h @@ -37,7 +37,7 @@ static inline void pud_populate(struct mm_struct *mm, pud= _t *pud, pmd_t *pmd) } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { pmd_set(pmd, (unsigned long)pte); } diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/boo= k3s64/radix_pgtable.c index 15e88f1439ec..a70063cd6f64 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -104,7 +104,7 @@ static int early_map_kernel_page(unsigned long ea, unsi= gned long pa, if (!pmd_present(*pmdp)) { ptep =3D early_alloc_pgtable(PAGE_SIZE, nid, region_start, region_end); - pmd_populate_kernel(&init_mm, pmdp, ptep); + pmd_populate_kernel(&init_mm, pmdp, ptep, ea); } ptep =3D pte_offset_kernel(pmdp, ea); =20 diff --git a/arch/powerpc/mm/kasan/init_32.c b/arch/powerpc/mm/kasan/init_3= 2.c index aa9aa11927b2..22df07fd1af5 100644 --- a/arch/powerpc/mm/kasan/init_32.c +++ b/arch/powerpc/mm/kasan/init_32.c @@ -47,7 +47,7 @@ int __init kasan_init_shadow_page_tables(unsigned long k_= start, unsigned long k_ if (!new) return -ENOMEM; kasan_populate_pte(new, PAGE_KERNEL); - pmd_populate_kernel(&init_mm, pmd, new); + pmd_populate_kernel(&init_mm, pmd, new, k_cur); } return 0; } @@ -187,6 +187,6 @@ void __init kasan_early_init(void) =20 do { next =3D pgd_addr_end(addr, end); - pmd_populate_kernel(&init_mm, pmd, kasan_early_shadow_pte); + pmd_populate_kernel(&init_mm, pmd, kasan_early_shadow_pte, addr); } while (pmd++, addr =3D next, addr !=3D end); } diff --git a/arch/powerpc/mm/kasan/init_book3e_64.c b/arch/powerpc/mm/kasan= /init_book3e_64.c index 11519e88dc6b..05ccdb88ff51 100644 --- a/arch/powerpc/mm/kasan/init_book3e_64.c +++ b/arch/powerpc/mm/kasan/init_book3e_64.c @@ -54,7 +54,7 @@ static int __init kasan_map_kernel_page(unsigned long ea,= unsigned long pa, pgpr if (kasan_pte_table(*pmdp)) { ptep =3D memblock_alloc(PTE_TABLE_SIZE, PTE_TABLE_SIZE); memcpy(ptep, kasan_early_shadow_pte, PTE_TABLE_SIZE); - pmd_populate_kernel(&init_mm, pmdp, ptep); + pmd_populate_kernel(&init_mm, pmdp, ptep, ea); } ptep =3D pte_offset_kernel(pmdp, ea); =20 @@ -93,9 +93,12 @@ void __init kasan_early_init(void) __set_pte_at(&init_mm, (unsigned long)kasan_early_shadow_page, &kasan_early_shadow_pte[i], zero_pte, 0); =20 - for (i =3D 0; i < PTRS_PER_PMD; i++) + addr =3D KASAN_SHADOW_START + for (i =3D 0; i < PTRS_PER_PMD; i++) { pmd_populate_kernel(&init_mm, &kasan_early_shadow_pmd[i], - kasan_early_shadow_pte); + kasan_early_shadow_pte, addr); + addr +=3D PMD_SIZE; + } =20 for (i =3D 0; i < PTRS_PER_PUD; i++) pud_populate(&init_mm, &kasan_early_shadow_pud[i], diff --git a/arch/powerpc/mm/kasan/init_book3s_64.c b/arch/powerpc/mm/kasan= /init_book3s_64.c index 9300d641cf9a..79569734dc29 100644 --- a/arch/powerpc/mm/kasan/init_book3s_64.c +++ b/arch/powerpc/mm/kasan/init_book3s_64.c @@ -55,6 +55,7 @@ void __init kasan_init(void) phys_addr_t start, end; u64 i; pte_t zero_pte =3D pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KER= NEL); + void *vaddr_start =3D __va(start); =20 if (!early_radix_enabled()) { pr_warn("KASAN not enabled as it requires radix!"); @@ -68,9 +69,11 @@ void __init kasan_init(void) __set_pte_at(&init_mm, (unsigned long)kasan_early_shadow_page, &kasan_early_shadow_pte[i], zero_pte, 0); =20 - for (i =3D 0; i < PTRS_PER_PMD; i++) + for (i =3D 0; i < PTRS_PER_PMD; i++) { pmd_populate_kernel(&init_mm, &kasan_early_shadow_pmd[i], - kasan_early_shadow_pte); + kasan_early_shadow_pte, + vaddr_start + i * PMD_SIZE); + } =20 for (i =3D 0; i < PTRS_PER_PUD; i++) pud_populate(&init_mm, &kasan_early_shadow_pud[i], diff --git a/arch/powerpc/mm/nohash/book3e_pgtable.c b/arch/powerpc/mm/noha= sh/book3e_pgtable.c index 1c5e4ecbebeb..930bdd7a3774 100644 --- a/arch/powerpc/mm/nohash/book3e_pgtable.c +++ b/arch/powerpc/mm/nohash/book3e_pgtable.c @@ -107,7 +107,7 @@ int __ref map_kernel_page(unsigned long ea, phys_addr_t= pa, pgprot_t prot) pmdp =3D pmd_offset(pudp, ea); if (!pmd_present(*pmdp)) { ptep =3D early_alloc_pgtable(PTE_TABLE_SIZE); - pmd_populate_kernel(&init_mm, pmdp, ptep); + pmd_populate_kernel(&init_mm, pmdp, ptep, ea); } ptep =3D pte_offset_kernel(pmdp, ea); } diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c index cfd622ebf774..e6fbaf3e9072 100644 --- a/arch/powerpc/mm/pgtable_32.c +++ b/arch/powerpc/mm/pgtable_32.c @@ -43,7 +43,7 @@ notrace void __init early_ioremap_init(void) =20 for (; (s32)(FIXADDR_TOP - addr) > 0; addr +=3D PGDIR_SIZE, ptep +=3D PTRS_PER_PTE, pmdp++) - pmd_populate_kernel(&init_mm, pmdp, ptep); + pmd_populate_kernel(&init_mm, pmdp, ptep, addr); =20 early_ioremap_setup(); } @@ -64,7 +64,7 @@ pte_t __init *early_pte_alloc_kernel(pmd_t *pmdp, unsigne= d long va) if (pmd_none(*pmdp)) { pte_t *ptep =3D early_alloc_pgtable(PTE_FRAG_SIZE); =20 - pmd_populate_kernel(&init_mm, pmdp, ptep); + pmd_populate_kernel(&init_mm, pmdp, ptep, va); } return pte_offset_kernel(pmdp, va); } diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgal= loc.h index deaf971253a2..d619daeded7f 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -16,7 +16,7 @@ #include =20 static inline void pmd_populate_kernel(struct mm_struct *mm, - pmd_t *pmd, pte_t *pte) + pmd_t *pmd, pte_t *pte, unsigned long vaddr) { unsigned long pfn =3D virt_to_pfn(pte); =20 diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 671b686c0158..085123ad4fa8 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -176,7 +176,7 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t = *src_pmdp, unsigned long if (!dst_ptep) return -ENOMEM; =20 - pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep, 0); } =20 dst_ptep =3D pte_offset_kernel(dst_pmdp, start); diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgallo= c.h index 7b84ef6dc4b6..4143b3f9d610 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -131,7 +131,7 @@ static inline void pmd_populate(struct mm_struct *mm, set_pmd(pmd, __pmd(_SEGMENT_ENTRY | __pa(pte))); } =20 -#define pmd_populate_kernel(mm, pmd, pte) pmd_populate(mm, pmd, pte) +#define pmd_populate_kernel(mm, pmd, pte, vaddr) pmd_populate(mm, pmd, pte) =20 /* * page table entry allocation/free routines. diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index 5d8577ab1591..04b29eb9712b 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -21,7 +21,7 @@ extern void pmd_free(struct mm_struct *mm, pmd_t *pmd); #endif =20 static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, - pte_t *pte) + pte_t *pte, unsigned long vaddr) { set_pmd(pmd, __pmd((unsigned long)pte)); } diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c index bf1b54055316..c862572dbec8 100644 --- a/arch/sh/mm/init.c +++ b/arch/sh/mm/init.c @@ -157,7 +157,7 @@ static pte_t * __init one_page_table_init(pmd_t *pmd) if (!pte) panic("%s: Failed to allocate %lu bytes align=3D0x%lx\n", __func__, PAGE_SIZE, PAGE_SIZE); - pmd_populate_kernel(&init_mm, pmd, pte); + pmd_populate_kernel(&init_mm, pmd, pte, 0); BUG_ON(pte !=3D pte_offset_kernel(pmd, 0)); } =20 diff --git a/arch/sparc/include/asm/pgalloc_32.h b/arch/sparc/include/asm/p= galloc_32.h index 4f73e87b22a3..558afcbd9016 100644 --- a/arch/sparc/include/asm/pgalloc_32.h +++ b/arch/sparc/include/asm/pgalloc_32.h @@ -53,7 +53,8 @@ static inline void free_pmd_fast(pmd_t * pmd) #define pmd_populate(mm, pmd, pte) pmd_set(pmd, pte) =20 void pmd_set(pmd_t *pmdp, pte_t *ptep); -#define pmd_populate_kernel pmd_populate +#define pmd_populate_kernel(mm, pmd, pte, vaddr) \ + pmd_populate(mm, pmd, pte) =20 pgtable_t pte_alloc_one(struct mm_struct *mm); =20 diff --git a/arch/sparc/include/asm/pgalloc_64.h b/arch/sparc/include/asm/p= galloc_64.h index caa7632be4c2..185ad9637442 100644 --- a/arch/sparc/include/asm/pgalloc_64.h +++ b/arch/sparc/include/asm/pgalloc_64.h @@ -69,8 +69,8 @@ void pte_free(struct mm_struct *mm, pgtable_t ptepage); #define pte_free_defer pte_free_defer void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable); =20 -#define pmd_populate_kernel(MM, PMD, PTE) pmd_set(MM, PMD, PTE) -#define pmd_populate(MM, PMD, PTE) pmd_set(MM, PMD, PTE) +#define pmd_populate_kernel(MM, PMD, PTE, VADDR) pmd_set(MM, PMD, PTE) +#define pmd_populate(MM, PMD, PTE) pmd_set(MM, PMD, PTE) =20 void pgtable_free(void *table, bool is_page); =20 diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index 1ca9054d9b97..32b3c89f869d 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -5,7 +5,7 @@ * Copyright (C) 1996-1999 David S. Miller (davem@caip.rutgers.edu) * Copyright (C) 1997-1999 Jakub Jelinek (jj@sunsite.mff.cuni.cz) */ -=20 + #include #include #include @@ -1843,7 +1843,7 @@ static unsigned long __ref kernel_map_range(unsigned = long pstart, if (!new) goto err_alloc; alloc_bytes +=3D PAGE_SIZE; - pmd_populate_kernel(&init_mm, pmd, new); + pmd_populate_kernel(&init_mm, pmd, new, vstart); } =20 pte =3D pte_offset_kernel(pmd, vstart); @@ -2404,11 +2404,11 @@ void __init paging_init(void) * work. */ init_mm.pgd +=3D ((shift) / (sizeof(pgd_t))); -=09 + memset(swapper_pg_dir, 0, sizeof(swapper_pg_dir)); =20 inherit_prom_mappings(); -=09 + /* Ok, we can use our TLB miss and window trap handlers safely. */ setup_tba(); =20 diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index de5e31c64793..300431ff61bb 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -1,5 +1,5 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/*=20 +/* * Copyright (C) 2000, 2001, 2002 Jeff Dike (jdike@karaya.com) * Copyright 2003 PathScale, Inc. * Derived from include/asm-i386/pgalloc.h and include/asm-i386/pgtable.h @@ -12,7 +12,7 @@ =20 #include =20 -#define pmd_populate_kernel(mm, pmd, pte) \ +#define pmd_populate_kernel(mm, pmd, pte, vaddr) \ set_pmd(pmd, __pmd(_PAGE_TABLE + (unsigned long) __pa(pte))) =20 #define pmd_populate(mm, pmd, pte) \ diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59beb..3bc5e0cc7b38 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -62,7 +62,8 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb,= struct page *pte, } =20 static inline void pmd_populate_kernel(struct mm_struct *mm, - pmd_t *pmd, pte_t *pte) + pmd_t *pmd, pte_t *pte, + unsigned long vaddr) { paravirt_alloc_pte(mm, __pa(pte) >> PAGE_SHIFT); set_pmd(pmd, __pmd(__pa(pte) | _PAGE_TABLE)); diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 7e177856ee4f..ee4a73842466 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -73,7 +73,15 @@ static inline void fname##_init(struct mm_struct *mm, \ DEFINE_POPULATE(p4d_populate, p4d, pud, init) DEFINE_POPULATE(pgd_populate, pgd, p4d, init) DEFINE_POPULATE(pud_populate, pud, pmd, init) -DEFINE_POPULATE(pmd_populate_kernel, pmd, pte, init) + +static inline void pmd_populate_kernel_init(struct mm_struct *mm, + pmd_t *arg1, pte_t *arg2, unsigned long arg3, bool init) +{ + if (init) + pmd_populate_kernel_safe(mm, arg1, arg2); + else + pmd_populate_kernel(mm, arg1, arg2, arg3); +} =20 #define DEFINE_ENTRY(type1, type2, init) \ static inline void set_##type1##_init(type1##_t *arg1, \ @@ -286,7 +294,7 @@ static pte_t *fill_pte(pmd_t *pmd, unsigned long vaddr) { if (pmd_none(*pmd)) { pte_t *pte =3D (pte_t *) spp_getpage(); - pmd_populate_kernel(&init_mm, pmd, pte); + pmd_populate_kernel(&init_mm, pmd, pte, vaddr); if (pte !=3D pte_offset_kernel(pmd, 0)) printk(KERN_ERR "PAGETABLE BUG #03!\n"); } @@ -575,7 +583,7 @@ phys_pmd_init(pmd_t *pmd_page, unsigned long paddr, uns= igned long paddr_end, paddr_last =3D phys_pte_init(pte, paddr, paddr_end, new_prot, init); =20 spin_lock(&init_mm.page_table_lock); - pmd_populate_kernel_init(&init_mm, pmd, pte, init); + pmd_populate_kernel_init(&init_mm, pmd, pte, init, __va(paddr)); spin_unlock(&init_mm.page_table_lock); } update_page_count(PG_LEVEL_2M, pages); diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index aa7d279321ea..8844047fdaad 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -888,7 +888,7 @@ void __init early_ioremap_init(void) =20 pmd =3D early_ioremap_pmd(fix_to_virt(FIX_BTMAP_BEGIN)); memset(bm_pte, 0, sizeof(bm_pte)); - pmd_populate_kernel(&init_mm, pmd, bm_pte); + pmd_populate_kernel(&init_mm, pmd, bm_pte, fix_to_virt(FIX_BTMAP_BEGIN)); =20 /* * The boot-ioremap range spans multiple pmds, for which diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 9dddf19a5571..95ae9e12fe41 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -53,7 +53,7 @@ static void __init kasan_populate_pmd(pmd_t *pmd, unsigne= d long addr, } =20 p =3D early_alloc(PAGE_SIZE, nid, true); - pmd_populate_kernel(&init_mm, pmd, p); + pmd_populate_kernel(&init_mm, pmd, p, addr); } =20 pte =3D pte_offset_kernel(pmd, addr); diff --git a/arch/xtensa/include/asm/pgalloc.h b/arch/xtensa/include/asm/pg= alloc.h index 7fc0f9126dd3..5359e4091b9a 100644 --- a/arch/xtensa/include/asm/pgalloc.h +++ b/arch/xtensa/include/asm/pgalloc.h @@ -21,7 +21,7 @@ * inside the pgd, so has no extra memory associated with it. */ =20 -#define pmd_populate_kernel(mm, pmdp, ptep) \ +#define pmd_populate_kernel(mm, pmdp, ptep, vaddr) \ (pmd_val(*(pmdp)) =3D ((unsigned long)ptep)) #define pmd_populate(mm, pmdp, page) \ (pmd_val(*(pmdp)) =3D ((unsigned long)page_to_virt(page))) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b0ee64225de..7162667c0e37 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2802,7 +2802,7 @@ static inline void mm_dec_nr_ptes(struct mm_struct *m= m) {} #endif =20 int __pte_alloc(struct mm_struct *mm, pmd_t *pmd); -int __pte_alloc_kernel(pmd_t *pmd); +int __pte_alloc_kernel(pmd_t *pmd, unsigned long vaddr); =20 #if defined(CONFIG_MMU) =20 @@ -2997,7 +2997,7 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pm= d_t *pmd, NULL : pte_offset_map_lock(mm, pmd, address, ptlp)) =20 #define pte_alloc_kernel(pmd, address) \ - ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \ + ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd, address))? \ NULL: pte_offset_kernel(pmd, address)) =20 #if USE_SPLIT_PMD_PTLOCKS diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index da177e49d956..cfbe3695fffb 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -58,7 +58,7 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *hea= d, unsigned long start, if (!pgtable) return -ENOMEM; =20 - pmd_populate_kernel(&init_mm, &__pmd, pgtable); + pmd_populate_kernel(&init_mm, &__pmd, pgtable, start); =20 for (i =3D 0; i < PTRS_PER_PTE; i++, addr +=3D PAGE_SIZE) { pte_t entry, *pte; @@ -81,7 +81,7 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *hea= d, unsigned long start, =20 /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); - pmd_populate_kernel(&init_mm, pmd, pgtable); + pmd_populate_kernel(&init_mm, pmd, pgtable, start); if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, start + PMD_SIZE); } else { diff --git a/mm/kasan/init.c b/mm/kasan/init.c index 89895f38f722..813f8e8a801c 100644 --- a/mm/kasan/init.c +++ b/mm/kasan/init.c @@ -117,7 +117,8 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned= long addr, =20 if (IS_ALIGNED(addr, PMD_SIZE) && end - addr >=3D PMD_SIZE) { pmd_populate_kernel(&init_mm, pmd, - lm_alias(kasan_early_shadow_pte)); + lm_alias(kasan_early_shadow_pte), + addr); continue; } =20 @@ -131,7 +132,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned= long addr, if (!p) return -ENOMEM; =20 - pmd_populate_kernel(&init_mm, pmd, p); + pmd_populate_kernel(&init_mm, pmd, p, addr); } zero_pte_populate(pmd, addr, next); } while (pmd++, addr =3D next, addr !=3D end); @@ -158,7 +159,8 @@ static int __ref zero_pud_populate(p4d_t *p4d, unsigned= long addr, lm_alias(kasan_early_shadow_pmd)); pmd =3D pmd_offset(pud, addr); pmd_populate_kernel(&init_mm, pmd, - lm_alias(kasan_early_shadow_pte)); + lm_alias(kasan_early_shadow_pte), + addr); continue; } =20 @@ -204,7 +206,8 @@ static int __ref zero_p4d_populate(pgd_t *pgd, unsigned= long addr, lm_alias(kasan_early_shadow_pmd)); pmd =3D pmd_offset(pud, addr); pmd_populate_kernel(&init_mm, pmd, - lm_alias(kasan_early_shadow_pte)); + lm_alias(kasan_early_shadow_pte), + addr); continue; } =20 @@ -267,7 +270,8 @@ int __ref kasan_populate_early_shadow(const void *shado= w_start, lm_alias(kasan_early_shadow_pmd)); pmd =3D pmd_offset(pud, addr); pmd_populate_kernel(&init_mm, pmd, - lm_alias(kasan_early_shadow_pte)); + lm_alias(kasan_early_shadow_pte), + addr); continue; } =20 diff --git a/mm/memory.c b/mm/memory.c index d2155ced45f8..67807ade9a0e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -447,7 +447,7 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd) return 0; } =20 -int __pte_alloc_kernel(pmd_t *pmd) +int __pte_alloc_kernel(pmd_t *pmd, unsigned long vaddr) { pte_t *new =3D pte_alloc_one_kernel(&init_mm); if (!new) @@ -456,7 +456,7 @@ int __pte_alloc_kernel(pmd_t *pmd) spin_lock(&init_mm.page_table_lock); if (likely(pmd_none(*pmd))) { /* Has another populated it ? */ smp_wmb(); /* See comment in pmd_install() */ - pmd_populate_kernel(&init_mm, pmd, new); + pmd_populate_kernel(&init_mm, pmd, new, vaddr); new =3D NULL; } spin_unlock(&init_mm.page_table_lock); diff --git a/mm/percpu.c b/mm/percpu.c index 4e11fc1e6def..fc83cf64baf6 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3238,7 +3238,7 @@ void __init __weak pcpu_populate_pte(unsigned long ad= dr) new =3D memblock_alloc(PTE_TABLE_SIZE, PTE_TABLE_SIZE); if (!new) goto err_alloc; - pmd_populate_kernel(&init_mm, pmd, new); + pmd_populate_kernel(&init_mm, pmd, new, addr); } =20 return; diff --git a/mm/pgalloc-track.h b/mm/pgalloc-track.h index e9e879de8649..ac983705a054 100644 --- a/mm/pgalloc-track.h +++ b/mm/pgalloc-track.h @@ -45,7 +45,8 @@ static inline pmd_t *pmd_alloc_track(struct mm_struct *mm= , pud_t *pud, =20 #define pte_alloc_kernel_track(pmd, address, mask) \ ((unlikely(pmd_none(*(pmd))) && \ - (__pte_alloc_kernel(pmd) || ({*(mask)|=3DPGTBL_PMD_MODIFIED;0;})))?\ + (__pte_alloc_kernel(pmd, address) || \ + ({*(mask)|=3DPGTBL_PMD_MODIFIED;0;})))?\ NULL: pte_offset_kernel(pmd, address)) =20 #endif /* _LINUX_PGALLOC_TRACK_H */ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a2cbe44c48e1..6085c8339b65 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -191,7 +191,7 @@ pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsi= gned long addr, int node) void *p =3D vmemmap_alloc_block_zero(PAGE_SIZE, node); if (!p) return NULL; - pmd_populate_kernel(&init_mm, pmd, p); + pmd_populate_kernel(&init_mm, pmd, p, addr); } return pmd; } --=20 2.39.2 From nobody Mon Feb 9 10:34:32 2026 Received: from mx0b-00823401.pphosted.com (mx0b-00823401.pphosted.com [148.163.152.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E05A0137C23 for ; Tue, 16 Apr 2024 19:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.152.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713295190; cv=none; b=KiJ1q9wmAoDzyGXx3mVaRvWQ8ziHon0cXPoKRogz+jzds8yk1S+3VVMABsu9M6zYP6vUkSjqdvIQVHRwPUDt9ew8OGA8u0lLlu6BPy643qLbKmuJmC/z/S1ZcLm9VQNeI7jBReIPyiEWuF5wUcM+toSef9cOX4+vpxPmqeD8FQc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713295190; c=relaxed/simple; bh=1/QhQq/SNBvhM5Qukg8qId++gk6cla4fzJPkLM7SWYI=; h=Message-Id:In-Reply-To:References:To:Cc:From:Subject:Date; b=opFDhlfEk5kYNkKncZcZmS6PxmImN9GsJvFV8kMH9kxTcXWaKdzst2YHpX7iMrhM0NMaII1KjgfxSbhbrdzsqGNAYgVn36dZHF2nRlWvFAEG/sq+UBjGxGCAln0/NNaZHvkJhyhIrTfNF+FRwRk66agiOjAKyReOru81saJ2xrM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com; spf=pass smtp.mailfrom=motorola.com; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b=RP/rT0lu; arc=none smtp.client-ip=148.163.152.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=motorola.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b="RP/rT0lu" Received: from pps.filterd (m0355089.ppops.net [127.0.0.1]) by mx0b-00823401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 43GGKltB014625; Tue, 16 Apr 2024 19:19:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=motorola.com; h= message-id:in-reply-to:references:to:cc:from:subject:date; s= DKIM202306; bh=bxlhcCLLu2nVwVVshKc9HTXBGPT+xoKrYdwmFjIADko=; b=R P/rT0luOjR5Tc+kJKMZ7XmGq7XLTVOYOty15m6179Fn2yM62v4KuHW3xu80sOZH6 0uxTedg/kHgnCirjUr67K3jDCBayRawwd5ZrFIKtcgcC1y2Y+eWgN+OSEHJ63HHU OQ0wix/aQOy/wBP/4X9C/9cLB5QbdgcgcbOg5EUO79WIkINVPFg+Fk3Hmi8DS33o WD2ntpRpoZGV0f4tjklC8QLJT6E5eFwHq8qjSjSGtSY1cvRem+UNKbFOYFR1zv1r JChKCLYb1VT7WyBbPCAkhVpy6enF21Sas6lIDBW6FhcjcbCLmXTRNhNDYoObSz+r S5xQvNsAkQK0dLy5oDIjw== Received: from va32lpfpp02.lenovo.com ([104.232.228.22]) by mx0b-00823401.pphosted.com (PPS) with ESMTPS id 3xhctp42v1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 16 Apr 2024 19:19:22 +0000 (GMT) Received: from va32lmmrp02.lenovo.com (va32lmmrp02.mot.com [10.62.176.191]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by va32lpfpp02.lenovo.com (Postfix) with ESMTPS id 4VJv3t0jpxz53xyZ; Tue, 16 Apr 2024 19:19:22 +0000 (UTC) Received: from ilclbld243.mot.com (ilclbld243.mot.com [100.64.22.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: mbland) by va32lmmrp02.lenovo.com (Postfix) with ESMTPSA id 4VJv3t0W4Hz2SlV8; Tue, 16 Apr 2024 19:19:22 +0000 (UTC) Message-Id: <20240416122254.868007168-5-mbland@motorola.com> In-Reply-To: <20240416122254.868007168-1-mbland@motorola.com> References: <20240416122254.868007168-1-mbland@motorola.com> To: linux-arm-kernel@lists.infradead.org Cc: Maxwell Bland , Catalin Marinas , Will Deacon , Ard Biesheuvel , Maxwell Bland , Mark Rutland , Greg Kroah-Hartman , Christoph Hellwig , Christophe Leroy , David Hildenbrand , Conor Dooley , linux-kernel@vger.kernel.org From: Maxwell Bland Subject: [PATCH 4/5 RESEND] arm64: dynamic enforcement of PXNTable Date: Tue, 16 Apr 2024 14:18:18 -0500 X-Proofpoint-GUID: DROmJMmX7G07KTG0qxdXMD6DZ-9Y4P3A X-Proofpoint-ORIG-GUID: DROmJMmX7G07KTG0qxdXMD6DZ-9Y4P3A X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-16_17,2024-04-16_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxscore=0 impostorscore=0 suspectscore=0 phishscore=0 adultscore=0 mlxlogscore=738 clxscore=1015 malwarescore=0 lowpriorityscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2404010003 definitions=main-2404160122 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" PXNTable is enforced during the init process to ensure that regions of user memory and kernel data cannot be executed from, preventing attacks which write to writable kernel pages and then modify the kernel's page tables to make this code executable. This patch ensures this protection is also preserved for dynamically allocated pages/pagetables, making it so that all PMDs populated outside of the module code region are PXNTable by default. Signed-off-by: Maxwell Bland --- arch/arm64/include/asm/pgalloc.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgal= loc.h index 5785272144e8..2376b4e7915c 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -12,6 +12,7 @@ #include #include #include +#include =20 #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE @@ -119,6 +120,12 @@ static inline void __pmd_populate(pmd_t *pmdp, phys_ad= dr_t ptep, set_pmd(pmdp, __pmd(__phys_to_pmd_val(ptep) | prot)); } =20 +static inline bool vaddr_is_data(unsigned long vaddr) +{ + return ((vaddr + PMD_SIZE < MODULES_ASLR_START || vaddr >=3D MODULES_ASLR= _END) && + (vaddr + PMD_SIZE < (unsigned long) _text || vaddr >=3D (unsigned long) = _etext)); +} + /* * Populate the pmdp entry with a pointer to the pte. This pmd is part * of the mm address space. @@ -127,8 +134,11 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep, unsigned long vaddr) { + pmdval_t pmd =3D PMD_TYPE_TABLE | PMD_TABLE_UXN; VM_BUG_ON(mm && mm !=3D &init_mm); - __pmd_populate(pmdp, __pa(ptep), PMD_TYPE_TABLE | PMD_TABLE_UXN); + if (vaddr_is_data(vaddr)) + pmd |=3D PMD_TABLE_PXN; + __pmd_populate(pmdp, __pa(ptep), pmd); } =20 static inline void --=20 2.39.2 From nobody Mon Feb 9 10:34:32 2026 Received: from mx0b-00823401.pphosted.com (mx0b-00823401.pphosted.com [148.163.152.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0491B133284; Tue, 16 Apr 2024 17:25:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.152.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713288350; cv=none; b=HdcZsUsORrmCAo8kfifV2kmSenPXzc46j2dJG5dvXMiBHb5FHZllL2PUkLqx5u9IE1R2bnQfmJzocM0kVe8DlAqEyUutz4PW7xrIJ7Hw1wNi3f7YG8DBacBrTgtGmCyjs5GXhMEo+9vzy9ouC6nhlYibsSnu/ehsTw7fHd5AOso= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713288350; c=relaxed/simple; bh=8qCKzY35EGzYV4nWpAHAjd7zJfdPfc7UJRsZu59DAXQ=; h=Message-Id:In-Reply-To:References:To:Cc:From:Date:Subject; b=oPoarQ/kEipGlukJcjVaJ0Rb3rK0cMLTRNK/DmrAc+rXWi4Me0TAUcnysQUTIbaw9xP0BxLsXjyKWCzdaQSexxd/G0Qtns1bDKj2vX0HrPxCcFFJU+kiit0jKdBoF5sBMjgP4CSEIzLLLTK6gKjj4dk89AZPU+9TFX1T5t1fDAY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com; spf=pass smtp.mailfrom=motorola.com; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b=fVFZT85i; arc=none smtp.client-ip=148.163.152.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=motorola.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=motorola.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=motorola.com header.i=@motorola.com header.b="fVFZT85i" Received: from pps.filterd (m0355091.ppops.net [127.0.0.1]) by mx0b-00823401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 43GFQBHE026682; Tue, 16 Apr 2024 17:24:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=motorola.com; h= message-id:in-reply-to:references:to:cc:from:date:subject; s= DKIM202306; bh=Oc2SUwX6uzMkXCJf4ZSU5CLVXrP2N2CrXvm6EA3BRc0=; b=f VFZT85i9fG9UZIMquW7lOcKAq20DOeCPjOdPiSoX40Uy/gdhpas7agKPawUQhT7x jlu6AEhZt7ljLk2n20UpKTnolmUli9RA4S94pfqgo82LJmyfQGSWX+2ckqN3E3a6 jp+/jciyo6S9UM3hNfRiVNfOIKTGmn3Gd7Kqf7Fg+ODOygCLrVJKEFbUu3tVWAti fadgXso/3HtOEdqpW7Sq+tQ8RIqcbtaSSGSjhS7q9YsinUy0LnphQn2+GK2iDP1m aaqJyHM7uz4vWoEonBp43YZWPoTF4kyTlWY8hHQneojPAs3B2gfooI2z9rSq2qK+ 8Td0+HxT5kbdpLFNyLd1A== Received: from va32lpfpp04.lenovo.com ([104.232.228.24]) by mx0b-00823401.pphosted.com (PPS) with ESMTPS id 3xhjbek979-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 16 Apr 2024 17:24:57 +0000 (GMT) Received: from va32lmmrp01.lenovo.com (va32lmmrp01.mot.com [10.62.177.113]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by va32lpfpp04.lenovo.com (Postfix) with ESMTPS id 4VJrWs1Wv3zj9hH; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Received: from ilclbld243.mot.com (ilclbld243.mot.com [100.64.22.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: mbland) by va32lmmrp01.lenovo.com (Postfix) with ESMTPSA id 4VJrWs17Mmz2VZS6; Tue, 16 Apr 2024 17:24:57 +0000 (UTC) Message-Id: <20240416122254.868007168-6-mbland@motorola.com> In-Reply-To: <20240416122254.868007168-1-mbland@motorola.com> References: <20240416122254.868007168-1-mbland@motorola.com> To: linux-mm@kvack.org Cc: Maxwell Bland , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Andrew Morton , Ard Biesheuvel , Mark Rutland , Maxwell Bland , Alexandre Ghiti , Yu Chien Peter Lin , Song Shuai , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org From: Maxwell Bland Date: Mon, 15 Apr 2024 14:51:32 -0500 Subject: [PATCH 5/5] ptdump: add state parameter for non-leaf callback X-Proofpoint-ORIG-GUID: Obuaco9Ts8gQobghqAGopelkIlnGG11N X-Proofpoint-GUID: Obuaco9Ts8gQobghqAGopelkIlnGG11N X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-16_14,2024-04-16_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 spamscore=0 impostorscore=0 suspectscore=0 bulkscore=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 adultscore=0 mlxlogscore=960 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2404010003 definitions=main-2404160108 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" ptdump can now note non-leaf descriptor entries, a useful addition for debugging table descriptor permissions when working on related code Signed-off-by: Maxwell Bland --- arch/arm64/mm/ptdump.c | 6 ++++-- arch/powerpc/mm/ptdump/ptdump.c | 2 ++ arch/riscv/mm/ptdump.c | 6 ++++-- arch/s390/mm/dump_pagetables.c | 6 ++++-- arch/x86/mm/dump_pagetables.c | 3 ++- include/linux/ptdump.h | 1 + mm/ptdump.c | 13 +++++++++++++ 7 files changed, 30 insertions(+), 7 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 796231a4fd63..1a6f4a3513e5 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -299,7 +299,8 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info= *info) .range =3D (struct ptdump_range[]){ {info->base_addr, end}, {0, 0} - } + }, + .note_non_leaf =3D false } }; =20 @@ -335,7 +336,8 @@ bool ptdump_check_wx(void) .range =3D (struct ptdump_range[]) { {_PAGE_OFFSET(vabits_actual), ~0UL}, {0, 0} - } + }, + .note_non_leaf =3D false } }; =20 diff --git a/arch/powerpc/mm/ptdump/ptdump.c b/arch/powerpc/mm/ptdump/ptdum= p.c index 9dc239967b77..89e673f5fd3d 100644 --- a/arch/powerpc/mm/ptdump/ptdump.c +++ b/arch/powerpc/mm/ptdump/ptdump.c @@ -307,6 +307,7 @@ static int ptdump_show(struct seq_file *m, void *v) .ptdump =3D { .note_page =3D note_page, .range =3D ptdump_range, + .note_non_leaf =3D false } }; =20 @@ -340,6 +341,7 @@ bool ptdump_check_wx(void) .ptdump =3D { .note_page =3D note_page, .range =3D ptdump_range, + .note_non_leaf =3D false } }; =20 diff --git a/arch/riscv/mm/ptdump.c b/arch/riscv/mm/ptdump.c index 1289cc6d3700..b355633afcaf 100644 --- a/arch/riscv/mm/ptdump.c +++ b/arch/riscv/mm/ptdump.c @@ -328,7 +328,8 @@ static void ptdump_walk(struct seq_file *s, struct ptd_= mm_info *pinfo) .range =3D (struct ptdump_range[]) { {pinfo->base_addr, pinfo->end}, {0, 0} - } + }, + .note_non_leaf =3D false } }; =20 @@ -350,7 +351,8 @@ bool ptdump_check_wx(void) .range =3D (struct ptdump_range[]) { {KERN_VIRT_START, ULONG_MAX}, {0, 0} - } + }, + .note_non_leaf =3D false } }; =20 diff --git a/arch/s390/mm/dump_pagetables.c b/arch/s390/mm/dump_pagetables.c index ffd07ed7b4af..6468cfd53e2a 100644 --- a/arch/s390/mm/dump_pagetables.c +++ b/arch/s390/mm/dump_pagetables.c @@ -200,7 +200,8 @@ bool ptdump_check_wx(void) .range =3D (struct ptdump_range[]) { {.start =3D 0, .end =3D max_addr}, {.start =3D 0, .end =3D 0}, - } + }, + .note_non_leaf =3D false }, .seq =3D NULL, .level =3D -1, @@ -239,7 +240,8 @@ static int ptdump_show(struct seq_file *m, void *v) .range =3D (struct ptdump_range[]) { {.start =3D 0, .end =3D max_addr}, {.start =3D 0, .end =3D 0}, - } + }, + .note_non_leaf =3D false }, .seq =3D m, .level =3D -1, diff --git a/arch/x86/mm/dump_pagetables.c b/arch/x86/mm/dump_pagetables.c index 89079ea73e65..43f00dfb955f 100644 --- a/arch/x86/mm/dump_pagetables.c +++ b/arch/x86/mm/dump_pagetables.c @@ -380,7 +380,8 @@ bool ptdump_walk_pgd_level_core(struct seq_file *m, .ptdump =3D { .note_page =3D note_page, .effective_prot =3D effective_prot, - .range =3D ptdump_ranges + .range =3D ptdump_ranges, + .note_non_leaf =3D false }, .level =3D -1, .to_dmesg =3D dmesg, diff --git a/include/linux/ptdump.h b/include/linux/ptdump.h index 8dbd51ea8626..b3e793a5c77f 100644 --- a/include/linux/ptdump.h +++ b/include/linux/ptdump.h @@ -16,6 +16,7 @@ struct ptdump_state { int level, u64 val); void (*effective_prot)(struct ptdump_state *st, int level, u64 val); const struct ptdump_range *range; + bool note_non_leaf; }; =20 bool ptdump_walk_pgd_level_core(struct seq_file *m, diff --git a/mm/ptdump.c b/mm/ptdump.c index 106e1d66e9f9..97da7a765b22 100644 --- a/mm/ptdump.c +++ b/mm/ptdump.c @@ -41,6 +41,9 @@ static int ptdump_pgd_entry(pgd_t *pgd, unsigned long add= r, if (st->effective_prot) st->effective_prot(st, 0, pgd_val(val)); =20 + if (st->note_non_leaf && !pgd_leaf(val)) + st->note_page(st, addr, 0, pgd_val(val)); + if (pgd_leaf(val)) { st->note_page(st, addr, 0, pgd_val(val)); walk->action =3D ACTION_CONTINUE; @@ -64,6 +67,9 @@ static int ptdump_p4d_entry(p4d_t *p4d, unsigned long add= r, if (st->effective_prot) st->effective_prot(st, 1, p4d_val(val)); =20 + if (st->note_non_leaf && !p4d_leaf(val)) + st->note_page(st, addr, 1, p4d_val(val)); + if (p4d_leaf(val)) { st->note_page(st, addr, 1, p4d_val(val)); walk->action =3D ACTION_CONTINUE; @@ -87,6 +93,9 @@ static int ptdump_pud_entry(pud_t *pud, unsigned long add= r, if (st->effective_prot) st->effective_prot(st, 2, pud_val(val)); =20 + if (st->note_non_leaf && !pud_leaf(val)) + st->note_page(st, addr, 2, pud_val(val)); + if (pud_leaf(val)) { st->note_page(st, addr, 2, pud_val(val)); walk->action =3D ACTION_CONTINUE; @@ -108,6 +117,10 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long = addr, =20 if (st->effective_prot) st->effective_prot(st, 3, pmd_val(val)); + + if (st->note_non_leaf && !pmd_leaf(val)) + st->note_page(st, addr, 3, pmd_val(val)); + if (pmd_leaf(val)) { st->note_page(st, addr, 3, pmd_val(val)); walk->action =3D ACTION_CONTINUE; --=20 2.39.2