From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB07D46D0A9 for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; cv=none; b=R+1mtAnIjPsrlus99kHWeqzUtfvZ8dW3/yBtsLXhh/u7sizlgmhc2A0ZfiriXfNhbuoj/6od3newfdB+BwNoC2/GNBQupxotfFz4eLqvu264PXVBSiP6tJMszplol2arTn8cZhQB1cckby3yEIetLc4iPHRVJFBGTTN9mLUpYbM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; c=relaxed/simple; bh=dSS0bMSoY0Q3KIDio1ufZOuk+k4Y5Q20zxzxymAbtic=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=lfAnO5XJrh3R9BD008upjXxL+ASpIlG310rQkojPyzFJaNPxvuOsJRgZJFoyNt9jG/EG4yfZDgU0PrKkqReuvbocwslkoLTyfRQVuNElr0IWp0VZPjuj3N2+0foIOHToiGstJanKWMxGDxZlYyuTp3XmTEGhoiBk6kgHr1yJopc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=X57abhX2; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="X57abhX2" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=mZpijvlSw2xNIsfL6IQ6jLABbu5LnLQd4TTudOzRHd8=; b=X57abhX2xJestC1P2bJzdl/sfZqWq7S2z/DnrQR2S6d5a9iNjQpzAs2nrZVVgINhqAiK9mlEe 8Un9AB4ywk8zAwPRdHmdZkkfEjyoRP/9lB50ajE6W2cmsfC1XR+YoRU9zHH+zlyOCfqwx3cge8Y 0suoxFbuxnwTqeM9jTi9eDY= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KV1gnVzJ4673; Fri, 28 Aug 2026 00:24:14 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id C345340572; Fri, 28 Aug 2026 00:24:35 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:35 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 01/18] mm: arm64 add Kconfig option for kernel replication Date: Fri, 28 Aug 2026 00:11:41 +0800 Message-ID: <20260827161158.3618409-2-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- mm/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..1302268b72b6 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1446,6 +1446,16 @@ config LOCK_MM_AND_FIND_VMA bool depends on !STACK_GROWSUP =20 +config KERNEL_REPLICATION + bool "Enable kernel text and ro-data replication across NUMA nodes" + default n + depends on ARM64 && MMU && NUMA && !MAXSMP + + help + Creates per-NUMA node replicas of kernel text and ro-data sections. + Page tables are replicated partially, according to replicated kernel me= mory range. + If unsure, say "n". + config IOMMU_MM_DATA bool =20 --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB377483BFA for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847889; cv=none; b=EhqzjM1iCJHX9JSfm85XZmBA5+kQtyK+/+3N8zl5ZhQ2WztWS9oM609guupZLlUt4uXSniyA8aplJQJM6d0KdAJjyQmu5Z3c+qmVHeikYXjehFpJUwyVJsvMRRXhI27x3n07eA1FDCbUwr8F5BTOZ6vEoYA6JgVwwAhpSjrluUA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847889; c=relaxed/simple; bh=pgoI14c19xZ43wko77ZwN8GGic+HffwtfhmOQJFlreQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kwbqeiuz1n1nhZcna8Ghbk1K4BTJGtRaNVHmMk/R4Y6Ti3Qk3AEx9CdV9tbJm19lRuQkaVpjKQ1h9DuJ76KX+1IZsjsIqeI9IVTeVkmzfIHkTMEiPY2fspxcnPJ2bKC+OiOYHhD6w4fA0y1Y5q3tq8eEk5yCZ4HSVcKAk26EtT8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=rrMP/6Ig; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="rrMP/6Ig" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=oMZ5AFZhqeXmEWrdBGYUKt+USNzuDSrHR09vQq45iZE=; b=rrMP/6IgeCyguD1XBUymEDs46U3PH2nQcOVjRM7NWTwXIBiKAc+v2wRktTvtj3dnhx+/A68o2 ujUyEiKFJ0FYG9oClz7vuPt/2Wbq9ooYp17fibHkCYasLVZkNzjTArv/0GTYFmcY9o+umZmAbaw okn+6XU8oRhyoEmAUSQddQg= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KC66nqzHnH5L; Fri, 28 Aug 2026 00:23:59 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id D785540570; Fri, 28 Aug 2026 00:24:35 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:35 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 02/18] arm64: align kernel text and rodata Date: Fri, 28 Aug 2026 00:11:42 +0800 Message-ID: <20260827161158.3618409-3-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" In order to comply with AArch64 page table entries type - table and block descriptor format, we need to use correct alignment for replicated memory areas. Physical blocks allocated for replicated areas must have the same offset for all replicas and same relative offset to the virtual address. For simplicity, we just use an explicit alignment for kernel sections. If 4K pages are used, 2MB alignment should be applied. Otherwise CONT_PTE_SIZE alignment is used. Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/kernel/vmlinux.lds.S | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.ld= s.S index af1d72020976..aaf07e926c54 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -192,6 +192,13 @@ SECTIONS _text =3D .; HEAD_TEXT } +#ifdef CONFIG_KERNEL_REPLICATION +#ifdef CONFIG_ARM64_4K_PAGES + . =3D ALIGN(PMD_SIZE); +#else + . =3D ALIGN(CONT_PTE_SIZE); +#endif +#endif .text : ALIGN(SEGMENT_ALIGN) { /* Real text segment */ _stext =3D .; /* Text and read-only data */ IRQENTRY_TEXT @@ -207,10 +214,25 @@ SECTIONS } =20 . =3D ALIGN(SEGMENT_ALIGN); +#ifdef CONFIG_KERNEL_REPLICATION +#ifdef CONFIG_ARM64_4K_PAGES + . =3D ALIGN(PMD_SIZE); +#else + . =3D ALIGN(CONT_PTE_SIZE); +#endif +#endif _etext =3D .; /* End of text section */ =20 /* everything from this point to __init_begin will be marked RO NX */ +#ifdef CONFIG_KERNEL_REPLICATION +#ifdef CONFIG_ARM64_4K_PAGES + RO_DATA(PMD_SIZE) +#else + RO_DATA(CONT_PTE_SIZE) +#endif +#else RO_DATA(PAGE_SIZE) +#endif =20 HYPERVISOR_RODATA_SECTIONS =20 --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB1EE47CA87 for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847889; cv=none; b=j+EGMRDhZEgp9LEywaPBF1cIPLxl7T5GNQ9DM/OoEchopuukAgbUN83NhHJ70oIfd9GoeTlCEe3LBcj5bP9TpjaX4Vws3YiGTDcoObZe7ucMoJN4jEj0tNeHX4QWfxlGaZ6fZ2eab1Kxz+ZuToUi5JHR6gfnx11U+KSmw+HWcQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847889; c=relaxed/simple; bh=wsoq5ycG3ToejydWj8Zt1+Cz6fm27uvK0qo+ezLvB6o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fituS+DY7ennWxUY5nOeNrYX/HLf6OUbpK3g48+OAijDd4PQUyTGeaYJx9tp5VcJHatzuSIsfSojhZCYq7n2a7Pspa3kQLuXx44BLeA+kLFStU9cDlkvGBLrMVR5B6/vRCqsc6grWMGvYbJoNyVAxoswKJuTXJh/F6MkJDlv3Jc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=SKY3L7vC; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="SKY3L7vC" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=UfxEk7T6JyEQqsUbsOIOBOw3o1DLsBcu07jRZivHztM=; b=SKY3L7vCGkgTSi/KQ22wZIdzfe/u4jTApdIfM0GcFZOaP2qSaY8c/zr5l1/36iDTjFDpDUA7O UrwwRuiQncZvbhNNPoZvfOw1Z+UzhAVIvBPmVXFX0z5iJfvI9onj3Yu6VZSipEiJJQo/FQPTKvD IEPRflEz2cCmIVDQejLXU5M= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KV2tftzJ46Dv; Fri, 28 Aug 2026 00:24:14 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id EBE4940572; Fri, 28 Aug 2026 00:24:35 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:35 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 03/18] mm: allow per-NUMA node local P4D/PUD/PMD/PTE allocation Date: Fri, 28 Aug 2026 00:11:43 +0800 Message-ID: <20260827161158.3618409-4-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- include/asm-generic/pgalloc.h | 90 +++++++++++++++++++++++++++++ include/asm-generic/pgtable-nop4d.h | 5 ++ include/asm-generic/pgtable-nopmd.h | 5 ++ include/asm-generic/pgtable-nopud.h | 5 ++ include/linux/mm.h | 87 +++++++++++++++++++++++++++- mm/memory.c | 68 ++++++++++++++++++++++ 6 files changed, 259 insertions(+), 1 deletion(-) diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 051aa1331051..79d10806c0fc 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -85,6 +85,26 @@ static inline pgtable_t __pte_alloc_one_noprof(struct mm= _struct *mm, gfp_t gfp) } #define __pte_alloc_one(...) alloc_hooks(__pte_alloc_one_noprof(__VA_ARGS_= _)) =20 +#ifdef CONFIG_KERNEL_REPLICATION +static inline pgtable_t __pte_alloc_one_node_noprof(unsigned int nid, + struct mm_struct *mm, gfp_t gfp) +{ + struct ptdesc *ptdesc; + + ptdesc =3D pagetable_alloc_node_noprof(nid, gfp, 0); + if (!ptdesc) + return NULL; + if (!pagetable_pte_ctor(mm, ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } + + return ptdesc_page(ptdesc); +} + +#define __pte_alloc_one_node(...) alloc_hooks(__pte_alloc_one_node_noprof(= __VA_ARGS__)) +#endif + #ifndef __HAVE_ARCH_PTE_ALLOC_ONE /** * pte_alloc_one - allocate a page for PTE-level user page table @@ -99,6 +119,17 @@ static inline pgtable_t pte_alloc_one_noprof(struct mm_= struct *mm) return __pte_alloc_one_noprof(mm, GFP_PGTABLE_USER); } #define pte_alloc_one(...) alloc_hooks(pte_alloc_one_noprof(__VA_ARGS__)) + +#ifdef CONFIG_KERNEL_REPLICATION +static inline pgtable_t pte_alloc_one_node_noprof(unsigned int nid, + struct mm_struct *mm) +{ + return __pte_alloc_one_node(nid, mm, GFP_PGTABLE_USER | __GFP_THISNODE); +} + +#define pte_alloc_one_node(...) alloc_hooks(pte_alloc_one_node_noprof(__VA= _ARGS__)) +#endif + #endif =20 /* @@ -154,6 +185,33 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_st= ruct *mm, unsigned long ad return ptdesc_address(ptdesc); } #define pmd_alloc_one(...) alloc_hooks(pmd_alloc_one_noprof(__VA_ARGS__)) + +#ifdef CONFIG_KERNEL_REPLICATION +static inline pmd_t *pmd_alloc_one_node_noprof(unsigned int nid, + struct mm_struct *mm, + unsigned long addr) +{ + struct ptdesc *ptdesc; + gfp_t gfp =3D GFP_PGTABLE_USER; + + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + + gfp |=3D __GFP_THISNODE; + + ptdesc =3D pagetable_alloc_node_noprof(nid, gfp, 0); + if (!ptdesc) + return NULL; + if (!pagetable_pmd_ctor(mm, ptdesc)) { + pagetable_free(ptdesc); + return NULL; + } + return ptdesc_address(ptdesc); +} + +#define pmd_alloc_one_node(...) alloc_hooks(pmd_alloc_one_node_noprof(__VA= _ARGS__)) +#endif /* CONFIG_KERNEL_REPLICATION */ + #endif =20 #ifndef __HAVE_ARCH_PMD_FREE @@ -191,6 +249,27 @@ static inline pud_t *__pud_alloc_one_noprof(struct mm_= struct *mm, unsigned long } #define __pud_alloc_one(...) alloc_hooks(__pud_alloc_one_noprof(__VA_ARGS_= _)) =20 +#ifdef CONFIG_KERNEL_REPLICATION +static inline pud_t *__pud_alloc_one_node_noprof(unsigned int nid, + struct mm_struct *mm, + unsigned long addr) +{ + gfp_t gfp =3D GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm =3D=3D &init_mm) + gfp =3D GFP_PGTABLE_KERNEL; + + gfp |=3D __GFP_THISNODE; + ptdesc =3D pagetable_alloc_node_noprof(nid, gfp, 0); + if (!ptdesc) + return NULL; + return ptdesc_address(ptdesc); +} + +#define __pud_alloc_one_node(...) alloc_hooks(__pud_alloc_one_node_noprof(= __VA_ARGS__)) +#endif /* CONFIG_KERNEL_REPLICATION */ + #ifndef __HAVE_ARCH_PUD_ALLOC_ONE /** * pud_alloc_one - allocate memory for a PUD-level page table @@ -206,6 +285,17 @@ static inline pud_t *pud_alloc_one_noprof(struct mm_st= ruct *mm, unsigned long ad return __pud_alloc_one_noprof(mm, addr); } #define pud_alloc_one(...) alloc_hooks(pud_alloc_one_noprof(__VA_ARGS__)) + +#ifdef CONFIG_KERNEL_REPLICATION +static inline pud_t *pud_alloc_one_node_noprof(unsigned int nid, + struct mm_struct *mm, unsigned long addr) +{ + return __pud_alloc_one_node(nid, mm, addr); +} + +#define pud_alloc_one_node(...) alloc_hooks(pud_alloc_one_node_noprof(__VA= _ARGS__)) +#endif /* CONFIG_KERNEL_REPLICATION */ + #endif =20 static inline void __pud_free(struct mm_struct *mm, pud_t *pud) diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgta= ble-nop4d.h index 89c21f84cffb..95c03ceeeea0 100644 --- a/include/asm-generic/pgtable-nop4d.h +++ b/include/asm-generic/pgtable-nop4d.h @@ -48,6 +48,11 @@ static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned lon= g address) * inside the pgd, so has no extra memory associated with it. */ #define p4d_alloc_one(mm, address) NULL + +#ifdef CONFIG_KERNEL_REPLICATION +#define p4d_alloc_one_node(nid, mm, address) NULL +#endif + #define p4d_free(mm, x) do { } while (0) #define p4d_free_tlb(tlb, x, a) do { } while (0) =20 diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgta= ble-nopmd.h index 36b6490ed180..9849760b3751 100644 --- a/include/asm-generic/pgtable-nopmd.h +++ b/include/asm-generic/pgtable-nopmd.h @@ -60,6 +60,11 @@ static inline pmd_t * pmd_offset(pud_t * pud, unsigned l= ong address) * inside the pud, so has no extra memory associated with it. */ #define pmd_alloc_one(mm, address) NULL + +#ifdef CONFIG_KERNEL_REPLICATION +#define pmd_alloc_one_node(nid, mm, address) NULL +#endif + static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) { } diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgta= ble-nopud.h index 356cbfbaab24..0acbac9961d0 100644 --- a/include/asm-generic/pgtable-nopud.h +++ b/include/asm-generic/pgtable-nopud.h @@ -56,6 +56,11 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned lon= g address) * inside the p4d, so has no extra memory associated with it. */ #define pud_alloc_one(mm, address) NULL + +#ifdef CONFIG_KERNEL_REPLICATION +#define pud_alloc_one_node(nid, mm, address) NULL +#endif + #define pud_free(mm, x) do { } while (0) #define pud_free_tlb(tlb, x, a) do { } while (0) =20 diff --git a/include/linux/mm.h b/include/linux/mm.h index dd09c438fa23..95213a81907c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3595,8 +3595,24 @@ static inline int __p4d_alloc(struct mm_struct *mm, = pgd_t *pgd, { return 0; } -#else + +#ifdef CONFIG_KERNEL_REPLICATION +static inline int __p4d_alloc_node(unsigned int nid, + struct mm_struct *mm, + pgd_t *pgd, unsigned long address) +{ + return 0; +} +#endif + +#else /* !__PAGETABLE_P4D_FOLDED */ int __p4d_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address); + +#ifdef CONFIG_KERNEL_REPLICATION +int __p4d_alloc_node(unsigned int nid, struct mm_struct *mm, + pgd_t *pgd, unsigned long address); +#endif + #endif =20 #if defined(__PAGETABLE_PUD_FOLDED) || !defined(CONFIG_MMU) @@ -3605,12 +3621,27 @@ static inline int __pud_alloc(struct mm_struct *mm,= p4d_t *p4d, { return 0; } + +#ifdef CONFIG_KERNEL_REPLICATION +static inline int __pud_alloc_node(unsigned int nid, + struct mm_struct *mm, + p4d_t *p4d, unsigned long address) +{ + return 0; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static inline void mm_inc_nr_puds(struct mm_struct *mm) {} static inline void mm_dec_nr_puds(struct mm_struct *mm) {} =20 #else int __pud_alloc(struct mm_struct *mm, p4d_t *p4d, unsigned long address); =20 +#ifdef CONFIG_KERNEL_REPLICATION +int __pud_alloc_node(unsigned int nid, + struct mm_struct *mm, + p4d_t *p4d, unsigned long address); +#endif /* CONFIG_KERNEL_REPLICATION */ static inline void mm_inc_nr_puds(struct mm_struct *mm) { if (mm_pud_folded(mm)) @@ -3633,12 +3664,27 @@ static inline int __pmd_alloc(struct mm_struct *mm,= pud_t *pud, return 0; } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static inline int __pmd_alloc_node(unsigned int nid, + struct mm_struct *mm, + pud_t *pud, unsigned long address) +{ + return 0; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static inline void mm_inc_nr_pmds(struct mm_struct *mm) {} static inline void mm_dec_nr_pmds(struct mm_struct *mm) {} =20 #else int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address); =20 +#ifdef CONFIG_KERNEL_REPLICATION +int __pmd_alloc_node(unsigned int nid, + struct mm_struct *mm, + pud_t *pud, unsigned long address); +#endif /* CONFIG_KERNEL_REPLICATION */ + static inline void mm_inc_nr_pmds(struct mm_struct *mm) { if (mm_pmd_folded(mm)) @@ -3710,6 +3756,32 @@ static inline pmd_t *pmd_alloc(struct mm_struct *mm,= pud_t *pud, unsigned long a return (unlikely(pud_none(*pud)) && __pmd_alloc(mm, pud, address))? NULL: pmd_offset(pud, address); } + +#ifdef CONFIG_KERNEL_REPLICATION +static inline p4d_t *p4d_alloc_node(unsigned int nid, + struct mm_struct *mm, + pgd_t *pgd, unsigned long address) +{ + return (unlikely(pgd_none(*pgd)) && __p4d_alloc_node(nid, mm, pgd, addres= s)) ? + NULL : p4d_offset(pgd, address); +} + +static inline pud_t *pud_alloc_node(unsigned int nid, + struct mm_struct *mm, + p4d_t *p4d, unsigned long address) +{ + return (unlikely(p4d_none(*p4d)) && __pud_alloc_node(nid, mm, p4d, addres= s)) ? + NULL : pud_offset(p4d, address); +} + +static inline pmd_t *pmd_alloc_node(unsigned int nid, + struct mm_struct *mm, + pud_t *pud, unsigned long address) +{ + return (unlikely(pud_none(*pud)) && __pmd_alloc_node(nid, mm, pud, addres= s)) ? + NULL : pmd_offset(pud, address); +} +#endif /* CONFIG_KERNEL_REPLICATION */ #endif /* CONFIG_MMU */ =20 enum pt_flags { @@ -3812,6 +3884,19 @@ static inline void pagetable_free_kernel(struct ptde= sc *pt) __pagetable_free(pt); } #endif + +#ifdef CONFIG_KERNEL_REPLICATION + +static inline struct ptdesc *pagetable_alloc_node_noprof(int nid, gfp_t gf= p, + unsigned int order) +{ + struct page *page =3D alloc_pages_node_noprof(nid, gfp | __GFP_COMP, orde= r); + + return page_ptdesc(page); +} + +#endif + /** * pagetable_free - Free pagetables * @pt: The page table descriptor diff --git a/mm/memory.c b/mm/memory.c index 8b0c2c735d3d..05853e4fb266 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6906,6 +6906,28 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vm= a, unsigned long address, EXPORT_SYMBOL_GPL(handle_mm_fault); =20 #ifndef __PAGETABLE_P4D_FOLDED + +#ifdef CONFIG_KERNEL_REPLICATION +int __p4d_alloc_node(unsigned int nid, + struct mm_struct *mm, + pgd_t *pgd, unsigned long address) +{ + p4d_t *new =3D p4d_alloc_one_node(nid, mm, address); + if (!new) + return -ENOMEM; + + spin_lock(&mm->page_table_lock); + if (pgd_present(*pgd)) { /* Another has populated it */ + p4d_free(mm, new); + } else { + smp_wmb(); /* See comment in pmd_install() */ + pgd_populate(mm, pgd, new); + } + spin_unlock(&mm->page_table_lock); + return 0; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + /* * Allocate p4d page table. * We've already handled the fast-path in-line. @@ -6929,6 +6951,28 @@ int __p4d_alloc(struct mm_struct *mm, pgd_t *pgd, un= signed long address) #endif /* __PAGETABLE_P4D_FOLDED */ =20 #ifndef __PAGETABLE_PUD_FOLDED + +#ifdef CONFIG_KERNEL_REPLICATION +int __pud_alloc_node(unsigned int nid, + struct mm_struct *mm, + p4d_t *p4d, unsigned long address) +{ + pud_t *new =3D pud_alloc_one_node(nid, mm, address); + if (!new) + return -ENOMEM; + + spin_lock(&mm->page_table_lock); + if (!p4d_present(*p4d)) { + mm_inc_nr_puds(mm); + smp_wmb(); /* See comment in pmd_install() */ + p4d_populate(mm, p4d, new); + } else /* Another has populated it */ + pud_free(mm, new); + spin_unlock(&mm->page_table_lock); + return 0; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + /* * Allocate page upper directory. * We've already handled the fast-path in-line. @@ -6974,6 +7018,30 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, un= signed long address) spin_unlock(ptl); return 0; } + +#ifdef CONFIG_KERNEL_REPLICATION +int __pmd_alloc_node(unsigned int nid, + struct mm_struct *mm, + pud_t *pud, unsigned long address) +{ + spinlock_t *ptl; + pmd_t *new =3D pmd_alloc_one_node(nid, mm, address); + if (!new) + return -ENOMEM; + + ptl =3D pud_lock(mm, pud); + if (!pud_present(*pud)) { + mm_inc_nr_pmds(mm); + smp_wmb(); /* See comment in pmd_install() */ + pud_populate(mm, pud, new); + } else { /* Another has populated it */ + pmd_free(mm, new); + } + spin_unlock(ptl); + return 0; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + #endif /* __PAGETABLE_PMD_FOLDED */ =20 static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB430483BFF for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; cv=none; b=G0ErVpK/+sm3sGMzCsXnDBnB3HCcSh/Up1W1uYCPwtqU5z9zgdfTH8vY6byiHnY5R2wJ9zOWJfFBYv+6y9iNvht1KmsIXcUu/VOSkhAuR1KNrTesPNTrFR4AL5ypAYXkb4yrX50JfWC0nhGgXl9b1XE51nW8oy+Rt+yLcWGlLzA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; c=relaxed/simple; bh=5ciELrr8Kk3fh+UFlH1sxexZsIYTzsnitGwn4d2bgZs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DLsR1j6OBRyQ0yWxxjcDeM5a6Ih5cQLq0+55Dd1YUdO2LzxfsULN8cGa1jRnXrhxMF2favAQjX81ud3gAm/7jYah/HzYA2tAA4WfCWP4wc4VvuXsaaMSw+IOmY+L+SwOlm8Az2Hsbqz3uTCXcOqEvZxll1IDgoavGhRsxLraa48= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=EywF5R3E; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="EywF5R3E" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=CEcX3HXLMzVOuLicTphBIj/R1Bu+Rm00Soec5tDKcys=; b=EywF5R3EdvPAq5zPDcW+XuVFbk1/bBUity+jgaldkQtH15J6yMmiQLRhCnW/+p3b8uDcX0TO1 Ax6VCwo0moAgqjpzspUqI4gkuwNmyv2ejH8HJreRveH306WGRUQxS41RlGX/9zjTKTQij6rPkqD t9bp17Vo0C1A4fOiICCfyes= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KD2c0vzHnH3v; Fri, 28 Aug 2026 00:24:00 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 5EC9B40570; Fri, 28 Aug 2026 00:24:36 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:35 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 04/18] arm64: add arch callbacks for kernel replication Date: Fri, 28 Aug 2026 00:11:44 +0800 Message-ID: <20260827161158.3618409-5-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/include/asm/numa_replication.h | 54 +++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 arch/arm64/include/asm/numa_replication.h diff --git a/arch/arm64/include/asm/numa_replication.h b/arch/arm64/include= /asm/numa_replication.h new file mode 100644 index 000000000000..441a399f9c51 --- /dev/null +++ b/arch/arm64/include/asm/numa_replication.h @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_NUMA_REPLICATION_H +#define __ASM_NUMA_REPLICATION_H + +#ifdef CONFIG_KERNEL_REPLICATION +#include +#include +#include +#include +#include +#include +#include + +#define PAGE_TABLE_REPLICATION_LEFT ((max((u64)_end - SZ_2G, (u64)MODULES= _VADDR)) & PGDIR_MASK) +#define PAGE_TABLE_REPLICATION_RIGHT ((((u64)_end + SZ_2G) & PGDIR_MASK) += PGDIR_SIZE - 1) + +static inline pgd_t *numa_replicate_pgt_pgd(int nid) +{ + pgd_t *new_pgd; + struct page *pgd_page; + + pgd_page =3D alloc_pages_node_noprof(nid, GFP_PGTABLE_KERNEL, 2); + BUG_ON(pgd_page =3D=3D NULL); + + new_pgd =3D (pgd_t *)page_address(pgd_page); + new_pgd +=3D (PAGE_SIZE * 2 / sizeof(pgd_t)); //Extra pages for KPTI + copy_page(new_pgd, swapper_pg_dir); + + return new_pgd; +} + +static inline void numa_load_replicated_pgd(pgd_t *pgd) +{ + cpu_replace_ttbr1(pgd); + local_flush_tlb_all(); +} + +static inline ssize_t numa_cpu_dump(struct seq_file *m) +{ + seq_printf(m, "NODE: #%02d, CPU: #%04d, ttbr1_el1: 0x%p, COMM: %s\n", + numa_node_id(), + smp_processor_id(), + (void *)read_sysreg(ttbr1_el1), + current->group_leader->comm); + return 0; +} + +static inline void numa_sync_text_replicas(unsigned long start, unsigned l= ong end) +{ + caches_clean_inval_pou(start, end); + icache_inval_all_pou(); +} +#endif /* CONFIG_KERNEL_REPLICATION */ +#endif /* __ASM_NUMA_REPLICATION_H */ --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB10F46EF87 for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847891; cv=none; b=N8QMUhpE0zyggoDpsvimLOUgqcamWJzizxTsUoqOs6spqehckEhI2mkIXdEfsqCAUqTFjllB8JAhUwv8L+hMEOuj9s0A+LTTJaWisJUK/GS5aNzbk7oRIqZniG9vF4gpx4t/dsbBJzzKDEBVfL58/vhhAsqHbL0IBWDjXeeggSI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847891; c=relaxed/simple; bh=myF9nUyNOrWsUYnQdiGqU/xU6L1O7Ec2QLtHgUSVHAg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=bqxjthVUZR5VzLiKRBU/uAZlf5aA1WT35Kfkl6MG0MiTWvTNK27Wvifa/9uJ/5jGtfL/eA9/LhHuXdJYTbIXjNkQVZqgrnQ4cbwTl7ZA1NTmEHVOhpfXDE1KnELXtEvEz4OR36oYi+BLfuOl49psCLu1gOVFtSfnvrNdGJqiEmY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=2vyssauP; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="2vyssauP" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=0XCpW5x2XMrFIuKJOAuq9yTaq6svDVv7iaV8fimo4Ec=; b=2vyssauPjk6VH8NkuDtsmgc01fHaaBh+AtFJ/Uqy3Y/Sa9M7GMOXeYlq1BQeRxV30kfBCpukO aMdlH6P3npTN9LuNOspLX8cWn5Uv/RvSNrfupyVFQsqRJwdII4pIuru0XhMYJ11TzAw98l9nLaf /Qp+Oy7j5GjC6AjtS3uPEg4= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KV6Vk9zJ46Yq; Fri, 28 Aug 2026 00:24:14 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 7396940572; Fri, 28 Aug 2026 00:24:36 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 05/18] mm: per-NUMA node replication core infrastructure Date: Fri, 28 Aug 2026 00:11:45 +0800 Message-ID: <20260827161158.3618409-6-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" In current design mutable kernel data modifications don't require synchronization between translation tables due to on 64-bit platforms all physical memory already mapped in kernel space and this mapping is persistent. No need to synchronize userspace at all. Due to separate ttbrs for userspace and kernel are used TT overview: NODE 0 NODE 1 USER KERNEL USER KERNEL --------------------- --------------------- PGD | | | | | | | | |*| | | | | | | | | |*| --------------------- --------------------- | | ------------------- ------------------- | | --------------------- --------------------- PUD | | | | | | | |*|*| | | | | | | | |*|*| --------------------- --------------------- | | ------------------- ------------------- | | --------------------- --------------------- PMD |READ-ONLY|MUTABLE | |READ-ONLY|MUTABLE | --------------------- --------------------- | | | | | -------------------------- | | | -------- ------- -------- PHYS | | | | | | MEM -------- ------- -------- <------> <------> NODE 0 Shared NODE 1 between nodes * - entries unique in each table Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Co-developed-by: Artem Kuzin Signed-off-by: Artem Kuzin Co-developed-by: Nikita Panov Signed-off-by: Nikita Panov --- include/linux/mm_types.h | 3 + include/linux/numa_kernel_replication.h | 89 +++ mm/Makefile | 2 + mm/numa_kernel_replication.c | 740 ++++++++++++++++++++++++ 4 files changed, 834 insertions(+) create mode 100644 include/linux/numa_kernel_replication.h create mode 100644 mm/numa_kernel_replication.c diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6d815f6440c9..06c42b7918a5 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1419,6 +1419,9 @@ struct mm_struct { #ifdef CONFIG_MM_ID mm_id_t mm_id; #endif /* CONFIG_MM_ID */ +#ifdef CONFIG_KERNEL_REPLICATION + pgd_t **pgd_numa; +#endif } __randomize_layout; =20 /* diff --git a/include/linux/numa_kernel_replication.h b/include/linux/numa_k= ernel_replication.h new file mode 100644 index 000000000000..34aa063d42e6 --- /dev/null +++ b/include/linux/numa_kernel_replication.h @@ -0,0 +1,89 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_NUMA_REPLICATION_H +#define _LINUX_NUMA_REPLICATION_H + +#ifdef CONFIG_KERNEL_REPLICATION + +#include +#include +#include +#include + +#include + +extern nodemask_t replica_nodes; + +#define for_each_memory_node(nid) \ + for (nid =3D first_node(replica_nodes); \ + nid !=3D MAX_NUMNODES; \ + nid =3D next_node(nid, replica_nodes)) + +#define this_node_pgd(mm) ((mm)->pgd_numa[numa_node_id()]) +#define per_node_pgd(mm, nid) ((mm)->pgd_numa[nid]) + +static inline bool numa_addr_has_replica(const void *addr) +{ + return ((unsigned long)addr >=3D PAGE_TABLE_REPLICATION_LEFT) && + ((unsigned long)addr <=3D PAGE_TABLE_REPLICATION_RIGHT); +} + +void __init numa_replication_init(void); +void __init numa_replicate_kernel_text(void); +void numa_replicate_kernel_rodata(void); +void numa_replication_fini(void); + +void numa_setup_pgd(void); +void __init_or_module *numa_get_replica(void *vaddr, int nid); +int numa_get_memory_node(int nid); +void dump_mm_pgtables(struct mm_struct *mm, + unsigned long start, unsigned long end); + +#else + +#include + +#define this_node_pgd(mm) ((mm)->pgd) +#define per_node_pgd(mm, nid) ((mm)->pgd) + +static inline void numa_setup_pgd(void) +{ +} + +static inline void __init numa_replication_init(void) +{ +} + +static inline void __init numa_replicate_kernel_text(void) +{ +} + +static inline void numa_replicate_kernel_rodata(void) +{ +} + +static inline void numa_replication_fini(void) +{ +} + +static inline bool numa_addr_has_replica(const void *addr) +{ + return false; +} + +static inline bool is_text_replicated(void) +{ + return false; +} + +static inline void *numa_get_replica(void *vaddr, int nid) +{ + return lm_alias(vaddr); +} + +static inline void dump_mm_pgtables(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +#endif /*CONFIG_KERNEL_REPLICATION*/ +#endif /*_LINUX_NUMA_REPLICATION_H*/ diff --git a/mm/Makefile b/mm/Makefile index e7245cb88c66..e2bbc04a48d3 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -148,3 +148,5 @@ obj-$(CONFIG_EXECMEM) +=3D execmem.o obj-$(CONFIG_TMPFS_QUOTA) +=3D shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) +=3D tests/lazy_mmu_mode_kunit.o obj-$(CONFIG_MEM_ALLOC_PROFILING) +=3D alloc_tag.o +obj-$(CONFIG_KERNEL_REPLICATION) +=3D numa_kernel_replication.o + diff --git a/mm/numa_kernel_replication.c b/mm/numa_kernel_replication.c new file mode 100644 index 000000000000..082aabc6b8bc --- /dev/null +++ b/mm/numa_kernel_replication.c @@ -0,0 +1,740 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#define KERNEL_TEXT_START ((unsigned long)&_stext) +#define KERNEL_TEXT_END ((unsigned long)&_etext) + +#define KERNEL_RODATA_START ((unsigned long)&__start_rodata) +#define KERNEL_RODATA_END ((unsigned long)&__end_rodata) + +#define PMD_ALLOC_ORDER (PMD_SHIFT-PAGE_SHIFT) +#define PAGES_PER_PMD (1 << PMD_ALLOC_ORDER) + +#define replication_log(data, fmt, args...) \ +({ \ + if (data && data->m) \ + seq_printf(data->m, fmt, ##args); \ + else \ + pr_info(KERN_CONT fmt, ##args); \ +}) + +struct numa_node_desc { + pgd_t *pgd; + void *text_vaddr; + void *rodata_vaddr; +}; + +static struct numa_node_desc __initdata_or_module node_desc[MAX_NUMNODES]; + +struct dump_data { + struct seq_file *m; +}; + +struct dump_config { + int pgd_extra_info:1; + int p4d_extra_info:1; + int pud_extra_info:1; + int pmd_extra_info:1; + int pte_extra_info:1; + struct dump_data *data; +}; + +static bool text_replicated; +/* + * The first ready NUMA node, used as a source node + * for kernel text and rodata replication + */ +static unsigned int master_node =3D INT_MAX; +/* + * The case when machine has memoryless nodes is rare + * but possible. To handle memoryless nodes properly + * kernel replication maintains mapping node -> node with memory + * for all NUMA nodes. + */ +static int node_to_memory_node[MAX_NUMNODES]; + +static bool pgtables_extra; +static DEFINE_SPINLOCK(debugfs_lock); + +bool is_text_replicated(void) +{ + return text_replicated; +} + +static void binary_dump(struct dump_data *data, unsigned long value) +{ + int i; + + for (i =3D BITS_PER_LONG - 1; i >=3D 0; i--) { + if ((BITS_PER_LONG - 1 - i) % BITS_PER_BYTE =3D=3D 0) + replication_log(data, "%-9d", i); + } + replication_log(data, "%d\n", 0); + + for (i =3D BITS_PER_LONG - 1; i >=3D 0; i--) { + if ((BITS_PER_LONG - 1 - i) % BITS_PER_BYTE =3D=3D 0) + replication_log(data, "|"); + + replication_log(data, "%d", (1UL << i) & value ? 1 : 0); + } + replication_log(data, "|\n"); +} + +static int pgd_callback(pgd_t *pgd, + unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + unsigned long val =3D pgd_val(*pgd); + struct dump_config *c =3D (struct dump_config *)walk->private; + + if (!val) + return 0; + + addr =3D addr & PGDIR_MASK; + next =3D (addr & PGDIR_MASK) - 1 + PGDIR_SIZE; + + replication_log(c->data, + "PGD ADDR: 0x%p PGD VAL: 0x%016lx [%p --- %p]\n", + pgd, val, (void *)addr, (void *)next); + + if (c->pgd_extra_info) + binary_dump(c->data, val); + + return 0; +} + +static int p4d_callback(p4d_t *p4d, + unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + unsigned long val =3D p4d_val(*p4d); + struct dump_config *c =3D (struct dump_config *)walk->private; + + if (!val) + return 0; + + addr =3D addr & P4D_MASK; + next =3D (addr & P4D_MASK) - 1 + P4D_SIZE; + + replication_log(c->data, + "P4D ADDR: 0x%p P4D VAL: 0x%016lx [%p --- %p]\n", + p4d, val, (void *)addr, (void *)next); + + if (c->p4d_extra_info) + binary_dump(c->data, val); + + return 0; +} + +static int pud_callback(pud_t *pud, + unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + unsigned long val =3D pud_val(*pud); + struct dump_config *c =3D (struct dump_config *)walk->private; + + if (!val) + return 0; + + addr =3D addr & PUD_MASK; + next =3D (addr & PUD_MASK) - 1 + PUD_SIZE; + + replication_log(c->data, + "PUD ADDR: 0x%p PUD VAL: 0x%016lx huge(%d) [%p --- %p]\n", + pud, val, !pud_table(*pud), (void *)addr, (void *)next); + + if (c->pud_extra_info) + binary_dump(c->data, val); + + return 0; +} + +static int pmd_callback(pmd_t *pmd, + unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + unsigned long val =3D pmd_val(*pmd); + unsigned long paddr =3D pmd_pfn(*pmd) << PAGE_SHIFT; + struct dump_config *c =3D (struct dump_config *)walk->private; + + if (!val) + return 0; + + addr =3D addr & PMD_MASK; + next =3D (addr & PMD_MASK) - 1 + PMD_SIZE; + + replication_log(c->data, + "PMD ADDR: 0x%p PMD VAL: 0x%016lx huge(%d) [%p --- %p] to %p\n", + pmd, val, !pmd_table(*pmd), (void *)addr, (void *)next, (void *)paddr); + + if (c->pmd_extra_info) + binary_dump(c->data, val); + + return 0; +} + +static int pte_callback(pte_t *pte, + unsigned long addr, unsigned long next, + struct mm_walk *walk) +{ + unsigned long val =3D pte_val(*pte); + unsigned long paddr =3D pte_pfn(*pte) << PAGE_SHIFT; + struct dump_config *c =3D (struct dump_config *)walk->private; + + if (!val) + return 0; + + addr =3D addr & PAGE_MASK; + next =3D (addr & PAGE_MASK) - 1 + PAGE_SIZE; + + replication_log(c->data, + "PTE ADDR: 0x%p PTE VAL: 0x%016lx [%p --- %p] to %p\n", + pte, val, (void *)addr, (void *)next, (void *)paddr); + + if (c->pte_extra_info) + binary_dump(c->data, val); + + return 0; +} + +static int pte_hole_callback(unsigned long addr, unsigned long next, + int depth, struct mm_walk *walk) +{ + struct dump_config *c =3D (struct dump_config *)walk->private; + + replication_log(c->data, "%*chole\n", depth * 2, ' '); + + return 0; +} + +static void dump_pgtables(struct mm_struct *mm, + struct dump_data *data, + unsigned long start, unsigned long end) +{ + int nid =3D 0; + int extra =3D pgtables_extra ? 1 : 0; + bool locked =3D false; + struct dump_config conf =3D { + .pgd_extra_info =3D extra, + .p4d_extra_info =3D extra, + .pud_extra_info =3D extra, + .pmd_extra_info =3D extra, + .pte_extra_info =3D extra, + .data =3D data, + }; + + const struct mm_walk_ops ops =3D { + .pgd_entry =3D pgd_callback, + .p4d_entry =3D p4d_callback, + .pud_entry =3D pud_callback, + .pmd_entry =3D pmd_callback, + .pte_entry =3D pte_callback, + .pte_hole =3D pte_hole_callback + }; + + BUG_ON(data && data->m =3D=3D NULL); + + start =3D start & PAGE_MASK; + end =3D (end & PAGE_MASK) - 1 + PAGE_SIZE; + + replication_log(data, + "----PER-NUMA NODE KERNEL REPLICATION ENABLED----\n"); + + if (rwsem_is_locked(&mm->mmap_lock)) + locked =3D true; + else + mmap_read_lock(mm); + + for_each_memory_node(nid) { + replication_log(data, "NUMA node id #%d\n", nid); + replication_log(data, "PGD: %p PGD phys: %p\n", + mm->pgd_numa[nid], (void *)virt_to_phys(mm->pgd_numa[nid])); + walk_kernel_page_table_range(start, end, &ops, mm->pgd_numa[nid], &conf); + } + + if (!locked) + mmap_read_unlock(mm); + + replication_log(data, + "----PER-NUMA NODE KERNEL REPLICATION ENABLED----\n"); +} + +static void dump_kernel_pgtables(struct dump_data *data, + unsigned long start, unsigned long end) +{ + dump_pgtables(&init_mm, data, start, end); +} + +void dump_mm_pgtables(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + dump_pgtables(mm, NULL, start, end); +} + +static void cpu_dump(void *info) +{ + struct dump_data *data =3D (struct dump_data *)info; + + spin_lock(&debugfs_lock); + numa_cpu_dump(data->m); + spin_unlock(&debugfs_lock); +} + +static int stats_show(struct seq_file *m, void *v) +{ + int cpu; + struct dump_data data =3D { + .m =3D m, + }; + + for_each_online_cpu(cpu) + smp_call_function_single(cpu, cpu_dump, &data, 1); + + return 0; +} + +DEFINE_SHOW_ATTRIBUTE(stats); + +static int pgtables_show(struct seq_file *m, void *v) +{ + struct dump_data data =3D { + .m =3D m, + }; + + dump_kernel_pgtables(&data, + KERNEL_TEXT_START, KERNEL_RODATA_END - 1); + + return 0; +} + +DEFINE_SHOW_ATTRIBUTE(pgtables); + +static void debugfs_init(void) +{ + struct dentry *dir; + static struct dentry *debugfs_dir; + + debugfs_dir =3D debugfs_create_dir("numa_replication", NULL); + if (IS_ERR(debugfs_dir)) { + pr_err("Failed to create debugfs entry for NUMA" + " replication: %ld\n", + PTR_ERR(debugfs_dir)); + return; + } + dir =3D debugfs_create_file("stats", 0400, debugfs_dir, + NULL, &stats_fops); + if (IS_ERR(dir)) { + pr_err("Failed to create debugfs entry for NUMA" + " replication stats: %ld\n", + PTR_ERR(dir)); + return; + } + + dir =3D debugfs_create_file("pgtables_kernel", 0400, debugfs_dir, + NULL, &pgtables_fops); + if (IS_ERR(dir)) { + pr_err("Failed to create debugfs entry for NUMA" + " replication pgtables: %ld\n", + PTR_ERR(dir)); + return; + } + + debugfs_create_bool("pgtables_kernel_extra", 0600, debugfs_dir, + &pgtables_extra); +} + +/* + * The case, when machine has memoryless NUMA nodes + * should be handled in a special way. To do this we + * create node<->memory mapping to have an information + * about the node with memory that memoryless node can use. + */ +static void init_node_to_memory_mapping(void) +{ + int nid; + + for_each_online_node(nid) { + int memory_nid; + int min_dist =3D INT_MAX; + + node_to_memory_node[nid] =3D nid; + for_each_memory_node(memory_nid) { + int dist =3D node_distance(nid, memory_nid); + + if (dist < min_dist) { + min_dist =3D dist; + node_to_memory_node[nid] =3D memory_nid; + } + } + pr_info("For node %d memory is on the node - %d\n", + nid, node_to_memory_node[nid]); + } +} + +int numa_get_memory_node(int nid) +{ + return node_to_memory_node[nid]; +} + +/* + * The function creates replica of particular memory area + * and install replicated memory in translation table of + * required NUMA node. + */ +static void replicate_memory(void *dst, unsigned long start, unsigned long= end, int nid) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + pgprot_t prot; + unsigned int offset_in_pages =3D 0; + unsigned long vaddr =3D start; + struct page *pages =3D virt_to_page(dst); + + memcpy(dst, lm_alias(start), end - start); + while (vaddr < end) { + pgd =3D pgd_offset_pgd(node_desc[nid].pgd, vaddr); + p4d =3D p4d_offset(pgd, vaddr); + pud =3D pud_offset(p4d, vaddr); + pmd =3D pmd_offset(pud, vaddr); + + if (pmd_leaf(*pmd)) { + prot =3D pmd_pgprot(*pmd); + + set_pmd(pmd, pfn_pmd(page_to_pfn(pages) + offset_in_pages, prot)); + offset_in_pages +=3D PAGES_PER_PMD; + vaddr +=3D PMD_SIZE; + continue; + } + pte =3D pte_offset_kernel(pmd, vaddr); + prot =3D pte_pgprot(*pte); + __set_pte(pte, pfn_pte(page_to_pfn(pages) + offset_in_pages, prot)); + offset_in_pages++; + vaddr +=3D PAGE_SIZE; + } +} + +static void __init replicate_kernel_text(int nid) +{ + replicate_memory(node_desc[nid].text_vaddr, + KERNEL_TEXT_START, KERNEL_TEXT_END, nid); + numa_sync_text_replicas((unsigned long)node_desc[nid].text_vaddr, + (unsigned long)node_desc[nid].text_vaddr + (KERNEL_TEXT_END - KERNEL_TEX= T_START)); +} + +static void replicate_kernel_rodata(int nid) +{ + replicate_memory(node_desc[nid].rodata_vaddr, + KERNEL_RODATA_START, KERNEL_RODATA_END, nid); +} + +//'-1' in next functions have only one purpose - prevent unsgined long ove= rflow +static void replicate_pgt_pte(pud_t *dst, pud_t *src, + unsigned long start, unsigned long end, + unsigned int nid) +{ + unsigned long left =3D start & PMD_MASK; + unsigned long right =3D (end & PMD_MASK) - 1 + PMD_SIZE; + unsigned long addr; + + pmd_t *clone_pmd =3D pmd_offset(dst, left); + pmd_t *orig_pmd =3D pmd_offset(src, left); + + for (addr =3D left; + (addr >=3D left && addr < right); addr +=3D PMD_SIZE) { + pgtable_t new_pte; + + if (pmd_none(*orig_pmd) || !pmd_table(*orig_pmd) || + pmd_val(*orig_pmd) =3D=3D 0) + goto skip; + + pmd_clear(clone_pmd); + new_pte =3D pte_alloc_one_node(nid, &init_mm); + pmd_populate_kernel(&init_mm, clone_pmd, page_to_virt(new_pte)); + BUG_ON(new_pte =3D=3D NULL); + + copy_page(page_to_virt(pmd_pgtable(*clone_pmd)), + page_to_virt(pmd_pgtable(*orig_pmd))); +skip: + clone_pmd++; + orig_pmd++; + } +} + +//'-1' in next functions have only one purpose - prevent unsgined long ove= rflow +static void replicate_pgt_pmd(p4d_t *dst, p4d_t *src, + unsigned long start, unsigned long end, + unsigned int nid) +{ + unsigned long left =3D start & PUD_MASK; + unsigned long right =3D (end & PUD_MASK) - 1 + PUD_SIZE; + + pud_t *clone_pud =3D pud_offset(dst, left); + pud_t *orig_pud =3D pud_offset(src, left); + + for (unsigned long addr =3D left; + (addr >=3D left && addr < right); addr +=3D PUD_SIZE) { + pmd_t *new_pmd; + + if (pud_none(*orig_pud) || !pud_table(*orig_pud) || + pud_val(*orig_pud) =3D=3D 0) + goto skip; + + pud_clear(clone_pud); + new_pmd =3D pmd_alloc_node(nid, &init_mm, clone_pud, addr); + BUG_ON(new_pmd =3D=3D NULL); + + copy_page(pud_pgtable(*clone_pud), pud_pgtable(*orig_pud)); + + replicate_pgt_pte(clone_pud, orig_pud, max(addr, start), + min(addr - 1 + PUD_SIZE, end), nid); +skip: + clone_pud++; + orig_pud++; + } +} + +static void replicate_pgt_pud(pgd_t *dst, pgd_t *src, + unsigned long start, unsigned long end, + unsigned int nid) +{ + unsigned long left =3D start & P4D_MASK; + unsigned long right =3D (end & P4D_MASK) - 1 + P4D_SIZE; + + p4d_t *clone_p4d =3D p4d_offset(dst, left); + p4d_t *orig_p4d =3D p4d_offset(src, left); + + for (unsigned long addr =3D left; + (addr >=3D left && addr < right); addr +=3D P4D_SIZE) { + pud_t *new_pud; + + if (p4d_none(*orig_p4d) || p4d_val(*orig_p4d) =3D=3D 0) + goto skip; + + p4d_clear(clone_p4d); + new_pud =3D pud_alloc_node(nid, &init_mm, clone_p4d, addr); + BUG_ON(new_pud =3D=3D NULL); + + copy_page(p4d_pgtable(*clone_p4d), p4d_pgtable(*orig_p4d)); + /* + * start and end passed to the next function must be in + * range of p4ds, so min and max are used here + */ + replicate_pgt_pmd(clone_p4d, orig_p4d, max(addr, start), + min(addr - 1 + P4D_SIZE, end), nid); +skip: + clone_p4d++; + orig_p4d++; + } +} + +static void replicate_pgt_p4d(pgd_t *dst, pgd_t *src, + unsigned long start, unsigned long end, + unsigned int nid) +{ + unsigned long left =3D start & PGDIR_MASK; + unsigned long right =3D (end & PGDIR_MASK) - 1 + PGDIR_SIZE; + + pgd_t *clone_pgd =3D pgd_offset_pgd(dst, left); + pgd_t *orig_pgd =3D pgd_offset_pgd(src, left); + + for (unsigned long addr =3D left; + (addr >=3D left && addr < right); addr +=3D PGDIR_SIZE) { + p4d_t *new_p4d; + + /* TODO: remove last condition and do something better + * In the case of a folded P4D level, pgd_none and pgd_huge + * always return 0, so we might start to replicate empty entries. + * We obviously want to avoid this, so the last check is performed here. + */ + if (pgd_none(*orig_pgd) || pgd_val(*orig_pgd) =3D=3D 0) + goto skip; + + pgd_clear(clone_pgd); + new_p4d =3D p4d_alloc_node(nid, &init_mm, clone_pgd, addr); + BUG_ON(new_p4d =3D=3D NULL); + + copy_page((void *)pgd_page_vaddr(*clone_pgd), + (void *)pgd_page_vaddr(*orig_pgd)); + replicate_pgt_pud(clone_pgd, orig_pgd, max(addr, start), + min(addr - 1 + PGDIR_SIZE, end), nid); +skip: + clone_pgd++; + orig_pgd++; + } +} + +static void replicate_pgt(int nid, unsigned long start, unsigned long end) +{ + replicate_pgt_p4d(node_desc[nid].pgd, init_mm.pgd, start, end, nid); +} + +/* + * Page tables replication works in a way when first + * pgd level replicated and then the replication of the + * left part if done. The only part of pagetable that + * contains text and rodata is replicated. Obviously a + * part of upper layer entries of page table should be + * replicated too. As result, the pgd, p4d, pud and pmd + * layers are touched by replication. In particular, the + * page table sub-tree that cover kernel text and rodata. + */ +static void replicate_pgtables(void) +{ + int nid; + + init_mm.pgd_numa =3D (pgd_t **)kmalloc(sizeof(pgd_t *) * MAX_NUMNODES, GF= P_PGTABLE_KERNEL); + BUG_ON(!init_mm.pgd_numa); + + for_each_memory_node(nid) { + node_desc[nid].pgd =3D numa_replicate_pgt_pgd(nid); + replicate_pgt(nid, PAGE_TABLE_REPLICATION_LEFT, + PAGE_TABLE_REPLICATION_RIGHT); + } + + init_mm.pgd =3D node_desc[numa_get_memory_node(master_node)].pgd; + + for_each_online_node(nid) { + int memory_nid =3D numa_get_memory_node(nid); + + init_mm.pgd_numa[nid] =3D node_desc[memory_nid].pgd; + } +} + +/* + * Kernel text replication includes two steps: + * 1. page tables replication for init_mm + * 2. kernel text pages replication and + * corresponding page table update. + * 3. setup page table, related to + * current NUMA node on current cpu, + * for other NUMA cpus page tables will + * be updated later, during cpu initialization. + * Master node - the first NUMA node, used as + * a source for replicas. Memory for master node + * is expected to be already local. + */ +void __init numa_replicate_kernel_text(void) +{ + int nid; + + replicate_pgtables(); + + for_each_memory_node(nid) { + if (nid =3D=3D master_node) + continue; + replicate_kernel_text(nid); + } + + text_replicated =3D true; + + numa_setup_pgd(); +} + +void numa_replicate_kernel_rodata(void) +{ + int nid; + + for_each_memory_node(nid) { + if (nid =3D=3D master_node) + continue; + replicate_kernel_rodata(nid); + } + + flush_tlb_all(); +} + +void numa_setup_pgd(void) +{ + numa_load_replicated_pgd(init_mm.pgd_numa[numa_node_id()]); +} + +void __init_or_module *numa_get_replica(void *vaddr, int nid) +{ + unsigned long addr =3D (unsigned long)vaddr; + unsigned long offset =3D addr - KERNEL_TEXT_START; + + BUG_ON(addr < KERNEL_TEXT_START || addr >=3D KERNEL_TEXT_END); + BUG_ON(node_desc[nid].text_vaddr =3D=3D NULL); + BUG_ON(numa_get_memory_node(nid) !=3D nid); + + return node_desc[nid].text_vaddr + offset; +} + +nodemask_t __ro_after_init replica_nodes =3D { { [0] =3D 1UL } }; + +void __init numa_replication_init(void) +{ + int nid; + + unsigned long align =3D PAGE_SIZE; +#ifdef CONFIG_ARM64_4K_PAGES + align =3D HPAGE_SIZE; +#else + align =3D CONT_PTE_SIZE; +#endif + nodes_clear(replica_nodes); + + for_each_node_state(nid, N_MEMORY) { + __node_set(nid, &replica_nodes); + } + + for_each_memory_node(nid) + pr_info("Memory node: %d\n", nid); + + init_node_to_memory_mapping(); + master_node =3D page_to_nid(virt_to_page(lm_alias((void *)KERNEL_TEXT_STA= RT))); + + pr_info("Master Node: #%d\n", master_node); + for_each_memory_node(nid) { + if (nid =3D=3D master_node) { + node_desc[nid].text_vaddr =3D lm_alias((void *)KERNEL_TEXT_START); + node_desc[nid].rodata_vaddr =3D lm_alias((void *)KERNEL_RODATA_START); + } else { + node_desc[nid].text_vaddr =3D memblock_alloc_try_nid( + (KERNEL_TEXT_END - KERNEL_TEXT_START), + align, 0, MEMBLOCK_ALLOC_ANYWHERE, nid); + + node_desc[nid].rodata_vaddr =3D memblock_alloc_try_nid( + (KERNEL_RODATA_END - KERNEL_RODATA_START), + align, 0, MEMBLOCK_ALLOC_ANYWHERE, nid); + } + + BUG_ON(node_desc[nid].text_vaddr =3D=3D NULL); + BUG_ON(node_desc[nid].rodata_vaddr =3D=3D NULL); + } +} + +void numa_replication_fini(void) +{ + int nid; + + /* + * Clear addresses form linear space + */ + for_each_memory_node(nid) { + node_desc[nid].text_vaddr =3D NULL; + node_desc[nid].rodata_vaddr =3D NULL; + } + + debugfs_init(); + + pr_info("Replicated page table : [%p --- %p]\n", + (void *)PAGE_TABLE_REPLICATION_LEFT, + (void *)PAGE_TABLE_REPLICATION_RIGHT); + + dump_kernel_pgtables(NULL, KERNEL_TEXT_START, KERNEL_RODATA_END - 1); +} --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB4F4484241 for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; cv=none; b=P4tDyLNrCzf5DYAtdHD1yV3rpT21PbatdY3SGohCahr2o8Pzz7ouuIt0dqJjKTK/Hl4XRDBzmLv4BoyGCrqWpO2EfgKvYMy8ZzT24aryJY6Y+UDBUi2Se9KRpxx7wPAiOpWS0fHHurrzH2vqBjgHeLRF0vdUtCn7pS3I+iolAhw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; c=relaxed/simple; bh=3Zi2sFCCUVjDBomdDDl7ydJXjOBy1jc6tc/I4IKpFzE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=dqngE39uuhK10b0tqu8FvIKMGYkgnAvHCVv5KBgkVfiDXsklcyiJn+mnA0/CDlj750lRJplQP+ZgyK9tnF1SgADsBdS7ew4IyZjKaIM8h8KR9tyzonY2jpA87FAJ48fvkJC5VU0Ly5PEs6r/wOnXgiVh/AGNHk52bhe7uAe1Z/g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=ggXzahVa; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="ggXzahVa" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=ubyjPs0XPl7ANEpl0YX14bjSE3xYgWesyhGRSLOi9nc=; b=ggXzahVaaBfCtHNRVFOrt6bSpyceyYC+W328qh5SyVVXpU7ve6ADh6fJGtgRPWvJ1Ro+sDJKs grtkf31gGgNDbjaZXxwrccLWpDTBTV9bFUMtSs+XQZKgw7pfzz3bMSYuJ9U/ZgkMRlD0E1zNXPT VnMYHbLlgtfayKqSH8rHPFE= Received: from mail.maildlp.com (unknown [172.18.224.107]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KD6DMWzHnH3v; Fri, 28 Aug 2026 00:24:00 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id DB7054058E; Fri, 28 Aug 2026 00:24:36 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 06/18] mm: add support of memory protection for NUMA replicas Date: Fri, 28 Aug 2026 00:11:46 +0800 Message-ID: <20260827161158.3618409-7-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- include/linux/set_memory.h | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 3030d9245f5a..03fccacef037 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -7,11 +7,32 @@ =20 #ifdef CONFIG_ARCH_HAS_SET_MEMORY #include + +#ifdef CONFIG_KERNEL_REPLICATION +int numa_set_memory_ro(unsigned long addr, int numpages); +int numa_set_memory_rw(unsigned long addr, int numpages); +int numa_set_memory_x(unsigned long addr, int numpages); +int numa_set_memory_nx(unsigned long addr, int numpages); +#else + +#define numa_set_memory_ro set_memory_ro +#define numa_set_memory_rw set_memory_rw +#define numa_set_memory_x set_memory_x +#define numa_set_memory_nx set_memory_nx + +#endif /* CONFIG_KERNEL_REPLICATION */ + #else static inline int __must_check set_memory_ro(unsigned long addr, int numpa= ges) { return 0; } static inline int __must_check set_memory_rw(unsigned long addr, int numpa= ges) { return 0; } static inline int __must_check set_memory_x(unsigned long addr, int numpa= ges) { return 0; } static inline int __must_check set_memory_nx(unsigned long addr, int numpa= ges) { return 0; } + +#define numa_set_memory_ro set_memory_ro +#define numa_set_memory_rw set_memory_rw +#define numa_set_memory_x set_memory_x +#define numa_set_memory_nx set_memory_nx + #endif =20 #ifndef set_memory_rox @@ -24,6 +45,20 @@ static inline int set_memory_rox(unsigned long addr, int= numpages) } #endif =20 +#ifndef numa_set_memory_rox +#ifdef CONFIG_KERNEL_REPLICATION +static inline int numa_set_memory_rox(unsigned long addr, int numpages) +{ + int ret =3D numa_set_memory_ro(addr, numpages); + if (ret) + return ret; + return numa_set_memory_x(addr, numpages); +} +#else +#define numa_set_memory_rox set_memory_rox +#endif +#endif + #ifndef CONFIG_ARCH_HAS_SET_DIRECT_MAP static inline int set_direct_map_invalid_noflush(struct page *page) { --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB637485CCB for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847893; cv=none; b=XrYzoA2j1GgbYMvl12uMUw3qCE/FfkxtqkRlv7/EnfCUAS+vwrjf2wPTvRkyo0VxwP9IK48Ze4mW5SPg2Fqi9IOSKEjE9hb6gMMU2wFOdhNwQNo/4WFgQVJfnq4v63yWfQj+nJyIgzNJ0cGV9aTJBl9Sb38OwVTXtIljAK570i8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847893; c=relaxed/simple; bh=CYqTiCgLA71tvPjxzXDbgehBFPEQs0cWBX5qVOFNlX0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Tu0uP34vk9883LHXl0Co6tmsva1UVN/HMa638g3gvohkC+nsZ/kaRlhBwjgen9UBVwbj3hw4bL0cxJARFki0hHUyBMBy4trcGllFEfAVDiCWk3ZJIXSOEzir/EQXROqWSQIJJWMeAFjXUfoY2RBRkVF7z1csuOKoP9uLAzsGTgM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=C5IJTV4J; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="C5IJTV4J" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=nKuenaCAMTLlNwdMoPfzm2hy+Pt0nBKVk4JpQv+n4NM=; b=C5IJTV4J1n9lw97V91/ERsJm3aHcbmynLEqmLveBrW9NAaQwfyJydojAtXyn2UNJqUnmRe9/R /2NlQfieQix/TjmZIknLChLnXdZdbA6mhO+V26YMdxKCAX2+4toDfUmszt5rt6Z8uN6/UWCOrqY YeCb5qZebuNEjUDCVmnBgbg= Received: from mail.maildlp.com (unknown [172.18.224.107]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KW1nY9zJ469r; Fri, 28 Aug 2026 00:24:15 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id C651A4058D; Fri, 28 Aug 2026 00:24:36 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 07/18] arm64: add support of memory protection for NUMA replicas Date: Fri, 28 Aug 2026 00:11:47 +0800 Message-ID: <20260827161158.3618409-8-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/mm/pageattr.c | 72 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 70 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index bbe98ac9ad8c..50a76385b184 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -9,6 +9,7 @@ #include #include #include +#include =20 #include #include @@ -102,7 +103,7 @@ bool can_set_direct_map(void) arm64_kfence_can_set_direct_map() || is_realm_world(); } =20 -static int update_range_prot(unsigned long start, unsigned long size, +static int update_range_prot_pgd(pgd_t *pgtable, unsigned long start, unsi= gned long size, pgprot_t set_mask, pgprot_t clear_mask) { struct page_change_data data; @@ -123,12 +124,19 @@ static int update_range_prot(unsigned long start, uns= igned long size, * must be eliminated by splitting the mapping. */ ret =3D walk_kernel_page_table_range_lockless(start, start + size, - &pageattr_ops, NULL, &data); + &pageattr_ops, pgtable, &data); lazy_mmu_mode_disable(); =20 return ret; } =20 +static int update_range_prot(unsigned long start, unsigned long size, + pgprot_t set_mask, pgprot_t clear_mask) +{ + return update_range_prot_pgd(NULL, start, size, set_mask, clear_mask); +} + + static int __change_memory_common(unsigned long start, unsigned long size, pgprot_t set_mask, pgprot_t clear_mask) { @@ -147,6 +155,22 @@ static int __change_memory_common(unsigned long start,= unsigned long size, return ret; } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static int __change_memory_common_replicas(unsigned long start, unsigned l= ong size, + pgprot_t set_mask, pgprot_t clear_mask) +{ + int nid; + int ret; + + for_each_memory_node(nid) { + ret |=3D update_range_prot_pgd(per_node_pgd(&init_mm, nid), start, size,= set_mask, clear_mask); + } + + flush_tlb_kernel_range(start, start + size); + return ret; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static int change_memory_common(unsigned long addr, int numpages, pgprot_t set_mask, pgprot_t clear_mask) { @@ -211,6 +235,20 @@ static int change_memory_common(unsigned long addr, in= t numpages, return __change_memory_common(start, size, set_mask, clear_mask); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static int numa_change_memory_common(unsigned long addr, int numpages, + pgprot_t set_mask, pgprot_t clear_mask) +{ + int ret; + + ret =3D change_memory_common(addr, numpages, set_mask, clear_mask); + if (ret) + return ret; + + return __change_memory_common_replicas(addr, numpages * PAGE_SIZE, set_ma= sk, clear_mask); +} +#endif /* CONFIG_KERNEL_REPLICATION */ + int set_memory_ro(unsigned long addr, int numpages) { return change_memory_common(addr, numpages, @@ -239,6 +277,36 @@ int set_memory_x(unsigned long addr, int numpages) __pgprot(PTE_PXN)); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +int numa_set_memory_x(unsigned long addr, int numpages) +{ + return numa_change_memory_common(addr, numpages, + __pgprot(PTE_MAYBE_GP), + __pgprot(PTE_PXN)); +} + +int numa_set_memory_nx(unsigned long addr, int numpages) +{ + return numa_change_memory_common(addr, numpages, + __pgprot(PTE_PXN), + __pgprot(PTE_MAYBE_GP)); +} + +int numa_set_memory_ro(unsigned long addr, int numpages) +{ + return numa_change_memory_common(addr, numpages, + __pgprot(PTE_RDONLY), + __pgprot(PTE_WRITE)); +} + +int numa_set_memory_rw(unsigned long addr, int numpages) +{ + return numa_change_memory_common(addr, numpages, + __pgprot(PTE_WRITE), + __pgprot(PTE_RDONLY)); +} +#endif /*CONFIG_KERNEL_REPLICATION*/ + int set_memory_valid(unsigned long addr, int numpages, int enable) { if (enable) --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB59E4854FF for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; cv=none; b=DO7PWxJevIgwWmytpfd0fHi9mJlfvD1KGthW3SS4L9cx4Ar1tA3NFSdTFp+N85hPFuK3utKi1mIgnBeJc3gfDV88pYR+8F9zA+UG8a32XzNUdZHIt9XUcbnFETULEI652FakSiKUr/fzcBNqRWNW7TmyFqINrr5YC8sfJnMX5ns= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; c=relaxed/simple; bh=ol9WO3YlM+anLjgz3GOTOizHJQ77u6h0Fn+YK1PzW1o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ScWfc0i6CzrUuZkrms2ALat9k4Kgu8LaDIpse6Ed3hf5qQ+lTuoGArpeL0XnKZ3uOgzKd01pFWTFPSAuwRzn+zHWLNYy9tUg/ldJlSPV4XJm199hhx1eaUy7ps5dw1XtzvNlgv1+8dckaDhidmPKtZLWvezd5hFYv+S+ndlDlMw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=JLBHyYAH; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="JLBHyYAH" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=oNCReT8Lzve2ehkzvtK73NbFcLcN4IwvjOltd7sg6Kg=; b=JLBHyYAHtKjETVOWSQdRRaXKVmJlMrZo/0ndCMHzFlXt3vDgP2gdxYdssFZ4wAwCG6dV9MMtb JQbMX6trjFE7wWXqdyI1ynnBZ6QPKRw865Tiqf/uYnhHJiska3VMxXEcATghutitWDhxgKETTer pJol0tV+AUj0F9LOFfq2Oww= Received: from mail.maildlp.com (unknown [172.18.224.107]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KW2znVzJ469r; Fri, 28 Aug 2026 00:24:15 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id F01ED4058D; Fri, 28 Aug 2026 00:24:36 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 08/18] mm: set memory permissions for BPF handlers replicas Date: Fri, 28 Aug 2026 00:11:48 +0800 Message-ID: <20260827161158.3618409-9-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/net/bpf_jit_comp.c | 4 ++-- include/asm-generic/set_memory.h | 14 ++++++++++++++ include/linux/set_memory.h | 14 -------------- kernel/bpf/core.c | 4 ++-- kernel/bpf/trampoline.c | 2 +- 5 files changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c18e005a41db..588d1b2a2753 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -3236,10 +3236,10 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke= _type old_t, /* non-zero plt_target indicates we're patching a bpf prog, * which is read only. */ - if (set_memory_rw(PAGE_MASK & ((uintptr_t)&plt->target), 1)) + if (numa_set_memory_rw(PAGE_MASK & ((uintptr_t)&plt->target), 1)) return -EFAULT; WRITE_ONCE(plt->target, plt_target); - set_memory_ro(PAGE_MASK & ((uintptr_t)&plt->target), 1); + numa_set_memory_ro(PAGE_MASK & ((uintptr_t)&plt->target), 1); /* since plt target points to either the new trampoline * or dummy_tramp, even if another CPU reads the old plt * target value before fetching the bl instruction to plt, diff --git a/include/asm-generic/set_memory.h b/include/asm-generic/set_mem= ory.h index c86abf6bc7ba..4c97978d2bfb 100644 --- a/include/asm-generic/set_memory.h +++ b/include/asm-generic/set_memory.h @@ -10,4 +10,18 @@ int set_memory_rw(unsigned long addr, int numpages); int set_memory_x(unsigned long addr, int numpages); int set_memory_nx(unsigned long addr, int numpages); =20 +#ifdef CONFIG_KERNEL_REPLICATION +int numa_set_memory_ro(unsigned long addr, int numpages); +int numa_set_memory_rw(unsigned long addr, int numpages); +int numa_set_memory_x(unsigned long addr, int numpages); +int numa_set_memory_nx(unsigned long addr, int numpages); +#else + +#define numa_set_memory_ro set_memory_ro +#define numa_set_memory_rw set_memory_rw +#define numa_set_memory_x set_memory_x +#define numa_set_memory_nx set_memory_nx + +#endif /* CONFIG_KERNEL_REPLICATION */ + #endif diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 03fccacef037..00b21d1c4331 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -8,20 +8,6 @@ #ifdef CONFIG_ARCH_HAS_SET_MEMORY #include =20 -#ifdef CONFIG_KERNEL_REPLICATION -int numa_set_memory_ro(unsigned long addr, int numpages); -int numa_set_memory_rw(unsigned long addr, int numpages); -int numa_set_memory_x(unsigned long addr, int numpages); -int numa_set_memory_nx(unsigned long addr, int numpages); -#else - -#define numa_set_memory_ro set_memory_ro -#define numa_set_memory_rw set_memory_rw -#define numa_set_memory_x set_memory_x -#define numa_set_memory_nx set_memory_nx - -#endif /* CONFIG_KERNEL_REPLICATION */ - #else static inline int __must_check set_memory_ro(unsigned long addr, int numpa= ges) { return 0; } static inline int __must_check set_memory_rw(unsigned long addr, int numpa= ges) { return 0; } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 8b294dfc1ad4..854db8e9dcf7 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -939,7 +939,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fil= l_hole_t bpf_fill_ill_ins if (!bpf_jit_mem_is_rox()) { bpf_fill_ill_insns(pack->ptr, BPF_PROG_PACK_SIZE); set_vm_flush_reset_perms(pack->ptr); - err =3D set_memory_rox((unsigned long)pack->ptr, + err =3D numa_set_memory_rox((unsigned long)pack->ptr, BPF_PROG_PACK_SIZE / PAGE_SIZE); if (err) goto out; @@ -977,7 +977,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t= bpf_fill_ill_insns, bool =20 bpf_fill_ill_insns(ptr, size); set_vm_flush_reset_perms(ptr); - err =3D set_memory_rox((unsigned long)ptr, + err =3D numa_set_memory_rox((unsigned long)ptr, size / PAGE_SIZE); if (err) { bpf_jit_free_exec(ptr); diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 90b70ea0d370..66656eb4714a 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -1508,7 +1508,7 @@ void __weak arch_free_bpf_trampoline(void *image, uns= igned int size) int __weak arch_protect_bpf_trampoline(void *image, unsigned int size) { WARN_ON_ONCE(size > PAGE_SIZE); - return set_memory_rox((long)image, 1); + return numa_set_memory_rox((long)image, 1); } =20 int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 fl= ags, --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB2B1481AB0 for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847892; cv=none; b=fRAIvjvhDr71KSGeF9NJGOpjZxkoaw3vhkrFdQFlkcNLHFwaYt+u+FS/ezEimc5xEJS6LhO9dhUnEkavYjqa0rky2dIkca30YL3yEHTL3tedJfvw5t+1DcNnIogpjUVrqo/nfvrnF8uXIhh2cJoERN4K64KM/vLQHlEaXjU6sjs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847892; c=relaxed/simple; bh=W/gXQkYRBcjjjEx3z1zIU6fBE9IUQGYNyKmE1u0y+Kk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ONpxqwM5KDcbZhcUQiQu+GNqBzaIHYgvk4nc08SrA3C3FTDGncaF9Jm60yMPFJvkU8UMNdQW7vLC2CUVlQazcYiXwV8W4PkDDXkyx4Jvik6k+u2ktxiY8bhL0Fhgsp29uEcpL9pUjBpprcOpHHk1ePxSSVHBWys7aYiXeUwWDfk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=TcSB835q; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="TcSB835q" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=qaJ1PYCrq7Z4YDNZIpLbpgluhtpESAtKia0lPi28zuQ=; b=TcSB835qmyqhdBzyqaw3s86lSkHss40WlOP4r/nQsE4HHZ0Y2Ss8V3Mo5FNl4ozP2wsVLqN8l MDaOylHR44Y1T4HKKFaTqtVHh4xYkVpUJlzXHrlCVoBKuC2DS15wP6DufUpJQ3D7hlP9x2Nygm9 gR1BN/IrmvkdvTJrfcN9q3Y= Received: from mail.maildlp.com (unknown [172.18.224.107]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KF0MjSzHnH3v; Fri, 28 Aug 2026 00:24:01 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 1057B4058E; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 09/18] mm: add replicas allocation support for vmalloc Date: Fri, 28 Aug 2026 00:11:49 +0800 Message-ID: <20260827161158.3618409-10-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" In order to support generic vmalloc, several modifications are required. If the allocated region belongs to the non-replicated part of the table, then the normal vmalloc mechanism is suitable. If the allocated region belongs to the replicated part of the table, a replicated table for each replica node must be created for this memory. This region might be replicated after its initialization, for example, to support replication of text and ro-data for loadable kernel modules. Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Co-developed-by: Artem Kuzin Signed-off-by: Artem Kuzin Co-developed-by: Nikita Panov Signed-off-by: Nikita Panov --- include/linux/mm.h | 2 + include/linux/vmalloc.h | 18 ++ mm/memory.c | 61 ++++++ mm/vmalloc.c | 454 +++++++++++++++++++++++++++++++++------- 4 files changed, 461 insertions(+), 74 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 95213a81907c..8a7083c02203 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1885,6 +1885,8 @@ int region_intersects(resource_size_t offset, size_t = size, unsigned long flags, struct page *vmalloc_to_page(const void *addr); unsigned long vmalloc_to_pfn(const void *addr); =20 +struct page *walk_to_page_node(int nid, const void *addr); + /* * Determine if an address is within the vmalloc range * diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index aed121d729b0..44eda2175a28 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -31,6 +31,10 @@ struct iov_iter; /* in uio.h */ #define VM_MAP_PUT_PAGES 0x00000200 /* put pages and free array in vfree */ #define VM_ALLOW_HUGE_VMAP 0x00000400 /* Allow for huge pages on arch= s with HAVE_ARCH_HUGE_VMALLOC */ =20 +#ifdef CONFIG_KERNEL_REPLICATION +#define VM_NUMA_SHARED 0x00002000 /* Pages shared between per-NUMA node T= T*/ +#endif + #if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \ !defined(CONFIG_KASAN_VMALLOC) #define VM_DEFER_KMEMLEAK 0x00000800 /* defer kmemleak object creation */ @@ -66,6 +70,10 @@ struct vm_struct { phys_addr_t phys_addr; const void *caller; unsigned long requested_size; +#ifdef CONFIG_KERNEL_REPLICATION + int node; + bool replicated; +#endif }; =20 struct vmap_area { @@ -177,6 +185,16 @@ extern void *__vmalloc_node_range_noprof(unsigned long= size, unsigned long align const void *caller) __alloc_size(1); #define __vmalloc_node_range(...) alloc_hooks(__vmalloc_node_range_noprof(= __VA_ARGS__)) =20 +#ifdef CONFIG_KERNEL_REPLICATION + /* + * DO NOT USE this function if you don't understand what it is doing + * Use only in pair with vmalloc(vm_flags|=3DVM_NUMA_SHARED) + */ +int __vmalloc_node_replicate_range(const void *addr, gfp_t gfp_mask, + pgprot_t prot, unsigned long vm_flags); +void vunmap_range_replicas(unsigned long addr, unsigned long end); +#endif + void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t= gfp_mask, int node, const void *caller) __alloc_size(1); #define __vmalloc_node(...) alloc_hooks(__vmalloc_node_noprof(__VA_ARGS__)) diff --git a/mm/memory.c b/mm/memory.c index 05853e4fb266..49e5a502abd8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -78,6 +78,7 @@ #include #include #include +#include =20 #include =20 @@ -7810,4 +7811,64 @@ void vma_pgtable_walk_end(struct vm_area_struct *vma) { if (is_vm_hugetlb_page(vma)) hugetlb_vma_unlock_read(vma); + +} +/** + * Walk in replicated tranlation table specified by nid. + * If kernel replication is disabled or text is not replicated yet, + * value of nid is not used + */ +struct page *walk_to_page_node(int nid, const void *vmalloc_addr) +{ + unsigned long addr =3D (unsigned long)vmalloc_addr; + struct page *page =3D NULL; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *ptep, pte; + + if (is_text_replicated()) + pgd =3D pgd_offset_pgd(per_node_pgd(&init_mm, nid), addr); + else + pgd =3D pgd_offset_pgd(init_mm.pgd, addr); + + if (pgd_none(*pgd)) + return NULL; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return NULL; /* XXX: no allowance for huge pgd */ + if (WARN_ON_ONCE(pgd_bad(*pgd))) + return NULL; + + p4d =3D p4d_offset(pgd, addr); + if (p4d_none(*p4d)) + return NULL; + if (p4d_leaf(*p4d)) + return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(p4d_bad(*p4d))) + return NULL; + + pud =3D pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + if (pud_leaf(*pud)) + return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pud_bad(*pud))) + return NULL; + + pmd =3D pmd_offset(pud, addr); + if (pmd_none(*pmd)) + return NULL; + if (pmd_leaf(*pmd)) + return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pmd_bad(*pmd))) + return NULL; + + ptep =3D pte_offset_map(pmd, addr); + pte =3D *ptep; + if (pte_present(pte)) + page =3D pte_page(pte); + pte_unmap(ptep); + + return page; } diff --git a/mm/vmalloc.c b/mm/vmalloc.c index bea9f76ed7e7..da6facf64db8 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -10,6 +10,7 @@ =20 #include #include +#include #include #include #include @@ -318,7 +319,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long add= r, unsigned long end, return err; } =20 -static int vmap_range_noflush(unsigned long addr, unsigned long end, +static int vmap_range_noflush_pgd(pgd_t *pgtable, unsigned long addr, unsi= gned long end, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { @@ -337,7 +338,7 @@ static int vmap_range_noflush(unsigned long addr, unsig= ned long end, BUG_ON(addr >=3D end); =20 start =3D addr; - pgd =3D pgd_offset_k(addr); + pgd =3D pgd_offset_pgd(pgtable, addr); do { next =3D pgd_addr_end(addr, end); err =3D vmap_p4d_range(pgd, addr, next, phys_addr, prot, @@ -352,6 +353,13 @@ static int vmap_range_noflush(unsigned long addr, unsi= gned long end, return err; } =20 +static int vmap_range_noflush(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + return vmap_range_noflush_pgd(init_mm.pgd, addr, end, phys_addr, prot, ma= x_page_shift); +} + int vmap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { @@ -490,18 +498,17 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned lon= g addr, unsigned long end, } =20 /* - * vunmap_range_noflush is similar to vunmap_range, but does not - * flush caches or TLBs. + * __vunmap_range_noflush_pgd is similar to vunmap_range, but does not + * flush caches or TLBs, and able to work with pgd granularity. * * The caller is responsible for calling flush_cache_vmap() before calling * this function, and flush_tlb_kernel_range after it has returned * successfully (and before the addresses are expected to cause a page fau= lt * or be re-mapped for something else, if TLB flushes are being delayed or * coalesced). - * - * This is an internal function only. Do not use outside mm/. */ -void __vunmap_range_noflush(unsigned long start, unsigned long end) +static void __vunmap_range_noflush_pgd(pgd_t *pgtable, + unsigned long start, unsigned long end) { unsigned long next; pgd_t *pgd; @@ -509,7 +516,7 @@ void __vunmap_range_noflush(unsigned long start, unsign= ed long end) pgtbl_mod_mask mask =3D 0; =20 BUG_ON(addr >=3D end); - pgd =3D pgd_offset_k(addr); + pgd =3D pgd_offset_pgd(pgtable, addr); do { next =3D pgd_addr_end(addr, end); if (pgd_bad(*pgd)) @@ -523,6 +530,17 @@ void __vunmap_range_noflush(unsigned long start, unsig= ned long end) arch_sync_kernel_mappings(start, end); } =20 +/* + * vunmap_range_noflush is similar to __vunmap_range_noflush_pgd, but works + * only with init_mm->pgd. + * + * This is an internal function only. Do not use outside mm/. + */ +void __vunmap_range_noflush(unsigned long start, unsigned long end) +{ + __vunmap_range_noflush_pgd(init_mm.pgd, start, end); +} + void vunmap_range_noflush(unsigned long start, unsigned long end) { kmsan_vunmap_range_noflush(start, end); @@ -545,6 +563,18 @@ void vunmap_range(unsigned long addr, unsigned long en= d) flush_tlb_kernel_range(addr, end); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +void vunmap_range_replicas(unsigned long addr, unsigned long end) +{ + int nid; + + flush_cache_vunmap(addr, end); + for_each_memory_node(nid) + __vunmap_range_noflush_pgd(init_mm.pgd_numa[nid], addr, end); + flush_tlb_kernel_range(addr, end); +} +#endif + static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) @@ -643,7 +673,8 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned lo= ng addr, return 0; } =20 -static int vmap_small_pages_range_noflush(unsigned long addr, unsigned lon= g end, +static int vmap_small_pages_range_noflush_pgd(pgd_t *pgtable, + unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages) { unsigned long start =3D addr; @@ -654,7 +685,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, pgtbl_mod_mask mask =3D 0; =20 BUG_ON(addr >=3D end); - pgd =3D pgd_offset_k(addr); + pgd =3D pgd_offset_pgd(pgtable, addr); do { next =3D pgd_addr_end(addr, end); if (pgd_bad(*pgd)) @@ -671,7 +702,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, } =20 /* - * vmap_pages_range_noflush is similar to vmap_pages_range, but does not + * __vmap_pages_range_noflush_pgd is similar to vmap_pages_range, but does= not * flush caches. * * The caller is responsible for calling flush_cache_vmap() after this @@ -679,7 +710,7 @@ static int vmap_small_pages_range_noflush(unsigned long= addr, unsigned long end, * * This is an internal function only. Do not use outside mm/. */ -int __vmap_pages_range_noflush(unsigned long addr, unsigned long end, +static int __vmap_pages_range_noflush_pgd(pgd_t *pgtable, unsigned long ad= dr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { unsigned int i, nr =3D (end - addr) >> PAGE_SHIFT; @@ -688,12 +719,13 @@ int __vmap_pages_range_noflush(unsigned long addr, un= signed long end, =20 if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || page_shift =3D=3D PAGE_SHIFT) - return vmap_small_pages_range_noflush(addr, end, prot, pages); + return vmap_small_pages_range_noflush_pgd(pgtable, addr, end, + prot, pages); =20 for (i =3D 0; i < nr; i +=3D 1U << (page_shift - PAGE_SHIFT)) { int err; =20 - err =3D vmap_range_noflush(addr, addr + (1UL << page_shift), + err =3D vmap_range_noflush_pgd(pgtable, addr, addr + (1UL << page_shift), page_to_phys(pages[i]), prot, page_shift); if (err) @@ -705,7 +737,7 @@ int __vmap_pages_range_noflush(unsigned long addr, unsi= gned long end, return 0; } =20 -int vmap_pages_range_noflush(unsigned long addr, unsigned long end, +static int vmap_pages_range_noflush_pgd(pgd_t *pgtable, unsigned long addr= , unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift, gfp_t gfp_mask) { @@ -714,20 +746,35 @@ int vmap_pages_range_noflush(unsigned long addr, unsi= gned long end, =20 if (ret) return ret; - return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + return __vmap_pages_range_noflush_pgd(pgtable, addr, end, prot, pages, pa= ge_shift); } =20 -static int __vmap_pages_range(unsigned long addr, unsigned long end, +int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift, + gfp_t gfp_mask) +{ + return vmap_pages_range_noflush_pgd(init_mm.pgd, addr, end, prot, pages, + page_shift, gfp_mask); +} + +static int __vmap_pages_range_pgd(pgd_t *pgtable, unsigned long addr, unsi= gned long end, pgprot_t prot, struct page **pages, unsigned int page_shift, gfp_t gfp_mask) { int err; =20 - err =3D vmap_pages_range_noflush(addr, end, prot, pages, page_shift, gfp_= mask); + err =3D vmap_pages_range_noflush_pgd(pgtable, addr, end, prot, pages, pag= e_shift, gfp_mask); flush_cache_vmap(addr, end); return err; } =20 +static int __vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift, + gfp_t gfp_mask) +{ + return __vmap_pages_range_pgd(init_mm.pgd, addr, end, prot, pages, page_s= hift, gfp_mask); +} + /** * vmap_pages_range - map pages to a kernel virtual address * @addr: start of the VM area to map @@ -821,57 +868,12 @@ EXPORT_SYMBOL_GPL(is_vmalloc_or_module_addr); */ struct page *vmalloc_to_page(const void *vmalloc_addr) { - unsigned long addr =3D (unsigned long) vmalloc_addr; - struct page *page =3D NULL; - pgd_t *pgd =3D pgd_offset_k(addr); - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep, pte; - /* * XXX we might need to change this if we add VIRTUAL_BUG_ON for * architectures that do not vmalloc module space */ VIRTUAL_BUG_ON(!is_vmalloc_or_module_addr(vmalloc_addr)); - - if (pgd_none(*pgd)) - return NULL; - if (WARN_ON_ONCE(pgd_leaf(*pgd))) - return NULL; /* XXX: no allowance for huge pgd */ - if (WARN_ON_ONCE(pgd_bad(*pgd))) - return NULL; - - p4d =3D p4d_offset(pgd, addr); - if (p4d_none(*p4d)) - return NULL; - if (p4d_leaf(*p4d)) - return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); - if (WARN_ON_ONCE(p4d_bad(*p4d))) - return NULL; - - pud =3D pud_offset(p4d, addr); - if (pud_none(*pud)) - return NULL; - if (pud_leaf(*pud)) - return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - if (WARN_ON_ONCE(pud_bad(*pud))) - return NULL; - - pmd =3D pmd_offset(pud, addr); - if (pmd_none(*pmd)) - return NULL; - if (pmd_leaf(*pmd)) - return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - if (WARN_ON_ONCE(pmd_bad(*pmd))) - return NULL; - - ptep =3D pte_offset_kernel(pmd, addr); - pte =3D ptep_get(ptep); - if (pte_present(pte)) - page =3D pte_page(pte); - - return page; + return walk_to_page_node(first_memory_node, vmalloc_addr); } EXPORT_SYMBOL(vmalloc_to_page); =20 @@ -2490,13 +2492,39 @@ static void free_vmap_area_noflush(struct vmap_area= *va) schedule_work(&drain_vmap_work); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static void vunmap_range_noflush_replicas(unsigned long start, unsigned lo= ng end) +{ + if (numa_addr_has_replica((void *)start)) { + int node; + kmsan_vunmap_range_noflush(start, end); + + /** + * In some scenarios we might clear + * empty entries here, which is totally fine + */ + for_each_memory_node(node) + __vunmap_range_noflush_pgd(per_node_pgd(&init_mm, node), start, end); + } else { + vunmap_range_noflush(start, end); + } +} +#else +static void vunmap_range_noflush_replicas(unsigned long start, unsigned lo= ng end) +{ + vunmap_range_noflush(start, end); +} +#endif + /* * Free and unmap a vmap area */ static void free_unmap_vmap_area(struct vmap_area *va) { flush_cache_vunmap(va->va_start, va->va_end); - vunmap_range_noflush(va->va_start, va->va_end); + + vunmap_range_noflush_replicas(va->va_start, va->va_end); + if (debug_pagealloc_enabled_static()) flush_tlb_kernel_range(va->va_start, va->va_end); =20 @@ -3193,7 +3221,11 @@ void __init vm_area_register_early(struct vm_struct = *vm, size_t align) BUG_ON(vmap_initialized); =20 for (p =3D &vmlist; (cur =3D *p) !=3D NULL; p =3D &cur->next) { - if ((unsigned long)cur->addr - addr >=3D vm->size) + /* Why change this? vm_structs for kerenl text + * might end up outside of vmalloc region. (kernel replcation + nokaslr = case) + * In that case, underflow will happen and normal vm will be added in th= e beginning + */ + if ((unsigned long)cur->addr >=3D addr + vm->size) break; addr =3D ALIGN((unsigned long)cur->addr + cur->size, align); } @@ -3357,17 +3389,74 @@ struct vm_struct *remove_vm_area(const void *addr) return vm; } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static inline void set_direct_map_page_replicas(const struct vm_struct *ar= ea, + struct page *page, + int (*set_direct_map)(struct page *page)) +{ + if (area->replicated) { + struct page *cursor; + + list_for_each_entry(cursor, &page->lru, lru) { + if (page_address(cursor)) + set_direct_map(cursor); + } + } +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static inline void set_area_direct_map(const struct vm_struct *area, int (*set_direct_map)(struct page *page)) { unsigned long i; =20 /* HUGE_VMALLOC passes small pages to set_direct_map */ - for (i =3D 0; i < area->nr_pages; i++) + for (i =3D 0; i < area->nr_pages; i++) { if (page_address(area->pages[i])) set_direct_map(area->pages[i]); +#ifdef CONFIG_KERNEL_REPLICATION + set_direct_map_page_replicas(area, + area->pages[i], set_direct_map); +#endif /* CONFIG_KERNEL_REPLICATION */ + } } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static void vm_account_replicated_range(struct vm_struct *area, + struct page *page, + unsigned long *s, + unsigned long *e, + int *flush) +{ + int flush_dmap =3D 0; + unsigned long start =3D ULONG_MAX, end =3D 0; + unsigned int page_order =3D vm_area_page_order(area); + + if (area->replicated) { + struct page *cursor; + + list_for_each_entry(cursor, &page->lru, lru) { + unsigned long addr =3D (unsigned long)page_address(cursor); + + if (addr) { + unsigned long page_size; + + page_size =3D PAGE_SIZE << page_order; + start =3D min(addr, start); + end =3D max(addr + page_size, end); + flush_dmap =3D 1; + } + } + } + + if (flush_dmap) + *flush =3D flush_dmap; + + *s =3D start; + *e =3D end; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + /* * Flush the vm mapping and reset the direct map. */ @@ -3393,6 +3482,10 @@ static void vm_reset_perms(struct vm_struct *area) end =3D max(addr + page_size, end); flush_dmap =3D 1; } +#ifdef CONFIG_KERNEL_REPLICATION + vm_account_replicated_range(area, area->pages[i], + &start, &end, &flush_dmap); +#endif /* CONFIG_KERNEL_REPLICATION */ } =20 /* @@ -3438,6 +3531,30 @@ void vfree_atomic(const void *addr) schedule_work(&p->wq); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static void vfree_page_replicas(struct vm_struct *area, struct page *page) +{ + if (area->replicated) { + struct page *cursor, *tmp; + + list_for_each_entry_safe(cursor, tmp, &page->lru, lru) { + BUG_ON(!cursor); + + list_del(&cursor->lru); + mod_lruvec_page_state(cursor, NR_VMALLOC, -1); + /* + * High-order allocs for huge vmallocs are split, so + * can be freed as an array of order-0 allocations + */ + __free_pages(cursor, 0); + cond_resched(); + } + } +} +#else +static void vfree_page_replicas(struct vm_struct *area, struct page *page)= { } +#endif /* CONFIG_KERNEL_REPLICATION */ + /* * vm_area_free_pages - free a range of pages from a vmalloc allocation * @vm: the vm_struct containing the pages @@ -3455,8 +3572,10 @@ static void vm_area_free_pages(struct vm_struct *vm,= unsigned long start_idx, unsigned long i; =20 if (!(vm->flags & VM_MAP_PUT_PAGES)) { - for (i =3D start_idx; i < end_idx; i++) + for (i =3D start_idx; i < end_idx; i++) { + vfree_page_replicas(vm, vm->pages[i]); mod_lruvec_page_state(vm->pages[i], NR_VMALLOC, -1); + } } free_pages_bulk(vm->pages + start_idx, end_idx - start_idx); =20 @@ -3862,6 +3981,58 @@ memalloc_restore_scope(unsigned int flags) memalloc_flags_restore(flags); } =20 +static int __vmap_pages_range_retry(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift, + gfp_t gfp_mask, bool nofail) +{ + int ret; + + do { + ret =3D __vmap_pages_range(addr, end, prot, pages, + page_shift, gfp_mask); + if (nofail && (ret < 0)) + schedule_timeout_uninterruptible(1); + } while (nofail && (ret < 0)); + + return ret; +} + +#ifdef CONFIG_KERNEL_REPLICATION +static int __vmap_pages_range_retry_replica_aware(unsigned long addr, unsi= gned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift, + gfp_t gfp_mask, bool nofail, unsigned long flags) +{ + int ret; + + if (flags & VM_NUMA_SHARED) { + int nid; + + ret =3D kmsan_vmap_pages_range_noflush(addr, end, prot, pages, + page_shift, gfp_mask); + for_each_memory_node(nid) { + pgd_t *pgd =3D per_node_pgd(&init_mm, nid); + + do { + ret =3D __vmap_pages_range_noflush_pgd(pgd, addr, end, prot, pages, + page_shift); + if (nofail && (ret < 0)) + schedule_timeout_uninterruptible(1); + } while (nofail && (ret < 0)); + + if (ret < 0) + goto out; + } + + flush_cache_vmap(addr, end); + } else { + ret =3D __vmap_pages_range_retry(addr, end, prot, pages, + page_shift, gfp_mask, nofail); + } +out: + return ret; +} +#endif + static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, pgprot_t prot, unsigned int page_shift, int node) @@ -3942,12 +4113,14 @@ static void *__vmalloc_area_node(struct vm_struct *= area, gfp_t gfp_mask, * by the scope API */ flags =3D memalloc_apply_gfp_scope(gfp_mask); - do { - ret =3D __vmap_pages_range(addr, addr + size, prot, area->pages, - page_shift, nested_gfp); - if (nofail && (ret < 0)) - schedule_timeout_uninterruptible(1); - } while (nofail && (ret < 0)); + +#ifdef CONFIG_KERNEL_REPLICATION + ret =3D __vmap_pages_range_retry_replica_aware(addr, addr + size, prot, a= rea->pages, + page_shift, nested_gfp, nofail, area->flags); +#else + ret =3D __vmap_pages_range_retry(addr, addr + size, prot, area->pages, + page_shift, nested_gfp, nofail); +#endif memalloc_restore_scope(flags); =20 if (ret < 0) { @@ -4076,6 +4249,16 @@ void *__vmalloc_node_range_noprof(unsigned long size= , unsigned long align, goto fail; } =20 +#ifdef CONFIG_KERNEL_REPLICATION + /* Check, that for requested region correct replication flag was passed */ + if (WARN_ON_ONCE(numa_addr_has_replica(area->addr) && !(vm_flags & VM_NUM= A_SHARED))) + vm_flags |=3D VM_NUMA_SHARED; + + if (WARN_ON_ONCE(!numa_addr_has_replica(area->addr) && (vm_flags & VM_NUM= A_SHARED))) + vm_flags &=3D ~VM_NUMA_SHARED; + + area->node =3D node; +#endif /* * Prepare arguments for __vmalloc_area_node() and * kasan_unpoison_vmalloc(). @@ -4165,6 +4348,129 @@ void *__vmalloc_node_noprof(unsigned long size, uns= igned long align, return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_EN= D, gfp_mask, PAGE_KERNEL, 0, node, caller); } + +#ifdef CONFIG_KERNEL_REPLICATION +static void numa_replicate_page_range(struct page **src, struct page **dst= , int nr_pages) +{ + int i; + void *from, *to; + + for (i =3D 0; i < nr_pages; i++) { + from =3D kmap_local_page(src[i]); + to =3D kmap_local_page(dst[i]); + + copy_page(to, from); + + kunmap_local(to); + kunmap_local(from); + } +} + +int __vmalloc_node_replicate_range(const void *addr, gfp_t gfp_mask, + pgprot_t prot, unsigned long vm_flags) +{ + int i, ret, node =3D 0; + struct vm_struct *area; + unsigned int page_order; + unsigned int nr_allocated; + struct page **pages; + unsigned long area_start, area_end; + const gfp_t nested_gfp =3D (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; + unsigned long array_size; + unsigned int flags; + + + gfp_mask |=3D __GFP_NOWARN; + if (!(gfp_mask & (GFP_DMA | GFP_DMA32))) + gfp_mask |=3D __GFP_HIGHMEM; + + if (unlikely(!numa_addr_has_replica(addr))) + return -EINVAL; + + area =3D find_vm_area(addr); + if (unlikely(!area)) + return -ENOENT; + + if (area->node =3D=3D NUMA_NO_NODE) + return -EINVAL; + + array_size =3D sizeof(struct page *) * area->nr_pages; + if (array_size > PAGE_SIZE) + pages =3D __vmalloc(array_size, nested_gfp); + else + pages =3D kmalloc(array_size, nested_gfp); + + if (!pages) + return -ENOMEM; + + page_order =3D vm_area_page_order(area); + for (i =3D 0; i < area->nr_pages; i++) + INIT_LIST_HEAD(&area->pages[i]->lru); + + area_start =3D (unsigned long)area->addr; + area_end =3D (unsigned long)(area->addr + area->nr_pages * PAGE_SIZE); + + for_each_memory_node(node) { + if (area->node =3D=3D node) + continue; + + nr_allocated =3D vm_area_alloc_pages(gfp_mask | __GFP_NOWARN, + node, page_order, area->nr_pages, pages); + if (nr_allocated !=3D area->nr_pages) + goto fail_alloc_pages; + + for (i =3D 0; i < area->nr_pages; i++) + list_add(&pages[i]->lru, &area->pages[i]->lru); + + __vunmap_range_noflush_pgd(init_mm.pgd_numa[node], + area_start, area_end); + + /* + * We can't fail here (hopefully) + * Possible errors: not enough memory for tables and not empty entries. + * Both unrealistic because we just cleared entries in existed tables. + */ + + numa_replicate_page_range(area->pages, pages, area->nr_pages); + + flags =3D memalloc_apply_gfp_scope(gfp_mask); + + + ret =3D __vmap_pages_range_noflush_pgd(per_node_pgd(&init_mm, node), are= a_start, + area_end, prot, area->pages, page_order + PAGE_SHIFT); + + memalloc_restore_scope(flags); + if (ret !=3D 0) + goto fail_map_pages; + + for (i =3D 0; i < area->nr_pages; i++) + pages[i] =3D NULL; + } + kvfree(pages); + flush_cache_vmap(area_start, area_end); + flush_tlb_kernel_range(area_start, area_end); + area->replicated =3D true; + + return 0; +fail_alloc_pages: + for (i =3D 0; i < nr_allocated; i++) + __free_pages(pages[i], 0); + +fail_map_pages: + kfree(pages); + for (i =3D 0; i < area->nr_pages; i++) { + struct page *page, *tmp; + + list_for_each_entry_safe(page, tmp, &area->pages[i]->lru, lru) { + list_del(&page->lru); + __free_pages(page, 0); + } + } + + return ret; +} +#endif /* CONFIG_KERNEL_REPLICATION */ + /* * This is only for performance analysis of vmalloc and stress purpose. * It is required by vmalloc test module, therefore do not use it other --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A42A148EBDF for ; Thu, 27 Aug 2026 16:24:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847894; cv=none; b=h9/RuBqI23EXYXstSVAE+gKmB717fFWyiJYmVfJCCXpmLEHyAvPNNvTyJoeh3XC+3PYiMm8eyC4X/yIEL1mH2RfIF+vPWIr9e1jdnoPBzFbmFt+lrJfnKQvqYchRkMiTlmVw1xqA239j++xXBTnmdh6ZR9S6OBqw7FZbDlkuvRg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847894; c=relaxed/simple; bh=OLZ3zWUKxXdQK5KLK8apCMbNQiforjv+K2b+8gGYLJM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=EMTALPfq2+W638ejPFYoOc4zP8/gWumte+3/SaMbQh2jCBgm1L0G3q3iDhHeOu/i/3xNHX6lywb/zx8oxrFdZnx6DdX2qQFJCUGIGVkT3Y6mcOD3M8hnjSpCQ/cmld+spQ7z9P69wFwpYPnUdtNeqCRNO9QVHcQ3EajooHQeUb0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=F63kYZZR; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="F63kYZZR" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=gzG5ol8Yr6RKYwTAU+BAIanpIkNMA3tETIj6mvgS0RQ=; b=F63kYZZRaWckjPUhtZOR2u+9xVDb4ayB+XJ7djeLCD8+HZj9Ituii5PFkTq09FW8YQWa/kROT 427t61dwdHdiaK2yyzTeCLrtvj4uBWR5I3vJ1UmSPJ72M17crIgWvQ4lDFSBleLV0vCWWAvY/PI IwFkSVRGhZ2SPjPU+HD5Af8= Received: from mail.maildlp.com (unknown [172.18.224.107]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KW4D2GzJ46Yq; Fri, 28 Aug 2026 00:24:15 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 25DEC4058D; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 10/18] arm64: enable per-NUMA node kernel text and rodata replication Date: Fri, 28 Aug 2026 00:11:50 +0800 Message-ID: <20260827161158.3618409-11-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" During boot memory for replicas is allocated, local translation tables are created, original text and rodata are copied to replicas, and replicas are mapped to local tables. On startup of the secondary CPUs, after minimal initialization, the local pgtable is loaded to ttbr1. Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/include/asm/pgtable.h | 4 +++ arch/arm64/kernel/smp.c | 8 ++++++ arch/arm64/mm/context.c | 1 + arch/arm64/mm/init.c | 49 ++++++++++++++++++++++++++++++++ arch/arm64/mm/kasan_init.c | 2 ++ arch/arm64/mm/mmu.c | 38 ++++++++++++++++++++++++- 6 files changed, 101 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 6000905a2e86..f11d77829890 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -20,7 +20,11 @@ * VMALLOC_START: beginning of the kernel vmalloc space * VMALLOC_END: extends to the available space below vmemmap */ +#ifdef CONFIG_KERNEL_REPLICATION +#define VMALLOC_START ((MODULES_END & PGDIR_MASK) + PGDIR_SIZE) +#else /* !CONFIG_KERNEL_REPLICATION */ #define VMALLOC_START (MODULES_END) +#endif /* CONFIG_KERNEL_REPLICATION */ #if VA_BITS =3D=3D VA_BITS_MIN #define VMALLOC_END (VMEMMAP_START - SZ_8M) #else diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index a61dc3016a11..4832a89ba992 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -36,6 +36,7 @@ #include #include #include +#include =20 #include #include @@ -208,6 +209,13 @@ asmlinkage notrace void secondary_start_kernel(void) mmgrab(mm); current->active_mm =3D mm; =20 + /* + * Setup per-NUMA node page table if kernel + * replication is enabled. Option supported + * only for 64-bit mode. + */ + numa_setup_pgd(); + /* * TTBR0 is only used for the identity mapping at this stage. Make it * point to zero page to avoid speculatively fetching new entries. diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 0f4a28b87469..3afdae62784e 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -11,6 +11,7 @@ #include #include #include +#include =20 #include #include diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index fbf215ecc7d0..3c2394b20c03 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -331,6 +331,47 @@ void __init bootmem_init(void) memblock_dump_all(); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +/* + * It is necessary to preallocate vmalloc pages in advance, + * otherwise the replicated page-tables can be incomplete. + */ +void __init preallocate_vmalloc_pages(void) +{ + unsigned long addr; + + for (addr =3D MODULES_VADDR; addr <=3D VMALLOC_END && addr !=3D 0UL; + addr =3D ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd =3D pgd_offset_k(addr); + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + int pte; + + p4d =3D p4d_alloc(&init_mm, pgd, addr); + /* + * No need to check p4d here due to + * only 4-stage page table is possible + */ + pud =3D pud_alloc(&init_mm, p4d, addr); + if (!pud) + panic("Failed to pre-allocate pud pages for vmalloc area\n"); + if (!mm_pud_folded(&init_mm)) + continue; + + pmd =3D pmd_alloc(&init_mm, pud, addr); + if (!pmd) + panic("Failed to pre-allocate pmd pages for vmalloc area\n"); + if (!mm_pmd_folded(&init_mm)) + continue; + + pte =3D pte_alloc(&init_mm, pmd); + if (pte) + panic("Failed to pre-allocate pte pages for vmalloc area\n"); + } +} +#endif /* CONFIG_KERNEL_REPLICATION */ + void __init arch_setup_zero_pages(void) { __zero_page =3D phys_to_page(__pa_symbol(empty_zero_page)); @@ -401,7 +442,15 @@ void free_initmem(void) * prevents the region from being reused for kernel modules, which * is not supported by kallsyms. */ +#ifdef CONFIG_KERNEL_REPLICATION + /* + * In case of replicated kernel the per-NUMA node vmalloc + * memory should be released. + */ + vunmap_range_replicas((u64)__init_begin, (u64)__init_end); +#else vunmap_range((u64)__init_begin, (u64)__init_end); +#endif /* CONFIG_KERNEL_REPLICATION */ } =20 void dump_mem_limit(void) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 45fbdce684c8..37eb408830f0 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -345,7 +345,9 @@ static void __init kasan_init_shadow(void) kasan_populate_early_shadow(kasan_mem_to_shadow((void *)PAGE_END), (void *)mod_shadow_start); =20 +#ifndef CONFIG_KERNEL_REPLICATION BUILD_BUG_ON(VMALLOC_START !=3D MODULES_END); +#endif kasan_populate_early_shadow((void *)vmalloc_shadow_end, (void *)KASAN_SHADOW_END); =20 diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 79d90226fd5d..fa12461c0d8e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -30,6 +30,7 @@ #include #include #include +#include =20 #include #include @@ -1039,6 +1040,22 @@ void __init create_pgd_mapping(struct mm_struct *mm,= phys_addr_t phys, pgd_pgtable_alloc_special_mm, flags); } =20 +static void populate_mappings_prot(phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot) +{ +#ifdef CONFIG_KERNEL_REPLICATION + int nid; + + for_each_memory_node(nid) { + early_create_pgd_mapping(per_node_pgd(&init_mm, nid), + page_to_phys(walk_to_page_node(nid, (void *)virt)), + virt, size, prot, NULL, 0); + } +#else + early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0); +#endif /* CONFIG_KERNEL_REPLICATION */ +} + static void update_mapping_prot(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) { @@ -1048,7 +1065,7 @@ static void update_mapping_prot(phys_addr_t phys, uns= igned long virt, return; } =20 - early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0); + populate_mappings_prot(phys, virt, size, prot); =20 /* flush the TLBs after updating live kernel mappings */ flush_tlb_kernel_range(virt, virt + size); @@ -1390,6 +1407,21 @@ static pgprot_t __init kernel_exec_prot(void) return rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static void __init populate_trampoline_mappings(void) +{ + int nid; + + /* Copy trampoline mappings in replicated tables */ + for_each_memory_node(nid) { + memcpy(per_node_pgd(&init_mm, nid) - (PAGE_SIZE * 2 / sizeof(pgd_t)), + tramp_pg_dir, PGD_SIZE); + } + /* Be sure that replicated page table can be observed properly */ + dsb(ishst); +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static int __init map_entry_trampoline(void) { int i; @@ -1418,6 +1450,10 @@ static int __init map_entry_trampoline(void) __set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i, pa_start + i * PAGE_SIZE, PAGE_KERNEL_RO); =20 +#ifdef CONFIG_KERNEL_REPLICATION + populate_trampoline_mappings(); +#endif /* CONFIG_KERNEL_REPLICATION */ + return 0; } core_initcall(map_entry_trampoline); --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE06C486624 for ; Thu, 27 Aug 2026 16:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; cv=none; b=hBxWlsblcQ2gLLfB3MNFSHZVrxd7V/Y1g7vEd4ngetOK04Y0mEVXhMCSou8kQ9Hm8DgpoVJBM2DXfZu07OEsuc+9BMZdG0hF5cAAz22E9CKevRXHga7c6/7aV1fAc6gmyhX7ps2LaV8EmKiXyKvIAZ8dWCtN/upHZXQaOluOBUk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847890; c=relaxed/simple; bh=Fucb+sl9nfCsiqQfaI/j3sVJ9rSyNnuwL/cuXtatQmY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=a0+XZRGAXm6rHLqBydq/n25OJsKPwgKWNzdRGxkE44wOAD1D5lHfC1/8lw203Mq6ZQJSwvtfPzySG/CwqvP3c8PCZaN0yWmTosyNCpPDs2CYrAUeG97h4LRzzU5wANRvjr4h6SSEyGfbfqg+OiTSu0LaAFAzBuvIiEJwsayWBfs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=wtbL8sbN; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="wtbL8sbN" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=uOTDgBkjYJFHvNQbfBQWTD/6fFe89tZAtGOjlb0y990=; b=wtbL8sbN6tNfhYL36YDbdr/ivXINVqVkk90pUmq92rN8cLBL20/oYfcOC+UfURGULymSn/Yf1 u0XqUbwXc5kYu2UuEba9a35jtyCNeRQF9KydgYEbxyPLIDONF88jLBvoo555mEitZA4/pudQ3PI /DPolqoyPJU9gC8IkMbs54k= Received: from mail.maildlp.com (unknown [172.18.224.107]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KF1XKkzHnH5S; Fri, 28 Aug 2026 00:24:01 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 3ACEA4058E; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:36 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 11/18] mm: enable per-NUMA node kernel text and rodata replication Date: Fri, 28 Aug 2026 00:11:51 +0800 Message-ID: <20260827161158.3618409-12-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/mm/context.c | 1 - arch/arm64/mm/mmu.c | 8 ++++++-- include/linux/mm.h | 3 +++ init/main.c | 17 +++++++++++++++++ mm/mm_init.c | 3 +++ 5 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 3afdae62784e..0f4a28b87469 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -11,7 +11,6 @@ #include #include #include -#include =20 #include #include diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index fa12461c0d8e..4bc8b748e303 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1040,10 +1040,10 @@ void __init create_pgd_mapping(struct mm_struct *mm= , phys_addr_t phys, pgd_pgtable_alloc_special_mm, flags); } =20 +#ifdef CONFIG_KERNEL_REPLICATION static void populate_mappings_prot(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) { -#ifdef CONFIG_KERNEL_REPLICATION int nid; =20 for_each_memory_node(nid) { @@ -1051,10 +1051,14 @@ static void populate_mappings_prot(phys_addr_t phys= , unsigned long virt, page_to_phys(walk_to_page_node(nid, (void *)virt)), virt, size, prot, NULL, 0); } +} #else +static void populate_mappings_prot(phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot) +{ early_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0); -#endif /* CONFIG_KERNEL_REPLICATION */ } +#endif /* CONFIG_KERNEL_REPLICATION */ =20 static void update_mapping_prot(phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8a7083c02203..0f3592038823 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4240,6 +4240,9 @@ extern int __meminit early_pfn_to_nid(unsigned long p= fn); #endif =20 extern void mem_init(void); +#ifdef CONFIG_KERNEL_REPLICATION +extern void preallocate_vmalloc_pages(void); +#endif extern void __init mmap_init(void); =20 extern void __show_mem(unsigned int flags, const nodemask_t *nodemask, int= max_zone_idx); diff --git a/init/main.c b/init/main.c index d9d936707c19..249307e46818 100644 --- a/init/main.c +++ b/init/main.c @@ -109,6 +109,7 @@ #include #include #include +#include #include =20 #include @@ -1038,12 +1039,20 @@ void start_kernel(void) * These use large bootmem allocations and must precede * initalization of page allocator */ + numa_replication_init(); setup_log_buf(0); vfs_caches_init_early(); sort_main_extable(); trap_init(); mm_core_init(); maple_tree_init(); + /* + * Kernel text replication should be done before + * alloc/init first mm struct, due to it is necessary + * to setup per-NUMA node translation tables and kernel + * instances properly. + */ + numa_replicate_kernel_text(); poking_init(); ftrace_init(); =20 @@ -1571,6 +1580,14 @@ static int __ref kernel_init(void *unused) free_initmem(); mark_readonly(); =20 + /* + * RODATA replication is done here due to + * it is necessary to finalize the kernel + * and modules initialization before + */ + numa_replicate_kernel_rodata(); + numa_replication_fini(); + /* * Kernel mappings are now finalized - update the userspace page-table * to finalize PTI. diff --git a/mm/mm_init.c b/mm/mm_init.c index 1533aebafb68..d2b13d8887c8 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2679,6 +2679,9 @@ void __init mm_core_init(void) ptlock_cache_init(); pgtable_cache_init(); debug_objects_mem_init(); +#ifdef CONFIG_KERNEL_REPLICATION + preallocate_vmalloc_pages(); +#endif vmalloc_init(); /* If no deferred init page_ext now, as vmap is fully initialized */ if (!deferred_struct_pages) --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A17A48D89B for ; Thu, 27 Aug 2026 16:24:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847894; cv=none; b=ExoLo++KPc/0wfT1Fdh9FrppP8tn68bkzoDk1CfQeDgaCw9f8gwGxbBytvLQLUUFUf2M70s/hdSxJsSqZpKS/Ezc7d06fh3b/BsxFI6VMhPLj5S3ZOIBlb8U8tBiduES1ywu2Dj4Sc6LKPc9hFrhzSUYwmp34sy0IzoxcpGW4y4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847894; c=relaxed/simple; bh=5REAQAlkVUwymRdU7RMs3rA+NWvSBeETX2y/IIFVBpQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Vm3L+iNPoYhUBp5adX0yYHPyYFOrABVi6Kwq6L80SZVSR8mGzXwxUXF+//6JGDZy+vk/oVbE4nfef49Y7jEMyqm497RNbsqt/hAJM3S8J4WLgpGI0yH1TnWy81uMpH6pZcC6tfC4znxhUhDDB7ctuq2iu5rXb9DDPqvtX1T6bgY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=CFxplSWM; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="CFxplSWM" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=MKq7Nh2gcYohgYrYR5axiDJ4p1s9We/3CeTSchjvn7A=; b=CFxplSWMhRpP8AAbiY91FEFGTrBwzp0iMlr33Qf/OYA54NMunuxvtLsKNFYazG/V7FN0tYbCW qVnSK4U881PmSGi5eEBaNy7Ih6oUsQdS4opo6Pbu87KJhzl/SLUUsws+oXRpCITmnkyZjeHgHeG yehCK3Lv097Ecr7CG8SvsTI= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KX0GPXzJ46Yq; Fri, 28 Aug 2026 00:24:16 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 92A1F40570; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:37 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 12/18] arm64: make power management aware about kernel replication Date: Fri, 28 Aug 2026 00:11:52 +0800 Message-ID: <20260827161158.3618409-13-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/include/asm/mmu_context.h | 4 ++++ arch/arm64/kernel/hibernate.c | 5 +++++ arch/arm64/kernel/sleep.S | 8 ++++++++ mm/numa_kernel_replication.c | 5 +++++ 4 files changed, 22 insertions(+) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/= mmu_context.h index 803b68758152..f364281d3442 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -136,10 +136,14 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr= 0, unsigned long t0sz) =20 void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp); =20 +#ifndef CONFIG_KERNEL_REPLICATION static inline void cpu_enable_swapper_cnp(void) { __cpu_replace_ttbr1(lm_alias(swapper_pg_dir), true); } +#else +void cpu_enable_swapper_cnp(void); +#endif =20 static inline void cpu_replace_ttbr1(pgd_t *pgdp) { diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 7bf117427777..e9f5ddf37c4a 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -16,6 +16,7 @@ #include #include #include +#include =20 #include #include @@ -113,7 +114,11 @@ int arch_hibernation_header_save(void *addr, unsigned = int max_size) return -EOVERFLOW; =20 arch_hdr_invariants(&hdr->invariants); +#ifdef CONFIG_KERNEL_REPLICATION + hdr->ttbr1_el1 =3D virt_to_phys(this_node_pgd(&init_mm)); +#else hdr->ttbr1_el1 =3D __pa_symbol(swapper_pg_dir); +#endif /* CONFIG_KERNEL_REPLICATION */ hdr->reenter_kernel =3D _cpu_resume; =20 /* We can't use __hyp_get_vectors() because kvm may still be loaded */ diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index f093cdf71be1..8ad66ed70122 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -5,6 +5,10 @@ #include #include =20 +#ifdef CONFIG_KERNEL_REPLICATION +.extern numa_setup_pgd +#endif + .text /* * Implementation of MPIDR_EL1 hash algorithm through shifting @@ -144,6 +148,10 @@ SYM_FUNC_START(_cpu_resume) bl kasan_unpoison_task_stack_below #endif =20 +#ifdef CONFIG_KERNEL_REPLICATION + bl numa_setup_pgd +#endif + ldp x19, x20, [x29, #16] ldp x21, x22, [x29, #32] ldp x23, x24, [x29, #48] diff --git a/mm/numa_kernel_replication.c b/mm/numa_kernel_replication.c index 082aabc6b8bc..b1c5cd2a1fa1 100644 --- a/mm/numa_kernel_replication.c +++ b/mm/numa_kernel_replication.c @@ -67,6 +67,11 @@ static int node_to_memory_node[MAX_NUMNODES]; static bool pgtables_extra; static DEFINE_SPINLOCK(debugfs_lock); =20 +void cpu_enable_swapper_cnp(void) +{ + __cpu_replace_ttbr1(this_node_pgd(&init_mm), true); +} + bool is_text_replicated(void) { return text_replicated; --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D20AB48F821 for ; Thu, 27 Aug 2026 16:24:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; cv=none; b=IkWLpu9Fg7HToHaGerDI6KNtBks1V26rfeuPFQozqWi3THpDms9z5lA4aSH13X79CKjSYZFRapT+FBEArRA/ge6f3TtYTyYqa04dWcCk4kB0YG0KayQqDZcA4PRfGPvC5ztXtwkVACVfjGFAgkrqLhcte1RF7+mJdds6O+e8NH8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; c=relaxed/simple; bh=BGWZvSQWJSKKeNHeHVkuEinhxork/JYU52fkZCLmHGU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Jmm4aQQWToiJx3sUrCDL53XGOhB5sefFLlHuHouIpvf2ao8D8nmBiKSX5WA4lfvza69UaKrxHPAL2UYasyP9inCZLghsjwh1dccSYApbz+wUnGeK2c2+Sj9/tEEQ5HGj9p7zvK8Ld1JvTvcQaZ6+QBubj7rArrUkJ2LgaJaaigI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=xLmUHxdt; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="xLmUHxdt" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=xAGNWJtbQb3AJ1HJQX61n/LHCCEcBpCaICJCBWpxqGI=; b=xLmUHxdtznvn14bKKb3Ae8Py9BGbsGZqW/mdynm2ysF18alkLLyUC/mTjbs9FM+IA4EMsY9Eg zCsPR8FkLdAC+gJ2+MTxLuwMlQQc1uJIXoR8zHE4WD/zGUWFsZXsiVf3Rr2lo/hmf4hDucR9LZt dUOIyHkNGI1TeXQ0Xw9IW6M= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KF4jbczHnH3h; Fri, 28 Aug 2026 00:24:01 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id A74DA40572; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:37 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 13/18] arm64: make kernel text patching aware about replicas Date: Fri, 28 Aug 2026 00:11:53 +0800 Message-ID: <20260827161158.3618409-14-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/kernel/alternative.c | 33 +++++++++++- arch/arm64/kernel/patching.c | 96 +++++++++++++++++++++++++++++++++ 2 files changed, 128 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternativ= e.c index f5ec7e7c1d3f..e85644d64bbd 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -10,6 +10,7 @@ =20 #include #include +#include #include #include #include @@ -139,6 +140,36 @@ static noinstr void clean_dcache_range_nopatch(u64 sta= rt, u64 end) } while (cur +=3D d_size, cur < end); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static void __write_alternatives(struct alt_instr *alt, + alternative_cb_t alt_cb, + __le32 *origptr, __le32 *updptr, + int nr_inst) +{ + if (is_text_replicated() && is_kernel_text((unsigned long)origptr)) { + int nid; + + for_each_memory_node(nid) { + __le32 *ptr =3D numa_get_replica(origptr, nid); + + alt_cb(alt, origptr, ptr, nr_inst); + clean_dcache_range_nopatch((u64)ptr, + (u64)(ptr + nr_inst)); + } + } else { + alt_cb(alt, origptr, updptr, nr_inst); + } +} +#else +static void __write_alternatives(struct alt_instr *alt, + alternative_cb_t alt_cb, + __le32 *origptr, __le32 *updptr, + int nr_inst) +{ + alt_cb(alt, origptr, updptr, nr_inst); +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static int __apply_alternatives(const struct alt_region *region, bool is_module, unsigned long *cpucap_mask) @@ -174,7 +205,7 @@ static int __apply_alternatives(const struct alt_region= *region, alt_cb =3D patch_alternative; } =20 - alt_cb(alt, origptr, updptr, nr_inst); + __write_alternatives(alt, alt_cb, origptr, updptr, nr_inst); =20 if (!is_module) { clean_dcache_range_nopatch((u64)origptr, diff --git a/arch/arm64/kernel/patching.c b/arch/arm64/kernel/patching.c index 09f019c6547a..8bac14198459 100644 --- a/arch/arm64/kernel/patching.c +++ b/arch/arm64/kernel/patching.c @@ -5,6 +5,7 @@ #include #include #include +#include =20 #include #include @@ -15,6 +16,7 @@ =20 static DEFINE_RAW_SPINLOCK(patch_lock); =20 +#ifndef CONFIG_KERNEL_REPLICATION static bool is_exit_text(unsigned long addr) { /* discarded with init text/data */ @@ -42,6 +44,17 @@ static void __kprobes *patch_map(void *addr, int fixmap) =20 return (void *)set_fixmap_offset(fixmap, phys); } +#else +static void __kprobes *patch_map(void *addr, int fixmap, int nid) +{ + struct page *page; + + page =3D walk_to_page_node(nid, addr); + BUG_ON(!page); + + return (void *)set_fixmap_offset(fixmap, page_to_phys(page) + offset_in_p= age(addr)); +} +#endif /* CONFIG_KERNEL_REPLICATION */ =20 static void __kprobes patch_unmap(int fixmap) { @@ -63,6 +76,28 @@ int __kprobes aarch64_insn_read(void *addr, u32 *insnp) return ret; } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static int __kprobes __aarch64_insn_write(void *addr, __le32 insn) +{ + int nid; + void *waddr =3D addr; + unsigned long flags =3D 0; + int ret; + + raw_spin_lock_irqsave(&patch_lock, flags); + for_each_memory_node(nid) { + waddr =3D patch_map(addr, FIX_TEXT_POKE0, nid); + ret =3D copy_to_kernel_nofault(waddr, &insn, AARCH64_INSN_SIZE); + patch_unmap(FIX_TEXT_POKE0); + + if (ret || !is_text_replicated()) + break; + } + raw_spin_unlock_irqrestore(&patch_lock, flags); + + return ret; +} +#else static int __kprobes __aarch64_insn_write(void *addr, __le32 insn) { void *waddr =3D addr; @@ -79,12 +114,37 @@ static int __kprobes __aarch64_insn_write(void *addr, = __le32 insn) =20 return ret; } +#endif /* CONFIG_KERNEL_REPLICATION */ =20 int __kprobes aarch64_insn_write(void *addr, u32 insn) { return __aarch64_insn_write(addr, cpu_to_le32(insn)); } =20 +#ifdef CONFIG_KERNEL_REPLICATION +noinstr int aarch64_insn_write_literal_u64(void *addr, u64 val) +{ + int nid; + u64 *waddr; + unsigned long flags; + int ret; + + raw_spin_lock_irqsave(&patch_lock, flags); + for_each_memory_node(nid) { + waddr =3D patch_map(addr, FIX_TEXT_POKE0, nid); + + ret =3D copy_to_kernel_nofault(waddr, &val, sizeof(val)); + + patch_unmap(FIX_TEXT_POKE0); + + if (ret || !is_text_replicated()) + break; + } + raw_spin_unlock_irqrestore(&patch_lock, flags); + + return ret; +} +#else noinstr int aarch64_insn_write_literal_u64(void *addr, u64 val) { u64 *waddr; @@ -101,9 +161,44 @@ noinstr int aarch64_insn_write_literal_u64(void *addr,= u64 val) =20 return ret; } +#endif /* CONFIG_KERNEL_REPLICATION */ =20 typedef void text_poke_f(void *dst, void *src, size_t patched, size_t len); =20 +#ifdef CONFIG_KERNEL_REPLICATION +static void *__text_poke(text_poke_f func, void *addr, void *src, size_t l= en) +{ + unsigned long flags; + size_t patched =3D 0; + size_t size; + void *waddr; + void *ptr; + + raw_spin_lock_irqsave(&patch_lock, flags); + + while (patched < len) { + int nid; + + ptr =3D addr + patched; + size =3D min(PAGE_SIZE - offset_in_page(ptr), len - patched); + + for_each_memory_node(nid) { + waddr =3D patch_map(ptr, FIX_TEXT_POKE0, nid); + func(waddr, src, patched, size); + patch_unmap(FIX_TEXT_POKE0); + + if (!is_text_replicated()) + break; + } + patched +=3D size; + } + raw_spin_unlock_irqrestore(&patch_lock, flags); + + flush_icache_range((uintptr_t)addr, (uintptr_t)addr + len); + + return addr; +} +#else static void *__text_poke(text_poke_f func, void *addr, void *src, size_t l= en) { unsigned long flags; @@ -130,6 +225,7 @@ static void *__text_poke(text_poke_f func, void *addr, = void *src, size_t len) =20 return addr; } +#endif =20 static void text_poke_memcpy(void *dst, void *src, size_t patched, size_t = len) { --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24CD948F839 for ; Thu, 27 Aug 2026 16:24:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; cv=none; b=Ub4agkd05Q8ZA0lPsaNb+pfTBtrj0PAZsG169knG2AJlLXkUYVZn4RyP/t6qwzUeolvMQp+ZSYhnP8fZCTjkiW+jLEjitpISu/Ts/KEGUAIAXodjrMXdCe3pcsB38trTkSm/BB3jUvL9ZSPxFK0HRqst0tAfgZoMvjVJlpJZrAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; c=relaxed/simple; bh=SxoyZOswqKGCE5Si5I0REoVrym50xz6jU8J8fzcZOfs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TUWlwECjRVHlHm6A2QlZlWvtrVwwbMUqqMtNNMEPy2B0QbLX9s2o7utbktCCmjVoPpXMmXxWrY0txmQ2FX2fdpgm4sVZaxsmay15DPTFAa9Fv9uoDifQmDKng0FU9TG9F83qbTCjZHRQVFV11OZwI9KR+3f2TVY+tSIf1Qq8n3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=To0G6wcE; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="To0G6wcE" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=ftKn+49W8yK1HAHXVqh6PcSXmf+Z7/mnV4iNORjKKA4=; b=To0G6wcEbxhMZWTK5DME5yHn/K70A7erPo6Vqiwugi/mKREl4Ejl8JHdOMkCXABbkqafCNh4j ECKo1G8HoT75N7J7N+yHuE5w1okfaAs4WC6+f/fxaPQwgHlmeNsruHPfm7iqHhU5lc4FldADO1v FZMO8OtdNCCJ+DMl+6mVL8w= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KX1Sd7zJ46Yq; Fri, 28 Aug 2026 00:24:16 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id BBD2540570; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:37 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 14/18] arm64: add correct alignment to kimage in efi code Date: Fri, 28 Aug 2026 00:11:54 +0800 Message-ID: <20260827161158.3618409-15-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/include/asm/efi.h | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index e8a9783235cb..08ec4d8b48cd 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -87,8 +87,6 @@ static inline unsigned long efi_get_max_initrd_addr(unsig= ned long image_addr) =20 static inline unsigned long efi_get_kimg_min_align(void) { - extern bool efi_nokaslr; - /* * Although relocatable kernels can fix up the misalignment with * respect to MIN_KIMG_ALIGN, the resulting virtual text addresses are @@ -97,7 +95,23 @@ static inline unsigned long efi_get_kimg_min_align(void) * 2M alignment if KASLR was explicitly disabled, even if it was not * going to be activated to begin with. */ + +#ifdef CONFIG_KERNEL_REPLICATION + /* If kernel replication is enabled, the special alignment is necessary. + * Due to this fact for now we map kernel by huge pages even + * in case of KASLR enabled. Ugly but works. + */ +#ifdef CONFIG_ARM64_4K_PAGES + return HPAGE_SIZE; +#else + return CONT_PTE_SIZE; +#endif + +#else + extern bool efi_nokaslr; + return efi_nokaslr ? MIN_KIMG_ALIGN : EFI_KIMG_ALIGN; +#endif } =20 #define EFI_ALLOC_ALIGN SZ_64K --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2424948F830 for ; Thu, 27 Aug 2026 16:24:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847896; cv=none; b=LddNmNyKWoCKfSbk2tHYb1heOYUcA2Ty75RiBxJiWg92amUTeivrS81ejdscu14hTjBJIWa7oo+xfvl8zTTG/78+bqvx5Ju3z49ziXUyp+LvtzJZUhumA/X1nv7TpAH0xPEchk6kaDgm4Cc9srd21+dXYbCJn/TVydFv+AvxXsY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847896; c=relaxed/simple; bh=Hzhaxw/Y44kMWGiScMy4uMzYjI1VVlV4Rpa3MtTyC1U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PtX6CPdjN919b9+RyfIfDMne9jdLuoDlbu7RNNLKKLNJVFWc/dqHk3RiFab2SFRxRQq1HQtCq9ZDy4lnS9BLz6pusHHFbQehwKzwNqSLP0VqgstB0DO47jT8fYUFq+3IrwJn0qO+fV21q0vk5KY5a3TQ+I5a1NLGXyRimqLqdIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=CccpWaUs; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="CccpWaUs" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=fUa7pS4OHibv7k1aZNkFDzTCnQ9Tn89o+Rildjw0B6k=; b=CccpWaUsRgM2J3CMlge2JDc32pT8dih8KCHNscYv2r+jgWhSa/BbR/8RILkH8O5kbpzttbHXn 6tbp/SC/chqJ3BbIXL3K7I04ctuktFcHeXy6i4fEkUR+lXXsoWpfO3tQGirIX0pTiGi04wxJ/z+ pCpZ1bHP36L3Y9WHJC9YEtw= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KF5v67zHnH3h; Fri, 28 Aug 2026 00:24:01 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id D049140572; Fri, 28 Aug 2026 00:24:37 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:37 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 15/18] arm64: add support of NUMA replication for ptdump Date: Fri, 28 Aug 2026 00:11:55 +0800 Message-ID: <20260827161158.3618409-16-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/mm/ptdump.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 5a76c59b5ada..17c4de4b9b3e 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -18,6 +18,7 @@ #include #include #include +#include =20 #include #include @@ -328,7 +329,7 @@ static struct ptdump_info kernel_ptdump_info __ro_after= _init =3D { .mm =3D &init_mm, }; =20 -bool ptdump_check_wx(void) +static bool ptdump_check_wx_pgd(struct mm_struct *mm, pgd_t *pgd) { struct ptdump_pg_state st =3D { .seq =3D NULL, @@ -353,7 +354,7 @@ bool ptdump_check_wx(void) } }; =20 - ptdump_walk_pgd(&st.ptdump, &init_mm, NULL); + ptdump_walk_pgd(&st.ptdump, mm, pgd); =20 if (st.wx_pages || st.uxn_pages) { pr_warn("Checked W+X mappings: FAILED, %lu W+X pages found, %lu non-UXN = pages found\n", @@ -367,6 +368,25 @@ bool ptdump_check_wx(void) } } =20 + +#ifdef CONFIG_KERNEL_REPLICATION +bool ptdump_check_wx(void) +{ + bool res =3D false; + int nid; + + for_each_memory_node(nid) + res |=3D ptdump_check_wx_pgd(&init_mm, per_node_pgd(&init_mm, nid)); + + return res; +} +#else +bool ptdump_check_wx(void) +{ + return ptdump_check_wx_pgd(&init_mm, init_mm->pgd); +} +#endif + static int __init ptdump_init(void) { u64 page_offset =3D _PAGE_OFFSET(vabits_actual); --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2439448F833 for ; Thu, 27 Aug 2026 16:24:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; cv=none; b=GIlea0XcGLoP8+vfFnTrWqCNlLdwTkq2xXMagP5b5BARK3BAeerUwBdNZJxc3RlN5Gjl4pYsk2qaqldbb1+pu48at+3gUm7EtjQ66oHe39rMDnWNIo5Ai9IVR9seok5Z+nTnvjcVP96PVJDognbHIykdV54wy+blGlMgm9u1RWs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; c=relaxed/simple; bh=VA2qbvtc9D3WAUublEK754KUy3iM7l5IKInc4JhiwpI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=YKom9/H2uOxsp5QxBReACk/lbeDIZJUqc4LXgXVfLoMdvcIIIOdqpoxpzKsK6owrRKl46Xyox5d9Ah4ByrJET3xOTEpuJnXVB/SSw6ybd3CZOUGLDIokmKCVZetp0B1qPLgOlte+iWLm/uNb9OSzuC73KLjsgYK0uuLEA4RXGkM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=2TvuhQxU; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="2TvuhQxU" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=5euHgUF21bp60qTQigBwKjH4x6Qvg0qe8ONZJ9cd938=; b=2TvuhQxUFr+fy+TYSZfwfzt2uKWRPMUlYGvkIcqkpSYVM6SYMIi2cNT3oGpFWAEDEkYLVmcp3 27DLLJT6/TOlAQbpxu6X1KM8Trmh2Y7QCzb+ecElRL9a3atlGgKwmDr08QkmlRpPCcLbQ5ati4P s1CxhEaZ6wolohHl03XhMkk= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KG1l3lzHnH3h; Fri, 28 Aug 2026 00:24:02 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 4117C40570; Fri, 28 Aug 2026 00:24:38 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:37 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 16/18] arm64: add kernel modules text and rodata replication support Date: Fri, 28 Aug 2026 00:11:56 +0800 Message-ID: <20260827161158.3618409-17-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- arch/arm64/kernel/module.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index 24adb581af0e..61f605f74d86 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -18,6 +18,7 @@ #include #include #include +#include =20 #include #include @@ -25,6 +26,16 @@ #include #include =20 +#ifdef CONFIG_KERNEL_REPLICATION +void module_replicate(void *ptr) +{ + gfp_t gfp_mask =3D GFP_KERNEL; + + __vmalloc_node_replicate_range(ptr, gfp_mask, + PAGE_KERNEL, 0); +} +#endif /*CONFIG_KERNEL_REPLICATION*/ + enum aarch64_reloc_op { RELOC_OP_NONE, RELOC_OP_ABS, --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24F9948F83A for ; Thu, 27 Aug 2026 16:24:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; cv=none; b=rtkyfVgsNkdEsAm/6NOxELyQ8af/hu4vb2JxF/NotWigj0YU173/+eIj+7ZZApmNoAM6M2BmF2fX0OKQXVp5MQ8X2WT3TRGM2BnXUc4s3DIi/wjq6ri1ySocYzMIpVN2/AuuFDWZzo8qKV2cKiLYflGce2aP5fMTKYJDagBC3/w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847895; c=relaxed/simple; bh=iqURKcoLb7dL5bEZlTtyjlOhUI+gBlTukvf1yCzZzxQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gGq191QKnBG2u7xWxxJfNRZuChs1GyUiyBSkvftZpLANP3IRkjmQqW4P5N2tSU/lGFmRHoN5xI824kmTZ3Wo4/Opl6MQVfbOqlBwGiD1v+XwoZ4zAo7NOCjrP7McPpdetGPuogiqrNg+/knqGt8wmkD8qZy1PPip9TblOmh5S8Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=s4BuyPR4; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="s4BuyPR4" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=mJEDJhtFMNSnLhy3o2OpHLDFK+OrNY4mUlH1+uIwElw=; b=s4BuyPR4z6gvxyyhsQPmUQLxP8SwiElyHOlbrC1Eo4Rgkt/YbyWTj7n33YgGtw0nWyN69mN76 NsVuPhOXAWVkrROAXHp01c/lc4cxcZuHQq/5FRCUYaAjOnDpMs3T63Y5vAGwjSZpB6mzq+tZFFj YQyUpUQJBZabrg0CIZAOjmw= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KX5cWQzJ46Yq; Fri, 28 Aug 2026 00:24:16 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 55E1340572; Fri, 28 Aug 2026 00:24:38 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:37 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 17/18] mm: init kernel modules with replication support Date: Fri, 28 Aug 2026 00:11:57 +0800 Message-ID: <20260827161158.3618409-18-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" Acked-by: Artem Kuzin Acked-by: Alexander Grubnikov Acked-by: Ilya Hanov Acked-by: Denis Darvish Signed-off-by: Nikita Panov --- include/linux/moduleloader.h | 4 ++++ kernel/module/main.c | 17 ++++++++++++++++ kernel/module/strict_rwx.c | 12 +++++------ mm/execmem.c | 39 +++++++++++++++++++++++++++++++----- 4 files changed, 61 insertions(+), 11 deletions(-) diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h index e395461d59e5..1d100dfc8251 100644 --- a/include/linux/moduleloader.h +++ b/include/linux/moduleloader.h @@ -25,6 +25,10 @@ int module_frob_arch_sections(Elf_Ehdr *hdr, /* Additional bytes needed by arch in front of individual sections */ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int sec= tion); =20 +#ifdef CONFIG_KERNEL_REPLICATION +void module_replicate(void *ptr); +#endif /* CONFIG_KERNEL_REPLICATION */ + /* Determines if the section name is an init section (that is only used du= ring * module loading). */ diff --git a/kernel/module/main.c b/kernel/module/main.c index d0e1e0bd2ad0..d5c8f041be48 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -60,6 +60,7 @@ #include #include #include +#include #include #include "internal.h" =20 @@ -1338,6 +1339,18 @@ void __weak module_arch_freeing_init(struct module *= mod) { } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static int sections_to_replicate[] =3D {MOD_TEXT, MOD_RODATA}; + +static void module_replicate_sections(struct module *mod) +{ + int i; + + for (i =3D 0; i < ARRAY_SIZE(sections_to_replicate); i++) + module_replicate(mod->mem[sections_to_replicate[i]].base); +} +#endif /* CONFIG_KERNEL_REPLICATION */ + static int module_memory_alloc(struct module *mod, enum mod_mem_type type) { unsigned int size =3D PAGE_ALIGN(mod->mem[type].size); @@ -3330,6 +3343,10 @@ static int complete_formation(struct module *mod, st= ruct load_info *info) module_bug_finalize(info->hdr, info->sechdrs, mod); module_cfi_finalize(info->hdr, info->sechdrs, mod); =20 +#ifdef CONFIG_KERNEL_REPLICATION + module_replicate_sections(mod); +#endif + err =3D module_enable_rodata_ro(mod); if (err) goto out_strict_rwx; diff --git a/kernel/module/strict_rwx.c b/kernel/module/strict_rwx.c index 8fd438529fbc..07ad1af69eaa 100644 --- a/kernel/module/strict_rwx.c +++ b/kernel/module/strict_rwx.c @@ -39,9 +39,9 @@ int module_enable_text_rox(const struct module *mod) if (mem->is_rox) ret =3D execmem_restore_rox(mem->base, mem->size); else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX)) - ret =3D module_set_memory(mod, type, set_memory_rox); + ret =3D module_set_memory(mod, type, numa_set_memory_rox); else - ret =3D module_set_memory(mod, type, set_memory_x); + ret =3D module_set_memory(mod, type, numa_set_memory_x); if (ret) return ret; } @@ -55,10 +55,10 @@ int module_enable_rodata_ro(const struct module *mod) if (!IS_ENABLED(CONFIG_STRICT_MODULE_RWX) || !rodata_enabled) return 0; =20 - ret =3D module_set_memory(mod, MOD_RODATA, set_memory_ro); + ret =3D module_set_memory(mod, MOD_RODATA, numa_set_memory_ro); if (ret) return ret; - ret =3D module_set_memory(mod, MOD_INIT_RODATA, set_memory_ro); + ret =3D module_set_memory(mod, MOD_INIT_RODATA, numa_set_memory_ro); if (ret) return ret; =20 @@ -70,7 +70,7 @@ int module_enable_rodata_ro_after_init(const struct modul= e *mod) if (!IS_ENABLED(CONFIG_STRICT_MODULE_RWX) || !rodata_enabled) return 0; =20 - return module_set_memory(mod, MOD_RO_AFTER_INIT, set_memory_ro); + return module_set_memory(mod, MOD_RO_AFTER_INIT, numa_set_memory_ro); } =20 int module_enable_data_nx(const struct module *mod) @@ -79,7 +79,7 @@ int module_enable_data_nx(const struct module *mod) return 0; =20 for_class_mod_mem_type(type, data) { - int ret =3D module_set_memory(mod, type, set_memory_nx); + int ret =3D module_set_memory(mod, type, numa_set_memory_nx); =20 if (ret) return ret; diff --git a/mm/execmem.c b/mm/execmem.c index 74a178a87e75..ea4b15c8a788 100644 --- a/mm/execmem.c +++ b/mm/execmem.c @@ -16,6 +16,7 @@ #include #include #include +#include =20 #include =20 @@ -26,8 +27,9 @@ static struct execmem_info *execmem_info __ro_after_init; static struct execmem_info default_execmem_info __ro_after_init; =20 #ifdef CONFIG_MMU -static void *execmem_vmalloc(struct execmem_range *range, size_t size, - pgprot_t pgprot, unsigned long vm_flags) + +static void *execmem_vmalloc_node(struct execmem_range *range, size_t size, + pgprot_t pgprot, unsigned long vm_flags, int node) { bool kasan =3D range->flags & EXECMEM_KASAN_SHADOW; gfp_t gfp_flags =3D GFP_KERNEL | __GFP_NOWARN; @@ -40,13 +42,13 @@ static void *execmem_vmalloc(struct execmem_range *rang= e, size_t size, vm_flags |=3D VM_DEFER_KMEMLEAK; =20 p =3D __vmalloc_node_range(size, align, start, end, gfp_flags, - pgprot, vm_flags, NUMA_NO_NODE, + pgprot, vm_flags, node, __builtin_return_address(0)); if (!p && range->fallback_start) { start =3D range->fallback_start; end =3D range->fallback_end; p =3D __vmalloc_node_range(size, align, start, end, gfp_flags, - pgprot, vm_flags, NUMA_NO_NODE, + pgprot, vm_flags, node, __builtin_return_address(0)); } =20 @@ -63,6 +65,26 @@ static void *execmem_vmalloc(struct execmem_range *range= , size_t size, return p; } =20 +#ifdef CONFIG_KERNEL_REPLICATION +static void *execmem_vmalloc_type(struct execmem_range *range, size_t size, + pgprot_t pgprot, unsigned long vm_flags, enum execmem_type type) +{ + if (is_text_replicated() && (type =3D=3D EXECMEM_MODULE_TEXT || type =3D= =3D EXECMEM_MODULE_DATA)) + /* Need to specify some numa node id for correct allocation and further = replication */ + return execmem_vmalloc_node(range, size, pgprot, + vm_flags | VM_NUMA_SHARED, numa_node_id()); + else + return execmem_vmalloc_node(range, size, pgprot, + vm_flags, NUMA_NO_NODE); +} +#else +static void *execmem_vmalloc_type(struct execmem_range *range, size_t size, + pgprot_t pgprot, unsigned long vm_flags, enum execmem_type type) +{ + return execmem_vmalloc_node(range, size, pgprot, vm_flags, NUMA_NO_NODE); +} +#endif + struct vm_struct *execmem_vmap(size_t size) { struct execmem_range *range =3D &execmem_info->ranges[EXECMEM_MODULE_DATA= ]; @@ -87,6 +109,13 @@ static void *execmem_vmalloc(struct execmem_range *rang= e, size_t size, #endif /* CONFIG_MMU */ =20 #ifdef CONFIG_ARCH_HAS_EXECMEM_ROX + +static void *execmem_vmalloc(struct execmem_range *range, size_t size, + pgprot_t pgprot, unsigned long vm_flags) +{ + return execmem_vmalloc_node(range, size, pgprot, vm_flags, NUMA_NO_NODE); +} + struct execmem_cache { struct mutex mutex; struct maple_tree busy_areas; @@ -475,7 +504,7 @@ void *execmem_alloc(enum execmem_type type, size_t size) if (use_cache) p =3D execmem_cache_alloc(range, size); else - p =3D execmem_vmalloc(range, size, pgprot, vm_flags); + p =3D execmem_vmalloc_type(range, size, pgprot, vm_flags, type); =20 return kasan_reset_tag(p); } --=20 2.34.1 From nobody Sun Sep 27 00:40:21 2026 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEC5C48CD55 for ; Thu, 27 Aug 2026 16:24:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.176.79.56 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847896; cv=none; b=kusN7+n0tTW+/NKny/0uBSat+lHXWqnJkGoJktljsStUZZZBAmvrRBm++WIrKkXlGenQTRA51ISH2mKCc6YGNiQPQpH6P6ElxNfhdWO48MavweDTryPG0Owrr0JM0GdK0EwiahVZ1KfEU356qq8wggAcJDYrTolQlJRBcC7PzeM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787847896; c=relaxed/simple; bh=CpZb6YjuVkK99rCOZMPlVhsAOdgy/X0lF2+TvxsNdaA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=utpol47zTgL86Fut8vsF1BFlQPZYFqn7zTvKY2/XjB2qbhf58g2H9cN44pilBhBuNpjcdInIOZAql1zVrl2ehgKaTsf49KzjhnrtAFMvb+yhsAP0bgEJkC2/kK9NzIotuBFYc2JlAEFgazleL6pHEtlcupm1gLxBE0ysHGNmrB8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=GA3eVHRa; arc=none smtp.client-ip=185.176.79.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="GA3eVHRa" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=ssuN6l1WxJ9Y104+n1a6EbIEvAWDLhzTTWp5ogAfBY4=; b=GA3eVHRaMB1sUAyee8x3oE8sFFq55hRTx1otDxbGFRORJ9LQh//eL8/dJQ9ws45KU0NRVm5kY 3FRlm4C7LukM49ewJKJsxRUAo2VZJ4zTkmIvYphhh9aAJ5aukhsEoh+2QWgz6ozklCrbDu7Uyg3 MMAYPYoCnAbFVUKjN1E/fHs= Received: from mail.maildlp.com (unknown [172.18.224.150]) by frasgout.his.huawei.com (SkyGuard) with ESMTPS id 4hW6KX6CxPzJ46ZW; Fri, 28 Aug 2026 00:24:16 +0800 (CST) Received: from mscpeml500004.china.huawei.com (unknown [7.188.26.250]) by mail.maildlp.com (Postfix) with ESMTPS id 6AA4840570; Fri, 28 Aug 2026 00:24:38 +0800 (CST) Received: from mscphis00949.huawei.com (10.123.66.11) by mscpeml500004.china.huawei.com (7.188.26.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Thu, 27 Aug 2026 19:24:38 +0300 From: Nikita Panov To: , , , , , , , , , , , , CC: , , , , , Subject: [RFC PATCH 18/18] mm: introduce kernel cmdline option "kernel_replication=" Date: Fri, 28 Aug 2026 00:11:58 +0800 Message-ID: <20260827161158.3618409-19-panov.nikita@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260827161158.3618409-1-panov.nikita@huawei.com> References: <20260827161158.3618409-1-panov.nikita@huawei.com> Reply-To: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: mscpeml100003.china.huawei.com (10.199.174.67) To mscpeml500004.china.huawei.com (7.188.26.250) Content-Type: text/plain; charset="utf-8" The new cmdline option allows enabling/disabling kernel replication feature. By default, the replication feature is disabled. This allows to set CONFIG_KERNEL_REPLICATION=3Dy by default, but enabling this feature is done via kernel cmdline. Signed-off-by: Nikita Panov --- .../admin-guide/kernel-parameters.txt | 7 ++ include/linux/numa_kernel_replication.h | 27 ++++++- kernel/module/main.c | 3 + mm/numa_kernel_replication.c | 78 ++++++++++++++++++- mm/vmalloc.c | 2 +- 5 files changed, 113 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index d55524e3b724..133afdd7dab5 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3039,6 +3039,13 @@ Kernel parameters for Movable pages. "nn[KMGTPE]", "nn%", and "mirror" are exclusive, so you cannot specify multiple forms. =20 + kernel_replication=3D + [ARM64] + Format: [on|off] + If CONFIG_KERNEL_REPLICATION is set, it allows + enabling/disabling the kernel replication feature + via cmdline. Default value is off. + kfence.burst=3D [MM,KFENCE] The number of additional successive allocations to be attempted through KFENCE for each sample interval. diff --git a/include/linux/numa_kernel_replication.h b/include/linux/numa_k= ernel_replication.h index 34aa063d42e6..23069ca82fa8 100644 --- a/include/linux/numa_kernel_replication.h +++ b/include/linux/numa_kernel_replication.h @@ -18,8 +18,31 @@ extern nodemask_t replica_nodes; nid !=3D MAX_NUMNODES; \ nid =3D next_node(nid, replica_nodes)) =20 -#define this_node_pgd(mm) ((mm)->pgd_numa[numa_node_id()]) -#define per_node_pgd(mm, nid) ((mm)->pgd_numa[nid]) +bool is_text_replicated(void); + +static inline pgd_t *this_node_pgd(struct mm_struct *mm) +{ + if (is_text_replicated()) + return mm->pgd_numa[numa_node_id()]; + else + return mm->pgd; +} + +static inline pgd_t *per_node_pgd(struct mm_struct *mm, int nid) +{ + if (is_text_replicated()) + return mm->pgd_numa[nid]; + else + return mm->pgd; +} + +static inline pgd_t **per_node_pgd_ptr(struct mm_struct *mm, int nid) +{ + if (is_text_replicated()) + return &mm->pgd_numa[nid]; + else + return &mm->pgd; +} =20 static inline bool numa_addr_has_replica(const void *addr) { diff --git a/kernel/module/main.c b/kernel/module/main.c index d5c8f041be48..25c8816588ce 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1346,6 +1346,9 @@ static void module_replicate_sections(struct module *= mod) { int i; =20 + if (!is_text_replicated()) + return; + for (i =3D 0; i < ARRAY_SIZE(sections_to_replicate); i++) module_replicate(mod->mem[sections_to_replicate[i]].base); } diff --git a/mm/numa_kernel_replication.c b/mm/numa_kernel_replication.c index b1c5cd2a1fa1..c8c3424fa20c 100644 --- a/mm/numa_kernel_replication.c +++ b/mm/numa_kernel_replication.c @@ -64,6 +64,8 @@ static unsigned int master_node =3D INT_MAX; */ static int node_to_memory_node[MAX_NUMNODES]; =20 +static bool kernel_replication_enabled; + static bool pgtables_extra; static DEFINE_SPINLOCK(debugfs_lock); =20 @@ -253,6 +255,9 @@ static void dump_pgtables(struct mm_struct *mm, start =3D start & PAGE_MASK; end =3D (end & PAGE_MASK) - 1 + PAGE_SIZE; =20 + if (!mm->pgd_numa) + return; + replication_log(data, "----PER-NUMA NODE KERNEL REPLICATION ENABLED----\n"); =20 @@ -619,6 +624,18 @@ static void replicate_pgtables(void) } } =20 +static void __init numa_replicate_kernel_text_disabled(void) +{ + int nid; + + init_mm.pgd_numa =3D (pgd_t **)kmalloc(sizeof(pgd_t *) * MAX_NUMNODES, GF= P_PGTABLE_KERNEL); + BUG_ON(!init_mm.pgd_numa); + for_each_online_node(nid) { + init_mm.pgd_numa[nid] =3D init_mm.pgd; + } +} + + /* * Kernel text replication includes two steps: * 1. page tables replication for init_mm @@ -636,6 +653,11 @@ void __init numa_replicate_kernel_text(void) { int nid; =20 + if (!kernel_replication_enabled) { + numa_replicate_kernel_text_disabled(); + return; + } + replicate_pgtables(); =20 for_each_memory_node(nid) { @@ -653,6 +675,10 @@ void numa_replicate_kernel_rodata(void) { int nid; =20 + if (!kernel_replication_enabled) { + return; + } + for_each_memory_node(nid) { if (nid =3D=3D master_node) continue; @@ -664,7 +690,7 @@ void numa_replicate_kernel_rodata(void) =20 void numa_setup_pgd(void) { - numa_load_replicated_pgd(init_mm.pgd_numa[numa_node_id()]); + numa_load_replicated_pgd(this_node_pgd(&init_mm)); } =20 void __init_or_module *numa_get_replica(void *vaddr, int nid) @@ -679,8 +705,48 @@ void __init_or_module *numa_get_replica(void *vaddr, i= nt nid) return node_desc[nid].text_vaddr + offset; } =20 +static int __init setup_kernel_replication(char *str) +{ + int ret =3D 0; + + if (!str) + goto out; + if (!strcmp(str, "on")) { + kernel_replication_enabled =3D true; + pr_info("Kernel replication enabled via cmdline\n"); + ret =3D 1; + } else if (!strcmp(str, "off")) { + kernel_replication_enabled =3D false; + pr_info("Kernel replication disabled via cmdline\n"); + ret =3D 1; + } +out: + if (!ret) + pr_warn("kernel_replication=3D cannot parse, ignored\n"); + return ret; +} +__setup("kernel_replication=3D", setup_kernel_replication); + + nodemask_t __ro_after_init replica_nodes =3D { { [0] =3D 1UL } }; =20 +/* + * Let us pretend, that we have only single node fore replicas. + * Do not replicate anything. + */ +static void __init numa_replication_init_disabled(void) +{ + int nid; + + __node_set(0, &replica_nodes); + for_each_online_node(nid) { + node_to_memory_node[nid] =3D 0; + } + + node_desc[0].text_vaddr =3D lm_alias((void *)KERNEL_TEXT_START); + node_desc[0].rodata_vaddr =3D lm_alias((void *)KERNEL_RODATA_START); +} + void __init numa_replication_init(void) { int nid; @@ -693,6 +759,16 @@ void __init numa_replication_init(void) #endif nodes_clear(replica_nodes); =20 + if (kernel_replication_enabled) + pr_info("WARNING! WARNING! WARNING! Kernel replication enabled WARNING! = WARNING! WARNING!\n"); + else + pr_info("Kernel replication disabled\n"); + + if (!kernel_replication_enabled) { + numa_replication_init_disabled(); + return; + } + for_each_node_state(nid, N_MEMORY) { __node_set(nid, &replica_nodes); } diff --git a/mm/vmalloc.c b/mm/vmalloc.c index da6facf64db8..532298ab4263 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -4422,7 +4422,7 @@ int __vmalloc_node_replicate_range(const void *addr, = gfp_t gfp_mask, for (i =3D 0; i < area->nr_pages; i++) list_add(&pages[i]->lru, &area->pages[i]->lru); =20 - __vunmap_range_noflush_pgd(init_mm.pgd_numa[node], + __vunmap_range_noflush_pgd(per_node_pgd(&init_mm, node), area_start, area_end); =20 /* --=20 2.34.1