From nobody Mon May 25 05:55:43 2026 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36BCF3CF026 for ; Mon, 18 May 2026 08:50:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.110 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094212; cv=none; b=kqDUFXSyJDeUngRZgZ0Hb6poG4VdImhAqY1fjvXbRMOeR9YCQFdmRvqEZaj/fDmFxAZiBUP0+1Fw1ZZr187TIsRyREn/OV1Eay7eYGfOnWrnbogACNaaJ/fIxzAv8JMOw//BjGAFfwrFQNEOiMHrQW+amGConcSLEr10NVordZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094212; c=relaxed/simple; bh=L8cJZcOP0t2KS3Mqq+rQ0PYMUP9fcw0ya0FdL8OTCwE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GP3hPDmMweKM+jwr6aqzx/6M8oL2sNj1Lar62RK6UqHXkcyjuhC77Y7V2LXi/DY/v8V0m/cOPkaJ/BLChPLru6oigfxhay2GPFhj0LhOSWJP4enQzl1ytcRQE8H5gUKK1ThjFEHzqIVyqLzyb4dkzYhWTzfDI5A/jO7Y81p8sSI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=EbrHZJiq; arc=none smtp.client-ip=115.124.30.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="EbrHZJiq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094208; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=snbZ8y+F0BUU/EmaB78zaC79H5axWqPJEhlqfbfpDD4=; b=EbrHZJiqWlCv2Zky3PkM/DRRyM2b4wAvONN0eZcp0j5IGEqxYav5Oiold1/xReSGCXgcbQrG5ewFHed95fwwWC9Y26j550av+DKbZPCXyQUidvnfRmGfrYBZB6t3fywYWfrt+/xO2YSvhq/B6xpIOf/wsxVXS5GfRJMN8D14HII= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R301e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032089153;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=30;SR=0;TI=SMTPD_---0X3647I9_1779094202; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X3647I9_1779094202 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:04 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com, Mauro Carvalho Chehab , Jonathan Cameron Subject: [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user() Date: Mon, 18 May 2026 16:49:45 +0800 Message-ID: <20260518084956.2538442-2-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tong Tiangen x86/powerpc has it's implementation of copy_mc_to_user(), we add generic fallback in include/linux/uaccess.h prepare for other architechures to enable CONFIG_ARCH_HAS_COPY_MC. Signed-off-by: Tong Tiangen Acked-by: Michael Ellerman Reviewed-by: Mauro Carvalho Chehab Reviewed-by: Jonathan Cameron --- arch/powerpc/include/asm/uaccess.h | 1 + arch/x86/include/asm/uaccess.h | 1 + include/linux/uaccess.h | 8 ++++++++ 3 files changed, 10 insertions(+) diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/= uaccess.h index e98c628e3899..073de098d45a 100644 --- a/arch/powerpc/include/asm/uaccess.h +++ b/arch/powerpc/include/asm/uaccess.h @@ -432,6 +432,7 @@ copy_mc_to_user(void __user *to, const void *from, unsi= gned long n) =20 return n; } +#define copy_mc_to_user copy_mc_to_user #endif =20 extern size_t copy_from_user_flushcache(void *dst, const void __user *src,= size_t size); diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 3a0dd3c2b233..308b0854d1d5 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -496,6 +496,7 @@ copy_mc_to_kernel(void *to, const void *from, unsigned = len); =20 unsigned long __must_check copy_mc_to_user(void __user *to, const void *from, unsigned len); +#define copy_mc_to_user copy_mc_to_user #endif =20 /* diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 56328601218c..c53a65394f80 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -250,6 +250,14 @@ copy_mc_to_kernel(void *dst, const void *src, size_t c= nt) } #endif =20 +#ifndef copy_mc_to_user +static inline unsigned long __must_check +copy_mc_to_user(void *dst, const void *src, size_t cnt) +{ + return copy_to_user(dst, src, cnt); +} +#endif + static __always_inline void pagefault_disabled_inc(void) { current->pagefault_disabled++; --=20 2.39.3 From nobody Mon May 25 05:55:43 2026 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BFE83C4562 for ; Mon, 18 May 2026 08:50:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094213; cv=none; b=EMGqLO6qjqxaI2M+turbXcEySc7VXN7h2+eOA15pa+Ggt/C6RbyyD9hzEaui30sixFwVYqgOFbUSYGklNtvJOJ+WPMhk8KZIy5kRrMIRkR8As/Jq4Zo+2oy5VM4SoyQdBH4Or9kWDLUuRKW5MpVH0o0PuO4d6bmLCnWiycCTr+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094213; c=relaxed/simple; bh=BjZ7Hs8hJdW2c8bPyXr0pTGVRxoq0lIMP6xt0HrU+5w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cP/c0peaMvEuSWV325v68QHLkBIx5ZlQo3yujDHXuuCHPNnZ3zUj/fVv/kd2Ovp4oojN5KCNP1kDNDJthNtzr54ZdjyHqZ5ZUB1DdKy+1yYqIecSVAhp32u6idczB1l4MY2iQkQYYf8ruQyUkoR3OADlyPEHwfP76nNHA1LjoMQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=xP2rOROc; arc=none smtp.client-ip=115.124.30.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="xP2rOROc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094209; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=UoNLybuMUSeU440nlltv1/CuGYWJcpqCWHdxGW4hf3M=; b=xP2rOROcyWq0VP5vlF3d2TJfJw3hiU6kyOhWlrObbNhQfZ5gSfreVI35SsZKD1e4ywilrwW13ACTXIx2fCQwnHXYWq4iYwIy5hdNbeVznf/Bya3lGPLM52zn77mqbRK1FPL3Sj9J4d/891dHxFIKwPviovj57dABGFKR+xx6P9U= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam011083073210;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=28;SR=0;TI=SMTPD_---0X36F6m8_1779094205; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X36F6m8_1779094205 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:07 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com Subject: [PATCH v14 2/8] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption Date: Mon, 18 May 2026 16:49:46 +0800 Message-ID: <20260518084956.2538442-3-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When a GHES SEA (Synchronous External Abort) fires while the CPU was executing in kernel mode, it typically means that kernel code itself consumed a poisoned memory location -- e.g. copy_from_user() / copy_to_user() invoked from a ioctl() or write() syscall touched a poisoned user page or page-cache page on behalf of the task. The expected behaviour in that case is that the faulting kernel helper returns via its extable fixup and the syscall returns an error (e.g. -EFAULT) to user space. It is NOT appropriate to deliver SIGBUS to the current task: the task did not directly dereference the poisoned address, the kernel did on its behalf, and the kernel is able to recover. Up to now ghes_handle_memory_failure() unconditionally promoted any synchronous recoverable memory error to MF_ACTION_REQUIRED, which ends up SIGBUS on current -- regardless of whether the poison was consumed from user space or from inside the kernel on the task's behalf. That kills tasks that should instead have seen a plain syscall error. To fix this, the execution mode in which the exception was taken must be captured at the arch-level entry point, where pt_regs (and hence user_mode(regs)) are still available. The estatus node that later drains the error in IRQ / process context no longer has access to the original regs. Introduce: enum context { NO_USE =3D -1, IN_KERNEL =3D 0, IN_USER =3D 1 }; and plumb the value all the way down to the queued estatus node: * Add an 'enum context context' field to struct ghes_estatus_node and record it in ghes_in_nmi_queue_one_entry(). * Extend ghes_notify_sea() and the internal ghes_in_nmi_spool_from_list() with an enum context parameter. Then consume the recorded context in ghes_handle_memory_failure() for the GHES_SEV_RECOVERABLE / sync path: flags =3D sync && context =3D=3D IN_USER ? MF_ACTION_REQUIRED : 0; i.e. MF_ACTION_REQUIRED (and thus SIGBUS via the task_work path) is only raised for user-mode poison consumption. Synchronous errors taken in kernel mode fall back to memory_failure_queue() with flags=3D0, asynchronously isolating the poisoned page while letting the faulting kernel helper's extable fixup return -EFAULT to user space. Paths that pass NO_USE are unaffected: sync is false for them, so flags stays 0 as before. Signed-off-by: Ruidong Tian --- arch/arm64/kernel/acpi.c | 2 +- drivers/acpi/apei/ghes.c | 36 ++++++++++++++++++++---------------- include/acpi/ghes.h | 6 ++++-- 3 files changed, 25 insertions(+), 19 deletions(-) diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c index 5891f92c2035..40d4a2913d51 100644 --- a/arch/arm64/kernel/acpi.c +++ b/arch/arm64/kernel/acpi.c @@ -409,7 +409,7 @@ int apei_claim_sea(struct pt_regs *regs) */ local_daif_restore(DAIF_ERRCTX); nmi_enter(); - err =3D ghes_notify_sea(); + err =3D ghes_notify_sea(user_mode(regs)); nmi_exit(); =20 /* diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c index 3236a3ce79d6..6f265893cddf 100644 --- a/drivers/acpi/apei/ghes.c +++ b/drivers/acpi/apei/ghes.c @@ -529,7 +529,7 @@ static bool ghes_do_memory_failure(u64 physical_addr, i= nt flags) } =20 static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdat= a, - int sev, bool sync) + int sev, bool sync, enum context context) { int flags =3D -1; int sec_sev =3D ghes_severity(gdata->error_severity); @@ -543,7 +543,7 @@ static bool ghes_handle_memory_failure(struct acpi_hest= _generic_data *gdata, (gdata->flags & CPER_SEC_ERROR_THRESHOLD_EXCEEDED)) flags =3D MF_SOFT_OFFLINE; if (sev =3D=3D GHES_SEV_RECOVERABLE && sec_sev =3D=3D GHES_SEV_RECOVERABL= E) - flags =3D sync ? MF_ACTION_REQUIRED : 0; + flags =3D sync && context =3D=3D IN_USER ? MF_ACTION_REQUIRED : 0; =20 if (flags !=3D -1) return ghes_do_memory_failure(mem_err->physical_addr, flags); @@ -552,10 +552,10 @@ static bool ghes_handle_memory_failure(struct acpi_he= st_generic_data *gdata, } =20 static bool ghes_handle_arm_hw_error(struct acpi_hest_generic_data *gdata, - int sev, bool sync) + int sev, bool sync, enum context context) { struct cper_sec_proc_arm *err =3D acpi_hest_get_payload(gdata); - int flags =3D sync ? MF_ACTION_REQUIRED : 0; + int flags =3D sync && context =3D=3D IN_USER ? MF_ACTION_REQUIRED : 0; int length =3D gdata->error_data_length; char error_type[120]; bool queued =3D false; @@ -910,7 +910,8 @@ static void ghes_log_hwerr(int sev, guid_t *sec_type) } =20 static void ghes_do_proc(struct ghes *ghes, - const struct acpi_hest_generic_status *estatus) + const struct acpi_hest_generic_status *estatus, + enum context context) { int sev, sec_sev; struct acpi_hest_generic_data *gdata; @@ -937,11 +938,11 @@ static void ghes_do_proc(struct ghes *ghes, atomic_notifier_call_chain(&ghes_report_chain, sev, mem_err); =20 arch_apei_report_mem_error(sev, mem_err); - queued =3D ghes_handle_memory_failure(gdata, sev, sync); + queued =3D ghes_handle_memory_failure(gdata, sev, sync, context); } else if (guid_equal(sec_type, &CPER_SEC_PCIE)) { ghes_handle_aer(gdata); } else if (guid_equal(sec_type, &CPER_SEC_PROC_ARM)) { - queued =3D ghes_handle_arm_hw_error(gdata, sev, sync); + queued =3D ghes_handle_arm_hw_error(gdata, sev, sync, context); } else if (guid_equal(sec_type, &CPER_SEC_CXL_PROT_ERR)) { struct cxl_cper_sec_prot_err *prot_err =3D acpi_hest_get_payload(gdata); =20 @@ -1190,7 +1191,7 @@ static int ghes_proc(struct ghes *ghes) if (ghes_print_estatus(NULL, ghes->generic, estatus)) ghes_estatus_cache_add(ghes->generic, estatus); } - ghes_do_proc(ghes, estatus); + ghes_do_proc(ghes, estatus, NO_USE); =20 out: ghes_clear_estatus(ghes, estatus, buf_paddr, FIX_APEI_GHES_IRQ); @@ -1297,7 +1298,7 @@ static void ghes_proc_in_irq(struct irq_work *irq_wor= k) len =3D cper_estatus_len(estatus); node_len =3D GHES_ESTATUS_NODE_LEN(len); =20 - ghes_do_proc(estatus_node->ghes, estatus); + ghes_do_proc(estatus_node->ghes, estatus, estatus_node->context); =20 if (!ghes_estatus_cached(estatus)) { generic =3D estatus_node->generic; @@ -1335,7 +1336,8 @@ static void ghes_print_queued_estatus(void) } =20 static int ghes_in_nmi_queue_one_entry(struct ghes *ghes, - enum fixed_addresses fixmap_idx) + enum fixed_addresses fixmap_idx, + enum context context) { struct acpi_hest_generic_status *estatus, tmp_header; struct ghes_estatus_node *estatus_node; @@ -1364,6 +1366,7 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *g= hes, if (!estatus_node) return -ENOMEM; =20 + estatus_node->context =3D context; estatus_node->ghes =3D ghes; estatus_node->generic =3D ghes->generic; estatus =3D GHES_ESTATUS_FROM_NODE(estatus_node); @@ -1398,14 +1401,15 @@ static int ghes_in_nmi_queue_one_entry(struct ghes = *ghes, } =20 static int ghes_in_nmi_spool_from_list(struct list_head *rcu_list, - enum fixed_addresses fixmap_idx) + enum fixed_addresses fixmap_idx, + enum context context) { int ret =3D -ENOENT; struct ghes *ghes; =20 rcu_read_lock(); list_for_each_entry_rcu(ghes, rcu_list, list) { - if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx)) + if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, context)) ret =3D 0; } rcu_read_unlock(); @@ -1488,7 +1492,7 @@ static LIST_HEAD(ghes_sea); * Return 0 only if one of the SEA error sources successfully reported an = error * record sent from the firmware. */ -int ghes_notify_sea(void) +int ghes_notify_sea(enum context context) { static DEFINE_RAW_SPINLOCK(ghes_notify_lock_sea); int rv; @@ -1497,7 +1501,7 @@ int ghes_notify_sea(void) return -ENOENT; =20 raw_spin_lock(&ghes_notify_lock_sea); - rv =3D ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA); + rv =3D ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA, context); raw_spin_unlock(&ghes_notify_lock_sea); =20 return rv; @@ -1552,7 +1556,7 @@ static int ghes_notify_nmi(unsigned int cmd, struct p= t_regs *regs) return ret; =20 raw_spin_lock(&ghes_notify_lock_nmi); - if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI)) + if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI, NO_USE)) ret =3D NMI_HANDLED; raw_spin_unlock(&ghes_notify_lock_nmi); =20 @@ -1606,7 +1610,7 @@ static void ghes_nmi_init_cxt(void) static int __ghes_sdei_callback(struct ghes *ghes, enum fixed_addresses fixmap_idx) { - if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx)) { + if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, NO_USE)) { irq_work_queue(&ghes_proc_irq_work); =20 return 0; diff --git a/include/acpi/ghes.h b/include/acpi/ghes.h index 8d7e5caef3f1..646cd5c3c0ca 100644 --- a/include/acpi/ghes.h +++ b/include/acpi/ghes.h @@ -33,10 +33,12 @@ struct ghes { void __iomem *error_status_vaddr; }; =20 +enum context {NO_USE =3D -1, IN_KERNEL =3D 0, IN_USER =3D 1}; struct ghes_estatus_node { struct llist_node llnode; struct acpi_hest_generic *generic; struct ghes *ghes; + enum context context; }; =20 struct ghes_estatus_cache { @@ -135,9 +137,9 @@ static inline void *acpi_hest_get_next(struct acpi_hest= _generic_data *gdata) section =3D acpi_hest_get_next(section)) =20 #ifdef CONFIG_ACPI_APEI_SEA -int ghes_notify_sea(void); +int ghes_notify_sea(enum context context); #else -static inline int ghes_notify_sea(void) { return -ENOENT; } +static inline int ghes_notify_sea(enum context context) { return -ENOENT; } #endif =20 struct notifier_block; --=20 2.39.3 From nobody Mon May 25 05:55:43 2026 Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9D173CF026 for ; Mon, 18 May 2026 08:50:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.101 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094222; cv=none; b=o0W+ghpb6EqBwUfPez6GHAQ152oAECgMf/GMqreZNymAWtjt7cwuyGrVk407msDERXsUdlQXyjNKUV53nUBZfUtZQqW4F5yozyyg8brqwFMbDlfnDynoT2K2NO/UEum6LMtgxytmz0iOw83NB2E7S3eyESLQ3iV287A22DevZz4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094222; c=relaxed/simple; bh=HZ9c2hz3syTn5/7vCHrKSUGzFyDvNIewrtmfbSQVG1Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a5Z60pbhegoHQ+vdTXIN+9Caur0++1lMSaPjLvQ4l8hr+KlDxdLb2LGA740DnIvwwCDrqgwscGq+ogbnhhSe7Yotjcfx1yfA995JBfF1GcgfWVPeWX7/dbgxuLoJ/q0LBHYZoSgwIln2TV0J/fqgof2tOxtHg/64gMp6owB65sM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=v691m1nl; arc=none smtp.client-ip=115.124.30.101 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="v691m1nl" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094212; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=XZNjExq7H0YZ5/JkBFHk426Iyxdv38aH11enOzWM1m0=; b=v691m1nlFa3KOjQrATQJFVOEAwu2d5w6W/1D1ueAAucd4roplnrHWFwHa7Kt7s/yDxCq+5Xm22jqYInqkcSPwYZto8P3raSk+tjE2TeMsAHxjG/KthSJfrL/kC774USzFSRXBjt1D2oecR0+hTXLt15SEMigb4avI2/16YiQt0I= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032089153;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=28;SR=0;TI=SMTPD_---0X3647KN_1779094209; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X3647KN_1779094209 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:10 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com Subject: [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC Date: Mon, 18 May 2026 16:49:47 +0800 Message-ID: <20260518084956.2538442-4-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tong Tiangen For the arm64 kernel, when it processes hardware memory errors for synchronize notifications(do_sea()), if the errors is consumed within the kernel, the current processing is panic. However, it is not optimal. Take copy_from/to_user for example, If ld* triggers a memory error, even in kernel mode, only the associated process is affected. Killing the user process and isolating the corrupt page is a better choice. Add new fixup type EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR to identify insn that can recover from memory errors triggered by access to kernel memory, and this fixup type is used in __arch_copy_to_user(), This make the regular copy_to_user() will handle kernel memory errors. [Ruidong: handle EX_TYPE_UACCESS_CPY in fixup_exception_me()] Signed-off-by: Tong Tiangen Signed-off-by: Ruidong Tian --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/asm-extable.h | 22 +++++++++++++++++++- arch/arm64/include/asm/asm-uaccess.h | 4 ++++ arch/arm64/include/asm/extable.h | 1 + arch/arm64/lib/copy_to_user.S | 10 +++++----- arch/arm64/mm/extable.c | 21 +++++++++++++++++++ arch/arm64/mm/fault.c | 30 ++++++++++++++++++++-------- 7 files changed, 75 insertions(+), 14 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fe60738e5943..831b20d45893 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -21,6 +21,7 @@ config ARM64 select ARCH_HAS_CACHE_LINE_SIZE select ARCH_HAS_CC_PLATFORM select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION + select ARCH_HAS_COPY_MC if ACPI_APEI_GHES select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/include/asm/= asm-extable.h index d67e2fdd1aee..4980023f2fbd 100644 --- a/arch/arm64/include/asm/asm-extable.h +++ b/arch/arm64/include/asm/asm-extable.h @@ -11,6 +11,8 @@ #define EX_TYPE_KACCESS_ERR_ZERO 3 #define EX_TYPE_UACCESS_CPY 4 #define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 5 +/* kernel access memory error safe */ +#define EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR 6 =20 /* Data fields for EX_TYPE_UACCESS_ERR_ZERO */ #define EX_DATA_REG_ERR_SHIFT 0 @@ -42,7 +44,7 @@ (.L__gpr_num_##gpr << EX_DATA_REG_##reg##_SHIFT) =20 #define _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) \ - __ASM_EXTABLE_RAW(insn, fixup, \ + __ASM_EXTABLE_RAW(insn, fixup, \ EX_TYPE_UACCESS_ERR_ZERO, \ ( \ EX_DATA_REG(ERR, err) | \ @@ -55,6 +57,17 @@ #define _ASM_EXTABLE_UACCESS(insn, fixup) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, wzr, wzr) =20 +#define _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, err, zero) \ + __ASM_EXTABLE_RAW(insn, fixup, \ + EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR, \ + ( \ + EX_DATA_REG(ERR, err) | \ + EX_DATA_REG(ZERO, zero) \ + )) + +#define _ASM_EXTABLE_KACCESS_MEM_ERR(insn, fixup) \ + _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, wzr, wzr) + /* * Create an exception table entry for uaccess `insn`, which will branch t= o `fixup` * when an unhandled fault is taken. @@ -76,6 +89,13 @@ .macro _asm_extable_uaccess_cpy, insn, fixup, uaccess_is_write __ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_UACCESS_CPY, \uaccess_is_write) .endm +/* + * Create an exception table entry for kaccess `insn`, which will branch to + * `fixup` when an unhandled fault is taken. + */ + .macro _asm_extable_kaccess_mem_err, insn, fixup + _ASM_EXTABLE_KACCESS_MEM_ERR(\insn, \fixup) + .endm =20 #else /* __ASSEMBLER__ */ =20 diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/= asm-uaccess.h index 12aa6a283249..c8f0af5fde63 100644 --- a/arch/arm64/include/asm/asm-uaccess.h +++ b/arch/arm64/include/asm/asm-uaccess.h @@ -57,6 +57,10 @@ alternative_else_nop_endif .endm #endif =20 +#define KERNEL_MEM_ERR(l, x...) \ +9999: x; \ + _asm_extable_kaccess_mem_err 9999b, l + #define USER(l, x...) \ 9999: x; \ _asm_extable_uaccess 9999b, l diff --git a/arch/arm64/include/asm/extable.h b/arch/arm64/include/asm/exta= ble.h index 9dc39612bdf5..47c851d7df4f 100644 --- a/arch/arm64/include/asm/extable.h +++ b/arch/arm64/include/asm/extable.h @@ -48,4 +48,5 @@ bool ex_handler_bpf(const struct exception_table_entry *e= x, #endif /* !CONFIG_BPF_JIT */ =20 bool fixup_exception(struct pt_regs *regs, unsigned long esr); +bool fixup_exception_me(struct pt_regs *regs); #endif diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/copy_to_user.S index 819f2e3fc7a9..991d94ecc1a8 100644 --- a/arch/arm64/lib/copy_to_user.S +++ b/arch/arm64/lib/copy_to_user.S @@ -20,7 +20,7 @@ * x0 - bytes not copied */ .macro ldrb1 reg, ptr, val - ldrb \reg, [\ptr], \val + KERNEL_MEM_ERR(9998f, ldrb \reg, [\ptr], \val) .endm =20 .macro strb1 reg, ptr, val @@ -28,7 +28,7 @@ .endm =20 .macro ldrh1 reg, ptr, val - ldrh \reg, [\ptr], \val + KERNEL_MEM_ERR(9998f, ldrh \reg, [\ptr], \val) .endm =20 .macro strh1 reg, ptr, val @@ -36,7 +36,7 @@ .endm =20 .macro ldr1 reg, ptr, val - ldr \reg, [\ptr], \val + KERNEL_MEM_ERR(9998f, ldr \reg, [\ptr], \val) .endm =20 .macro str1 reg, ptr, val @@ -44,7 +44,7 @@ .endm =20 .macro ldp1 reg1, reg2, ptr, val - ldp \reg1, \reg2, [\ptr], \val + KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, [\ptr], \val) .endm =20 .macro stp1 reg1, reg2, ptr, val @@ -74,7 +74,7 @@ SYM_FUNC_START(__arch_copy_to_user) 9997: cmp dst, dstin b.ne 9998f // Before being absolutely sure we couldn't copy anything, try harder - ldrb tmp1w, [srcin] +KERNEL_MEM_ERR(9998f, ldrb tmp1w, [srcin]) USER(9998f, sttrb tmp1w, [dst]) add dst, dst, #1 9998: sub x0, end, dst // bytes not copied diff --git a/arch/arm64/mm/extable.c b/arch/arm64/mm/extable.c index 6e0528831cd3..f78ac7e92845 100644 --- a/arch/arm64/mm/extable.c +++ b/arch/arm64/mm/extable.c @@ -110,7 +110,28 @@ bool fixup_exception(struct pt_regs *regs, unsigned lo= ng esr) return ex_handler_uaccess_cpy(ex, regs, esr); case EX_TYPE_LOAD_UNALIGNED_ZEROPAD: return ex_handler_load_unaligned_zeropad(ex, regs); + case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR: + return false; } =20 BUG(); } + +bool fixup_exception_me(struct pt_regs *regs) +{ + const struct exception_table_entry *ex; + + ex =3D search_exception_tables(instruction_pointer(regs)); + if (!ex) + return false; + + switch (ex->type) { + case EX_TYPE_UACCESS_CPY: + return ex_handler_uaccess_cpy(ex, regs, 0); + case EX_TYPE_UACCESS_ERR_ZERO: + case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR: + return ex_handler_uaccess_err_zero(ex, regs); + } + + return false; +} diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 0f3c5c7ca054..efbda54770be 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -858,21 +858,35 @@ static int do_bad(unsigned long far, unsigned long es= r, struct pt_regs *regs) return 1; /* "fault" */ } =20 +/* + * APEI claimed this as a firmware-first notification. + * Some processing deferred to task_work before ret_to_user(). + */ +static int do_apei_claim_sea(struct pt_regs *regs) +{ + int ret; + + ret =3D apei_claim_sea(regs); + if (ret) + return ret; + + if (!user_mode(regs) && IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC)) { + if (!fixup_exception_me(regs)) + return -ENOENT; + } + + return ret; +} + static int do_sea(unsigned long far, unsigned long esr, struct pt_regs *re= gs) { const struct fault_info *inf; unsigned long siaddr; =20 - inf =3D esr_to_fault_info(esr); - - if (user_mode(regs) && apei_claim_sea(regs) =3D=3D 0) { - /* - * APEI claimed this as a firmware-first notification. - * Some processing deferred to task_work before ret_to_user(). - */ + if (do_apei_claim_sea(regs) =3D=3D 0) return 0; - } =20 + inf =3D esr_to_fault_info(esr); if (esr & ESR_ELx_FnV) { siaddr =3D 0; } else { --=20 2.39.3 From nobody Mon May 25 05:55:43 2026 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F825387585 for ; Mon, 18 May 2026 08:50:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.110 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094220; cv=none; b=OpNgruc6fRGDjIbI9rlqTlb1kAjGIsO2hxCslg72En19q3u0LTJxDNV8GcMsQ8hhyGlO2+4clQej9GdSkd42ck38kXBFh3Yf4gH/20aIcsiqogUR4NMKAFox2p0lj7nX+NPQfSVtJEm7byu1l4Vgl6aZka5vtJOTxdnbx/0T7eE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094220; c=relaxed/simple; bh=brmnOeKWNpliocbmYOYS4ppxivkLB3jaPhueQwCKquo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QvoVl9kiUDqbyfxV3OREqOvPLjqN+2ob3DtiMnqNkf7nRfFTkOI/NQQ14f0Q62nilczmHe3DZipw7mg/+G3427ti9nVX/Qm7p0tS/8EudZ8CRlLs9Cj/+rII2v91QoYgyeY6jJTIxsgTfEgolXbosSluz8LKkZya8A9pyjezK8A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=HcmhSZ8x; arc=none smtp.client-ip=115.124.30.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="HcmhSZ8x" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094216; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=OfwkogOixf5GMwu+2dp6NmTbHKTViLblOsXgNerkr74=; b=HcmhSZ8xVgNnyLKrqJQMpFFAcwESQpuUg5lvl6ybmjABSKtutGSJ594Rtn5xMK012n1AVJCu6PFz7NeCc6NLL7M1mJgYkuh1sj6hTHYDgSqgKZa8XdnH7CQaJST6xlmVWo/EdFA/UqWAhuV/7GpnYQHGyqFn1AI4CFtirzxdsds= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032089153;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=30;SR=0;TI=SMTPD_---0X3647Lr_1779094212; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X3647Lr_1779094212 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:13 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com, Jonathan Cameron , Mauro Carvalho Chehab Subject: [PATCH v14 4/8] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage() Date: Mon, 18 May 2026 16:49:48 +0800 Message-ID: <20260518084956.2538442-5-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tong Tiangen Currently, copy_mc_[user]_highpage() returns zero on success, or in case of failures, the number of bytes that weren't copied. While tracking the number of not copied works fine for x86 and PPC, There are some difficulties in doing the same thing on ARM64 because there is no available caller-saved register in copy_page()(lib/copy_page.S) to save "bytes not copied", and the following copy_mc_page() will also encounter the same problem. Consider the caller of copy_mc_[user]_highpage() cannot do any processing on the remaining data(The page has hardware errors), they only check if copy was succeeded or not, make the interface more generic by using an error code when copy fails (-EFAULT) or return zero on success. Signed-off-by: Tong Tiangen Reviewed-by: Jonathan Cameron Reviewed-by: Mauro Carvalho Chehab --- include/linux/highmem.h | 8 ++++---- mm/khugepaged.c | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index af03db851a1d..18dc4aca4aa1 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -427,8 +427,8 @@ static inline void copy_highpage(struct page *to, struc= t page *from) /* * If architecture supports machine check exception handling, define the * #MC versions of copy_user_highpage and copy_highpage. They copy a memory - * page with #MC in source page (@from) handled, and return the number - * of bytes not copied if there was a #MC, otherwise 0 for success. + * page with #MC in source page (@from) handled, and return -EFAULT if the= re + * was a #MC, otherwise 0 for success. */ static inline int copy_mc_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma) @@ -447,7 +447,7 @@ static inline int copy_mc_user_highpage(struct page *to= , struct page *from, if (ret) memory_failure_queue(page_to_pfn(from), 0); =20 - return ret; + return ret ? -EFAULT : 0; } =20 static inline int copy_mc_highpage(struct page *to, struct page *from) @@ -466,7 +466,7 @@ static inline int copy_mc_highpage(struct page *to, str= uct page *from) if (ret) memory_failure_queue(page_to_pfn(from), 0); =20 - return ret; + return ret ? -EFAULT : 0; } #else static inline int copy_mc_user_highpage(struct page *to, struct page *from, diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b8452dbdb043..cf1b78eed3c3 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -810,7 +810,7 @@ static enum scan_result __collapse_huge_page_copy(pte_t= *pte, struct folio *foli continue; } src_page =3D pte_page(pteval); - if (copy_mc_user_highpage(page, src_page, src_addr, vma) > 0) { + if (copy_mc_user_highpage(page, src_page, src_addr, vma)) { result =3D SCAN_COPY_MC; break; } @@ -2143,7 +2143,7 @@ static enum scan_result collapse_file(struct mm_struc= t *mm, unsigned long addr, } =20 for (i =3D 0; i < nr_pages; i++) { - if (copy_mc_highpage(dst, folio_page(folio, i)) > 0) { + if (copy_mc_highpage(dst, folio_page(folio, i))) { result =3D SCAN_COPY_MC; goto rollback; } --=20 2.39.3 From nobody Mon May 25 05:55:43 2026 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D8D33DEAC7 for ; Mon, 18 May 2026 08:50:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094223; cv=none; b=Ujbd8riJ9OsoFz+QluN/vavxf85l6NLi4+bQbk2pHyU3xORtWC3RtcVSi0jDg+los9Scc8emnwddGfNWkdfB/s74aCqytjCNtg3P4NCRSm8NaNMQF0Ta7L8piHdNGUxMjOiMXFrjzB8MJuo/RW2CZYftU2ShYXeAE4SS/eIZswE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094223; c=relaxed/simple; bh=p9pWhsyE0K2omOY2mkfUoArzJ0FhQvOv/6kubwepvHY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=A4DXOY5uBgrsmLAVnpa8nB22dEjiTKV4fsNLM2Fe/2DGbmNXcjB2GlowJ3Rf0wQvB3+8g11wTrumCECaqXXjND7FltL1MxPqF+cAfTKPXZmP2XFw57J1PBhwPJmtatC/rF44NgIoZyhNHI1og6fsQDJ+GxKpqQ9XdGF85GjcPUk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Og2bnEu1; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Og2bnEu1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094218; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=1XKHXutbZ1417bPP+b3eJaiLcyB5W21y2+Eld3tQs3k=; b=Og2bnEu1vpu8lc22W8bI7X0vBGWVOkrmE3VTdLlI2VNbjQFnSBLVd2k567+UJzTTCCYqa2zuk/b1JT64y9AY2uESAXHdzr4fL3TCXa8WwfWA+8L+TcSR0DgfY4rihVkAsn2qHK01o9LF95xBCH0ATifaXOloCl/igu/+uIEJdLw= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R371e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045133197;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=28;SR=0;TI=SMTPD_---0X36F6or_1779094215; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X36F6or_1779094215 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:16 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com Subject: [PATCH v14 5/8] arm64: support copy_mc_[user]_highpage() Date: Mon, 18 May 2026 16:49:49 +0800 Message-ID: <20260518084956.2538442-6-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tong Tiangen Currently, many scenarios that can tolerate memory errors when copying page have been supported in the kernel[1~5], all of which are implemented by copy_mc_[user]_highpage(). arm64 should also support this mechanism. Due to mte, arm64 needs to have its own copy_mc_[user]_highpage() architecture implementation, macros __HAVE_ARCH_COPY_MC_HIGHPAGE and __HAVE_ARCH_COPY_MC_USER_HIGHPAGE have been added to control it. Add new helper copy_mc_page() which provide a page copy implementation with hardware memory error safe. The code logic of copy_mc_page() is the same as copy_page(), the main difference is that the ldp insn of copy_mc_page() contains the fixup type EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR, therefore, the main logic is extracted to copy_page_template.S. In addition, the fixup of MOPS insn is not considered at present. [Ruidong: add FEAT_MOPS support] [1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, tak= e page offline") [2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy= -on-write faults") [3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_nee= d_to_copy()") [4] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous me= mory") [5] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed = memory") Signed-off-by: Tong Tiangen Signed-off-by: Ruidong Tian --- arch/arm64/include/asm/mte.h | 9 ++++ arch/arm64/include/asm/page.h | 10 ++++ arch/arm64/lib/Makefile | 2 + arch/arm64/lib/copy_mc_page.S | 44 +++++++++++++++++ arch/arm64/lib/copy_page.S | 62 ++---------------------- arch/arm64/lib/copy_page_template.S | 71 +++++++++++++++++++++++++++ arch/arm64/lib/mte.S | 29 +++++++++++ arch/arm64/mm/copypage.c | 75 +++++++++++++++++++++++++++++ include/linux/highmem.h | 8 +++ 9 files changed, 253 insertions(+), 57 deletions(-) create mode 100644 arch/arm64/lib/copy_mc_page.S create mode 100644 arch/arm64/lib/copy_page_template.S diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 7f7b97e09996..a0b1757f4847 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -98,6 +98,11 @@ static inline bool try_page_mte_tagging(struct page *pag= e) void mte_zero_clear_page_tags(void *addr); void mte_sync_tags(pte_t pte, unsigned int nr_pages); void mte_copy_page_tags(void *kto, const void *kfrom); + +#ifdef CONFIG_ARCH_HAS_COPY_MC +int mte_copy_mc_page_tags(void *kto, const void *kfrom); +#endif + void mte_thread_init_user(void); void mte_thread_switch(struct task_struct *next); void mte_cpu_setup(void); @@ -134,6 +139,10 @@ static inline void mte_sync_tags(pte_t pte, unsigned i= nt nr_pages) static inline void mte_copy_page_tags(void *kto, const void *kfrom) { } +static inline int mte_copy_mc_page_tags(void *kto, const void *kfrom) +{ + return 0; +} static inline void mte_thread_init_user(void) { } diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index e25d0d18f6d7..f65818ee614a 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -29,6 +29,16 @@ void copy_user_highpage(struct page *to, struct page *fr= om, void copy_highpage(struct page *to, struct page *from); #define __HAVE_ARCH_COPY_HIGHPAGE =20 +#ifdef CONFIG_ARCH_HAS_COPY_MC +int copy_mc_page(void *to, const void *from); +int copy_mc_highpage(struct page *to, struct page *from); +#define __HAVE_ARCH_COPY_MC_HIGHPAGE + +int copy_mc_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma); +#define __HAVE_ARCH_COPY_MC_USER_HIGHPAGE +#endif + struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, unsigned long vaddr); #define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 448c917494f3..1f4c3f743a20 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -7,6 +7,8 @@ lib-y :=3D clear_user.o delay.o copy_from_user.o \ =20 lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) +=3D uaccess_flushcache.o =20 +lib-$(CONFIG_ARCH_HAS_COPY_MC) +=3D copy_mc_page.o + obj-$(CONFIG_FUNCTION_ERROR_INJECTION) +=3D error-inject.o =20 obj-$(CONFIG_ARM64_MTE) +=3D mte.o diff --git a/arch/arm64/lib/copy_mc_page.S b/arch/arm64/lib/copy_mc_page.S new file mode 100644 index 000000000000..ad1371e9e687 --- /dev/null +++ b/arch/arm64/lib/copy_mc_page.S @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * Copy a page from src to dest (both are page aligned) with memory error = safe + * + * Parameters: + * x0 - dest + * x1 - src + * Returns: + * x0 - Return 0 if copy success, or -EFAULT if anything goes wrong + * while copying. + */ + .macro ldp1 reg1, reg2, ptr, val + KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, [\ptr, \val]) + .endm + + .macro cpy1 dst, src, count + .arch_extension mops + USER_CPY(9998f, 0, cpyfprt [\dst]!, [\src]!, \count!) + USER_CPY(9998f, 0, cpyfmrt [\dst]!, [\src]!, \count!) + USER_CPY(9998f, 0, cpyfert [\dst]!, [\src]!, \count!) + .endm + +SYM_FUNC_START(__pi_copy_mc_page) +#include "copy_page_template.S" + + mov x0, #0 + ret + +9998: mov x0, #-EFAULT + ret + +SYM_FUNC_END(__pi_copy_mc_page) +SYM_FUNC_ALIAS(copy_mc_page, __pi_copy_mc_page) +EXPORT_SYMBOL(copy_mc_page) diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S index e6374e7e5511..d0186bbf99f1 100644 --- a/arch/arm64/lib/copy_page.S +++ b/arch/arm64/lib/copy_page.S @@ -17,65 +17,13 @@ * x0 - dest * x1 - src */ -SYM_FUNC_START(__pi_copy_page) -#ifdef CONFIG_AS_HAS_MOPS - .arch_extension mops -alternative_if_not ARM64_HAS_MOPS - b .Lno_mops -alternative_else_nop_endif - - mov x2, #PAGE_SIZE - cpypwn [x0]!, [x1]!, x2! - cpymwn [x0]!, [x1]!, x2! - cpyewn [x0]!, [x1]!, x2! - ret -.Lno_mops: -#endif - ldp x2, x3, [x1] - ldp x4, x5, [x1, #16] - ldp x6, x7, [x1, #32] - ldp x8, x9, [x1, #48] - ldp x10, x11, [x1, #64] - ldp x12, x13, [x1, #80] - ldp x14, x15, [x1, #96] - ldp x16, x17, [x1, #112] - - add x0, x0, #256 - add x1, x1, #128 -1: - tst x0, #(PAGE_SIZE - 1) =20 - stnp x2, x3, [x0, #-256] - ldp x2, x3, [x1] - stnp x4, x5, [x0, #16 - 256] - ldp x4, x5, [x1, #16] - stnp x6, x7, [x0, #32 - 256] - ldp x6, x7, [x1, #32] - stnp x8, x9, [x0, #48 - 256] - ldp x8, x9, [x1, #48] - stnp x10, x11, [x0, #64 - 256] - ldp x10, x11, [x1, #64] - stnp x12, x13, [x0, #80 - 256] - ldp x12, x13, [x1, #80] - stnp x14, x15, [x0, #96 - 256] - ldp x14, x15, [x1, #96] - stnp x16, x17, [x0, #112 - 256] - ldp x16, x17, [x1, #112] - - add x0, x0, #128 - add x1, x1, #128 - - b.ne 1b - - stnp x2, x3, [x0, #-256] - stnp x4, x5, [x0, #16 - 256] - stnp x6, x7, [x0, #32 - 256] - stnp x8, x9, [x0, #48 - 256] - stnp x10, x11, [x0, #64 - 256] - stnp x12, x13, [x0, #80 - 256] - stnp x14, x15, [x0, #96 - 256] - stnp x16, x17, [x0, #112 - 256] + .macro ldp1 reg1, reg2, ptr, val + ldp \reg1, \reg2, [\ptr, \val] + .endm =20 +SYM_FUNC_START(__pi_copy_page) +#include "copy_page_template.S" ret SYM_FUNC_END(__pi_copy_page) SYM_FUNC_ALIAS(copy_page, __pi_copy_page) diff --git a/arch/arm64/lib/copy_page_template.S b/arch/arm64/lib/copy_page= _template.S new file mode 100644 index 000000000000..d466b51c8ed9 --- /dev/null +++ b/arch/arm64/lib/copy_page_template.S @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2012 ARM Ltd. + */ + +/* + * Copy a page from src to dest (both are page aligned) + * + * Parameters: + * x0 - dest + * x1 - src + */ +dstin .req x0 +src .req x1 + +#ifdef CONFIG_AS_HAS_MOPS + .arch_extension mops +alternative_if_not ARM64_HAS_MOPS + b .Lno_mops +alternative_else_nop_endif + mov x2, #PAGE_SIZE + cpy1 dst, src, x2 + b .Lexitfunc +.Lno_mops: +#endif + + ldp1 x2, x3, x1, #0 + ldp1 x4, x5, x1, #16 + ldp1 x6, x7, x1, #32 + ldp1 x8, x9, x1, #48 + ldp1 x10, x11, x1, #64 + ldp1 x12, x13, x1, #80 + ldp1 x14, x15, x1, #96 + ldp1 x16, x17, x1, #112 + + add x0, x0, #256 + add x1, x1, #128 +1: + tst x0, #(PAGE_SIZE - 1) + + stnp x2, x3, [x0, #-256] + ldp1 x2, x3, x1, #0 + stnp x4, x5, [x0, #16 - 256] + ldp1 x4, x5, x1, #16 + stnp x6, x7, [x0, #32 - 256] + ldp1 x6, x7, x1, #32 + stnp x8, x9, [x0, #48 - 256] + ldp1 x8, x9, x1, #48 + stnp x10, x11, [x0, #64 - 256] + ldp1 x10, x11, x1, #64 + stnp x12, x13, [x0, #80 - 256] + ldp1 x12, x13, x1, #80 + stnp x14, x15, [x0, #96 - 256] + ldp1 x14, x15, x1, #96 + stnp x16, x17, [x0, #112 - 256] + ldp1 x16, x17, x1, #112 + + add x0, x0, #128 + add x1, x1, #128 + + b.ne 1b + + stnp x2, x3, [x0, #-256] + stnp x4, x5, [x0, #16 - 256] + stnp x6, x7, [x0, #32 - 256] + stnp x8, x9, [x0, #48 - 256] + stnp x10, x11, [x0, #64 - 256] + stnp x12, x13, [x0, #80 - 256] + stnp x14, x15, [x0, #96 - 256] + stnp x16, x17, [x0, #112 - 256] +.Lexitfunc: diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 5018ac03b6bf..9d4eeb76a838 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -80,6 +80,35 @@ SYM_FUNC_START(mte_copy_page_tags) ret SYM_FUNC_END(mte_copy_page_tags) =20 +#ifdef CONFIG_ARCH_HAS_COPY_MC +/* + * Copy the tags from the source page to the destination one with memory e= rror safe + * x0 - address of the destination page + * x1 - address of the source page + * Returns: + * x0 - Return 0 if copy success, or + * -EFAULT if anything goes wrong while copying. + */ +SYM_FUNC_START(mte_copy_mc_page_tags) + mov x2, x0 + mov x3, x1 + multitag_transfer_size x5, x6 +1: +KERNEL_MEM_ERR(2f, ldgm x4, [x3]) + stgm x4, [x2] + add x2, x2, x5 + add x3, x3, x5 + tst x2, #(PAGE_SIZE - 1) + b.ne 1b + + mov x0, #0 + ret + +2: mov x0, #-EFAULT + ret +SYM_FUNC_END(mte_copy_mc_page_tags) +#endif + /* * Read tags from a user buffer (one tag per byte) and set the correspondi= ng * tags at the given kernel address. Used by PTRACE_POKEMTETAGS. diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c index cd5912ba617b..9fd773baf17b 100644 --- a/arch/arm64/mm/copypage.c +++ b/arch/arm64/mm/copypage.c @@ -72,3 +72,78 @@ void copy_user_highpage(struct page *to, struct page *fr= om, flush_dcache_page(to); } EXPORT_SYMBOL_GPL(copy_user_highpage); + +#ifdef CONFIG_ARCH_HAS_COPY_MC +/* + * Return -EFAULT if anything goes wrong while copying page or mte. + */ +int copy_mc_highpage(struct page *to, struct page *from) +{ + void *kto =3D page_address(to); + void *kfrom =3D page_address(from); + struct folio *src =3D page_folio(from); + struct folio *dst =3D page_folio(to); + unsigned int i, nr_pages; + int ret; + + ret =3D copy_mc_page(kto, kfrom); + if (ret) + return -EFAULT; + + if (kasan_hw_tags_enabled()) + page_kasan_tag_reset(to); + + if (!system_supports_mte()) + return 0; + + if (folio_test_hugetlb(src)) { + if (!folio_test_hugetlb_mte_tagged(src) || + from !=3D folio_page(src, 0)) + return 0; + + WARN_ON_ONCE(!folio_try_hugetlb_mte_tagging(dst)); + + /* + * Populate tags for all subpages. + * + * Don't assume the first page is head page since + * huge page copy may start from any subpage. + */ + nr_pages =3D folio_nr_pages(src); + for (i =3D 0; i < nr_pages; i++) { + kfrom =3D page_address(folio_page(src, i)); + kto =3D page_address(folio_page(dst, i)); + ret =3D mte_copy_mc_page_tags(kto, kfrom); + if (ret) + return -EFAULT; + } + folio_set_hugetlb_mte_tagged(dst); + } else if (page_mte_tagged(from)) { + /* It's a new page, shouldn't have been tagged yet */ + WARN_ON_ONCE(!try_page_mte_tagging(to)); + + ret =3D mte_copy_mc_page_tags(kto, kfrom); + if (ret) + return -EFAULT; + set_page_mte_tagged(to); + } + + return 0; +} +EXPORT_SYMBOL(copy_mc_highpage); + +int copy_mc_user_highpage(struct page *to, struct page *from, + unsigned long vaddr, struct vm_area_struct *vma) +{ + int ret; + + ret =3D copy_mc_highpage(to, from); + if (ret) + return ret; + + flush_dcache_page(to); + + return 0; +} +EXPORT_SYMBOL_GPL(copy_mc_user_highpage); +#endif diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 18dc4aca4aa1..f168c9d4ad0e 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -424,6 +424,7 @@ static inline void copy_highpage(struct page *to, struc= t page *from) #endif =20 #ifdef copy_mc_to_kernel +#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE /* * If architecture supports machine check exception handling, define the * #MC versions of copy_user_highpage and copy_highpage. They copy a memory @@ -449,7 +450,9 @@ static inline int copy_mc_user_highpage(struct page *to= , struct page *from, =20 return ret ? -EFAULT : 0; } +#endif =20 +#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE static inline int copy_mc_highpage(struct page *to, struct page *from) { unsigned long ret; @@ -468,20 +471,25 @@ static inline int copy_mc_highpage(struct page *to, s= truct page *from) =20 return ret ? -EFAULT : 0; } +#endif #else +#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE static inline int copy_mc_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma) { copy_user_highpage(to, from, vaddr, vma); return 0; } +#endif =20 +#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE static inline int copy_mc_highpage(struct page *to, struct page *from) { copy_highpage(to, from); return 0; } #endif +#endif =20 static inline void memcpy_page(struct page *dst_page, size_t dst_off, struct page *src_page, size_t src_off, --=20 2.39.3 From nobody Mon May 25 05:55:43 2026 Received: from out30-98.freemail.mail.aliyun.com (out30-98.freemail.mail.aliyun.com [115.124.30.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EFD33E3167 for ; Mon, 18 May 2026 08:50:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.98 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094224; cv=none; b=OJYhTbMrAodvvv7ZDVBD/fvqy9F1Kb2NxWm/wPCLIvgBKlrRN0IaK/ULpk+rrZOgXSy2KMBK26+rHHAlLvq9sG9kyrIYv2bRqtHa1CFLxkoEhdIafljPYQXc8WUHtJ1o8v0iotphBj0XQ6a8XSdmOXwARxkAJH223VV/Ag1ml/k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094224; c=relaxed/simple; bh=bTYO4rqIz0w/9rl74JvuqUmOSSKuRxR0m+h9pb/queQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IqFk7zc670XTsJ/g3DhtpgojP2fQERfSN6/mAC35CCwENbx+0yes/rLI9rIEBOMh2tlpaN/u9pA6q3bJHOF9jgxd5y/FvwUdIFgXbixtLHE3rFLfFXC/WQkyvEwSPRdGPSDh6mjO0PXeeTkq5pjPDfTXRV/nxk0qO95HLJFcMhQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Vbl42/ll; arc=none smtp.client-ip=115.124.30.98 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Vbl42/ll" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094220; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=JfppSwEbo8CVHsJ/3tFLMi3O9oZ5mHSNY9AbWVhXA+M=; b=Vbl42/llSQ2EUXEwsMBx/H/utodf96qMtOvziDNl7WLhnVPLCMmDu5vUykNzMmeka77AphVz+ILyhr5AqjYLxuqJ/dhvt1tF7k0VYiJ4PeGV0PNs/z5L9b5YRBdYSzwzIqYOWBKNWQiT+ZYy7yvAlre7hsVG4xvVWFFFixeH7cE= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032089153;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=28;SR=0;TI=SMTPD_---0X3647Nb_1779094217; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X3647Nb_1779094217 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:19 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com Subject: [PATCH v14 6/8] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests Date: Mon, 18 May 2026 16:49:50 +0800 Message-ID: <20260518084956.2538442-7-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add KUnit tests for copy_page() and copy_mc_page(), modeled after the existing memcpy_test() style: a static page-aligned src and a two-page dst, filled with random bytes plus non-zero edges, then verify byte-for-byte equality and that the adjacent page is untouched. The copy_mc_page() case additionally checks the return value is 0 on clean memory and is gated on CONFIG_ARCH_HAS_COPY_MC. Signed-off-by: Ruidong Tian --- lib/tests/memcpy_kunit.c | 67 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c index d36933554e46..85df53ccfb0c 100644 --- a/lib/tests/memcpy_kunit.c +++ b/lib/tests/memcpy_kunit.c @@ -493,6 +493,67 @@ static void memmove_overlap_test(struct kunit *test) } } =20 +/* --- Page-sized copy tests --- */ + +static u8 page_src[PAGE_SIZE] __aligned(PAGE_SIZE); +static u8 page_dst[PAGE_SIZE * 2] __aligned(PAGE_SIZE); +static const u8 page_zero[PAGE_SIZE] __aligned(PAGE_SIZE); + +static void init_page(struct kunit *test) +{ + /* Get many bit patterns. */ + get_random_bytes(page_src, PAGE_SIZE); + + /* Make sure we have non-zero edges. */ + set_random_nonzero(test, &page_src[0]); + set_random_nonzero(test, &page_src[PAGE_SIZE - 1]); + + /* Explicitly zero the entire destination. */ + memset(page_dst, 0, ARRAY_SIZE(page_dst)); +} + +static void copy_page_test(struct kunit *test) +{ + init_page(test); + + /* Copy. */ + copy_page(page_dst, page_src); + + /* Verify byte-for-byte exact. */ + KUNIT_ASSERT_EQ_MSG(test, + memcmp(page_dst, page_src, PAGE_SIZE), 0, + "copy_page content mismatch with random data"); + + /* Verify no overflow into second page. */ + KUNIT_ASSERT_EQ_MSG(test, + memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0, + "copy_page overflow into adjacent page"); +} + +#ifdef CONFIG_ARCH_HAS_COPY_MC +static void copy_mc_page_test(struct kunit *test) +{ + int ret; + + init_page(test); + + /* Copy and check return value. */ + ret =3D copy_mc_page(page_dst, page_src); + KUNIT_ASSERT_EQ_MSG(test, ret, 0, + "copy_mc_page returned %d on clean memory", ret); + + /* Verify byte-for-byte exact. */ + KUNIT_ASSERT_EQ_MSG(test, + memcmp(page_dst, page_src, PAGE_SIZE), 0, + "copy_mc_page content mismatch with random data"); + + /* Verify no overflow into second page. */ + KUNIT_ASSERT_EQ_MSG(test, + memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0, + "copy_mc_page overflow into adjacent page"); +} +#endif /* CONFIG_ARCH_HAS_COPY_MC */ + static struct kunit_case memcpy_test_cases[] =3D { KUNIT_CASE(memset_test), KUNIT_CASE(memcpy_test), @@ -500,6 +561,10 @@ static struct kunit_case memcpy_test_cases[] =3D { KUNIT_CASE_SLOW(memmove_test), KUNIT_CASE_SLOW(memmove_large_test), KUNIT_CASE_SLOW(memmove_overlap_test), + KUNIT_CASE(copy_page_test), +#ifdef CONFIG_ARCH_HAS_COPY_MC + KUNIT_CASE(copy_mc_page_test), +#endif {} }; =20 @@ -510,5 +575,5 @@ static struct kunit_suite memcpy_test_suite =3D { =20 kunit_test_suite(memcpy_test_suite); =20 -MODULE_DESCRIPTION("test cases for memcpy(), memmove(), and memset()"); +MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_= page()"); MODULE_LICENSE("GPL"); --=20 2.39.3 From nobody Mon May 25 05:55:43 2026 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A0313E3D96 for ; Mon, 18 May 2026 08:50:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.110 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094234; cv=none; b=DYUXIJrA4/az6H5aIRV62Lj1n08xlRfgaseMjWI43VbLz5hV2lpNJdQfEl9WxbHCLnSgmFUzqZWzwqk5SrDHWbv+5SLPm8UYzcSRpY8cHF3r9+NSQ2RcuTm/Uxj5zVnjKXS8MGbSeMhbUb/n0bP/k74JbZ6YE3gIkbQaUc8EgTc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094234; c=relaxed/simple; bh=Ugk1pQg010Nxjcz1jy67Lxl/IBBRCWcf14/vg3uknpw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DA8PtRQ5zfc0wCme7GMjKG5+KCcpEvw/d9Sj+/McAQT9YJFTkf8Hg7aTabFPMo9hS8trKNwkEazju1ZJLmvpZRZavFo6Yg6qdTeg7zH8gaYdGzkTCamvVg6h+kSSCEpSUmmo0cOe6so1RwtFSl15jTAL67UmEzr0HBZkmIIIEw8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=XNXeCiil; arc=none smtp.client-ip=115.124.30.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="XNXeCiil" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094223; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=0XgoFNOITCWZNsicdGaSoSFLXre/X1GKvpXRqiN0JGs=; b=XNXeCiilhOw+pC7W70AGTd0gubm4j75r1+1mD++meY5mkwKmr4BVyiL7eFpaUYGTkP/+g6u40Qoj3J8b3uov5tUr+Sz77EV3rNyBw/r/WMfxG/1YSQkn2mHFgAQq79E4NOSYsCdbOmuZYw5Fr6q6ZnAK1Ia7GUXH74TQCngF9pY= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032089153;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=28;SR=0;TI=SMTPD_---0X3647Ob_1779094220; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X3647Ob_1779094220 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:22 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com Subject: [PATCH v14 7/8] arm64: introduce copy_mc_to_kernel() implementation Date: Mon, 18 May 2026 16:49:51 +0800 Message-ID: <20260518084956.2538442-8-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tong Tiangen The copy_mc_to_kernel() helper is memory copy implementation that handles source exceptions. It can be used in memory copy scenarios that tolerate hardware memory errors(e.g: pmem_read/dax_copy_to_iter). Currently, only x86 and ppc support this helper, Add this for ARM64 as well, if ARCH_HAS_COPY_MC is defined, by implementing copy_mc_to_kernel() and memcpy_mc() functions. Because there is no caller-saved GPR is available for saving "bytes not copied" in memcpy(), the memcpy_mc() is referenced to the implementation of copy_from_user(). In addition, the fixup of MOPS insn is not considered at present. [Ruidong: refactor memcpy_mc on top of the new memcpy implementation.] Signed-off-by: Tong Tiangen Signed-off-by: Ruidong Tian --- arch/arm64/include/asm/string.h | 5 + arch/arm64/include/asm/uaccess.h | 17 +++ arch/arm64/lib/Makefile | 2 +- arch/arm64/lib/memcpy.S | 253 +++---------------------------- arch/arm64/lib/memcpy_mc.S | 56 +++++++ arch/arm64/lib/memcpy_template.S | 249 ++++++++++++++++++++++++++++++ mm/kasan/shadow.c | 12 ++ 7 files changed, 359 insertions(+), 235 deletions(-) create mode 100644 arch/arm64/lib/memcpy_mc.S create mode 100644 arch/arm64/lib/memcpy_template.S diff --git a/arch/arm64/include/asm/string.h b/arch/arm64/include/asm/strin= g.h index 3a3264ff47b9..23eca4fb24fa 100644 --- a/arch/arm64/include/asm/string.h +++ b/arch/arm64/include/asm/string.h @@ -35,6 +35,10 @@ extern void *memchr(const void *, int, __kernel_size_t); extern void *memcpy(void *, const void *, __kernel_size_t); extern void *__memcpy(void *, const void *, __kernel_size_t); =20 +#define __HAVE_ARCH_MEMCPY_MC +extern int memcpy_mc(void *, const void *, __kernel_size_t); +extern int __memcpy_mc(void *, const void *, __kernel_size_t); + #define __HAVE_ARCH_MEMMOVE extern void *memmove(void *, const void *, __kernel_size_t); extern void *__memmove(void *, const void *, __kernel_size_t); @@ -57,6 +61,7 @@ void memcpy_flushcache(void *dst, const void *src, size_t= cnt); */ =20 #define memcpy(dst, src, len) __memcpy(dst, src, len) +#define memcpy_mc(dst, src, len) __memcpy_mc(dst, src, len) #define memmove(dst, src, len) __memmove(dst, src, len) #define memset(s, c, n) __memset(s, c, n) =20 diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uacc= ess.h index b0c83a08dda9..93277eca2268 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -499,5 +499,22 @@ static inline size_t probe_subpage_writeable(const cha= r __user *uaddr, } =20 #endif /* CONFIG_ARCH_HAS_SUBPAGE_FAULTS */ +#ifdef CONFIG_ARCH_HAS_COPY_MC +/** + * copy_mc_to_kernel - memory copy that handles source exceptions + * + * @to: destination address + * @from: source address + * @size: number of bytes to copy + * + * Return 0 for success, or bytes not copied. + */ +static inline unsigned long __must_check +copy_mc_to_kernel(void *to, const void *from, unsigned long size) +{ + return memcpy_mc(to, from, size); +} +#define copy_mc_to_kernel copy_mc_to_kernel +#endif =20 #endif /* __ASM_UACCESS_H */ diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 1f4c3f743a20..a5820e6c33d4 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -7,7 +7,7 @@ lib-y :=3D clear_user.o delay.o copy_from_user.o \ =20 lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) +=3D uaccess_flushcache.o =20 -lib-$(CONFIG_ARCH_HAS_COPY_MC) +=3D copy_mc_page.o +lib-$(CONFIG_ARCH_HAS_COPY_MC) +=3D copy_mc_page.o memcpy_mc.o =20 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) +=3D error-inject.o =20 diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S index 9b99106fb95f..ef6aea2de9b4 100644 --- a/arch/arm64/lib/memcpy.S +++ b/arch/arm64/lib/memcpy.S @@ -15,247 +15,32 @@ * */ =20 -#define L(label) .L ## label + .macro ldrb1 reg, addr:vararg + ldrb \reg, \addr + .endm =20 -#define dstin x0 -#define src x1 -#define count x2 -#define dst x3 -#define srcend x4 -#define dstend x5 -#define A_l x6 -#define A_lw w6 -#define A_h x7 -#define B_l x8 -#define B_lw w8 -#define B_h x9 -#define C_l x10 -#define C_lw w10 -#define C_h x11 -#define D_l x12 -#define D_h x13 -#define E_l x14 -#define E_h x15 -#define F_l x16 -#define F_h x17 -#define G_l count -#define G_h dst -#define H_l src -#define H_h srcend -#define tmp1 x14 + .macro ldr1 reg, addr:vararg + ldr \reg, \addr + .endm =20 -/* This implementation handles overlaps and supports both memcpy and memmo= ve - from a single entry point. It uses unaligned accesses and branchless - sequences to keep the code small, simple and improve performance. + .macro ldp1 reg1, reg2, addr:vararg + ldp \reg1, \reg2, \addr + .endm =20 - Copies are split into 3 main cases: small copies of up to 32 bytes, med= ium - copies of up to 128 bytes, and large copies. The overhead of the overl= ap - check is negligible since it is only required for large copies. + .macro ret1 + ret + .endm =20 - Large copies use a software pipelined loop processing 64 bytes per iter= ation. - The destination pointer is 16-byte aligned to minimize unaligned access= es. - The loop tail is handled by always copying 64 bytes from the end. -*/ + .macro cpy1 dst, src, count + .arch_extension mops + cpyp [\dst]!, [\src]!, \count! + cpym [\dst]!, [\src]!, \count! + cpye [\dst]!, [\src]!, \count! + .endm =20 -SYM_FUNC_START_LOCAL(__pi_memcpy_generic) - add srcend, src, count - add dstend, dstin, count - cmp count, 128 - b.hi L(copy_long) - cmp count, 32 - b.hi L(copy32_128) - - /* Small copies: 0..32 bytes. */ - cmp count, 16 - b.lo L(copy16) - ldp A_l, A_h, [src] - ldp D_l, D_h, [srcend, -16] - stp A_l, A_h, [dstin] - stp D_l, D_h, [dstend, -16] - ret - - /* Copy 8-15 bytes. */ -L(copy16): - tbz count, 3, L(copy8) - ldr A_l, [src] - ldr A_h, [srcend, -8] - str A_l, [dstin] - str A_h, [dstend, -8] - ret - - .p2align 3 - /* Copy 4-7 bytes. */ -L(copy8): - tbz count, 2, L(copy4) - ldr A_lw, [src] - ldr B_lw, [srcend, -4] - str A_lw, [dstin] - str B_lw, [dstend, -4] - ret - - /* Copy 0..3 bytes using a branchless sequence. */ -L(copy4): - cbz count, L(copy0) - lsr tmp1, count, 1 - ldrb A_lw, [src] - ldrb C_lw, [srcend, -1] - ldrb B_lw, [src, tmp1] - strb A_lw, [dstin] - strb B_lw, [dstin, tmp1] - strb C_lw, [dstend, -1] -L(copy0): - ret - - .p2align 4 - /* Medium copies: 33..128 bytes. */ -L(copy32_128): - ldp A_l, A_h, [src] - ldp B_l, B_h, [src, 16] - ldp C_l, C_h, [srcend, -32] - ldp D_l, D_h, [srcend, -16] - cmp count, 64 - b.hi L(copy128) - stp A_l, A_h, [dstin] - stp B_l, B_h, [dstin, 16] - stp C_l, C_h, [dstend, -32] - stp D_l, D_h, [dstend, -16] - ret - - .p2align 4 - /* Copy 65..128 bytes. */ -L(copy128): - ldp E_l, E_h, [src, 32] - ldp F_l, F_h, [src, 48] - cmp count, 96 - b.ls L(copy96) - ldp G_l, G_h, [srcend, -64] - ldp H_l, H_h, [srcend, -48] - stp G_l, G_h, [dstend, -64] - stp H_l, H_h, [dstend, -48] -L(copy96): - stp A_l, A_h, [dstin] - stp B_l, B_h, [dstin, 16] - stp E_l, E_h, [dstin, 32] - stp F_l, F_h, [dstin, 48] - stp C_l, C_h, [dstend, -32] - stp D_l, D_h, [dstend, -16] - ret - - .p2align 4 - /* Copy more than 128 bytes. */ -L(copy_long): - /* Use backwards copy if there is an overlap. */ - sub tmp1, dstin, src - cbz tmp1, L(copy0) - cmp tmp1, count - b.lo L(copy_long_backwards) - - /* Copy 16 bytes and then align dst to 16-byte alignment. */ - - ldp D_l, D_h, [src] - and tmp1, dstin, 15 - bic dst, dstin, 15 - sub src, src, tmp1 - add count, count, tmp1 /* Count is now 16 too large. */ - ldp A_l, A_h, [src, 16] - stp D_l, D_h, [dstin] - ldp B_l, B_h, [src, 32] - ldp C_l, C_h, [src, 48] - ldp D_l, D_h, [src, 64]! - subs count, count, 128 + 16 /* Test and readjust count. */ - b.ls L(copy64_from_end) - -L(loop64): - stp A_l, A_h, [dst, 16] - ldp A_l, A_h, [src, 16] - stp B_l, B_h, [dst, 32] - ldp B_l, B_h, [src, 32] - stp C_l, C_h, [dst, 48] - ldp C_l, C_h, [src, 48] - stp D_l, D_h, [dst, 64]! - ldp D_l, D_h, [src, 64]! - subs count, count, 64 - b.hi L(loop64) - - /* Write the last iteration and copy 64 bytes from the end. */ -L(copy64_from_end): - ldp E_l, E_h, [srcend, -64] - stp A_l, A_h, [dst, 16] - ldp A_l, A_h, [srcend, -48] - stp B_l, B_h, [dst, 32] - ldp B_l, B_h, [srcend, -32] - stp C_l, C_h, [dst, 48] - ldp C_l, C_h, [srcend, -16] - stp D_l, D_h, [dst, 64] - stp E_l, E_h, [dstend, -64] - stp A_l, A_h, [dstend, -48] - stp B_l, B_h, [dstend, -32] - stp C_l, C_h, [dstend, -16] - ret - - .p2align 4 - - /* Large backwards copy for overlapping copies. - Copy 16 bytes and then align dst to 16-byte alignment. */ -L(copy_long_backwards): - ldp D_l, D_h, [srcend, -16] - and tmp1, dstend, 15 - sub srcend, srcend, tmp1 - sub count, count, tmp1 - ldp A_l, A_h, [srcend, -16] - stp D_l, D_h, [dstend, -16] - ldp B_l, B_h, [srcend, -32] - ldp C_l, C_h, [srcend, -48] - ldp D_l, D_h, [srcend, -64]! - sub dstend, dstend, tmp1 - subs count, count, 128 - b.ls L(copy64_from_start) - -L(loop64_backwards): - stp A_l, A_h, [dstend, -16] - ldp A_l, A_h, [srcend, -16] - stp B_l, B_h, [dstend, -32] - ldp B_l, B_h, [srcend, -32] - stp C_l, C_h, [dstend, -48] - ldp C_l, C_h, [srcend, -48] - stp D_l, D_h, [dstend, -64]! - ldp D_l, D_h, [srcend, -64]! - subs count, count, 64 - b.hi L(loop64_backwards) - - /* Write the last iteration and copy 64 bytes from the start. */ -L(copy64_from_start): - ldp G_l, G_h, [src, 48] - stp A_l, A_h, [dstend, -16] - ldp A_l, A_h, [src, 32] - stp B_l, B_h, [dstend, -32] - ldp B_l, B_h, [src, 16] - stp C_l, C_h, [dstend, -48] - ldp C_l, C_h, [src] - stp D_l, D_h, [dstend, -64] - stp G_l, G_h, [dstin, 48] - stp A_l, A_h, [dstin, 32] - stp B_l, B_h, [dstin, 16] - stp C_l, C_h, [dstin] - ret -SYM_FUNC_END(__pi_memcpy_generic) - -#ifdef CONFIG_AS_HAS_MOPS - .arch_extension mops SYM_FUNC_START(__pi_memcpy) -alternative_if_not ARM64_HAS_MOPS - b __pi_memcpy_generic -alternative_else_nop_endif - - mov dst, dstin - cpyp [dst]!, [src]!, count! - cpym [dst]!, [src]!, count! - cpye [dst]!, [src]!, count! - ret +#include "memcpy_template.S" SYM_FUNC_END(__pi_memcpy) -#else -SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic) -#endif =20 SYM_FUNC_ALIAS(__memcpy, __pi_memcpy) EXPORT_SYMBOL(__memcpy) diff --git a/arch/arm64/lib/memcpy_mc.S b/arch/arm64/lib/memcpy_mc.S new file mode 100644 index 000000000000..90624d35af4b --- /dev/null +++ b/arch/arm64/lib/memcpy_mc.S @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2012-2021, Arm Limited. + * + * Adapted from the original at: + * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d922= 9/string/aarch64/memcpy.S + */ + +#include +#include +#include + +/* Assumptions: + * + * ARMv8-a, AArch64, unaligned accesses. + * + */ + + .macro ldrb1 reg, addr:vararg + KERNEL_MEM_ERR(9998f, ldrb \reg, \addr) + .endm + + .macro ldr1 reg, addr:vararg + KERNEL_MEM_ERR(9998f, ldr \reg, \addr) + .endm + + .macro ldp1 reg1, reg2, addr:vararg + KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, \addr) + .endm + + .macro ret1 + mov x0, #0 + ret + .endm + + .macro cpy1 dst, src, count + .arch_extension mops + USER_CPY(9998f, 0, cpyp [\dst]!, [\src]!, \count!) + USER_CPY(9996f, 0, cpym [\dst]!, [\src]!, \count!) + USER_CPY(9996f, 0, cpye [\dst]!, [\src]!, \count!) + .endm + +SYM_FUNC_START(__memcpy_mc) +#include "memcpy_template.S" + + // Exception fixups +9996: b.cs 9998f + // Registers are in Option A format + add dst, dst, count +9998: sub x0, dstend, dstin // bytes not copied + ret +SYM_FUNC_END(__memcpy_mc) + +EXPORT_SYMBOL(__memcpy_mc) +SYM_FUNC_ALIAS_WEAK(memcpy_mc, __memcpy_mc) +EXPORT_SYMBOL(memcpy_mc) diff --git a/arch/arm64/lib/memcpy_template.S b/arch/arm64/lib/memcpy_templ= ate.S new file mode 100644 index 000000000000..205516c6e076 --- /dev/null +++ b/arch/arm64/lib/memcpy_template.S @@ -0,0 +1,249 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2012-2021, Arm Limited. + * + * Adapted from the original at: + * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d922= 9/string/aarch64/memcpy.S + */ + +#include +#include + +/* Assumptions: + * + * ARMv8-a, AArch64, unaligned accesses. + * + */ + +#define L(label) .L ## label + +#define dstin x0 +#define src x1 +#define count x2 +#define dst x3 +#define srcend x4 +#define dstend x5 +#define A_l x6 +#define A_lw w6 +#define A_h x7 +#define B_l x8 +#define B_lw w8 +#define B_h x9 +#define C_l x10 +#define C_lw w10 +#define C_h x11 +#define D_l x12 +#define D_h x13 +#define E_l x14 +#define E_h x15 +#define F_l x16 +#define F_h x17 +#define G_l count +#define G_h dst +#define H_l src +#define H_h srcend +#define tmp1 x14 + +/* This implementation handles overlaps and supports both memcpy and memmo= ve + from a single entry point. It uses unaligned accesses and branchless + sequences to keep the code small, simple and improve performance. + + Copies are split into 3 main cases: small copies of up to 32 bytes, med= ium + copies of up to 128 bytes, and large copies. The overhead of the overl= ap + check is negligible since it is only required for large copies. + + Large copies use a software pipelined loop processing 64 bytes per iter= ation. + The destination pointer is 16-byte aligned to minimize unaligned access= es. + The loop tail is handled by always copying 64 bytes from the end. +*/ + +#ifdef CONFIG_AS_HAS_MOPS +alternative_if_not ARM64_HAS_MOPS + b L(no_mops): +alternative_else_nop_endif + + cpy1 dst, src, count + ret1 +#endif + +L(no_mops): + add srcend, src, count + add dstend, dstin, count + cmp count, 128 + b.hi L(copy_long) + cmp count, 32 + b.hi L(copy32_128) + + /* Small copies: 0..32 bytes. */ + cmp count, 16 + b.lo L(copy16) + ldp1 A_l, A_h, [src] + ldp1 D_l, D_h, [srcend, -16] + stp A_l, A_h, [dstin] + stp D_l, D_h, [dstend, -16] + ret1 + + /* Copy 8-15 bytes. */ +L(copy16): + tbz count, 3, L(copy8) + ldr1 A_l, [src] + ldr1 A_h, [srcend, -8] + str A_l, [dstin] + str A_h, [dstend, -8] + ret1 + + .p2align 3 + /* Copy 4-7 bytes. */ +L(copy8): + tbz count, 2, L(copy4) + ldr1 A_lw, [src] + ldr1 B_lw, [srcend, -4] + str A_lw, [dstin] + str B_lw, [dstend, -4] + ret1 + + /* Copy 0..3 bytes using a branchless sequence. */ +L(copy4): + cbz count, L(copy0) + lsr tmp1, count, 1 + ldrb1 A_lw, [src] + ldrb1 C_lw, [srcend, -1] + ldrb1 B_lw, [src, tmp1] + strb A_lw, [dstin] + strb B_lw, [dstin, tmp1] + strb C_lw, [dstend, -1] +L(copy0): + ret1 + + .p2align 4 + /* Medium copies: 33..128 bytes. */ +L(copy32_128): + ldp1 A_l, A_h, [src] + ldp1 B_l, B_h, [src, 16] + ldp1 C_l, C_h, [srcend, -32] + ldp1 D_l, D_h, [srcend, -16] + cmp count, 64 + b.hi L(copy128) + stp A_l, A_h, [dstin] + stp B_l, B_h, [dstin, 16] + stp C_l, C_h, [dstend, -32] + stp D_l, D_h, [dstend, -16] + ret1 + + .p2align 4 + /* Copy 65..128 bytes. */ +L(copy128): + ldp1 E_l, E_h, [src, 32] + ldp1 F_l, F_h, [src, 48] + cmp count, 96 + b.ls L(copy96) + ldp1 G_l, G_h, [srcend, -64] + ldp1 H_l, H_h, [srcend, -48] + stp G_l, G_h, [dstend, -64] + stp H_l, H_h, [dstend, -48] +L(copy96): + stp A_l, A_h, [dstin] + stp B_l, B_h, [dstin, 16] + stp E_l, E_h, [dstin, 32] + stp F_l, F_h, [dstin, 48] + stp C_l, C_h, [dstend, -32] + stp D_l, D_h, [dstend, -16] + ret1 + + .p2align 4 + /* Copy more than 128 bytes. */ +L(copy_long): + /* Use backwards copy if there is an overlap. */ + sub tmp1, dstin, src + cbz tmp1, L(copy0) + cmp tmp1, count + b.lo L(copy_long_backwards) + + /* Copy 16 bytes and then align dst to 16-byte alignment. */ + + ldp1 D_l, D_h, [src] + and tmp1, dstin, 15 + bic dst, dstin, 15 + sub src, src, tmp1 + add count, count, tmp1 /* Count is now 16 too large. */ + ldp1 A_l, A_h, [src, 16] + stp D_l, D_h, [dstin] + ldp1 B_l, B_h, [src, 32] + ldp1 C_l, C_h, [src, 48] + ldp1 D_l, D_h, [src, 64]! + subs count, count, 128 + 16 /* Test and readjust count. */ + b.ls L(copy64_from_end) + +L(loop64): + stp A_l, A_h, [dst, 16] + ldp1 A_l, A_h, [src, 16] + stp B_l, B_h, [dst, 32] + ldp1 B_l, B_h, [src, 32] + stp C_l, C_h, [dst, 48] + ldp1 C_l, C_h, [src, 48] + stp D_l, D_h, [dst, 64]! + ldp1 D_l, D_h, [src, 64]! + subs count, count, 64 + b.hi L(loop64) + + /* Write the last iteration and copy 64 bytes from the end. */ +L(copy64_from_end): + ldp1 E_l, E_h, [srcend, -64] + stp A_l, A_h, [dst, 16] + ldp1 A_l, A_h, [srcend, -48] + stp B_l, B_h, [dst, 32] + ldp1 B_l, B_h, [srcend, -32] + stp C_l, C_h, [dst, 48] + ldp1 C_l, C_h, [srcend, -16] + stp D_l, D_h, [dst, 64] + stp E_l, E_h, [dstend, -64] + stp A_l, A_h, [dstend, -48] + stp B_l, B_h, [dstend, -32] + stp C_l, C_h, [dstend, -16] + ret1 + + .p2align 4 + + /* Large backwards copy for overlapping copies. + Copy 16 bytes and then align dst to 16-byte alignment. */ +L(copy_long_backwards): + ldp1 D_l, D_h, [srcend, -16] + and tmp1, dstend, 15 + sub srcend, srcend, tmp1 + sub count, count, tmp1 + ldp1 A_l, A_h, [srcend, -16] + stp D_l, D_h, [dstend, -16] + ldp1 B_l, B_h, [srcend, -32] + ldp1 C_l, C_h, [srcend, -48] + ldp1 D_l, D_h, [srcend, -64]! + sub dstend, dstend, tmp1 + subs count, count, 128 + b.ls L(copy64_from_start) + +L(loop64_backwards): + stp A_l, A_h, [dstend, -16] + ldp1 A_l, A_h, [srcend, -16] + stp B_l, B_h, [dstend, -32] + ldp1 B_l, B_h, [srcend, -32] + stp C_l, C_h, [dstend, -48] + ldp1 C_l, C_h, [srcend, -48] + stp D_l, D_h, [dstend, -64]! + ldp1 D_l, D_h, [srcend, -64]! + subs count, count, 64 + b.hi L(loop64_backwards) + + /* Write the last iteration and copy 64 bytes from the start. */ +L(copy64_from_start): + ldp1 G_l, G_h, [src, 48] + stp A_l, A_h, [dstend, -16] + ldp1 A_l, A_h, [src, 32] + stp B_l, B_h, [dstend, -32] + ldp1 B_l, B_h, [src, 16] + stp C_l, C_h, [dstend, -48] + ldp1 C_l, C_h, [src] + stp D_l, D_h, [dstend, -64] + stp G_l, G_h, [dstin, 48] + stp A_l, A_h, [dstin, 32] + stp B_l, B_h, [dstin, 16] + stp C_l, C_h, [dstin] + ret1 diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index d286e0a04543..3128f0d9cc46 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -79,6 +79,18 @@ void *memcpy(void *dest, const void *src, size_t len) } #endif =20 +#ifdef __HAVE_ARCH_MEMCPY_MC +#undef memcpy_mc +int memcpy_mc(void *dest, const void *src, size_t len) +{ + if (!kasan_check_range(src, len, false, _RET_IP_) || + !kasan_check_range(dest, len, true, _RET_IP_)) + return (int)len; + + return __memcpy_mc(dest, src, len); +} +#endif + void *__asan_memset(void *addr, int c, ssize_t len) { if (!kasan_check_range(addr, len, true, _RET_IP_)) --=20 2.39.3 From nobody Mon May 25 05:55:43 2026 Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 257893C4562 for ; Mon, 18 May 2026 08:50:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.118 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094231; cv=none; b=F7gZR0IM5krcalfx2O00ZAJq5GM06IGYGPRELsKdT4O0EYo9zHiQENgH8+Ked7yRWvtFuN5Feu8FEsLOVqiCDZ9+5JoseTZ1laHlwmuqu7ZkaAMRcg/3gmtB5jD02L0CGhMrJ5WNDvM/zwZqYMim2NtibOHOx+iH4gAioXO48jk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779094231; c=relaxed/simple; bh=DOVBnHCxu7YKSTfrlpBII4fuOvBLEdzNgSVfMLDoGRs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HJGbsD0fzN6cF37K1zxQ0XOwaibn+62RHmMpIFGYkyBH32lxBSulD4w24mT1BClEr1o+BFayvR8hqytzmx/1ccsYbbHu9zlUcoUxzqWAWbPE+3GWQYvyOW3uWHP4jJWzA2OACpADk6Ii2e26XHyED9uziTyB5YOd2tk00Z1HqSY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=uJoZGfGR; arc=none smtp.client-ip=115.124.30.118 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="uJoZGfGR" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779094227; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=4ZlFlL/ldqFxlJ/mCuoKEqDloBHgz/6+XLcrhP5/h8w=; b=uJoZGfGR2uabJ0rfA3C8SFKfpGjjGGUQuFOZlnA8fZcspelZrbuRcVnJ3RI4zeamr2LYLH4AZPW9DSJxXcUoTjCtT4R+pCcSI0S/l1pcw2eJji3+Y9Tu5YbsprOcziR2kj0GcU6QxNoOz00AGfukC30ObrQeOy9GdvzLa4pctV8= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R211e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037009110;MF=tianruidong@linux.alibaba.com;NM=1;PH=DS;RN=28;SR=0;TI=SMTPD_---0X365dBE_1779094223; Received: from localhost(mailfrom:tianruidong@linux.alibaba.com fp:SMTPD_---0X365dBE_1779094223 cluster:ay36) by smtp.aliyun-inc.com; Mon, 18 May 2026 16:50:25 +0800 From: Ruidong Tian To: catalin.marinas@arm.com, will@kernel.org, rafael@kernel.org, tony.luck@intel.com, guohanjun@huawei.com, mchehab@kernel.org, xueshuai@linux.alibaba.com, tongtiangen@huawei.com, james.morse@arm.com, robin.murphy@arm.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, mpe@ellerman.id.au, npiggin@gmail.com, ryabinin.a.a@gmail.com, glider@google.com, christophe.leroy@csgroup.eu, aneesh.kumar@kernel.org, naveen.n.rao@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, tianruidong@linux.alibaba.com Subject: [PATCH v14 8/8] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test Date: Mon, 18 May 2026 16:49:52 +0800 Message-ID: <20260518084956.2538442-9-tianruidong@linux.alibaba.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> References: <20260518084956.2538442-1-tianruidong@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memcpy_mc() is the Machine-Check safe memcpy variant that returns the number of bytes NOT copied on a hardware memory error, or 0 on success. Add two test cases modeled after the existing memcpy_test() and memcpy_large_test() implementations: Signed-off-by: Ruidong Tian --- lib/tests/memcpy_kunit.c | 113 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 112 insertions(+), 1 deletion(-) diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c index 85df53ccfb0c..b4b2dafb50f1 100644 --- a/lib/tests/memcpy_kunit.c +++ b/lib/tests/memcpy_kunit.c @@ -552,6 +552,115 @@ static void copy_mc_page_test(struct kunit *test) memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0, "copy_mc_page overflow into adjacent page"); } +/* + * memcpy_mc() is a Machine-Check safe memcpy variant. + * Signature: int memcpy_mc(void *dst, const void *src, size_t len) + * Returns: 0 on success, or number of bytes NOT copied on MC error. + * + * In the normal (no-poison) path it must behave identically to memcpy() + * and always return 0. + */ +static void memcpy_mc_test(struct kunit *test) +{ +#define TEST_OP "memcpy_mc" + struct some_bytes control =3D { + .data =3D { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + }, + }; + struct some_bytes zero =3D { }; + struct some_bytes middle =3D { + .data =3D { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + }, + }; + struct some_bytes three =3D { + .data =3D { 0x00, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + }, + }; + struct some_bytes dest =3D { }; + int ret, count; + u8 *ptr; + + /* Verify static initializers. */ + check(control, 0x20); + check(zero, 0); + compare("static initializers", dest, zero); + + /* Verify assignment. */ + dest =3D control; + compare("direct assignment", dest, control); + + /* Verify complete overwrite. */ + ret =3D memcpy_mc(dest.data, control.data, sizeof(dest.data)); + KUNIT_ASSERT_EQ(test, ret, 0); + compare("complete overwrite", dest, control); + + /* Verify middle overwrite: 7 bytes at offset 12. */ + dest =3D control; + ret =3D memcpy_mc(dest.data + 12, zero.data, 7); + KUNIT_ASSERT_EQ(test, ret, 0); + compare("middle overwrite", dest, middle); + + /* Verify zero-length copy is a no-op. */ + dest =3D control; + ret =3D memcpy_mc(dest.data, zero.data, 0); + KUNIT_ASSERT_EQ(test, ret, 0); + compare("zero length", dest, control); + + /* Verify argument side-effects aren't repeated. */ + dest =3D control; + ptr =3D dest.data; + count =3D 1; + memcpy(ptr++, zero.data, count++); + ptr +=3D 8; + memcpy(ptr++, zero.data, count++); + compare("argument side-effects", dest, three); +#undef TEST_OP +} + +static void memcpy_mc_large_test(struct kunit *test) +{ + init_large(test); + + /* Sweep 1..1024 bytes x shifting offset to cover all template paths. */ + for (int bytes =3D 1; bytes <=3D ARRAY_SIZE(large_src); bytes++) { + for (int offset =3D 0; offset < ARRAY_SIZE(large_src); offset++) { + int right_zero_pos =3D offset + bytes; + int right_zero_size =3D ARRAY_SIZE(large_dst) - right_zero_pos; + int ret; + + ret =3D memcpy_mc(large_dst + offset, large_src, bytes); + KUNIT_ASSERT_EQ_MSG(test, ret, 0, + "memcpy_mc returned %d with size %d at offset %d", + ret, bytes, offset); + + /* No write before copy area. */ + KUNIT_ASSERT_EQ_MSG(test, + memcmp(large_dst, large_zero, offset), 0, + "with size %d at offset %d", bytes, offset); + /* No write after copy area. */ + KUNIT_ASSERT_EQ_MSG(test, + memcmp(&large_dst[right_zero_pos], large_zero, + right_zero_size), 0, + "with size %d at offset %d", bytes, offset); + /* Byte-for-byte exact. */ + KUNIT_ASSERT_EQ_MSG(test, + memcmp(large_dst + offset, large_src, bytes), 0, + "with size %d at offset %d", bytes, offset); + + memset(large_dst + offset, 0, bytes); + } + cond_resched(); + } +} #endif /* CONFIG_ARCH_HAS_COPY_MC */ =20 static struct kunit_case memcpy_test_cases[] =3D { @@ -564,6 +673,8 @@ static struct kunit_case memcpy_test_cases[] =3D { KUNIT_CASE(copy_page_test), #ifdef CONFIG_ARCH_HAS_COPY_MC KUNIT_CASE(copy_mc_page_test), + KUNIT_CASE(memcpy_mc_test), + KUNIT_CASE_SLOW(memcpy_mc_large_test), #endif {} }; @@ -575,5 +686,5 @@ static struct kunit_suite memcpy_test_suite =3D { =20 kunit_test_suite(memcpy_test_suite); =20 -MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_= page()"); +MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset(), copy_pag= e() and memcpy_mc()"); MODULE_LICENSE("GPL"); --=20 2.39.3