From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9D41A36B90D; Mon, 24 Aug 2026 16:05:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587554; cv=none; b=ECKQVoOoAQNpKAoiAasmwKgkPzyDI7/7I7o4c4sY03cB5s+SmDDG1NO3VcfITeKhRYuxvBp3YNKXSiw72d2QvwTz0oPnJE7vj3wn+w7ex4zIk5xfHjRM0wEWeW0uSiwJhc8IEnggeBywhnjUtxJPB0dQHcSTaSrQ9DFMiAKbOtI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587554; c=relaxed/simple; bh=z3HgR5JrV87UUef5omwF+7GRZHLLMVnWFEywPKlYZN8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N/sj/DoyaQb47X3AbYuArKAePMPcZUflJ7O7nJCst1rzJP/98iu4MzpWVrSMysf4u8NuK1W/eDVd73uGDsCm3WGJh2d5nNPLGKtuKHPJaJ7sT/4pCH32hIWfG2pq2C2z2wRO7p6/AVeUNet/Obe0xyCfi/F0EXtUZ8ikfmBehDI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=MzfdHSGU; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="MzfdHSGU" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C63FC1756; Mon, 24 Aug 2026 09:05:46 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 67FFE3F85F; Mon, 24 Aug 2026 09:05:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587550; bh=z3HgR5JrV87UUef5omwF+7GRZHLLMVnWFEywPKlYZN8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MzfdHSGUI4MR4hpSZEjGYjwit9tca6UiDVjwbEHbeT9dUPE7UHQ86d+HRFpo98vex uwdglyCvZ9EqcSEXQhaNrCFd3FdwW9vV37RTPRQ7R85BMRMoAFvajoPp1TuM/0tmJ7 7w742yIUkmItIqYVPtE8QtBTxCTTT7GLQC3vNQVc= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 1/7] arm64: uaccess: Add batched kernel nofault accessors Date: Mon, 24 Aug 2026 17:04:46 +0100 Message-ID: <20260824160523.3907021-2-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With Hardware Tag-Based KASAN in asynchronous or asymmetric mode, arm64 sets and clears PSTATE.TCO around every kernel nofault load or store. A loop pays that cost for every access even though tag checking can stay disabled until the operation finishes. Separate TCO management from the fault-tolerant access and add begin and end hooks for callers that want to batch several accesses. Keep the existing accessors self-contained, and provide aliases and no-op hooks for architectures that do not need special handling. A context switch re-enables tag checking, so a batched region must not schedule. Continue to evaluate accessor arguments before overriding TCO, as those expressions may block. Signed-off-by: Muhammad Usama Anjum --- arch/arm64/include/asm/uaccess.h | 71 ++++++++++++++++++++++++-------- include/linux/uaccess.h | 16 +++++++ 2 files changed, 69 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uacc= ess.h index 9f5bd9c69c249..1a14eb2a51dce 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -270,28 +270,43 @@ do { \ #define get_user __get_user =20 /* - * We must not call into the scheduler between __mte_enable_tco_async() and - * __mte_disable_tco_async(). As `dst` and `src` may contain blocking - * functions, we must evaluate these outside of the critical section. + * Nofault load without TCO management for use inside a + * __begin/__end_kernel_nofault_bare() region. */ -#define __get_kernel_nofault(dst, src, type, err_label) \ +#define __get_kernel_nofault_bare(dst, src, type, err_label) \ do { \ __typeof__(dst) __gkn_dst =3D (dst); \ __typeof__(src) __gkn_src =3D (src); \ do { \ __label__ __gkn_label; \ - \ - __mte_enable_tco_async(); \ __raw_get_mem("ldr", *((type *)(__gkn_dst)), \ (__force type *)(__gkn_src), __gkn_label, K); \ - __mte_disable_tco_async(); \ break; \ __gkn_label: \ - __mte_disable_tco_async(); \ goto err_label; \ } while (0); \ } while (0) =20 +/* + * We must not call into the scheduler between __mte_enable_tco_async() and + * __mte_disable_tco_async(). As dst and src may contain blocking function= s, + * evaluate them before overriding TCO. + */ +#define __get_kernel_nofault(dst, src, type, err_label) \ +do { \ + __label__ __gkn_tco_err; \ + __typeof__(dst) __gkn_tco_dst =3D (dst); \ + __typeof__(src) __gkn_tco_src =3D (src); \ + __mte_enable_tco_async(); \ + __get_kernel_nofault_bare(__gkn_tco_dst, __gkn_tco_src, type, \ + __gkn_tco_err); \ + __mte_disable_tco_async(); \ + break; \ +__gkn_tco_err: \ + __mte_disable_tco_async(); \ + goto err_label; \ +} while (0) + #define __put_mem_asm(store, reg, x, addr, label, type) \ asm goto( \ "1: " store " " reg "0, [%1]\n" \ @@ -366,28 +381,48 @@ do { \ =20 #define put_user __put_user =20 -/* - * We must not call into the scheduler between __mte_enable_tco_async() and - * __mte_disable_tco_async(). As `dst` and `src` may contain blocking - * functions, we must evaluate these outside of the critical section. - */ -#define __put_kernel_nofault(dst, src, type, err_label) \ +/* Nofault store without TCO management; see __get_kernel_nofault_bare. */ +#define __put_kernel_nofault_bare(dst, src, type, err_label) \ do { \ __typeof__(dst) __pkn_dst =3D (dst); \ __typeof__(src) __pkn_src =3D (src); \ \ do { \ __label__ __pkn_err; \ - __mte_enable_tco_async(); \ __raw_put_mem("str", *((type *)(__pkn_src)), \ (__force type *)(__pkn_dst), __pkn_err, K); \ - __mte_disable_tco_async(); \ break; \ __pkn_err: \ - __mte_disable_tco_async(); \ goto err_label; \ } while (0); \ -} while(0) +} while (0) + +/* + * We must not call into the scheduler between __mte_enable_tco_async() and + * __mte_disable_tco_async(). As `dst` and `src` may contain blocking + * functions, we must evaluate these outside of the critical section. + */ +#define __put_kernel_nofault(dst, src, type, err_label) \ +do { \ + __label__ __pkn_tco_err; \ + __typeof__(dst) __pkn_tco_dst =3D (dst); \ + __typeof__(src) __pkn_tco_src =3D (src); \ + __mte_enable_tco_async(); \ + __put_kernel_nofault_bare(__pkn_tco_dst, __pkn_tco_src, type, \ + __pkn_tco_err); \ + __mte_disable_tco_async(); \ + break; \ +__pkn_tco_err: \ + __mte_disable_tco_async(); \ + goto err_label; \ +} while (0) + +/* + * A context switch re-enables tag checking, hence the no-scheduling + * requirement for a bare nofault region. + */ +#define __begin_kernel_nofault_bare() __mte_enable_tco_async() +#define __end_kernel_nofault_bare() __mte_disable_tco_async() =20 extern unsigned long __must_check __arch_copy_from_user(void *to, const vo= id __user *from, unsigned long n); #define raw_copy_from_user(to, from, n) \ diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index eddbbb65ccc4f..7ae1854673471 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -637,6 +637,22 @@ do { \ =20 #endif /* !__get_kernel_nofault */ =20 +/* + * Architectures may use the begin/end hooks to establish state shared by a + * sequence of bare nofault accesses. Every path out of the region must ca= ll + * the end hook. The region, including expressions passed to the bare + * accessors, must not call into the scheduler. + */ +#ifndef __get_kernel_nofault_bare +#define __get_kernel_nofault_bare __get_kernel_nofault +#define __put_kernel_nofault_bare __put_kernel_nofault +#endif + +#ifndef __begin_kernel_nofault_bare +#define __begin_kernel_nofault_bare() do {} while (0) +#define __end_kernel_nofault_bare() do {} while (0) +#endif + /** * get_kernel_nofault(): safely attempt to read from a location * @val: read into this variable --=20 2.47.3 From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id ED77536897F; Mon, 24 Aug 2026 16:05:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587558; cv=none; b=Tn9Pk8auMwiKjqOt/oIvkYM48pYeWJE/ajcsp3KN9fy+Xa5GyFa3ZmcuwJJhVcQlqU1vEsvnqUYB9bSF3QfFoEIqV3+gfsl6SyOpDeSWoaSOBXy4kSfXVy+4GhOcCLXwB5GLYjKEH5KYF3m9qc5CkzPYWVcDnn8gh8vcO+51BLo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587558; c=relaxed/simple; bh=mAnn8lCC8CIaJdHDU1rhT42Z2WA5jMjaMEMo8Y0pB/s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WxPYlkVbPwRFxn1MEQVHPAPLkqn2aMEi8IH+Ebq8xx351sjK7Va2qe+Hqv0MaFqBCo7AfqMqh3c/YgbuS00BAmL1OW+6oxYXTfdKUjOZSutjz/eZRg3DS5ip6VWJbx1NkVUHa3fb+RCbbKJh0dwNfsEcu9WTxT27BRcJysiWINU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=KXcATGYi; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="KXcATGYi" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 57321176C; Mon, 24 Aug 2026 09:05:50 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E50133F85F; Mon, 24 Aug 2026 09:05:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587554; bh=mAnn8lCC8CIaJdHDU1rhT42Z2WA5jMjaMEMo8Y0pB/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KXcATGYiblCFLHpkpJ3dWLchgVGcO9Fkn7aLt+0hDsnP3bpXXPHFXftjmt7pR58vR 2bXx6QlmsEr8OZamAZdP5OCw7v/hT+jfDQRoSou7GvXm1ImM5oCovGMfQyOGW9P/Gn em69FAcrIUlVaiYSuADP1SL6CUzdUrdZlVbQffZ4= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 2/7] uaccess: Add scope guard for bare kernel nofault regions Date: Mon, 24 Aug 2026 17:04:47 +0100 Message-ID: <20260824160523.3907021-3-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A batched nofault region must run its end hook on every exit. Pairing the hooks by hand makes early returns and error paths easy to get wrong. Add a scope guard that starts the region on entry and ends it when the scope is left. Callers can keep their natural control flow while the architecture state remains balanced on success and failure. No functional change until a caller uses the guard. Signed-off-by: Muhammad Usama Anjum --- include/linux/uaccess.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 7ae1854673471..b25e778ddc38a 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -653,6 +653,10 @@ do { \ #define __end_kernel_nofault_bare() do {} while (0) #endif =20 +DEFINE_LOCK_GUARD_0(__kernel_nofault_bare, + __begin_kernel_nofault_bare(), + __end_kernel_nofault_bare()) + /** * get_kernel_nofault(): safely attempt to read from a location * @val: read into this variable --=20 2.47.3 From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B3AEE439913; Mon, 24 Aug 2026 16:05:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587561; cv=none; b=sb71T+6WTpMNG+rVUDMY617sFD9A5I2i6ZHapcp6GkiKyPeHvlRs2inQdGT/x+McSjd1s88kkIBEwmsNLx6YMNt6sSeSbKHhPGB+MtsdiMohwFd5xxdkhkREeM6fh7Yoc4+5LJAcgYx2r+vNuT+pKo6pk/kOepwMpFCcG6M98YE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587561; c=relaxed/simple; bh=m2SBTQpkWSQqtHWMxAh9TvwsKocaEw70262S7IijE4E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KvWvY7qgUNJSX8xm+j/l/P0I89hQL7X9rN3SMhRYxuDyc9BY3YGyrneymQocC4bFAf4R4ZReNDxLEeIdHfsH8YEbzIS7GH1OErff+2civZcokBZA+jqPhfrFpvk4Ov0RGF1QxmVqNxvHf5hz6jTgiV019KQ4R2SJrv01poB9mGE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=IJ4nzVeh; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="IJ4nzVeh" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DC93A1AC1; Mon, 24 Aug 2026 09:05:53 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 769D03F85F; Mon, 24 Aug 2026 09:05:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587557; bh=m2SBTQpkWSQqtHWMxAh9TvwsKocaEw70262S7IijE4E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IJ4nzVehS8Hj4+HT2oldjrWLZfDgo41bmtZTUlst4kclYjz6JGUKaYe6MBCCvzi8t 7KG05+LCjlzm0mxwo5U1+zEPCt1LJnfci69d5G/n9XN09VbrmnvKqGB7eAhMcCqd+S ZBYO+C1F8Knznuik3pACO1Cht3v7ezOhwpBKTvfg= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 3/7] maccess: Skip setup for zero-sized kernel nofault copies Date: Mon, 24 Aug 2026 17:04:48 +0100 Message-ID: <20260824160523.3907021-4-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A zero-sized kernel nofault copy does not enter an access loop, but it still disables and re-enables page faults. Zero sizes are valid. BPF probe-read helpers accept them, and KGDB memory packets may carry a zero length. Return before changing page-fault state when there is nothing to copy. For reads, keep architecture-specific address validation before the fast path so its behavior is unchanged. Signed-off-by: Muhammad Usama Anjum --- mm/maccess.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/maccess.c b/mm/maccess.c index 486559d688583..c59a0e092d24a 100644 --- a/mm/maccess.c +++ b/mm/maccess.c @@ -35,6 +35,8 @@ long copy_from_kernel_nofault(void *dst, const void *src,= size_t size) =20 if (!copy_from_kernel_nofault_allowed(src, size)) return -ERANGE; + if (!size) + return 0; =20 pagefault_disable(); if (!(align & 7)) @@ -65,6 +67,9 @@ long copy_to_kernel_nofault(void *dst, const void *src, s= ize_t size) { unsigned long align =3D 0; =20 + if (!size) + return 0; + if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) align =3D (unsigned long)dst | (unsigned long)src; =20 --=20 2.47.3 From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5B42741F347; Mon, 24 Aug 2026 16:06:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587565; cv=none; b=Yrtwp9W4Xyy9D6enl3Z6Yi71bUxgPK1HQMzyKQ3lKMyhyPjlx1zZ6lH3Suxk59ZJHgvYnrrQTXDUZQiHhG0izGzK1HBZB6N3W2syuTaPIall5H1KpQwL9VqYtIyub/DuZf2cKiDnSNVEcY33kiT9qU9ZBhwb+Tp1KaXfZFPctB8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587565; c=relaxed/simple; bh=nodn3/K9trBK+2wWrkLJGeFqd0INql6u2EHBqnLbWuc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oRP1LvrD8PeZnLaTu1qOOB9Qfuq9fJwCfDD0M2uVUxmm0og5AYQaTxEdZGTydHLXO+sK2OZ77gKLT5cu/QuqQj8zJw9M0CubJSz+ki175nRiOi/ODKGScjYEHlfmtyOK3wOIVsCAU2LA/vkqMLXLoJ5C7O5J5HrWbL6+PJX4JbQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=mNGt+rmw; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="mNGt+rmw" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 66CAC1BA8; Mon, 24 Aug 2026 09:05:57 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0779D3F85F; Mon, 24 Aug 2026 09:05:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587561; bh=nodn3/K9trBK+2wWrkLJGeFqd0INql6u2EHBqnLbWuc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mNGt+rmw+m3ruCwnhja8afWvDAIEG0qPzwe79zxnTqszfW0fgkmn7kM6QqJS5/+ej jNIvAv0fF2vEvSYW8xIbW03slFbzO7/T4PlF0kwnb3MRc3Itdj++wKmSqG7vbSAzpV QanHuh35qBM3KtYjRoJ5HgcXGbly8RHUXJTfOLGQ= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 4/7] maccess: Use a scoped guard for page faults Date: Mon, 24 Aug 2026 17:04:49 +0100 Message-ID: <20260824160523.3907021-5-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel nofault copy and string paths open-code page-fault disable and enable around label-based loops, duplicating cleanup on success and failure. Use a page-fault scope guard instead. Leaving the scope now re-enables page faults on both paths without separate cleanup at the fault label. No functional change. Signed-off-by: Muhammad Usama Anjum --- mm/maccess.c | 53 +++++++++++++++++++++++++--------------------------- 1 file changed, 25 insertions(+), 28 deletions(-) diff --git a/mm/maccess.c b/mm/maccess.c index c59a0e092d24a..f695ceefe6fcc 100644 --- a/mm/maccess.c +++ b/mm/maccess.c @@ -38,18 +38,17 @@ long copy_from_kernel_nofault(void *dst, const void *sr= c, size_t size) if (!size) return 0; =20 - pagefault_disable(); - if (!(align & 7)) - copy_from_kernel_nofault_loop(dst, src, size, u64, Efault); - if (!(align & 3)) - copy_from_kernel_nofault_loop(dst, src, size, u32, Efault); - if (!(align & 1)) - copy_from_kernel_nofault_loop(dst, src, size, u16, Efault); - copy_from_kernel_nofault_loop(dst, src, size, u8, Efault); - pagefault_enable(); + scoped_guard(pagefault) { + if (!(align & 7)) + copy_from_kernel_nofault_loop(dst, src, size, u64, Efault); + if (!(align & 3)) + copy_from_kernel_nofault_loop(dst, src, size, u32, Efault); + if (!(align & 1)) + copy_from_kernel_nofault_loop(dst, src, size, u16, Efault); + copy_from_kernel_nofault_loop(dst, src, size, u8, Efault); + } return 0; Efault: - pagefault_enable(); return -EFAULT; } EXPORT_SYMBOL_GPL(copy_from_kernel_nofault); @@ -73,18 +72,17 @@ long copy_to_kernel_nofault(void *dst, const void *src,= size_t size) if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) align =3D (unsigned long)dst | (unsigned long)src; =20 - pagefault_disable(); - if (!(align & 7)) - copy_to_kernel_nofault_loop(dst, src, size, u64, Efault); - if (!(align & 3)) - copy_to_kernel_nofault_loop(dst, src, size, u32, Efault); - if (!(align & 1)) - copy_to_kernel_nofault_loop(dst, src, size, u16, Efault); - copy_to_kernel_nofault_loop(dst, src, size, u8, Efault); - pagefault_enable(); + scoped_guard(pagefault) { + if (!(align & 7)) + copy_to_kernel_nofault_loop(dst, src, size, u64, Efault); + if (!(align & 3)) + copy_to_kernel_nofault_loop(dst, src, size, u32, Efault); + if (!(align & 1)) + copy_to_kernel_nofault_loop(dst, src, size, u16, Efault); + copy_to_kernel_nofault_loop(dst, src, size, u8, Efault); + } return 0; Efault: - pagefault_enable(); return -EFAULT; } =20 @@ -97,18 +95,17 @@ long strncpy_from_kernel_nofault(char *dst, const void = *unsafe_addr, long count) if (!copy_from_kernel_nofault_allowed(unsafe_addr, count)) return -ERANGE; =20 - pagefault_disable(); - do { - __get_kernel_nofault(dst, src, u8, Efault); - dst++; - src++; - } while (dst[-1] && src - unsafe_addr < count); - pagefault_enable(); + scoped_guard(pagefault) { + do { + __get_kernel_nofault(dst, src, u8, Efault); + dst++; + src++; + } while (dst[-1] && src - unsafe_addr < count); + } =20 dst[-1] =3D '\0'; return src - unsafe_addr; Efault: - pagefault_enable(); dst[0] =3D '\0'; return -EFAULT; } --=20 2.47.3 From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0446C45FFBD; Mon, 24 Aug 2026 16:06:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587569; cv=none; b=NJ9Ywgh0mzwXJu/d+/2dk8QaGGBLVpGCHWWYGpqoO3OicsYdXnwfRjOei7kQsTgF7NQEcEX+oZY3OqJppTLTzgRkCVjtOjNXL2M1H6Fr0Es6/Y78Ol4bgKnpqSdKycS9h6YIownIO0gX3ClXfCL552pT10i5mpswO4tWdDI3YXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587569; c=relaxed/simple; bh=nodn3/K9trBK+2wWrkLJGeFqd0INql6u2EHBqnLbWuc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XVH8phTNaGNlC2L6NklNkDKKHY4PuJD3d9csZ7HtRfzVoBP8TIbwzyBPTHVe2WtZXyvggz3PPR+ghyB0hfiUCgg1sG8B4Iu2HVxHUkyToV9nltCzKGvTsdO3fdHhUrM98d98zEOB66sTCfLLdDr3ERlj2pGVKKemE1BDHXfNLoU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=cM4OF4zK; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="cM4OF4zK" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E2D1F16F2; Mon, 24 Aug 2026 09:06:00 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 846553F85F; Mon, 24 Aug 2026 09:06:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587564; bh=nodn3/K9trBK+2wWrkLJGeFqd0INql6u2EHBqnLbWuc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cM4OF4zKGJ1P/SjzDr6eXMRdqUQP3re7P3X9242MP8mexIfZldlzw0sfMcbAUSh1L 1a49U9Gw7XBFBsIGu7em0nv2/uib4e3xK/sCEUzr4sS4BhFi9PqpXz8JGLANNTKTay tyBid6SLPtTSmY70IX5SC9k9RdCKMRCJUje9FtAk= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 4/7] maccess: Use a scoped guard to re-enable page faults Date: Mon, 24 Aug 2026 17:04:50 +0100 Message-ID: <20260824160523.3907021-6-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel nofault copy and string paths open-code page-fault disable and enable around label-based loops, duplicating cleanup on success and failure. Use a page-fault scope guard instead. Leaving the scope now re-enables page faults on both paths without separate cleanup at the fault label. No functional change. Signed-off-by: Muhammad Usama Anjum --- mm/maccess.c | 53 +++++++++++++++++++++++++--------------------------- 1 file changed, 25 insertions(+), 28 deletions(-) diff --git a/mm/maccess.c b/mm/maccess.c index c59a0e092d24a..f695ceefe6fcc 100644 --- a/mm/maccess.c +++ b/mm/maccess.c @@ -38,18 +38,17 @@ long copy_from_kernel_nofault(void *dst, const void *sr= c, size_t size) if (!size) return 0; =20 - pagefault_disable(); - if (!(align & 7)) - copy_from_kernel_nofault_loop(dst, src, size, u64, Efault); - if (!(align & 3)) - copy_from_kernel_nofault_loop(dst, src, size, u32, Efault); - if (!(align & 1)) - copy_from_kernel_nofault_loop(dst, src, size, u16, Efault); - copy_from_kernel_nofault_loop(dst, src, size, u8, Efault); - pagefault_enable(); + scoped_guard(pagefault) { + if (!(align & 7)) + copy_from_kernel_nofault_loop(dst, src, size, u64, Efault); + if (!(align & 3)) + copy_from_kernel_nofault_loop(dst, src, size, u32, Efault); + if (!(align & 1)) + copy_from_kernel_nofault_loop(dst, src, size, u16, Efault); + copy_from_kernel_nofault_loop(dst, src, size, u8, Efault); + } return 0; Efault: - pagefault_enable(); return -EFAULT; } EXPORT_SYMBOL_GPL(copy_from_kernel_nofault); @@ -73,18 +72,17 @@ long copy_to_kernel_nofault(void *dst, const void *src,= size_t size) if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) align =3D (unsigned long)dst | (unsigned long)src; =20 - pagefault_disable(); - if (!(align & 7)) - copy_to_kernel_nofault_loop(dst, src, size, u64, Efault); - if (!(align & 3)) - copy_to_kernel_nofault_loop(dst, src, size, u32, Efault); - if (!(align & 1)) - copy_to_kernel_nofault_loop(dst, src, size, u16, Efault); - copy_to_kernel_nofault_loop(dst, src, size, u8, Efault); - pagefault_enable(); + scoped_guard(pagefault) { + if (!(align & 7)) + copy_to_kernel_nofault_loop(dst, src, size, u64, Efault); + if (!(align & 3)) + copy_to_kernel_nofault_loop(dst, src, size, u32, Efault); + if (!(align & 1)) + copy_to_kernel_nofault_loop(dst, src, size, u16, Efault); + copy_to_kernel_nofault_loop(dst, src, size, u8, Efault); + } return 0; Efault: - pagefault_enable(); return -EFAULT; } =20 @@ -97,18 +95,17 @@ long strncpy_from_kernel_nofault(char *dst, const void = *unsafe_addr, long count) if (!copy_from_kernel_nofault_allowed(unsafe_addr, count)) return -ERANGE; =20 - pagefault_disable(); - do { - __get_kernel_nofault(dst, src, u8, Efault); - dst++; - src++; - } while (dst[-1] && src - unsafe_addr < count); - pagefault_enable(); + scoped_guard(pagefault) { + do { + __get_kernel_nofault(dst, src, u8, Efault); + dst++; + src++; + } while (dst[-1] && src - unsafe_addr < count); + } =20 dst[-1] =3D '\0'; return src - unsafe_addr; Efault: - pagefault_enable(); dst[0] =3D '\0'; return -EFAULT; } --=20 2.47.3 From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AFB9F4611E1; Mon, 24 Aug 2026 16:06:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587574; cv=none; b=oi5bRNotO2xhAgnCqL6PKeZI5umkiWnTrdke9wLCx77K6AOC/VpqrZ1Zdy3fEyjcUvHvAEwx9+orYsqIXCafqHeEmZd+pd0IxI0twLcpdIcyWqjvKoTw9Ax73e6KQqwRA7yQ7Ojr583t8S1uV28VYgjqxZT6NwAVsW13Rt69rDQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587574; c=relaxed/simple; bh=RrebhXATXwELFIUm3sfpDdh92ACD3QNlTwQzpg9AkNk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qYsER3GCw3PRI/SDcYgJfsnrilb8btt3LZwmq22RHt+8rntGbsS+/z4XYFqTFeY27pkcJcEO8HY/aaCIh3Cnngbdie+VGecdTF7GjJDVJrylEgphVI9IZ2IzW8+WivJarGEVznmg7j9iWbzXpljedx/mBpbIcoVHG4cW3GmIomM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=Jc91idwU; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="Jc91idwU" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 684411756; Mon, 24 Aug 2026 09:06:04 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0C5DE3F85F; Mon, 24 Aug 2026 09:06:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587568; bh=RrebhXATXwELFIUm3sfpDdh92ACD3QNlTwQzpg9AkNk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jc91idwUPeyzoMBCcMqqcJfBY/NipSip+nx0Vu2SSHMLt0GB1cy1vcJ0XIHfHuDVt ZfysIhqHXV5A+RsGPMQs5ObrYLtPAo5PLdYurj6y9VDAQ+21gPimS5kpua6pOqy5p/ cu3IO/eY89jF1Cl+5pzRLPOStBWlPPk/3hQnsiX0= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 5/7] maccess: Batch TCO handling in kernel nofault loops Date: Mon, 24 Aug 2026 17:04:51 +0100 Message-ID: <20260824160523.3907021-7-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With Hardware Tag-Based KASAN in asynchronous or asymmetric mode, every arm64 kernel nofault access sets and clears PSTATE.TCO. Copy and string loops repeat that pair even though tag checking can stay disabled for the whole operation. Cover each non-empty operation with one bare nofault region and use bare accessors in the loop. Leaving the region restores TCO before page faults are enabled again, including after an access fault. Existing empty-work checks ensure that every new region performs at least one access. The number of dynamic MSR TCO executions therefore changes as follows: Work Before After N nofault accesses 2N 2 4 KiB nofault copy 1,024 2 N-byte strncpy 2N 2 The 4 KiB case assumes 512 64-bit accesses. These figures come from the control flow rather than a runtime measurement, so the time saved depends on the CPU and workload. Generic fallbacks leave other architectures unchanged. Signed-off-by: Muhammad Usama Anjum --- mm/maccess.c | 48 +++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/mm/maccess.c b/mm/maccess.c index f695ceefe6fcc..87486bf98d15a 100644 --- a/mm/maccess.c +++ b/mm/maccess.c @@ -19,7 +19,7 @@ bool __weak copy_from_kernel_nofault_allowed(const void *= unsafe_src, */ #define copy_from_kernel_nofault_loop(dst, src, len, type, err_label) \ while (len >=3D sizeof(type)) { \ - __get_kernel_nofault(dst, src, type, err_label); \ + __get_kernel_nofault_bare(dst, src, type, err_label); \ kmsan_check_memory(src, sizeof(type)); \ dst +=3D sizeof(type); \ src +=3D sizeof(type); \ @@ -39,13 +39,15 @@ long copy_from_kernel_nofault(void *dst, const void *sr= c, size_t size) return 0; =20 scoped_guard(pagefault) { - if (!(align & 7)) - copy_from_kernel_nofault_loop(dst, src, size, u64, Efault); - if (!(align & 3)) - copy_from_kernel_nofault_loop(dst, src, size, u32, Efault); - if (!(align & 1)) - copy_from_kernel_nofault_loop(dst, src, size, u16, Efault); - copy_from_kernel_nofault_loop(dst, src, size, u8, Efault); + scoped_guard(__kernel_nofault_bare) { + if (!(align & 7)) + copy_from_kernel_nofault_loop(dst, src, size, u64, Efault); + if (!(align & 3)) + copy_from_kernel_nofault_loop(dst, src, size, u32, Efault); + if (!(align & 1)) + copy_from_kernel_nofault_loop(dst, src, size, u16, Efault); + copy_from_kernel_nofault_loop(dst, src, size, u8, Efault); + } } return 0; Efault: @@ -55,7 +57,7 @@ EXPORT_SYMBOL_GPL(copy_from_kernel_nofault); =20 #define copy_to_kernel_nofault_loop(dst, src, len, type, err_label) \ while (len >=3D sizeof(type)) { \ - __put_kernel_nofault(dst, src, type, err_label); \ + __put_kernel_nofault_bare(dst, src, type, err_label); \ instrument_write(dst, sizeof(type)); \ dst +=3D sizeof(type); \ src +=3D sizeof(type); \ @@ -73,13 +75,15 @@ long copy_to_kernel_nofault(void *dst, const void *src,= size_t size) align =3D (unsigned long)dst | (unsigned long)src; =20 scoped_guard(pagefault) { - if (!(align & 7)) - copy_to_kernel_nofault_loop(dst, src, size, u64, Efault); - if (!(align & 3)) - copy_to_kernel_nofault_loop(dst, src, size, u32, Efault); - if (!(align & 1)) - copy_to_kernel_nofault_loop(dst, src, size, u16, Efault); - copy_to_kernel_nofault_loop(dst, src, size, u8, Efault); + scoped_guard(__kernel_nofault_bare) { + if (!(align & 7)) + copy_to_kernel_nofault_loop(dst, src, size, u64, Efault); + if (!(align & 3)) + copy_to_kernel_nofault_loop(dst, src, size, u32, Efault); + if (!(align & 1)) + copy_to_kernel_nofault_loop(dst, src, size, u16, Efault); + copy_to_kernel_nofault_loop(dst, src, size, u8, Efault); + } } return 0; Efault: @@ -96,11 +100,13 @@ long strncpy_from_kernel_nofault(char *dst, const void= *unsafe_addr, long count) return -ERANGE; =20 scoped_guard(pagefault) { - do { - __get_kernel_nofault(dst, src, u8, Efault); - dst++; - src++; - } while (dst[-1] && src - unsafe_addr < count); + scoped_guard(__kernel_nofault_bare) { + do { + __get_kernel_nofault_bare(dst, src, u8, Efault); + dst++; + src++; + } while (dst[-1] && src - unsafe_addr < count); + } } =20 dst[-1] =3D '\0'; --=20 2.47.3 From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 627D644C51B; Mon, 24 Aug 2026 16:06:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587575; cv=none; b=SbxLg7cRSJxYd1NXXxbRJgyl0Tp5xSxUsm2WHw2Twt2kdO8JBpOYrtDQPLYpuHWdoYioYb3VSKaOyUP75rNOEfuhVPZdpwWTuhw8Eib8f4+Yjc2Ndtz9dBPd4maieZUZ/lpZQewHfFxmBC5WE0l+otwBqX3QJ81+Vefll9vUDNU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587575; c=relaxed/simple; bh=Rwi4AzekxF8PcB6UHH8vOzSHeIKlfPuKVBoBNngyqjk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mBIBQ4Lzu7SlXueWSUf22JY/FovrComPqm1eVKMNzrxj3V1R1dF4Pk2JBcq6bkd3itX/3UkucAT1h4/LJ2zgH8IHGi2ujWUTzNdFqlPE5Et7JTwF65KFrmLX7uK6uNR3CJORWP1kKvXkbz1Muz+M214WEeoBScZ39w+RBue9kkg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=dJsvLuXE; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="dJsvLuXE" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E38FB176C; Mon, 24 Aug 2026 09:06:07 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 885403F85F; Mon, 24 Aug 2026 09:06:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587571; bh=Rwi4AzekxF8PcB6UHH8vOzSHeIKlfPuKVBoBNngyqjk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dJsvLuXExRZfSJVbv3K6HCiW3BR7TJXaVrPv1IYrS8GDrZipSySzPSoKX1sDg/TLi rcgwUtzijkwFuJ82JRFRbQoUfk2VhTL8yreQOzRe8wWaiRJ4JRUZTANWfRepEescVY U3NBHGGKRha8TpysreQuCpCiYPx/pdEPgbg4PJ+s= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 6/7] bpf: Skip setup for zero-length string kfunc operations Date: Mon, 24 Aug 2026 17:04:52 +0100 Message-ID: <20260824160523.3907021-8-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A zero limit is valid for several length-bounded BPF string operations. Their loops perform no load in that case, but they still enter and leave a page-fault-disabled region. Return the existing empty result before changing page-fault state. Keep address validation first so an invalid pointer continues to return -ERANGE. Signed-off-by: Muhammad Usama Anjum --- kernel/bpf/helpers.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index b3cc5c8fc8756..3574a9a5721ec 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3738,6 +3738,8 @@ static int __bpf_strncasecmp(const char *s1, const ch= ar *s2, bool ignore_case, s !copy_from_kernel_nofault_allowed(s2, 1)) { return -ERANGE; } + if (!len) + return 0; =20 guard(pagefault)(); for (i =3D 0; i < len && i < XATTR_SIZE_MAX; i++) { @@ -3837,6 +3839,8 @@ __bpf_kfunc int bpf_strnchr(const char *s__ign, size_= t count, char c) =20 if (!copy_from_kernel_nofault_allowed(s__ign, 1)) return -ERANGE; + if (!count) + return -ENOENT; =20 guard(pagefault)(); for (i =3D 0; i < count && i < XATTR_SIZE_MAX; i++) { @@ -3956,6 +3960,8 @@ __bpf_kfunc int bpf_strnlen(const char *s__ign, size_= t count) =20 if (!copy_from_kernel_nofault_allowed(s__ign, 1)) return -ERANGE; + if (!count) + return 0; =20 guard(pagefault)(); for (i =3D 0; i < count && i < XATTR_SIZE_MAX; i++) { --=20 2.47.3 From nobody Mon Sep 28 08:07:24 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9466D468C38; Mon, 24 Aug 2026 16:06:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587577; cv=none; b=GDON+3OH8nWI88YhC+KSmjgw04sYtckhUG06rPHgcXsrYf3UKaTheo7Kmb0XVvIIHOAUHiTzBICA9PD8vjQy1ZiTLZGqHLv2+Q/eUaqTPs3Rx621dYYfP4rkeKPllzfHkz1DSjkLy6ousJhrShTeNvpk5QD1YCTru4Vrj2zd4mg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787587577; c=relaxed/simple; bh=Eti3rTlv+cxAVxk1vV0UCcKrMWkCOHT4UCvaZN/ysVQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SQYz4megbcGDIqh9xpebUlvnBF5gc5oZIr1uZXHC5jz+bH32B3ibB0KELrF2noCY6erL9QiHO1Em75PSmA2iPkP4WlckzD8gEboqYwhNn3b4xWuxFkSTtMBIDZvsVDkDML9Tr6j7OdePtWjrdwOC1pcB8zBEWSczXGvYgkJ3xDg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=n5AF6xbs; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="n5AF6xbs" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6BE991AC1; Mon, 24 Aug 2026 09:06:11 -0700 (PDT) Received: from e142334-100.cambridge.arm.com (e142334-100.cambridge.arm.com [10.2.198.93]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 102303F85F; Mon, 24 Aug 2026 09:06:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787587575; bh=Eti3rTlv+cxAVxk1vV0UCcKrMWkCOHT4UCvaZN/ysVQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n5AF6xbsvYOJDb232Qg50DCemCE4qaYKTCUYp0BZbfqNZ6nujWbXQrRHrQjDCm99n akrUnA9aq9/VXusaTW0mhrDpjwetDpb7unbe70ZAAywW7Ln4KarRk8gPk63sHOgK51 kLE5ugmLz2WbYPD7/SK3iKuatOMbzXplh0cSB1Mg= From: Muhammad Usama Anjum To: Catalin Marinas , Will Deacon , Mark Rutland , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , linux-arm-kernel@lists.infradead.org (moderated list:ARM64 PORT (AARCH64 ARCHITECTURE)), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), linux-mm@kvack.org (open list:MEMORY MANAGEMENT - CORE) Cc: Muhammad Usama Anjum Subject: [PATCH 7/7] bpf: Batch TCO handling in string kfuncs Date: Mon, 24 Aug 2026 17:04:53 +0100 Message-ID: <20260824160523.3907021-9-usama.anjum@arm.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824160523.3907021-1-usama.anjum@arm.com> References: <20260824160523.3907021-1-usama.anjum@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" BPF string kfuncs cannot rely on NUL termination, so they scan memory with kernel nofault loads. With Hardware Tag-Based KASAN in asynchronous or asymmetric mode, every arm64 load sets and clears PSTATE.TCO. Use bare loads and hold one nofault region across each string operation. The scope guard restores TCO before page faults are enabled again on every exit, including an access fault. A character comparison performs two nofault loads. For N compared characters, the number of dynamic MSR TCO executions therefore falls from 4N to 2. Signed-off-by: Muhammad Usama Anjum --- kernel/bpf/helpers.c | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 3574a9a5721ec..f8c557161584e 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3726,7 +3726,8 @@ __bpf_kfunc void __bpf_trap(void) * * Since strings are not necessarily %NUL-terminated, we cannot directly c= all * in-kernel implementations. Instead, we open-code the implementations us= ing - * __get_kernel_nofault instead of plain dereference to make them safe. + * __get_kernel_nofault_bare under guard(__kernel_nofault_bare)() to make = them + * safe. */ =20 static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_c= ase, size_t len) @@ -3742,9 +3743,10 @@ static int __bpf_strncasecmp(const char *s1, const c= har *s2, bool ignore_case, s return 0; =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < len && i < XATTR_SIZE_MAX; i++) { - __get_kernel_nofault(&c1, s1, char, err_out); - __get_kernel_nofault(&c2, s2, char, err_out); + __get_kernel_nofault_bare(&c1, s1, char, err_out); + __get_kernel_nofault_bare(&c2, s2, char, err_out); if (ignore_case) { c1 =3D tolower(c1); c2 =3D tolower(c2); @@ -3843,8 +3845,9 @@ __bpf_kfunc int bpf_strnchr(const char *s__ign, size_= t count, char c) return -ENOENT; =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < count && i < XATTR_SIZE_MAX; i++) { - __get_kernel_nofault(&sc, s__ign, char, err_out); + __get_kernel_nofault_bare(&sc, s__ign, char, err_out); if (sc =3D=3D c) return i; if (sc =3D=3D '\0') @@ -3897,8 +3900,9 @@ __bpf_kfunc int bpf_strchrnul(const char *s__ign, cha= r c) return -ERANGE; =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < XATTR_SIZE_MAX; i++) { - __get_kernel_nofault(&sc, s__ign, char, err_out); + __get_kernel_nofault_bare(&sc, s__ign, char, err_out); if (sc =3D=3D '\0' || sc =3D=3D c) return i; s__ign++; @@ -3929,8 +3933,9 @@ __bpf_kfunc int bpf_strrchr(const char *s__ign, int c) return -ERANGE; =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < XATTR_SIZE_MAX; i++) { - __get_kernel_nofault(&sc, s__ign, char, err_out); + __get_kernel_nofault_bare(&sc, s__ign, char, err_out); if (sc =3D=3D c) last =3D i; if (sc =3D=3D '\0') @@ -3964,8 +3969,9 @@ __bpf_kfunc int bpf_strnlen(const char *s__ign, size_= t count) return 0; =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < count && i < XATTR_SIZE_MAX; i++) { - __get_kernel_nofault(&c, s__ign, char, err_out); + __get_kernel_nofault_bare(&c, s__ign, char, err_out); if (c =3D=3D '\0') return i; s__ign++; @@ -4014,12 +4020,13 @@ __bpf_kfunc int bpf_strspn(const char *s__ign, cons= t char *accept__ign) } =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < XATTR_SIZE_MAX; i++) { - __get_kernel_nofault(&cs, s__ign, char, err_out); + __get_kernel_nofault_bare(&cs, s__ign, char, err_out); if (cs =3D=3D '\0') return i; for (j =3D 0; j < XATTR_SIZE_MAX; j++) { - __get_kernel_nofault(&ca, accept__ign + j, char, err_out); + __get_kernel_nofault_bare(&ca, accept__ign + j, char, err_out); if (cs =3D=3D ca || ca =3D=3D '\0') break; } @@ -4058,12 +4065,13 @@ __bpf_kfunc int bpf_strcspn(const char *s__ign, con= st char *reject__ign) } =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < XATTR_SIZE_MAX; i++) { - __get_kernel_nofault(&cs, s__ign, char, err_out); + __get_kernel_nofault_bare(&cs, s__ign, char, err_out); if (cs =3D=3D '\0') return i; for (j =3D 0; j < XATTR_SIZE_MAX; j++) { - __get_kernel_nofault(&cr, reject__ign + j, char, err_out); + __get_kernel_nofault_bare(&cr, reject__ign + j, char, err_out); if (cs =3D=3D cr || cr =3D=3D '\0') break; } @@ -4090,9 +4098,10 @@ static int __bpf_strnstr(const char *s1, const char = *s2, size_t len, } =20 guard(pagefault)(); + guard(__kernel_nofault_bare)(); for (i =3D 0; i < XATTR_SIZE_MAX; i++) { for (j =3D 0; i + j <=3D len && j < XATTR_SIZE_MAX; j++) { - __get_kernel_nofault(&c2, s2 + j, char, err_out); + __get_kernel_nofault_bare(&c2, s2 + j, char, err_out); if (c2 =3D=3D '\0') return i; /* @@ -4102,7 +4111,7 @@ static int __bpf_strnstr(const char *s1, const char *= s2, size_t len, */ if (i + j =3D=3D len) break; - __get_kernel_nofault(&c1, s1 + j, char, err_out); + __get_kernel_nofault_bare(&c1, s1 + j, char, err_out); =20 if (ignore_case) { c1 =3D tolower(c1); --=20 2.47.3