From nobody Tue Feb 10 02:43:51 2026 Received: from canpmsgout06.his.huawei.com (canpmsgout06.his.huawei.com [113.46.200.221]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4778032E696; Wed, 28 Jan 2026 03:20:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=113.46.200.221 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769570407; cv=none; b=YKXJAxBkIbaCVO0lfqnKHKPjtQA7oV0Xr5nhigra6yUKJi2x8AzyLfCRfCv9f5j22EAWuw1HjEtGt2WbbYQ8tccy6I00SWJT+1B8H6g9xzDMOqn/0U6GM+91G+7TS+F3HfN8MIjen38dAjWSo7oadI3nGZOY6kGbPB9wow3f/RY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769570407; c=relaxed/simple; bh=pHYdsKdt0rbRD+OijXRThbBYLnxRNICeNwA1RurEF14=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HT7qG+uuH24CBSe1OI1LrUQantWC07wwhFyqoz8LVXsW4oOIe91P4niawMjB3PX+n/uU69j7BzDXlmu/Sm8gvXy0a3qn2aXjilAUn/DlBToWVAWCLiy5xfIMw5l9CuOH3diwS7SuOoAZQDZ3PwkdGzHYx9VUzm1+v5nrujcmeYk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=UZ2mYVUN; arc=none smtp.client-ip=113.46.200.221 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="UZ2mYVUN" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=/winTl3WYeNhZLFo886OhmhgHxj8matd5TvztlmrpLM=; b=UZ2mYVUNqPjG+gCmvbBhfM11qgbIbyFE7mlW7xQikSPioWdC3csAAY++lS7cdaTgFBGUuK5kj 6QJA46WpH7lukA83W4WAu/KqR+L4h/Tgyj2IzN0yxiqyiq6ad5lsfjGVXCFKDFpIoLF8PWTpVnw rVCgJcX4Btjeyya/KbS0PTU= Received: from mail.maildlp.com (unknown [172.19.162.140]) by canpmsgout06.his.huawei.com (SkyGuard) with ESMTPS id 4f16r226ZWzRhrt; Wed, 28 Jan 2026 11:16:34 +0800 (CST) Received: from dggpemf500011.china.huawei.com (unknown [7.185.36.131]) by mail.maildlp.com (Postfix) with ESMTPS id 5387D2021A; Wed, 28 Jan 2026 11:20:02 +0800 (CST) Received: from huawei.com (10.90.53.73) by dggpemf500011.china.huawei.com (7.185.36.131) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Wed, 28 Jan 2026 11:20:00 +0800 From: Jinjie Ruan To: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , CC: Subject: [PATCH v11 13/14] entry: Inline syscall_exit_work() and syscall_trace_enter() Date: Wed, 28 Jan 2026 11:19:33 +0800 Message-ID: <20260128031934.3906955-14-ruanjinjie@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260128031934.3906955-1-ruanjinjie@huawei.com> References: <20260128031934.3906955-1-ruanjinjie@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems200002.china.huawei.com (7.221.188.68) To dggpemf500011.china.huawei.com (7.185.36.131) After switch arm64 to Generic Entry, a new hotspot syscall_exit_work() appeared because syscall_exit_work() is not inlined, so inline syscall_exit_work(). Also inline syscall_trace_enter() to align with syscall_exit_work(). On v6.19-rc1 with audit on, inline both syscall_trace_enter() and syscall_exit_work() has 4% performance uplift on perf bench basic syscall on kunpeng920 as below: | Metric | W/O this patch | With this patch | Change | | ---------- | -------------- | --------------- | ------ | | Total time | 2.353 [sec] | 2.264 [sec] | =E2=86=933.8% | | usecs/op | 0.235374 | 0.226472 | =E2=86=933.8% | | ops/sec | 4,248,588 | 4,415,554 | =E2=86=913.9% | Signed-off-by: Jinjie Ruan --- include/linux/entry-common.h | 101 ++++++++++++++++++++++++++- kernel/entry/common.h | 7 -- kernel/entry/syscall-common.c | 95 ++----------------------- kernel/entry/syscall_user_dispatch.c | 4 +- 4 files changed, 105 insertions(+), 102 deletions(-) delete mode 100644 kernel/entry/common.h diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index 48bdde74a3e1..c2d772c70b7c 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -2,6 +2,7 @@ #ifndef __LINUX_ENTRYCOMMON_H #define __LINUX_ENTRYCOMMON_H =20 +#include #include #include #include @@ -64,7 +65,63 @@ static __always_inline int arch_ptrace_report_syscall_en= try(struct pt_regs *regs } #endif =20 -long syscall_trace_enter(struct pt_regs *regs, unsigned long work); +static inline void syscall_enter_audit(struct pt_regs *regs, long syscall) +{ + if (unlikely(audit_context())) { + unsigned long args[6]; + + syscall_get_arguments(current, regs, args); + audit_syscall_entry(syscall, args[0], args[1], args[2], args[3]); + } +} + +void __trace_sys_enter(struct pt_regs *regs, long syscall); +bool syscall_user_dispatch(struct pt_regs *regs); + +static __always_inline long syscall_trace_enter(struct pt_regs *regs, unsi= gned long work) +{ + long syscall, ret =3D 0; + + /* + * Handle Syscall User Dispatch. This must comes first, since + * the ABI here can be something that doesn't make sense for + * other syscall_work features. + */ + if (work & SYSCALL_WORK_SYSCALL_USER_DISPATCH) { + if (syscall_user_dispatch(regs)) + return -1L; + } + + /* Handle ptrace */ + if (work & (SYSCALL_WORK_SYSCALL_TRACE | SYSCALL_WORK_SYSCALL_EMU)) { + ret =3D arch_ptrace_report_syscall_entry(regs); + if (ret || (work & SYSCALL_WORK_SYSCALL_EMU)) + return -1L; + } + + /* Do seccomp after ptrace, to catch any tracer changes. */ + if (work & SYSCALL_WORK_SECCOMP) { + ret =3D __secure_computing(); + if (ret =3D=3D -1L) + return ret; + } + + /* Either of the above might have changed the syscall number */ + syscall =3D syscall_get_nr(current, regs); + + if (unlikely(work & SYSCALL_WORK_SYSCALL_TRACEPOINT)) { + __trace_sys_enter(regs, syscall); + /* + * Probes or BPF hooks in the tracepoint may have changed the + * system call number as well. + */ + syscall =3D syscall_get_nr(current, regs); + } + + syscall_enter_audit(regs, syscall); + + return ret ? : syscall; +} =20 /** * syscall_enter_from_user_mode_work - Check and handle work before invoki= ng @@ -131,6 +188,19 @@ static __always_inline long syscall_enter_from_user_mo= de(struct pt_regs *regs, l return ret; } =20 +/* + * If SYSCALL_EMU is set, then the only reason to report is when + * SINGLESTEP is set (i.e. PTRACE_SYSEMU_SINGLESTEP). This syscall + * instruction has been already reported in syscall_enter_from_user_mode(). + */ +static __always_inline bool report_single_step(unsigned long work) +{ + if (work & SYSCALL_WORK_SYSCALL_EMU) + return false; + + return work & SYSCALL_WORK_SYSCALL_EXIT_TRAP; +} + /** * arch_ptrace_report_syscall_exit - Architecture specific * ptrace_report_syscall_exit. @@ -151,6 +221,8 @@ static __always_inline void arch_ptrace_report_syscall_= exit(struct pt_regs *regs } #endif =20 +void __trace_sys_exit(struct pt_regs *regs, long ret); + /** * syscall_exit_work - Handle work before returning to user mode * @regs: Pointer to current pt_regs @@ -158,7 +230,32 @@ static __always_inline void arch_ptrace_report_syscall= _exit(struct pt_regs *regs * * Do one-time syscall specific work. */ -void syscall_exit_work(struct pt_regs *regs, unsigned long work); +static __always_inline void syscall_exit_work(struct pt_regs *regs, unsign= ed long work) +{ + bool step; + + /* + * If the syscall was rolled back due to syscall user dispatching, + * then the tracers below are not invoked for the same reason as + * the entry side was not invoked in syscall_trace_enter(): The ABI + * of these syscalls is unknown. + */ + if (work & SYSCALL_WORK_SYSCALL_USER_DISPATCH) { + if (unlikely(current->syscall_dispatch.on_dispatch)) { + current->syscall_dispatch.on_dispatch =3D false; + return; + } + } + + audit_syscall_exit(regs); + + if (work & SYSCALL_WORK_SYSCALL_TRACEPOINT) + __trace_sys_exit(regs, syscall_get_return_value(current, regs)); + + step =3D report_single_step(work); + if (step || work & SYSCALL_WORK_SYSCALL_TRACE) + arch_ptrace_report_syscall_exit(regs, step); +} =20 /** * syscall_exit_to_user_mode_work - Handle work before returning to user m= ode diff --git a/kernel/entry/common.h b/kernel/entry/common.h deleted file mode 100644 index f6e6d02f07fe..000000000000 --- a/kernel/entry/common.h +++ /dev/null @@ -1,7 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _COMMON_H -#define _COMMON_H - -bool syscall_user_dispatch(struct pt_regs *regs); - -#endif diff --git a/kernel/entry/syscall-common.c b/kernel/entry/syscall-common.c index bb5f61f5629d..10231d30405e 100644 --- a/kernel/entry/syscall-common.c +++ b/kernel/entry/syscall-common.c @@ -1,103 +1,16 @@ // SPDX-License-Identifier: GPL-2.0 =20 -#include #include -#include "common.h" =20 #define CREATE_TRACE_POINTS #include =20 -static inline void syscall_enter_audit(struct pt_regs *regs, long syscall) +void __trace_sys_enter(struct pt_regs *regs, long syscall) { - if (unlikely(audit_context())) { - unsigned long args[6]; - - syscall_get_arguments(current, regs, args); - audit_syscall_entry(syscall, args[0], args[1], args[2], args[3]); - } + trace_sys_enter(regs, syscall); } =20 -long syscall_trace_enter(struct pt_regs *regs, unsigned long work) +void __trace_sys_exit(struct pt_regs *regs, long ret) { - long syscall, ret =3D 0; - - /* - * Handle Syscall User Dispatch. This must comes first, since - * the ABI here can be something that doesn't make sense for - * other syscall_work features. - */ - if (work & SYSCALL_WORK_SYSCALL_USER_DISPATCH) { - if (syscall_user_dispatch(regs)) - return -1L; - } - - /* Handle ptrace */ - if (work & (SYSCALL_WORK_SYSCALL_TRACE | SYSCALL_WORK_SYSCALL_EMU)) { - ret =3D arch_ptrace_report_syscall_entry(regs); - if (ret || (work & SYSCALL_WORK_SYSCALL_EMU)) - return -1L; - } - - /* Do seccomp after ptrace, to catch any tracer changes. */ - if (work & SYSCALL_WORK_SECCOMP) { - ret =3D __secure_computing(); - if (ret =3D=3D -1L) - return ret; - } - - /* Either of the above might have changed the syscall number */ - syscall =3D syscall_get_nr(current, regs); - - if (unlikely(work & SYSCALL_WORK_SYSCALL_TRACEPOINT)) { - trace_sys_enter(regs, syscall); - /* - * Probes or BPF hooks in the tracepoint may have changed the - * system call number as well. - */ - syscall =3D syscall_get_nr(current, regs); - } - - syscall_enter_audit(regs, syscall); - - return ret ? : syscall; -} - -/* - * If SYSCALL_EMU is set, then the only reason to report is when - * SINGLESTEP is set (i.e. PTRACE_SYSEMU_SINGLESTEP). This syscall - * instruction has been already reported in syscall_enter_from_user_mode(). - */ -static inline bool report_single_step(unsigned long work) -{ - if (work & SYSCALL_WORK_SYSCALL_EMU) - return false; - - return work & SYSCALL_WORK_SYSCALL_EXIT_TRAP; -} - -void syscall_exit_work(struct pt_regs *regs, unsigned long work) -{ - bool step; - - /* - * If the syscall was rolled back due to syscall user dispatching, - * then the tracers below are not invoked for the same reason as - * the entry side was not invoked in syscall_trace_enter(): The ABI - * of these syscalls is unknown. - */ - if (work & SYSCALL_WORK_SYSCALL_USER_DISPATCH) { - if (unlikely(current->syscall_dispatch.on_dispatch)) { - current->syscall_dispatch.on_dispatch =3D false; - return; - } - } - - audit_syscall_exit(regs); - - if (work & SYSCALL_WORK_SYSCALL_TRACEPOINT) - trace_sys_exit(regs, syscall_get_return_value(current, regs)); - - step =3D report_single_step(work); - if (step || work & SYSCALL_WORK_SYSCALL_TRACE) - arch_ptrace_report_syscall_exit(regs, step); + trace_sys_exit(regs, ret); } diff --git a/kernel/entry/syscall_user_dispatch.c b/kernel/entry/syscall_us= er_dispatch.c index a9055eccb27e..d89dffcc2d64 100644 --- a/kernel/entry/syscall_user_dispatch.c +++ b/kernel/entry/syscall_user_dispatch.c @@ -2,6 +2,8 @@ /* * Copyright (C) 2020 Collabora Ltd. */ + +#include #include #include #include @@ -15,8 +17,6 @@ =20 #include =20 -#include "common.h" - static void trigger_sigsys(struct pt_regs *regs) { struct kernel_siginfo info; --=20 2.34.1