From nobody Thu Dec 18 03:21:15 2025 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44B002153C4 for ; Wed, 7 May 2025 10:43:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.133 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746614615; cv=none; b=JRj+kKdv84yeaJ/OnemiXI77JrMNYYF7WyqLsHL5Z10kaeobeSba0Mtcm5m8urgKa/sKOVzBz4quS5jsOJPIITtXlVAhdKhJfwYh14KMjADHqWfSWOoy5C2YxYMYU5w0Tu4oJMJJxtpfAHXcCBQVmpQFcLC+GAyxCLucjaPujTs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746614615; c=relaxed/simple; bh=tW2E658I3M6uqqha9+slKauAhawjMd6b6ubXyPWOciQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=btC8RzSnwgQeV1KVYF5mOPGFJgVI/LMbswXBvydcyClkxHp6uf/OWaITA4AMXpmajMldB4Q1gRRGi4UX4aTgzcteh+wF91sTEXUOP8kz7EVJHZRWaNWXUst1Yjt04uuMPLhy1HTWs4MuGiFWPvEADS2s0cmhbvvfEiAJJFGA7qI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=G08ooDU5; arc=none smtp.client-ip=115.124.30.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="G08ooDU5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1746614604; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=4vWtkGh0fb3l4BwRAliZH3bthkTUpQ9fw5yBkTW9YV8=; b=G08ooDU5EyEP42LITK3pgcu89P2e3NzgtNeSCmj+QUicLWq0NdBtHKnBNLYoEQ7SE3vZgIEUhCXoOYTtjbT0kH82x/QGlb8jG27JP4w9/ZUODnoXnjIKsb57LcKZA6cRf0ZEpt54ey+x+GjSHEUhcr/JLvhi02kGtmVeAB9XwFM= Received: from localhost(mailfrom:feng.tang@linux.alibaba.com fp:SMTPD_---0WZpoLJg_1746614603 cluster:ay36) by smtp.aliyun-inc.com; Wed, 07 May 2025 18:43:23 +0800 From: Feng Tang To: Andrew Morton , Petr Mladek , Steven Rostedt , Lance Yang , linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [PATCH RFC 1/3] kernel/panic: generalize panic_print's function to show sys info Date: Wed, 7 May 2025 18:43:20 +0800 Message-Id: <20250507104322.30700-2-feng.tang@linux.alibaba.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250507104322.30700-1-feng.tang@linux.alibaba.com> References: <20250507104322.30700-1-feng.tang@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" panic_print was introduced to help debugging kernel panic by dumping different kinds of system information like tasks' call stack, memory, ftrace buffer etc. Acutually this function could help debugging cases like task-hung, soft/hard lockup too, where user may need the snapshot of system info at that time. Extract sys_show_info() function out to be used by other kernel parts for debugging. Signed-off-by: Feng Tang --- include/linux/panic.h | 12 ++++++++++++ kernel/panic.c | 44 +++++++++++++++++++++---------------------- 2 files changed, 34 insertions(+), 22 deletions(-) diff --git a/include/linux/panic.h b/include/linux/panic.h index 2494d51707ef..a6b538936a67 100644 --- a/include/linux/panic.h +++ b/include/linux/panic.h @@ -16,6 +16,18 @@ extern void oops_enter(void); extern void oops_exit(void); extern bool oops_may_print(void); =20 +/* Currently SYS_PRINT_ALL_PRINTK_MSG is only used for panic case */ +#define SYS_PRINT_TASK_INFO 0x00000001 +#define SYS_PRINT_MEM_INFO 0x00000002 +#define SYS_PRINT_TIMER_INFO 0x00000004 +#define SYS_PRINT_LOCK_INFO 0x00000008 +#define SYS_PRINT_FTRACE_INFO 0x00000010 +#define SYS_PRINT_ALL_PRINTK_MSG 0x00000020 +#define SYS_PRINT_ALL_CPU_BT 0x00000040 +#define SYS_PRINT_BLOCKED_TASKS 0x00000080 + +extern void sys_show_info(unsigned long info_mask); + extern bool panic_triggering_all_cpu_backtrace; extern int panic_timeout; extern unsigned long panic_print; diff --git a/kernel/panic.c b/kernel/panic.c index a3889f38153d..4fd9499f6505 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -69,14 +69,6 @@ bool panic_triggering_all_cpu_backtrace; int panic_timeout =3D CONFIG_PANIC_TIMEOUT; EXPORT_SYMBOL_GPL(panic_timeout); =20 -#define PANIC_PRINT_TASK_INFO 0x00000001 -#define PANIC_PRINT_MEM_INFO 0x00000002 -#define PANIC_PRINT_TIMER_INFO 0x00000004 -#define PANIC_PRINT_LOCK_INFO 0x00000008 -#define PANIC_PRINT_FTRACE_INFO 0x00000010 -#define PANIC_PRINT_ALL_PRINTK_MSG 0x00000020 -#define PANIC_PRINT_ALL_CPU_BT 0x00000040 -#define PANIC_PRINT_BLOCKED_TASKS 0x00000080 unsigned long panic_print; =20 ATOMIC_NOTIFIER_HEAD(panic_notifier_list); @@ -208,31 +200,39 @@ void nmi_panic(struct pt_regs *regs, const char *msg) } EXPORT_SYMBOL(nmi_panic); =20 -static void panic_print_sys_info(bool console_flush) +void sys_show_info(unsigned long info_mask) { - if (console_flush) { - if (panic_print & PANIC_PRINT_ALL_PRINTK_MSG) - console_flush_on_panic(CONSOLE_REPLAY_ALL); - return; - } - - if (panic_print & PANIC_PRINT_TASK_INFO) + if (info_mask & SYS_PRINT_TASK_INFO) show_state(); =20 - if (panic_print & PANIC_PRINT_MEM_INFO) + if (info_mask & SYS_PRINT_MEM_INFO) show_mem(); =20 - if (panic_print & PANIC_PRINT_TIMER_INFO) + if (info_mask & SYS_PRINT_TIMER_INFO) sysrq_timer_list_show(); =20 - if (panic_print & PANIC_PRINT_LOCK_INFO) + if (info_mask & SYS_PRINT_LOCK_INFO) debug_show_all_locks(); =20 - if (panic_print & PANIC_PRINT_FTRACE_INFO) + if (info_mask & SYS_PRINT_FTRACE_INFO) ftrace_dump(DUMP_ALL); =20 - if (panic_print & PANIC_PRINT_BLOCKED_TASKS) + if (info_mask & SYS_PRINT_BLOCKED_TASKS) show_state_filter(TASK_UNINTERRUPTIBLE); + + if (info_mask & SYS_PRINT_ALL_CPU_BT) + trigger_all_cpu_backtrace(); +} + +static void panic_print_sys_info(bool console_flush) +{ + if (console_flush) { + if (panic_print & SYS_PRINT_ALL_PRINTK_MSG) + console_flush_on_panic(CONSOLE_REPLAY_ALL); + return; + } + + sys_show_info(panic_print); } =20 void check_panic_on_warn(const char *origin) @@ -255,7 +255,7 @@ void check_panic_on_warn(const char *origin) */ static void panic_other_cpus_shutdown(bool crash_kexec) { - if (panic_print & PANIC_PRINT_ALL_CPU_BT) { + if (panic_print & SYS_PRINT_ALL_CPU_BT) { /* Temporary allow non-panic CPUs to write their backtraces. */ panic_triggering_all_cpu_backtrace =3D true; trigger_all_cpu_backtrace(); --=20 2.39.5 (Apple Git-154) From nobody Thu Dec 18 03:21:15 2025 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE1DA21480D for ; Wed, 7 May 2025 10:43:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746614611; cv=none; b=Xz31qjhvBY3xq/66Qu3ZkJVdupT7RcsmtM2NijmVZSBh2nME0UhvlUEdypCymCEYBOfaraZzjHlTHBvpY3YgOyjyJB+bbD4e2tU/cOMeAtFXBw3AAyZjZsJ3VEr2ufQDiDFvdqFjS6nsB70OIKGljTOnhptXpHA31HQ8WJsl9ds= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746614611; c=relaxed/simple; bh=cYqyqAfakJpWQEz5sImgF3obdoE1LtUan2iUEcKmPQ8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RNww5485nwddlgpkLgQEQOgp0zV4ktrd9B7S1CoOiNHK7sW/hW0xnkLZP5D65La6eGqGBrTBmCxc3ckfqrpGj8ufOBi1PAYwkwfTWFZxEuVImRdbVa5YG/m2rxvz/+zluyD/GwACflBhhuOED+z/x2qiwPXH330mgs1YqG/8CMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=LOeG5q+u; arc=none smtp.client-ip=115.124.30.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="LOeG5q+u" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1746614605; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=j/B0xu3DJe4JpL8FQQd96mEM5+oX8xJXLKWeAxyHaSg=; b=LOeG5q+u4SE1SI/KiT8/vPmKRaSa0ohIgDSmsjcBe66OKNPRTXGqOApRuS8RQOH/UIPFTdcx6d5+QQOHdcVtdvyUonH+lKPGO6PZczbSbGjxUvNf+hKWQgGvhNhmIw7fbyJAYf7vzzPNaL76o0JMBaxl4kKvTr+y/e7yoc2UObI= Received: from localhost(mailfrom:feng.tang@linux.alibaba.com fp:SMTPD_---0WZpgPjC_1746614604 cluster:ay36) by smtp.aliyun-inc.com; Wed, 07 May 2025 18:43:24 +0800 From: Feng Tang To: Andrew Morton , Petr Mladek , Steven Rostedt , Lance Yang , linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [PATCH RFC 2/3] kernel/hung_task: add option to dump system info when hung task detected Date: Wed, 7 May 2025 18:43:21 +0800 Message-Id: <20250507104322.30700-3-feng.tang@linux.alibaba.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250507104322.30700-1-feng.tang@linux.alibaba.com> References: <20250507104322.30700-1-feng.tang@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel panic code utilizes sys_show_info() to dump needed system information to help debugging. Similarly, add this debug option for task hung case, and 'hungtask_print' is the knob to control what information should be printed out. Also clean up the code about dumping locks and triggering backtrace for all CPUs. One todo may be to merge this 'hungtask_print' with some sysctl knobs in hung_task.c. Signed-off-by: Feng Tang --- kernel/hung_task.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/kernel/hung_task.c b/kernel/hung_task.c index dc898ec93463..8229637be2c7 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -58,12 +58,20 @@ static unsigned long __read_mostly sysctl_hung_task_che= ck_interval_secs; static int __read_mostly sysctl_hung_task_warnings =3D 10; =20 static int __read_mostly did_panic; -static bool hung_task_show_lock; static bool hung_task_call_panic; -static bool hung_task_show_all_bt; =20 static struct task_struct *watchdog_task; =20 +/* + * A bitmask to control what kinds of system info to be printed when a + * hung task is detected, it could be task, memory, lock etc. Refer panic.h + * for details of bit definition. + */ +unsigned long hungtask_print; +core_param(hungtask_print, hungtask_print, ulong, 0644); + +static unsigned long cur_hungtask_print; + #ifdef CONFIG_SMP /* * Should we dump all CPUs backtraces in a hung task event? @@ -163,11 +171,12 @@ static void check_hung_task(struct task_struct *t, un= signed long timeout) */ sysctl_hung_task_detect_count++; =20 + cur_hungtask_print =3D hungtask_print; trace_sched_process_hang(t); =20 if (sysctl_hung_task_panic) { console_verbose(); - hung_task_show_lock =3D true; + cur_hungtask_print |=3D SYS_PRINT_LOCK_INFO; hung_task_call_panic =3D true; } =20 @@ -190,10 +199,10 @@ static void check_hung_task(struct task_struct *t, un= signed long timeout) " disables this message.\n"); sched_show_task(t); debug_show_blocker(t); - hung_task_show_lock =3D true; + cur_hungtask_print |=3D SYS_PRINT_LOCK_INFO; =20 if (sysctl_hung_task_all_cpu_backtrace) - hung_task_show_all_bt =3D true; + cur_hungtask_print |=3D SYS_PRINT_ALL_CPU_BT; if (!sysctl_hung_task_warnings) pr_info("Future hung task reports are suppressed, see sysctl kernel.hun= g_task_warnings\n"); } @@ -242,7 +251,7 @@ static void check_hung_uninterruptible_tasks(unsigned l= ong timeout) if (test_taint(TAINT_DIE) || did_panic) return; =20 - hung_task_show_lock =3D false; + cur_hungtask_print =3D 0; rcu_read_lock(); for_each_process_thread(g, t) { unsigned int state; @@ -266,14 +275,8 @@ static void check_hung_uninterruptible_tasks(unsigned = long timeout) } unlock: rcu_read_unlock(); - if (hung_task_show_lock) - debug_show_all_locks(); - - if (hung_task_show_all_bt) { - hung_task_show_all_bt =3D false; - trigger_all_cpu_backtrace(); - } =20 + sys_show_info(cur_hungtask_print); if (hung_task_call_panic) panic("hung_task: blocked tasks"); } --=20 2.39.5 (Apple Git-154) From nobody Thu Dec 18 03:21:15 2025 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E33A21505C for ; Wed, 7 May 2025 10:43:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746614616; cv=none; b=ZzeAlUqLrf6+K/+VHOuaScEDgBURAn9JmQIULx61IfV1a5fQX6/+biLGOK50fWyqR53L3WVRsO9Rnxd9TLXlt+evBaft/wEwYxx5LeRvUgvY0Vjh7lMhnPZL2rbBV9aC4LWLF9Tbk3D+1ZCrunduiVNfzS/UdLC1UAZd9CjB3h4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746614616; c=relaxed/simple; bh=Qo/ZrceoQTXto2/HZEMEwY9kyey3PehYg+COl4yhazc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=L6GGMCmCQtg9Zd0qe6RoLUCwEKZTj/75iOv6f0pIqiArqr5aXbyN8X39frRawvxZVdlPOCWGwqrJWbjMlTHW5GWPe3G3WoINPY5SIrlGeXBTPs1fpbSn2FToHGq+j0T7McUf4sPgM4ghT/ebqntceZJjSdq4trJvoqIkbM5W8+g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=P7sLMU/h; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="P7sLMU/h" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1746614605; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=jZyeNUllhEjWwZXWgeTBRRxB9T75NXtWWR44tfcSckE=; b=P7sLMU/hKClolkviU21p+2v7Ma1g3Lf5jW/gC+aLpGszbUJY1b0KijLYqve3AueEhwDAh0I9RiPP7HvWQPvgtUnfOr38PpJjfQGIQ3VULbC9piv8vdzKHAv1i0lmGA4I+jFJTX+w7TsmLC6DbiV137a/IgRJusxoAbV6Emyqbg8= Received: from localhost(mailfrom:feng.tang@linux.alibaba.com fp:SMTPD_---0WZpa-8q_1746614604 cluster:ay36) by smtp.aliyun-inc.com; Wed, 07 May 2025 18:43:25 +0800 From: Feng Tang To: Andrew Morton , Petr Mladek , Steven Rostedt , Lance Yang , linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [PATCH RFC 3/3] kernel/watchdog: add option to dump system info when system is locked up Date: Wed, 7 May 2025 18:43:22 +0800 Message-Id: <20250507104322.30700-4-feng.tang@linux.alibaba.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250507104322.30700-1-feng.tang@linux.alibaba.com> References: <20250507104322.30700-1-feng.tang@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel panic code utilizes sys_show_info() to dump needed system information to help debugging. Similarly, add this debug option for software/hardware lockup cases, and 'lockup_print' is the knob to control what information should be printed out. Signed-off-by: Feng Tang --- kernel/watchdog.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 9fa2af9dbf2c..60afcb0247ab 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -52,6 +52,14 @@ static int __read_mostly watchdog_hardlockup_available; struct cpumask watchdog_cpumask __read_mostly; unsigned long *watchdog_cpumask_bits =3D cpumask_bits(&watchdog_cpumask); =20 +/* + * A bitmask to control what kinds of system info to be printed when a + * software/hardware lockup is detected, it could be task, memory, lock et= c. + * Refer panic.h for details of bit definition. + */ +unsigned long lockup_print; +core_param(lockup_print, lockup_print, ulong, 0644); + #ifdef CONFIG_HARDLOCKUP_DETECTOR =20 # ifdef CONFIG_SMP @@ -212,6 +220,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct= pt_regs *regs) clear_bit_unlock(0, &hard_lockup_nmi_warn); } =20 + sys_show_info(lockup_print); if (hardlockup_panic) nmi_panic(regs, "Hard LOCKUP"); =20 @@ -774,6 +783,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hr= timer *hrtimer) } =20 add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); + + sys_show_info(lockup_print); if (softlockup_panic) panic("softlockup: hung tasks"); } --=20 2.39.5 (Apple Git-154)