From nobody Fri Dec 19 21:33:37 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C0651E8326 for ; Wed, 19 Feb 2025 13:00:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739970053; cv=none; b=RqwO7BTt3li/ID7JcCFNOVRUIDGlqKTtrcMm74stbU9b5C+0bAm5MqHgAMK3i06fVzsvSuVQtN82lbTBbvpxJA56653xDnAx8ax5Awr4hYIrNBGlfR099rIpr8mKpbGg7cPpXWkgtxrJHu54lpjM72YhLarLQFco08MDKGCkM5U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739970053; c=relaxed/simple; bh=sb6Wf5NKqPEY4wAFsfQBGRH2U+v7+V075jzr4+QqN74=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MGR9e+r6XLR/zKbz+9yinlRdU6fDyqcGiddgdV7p6ylP0hu3VuXu7LUU2hviRKjEx21Nax76mg0zupFekQiIRXB+T81h4NBN92vN4cSmwm3l6u0wzVZZp7ZqL0KoGM1qTR6mbSa/E3QI5ho4WZCswhS65Z3AhMGZXWpblAi9Cto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=emwPC9J8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="emwPC9J8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09E6BC4CED1; Wed, 19 Feb 2025 13:00:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739970052; bh=sb6Wf5NKqPEY4wAFsfQBGRH2U+v7+V075jzr4+QqN74=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=emwPC9J84b5EDAVdhlCu/qnrEWrlwHqSZ845uY8rc7wTOvKfRyIC21S2OCeAGMmyj uLkWNuDwWw4Rqh/KlExx4VCwxuQ0iazs6v5L+gkOByiiUG3gKVV+xXxAGV6BEakCnX ZR+QJRwN1lXz7LMiw9FXtzZ619RcRWWi5DnCiAlR8dfr0Fpchlu0Rm0dIsddtVljpd zFfxTkAv0eTh3BIQVHEcaUg7OpXuerkJZCn10ZNUkVUNd6Psd2sX3kxPbT+tc71/J+ 8a/KE+9PYvCv6VvjiMrmu/fJ1oLEZYNdVeIetSIEveKDj/cYrOU4VryM/SIeadapCg dRqC7Wk9Ou5Cg== From: "Masami Hiramatsu (Google)" To: Peter Zijlstra , Ingo Molnar , Will Deacon , Andrew Morton Cc: Boqun Feng , Waiman Long , Joel Granados , Masami Hiramatsu , Anna Schumaker , Lance Yang , Kent Overstreet , Yongliang Gao , Steven Rostedt , Tomasz Figa , Sergey Senozhatsky , linux-kernel@vger.kernel.org Subject: [PATCH 1/2] hung_task: Show the blocker task if the task is hung on mutex Date: Wed, 19 Feb 2025 22:00:49 +0900 Message-ID: <173997004932.2137198.7959507113210521328.stgit@mhiramat.tok.corp.google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <173997003868.2137198.9462617208992136056.stgit@mhiramat.tok.corp.google.com> References: <173997003868.2137198.9462617208992136056.stgit@mhiramat.tok.corp.google.com> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Masami Hiramatsu (Google) The "hung_task" shows a long-time uninterruptible slept task, but most often, it's blocked on a mutex acquired by another task. Without dumping such a task, investigating the root cause of the hung task problem is very difficult. Fortunately CONFIG_DEBUG_MUTEXES=3Dy allows us to identify the mutex blocking the task. And the mutex has "owner" information, which can be used to find the owner task and dump it with hung tasks. With this change, the hung task shows blocker task's info like below; INFO: task cat:113 blocked for more than 122 seconds. Not tainted 6.14.0-rc3-00002-g6afe972e1b9b #152 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:cat state:D stack:13432 pid:113 tgid:113 ppid:103 = task_flags:0x400100 flags:0x00000002 Call Trace: __schedule+0x731/0x960 ? schedule_preempt_disabled+0x54/0xa0 schedule+0xb7/0x140 ? __mutex_lock+0x51d/0xa50 ? __mutex_lock+0x51d/0xa50 schedule_preempt_disabled+0x54/0xa0 __mutex_lock+0x51d/0xa50 ? current_time+0x3a/0x120 read_dummy+0x23/0x70 full_proxy_read+0x6a/0xc0 vfs_read+0xc2/0x340 ? __pfx_direct_file_splice_eof+0x10/0x10 ? do_sendfile+0x1bd/0x2e0 ksys_read+0x76/0xe0 do_syscall_64+0xe3/0x1c0 ? exc_page_fault+0xa9/0x1d0 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x4840cd RSP: 002b:00007ffe632b76c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004840cd RDX: 0000000000001000 RSI: 00007ffe632b7710 RDI: 0000000000000003 RBP: 00007ffe632b7710 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000001000 R13: 000000003a8b63a0 R14: 0000000000000001 R15: ffffffffffffffff INFO: task cat:113 is blocked on a mutex owned by task cat:112. task:cat state:S stack:13432 pid:112 tgid:112 ppid:103 = task_flags:0x400100 flags:0x00000002 Call Trace: __schedule+0x731/0x960 ? schedule_timeout+0xa8/0x120 schedule+0xb7/0x140 schedule_timeout+0xa8/0x120 ? __pfx_process_timeout+0x10/0x10 msleep_interruptible+0x3e/0x60 read_dummy+0x2d/0x70 full_proxy_read+0x6a/0xc0 vfs_read+0xc2/0x340 ? __pfx_direct_file_splice_eof+0x10/0x10 ? do_sendfile+0x1bd/0x2e0 ksys_read+0x76/0xe0 do_syscall_64+0xe3/0x1c0 ? exc_page_fault+0xa9/0x1d0 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x4840cd RSP: 002b:00007ffd69513748 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004840cd RDX: 0000000000001000 RSI: 00007ffd69513790 RDI: 0000000000000003 RBP: 00007ffd69513790 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000001000 R13: 0000000029d8d3a0 R14: 0000000000000001 R15: ffffffffffffffff Signed-off-by: Masami Hiramatsu (Google) --- kernel/hung_task.c | 38 ++++++++++++++++++++++++++++++++++++++ kernel/locking/mutex-debug.c | 1 + kernel/locking/mutex.c | 9 +++++++++ kernel/locking/mutex.h | 6 ++++++ 4 files changed, 54 insertions(+) diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 04efa7a6e69b..d1ce69504090 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -25,6 +25,8 @@ =20 #include =20 +#include "locking/mutex.h" + /* * The number of tasks checked: */ @@ -93,6 +95,41 @@ static struct notifier_block panic_block =3D { .notifier_call =3D hung_task_panic, }; =20 + +#ifdef CONFIG_DEBUG_MUTEXES +static void debug_show_blocker(struct task_struct *task) +{ + struct task_struct *g, *t; + unsigned long owner; + struct mutex *lock; + + if (!task->blocked_on) + return; + + lock =3D task->blocked_on->mutex; + if (unlikely(!lock)) { + pr_err("INFO: task %s:%d is blocked on a mutex, but the mutex is not fou= nd.\n", + task->comm, task->pid); + return; + } + owner =3D debug_mutex_get_owner(lock); + if (likely(owner)) { + /* Ensure the owner information is correct. */ + for_each_process_thread(g, t) + if ((unsigned long)t =3D=3D owner) { + pr_err("INFO: task %s:%d is blocked on a mutex owned by task %s:%d.\n", + task->comm, task->pid, t->comm, t->pid); + sched_show_task(t); + return; + } + } + pr_err("INFO: task %s:%d is blocked on a mutex, but the owner is not foun= d.\n", + task->comm, task->pid); +} +#else +#define debug_show_blocker(t) do {} while (0) +#endif + static void check_hung_task(struct task_struct *t, unsigned long timeout) { unsigned long switch_count =3D t->nvcsw + t->nivcsw; @@ -152,6 +189,7 @@ static void check_hung_task(struct task_struct *t, unsi= gned long timeout) pr_err("\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\"" " disables this message.\n"); sched_show_task(t); + debug_show_blocker(t); hung_task_show_lock =3D true; =20 if (sysctl_hung_task_all_cpu_backtrace) diff --git a/kernel/locking/mutex-debug.c b/kernel/locking/mutex-debug.c index 6e6f6071cfa2..94282c760cee 100644 --- a/kernel/locking/mutex-debug.c +++ b/kernel/locking/mutex-debug.c @@ -30,6 +30,7 @@ void debug_mutex_lock_common(struct mutex *lock, struct m= utex_waiter *waiter) { memset(waiter, MUTEX_DEBUG_INIT, sizeof(*waiter)); waiter->magic =3D waiter; + waiter->mutex =3D lock; INIT_LIST_HEAD(&waiter->list); waiter->ww_ctx =3D MUTEX_POISON_WW_CTX; } diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index b36f23de48f1..db093cc0a220 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -72,6 +72,15 @@ static inline unsigned long __owner_flags(unsigned long = owner) return owner & MUTEX_FLAGS; } =20 +#ifdef CONFIG_DEBUG_MUTEXES +/* Do not use the return value as a pointer directly. */ +unsigned long debug_mutex_get_owner(struct mutex *lock) +{ + unsigned long owner =3D atomic_long_read(&lock->owner); + + return (unsigned long)__owner_task(owner); +} +#endif /* * Returns: __mutex_owner(lock) on failure or NULL on success. */ diff --git a/kernel/locking/mutex.h b/kernel/locking/mutex.h index cbff35b9b7ae..0d884e6b3a66 100644 --- a/kernel/locking/mutex.h +++ b/kernel/locking/mutex.h @@ -16,6 +16,7 @@ struct mutex_waiter { struct task_struct *task; struct ww_acquire_ctx *ww_ctx; #ifdef CONFIG_DEBUG_MUTEXES + struct mutex *mutex; void *magic; #endif }; @@ -61,6 +62,7 @@ extern void debug_mutex_remove_waiter(struct mutex *lock,= struct mutex_waiter *w extern void debug_mutex_unlock(struct mutex *lock); extern void debug_mutex_init(struct mutex *lock, const char *name, struct lock_class_key *key); +extern unsigned long debug_mutex_get_owner(struct mutex *lock); #else /* CONFIG_DEBUG_MUTEXES */ # define debug_mutex_lock_common(lock, waiter) do { } while (0) # define debug_mutex_wake_waiter(lock, waiter) do { } while (0) @@ -69,4 +71,8 @@ extern void debug_mutex_init(struct mutex *lock, const ch= ar *name, # define debug_mutex_remove_waiter(lock, waiter, ti) do { } while (0) # define debug_mutex_unlock(lock) do { } while (0) # define debug_mutex_init(lock, name, key) do { } while (0) +static inline unsigned long debug_mutex_get_owner(struct mutex *lock) +{ + return 0; +} #endif /* !CONFIG_DEBUG_MUTEXES */ From nobody Fri Dec 19 21:33:37 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C25F1EA7EA for ; Wed, 19 Feb 2025 13:01:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739970062; cv=none; b=qtmus0bcU67KqRd8wJHbTBqhq9SnYGTOc7rQ533X35ujz94pSue42D5RwnzFki38tfI8Wa/cZLLsJWSvamWZy0/dycznZ0lC6PFTRYMPbVBOVybqS4pLAscEOIybU2NMkEwb5NJz4+AZoPYFV0NNSJ6u7Xe/cuUtxYhPt8rv3gc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739970062; c=relaxed/simple; bh=yRt/VaZXQuh0DYvzuCq9THX6LnIgF5KQbpuqcMQ4SZs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fsQ6t/14XaEROZT++0mpUTkpLtAUsNpPz8JuSPR9AKBDU6y2a43ry8150Xlvu1kSg/v+bZcERsNUIG/kdntV2TINPl0tTLqtVtXQ2s8Wq4T6JTKP31laMFWR+H79fVz/FMPycohDBL7j/SekWD0+yzSqn+L1J7QNrIbZCMI9cfk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lCcnFldD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lCcnFldD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D690CC4CED1; Wed, 19 Feb 2025 13:00:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739970061; bh=yRt/VaZXQuh0DYvzuCq9THX6LnIgF5KQbpuqcMQ4SZs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lCcnFldD0m+q/ylAlrkWh8wDO68zQUs+Fhb+80rATiw1vEBeLLczY8edQtLr6IImG 3BbLY0OILVuMSdiVFvaCkfG2IW/YwW2X2/iLIQl7Eff8xpp6PEODAgnvaeFPQIoo+e jNzEUCsNInV4iAX3kznlG26MnJ6DwpVRWH9LlqfNRXMLDJnsE7B+8LdZkC+MkpJZna KVRNSBPjgMPUij2X33DD4DdOuqDYRc++Gkzvc3ZmWLQc5bMdXHhHkN4iU/r0lzOT07 PyU/m/hL+YmuAOWIc8bS58GlQQ7MPTl1x9lNgkQA6pKJedBDf7d8hc88Gz7kqBDv7J bKUS9OnwgPNWQ== From: "Masami Hiramatsu (Google)" To: Peter Zijlstra , Ingo Molnar , Will Deacon , Andrew Morton Cc: Boqun Feng , Waiman Long , Joel Granados , Masami Hiramatsu , Anna Schumaker , Lance Yang , Kent Overstreet , Yongliang Gao , Steven Rostedt , Tomasz Figa , Sergey Senozhatsky , linux-kernel@vger.kernel.org Subject: [PATCH 2/2] samples: Add hung_task detector mutex blocking sample Date: Wed, 19 Feb 2025 22:00:58 +0900 Message-ID: <173997005819.2137198.2020111976324308587.stgit@mhiramat.tok.corp.google.com> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <173997003868.2137198.9462617208992136056.stgit@mhiramat.tok.corp.google.com> References: <173997003868.2137198.9462617208992136056.stgit@mhiramat.tok.corp.google.com> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Masami Hiramatsu (Google) Add a hung_task detector mutex blocking test sample code. This module will create a dummy file on the debugfs. That file will cause the read process to sleep for enough long time (256 seconds) while holding a mutex. As a result, the second process will wait on the mutex for a prolonged duration and be detected by the hung_task detector. Usage is; > cd /sys/kernel/debug/hung_task > cat mutex & cat mutex and wait for hung_task message. Signed-off-by: Masami Hiramatsu (Google) --- samples/Kconfig | 9 +++++ samples/Makefile | 1 + samples/hung_task/Makefile | 2 + samples/hung_task/hung_task_mutex.c | 66 +++++++++++++++++++++++++++++++= ++++ 4 files changed, 78 insertions(+) create mode 100644 samples/hung_task/Makefile create mode 100644 samples/hung_task/hung_task_mutex.c diff --git a/samples/Kconfig b/samples/Kconfig index 820e00b2ed68..09011be2391a 100644 --- a/samples/Kconfig +++ b/samples/Kconfig @@ -300,6 +300,15 @@ config SAMPLE_CHECK_EXEC demonstrate how they should be used with execveat(2) + AT_EXECVE_CHECK. =20 +config SAMPLE_HUNG_TASK + tristate "Hung task detector test code" + depends on DETECT_HUNG_TASK && DEBUG_FS + help + Build a module which provide a simple debugfs file. If user reads + the file, it will sleep long time (256 seconds) with holding a + mutex. Thus if there are 2 or more processes read this file, it + will be detected by the hung_task watchdog. + source "samples/rust/Kconfig" =20 source "samples/damon/Kconfig" diff --git a/samples/Makefile b/samples/Makefile index f24cd0d72dd0..bf6e6fca5410 100644 --- a/samples/Makefile +++ b/samples/Makefile @@ -42,3 +42,4 @@ obj-$(CONFIG_SAMPLE_FPROBE) +=3D fprobe/ obj-$(CONFIG_SAMPLES_RUST) +=3D rust/ obj-$(CONFIG_SAMPLE_DAMON_WSSE) +=3D damon/ obj-$(CONFIG_SAMPLE_DAMON_PRCL) +=3D damon/ +obj-$(CONFIG_SAMPLE_HUNG_TASK) +=3D hung_task/ diff --git a/samples/hung_task/Makefile b/samples/hung_task/Makefile new file mode 100644 index 000000000000..fe9dde799880 --- /dev/null +++ b/samples/hung_task/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_SAMPLE_HUNG_TASK) +=3D hung_task_mutex.o \ No newline at end of file diff --git a/samples/hung_task/hung_task_mutex.c b/samples/hung_task/hung_t= ask_mutex.c new file mode 100644 index 000000000000..7a29f2246d22 --- /dev/null +++ b/samples/hung_task/hung_task_mutex.c @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * hung_task_mutex.c - Sample code which causes hung task by mutex + * + * Usage: load this module and read `/hung_task/mutex` + * by 2 or more processes. + * + * This is for testing kernel hung_task error message. + * Note that this will make your system freeze and maybe + * cause panic. So do not use this except for the test. + */ + +#include +#include +#include +#include +#include + +#define HUNG_TASK_DIR "hung_task" +#define HUNG_TASK_FILE "mutex" +#define SLEEP_SECOND 256 + +static const char dummy_string[] =3D "This is a dummy string."; +static DEFINE_MUTEX(dummy_mutex); +struct dentry *hung_task_dir; + +static ssize_t read_dummy(struct file *file, char __user *user_buf, + size_t count, loff_t *ppos) +{ + /* If the second task waits on the lock, it is uninterruptible sleep. */ + guard(mutex)(&dummy_mutex); + + /* When the first task sleep here, it is interruptible. */ + msleep_interruptible(SLEEP_SECOND * 1000); + + return simple_read_from_buffer(user_buf, count, ppos, + dummy_string, sizeof(dummy_string)); +} + +static const struct file_operations hung_task_fops =3D { + .read =3D read_dummy, +}; + +static int __init hung_task_sample_init(void) +{ + hung_task_dir =3D debugfs_create_dir(HUNG_TASK_DIR, NULL); + if (IS_ERR(hung_task_dir)) + return PTR_ERR(hung_task_dir); + + debugfs_create_file(HUNG_TASK_FILE, 0400, hung_task_dir, + NULL, &hung_task_fops); + + return 0; +} + +static void __exit hung_task_sample_exit(void) +{ + debugfs_remove_recursive(hung_task_dir); +} + +module_init(hung_task_sample_init); +module_exit(hung_task_sample_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Masami Hiramatsu"); +MODULE_DESCRIPTION("Simple sleep under mutex file for testing hung task");