From nobody Tue Nov 26 02:35:07 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C50345026 for ; Tue, 22 Oct 2024 11:48:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729597694; cv=none; b=FLMbTe0HI8h1/9Yc9Up4y/E3rtl9HIugyDboRg3/QamPxKotkPtD2ZunlFqxc9ZAj62kVT4WQIGgvc1wdZjWwzAIB92xXB74vr2Va3rrCpaU+y9h8jvIFEdE7qni7jkg//vDMcmjI125H2zDBcYt6hrbuFLWLselY5GXeeDXXtI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729597694; c=relaxed/simple; bh=Bqeo9yMnwygaahE35FD8nIgjrAjptFnrJVmtoyrIb+w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xa+7Q2qH5R5GoVZAlISCD6hrBuwYj+9h0oA6SjeIyYnpZJoook327oFgUZ0m6tvnQ0zlzG+GjzgrvPSCkMMpqyBBSGWT/D33PUP2HNk/0BfikIjP2ESo5J3K9Kt4l3978wtm7ue81De5kDk0YO0LS4XB66knJ+t8ijFqlubDQVc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aYvXo9ZT; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aYvXo9ZT" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-20caccadbeeso60121905ad.2 for ; Tue, 22 Oct 2024 04:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729597692; x=1730202492; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hKl6U/5kYqIul35IkUGRHOkC+Sxs+wAsHazeOfjnayo=; b=aYvXo9ZT6rX5sSNKgHW2RQUalCZYiz/YmBoHYTQUX+dVsCTmTlnPkVhRL2DW7HOL7J knJRBdLjyTztjkRuWy8zEOFeoMT3sUsg56XWqirzqB2X5VS9chtrCbW0Cu9cjVjCknCW fUKZs8bJGIkuiu4+5j7TamgvEDFgi5kcj4jv9Q3Ldyv1oq2jRiCUNCQ29jZnHcZ6cLcB sNCGANwn90NuF8amMt4PG8vOG5QnstSO75eBO9Kb6CyfRk5cY8QAK/Bd2VI3/kLPgPiX q3flEcAUVDMDRdWroNx7hBRpBZMwKzEPKdO/Rhbv2063HesFzKasLe3Lb/j31IYdhwzv dY/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729597692; x=1730202492; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hKl6U/5kYqIul35IkUGRHOkC+Sxs+wAsHazeOfjnayo=; b=p6g/4rJCYKSbC84SQnTtFMVXv0JBglG/Q/n1yl0+OixgeIE0vpFx4wPcA5mNzZ3POq BSm6VgJ8piDSDHFxBl7DwfzyBHKWmXIqILFvQeNHE1TCzY8kxPVmYy9AYjk2YiPAXhgD JDXHBjN4ISAumYbFhuqo+cvo9WKGJSy509MVII011Y0rU+aUS4W31R7YVY4+4voHPl9r SNYdsw5MJhZqx+iUAlTnhYmpeTY7XO6egYEzzOrdaoSkfqFc+/WyTX5afaZcGmb9/MFs hyRPlfEbsrGPzf6QRJCN0ihAF9gtZ6olGWcdVokQFehm7wl1W0Qn0uQLqcT/ynYMbGSV ky+g== X-Forwarded-Encrypted: i=1; AJvYcCUiGdE1R/QQOYiBu1qOTtyGvY8kJ+PEW6fC3NJrO5fqmB1bDwBoH50+Z9gnIAcIMEdpJdTjSYygj1148dA=@vger.kernel.org X-Gm-Message-State: AOJu0Yze5u0PgXqnHGYew3xjWFNSS7LSGBcWizyKYW7a/A7s2nCSx4lD FlXHaDrAagWncEqR3n/bDFbs3O+RRtmBrNBMf155qbrHO5SF3XhC X-Google-Smtp-Source: AGHT+IHiGn0rGlYqLqA+F/OWg8LpaBp6Ymm27XpKX4Az/xFYXPaV3FzGzu4YKfbwVcSJ4WTJHHB2HQ== X-Received: by 2002:a17:902:ea0a:b0:20b:70b4:69d8 with SMTP id d9443c01a7336-20e984a5a57mr20839455ad.37.1729597692576; Tue, 22 Oct 2024 04:48:12 -0700 (PDT) Received: from EBJ9932692.tcent.cn ([43.129.202.66]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7ef0e13bsm41147145ad.65.2024.10.22.04.48.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Oct 2024 04:48:12 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: cunhuang@tencent.com, leonylgao@tencent.com, j.granados@samsung.com, jsiddle@redhat.com, kent.overstreet@linux.dev, 21cnbao@gmail.com, ryan.roberts@arm.com, david@redhat.com, ziy@nvidia.com, libang.li@antgroup.com, baolin.wang@linux.alibaba.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lance Yang , Mingzhe Yang Subject: [PATCH 1/2] hung_task: add detect count for hung tasks Date: Tue, 22 Oct 2024 19:47:35 +0800 Message-ID: <20241022114736.83285-2-ioworker0@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241022114736.83285-1-ioworker0@gmail.com> References: <20241022114736.83285-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds a counter, hung_task_detect_count, to track the number of times hung tasks are detected. This counter provides a straightforward way to monitor hung task events without manually checking dmesg logs. With this counter in place, system issues can be spotted quickly, allowing admins to step in promptly before system load spikes occur, even if the hung_task_warnings value has been decreased to 0 well before. Signed-off-by: Mingzhe Yang Signed-off-by: Lance Yang --- kernel/hung_task.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 959d99583d1c..229ff3d4e501 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -30,6 +30,11 @@ */ static int __read_mostly sysctl_hung_task_check_count =3D PID_MAX_LIMIT; =20 +/* + * Total number of tasks detected as hung since boot: + */ +static unsigned long __read_mostly sysctl_hung_task_detect_count; + /* * Limit number of tasks checked in a batch. * @@ -115,6 +120,12 @@ static void check_hung_task(struct task_struct *t, uns= igned long timeout) if (time_is_after_jiffies(t->last_switch_time + timeout * HZ)) return; =20 + /* + * This counter tracks the total number of tasks detected as hung + * since boot. + */ + sysctl_hung_task_detect_count++; + trace_sched_process_hang(t); =20 if (sysctl_hung_task_panic) { @@ -314,6 +325,13 @@ static struct ctl_table hung_task_sysctls[] =3D { .proc_handler =3D proc_dointvec_minmax, .extra1 =3D SYSCTL_NEG_ONE, }, + { + .procname =3D "hung_task_detect_count", + .data =3D &sysctl_hung_task_detect_count, + .maxlen =3D sizeof(unsigned long), + .mode =3D 0444, + .proc_handler =3D proc_dointvec, + }, }; =20 static void __init hung_task_sysctl_init(void) --=20 2.45.2 From nobody Tue Nov 26 02:35:07 2024 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7ED1519AA5F for ; Tue, 22 Oct 2024 11:48:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729597700; cv=none; b=WF/N0qibnuw9QimSzhwfQCqUJ3FXvbRaOTBkP7QT4BO84r9GHEWEVpzFMTDAKpqM8LQ44GMuS15k3g742apbf/V/TNGGjPz8cV+vFEkRdugCxmPznqmc08J6BhASG1sYM6qqKQ0+HuyTd2+XLOxGPkmKCQDeeRpxwVAWNLpSce8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729597700; c=relaxed/simple; bh=D/ynVHYmqi2Ckh7aFWJwqCLmolJOnpYqtWvCpDp09O8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PYalKjpaNkQ2On32w9y/TZQVP9IZrakV8e3s3jEp9TwHzJrkf9osnjIVhpFK45v1EIHFiUOa7bfAyq1GDcwAIAYyMekIKsI6lJ+htGQpA/41OrBVoOaaQndz7n0+aYdZjDDySjWq+bUZV0IQkDsd3P5KFjni4n7Dew43CjXgLsM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ka+z0jPY; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ka+z0jPY" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-20c693b68f5so58780845ad.1 for ; Tue, 22 Oct 2024 04:48:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729597699; x=1730202499; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BAlFXpVPV8uFWMmnJmNJ4B3bgnH5ueF+92q0u74/c1Q=; b=ka+z0jPYwXn8wuxQFGqkueMPImjbSiseU9vdxhROsJMltZDxPTJ+sHk+j/Izt1EZQW RpkCsoYUGcG+f/EPdKyAYCneIEW7S8Wu3pDGLQmuWXaxC07l2W1/HJOWLhbmZa01tD6V viDniqCIJIFvgMdjI5IugKPbZvnbksdPRiUyiR4vJAxeNqyRYvKcRg5djkSr4Q8mywD1 o+znp0ljGOyl7MIJXoaja4EAm9uld/SCG8WCls+t/5VoivWusflN6ul9mWHGu3Ks2pd6 KKC8MS17BBM9/W861ALmVFgdIsN2xyRj16zYXLC65Gnoa9yUTH+8RaNMuG811F4GIZ3G jnUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729597699; x=1730202499; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BAlFXpVPV8uFWMmnJmNJ4B3bgnH5ueF+92q0u74/c1Q=; b=H3eDoKH31/ahRNNznHswD/4M/PzfdCkLbOz9X5qhnPxGbkSZ/Lq31KcmlaM/LMFzse iEI9SU5NSckjrIyXOLTSmkUbQaDwmcl2KPFcX0pbiC65xIfsehsU6m/M1pJclwGE5YPr zfVAG3Ev+zmv7kQv0zO8sNj0raLAVKqgwLiNFXUOh6QHHLGUM3apijA0Vuq3im5/ZnYa LLZumTX4b/ZNQQr7urOf5rtc7rZiQo5i2ESMHsXjXRyfSXrsOJ5prEujTmtzLchnMXQ/ CUC2oVOrzvuWn+EUi/4DPGF9/nRuIdKjzJCrzceqW7V0CeKKG6Po5zsTJkesHxYfrGb+ YSaA== X-Forwarded-Encrypted: i=1; AJvYcCW+So5L5m+Cb7ExyzEHy6Eq3j7/kDINSnqxOfaItqVFGuuslqLiXzlHOFypcucpUn7gmecvU4xHUQeMnaY=@vger.kernel.org X-Gm-Message-State: AOJu0YwArRs4D8K4s1XTIVhiStjxrsHzoNbT1+RfXH/hVSU7y7iliZTj j490XonBxkMiVEStKhClmnNTw+NGtHWJjhT/P5uBJLZq13qucE5lDWVWcA== X-Google-Smtp-Source: AGHT+IG0hzVA6fwCTy7pJ9a3mstX945h4KMDaq2ak8MlWEnvH6gaqUQZykyWc+97P6hdZN/L/wWRag== X-Received: by 2002:a17:903:2344:b0:20b:5439:f194 with SMTP id d9443c01a7336-20e984706f9mr31355125ad.16.1729597698507; Tue, 22 Oct 2024 04:48:18 -0700 (PDT) Received: from EBJ9932692.tcent.cn ([43.129.202.66]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7ef0e13bsm41147145ad.65.2024.10.22.04.48.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Oct 2024 04:48:18 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: cunhuang@tencent.com, leonylgao@tencent.com, j.granados@samsung.com, jsiddle@redhat.com, kent.overstreet@linux.dev, 21cnbao@gmail.com, ryan.roberts@arm.com, david@redhat.com, ziy@nvidia.com, libang.li@antgroup.com, baolin.wang@linux.alibaba.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lance Yang , Mingzhe Yang Subject: [PATCH 2/2] hung_task: add docs for hung_task_detect_count Date: Tue, 22 Oct 2024 19:47:36 +0800 Message-ID: <20241022114736.83285-3-ioworker0@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241022114736.83285-1-ioworker0@gmail.com> References: <20241022114736.83285-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit introduces documentation for hung_task_detect_count in kernel.rst. Signed-off-by: Mingzhe Yang Signed-off-by: Lance Yang --- Documentation/admin-guide/sysctl/kernel.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/ad= min-guide/sysctl/kernel.rst index f8bc1630eba0..b2b36d0c3094 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -401,6 +401,15 @@ The upper bound on the number of tasks that are checke= d. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. =20 =20 +hung_task_detect_count +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Indicates the total number of tasks that have been detected as hung since +the system boot. + +This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. + + hung_task_timeout_secs =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 --=20 2.45.2