From nobody Thu Dec 18 01:00:58 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B52BDEE49A0 for ; Mon, 21 Aug 2023 20:05:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230234AbjHUUFJ (ORCPT ); Mon, 21 Aug 2023 16:05:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230229AbjHUUFH (ORCPT ); Mon, 21 Aug 2023 16:05:07 -0400 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72A00129 for ; Mon, 21 Aug 2023 13:05:04 -0700 (PDT) Received: from [2601:18c:8180:ac39:6e0b:84ff:fee2:98bb] (helo=imladris.surriel.com) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1qYB8H-0005n1-1a; Mon, 21 Aug 2023 16:04:13 -0400 Date: Mon, 21 Aug 2023 16:04:09 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: kernel-team@meta.com, Peter Zijlstra , "Paul E. McKenney" , Valentin Schneider , Juergen Gross Subject: [PATCH,RFC] smp,csd: throw an error if a CSD lock is stuck for too long Message-ID: <20230821160409.663b8ba9@imladris.surriel.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: riel@surriel.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The CSD lock seems to get stuck in 2 "modes". When it gets stuck temporarily, it usually gets released in a few seconds, and sometimes up to one or two minutes. If the CSD lock stays stuck for more than several minutes, it never seems to get unstuck, and gradually more and more things in the system end up also getting stuck. In the latter case, we should just give up, so the system can dump out a little more information about what went wrong, and, with panic_on_oops and a kdump kernel loaded, dump a whole bunch more information about what might have gone wrong. Question: should this have its own panic_on_ipistall switch in /proc/sys/kernel, or maybe piggyback on panic_on_oops in a different way than via BUG_ON? Signed-off-by: Rik van Riel Reviewed-by: Paul E. McKenney --- kernel/smp.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/kernel/smp.c b/kernel/smp.c index 385179dae360..8b808bff15e6 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -228,6 +228,7 @@ static bool csd_lock_wait_toolong(struct __call_single_= data *csd, u64 ts0, u64 * } =20 ts2 =3D sched_clock(); + /* How long since we last checked for a stuck CSD lock.*/ ts_delta =3D ts2 - *ts1; if (likely(ts_delta <=3D csd_lock_timeout_ns || csd_lock_timeout_ns =3D= =3D 0)) return false; @@ -241,9 +242,17 @@ static bool csd_lock_wait_toolong(struct __call_single= _data *csd, u64 ts0, u64 * else cpux =3D cpu; cpu_cur_csd =3D smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func= and info. */ + /* How long since this CSD lock was stuck. */ + ts_delta =3D ts2 - ts0; pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %llu n= s for CPU#%02d %pS(%ps).\n", - firsttime ? "Detected" : "Continued", *bug_id, raw_smp_processor_id(), = ts2 - ts0, + firsttime ? "Detected" : "Continued", *bug_id, raw_smp_processor_id(), = ts_delta, cpu, csd->func, csd->info); + /* + * If the CSD lock is still stuck after 5 minutes, it is unlikely + * to become unstuck. Use a signed comparison to avoid triggering + * on underflows when the TSC is out of sync between sockets. + */ + BUG_ON((s64)ts_delta > 300000000000LL); if (cpu_cur_csd && csd !=3D cpu_cur_csd) { pr_alert("\tcsd: CSD lock (#%d) handling prior %pS(%ps) request.\n", *bug_id, READ_ONCE(per_cpu(cur_csd_func, cpux)), --=20 2.41.0