From nobody Wed Nov 27 03:43:30 2024 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2044.outbound.protection.outlook.com [40.107.236.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9608F15359A for ; Mon, 14 Oct 2024 09:04:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.44 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896679; cv=fail; b=ciM3/bwJgy71LWEqGAWcPsDENgraSKYEDmOv/FdDS5ZPLiI/o4DrrYg+wWn7oJuC7+RCaCngvedYBRTmbgn0Z1qemXR7NQEhiGorZUi96k8XQLNRm7KZbUnE1Jn0xqd/Ow1Zp0n7KVSpSfVHuP69kJFJ7utlgxErvGjmHgVBFno= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896679; c=relaxed/simple; bh=SgWvRHqKNQR4/LVi8FqzjkDWI1RoB2seXSJZm8Le2FM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=D978YX2xFZHG7YNar94dBQfYpyGLpQQ/mRvkFUddBUEdz8UwOcYl1Mwm8dfQoA/o/HVJF4TjWW1bTCxBn6y6Mhd+YB8vubTRLGJ7xQiCQc/9QyhKmofdZNPwFu6MVhKmuKaAhu77mYASr8DghIn2pnVi+Gt3awuB4ptkelZ243s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=nhsSA86D; arc=fail smtp.client-ip=40.107.236.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="nhsSA86D" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Yf2OA6Ek6+LLW5amOnK/0bZBsKfRE6RvY8Sr7kDMTTtROXj+WBZfCMHPFqCkOiALe4AmEKrGEIdZWJzMeanZSs4BWlrb0Xx/p2nMy1CYee9m1+uuXckmSWB0YprM0z6jgpAVEAhZZHhqm5HudwOSEIuClYxTPeHhQ1T5GDocMXuNTFAnPywxs6pzaWJGV65gCMExGLGYr6ZDKW5OLuZUhJgPNiODsNXOmGFtnm78wLCGvqkOc1xrpEIdQORBr2WX7Ab4l+iVrpuCCPnfE+TMp3PBhee3uV8yL55hbXanTzM8BShF/I6Mm+/8I/hjDLr01zB4D3pbOkzD4ogpgoMkXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XWxGa+4WYEQl1q2pIojW3nX+ChwXIg18j8iOrMFPano=; b=S7Yp2KWW+jj0vkODrqjubNS29OZMbvsxWmP7AYoYvpkEIH0Shpsi5OPxscfd8iYHt9BYcq7DyWq2ZXhOK6EdVwUI/xbgZxtliSBM+54mxmf6ioN4oJNdxGri2r2UtjV3ETunp1pE23li8KVcj/xjibOrYz2hlW4foZGsFI8oTxle1DyZCYNZx/I/Bj6+72o8jI7eOeZmKJurASwmtUCMWZQkLkvxnBp5/jWOdbvcBfvjx02cEMeI07JPpFD/l1kW1dEi3JIMZMOvGwp+8P+B1cIyZ1ITHuwVbPgQU/pqyDvqBY8e0YFLkcHURIu9j1i3ALSqCe5RBuAbB9hoVc6zZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XWxGa+4WYEQl1q2pIojW3nX+ChwXIg18j8iOrMFPano=; b=nhsSA86Dr6kyAM2MgJ5Q2ocHeVfrhl4bneDOoH1FhHENLfjmKQ8dHStwUssA3ht3r5W7bS7/VasNC4PbROtM5x4iWJdjC4EhTvyXsUZi3gO5G6uVjRC6mzz8uz4Q0qf8/uTbsKKcmIKTjf0vujpK1orAoTcANQTC7FJDFissqhA= Received: from CYZPR12CA0018.namprd12.prod.outlook.com (2603:10b6:930:8b::10) by CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.25; Mon, 14 Oct 2024 09:04:33 +0000 Received: from CY4PEPF0000EE3B.namprd03.prod.outlook.com (2603:10b6:930:8b:cafe::72) by CYZPR12CA0018.outlook.office365.com (2603:10b6:930:8b::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.27 via Frontend Transport; Mon, 14 Oct 2024 09:04:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EE3B.mail.protection.outlook.com (10.167.242.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Mon, 14 Oct 2024 09:04:32 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 14 Oct 2024 04:04:22 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Thomas Gleixner , , Sebastian Andrzej Siewior CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Paul E. McKenney" , Rik van Riel , "Neeraj Upadhyay" , Leonardo Bras , Thorsten Blum , Zqiang , Tejun Heo , NeilBrown , Caleb Sander Mateos , "Gautham R . Shenoy" , Chen Yu , Julia Lawall , "K Prateek Nayak" Subject: [PATCH v3 1/5] softirq: Allow raising SCHED_SOFTIRQ from SMP-call-function on RT kernel Date: Mon, 14 Oct 2024 09:03:35 +0000 Message-ID: <20241014090339.2478-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014090339.2478-1-kprateek.nayak@amd.com> References: <20241014090339.2478-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE3B:EE_|CH3PR12MB7763:EE_ X-MS-Office365-Filtering-Correlation-Id: 2b615006-2254-4c9d-ef65-08dcec2f380a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?/+fWhIccpeEdJilFcIy8mULhphwyndSJ/ESUq5XicIwuAxd2JwdqSeMqICiO?= =?us-ascii?Q?Ij4fN0S0Nsypst0BJyIPYmDM9ZZojMjRss2GkT1o95W5GQPmwptEGIL34RpK?= =?us-ascii?Q?VJqsjZFBG4MPTQvD/mrupd9UQZuUVuiWhwKVrPAtX7uqhZuX711ItUuWdf3s?= =?us-ascii?Q?hr1cZfe43O5Qnfgal3hUdTEqJphFno+6IYpM2tKvl9zDGLekFzJgBKxRNXKA?= =?us-ascii?Q?mKSMzWBfcD7dLZxkI8/WZGaDLGt8UOVHWN7XsjuzVw2oYcRTdn4gl8POeWQs?= =?us-ascii?Q?qNLL/2Cv9kdBkPCaB47L0V+99ObArN4s6rax4L9m6/nLvBYRkFV77Hv69QM9?= =?us-ascii?Q?jyt5AoU2au9w0+IA7loP+X93bfnn2pjbh/VbfEcdVWnWi2TFfxlZ8KguhHA1?= =?us-ascii?Q?vFZqKQt7TuEf5ZKv8K4zq3kdZBNiDsAowz0JPhWA1LY1CwSOOW6w7uAZNzdj?= =?us-ascii?Q?EenB462OHDD0X20R0GrzB2Q4WxVws2t0O+H5wmWJYvoBcmW7/3L/5Du1LmcK?= =?us-ascii?Q?KuteXdZKs8uezRTZDGCrJuQUlG4wYSrXwol4SyAK1vXtU3z9kmGLjjmlxudm?= =?us-ascii?Q?raHe/5LpzZbSq4KcJwyK+AJwe7S50Th8gTFy4u66mJECYee5a5OGdWnJ8Hxf?= =?us-ascii?Q?+JZ5eG9yQ/FnyFb8iPPoe9of2ddc0iimVHcU0/TtKu5K+m+Z4oEnidCransy?= =?us-ascii?Q?tUk94EQj/+xqIMB2DfpzLtuqTWgqdUseDONffPGUxRP+3G34/iWLwfRV1p76?= =?us-ascii?Q?t6nzqV4PEUKD/wKaWx1Vglp2DZ20v/U/cWCink/mG19zzwuRhRU7Y0n8WXOn?= =?us-ascii?Q?1Ms6p84LDow9ChmD3VakuE7MBSXSFzjNL7CYSTz6QmXiWlnrPDiydg5uKdLN?= =?us-ascii?Q?4jUMj+U05AluIRyXtgU/51uv9stEBba6P+ah3EnJ+wLXVUiOQ1ibENaztC8x?= =?us-ascii?Q?ArYdZpjZ0MLIXA59e2PAFVjbXndGkCBg5d8KbwIm5N9qZeQlfbUyQn3yOy7l?= =?us-ascii?Q?RhZt93FeVXux9Qlz9Cf/C4813VNWo0WMQQaL2O4RKWAzIJVJMOYDEagYB7A1?= =?us-ascii?Q?prfTGQ+B0l0CsEX0RHPn9xBHQOem6pxbjKJe4Ou6oPZPImK2bI3Rikt6f8ki?= =?us-ascii?Q?ppH37rb6b/RS1nTnUUsnES244vvkXyoGNttk82ZEDqTmZUtsz3JU9QzYGShz?= =?us-ascii?Q?1FpRZTX6sH4H+ArcHpKVtBUhfQo/JJMyqGD+QlD2imP7qi93hJ4sSqUJad8K?= =?us-ascii?Q?trXwg6olJ1CamzWwhXGpfikQfv3seF43o6gPI0tUQm7IAqV68Y7wahea7h2g?= =?us-ascii?Q?pCBGzwDcnPcbHJrqq2fJo0HaETnm4tinWns+Ua/tvf4CAEXRYpEcNyT7e1g2?= =?us-ascii?Q?C+OK6RIpzyszsTpzqIeRoRRqWtdz?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2024 09:04:32.5907 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2b615006-2254-4c9d-ef65-08dcec2f380a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3B.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7763 Content-Type: text/plain; charset="utf-8" do_softirq_post_smp_call_flush() on PREEMPT_RT kernels carries a WARN_ON_ONCE() for any SOFTIRQ being raised from an SMP-call-function. Since do_softirq_post_smp_call_flush() is called with preempt disabled, raising a SOFTIRQ during flush_smp_call_function_queue() can lead to longer preempt disabled sections. Since commit b2a02fc43a1f ("smp: Optimize send_call_function_single_ipi()") IPIs to an idle CPU in TIF_POLLING_NRFLAG mode can be optimized out by instead setting TIF_NEED_RESCHED bit in idle task's thread_info and relying on the flush_smp_call_function_queue() in the idle-exit path to run the SMP-call-function. To trigger an idle load balancing, the scheduler queues nohz_csd_function() responsible for triggering an idle load balancing on a target nohz idle CPU and sends an IPI. Only now, this IPI is optimized out and the SMP-call-function is executed from flush_smp_call_function_queue() in do_idle() which can raise a SCHED_SOFTIRQ to trigger the balancing. So far, this went undetected since, the need_resched() check in nohz_csd_function() would make it bail out of idle load balancing early as the idle thread does not clear TIF_POLLING_NRFLAG before calling flush_smp_call_function_queue(). The need_resched() check was added with the intent to catch a new task wakeup, however, it has recently discovered to be unnecessary and will be removed soon. As such, nohz_csd_function() will raise a SCHED_SOFTIRQ from flush_smp_call_function_queue() to trigger an idle load balance on an idle target. nohz_csd_function() bails out early if "idle_cpu()" check for the target CPU returns false and should not delay a newly woken up task from running. Account for this and prevent a WARN_ON_ONCE() when SCHED_SOFTIRQ is raised from flush_smp_call_function_queue(). Signed-off-by: K Prateek Nayak --- v2..v3: o Added more details on SCHED_SOFTIRQ and how the idle load balancer will bail out if there is a new task that has woken up on the idle CPU and should not add more latency for task to run on CPU. --- kernel/softirq.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index b756d6b3fd09..d89be0affe46 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -280,17 +280,24 @@ static inline void invoke_softirq(void) wakeup_softirqd(); } =20 +#define SCHED_SOFTIRQ_MASK BIT(SCHED_SOFTIRQ) + /* * flush_smp_call_function_queue() can raise a soft interrupt in a function - * call. On RT kernels this is undesired and the only known functionality - * in the block layer which does this is disabled on RT. If soft interrupts - * get raised which haven't been raised before the flush, warn so it can be + * call. On RT kernels this is undesired and the only known functionalities + * are in the block layer which is disabled on RT, and in the scheduler for + * idle load balancing. If soft interrupts get raised which haven't been + * raised before the flush, warn if it is not a SCHED_SOFTIRQ so it can be * investigated. */ void do_softirq_post_smp_call_flush(unsigned int was_pending) { - if (WARN_ON_ONCE(was_pending !=3D local_softirq_pending())) + unsigned int is_pending =3D local_softirq_pending(); + + if (unlikely(was_pending !=3D is_pending)) { + WARN_ON_ONCE(was_pending !=3D (is_pending & ~SCHED_SOFTIRQ_MASK)); invoke_softirq(); + } } =20 #else /* CONFIG_PREEMPT_RT */ --=20 2.34.1 From nobody Wed Nov 27 03:43:30 2024 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2083.outbound.protection.outlook.com [40.107.244.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F280714A0AA for ; Mon, 14 Oct 2024 09:05:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.83 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896706; cv=fail; b=towGsEQRwvheO71dt1k5rZIU7bHRL8jyQUlJDB1BF6Y6CVY311940b6IcBVAM1G9JEkbUUKIb+kwhThN6V3k1vdAg3W6EBPeof+YMWDxntZ7LgEzzGOXPx+mLcYHiVC7uIT54Y1pqzeAIaaydxFHZk4Q0aoPcgdLX6pzldNNKLk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896706; c=relaxed/simple; bh=FXY7bGQUPwQcqNYup3fSwLb5bEp+aqsTt9/xAaKbFyo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=YPxqx4NeXS2GXYJYHE+v5uWG1Lne2wUgHbYX/ZWBdsWvARhhPnq6kStuIR8KgqAVJ+FqV5cMEZKHe894vy3wamHUDLWqWLslasjkVx4LdJ3Pnu4+NU4+4Q0xhuLLakzeThAn3mruwYCeWG6DCPsvElDvuP/7B9YLrvBrhSnenyE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Bs8haIZQ; arc=fail smtp.client-ip=40.107.244.83 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Bs8haIZQ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=huvoGuw0kASHzWPBYHwcAGSCXVsY+VD9ZQ5KYMFtoaoZhBEjGBzcfMUy2qhFMNnOxiFJ2JCAJDea9ci2IqFagoj02mRClGEz1ED+sEf/vGzHPIAjijc/75WjzeF3TVEp/6u+jNn2xrAiBsXaQQHVBmCo4SrNV7Xe/uoG8RzwA6shjU+30EZt2RNcif52isBaRWAIF8FbxtkRBgr8X75Rh/HxVywQY2tMwhd9iEWi5r2j6k50XwuwLPkNiODr60wcVOTGjYgo3iDydnlEXxdV/ubV/SsD/R15nrM4a8zxv8CoEuCXxbK8AcmuoYXsNTvmUOEyPx989G+xuqxNZbb/gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jUHBiWBboSSLzj3A5G9NzZV2oiynraVsa7bVUdd/p4A=; b=Ug22hDAe8cD3OyPNjIeaeAMS5Dj1Z903hLH40iPPOFg87eC6fG2lo2imKdUc3WJf48fUvUefH1gBsNmCOIGPfeIdnGlpJAjYA7iieMYTqFzxH4UUl4KaevSivR6Itbp2MXttxeiQBO192CC2jM4dseV5FCKzmKFwFUlshihQ6LKpEd8X8O2XsORwI8/ltljqgdfjY11vpI93Vgh6anvZ0frKFDQNMNNrlnrKCocRirbDNH32frphAiAoOsdMv2XFP+jeuVJWiARRh/Im6H3LVdlWSZdP+0Zg0cpxexovajrUvGGenBO1KJj5cPvqW4uumLifxixsSIPfzGsAArKNMQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jUHBiWBboSSLzj3A5G9NzZV2oiynraVsa7bVUdd/p4A=; b=Bs8haIZQ38LHSxvaYfVSCeXYoLjF2d+RkLFiUYNAvuQfCZULRVK7eOZHRsA6f2rPUDwEHUvE1PQ2VROZjAIkhRQurSjyyJAjh+ieJeBywIES6zu61KE5UNpKVZVsW84AJSDX2mmozdc1D+VA/NKuG+JQ9O2MCpdgAa+VrceEnjA= Received: from CY5PR14CA0026.namprd14.prod.outlook.com (2603:10b6:930:2::34) by SJ2PR12MB9086.namprd12.prod.outlook.com (2603:10b6:a03:55f::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.26; Mon, 14 Oct 2024 09:05:00 +0000 Received: from CY4PEPF0000EE3D.namprd03.prod.outlook.com (2603:10b6:930:2:cafe::12) by CY5PR14CA0026.outlook.office365.com (2603:10b6:930:2::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.27 via Frontend Transport; Mon, 14 Oct 2024 09:05:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EE3D.mail.protection.outlook.com (10.167.242.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Mon, 14 Oct 2024 09:04:59 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 14 Oct 2024 04:04:46 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Thomas Gleixner , , Sebastian Andrzej Siewior CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Paul E. McKenney" , Rik van Riel , "Neeraj Upadhyay" , Leonardo Bras , Thorsten Blum , Zqiang , Tejun Heo , NeilBrown , Caleb Sander Mateos , "Gautham R . Shenoy" , Chen Yu , Julia Lawall , "K Prateek Nayak" Subject: [PATCH v3 2/5] sched/core: Remove the unnecessary need_resched() check in nohz_csd_func() Date: Mon, 14 Oct 2024 09:03:36 +0000 Message-ID: <20241014090339.2478-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014090339.2478-1-kprateek.nayak@amd.com> References: <20241014090339.2478-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE3D:EE_|SJ2PR12MB9086:EE_ X-MS-Office365-Filtering-Correlation-Id: d22b4590-286e-4597-58ae-08dcec2f482d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?5h1Om1zqUoqi0mBo5ZC2VbBA4oAw/RuLbkFIb5c7No3BN+E7cWYKQWz0mLrN?= =?us-ascii?Q?Jt8VW1rFNNueB1piec9mT+21VebPTuU7LnlfY3NKiQ3hen6f6TjaL38NQ0mL?= =?us-ascii?Q?RhsuqpdDAOkuSWZ0Gdio7p7Ny8rM9+CpNWqVTE1ZhUUOuSQrx8hsqwyKow2S?= =?us-ascii?Q?EOSaRyVHDhQapUENFcWJiUlVUn1pKkk4q/KhN2MoVXydSS6UJx8o1cloEcld?= =?us-ascii?Q?t8xHsXneXhk9H/n+C5RqeWzyt0T99aVDCvolHVegxgZMo+9awGw3B5ekCO76?= =?us-ascii?Q?uIK2660lgSbVT+aBctmih0q8rsejg8kcGyk0i42o3a92PsMzlCJg5ZOAGvQo?= =?us-ascii?Q?FQz8fAH91UP49qeujFjAvHs0UF6iJG29JEJINW5unWikhZxw1PCwFusSa2Ss?= =?us-ascii?Q?6VLTpK046UxBKzqfHEn/8CL+lt2a6MBNAgKbFzcyab6e61pGGh+v1jn9nRI4?= =?us-ascii?Q?DgBaeL14mltWorJKGsNK2x21UhRhxebz7Sepu3cNbXRrhkKPFc0SFMQF674A?= =?us-ascii?Q?QZe5PS3OYkvYunmmHbGUnpA/a6hsTXFr+uJEurf5TtnCbPtM2bj+LGXI/sxw?= =?us-ascii?Q?++kiEBPBWTMiVttMz7gxicb2+fJ1tQh35y7Q5Ig0xWKyoKOWlRue4ucq118t?= =?us-ascii?Q?NlLhcoVMn215NtC+nPy1ZS68/NwEDS3AzCfaFu659XcO0CgFRVEIZt6lPRuX?= =?us-ascii?Q?bhxZV8XSJU5y3vw2pZCrRGkTeI0Dzfby8GAek7/gruZc5ny9hLumNZN17F+r?= =?us-ascii?Q?nLJBMeSgCs0SzPHQIhIyjqND10oaVIKvk6LKoL/oCkjDtXknJ37Wxehrqhnp?= =?us-ascii?Q?K/zgWXcA7ckYHp6LcxstpndwLYGGjcwzpbDU46dW5jcpUXHfsIwOQ/gyrU2h?= =?us-ascii?Q?84vfEqHekPUj4AKHhyodhbxRh4RBZxqfh7f88nN7kTMLaO4XZZmtu+cWFhzn?= =?us-ascii?Q?FdA7tHR52R+F5UDXMiaycclTGoUltvrU4I6LQcKCSPmuU5nOPk79OuiRv2y6?= =?us-ascii?Q?GhGVsqXFE9M+nAz84gXSbZoj41AcQjJUk7g3YFv5732Oehf2emaV0L0yxbYu?= =?us-ascii?Q?0SqylqTNNk+VuZMW3VPpDbdks5NQs2IgRSciNMaJXnQinJmwMFI0pc3Hz5DL?= =?us-ascii?Q?d7iiLbtvEiUwhqqK7kYG69cGqlVMC5NQU53KWzZUNNBXLUlmafIDeut7pUi5?= =?us-ascii?Q?UQsJ9BBSN+w6+mjdaHkYhGWnYq/BMTA6OZoFvT8xVYPkqEf1zoFlDA2x4Pm1?= =?us-ascii?Q?xHwa+1HftUy9CEFN15tiL6BUzu/USzrZp7t+m2x3wOoASKTyUPGv7D0kRZGE?= =?us-ascii?Q?iAxKSZQOsSyQvlKMms6wDmb6LuqryH/H9v7/gxnoUBLy1XoZmztuNCMAhUbl?= =?us-ascii?Q?Pk8HU1yhPkRmbIdentPpKpbmfBd3?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2024 09:04:59.6598 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d22b4590-286e-4597-58ae-08dcec2f482d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3D.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB9086 Content-Type: text/plain; charset="utf-8" The need_resched() check currently in nohz_csd_func() can be tracked to have been added in scheduler_ipi() back in 2011 via commit ca38062e57e9 ("sched: Use resched IPI to kick off the nohz idle balance") Since then, it has travelled quite a bit but it seems like an idle_cpu() check currently is sufficient to detect the need to bail out from an idle load balancing. To justify this removal, consider all the following case where an idle load balancing could race with a task wakeup: o Since commit f3dd3f674555b ("sched: Remove the limitation of WF_ON_CPU on wakelist if wakee cpu is idle") a target perceived to be idle (target_rq->nr_running =3D=3D 0) will return true for ttwu_queue_cond(target) which will offload the task wakeup to the idle target via an IPI. In all such cases target_rq->ttwu_pending will be set to 1 before queuing the wake function. If an idle load balance races here, following scenarios are possible: - The CPU is not in TIF_POLLING_NRFLAG mode in which case an actual IPI is sent to the CPU to wake it out of idle. If the nohz_csd_func() queues before sched_ttwu_pending(), the idle load balance will bail out since idle_cpu(target) returns 0 since target_rq->ttwu_pending is 1. If the nohz_csd_func() is queued after sched_ttwu_pending() it should see rq->nr_running to be non-zero and bail out of idle load balancing. - The CPU is in TIF_POLLING_NRFLAG mode and instead of an actual IPI, the sender will simply set TIF_NEED_RESCHED for the target to put it out of idle and flush_smp_call_function_queue() in do_idle() will execute the call function. Depending on the ordering of the queuing of nohz_csd_func() and sched_ttwu_pending(), the idle_cpu() check in nohz_csd_func() should either see target_rq->ttwu_pending =3D 1 or target_rq->nr_running to be non-zero if there is a genuine task wakeup racing with the idle load balance kick. o The waker CPU perceives the target CPU to be busy (targer_rq->nr_running !=3D 0) but the CPU is in fact going idle and due to a series of unfortunate events, the system reaches a case where the waker CPU decides to perform the wakeup by itself in ttwu_queue() on the target CPU but target is concurrently selected for idle load balance (XXX: Can this happen? I'm not sure, but we'll consider the mother of all coincidences to estimate the worst case scenario). ttwu_do_activate() calls enqueue_task() which would increment "rq->nr_running" post which it calls wakeup_preempt() which is responsible for setting TIF_NEED_RESCHED (via a resched IPI or by setting TIF_NEED_RESCHED on a TIF_POLLING_NRFLAG idle CPU) The key thing to note in this case is that rq->nr_running is already non-zero in case of a wakeup before TIF_NEED_RESCHED is set which would lead to idle_cpu() check returning false. In all cases, it seems that need_resched() check is unnecessary when checking for idle_cpu() first since an impending wakeup racing with idle load balancer will either set the "rq->ttwu_pending" or indicate a newly woken task via "rq->nr_running". Chasing the reason why this check might have existed in the first place, I came across Peter's suggestion on the fist iteration of Suresh's patch from 2011 [1] where the condition to raise the SCHED_SOFTIRQ was: sched_ttwu_do_pending(list); if (unlikely((rq->idle =3D=3D current) && rq->nohz_balance_kick && !need_resched())) raise_softirq_irqoff(SCHED_SOFTIRQ); Since the condition to raise the SCHED_SOFIRQ was preceded by sched_ttwu_do_pending() (which is equivalent of sched_ttwu_pending()) in the current upstream kernel, the need_resched() check was necessary to catch a newly queued task. Peter suggested modifying it to: if (idle_cpu() && rq->nohz_balance_kick && !need_resched()) raise_softirq_irqoff(SCHED_SOFTIRQ); where idle_cpu() seems to have replaced "rq->idle =3D=3D current" check. Even back then, the idle_cpu() check would have been sufficient to catch a new task being enqueued. Since commit b2a02fc43a1f ("smp: Optimize send_call_function_single_ipi()") overloads the interpretation of TIF_NEED_RESCHED for TIF_POLLING_NRFLAG idling, remove the need_resched() check in nohz_csd_func() to raise SCHED_SOFTIRQ based on Peter's suggestion. Link: https://lore.kernel.org/all/1317670590.20367.38.camel@twins/ [1] Link: https://lore.kernel.org/lkml/20240615014521.GR8774@noisy.programming.= kicks-ass.net/ Fixes: b2a02fc43a1f ("smp: Optimize send_call_function_single_ipi()") Suggested-by: Peter Zijlstra Signed-off-by: K Prateek Nayak --- v2..v3: o No changes. --- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 43e453ab7e20..424c652a9ddc 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1237,7 +1237,7 @@ static void nohz_csd_func(void *info) WARN_ON(!(flags & NOHZ_KICK_MASK)); =20 rq->idle_balance =3D idle_cpu(cpu); - if (rq->idle_balance && !need_resched()) { + if (rq->idle_balance) { rq->nohz_idle_balance =3D flags; raise_softirq_irqoff(SCHED_SOFTIRQ); } --=20 2.34.1 From nobody Wed Nov 27 03:43:30 2024 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2058.outbound.protection.outlook.com [40.107.94.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A2F41BC58 for ; Mon, 14 Oct 2024 09:05:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896745; cv=fail; b=aNjllPl891pTFm+TDWPadRbZ9ieu/DtQ80vQP13FlA43JDqw+NLyhpFD7+Mq0Honi814VziW8pR4zryyBV7AZC+RPpwPOfdoQvXZmGxix3tKvGZfUxW19e90kVHYGATpVwE3ZNZpimUQ0gv3uuYT4qR9Hr8ctOS1SvonVCrydRI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896745; c=relaxed/simple; bh=HqANVoLm9lJ1cGqDX0awhkfs2uqW6m3IAcdYjHuOKe8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=i+CRR1dQKf7SRKJQCm3Yy2YtBPsO5hrFQIHp5xFes8bDAo6/2eC5hCUQiftXPdUU4LeUgEgpndZ5KMCiQQcGeLk+tbr9KiKgOg7jKvzjywk2Xdl9OvhKEHs0LOaTUlTpuOORAmM+uXxZcg/yBy62UZefzoZcMLgAcOk4GsrWV0s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=nmhuhY9/; arc=fail smtp.client-ip=40.107.94.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="nmhuhY9/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=g7ahGJlP2XqgGJly5Gi8KDtaDTGm35029jcx7fh5FJL01B9P8FdmeU1aJ4vr3AHstp6rsWfae+v6Aq7ze8moz33y1VG6aqp8R8zdbbKh6zQkOHirzQa8Hxy70ikwfNvY/pr/7l0yV42HxTM4H3fXBfODXWK3A9FBT+wBXnrsfZ1s8o2rhQomxMk7+OSpQI6LD5UdasGyJPsCFahg/cYZLpqj4yK0YFxhNagc+r291NHQc1GtPF2FCG3W8YiOthuwc/D676bgbwYIG/Ww+hVmgMCCTKA60gDgqudH/9u0/tsnIG3o3AY2WEvvkEJT6VxYyeBK0ScLyxUWvx0WhSojaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TNQIcqaZGL+cQAHDNKvBCTIxFOTjoNSywCxNT443MpU=; b=GpMLnp5zLa0y/JFMp4YemtRu/h51s0HKEQkMKstrCZM/+B2JTzO0PC2k+NuWf6wudUEpo4Hd+R4SWYkqbTOMaxHSeIMYMCb3m6VAginc+OC6G13T8ApbtpIQ8vRQsR8biZyHfcpDNW14sC6/5bL4CJ4nl4h+GNo6JDQRkNntcGNfhl8CdUHH1Z75LA7L7ZhNYd1jm4F1F5cNinLeI9YwLA9GLfm2wO5OvBOKZnVeLpoelB/c704/diKJyTrHdNKV84zZh3YvEYH3b9HwbfDY23US6LiXrfHERYI4BlwwAGyrrYxvIzDnFsHUEy7zCvP0x7rWthqMWSlEtf/I0TgYDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TNQIcqaZGL+cQAHDNKvBCTIxFOTjoNSywCxNT443MpU=; b=nmhuhY9/AD7IYrWu0f4DVYlOTJnjqhg8Fqq7kVuoChE6II36jvB8aTyT71nzu8u+nTwaJcN9lAQNuSltx7uJcijWoGHyWMwkBzlo7GwJDdQSwJ21LYSMUzUM2s/+aGh1eYndsNkNzYX3wKZqZ/+RSI1a4XHXw4to8drZW+dkpFI= Received: from PH8PR05CA0017.namprd05.prod.outlook.com (2603:10b6:510:2cc::29) by IA1PR12MB6283.namprd12.prod.outlook.com (2603:10b6:208:3e5::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.22; Mon, 14 Oct 2024 09:05:39 +0000 Received: from CY4PEPF0000EE3F.namprd03.prod.outlook.com (2603:10b6:510:2cc:cafe::5f) by PH8PR05CA0017.outlook.office365.com (2603:10b6:510:2cc::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.16 via Frontend Transport; Mon, 14 Oct 2024 09:05:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EE3F.mail.protection.outlook.com (10.167.242.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Mon, 14 Oct 2024 09:05:38 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 14 Oct 2024 04:05:14 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Thomas Gleixner , , Sebastian Andrzej Siewior CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Paul E. McKenney" , Rik van Riel , "Neeraj Upadhyay" , Leonardo Bras , Thorsten Blum , Zqiang , Tejun Heo , NeilBrown , Caleb Sander Mateos , "Gautham R . Shenoy" , Chen Yu , Julia Lawall , "K Prateek Nayak" Subject: [PATCH v3 3/5] softirq: Mask reads of softirq_ctrl.cnt with SOFTIRQ_MASK for PREEMPT_RT Date: Mon, 14 Oct 2024 09:03:37 +0000 Message-ID: <20241014090339.2478-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014090339.2478-1-kprateek.nayak@amd.com> References: <20241014090339.2478-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE3F:EE_|IA1PR12MB6283:EE_ X-MS-Office365-Filtering-Correlation-Id: 7d4adb82-72e1-4244-c042-08dcec2f5f8d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700013|376014|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?/kx9X70zizwcHW1oJaBjts78FaHr+KxwXRWuEFS0uOwJIjJgXugMWVvHWc/9?= =?us-ascii?Q?gB3JhXvA9bm1D8SEfTJniUZz871pvVw0A8N24FJ2XE9d1UqSG/8HoPvunDxp?= =?us-ascii?Q?cRMNpKO164Qr328MP81wARvpy7raJGcg68HNlu6e18cgrNWXCiGQS5Z7p7UI?= =?us-ascii?Q?eYq+frSEdHuVgBl1k1d8EZWqPU73h93a26KR5Ii+RMTxC999oh7SHw+vPTyU?= =?us-ascii?Q?L2HJ0nwKhgmb5kjNw4MtFDAbQNwFEu+f9SnB9tIx3zrclF3e0Ki/uVW8DsGq?= =?us-ascii?Q?ma/QEcDm3AoU2IETTA5jItmQyTDbi+6zPFroV9JUS3jqPPLKXWDzdgRInfeJ?= =?us-ascii?Q?vgVTF+hDsMfeeyCVw5nn23iktl20ABReut8K28dRlGZSR8vas5i6aSdFYQWZ?= =?us-ascii?Q?Gu2mx9u5/m5szdDlcWqWdqlhhwDnEd/uvg1Hhqkd/8antCPMW5iEuq5KgrlY?= =?us-ascii?Q?GWLjM39Kdflc0NX8FM2M+47EvRl4fRtKyYrjmfivGv+/8xRsPjZ2OKet//5Z?= =?us-ascii?Q?xbzcExgXeZQYRjD4osEstDCN9zqo3uwQqyKzzTv/sv5mUW890PGuL27LR2yy?= =?us-ascii?Q?7+juzqChL0n8nkQdb95k7iktr1C6ZrEdUSE1qMbRa6qfbmDnHWBAuakCHF9t?= =?us-ascii?Q?PMM7/PZuWq1d2ZwqlO5W7nAZj8apKSkMWSp4p4RQFOXLXJ9F/2CMHJabOXF5?= =?us-ascii?Q?STix2bH/ggCCP9NNp70UcVGuBQ/PAtRcqA8X61NJAAtVURrH3uwoVY0G8QuY?= =?us-ascii?Q?EMVG03zEjM1U8VS4Muq061/cBrmBnkhx9PMDes90tAGoGKyfJbU+bRnAk7Ky?= =?us-ascii?Q?CGuyz+MdhrkryIli24H3ZBo8rn4zXGZLGLsGmiqrZcptv8xvd4HtJkbTUtvB?= =?us-ascii?Q?YgP6rozOreQQ3wjwxJ8bdHIvzYw9DTNwBeaxcw00dR95/AGje1bA3R905fIf?= =?us-ascii?Q?4zkz99ZfcxKNYqtglaDN5N9PneNeTUwiHSdZeHov+AECGQkOc34tBRhxNHLE?= =?us-ascii?Q?hAXN8bfIpjTnm4bBLTc9jGsPMnidD80fKrl9diOTaeyoZFSfHSNGJ3dLynOq?= =?us-ascii?Q?Azmkh1YZWl36u6ACJR+RFzhaB7MCkGJ1sk+R6lRW0jzjPk1Vl2rRAKobj+J9?= =?us-ascii?Q?Cv5IM22XG+NVVewigswpigONjHXFdPzVjqvwftg+w61odSqssStgeWIPvSjd?= =?us-ascii?Q?Fi9hDGTTxpWYe0p9NIMQtcCq9j4jeGldF2HOsa8zTbCASIPwBTJv31SiD1EX?= =?us-ascii?Q?2oKpGGJW73O9thUP9Rtxu0s58MciEuYY9s0MXCCNyapTPt9/f8DjpYlNmvrD?= =?us-ascii?Q?94+Nld3OkBVLmrb5C6JA2iUZiVJpFtbtPEBbFqSx2/1JyQTC61UicHu0eOBZ?= =?us-ascii?Q?Q3xlBv4=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(36860700013)(376014)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2024 09:05:38.8948 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7d4adb82-72e1-4244-c042-08dcec2f5f8d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3F.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6283 Content-Type: text/plain; charset="utf-8" On PREEMPT_RT kernels, softirq_ctrl.cnt tracks the softirq count and set the task::softirqs_disabled_cnt accordingly. This count is always 'changed by SOFTIRQ_OFFSET when entering and leaving softirq processing' or 'SOFTIRQ_DISABLE_OFFSET (=3D 2 * SOFTIRQ_OFFSET) on local_bh_disable or local_bh_enable", as stated in the comment about softirq_ctrl declaration. Soon, the softirq_ctrl.cnt will also be used by !PREEMPT_RT kernels to track an impending call to do_softirq() in order to prevent pointless wakeup of ksoftirqd which will use the lower bits of softirq_ctrl.cnt. Mask all the current reads of softirq_ctrl.cnt on PREEMPT_RT kernels with SOFTIRQ_MASK to track only multiples of SOFTIRQ_OFFSET based changes. No functional change intended. Signed-off-by: K Prateek Nayak --- v2..v3: o No changes. --- kernel/softirq.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index d89be0affe46..fc8c3a7dbe35 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -138,7 +138,7 @@ static DEFINE_PER_CPU(struct softirq_ctrl, softirq_ctrl= ) =3D { */ bool local_bh_blocked(void) { - return __this_cpu_read(softirq_ctrl.cnt) !=3D 0; + return (__this_cpu_read(softirq_ctrl.cnt) & SOFTIRQ_MASK) !=3D 0; } =20 void __local_bh_disable_ip(unsigned long ip, unsigned int cnt) @@ -155,7 +155,8 @@ void __local_bh_disable_ip(unsigned long ip, unsigned i= nt cnt) /* Required to meet the RCU bottomhalf requirements. */ rcu_read_lock(); } else { - DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt)); + DEBUG_LOCKS_WARN_ON(this_cpu_read(softirq_ctrl.cnt) & + SOFTIRQ_MASK); } } =20 @@ -163,7 +164,7 @@ void __local_bh_disable_ip(unsigned long ip, unsigned i= nt cnt) * Track the per CPU softirq disabled state. On RT this is per CPU * state to allow preemption of bottom half disabled sections. */ - newcnt =3D __this_cpu_add_return(softirq_ctrl.cnt, cnt); + newcnt =3D __this_cpu_add_return(softirq_ctrl.cnt, cnt) & SOFTIRQ_MASK; /* * Reflect the result in the task state to prevent recursion on the * local lock and to make softirq_count() & al work. @@ -184,7 +185,7 @@ static void __local_bh_enable(unsigned int cnt, bool un= lock) int newcnt; =20 DEBUG_LOCKS_WARN_ON(current->softirq_disable_cnt !=3D - this_cpu_read(softirq_ctrl.cnt)); + (this_cpu_read(softirq_ctrl.cnt) & SOFTIRQ_MASK)); =20 if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS) && softirq_count() =3D=3D cnt) { raw_local_irq_save(flags); @@ -192,7 +193,7 @@ static void __local_bh_enable(unsigned int cnt, bool un= lock) raw_local_irq_restore(flags); } =20 - newcnt =3D __this_cpu_sub_return(softirq_ctrl.cnt, cnt); + newcnt =3D __this_cpu_sub_return(softirq_ctrl.cnt, cnt) & SOFTIRQ_MASK; current->softirq_disable_cnt =3D newcnt; =20 if (!newcnt && unlock) { @@ -212,7 +213,7 @@ void __local_bh_enable_ip(unsigned long ip, unsigned in= t cnt) lockdep_assert_irqs_enabled(); =20 local_irq_save(flags); - curcnt =3D __this_cpu_read(softirq_ctrl.cnt); + curcnt =3D __this_cpu_read(softirq_ctrl.cnt) & SOFTIRQ_MASK; =20 /* * If this is not reenabling soft interrupts, no point in trying to --=20 2.34.1 From nobody Wed Nov 27 03:43:30 2024 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2075.outbound.protection.outlook.com [40.107.244.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8380A156678 for ; Mon, 14 Oct 2024 09:06:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.75 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896768; cv=fail; b=rgt+2OrLkJ0Y6nTho+GSnJna7UzoIH/wuhmq3ul99TdpaN7e1ibvWp3cuLxg8KAJpOL/l1GuNK/z2EFyrIhNjH5aE71qthN+N5jlI/AdgQI6Aoi+qQmiLbXSU0U2oxP4LAxVhYl/xOKYKm8lvrVu+x8TzGXwYa/PTswoNoVRNzc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896768; c=relaxed/simple; bh=R7+nRUahrBD4bBgAcevzrlRasguwE8JpM1AOq4sal0E=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kf6qIFtl62c4ohXbV8gyRC5TXaOAMWj9I7NNla/wXMGqoHUPTBJGyFOtlcPB8lQFnAHRlYqbcxdkYE1D6CJzmT7tWsrr4hE6T1o7lRY5S6RM6PuaK7Up+oClBAdMX2bIoxbLPtIoJr39IZYGZP+0h1Qi2yOyS+YtjQsxbc/q7cQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=ZAITi7KG; arc=fail smtp.client-ip=40.107.244.75 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="ZAITi7KG" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KbuM6grMQPz3a7umC9hm+LCfPGgxwPB9By7n79tSC5tiVeNP81pTPLpBG9tb/6Qpg6SHGCHrPtCT+vpJG5pYjIWuBH9vrvQnbs8vsqMjMw1poskkte3NlPLoSVetmM4bBsV7pb1yQqrhz7UbmdO+GSb/1czDzshSxlhMiwHJ0EyvcDOFS1ZqUrmv+dBZdLLDBDr1UYRYnSn4UwyPAehcHuH02uOeGQL+F4mhIkjrIji3lsni5jUq7RkKy/Nydn1Q6uBCOPQWZl7MS0XwGCtk8NhOR9a0wNCFSdZ8VTbGpG1hLQtjvOpTckzVq7jLr+9BcSHKoll1fSkhfGYJs2OobQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=J6h5i5pMa+w0iivB0qlD2WFylp3lL9ATtzHNSMFne8c=; b=jeLQxRw3MXmOFiTewbIM9oDgPwDzsofr5vvo/mxlLWB7PQCfPue5wNv/rvCEGaI97iq6uJYZAv+7Ae7GZhZjI7OfSikoEM8kIoiBpN61+NlpdxS761/UCd7kbEgkWBQXvipiF99xmus+apkjZRCPlZ0zl4KC76XV8eJEuB2/Ci5DjAJL9Ohdxlt6RVbAS0rj/bIam92OzrjjIwHgtF3e/nOf8B26lF+hqd9lfvO9Ov7sZboiwJr6yOchSpovyjEwPE95hNEVXbFe8MMVBE4Hzc+ogjjoEuiVwyJjXMbviQDGecFDXucVdohUZe+jSHu710QeivHiTSgTDpLBHT/5/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J6h5i5pMa+w0iivB0qlD2WFylp3lL9ATtzHNSMFne8c=; b=ZAITi7KGhqdQoOS36vC3UEG2IDUKjAqd+l2rbQw8xMAUv8UTtPqGi2IChcUAhuNMMCRiqFeEdI3MVk41yVcoNxaWkV2b0dAjo8dtgTm13B1HzML99eydhM6n+zo8gtBJW7+6AJoYGRd/wEsi8YDuKTlcL4Q02BHjbKR9VA74mEs= Received: from PH7PR17CA0061.namprd17.prod.outlook.com (2603:10b6:510:325::11) by DS0PR12MB6656.namprd12.prod.outlook.com (2603:10b6:8:d2::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.25; Mon, 14 Oct 2024 09:06:01 +0000 Received: from CY4PEPF0000EE3C.namprd03.prod.outlook.com (2603:10b6:510:325:cafe::94) by PH7PR17CA0061.outlook.office365.com (2603:10b6:510:325::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.26 via Frontend Transport; Mon, 14 Oct 2024 09:06:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EE3C.mail.protection.outlook.com (10.167.242.13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Mon, 14 Oct 2024 09:06:00 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 14 Oct 2024 04:05:52 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Thomas Gleixner , , Sebastian Andrzej Siewior CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Paul E. McKenney" , Rik van Riel , "Neeraj Upadhyay" , Leonardo Bras , Thorsten Blum , Zqiang , Tejun Heo , NeilBrown , Caleb Sander Mateos , "Gautham R . Shenoy" , Chen Yu , Julia Lawall , "K Prateek Nayak" Subject: [PATCH v3 4/5] softirq: Unify should_wakeup_ksoftirqd() Date: Mon, 14 Oct 2024 09:03:38 +0000 Message-ID: <20241014090339.2478-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014090339.2478-1-kprateek.nayak@amd.com> References: <20241014090339.2478-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE3C:EE_|DS0PR12MB6656:EE_ X-MS-Office365-Filtering-Correlation-Id: 433099dc-0726-49f6-9bcf-08dcec2f6c5b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?8Q+fnqxWlFxUfZ0cJjedsRoQK6x1VpBaO9ks7M3dN6OAEy8RJairOhf2AwAD?= =?us-ascii?Q?PwIJiDlK2b4rFhL20LrT0nNAxm9jyckE37QK2UA2VF88++2silawTjwjc50l?= =?us-ascii?Q?ctDhcqmpWaiGksbENkYptrtSRhdApP8MiZCoVInjzvi/6ViS8L1GDWxgN957?= =?us-ascii?Q?UmIaPVq+qvStQG2pENFGYmprecDRY8qd2TBk8Y7MQIcFBWzQvnD25jXTxqaf?= =?us-ascii?Q?z61a11E5aLRDzsHoL1bM/Og0PacbFdC1W7QijOL5ocbB4TQdFGrALnGbVAXk?= =?us-ascii?Q?UxsfNIUczP7s82pRUi04CL78rndozT8Vw8sTuobbd/TVTmbgkrFZWq6UZYqu?= =?us-ascii?Q?rzgN21lS+DsRsjH3QhBMV03sE1KKWhZZzVgiorPgVOIIUIBzcPHFRYmHBxCn?= =?us-ascii?Q?AFIzrUeDj/9Vg+Qs1gn+avtBSm20rSZg6nKzpA4XUa3rgI7xwDGWz5nyO3WK?= =?us-ascii?Q?/Ik56G6V8vyOnuNFi8ixi6Nfj2ZxHIoGwandbMJEswoH5oVO+1TDundgvS9p?= =?us-ascii?Q?Y7vE4yPPcrfFYeL0YhmK2tHEzGXgWwlWmMDiOs6bAmEs6AsFX0CZuPHDA2Or?= =?us-ascii?Q?47GKLSBB7CqhJ9uNX1Vx6DT+8jf64QNufwO5knqOqOyhaTGirnk7ucAn79/q?= =?us-ascii?Q?DmIyLnpBJDaLg6LUFkRG8C8a3R8mMGH5hPgMuSy4mN4f4yONHhjUWykJEHs/?= =?us-ascii?Q?tysnNrQw6NxDJOLFuF3+10MGU8Sv0unFrZxIVTnctCm9UODP7eD2OhBuGQtL?= =?us-ascii?Q?d/JvUTFv8on4HJnJ0IJal696EUp9afX+zZCuJ5CGwDGe94DyrZQpoe3msg2I?= =?us-ascii?Q?lOcaIDJLyuhAfoWSOvEMHwJiF9C8hjnRnU2ZwXULxS0n1qAewn70IvcfGYAF?= =?us-ascii?Q?pJ0KrgSfYUXULt949ePlEoB69EViNFXc67SzVJQCZ158e3UMmCvdiyMhOTd9?= =?us-ascii?Q?VG9DdFK+gzyvihoxxyhuog8iODBFEqQ4o4Hp4TUMBkynFcxW+img92aS2yTz?= =?us-ascii?Q?djB9rjO2YRRHOZ+HLMrG3auNTb24rf6RRtkCQhslqtHrnVnZ4LN5lkP7F0X2?= =?us-ascii?Q?FIr40xqGInKGhrlL5R8T2KlaYEC/LIVeDOFRZ/MwKZVGLAFH/sfbVs3g1jkg?= =?us-ascii?Q?nFnRFc4wBxCm2yTkM0P/psEjCX54P0DHBCQijSArmuwIOWjK95ab4VcO8MxA?= =?us-ascii?Q?7SGXsjtu9ewtIzxyaWVST3WsliRDyHFZbQHJzrVcIpvgaafaLN/fvapuzji7?= =?us-ascii?Q?3yyKlULfu0TqCxfBQYeBZW1h9W8SV6x+99Cvw+liw5+8t7OyH5iKHjunJZJv?= =?us-ascii?Q?qrrmeVbvEmJoAZhuMymZ/emRQiBLw94HaqHbpUT+6iHyMMgWtyTXCUSXVFJZ?= =?us-ascii?Q?T74gBrk0hJZPnPYccj3uVfDjPKVW?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2024 09:06:00.3610 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 433099dc-0726-49f6-9bcf-08dcec2f6c5b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3C.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6656 Content-Type: text/plain; charset="utf-8" Define softirq_ctrl::cnt for !PREEMPT_RT kernels too and unify should_wakeup_ksoftirqd() to return based on softirq_ctrl::cnt. Since these counts can change quite frequently when running interrupt heavy benchmark, declare per-cpu softirq_ctrl as cacheline aligned. No functional changes intended since !PREEMPT_RT kernels do not increment the softirq_ctrl::cnt (yet) and should always return true mimicking the current behavior. Signed-off-by: K Prateek Nayak --- v2..v3: o Removed ifdefs around local_lock_t. (Peter) --- kernel/softirq.c | 51 ++++++++++++++++++++++-------------------------- 1 file changed, 23 insertions(+), 28 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index fc8c3a7dbe35..0730c2b43ae4 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -88,23 +88,6 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); #endif =20 -/* - * SOFTIRQ_OFFSET usage: - * - * On !RT kernels 'count' is the preempt counter, on RT kernels this appli= es - * to a per CPU counter and to task::softirqs_disabled_cnt. - * - * - count is changed by SOFTIRQ_OFFSET on entering or leaving softirq - * processing. - * - * - count is changed by SOFTIRQ_DISABLE_OFFSET (=3D 2 * SOFTIRQ_OFFSET) - * on local_bh_disable or local_bh_enable. - * - * This lets us distinguish between whether we are currently processing - * softirq and whether we just have bh disabled. - */ -#ifdef CONFIG_PREEMPT_RT - /* * RT accounts for BH disabled sections in task::softirqs_disabled_cnt and * also in per CPU softirq_ctrl::cnt. This is necessary to allow tasks in a @@ -122,10 +105,32 @@ struct softirq_ctrl { int cnt; }; =20 -static DEFINE_PER_CPU(struct softirq_ctrl, softirq_ctrl) =3D { +static DEFINE_PER_CPU_ALIGNED(struct softirq_ctrl, softirq_ctrl) =3D { .lock =3D INIT_LOCAL_LOCK(softirq_ctrl.lock), }; =20 +static inline bool should_wake_ksoftirqd(void) +{ + return !this_cpu_read(softirq_ctrl.cnt); +} + +/* + * SOFTIRQ_OFFSET usage: + * + * On !RT kernels 'count' is the preempt counter, on RT kernels this appli= es + * to a per CPU counter and to task::softirqs_disabled_cnt. + * + * - count is changed by SOFTIRQ_OFFSET on entering or leaving softirq + * processing. + * + * - count is changed by SOFTIRQ_DISABLE_OFFSET (=3D 2 * SOFTIRQ_OFFSET) + * on local_bh_disable or local_bh_enable. + * + * This lets us distinguish between whether we are currently processing + * softirq and whether we just have bh disabled. + */ +#ifdef CONFIG_PREEMPT_RT + /** * local_bh_blocked() - Check for idle whether BH processing is blocked * @@ -270,11 +275,6 @@ static inline void ksoftirqd_run_end(void) static inline void softirq_handle_begin(void) { } static inline void softirq_handle_end(void) { } =20 -static inline bool should_wake_ksoftirqd(void) -{ - return !this_cpu_read(softirq_ctrl.cnt); -} - static inline void invoke_softirq(void) { if (should_wake_ksoftirqd()) @@ -419,11 +419,6 @@ static inline void ksoftirqd_run_end(void) local_irq_enable(); } =20 -static inline bool should_wake_ksoftirqd(void) -{ - return true; -} - static inline void invoke_softirq(void) { if (!force_irqthreads() || !__this_cpu_read(ksoftirqd)) { --=20 2.34.1 From nobody Wed Nov 27 03:43:30 2024 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2064.outbound.protection.outlook.com [40.107.220.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 147A813CFBD for ; Mon, 14 Oct 2024 09:06:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.64 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896790; cv=fail; b=tbIRuUmrZBbjfSjktWMTmLfQzenjFUiOjFymi6AQ3jLSJM+GV/b5IN2lhBRWvYZyUgYLly2d/5cA4ScyAWkUW2ZbGu6yYIBFXDJXTY5IOfxZrrjd6vTEf86whAO1r4HP0b7H/aTCNRFHnzS/FJmjVl3ZstcVT7SOoINyv37Zv74= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728896790; c=relaxed/simple; bh=sIyzz3fvlpW3TOgaOOWGImoYuQ78AypeefgW9lyqiGY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SnpF/Ti3uW2v6oyHiAvnz6fsx2nXQZbqa0e8omN5H6MAsvqvVH4mtOFbXtV490x6gLvXJ790piI3XBLTskTNHzaYEW4jDZ77Bou/tS5yWDIduObp3hcHu4mmFi2TAeq7Q2X83Z+CoSzJq1P1/LjTDWHzz9VsGMNoKrfxY2nfY0o= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=nQvLh3pe; arc=fail smtp.client-ip=40.107.220.64 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="nQvLh3pe" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yamCgRwaF8/4GdtsiNhHzbSKX2WpGs6WGIWizNBfRTMmfLPyRKKJWVZcY5X52Nxs1uUOsgOfZcnFwLiQFcegUbe7d7yC8uHGuNEyWU5vtV1hXyFpLIvgVlskliw10Eax9fd3NQ6HxXN9vxE5DPaVJWi2PJQcqq0Dj/2M7WEfN43jdkg94KlAxg6PtoLWUFtvjH/qn4Waka6dm9WJNBMITa8xQUO5W+/04wacF2F10YMewqstvHO+P3Mv/zuWfgvBvzICuHe9NhxODSOTU/yzW1a926+Dp8AoRqAHGou/zOGuE5guB5TCGpUaMzJUu4vrlvTRH+MU9PxUxsD+zicNDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=D7V2FYQZbGLGP2iose40F3ChidVJFQAWWV9OIk5z96g=; b=CrHguMNcNqHHYZq7rNb7WbijeeJrK2D34bYJIoDbPJZ/TuWxKvi5ffFSpOYPYP1o+WZJBWoPFN1D8pWohJeJVlBISMuSV8NHxRFeb8MlPWnKdm/hpiUBF/S0cbir9PjqyjB1svuYvXvf4fSbDChxT9qdT3HhaQIJEv15Y5MLQI/rt92Oj+O8/5r+QnWv4Uv3ZHhA0CVFTFJKqnnXJPNV+RilixpqtPrVNtbLucPjPeZ0GvS4bVf/LCYZC4SHRWAudieJDkLwfz4mv5rwMUaSg3MR41uQ2LO2NuA8AkAgVbGNINi1hZn5P8ZRElwh/ttaV402ZTX3LU37L0ZHZu5qiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D7V2FYQZbGLGP2iose40F3ChidVJFQAWWV9OIk5z96g=; b=nQvLh3peQHGWY+MUa545aRQFhyNhjSOK5wkVaz1vD5kAtsXj29oUnWNFsTgiwppUX1uTgBsWqeB1O73EZZC7Zrzn7W1nyTECmiQIeUs7oY4y8iYLR7giBYP7vhc+F1am+9InThP0IKRqkqQjLy3Ac2WLPG5I3RKJ+pRDzeO/R1Y= Received: from PH8PR05CA0022.namprd05.prod.outlook.com (2603:10b6:510:2cc::10) by MW5PR12MB5623.namprd12.prod.outlook.com (2603:10b6:303:199::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.22; Mon, 14 Oct 2024 09:06:24 +0000 Received: from CY4PEPF0000EE3F.namprd03.prod.outlook.com (2603:10b6:510:2cc:cafe::4d) by PH8PR05CA0022.outlook.office365.com (2603:10b6:510:2cc::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Mon, 14 Oct 2024 09:06:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EE3F.mail.protection.outlook.com (10.167.242.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8048.13 via Frontend Transport; Mon, 14 Oct 2024 09:06:23 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 14 Oct 2024 04:06:13 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Thomas Gleixner , , Sebastian Andrzej Siewior CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Paul E. McKenney" , Rik van Riel , "Neeraj Upadhyay" , Leonardo Bras , Thorsten Blum , Zqiang , Tejun Heo , NeilBrown , Caleb Sander Mateos , "Gautham R . Shenoy" , Chen Yu , Julia Lawall , "K Prateek Nayak" , Julia Lawall Subject: [PATCH v3 5/5] softirq: Avoid unnecessary wakeup of ksoftirqd when a call to do_sofirq() is pending Date: Mon, 14 Oct 2024 09:03:39 +0000 Message-ID: <20241014090339.2478-6-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014090339.2478-1-kprateek.nayak@amd.com> References: <20241014090339.2478-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EE3F:EE_|MW5PR12MB5623:EE_ X-MS-Office365-Filtering-Correlation-Id: 24ab3f0c-705b-4799-fe49-08dcec2f7a45 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ezrnlsbXhW2AZQ/U3MkTn8NL3NOc+ITDakWE+RdkFkf61MHDBvpIq1nqI7hG?= =?us-ascii?Q?71C4nC4/0GfgxBeSCqZ9qNbqwOV1I8yZI7rg/bM4+0fe0KaBkfPz6lOxxV1i?= =?us-ascii?Q?R3qSd/Gb9u4eKjMI1xqjPEBB2026yNJb04PdNTGHoJo0W5CQkXNB+9ZrvDbF?= =?us-ascii?Q?yWYnVj3lnHP93Lm7jfgzcqTipd3Xee7Kkh8hoUS+UbUJP3caFaxJP3g681o0?= =?us-ascii?Q?tbn3wRD+1unlP144D6zlueKzigXHm2tGCnIJr4LXAyK2POH9eY0ocVDpIawJ?= =?us-ascii?Q?Cwpc5/vsnsfnn1ls8byWBHHldMurmshVKa9g7/4tOsiwfEGukKQ+ZQqK9zYM?= =?us-ascii?Q?92gUwXlFRHQRyoFzfimQ2OOo5zytiLO3YDihIQ7JQIuMwSuLlb/V54ikO4Xr?= =?us-ascii?Q?7u9vqMtAZo99WocaNwY5eSXYnri/kJjK11mF4FYMrvYOWmssBPP6qAmMPUyr?= =?us-ascii?Q?SeEdG3uLFv1JpagVeWd/q80cnFmVAbg/4ozSn3gXJIimkPfwmFkX/ZEZIwEM?= =?us-ascii?Q?2HvjiDlLixxB5pHsAvCTeMlh7XuB3tzAUJfPDfUs+ZPOdh/v93PmPxJkLQz3?= =?us-ascii?Q?XEdEZYSGvGPLm/vqrfrnk6VS1JCMvnI7iYvw+tSSlFiuNoYtaDIEDKrHYwaH?= =?us-ascii?Q?ZiOQcIDN7eLnVU1QtP05ChoR5fZQfWanyTj/aSOeGf47ScyNaAb+lQoK+/aA?= =?us-ascii?Q?Vn6at0rTC4mkSaU+0+PpUTJmIvPAI8RO5LOXjmIkyKJcop0l2bQULLKUu4C0?= =?us-ascii?Q?q4b5oj15AU+/TZHdhQvypUwb4drhlmhgNxqEVizQdyuLNYPnjVC7Nph4Cxzw?= =?us-ascii?Q?2aOHP5KVszWnTLeBKG8ZpF6PK66c+3n7cMYV1HxlkXNWnsUTeJtJsM+K4z5f?= =?us-ascii?Q?jyEhqlpK7FtQUgUOwMZJieZ20gnGNt+nqPxjUhuOzUOqf6EyvPdY+Rp3QIP3?= =?us-ascii?Q?B0QiiXkGT/BMr0WK+AqNr50qf4fuqPAw6tA3o6dTzmTvkHefw1eVb8bDbdEg?= =?us-ascii?Q?oKc/ho3S3es0MzDsZPFVeU5gawIR8KDp9YsbQtGTz7RgepwxpxD35lai6xIk?= =?us-ascii?Q?fQyW5czrN4RYhXiLu86rdIXMDGPiY+pDWgtO6ItIYAUEjH9fYkYGW11ZlVvl?= =?us-ascii?Q?A8fxf6IUhTfKnk9nNMY5W6DJWKEaYWTxKwcnzNa0rGDSmwxCD/k1TBzQqQuw?= =?us-ascii?Q?WoZ2/KhKdIJ1grfVS1QhBaHqn6kKV3ph9itEBX98WJ1TfJmaAORtzrilXPob?= =?us-ascii?Q?Vpq32Iw1slT+LV87F2ETVE0woN2ghNEEl6wQeFlcJ1lfSeixY0EMX4B/SMzS?= =?us-ascii?Q?rxYToydO4VgZEEmzsQyzh+qECfgN+2kq6hV67QfMr3+HQ1NgLjEyxNKulx60?= =?us-ascii?Q?eI1Om7Aw9vi303E8DaAsk2ZJoMjn?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2024 09:06:23.7230 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 24ab3f0c-705b-4799-fe49-08dcec2f7a45 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3F.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR12MB5623 Content-Type: text/plain; charset="utf-8" Since commit b2a02fc43a1f4 ("smp: Optimize send_call_function_single_ipi()"), sending an actual interrupt to an idle CPU in TIF_POLLING_NRFLAG mode can be avoided by queuing the SMP call function on the call function queue of the CPU and setting the TIF_NEED_RESCHED bit in idle task's thread info. The call function is handled in the idle exit path when do_idle() calls flush_smp_call_function_queue(). However, since flush_smp_call_function_queue() is executed in idle thread's context, in_interrupt() check within a call function will return false. raise_softirq() uses this check to decide whether to wake ksoftirqd, since, a softirq raised from an interrupt context will be handled at irq exit. In all other cases, raise_softirq() wakes up ksoftirqd to handle the softirq on !PREEMPT_RT kernel. Adding a trace_printk() in nohz_csd_func() at the spot of raising SCHED_SOFTIRQ and enabling trace events for sched_switch, sched_wakeup, and softirq_entry (for SCHED_SOFTIRQ vector alone) helps observing the current behavior: -0 [000] dN.1.: nohz_csd_func: Raising SCHED_SOFTIRQ from n= ohz_csd_func -0 [000] dN.4.: sched_wakeup: comm=3Dksoftirqd/0 pid=3D16 p= rio=3D120 target_cpu=3D000 -0 [000] .Ns1.: softirq_entry: vec=3D7 [action=3DSCHED] -0 [000] .Ns1.: softirq_exit: vec=3D7 [action=3DSCHED] -0 [000] d..2.: sched_switch: prev_comm=3Dswapper/0 prev_pi= d=3D0 prev_prio=3D120 prev_state=3DR =3D=3D> next_comm=3Dksoftirqd/0 next_p= id=3D16 next_prio=3D120 ksoftirqd/0-16 [000] d..2.: sched_switch: prev_comm=3Dksoftirqd/0 prev_= pid=3D16 prev_prio=3D120 prev_state=3DS =3D=3D> next_comm=3Dswapper/0 next_= pid=3D0 next_prio=3D120 ... ksoftirqd is woken up before the idle thread calls do_softirq_post_smp_call_flush() which can make the runqueue appear busy and prevent the idle load balancer from pulling task from an overloaded runqueue towards itself[1]. Since flush_smp_call_function_queue() calls do_softirq_post_smp_call_flush(), waking up ksoftirqd is not necessary since the softirqs raised by the call functions will be handled soon after the call function queue is flushed. Introduce two new APIs: - set_do_softirq_pending(): Increments per-cpu softirq_ctrl::cnt by 1 to indicate a pending call to do_softirq(). - clr_do_softirq_pending(): Decrements per-cpu softirq_ctrl::cnt by 1 just before calling do_softirq(). Call set_do_softirq_pending() before __flush_smp_call_function_queue() within flush_smp_call_function_queue() to indicate a pending call to do_softirq() and clr_do_softirq_pending() before calling do_softirq_post_smp_call_flush() to mark the promise being fulfilled. Since this impending call is tracked by softirq_ctrl::cnt, should_wakeup_ksoftirqd() will return false and prevent a pointless wakeup of ksoftirqd. Following are the observations with the changes when enabling the same set of events: -0 [000] dN.1.: nohz_csd_func: Raising SCHED_SOFTIRQ for= nohz_idle_balance -0 [000] dN.1.: softirq_raise: vec=3D7 [action=3DSCHED] -0 [000] .Ns1.: softirq_entry: vec=3D7 [action=3DSCHED] No unnecessary ksoftirqd wakeups are seen from idle task's context to service the softirq. Fixes: b2a02fc43a1f ("smp: Optimize send_call_function_single_ipi()") Reported-by: Julia Lawall Closes: https://lore.kernel.org/lkml/fcf823f-195e-6c9a-eac3-25f870cb35ac@in= ria.fr/ [1] Suggested-by: Peter Zijlstra # Reuse softirq_ctrl.cn= t from PREEMPT_RT Signed-off-by: K Prateek Nayak --- v2..v3: o Updated the traces with one where SCHED_SOFTIRQ is raised from flush_smp_call_function_queue() and not from hard IRQ context. --- kernel/sched/smp.h | 9 +++++++++ kernel/smp.c | 2 ++ kernel/softirq.c | 14 ++++++++++++++ 3 files changed, 25 insertions(+) diff --git a/kernel/sched/smp.h b/kernel/sched/smp.h index 21ac44428bb0..83f70626ff1e 100644 --- a/kernel/sched/smp.h +++ b/kernel/sched/smp.h @@ -9,7 +9,16 @@ extern void sched_ttwu_pending(void *arg); extern bool call_function_single_prep_ipi(int cpu); =20 #ifdef CONFIG_SMP +/* + * Used to indicate a pending call to do_softirq() from + * flush_smp_call_function_queue() + */ +extern void set_do_softirq_pending(void); +extern void clr_do_softirq_pending(void); + extern void flush_smp_call_function_queue(void); #else +static inline void set_do_softirq_pending(void) { } +static inline void clr_do_softirq_pending(void) { } static inline void flush_smp_call_function_queue(void) { } #endif diff --git a/kernel/smp.c b/kernel/smp.c index f25e20617b7e..be0f84d62475 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -612,7 +612,9 @@ void flush_smp_call_function_queue(void) local_irq_save(flags); /* Get the already pending soft interrupts for RT enabled kernels */ was_pending =3D local_softirq_pending(); + set_do_softirq_pending(); __flush_smp_call_function_queue(true); + clr_do_softirq_pending(); if (local_softirq_pending()) do_softirq_post_smp_call_flush(was_pending); =20 diff --git a/kernel/softirq.c b/kernel/softirq.c index 0730c2b43ae4..3a6b3e67ea24 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -99,6 +99,10 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); * * The per CPU counter prevents pointless wakeups of ksoftirqd in case that * the task which is in a softirq disabled section is preempted or blocks. + * + * The bottom bits of softirq_ctrl::cnt is used to indicate an impending c= all + * to do_softirq() to prevent pointless wakeups of ksoftirqd since the CPU + * promises to handle softirqs soon. */ struct softirq_ctrl { local_lock_t lock; @@ -109,6 +113,16 @@ static DEFINE_PER_CPU_ALIGNED(struct softirq_ctrl, sof= tirq_ctrl) =3D { .lock =3D INIT_LOCAL_LOCK(softirq_ctrl.lock), }; =20 +inline void set_do_softirq_pending(void) +{ + __this_cpu_inc(softirq_ctrl.cnt); +} + +inline void clr_do_softirq_pending(void) +{ + __this_cpu_dec(softirq_ctrl.cnt); +} + static inline bool should_wake_ksoftirqd(void) { return !this_cpu_read(softirq_ctrl.cnt); --=20 2.34.1