From nobody Fri Apr 3 05:20:00 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61791C6FA89 for ; Thu, 15 Sep 2022 14:23:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230171AbiIOOX4 (ORCPT ); Thu, 15 Sep 2022 10:23:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230083AbiIOOXq (ORCPT ); Thu, 15 Sep 2022 10:23:46 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA1A324958; Thu, 15 Sep 2022 07:23:45 -0700 (PDT) Date: Thu, 15 Sep 2022 14:23:42 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1663251824; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EZi+lu5Eqhg5mHLXKWBgYIAlgzIj4opQNwT9tCgnZ/U=; b=SivsOrmxfzncYPO/0EBPgquaEqwgZhUX4k7kaPzvJkf0VUPfj/+zCCpbbMJTqJLchtJhAO C5fWKvn1UNSpGDqAHbI5euHykAJqr6U+dzt1JgETONj3QjCal1n10dkzOBM+WeikIp+Nth XVx8RYGy9KEnyOj/IyTRO0EjlJcpjyqZBDJepcZ+/SUcR8oxH7e3334RcHAh+6K7h19FUi hpC/yZpfieu6+T/XieiiciJUPPzZrxu/gHIp25QX/Tu0WMO1z4Sk8T3xyAsCVoE1RLxJ4Z vHge/sZhWq7nllcrcn0HrSeO1HeuAGdRGUJ7940Da3x9p0H2Itdw0N0X+5z57w== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1663251824; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EZi+lu5Eqhg5mHLXKWBgYIAlgzIj4opQNwT9tCgnZ/U=; b=jr+Eq1dBLoon5woyQ4tuYj/K7TRu4y2PCRDoYE1W5gXiW/pkh2RuBlTnxwsZWWjIM5beH+ bofQ4lpdPuE3QhCQ== From: "tip-bot2 for Gokul krishna Krishnakumar" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: locking/core] locking/rwsem: Disable preemption while trying for rwsem lock Cc: Gokul krishna Krishnakumar , Mukesh Ojha , "Peter Zijlstra (Intel)" , Waiman Long , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <1662661467-24203-1-git-send-email-quic_mojha@quicinc.com> References: <1662661467-24203-1-git-send-email-quic_mojha@quicinc.com> MIME-Version: 1.0 Message-ID: <166325182289.401.441252209836174142.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The following commit has been merged into the locking/core branch of tip: Commit-ID: 48dfb5d2560d36fb16c7d430c229d1604ea7d185 Gitweb: https://git.kernel.org/tip/48dfb5d2560d36fb16c7d430c229d1604= ea7d185 Author: Gokul krishna Krishnakumar AuthorDate: Thu, 08 Sep 2022 23:54:27 +05:30 Committer: Peter Zijlstra CommitterDate: Thu, 15 Sep 2022 16:14:02 +02:00 locking/rwsem: Disable preemption while trying for rwsem lock Make the region inside the rwsem_write_trylock non preemptible. We observe RT task is hogging CPU when trying to acquire rwsem lock which was acquired by a kworker task but before the rwsem owner was set. Here is the scenario: 1. CFS task (affined to a particular CPU) takes rwsem lock. 2. CFS task gets preempted by a RT task before setting owner. 3. RT task (FIFO) is trying to acquire the lock, but spinning until RT throttling happens for the lock as the lock was taken by CFS task. This patch attempts to fix the above issue by disabling preemption until owner is set for the lock. While at it also fix the issues at the places where rwsem_{set,clear}_owner() are called. This also adds lockdep annotation of preemption disable in rwsem_{set,clear}_owner() on Peter Z. suggestion. Signed-off-by: Gokul krishna Krishnakumar Signed-off-by: Mukesh Ojha Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Waiman Long Link: https://lore.kernel.org/r/1662661467-24203-1-git-send-email-quic_mojh= a@quicinc.com --- kernel/locking/rwsem.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index 65f0262..4487359 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -133,14 +133,19 @@ * the owner value concurrently without lock. Read from owner, however, * may not need READ_ONCE() as long as the pointer value is only used * for comparison and isn't being dereferenced. + * + * Both rwsem_{set,clear}_owner() functions should be in the same + * preempt disable section as the atomic op that changes sem->count. */ static inline void rwsem_set_owner(struct rw_semaphore *sem) { + lockdep_assert_preemption_disabled(); atomic_long_set(&sem->owner, (long)current); } =20 static inline void rwsem_clear_owner(struct rw_semaphore *sem) { + lockdep_assert_preemption_disabled(); atomic_long_set(&sem->owner, 0); } =20 @@ -251,13 +256,16 @@ static inline bool rwsem_read_trylock(struct rw_semap= hore *sem, long *cntp) static inline bool rwsem_write_trylock(struct rw_semaphore *sem) { long tmp =3D RWSEM_UNLOCKED_VALUE; + bool ret =3D false; =20 + preempt_disable(); if (atomic_long_try_cmpxchg_acquire(&sem->count, &tmp, RWSEM_WRITER_LOCKE= D)) { rwsem_set_owner(sem); - return true; + ret =3D true; } =20 - return false; + preempt_enable(); + return ret; } =20 /* @@ -1352,8 +1360,10 @@ static inline void __up_write(struct rw_semaphore *s= em) DEBUG_RWSEMS_WARN_ON((rwsem_owner(sem) !=3D current) && !rwsem_test_oflags(sem, RWSEM_NONSPINNABLE), sem); =20 + preempt_disable(); rwsem_clear_owner(sem); tmp =3D atomic_long_fetch_add_release(-RWSEM_WRITER_LOCKED, &sem->count); + preempt_enable(); if (unlikely(tmp & RWSEM_FLAG_WAITERS)) rwsem_wake(sem); }