From nobody Fri Dec 19 16:01:55 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2851143AC5; Sun, 24 Mar 2024 22:35:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711319704; cv=none; b=pAZ8kbDZD0zrU5IhueI5+OjaZCN7pLukjC84fS95WXofFu1JOL5WP8RJmLdCA4peF990FIxPRL0P4ih53oQ38fnKsAYNOoAQ37PQeAJqFFTlQXniN6aib9ScmO0HLwZk3jgC68JLdP5DKmhWG37F/Jxv8g09+WyN6YRX9OTT32Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711319704; c=relaxed/simple; bh=CYKCU5m1Lbnhi0N/OhAGJKYTjm7/p8K6CjRh77KXypA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nXR/CNxj6oGu9MXP2ipQa61F5NuSvB9qD2jTigwsA5noTyZWeEnO1jaP6SsZoA1gBlQLfGmFH/sBwN/dm9G9Yo+nqh667x0b9QqB5/Qc6QlgXMetTJ2CdHVNBHxmyFcBBlAdVniYhIzkEbbxhpBiz6MDrnXoafaSvC1TlztZrqg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IqOFuSlk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IqOFuSlk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0F70FC433B2; Sun, 24 Mar 2024 22:35:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711319703; bh=CYKCU5m1Lbnhi0N/OhAGJKYTjm7/p8K6CjRh77KXypA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IqOFuSlkXrbea8LnbKK4PZv+050a2lolToSG3k6bHnAXNIQWfPtF4Tjt9Q9l4ncyW U1WizDbiDf/yLDGjoU80MrP9BnSsH1ZkwmMEOsa0LahgPmsGCDFOdx5SqU0nQ+DCN0 X7jsdxAsq8IJ4ldQqd2+koMY83xLaciyyXo+vDTtkl/RmW11ETtc/NvG0gijx8Cfrh qcAUtYzu8kIfh67uSvROTPDWct9BJeX9fE+OymgSGXzfoGHL1DyEmLWSOZWHfAHv6s l5bqidlxJ6PXnQsvAw+pNoA9fLuxQCSG5s0x/HcXct5AmYXkZujwGBxolDYYusiXs1 AYCVyNKG+MjUQ== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Tejun Heo , Lai Jiangshan , Sasha Levin Subject: [PATCH 6.8 006/715] workqueue: Move nr_active handling into helpers Date: Sun, 24 Mar 2024 18:23:05 -0400 Message-ID: <20240324223455.1342824-7-sashal@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240324223455.1342824-1-sashal@kernel.org> References: <20240324223455.1342824-1-sashal@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Tejun Heo [ Upstream commit 1c270b79ce0b8290f146255ea9057243f6dd3c17 ] __queue_work(), pwq_dec_nr_in_flight() and wq_adjust_max_active() were open-coding nr_active handling, which is fine given that the operations are trivial. However, the planned unbound nr_active update will make them more complicated, so let's move them into helpers. - pwq_tryinc_nr_active() is added. It increments nr_active if under max_active limit and return a boolean indicating whether inc was successful. Note that the function is structured to accommodate future changes. __queue_work() is updated to use the new helper. - pwq_activate_first_inactive() is updated to use pwq_tryinc_nr_active() and thus no longer assumes that nr_active is under max_active and returns a boolean to indicate whether a work item has been activated. - wq_adjust_max_active() no longer tests directly whether a work item can be activated. Instead, it's updated to use the return value of pwq_activate_first_inactive() to tell whether a work item has been activated. - nr_active decrement and activating the first inactive work item is factored into pwq_dec_nr_active(). v3: - WARN_ON_ONCE(!WORK_STRUCT_INACTIVE) added to __pwq_activate_work() as now we're calling the function unconditionally from pwq_activate_first_inactive(). v2: - wq->max_active now uses WRITE/READ_ONCE() as suggested by Lai. Signed-off-by: Tejun Heo Reviewed-by: Lai Jiangshan Stable-dep-of: 5797b1c18919 ("workqueue: Implement system-wide nr_active en= forcement for unbound workqueues") Signed-off-by: Sasha Levin --- kernel/workqueue.c | 86 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 67 insertions(+), 19 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 7e1b0238158ea..80733046ee012 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1464,11 +1464,14 @@ static bool pwq_is_empty(struct pool_workqueue *pwq) static void __pwq_activate_work(struct pool_workqueue *pwq, struct work_struct *work) { + unsigned long *wdb =3D work_data_bits(work); + + WARN_ON_ONCE(!(*wdb & WORK_STRUCT_INACTIVE)); trace_workqueue_activate_work(work); if (list_empty(&pwq->pool->worklist)) pwq->pool->watchdog_ts =3D jiffies; move_linked_works(work, &pwq->pool->worklist, NULL); - __clear_bit(WORK_STRUCT_INACTIVE_BIT, work_data_bits(work)); + __clear_bit(WORK_STRUCT_INACTIVE_BIT, wdb); } =20 /** @@ -1493,12 +1496,66 @@ static bool pwq_activate_work(struct pool_workqueue= *pwq, return true; } =20 -static void pwq_activate_first_inactive(struct pool_workqueue *pwq) +/** + * pwq_tryinc_nr_active - Try to increment nr_active for a pwq + * @pwq: pool_workqueue of interest + * + * Try to increment nr_active for @pwq. Returns %true if an nr_active coun= t is + * successfully obtained. %false otherwise. + */ +static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq) +{ + struct workqueue_struct *wq =3D pwq->wq; + struct worker_pool *pool =3D pwq->pool; + bool obtained; + + lockdep_assert_held(&pool->lock); + + obtained =3D pwq->nr_active < READ_ONCE(wq->max_active); + + if (obtained) + pwq->nr_active++; + return obtained; +} + +/** + * pwq_activate_first_inactive - Activate the first inactive work item on = a pwq + * @pwq: pool_workqueue of interest + * + * Activate the first inactive work item of @pwq if available and allowed = by + * max_active limit. + * + * Returns %true if an inactive work item has been activated. %false if no + * inactive work item is found or max_active limit is reached. + */ +static bool pwq_activate_first_inactive(struct pool_workqueue *pwq) +{ + struct work_struct *work =3D + list_first_entry_or_null(&pwq->inactive_works, + struct work_struct, entry); + + if (work && pwq_tryinc_nr_active(pwq)) { + __pwq_activate_work(pwq, work); + return true; + } else { + return false; + } +} + +/** + * pwq_dec_nr_active - Retire an active count + * @pwq: pool_workqueue of interest + * + * Decrement @pwq's nr_active and try to activate the first inactive work = item. + */ +static void pwq_dec_nr_active(struct pool_workqueue *pwq) { - struct work_struct *work =3D list_first_entry(&pwq->inactive_works, - struct work_struct, entry); + struct worker_pool *pool =3D pwq->pool; =20 - pwq_activate_work(pwq, work); + lockdep_assert_held(&pool->lock); + + pwq->nr_active--; + pwq_activate_first_inactive(pwq); } =20 /** @@ -1516,14 +1573,8 @@ static void pwq_dec_nr_in_flight(struct pool_workque= ue *pwq, unsigned long work_ { int color =3D get_work_color(work_data); =20 - if (!(work_data & WORK_STRUCT_INACTIVE)) { - pwq->nr_active--; - if (!list_empty(&pwq->inactive_works)) { - /* one down, submit an inactive one */ - if (pwq->nr_active < READ_ONCE(pwq->wq->max_active)) - pwq_activate_first_inactive(pwq); - } - } + if (!(work_data & WORK_STRUCT_INACTIVE)) + pwq_dec_nr_active(pwq); =20 pwq->nr_in_flight[color]--; =20 @@ -1825,13 +1876,11 @@ static void __queue_work(int cpu, struct workqueue_= struct *wq, * @work must also queue behind existing inactive work items to maintain * ordering when max_active changes. See wq_adjust_max_active(). */ - if (list_empty(&pwq->inactive_works) && - pwq->nr_active < READ_ONCE(pwq->wq->max_active)) { + if (list_empty(&pwq->inactive_works) && pwq_tryinc_nr_active(pwq)) { if (list_empty(&pool->worklist)) pool->watchdog_ts =3D jiffies; =20 trace_workqueue_activate_work(work); - pwq->nr_active++; insert_work(pwq, work, &pool->worklist, work_flags); kick_pool(pool); } else { @@ -4680,9 +4729,8 @@ static void wq_adjust_max_active(struct workqueue_str= uct *wq) /* this function can be called during early boot w/ irq disabled */ raw_spin_lock_irqsave(&pwq->pool->lock, flags); =20 - while (!list_empty(&pwq->inactive_works) && - pwq->nr_active < wq->max_active) - pwq_activate_first_inactive(pwq); + while (pwq_activate_first_inactive(pwq)) + ; =20 kick_pool(pwq->pool); =20 --=20 2.43.0