From nobody Tue Dec 2 00:25:43 2025 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E3DEE298CDE for ; Tue, 25 Nov 2025 06:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052342; cv=none; b=cHAq0+Wyse27eV4caG1bZQecSwGfJaxJHoQTfYLyoSJtaSmCSOFyAwuFhdwusPdwEyQhm7PwBPDS5/ozmGVda8cuJvJ1rh2b3gn/aY2HaHkCwhM8nIc4dDlThUkFRIU4og55RXW59EVukp5sCUVEba+J1w8f+hSk4u73GsYk9ZA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052342; c=relaxed/simple; bh=kzsApZ8k7L9zdQootOUd+LETAxNM7N9sI8AEUxviAsw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sBmOIV2BkAK0bVSrKF+OXOPgLhUksxduQzQJ+ehgJXCFsbKu1yk1Pq+dFrgp623ueDrzn/hNvhpy/WnYuKSjciY05qr/DEUDW3zuzrX9BbaV8FRlYtNwqX6GuC78pzbh+d1UarZNimoYg+S4NtAuC2zqUFYWId14TIEcbYWngf0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cW9Yaq6u; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cW9Yaq6u" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-7a9c64dfa8aso3979324b3a.3 for ; Mon, 24 Nov 2025 22:32:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764052340; x=1764657140; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c/8nTaqZzY56h+wYOzS7tOWNX+nlZk/rLssu/KjYUAY=; b=cW9Yaq6uvtTp3+xXxJN6RcQhymlMshDi+WC8gORs8+owKAh5H9z3zQcHMBnB4GGfmX fdOKUJ9CFbTi7RHz/5LCOU883PweAvyvCxZCAAnDqOlnoKPS2raUQ1wfC45tlKq05Svs 1gyRSHJFIY+X7I2jQgoBm0jydRiFOrZ3JP90hH+YKhPFK59jAdYGhdIycBRUK/mn83n7 OTGgH5OUCTFniRcQOgfCnrgUt1aNawiv3C1Ty/VwvFrz6ROCMJRMYad5Sca+vrlyBYOC pM2syZbPX3XDMJx6p/8I9zJfEE+6mIhzTRtQl4hhnUANN8bPgFPLpVX7sZr2IM5w39bE 9yxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764052340; x=1764657140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=c/8nTaqZzY56h+wYOzS7tOWNX+nlZk/rLssu/KjYUAY=; b=oAqaxvABO9h2Ca7K/98BDhfjfclMIqwE3yLl4eSRNjEbYlbo5uxnNPnB9nikJf/Z/N +AHDXfkK5/rEWrJ6F15xbkUJSVuyHpuTBQBPgRjrKnU7WLcg2C2cyaHd3FqYVYcwEbLZ hJEJoJay7tjH6KbEv8M5c5kVbG9xswS5F2fSDh2dc/IezCmxg64Nxl5oy4b1uyXG7X7O gK7tEpGEYtaMPqkEKAL9hFdIRt2k/+VUo0f03wiVbHQw+pNMuyTOPWIgAsW1VMuKm9j7 3bwnGIT5cnaHzqH0nUG91vjRnl9sSQtrG9Ze/xdTS/jxB21NntUaO9PJt0Rp8bRxOIUz Z/eg== X-Gm-Message-State: AOJu0YxAEpQATZtx3j3AGBLPH5p3vIAZDodTZS4tQu2lab/F8HILMOm7 xFziePNrjsrrdRorYFkRDQZw5H+PSMM4D4wynNA/cavO261t1OBfvatdObVU2Gbq X-Gm-Gg: ASbGncv8ETWP8B8Rsoy3TaiZSsTpwrWU3PRtV+88QcwvMwCZvFh5z2xFF7fUaBSj4rS rUzX+IeBMDSBwoI1zB8OYhYJ3PuY9IOlMsESK2PVRYgGw4LfIBSVxgpsH2tpcxwHwgSabmtVzLO hHMLFQmqtUtj0GUfcY4N5qxNEfhxtTqAdHvOooBw1Gn5xt7izp4/Bi6+7CbBYJOyjn19fGu18eh n/LZPBhV6IkLdFlygBuI5KujiBcMj1B6bcQIIj81DhrPQCYREtHE70LXvFPelEcbYgEaj+IS9os PsLuG2IJ3dakK4ixUZTTGRi5GxhjaGP8b6VW5V++Sj5S2VPZSMaVd0sL2hk2aeyoND1TKV3IptF 9c7ddGmdv0S/j2SRgZw9e+0ktX2g6IwmAeDfppe1rCojrH217HRl124VuYRRn0btay1W8N7TSci y1Kvnktf/U+ds= X-Google-Smtp-Source: AGHT+IG8Aj1LNsOBYeqbsFiew9Kaim6+3OzL1fGJPrX0hh3JPJ8Nn+KzfiRnmui+CrDnCW8oeLq9WQ== X-Received: by 2002:a05:7022:7f11:b0:11b:9386:a38b with SMTP id a92af1059eb24-11cbba87ea7mr899017c88.46.1764052339740; Mon, 24 Nov 2025 22:32:19 -0800 (PST) Received: from localhost ([240b:4004:a2:7900:ecf3:dec8:8c1e:4f6]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-11c93de6d5csm53578599c88.4.2025.11.24.22.32.19 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Nov 2025 22:32:19 -0800 (PST) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Tejun Heo , ying chen , Lai Jiangshan , Lai Jiangshan Subject: [PATCH V4 1/4] workqueue: Loop over in rescuer until all its work is done Date: Tue, 25 Nov 2025 14:36:14 +0800 Message-Id: <20251125063617.671199-2-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20251125063617.671199-1-jiangshanlai@gmail.com> References: <20251125063617.671199-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Simplify the rescuer work by looping directly in the rescuer rather than adding the pwq back to the maydays list. This also helps when max_requests is 1 or small but pwq->inactive_works has a large number of pending work items. This might hurt fairness among PWQs and the rescuer could end up being stuck on one PWQ indefinitely, but the rescuer's objective is to make forward progress rather than ensure fairness. Fairness can be further improved in future by assigning work items to the rescuer one by one and this is a temporary change to ease the transition. Signed-off-by: Lai Jiangshan --- kernel/workqueue.c | 24 +----------------------- 1 file changed, 1 insertion(+), 23 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 2654fbd481a1..02386e6eb409 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -3520,31 +3520,9 @@ static int rescuer_thread(void *__rescuer) =20 WARN_ON_ONCE(!list_empty(&rescuer->scheduled)); =20 - if (assign_rescuer_work(pwq, rescuer)) { + while (assign_rescuer_work(pwq, rescuer)) process_scheduled_works(rescuer); =20 - /* - * The above execution of rescued work items could - * have created more to rescue through - * pwq_activate_first_inactive() or chained - * queueing. Let's put @pwq back on mayday list so - * that such back-to-back work items, which may be - * being used to relieve memory pressure, don't - * incur MAYDAY_INTERVAL delay inbetween. - */ - if (pwq->nr_active && need_to_create_worker(pool)) { - raw_spin_lock(&wq_mayday_lock); - /* - * Queue iff somebody else hasn't queued it already. - */ - if (list_empty(&pwq->mayday_node)) { - get_pwq(pwq); - list_add_tail(&pwq->mayday_node, &wq->maydays); - } - raw_spin_unlock(&wq_mayday_lock); - } - } - /* * Leave this pool. Notify regular workers; otherwise, we end up * with 0 concurrency and stalling the execution. --=20 2.19.1.6.gb485710b From nobody Tue Dec 2 00:25:43 2025 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 517C429CB57 for ; Tue, 25 Nov 2025 06:32:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052346; cv=none; b=Mgxa2KINfQHp6EC/lVG961bDyUxMMdJVCHfzSvmuXq1E5MWWPG9CCrw+uxO7yY1Xhi7DKySqDC8KmkuIDy0VWvPsMb6OQO7gBBKOjNLLAH8Wsoi2oa4a7WjBst5R0NqLCh91U39snQ5kKabOl/dOpm4/OEjWbBBFuCj0ALU/bbc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052346; c=relaxed/simple; bh=X9TvG6zxHwZqc60PhR8CQyzRniUo8show66coxNAmm0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rmHmthG45n3qn4+qLGzpAT8buhlIPjAt9KLsfZE5BLH7UFZr+yHL0f5+H39/du7CxHsKGhvn+5jLRAcJ91mPSp1iVOXpTvWwAAbznmKY2mqRXpPBAE0+Gei0isLR7T7lEDNGrc+y1hfxsmtPIa3hSdxgXqacbxIq9LhYmbVVsAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FWVKdbUA; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FWVKdbUA" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-7c66822dd6dso1608880b3a.0 for ; Mon, 24 Nov 2025 22:32:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764052343; x=1764657143; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=il2RpXEIXVNw+BA9yi89csS66MG4gPZKPafwjtYIz/Q=; b=FWVKdbUAqZXSyZAWzXKsPYcLmYi/gUUyk5j6R95CJcTDhYXbYBaQOQCsWEfVtsKmn1 Z8DQTanIYEN7uCiat8fswW7e6oqEQybZ9eBqBXi8ECHFnC2Ceie40LOD5Xf2rRenhC5o Z/PlDeIqnIEw8kez+NlPQCpvVSw9LE3E5G96XKTPUJ+1o9F1XbViTDV2W+EsXZQq8fXb /qXkZUZng5y5oFHwx1GWTxkXXzz1vbyfJN52PkW0zB0D7JNXQy6ool+lhWEwTsXth5DI ZeEV1gGyewMYYui4jqR4ViLqZGrHV21HsIBymZrBqaZqSeXL4OUKMOob0gxub4b9mkD7 +S9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764052343; x=1764657143; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=il2RpXEIXVNw+BA9yi89csS66MG4gPZKPafwjtYIz/Q=; b=PmkKqWg1rL02CSonO7NkMTVNjrvAcPmK0fhXYcs5cuxaGCF+sjIZdb/0Z/d80Mn7wk +1zp+I9FkrUZfmuy5lOuA0SRJgpFMk3vEFchUDDCUmh2uVJxEtiprgJyPfgfDizccAul N1TV9hMKteZFgJfZuKeqh9c2d3hVxT7xH7aM5Ve1qOj0o/RW+ETs6jldCutmiWWGhETJ hvF+3YlBRiN2IsRWCnLtHOItBEvOM4D4FGwKm6heIt/KOuDFTebLhP8MTyEociSMEUys tlNOr3HAZqgyu7goBTb8azqIiKTpyVu9Ji9jTqh8CRXPkkE65LMUkVdomjTDOVOPXB/G ZdTg== X-Gm-Message-State: AOJu0YyII1qqOAqZgQB97DDX+BNbnRUTt5Z8i42jmanWOiPV+dgpZVqP 9/m6yZg+yJs8lTQmtHkNv9gxfVbkWbzWDpY3YmQQA6tjy8ZbJUznkPualmwga4mG X-Gm-Gg: ASbGnctxFfECEnHxaCfCWFyc8f0ddmdRqZvytsOOeS+l34naP/LUJGMh9b/BUcRYitm b682S4ZXLi400JTywNGXSApfkOURF5YE31i5Wdljzncj9bp6SX+OOpcaSlU+7XOVJS0hp9l5ZN0 SskLRbHqeRjkvEl4L9pvAwWzm11XERebCe0PEpZjw8MdM1ax5IezpZNc3WqUCKl8DCbsxGEn6vP DOOPuBK+duKDRFR/hTSoAMZLTTnbrz9/CxVzhh73RxrZ/d9Ij+rrDe+seniVNtMcWTrJrAKRUo7 7DI+1r5xgmfpLlAXPL/H4uWYAKs8FbNdPIRRlCdRCz/MG9Vv9TLkORMmNtr8v5HTzJnAxVG6wH/ ytIbM7n3XMW2yFEK+3j/LgtVyEfDn9BMTqGUcj7mvm793k01By+2K8b3y4nV3SQIiLOZOpP7Ox1 1qDFw+uYs24Zc= X-Google-Smtp-Source: AGHT+IFdsC2utimf3qvk2fooicbHSBNEGLv1NFWm4ha6niJGOxk1QfjKzFA+gz2+ey1o8TG7dg4eFg== X-Received: by 2002:a05:6a20:a114:b0:342:fa5:8b20 with SMTP id adf61e73a8af0-3614f5aec2cmr14618378637.30.1764052343284; Mon, 24 Nov 2025 22:32:23 -0800 (PST) Received: from localhost ([240b:4000:bb:1700:7b36:494d:5625:5a0]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7c3f0243b2fsm16616310b3a.32.2025.11.24.22.32.22 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Nov 2025 22:32:23 -0800 (PST) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Tejun Heo , ying chen , Lai Jiangshan , Lai Jiangshan Subject: [PATCH V4 2/4] workqueue: Process rescuer work items one-by-one using a cursor Date: Tue, 25 Nov 2025 14:36:15 +0800 Message-Id: <20251125063617.671199-3-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20251125063617.671199-1-jiangshanlai@gmail.com> References: <20251125063617.671199-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Previously, the rescuer scanned for all matching work items at once and processed them within a single rescuer thread, which could cause one blocking work item to stall all others. Make the rescuer process work items one-by-one instead of slurping all matches in a single pass. Break the rescuer loop after finding and processing the first matching work item, then restart the search to pick up the next. This gives normal worker threads a chance to process other items which gives them the opportinity to be processed instead of waiting on the rescuer's queue and prevents a blocking work item from stalling the rest once memory pressure is relieved. Introduce a dummy cursor work item to avoid potentially O(N^2) rescans of the work list. The marker records the resume position for the next scan, eliminating redundant traversals. Cc: ying chen Reported-by: ying chen Fixes: e22bee782b3b ("workqueue: implement concurrency managed dynamic work= er pool") Signed-off-by: Lai Jiangshan --- kernel/workqueue.c | 55 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 48 insertions(+), 7 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 02386e6eb409..06cd3d6ff7e1 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -286,6 +286,7 @@ struct pool_workqueue { struct list_head pending_node; /* LN: node on wq_node_nr_active->pending_= pwqs */ struct list_head pwqs_node; /* WR: node on wq->pwqs */ struct list_head mayday_node; /* MD: node on wq->maydays */ + struct work_struct mayday_cursor; /* L: cursor on pool->worklist */ =20 u64 stats[PWQ_NR_STATS]; =20 @@ -1120,6 +1121,12 @@ static struct worker *find_worker_executing_work(str= uct worker_pool *pool, return NULL; } =20 +static void mayday_cursor_func(struct work_struct *work) +{ + /* should not be processed, only for marking position */ + BUG(); +} + /** * move_linked_works - move linked works to a list * @work: start of series of works to be scheduled @@ -1182,6 +1189,16 @@ static bool assign_work(struct work_struct *work, st= ruct worker *worker, =20 lockdep_assert_held(&pool->lock); =20 + /* The cursor work should not be processed */ + if (unlikely(work->func =3D=3D mayday_cursor_func)) { + /* only worker_thread() can possibly take this branch */ + WARN_ON_ONCE(worker->rescue_wq); + if (nextp) + *nextp =3D list_next_entry(work, entry); + list_del_init(&work->entry); + return false; + } + /* * A single work shouldn't be executed concurrently by multiple workers. * __queue_work() ensures that @work doesn't jump to a different pool @@ -3436,22 +3453,33 @@ static int worker_thread(void *__worker) static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker = *rescuer) { struct worker_pool *pool =3D pwq->pool; + struct work_struct *cursor =3D &pwq->mayday_cursor; struct work_struct *work, *n; =20 + /* search from the start or cursor if available */ + if (list_empty(&cursor->entry)) { + work =3D list_first_entry(&pool->worklist, struct work_struct, entry); + } else { + work =3D list_next_entry(cursor, entry); + /* It will be at a new position or not need cursor anymore */ + list_del_init(&cursor->entry); + } + /* need rescue? */ if (!pwq->nr_active || !need_to_create_worker(pool)) return false; =20 - /* - * Slurp in all works issued via this workqueue and - * process'em. - */ - list_for_each_entry_safe(work, n, &pool->worklist, entry) { - if (get_work_pwq(work) =3D=3D pwq && assign_work(work, rescuer, &n)) + /* find the next work item to rescue */ + list_for_each_entry_safe_from(work, n, &pool->worklist, entry) { + if (get_work_pwq(work) =3D=3D pwq && assign_work(work, rescuer, &n)) { pwq->stats[PWQ_STAT_RESCUED]++; + /* put the cursor for next search */ + list_add_tail(&cursor->entry, &n->entry); + return true; + } } =20 - return !list_empty(&rescuer->scheduled); + return false; } =20 /** @@ -5135,6 +5163,19 @@ static void init_pwq(struct pool_workqueue *pwq, str= uct workqueue_struct *wq, INIT_LIST_HEAD(&pwq->pwqs_node); INIT_LIST_HEAD(&pwq->mayday_node); kthread_init_work(&pwq->release_work, pwq_release_workfn); + + /* + * Set the dummy cursor work with valid function and get_work_pwq(). + * + * The cursor work should only be in the pwq->pool->worklist, and + * should not be treated as a processable work item. + * + * WORK_STRUCT_PENDING and WORK_STRUCT_INACTIVE just make it less + * surprise for kernel debuging tools and reviewers. + */ + INIT_WORK(&pwq->mayday_cursor, mayday_cursor_func); + atomic_long_set(&pwq->mayday_cursor.data, (unsigned long)pwq | + WORK_STRUCT_PENDING | WORK_STRUCT_PWQ | WORK_STRUCT_INACTIVE); } =20 /* sync @pwq with the current state of its associated wq and link it */ --=20 2.19.1.6.gb485710b From nobody Tue Dec 2 00:25:43 2025 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D62AC2BE64A for ; Tue, 25 Nov 2025 06:32:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052349; cv=none; b=f3yGWJfE2BYyk2BVpZ+V3osV+VUOfU+hU3C9s9KInhDnUhPTug9q8VlijQYlJphaCkAfsT2DAUlUAcjrRAJhhosRixvBGYMx5ANaj3BW+kRkp4bxCWVMjDM9JkI8vOBjKtoEQW39YASpQA/ROmqQCOBH4aayqzOjDb+DxF6+mFw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052349; c=relaxed/simple; bh=iWOhizPZ5b8MlafjCIMS4s/7fP/UjGGHBzJjqhvrQMo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JnX/13YJTthFwQw7roC6+9SvUXPSs+OnL/aOK8HKR3gZdgEfc3BzEG8CJs1USa1QaUQdmtiRErd/wR7Lc6YDIEV4BWXGYrnX8o+SvqseNcvR+x8ZO+y3hbSdKewOoJyzxRT1qQ0qMdv3Lj9YqHGWO3q5WIv9BhVJzpuDJ4VzZrE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YLobadO1; arc=none smtp.client-ip=209.85.215.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YLobadO1" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-bc0e89640b9so3294880a12.1 for ; Mon, 24 Nov 2025 22:32:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764052347; x=1764657147; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=boOqRcUxmNCW9UWJksRZe9rQfSJ3rUsWo5BpoBtUE1Q=; b=YLobadO1lIAyf4fSJUauBdtmLtCwJYL63rTZfchbHlHZVGeHGtJqrjqPhg6mKWmopR UvWJDkReGgTNbKDpH/RBiMgUp83a9EZpBFRohj6+LyJaEuMt/g5Bo/oWhCTWLYRuK5xm BIMJ/b8y1cDK8szqmu8JRj8DPLsblod2MIMksa+ser6apI90rU57TH3Em/DpHT6zaZNo td+8xgiknKL0KrDwRG7CTofb/WJASF0FsmhI2oIof75l9Dg6gWC1dAEna6nTuTPLw4+x Ud9d9T8K1xyQ+1nD0csLO/NrVI4IzCTAqmFbpGyyo8OEQqvcMf5SBILMWQ+mBcAdA3RS 0VXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764052347; x=1764657147; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=boOqRcUxmNCW9UWJksRZe9rQfSJ3rUsWo5BpoBtUE1Q=; b=fablWaIPaGtQwfNwhWGLmHx6W+aw1gDNoxaigZvUIZORQYkZZyD+hjKgJKYJ45+nU7 aKp0APKFKa9/d71yBty3E5LJh0OyI6gTKTZplk5txUtFv+UQsMFY8luPR6KPqGYts7c6 0KN6pCPb36SFnpWQ2GktON6fPnzAEnu3jUnJLqXBx9Z2afka/KWb6zVQNhl8ADYPrXX/ aQxgZjDIDeiM3O04BcaCC5vfPipNbLYIgvY0Yo3a6KhjcMBFXfUvJRLymLoqR3w4rm0O U+xNVpA9lXnmRWUilL6LxDg5pqQfHPtKStNSei5TaKHedNe3qmAljJFQsEvGnvVqlJOr hRcQ== X-Gm-Message-State: AOJu0YyqnsOMMTYaEUdF0z/OzLYnNLPBVxp/GUgxixpUqZmcXKZBXl9l KdqL6LtIapAJYqZMTFy9HUdcbEyF0oWstf4S/tDJJAl/r89reJqN0KKBJsJhi6nf X-Gm-Gg: ASbGncu+n8ppU6Te3l2jQ2skctmjqUwNvxR2kDtnj5XCs/kOSb8QbuTEYezZ9YI1sh3 EPba5PsHdaTyrBgfkwoxyofUKPPIqXH/Oa19ezq1Czt4DCHxNzrHNfyiXDlKrFa/gC1CVosvSJ2 z8JQH6FHnVGBiNNcBHyt+BADqZxkGP7b3jw3sgKnP46NsiM5g0FW8amUtja5oXVwG5di3GusY4l f/X6XV3miXi5MeGhbTxu6RtPgRprnHwlDgVW//ytR+sf+gZ06gZOxNplhTMlUUUAikQj2YlDEn+ ERLc8Se3KpwvpXrNd+UKlCmEBg9SpdqrHZIPxbd9UWQNPW1u8vpEsFKaqvghC7XAi0rIb65LhEy hPhwImbYrDFP7Kg6BZNuPu8N8vmxruQN1djsb9ldOinXOIKfZlq3pG2Ub6Ca0vGsFcKcSBcckMR kqpdOi6Kk74Yo= X-Google-Smtp-Source: AGHT+IExsyhi3g7T+UDHjmiRwUzOanMTtYwusz0GkcrpQIjjZD5JSIpoM6DBI1uQBTh72N3IrO7FBQ== X-Received: by 2002:a05:701b:2719:b0:11b:b2f5:a1f4 with SMTP id a92af1059eb24-11c9d863125mr9036261c88.39.1764052346769; Mon, 24 Nov 2025 22:32:26 -0800 (PST) Received: from localhost ([240b:4004:a2:7900:ecf3:dec8:8c1e:4f5]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-11cc631c236sm6681959c88.7.2025.11.24.22.32.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Nov 2025 22:32:26 -0800 (PST) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Tejun Heo , ying chen , Lai Jiangshan , Lai Jiangshan Subject: [PATCH V4 3/4] workqueue: Limit number of processed works in rescuer per turn Date: Tue, 25 Nov 2025 14:36:16 +0800 Message-Id: <20251125063617.671199-4-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20251125063617.671199-1-jiangshanlai@gmail.com> References: <20251125063617.671199-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Currently the rescuer keeps looping until all work on a PWQ is done, and this may hurt fairness among PWQs, as the rescuer could remain stuck on one PWQ indefinitely. Introduce RESCUER_BATCH to control the maximum number of work items the rescuer processes in each turn, and move on to other PWQs when the limit is reached. Signed-off-by: Lai Jiangshan --- kernel/workqueue.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 06cd3d6ff7e1..7cec9755b4e1 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -117,6 +117,8 @@ enum wq_internal_consts { MAYDAY_INTERVAL =3D HZ / 10, /* and then every 100ms */ CREATE_COOLDOWN =3D HZ, /* time to breath after fail */ =20 + RESCUER_BATCH =3D 16, /* process items per turn */ + /* * Rescue workers are used only on emergencies and shared by * all cpus. Give MIN_NICE. @@ -3450,7 +3452,15 @@ static int worker_thread(void *__worker) goto woke_up; } =20 -static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker = *rescuer) +/* + * Try to assign one work item from @pwq to @rescuer. + * + * Returns true if a work item was successfully assigned, false otherwise. + * If @throttled and other PWQs are in mayday, requeue mayday for this PWQ + * and let the rescuer handle other PWQs first. + * If this is the only PWQ in mayday, process it regardless of @throttled. + */ +static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker = *rescuer, bool throttled) { struct worker_pool *pool =3D pwq->pool; struct work_struct *cursor =3D &pwq->mayday_cursor; @@ -3471,7 +3481,21 @@ static bool assign_rescuer_work(struct pool_workqueu= e *pwq, struct worker *rescu =20 /* find the next work item to rescue */ list_for_each_entry_safe_from(work, n, &pool->worklist, entry) { - if (get_work_pwq(work) =3D=3D pwq && assign_work(work, rescuer, &n)) { + if (get_work_pwq(work) !=3D pwq) + continue; + /* + * If throttled, update the cursor, requeue a mayday for this + * PWQ, and move on to other PWQs. If there are no other PWQs + * in mayday, continue processing this one. + */ + if (throttled && !list_empty(&pwq->wq->maydays)) { + list_add_tail(&cursor->entry, &work->entry); + raw_spin_lock(&wq_mayday_lock); /* for wq->maydays */ + send_mayday(work); + raw_spin_unlock(&wq_mayday_lock); + return false; + } + if (assign_work(work, rescuer, &n)) { pwq->stats[PWQ_STAT_RESCUED]++; /* put the cursor for next search */ list_add_tail(&cursor->entry, &n->entry); @@ -3536,6 +3560,7 @@ static int rescuer_thread(void *__rescuer) struct pool_workqueue *pwq =3D list_first_entry(&wq->maydays, struct pool_workqueue, mayday_node); struct worker_pool *pool =3D pwq->pool; + unsigned int count =3D 0; =20 __set_current_state(TASK_RUNNING); list_del_init(&pwq->mayday_node); @@ -3548,7 +3573,7 @@ static int rescuer_thread(void *__rescuer) =20 WARN_ON_ONCE(!list_empty(&rescuer->scheduled)); =20 - while (assign_rescuer_work(pwq, rescuer)) + while (assign_rescuer_work(pwq, rescuer, ++count > RESCUER_BATCH)) process_scheduled_works(rescuer); =20 /* --=20 2.19.1.6.gb485710b From nobody Tue Dec 2 00:25:43 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0DE12BE7B6 for ; Tue, 25 Nov 2025 06:32:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052352; cv=none; b=g2ETBGC/BiJ9B457+HMMhiFL6QKLAdFAd6D5OBfGjvAa0ekchuPFDB8bE+R2CI9XJ/pDRw+XI3zOuLdjldQkpDB4o0nOX7jV25fDKBCydXX3l9lepVIpTwymgRR3EcRt9YBqNMdMFjBkx3wvIw9GS02h4R5Cm28r9t53e6St7ZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764052352; c=relaxed/simple; bh=60w5pS9yBSx1jjubcjS32HjXJo9fhAM3P6pHnvtYbBU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ff8VjN9RpkaArDeY5hxx5LHYWrZKzTiXJyVgl9kkJUfYSDbw8/vZjNxDTs5pp6WTGV5VIH6bwXGf2gkEbrvsD/QARwZhmONKwPXav93ZjP+3sf8NdpuRE+bltXLtR2btt+P7jicFUsZEOcfTbtNTxRGU/rFMOHGTXeID0+uo/hM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FpAzqlRY; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FpAzqlRY" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2980d9b7df5so63649615ad.3 for ; Mon, 24 Nov 2025 22:32:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764052350; x=1764657150; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LPmid93vVx5WXKVpvZGHfLR1iexFd/wtP/8PCZcFv7U=; b=FpAzqlRYzfahPUvd47bashl0TezZdfx3bKqW3dvSH1GZ/7MYrT/gb91tZ6z3EgRLul rFYNbdxFG1V6jkNqX4wWw8TgeUd20/W7sFPPMxKrFnJ6GR0ZnnQkVEiLdw5FgKxUaRio /KGv3o+ir6hX8vkInXiOmTb2ONiZxDxhcap2iJbsaFLW8TyAVqDrfV8JDSvs6O5+5SsC omLeUsqjmPok7dQ0AuwezuS3IXCIQFhm7af6iOa5p6JZC+77U4TjPfJHJKmhy+JojuAe SR5df/VszCEZm2P11XDkxPtPN/iy8qDN3TYaWQTrqX/vjIXYCp0ExoKJa60VX9gqA9Ye xb3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764052350; x=1764657150; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=LPmid93vVx5WXKVpvZGHfLR1iexFd/wtP/8PCZcFv7U=; b=dGLhGZSRsCm0pT1j6FuOtFG+PMjBkuJfW5jB/AbbIqYHprMMHIzuMfIqtlj4CSpoQA fm3ItQug+02Z8Dz4HelFkVg0WT9QYmCafVENTZB1AT6HGxcRvP/mtyxjOtZdXHS1PPpP ofkO8f6kATZufFVaP5IGTzIdRciUr5cOGQrBXgQXcJGe//QtFZKdZGe12xPkEeO0tZMx 3bErmpWV8j43NMpxOt4uG9HMEsI38cPlqASkDjQp0Z1N5qCqlBUHKmcXTLgdtbaWKX0d c3J98y9ohB1TwvWTCFjP7EhAUD7gtOTO8uSNCKzzzxEn5yq6B9ZxOsrhBZ1LVI8o/+e7 x3xw== X-Gm-Message-State: AOJu0YzpCCox6fSIULauiEsC4eUeA8636wN90ti4WxIsN9rTpS0seryL F7sPQBGYciXLt6p9yxsgkvaJgiB//EDQo5rSuFlHqsfL6jM5lEceqjT2qJxdP9sK X-Gm-Gg: ASbGncvyj2CWPGy4j1ipQ57tPuC1OaHp3L3Wso/T/HhyPizC+LGDgygNPV6c/bRDVUo gqlujG/zQbky7zJz7qFSsH5HZzA/mDtAh3MUG6OykC87SCkDpncdMmdRTgCRuohnDL+pQB6w1SN 2qym+m/4GVV21unfgEKqbMCGTROITeKfaOBeI92sQYnVVkkDWoBwnY7bpmjaldgbdMyGQ/RcKYx G8aF/to5tNr+FbbCDZtQv+zqHDZAb6d+BnZZz2l123hddXs9l8zSNd+MLxsrzhVZ34ISVnvlQpp DZXTLSpB0+E5BwKHHENwJC42+3J5MM7QOChXiHUx80x5WmLtVhHNFnmYad+nJ93aANWO88HeACr 1nOiwIYn+QVug/u7qm11U/tGTw/0vMJSzh86iJEmREwbsDrew2LFsyDpzLkPrdSYgcfSXtKhrKL QhP0wOR5EeHWg/ X-Google-Smtp-Source: AGHT+IGuoDbcZr3RSab9H8RYzJ76vTNB+NbXc1HA7gxF/haRyOSam/ZqrRv+OAet+D+e3V5nB+mIPg== X-Received: by 2002:a17:902:ef07:b0:294:cc1d:e2b6 with SMTP id d9443c01a7336-29b6bf9c727mr159206525ad.59.1764052349890; Mon, 24 Nov 2025 22:32:29 -0800 (PST) Received: from localhost ([240b:4000:bb:1701:388a:b68c:56c7:adeb]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-29b5b273b06sm153667055ad.74.2025.11.24.22.32.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Nov 2025 22:32:29 -0800 (PST) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: Tejun Heo , ying chen , Lai Jiangshan , Lai Jiangshan Subject: [PATCH V4 4/4] workqueue: Process extra works in rescuer when there are no more to rescue Date: Tue, 25 Nov 2025 14:36:17 +0800 Message-Id: <20251125063617.671199-5-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20251125063617.671199-1-jiangshanlai@gmail.com> References: <20251125063617.671199-1-jiangshanlai@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lai Jiangshan Make the rescuer process more work on the last pwq when there are no more to rescue for the whole workqueue to help the regular workers in case it is a temporary memory pressure relief and to reduce relapse. Also throttle the number of extra works. Signed-off-by: Lai Jiangshan --- kernel/workqueue.c | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 7cec9755b4e1..12d7d6503482 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -3459,6 +3459,9 @@ static int worker_thread(void *__worker) * If @throttled and other PWQs are in mayday, requeue mayday for this PWQ * and let the rescuer handle other PWQs first. * If this is the only PWQ in mayday, process it regardless of @throttled. + * If this is the last PWQ leaving mayday, but the pool is in semi-mayday + * (idle workers exist while memory pressure is still on), process it unle= ss + * @throttled. */ static bool assign_rescuer_work(struct pool_workqueue *pwq, struct worker = *rescuer, bool throttled) { @@ -3475,10 +3478,38 @@ static bool assign_rescuer_work(struct pool_workque= ue *pwq, struct worker *rescu list_del_init(&cursor->entry); } =20 - /* need rescue? */ - if (!pwq->nr_active || !need_to_create_worker(pool)) + /* have work items to rescue? */ + if (!pwq->nr_active) return false; =20 + /* need rescue? */ + if (!need_to_create_worker(pool)) { + /* + * The pool has idle workers and doesn't need the rescuer, so it + * could simply return false here. + * + * However, the memory pressure might not be fully relieved. + * In PERCPU pool with concurrency enabled, having idle workers + * does not necessarily mean memory pressure is gone; it may + * simply mean regular workers have woken up, completed their + * work, and gone idle again due to concurrency limits. + * + * In this case, those working workers may later sleep again, + * the pool may run out of idle workers, and it will have to + * allocate new ones and wait for the timer to send mayday, + * causing unnecessary delay - especially if memory pressure + * was never resolved throughout. + * + * Do more work if memory pressure is still on to reduce + * relapse, using (pool->flags & POOL_MANAGER_ACTIVE), though + * not precisely, unless there are other PWQs needing help or + * this PWQ is throttled. + */ + if (!(pool->flags & POOL_MANAGER_ACTIVE) || + !list_empty(&pwq->wq->maydays) || throttled) + return false; + } + /* find the next work item to rescue */ list_for_each_entry_safe_from(work, n, &pool->worklist, entry) { if (get_work_pwq(work) !=3D pwq) --=20 2.19.1.6.gb485710b