From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 670061F427C for ; Fri, 7 Aug 2026 03:52:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074757; cv=none; b=eYUPluh8VcKz6g3LC4knVYVrZDFBBgym0vip9Ui9CrAs5rlpus8NIWw7JqGsBI4AODVbepqrXtcdoWvmpXPDeBBSPujvzwy8f8T9ceGeniU95tmrxpfWn5eRkELmuMqvcZMBaj4/BeVtmn+w23QaBTIt3C3fFnHj3pWxka4SUJE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074757; c=relaxed/simple; bh=bk3eLBZGyN9A3ZWCcIMLG47lxkE2OoQIRajzwrLZzCs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=T1ZZecIxU9d/ZzoN+gcE/0JDHRUnG8/l8pqSU7CgIanjpK4vqi/R43oJXSfDNb4HNDYG4s9brqt3QDSGcJkVhsoVEOC4QYQ+OC9at2UZHYij8GgzatLzAtqVWcb6HLfPFof6F0DrMdM9g2Uxosa9cxXZSr+bDc6HywPDJzDp4kY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HkTgcsLE; arc=none smtp.client-ip=209.85.210.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HkTgcsLE" Received: by mail-pf1-f199.google.com with SMTP id d2e1a72fcca58-8486ffba174so6730912b3a.1 for ; Thu, 06 Aug 2026 20:52:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074756; x=1786679556; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0HxBWbJx8Yu4nhKNYzoLJtzmsIzjapjobp3VL0mMJ0U=; b=HkTgcsLEGWwc7XBoGWKKp7FcCID4hNCPAQJcXkaXfDza8Iyl8nJ3DXjoU1jlNlIshT tCeZG/keIPyInO6cK8t+1MUUYGtAqxycXeW0OmXAnPNx6wqF5qSVuPX9H78/trMpw4QB axnqOrVh0Vv3HK6LoOOKRX61wlVM1WWxZIAKZggsyqRB2znLmFkcKMrX6SQsNTFk6Q5a a3+jdVH641Us/fI/8vvLbiPA9YbfLswtCux8vSQpO+MTySWK4rVsLsTYrxhahthaJcr/ V4gMVLP6RC9O/aDCDKHLzeuZjrcm1kRqFCKqCmWHVpAYUHXLguC0TtJGVVG8cvZzQ/e9 gyUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074756; x=1786679556; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0HxBWbJx8Yu4nhKNYzoLJtzmsIzjapjobp3VL0mMJ0U=; b=E7ixE0jtWD5p+sVOWv3OYt8jC9vyTu8iht/QKUL5MkY0uZQRB0Ap6hLAsgo5K07Q9y ptsUYnGdVDvp2e6b6IRAM/WZyjJ3P8QH/j6W+t33cTMQSvlF4uVB6FYLXGOk+I4B7GBU pA2TUi2QZoM/OkaO2wEM2I9RIqltqKfP9RCVHYqfPjVAMArNPpHrqeltv2v9vqB7G0iJ qL2UOeaCNDKMDy0ZhuKYjVGCDalpLgWHaQ98p57iAyHqEiLGr5RT34WEOOM17nfy9ecc wrk9CB1Sp7k0X+LaSGPP//Sxj1l0IffnxQPimQoZHJ2UTMWEijVXtNR9AOrZpBbmt/Ar BkLw== X-Gm-Message-State: AOJu0YztBlp73F4ez4GBpKplifrwj7TgjREcqKEQzi8+YYf16zevdsoM eXcDw5VIr27nHTbXAmfwWzj93uuNof3kv3VmF8IHdC66Jo0EVNKEni2RpcI+zBZrApDpzkt17i/ M9cXWxBkW9o+rGuL3bdsCZ0rbEaIQTNm1DwGKc7+theB/LdjyfmrRcbGjc/4AZmWgFKfmyXb/Ov 04jSAL7L/8/+H6I+b3GbiMoLJGt+1EIiUnQkjXDREupsgZECWo X-Received: from pfet15.prod.google.com ([2002:aa7:938f:0:b0:82f:d8c0:fef8]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2d8f:b0:848:48c6:480c with SMTP id d2e1a72fcca58-84f2e00e9ddmr21193745b3a.24.1786074755287; Thu, 06 Aug 2026 20:52:35 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:07 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-2-jstultz@google.com> Subject: [RESEND][PATCH v31 1/9] sched/deadline: Ignore proxy-exec sched_yield() From: John Stultz To: LKML Cc: Christian Loehle , Juri Lelli , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christian Loehle With proxy execution, rq->curr is the execution context while rq->donor is the donating context. rq->curr's sched_yield() is dispatched through the donor class so that proxy execution follows the effective scheduling context. For SCHED_DEADLINE, this is too strong. yield_task_dl() does not just ask for another task of equal priority to get to run, it marks the current DL entity as yielded and forces it to sleep until replenishment. These yield semantics are fundamentally different from FIFO/RR (where if no equal-priority tasks are runnable, no harm done, they get picked again immediately) or OTHER (also doesn't cause priority inversion), so do not mix these semantics by ignoring a sched_yield() on DL donors. Fixes: 127b90315ca0 ("sched/proxy: Yield the donor task") Acked-by: Juri Lelli Signed-off-by: Christian Loehle Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/deadline.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 0f858b98c9aa3..3e89b3abeb278 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2574,6 +2574,9 @@ static bool dequeue_task_dl(struct rq *rq, struct tas= k_struct *p, int flags) */ static void yield_task_dl(struct rq *rq) { + if (sched_proxy_exec() && rq->curr !=3D rq->donor) + return; + /* * We make the task go to sleep until its current deadline by * forcing its runtime to zero. This way, update_curr_dl() stops --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 923EC54781 for ; Fri, 7 Aug 2026 03:52:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074759; cv=none; b=t4H+aL/RRDFjwstr6uzCDD1oO9+qtnEWfa8gTTv5d+Zx99OW+MGHonfjTJDHB5nn99vdjAla3T0JzcFEbhNX8LSTYVZ+Wc3A27njpRniRG73s7TpDHFC34xYHlQekb9GVWViP1vahdAwO9swuuxr1wxAJCT1qDLGnZL5Vn0vXCI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074759; c=relaxed/simple; bh=BveIFHm8f1pxHIEtlh4aRe8r6xMPj1s9NARTeBSlw4c=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=r8YsyRWVZNFEks10iG7+vKgWuohOahuyuoJgY7L8oKBw3bJtlzW5fEXGBsYifKC83HxAhzCXp9/nsmfJVoLytcER7pjhA5tBTHp/hUOuKJYl75zoeSiL9bdyIkcvpnEPEaf7rHTlqbW8pVya8aezz3pstFGXoiwin2B6Sci2TCM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AYuGXELe; arc=none smtp.client-ip=209.85.210.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AYuGXELe" Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-8488ac68185so7154427b3a.2 for ; Thu, 06 Aug 2026 20:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074757; x=1786679557; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=nOytJ7VS6kPnWFg638bBnd067nfAO6608KAQO4VuI2U=; b=AYuGXELeqIHyCD/LnXjtvUNByltj5oMbFnZ7Jxz4E5ptR1IduKRZRCxHvZ/3u2tjQe 6W2D5yqvwE4yYA2ngcfesjT5OmUYSwIduESrcH1fA2n22ojGccYPLU23u+U056woBzge jsEW/hFxT5hmwuHho28m238YsPBkPp3hglHTDZWWVGXgW9mHIbBsvGltrRzCXS2j3dYW fet9SjXcq06N35V2Gg0qZDpq2MQOqO/CuXfs/bxIcgYf3dPxudXjgwJhC1co8YtC3lH4 VHtim8QOo5Zw8rbO85CRwYl3t9pv20nxlzOc+iTihoSZrfGzP6sYIV74w8cC9sIqK6iT q4Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074757; x=1786679557; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=nOytJ7VS6kPnWFg638bBnd067nfAO6608KAQO4VuI2U=; b=iucLz6nuvd9iIrHUUEZ3KCU94Yf6VKnfducdAX6hGcxm9DGxnIvvimH71KnMNJS3O5 DaMTgGrIditoeOHqiuJ3Z6AV8ImX0iYXl32Am5nw8p/lpEcIZDLe8oSDjTiXcAFDAeMw fIngxpvc/3ovzNFukkHqGPRgmDhjDyBW49v6xzEDYuwi5Lze72zKyCLlSkIu9U7An6du oqh+op0qvLA6qyZ2ZbvuJka/+e45b+tC3XUO9pZxIZZdrjJnOcZljCkmmcR9upPbVdLG pmUlB/fM00DoWRCig9DgrrCSBCQckW4kn2xa9Zpcl6x3VyVgvmn7xktxkxcwCxv4bLZ4 s/EQ== X-Gm-Message-State: AOJu0Yw1jNkGA3E6ni0+QWgzDc5yW7JI+IR4WuEW0FlKKrHv6qYlwk1G qQoWKXf4lb3xtIdGpusuEX5SiDMwr8wPgWodz7+5zl7VsBHBFU3v6Laye9tN+bRqQ2WimWd55Tf oZL54eiSpY6xC0I6NGLnNC0VxKoUrrIY7IcfNvvh+17Fin3aO1paxCE6PuMJB9RUQcR8IDKpSHq zE7OCEwOweMrOFd0jLiY7zYL1O3Yx66xBa9ZpgR9/n91d1CJev X-Received: from pfbeb25.prod.google.com ([2002:a05:6a00:4c99:b0:848:49f6:6f5c]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:298e:b0:845:dffa:3740 with SMTP id d2e1a72fcca58-84f2e0151aemr24279859b3a.4.1786074756345; Thu, 06 Aug 2026 20:52:36 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:08 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-3-jstultz@google.com> Subject: [RESEND][PATCH v31 2/9] sched/core: Don't steal a proxy-exec donor From: John Stultz To: LKML Cc: Vasily Gorbik , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vasily Gorbik try_steal_cookie() avoids stealing src->core_pick and src->curr before moving a task with the same cookie via move_queued_task_locked(). With proxy-exec, src->donor is the current scheduling context and may differ from src->curr. Stealing it migrates a task that the source rq still treats as current, leaving src's scheduler state for that task stale. For CFS this means cfs_rq->curr points at the stolen entity, and the next pick on the source rq hits the WARN_ON_ONCE in put_prev_entity(). Commit 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()") tweaked the fair class logic so that the donor task isn't migrated away while we're running the proxy. Do it similarly for try_steal_cookie() and skip src->donor as well. Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_prox= y_task()") Signed-off-by: Vasily Gorbik Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f61..e7074ba54a91f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6474,7 +6474,7 @@ static bool try_steal_cookie(int this, int that) return false; =20 do { - if (p =3D=3D src->core_pick || p =3D=3D src->curr) + if (p =3D=3D src->core_pick || p =3D=3D src->curr || p =3D=3D src->donor) goto next; =20 if (!is_cpu_allowed(p, this)) --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CD0118DB35 for ; Fri, 7 Aug 2026 03:52:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074759; cv=none; b=bN0BNWwy4TTzBV+1+H7jTtIoXXgR7dpMyaqi+9hezEINW5Kmz8xI/WZV1sFpnTH94j81B1Wy3JscStGVoIHCkQih5IZjbD2j+4T2kATedXdsBiY34aNcv2cutK+K0zbql/Vc10+TTx9goxwk1aOXTB3u0sBm53iDYaNvwmrknWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074759; c=relaxed/simple; bh=uj04ijSYqu89EY6ysFzpr6q1PmxTWc2u9ZJuvw6Q9Fw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DbZs/bKv+SdJ18knr9F3KWZQiSKrwdznK0F0yqnjaE1AfF3yH/ijrJ8ffowzYk2eN8oxcU4dE064sU96M5Q/Vdu5dYOT4SHHuGTXltrpW89og/VRHhvnSDW2vQx1Ord/pkxtKpkfg+DWRlYmjTifr+GaKrE3cpHUDyfTbCQIyGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cQvAU2FJ; arc=none smtp.client-ip=209.85.210.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cQvAU2FJ" Received: by mail-pf1-f199.google.com with SMTP id d2e1a72fcca58-84a251c2e3eso2059841b3a.1 for ; Thu, 06 Aug 2026 20:52:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074758; x=1786679558; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=2nWwZDm3oo2dBB7Ku1eqiStfIQeQxnTlmjem0Y5982A=; b=cQvAU2FJArlCiC77myaZCIqJc+PB5QoP+wYZ9qEEPuPr2yJ6mCCs2SQqCA+VP8eWte zYf19728GFzml4xEzSo8xTxv8Cu6o0rbkRw466I2ETLQIPKgJ5Y7Bp1p9mKOd9HGq1Va uAqRqo5iI+S+S4ASa1u4qU1CcquOSlCtYbB7Pr+SeOsD6mfdcgySgHgm4H8UpoOImxnQ 8Ps5dVT6w8x+WeucxyQA4be66aAklFNhJMUdekcd2R/7K/Go6roHnlY3gF34BCpmcQEW 9cZI9k6oEzWkan0HQo1yIF5arv5n+FtHNSauvi6e/FMcg9a+07vWyW2yAWij+eeXfOLf 1UPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074758; x=1786679558; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=2nWwZDm3oo2dBB7Ku1eqiStfIQeQxnTlmjem0Y5982A=; b=IR/HOoVwomFa57z5bDn6lvoWLXVG7+gwJ1avHQZs0mvwS97p6a2nkIu4DHIfrLu+2O xqAY8KPeh8zznAPk213i1vO2LdNI3B6ORTluU9I375FCN9nJ4qp1hAFb8pT6Emyualkd ELNrBrYU4omlIKZDj8+dKf6lyVHK8Wk78s0XuRl4rBi9gJ5Kf7fEsNp3UliUHVt4nFK3 WQkJn/s6dIpP4jPFU0yPlJ18XaBdc7Iwr1VHS/5FnrLGPeCdnU0W0PA9cOa9fCzCRYdV Myd6Htur/nyMCDXfMFJd12RbfxdRt2PzdjXxuWWYFpH+IGUh8jEm7TroJq1J6oZYaRgP 4Erg== X-Gm-Message-State: AOJu0YyyywSWkGBq025NB/bLbhEhdq3zSbggZjil6DHv/hx9wVV/NxP6 pa9KG5olu1TWSUlRE+H/nqPWMKp2j7DXlFZr8v9Zjnf/Z62eoEMhDsixpV8QUH3ORqWNzzEh9wM /cka7bTRFbYI+vY+pXkmB/ITxa9Ijxvru40pKXMVkNGnyPSWMVg1m1ZqA+3xVsb323b4DrqJjFB 4BeGMdlt/8TyglXoZ9FSeFvf3wxwuVES1Jc+QSZBIHmZUDGpIe X-Received: from pfbko19.prod.google.com ([2002:a05:6a00:4613:b0:847:926b:dc17]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:3686:b0:847:950b:b71a with SMTP id d2e1a72fcca58-84f47fe6caemr11247546b3a.16.1786074757249; Thu, 06 Aug 2026 20:52:37 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:09 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-4-jstultz@google.com> Subject: [RESEND][PATCH v31 3/9] sched/core: Avoid migrating blocked_on tasks From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Its not useful to migrate blocked_on tasks from other runqueues as the proxy logic will just migrate it back to the owner's rq. So skip blocked_on tasks here. Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e7074ba54a91f..8e661b5f133d7 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6477,6 +6477,9 @@ static bool try_steal_cookie(int this, int that) if (p =3D=3D src->core_pick || p =3D=3D src->curr || p =3D=3D src->donor) goto next; =20 + if (task_is_blocked(p)) + goto next; + if (!is_cpu_allowed(p, this)) goto next; =20 --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6116E3A6B8E for ; Fri, 7 Aug 2026 03:52:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074760; cv=none; b=oD2VuW5LsphZ07MuQNj8hlUt64Ffdj2epXDfJDXgHJo96YurwDAmGrrpotguG+R9qPVkxJ56BZMrhUI/ErZpY+JPneT/mCo+zZUxlwyaxKoX8RIzFMlVEEBhAYTQ129tiQbLDA9I3PTjMDJ1J/rveEZ+YbS/nuYYXDgCdk/XCjY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074760; c=relaxed/simple; bh=DOTE5M8hb/JpZ0fhN/N8sxeueCXjN5uVDpxbKQF6ph8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cMOTB46QKHQQBK5oY4vQeQoab7opI/Z7CEi0Dq6SCI1ud8R6TeG/wxyCujALzLV5I0PX08FgtXdpj+hr9HfmwuGx73PBFxhUCHo0pAa1wl6aLMz7oGtmD84Gsam/tw4ismjQWKAw6+y8xZPwLp0w0Tu6ewLhzhUb267NNr0X828= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WZU3nYue; arc=none smtp.client-ip=209.85.215.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WZU3nYue" Received: by mail-pg1-f199.google.com with SMTP id 41be03b00d2f7-caf5fa127d4so4870266a12.0 for ; Thu, 06 Aug 2026 20:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074759; x=1786679559; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=7bVqdXkuV00QULLPySg1D37iePHKV2Z9Jhad4OgmOq0=; b=WZU3nYueknho7pKNvdlMK73Fl32WKXfkWlYCRwcpwNKKG6g+A3ic6xkFqEAyiMY3FK cZgTDjFQO7x83pWCRDE0ZLw+/UYUPNlHL0+cJ8WOWWa+rnMm2HOvUb+5iIb4Mv/UBSvK g4dyygEvAZmnciiTDR2cGkKkha6Wzxna76pvBwP7R+zzjkJNPpk7+gHn4BljzDKSCqXm CyHPF+gvogWkKo8WJINdIRYagZzfIgegXBerO6ub6MDA3T9kNQ+APaEFolME+ptD1LL2 M016AUC+6P6lHp9ajAqZIHanHSX4quv1xkOm4GSJ0wHVMHULSz22MaGjFL+ikCI0qXVA QHvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074759; x=1786679559; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7bVqdXkuV00QULLPySg1D37iePHKV2Z9Jhad4OgmOq0=; b=OPCuwcCxWPCzh5DhIeI6RRE7MfKcfiPmQCDHNDrI7pAZIRUR9VKykmvk93tL8tEJfp A40QAukJ2v7jDWdejcFLyLHIQvuYZPIPrmxUOMrRtats3VzxQERJUEuLhjDsetWL94Sn DbZJEaeGEoRWgdXkn0RlcotmzNGpQ1Qek2aEdgLoVdTVr61bzv0JixVYXsusqx2IFZqi y71MDtcoiQQkuejKWghP+UJKrGISHFOQKT3Hvsjt2eSAhap4twDUWJidfXKojxZ3/XqI 2ZMy7ySyw2nGW+ZKXHZOF+OuBlj468RhfqyTt8bGj7J7FPyoreB7Tq+IPL48FpqaTXTv erGg== X-Gm-Message-State: AOJu0YzXTaG9KX5Bfom76cSs8UhLO7UkVzQT+3bGuUxz9143XTC6uHP4 +rPlFy5r1zUTQY0JnMniCBTf02BEXgajQDCPxx26C3q+Pv45vLFAu5bSc0bGfgHsfrs82GxdZeJ VkAOBQulyBsUedIS2RlfzCH7J2ARxq+GW4dnMa+Y4lrdy/6MD82YqU9rkd4gjxFfKQ6Ub6MvomP ct7raGb1Bg3LOAAFdrJ4D/MjiSyDuStU4JjIxcMvVQkUuVNkLR X-Received: from pfblc11.prod.google.com ([2002:a05:6a00:4f4b:b0:84a:1bf6:cb4a]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:847:b0:84e:5e4:b9a2 with SMTP id d2e1a72fcca58-84f5e1136d6mr683972b3a.36.1786074758272; Thu, 06 Aug 2026 20:52:38 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:10 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-5-jstultz@google.com> Subject: [RESEND][PATCH v31 4/9] sched/core: Don't proxy-exec unmatched cookie lock owners From: John Stultz To: LKML Cc: Vasily Gorbik , K Prateek Nayak , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vasily Gorbik Core scheduling chooses a core-wide cookie before __schedule() installs the next task. With proxy-exec enabled, that task becomes the donor/scheduling context, and find_proxy_task() may then replace the execution context with the runnable mutex owner. If its cookie differs from the selected core cookie, running it would bypass core scheduling's cookie selection. When the final mutex owner found by find_proxy_task() does not match the selected core cookie, stop proxying the donor. If the current execution context is already in the blocked chain, fall back to idle like the existing proxy-exec retry paths do. Otherwise deactivate the donor and let __schedule() pick again. The mutex owner can be picked later under its own cookie. Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_prox= y_task()") Reported-by: K Prateek Nayak Link: https://lore.kernel.org/lkml/10282ce9-f4ae-498f-9b57-f4e1e61fffbc@amd= .com/ Signed-off-by: Vasily Gorbik [jstultz: Added tweak to ensure we deactivate donor, not runnable owner] Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 8e661b5f133d7..36e1db67a8374 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7004,6 +7004,14 @@ find_proxy_task(struct rq *rq, struct task_struct *d= onor, struct rq_flags *rf) owner->blocked_donor =3D p; } WARN_ON_ONCE(owner && !owner->on_rq); + + if (owner && !sched_cpu_cookie_match(rq, owner)) { + if (curr_in_chain) + return proxy_resched_idle(rq); + p =3D donor; /* Deactivate the donor, not the runnable owner */ + clear_task_blocked_on(p, NULL); + goto deactivate; + } return owner; =20 deactivate: --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DC2C34B1A5 for ; Fri, 7 Aug 2026 03:52:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074761; cv=none; b=cAp+609/dciOBN9wkdQSiAmRR4GSkSTgxiYcBVY//wRhWWFOaLuI+cRW9+Jg4T9R2yyBu85CVybdGhFI9vz8AunKudpJqnX0g/dCTla37EZLYRWk/OEbBgMqttTCPwAD8SXZmb8dcG6heuiXDLuGWR2MT5udyfmUq/LZkb5VqC0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074761; c=relaxed/simple; bh=5yvP+idFJpuYqgACMllwQRMMhFNd/wSLjHeOdVmOm4c=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OMIg/zuEHTvan7PREb4rmdRnxVFG5IOHtPxmlTYAF/znm7axVQTLxcl0ctopTEssLnUCwE8euVl2P4YS2prmRGuwuUN2rKx7ULIAEy50SJ0ClbCYLkcTmuRpbVqk5eFGIgYVtcoBG89pPZ4QSdePLCR2CV/ZwQaMmpWAjj4FnN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=V7kcZ8lC; arc=none smtp.client-ip=209.85.216.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="V7kcZ8lC" Received: by mail-pj1-f69.google.com with SMTP id 98e67ed59e1d1-38f5ac7416eso3578251a91.3 for ; Thu, 06 Aug 2026 20:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074759; x=1786679559; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0h3BDRkU0A8iDDcvEIaMLz4VqHCGOefnCdxoKfqo/7o=; b=V7kcZ8lCVuZw9XoR45ulUzOxjWBXk0MPk5+lIKl+Xn3gr//O1AK8yy4Fcpx0alB64n L3xHItm1fAC9RGhhs0BDvUN5IAQ72RlBI+lrLSHZEVicnJEW2tXIBHMUeUs+GFWXbY1W fTMgSgAGXNcwBFyUoAY1O5w62Mv5u43gxBk3AUVfm6NTnbjN2wqYsn3knLJQxJ8IOapd 2wK7JljXFWsFzUYd+qCkKaEDH851TG6Qr+Udy7EwGAYu/I+ZosrgT5bTq7bfujIDL3LH avScHpjYUca8yOKJa8dGF8/YF4dX7zSSl5XyQqs7uUX3SbUHvJOMNt8ly9tLZcfgfCBh QJsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074759; x=1786679559; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0h3BDRkU0A8iDDcvEIaMLz4VqHCGOefnCdxoKfqo/7o=; b=PmaoL/mcvR2/eg+7Zd2OHb6sKpaosPTUTESgKjssx96zcbr1fFlrrKiP7UxJ9k80R1 vKne10i4BeQInq0S+JbCuQgMpF9SGteDbXX/4lMkaK4Ogp2PtngMaAXZIPv5jAAecyl9 fFEUQpyNLbQiuC/38cCCZy3xf30QeHZMdWmf4ylQIIpPI+RBuo8hCmGJtXwJ+rUI0C+w uhcFRLPi11ObPRlT9ARSFTDqrZSBba3yiza/bGmtt2e0l7pSO6raubh0ilDMg2XNwEPk j0aF8GZwcfqqyQwTJ5yUDbBN2Yj/QWHpAfsOkD+9h6BsX3oBAB9PNpjIFz/6YuExRCNi vfqw== X-Gm-Message-State: AOJu0YzvFVfbOYVjuWn9Ha4wp2LsHdA0lkYl83cbodhud8JddXTOmXiS UdgmO1kV/JSSyV0o+LhTCXTVCg6xcTTHVImFTXNaYmxBGY6dxyqbTYY0aB9EWxU908wo3hKIm0g I5XIXrtQ4bMr0TbAtvX5DlxUoNfKh1evvdvH97GIgYmWzTMEdK/1Nl0hrooocfoKjeTBZUaHuvb cK//qAGrQaT/ImXCij5gz+FgOZJDImheqn2qahHdgbjEZ1hdXa X-Received: from pjbgc15.prod.google.com ([2002:a17:90b:310f:b0:38e:1db:751c]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:dfce:b0:38e:c232:9d3f with SMTP id 98e67ed59e1d1-3903c547f80mr21119534a91.5.1786074759186; Thu, 06 Aug 2026 20:52:39 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:11 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-6-jstultz@google.com> Subject: [RESEND][PATCH v31 5/9] sched: Switch rq->next_class in proxy_reset_donor() From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , Vasily Gorbik , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similar to commit 04f80f8b12a0 ("sched: Switch rq->next_class on proxy_resched_idle()"), we should also set the rq->next_class when we call proxy_reset_donor(). Fixes: f13beb010e4a ("sched: Have try_to_wake_up() handle return-migration = for PROXY_WAKING case") Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: Vasily Gorbik Cc: kernel-team@android.com --- kernel/sched/core.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 36e1db67a8374..564762ed36f2d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3750,6 +3750,7 @@ static inline void proxy_reset_donor(struct rq *rq) WARN_ON_ONCE(rq->donor =3D=3D rq->curr); =20 put_prev_set_next_task(rq, rq->donor, rq->curr); + rq->next_class =3D rq->curr->sched_class; rq_set_donor(rq, rq->curr); zap_balance_callbacks(rq); resched_curr(rq); --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECD423AAF59 for ; Fri, 7 Aug 2026 03:52:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074762; cv=none; b=lR6kQGU135XUK2u/ZNHDtL9v/23vDV6mpTPYj6+ojQMlHw4RQ9WaRWqHW0Abk841wDfStFKvF5q0tvU/t7W/OFWAIJrd28LEy29duKMcSvM4n6spy8/ICZ76bjIN8ALMKdTVhfm92CAsRFBgDTIiVOuzEUAlfB5a7G6yE+vZMLk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074762; c=relaxed/simple; bh=Mf9VzjKnGumLQ0UESxtM5HUID11ySTvu2WV0vwKzJyk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rvzx0MNXMjrtYuR2rw6cOafwi8A4sn/151jzKK8oQLbvDtCTo4DeLXaAW0Om+pQU0BCaMTYeRtbE3j0gWWP0+iCJ53tlUEAifha/dzg3nWr3oz+rq3/TFHDP8UnO2tQhpukfvEFzPRZ7E4so14xRLjINr2jvdWl1MWq9i6jwYQ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vw8Vsb7L; arc=none smtp.client-ip=209.85.216.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vw8Vsb7L" Received: by mail-pj1-f69.google.com with SMTP id 98e67ed59e1d1-38e1118e4abso4321473a91.0 for ; Thu, 06 Aug 2026 20:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074760; x=1786679560; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=v74fkH+T9qp22ctkQvaUkuJE0FnZiEKV1z5gp8lIp44=; b=vw8Vsb7LeIIl5lnz4VrIP8bsEVJKbOzTMp+mH2171gftpu2dhBYIC7ITc0BlEL6ppq GRmkurSqVln08HQqgF65d9gtlv2BW3bf7bwfXdLik1IrJCNOBR/5ZE3mQjP0LgGlMnnt xQ461de28FfS4KAIpIg3T9KcU8IYAEdiBYScfchG6H8reb8HQOed2Y34UMXyRwrqEKKy PFP1aYE26KgH8HqzMrrQbhwitn89ahEAKo6chnJaFnwweMftGrGFZz5kAXEs/yCNFBgH KHzZVQiEP0jLPpPX0LRLoIDTGdze4qujsVdMafqsug8ghN5o/++L4pmgT20bIA4o1QoA Zj3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074760; x=1786679560; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=v74fkH+T9qp22ctkQvaUkuJE0FnZiEKV1z5gp8lIp44=; b=InPk+UYi2orBkoEHI7yOKX8UmO+piicF6hnoL9P7I8Vo+ZeTqTfd0Tcfs6dQqhB6hu x3EAYNvvNIILRy/bOK034ZdnyxqCJ7DSfreRxEeEw58kU76isSM0VpffzszaEkpG854T bLYfdtqu/JyRPDK97uAtSxfBPQ82U10448xKzPaEDojbwJSxiOR4LHV4i79IOg9ffokO 6wpZfkZ4ZFPz6NE8XlzHgUmDdXrCVVW2Lj4t7eTz0BBlG42NhiOgV55lrO9T+eQnCZZg saaZ80/tF85CB5LoX2Yjb/LUDcrcuaiJgLPd9miSLfJTWnCMztQ664fdps1YGWtPEPw+ mYgg== X-Gm-Message-State: AOJu0Yy6c6obk9HHED3WQOmZwRW3XHzL8dgth+YCICXaT7D5tvr/r0HZ 5fYmH+25zCb1Uxn90N3hX+0XYMNvYHF/YIUfZLb6uRW/2AxcRl2syDFM64gtAwe0q/ek+tpagpx wxJzXEvI5Hb8D7fU8DralcRzFK7aHV0tu7LJQ6o4R+xYXedimcU4kVTbvc7LKs50OlGk7ovHxdH JCyy+Zl2wCK0D9QRaVxOASEp1x3+XESnrwKZZMDhyscFv9feXx X-Received: from pjbhk8.prod.google.com ([2002:a17:90b:2248:b0:381:27d7:fc1]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:57c3:b0:38e:bfe:81e9 with SMTP id 98e67ed59e1d1-3903c5589e0mr21693806a91.1.1786074760059; Thu, 06 Aug 2026 20:52:40 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:12 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-7-jstultz@google.com> Subject: [RESEND][PATCH v31 6/9] sched: Break out core of attach_tasks() helper into sched.h From: John Stultz To: LKML Cc: John Stultz , K Prateek Nayak , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pull the core of attach_tasks() out into sched.h so it can be used more generically. Suggested-by: K Prateek Nayak Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/fair.c | 16 +--------------- kernel/sched/sched.h | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee13..6d2fa0cd77779 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11035,21 +11035,7 @@ static int detach_tasks(struct lb_env *env) */ static void attach_tasks(struct lb_env *env) { - struct list_head *tasks =3D &env->tasks; - struct task_struct *p; - struct rq_flags rf; - - rq_lock(env->dst_rq, &rf); - update_rq_clock(env->dst_rq); - - while (!list_empty(tasks)) { - p =3D list_first_entry(tasks, struct task_struct, se.group_node); - list_del_init(&p->se.group_node); - - attach_task(env->dst_rq, p); - } - - rq_unlock(env->dst_rq, &rf); + __attach_tasks(env->dst_rq, &env->tasks); } =20 #ifdef CONFIG_NO_HZ_COMMON diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56acf502ba260..56d9c09c485e6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3100,6 +3100,25 @@ static inline void attach_one_task(struct rq *rq, st= ruct task_struct *p) attach_task(rq, p); } =20 +/* + * __attach_tasks() - attaches a list of tasks (using se.group_node) to + * the new rq + */ +static inline void __attach_tasks(struct rq *rq, struct list_head *tasks) +{ + guard(rq_lock)(rq); + update_rq_clock(rq); + + while (!list_empty(tasks)) { + struct task_struct *p; + + p =3D list_first_entry(tasks, struct task_struct, se.group_node); + list_del_init(&p->se.group_node); + + attach_task(rq, p); + } +} + #ifdef CONFIG_PREEMPT_RT # define SCHED_NR_MIGRATE_BREAK 8 #else --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B438C3ABD82 for ; Fri, 7 Aug 2026 03:52:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074763; cv=none; b=D1hfZXK65EsUOHYjVbz7XmjrAQbUrezozuVAiuFLknW7VPB6YX9eTN+bCJQpps07MClXy5ccbEnigaGK9wxO7fPq1S+q28pJZOuLDO9JEQTU1I9NzJCr19iEMcfRzdxVHwjuNC0Xg5+/L+jrqMvDOKq7xmLLyU+FcKIAMFpASRo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074763; c=relaxed/simple; bh=CSOVsS8UWQGlIziI7NKPdi5vrcIQ+oL85iiVMBgLlvA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qgwatiHZU947zpwfz+dfv7RG0l0cWnQYS5b0i4SXT1bd9fp9+HBxMLhLmTvjmR/RLcwMJhdOP/vFCZX9pBFtVQL43dX4Ks/hIJnp18sOOSZv5X+CavUerXJYWPV8tt12z+SkxQsiWlIcevL3qjVFtfQBtOyZv3HLt2TzbsshFys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MI+BtNTb; arc=none smtp.client-ip=209.85.216.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MI+BtNTb" Received: by mail-pj1-f69.google.com with SMTP id 98e67ed59e1d1-38e54b6556aso3260130a91.2 for ; Thu, 06 Aug 2026 20:52:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074761; x=1786679561; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=v3lhC9D0tWuMtdDZrYkdJhxklzGgK946m8G3sbfQfYM=; b=MI+BtNTbxAA1pGe3jabswWRt5x1sziNCvTrhlWYPuqVg0NGC86t14Bva7x3fVrlJnN v+Sqkoh9lvh/lVUwJ4WIw/NTiwLU4in4tf3+ffAXehI4H1p/4+qeV2TgmivFpYZ3zn9c KS81Ond8cu4RKE4hAoocG9JtKOxMwxVhA2VDuRIgAAhzAB3W3FRy2/3SiFkANdZTzcZl 5FFLdDyVx14+8bdjHdfnObIenKgUriWNQwKMdW0bYJRxDw9qDNmPQKaDTyn2gog5ANfQ dfD8GOg24YrImddvdS/z6URP9IQcdGDlzygJMItOdKXbxqLxSfOAA9ivbJ/ZJScPZbby Pz+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074761; x=1786679561; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=v3lhC9D0tWuMtdDZrYkdJhxklzGgK946m8G3sbfQfYM=; b=KqcHIJVEbdp8kg6x5BTNbotc60oGEW63/Kr5kJxYFW0FYBzRiuFoNBMvVa33fjXp+3 EF4DYFE1aZwTpI1Cbo68oqtBYPiEltrxOpK23W62XScmedLuCecwNHk2aOByHfbIBiRc K5kGSGt1eckkDK2pQc0JeWPfmWLu44JO1bVohILq+XEHziEGfsChuf5M6KCe7gMRCgRH aTrXg9Zpw1IMZS+AYXSmLLcdhLiBKYOrTi6YaHe8x0C+kx58FqJDSd7/lQWAoruFrwi1 FRSxpPLZucifWql7kJEKJb+Lbl+QxoxPWO7OZXAU08YhDNG2F8ZN33OkwysAdkuJtmNf mrRw== X-Gm-Message-State: AOJu0YyYup7OE0rWQ6RU2rXB4UEvNWj1ahurtfFXbpQCPzRmmiDfhDjl 4Z2g+tPYNBtuWtXI2WIB5s3FJY4iptXKSd4PohPlfL+xl2l9p6DlsheygVx8uPX5LwVGi422xJW +z75JY7wlvOI+Qc2X98WQcpmT1NVgXEMKBkKnOoO7wxhO+viYcNnx7jjK7MTUxHk/4ppRzezwbO kZanXGoYSHDQt7CTnVWInhOKVWxe+rLAtKi7gi/cfSv95SJs6u X-Received: from pjbnd7.prod.google.com ([2002:a17:90b:4cc7:b0:38e:b8e0:d97d]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4e8d:b0:381:cef1:11ac with SMTP id 98e67ed59e1d1-3903c58ed40mr20127680a91.10.1786074760894; Thu, 06 Aug 2026 20:52:40 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:13 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-8-jstultz@google.com> Subject: [RESEND][PATCH v31 7/9] sched: Migrate whole chain in proxy_migrate_task() From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of migrating one task each time through find_proxy_task(), we can walk up the blocked_donor ptrs and migrate the entire current chain in one go. This was broken out of earlier patches and held back while the series was being stabilized, but I wanted to re-introduce it. Signed-off-by: John Stultz --- v12: * Earlier this was re-using blocked_node, but I hit a race with activating blocked entities, and to avoid it introduced a new migration_node listhead v18: * Add init_task initialization of migration_node as suggested by Suleiman v22: * Move migration_node under CONFIG_SCHED_PROXY_EXEC as suggested by K Prateek v25: * Use se.group_node instead of adding migration_node, as suggsested by K Prateek * Integrated attach_tasks() cleanups suggested by K Prateek Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 564762ed36f2d..1bf60d78c9208 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6819,9 +6819,9 @@ static void proxy_migrate_task(struct rq *rq, struct = rq_flags *rf, __must_hold(__rq_lockp(rq)) { struct rq *target_rq =3D cpu_rq(target_cpu); + LIST_HEAD(migrate_list); =20 lockdep_assert_rq_held(rq); - WARN_ON(p =3D=3D rq->curr); /* * Since we are migrating a blocked donor, it could be rq->donor, * and we want to make sure there aren't any references from this @@ -6834,13 +6834,20 @@ static void proxy_migrate_task(struct rq *rq, struc= t rq_flags *rf, * before we release the lock. */ proxy_resched_idle(rq); - - deactivate_task(rq, p, DEQUEUE_NOCLOCK); - proxy_set_task_cpu(p, target_cpu); - + for (; p; p =3D p->blocked_donor) { + WARN_ON(p =3D=3D rq->curr); + deactivate_task(rq, p, DEQUEUE_NOCLOCK); + proxy_set_task_cpu(p, target_cpu); + /* + * We can re-use se.group_node to migrate the thing, + * because @p is deactivated (won't be balanced) and + * we hold the rq_lock. + */ + list_add(&p->se.group_node, &migrate_list); + } proxy_release_rq_lock(rq, rf); =20 - attach_one_task(target_rq, p); + __attach_tasks(target_rq, &migrate_list); =20 proxy_reacquire_rq_lock(rq, rf); } --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1EE73B0580 for ; Fri, 7 Aug 2026 03:52:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074765; cv=none; b=Nn4eZE5P94CYmXbT61/3FTjwOPTjf82SQF4cHC53DCSmrG8ArrWn4O8Db5QgPAr1dcfhB/bJ8V9RKHPdGHzhpof7bJp/UzfIwtN5vzifYBe1IA/Jhbt2IHhLZnvPLGV8B8j5BapPUV9z6ym4XA51Sw8DRsiUI2P5vAQIruXTHcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074765; c=relaxed/simple; bh=DrxmwGLT3LaYqeav3iYEvmBV3uA0UpKO1ue4s78ljEo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=m1XgThDd/xuvDP1LdsJF/wSHo//BgPQlvwCix81k6f4F+fIhjt+Yxr+0d3bPRP/kbxrG9Ll3tkXDKWUy0nSlZMlfBOezN1Q1K/0g1W5ulG0hv2gld0vOtNs3eWq9rExLBXb7I6GM1SXgQXKJASG7MZsG/HIo1Ip0DrOCPCPOMMc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PANxT5hk; arc=none smtp.client-ip=209.85.210.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PANxT5hk" Received: by mail-pf1-f199.google.com with SMTP id d2e1a72fcca58-8486ffba174so6731165b3a.1 for ; Thu, 06 Aug 2026 20:52:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074762; x=1786679562; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=81WrUODFT8LXaVnxK7UXUXgW6pKnmShsuXkajQL0M7Y=; b=PANxT5hkhzORaO6opC6mftPzoYJzwxPeXhaS9vV5h6cuia22mIAj+5vpapyImkOs6D 2XFSxufrIAGdq+Z5vyCa7EePz4qNbmpVMV+hBS2XhFpLBO9AHafc/sHVO+ppvvfCOT6C yeeLbagxVwRMylAvlcwv2DfnS+BmKvQcgxa32Gs5Qds+8HXWlaPGLwR5r6IyyS31VwuI 1taPsGHR1toPHkD6WVRjSC3g8450XytZdutXKd0VN8MP4TPJy2zlQoY+p8yvbf0LQaCb 1aDAZ7k4toamKNL80Rx9kGROHuh67jmVw5oVo1KhdsNeakycwZDJVqxPDxQEBdJI2oYT MyTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074762; x=1786679562; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=81WrUODFT8LXaVnxK7UXUXgW6pKnmShsuXkajQL0M7Y=; b=Yj+IM5Qh80lUKBw/4bfnkDTMVNm6l/p0GaRCV3M76+v91xQknbh6RTMDZem3JJZ1cl GO8Ia9ev3597Samt/WlqlAKridE6rRsKHTXxmn7km9CZBA6t5CSi9ew3ox7k7gTaF3y4 f4Kn07OY6FP08CDrwZUUOac+wsd8Lkk00S1q69weeZbua99tONpKX6QFDrWy4NlD1StV NyuiYgPPZdaPw1abDHb1scQV74KcMA3H5CeWHPiR7kqYzxhf7DkayDI0Na3UAyS10RPx f/aF6e0HfHViFU1kYrJcw55tR29GZoe5OBOFzmOAzzVpUbTQVjiPHgxM/W81oLyxYNRp T05Q== X-Gm-Message-State: AOJu0YxqQgJAV3SmU8kkZbXukmheB9cMBfl5XDaG7hDXtyXEWsBULW+r AE415KaXf9Av+fG4ulPV3aqNi9uvSjs0de1aHkfp8EJlwz5msHa0Muhj4JoyE4d/ZWFm+PSK54p cpoGEv0nUaPlgTISojh3q9zzdWRMs4/LPGn79az5K3SfOw+h+WMcWDmf6ulIXrqyh+zCNL2mltx OdA/eZ+PxTlGrs8bE665MJf/2Dc3jhZyFRGmh7qRYj+VjJv+Bh X-Received: from pfaw13.prod.google.com ([2002:a05:6a00:ab8d:b0:848:4460:aed3]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:17aa:b0:84e:e741:174f with SMTP id d2e1a72fcca58-84f2dfc8a36mr19358848b3a.7.1786074761731; Thu, 06 Aug 2026 20:52:41 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:14 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-9-jstultz@google.com> Subject: [RESEND][PATCH v31 8/9] sched: Add deactivated (sleeping) owner handling to find_proxy_task() From: John Stultz To: LKML Cc: Peter Zijlstra , Juri Lelli , Valentin Schneider , "Connor O'Brien" , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra If the blocked_on chain resolves to a sleeping owner, deactivate the donor task, and enqueue it on the sleeping owner task. Then re-activate it later when the owner is woken up. NOTE: This has been particularly challenging to get working properly, and some of the locking is particularly awkward. I'd very much appreciate review and feedback for ways to simplify this. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Juri Lelli Signed-off-by: Valentin Schneider Signed-off-by: Connor O'Brien [jstultz: This was broken out from the larger proxy() patch] Signed-off-by: John Stultz --- v5: * Split out from larger proxy patch v6: * Major rework, replacing the single list head per task with per-task list head and nodes, creating a tree structure so we only wake up descendants of the task woken. * Reworked the locking to take the task->pi_lock, so we can avoid mid-chain wakeup races from try_to_wake_up() called by the ww_mutex logic. v7: * Drop unnecessary __nested lock annotation, as we already drop the lock prior. * Add comments on #else & #endif lines, and clearer function names, and commit message tweaks as suggested by Metin Kaya * Move activate_blocked_entities() call from ttwu_queue to try_to_wake_up() to simplify locking. Thanks to questions from Metin Kaya * Fix irqsave/irqrestore usage now we call this outside where the pi_lock is held * Fix activate_blocked_entitites not preserving wake_cpu * Fix for UP builds v8: * Minor checkpatch fixup * Drop proxy_deactivate and cleanups suggested by Metin v9: * Fix bug causing possibly uninitialized cpu value to be used with activate_blocked_entities() * Improved comment around preserving wake_cpu suggested by Metin * Add additional lockdep asserts, suggested by Metin * Tweaked placement of lockdep assert, suggested by Metin * Fixed comment referring to structure entry name * Fix to call proxy_resched_idle() _prior_ to calling proxy_enqueue_on_owner() where we deactivate the task, this avoids stale references to rq_selected() when the task may have been migrated to another rq. * Fix to remove the blocked_head list at the start of activate_blocked_entities() so we only do a finite amount of work, avoiding a potential livelock of two cpus removing and adding tasks to the list at the same time if the owner went back to sleep while blocked entities were being woken. v11: * Big rework to get rid of recursion. Had to add another list item to the task_stuct to do this as we are in atomic context and cannot allocate memory while activating blocked entities. Will need to watch carefully for bugs, as switching to a list_head in the task_struct instead of a pointer on the stack opens up the potential for races on the shared state, but I think I've got the locking sorted. * Moved proxy_set_task_cpu helper to earlier in the series * Minor rework for try_to_deactivate_task changes * Minor variable name cleanups suggested by Metin v13: * Switch to use donor from next for proxy_enqueue_on_owner * Switch to using block_task instead of deactivate_task v14: * Ensure we call block_task() last in proxy_enqueue_on_owner and not touch it again to avoid races where it might be activated on another cpu * Make sure we activate blocked_entities when we exit from ttwu * Fix to enqueue the last task in the chain (p) on the blocked owner instead of donor, so that we preserve the chain structure so mid-chain wakeups propagate properly * Rework of sleeping_owner handling so that we properly deal with delayed-dequeued (sched_delayed) tasks (also removes now unused proxy_deactivate() logic) v15: * Rework do_activate_task to be activate_task() and have it call __activate_task(), suggested by Carlos Llamas * Put task_struct additions under CONFIG_SCHED_PROXY_EXEC v16: * Rework do_activate_blocked_waiter locking to use scoped_guard * Rework find_proxy_task() logic to use guard v18: * Integrate Suleiman's suggested optimization to check sleeping owner status before task_cpu, to avoid unnecssarily proxy-migrating tasks to then just dequeue them. * Add on_cpu check to fix for a very hard to reproduce race where a late blocked_task_activation() happens while the task is already on_cpu elsewhere. When we get to __schedule() we block the task (!on_rq) but haven't yet switched away. blocked_task_activation() would then incorrectly activate on a different runqueue. * Add init_task initialization for sleeping owner lists, as suggested by Suleiman v19: * Build fixup for !CONFIG_SMP v22: * Rework to avoid gotos in guard() scopes, using break and switch() on action values. Suggested by K Prateek. v25: * Fix elevated nr_uninterruptible and nr_iowait counts, which were causing bad loadavg values. Reported and fixed by David Stevens v28: * In __proxy_remove_from_sleeping_owner() we call put_task(), which might free the owner while we are still holing the owners->blocked_lock. So be sure to get_task()/put_task() around the owner usage to ensure we don't prematurely free the task. * Also reorder the put_task/unlock lines in activate_blocked_waiters(), to avoid a similar issue v30: * Optimize activate_blocked_waiters() so we don't do so much unnecessary work when proxy-exec is enabled. If there are no blocked waiters, we can return early. v31: * K Prateek's noticed a potential race where if a sleeping owner is woken in paralell with the waiter enquing itself on the sleeping owner, the waiting task could get stuck on that owner until it sleeps and is woken up again. So double check after we grab the blocked_lock to avoid this. * Maria Yu and Tengfei Fan reported an issue with rq->nr_iowait values getting out of balance, and provided a helpful reproducer. To fix this, make sure we do the atomic_dec() in do_activate_blocked_waiter() before we call proxy_set_task_cpu() so we always dec on the same rq we inc'ed. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- include/linux/sched.h | 7 + init/init_task.c | 6 + kernel/fork.c | 6 + kernel/sched/core.c | 317 +++++++++++++++++++++++++++++++++++++++--- 4 files changed, 315 insertions(+), 21 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d10..8c2ba6dce58f4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1251,6 +1251,13 @@ struct task_struct { =20 struct mutex *blocked_on; /* lock we're blocked on */ raw_spinlock_t blocked_lock; +#ifdef CONFIG_SCHED_PROXY_EXEC + struct list_head blocked_head; /* tasks blocked on this task */ + struct list_head blocked_node; /* our entry on someone elses blocked_he= ad */ + /* Node for list of tasks to process blocked_head list for blocked entiti= y activations */ + struct list_head blocked_activation_node; + struct task_struct *sleeping_owner; /* task our blocked_node is enqueued= on */ +#endif =20 /* * The task that is boosting this task; a back link for the current diff --git a/init/init_task.c b/init/init_task.c index b67ef6040a655..809282a2741d7 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -211,6 +211,12 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES)= =3D { &init_task.alloc_lock), #endif .blocked_donor =3D NULL, +#ifdef CONFIG_SCHED_PROXY_EXEC + .blocked_head =3D LIST_HEAD_INIT(init_task.blocked_head), + .blocked_node =3D LIST_HEAD_INIT(init_task.blocked_node), + .blocked_activation_node =3D LIST_HEAD_INIT(init_task.blocked_activation_= node), + .sleeping_owner =3D NULL, +#endif #ifdef CONFIG_RT_MUTEXES .pi_waiters =3D RB_ROOT_CACHED, .pi_top_task =3D NULL, diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5a..6dfde115fe34e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2247,6 +2247,12 @@ __latent_entropy struct task_struct *copy_process( =20 p->blocked_on =3D NULL; /* not blocked yet */ p->blocked_donor =3D NULL; /* nobody is boosting p yet */ +#ifdef CONFIG_SCHED_PROXY_EXEC + INIT_LIST_HEAD(&p->blocked_head); + INIT_LIST_HEAD(&p->blocked_node); + INIT_LIST_HEAD(&p->blocked_activation_node); + p->sleeping_owner =3D NULL; +#endif =20 #ifdef CONFIG_BCACHE p->sequential_io =3D 0; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1bf60d78c9208..d813a9c2f0c35 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2216,7 +2216,7 @@ inline bool dequeue_task(struct rq *rq, struct task_s= truct *p, int flags) return p->sched_class->dequeue_task(rq, p, flags); } =20 -void activate_task(struct rq *rq, struct task_struct *p, int flags) +static inline void __activate_task(struct rq *rq, struct task_struct *p, i= nt flags) { if (task_on_rq_migrating(p)) flags |=3D ENQUEUE_MIGRATED; @@ -2227,6 +2227,71 @@ void activate_task(struct rq *rq, struct task_struct= *p, int flags) ASSERT_EXCLUSIVE_WRITER(p->on_rq); } =20 +#ifdef CONFIG_SCHED_PROXY_EXEC +static inline +void __proxy_remove_from_sleeping_owner(struct task_struct *owner, struct = task_struct *p) +{ + lockdep_assert_held(&owner->blocked_lock); + + if (p->sleeping_owner =3D=3D owner) { + list_del_init(&p->blocked_node); + WRITE_ONCE(p->sleeping_owner, NULL); + put_task_struct(owner); // matches get in proxy_enqueue_on_owner + } +} + +static inline void proxy_remove_from_sleeping_owner(struct task_struct *p) +{ + struct task_struct *owner =3D READ_ONCE(p->sleeping_owner); + + if (owner) { + /* + * __proxy_remove_from_sleeping_owner() does a + * put on owner to match the get done in + * proxy_enqueue_on_owner(). If that put is the + * last one and it frees owner, we'd be freeing + * a lock we held. So get/put owner around its + * usage her to ensure that doesn't happen. + */ + get_task_struct(owner); + raw_spin_lock(&owner->blocked_lock); + __proxy_remove_from_sleeping_owner(owner, p); + raw_spin_unlock(&owner->blocked_lock); + put_task_struct(owner); + } +} + +void activate_task(struct rq *rq, struct task_struct *p, int en_flags) +{ + if (!sched_proxy_exec()) { + __activate_task(rq, p, en_flags); + return; + } + + lockdep_assert_rq_held(rq); + proxy_remove_from_sleeping_owner(p); + /* + * By calling __activate_task() with blocked_lock held, we + * order against the find_proxy_task() blocked_task case + * such that no more blocked tasks will be enqueued on p + * once we release p->blocked_lock. + */ + raw_spin_lock(&p->blocked_lock); + WARN_ON(task_cpu(p) !=3D cpu_of(rq)); + __activate_task(rq, p, en_flags); + raw_spin_unlock(&p->blocked_lock); +} +#else +static inline void proxy_remove_from_sleeping_owner(struct task_struct *p) +{ +} + +void activate_task(struct rq *rq, struct task_struct *p, int en_flags) +{ + __activate_task(rq, p, en_flags); +} +#endif + void deactivate_task(struct rq *rq, struct task_struct *p, int flags) { WARN_ON_ONCE(flags & DEQUEUE_SLEEP); @@ -3756,6 +3821,169 @@ static inline void proxy_reset_donor(struct rq *rq) resched_curr(rq); } =20 +static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) +{ + unsigned int wake_cpu; + + /* + * Since we are enqueuing a blocked task on a cpu it may + * not be able to run on, preserve wake_cpu when we + * __set_task_cpu so we can return the task to where it + * was previously runnable. + */ + wake_cpu =3D p->wake_cpu; + __set_task_cpu(p, cpu); + p->wake_cpu =3D wake_cpu; +} + +static void do_activate_blocked_waiter(struct rq *target_rq, struct task_s= truct *p, int en_flags) +{ + unsigned int state; + struct rq_flags rf; + int target_cpu =3D cpu_of(target_rq); + + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + state =3D READ_ONCE(p->__state); + /* Avoid racing with ttwu */ + if (state =3D=3D TASK_WAKING) + return; + + if (READ_ONCE(p->on_rq)) { + /* + * We raced with a non mutex handoff activation of p. + * That activation will also take care of activating + * all of the tasks after p in the blocked_head list, + * so we're done here. + */ + return; + } + if (task_on_cpu(task_rq(p), p)) { + /* + * Its possible this activation is very late, and + * we already were woken up and are running on a + * different cpu. If that task blocked, it could be + * dequeued (so on_rq =3D=3D 0), but still on_cpu. + * Bail in this case, as we definitely don't want to + * activate a task when its on_cpu elsewhere. + */ + return; + } + /* + * Have to make sure we handle nr_iowait adjustment before + * we call proxy_set_task_cpu() to ensure we are adjusting + * the same runqueue we left (where block_task() + * incremented nr_iowait). + */ + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } + proxy_set_task_cpu(p, target_cpu); + rq_lock_irqsave(target_rq, &rf); + /* + * proxy_enqueue_on_owner() called block_task() which + * increments nr_uninterruptible, so we need to reverse + * that when we activate the blocked waiter + */ + if (p->sched_contributes_to_load) + target_rq->nr_uninterruptible--; + update_rq_clock(target_rq); + activate_task(target_rq, p, en_flags); + resched_curr(target_rq); + rq_unlock_irqrestore(target_rq, &rf); + } +} + +static void activate_blocked_waiters(struct rq *target_rq, + struct task_struct *owner, + int wake_flags) +{ + struct list_head bal_head; + unsigned long flags; + int en_flags =3D ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK; + + if (!sched_proxy_exec()) + return; + + /* + * A whole bunch of waiting donor tasks back this blocked + * lock owner task, wake them all up to give this task its + * 'fair' share. + * + * This is a little unique here and the locking is messy. + * At this point we only hold the blocked_lock, so the + * owner task may be able to run and do all sorts of + * things while we are processing the blocked_head list, + * including going back to sleep, which can cause tasks + * to be added to the owners->blocked_head while we are + * processing it! + * Thus, we pull the entire list off the owner->blocked_head + * here so that we will only process a finite amount of + * tasks. Tasks added after this will be processed by the + * future wake events. + * Even though we have pulled the list off the blocked_head + * the removed list is *still* "owned" and serialized by + * the owner->blocked_lock! As we have to serialize against + * mid-chain wakeups, who may try to remove themselves from + * the list. + */ + raw_spin_lock_irqsave(&owner->blocked_lock, flags); + if (!list_empty(&owner->blocked_activation_node)) { + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + return; + } + + if (list_empty(&owner->blocked_head)) { + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + return; + } + + get_task_struct(owner); + INIT_LIST_HEAD(&bal_head); + list_add_tail(&owner->blocked_activation_node, &bal_head); + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + + if (wake_flags & WF_MIGRATED) + en_flags |=3D ENQUEUE_MIGRATED; + + while (!list_empty(&bal_head)) { + struct list_head tmp_head; + + INIT_LIST_HEAD(&tmp_head); + owner =3D list_first_entry(&bal_head, struct task_struct, blocked_activa= tion_node); + + raw_spin_lock_irqsave(&owner->blocked_lock, flags); + list_replace_init(&owner->blocked_head, &tmp_head); + list_del_init(&owner->blocked_activation_node); + while (!list_empty(&tmp_head)) { + struct task_struct *p; + + p =3D list_first_entry(&tmp_head, + struct task_struct, + blocked_node); + WARN_ON(p =3D=3D owner); + WARN_ON(p->sleeping_owner !=3D owner); + __proxy_remove_from_sleeping_owner(owner, p); + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + + do_activate_blocked_waiter(target_rq, p, en_flags); + + raw_spin_lock_irqsave(&p->blocked_lock, flags); + if (list_empty(&p->blocked_activation_node)) { + get_task_struct(p); + list_add_tail(&p->blocked_activation_node, &bal_head); + } + raw_spin_unlock_irqrestore(&p->blocked_lock, flags); + + raw_spin_lock_irqsave(&owner->blocked_lock, flags); + } + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + put_task_struct(owner); // put matches get prior to adding to local bal_= head + } +} + +static inline struct task_struct *proxy_resched_idle(struct rq *rq); + /* * Checks to see if task p has been proxy-migrated to another rq * and needs to be returned. If so, we deactivate the task here @@ -3800,6 +4028,10 @@ static inline bool proxy_needs_return(struct rq *rq,= struct task_struct *p) { return false; } + +static inline void activate_blocked_waiters(struct rq *target_rq, + struct task_struct *owner, + int wake_flags) {} #endif /* CONFIG_SCHED_PROXY_EXEC */ =20 static void @@ -3873,8 +4105,10 @@ static int ttwu_runnable(struct task_struct *p, int = wake_flags) =20 update_rq_clock(rq); if (p->is_blocked) { - if (p->se.sched_delayed) + if (p->se.sched_delayed) { + proxy_remove_from_sleeping_owner(p); enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED); + } if (proxy_needs_return(rq, p)) return 0; } @@ -3903,13 +4137,19 @@ void sched_ttwu_pending(void *arg) update_rq_clock(rq); =20 llist_for_each_entry_safe(p, t, llist, wake_entry.llist) { + int wake_flags; if (WARN_ON_ONCE(p->on_cpu)) smp_cond_load_acquire(&p->on_cpu, !VAL); =20 if (WARN_ON_ONCE(task_cpu(p) !=3D cpu_of(rq))) set_task_cpu(p, cpu_of(rq)); =20 - ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); + wake_flags =3D p->sched_remote_wakeup ? WF_MIGRATED : 0; + ttwu_do_activate(rq, p, wake_flags, &rf); + rq_unlock(rq, &rf); + activate_blocked_waiters(rq, p, wake_flags); + rq_lock(rq, &rf); + update_rq_clock(rq); } =20 /* @@ -4416,6 +4656,7 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) ttwu_queue(p, cpu, wake_flags); } out: + activate_blocked_waiters(cpu_rq(task_cpu(p)), p, wake_flags); if (success) ttwu_stat(p, task_cpu(p), wake_flags); =20 @@ -6731,21 +6972,6 @@ static bool try_to_block_task(struct rq *rq, struct = task_struct *p, } =20 #ifdef CONFIG_SCHED_PROXY_EXEC -static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) -{ - unsigned int wake_cpu; - - /* - * Since we are enqueuing a blocked task on a cpu it may - * not be able to run on, preserve wake_cpu when we - * __set_task_cpu so we can return the task to where it - * was previously runnable. - */ - wake_cpu =3D p->wake_cpu; - __set_task_cpu(p, cpu); - p->wake_cpu =3D wake_cpu; -} - static inline struct task_struct *proxy_resched_idle(struct rq *rq) { put_prev_set_next_task(rq, rq->donor, rq->idle); @@ -6852,6 +7078,29 @@ static void proxy_migrate_task(struct rq *rq, struct= rq_flags *rf, proxy_reacquire_rq_lock(rq, rf); } =20 +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owne= r, + struct task_struct *p) +{ + lockdep_assert_rq_held(rq); + lockdep_assert_held(&owner->blocked_lock); + /* + * ttwu_activate() will pick them up and place them on whatever rq + * @owner will run next. + */ + WARN_ON(p =3D=3D owner); + WARN_ON(!p->on_rq); + WARN_ON(p->sleeping_owner); + get_task_struct(owner); + WRITE_ONCE(p->sleeping_owner, owner); + /* + * ttwu_do_activate must not have a chance to activate p + * elsewhere before it's fully extricated from its old rq. + */ + list_add(&p->blocked_node, &owner->blocked_head); + proxy_resched_idle(rq); + block_task(rq, p, READ_ONCE(p->__state)); +} + /* * Find runnable lock owner to proxy for mutex blocked donor * @@ -6938,11 +7187,37 @@ find_proxy_task(struct rq *rq, struct task_struct *= donor, struct rq_flags *rf) } =20 if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) { - /* XXX Don't handle blocked owners/delayed dequeue yet */ + /* + * rq->curr must not be added to the blocked_head list or else + * ttwu_do_activate could enqueue it elsewhere before it switches + * out here. The approach to avoid this is the same as in the + * migrate_task case. + */ if (curr_in_chain) return proxy_resched_idle(rq); - __clear_task_blocked_on(p, NULL); - goto deactivate; + /* + * If !@owner->on_rq, holding @rq->lock will not pin the task, + * so we cannot drop @mutex->wait_lock until we're sure its a blocked + * task on this rq. + * + * We use @owner->blocked_lock to serialize against ttwu_activate(). + * Either we see its new owner->on_rq or it will see our list_add(). + */ + WARN_ON(owner =3D=3D p); + raw_spin_unlock(&p->blocked_lock); + raw_spin_lock(&owner->blocked_lock); + /* + * Before actually adding to the sleeping owner, double check + * we didn't race with an owner wakeup before grabbing the + * owner's blocked_lock. + */ + if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) + proxy_enqueue_on_owner(rq, owner, p); + + raw_spin_unlock(&owner->blocked_lock); + raw_spin_lock(&p->blocked_lock); + + return NULL; /* retry task selection */ } =20 owner_cpu =3D task_cpu(owner); --=20 2.55.0.654.g21b8a5bc05-goog From nobody Tue Sep 29 13:57:27 2026 Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B07FE3B0AD0 for ; Fri, 7 Aug 2026 03:52:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074765; cv=none; b=hTVo9cVoyOK471S9TakpnfaSU0tk/L59Q3pL6SL6uHj52wHIIdXwEZGYvmF7P2XPG2XZbmKqxtIysKujJaq8i29aPEzjJP0RX85t39EqpquhaZ7GL6krCTVaj1tGkOwEp1yDuqLB5f1mOArH55NMWNqLv2uop1151JOKV0E9qjk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786074765; c=relaxed/simple; bh=ytutfxjM7Kr/Xh6NGVjWZUggzWWeKNv3va6vplznXTg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=afOFk4371vP4T18gor4Hzy2F5MiyHud/Mxi7sJLAQ100oF6vcI3O45ANE4qQLiCBcQc85UnIiBRSagEKGcGWsd7BecwSa5qzZ95J/ltai/NTnjUY6kssysPL819/Vj0SV7QPHs2bvROd/uUSOGMZa88jGJPQQY6X5BYz+lCCzIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cEQmUaHh; arc=none smtp.client-ip=209.85.210.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cEQmUaHh" Received: by mail-pf1-f200.google.com with SMTP id d2e1a72fcca58-84865f326efso3498689b3a.0 for ; Thu, 06 Aug 2026 20:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786074763; x=1786679563; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=xhfpKoaMDL+R4VkwqXx0RUTIjsiiOPWkf+aiH45sxeI=; b=cEQmUaHhtASaznLtGpzQReCIEHSPlAmLBsdqavJcnNg/y1848tqB9BqZ8i7wj437+I mAa8WQO5iFoy8AtvAGPvEV7feard82l0q018dNgtirPgrfhkdSdnNykmTe/Nuta6xB8Y KjJOWWR+FG7NZG6aem68AH021SWUo8zBuxPKNUhrkdfUrNrl7QXwg/CpZQlWyodJPFw5 2d8m3nPp3GYfAYLzBZ9ZaqyCPtT1tbevDbX2GTiqsVvhzW36C8dw88FBHbu4+4v9Ab7L pSJUeX9cO5o6pgnjbb/eQaEk3uv721c0+x0m3aWkJ7lhxBuPKZTlaxLz3h/82GJQ1OzG 5ENQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786074763; x=1786679563; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xhfpKoaMDL+R4VkwqXx0RUTIjsiiOPWkf+aiH45sxeI=; b=qtald89cY8uR3L9D/Ri9UZz7wbOYir9vnKmh+rURV3KCqMMBrOkWm1VmRTBLF4Ze6p fPya/q2h+mxT4Gqc7Mpfz7SV06083O49+7cVIdIYd4NUG8cpqNpkfLtAYqgdWDaVz/Co fIao7gz+ZevR64ulcaajh2jNj55s/WEilb9TwJy2eHn2aQmpHHKm0wkRnNkIEE5AYXs/ JpQTI8yY6SQ7XXasxdNQdEniPQzwC9V8FOlvDszZRz/+v4OkPXOXV/hcbAdigHN9MW3z 10LY5Y4dCoQMHFoQ+9dU0hG5MjqZrjxtpk93XyD1xeJD96tZqOqAyVlUyADUyZRcalZA 7Teg== X-Gm-Message-State: AOJu0YzeSBbbEtPOsru4EHhnfAoVuS3vQHzkW4/UiLrlS8qCeFyXOCBl l/PdS2zHBKmVOhnYTHxckjePmILwvUED9WorkB/3Opn+KOuOWVXN1NajTSLxQE8G7SIBnHKDNep sSWSKYi6S6TkLRPxBHxtEN+IHFvL2c3/ljPzEnuwxuGpa3BKVNmSCl9xJY3rzg5/EFOJIXwTUkf IuZPeWN3pb9jZEIPE8tQQj1BrVho6ultnOHc/W5GAzs/CVa/Qo X-Received: from pfwp49.prod.google.com ([2002:a05:6a00:26f1:b0:84a:894f:23f2]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:181e:b0:848:2ae4:d2ba with SMTP id d2e1a72fcca58-84f2e05e84emr21553543b3a.28.1786074762558; Thu, 06 Aug 2026 20:52:42 -0700 (PDT) Date: Fri, 7 Aug 2026 03:52:15 +0000 In-Reply-To: <20260807035232.1881495-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260807035232.1881495-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.654.g21b8a5bc05-goog Message-ID: <20260807035232.1881495-10-jstultz@google.com> Subject: [RESEND][PATCH v31 9/9] sched: Distinguish proxy activations from wakeups From: John Stultz To: LKML Cc: Andrea Righi , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Andrea Righi Sleeping-owner handling reactivates blocked waiters with wakeup enqueue semantics so scheduling classes can restore their runnable accounting. Unlike a normal wakeup, the waiter remains blocked and is only made runnable to donate its scheduling context. Add ENQUEUE_PROXY to identify this activation and set it alongside ENQUEUE_WAKEUP. Consume the internal marker in sched_ext and suppress SCX_ENQ_WAKEUP before invoking BPF so the activation can be handled as a blocked proxy donor. Signed-off-by: Andrea Righi Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 2 +- kernel/sched/ext/ext.c | 8 ++++++++ kernel/sched/sched.h | 2 ++ 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d813a9c2f0c35..c54e9fedc9cd8 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3900,7 +3900,7 @@ static void activate_blocked_waiters(struct rq *targe= t_rq, { struct list_head bal_head; unsigned long flags; - int en_flags =3D ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK; + int en_flags =3D ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK | ENQUEUE_PROXY; =20 if (!sched_proxy_exec()) return; diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c index e3fa7b2fac9df..834c8b6dae6c9 100644 --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -2021,6 +2021,14 @@ static void enqueue_task_scx(struct rq *rq, struct t= ask_struct *p, int core_enq_ int sticky_cpu =3D p->scx.sticky_cpu; u64 enq_flags =3D core_enq_flags | rq->scx.extra_enq_flags; =20 + /* + * Sleeping-owner activation uses wakeup semantics for the core + * scheduling classes, but the donor remains blocked. Expose it to BPF + * as a blocked-donor admission rather than a full wakeup. + */ + if (core_enq_flags & ENQUEUE_PROXY) + enq_flags &=3D ~(ENQUEUE_PROXY | SCX_ENQ_WAKEUP); + if (enq_flags & ENQUEUE_WAKEUP) rq->scx.flags |=3D SCX_RQ_IN_WAKEUP; =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56d9c09c485e6..e38988574cfa2 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2540,6 +2540,7 @@ extern const u32 sched_prio_to_wmult[40]; * ENQUEUE_REPLENISH - CBS (replenish runtime and postpone deadline) * ENQUEUE_MIGRATED - the task was migrated during wakeup * ENQUEUE_RQ_SELECTED - ->select_task_rq() was called + * ENQUEUE_PROXY - activate a blocked donor behind a waking owner * * XXX SAVE/RESTORE in combination with CLASS doesn't really make sense, b= ut * SCHED_DEADLINE seems to rely on this for now. @@ -2571,6 +2572,7 @@ extern const u32 sched_prio_to_wmult[40]; #define ENQUEUE_MIGRATED 0x00040000 #define ENQUEUE_INITIAL 0x00080000 #define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_PROXY 0x00200000 =20 #define RETRY_TASK ((void *)-1UL) =20 --=20 2.55.0.654.g21b8a5bc05-goog