From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03F9F33F59D for ; Fri, 24 Jul 2026 18:12:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916765; cv=none; b=MkGZX/HIIVSqK8tAClB+HpXrHBjx+x6HdLEQX9ZNO307FPDugB0S6IsTUr5TIRIwcuTx2UAADsxYfvIVdUUt4WkyA/8YWHSwKgFeMV2HW3Fr+BymGjtT+ZcmKetGK7cz/IJPNEmO/OJVMvoDAO9TN0N3z9asz31XutlHj6mXiiw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916765; c=relaxed/simple; bh=ovv5Ac47snxxQ+Z9R6rGJxumNVj0VNE9iu5ZB64X1mU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WGpQ3FbP3wJrLhPKEKQ7KUfgq/xmnpjCsflfytf5BgZdXf5lkRxbRcKbYpLv8H84SomV7eMD65xPTXsgwexY+2dW0s0wZ6LPgIAm57y8zcIw2XnK+O8UVuSIa0UKBmbxfQ7KFLV/9Z/h7QN8Dtm7MHIrRoN71fUKXoudDtMhaBI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CDoFm3L2; arc=none smtp.client-ip=209.85.214.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CDoFm3L2" Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-2cfca8558d2so8806815ad.2 for ; Fri, 24 Jul 2026 11:12:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916763; x=1785521563; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=xI6yi0/ES/KMSLcG2AQHgX5uE8KlWDgTw1GhLrDHkOQ=; b=CDoFm3L2nWQelsBpcnE0/hI/FuX7DfQ7CbD+uQUI8XMJzRX94NGWknQ2tmbFE4nU6O v1spSc0el6G/OXR6o5MivcUJmcdZlnMLaZLn1FuUdBvXl5dwtx7rE9caUb3ktWfA03s7 DyE/uT6nfpLpdNShwmQGTEIQPOxmoutDeWw4YFqbVW6khe0II90pnVavlS3K00o2/E2P svr3g1rwjNVQuJBY5AQZ5SYc8mKPPottjoSzTJF6uju4zEy0KR/MJVH/Lxjc9oT1/F6r gu7KxXJ5IuaWtdIU42xjK5myizJQaDHTDghiKNylWmmGcYDnaBCM0akRWL2KybPNW5aM thzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916763; x=1785521563; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xI6yi0/ES/KMSLcG2AQHgX5uE8KlWDgTw1GhLrDHkOQ=; b=g/lJm5d8TIIlZYQbHtklAphVM7//49VCCvZfuXq4B2xEz/R+eRb3qilIBi5/rWBHKT 0c9LHZsaE1bMPs4f+EvJ2WZZPPozNtnIyMsztGO2CFheuQwxk0X+GedMSYFdO91fDN2r +T7NBhgbcWbjv/uv87ZQY6EmWSnKta3BalCftMQKCJTb8ZZ7Fxyv9Wq25Bke1ZNM5DgF xAC6MEfaWKcvAIk62bMhKoh9rzYM/dsmGcm0S02kPym9/E2YISNoTX1ebHo0ATpjnnc3 EIJOPsJ+krlymdXfX2VDtQ+M4xr/faLF8KNwA+iMp6aBjA7FjlA6/8UqFDIj1TfISEar M1vQ== X-Gm-Message-State: AOJu0Yy5aufaLF8cobP180D7SPCjVvtfT6OJnsMQW0NH1A55yB/BNIrN gklNOzaN7vKUBI/2ksYzGVtHwckfmmMlAHp7kwvhfwP533UUmGg4Ytf4+sel/tjoOZJljfuqd+w bbwlE2kBXzVhzMXTuG6r/ZXa72np0duOeXRRfSadleMG+xyh+pFQiAmKv+z7Tux9Ibcfw15ykiR jyqMfXeyNsgymgUpsPCwSlG3HiKZ36XnQzLaGN2S730AuC7nZN X-Received: from plrp20.prod.google.com ([2002:a17:902:b094:b0:2cf:a0e2:93f7]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:f68e:b0:2cc:8267:31b5 with SMTP id d9443c01a7336-2cfa6b77a03mr96771255ad.19.1784916762892; Fri, 24 Jul 2026 11:12:42 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:15 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-2-jstultz@google.com> Subject: [PATCH v31 1/9] sched/deadline: Ignore proxy-exec sched_yield() From: John Stultz To: LKML Cc: Christian Loehle , Juri Lelli , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christian Loehle With proxy execution, rq->curr is the execution context while rq->donor is the donating context. rq->curr's sched_yield() is dispatched through the donor class so that proxy execution follows the effective scheduling context. For SCHED_DEADLINE, this is too strong. yield_task_dl() does not just ask for another task of equal priority to get to run, it marks the current DL entity as yielded and forces it to sleep until replenishment. These yield semantics are fundamentally different from FIFO/RR (where if no equal-priority tasks are runnable, no harm done, they get picked again immediately) or OTHER (also doesn't cause priority inversion), so do not mix these semantics by ignoring a sched_yield() on DL donors. Fixes: 127b90315ca0 ("sched/proxy: Yield the donor task") Acked-by: Juri Lelli Signed-off-by: Christian Loehle Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/deadline.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 0f858b98c9aa3..3e89b3abeb278 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2574,6 +2574,9 @@ static bool dequeue_task_dl(struct rq *rq, struct tas= k_struct *p, int flags) */ static void yield_task_dl(struct rq *rq) { + if (sched_proxy_exec() && rq->curr !=3D rq->donor) + return; + /* * We make the task go to sleep until its current deadline by * forcing its runtime to zero. This way, update_curr_dl() stops --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3DFA372B27 for ; Fri, 24 Jul 2026 18:12:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916766; cv=none; b=KNjYlj7vLsRXXOVMkv6Cnsx1uA/WswvO15qq4eyAuZ+VTaDt6JGZY+rzihj6+TImdiulP2Sxkr+BQRGw4CQhzGex638p24ah/jyDox+MLZTUJa/lzVrjF3FyJTHOf+KJEKF3XJzRE+doBfeYG/i4iyglVf4TVIxI9+5hK3NoLes= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916766; c=relaxed/simple; bh=gwBHrcBppSr7kgKgReFU0s7XkMcqC9NKaO0MOP/ng3Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=B4cDo2L5yDQj7Tm+n9/Z0IbG4waeVeItinYv2VkRIt6QenfRB4arVZai1/80D/77GInTwS9h2CrHAO8dT9snNRyUz5lCXCjuicVS01PLsPB4+oCmSAFVPSSnqZBp0sfbm9Qvre6V9dAa1ffAP5/tF8YHNSQGpFG3kM8S+5ga3R4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FPnp84cp; arc=none smtp.client-ip=209.85.214.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FPnp84cp" Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-2ccb6823efcso6538845ad.0 for ; Fri, 24 Jul 2026 11:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916764; x=1785521564; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=J7OIU3r/UTkoP1uvZH3CR6gme4w4msMs0MeYe2vy4AU=; b=FPnp84cprBgFzAejjlBdycaJKwKUywzbT4LAJDGvOtKkhtEaVwTpcRo9fWkkvJ553Y eAR014VeA+jPP61vKK/jAkfCKfI5tWlScFvWFMEVlFFyYi2/Ns2qelhFmOoA24MyhF/H 7GeAmfama5/Iagc9gNQ2KmGUTpwBMrXRpDeaYIwl0dtqeRqw5x2n65qktCOtI8YuCQap ARlIPO4NdBDTxwETMZYcOzQ6UA5hgvJaez4nDvhcT7nGd0XL72I6fUGN1oSgJ5nyWYTW RGCKY/2mhnPfKmISPrRTeZMJqkuVjEFN3CX2BseAoAwA4r9w0zL5QEDaq2cxOmuLIokP BXvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916764; x=1785521564; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=J7OIU3r/UTkoP1uvZH3CR6gme4w4msMs0MeYe2vy4AU=; b=os3Ejg9rxwQu8gHMqqpfnQANMgHtFcp+B5SsIn0ZSu8RdOz68Y/5Whvb74cFwfELoH IWTpENtfoUimyteL2kmBNkd3uOsxDaT/4eM67+PVTgMHQbYI5jN4juJQEr/6+/yQf3Bj 3u6je2j0zboVN4eq24sUFUZjJMzKmMvO2pgQVZg2ug7ZULPvF+SpDyO//cNBHScs7/iw Ec8rY76LreoYJkFAyU8Qf/Mr/Ta9t9xBoAdFBjHZ7Hq6XWs3bwreXJ7EqT3cLASUWuFf LbL8AMxQ+pu74JjMQv5QvsalPdz7ODcKijiETSNCxJV9TaXPZIx4+6xf7931AFZ2w8x9 FT6w== X-Gm-Message-State: AOJu0YyQCMQjh8d1BWLusesh9t5xSLuu76QIEUY9YcgBSwDzGduUllxe i6YKbbg1WwrEAhXPSJs2m1ijLacOfx6g1A+W3xSZX9xxfGumKHcMy3B4CSbvZGhLoJaJKIwzlL6 kFaUcqD3CkTksRXWhjmCuRS/1yKfSxiLbXC1qjrF7tT/4L2LocxFwAsnRV2isRoY2vWVmK4QfPC SC+nX7InjMEVeDjQSPvdJWXLLG5kymwMh0bXaDcLCzNjpGN1QR X-Received: from pla11.prod.google.com ([2002:a17:902:ffcb:b0:2cc:e407:8fef]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1c3:b0:2ce:d34a:5959 with SMTP id d9443c01a7336-2cfa6f97cedmr92633065ad.44.1784916763734; Fri, 24 Jul 2026 11:12:43 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:16 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-3-jstultz@google.com> Subject: [PATCH v31 2/9] sched/core: Don't steal a proxy-exec donor From: John Stultz To: LKML Cc: Vasily Gorbik , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vasily Gorbik try_steal_cookie() avoids stealing src->core_pick and src->curr before moving a task with the same cookie via move_queued_task_locked(). With proxy-exec, src->donor is the current scheduling context and may differ from src->curr. Stealing it migrates a task that the source rq still treats as current, leaving src's scheduler state for that task stale. For CFS this means cfs_rq->curr points at the stolen entity, and the next pick on the source rq hits the WARN_ON_ONCE in put_prev_entity(). Commit 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()") tweaked the fair class logic so that the donor task isn't migrated away while we're running the proxy. Do it similarly for try_steal_cookie() and skip src->donor as well. Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_prox= y_task()") Signed-off-by: Vasily Gorbik Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f61..e7074ba54a91f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6474,7 +6474,7 @@ static bool try_steal_cookie(int this, int that) return false; =20 do { - if (p =3D=3D src->core_pick || p =3D=3D src->curr) + if (p =3D=3D src->core_pick || p =3D=3D src->curr || p =3D=3D src->donor) goto next; =20 if (!is_cpu_allowed(p, this)) --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8862D3AEF3F for ; Fri, 24 Jul 2026 18:12:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916767; cv=none; b=K/MngJoHPsFUrv1KLYrQow3vtgnjsfFcnpv4OMf6tGOT8G/Hvrck9TlCjtQdTyOn9ur1RkJCe/T4qs+1s6TYW6/PEo2J9emXSuhyNafQO6O5VvNv/LvvOxUYFDlMMKGHO0g8f0PYr2V+PMM7W7w3syB7UxFGHrX+eF67ui5n1IM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916767; c=relaxed/simple; bh=FzcGsrUN17ifre3P9vJd3tfeebpT7itoQNejEk7m8SI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ao9/b8ruAuTP/PSgEaD4YVm5kJXGynlvwAR5DZTEM/b2KbMv7glB7kk5wqstHjjNYyt1uuRsxVoRraLq0QRKMLO4DBYLIw1SyR+aqANwEXT+b3vDZQGvMXV1C0XW5uRaP3lZS1SqF2hqDMEWgwl6K8hmISzj724g/zhbq3sFpfk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UUPoOm4/; arc=none smtp.client-ip=209.85.214.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UUPoOm4/" Received: by mail-pl1-f198.google.com with SMTP id d9443c01a7336-2caf4173b1cso12415595ad.3 for ; Fri, 24 Jul 2026 11:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916765; x=1785521565; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=WIEiccwdSFaUPusHTMl+DdTe7ivr6hPIsg8PNnTooow=; b=UUPoOm4/loplQqTwfymRnWydv7NMS/jSna6mxyBSaViqoULbDU4IASX9VjY2jgl+UT acrRcIQBYEMHH2iuu/lTYalNDUD6G9jwdD8VnCVnDecNQcNrIY2pLfM/8YBJ+XYZtbW0 D2UG2K7RpsEdVrgBvcf0SU2mre1voN3qGMR0yXDZojqvFqJEnGEi3mYDXdMgWLXkGQOu C1IPrBE/kVetIRcy2BQQr7RZ6y/UpLK+4ByBpHN+L5PEg0PX3YWW8SrKa7luYolJvWqF JaJTSSnNrzUdrc5rNYTUw3BxJjW6Kf5T9Jy/yDEdxjd0V5gcT4dbT2gYqNohmUFkjVCc /uYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916765; x=1785521565; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WIEiccwdSFaUPusHTMl+DdTe7ivr6hPIsg8PNnTooow=; b=IEKNQnjO/Dlf2Y/anmg0ZY5EICBLT8KoTES7pSt42R7muwLAMftYa7HNYr2KgeULZk IRHC7WQbnP670Edt1D8ad6yOsbGgAuzUPIy8midKFVUZXKz9yugRxlqsaGUImYDtNyJG 3zl/hHAU4TfQU162W+N1eqnOq8OBaX3NedkQan/Juy+YVpKl+V5fd3KBFSA2CpsBlSDF gnNpGT9AxkQnywPjqulShnG5xXz/Ob41TGduwxchqUeTcINyUL9UDzscu/pCjHrvk99O mX1AUfmLJ4fQ5tYsW6dXu37q/zYSyE9BPWWK8egroqQXIB1JUqrsKJAe+iufKUQE+mfs OvHQ== X-Gm-Message-State: AOJu0YznaHb9MDIibznVQ2ITFZgKcagmVrFMxEVXqviaZblW5xdeBs64 qj4IT5waHmMcBjtla9VCqarvg/5NPQv5t12gIbhzrrAyAlPfzdPn9xAxNHDI1WNTjj7yDVt6bmQ BWUg4yQd2LnCuGDdnAgR9rRdqDzr7NCBTVSWx7Am3MGJeMBxgRtSXp/A3Nnataq3lLD8Qpz59sw EJe2WKRs+WENvGZbsAzhrVvYujmfx/tClLP8Kz9E/heuniT9N5 X-Received: from plot12.prod.google.com ([2002:a17:902:8c8c:b0:2cc:7e9c:2c37]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e786:b0:2ca:6d87:cda0 with SMTP id d9443c01a7336-2cfa71b75afmr95463425ad.6.1784916764558; Fri, 24 Jul 2026 11:12:44 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:17 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-4-jstultz@google.com> Subject: [PATCH v31 3/9] sched/core: Avoid migrating blocked_on tasks From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Its not useful to migrate blocked_on tasks from other runqueues as the proxy logic will just migrate it back to the owner's rq. So skip blocked_on tasks here. Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e7074ba54a91f..8e661b5f133d7 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6477,6 +6477,9 @@ static bool try_steal_cookie(int this, int that) if (p =3D=3D src->core_pick || p =3D=3D src->curr || p =3D=3D src->donor) goto next; =20 + if (task_is_blocked(p)) + goto next; + if (!is_cpu_allowed(p, this)) goto next; =20 --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F7583C9ED9 for ; Fri, 24 Jul 2026 18:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916769; cv=none; b=p9nHglCcHCJu9P02X5JShVZ9bVLWGzHbeJoqRlaKlSNG0NmRFCln3YHjpekOV9/DjxB6B3/h6E0pZeoEPPmVBJq9PZERVUMH3JV898kgb+uyA+iC/pbq1zWXTvswCezkSBSfhYdTJXT3j2+Q2Z+0g3auvxpXNSIwyxKcuK3aElE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916769; c=relaxed/simple; bh=hfCyaujzoD2BnYDdd5euZDzhLPjmpwude885TQnSLDk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SLiturD9dxjmv7BAdQdSC3xWEmoWY3OOGaCddo70j7RkO7GwxnRpF3Wd1ANnZaBcDE1Cqm1+p0rX46wyAwK/sbVrYXW9BNgh0PihGNVp0bDdYhMPbR+EVwB+9ytBWN/S0v/eOtzNIpYTCrSXggmTzjGclNDoybibu0JEIfuGhfg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BC+3czXZ; arc=none smtp.client-ip=209.85.214.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BC+3czXZ" Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-2cc88e22f92so14971155ad.1 for ; Fri, 24 Jul 2026 11:12:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916766; x=1785521566; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=qBHfJAWhzNVyF+D9jhMXkxmBaLOzY2vh4135EZHyFR4=; b=BC+3czXZbW0xTfB3+FaljnWESLnhpITR9asWlN/TeN4y/TAZo1nbuEFjCKL9R8p12W w1ziHXNRM1PbJdzG4Pk6Tm+Qe2rpGY0ZteOXWR5mF7obQjN0nAXNxQUTeo5+0fgvEOD6 ZVehmezfcS/Ic3K+Hwdnv5eQlHyVyWV3MAD8q+OKidMsWkbAdaQPq6dTMu0TKFo9pNKP TA70lVCfwbl9HG63uSWG1GX5MATpcIzniZdXZt4k3Wqqq0SnWanfJEHmAalcGNiQaebt ZMOqT1aWZUaysIxQFs2y45E/wA2Uz8fUooMEDpWZ3Yffwi5s2T/mbdKJ77h8dyLD+YiF Hfbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916766; x=1785521566; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qBHfJAWhzNVyF+D9jhMXkxmBaLOzY2vh4135EZHyFR4=; b=XBamhde0r1fuBP7z0zI+U2Eccm9BsfLSzd4ZG93D/iFjDmt6r+HuZSoqSgekYiz4Wp Uwa/aEJLJrPfpwbI+lNtHnDX5DsW/wk3MSED/zNTnKwDKCGTZcoyoFG+GhNrGjigkSrK jWnkLeNbGmtO70TFe6YkhTt3CjGU8rGllmetAnjsJ9/ouOCMJajEqwlvV5ziwYUaXLx4 fuwkpxrcsnMM0t+OBJ3dlq08vx+/q7x+RAg8XxMF6e46CHs7iAH4XA7AsenEm7/RFV1m atF4VvkCSUdwhTu5Gwf+iHgulqvzU5VtgGNfGfmPM/WOCqFvCldvsY7sO2QvVCv0zpmW jtOg== X-Gm-Message-State: AOJu0Yx0U5zfBSsDxCyiMWMGXZQyStlHjCDjD8dttA1dZ9etMsBvvGGj Onp5ibOTG7oDHfn/P+Q1kZVR0b9qA9x/2CgrJ2RidM2GHTcptcwuTdj3hsAbGTKj7I/W0oW3B+f xIjtM/yH1wUQykzmQxhARPf6rl9+Oo4XkQrw0MHNxEnuY3TzU6UGp7M9bRjpBHy/eyVFWh0RNGh pZZ90tXQm9LaLd0E+Wef/q1Jc1RHFLtdhfHEXWRRkbi6cd4Jnq X-Received: from plblo15.prod.google.com ([2002:a17:903:434f:b0:2ca:d0f4:2f91]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:2385:b0:2ce:ed59:7044 with SMTP id d9443c01a7336-2cfa748ce5fmr93276075ad.30.1784916765351; Fri, 24 Jul 2026 11:12:45 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:18 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-5-jstultz@google.com> Subject: [PATCH v31 4/9] sched/core: Don't proxy-exec unmatched cookie lock owners From: John Stultz To: LKML Cc: Vasily Gorbik , K Prateek Nayak , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vasily Gorbik Core scheduling chooses a core-wide cookie before __schedule() installs the next task. With proxy-exec enabled, that task becomes the donor/scheduling context, and find_proxy_task() may then replace the execution context with the runnable mutex owner. If its cookie differs from the selected core cookie, running it would bypass core scheduling's cookie selection. When the final mutex owner found by find_proxy_task() does not match the selected core cookie, stop proxying the donor. If the current execution context is already in the blocked chain, fall back to idle like the existing proxy-exec retry paths do. Otherwise deactivate the donor and let __schedule() pick again. The mutex owner can be picked later under its own cookie. Fixes: 7de9d4f94638 ("sched: Start blocked_on chain processing in find_prox= y_task()") Reported-by: K Prateek Nayak Link: https://lore.kernel.org/lkml/10282ce9-f4ae-498f-9b57-f4e1e61fffbc@amd= .com/ Signed-off-by: Vasily Gorbik [jstultz: Added tweak to ensure we deactivate donor, not runnable owner] Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 8e661b5f133d7..36e1db67a8374 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7004,6 +7004,14 @@ find_proxy_task(struct rq *rq, struct task_struct *d= onor, struct rq_flags *rf) owner->blocked_donor =3D p; } WARN_ON_ONCE(owner && !owner->on_rq); + + if (owner && !sched_cpu_cookie_match(rq, owner)) { + if (curr_in_chain) + return proxy_resched_idle(rq); + p =3D donor; /* Deactivate the donor, not the runnable owner */ + clear_task_blocked_on(p, NULL); + goto deactivate; + } return owner; =20 deactivate: --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7715C327C00 for ; Fri, 24 Jul 2026 18:12:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916775; cv=none; b=PBf55F6n5YPLZkZYS9ujcdiGR9C8q5Gl3AxSXkxDGxswu+hitt8ktbEXTDSfJ+28odCz5IY5yjEWv2zTwL9OHhT6w9PkbyYp6hG/boNQl2t5aOy2ndnU+qhzxZ0WAWNo5C9t7UqHYjVWPCn9nsIFj6HOynIj59Z8OGmQK8CZf1w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916775; c=relaxed/simple; bh=9D5UkzbeLAreclXch87c0nr0VRwdKxvB8YfGnMcuC+o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GEds/wnMaEIOErJhf4IkuClt4oNmbTt9D99yWRNCDE+s3Yv5QeC8ZWetrBQvmkKpGsCLCBvEF5A33vFSDwgSiCzJ0nOncz0DQZQ7EuPGQ8L2o1C0S/4C+BNJFTFzqJ3w7yHPK8lscCln2Jr5NUfRG4eteUGcCw1BfI64lpD8efE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=IKZVJ9tp; arc=none smtp.client-ip=209.85.215.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="IKZVJ9tp" Received: by mail-pg1-f199.google.com with SMTP id 41be03b00d2f7-ca6bd8a190cso934412a12.0 for ; Fri, 24 Jul 2026 11:12:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916767; x=1785521567; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=YaTsrIC1+B3I6xZibbV+Jq2ZNzdvSf9XZbVKiCgTfV0=; b=IKZVJ9tpZpNX2EdEfQFbnxEnWxLQGGnROV5QYf0l+bLLhYrMuQNpTnaDhSiXgRclPz Xe55Ow7yeuwRc7OSc4IGpnaxJEEzomELdLrRgIcS/Hn1F2SKytkf9zUYmjKTAVwGEFIs 8HlRaetq/HB464db+VRh4/v+W3K48r+hX0rHv7jjFVBa0/mKnuoFjD5Nl6Qw+VxNvOtD ED34FTHkWOfsozVTuy2vEs5SBxGSlqZ0LYVaiCth5k+EOSv+g1lqNmOsQvf4ta6bNd3b KDoOAq0xBc481bV90KuYl8p1kHDlQyoNcJAND4muPYGiYoqQfID2PSZI+ckbHPjIH9+n NOSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916767; x=1785521567; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YaTsrIC1+B3I6xZibbV+Jq2ZNzdvSf9XZbVKiCgTfV0=; b=P7I6AGCrFsPQc7/kHgfdMUQp2C8sj8ya5LTVq6DZfFx5aRjDRVf2kxbypny48duri2 xw4zc9hS2SXFmEMsJ8McCJ/+cnnGKLUMugHwa2dlOLlvVqth2A9FD1WCOKc4OeXPoeVV DMlHQ1ebIE5HvNDTxcae/nomCEVEvAk6gc98jfRl3LnRcRlP1fjJmHlosDNnwEDQZEBo T949adeUZ0FDxT0C7qg66J2YW+vNLF6BeqNyu8TWIyt9V80NpqJgaiIB6aPdKPL11+G1 zD+l87EAm5lfrGkt7Oj165JS6PI+oLH2hewUQNgj3ZemMfEfQ+AS5Af4/FcMQSZx2/GJ GXrw== X-Gm-Message-State: AOJu0Yz9D2gv1aZkSOl5H/5K/bP7GOcv7cpokAIdZdmv4w9tIgXCcBB1 si+Y6mCoLndrulzaD5IkS6wAATpZDZRvG0xD+jjIiyVKZ3fSfSliW/sEGzq5UYIfp7MaG6YEiiU JLCnuaouvso1W/VYuWbM5sMKt0M7ZCdt7AzhBFlznC+Tra+6Lk55Q5bi2LbuIi6+KDzhaeBI1Qo 121hQ21pwmGADAkTW6JvJ3pfLGf10WxU8DU+AGixEbxY2VTV11 X-Received: from pfbc4.prod.google.com ([2002:a05:6a00:ad04:b0:84a:2e0d:e14f]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:1701:b0:848:44c5:f801 with SMTP id d2e1a72fcca58-84e2b8c0f0fmr9152752b3a.33.1784916766240; Fri, 24 Jul 2026 11:12:46 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:19 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-6-jstultz@google.com> Subject: [PATCH v31 5/9] sched: Switch rq->next_class in proxy_reset_donor() From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , Vasily Gorbik , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similar to commit 04f80f8b12a0 ("sched: Switch rq->next_class on proxy_resched_idle()"), we should also set the rq->next_class when we call proxy_reset_donor(). Fixes: f13beb010e4a ("sched: Have try_to_wake_up() handle return-migration = for PROXY_WAKING case") Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: Vasily Gorbik Cc: kernel-team@android.com --- kernel/sched/core.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 36e1db67a8374..564762ed36f2d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3750,6 +3750,7 @@ static inline void proxy_reset_donor(struct rq *rq) WARN_ON_ONCE(rq->donor =3D=3D rq->curr); =20 put_prev_set_next_task(rq, rq->donor, rq->curr); + rq->next_class =3D rq->curr->sched_class; rq_set_donor(rq, rq->curr); zap_balance_callbacks(rq); resched_curr(rq); --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1436533F59D for ; Fri, 24 Jul 2026 18:12:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916774; cv=none; b=TannmlAsRpPOHDPGnI0ybqsf9Y/YgB+AD/QbaSfU9AfxUQpb5GVvWhJbzToUXVT1rbObknkYP3Kp48JVYCzcU9YANnP4GiSyRxeAGtCm5PJNa0RgR2XUPDGnIpt1nZ5RHPP60KGmwWpLh7Or0PGSdugZKbPlQoz0etz3RQyqzK0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916774; c=relaxed/simple; bh=z0ZiB9KxT5aGyBJU2MIivLCaJvRQgJtm2WHHVgwy/UM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Y0gLIQLsuU1a3GWn6mkjMKeJemBa1+pAdYRT4emNBTw5wm5/MfqVCWhJ37D1F2t+g2DNngEMrmFVcMa+eTKLPqLbFVLlFKfTV/LvfTK2ddTEbxjjiveYj1crdi3rsn0pGBr1+t8BVruX4OtkUu7H3W3m7dinlfEgHwLdKL9In5s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=IKVKja23; arc=none smtp.client-ip=209.85.210.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="IKVKja23" Received: by mail-pf1-f200.google.com with SMTP id d2e1a72fcca58-84a67b16217so1080182b3a.3 for ; Fri, 24 Jul 2026 11:12:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916768; x=1785521568; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=KprZwAc5q4vY44wv9gOnWLxxrrdKt2iGwmqxm7X05+I=; b=IKVKja23POoN098zhv+giSpZz504CNZfF70jlkvB1YdR82TuaLefOYKBHKP6s7vMzb 4BhoDu59OZxFVwOY83+MGDyiI/ZHCf+zS7sqWEruJQulh6NwA94YziXwAhyddU0zAwRW GHQIViw5TxVtlaLUOnV8SvW1CDamrI389pas9qKRsgKC5FcakUJen7XWFqozDEEFHrHw GhsdhnA9XVC6TspL5s142+nTtZe4v4EhhE3pD2UvIapWTeCq9BNFRrScz+HxpVbRndVz VUH9Hc5GNRM0YRiPXnfXfX/lo5XXmNBSl/5wc4/pfVMFoWrzGnOrk4NEUjCjQfWUgFrg aYLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916768; x=1785521568; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KprZwAc5q4vY44wv9gOnWLxxrrdKt2iGwmqxm7X05+I=; b=bemLSCm8Lsgco4kMu/v+L5G6xwvrn7ZUqbQTGZ734XTE3HGo+DzHO6ipV7lE1xrIsg rUY44sydwcok4Di8REXd4atAtfTIHRQi1ebjO43dI4fbFUbwd1pwe/+TcBzQv7QpmmEe SxvhjENTviuprdN5g/XLfHsbt29SsL0e0hl2dldk6pzmcwEVJGdht8ToD3cyhxUMFe8W 5MyfXnrExWD9eGogClFwH/bpas/j0b//kTxC69AdiAx5lJNVXsLYD4Y+iOFoufszGmQm pQrHCHdRFUVQFMFfBl2fUDuCPYag9itNZLpLjfC4lLAQPDsIp7tPMV5vxFg8rLviVVZu 6w0Q== X-Gm-Message-State: AOJu0YyFQdI82uUml/JOgYSvt0GycGGplzKA2fUwwPEAdbOKP6p4jGT7 tlie1WQpBF5/S++jNsw1FylCRDzaiCBc5J2ctO1idCNlv7oM98NZCXZNu2IHjJ17wkTDlVlalyR MMTuSlNlTK63iUKZ7qFRioAtJh24mg7YOcX0wGN9h1HwZfziS4DamZTlgke1ZvkpDKTVekdYTyq pfcoK03CnmMCDGVlQrlwAHYNrJRdeK8uwHy52YanxXrSS35Myz X-Received: from pfbmb13.prod.google.com ([2002:a05:6a00:760d:b0:847:a7b8:c316]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:451a:b0:847:86b0:888c with SMTP id d2e1a72fcca58-84e2bb54669mr9285040b3a.48.1784916767308; Fri, 24 Jul 2026 11:12:47 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:20 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-7-jstultz@google.com> Subject: [PATCH v31 6/9] sched: Break out core of attach_tasks() helper into sched.h From: John Stultz To: LKML Cc: John Stultz , K Prateek Nayak , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pull the core of attach_tasks() out into sched.h so it can be used more generically. Suggested-by: K Prateek Nayak Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/fair.c | 16 +--------------- kernel/sched/sched.h | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee13..6d2fa0cd77779 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11035,21 +11035,7 @@ static int detach_tasks(struct lb_env *env) */ static void attach_tasks(struct lb_env *env) { - struct list_head *tasks =3D &env->tasks; - struct task_struct *p; - struct rq_flags rf; - - rq_lock(env->dst_rq, &rf); - update_rq_clock(env->dst_rq); - - while (!list_empty(tasks)) { - p =3D list_first_entry(tasks, struct task_struct, se.group_node); - list_del_init(&p->se.group_node); - - attach_task(env->dst_rq, p); - } - - rq_unlock(env->dst_rq, &rf); + __attach_tasks(env->dst_rq, &env->tasks); } =20 #ifdef CONFIG_NO_HZ_COMMON diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56acf502ba260..56d9c09c485e6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3100,6 +3100,25 @@ static inline void attach_one_task(struct rq *rq, st= ruct task_struct *p) attach_task(rq, p); } =20 +/* + * __attach_tasks() - attaches a list of tasks (using se.group_node) to + * the new rq + */ +static inline void __attach_tasks(struct rq *rq, struct list_head *tasks) +{ + guard(rq_lock)(rq); + update_rq_clock(rq); + + while (!list_empty(tasks)) { + struct task_struct *p; + + p =3D list_first_entry(tasks, struct task_struct, se.group_node); + list_del_init(&p->se.group_node); + + attach_task(rq, p); + } +} + #ifdef CONFIG_PREEMPT_RT # define SCHED_NR_MIGRATE_BREAK 8 #else --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 064363F39CA for ; Fri, 24 Jul 2026 18:12:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916774; cv=none; b=UDLRgrsGPJInFluO3bOuSlmRAmDE6odrQ/cfnebZM16FS0IPsorXOx6SdMK5gq21uNdwtFnDFhvonO0IxxRIp62/JFgI4TtSuwPFcy0Vor+IZBadM96adhjdxxclEGrS0c6O/JuwuP1y8TEI6PE8zN5gwSkAnJBi/Yw9mQEchGo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916774; c=relaxed/simple; bh=XuZeRBFbPHtq6svLOO6eZbyEBdkxv6YoZQTCKQVEngU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CaTJGpC4QdKXzuAK+mQZ52v8RB4RC0GOfIC2wppSgEqMR49sAB5Pqzxh20Zl9i1tgyUh/Btr94JxCdOhsyE8F7mCw4dbUzz0xXzAlxKguHJSN3QKc/lmOOvCVbH0e+z8HHwVbs3v2wmFmLTyQWVAtUFtGemJwxyXGLCIPV9KgDc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BPsUfOMv; arc=none smtp.client-ip=209.85.215.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BPsUfOMv" Received: by mail-pg1-f198.google.com with SMTP id 41be03b00d2f7-cb835525b10so768931a12.2 for ; Fri, 24 Jul 2026 11:12:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916768; x=1785521568; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=p/WuSLcSEfa3FwxJ8MnqaqaH4ucTQMrCIvVbyMRplsw=; b=BPsUfOMv90rypHhtOy+77tfI3U2UziAoMZ45/x8F1BklEF9ObRH7ERfkOo6u4t9APL MpVn7nVq2j7/sI0ZsqL8sHfx7+6mtQw3W4EGRDDiXly6zcJ2eTXp4MGXPCLVs52qWg9u PmPnRxUEfrdGGKnFB1mmCS9evNZxuEYgYNMbvigZEKP32SoPnfUZ6SjL3tLCkCKxLpNU 2QYPbTLl4A4YFdwjyxI/L5X08gaEwwA7Yk1qnQ4R1jk2u0hqsv02csaBEFXVWWez0+NY hFZ4TZsX89JiFWR5hDySEmKuh/UT0DHFlkyTTXdvmvYZ9ySkyq0W4l7d+IIa3OpyDtRX TnTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916768; x=1785521568; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=p/WuSLcSEfa3FwxJ8MnqaqaH4ucTQMrCIvVbyMRplsw=; b=QGZwbOn21fp64wks8UD21wG6bkY2scBt95WPJPoStG7lTKlJXh61zt+eXD7UbqH7Gg kHzOwcfXgHNq53JwTRvexcOYeZcvo4gZiBe0yREYjFfzLu/oDge8brSXZ3c+2bN0DN/U pSiZtcWGkw01omZAWESWq/5F83Y0Jci5ERyR4yxYzdFhCUGrSQ1a+iUrm/OhxXWiyfHq VzRDYUO82ajwqlTj877dAsqbQRfCTYceIx8PgQyIEc4yodju4IYaeO3H15CKJcpcQPd8 CPZHoq4SVDGbS66gfoQpaMv2D7Cs3WN5VprKZg9CsBTwJDiDtTLbC8B1aiKYPNzSIYZN Lw5A== X-Gm-Message-State: AOJu0YxTmO5+IZDktQpDzCeisDIc+8r2Bp7gPvodJVLpKyVO3CmzkalU vQr7H5W8gNEVhAX+Dp0Z8C9YMIurSSR+/nx8f2jv8H6QddFSCAwE79u+mNrAr5G8hHrRd/85ndd 96hILbudJxtnQmLJ7Db6if5P2/ewi3BDBjyl84MIK5fOf9cTByCDUrxNkYyqunRQs82oStG31Su GqfFoW4wedoLpp/BvByY70wFG6DkKVTX8RXnVvYcvhrE2G96Rg X-Received: from pgc10.prod.google.com ([2002:a05:6a02:2f8a:b0:c80:1fe6:ac57]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:4cc1:b0:3c3:8255:8c4a with SMTP id adf61e73a8af0-3c44aff5387mr9732228637.17.1784916768087; Fri, 24 Jul 2026 11:12:48 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:21 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-8-jstultz@google.com> Subject: [PATCH v31 7/9] sched: Migrate whole chain in proxy_migrate_task() From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Instead of migrating one task each time through find_proxy_task(), we can walk up the blocked_donor ptrs and migrate the entire current chain in one go. This was broken out of earlier patches and held back while the series was being stabilized, but I wanted to re-introduce it. Signed-off-by: John Stultz --- v12: * Earlier this was re-using blocked_node, but I hit a race with activating blocked entities, and to avoid it introduced a new migration_node listhead v18: * Add init_task initialization of migration_node as suggested by Suleiman v22: * Move migration_node under CONFIG_SCHED_PROXY_EXEC as suggested by K Prateek v25: * Use se.group_node instead of adding migration_node, as suggsested by K Prateek * Integrated attach_tasks() cleanups suggested by K Prateek Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 564762ed36f2d..1bf60d78c9208 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6819,9 +6819,9 @@ static void proxy_migrate_task(struct rq *rq, struct = rq_flags *rf, __must_hold(__rq_lockp(rq)) { struct rq *target_rq =3D cpu_rq(target_cpu); + LIST_HEAD(migrate_list); =20 lockdep_assert_rq_held(rq); - WARN_ON(p =3D=3D rq->curr); /* * Since we are migrating a blocked donor, it could be rq->donor, * and we want to make sure there aren't any references from this @@ -6834,13 +6834,20 @@ static void proxy_migrate_task(struct rq *rq, struc= t rq_flags *rf, * before we release the lock. */ proxy_resched_idle(rq); - - deactivate_task(rq, p, DEQUEUE_NOCLOCK); - proxy_set_task_cpu(p, target_cpu); - + for (; p; p =3D p->blocked_donor) { + WARN_ON(p =3D=3D rq->curr); + deactivate_task(rq, p, DEQUEUE_NOCLOCK); + proxy_set_task_cpu(p, target_cpu); + /* + * We can re-use se.group_node to migrate the thing, + * because @p is deactivated (won't be balanced) and + * we hold the rq_lock. + */ + list_add(&p->se.group_node, &migrate_list); + } proxy_release_rq_lock(rq, rf); =20 - attach_one_task(target_rq, p); + __attach_tasks(target_rq, &migrate_list); =20 proxy_reacquire_rq_lock(rq, rf); } --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 39B1E3AEF3F for ; Fri, 24 Jul 2026 18:12:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916776; cv=none; b=QANVzz8KZtDDnR/4rv7//BYAc8EqofOEsGf0FW57b4VrXS1ucbM+88MP3ym8VnHmLYYwped5DF9lIbPe0FzVhSspylinQUyCNvzukENKXHuZvZjzfIsDL05HGtits71BWapvcd99dipMDOWGdAZ0wxVnQHsLfEgs0lhT8EkHl9M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916776; c=relaxed/simple; bh=EAt9XKCuOBZM/9QeGVX+8NFf1RidHlUL67c2HPIzqy0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SQao/397U/8PCC2vOxfuuBTcbctsaZRgbgtNQh0PFAh5fIBj8doXmIiGIjISeTFOPFwxtCI8OMbX0k4nPqwBIAJF155OqEGEZZYuA8pk5f32+fZlzPKm1MwryCjralRtpPngFxNH/ygNP7ux24zuCs3+zM5myLF/Z3n1uQF72sE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=f8BcwuKM; arc=none smtp.client-ip=209.85.210.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="f8BcwuKM" Received: by mail-pf1-f199.google.com with SMTP id d2e1a72fcca58-84842381150so1087727b3a.3 for ; Fri, 24 Jul 2026 11:12:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916769; x=1785521569; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=EsmahXN3w4fGZ/2LSwiKUMnDuIvv0mLAiQp91z9Ua30=; b=f8BcwuKMGK6SM6w6wJlJzp9LbJ0p4ylDDigiEm1D1tK6SUZdsaOu5nYvFzr1LMTVw/ 8UDg5BQqSdL5xQRWtxg4EqYsO9VeMeCegEZBhAgE7JBIG1w/oCMhi5g6u2D4dwSp4hYD WqlV5ueThMlRgf/AMirACTW+Yl0mxNicLR3jEbn/L+rUYSCtSqIhVo3JYkuZXUDmg/K/ Ih99KkNA447zYArus9lV2cs/eIZ0w3dTvnQOaOS1cHKdfFFtZ0gZnE1Rkbv7TWenMxjw M885iSuu4ug7pP3ImJT76p51WpFchNjk2sfJm82vBcAgGDoI+R0KhP2hsw/r3hzT8K6N bBXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916769; x=1785521569; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EsmahXN3w4fGZ/2LSwiKUMnDuIvv0mLAiQp91z9Ua30=; b=kP09pmpYFrLABeVMsDztPiLR5qvw2KTCb7viEhhlppEvzce9aze3jZYTNWwtM00kU3 uF7IDmWW1LtUn4lghSiYl3Mnhg74he5v/c3Dr7v+7EpSQ1XywSiTE39fCfbYnER+/5H6 qTik7jmUfW8nUZdtjKH9aDU7k1NZ5nAnCbmslOU7BVVtel4Y8aB1otbK0/mbGiy968Ah sqz22gzzeZJuocEGf2T3K2K2sJYPJ/SsrUuoVvqbpIncL34CLa1jXzVskg6JQmNtUG+2 aKO5l03ACvhTxDhXnXh+rs42AMWaY+lQGJyqjWSfUU7kEGce3cVD8fWmkd8JKRVOhwTJ 7KKg== X-Gm-Message-State: AOJu0Yw3Tduw/PQ4qVj1HVn8K7gqlM3j9txBdgmYFr2CBfGokcuR98Vx BpBDVXj9ueuw8GfHI4u3/Rzqmj1aDyR83PuBJn4dZfH86p4U4r+uGoM9SKcjwYrBCMB9AMRB3KN R6fNAtqh64nf6ZYbZKmeEy90I+xWmEpcZIq8h9B73uPbYA2Yagn4v29UssSMZ8sq689AWZP1ScL bORGN2e/4nevuMl7ABR2e41k3vEok0b7M0Scb2RI0HIH/LMBVN X-Received: from pfbfi42.prod.google.com ([2002:a05:6a00:39aa:b0:84a:2dcb:5abe]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2e21:b0:848:5ca1:bae0 with SMTP id d2e1a72fcca58-84e2bb3728fmr8694708b3a.42.1784916768876; Fri, 24 Jul 2026 11:12:48 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:22 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-9-jstultz@google.com> Subject: [PATCH v31 8/9] sched: Add deactivated (sleeping) owner handling to find_proxy_task() From: John Stultz To: LKML Cc: Peter Zijlstra , Juri Lelli , Valentin Schneider , "Connor O'Brien" , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , Andrea Righi , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra If the blocked_on chain resolves to a sleeping owner, deactivate the donor task, and enqueue it on the sleeping owner task. Then re-activate it later when the owner is woken up. NOTE: This has been particularly challenging to get working properly, and some of the locking is particularly awkward. I'd very much appreciate review and feedback for ways to simplify this. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Juri Lelli Signed-off-by: Valentin Schneider Signed-off-by: Connor O'Brien [jstultz: This was broken out from the larger proxy() patch] Signed-off-by: John Stultz --- v5: * Split out from larger proxy patch v6: * Major rework, replacing the single list head per task with per-task list head and nodes, creating a tree structure so we only wake up descendants of the task woken. * Reworked the locking to take the task->pi_lock, so we can avoid mid-chain wakeup races from try_to_wake_up() called by the ww_mutex logic. v7: * Drop unnecessary __nested lock annotation, as we already drop the lock prior. * Add comments on #else & #endif lines, and clearer function names, and commit message tweaks as suggested by Metin Kaya * Move activate_blocked_entities() call from ttwu_queue to try_to_wake_up() to simplify locking. Thanks to questions from Metin Kaya * Fix irqsave/irqrestore usage now we call this outside where the pi_lock is held * Fix activate_blocked_entitites not preserving wake_cpu * Fix for UP builds v8: * Minor checkpatch fixup * Drop proxy_deactivate and cleanups suggested by Metin v9: * Fix bug causing possibly uninitialized cpu value to be used with activate_blocked_entities() * Improved comment around preserving wake_cpu suggested by Metin * Add additional lockdep asserts, suggested by Metin * Tweaked placement of lockdep assert, suggested by Metin * Fixed comment referring to structure entry name * Fix to call proxy_resched_idle() _prior_ to calling proxy_enqueue_on_owner() where we deactivate the task, this avoids stale references to rq_selected() when the task may have been migrated to another rq. * Fix to remove the blocked_head list at the start of activate_blocked_entities() so we only do a finite amount of work, avoiding a potential livelock of two cpus removing and adding tasks to the list at the same time if the owner went back to sleep while blocked entities were being woken. v11: * Big rework to get rid of recursion. Had to add another list item to the task_stuct to do this as we are in atomic context and cannot allocate memory while activating blocked entities. Will need to watch carefully for bugs, as switching to a list_head in the task_struct instead of a pointer on the stack opens up the potential for races on the shared state, but I think I've got the locking sorted. * Moved proxy_set_task_cpu helper to earlier in the series * Minor rework for try_to_deactivate_task changes * Minor variable name cleanups suggested by Metin v13: * Switch to use donor from next for proxy_enqueue_on_owner * Switch to using block_task instead of deactivate_task v14: * Ensure we call block_task() last in proxy_enqueue_on_owner and not touch it again to avoid races where it might be activated on another cpu * Make sure we activate blocked_entities when we exit from ttwu * Fix to enqueue the last task in the chain (p) on the blocked owner instead of donor, so that we preserve the chain structure so mid-chain wakeups propagate properly * Rework of sleeping_owner handling so that we properly deal with delayed-dequeued (sched_delayed) tasks (also removes now unused proxy_deactivate() logic) v15: * Rework do_activate_task to be activate_task() and have it call __activate_task(), suggested by Carlos Llamas * Put task_struct additions under CONFIG_SCHED_PROXY_EXEC v16: * Rework do_activate_blocked_waiter locking to use scoped_guard * Rework find_proxy_task() logic to use guard v18: * Integrate Suleiman's suggested optimization to check sleeping owner status before task_cpu, to avoid unnecssarily proxy-migrating tasks to then just dequeue them. * Add on_cpu check to fix for a very hard to reproduce race where a late blocked_task_activation() happens while the task is already on_cpu elsewhere. When we get to __schedule() we block the task (!on_rq) but haven't yet switched away. blocked_task_activation() would then incorrectly activate on a different runqueue. * Add init_task initialization for sleeping owner lists, as suggested by Suleiman v19: * Build fixup for !CONFIG_SMP v22: * Rework to avoid gotos in guard() scopes, using break and switch() on action values. Suggested by K Prateek. v25: * Fix elevated nr_uninterruptible and nr_iowait counts, which were causing bad loadavg values. Reported and fixed by David Stevens v28: * In __proxy_remove_from_sleeping_owner() we call put_task(), which might free the owner while we are still holing the owners->blocked_lock. So be sure to get_task()/put_task() around the owner usage to ensure we don't prematurely free the task. * Also reorder the put_task/unlock lines in activate_blocked_waiters(), to avoid a similar issue v30: * Optimize activate_blocked_waiters() so we don't do so much unnecessary work when proxy-exec is enabled. If there are no blocked waiters, we can return early. v31: * K Prateek's noticed a potential race where if a sleeping owner is woken in paralell with the waiter enquing itself on the sleeping owner, the waiting task could get stuck on that owner until it sleeps and is woken up again. So double check after we grab the blocked_lock to avoid this. * Maria Yu and Tengfei Fan reported an issue with rq->nr_iowait values getting out of balance, and provided a helpful reproducer. To fix this, make sure we do the atomic_dec() in do_activate_blocked_waiter() before we call proxy_set_task_cpu() so we always dec on the same rq we inc'ed. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- include/linux/sched.h | 7 + init/init_task.c | 6 + kernel/fork.c | 6 + kernel/sched/core.c | 317 +++++++++++++++++++++++++++++++++++++++--- 4 files changed, 315 insertions(+), 21 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d10..8c2ba6dce58f4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1251,6 +1251,13 @@ struct task_struct { =20 struct mutex *blocked_on; /* lock we're blocked on */ raw_spinlock_t blocked_lock; +#ifdef CONFIG_SCHED_PROXY_EXEC + struct list_head blocked_head; /* tasks blocked on this task */ + struct list_head blocked_node; /* our entry on someone elses blocked_he= ad */ + /* Node for list of tasks to process blocked_head list for blocked entiti= y activations */ + struct list_head blocked_activation_node; + struct task_struct *sleeping_owner; /* task our blocked_node is enqueued= on */ +#endif =20 /* * The task that is boosting this task; a back link for the current diff --git a/init/init_task.c b/init/init_task.c index b67ef6040a655..809282a2741d7 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -211,6 +211,12 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES)= =3D { &init_task.alloc_lock), #endif .blocked_donor =3D NULL, +#ifdef CONFIG_SCHED_PROXY_EXEC + .blocked_head =3D LIST_HEAD_INIT(init_task.blocked_head), + .blocked_node =3D LIST_HEAD_INIT(init_task.blocked_node), + .blocked_activation_node =3D LIST_HEAD_INIT(init_task.blocked_activation_= node), + .sleeping_owner =3D NULL, +#endif #ifdef CONFIG_RT_MUTEXES .pi_waiters =3D RB_ROOT_CACHED, .pi_top_task =3D NULL, diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5a..6dfde115fe34e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2247,6 +2247,12 @@ __latent_entropy struct task_struct *copy_process( =20 p->blocked_on =3D NULL; /* not blocked yet */ p->blocked_donor =3D NULL; /* nobody is boosting p yet */ +#ifdef CONFIG_SCHED_PROXY_EXEC + INIT_LIST_HEAD(&p->blocked_head); + INIT_LIST_HEAD(&p->blocked_node); + INIT_LIST_HEAD(&p->blocked_activation_node); + p->sleeping_owner =3D NULL; +#endif =20 #ifdef CONFIG_BCACHE p->sequential_io =3D 0; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1bf60d78c9208..d813a9c2f0c35 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2216,7 +2216,7 @@ inline bool dequeue_task(struct rq *rq, struct task_s= truct *p, int flags) return p->sched_class->dequeue_task(rq, p, flags); } =20 -void activate_task(struct rq *rq, struct task_struct *p, int flags) +static inline void __activate_task(struct rq *rq, struct task_struct *p, i= nt flags) { if (task_on_rq_migrating(p)) flags |=3D ENQUEUE_MIGRATED; @@ -2227,6 +2227,71 @@ void activate_task(struct rq *rq, struct task_struct= *p, int flags) ASSERT_EXCLUSIVE_WRITER(p->on_rq); } =20 +#ifdef CONFIG_SCHED_PROXY_EXEC +static inline +void __proxy_remove_from_sleeping_owner(struct task_struct *owner, struct = task_struct *p) +{ + lockdep_assert_held(&owner->blocked_lock); + + if (p->sleeping_owner =3D=3D owner) { + list_del_init(&p->blocked_node); + WRITE_ONCE(p->sleeping_owner, NULL); + put_task_struct(owner); // matches get in proxy_enqueue_on_owner + } +} + +static inline void proxy_remove_from_sleeping_owner(struct task_struct *p) +{ + struct task_struct *owner =3D READ_ONCE(p->sleeping_owner); + + if (owner) { + /* + * __proxy_remove_from_sleeping_owner() does a + * put on owner to match the get done in + * proxy_enqueue_on_owner(). If that put is the + * last one and it frees owner, we'd be freeing + * a lock we held. So get/put owner around its + * usage her to ensure that doesn't happen. + */ + get_task_struct(owner); + raw_spin_lock(&owner->blocked_lock); + __proxy_remove_from_sleeping_owner(owner, p); + raw_spin_unlock(&owner->blocked_lock); + put_task_struct(owner); + } +} + +void activate_task(struct rq *rq, struct task_struct *p, int en_flags) +{ + if (!sched_proxy_exec()) { + __activate_task(rq, p, en_flags); + return; + } + + lockdep_assert_rq_held(rq); + proxy_remove_from_sleeping_owner(p); + /* + * By calling __activate_task() with blocked_lock held, we + * order against the find_proxy_task() blocked_task case + * such that no more blocked tasks will be enqueued on p + * once we release p->blocked_lock. + */ + raw_spin_lock(&p->blocked_lock); + WARN_ON(task_cpu(p) !=3D cpu_of(rq)); + __activate_task(rq, p, en_flags); + raw_spin_unlock(&p->blocked_lock); +} +#else +static inline void proxy_remove_from_sleeping_owner(struct task_struct *p) +{ +} + +void activate_task(struct rq *rq, struct task_struct *p, int en_flags) +{ + __activate_task(rq, p, en_flags); +} +#endif + void deactivate_task(struct rq *rq, struct task_struct *p, int flags) { WARN_ON_ONCE(flags & DEQUEUE_SLEEP); @@ -3756,6 +3821,169 @@ static inline void proxy_reset_donor(struct rq *rq) resched_curr(rq); } =20 +static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) +{ + unsigned int wake_cpu; + + /* + * Since we are enqueuing a blocked task on a cpu it may + * not be able to run on, preserve wake_cpu when we + * __set_task_cpu so we can return the task to where it + * was previously runnable. + */ + wake_cpu =3D p->wake_cpu; + __set_task_cpu(p, cpu); + p->wake_cpu =3D wake_cpu; +} + +static void do_activate_blocked_waiter(struct rq *target_rq, struct task_s= truct *p, int en_flags) +{ + unsigned int state; + struct rq_flags rf; + int target_cpu =3D cpu_of(target_rq); + + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + state =3D READ_ONCE(p->__state); + /* Avoid racing with ttwu */ + if (state =3D=3D TASK_WAKING) + return; + + if (READ_ONCE(p->on_rq)) { + /* + * We raced with a non mutex handoff activation of p. + * That activation will also take care of activating + * all of the tasks after p in the blocked_head list, + * so we're done here. + */ + return; + } + if (task_on_cpu(task_rq(p), p)) { + /* + * Its possible this activation is very late, and + * we already were woken up and are running on a + * different cpu. If that task blocked, it could be + * dequeued (so on_rq =3D=3D 0), but still on_cpu. + * Bail in this case, as we definitely don't want to + * activate a task when its on_cpu elsewhere. + */ + return; + } + /* + * Have to make sure we handle nr_iowait adjustment before + * we call proxy_set_task_cpu() to ensure we are adjusting + * the same runqueue we left (where block_task() + * incremented nr_iowait). + */ + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } + proxy_set_task_cpu(p, target_cpu); + rq_lock_irqsave(target_rq, &rf); + /* + * proxy_enqueue_on_owner() called block_task() which + * increments nr_uninterruptible, so we need to reverse + * that when we activate the blocked waiter + */ + if (p->sched_contributes_to_load) + target_rq->nr_uninterruptible--; + update_rq_clock(target_rq); + activate_task(target_rq, p, en_flags); + resched_curr(target_rq); + rq_unlock_irqrestore(target_rq, &rf); + } +} + +static void activate_blocked_waiters(struct rq *target_rq, + struct task_struct *owner, + int wake_flags) +{ + struct list_head bal_head; + unsigned long flags; + int en_flags =3D ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK; + + if (!sched_proxy_exec()) + return; + + /* + * A whole bunch of waiting donor tasks back this blocked + * lock owner task, wake them all up to give this task its + * 'fair' share. + * + * This is a little unique here and the locking is messy. + * At this point we only hold the blocked_lock, so the + * owner task may be able to run and do all sorts of + * things while we are processing the blocked_head list, + * including going back to sleep, which can cause tasks + * to be added to the owners->blocked_head while we are + * processing it! + * Thus, we pull the entire list off the owner->blocked_head + * here so that we will only process a finite amount of + * tasks. Tasks added after this will be processed by the + * future wake events. + * Even though we have pulled the list off the blocked_head + * the removed list is *still* "owned" and serialized by + * the owner->blocked_lock! As we have to serialize against + * mid-chain wakeups, who may try to remove themselves from + * the list. + */ + raw_spin_lock_irqsave(&owner->blocked_lock, flags); + if (!list_empty(&owner->blocked_activation_node)) { + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + return; + } + + if (list_empty(&owner->blocked_head)) { + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + return; + } + + get_task_struct(owner); + INIT_LIST_HEAD(&bal_head); + list_add_tail(&owner->blocked_activation_node, &bal_head); + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + + if (wake_flags & WF_MIGRATED) + en_flags |=3D ENQUEUE_MIGRATED; + + while (!list_empty(&bal_head)) { + struct list_head tmp_head; + + INIT_LIST_HEAD(&tmp_head); + owner =3D list_first_entry(&bal_head, struct task_struct, blocked_activa= tion_node); + + raw_spin_lock_irqsave(&owner->blocked_lock, flags); + list_replace_init(&owner->blocked_head, &tmp_head); + list_del_init(&owner->blocked_activation_node); + while (!list_empty(&tmp_head)) { + struct task_struct *p; + + p =3D list_first_entry(&tmp_head, + struct task_struct, + blocked_node); + WARN_ON(p =3D=3D owner); + WARN_ON(p->sleeping_owner !=3D owner); + __proxy_remove_from_sleeping_owner(owner, p); + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + + do_activate_blocked_waiter(target_rq, p, en_flags); + + raw_spin_lock_irqsave(&p->blocked_lock, flags); + if (list_empty(&p->blocked_activation_node)) { + get_task_struct(p); + list_add_tail(&p->blocked_activation_node, &bal_head); + } + raw_spin_unlock_irqrestore(&p->blocked_lock, flags); + + raw_spin_lock_irqsave(&owner->blocked_lock, flags); + } + raw_spin_unlock_irqrestore(&owner->blocked_lock, flags); + put_task_struct(owner); // put matches get prior to adding to local bal_= head + } +} + +static inline struct task_struct *proxy_resched_idle(struct rq *rq); + /* * Checks to see if task p has been proxy-migrated to another rq * and needs to be returned. If so, we deactivate the task here @@ -3800,6 +4028,10 @@ static inline bool proxy_needs_return(struct rq *rq,= struct task_struct *p) { return false; } + +static inline void activate_blocked_waiters(struct rq *target_rq, + struct task_struct *owner, + int wake_flags) {} #endif /* CONFIG_SCHED_PROXY_EXEC */ =20 static void @@ -3873,8 +4105,10 @@ static int ttwu_runnable(struct task_struct *p, int = wake_flags) =20 update_rq_clock(rq); if (p->is_blocked) { - if (p->se.sched_delayed) + if (p->se.sched_delayed) { + proxy_remove_from_sleeping_owner(p); enqueue_task(rq, p, ENQUEUE_NOCLOCK | ENQUEUE_DELAYED); + } if (proxy_needs_return(rq, p)) return 0; } @@ -3903,13 +4137,19 @@ void sched_ttwu_pending(void *arg) update_rq_clock(rq); =20 llist_for_each_entry_safe(p, t, llist, wake_entry.llist) { + int wake_flags; if (WARN_ON_ONCE(p->on_cpu)) smp_cond_load_acquire(&p->on_cpu, !VAL); =20 if (WARN_ON_ONCE(task_cpu(p) !=3D cpu_of(rq))) set_task_cpu(p, cpu_of(rq)); =20 - ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); + wake_flags =3D p->sched_remote_wakeup ? WF_MIGRATED : 0; + ttwu_do_activate(rq, p, wake_flags, &rf); + rq_unlock(rq, &rf); + activate_blocked_waiters(rq, p, wake_flags); + rq_lock(rq, &rf); + update_rq_clock(rq); } =20 /* @@ -4416,6 +4656,7 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) ttwu_queue(p, cpu, wake_flags); } out: + activate_blocked_waiters(cpu_rq(task_cpu(p)), p, wake_flags); if (success) ttwu_stat(p, task_cpu(p), wake_flags); =20 @@ -6731,21 +6972,6 @@ static bool try_to_block_task(struct rq *rq, struct = task_struct *p, } =20 #ifdef CONFIG_SCHED_PROXY_EXEC -static inline void proxy_set_task_cpu(struct task_struct *p, int cpu) -{ - unsigned int wake_cpu; - - /* - * Since we are enqueuing a blocked task on a cpu it may - * not be able to run on, preserve wake_cpu when we - * __set_task_cpu so we can return the task to where it - * was previously runnable. - */ - wake_cpu =3D p->wake_cpu; - __set_task_cpu(p, cpu); - p->wake_cpu =3D wake_cpu; -} - static inline struct task_struct *proxy_resched_idle(struct rq *rq) { put_prev_set_next_task(rq, rq->donor, rq->idle); @@ -6852,6 +7078,29 @@ static void proxy_migrate_task(struct rq *rq, struct= rq_flags *rf, proxy_reacquire_rq_lock(rq, rf); } =20 +static void proxy_enqueue_on_owner(struct rq *rq, struct task_struct *owne= r, + struct task_struct *p) +{ + lockdep_assert_rq_held(rq); + lockdep_assert_held(&owner->blocked_lock); + /* + * ttwu_activate() will pick them up and place them on whatever rq + * @owner will run next. + */ + WARN_ON(p =3D=3D owner); + WARN_ON(!p->on_rq); + WARN_ON(p->sleeping_owner); + get_task_struct(owner); + WRITE_ONCE(p->sleeping_owner, owner); + /* + * ttwu_do_activate must not have a chance to activate p + * elsewhere before it's fully extricated from its old rq. + */ + list_add(&p->blocked_node, &owner->blocked_head); + proxy_resched_idle(rq); + block_task(rq, p, READ_ONCE(p->__state)); +} + /* * Find runnable lock owner to proxy for mutex blocked donor * @@ -6938,11 +7187,37 @@ find_proxy_task(struct rq *rq, struct task_struct *= donor, struct rq_flags *rf) } =20 if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) { - /* XXX Don't handle blocked owners/delayed dequeue yet */ + /* + * rq->curr must not be added to the blocked_head list or else + * ttwu_do_activate could enqueue it elsewhere before it switches + * out here. The approach to avoid this is the same as in the + * migrate_task case. + */ if (curr_in_chain) return proxy_resched_idle(rq); - __clear_task_blocked_on(p, NULL); - goto deactivate; + /* + * If !@owner->on_rq, holding @rq->lock will not pin the task, + * so we cannot drop @mutex->wait_lock until we're sure its a blocked + * task on this rq. + * + * We use @owner->blocked_lock to serialize against ttwu_activate(). + * Either we see its new owner->on_rq or it will see our list_add(). + */ + WARN_ON(owner =3D=3D p); + raw_spin_unlock(&p->blocked_lock); + raw_spin_lock(&owner->blocked_lock); + /* + * Before actually adding to the sleeping owner, double check + * we didn't race with an owner wakeup before grabbing the + * owner's blocked_lock. + */ + if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) + proxy_enqueue_on_owner(rq, owner, p); + + raw_spin_unlock(&owner->blocked_lock); + raw_spin_lock(&p->blocked_lock); + + return NULL; /* retry task selection */ } =20 owner_cpu =3D task_cpu(owner); --=20 2.55.0.229.g6434b31f56-goog From nobody Fri Jul 24 20:51:03 2026 Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48157372B27 for ; Fri, 24 Jul 2026 18:12:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.197 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916774; cv=none; b=ipnoJ34L7qQeeuF+Xr7oUGk9B5ysBv0fTAU/2jwCm+DICdkw2cb7GILZ5B868BDpGDmjMoQc2AYk5QM/3ghn6ApHmENZ6RBGpPQFxJtXNHVCz0d7k05nfEGLeRhp4KCtGHVC11fp6/5ExcpbPFLhA7iRPGK7HQlkxKwIeQvEq0k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784916774; c=relaxed/simple; bh=OqJ39F9aEBeNkacHvljqDrKwYBAVIL2tWzg9ofeIMJI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=c1DMAQMiWT5hSc2W9vxUnmwLtFw3xlX5Y8j9py8Dt4vdkZa85+N0Iu3u8no9wLsfMMuAEpzpkER4gMFJHWvh/HuIv3AFIYz7Ft5MMi3e4Mqd/8xNjV7S5dip2wk5Rumiq9NSoEnuwmnKjet189dSVoCi/3vUC4e2fzrfDWtD9W4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GtwD98yL; arc=none smtp.client-ip=209.85.210.197 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GtwD98yL" Received: by mail-pf1-f197.google.com with SMTP id d2e1a72fcca58-84842381150so1087749b3a.3 for ; Fri, 24 Jul 2026 11:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784916770; x=1785521570; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=kWbB0NWfOkfjR5Zf1PJALEvxKOiKfMCKTW3apXjrlUY=; b=GtwD98yLOEDzHdjkk85SV26UPEBIWT0faexhv8GyGiYs346LqllGmpiqY/ekfT1TXL 0Z6QBJVUb1ed2zrINCBHLX4y/g1PeuxIPXK3CZu1QZFdeefeArH0KVfPoQknIteBvkjC TQJM+9g4K5HAkwQkhTWV+wbANBdAHnFdlwZah79x9igS9jSqy51LOZR33HjSQyYzUjwq 6XWv7s+GaTOBR5DS1u0n1EcRJtmVZiyFDYgEcghGEAHGNAjIrCAVs5kcpOjnTbKGR2Mp CjGET/mr08s85xaylxTCRx1kp73IuZJFZGvqyV2sUFPB1PH7l2ZbNJmmh3ppc6C2CtaI v5nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784916770; x=1785521570; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=kWbB0NWfOkfjR5Zf1PJALEvxKOiKfMCKTW3apXjrlUY=; b=dqwjZQ3nfnlsGfaTApElLxKbqLaNAD79ma2HD+nhH0pACYvkIICYENtqn/BXY8KMEw P8MHnAnQ8/imVAw+xZU8mcSEiOt4G7zh/NwyPZnSxzMscLymu3AeLMAKv5KKy756iAFU qqCJ36odf/KbfIdrRXp9XjQZWQoMWx2PbTx+f92BfNbfwKLctcLQ6lXvtrTdWsIb32Ua dpvbJxDPLz2t25FCK+MewyxNLIk/SXRqRfe29tS+dwv9gn+yYAju30ZRF5Un/WybWQRT R69SEmFIzZZbx3ZLcHmDb7ih4+EShIWW6Rqw93XkTmxwmVSOaGX7MLLBhq8xSH2dkJuU UqeQ== X-Gm-Message-State: AOJu0YwaaQ5amdJWJpWRFBw87webi6VxngBFPMyUg6/uQ2EgGY31W1tM cm4H+7o79d/TknUlyIME1K/txilGXW1HSt3xOxi9dssJMngcs7hTlC+HOR9+9YbiZomBLL5xnQN T/w+WQ+Wkkdm7PI4Y3rSJRre8Pbs9qiUiY28C9keYQq7fnMqZGyVGvIHH0RSFlCO76qdqIxxO4J 7V1awwm0Ihr6Q/7eB5wPpYDrIhB6G7oyY0+KKz3lAeMB3aqqBd X-Received: from pfbhq5.prod.google.com ([2002:a05:6a00:6805:b0:848:416d:e7f3]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4b56:b0:84e:538:476c with SMTP id d2e1a72fcca58-84e2bbf9189mr8753440b3a.67.1784916769747; Fri, 24 Jul 2026 11:12:49 -0700 (PDT) Date: Fri, 24 Jul 2026 18:12:23 +0000 In-Reply-To: <20260724181238.3445275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260724181238.3445275-1-jstultz@google.com> X-Mailer: git-send-email 2.55.0.229.g6434b31f56-goog Message-ID: <20260724181238.3445275-10-jstultz@google.com> Subject: [PATCH v31 9/9] sched: Distinguish proxy activations from wakeups From: John Stultz To: LKML Cc: Andrea Righi , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Andrea Righi Sleeping-owner handling reactivates blocked waiters with wakeup enqueue semantics so scheduling classes can restore their runnable accounting. Unlike a normal wakeup, the waiter remains blocked and is only made runnable to donate its scheduling context. Add ENQUEUE_PROXY to identify this activation and set it alongside ENQUEUE_WAKEUP. Consume the internal marker in sched_ext and suppress SCX_ENQ_WAKEUP before invoking BPF so the activation can be handled as a blocked proxy donor. Signed-off-by: Andrea Righi Signed-off-by: John Stultz --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: Andrea Righi Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 2 +- kernel/sched/ext/ext.c | 8 ++++++++ kernel/sched/sched.h | 2 ++ 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d813a9c2f0c35..c54e9fedc9cd8 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3900,7 +3900,7 @@ static void activate_blocked_waiters(struct rq *targe= t_rq, { struct list_head bal_head; unsigned long flags; - int en_flags =3D ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK; + int en_flags =3D ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK | ENQUEUE_PROXY; =20 if (!sched_proxy_exec()) return; diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c index e3fa7b2fac9df..834c8b6dae6c9 100644 --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -2021,6 +2021,14 @@ static void enqueue_task_scx(struct rq *rq, struct t= ask_struct *p, int core_enq_ int sticky_cpu =3D p->scx.sticky_cpu; u64 enq_flags =3D core_enq_flags | rq->scx.extra_enq_flags; =20 + /* + * Sleeping-owner activation uses wakeup semantics for the core + * scheduling classes, but the donor remains blocked. Expose it to BPF + * as a blocked-donor admission rather than a full wakeup. + */ + if (core_enq_flags & ENQUEUE_PROXY) + enq_flags &=3D ~(ENQUEUE_PROXY | SCX_ENQ_WAKEUP); + if (enq_flags & ENQUEUE_WAKEUP) rq->scx.flags |=3D SCX_RQ_IN_WAKEUP; =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56d9c09c485e6..e38988574cfa2 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2540,6 +2540,7 @@ extern const u32 sched_prio_to_wmult[40]; * ENQUEUE_REPLENISH - CBS (replenish runtime and postpone deadline) * ENQUEUE_MIGRATED - the task was migrated during wakeup * ENQUEUE_RQ_SELECTED - ->select_task_rq() was called + * ENQUEUE_PROXY - activate a blocked donor behind a waking owner * * XXX SAVE/RESTORE in combination with CLASS doesn't really make sense, b= ut * SCHED_DEADLINE seems to rely on this for now. @@ -2571,6 +2572,7 @@ extern const u32 sched_prio_to_wmult[40]; #define ENQUEUE_MIGRATED 0x00040000 #define ENQUEUE_INITIAL 0x00080000 #define ENQUEUE_RQ_SELECTED 0x00100000 +#define ENQUEUE_PROXY 0x00200000 =20 #define RETRY_TASK ((void *)-1UL) =20 --=20 2.55.0.229.g6434b31f56-goog