From nobody Tue Oct 7 00:21:15 2025 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A18462F363A; Wed, 16 Jul 2025 10:19:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752661153; cv=none; b=hY85eWQnUmbKiac9bCnKhffEaKMzaNbeaiJ+HQxlEv2XcfWHGrT9sAkp6D7bdQBCcK244fu0xTE8+ZJI4IAlxge82Je5hTiQEgj2owPgCz1FBZdw/lJ1VcgqSYPiuGIbt08tbD+n0cF0aaHE3KTHN8uMyTxxDawKHU7IUY3Gqyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752661153; c=relaxed/simple; bh=l1zhxn/BfyNCWA3mmtvAUnxRgL2+WFtTa3GOqVM/z48=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=alh59/d6+qUGOtOYQeHO2p7lGuQS24LOSx/2l2Xb9453ybjPFC7djqQAZwfgy/kPDtE/Q0dApLNt/kSqSbG64KS7RNVUOhB3tS1T1uiU83hsUZo3DIHrulKOZd9d2fDqlRraBQhEOAscsfAWo/Z/5wra7SoouPgtgYacyVVtfjA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=arKt8cgX; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=RKQO9kQL; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="arKt8cgX"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="RKQO9kQL" Date: Wed, 16 Jul 2025 10:19:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1752661150; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nmaARJhqZxghvBxb4vLY1NO9AWORPBtjHNauEGEbdgc=; b=arKt8cgX9KaUNMgtPWZDNL2QeVqNl2YAYLBZgszNaHmM0ck3NVghjbuUgT7ueXlmubbzzN QTGVkNU/0bRzqOUTmf12dy44v9kb1DU2Z5GFiWYbNsWMVqdx1kdyWcIHyCr70nd7tzVuaT Dp4j0VzlnHjVjyIeqmjMzcI3Rm54CsDIt51Y9RujYzaZ7c8mhV6yb+gZlQ21CcIDkfY795 x0D1V1v1hVb+dTRKZWiLBo4Hc6GDpa7DUEHmPbVko8mzlKPLprNtSaTC2Ro0b6VOTzqywH Q4PnMmE/FPmqw1u6ETTt9cO/bujrgErhDTvx7W7zYNAZ5/ytN1xQ/PWtps+a0A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1752661150; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nmaARJhqZxghvBxb4vLY1NO9AWORPBtjHNauEGEbdgc=; b=RKQO9kQL33HawxR/9RoJ5nJ2rEZjYhGEh1x7WkU7j+aabvAtGON8rmay+YPLZu6GNGwvn4 IOg29+Ya1l4YcgCw== From: "tip-bot2 for Valentin Schneider" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched: Fix proxy/current (push,pull)ability Cc: Valentin Schneider , "Connor O'Brien" , John Stultz , "Peter Zijlstra (Intel)" , K Prateek Nayak , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20250712033407.2383110-8-jstultz@google.com> References: <20250712033407.2383110-8-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <175266114881.406.11249039491907002913.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/core branch of tip: Commit-ID: be39617e38e0b1939a6014d77ee6f14707d59b1b Gitweb: https://git.kernel.org/tip/be39617e38e0b1939a6014d77ee6f1470= 7d59b1b Author: Valentin Schneider AuthorDate: Sat, 12 Jul 2025 03:33:48=20 Committer: Peter Zijlstra CommitterDate: Mon, 14 Jul 2025 17:16:33 +02:00 sched: Fix proxy/current (push,pull)ability Proxy execution forms atomic pairs of tasks: The waiting donor task (scheduling context) and a proxy (execution context). The donor task, along with the rest of the blocked chain, follows the proxy wrt CPU placement. They can be the same task, in which case push/pull doesn't need any modification. When they are different, however, FIFO1 & FIFO42: ,-> RT42 | | blocked-on | v blocked_donor | mutex | | owner | v `-- RT1 RT1 RT42 CPU0 CPU1 ^ ^ | | overloaded !overloaded rq prio =3D 42 rq prio =3D 0 RT1 is eligible to be pushed to CPU1, but should that happen it will "carry" RT42 along. Clearly here neither RT1 nor RT42 must be seen as push/pullable. Unfortunately, only the donor task is usually dequeued from the rq, and the proxy'ed execution context (rq->curr) remains on the rq. This can cause RT1 to be selected for migration from logic like the rt pushable_list. Thus, adda a dequeue/enqueue cycle on the proxy task before __schedule returns, which allows the sched class logic to avoid adding the now current task to the pushable_list. Furthermore, tasks becoming blocked on a mutex don't need an explicit dequeue/enqueue cycle to be made (push/pull)able: they have to be running to block on a mutex, thus they will eventually hit put_prev_task(). Signed-off-by: Valentin Schneider Signed-off-by: Connor O'Brien Signed-off-by: John Stultz Signed-off-by: Peter Zijlstra (Intel) Tested-by: K Prateek Nayak Link: https://lkml.kernel.org/r/20250712033407.2383110-8-jstultz@google.com --- kernel/sched/core.c | 25 +++++++++++++++++++++++++ kernel/sched/deadline.c | 7 +++++++ kernel/sched/rt.c | 5 +++++ 3 files changed, 37 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index cb55d42..a0b1120 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6654,6 +6654,23 @@ find_proxy_task(struct rq *rq, struct task_struct *d= onor, struct rq_flags *rf) } #endif /* SCHED_PROXY_EXEC */ =20 +static inline void proxy_tag_curr(struct rq *rq, struct task_struct *owner) +{ + if (!sched_proxy_exec()) + return; + /* + * pick_next_task() calls set_next_task() on the chosen task + * at some point, which ensures it is not push/pullable. + * However, the chosen/donor task *and* the mutex owner form an + * atomic pair wrt push/pull. + * + * Make sure owner we run is not pushable. Unfortunately we can + * only deal with that by means of a dequeue/enqueue cycle. :-/ + */ + dequeue_task(rq, owner, DEQUEUE_NOCLOCK | DEQUEUE_SAVE); + enqueue_task(rq, owner, ENQUEUE_NOCLOCK | ENQUEUE_RESTORE); +} + /* * __schedule() is the main scheduler function. * @@ -6798,6 +6815,10 @@ picked: * changes to task_struct made by pick_next_task(). */ RCU_INIT_POINTER(rq->curr, next); + + if (!task_current_donor(rq, next)) + proxy_tag_curr(rq, next); + /* * The membarrier system call requires each architecture * to have a full memory barrier after updating @@ -6832,6 +6853,10 @@ picked: /* Also unlocks the rq: */ rq =3D context_switch(rq, prev, next, &rf); } else { + /* In case next was already curr but just got blocked_donor */ + if (!task_current_donor(rq, next)) + proxy_tag_curr(rq, next); + rq_unpin_lock(rq, &rf); __balance_callbacks(rq); raw_spin_rq_unlock_irq(rq); diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 1af06e4..e2d51f4 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2121,6 +2121,9 @@ static void enqueue_task_dl(struct rq *rq, struct tas= k_struct *p, int flags) if (dl_server(&p->dl)) return; =20 + if (task_is_blocked(p)) + return; + if (!task_current(rq, p) && !p->dl.dl_throttled && p->nr_cpus_allowed > 1) enqueue_pushable_dl_task(rq, p); } @@ -2415,6 +2418,10 @@ static void put_prev_task_dl(struct rq *rq, struct t= ask_struct *p, struct task_s update_curr_dl(rq); =20 update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 1); + + if (task_is_blocked(p)) + return; + if (on_dl_rq(&p->dl) && p->nr_cpus_allowed > 1) enqueue_pushable_dl_task(rq, p); } diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index be6e9bc..7936d43 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1440,6 +1440,9 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p,= int flags) =20 enqueue_rt_entity(rt_se, flags); =20 + if (task_is_blocked(p)) + return; + if (!task_current(rq, p) && p->nr_cpus_allowed > 1) enqueue_pushable_task(rq, p); } @@ -1716,6 +1719,8 @@ static void put_prev_task_rt(struct rq *rq, struct ta= sk_struct *p, struct task_s =20 update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 1); =20 + if (task_is_blocked(p)) + return; /* * The previous task needs to be made eligible for pushing * if it is still active