From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B7411E9B21 for ; Wed, 25 Jun 2025 20:31:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883481; cv=none; b=GdI6hM/hhORSGnnsEjqCOTUH1JH+uKgMx4BsvBiZFnzDL7hFITnJbOjl4HU3/CbdVB50A7+88GSkuyzPLddgFdo727mmpRrfZ7UPh72lqJVtYB2MXK55sc0UWa+VLukb8oBnFDAX2R0pHVHzner6SKHsEbWPaD9FIKtj5cZkMoI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883481; c=relaxed/simple; bh=vtzyhv9a/esCi91sq/GucpYona1JPGP894PNVA+Aruw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jnrDFs6AjWEP/jfbVgRtUCsXLaMcC9W6mZlK0TBfeDevv8zIcVcOuBg+gcSHLbwkrRwFy0QSYgRKJdNrC1Z+VM98U93MWCuz1THNKa49cnED40Dqq9VCZpPXPztNIiEpByalLulKEFqzhBJQq/wdAm9jTQT32rIE307aVbUX10c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QAs8f7Js; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QAs8f7Js" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-74913385dd8so441046b3a.0 for ; Wed, 25 Jun 2025 13:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883479; x=1751488279; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GtamtensSo2O4q2/DkAjxE9NIlfjTtUqP5/1W59cAKQ=; b=QAs8f7JsJLNtIijz++j4KJagTwZVANENt8qyoaY3VwoMS9rwZthLQxzZ0X3BkPiG/+ 4AzMjwlTk/zmYXmmDA+bo3s5frN5sX1A3zIt48NlwepNuA3tDW+3mywgyfUO1gH5bYtL iIO7xEV899ljCEvGt+X9NUvKIW4K28fxfRpCbYfA4mr7OgcclTLWhQ4z3DB4PxRuAArg HP/FSjD/Ph25PQ6ZRft8Tp+keZGBltiXjA9Mcao1V7yAg+DQ54QnJVqEmcljHwySNx2F XBwirNHJvjp3yKu0XskPZ9VwFmIn8bM5xCvXDvQgyVxD0a9IfB89xsbVjwqByPwvXtx6 9pdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883479; x=1751488279; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GtamtensSo2O4q2/DkAjxE9NIlfjTtUqP5/1W59cAKQ=; b=Ed50N05HFwcOMV8smgPS19fBC3uHbU5Rkpml1bYsXLYR6HBifZ28M4RetDu6vJ/i93 UQutSCqcrP3hQKnSO9gyWV5FSY0SW5NmrfG6R3vwEA6PV9Me8yz7vrb6yVhRssQ0KF3/ xKhUUHqaf6FrVrE44DH2VY+/u7b6W599o7h/8deZdJIg7itTBFJaZ4J9knHvsRjqrVdX eD1XOvtfYqxygLi87B7SB4jJqwb27NadOp5nD74W1+zhGCiUCrVMPzvFHkIjwzwMDBj1 aFa5N5/qvmBT8vPMA+zk/dQqLx8bbdSSnfvOI0j2FDxYD2LnryxvT2bJ1ikp3O8uODCK 9WmQ== X-Gm-Message-State: AOJu0YzlPdXg4316+9lzgQBnvyl2YMJAm7ht1teasgC13ScYDbjisOWO uIG5XE9QT8IrZZkTrCmJCg83tFkpqlIWWaadtgU/Zcjz+mKAkRZxe0MgjRVOcBu3sroMF9LQyYM FuqXPLb/hl1K84Fg4iBqnfTOC2HB7s4CjOSifgNHDqStGlNpTtdD7UL966PDDfLWNm0sZ0gz4bX j5THpkRtJWqONMgzDxfE3f182fm0Z3pBRuDgXT9KSpfZV5SG8/ X-Google-Smtp-Source: AGHT+IHDYmheI+wMzvJXJbNg18k/gImnOm8cJXM7KGNchZPmuVg3GX3LXq2eEP3VpdY3oAEqBe1+ruuSTegG X-Received: from pfjw3.prod.google.com ([2002:aa7:9a03:0:b0:746:247f:7384]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2e03:b0:736:31cf:2590 with SMTP id d2e1a72fcca58-74ad45bb45bmr5900906b3a.16.1750883478695; Wed, 25 Jun 2025 13:31:18 -0700 (PDT) Date: Wed, 25 Jun 2025 20:30:54 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-2-jstultz@google.com> Subject: [PATCH v18 1/8] sched: Add CONFIG_SCHED_PROXY_EXEC & boot argument to enable/disable From: John Stultz To: LKML Cc: John Stultz , K Prateek Nayak , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a CONFIG_SCHED_PROXY_EXEC option, along with a boot argument sched_proxy_exec=3D that can be used to disable the feature at boot time if CONFIG_SCHED_PROXY_EXEC was enabled. Also uses this option to allow the rq->donor to be different from rq->curr. Tested-by: K Prateek Nayak Signed-off-by: John Stultz --- v7: * Switch to CONFIG_SCHED_PROXY_EXEC/sched_proxy_exec=3D as suggested by Metin Kaya. * Switch boot arg from =3Ddisable/enable to use kstrtobool(), which supports =3Dyes|no|1|0|true|false|on|off, as also suggested by Metin Kaya, and print a message when a boot argument is used. v8: * Move CONFIG_SCHED_PROXY_EXEC under Scheduler Features as Suggested by Metin * Minor rework reordering with split sched contexts patch v12: * Rework for selected -> donor renaming v14: * Depend on !PREEMPT_RT to avoid build issues for now v15: * Depend on EXPERT while patch series upstreaming is in progress. v16: * Allow "sched_proxy_exec" without "=3Dtrue" to enable proxy-execution at boot time, in addition to the "sched_proxy_exec=3Dtrue" or "sched_proxy_exec=3Dfalse" options as suggested by Steven * Drop the "default n" in Kconfig as suggested by Steven * Add !SCHED_CLASS_EXT dependency until I can investigate if sched_ext can understand split contexts, as suggested by Peter v17: * Expanded the commit message a bit to clarify that the option allows the split contexts (rq->donor and rq->curr) to be different tasks, from feedback from Juri Lelli Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- .../admin-guide/kernel-parameters.txt | 5 ++++ include/linux/sched.h | 13 +++++++++ init/Kconfig | 12 ++++++++ kernel/sched/core.c | 29 +++++++++++++++++++ kernel/sched/sched.h | 12 ++++++++ 5 files changed, 71 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index f1f2c0874da9d..5b89464ca570a 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6387,6 +6387,11 @@ sa1100ir [NET] See drivers/net/irda/sa1100_ir.c. =20 + sched_proxy_exec=3D [KNL] + Enables or disables "proxy execution" style + solution to mutex-based priority inversion. + Format: + sched_verbose [KNL,EARLY] Enables verbose scheduler debug messages. =20 schedstats=3D [KNL,X86] Enable or disable scheduled statistics. diff --git a/include/linux/sched.h b/include/linux/sched.h index 4f78a64beb52c..ba2a02ead8c7e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1664,6 +1664,19 @@ struct task_struct { randomized_struct_fields_end } __attribute__ ((aligned (64))); =20 +#ifdef CONFIG_SCHED_PROXY_EXEC +DECLARE_STATIC_KEY_TRUE(__sched_proxy_exec); +static inline bool sched_proxy_exec(void) +{ + return static_branch_likely(&__sched_proxy_exec); +} +#else +static inline bool sched_proxy_exec(void) +{ + return false; +} +#endif + #define TASK_REPORT_IDLE (TASK_REPORT + 1) #define TASK_REPORT_MAX (TASK_REPORT_IDLE << 1) =20 diff --git a/init/Kconfig b/init/Kconfig index af4c2f0854554..6844b25303913 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -875,6 +875,18 @@ config UCLAMP_BUCKETS_COUNT =20 If in doubt, use the default value. =20 +config SCHED_PROXY_EXEC + bool "Proxy Execution" + # Avoid some build failures w/ PREEMPT_RT until it can be fixed + depends on !PREEMPT_RT + # Need to investigate how to inform sched_ext of split contexts + depends on !SCHED_CLASS_EXT + # Not particularly useful until we get to multi-rq proxying + depends on EXPERT + help + This option enables proxy execution, a mechanism for mutex-owning + tasks to inherit the scheduling context of higher priority waiters. + endmenu =20 # diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 8988d38d46a38..853157b27f384 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -119,6 +119,35 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_compute_energy_tp); =20 DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); =20 +#ifdef CONFIG_SCHED_PROXY_EXEC +DEFINE_STATIC_KEY_TRUE(__sched_proxy_exec); +static int __init setup_proxy_exec(char *str) +{ + bool proxy_enable =3D true; + + if (*str && kstrtobool(str + 1, &proxy_enable)) { + pr_warn("Unable to parse sched_proxy_exec=3D\n"); + return 0; + } + + if (proxy_enable) { + pr_info("sched_proxy_exec enabled via boot arg\n"); + static_branch_enable(&__sched_proxy_exec); + } else { + pr_info("sched_proxy_exec disabled via boot arg\n"); + static_branch_disable(&__sched_proxy_exec); + } + return 1; +} +#else +static int __init setup_proxy_exec(char *str) +{ + pr_warn("CONFIG_SCHED_PROXY_EXEC=3Dn, so it cannot be enabled or disabled= at boot time\n"); + return 0; +} +#endif +__setup("sched_proxy_exec", setup_proxy_exec); + /* * Debugging: various feature bits * diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 475bb5998295e..6b61e0c7f6e78 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1151,10 +1151,15 @@ struct rq { */ unsigned int nr_uninterruptible; =20 +#ifdef CONFIG_SCHED_PROXY_EXEC + struct task_struct __rcu *donor; /* Scheduling context */ + struct task_struct __rcu *curr; /* Execution context */ +#else union { struct task_struct __rcu *donor; /* Scheduler context */ struct task_struct __rcu *curr; /* Execution context */ }; +#endif struct sched_dl_entity *dl_server; struct task_struct *idle; struct task_struct *stop; @@ -1349,10 +1354,17 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues= ); #define cpu_curr(cpu) (cpu_rq(cpu)->curr) #define raw_rq() raw_cpu_ptr(&runqueues) =20 +#ifdef CONFIG_SCHED_PROXY_EXEC +static inline void rq_set_donor(struct rq *rq, struct task_struct *t) +{ + rcu_assign_pointer(rq->donor, t); +} +#else static inline void rq_set_donor(struct rq *rq, struct task_struct *t) { /* Do nothing */ } +#endif =20 #ifdef CONFIG_SCHED_CORE static inline struct cpumask *sched_group_span(struct sched_group *sg); --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D6152D660E for ; Wed, 25 Jun 2025 20:31:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883484; cv=none; b=tUpSWuzEw0hkIhj6YAxV09it9/Rc3JHsCiNkDUmgZpw8pNdT8SmaykoRIbSK3w3N7xzWMcBqF4mZEdIR6V995lJurmX9M+jzJLdGKM0VRZbvLrMCFo5A/c8zd80XLbQoT+c8AEMepfL5CL0W462bYmevtx9h3L+xIwKD02IFpoE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883484; c=relaxed/simple; bh=Ja7Ju/Y3i7DdRdteQV6S0vNj+DWaIxbHe/WOxhHFtk8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=EPUil172AEIRpsFERgeUXJWEFwEvifEzdIgBROdTbkzaEbdG6/tatZJQqx2IAZ0O3pOkslbhfZj4HOvHULm9P7HrRD3Gv34dwW4ybL8n88KS1oVlCTALDMrZ9O0XurWUuPcXORQFlSc/QDkPl/qJ1qzF3KardLfJPKNp6tgi0M8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NwFZiv8T; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NwFZiv8T" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-7489ac848f3so510991b3a.1 for ; Wed, 25 Jun 2025 13:31:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883480; x=1751488280; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GWNDDRSNPv4XDiBT9EiVhHZPV0Oi3E4yRSOXlnJKxBk=; b=NwFZiv8T7VLtF7sH14+zwsPja4fU+2ce3kTAH1he6bCaHXpf0mS3tmecC76s4pej8J gBprnMyWriVKv/w4K4z8fuR6ry+vJabvp1WRLHkAm0hJ+q/cWgMEPc4zT+z+8vPPXk0v J2sgfsvZAEGsIo+fZTDQ0bze6v3ouCdWKB/TgHXLYc5Cx9LKTosjPthLw9fyBhgCChPJ j7MBI2WfCGDIJQ3JkhysshTF6abvXI6WF0lxiHsdeasxQgd7rajyaK1P1dti875Ji914 DK3WTMIcAUrYQFK+Kf0zuIjaKHXnnmn6Um63yBDa9Bl9RSzc/eAzXeTUMABmAVwhj9mI uq4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883480; x=1751488280; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GWNDDRSNPv4XDiBT9EiVhHZPV0Oi3E4yRSOXlnJKxBk=; b=orTSxG8sHERy49XEJlU25PczQHT8pNINTrVLlE34VLSukh9C/69cLYS4pIIUE0L5b0 ClNXOfgRZrDugE343njlWfUMTZ6ajQ4Ja22vPXv2HudSMUyo5YTr+8gp/dTrpV16Q9Jp jGBjCqaOWBFRoDyWvZBpIy0fl12fndIwd6mbUaVac+8TFwNgqOjUeNuD7XT+AS6TuhR4 KEOTDTaaZ+8vub58dIt3IOrVoFBteOKSysUfPBr0aMcqbvWVIIAzsp+yqql6e53ItblQ jOjETHWisYg8mxsws/KCSqopgvBGWkrME4OO+Yd+rGXiTwv0k4uQZkgWW/LMt4+/HGfd cRXg== X-Gm-Message-State: AOJu0YxUfimSC2XYASS0axrMyqMzhecBlcnzm+sfPrVOg/sR5/EFwKBK hC/D3+qHlilNlNUryb9Walm15nr5vaJtyy/JbqMa8ahHRmPegICkinHFJ6fTXU7pSAdb8lTynBt ZUcNFqkGIfhtBOUzdbAxKWyThMxcmNbx429eJoawr962kkRIf42uNbpzbdCQQmACsW1MczPLkxb LZAP3w7kmTunpMRdt7TbMauy4ZiMkZAykRuxKaRBObCXOXk1Wo X-Google-Smtp-Source: AGHT+IF8uhC5Jb7nLGkPmnfcjJw5b37JLD48YRigHTmj1RNnQcvTlDdrL97Me1RO33ny1zJh0x/ZKcw+0+NQ X-Received: from pfme17.prod.google.com ([2002:aa7:98d1:0:b0:747:bd3b:4b63]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:22d4:b0:748:2ac2:f8c3 with SMTP id d2e1a72fcca58-74ad45d6782mr7979487b3a.24.1750883480302; Wed, 25 Jun 2025 13:31:20 -0700 (PDT) Date: Wed, 25 Jun 2025 20:30:55 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-3-jstultz@google.com> Subject: [PATCH v18 2/8] locking/mutex: Rework task_struct::blocked_on From: John Stultz To: LKML Cc: Peter Zijlstra , Juri Lelli , "Connor O'Brien" , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Track the blocked-on relation for mutexes, to allow following this relation at schedule time. task | blocked-on v mutex | owner v task This all will be used for tracking blocked-task/mutex chains with the prox-execution patch in a similar fashion to how priority inheritance is done with rt_mutexes. For serialization, blocked-on is only set by the task itself (current). And both when setting or clearing (potentially by others), is done while holding the mutex::wait_lock. Signed-off-by: Peter Zijlstra (Intel) [minor changes while rebasing] Signed-off-by: Juri Lelli Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Connor O'Brien [jstultz: Fix blocked_on tracking in __mutex_lock_common in error paths] Signed-off-by: John Stultz Tested-by: K Prateek Nayak --- v2: * Fixed blocked_on tracking in error paths that was causing crashes v4: * Ensure we clear blocked_on when waking ww_mutexes to die or wound. This is critical so we don't get circular blocked_on relationships that can't be resolved. v5: * Fix potential bug where the skip_wait path might clear blocked_on when that path never set it * Slight tweaks to where we set blocked_on to make it consistent, along with extra WARN_ON correctness checking * Minor comment changes v7: * Minor commit message change suggested by Metin Kaya * Fix WARN_ON conditionals in unlock path (as blocked_on might already be cleared), found while looking at issue Metin Kaya raised. * Minor tweaks to be consistent in what we do under the blocked_on lock, also tweaked variable name to avoid confusion with label, and comment typos, as suggested by Metin Kaya * Minor tweak for CONFIG_SCHED_PROXY_EXEC name change * Moved unused block of code to later in the series, as suggested by Metin Kaya * Switch to a tri-state to be able to distinguish from waking and runnable so we can later safely do return migration from ttwu * Folded together with related blocked_on changes v8: * Fix issue leaving task BO_BLOCKED when calling into optimistic spinning path. * Include helper to better handle BO_BLOCKED->BO_WAKING transitions v9: * Typo fixup pointed out by Metin * Cleanup BO_WAKING->BO_RUNNABLE transitions for the !proxy case * Many cleanups and simplifications suggested by Metin v11: * Whitespace fixup pointed out by Metin v13: * Refactor set_blocked_on helpers clean things up a bit v14: * Small build fixup with PREEMPT_RT v15: * Improve consistency of names for functions that assume blocked_lock is held, as suggested by Peter * Use guard instead of separate spinlock/unlock calls, also suggested by Peter * Drop blocked_on_state tri-state for now, as its not needed until later in the series, when we get to proxy-migration and return- migration. v16: * Clear blocked on before optimistic spinning Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- include/linux/sched.h | 5 +---- kernel/fork.c | 3 +-- kernel/locking/mutex-debug.c | 9 +++++---- kernel/locking/mutex.c | 22 ++++++++++++++++++++++ kernel/locking/ww_mutex.h | 18 ++++++++++++++++-- 5 files changed, 45 insertions(+), 12 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index ba2a02ead8c7e..aa7966141a090 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1238,10 +1238,7 @@ struct task_struct { struct rt_mutex_waiter *pi_blocked_on; #endif =20 -#ifdef CONFIG_DEBUG_MUTEXES - /* Mutex deadlock detection: */ - struct mutex_waiter *blocked_on; -#endif + struct mutex *blocked_on; /* lock we're blocked on */ =20 #ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER /* diff --git a/kernel/fork.c b/kernel/fork.c index 1ee8eb11f38ba..5f87f05aff4a0 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2123,9 +2123,8 @@ __latent_entropy struct task_struct *copy_process( lockdep_init_task(p); #endif =20 -#ifdef CONFIG_DEBUG_MUTEXES p->blocked_on =3D NULL; /* not blocked yet */ -#endif + #ifdef CONFIG_BCACHE p->sequential_io =3D 0; p->sequential_io_avg =3D 0; diff --git a/kernel/locking/mutex-debug.c b/kernel/locking/mutex-debug.c index 6e6f6071cfa27..758b7a6792b0c 100644 --- a/kernel/locking/mutex-debug.c +++ b/kernel/locking/mutex-debug.c @@ -53,17 +53,18 @@ void debug_mutex_add_waiter(struct mutex *lock, struct = mutex_waiter *waiter, { lockdep_assert_held(&lock->wait_lock); =20 - /* Mark the current thread as blocked on the lock: */ - task->blocked_on =3D waiter; + /* Current thread can't be already blocked (since it's executing!) */ + DEBUG_LOCKS_WARN_ON(task->blocked_on); } =20 void debug_mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *wa= iter, struct task_struct *task) { + struct mutex *blocked_on =3D READ_ONCE(task->blocked_on); + DEBUG_LOCKS_WARN_ON(list_empty(&waiter->list)); DEBUG_LOCKS_WARN_ON(waiter->task !=3D task); - DEBUG_LOCKS_WARN_ON(task->blocked_on !=3D waiter); - task->blocked_on =3D NULL; + DEBUG_LOCKS_WARN_ON(blocked_on && blocked_on !=3D lock); =20 INIT_LIST_HEAD(&waiter->list); waiter->task =3D NULL; diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index a39ecccbd106a..e2f59863a866e 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -644,6 +644,8 @@ __mutex_lock_common(struct mutex *lock, unsigned int st= ate, unsigned int subclas goto err_early_kill; } =20 + WARN_ON(current->blocked_on); + current->blocked_on =3D lock; set_current_state(state); trace_contention_begin(lock, LCB_F_MUTEX); for (;;) { @@ -680,6 +682,12 @@ __mutex_lock_common(struct mutex *lock, unsigned int s= tate, unsigned int subclas =20 first =3D __mutex_waiter_is_first(lock, &waiter); =20 + /* + * As we likely have been woken up by task + * that has cleared our blocked_on state, re-set + * it to the lock we are trying to aquire. + */ + current->blocked_on =3D lock; set_current_state(state); /* * Here we order against unlock; we must either see it change @@ -691,8 +699,11 @@ __mutex_lock_common(struct mutex *lock, unsigned int s= tate, unsigned int subclas =20 if (first) { trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN); + /* clear blocked_on as mutex_optimistic_spin may schedule() */ + current->blocked_on =3D NULL; if (mutex_optimistic_spin(lock, ww_ctx, &waiter)) break; + current->blocked_on =3D lock; trace_contention_begin(lock, LCB_F_MUTEX); } =20 @@ -700,6 +711,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int st= ate, unsigned int subclas } raw_spin_lock_irqsave(&lock->wait_lock, flags); acquired: + current->blocked_on =3D NULL; __set_current_state(TASK_RUNNING); =20 if (ww_ctx) { @@ -729,9 +741,11 @@ __mutex_lock_common(struct mutex *lock, unsigned int s= tate, unsigned int subclas return 0; =20 err: + current->blocked_on =3D NULL; __set_current_state(TASK_RUNNING); __mutex_remove_waiter(lock, &waiter); err_early_kill: + WARN_ON(current->blocked_on); trace_contention_end(lock, ret); raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q); debug_mutex_free_waiter(&waiter); @@ -942,6 +956,14 @@ static noinline void __sched __mutex_unlock_slowpath(s= truct mutex *lock, unsigne next =3D waiter->task; =20 debug_mutex_wake_waiter(lock, waiter); + /* + * Unlock wakeups can be happening in parallel + * (when optimistic spinners steal and release + * the lock), so blocked_on may already be + * cleared here. + */ + WARN_ON(next->blocked_on && next->blocked_on !=3D lock); + next->blocked_on =3D NULL; wake_q_add(&wake_q, next); } =20 diff --git a/kernel/locking/ww_mutex.h b/kernel/locking/ww_mutex.h index 37f025a096c9d..00db40946328e 100644 --- a/kernel/locking/ww_mutex.h +++ b/kernel/locking/ww_mutex.h @@ -284,6 +284,14 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER= *waiter, #ifndef WW_RT debug_mutex_wake_waiter(lock, waiter); #endif + /* + * When waking up the task to die, be sure to clear the + * blocked_on pointer. Otherwise we can see circular + * blocked_on relationships that can't resolve. + */ + WARN_ON(waiter->task->blocked_on && + waiter->task->blocked_on !=3D lock); + waiter->task->blocked_on =3D NULL; wake_q_add(wake_q, waiter->task); } =20 @@ -331,9 +339,15 @@ static bool __ww_mutex_wound(struct MUTEX *lock, * it's wounded in __ww_mutex_check_kill() or has a * wakeup pending to re-read the wounded state. */ - if (owner !=3D current) + if (owner !=3D current) { + /* + * When waking up the task to wound, be sure to clear the + * blocked_on pointer. Otherwise we can see circular + * blocked_on relationships that can't resolve. + */ + owner->blocked_on =3D NULL; wake_q_add(wake_q, owner); - + } return true; } =20 --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CAF92DECC1 for ; Wed, 25 Jun 2025 20:31:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883485; cv=none; b=rQ7P/0SJyt6dpPl22aTX6IDJElyT8s4IMIi0NwbwjC5p4MhkkL67L2x8ZbpxagKLBYIMVZg4DVnIjQrnwOU/lFdhLm4Fnr6hfZKUwQwtmOCIxIOSAvVdYkm6bawtK+d1HAxlcujWIp8duDh+JYk2Kjg6VU0ZVfskTKTEoIRkcr8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883485; c=relaxed/simple; bh=W+qmZSIaHerBC6mMLl7PYab0nmLTysV/UsZpoTQGwms=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qM7Ol8wEinIIem+n/OxTCiiabppVTk6DzSDz8RolEp0TcqLLPwleMMZnSrio6DbjFgKNzRfbdwvl/2AoM3O4/YClNnTG87br//a8pt1uHD0vlsL+bLuaohg6Uy61WFSLn4owfIWpWYyH1hA0iiCsu/ar5R7aEfJSAEzx2vz0eu0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZLzz8OK4; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZLzz8OK4" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-b31f112c90aso191708a12.0 for ; Wed, 25 Jun 2025 13:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883482; x=1751488282; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8i/fLaJcfoOBu+GE99WreXdg0Kg5hxNU37Cftd/kbl0=; b=ZLzz8OK4/1PhZ+qla4amjqZuAmySP1yG4lR7edGW6W00WCgOzum13LRmeRWQhYDDIE zSpr4C8v8SAmt86Mt7pqTcyQQnO4qWK/6xOBtUdJ3Xpy66wK/Mu3q/IAKUKgCVJOQfDT RqFP6tDHsnXZv8AsHXtvTtnirIxFqd2024d/+axXPhKkT9nidzvL3q6MEpgWz9E//6pH in6N60CE05QtCuMShhGwU4fIXF9+uVcIRc2QtB63JeJgsr3koPd3GZpMCzZC/zUVEJVV 0KlqEogTea6sTpgDgBq6xdMuNAvR5Vm6CI42hQ8JChd1QQENeQ0PrCw1Y25P/AGITuI0 Sv0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883482; x=1751488282; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8i/fLaJcfoOBu+GE99WreXdg0Kg5hxNU37Cftd/kbl0=; b=u5cawpWQkKQ9LFRLYQMzAUSF29Xl1PaT2ryGCSsp+xGAv8qMf/u6TU893EdKNev/JO tXVf6+3z9nvHhWPlXOpjNXJnxr9b4nwYeIGaD79cQg5UD3Y638QTZFNJklQXj/AC/VOm oGXRLSNqjU1vu/xE8EWQFgCYs2snvTnB8jGo/yNFd6oY8buv/OJUPT8cNZ5ZDRcleH1u QRDxWrOaVgriH7c+uhRxgB6hnMV9eV+OsSrh9oyZS5q83om4W7ABr486GMdMfc3UH2ZQ DEzqj6UoS5McAdE+8MZgatn7R/ckNfoleZS9qlVn0XDY+X7ln52VAtybI9tiIMuT4zoW UsyQ== X-Gm-Message-State: AOJu0YwVeIeZ/ImK+jvuIRRSnlZsPkCjGoBPv13jpM5J/pAuj7JAz2Ai 6IYjx8btDaAKbioWJJzUSuDFgPRDvpxtaf4kBTFc1hjE6mdhsRuxQNUyCHsH0YnXNNb/eSzI/Pj 1DVW5UmX0INhIkFH5D8yL8cAJE2VTsPRZCluEZIgUNN90Dz0shytaHU8T9TyNJjlEmIEAzJnk3r NRpwvcr2C7C0rhs7O/IL+XsNM/B1YsFs1VDZc2XEf1AJf4toe0 X-Google-Smtp-Source: AGHT+IHWEW7MnrGP8BNZ0jkqVpfSLJoF00onsFnffYMQy+IlxrmpUPVdQV7hWQVFIu03Lgnxan9LyfUj7FRQ X-Received: from pfqf28.prod.google.com ([2002:aa7:9d9c:0:b0:73c:29d8:b795]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:a11a:b0:1f3:20be:c18a with SMTP id adf61e73a8af0-2208d0ec8d6mr1160715637.10.1750883482137; Wed, 25 Jun 2025 13:31:22 -0700 (PDT) Date: Wed, 25 Jun 2025 20:30:56 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-4-jstultz@google.com> Subject: [PATCH v18 3/8] locking/mutex: Add p->blocked_on wrappers for correctness checks From: John Stultz To: LKML Cc: Valentin Schneider , "Connor O'Brien" , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Valentin Schneider This lets us assert mutex::wait_lock is held whenever we access p->blocked_on, as well as warn us for unexpected state changes. Signed-off-by: Valentin Schneider [fix conflicts, call in more places] Signed-off-by: Connor O'Brien [jstultz: tweaked commit subject, reworked a good bit] Signed-off-by: John Stultz Tested-by: K Prateek Nayak --- v2: * Added get_task_blocked_on() accessor v4: * Address READ_ONCE usage that was dropped in v2 * Reordered to be a later add on to the main patch series as Peter was unhappy with similar wrappers in other patches. v5: * Added some extra correctness checking in wrappers v7: * Tweaks to reorder this change in the patch series * Minor cleanup to set_task_blocked_on() suggested by Metin Kaya v15: * Split out into its own patch again. * Further improve assumption checks in helpers. v16: * Fix optimistic spin case that can call schedule() v17: * Fix typos caught by Metin Kaya * Add lockdep_assert_held_once and drop the READ_ONCE in __get_task_blocked_on(), as suggested by Juri Lelli Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- include/linux/sched.h | 54 ++++++++++++++++++++++++++++++++++-- kernel/locking/mutex-debug.c | 4 +-- kernel/locking/mutex.c | 32 ++++++++++----------- kernel/locking/ww_mutex.h | 6 ++-- 4 files changed, 70 insertions(+), 26 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index aa7966141a090..1d7f625adbb5e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -2177,6 +2178,57 @@ extern int __cond_resched_rwlock_write(rwlock_t *loc= k); __cond_resched_rwlock_write(lock); \ }) =20 +static inline void __set_task_blocked_on(struct task_struct *p, struct mut= ex *m) +{ + WARN_ON_ONCE(!m); + /* The task should only be setting itself as blocked */ + WARN_ON_ONCE(p !=3D current); + /* Currently we serialize blocked_on under the mutex::wait_lock */ + lockdep_assert_held_once(&m->wait_lock); + /* + * Check ensure we don't overwrite existing mutex value + * with a different mutex. Note, setting it to the same + * lock repeatedly is ok. + */ + WARN_ON_ONCE(p->blocked_on && p->blocked_on !=3D m); + p->blocked_on =3D m; +} + +static inline void set_task_blocked_on(struct task_struct *p, struct mutex= *m) +{ + guard(raw_spinlock_irqsave)(&m->wait_lock); + __set_task_blocked_on(p, m); +} + +static inline void __clear_task_blocked_on(struct task_struct *p, struct m= utex *m) +{ + WARN_ON_ONCE(!m); + /* Currently we serialize blocked_on under the mutex::wait_lock */ + lockdep_assert_held_once(&m->wait_lock); + /* + * There may be cases where we re-clear already cleared + * blocked_on relationships, but make sure we are not + * clearing the relationship with a different lock. + */ + WARN_ON_ONCE(m && p->blocked_on && p->blocked_on !=3D m); + p->blocked_on =3D NULL; +} + +static inline void clear_task_blocked_on(struct task_struct *p, struct mut= ex *m) +{ + guard(raw_spinlock_irqsave)(&m->wait_lock); + __clear_task_blocked_on(p, m); +} + +static inline struct mutex *__get_task_blocked_on(struct task_struct *p) +{ + struct mutex *m =3D p->blocked_on; + + if (m) + lockdep_assert_held_once(&m->wait_lock); + return m; +} + static __always_inline bool need_resched(void) { return unlikely(tif_need_resched()); @@ -2216,8 +2268,6 @@ extern bool sched_task_on_rq(struct task_struct *p); extern unsigned long get_wchan(struct task_struct *p); extern struct task_struct *cpu_curr_snapshot(int cpu); =20 -#include - /* * In order to reduce various lock holder preemption latencies provide an * interface to see if a vCPU is currently running or not. diff --git a/kernel/locking/mutex-debug.c b/kernel/locking/mutex-debug.c index 758b7a6792b0c..949103fd8e9b5 100644 --- a/kernel/locking/mutex-debug.c +++ b/kernel/locking/mutex-debug.c @@ -54,13 +54,13 @@ void debug_mutex_add_waiter(struct mutex *lock, struct = mutex_waiter *waiter, lockdep_assert_held(&lock->wait_lock); =20 /* Current thread can't be already blocked (since it's executing!) */ - DEBUG_LOCKS_WARN_ON(task->blocked_on); + DEBUG_LOCKS_WARN_ON(__get_task_blocked_on(task)); } =20 void debug_mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *wa= iter, struct task_struct *task) { - struct mutex *blocked_on =3D READ_ONCE(task->blocked_on); + struct mutex *blocked_on =3D __get_task_blocked_on(task); =20 DEBUG_LOCKS_WARN_ON(list_empty(&waiter->list)); DEBUG_LOCKS_WARN_ON(waiter->task !=3D task); diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index e2f59863a866e..80d778fedd605 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -644,8 +644,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int st= ate, unsigned int subclas goto err_early_kill; } =20 - WARN_ON(current->blocked_on); - current->blocked_on =3D lock; + __set_task_blocked_on(current, lock); set_current_state(state); trace_contention_begin(lock, LCB_F_MUTEX); for (;;) { @@ -685,9 +684,9 @@ __mutex_lock_common(struct mutex *lock, unsigned int st= ate, unsigned int subclas /* * As we likely have been woken up by task * that has cleared our blocked_on state, re-set - * it to the lock we are trying to aquire. + * it to the lock we are trying to acquire. */ - current->blocked_on =3D lock; + set_task_blocked_on(current, lock); set_current_state(state); /* * Here we order against unlock; we must either see it change @@ -699,11 +698,15 @@ __mutex_lock_common(struct mutex *lock, unsigned int = state, unsigned int subclas =20 if (first) { trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN); - /* clear blocked_on as mutex_optimistic_spin may schedule() */ - current->blocked_on =3D NULL; + /* + * mutex_optimistic_spin() can call schedule(), so + * clear blocked on so we don't become unselectable + * to run. + */ + clear_task_blocked_on(current, lock); if (mutex_optimistic_spin(lock, ww_ctx, &waiter)) break; - current->blocked_on =3D lock; + set_task_blocked_on(current, lock); trace_contention_begin(lock, LCB_F_MUTEX); } =20 @@ -711,7 +714,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int st= ate, unsigned int subclas } raw_spin_lock_irqsave(&lock->wait_lock, flags); acquired: - current->blocked_on =3D NULL; + __clear_task_blocked_on(current, lock); __set_current_state(TASK_RUNNING); =20 if (ww_ctx) { @@ -741,11 +744,11 @@ __mutex_lock_common(struct mutex *lock, unsigned int = state, unsigned int subclas return 0; =20 err: - current->blocked_on =3D NULL; + __clear_task_blocked_on(current, lock); __set_current_state(TASK_RUNNING); __mutex_remove_waiter(lock, &waiter); err_early_kill: - WARN_ON(current->blocked_on); + WARN_ON(__get_task_blocked_on(current)); trace_contention_end(lock, ret); raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q); debug_mutex_free_waiter(&waiter); @@ -956,14 +959,7 @@ static noinline void __sched __mutex_unlock_slowpath(s= truct mutex *lock, unsigne next =3D waiter->task; =20 debug_mutex_wake_waiter(lock, waiter); - /* - * Unlock wakeups can be happening in parallel - * (when optimistic spinners steal and release - * the lock), so blocked_on may already be - * cleared here. - */ - WARN_ON(next->blocked_on && next->blocked_on !=3D lock); - next->blocked_on =3D NULL; + __clear_task_blocked_on(next, lock); wake_q_add(&wake_q, next); } =20 diff --git a/kernel/locking/ww_mutex.h b/kernel/locking/ww_mutex.h index 00db40946328e..086fd5487ca77 100644 --- a/kernel/locking/ww_mutex.h +++ b/kernel/locking/ww_mutex.h @@ -289,9 +289,7 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER = *waiter, * blocked_on pointer. Otherwise we can see circular * blocked_on relationships that can't resolve. */ - WARN_ON(waiter->task->blocked_on && - waiter->task->blocked_on !=3D lock); - waiter->task->blocked_on =3D NULL; + __clear_task_blocked_on(waiter->task, lock); wake_q_add(wake_q, waiter->task); } =20 @@ -345,7 +343,7 @@ static bool __ww_mutex_wound(struct MUTEX *lock, * blocked_on pointer. Otherwise we can see circular * blocked_on relationships that can't resolve. */ - owner->blocked_on =3D NULL; + __clear_task_blocked_on(owner, lock); wake_q_add(wake_q, owner); } return true; --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB8CB2DFA35 for ; Wed, 25 Jun 2025 20:31:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883486; cv=none; b=KvlsdB0OCJPJpec8a99NqMhsFIzrqand1m8ML/7djyjbmCXOo3Je6XxxIHGHxUEoQfM5TGjPTwe2zL9WvQOrNNbZfl0k2aCAIjm26tjRTRhvdKEjomeM/Z/MJNY48jYEtFcBqMsPL2MZDjdZAxJoLy9HQiGMmjycgzQ3WG9pA58= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883486; c=relaxed/simple; bh=nLZ/ViKh9mCSmL/QUFoTZ524l89j3+XT4qT6eKR2NHc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CR5PLETLNZJcjKnzhOOtzbGOZHVTfKgY846l8tQDW9OHj2mdxo5soGYyiP2pfVe8vOsCsKaYWGCTR1Bh6w8R27yAP0WI3/dcTyjpMnD2IW/5xx/M4oyNMYZAXFva77W5HIvvtN9yLtn9JQmF5doMKCoKxRqlT/J7vyhLw0p5wlk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PyzaRGm+; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PyzaRGm+" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-3132c8437ffso293686a91.1 for ; Wed, 25 Jun 2025 13:31:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883484; x=1751488284; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fWMZEwaLzB4ZLAXRICQvTxcHO4sAmAm0N54ZCGAyYPs=; b=PyzaRGm+QBDX76TZmNSgaY6a30iD59r82MQrFwjfkiWb8nUwCnWkAMZ/pAJhKvN+W+ 3PNC1gs8Q7+nCwWaCv8DqJwBrMzdkHZW0Y0Xl96MECbabiX3C2fk2DKGRIBrisVAu14t 54hvK4TAbCJvC5lq7YG5K4Uam3uqdh7/kxC5hOUs4zp0pP+jAQwv0flR0IXMTzKz4QfL 2GY+VtsuuqjgpajTcGVQ6+dzBYgS1eOxqHYLUlWNHrBIZVBXF3cLe9mRQl/ckldE4P+y +YWqCo1pPJy6kJSQKg0R9LDkEnT7+L1yjAbd7Em3EZkY/sDkrh93Kkrq2inkEf0nUOYP 4kGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883484; x=1751488284; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fWMZEwaLzB4ZLAXRICQvTxcHO4sAmAm0N54ZCGAyYPs=; b=hlux4hpmGl7+hdE5w1vR3M69d5w8kb0DpHfDoyIz0P9c0FlVeSvzUllfGcW+FPJSCY IQ10GfgEphEDEWzlNNckrotXKeLb/xPYmY2eWVm+XNpYAYu85Ki+GZxrW9U+/TXq9vxp j0Z9hHN4PuydKu9a4Cm5UzSW/kgg6ycFIcUScIx2nSBkDl8V9Wb42Yvxcrr70H67lfv1 ZpcHqb/CHbWVMJBsskdHC2cnF8ZGD7XdgMwtuqC+zK5zxIxxU5RxtZZ4pj1L3NpnX2j3 okikOWHlHnQlnv0WheSY4EwFWqgksQO1gaxUWQblNa6DJPVjA+3rVkXBMSE/ksq8KehW PvBg== X-Gm-Message-State: AOJu0YzT5juEu6k4NVfd4O3LAQ1vrkwGbIoh/1ZpVMENS3tV84IXMy+p 4AHZowA0Ua6TjV3yBorZnBpe5iXrtDEnvcnp6CnxPXkGQZVlzf/jIL2bhmGIpZyHwLqMqHn45JE c+5Y1pY+SS8Q8vJ5wsX3Aks/Lbdu/oyMmZl19I4T4NVrhiG5niYTHIFOTRuPGlgvT41wiaoxb2z /t95ya9FsKDVugmYIGum8OHkYsF4Suw/NnsigTi1UYE1Kca6GB X-Google-Smtp-Source: AGHT+IF+V1NliafRtG0Qu5AbXO5KkZeLJZHhGeYYmtoo+Ilo+hYYd6k76OF3fjQfn6z52FmBUmPy2RU6z28w X-Received: from pjbdy12.prod.google.com ([2002:a17:90b:6cc:b0:311:462d:cb60]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:53c5:b0:312:1508:fb4e with SMTP id 98e67ed59e1d1-315f2675bbbmr6780975a91.17.1750883483656; Wed, 25 Jun 2025 13:31:23 -0700 (PDT) Date: Wed, 25 Jun 2025 20:30:57 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-5-jstultz@google.com> Subject: [PATCH v18 4/8] sched: Move update_curr_task logic into update_curr_se From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Absorb update_curr_task() into update_curr_se(), and in the process simplify update_curr_common(). This will make the next step a bit easier. Signed-off-by: John Stultz Tested-by: K Prateek Nayak --- Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/fair.c | 26 +++++++++----------------- 1 file changed, 9 insertions(+), 17 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 7a14da5396fb2..edcc7d59ecc3b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1155,6 +1155,14 @@ static s64 update_curr_se(struct rq *rq, struct sche= d_entity *curr) curr->exec_start =3D now; curr->sum_exec_runtime +=3D delta_exec; =20 + if (entity_is_task(curr)) { + struct task_struct *p =3D task_of(curr); + + trace_sched_stat_runtime(p, delta_exec); + account_group_exec_runtime(p, delta_exec); + cgroup_account_cputime(p, delta_exec); + } + if (schedstat_enabled()) { struct sched_statistics *stats; =20 @@ -1166,13 +1174,6 @@ static s64 update_curr_se(struct rq *rq, struct sche= d_entity *curr) return delta_exec; } =20 -static inline void update_curr_task(struct task_struct *p, s64 delta_exec) -{ - trace_sched_stat_runtime(p, delta_exec); - account_group_exec_runtime(p, delta_exec); - cgroup_account_cputime(p, delta_exec); -} - static inline bool did_preempt_short(struct cfs_rq *cfs_rq, struct sched_e= ntity *curr) { if (!sched_feat(PREEMPT_SHORT)) @@ -1211,13 +1212,8 @@ static inline bool do_preempt_short(struct cfs_rq *c= fs_rq, s64 update_curr_common(struct rq *rq) { struct task_struct *donor =3D rq->donor; - s64 delta_exec; =20 - delta_exec =3D update_curr_se(rq, &donor->se); - if (likely(delta_exec > 0)) - update_curr_task(donor, delta_exec); - - return delta_exec; + return update_curr_se(rq, &donor->se); } =20 /* @@ -1242,10 +1238,6 @@ static void update_curr(struct cfs_rq *cfs_rq) update_min_vruntime(cfs_rq); =20 if (entity_is_task(curr)) { - struct task_struct *p =3D task_of(curr); - - update_curr_task(p, delta_exec); - /* * If the fair_server is active, we need to account for the * fair_server time whether or not the task is running on --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36B3F2DFF25 for ; Wed, 25 Jun 2025 20:31:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883487; cv=none; b=JSkswwHnihwAqNFXFlvtRLCG++vaQ6MXxyPtaAPLAhXNa0zlkd0xqXwRQxxpacANQe9cE1K0LhFelYRzszHPKvMWS9EVJWH6gfeW81h5x9AQH62wtH1rR8q+ORYyO32BN+gK9YzuYDDhYlaUdJH1zzGNVOCpKuytQwVq5KRQ9Xw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883487; c=relaxed/simple; bh=sOZ4t17AJ5iVPp4wVyz1H8xJqd/fbypdGdkeH5mDiiI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uwIZACTJFGtA2G3ToU4jsRM3PmzXgcxnBVxxO6cS/nur/mRllzKyryQyZWAeicylKtrS5TCYmAG4RQwW+Clt9TZXSpfPZiFKyZ2DcYOxbmHbwdVYg/5DkEsSJd07fXyiwZ/jvPsADzTuxvR/rFaU0kR3U39iTdQd2Y6N1wgPz3w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=K42HQz2o; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="K42HQz2o" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-b2c36951518so240107a12.2 for ; Wed, 25 Jun 2025 13:31:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883485; x=1751488285; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fPeymT7NIAooWQEri2B3P7iuuC6t76eak7WRt7BTTBs=; b=K42HQz2oqDl0fWq/vbA4gozzTJUdR74ThuC3FOFmmP2qwqPzqEkmEroShUaFhudJnF 7M6HlYe6vo10FPOfREe4oxbahmNtbKX81ZGHrpqNqXsC7S1wSwOgGH0PF1FdM+f1phId gR8+XPPGSNzJcwTmzONqdvWbMxdaFf5iG83gYY0DmjeS2SJSBaWrUlnIxxb5rtPJE4aT lHK0EFVO/cxR/+gs9FRyo6dibTM7eo/gh/DdD3ux6qapXWcphfhviJXaUlC3p+jFQ/9E 49q0Y+77Z7nNxL7XIvKC1jAs6X/p91NOzwlFJieopTi0P2jm8O9WwOl2LvSO6OoVMLcH 0Qzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883485; x=1751488285; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fPeymT7NIAooWQEri2B3P7iuuC6t76eak7WRt7BTTBs=; b=QQZt5Zo280cqu2hsl7sxyPt2HhLXWW9N7VtgGPCWsOrICytRfJJW3jR1hAHEllTGgv 5IHEzGTt/aylKKtv1C15hWvcelHUAKkaP/KYVS0t/uBwYnU0yWALaCPayaBVMiDPpDb9 NNcAYh3CVE+4cXbcx3V1U/QXXtxMEa3qBwBxJD748P+pg8d4LWPgPb5rM/3i6+yMs6MY HzifPJy7MmvXqS9M6l/6tx60vp1d+smfgHmgs3ZvtnX9mIzcNMpAF3hGDcdt1DVOelWG wqrxaD2K+9s0Iw/+njcIvA4TaQRc0M/jWRYg+WDXSfybZcBnOXFnOGrrPDfplYOrQO3u tmjQ== X-Gm-Message-State: AOJu0YwlDx37/orOZX6OW41bqQerkXSZHlmwNOqPJQlE5cIdLUD8Syzx zW0OJsIk1ggeFMeZuzrk3AAfD/4dq60fkgaqEb/dQQt5P53PUa/wOVMv4klNR5aIEaoHJmkJQWw xnvib3nOngIMv4MkVjvQPqaJm2zaU7NANxHVNFN2Nal3qxeWU1aqYVzZ8Z6B53z2ZwcRTR1Fx8r WyMkpW0ccPAJ9t6Zx7ZVjz5gStnDSq651FEVPOuwzPUnmWkW2f X-Google-Smtp-Source: AGHT+IHWv7rw9xYClYLSpjznGzC3DSbsn/7oFCuojbsln+hYJ4FF6RN2RlQz5mNTrLiP9vNO64bcvauzZAdH X-Received: from pgar3.prod.google.com ([2002:a05:6a02:2e83:b0:b31:d4af:e567]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:2444:b0:1ee:e33d:f477 with SMTP id adf61e73a8af0-2207f19c2d0mr8118560637.15.1750883485284; Wed, 25 Jun 2025 13:31:25 -0700 (PDT) Date: Wed, 25 Jun 2025 20:30:58 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-6-jstultz@google.com> Subject: [PATCH v18 5/8] sched: Fix runtime accounting w/ split exec & sched contexts From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Without proxy-exec, we normally charge the "current" task for both its vruntime as well as its sum_exec_runtime. With proxy, however, we have two "current" contexts: the scheduler context and the execution context. We want to charge the execution context rq->curr (ie: proxy/lock holder) execution time to its sum_exec_runtime (so it's clear to userland the rq->curr task *is* running), as well as its thread group. However the rest of the time accounting (such a vruntime and cgroup accounting), we charge against the scheduler context (rq->donor) task, because it is from that task that the time is being "donated". If the donor and curr tasks are the same, then it's the same as without proxy. Signed-off-by: John Stultz Tested-by: K Prateek Nayak --- v16: * Renamed update_curr_se to update_se_times, as suggested by Steven Rostedt. * Reworded the commit message as suggested by Steven Rostedt v17: * Renamed update_se_times to update_se, as suggested by Peter * Added comment calrifying cfs_rq->curr being rq->donor.se as suggested by Peter * Reworked to fix issue Peter pointed out with thread group accounting being done on the donor, rather then the running execution context. * Slight rewording of the commit message to further clairifiy things. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index edcc7d59ecc3b..c34e0891193a7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1143,30 +1143,40 @@ static void update_tg_load_avg(struct cfs_rq *cfs_r= q) } #endif /* CONFIG_SMP */ =20 -static s64 update_curr_se(struct rq *rq, struct sched_entity *curr) +static s64 update_se(struct rq *rq, struct sched_entity *se) { u64 now =3D rq_clock_task(rq); s64 delta_exec; =20 - delta_exec =3D now - curr->exec_start; + delta_exec =3D now - se->exec_start; if (unlikely(delta_exec <=3D 0)) return delta_exec; =20 - curr->exec_start =3D now; - curr->sum_exec_runtime +=3D delta_exec; + se->exec_start =3D now; + if (entity_is_task(se)) { + struct task_struct *donor =3D task_of(se); + struct task_struct *running =3D rq->curr; + /* + * If se is a task, we account the time against the running + * task, as w/ proxy-exec they may not be the same. + */ + running->se.exec_start =3D now; + running->se.sum_exec_runtime +=3D delta_exec; =20 - if (entity_is_task(curr)) { - struct task_struct *p =3D task_of(curr); + trace_sched_stat_runtime(running, delta_exec); + account_group_exec_runtime(running, delta_exec); =20 - trace_sched_stat_runtime(p, delta_exec); - account_group_exec_runtime(p, delta_exec); - cgroup_account_cputime(p, delta_exec); + /* cgroup time is always accounted against the donor */ + cgroup_account_cputime(donor, delta_exec); + } else { + /* If not task, account the time against donor se */ + se->sum_exec_runtime +=3D delta_exec; } =20 if (schedstat_enabled()) { struct sched_statistics *stats; =20 - stats =3D __schedstats_from_se(curr); + stats =3D __schedstats_from_se(se); __schedstat_set(stats->exec_max, max(delta_exec, stats->exec_max)); } @@ -1213,7 +1223,7 @@ s64 update_curr_common(struct rq *rq) { struct task_struct *donor =3D rq->donor; =20 - return update_curr_se(rq, &donor->se); + return update_se(rq, &donor->se); } =20 /* @@ -1221,6 +1231,12 @@ s64 update_curr_common(struct rq *rq) */ static void update_curr(struct cfs_rq *cfs_rq) { + /* + * Note: cfs_rq->curr corresponds to the task picked to + * run (ie: rq->donor.se) which due to proxy-exec may + * not necessarily be the actual task running + * (rq->curr.se). This is easy to confuse! + */ struct sched_entity *curr =3D cfs_rq->curr; struct rq *rq =3D rq_of(cfs_rq); s64 delta_exec; @@ -1229,7 +1245,7 @@ static void update_curr(struct cfs_rq *cfs_rq) if (unlikely(!curr)) return; =20 - delta_exec =3D update_curr_se(rq, curr); + delta_exec =3D update_se(rq, curr); if (unlikely(delta_exec <=3D 0)) return; =20 --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94FDE2E0B64 for ; Wed, 25 Jun 2025 20:31:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883489; cv=none; b=jo3XZlVO0zhZsb7fcoqC9vGaO2W6IS1jM/lMVq/qVZrAO/Vc+6yYnEs2+zzck3nifb8sj61zLhYYgrwABKKvfX/imvwNX8NeehY43/BFn7II2dKNYLbOVi2eZ8ul6PqDdJXsIEhCozmYzuX4/sJRzBZMNXsHqBAYKd09Nk3bs3Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883489; c=relaxed/simple; bh=7Gbbe7DmrjCrYsDM9bnRRyN0PD8a0t+/UhqfWT8utV8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BIv5IzeYnz4vhbCBGG2U7jl60vUmAE/pfGHUlO8NR41PHo9wt897HMj2kNKZGYNnNRkqDzXQ9fK3tbIzumDM6UlaIWaSRovz+DSQ5JLKcb4DJRXP17wdsHNN97kDi93qDLAFAwVVZgbBobreeavjHNFhla1n4EDDjkrZsRAI5oo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kN+cbraF; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kN+cbraF" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-313c3915345so274419a91.3 for ; Wed, 25 Jun 2025 13:31:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883487; x=1751488287; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Vb+m0zzjHPVW7QgH/i+p2jYSCiuiyBNV0PMACERIPxY=; b=kN+cbraFspWqyc4L2CPS6S3BcAnlTfEOem7AeDkJOfNIsIprR6cfznnoS4ME5Qhjw6 +R/7m+812z7dHCgst2fGp4CTaBD6Ck00R8kz+0p6sf5mGI2fbnO+NpkI+8P1C5VTS11W sSacZRjmhn6wYSB7MNFMJAVqbGrVvEV3VIAlvScIcPbn/epBUBzl5/Ef95Sw0/O0RXs8 K8Gzwapo2fuyExvsZEktNzTn0msQo5gSngqkSIiuGY9LnESiylbN+wsUkSujwJJ9BRJe j4RUcRPTAeJhUscfPrvPtwqUV1x1rTFJy+CU5fqxtVELHEf3p/lS6XwxUCx2548fDxdV hlNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883487; x=1751488287; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Vb+m0zzjHPVW7QgH/i+p2jYSCiuiyBNV0PMACERIPxY=; b=RQOeB1rZq7MuJHAVQY7TsNcitJ0x3Zlezb8lib9tdJJlaf1p5wH8LGz5OaUDCd6swr yPX3IQkpwTySJA/o1iMHaHCinlQBEQjGZaeVDF4bD8XyDxNzP10JTJA5Vp0CHxvqx2pE GrveiaMKwmEyaI9QjN4qqa/eCtVN+dMdpM9iYQDv92k4eUCa95cg8uWS0w5gd+SQFvvs A5iBXPdxyt1F0MtoAla/wcQKDtb2agBYuZyi89fWu+2QPviL7sQJQ6jfH9GZ7OsyDdJK ejZ1yCVt3GVywe/H6XlAlzEZ0+wRJsqlkIz++9uvFKjjZZujFVpZ80LcsjydubHMd2MB 6szQ== X-Gm-Message-State: AOJu0YxLcRhRuD+tEcFV+2EdmG3SGeSbmdWyfLA8YKXY4pg8uX9cDDXX Xm4HFe3RRyS6YCi0ka5rIEnF5to9oQMGGJ/3+vOhXMrtJuOGBCqJNXG+rhiKMZOc9qRNfoaIeFs 4WdeBKgvHimx7P1JfxdwXrgqh+WJEjJ3JlDKLMWPdQAoZ2jsKgjE7CYOHoxhy1CUBJNqoUYGj7R Hsyo6UxmFSjzzb3lv2NL7nQhhS3KQ2BcsPbWyYFObiB4D678Na X-Google-Smtp-Source: AGHT+IEMawx/CbvxPo4TMaer7VeKx3+9OKJyDqLjKr+lWkY/rS1bfp0rx+UN29DaGPxjqwR+TeQlirmkjawr X-Received: from pjbpv1.prod.google.com ([2002:a17:90b:3c81:b0:313:2d44:397b]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2884:b0:311:ba2e:bdc9 with SMTP id 98e67ed59e1d1-315f26893bemr6517524a91.27.1750883486877; Wed, 25 Jun 2025 13:31:26 -0700 (PDT) Date: Wed, 25 Jun 2025 20:30:59 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-7-jstultz@google.com> Subject: [PATCH v18 6/8] sched: Add an initial sketch of the find_proxy_task() function From: John Stultz To: LKML Cc: John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a find_proxy_task() function which doesn't do much. When we select a blocked task to run, we will just deactivate it and pick again. The exception being if it has become unblocked after find_proxy_task() was called. This allows us to validate keeping blocked tasks on the runqueue and later deactivating them is working ok, stressing the failure cases for when a proxy isn't found. Greatly simplified from patch by: Peter Zijlstra (Intel) Juri Lelli Valentin Schneider Connor O'Brien [jstultz: Split out from larger proxy patch and simplified for review and testing.] Signed-off-by: John Stultz Tested-by: K Prateek Nayak --- v5: * Split out from larger proxy patch v7: * Fixed unused function arguments, spelling nits, and tweaks for clarity, pointed out by Metin Kaya * Fix build warning Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202311081028.yDLmCWgr-lkp@i= ntel.com/ v8: * Fixed case where we might return a blocked task from find_proxy_task() * Continued tweaks to handle avoiding returning blocked tasks v9: * Add zap_balance_callbacks helper to unwind balance_callbacks when we will re-call pick_next_task() again. * Add extra comment suggested by Metin * Typo fixes from Metin * Moved adding proxy_resched_idle earlier in the series, as suggested by Metin * Fix to call proxy_resched_idle() *prior* to deactivating next, to avoid crashes caused by stale references to next * s/PROXY/SCHED_PROXY_EXEC/ as suggested by Metin * Number of tweaks and cleanups suggested by Metin * Simplify proxy_deactivate as suggested by Metin v11: * Tweaks for earlier simplification in try_to_deactivate_task v13: * Rename rename "next" to "donor" in find_proxy_task() for clarity * Similarly use "donor" instead of next in proxy_deactivate * Refactor/simplify proxy_resched_idle * Moved up a needed fix from later in the series v15: * Tweaked some comments to better explain the initial sketch of find_proxy_task(), suggested by Qais * Build fixes for !CONFIG_SMP * Slight rework for blocked_on_state being added later in the series. * Move the zap_balance_callbacks to later in the patch series v16: * Move the enqueue_task_rt() out to later in the series, as suggested by K Prateek Nayak * Fixup whitespace error pointed out by K Prateek Nayak * Use put_prev_set_next_task as suggested by K Prateek Nayak * Try to rework find_proxy_task() locking to use guard and proxy_deactivate_task() in the way Peter suggested. v17: * Slightly simplified variable names per suggestion from Juri Lelli * Minor comment and commit message tweaks suggested by Peter Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 100 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 10 ++++- 2 files changed, 106 insertions(+), 4 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 853157b27f384..dc82d9b8bee2c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6614,7 +6614,8 @@ pick_next_task(struct rq *rq, struct task_struct *pre= v, struct rq_flags *rf) * Otherwise marks the task's __state as RUNNING */ static bool try_to_block_task(struct rq *rq, struct task_struct *p, - unsigned long *task_state_p) + unsigned long *task_state_p, + bool deactivate_cond) { unsigned long task_state =3D *task_state_p; int flags =3D DEQUEUE_NOCLOCK; @@ -6625,6 +6626,9 @@ static bool try_to_block_task(struct rq *rq, struct t= ask_struct *p, return false; } =20 + if (!deactivate_cond) + return false; + p->sched_contributes_to_load =3D (task_state & TASK_UNINTERRUPTIBLE) && !(task_state & TASK_NOLOAD) && @@ -6648,6 +6652,89 @@ static bool try_to_block_task(struct rq *rq, struct = task_struct *p, return true; } =20 +#ifdef CONFIG_SCHED_PROXY_EXEC +static inline struct task_struct *proxy_resched_idle(struct rq *rq) +{ + put_prev_set_next_task(rq, rq->donor, rq->idle); + rq_set_donor(rq, rq->idle); + set_tsk_need_resched(rq->idle); + return rq->idle; +} + +static bool __proxy_deactivate(struct rq *rq, struct task_struct *donor) +{ + unsigned long state =3D READ_ONCE(donor->__state); + + /* Don't deactivate if the state has been changed to TASK_RUNNING */ + if (state =3D=3D TASK_RUNNING) + return false; + /* + * Because we got donor from pick_next_task(), it is *crucial* + * that we call proxy_resched_idle() before we deactivate it. + * As once we deactivate donor, donor->on_rq is set to zero, + * which allows ttwu() to immediately try to wake the task on + * another rq. So we cannot use *any* references to donor + * after that point. So things like cfs_rq->curr or rq->donor + * need to be changed from next *before* we deactivate. + */ + proxy_resched_idle(rq); + return try_to_block_task(rq, donor, &state, true); +} + +static struct task_struct *proxy_deactivate(struct rq *rq, struct task_str= uct *donor) +{ + if (!__proxy_deactivate(rq, donor)) { + /* + * XXX: For now, if deactivation failed, set donor + * as unblocked, as we aren't doing proxy-migrations + * yet (more logic will be needed then). + */ + donor->blocked_on =3D NULL; + } + return NULL; +} + +/* + * Initial simple sketch that just deactivates the blocked task + * chosen by pick_next_task() so we can then pick something that + * isn't blocked. + */ +static struct task_struct * +find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags = *rf) +{ + struct mutex *mutex; + + mutex =3D donor->blocked_on; + /* Something changed in the chain, so pick again */ + if (!mutex) + return NULL; + /* + * By taking mutex->wait_lock we hold off concurrent mutex_unlock() + * and ensure @owner sticks around. + */ + guard(raw_spinlock)(&mutex->wait_lock); + + /* Check again that donor is blocked with blocked_lock held */ + if (!task_is_blocked(donor) || mutex !=3D __get_task_blocked_on(donor)) { + /* + * Something changed in the blocked_on chain and + * we don't know if only at this level. So, let's + * just bail out completely and let __schedule() + * figure things out (pick_again loop). + */ + return NULL; /* do pick_next_task() again */ + } + return proxy_deactivate(rq, donor); +} +#else /* SCHED_PROXY_EXEC */ +static struct task_struct * +find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags = *rf) +{ + WARN_ONCE(1, "This should never be called in the !SCHED_PROXY_EXEC case\n= "); + return donor; +} +#endif /* SCHED_PROXY_EXEC */ + /* * __schedule() is the main scheduler function. * @@ -6760,12 +6847,19 @@ static void __sched notrace __schedule(int sched_mo= de) goto picked; } } else if (!preempt && prev_state) { - try_to_block_task(rq, prev, &prev_state); + try_to_block_task(rq, prev, &prev_state, + !task_is_blocked(prev)); switch_count =3D &prev->nvcsw; } =20 - next =3D pick_next_task(rq, prev, &rf); +pick_again: + next =3D pick_next_task(rq, rq->donor, &rf); rq_set_donor(rq, next); + if (unlikely(task_is_blocked(next))) { + next =3D find_proxy_task(rq, next, &rf); + if (!next) + goto pick_again; + } picked: clear_tsk_need_resched(prev); clear_preempt_need_resched(); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 6b61e0c7f6e78..590a44c0215fb 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2290,6 +2290,14 @@ static inline int task_current_donor(struct rq *rq, = struct task_struct *p) return rq->donor =3D=3D p; } =20 +static inline bool task_is_blocked(struct task_struct *p) +{ + if (!sched_proxy_exec()) + return false; + + return !!p->blocked_on; +} + static inline int task_on_cpu(struct rq *rq, struct task_struct *p) { #ifdef CONFIG_SMP @@ -2499,7 +2507,7 @@ static inline void put_prev_set_next_task(struct rq *= rq, struct task_struct *prev, struct task_struct *next) { - WARN_ON_ONCE(rq->curr !=3D prev); + WARN_ON_ONCE(rq->donor !=3D prev); =20 __put_prev_set_next_dl_server(rq, prev, next); =20 --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B7022E11CD for ; Wed, 25 Jun 2025 20:31:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883491; cv=none; b=aFrVEtRTQEE71pSHTPHAXyBRhjT5e3BpCNSER5PbPFfBq7Rov0TD3DXfvE12/9lyb5bFbAuybY7hzZMb21iLIGzTklKOj5GQnJ3uoxNTbyd3LVgLSeYBwLbZA7pMONyzYReUnYLDED07c0JbJrYmM4DBCluiiTUy54DZ/525YLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883491; c=relaxed/simple; bh=XJ+zB5vfjgj/VuFNky+aSN6sseEPlMbV05wwGCrI/A0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dW5vJhYkTuWuoETCoBRZnGjFQcIhWWJx6qg1etnw7HB0E8djELnHsa3FCXPDRguy4XbHeBJzxgdtQ0qD8QK9qblAFnC7zFnCxtlrnDDH1tG7QxXm+oaMGWYGM01/gb79Vw8V2HlYa4Gu9gWe+YHbF7LYNPe3xl217A8tDjV+G+M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rMHXN94Q; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rMHXN94Q" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-b2eb60594e8so128413a12.1 for ; Wed, 25 Jun 2025 13:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883488; x=1751488288; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7Rq31Hw/FTi2WUEI+nbUuOFryI5Rf1pz1yynmS6irFs=; b=rMHXN94QnDs8G1/NFuqgHVv/7SEQbH9kSfyAKIQWa2+p2ThRThGBMDhQnv2M+UQtTb Mr6cE+9KknATc2IHdJKyPQSP+j+Rpk5teb1zVuoTsE27h+ADRGC2IhhpxyEoMuH2lCb+ kA/UHbeQvNb2i+MZVifRXnfuM5NaSPDmIKp1ZQJDZiGH+zT+iVEhKkRxTYmbrxXr0fiT N0XZeRucTUMyZuI/SOuh2/hgaTawVsxDSHGIIWIkiyBwKTnikagG2EhIgeMJIkl9v4kT +CTSBwQw3qBfXWG+9K41tM+HorBhX6vAsw/xbfUqVTczg21D67LmsPrnUU0tWkfMf7zp 6+5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883488; x=1751488288; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7Rq31Hw/FTi2WUEI+nbUuOFryI5Rf1pz1yynmS6irFs=; b=t+7Yyf81cf0o6BMCYmwHUDOMfSwZxOEwh/OkAgM1LXRoLLPXqSOmAqK1yL+0xIvnQL DrKel+5wUbv7BBF3smKqZuYE9Ip/uF03gwvIbdB4VQDsPS3Z24qc72QwzBXjxs6OvUKe bmhGnLHS4IdgJZJyeIrtf00C+EqaIxkh4wul2fmyTTN4vzQfzPQIpcMC6N/hYl8DvQyl mW2uCmgk3jsDzRTj+HzfwHvvYYHqhOHYpzbn0NpndpgC9iWzHzVQFLQQKpqHiDJOe1+R gK6PbBnClz8iCCMhsM38rqk6StXxcBgvBWKFZJ03nxiFohxbnOG6qJ+muN+1aH2+YRPu uMcw== X-Gm-Message-State: AOJu0YzMcblOm/7ByhED5GXGQHYqPfyOnYQUyKFbr9LZJzKGF48ADq1A 0wGz3Sune3S1k116GF44pElEKuonAqPwcTrvGs8xzc1ri//2Ur3Gi3A02MpyqG2n04v0tJ4ufxe S2OoY9WllSm/GBM84R35D8JXHsGH++7Y2Hwjsoo2io3+xe8DI5GjdMWXyMyw8GnnmUAMnPUn5il bY42SEPtEaFOs8A4bcM4BTZt+EbYOsIQNr8LRrVne4prnQ/95/ X-Google-Smtp-Source: AGHT+IEIJjMyvJHQ8RtqqWnbiH/IcHGJSBO4XdkrUZuvBBBiYERZZhj8nO0yMIxtjLUpxd30/hXbupIWFWZa X-Received: from pfbiv18.prod.google.com ([2002:a05:6a00:66d2:b0:748:ec4d:30ec]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:9009:b0:21a:b9d4:ad73 with SMTP id adf61e73a8af0-2207f2bd033mr8463572637.40.1750883488323; Wed, 25 Jun 2025 13:31:28 -0700 (PDT) Date: Wed, 25 Jun 2025 20:31:00 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-8-jstultz@google.com> Subject: [PATCH v18 7/8] sched: Fix proxy/current (push,pull)ability From: John Stultz To: LKML Cc: Valentin Schneider , "Connor O'Brien" , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Valentin Schneider Proxy execution forms atomic pairs of tasks: The waiting donor task (scheduling context) and a proxy (execution context). The donor task, along with the rest of the blocked chain, follows the proxy wrt CPU placement. They can be the same task, in which case push/pull doesn't need any modification. When they are different, however, FIFO1 & FIFO42: ,-> RT42 | | blocked-on | v blocked_donor | mutex | | owner | v `-- RT1 RT1 RT42 CPU0 CPU1 ^ ^ | | overloaded !overloaded rq prio =3D 42 rq prio =3D 0 RT1 is eligible to be pushed to CPU1, but should that happen it will "carry" RT42 along. Clearly here neither RT1 nor RT42 must be seen as push/pullable. Unfortunately, only the donor task is usually dequeued from the rq, and the proxy'ed execution context (rq->curr) remains on the rq. This can cause RT1 to be selected for migration from logic like the rt pushable_list. Thus, adda a dequeue/enqueue cycle on the proxy task before __schedule returns, which allows the sched class logic to avoid adding the now current task to the pushable_list. Furthermore, tasks becoming blocked on a mutex don't need an explicit dequeue/enqueue cycle to be made (push/pull)able: they have to be running to block on a mutex, thus they will eventually hit put_prev_task(). Signed-off-by: Valentin Schneider Signed-off-by: Connor O'Brien Signed-off-by: John Stultz Tested-by: K Prateek Nayak --- v3: * Tweaked comments & commit message v5: * Minor simplifications to utilize the fix earlier in the patch series. * Rework the wording of the commit message to match selected/ proxy terminology and expand a bit to make it more clear how it works. v6: * Dropped now-unused proxied value, to be re-added later in the series when it is used, as caught by Dietmar v7: * Unused function argument fixup * Commit message nit pointed out by Metin Kaya * Dropped unproven unlikely() and use sched_proxy_exec() in proxy_tag_curr, suggested by Metin Kaya v8: * More cleanups and typo fixes suggested by Metin Kaya v11: * Cleanup of comimt message suggested by Metin v12: * Rework for rq_selected -> rq->donor renaming v16: * Pulled logic from later patch in to avoid sched_balance migrating blocked tasks. * Moved enqueue_task_rt() logic from earlier into this patch as suggested by K Prateek Nayak * Simplified changes to enqueue_task_rt to match deadline's logic, as pointed out by Peter v17: * Add similar logic to put_prev_task_dl as pointed out by K Prateek Nayak Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/sched/core.c | 25 +++++++++++++++++++++++++ kernel/sched/deadline.c | 7 +++++++ kernel/sched/rt.c | 5 +++++ 3 files changed, 37 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index dc82d9b8bee2c..98a802e0a3f36 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6735,6 +6735,23 @@ find_proxy_task(struct rq *rq, struct task_struct *d= onor, struct rq_flags *rf) } #endif /* SCHED_PROXY_EXEC */ =20 +static inline void proxy_tag_curr(struct rq *rq, struct task_struct *owner) +{ + if (!sched_proxy_exec()) + return; + /* + * pick_next_task() calls set_next_task() on the chosen task + * at some point, which ensures it is not push/pullable. + * However, the chosen/donor task *and* the mutex owner form an + * atomic pair wrt push/pull. + * + * Make sure owner we run is not pushable. Unfortunately we can + * only deal with that by means of a dequeue/enqueue cycle. :-/ + */ + dequeue_task(rq, owner, DEQUEUE_NOCLOCK | DEQUEUE_SAVE); + enqueue_task(rq, owner, ENQUEUE_NOCLOCK | ENQUEUE_RESTORE); +} + /* * __schedule() is the main scheduler function. * @@ -6873,6 +6890,10 @@ static void __sched notrace __schedule(int sched_mod= e) * changes to task_struct made by pick_next_task(). */ RCU_INIT_POINTER(rq->curr, next); + + if (!task_current_donor(rq, next)) + proxy_tag_curr(rq, next); + /* * The membarrier system call requires each architecture * to have a full memory barrier after updating @@ -6907,6 +6928,10 @@ static void __sched notrace __schedule(int sched_mod= e) /* Also unlocks the rq: */ rq =3D context_switch(rq, prev, next, &rf); } else { + /* In case next was already curr but just got blocked_donor */ + if (!task_current_donor(rq, next)) + proxy_tag_curr(rq, next); + rq_unpin_lock(rq, &rf); __balance_callbacks(rq); raw_spin_rq_unlock_irq(rq); diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index ad45a8fea245e..4757c29d0979e 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2166,6 +2166,9 @@ static void enqueue_task_dl(struct rq *rq, struct tas= k_struct *p, int flags) if (dl_server(&p->dl)) return; =20 + if (task_is_blocked(p)) + return; + if (!task_current(rq, p) && !p->dl.dl_throttled && p->nr_cpus_allowed > 1) enqueue_pushable_dl_task(rq, p); } @@ -2465,6 +2468,10 @@ static void put_prev_task_dl(struct rq *rq, struct t= ask_struct *p, struct task_s update_curr_dl(rq); =20 update_dl_rq_load_avg(rq_clock_pelt(rq), rq, 1); + + if (task_is_blocked(p)) + return; + if (on_dl_rq(&p->dl) && p->nr_cpus_allowed > 1) enqueue_pushable_dl_task(rq, p); } diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index e40422c370335..b1fc9e6cf4a60 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1488,6 +1488,9 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p,= int flags) =20 enqueue_rt_entity(rt_se, flags); =20 + if (task_is_blocked(p)) + return; + if (!task_current(rq, p) && p->nr_cpus_allowed > 1) enqueue_pushable_task(rq, p); } @@ -1768,6 +1771,8 @@ static void put_prev_task_rt(struct rq *rq, struct ta= sk_struct *p, struct task_s =20 update_rt_rq_load_avg(rq_clock_pelt(rq), rq, 1); =20 + if (task_is_blocked(p)) + return; /* * The previous task needs to be made eligible for pushing * if it is still active --=20 2.50.0.727.gbf7dc18ff4-goog From nobody Wed Oct 8 16:39:47 2025 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A96632DCC0C for ; Wed, 25 Jun 2025 20:31:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883492; cv=none; b=ltVNrIv/QwlNTBZZs9+P5OiPb3jmSidCWfjeof0bHoWZujNVFi0PSSo3R5vGY7jYKq10zlPyKDjuYUx07nPoihtGyRwEw382KxHbWcw2ESoorMBxBDGUn9D6kCQL5zZTxammssaxIYLZQ0qRBI1O+4wfISFyTC5lDheo6QZnxbA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750883492; c=relaxed/simple; bh=eEGbTUhehreNHJJcHfYFZqQbpQHx5PlflpLC6KOfwLo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HOG4Xc6tbTCd+XpmoTlWCoc2lmsf9OdE9KrXc7V0hFpxusd8QFe8bA7d2YSehqIjjG7seXdMjA+YFTmr7SOZZXKSG0owC4O9o4ba+y4xCQKa3fbJA6+jJaBV1Nu2zB+P+VIgxbM0d9cWnkKzCI7GfNhSPvp04upRCVrMqok/gi0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Jo5R9Auf; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Jo5R9Auf" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-748f3d4c7e7so239161b3a.3 for ; Wed, 25 Jun 2025 13:31:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1750883490; x=1751488290; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yGWjwWeoIfXjnW6VujAMlFOzBqf8NJo/hzIiehZaI4M=; b=Jo5R9AufxBdN96dV06AmJ21iybtZU/8iHyOpOMs2rEsXeuBEOBM1x0+b4QssgW8PZ/ z1cXQxx/ZQourEkwATIljPOaNtSnxUcC1c5YxoSj9YGGCK5I0gYJjqjQBmkbYNBnyeui q5FcjJwE8ZnourSHueKyYXf6dNwxdkrjEkR4CNSxicSScKCuTPWz+r+5Atc3mCm8WED4 tJ2lJ3+zk/5uA/IGvpCSwb5fUvYkMVI3/aKUhVCcq7Qk94BZvm6QEDqBBqRgFeMrzmXC Vc19g6ach+rEtnb1KVWors6cZ5HeuAtEBoS62ZNqM7BOjgPqFbIu5Z6861xJ5YVVxKnr uMzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750883490; x=1751488290; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yGWjwWeoIfXjnW6VujAMlFOzBqf8NJo/hzIiehZaI4M=; b=AR/IB+WaQNp3blQIXO5jHt7Arms5+4sQskekVuwqpyuxDQlhsSmtvZJQvol0iwSMPK kbJNtc94JZ1fgv7cWheZBxi3yEX6h6UmvzWNGI2fTbokIwRQRNf1hCDwSfyv/m0PLwLN qLI1RRXfmum6RfDOi4PleJvGroDPQBTKfRwqQa37NT30sFH2YlrRyVO9p+NYR5uSjL4Y YGIo8JV1VgZr5ydjWRBfv3aqbB3BqZBvJO9JOXKoQx2jsak4nGRPNY0MsOx8vM5cfd1A TjqPap4rkZYP2v1zWS7OJRQVEbJ2jWpGrJpj54z4o3G6zUNO9uudY+LdvVezLINjv9yL qAZg== X-Gm-Message-State: AOJu0Yy1qaj0yorkrLhKD1HTky0kdE+crLrEvNzfj0ESTRBhIlfaRV3T trJEnPUl6CJn7jNqpixvbo8QBZ3Ntw1LdGjJXsQkH19NA+mxin0gTRGPYu14FhCVwLXjhqwVrgd f1y/tTCH+U0ifjUHvG1eKudKba9qr97MOPbudWF15q2CCYxUhTeRlir002e4vjmGSeh7tC4LsXR zadzByj9Jza7btFh5CE0ccLsRwwmZ5PwzCWWXz7bqXueg/85We X-Google-Smtp-Source: AGHT+IEqs5f8XRv7rHPGYoDkgavkdCUmUkG8ZyD9wKvrccGxxaEgqULooPwub1kfQqVHhIN1j9jPGtID56OS X-Received: from pgy33.prod.google.com ([2002:a63:1861:0:b0:b31:cdc2:ff97]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:3298:b0:1f5:591b:4f7a with SMTP id adf61e73a8af0-2207f31ac43mr7293136637.38.1750883489863; Wed, 25 Jun 2025 13:31:29 -0700 (PDT) Date: Wed, 25 Jun 2025 20:31:01 +0000 In-Reply-To: <20250625203110.2299275-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250625203110.2299275-1-jstultz@google.com> X-Mailer: git-send-email 2.50.0.727.gbf7dc18ff4-goog Message-ID: <20250625203110.2299275-9-jstultz@google.com> Subject: [PATCH v18 8/8] sched: Start blocked_on chain processing in find_proxy_task() From: John Stultz To: LKML Cc: Peter Zijlstra , Juri Lelli , Valentin Schneider , "Connor O'Brien" , John Stultz , Joel Fernandes , Qais Yousef , Ingo Molnar , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Mel Gorman , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , Daniel Lezcano , Suleiman Souhlal , kuyo chang , hupu , kernel-team@android.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra Start to flesh out the real find_proxy_task() implementation, but avoid the migration cases for now, in those cases just deactivate the donor task and pick again. To ensure the donor task or other blocked tasks in the chain aren't migrated away while we're running the proxy, also tweak the fair class logic to avoid migrating donor or mutex blocked tasks. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Juri Lelli Signed-off-by: Valentin Schneider Signed-off-by: Connor O'Brien [jstultz: This change was split out from the larger proxy patch] Signed-off-by: John Stultz Tested-by: K Prateek Nayak --- v5: * Split this out from larger proxy patch v7: * Minor refactoring of core find_proxy_task() function * Minor spelling and corrections suggested by Metin Kaya * Dropped an added BUG_ON that was frequently tripped v8: * Fix issue if proxy_deactivate fails, we don't leave task BO_BLOCKED * Switch to WARN_ON from BUG_ON checks v9: * Improve comments suggested by Metin * Minor cleanups v11: * Previously we checked next=3D=3Drq->idle && prev=3D=3Drq->idle, but I think we only really care if next=3D=3Drq->idle from find_proxy_task, as we will still want to resched regardless of what prev was. v12: * Commit message rework for selected -> donor rewording v13: * Address new delayed dequeue condition (deactivate donor for now) * Next to donor renaming in find_proxy_task * Improved comments for find_proxy_task * Rework for proxy_deactivate cleanup v14: * Fix build error from __mutex_owner() with CONFIG_PREEMPT_RT v15: * Reworks for moving blocked_on_state to later in the series v16: * Pull down fix from later in the series where a deactivated task could pass the (task_cpu(owner) =3D=3D this_cpu) check then have it be activated on a different cpu, so it passes the on_rq check. Thus double check the values in the opposite order to make sure nothing slips by. * Add resched_idle label to simplify common exit path * Get rid of preserve_need_resched flag and rework per Peter's suggestion * Rework find_proxy_task() to use guard to cleanup the exit gotos as Peter suggested. v18: * Minor tweak to order of logic in find_proxy_task() to enable an optimization later in the full series suggested by Suleiman * Use READ_ONCE calls to on_rq references as I hit a bug where the compiler was optimizing out some of the reads. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Mel Gorman Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: Daniel Lezcano Cc: Suleiman Souhlal Cc: kuyo chang Cc: hupu Cc: kernel-team@android.com --- kernel/locking/mutex.h | 3 +- kernel/sched/core.c | 143 ++++++++++++++++++++++++++++++++++------- kernel/sched/fair.c | 11 +++- 3 files changed, 132 insertions(+), 25 deletions(-) diff --git a/kernel/locking/mutex.h b/kernel/locking/mutex.h index cbff35b9b7ae3..2e8080a9bee37 100644 --- a/kernel/locking/mutex.h +++ b/kernel/locking/mutex.h @@ -6,7 +6,7 @@ * * Copyright (C) 2004, 2005, 2006 Red Hat, Inc., Ingo Molnar */ - +#ifndef CONFIG_PREEMPT_RT /* * This is the control structure for tasks blocked on mutex, which resides * on the blocked task's kernel stack: @@ -70,3 +70,4 @@ extern void debug_mutex_init(struct mutex *lock, const ch= ar *name, # define debug_mutex_unlock(lock) do { } while (0) # define debug_mutex_init(lock, name, key) do { } while (0) #endif /* !CONFIG_DEBUG_MUTEXES */ +#endif /* CONFIG_PREEMPT_RT */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 98a802e0a3f36..1c7573d0cd312 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -96,6 +96,7 @@ #include "../workqueue_internal.h" #include "../../io_uring/io-wq.h" #include "../smpboot.h" +#include "../locking/mutex.h" =20 EXPORT_TRACEPOINT_SYMBOL_GPL(ipi_send_cpu); EXPORT_TRACEPOINT_SYMBOL_GPL(ipi_send_cpumask); @@ -2965,8 +2966,15 @@ static int affine_move_task(struct rq *rq, struct ta= sk_struct *p, struct rq_flag struct set_affinity_pending my_pending =3D { }, *pending =3D NULL; bool stop_pending, complete =3D false; =20 - /* Can the task run on the task's current CPU? If so, we're done */ - if (cpumask_test_cpu(task_cpu(p), &p->cpus_mask)) { + /* + * Can the task run on the task's current CPU? If so, we're done + * + * We are also done if the task is the current donor, boosting a lock- + * holding proxy, (and potentially has been migrated outside its + * current or previous affinity mask) + */ + if (cpumask_test_cpu(task_cpu(p), &p->cpus_mask) || + (task_current_donor(rq, p) && !task_current(rq, p))) { struct task_struct *push_task =3D NULL; =20 if ((flags & SCA_MIGRATE_ENABLE) && @@ -6695,36 +6703,124 @@ static struct task_struct *proxy_deactivate(struct= rq *rq, struct task_struct *d } =20 /* - * Initial simple sketch that just deactivates the blocked task - * chosen by pick_next_task() so we can then pick something that - * isn't blocked. + * Find runnable lock owner to proxy for mutex blocked donor + * + * Follow the blocked-on relation: + * task->blocked_on -> mutex->owner -> task... + * + * Lock order: + * + * p->pi_lock + * rq->lock + * mutex->wait_lock + * + * Returns the task that is going to be used as execution context (the one + * that is actually going to be run on cpu_of(rq)). */ static struct task_struct * find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags = *rf) { + struct task_struct *owner =3D NULL; + int this_cpu =3D cpu_of(rq); + struct task_struct *p; struct mutex *mutex; =20 - mutex =3D donor->blocked_on; - /* Something changed in the chain, so pick again */ - if (!mutex) - return NULL; - /* - * By taking mutex->wait_lock we hold off concurrent mutex_unlock() - * and ensure @owner sticks around. - */ - guard(raw_spinlock)(&mutex->wait_lock); + /* Follow blocked_on chain. */ + for (p =3D donor; task_is_blocked(p); p =3D owner) { + mutex =3D p->blocked_on; + /* Something changed in the chain, so pick again */ + if (!mutex) + return NULL; + /* + * By taking mutex->wait_lock we hold off concurrent mutex_unlock() + * and ensure @owner sticks around. + */ + guard(raw_spinlock)(&mutex->wait_lock); =20 - /* Check again that donor is blocked with blocked_lock held */ - if (!task_is_blocked(donor) || mutex !=3D __get_task_blocked_on(donor)) { + /* Check again that p is blocked with wait_lock held */ + if (mutex !=3D __get_task_blocked_on(p)) { + /* + * Something changed in the blocked_on chain and + * we don't know if only at this level. So, let's + * just bail out completely and let __schedule() + * figure things out (pick_again loop). + */ + return NULL; + } + + owner =3D __mutex_owner(mutex); + if (!owner) { + __clear_task_blocked_on(p, mutex); + return p; + } + + if (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) { + /* XXX Don't handle blocked owners/delayed dequeue yet */ + return proxy_deactivate(rq, donor); + } + + if (task_cpu(owner) !=3D this_cpu) { + /* XXX Don't handle migrations yet */ + return proxy_deactivate(rq, donor); + } + + if (task_on_rq_migrating(owner)) { + /* + * One of the chain of mutex owners is currently migrating to this + * CPU, but has not yet been enqueued because we are holding the + * rq lock. As a simple solution, just schedule rq->idle to give + * the migration a chance to complete. Much like the migrate_task + * case we should end up back in find_proxy_task(), this time + * hopefully with all relevant tasks already enqueued. + */ + return proxy_resched_idle(rq); + } + + /* + * Its possible to race where after we check owner->on_rq + * but before we check (owner_cpu !=3D this_cpu) that the + * task on another cpu was migrated back to this cpu. In + * that case it could slip by our checks. So double check + * we are still on this cpu and not migrating. If we get + * inconsistent results, try again. + */ + if (!task_on_rq_queued(owner) || task_cpu(owner) !=3D this_cpu) + return NULL; + + if (owner =3D=3D p) { + /* + * It's possible we interleave with mutex_unlock like: + * + * lock(&rq->lock); + * find_proxy_task() + * mutex_unlock() + * lock(&wait_lock); + * donor(owner) =3D current->blocked_donor; + * unlock(&wait_lock); + * + * wake_up_q(); + * ... + * ttwu_runnable() + * __task_rq_lock() + * lock(&wait_lock); + * owner =3D=3D p + * + * Which leaves us to finish the ttwu_runnable() and make it go. + * + * So schedule rq->idle so that ttwu_runnable() can get the rq + * lock and mark owner as running. + */ + return proxy_resched_idle(rq); + } /* - * Something changed in the blocked_on chain and - * we don't know if only at this level. So, let's - * just bail out completely and let __schedule() - * figure things out (pick_again loop). + * OK, now we're absolutely sure @owner is on this + * rq, therefore holding @rq->lock is sufficient to + * guarantee its existence, as per ttwu_remote(). */ - return NULL; /* do pick_next_task() again */ } - return proxy_deactivate(rq, donor); + + WARN_ON_ONCE(owner && !owner->on_rq); + return owner; } #else /* SCHED_PROXY_EXEC */ static struct task_struct * @@ -6876,10 +6972,13 @@ static void __sched notrace __schedule(int sched_mo= de) next =3D find_proxy_task(rq, next, &rf); if (!next) goto pick_again; + if (next =3D=3D rq->idle) + goto keep_resched; } picked: clear_tsk_need_resched(prev); clear_preempt_need_resched(); +keep_resched: rq->last_seen_need_resched_ns =3D 0; =20 is_switch =3D prev !=3D next; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c34e0891193a7..ea4331ed6bc23 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9415,7 +9415,8 @@ int can_migrate_task(struct task_struct *p, struct lb= _env *env) * 2) throttled_lb_pair, or * 3) cannot be migrated to this CPU due to cpus_ptr, or * 4) running (obviously), or - * 5) are cache-hot on their current CPU. + * 5) are cache-hot on their current CPU or + * 6) are blocked on mutexes (if SCHED_PROXY_EXEC is enabled) */ if ((p->se.sched_delayed) && (env->migration_type !=3D migrate_load)) return 0; @@ -9437,6 +9438,9 @@ int can_migrate_task(struct task_struct *p, struct lb= _env *env) if (kthread_is_per_cpu(p)) return 0; =20 + if (task_is_blocked(p)) + return 0; + if (!cpumask_test_cpu(env->dst_cpu, p->cpus_ptr)) { int cpu; =20 @@ -9472,7 +9476,8 @@ int can_migrate_task(struct task_struct *p, struct lb= _env *env) /* Record that we found at least one task that could run on dst_cpu */ env->flags &=3D ~LBF_ALL_PINNED; =20 - if (task_on_cpu(env->src_rq, p)) { + if (task_on_cpu(env->src_rq, p) || + task_current_donor(env->src_rq, p)) { schedstat_inc(p->stats.nr_failed_migrations_running); return 0; } @@ -9515,6 +9520,8 @@ static void detach_task(struct task_struct *p, struct= lb_env *env) schedstat_inc(env->sd->lb_hot_gained[env->idle]); schedstat_inc(p->stats.nr_forced_migrations); } + WARN_ON(task_current(env->src_rq, p)); + WARN_ON(task_current_donor(env->src_rq, p)); =20 deactivate_task(env->src_rq, p, DEQUEUE_NOCLOCK); set_task_cpu(p, env->dst_cpu); --=20 2.50.0.727.gbf7dc18ff4-goog