From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E0DA16D4EF for ; Thu, 11 Jul 2024 13:00:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702858; cv=none; b=FEHYWQ1JWpJ61kkhcMoYVaPZmsR3v+fJRdjnnSqd+IOgSZOpUvtHFDBpP2m9Ob+VtcfLV3Jcms2ViW9Tvby8IxmYQSef32czbTj2P4OQJMZyoy6UbcSxenlXGhb0I3YlnHN3V3Aeid63VpWGm/K2LmiEUHMRNYCoIWRiz7VtJ1Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702858; c=relaxed/simple; bh=03eOuIbpk2CL8gHcNkcMr6D2tKWHDvUL5fMCqSTDmZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IN5yNVVTfSUFpewKsk/tkF46m3biF8O6Ckc/2HYs/4AjE7Otu4Z7Riuu3df6Q04mtC6XaWkK1VM8O2ZmpD4p9pUYZZlm/kXZNvrpy4ge7ChEsVyBXN/okXqCeBrMtXgjYqlU3HA0woQCE9hXOkqXIeC8rGTBRbHj586Rk7OrzhY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VjRC3VUy; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VjRC3VUy" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mgv6m6VmpgYAIYyr1Uojk4UY9y2hZWycgx24o6Dhrv4=; b=VjRC3VUytllNnOXTV2DqE4H0hHEQ2UTZPKPDE/K0mn/pVu7mtSBtDBe9mHOkQpPgdgGUkg Y3x/YhHcvlkRGW+lL9eQo432bJk3LrX7RmQyYn/FOQi6AYd98dRlnjnMmwdFQMariV2rdQ 48CXGs6n1R3dWdtyoE5kP8ZrDmmUtVM= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-316-I0jpcKX5O0eRlw_1D8M9ZQ-1; Thu, 11 Jul 2024 09:00:49 -0400 X-MC-Unique: I0jpcKX5O0eRlw_1D8M9ZQ-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 15AAD19772C9; Thu, 11 Jul 2024 13:00:45 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0DD7319560AA; Thu, 11 Jul 2024 13:00:36 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 01/10] rcuwait: Split type definition to its own header Date: Thu, 11 Jul 2024 14:59:55 +0200 Message-ID: <20240711130004.2157737-2-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" A later commit will add a struct irq_work member to struct task_struct, whi= ch would require to include the definition of struct irq_work. Thanks to header dependency hell, incudling in results in defining inline helpers using not-yet-defined fields (mm_struct, task_struct, various task states...). Break off the definition of struct rcuwait into its own header file. Signed-off-by: Valentin Schneider --- include/linux/rcuwait.h | 9 ++------- include/linux/rcuwait_types.h | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 7 deletions(-) create mode 100644 include/linux/rcuwait_types.h diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h index 27343424225cf..1f1ca7d38cdf8 100644 --- a/include/linux/rcuwait.h +++ b/include/linux/rcuwait.h @@ -5,17 +5,12 @@ #include #include =20 +#include + /* * rcuwait provides a way of blocking and waking up a single * task in an rcu-safe manner. - * - * The only time @task is non-nil is when a user is blocked (or - * checking if it needs to) on a condition, and reset as soon as we - * know that the condition has succeeded and are awoken. */ -struct rcuwait { - struct task_struct __rcu *task; -}; =20 #define __RCUWAIT_INITIALIZER(name) \ { .task =3D NULL, } diff --git a/include/linux/rcuwait_types.h b/include/linux/rcuwait_types.h new file mode 100644 index 0000000000000..60a4385a2c368 --- /dev/null +++ b/include/linux/rcuwait_types.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_RCUWAIT_TYPES_H_ +#define _LINUX_RCUWAIT_TYPES_H_ + +#include + +/* + * The only time @task is non-nil is when a user is blocked (or + * checking if it needs to) on a condition, and reset as soon as we + * know that the condition has succeeded and are awoken. + */ +struct rcuwait { + struct task_struct __rcu *task; +}; + +#endif --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2241E15F402 for ; Thu, 11 Jul 2024 13:01:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702863; cv=none; b=tWEYHJfX1T3yyvqR7qNYn45Fi3tpWlVKmjXaveYyNz63BIX8dxAhrEOgyxf75yBzl2DsQ7ogtowlXqCvayoWGAv/hHTdWT0iTA90AsS4mYsHvqDSP0c4QWikpu95X98BR5AKMePK/TF4z9rc7Q7U57JYo3Yfwj0HzR/hGHWSZR8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702863; c=relaxed/simple; bh=j48GshrK4SLyYIYU/4TmYMncSjreC5NozFzjgMwZI9g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kkzMaNDkjkOmGw1aJQJS5vHIbBauxYYMFvq9mT6nWVJ2IB2l1ou23xBEUNcN7lp9Aj7N0t80lf/HIHW17HBdJyUq0RDaZCQxb2qIcBRCAMUd3s8s1todLBce4iCwFCUydMo4S1+TAP2cy6qTVl5FnP5jNPA3R7prDWEPMblhbEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=RvljnccQ; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RvljnccQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702861; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+8MT3JoxFoj9B2SXJDisOC/gnRr4MwegwcdGr6Z7BqM=; b=RvljnccQIMAES5QpoLsqLlQ7uc/4MVfQr9C8uNRnG6Ano8OToeF/Jt/DRGlU/CN6FIifwp LI2UsEnpkUs5W/I/mdmVq9SSbZ5+kdBrd3+sdZmhXn5FIfEG3U45BLDno5sibxZPZ2xlXK M+f0IY453s7COBUf6iBwqR1S9UAXjQs= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-490-E1QTop2-MIW3fGQv61nn-A-1; Thu, 11 Jul 2024 09:00:58 -0400 X-MC-Unique: E1QTop2-MIW3fGQv61nn-A-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D42011955EAA; Thu, 11 Jul 2024 13:00:53 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 736A91955E85; Thu, 11 Jul 2024 13:00:45 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 02/10] irq_work: Split type definition to its own header Date: Thu, 11 Jul 2024 14:59:56 +0200 Message-ID: <20240711130004.2157737-3-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" A later commit will add a struct irq_work member to struct task_struct, whi= ch would require to include the definition of struct irq_work. Thanks to header dependency hell, incudling in results in defining inline helpers using not-yet-defined fields (mm_struct, task_struct, various task states...). Break off the definition of struct irq_work into its own header file. Signed-off-by: Valentin Schneider --- include/linux/irq_work.h | 8 ++------ include/linux/irq_work_types.h | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 6 deletions(-) create mode 100644 include/linux/irq_work_types.h diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h index 136f2980cba30..7f6d2af360d91 100644 --- a/include/linux/irq_work.h +++ b/include/linux/irq_work.h @@ -5,6 +5,8 @@ #include #include =20 +#include + /* * An entry can be in one of four states: * @@ -14,12 +16,6 @@ * busy NULL, 2 -> {free, claimed} : callback in progress, can be cla= imed */ =20 -struct irq_work { - struct __call_single_node node; - void (*func)(struct irq_work *); - struct rcuwait irqwait; -}; - #define __IRQ_WORK_INIT(_func, _flags) (struct irq_work){ \ .node =3D { .u_flags =3D (_flags), }, \ .func =3D (_func), \ diff --git a/include/linux/irq_work_types.h b/include/linux/irq_work_types.h new file mode 100644 index 0000000000000..108cbc514733b --- /dev/null +++ b/include/linux/irq_work_types.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_IRQ_WORK_TYPES_H +#define _LINUX_IRQ_WORK_TYPES_H + +#include +#include + +struct irq_work { + struct __call_single_node node; + void (*func)(struct irq_work *); + struct rcuwait irqwait; +}; + +#endif --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDE9115FD1B for ; Thu, 11 Jul 2024 13:01:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702873; cv=none; b=Q0dCcDvH1YeBR2ATaizWL3fFsxOxwXh75MwqVnTbDmlpliVstT+DQ/ev304FKYTzOTO+a64agJbtsA3F8qKSswJetSJJIc7x0Vv0feEgcx7ZPmL+GpNmP4PLWN3uP3Q+UpXkVUIR1NSehwHeonkbWu4XI50fq8wig7iasse6Xfg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702873; c=relaxed/simple; bh=tk3/eU+VtYT+/UhY7nwOfNv0uD4Ewrnc70U2kX7qZDc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=stqZRjsPEobSCQ5+gm3zpd1VPyaTYSlTrlAcfhdOlZvRqIM3SVb4etiLxKOXQDdBwM2sNn4rEZzczloltFBR9fme+NJw3f/cOdfeCVaKpyYfFkKMRd6aKGRxQGX5BFJxtccxncYBrjSu8D+H6aFbjAhsHF20D5sOL2G/BcQJdF0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ik3KuZel; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ik3KuZel" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702870; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Oe3ZcX+q5foJNAvfJR3Gr4N1bW3UV6I6D4EkwWMO3O0=; b=ik3KuZel+/MNbCrbxQmgD9Sa+ciKJuVuC8Hx9xFy2GqlFD2eF95ouLpQy5AUlI1jhyaorT azE01MS2TUGp80g6WUmmm2TntZQOM3GBOJWwlX2rHVmfB/ERfF7OF7naDtOATN99AIbiLC L/z0x75P1mHg0XnDBMX3yq5ktFYN1wE= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-324-fdulkfqJNUmCkKIxHRvfTg-1; Thu, 11 Jul 2024 09:01:07 -0400 X-MC-Unique: fdulkfqJNUmCkKIxHRvfTg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 63A4F19560B6; Thu, 11 Jul 2024 13:01:01 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 391341955E8E; Thu, 11 Jul 2024 13:00:54 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 03/10] task_work, sched: Add a _locked variant to task_work_cancel() Date: Thu, 11 Jul 2024 14:59:57 +0200 Message-ID: <20240711130004.2157737-4-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Later commits will need to issue a task_work_cancel() from within the scheduler with the task's ->pi_lock held. Add a _locked variant that expects p->pi_lock to be held. Expose it in a separate scheduler header file, as this really is a scheduler-only interface. Signed-off-by: Valentin Schneider Reviewed-by: Oleg Nesterov --- kernel/sched/task_work_sched.h | 14 +++++++ kernel/task_work.c | 67 ++++++++++++++++++++++++++-------- 2 files changed, 66 insertions(+), 15 deletions(-) create mode 100644 kernel/sched/task_work_sched.h diff --git a/kernel/sched/task_work_sched.h b/kernel/sched/task_work_sched.h new file mode 100644 index 0000000000000..e235da456427f --- /dev/null +++ b/kernel/sched/task_work_sched.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Scheduler internal task_work methods + */ +#ifndef _KERNEL_TASK_WORK_SCHED_H +#define _KERNEL_TASK_WORK_SCHED_H + +#include +#include + +struct callback_head * +task_work_cancel_locked(struct task_struct *task, task_work_func_t func); + +#endif diff --git a/kernel/task_work.c b/kernel/task_work.c index 95a7e1b7f1dab..81092bc2e7371 100644 --- a/kernel/task_work.c +++ b/kernel/task_work.c @@ -3,6 +3,8 @@ #include #include =20 +#include "sched/task_work_sched.h" + static struct callback_head work_exited; /* all we need is ->next =3D=3D N= ULL */ =20 /** @@ -74,33 +76,20 @@ int task_work_add(struct task_struct *task, struct call= back_head *work, return 0; } =20 -/** - * task_work_cancel_match - cancel a pending work added by task_work_add() - * @task: the task which should execute the work - * @match: match function to call - * @data: data to be passed in to match function - * - * RETURNS: - * The found work or NULL if not found. - */ -struct callback_head * -task_work_cancel_match(struct task_struct *task, +static struct callback_head * +task_work_cancel_match_locked(struct task_struct *task, bool (*match)(struct callback_head *, void *data), void *data) { struct callback_head **pprev =3D &task->task_works; struct callback_head *work; - unsigned long flags; =20 - if (likely(!task_work_pending(task))) - return NULL; /* * If cmpxchg() fails we continue without updating pprev. * Either we raced with task_work_add() which added the * new entry before this work, we will find it again. Or * we raced with task_work_run(), *pprev =3D=3D NULL/exited. */ - raw_spin_lock_irqsave(&task->pi_lock, flags); work =3D READ_ONCE(*pprev); while (work) { if (!match(work, data)) { @@ -109,6 +98,32 @@ task_work_cancel_match(struct task_struct *task, } else if (try_cmpxchg(pprev, &work, work->next)) break; } + + return work; +} + +/** + * task_work_cancel_match - cancel a pending work added by task_work_add() + * @task: the task which should execute the work + * @match: match function to call + * @data: data to be passed in to match function + * + * RETURNS: + * The found work or NULL if not found. + */ +struct callback_head * +task_work_cancel_match(struct task_struct *task, + bool (*match)(struct callback_head *, void *data), + void *data) +{ + unsigned long flags; + struct callback_head *work; + + if (likely(!task_work_pending(task))) + return NULL; + + raw_spin_lock_irqsave(&task->pi_lock, flags); + work =3D task_work_cancel_match_locked(task, match, data); raw_spin_unlock_irqrestore(&task->pi_lock, flags); =20 return work; @@ -136,6 +151,28 @@ task_work_cancel(struct task_struct *task, task_work_f= unc_t func) return task_work_cancel_match(task, task_work_func_match, func); } =20 +/** + * task_work_cancel - cancel a pending work added by task_work_add() + * @task: the task which should execute the work + * @func: identifies the work to remove + * + * Find the last queued pending work with ->func =3D=3D @func and remove + * it from queue. + * + * RETURNS: + * The found work or NULL if not found. + */ +struct callback_head * +task_work_cancel_locked(struct task_struct *task, task_work_func_t func) +{ + lockdep_assert_held(&task->pi_lock); + + if (likely(!task_work_pending(task))) + return NULL; + + return task_work_cancel_match_locked(task, task_work_func_match, func); +} + /** * task_work_run - execute the works added by task_work_add() * --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A695F160887 for ; Thu, 11 Jul 2024 13:01:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702886; cv=none; b=a8YMfaGbJYc4Ud3G7uj8zVWof12YxO+o/OmPpFS3/3NeVsVDJrhJMMT2qponxfnewCTG7fiLvJ/YE+fl6vKTtv5Cyd5QyLdLfr/UImFau+VB9gxbl9G4OSNlxLSMXXL08ZpleDqeeM+lXzNbCCYtM27Q9e5gqcx91kWM9yICt4c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702886; c=relaxed/simple; bh=ac39iJ85qdc9Ytj5FEcWzWjFfQVSQdQQK96665YcdAg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pwNshagZ21fptC5InVfcvRTRiM7qj4Lsh0yeNAW33qDDI9ikK6vmAmjT45R0iqkEUBYu2XgSaNjVDOhXJlfma9sngbGzWxqEC5V9bdZqHyD8kVx5h0LJXuLLuuFwBgPDLRJzyDMjlQAy3hfrKO0c4vk7uwmQ4rN1v/BiS3/EU/0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=K0n0eQ0O; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="K0n0eQ0O" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702883; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bY5N4wVYoZd5/9PoLPrXsqV7N1C2PIbxzBinwZyAijs=; b=K0n0eQ0Oxjl+mCDGj4ePcSGBu1Qk8Ru+vyXG/IXzOCfJjpgYMRwyYzOeJf0kQzjiOP+ida U2eh4oxABTjUWd0CfKVsZvl3qgJUGazDx7E6LXvdsO38OBaA6buReXEu8wI3iGsxeOLe5j flBfQfOTtYJLAeCzwx2rReGEutI/wyk= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-284-uymGDPAkPJiTvRpsqk9RfA-1; Thu, 11 Jul 2024 09:01:17 -0400 X-MC-Unique: uymGDPAkPJiTvRpsqk9RfA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8E3E41977009; Thu, 11 Jul 2024 13:01:10 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C08621955E85; Thu, 11 Jul 2024 13:01:01 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 04/10] sched/fair: Introduce sched_throttle_work Date: Thu, 11 Jul 2024 14:59:58 +0200 Message-ID: <20240711130004.2157737-5-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. Actual throttling of a task will happen in the return to user path, which will be implemented via a task_work callback. To ease reviewing, the infrastructure and helpers are added first, the actual behaviour will be implemented when switching to per-task throttling. Add a task_work node to struct task_struct, and have it initialised at sched_fork(). Signed-off-by: Valentin Schneider --- include/linux/sched.h | 1 + kernel/sched/core.c | 4 ++++ kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/sched.h | 2 ++ 4 files changed, 19 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 90691d99027e3..a4976eb5065fc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -812,6 +812,7 @@ struct task_struct { =20 #ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group; + struct callback_head sched_throttle_work; #endif =20 =20 diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 6d35c48239be0..b811670d2c362 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4329,6 +4329,10 @@ static void __sched_fork(unsigned long clone_flags, = struct task_struct *p) p->se.cfs_rq =3D NULL; #endif =20 +#ifdef CONFIG_CFS_BANDWIDTH + init_cfs_throttle_work(p); +#endif + #ifdef CONFIG_SCHEDSTATS /* Even if schedstat is disabled, there should not be garbage */ memset(&p->stats, 0, sizeof(p->stats)); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9057584ec06de..775547cdd3ce0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5759,6 +5759,18 @@ static int tg_throttle_down(struct task_group *tg, v= oid *data) return 0; } =20 +static void throttle_cfs_rq_work(struct callback_head *work) +{ + +} + +void init_cfs_throttle_work(struct task_struct *p) +{ + /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_= work() */ + p->sched_throttle_work.next =3D &p->sched_throttle_work; + init_task_work(&p->sched_throttle_work, throttle_cfs_rq_work); +} + static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { struct rq *rq =3D rq_of(cfs_rq); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 4c36cc6803617..943bca8263ffe 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2510,6 +2510,8 @@ extern void init_sched_dl_class(void); extern void init_sched_rt_class(void); extern void init_sched_fair_class(void); =20 +extern void init_cfs_throttle_work(struct task_struct *p); + extern void reweight_task(struct task_struct *p, const struct load_weight = *lw); =20 extern void resched_curr(struct rq *rq); --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7ACB71662E2 for ; Thu, 11 Jul 2024 13:01:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702895; cv=none; b=Fi/z+dY5cw7C2wXmTA+7/yYNkylaizEFlfbTFYEQLFA+NEcveRpJT1jSga4BxoWL+wqLoqxGz9MEFX5ddV0VuO5GqK3H/msuXf/d8FV2Q4Ml1p/oG2shDQfiSHjxQegt2YuPSU1PNjishtv6uAhdPUWsHwVi+7nB53KXBY7BLd0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702895; c=relaxed/simple; bh=8ueQdFLG87p+qfy4yt6xngF+VdvTIn2SFkuPJ3FVakA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vg6xrPCBxbDHoTSn0nok4qKdmeIOpngiaJUPoYHl+/etz/F+v/JPndn9twjhp51spuktLNg4nAEp+xSh5OQRXkvDUMNlwHHEOEn/V3GxisM9EY//SxP+c6IbFm5TrMo4MDYlko/KpH/OVwQx0XPw03LxmY5PIrIoEGQD+il/aaI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=YnS/AEfI; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YnS/AEfI" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702892; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cYNML39erjHIQYIq2MzxI1f2C9g8cKZz+w3h5LIB8k4=; b=YnS/AEfID74W0dSUJ4M4g2Pr7GTdi3RBUQQBV1npdH4oRG6jLAXwIqdjls5FJI9OhqdRU5 QtWSWsqKUs2ZQ4a70b5+bYTRllUfcu6h1Gws1kuKpgS+poTvT78KDZz1UjhOLCntzP/jXW 2BwW5p0catHSlUdljzzIA/kmw0nCHNI= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-692-jRWigaaEPvK4-17APSOBkg-1; Thu, 11 Jul 2024 09:01:29 -0400 X-MC-Unique: jRWigaaEPvK4-17APSOBkg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 403CE1954B0C; Thu, 11 Jul 2024 13:01:18 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EACDC1955E9F; Thu, 11 Jul 2024 13:01:10 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 05/10] sched/fair: Introduce an irq_work for cancelling throttle task_work Date: Thu, 11 Jul 2024 14:59:59 +0200 Message-ID: <20240711130004.2157737-6-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Moving towards per-task throttling, the soon-to-be task_work used for the actual throttling will need to be cancelled when a task is moving out of a throttled cfs_rq and into a non-throttled cfs_rq (or out of CFS altogether). Such code paths will have at least the rq lock held, sometimes both the rq and the p->pi_lock locks held. Functions such as migrate_task_rq_fair() don't have guarantees as to which of the two is held, as such the cancellation will need to happen in a separate context. It will be punted to irq_work context, the groundwork is added here and the irq_work callback will be implemented when switching to per-task throttling. Signed-off-by: Valentin Schneider --- include/linux/sched.h | 4 ++++ kernel/sched/fair.c | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index a4976eb5065fc..99a1e77d769db 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -46,6 +46,7 @@ #include #include #include +#include =20 /* task_struct member predeclarations (sorted alphabetically): */ struct audit_context; @@ -813,6 +814,9 @@ struct task_struct { #ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group; struct callback_head sched_throttle_work; +#ifdef CONFIG_CFS_BANDWIDTH + struct irq_work unthrottle_irq_work; +#endif #endif =20 =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 775547cdd3ce0..095357bd17f0e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5764,11 +5764,17 @@ static void throttle_cfs_rq_work(struct callback_he= ad *work) =20 } =20 +static void task_throttle_cancel_irq_work_fn(struct irq_work *work) +{ + /* Write me */ +} + void init_cfs_throttle_work(struct task_struct *p) { /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_= work() */ p->sched_throttle_work.next =3D &p->sched_throttle_work; init_task_work(&p->sched_throttle_work, throttle_cfs_rq_work); + p->unthrottle_irq_work =3D IRQ_WORK_INIT_HARD(task_throttle_cancel_irq_wo= rk_fn); } =20 static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E4F116DC1D for ; Thu, 11 Jul 2024 13:01:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702894; cv=none; b=qWR56AlRUdmxieIZzWj9eLthyUOQDBC0FDmk+sIAJ5ZUpFHTwD9jy/RvFfKjOv6kVVIAH5zYyvxvQdJjpaBuAwnN6g1/T0p9rbraDYAlbxc6fGd44zNyem2cTin+plL7MGsMIs+dfBW/aIQ5ce4+d3q9EKV9WchiiIX6crVPrIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702894; c=relaxed/simple; bh=j+9voxVbkXLZex9aF3BIDI+OpqtDuigja9qnrW3CW7w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pEZLBIlSiET6+3HegsuJ/rCtBhZsRU96gryY5S9cZw0x8joGdp67kevejZ0aNOo0gfgCMwlX8uwwpsdTjwYJ8Dd5xlD8NeuHZRZeAOEpwVMlS4ooauQde/7G1xy3TrOkSbWvLl4XoubgbhxPC78ptkv6OJRO9wwkX1NOXsb6QAs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=DDQfY8Nq; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="DDQfY8Nq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702891; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=39lyjke0SXVRF7E8CxU/t0CTJvnKYmxII1z60vz/0qI=; b=DDQfY8NquneqW1+NZShZr0kGENS5fR6RWCJ4ZJQxkeWdCYvGwzx1skJ+Oq6ezqKsh+ZIk9 HzFh33xclJUHbhct1Lfvl8vvN/YkyK9MuLso0mQoiiA8FuO0ZHBDx1SJBKEMuaX9dGvSwh p1mFspASlvhLTdX5UxNq88OSzXU6B4Y= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-146-4VTRME0cO9a6HJIqgaBDrA-1; Thu, 11 Jul 2024 09:01:29 -0400 X-MC-Unique: 4VTRME0cO9a6HJIqgaBDrA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CF9721953966; Thu, 11 Jul 2024 13:01:25 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8A1F2195606C; Thu, 11 Jul 2024 13:01:18 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 06/10] sched/fair: Prepare switched_from & switched_to for per-task throttling Date: Thu, 11 Jul 2024 15:00:00 +0200 Message-ID: <20240711130004.2157737-7-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add helpers to switched_from_fair() and switched_to_fair() to cover class changes. If switching from CFS, a task may need to be unthrottled. If switching to CFS, a task may need to be throttled. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 095357bd17f0e..acac0829c71f3 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5694,6 +5694,10 @@ static inline int throttled_hierarchy(struct cfs_rq = *cfs_rq) return cfs_bandwidth_used() && cfs_rq->throttle_count; } =20 +static inline bool task_needs_throttling(struct task_struct *p) { return f= alse; } +static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel(struct task_struct *p) { } + /* * Ensure that neither of the group entities corresponding to src_cpu or * dest_cpu are members of a throttled hierarchy when performing group @@ -6622,6 +6626,10 @@ static inline int throttled_lb_pair(struct task_grou= p *tg, return 0; } =20 +static inline bool task_needs_throttling(struct task_struct *p) { return f= alse; } +static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel(struct task_struct *p) { } + #ifdef CONFIG_FAIR_GROUP_SCHED void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth = *parent) {} static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq) {} @@ -12847,11 +12855,15 @@ static void attach_task_cfs_rq(struct task_struct= *p) static void switched_from_fair(struct rq *rq, struct task_struct *p) { detach_task_cfs_rq(p); + if (cfs_bandwidth_used()) + task_throttle_cancel(p); } =20 static void switched_to_fair(struct rq *rq, struct task_struct *p) { attach_task_cfs_rq(p); + if (cfs_bandwidth_used() && task_needs_throttling(p)) + task_throttle_setup(p); =20 set_task_max_allowed_capacity(p); =20 --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7DF715AAD3 for ; Thu, 11 Jul 2024 13:01:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702905; cv=none; b=Zq7qDuo6V39wr4FCft3gckzYW75milXTGo7N3DkoCH5x8S5j9Q7uGkS329GmVdfEbLYawlCuypwgEKn+8XkSP9yvqikw+2WJiJVzdfWFZKXJzs9opejO4RPOCQk77nfklQ+nTMe8b5ewUftWOPGaz56Ugt9xDacfpPJEZWnVZIQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702905; c=relaxed/simple; bh=3hQTshq3j/pEAuKohadW5LUjN4RXbzUfNWbS/gaH5rc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pHXqYvhRg1dkg0jjCvgSqspEGAnEAg9xV9KPyAsAfDPLAgzLrxNr8vu8k4BuECUG0vFXZUSKyQzkEKAXN0wySO2epWQOv5gC65W10Y54GLhCfecmJ01+l7HTjeL4SPF62e//Va+MCzBesgg6NtMCSxSFI3WI1ml/kMfjIu4omjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fqcmA5f6; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fqcmA5f6" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702903; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lQ8G2jooOuhKMZEVaAQ+rNv5KFn8ERDELUo5S05OR5Q=; b=fqcmA5f6RvvBvkDtT/JJqxGrD+s9ry7EccqXcT9QadHP7jcua9x1jZbP/4JyHPqqhXjusF t408dtKl/yez85AkilfxMVOkbAV/kS5kIsdNz4/8HZ3rkCdRwCR27To9Tr9qJmrtBlJc+e HOQUWFd+c7nTs1f7rdLqfnKuFIKINws= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-22-6BOpIVidMVuPnBl-0JpA5w-1; Thu, 11 Jul 2024 09:01:40 -0400 X-MC-Unique: 6BOpIVidMVuPnBl-0JpA5w-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5E81F1944B3E; Thu, 11 Jul 2024 13:01:33 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 23CF61956066; Thu, 11 Jul 2024 13:01:25 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 07/10] sched/fair: Prepare task_change_group_fair() for per-task throttling Date: Thu, 11 Jul 2024 15:00:01 +0200 Message-ID: <20240711130004.2157737-8-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add helpers to task_change_group_fair() to cover CPU cgroup migration. If changing to a throttled taskgroup/cfs_rq, the task needs to be throttled. Conversely, if the task is already throttled but changing to a taskgroup/cfs_rq that still has some runtime, the task must be unthrottle= d. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index acac0829c71f3..ec4cf7308a586 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12935,6 +12935,14 @@ static void task_change_group_fair(struct task_str= uct *p) #endif set_task_rq(p, task_cpu(p)); attach_task_cfs_rq(p); + + if (!cfs_bandwidth_used()) + return; + + if (task_needs_throttling(p)) + task_throttle_setup(p); + else + task_throttle_cancel(p); } =20 void free_fair_sched_group(struct task_group *tg) --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B401C16131C for ; Thu, 11 Jul 2024 13:01:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702913; cv=none; b=riuJ64m+JnDAp25Db+6i4XUyvrjMg71MAx8nOX4UuPhYkxbs5KDk6zCYFyaI+6pXxrum+NkBfIoXmf3kftzz1nVaK/I75LfrR3LH8JJKYDmL5uFOOccsNSEabA95FOmZ1fXt/BQNmUOIj2QtGqakity7lplGd52V+WF48oOVl1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702913; c=relaxed/simple; bh=kEbqFbMAMRFPsL2Ni9U3GMlxwu2JbJXIf/1GDFAi7j8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Knk0X1SyjLZ4l9B+p6EuJOA+EhJMoLJ9UfsMJWCArMioySoWpvddkJrNaqluK1ommH7wv6Y9Y9HO/8ym4b16VgtCyWLPnBZfOYVe0CFBjC16vRJSn3rHy+uObB/IgnGXmFiADFbg1UOewAQDgFkK+JU0CczLC6tx1q8Tx1390lE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=g4Tforgx; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="g4Tforgx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702910; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bo6AeZ1jwJY+2L4Z28XS0rJC0wllWNmRErvAYHJbVJg=; b=g4TforgxcNNMG6GREnuqwlaZiGqQ37DLsaXtD8/dFAz5+2MuxhI8i6uscDRYWVl75tZS2F 1+7BvfB8kHc/CcetNcOi8OVOj8CtVTwjCh9FzasTip8l2E0IT9ddM5rn2OeVnk9WxeKJkf WF/j4Wnq1tzP20KsroJBn53nMxgM+ZQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-648-AnhaU64YN2SPdUPPEE27_Q-1; Thu, 11 Jul 2024 09:01:46 -0400 X-MC-Unique: AnhaU64YN2SPdUPPEE27_Q-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2102919236B1; Thu, 11 Jul 2024 13:01:41 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A1D4119560AA; Thu, 11 Jul 2024 13:01:33 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 08/10] sched/fair: Prepare migrate_task_rq_fair() for per-task throttling Date: Thu, 11 Jul 2024 15:00:02 +0200 Message-ID: <20240711130004.2157737-9-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add helpers to migrate_task_rq_fair() to cover CPU migration. Even if the task stays within the same taskgroup, each cfs_rq has its own runtime accounting, thus the task needs to be throttled or unthrottled accordingly. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ec4cf7308a586..b2242307677ca 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5694,8 +5694,11 @@ static inline int throttled_hierarchy(struct cfs_rq = *cfs_rq) return cfs_bandwidth_used() && cfs_rq->throttle_count; } =20 +static inline bool task_has_throttle_work(struct task_struct *p) { return = false; } static inline bool task_needs_throttling(struct task_struct *p) { return f= alse; } +static inline bool task_needs_migrate_throttling(struct task_struct *p, un= signed int dst_cpu) { return false; } static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel_migrate(struct task_struct *p, int= dst_cpu) { } static inline void task_throttle_cancel(struct task_struct *p) { } =20 /* @@ -6626,8 +6629,11 @@ static inline int throttled_lb_pair(struct task_grou= p *tg, return 0; } =20 +static inline bool task_has_throttle_work(struct task_struct *p) { return = false; } static inline bool task_needs_throttling(struct task_struct *p) { return f= alse; } +static inline bool task_needs_migrate_throttling(struct task_struct *p, un= signed int dst_cpu) { return false; } static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel_migrate(struct task_struct *p, int= dst_cpu) { } static inline void task_throttle_cancel(struct task_struct *p) { } =20 #ifdef CONFIG_FAIR_GROUP_SCHED @@ -8308,6 +8314,24 @@ static void migrate_task_rq_fair(struct task_struct = *p, int new_cpu) se->avg.last_update_time =3D 0; =20 update_scan_period(p, new_cpu); + + if (!cfs_bandwidth_used()) + return; + /* + * When the runtime within a cfs_bandwidth is depleted, all underlying + * cfs_rq's can have (approximately) sched_cfs_bandwidth_slice() runtime + * remaining. + * + * This means all tg->cfs_rq[]'s do not get throttled at the exact same + * time: some may still have a bit of runtime left. Thus, even if the + * task is staying within the same cgroup, and under the same + * cfs_bandwidth, the cfs_rq it migrates to might have a different + * throttle status - resync is needed. + */ + if (task_needs_migrate_throttling(p, new_cpu)) + task_throttle_setup(p); + else if (task_has_throttle_work(p)) + task_throttle_cancel_migrate(p, new_cpu); } =20 static void task_dead_fair(struct task_struct *p) --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2505169AE3 for ; Thu, 11 Jul 2024 13:02:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702937; cv=none; b=DwTKA9tOVg3bFnG5D2NOWhLvP3qCdbBUIZ7vUQijWlwCwu40NoPpgoK6X8kkC7bblkvMIScCaEgGMWV5Zc+xAZKsJO+Umgzr8zoAHX7jTuXs6lBTTJ87AdzGGDuBJ1HTqwSnt0JqNw1+9g0dJrdkEsGyzQffiG2S2oGy0mN7Ry8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702937; c=relaxed/simple; bh=XQg0xC95fqr2rj2jwofKATx4tzAfNoZL3L3ocZIK2Yc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qCQxPtylYyHabirGcubWnNmZRWudvP6b5yO+OjDGvuX7PvrX/rRp+zAXF8h/a8aHv4SXlGa35+nagedn9XCBTV7Jv/dkBalXIviDXi2r7Kitr2AgL5AoQvS3RNLkH9Tv9tkuRrykFxjMDqy9KVSCkHdH955pgV+Xy/YejQ51Zr0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=B23sVS/K; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="B23sVS/K" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702934; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BecpANArIMV8DYuzp7MuPwP+wsa2agRYjWQ5hatmwiA=; b=B23sVS/KpoBngoElkHF8izZfK6Xdfvhia/B8i7FNs9SJdgQ/6XGz4Rrumg9+kIcfQTO4L4 XXQKG9BD9eiuDEa9s742OXUgKRIH9HohGaI0LTqscWTOjgPRhwqnwjTn2esu8zj4TRtlzU iGm0RUkTNyKGQKUnCxHXisWiT06haR4= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-573-j3VMWE-ePrmUOpn0P2CXbA-1; Thu, 11 Jul 2024 09:02:09 -0400 X-MC-Unique: j3VMWE-ePrmUOpn0P2CXbA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 43DA11977006; Thu, 11 Jul 2024 13:02:04 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4A6D51955E9F; Thu, 11 Jul 2024 13:01:41 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 09/10] sched/fair: Add a class->task_woken callback in preparation for per-task throttling Date: Thu, 11 Jul 2024 15:00:03 +0200 Message-ID: <20240711130004.2157737-10-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add a class->task_woken callback to handle tasks being woken into potentially throttled cfs_rq's. Conversely, a task flagged for throttle-at-kernel-exit may block and need to have its pending throttle removed if runtime was replenished by the time it got woken up. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b2242307677ca..0cec3e70f1277 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5776,6 +5776,17 @@ static void task_throttle_cancel_irq_work_fn(struct = irq_work *work) /* Write me */ } =20 +static void task_woken_fair(struct rq *rq, struct task_struct *p) +{ + if (!cfs_bandwidth_used()) + return; + + if (task_needs_throttling(p)) + task_throttle_setup(p); + else + task_throttle_cancel(p); +} + void init_cfs_throttle_work(struct task_struct *p) { /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_= work() */ @@ -13288,6 +13299,10 @@ DEFINE_SCHED_CLASS(fair) =3D { .task_change_group =3D task_change_group_fair, #endif =20 +#ifdef CONFIG_CFS_BANDWIDTH + .task_woken =3D task_woken_fair, +#endif + #ifdef CONFIG_SCHED_CORE .task_is_throttled =3D task_is_throttled_fair, #endif --=20 2.43.0 From nobody Fri Dec 19 04:06:48 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50A9816B739 for ; Thu, 11 Jul 2024 13:02:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702946; cv=none; b=capW55/jgeDIX3CunR3obdPQRashDmror4QoZS68wWQK0nA5M1DJdPFD+V+m53hD1Zl52PE5jTLYDaDo+CSEZDNkx+rCM17kEdJs/rm/nGXGgM0U4njIKBGyXlGjFAzld6i6enfmX59jJrV50OnM5GaDHAptwHXKypfFnb4po74= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702946; c=relaxed/simple; bh=ck9m3Um5OTNGL+iPZ3Ph2VLm2iJmwF9OHXTWyjvQvrQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I8vZbENjOMtIIEClo/TMldvY9ZI/4WQlcT2d0ktj7rLLtxtmKeEJvuxy2nLwldUblogYSOCMRE5O7GpsyDM72fYHXWkNRanIDklrJNwV5Oc87HnFbBiLe1DU3QtB2o5DRnBJCrNnFAMmk02oyxjVfog7LYYJM5n0F3qsV7CxBTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=P2/gUuiP; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="P2/gUuiP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702943; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FzQ5KjMRLj4g81cfSh32RRFGiouIEV/aENXMGz09G0s=; b=P2/gUuiPxVSiIasNtpmBWz39NP0LGc/2MTrYzTdyKav4mhbVMD6Le2O7s3CmLblrWWXTGB /YtS3gstLIAIOpc9mN5tNQa93NoMUSAtIsfXkGnTMn5aNeINAiS9FYH4+56kFWsXpQLMSJ b4HEpuj76Fx4BgCfKlyWWBthZzvB2Bs= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-600-w0Nh7MaYNbCd0g_UNUP4XA-1; Thu, 11 Jul 2024 09:02:21 -0400 X-MC-Unique: w0Nh7MaYNbCd0g_UNUP4XA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EE4111955EA9; Thu, 11 Jul 2024 13:02:11 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7194F1955EB3; Thu, 11 Jul 2024 13:02:04 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 10/10] sched/fair: Throttle CFS tasks on return to userspace Date: Thu, 11 Jul 2024 15:00:04 +0200 Message-ID: <20240711130004.2157737-11-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Content-Type: text/plain; charset="utf-8" As reported in [1], CFS bandwidth throttling is a source of headaches in PREEMPT_RT - generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling the cfs_rq of said CFS task. Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the cfs_rq has depleted its quota, we = wait for the task to be about to return to userspace. This approach also benefits !PREEMPT_RT, as it reduces latency caused by throttled tasks owning contended (kernel) resources. Concept =3D=3D=3D=3D=3D=3D=3D Upon throttling a cfs_rq, all tasks already enqueued get a task_work added, which lets the actual throttling happen in exit_to_user_mode(). Any new task migrated to or enqueued on such a cfs_rq similarly gets the task_work added. Previous patches have added helpers for all the relevant locations where the task_work may need to be either added or removed depending on the state of the cfs_rq the task is (to be) enqueued on: o sched_class change o cgroup migration o CPU migration o task wakeup Upon unthrottling, tasks are enqueued back onto their respective cfs_rq. Unlike the previous throttling implementation, cfs_rq's can be unthrottled while in a half-throttled state (i.e. some tasks have been removed from them, while others are still enqueued and runnable as they haven't reached exit_to_user_mode() yet), so the unthrottling process is a bit more involved, especially when it comes to maintaining *nr_running fiel= ds. Clocks =3D=3D=3D=3D=3D=3D Correctly handling the different cfs_rq->throttled_clock* is tricky, as unlike the current upstream approach where all tasks of a cfs_rq are throttled at the exact same time, here they each get throttled at a per-task, not-known-beforehand time. For instance, for the ->throttled_clock_pelt, ideally we would need a per-task snapshot of when the task gets really throttled in exit_to_user_mode(), rather than a single snapshot of when the cfs_rq runs out of runtime. This isn't implemented here. The ->throttled_clock_pelt is set when the cfs_rq runs out of runtime, which means the "grace period" given to the cfs_rq's tasks on their way to exit_to_user_mode() isn't accounted. Notable behaviour changes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Once a cfs_rq is ->throttled, its tasks can continue running until they hit exit_to_user_mode(). This means they can keep draining further runtime from their cfs_rq, which can end up draining more than one period's worth of runtime. I've tested a 10ms runtime / 100ms period cgroup with an always running task: upstream gets a "clean" periodic pattern of 10ms runtime every 100ms, whereas this gets something more like 40ms runtime every 400ms. [1]: https://lore.kernel.org/all/20231031160120.GE15024@noisy.programming.k= icks-ass.net/ Suggested-by: Peter Zijlstra Signed-off-by: Valentin Schneider --- include/linux/sched.h | 1 + kernel/sched/fair.c | 438 ++++++++++++++++++++++++++++++------------ kernel/sched/sched.h | 4 + 3 files changed, 320 insertions(+), 123 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 99a1e77d769db..29b9334738af1 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -815,6 +815,7 @@ struct task_struct { struct task_group *sched_task_group; struct callback_head sched_throttle_work; #ifdef CONFIG_CFS_BANDWIDTH + struct list_head throttle_node; struct irq_work unthrottle_irq_work; #endif #endif diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0cec3e70f1277..08cf7343aedb1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -54,6 +54,7 @@ #include "sched.h" #include "stats.h" #include "autogroup.h" +#include "task_work_sched.h" =20 /* * The initial- and re-scaling of tunables is configurable @@ -5694,12 +5695,114 @@ static inline int throttled_hierarchy(struct cfs_r= q *cfs_rq) return cfs_bandwidth_used() && cfs_rq->throttle_count; } =20 -static inline bool task_has_throttle_work(struct task_struct *p) { return = false; } -static inline bool task_needs_throttling(struct task_struct *p) { return f= alse; } -static inline bool task_needs_migrate_throttling(struct task_struct *p, un= signed int dst_cpu) { return false; } -static inline void task_throttle_setup(struct task_struct *p) { } -static inline void task_throttle_cancel_migrate(struct task_struct *p, int= dst_cpu) { } -static inline void task_throttle_cancel(struct task_struct *p) { } +static inline bool task_has_throttle_work(struct task_struct *p) +{ + return p->sched_throttle_work.next !=3D &p->sched_throttle_work; +} + +static inline bool task_needs_throttling(struct task_struct *p) +{ + return throttled_hierarchy(cfs_rq_of(&p->se)); +} + +static inline bool task_needs_migrate_throttling(struct task_struct *p, un= signed int dst_cpu) +{ + return throttled_hierarchy(task_group(p)->cfs_rq[dst_cpu]); +} + +static inline bool task_is_throttled(struct task_struct *p) +{ + return !list_empty(&p->throttle_node); +} + +static inline void task_throttle_setup_work(struct task_struct *p) +{ + /* + * Kthreads and exiting tasks don't return to userspace, so adding the + * work is pointless + */ + if (!(p->flags & (PF_EXITING | PF_KTHREAD))) + task_work_add(p, &p->sched_throttle_work, TWA_RESUME); +} + +static void throttle_cfs_rq_work(struct callback_head *work); +static inline void task_throttle_do_cancel_work(struct task_struct *p) +{ + /* + * If this returns NULL, it means the work got run, which per + * this being called is a bug: the task_work throttled the + * task when it didn't need to be. + */ + WARN_ON_ONCE(!task_work_cancel_locked(p, throttle_cfs_rq_work)); + p->sched_throttle_work.next =3D &p->sched_throttle_work; +} + +static inline void task_throttle_cancel_work(struct task_struct *p, int ds= t_cpu) +{ + /* + * The calling context may be holding p->pi_lock, which is also acquired + * by task_work_cancel_match(). + * + * Lock recursion is prevented by punting the work cancellation to the + * next IRQ enable. This is sent to the destination CPU rather than + * >this< CPU to prevent the task from resuming execution and getting + * throttled in its return to userspace. + */ + irq_work_queue_on(&p->unthrottle_irq_work, dst_cpu); +} + +static void task_throttle_cancel_irq_work_fn(struct irq_work *work) +{ + struct task_struct *p =3D container_of(work, struct task_struct, unthrott= le_irq_work); + int cpu =3D raw_smp_processor_id(); + + CLASS(task_rq_lock, rq_guard)(p); + WARN_ON_ONCE(task_cpu(p) !=3D cpu); + + if (task_has_throttle_work(p) && !task_needs_throttling(p)) + task_throttle_do_cancel_work(p); +} + +static inline void task_throttle_setup(struct task_struct *p) +{ + /* + * If already throttled-in-userspace, just transfer the throttle_node + * link to the new cfs_rq + * + * Else, if not yet throttled, set up the work. Also, the task may be + * running in userspace (e.g. this is called from sched_move_task()), + * so make sure it is running in kernelspace to get the kernel-exit + * throttle. + */ + if (task_is_throttled(p)) + list_move(&p->throttle_node, &cfs_rq_of(&p->se)->throttled_limbo_list); + else if (!task_has_throttle_work(p)) + task_throttle_setup_work(p); +} + +static void enqueue_task_fair(struct rq *rq, struct task_struct *p, int fl= ags); +static inline void __task_throttle_cancel(struct task_struct *p, unsigned = int cpu) +{ + /* + * Task musn't be throttled, either: + * o it's already throttled-in-userspace, unthrottle it + * o it has the task_work installed, remove it + */ + if (task_is_throttled(p)) { + list_del_init(&p->throttle_node); + enqueue_task_fair(cpu_rq(cpu), p, ENQUEUE_WAKEUP); + } else if (task_has_throttle_work(p)) { + task_throttle_cancel_work(p, cpu); + } +} +static inline void task_throttle_cancel(struct task_struct *p) +{ + __task_throttle_cancel(p, task_cpu(p)); +} +static inline void task_throttle_cancel_migrate(struct task_struct *p, uns= igned int dst_cpu) +{ + __task_throttle_cancel(p, dst_cpu); +} =20 /* * Ensure that neither of the group entities corresponding to src_cpu or @@ -5722,35 +5825,107 @@ static int tg_unthrottle_up(struct task_group *tg,= void *data) { struct rq *rq =3D data; struct cfs_rq *cfs_rq =3D tg->cfs_rq[cpu_of(rq)]; + struct sched_entity *se =3D tg->se[cpu_of(rq)]; + struct cfs_rq *pcfs_rq =3D cfs_rq_of(se); + long task_delta =3D 0, idle_task_delta =3D 0; + struct task_struct *p, *tmp; =20 cfs_rq->throttle_count--; - if (!cfs_rq->throttle_count) { - cfs_rq->throttled_clock_pelt_time +=3D rq_clock_pelt(rq) - - cfs_rq->throttled_clock_pelt; + if (cfs_rq->throttle_count) + return 0; =20 - /* Add cfs_rq with load or one or more already running entities to the l= ist */ - if (!cfs_rq_is_decayed(cfs_rq)) - list_add_leaf_cfs_rq(cfs_rq); + cfs_rq->throttled_clock_pelt_time +=3D rq_clock_pelt(rq) - + cfs_rq->throttled_clock_pelt; + + /* Add cfs_rq with load or one or more already running entities to the li= st */ + if (!cfs_rq_is_decayed(cfs_rq)) + list_add_leaf_cfs_rq(cfs_rq); =20 - if (cfs_rq->throttled_clock_self) { - u64 delta =3D rq_clock(rq) - cfs_rq->throttled_clock_self; + if (cfs_rq->throttled_clock_self) { + u64 delta =3D rq_clock(rq) - cfs_rq->throttled_clock_self; =20 - cfs_rq->throttled_clock_self =3D 0; + cfs_rq->throttled_clock_self =3D 0; =20 - if (SCHED_WARN_ON((s64)delta < 0)) - delta =3D 0; + if (SCHED_WARN_ON((s64)delta < 0)) + delta =3D 0; =20 - cfs_rq->throttled_clock_self_time +=3D delta; - } + cfs_rq->throttled_clock_self_time +=3D delta; + } + + /* + * Re-enqueue the tasks that have been throttled at this level. + * + * The task count is up-propagated via ->unthrottled_*h_nr_running, + * as we can't purely rely on h_nr_running post-enqueue: the unthrottle + * might happen when a cfs_rq still has some tasks enqueued, either still + * making their way to userspace, or freshly migrated to it. + */ + list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_= node) { + struct sched_entity *pse =3D &p->se; + + list_del_init(&p->throttle_node); + + enqueue_entity(cfs_rq, pse, ENQUEUE_WAKEUP); + task_delta++; + idle_task_delta +=3D task_has_idle_policy(p); + } + + /* + * Account tasks woken up in children; by this point all direct children + * have been visited. + */ + task_delta +=3D cfs_rq->unthrottled_h_nr_running; + idle_task_delta +=3D cfs_rq->unthrottled_idle_h_nr_running; + + cfs_rq->h_nr_running +=3D task_delta; + cfs_rq->idle_h_nr_running +=3D idle_task_delta; + + /* + * unthrottle_cfs_rq() needs a value to up-propagate above the + * freshly unthrottled cfs_rq. + */ + cfs_rq->unthrottled_h_nr_running =3D task_delta; + cfs_rq->unthrottled_idle_h_nr_running =3D idle_task_delta; + + /* Accumulate the delta in the parent's stash. Once all its children + * (i.e. all of this cfs_rq's siblings) have been visited, this value + * will be stable and used for its own count update. + */ + pcfs_rq->unthrottled_h_nr_running +=3D task_delta; + pcfs_rq->unthrottled_idle_h_nr_running +=3D idle_task_delta; + + /* + * If the cfs_rq became empty during throttling, then we dequeued + * it. It needs to be put back in the hierarchy if it or any of + * its children have now-unthrottled tasks. + */ + if (!se->on_rq && (cfs_rq->h_nr_running || cfs_rq->idle_h_nr_running)) { + enqueue_entity(pcfs_rq, se, ENQUEUE_WAKEUP); + } else { + update_load_avg(pcfs_rq, se, UPDATE_TG); + se_update_runnable(se); } =20 return 0; } =20 +static int tg_unthrottle_clear_up(struct task_group *tg, void *data) +{ + struct rq *rq =3D data; + struct cfs_rq *cfs_rq =3D tg->cfs_rq[cpu_of(rq)]; + + cfs_rq->unthrottled_h_nr_running =3D 0; + cfs_rq->unthrottled_idle_h_nr_running =3D 0; + + return 0; +} + static int tg_throttle_down(struct task_group *tg, void *data) { struct rq *rq =3D data; struct cfs_rq *cfs_rq =3D tg->cfs_rq[cpu_of(rq)]; + struct rb_node *node; + struct task_struct *p; =20 /* group is entering throttled state, stop time */ if (!cfs_rq->throttle_count) { @@ -5763,17 +5938,118 @@ static int tg_throttle_down(struct task_group *tg,= void *data) } cfs_rq->throttle_count++; =20 + /* + * If we've already visited this cfs_rq (e.g. it ran out of its own + * runtime sometime earlier and hasn't had a replenish yet), then + * there's nothing more to do. + */ + if (cfs_rq->throttle_count > 1) + return 0; + + WARN_ON_ONCE(!list_empty(&cfs_rq->throttled_limbo_list)); + /* + * rq_lock is held, current is (obviously) executing this in kernelspace. + * + * All other tasks enqueued on this rq have their saved PC at the + * context switch, so they will go through the kernel before returning + * to userspace. Thus, there are no tasks-in-userspace to handle, just + * install the task_work on all of them. + */ + node =3D rb_first(&cfs_rq->tasks_timeline.rb_root); + while (node) { + struct sched_entity *se =3D __node_2_se(node); + + if (!entity_is_task(se)) + goto next; + + p =3D task_of(se); + + if (!task_has_throttle_work(p)) + task_throttle_setup_work(p); +next: + node =3D rb_next(node); + } + return 0; } =20 -static void throttle_cfs_rq_work(struct callback_head *work) +static void throttle_one_task(struct cfs_rq *cfs_rq, struct task_struct *p) { + long task_delta, idle_task_delta; + struct sched_entity *se =3D &p->se; + + list_add(&p->throttle_node, &cfs_rq->throttled_limbo_list); =20 + task_delta =3D 1; + idle_task_delta =3D cfs_rq_is_idle(cfs_rq) ? 1 : 0; + + for_each_sched_entity(se) { + cfs_rq =3D cfs_rq_of(se); + + if (!se->on_rq) + return; + + dequeue_entity(cfs_rq, se, DEQUEUE_SLEEP); + cfs_rq->h_nr_running -=3D task_delta; + cfs_rq->idle_h_nr_running -=3D idle_task_delta; + + if (cfs_rq->load.weight) { + /* Avoid re-evaluating load for this entity: */ + se =3D parent_entity(se); + break; + } + } + + for_each_sched_entity(se) { + cfs_rq =3D cfs_rq_of(se); + /* throttled entity or throttle-on-deactivate */ + if (!se->on_rq) + goto throttle_done; + + update_load_avg(cfs_rq, se, 0); + se_update_runnable(se); + cfs_rq->h_nr_running -=3D task_delta; + cfs_rq->h_nr_running -=3D idle_task_delta; + } + +throttle_done: + /* At this point se is NULL and we are at root level*/ + sub_nr_running(rq_of(cfs_rq), 1); } =20 -static void task_throttle_cancel_irq_work_fn(struct irq_work *work) +static void throttle_cfs_rq_work(struct callback_head *work) { - /* Write me */ + struct task_struct *p =3D container_of(work, struct task_struct, sched_th= rottle_work); + struct sched_entity *se; + struct rq *rq; + struct cfs_rq *cfs_rq; + + WARN_ON_ONCE(p !=3D current); + p->sched_throttle_work.next =3D &p->sched_throttle_work; + /* + * If task is exiting, then there won't be a return to userspace, so we + * don't have to bother with any of this. + */ + if ((p->flags & PF_EXITING)) + return; + + CLASS(task_rq_lock, rq_guard)(p); + rq =3D rq_guard.rq; + se =3D &p->se; + cfs_rq =3D cfs_rq_of(se); + + /* + * If not in limbo, then either replenish has happened or this task got + * migrated out of the throttled cfs_rq, move along + */ + if (!cfs_rq->throttle_count) + return; + + update_rq_clock(rq); + + throttle_one_task(cfs_rq, p); + + resched_curr(rq); } =20 static void task_woken_fair(struct rq *rq, struct task_struct *p) @@ -5792,6 +6068,7 @@ void init_cfs_throttle_work(struct task_struct *p) /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_= work() */ p->sched_throttle_work.next =3D &p->sched_throttle_work; init_task_work(&p->sched_throttle_work, throttle_cfs_rq_work); + INIT_LIST_HEAD(&p->throttle_node); p->unthrottle_irq_work =3D IRQ_WORK_INIT_HARD(task_throttle_cancel_irq_wo= rk_fn); } =20 @@ -5799,8 +6076,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { struct rq *rq =3D rq_of(cfs_rq); struct cfs_bandwidth *cfs_b =3D tg_cfs_bandwidth(cfs_rq->tg); - struct sched_entity *se; - long task_delta, idle_task_delta, dequeue =3D 1; + long dequeue =3D 1; =20 raw_spin_lock(&cfs_b->lock); /* This will start the period timer if necessary */ @@ -5818,70 +6094,24 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) list_add_tail_rcu(&cfs_rq->throttled_list, &cfs_b->throttled_cfs_rq); } + raw_spin_unlock(&cfs_b->lock); =20 if (!dequeue) return false; /* Throttle no longer required. */ =20 - se =3D cfs_rq->tg->se[cpu_of(rq_of(cfs_rq))]; - - /* freeze hierarchy runnable averages while throttled */ + /* Flag the hierarchy for throttle-at-user-entry */ rcu_read_lock(); walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq); rcu_read_unlock(); =20 - task_delta =3D cfs_rq->h_nr_running; - idle_task_delta =3D cfs_rq->idle_h_nr_running; - for_each_sched_entity(se) { - struct cfs_rq *qcfs_rq =3D cfs_rq_of(se); - /* throttled entity or throttle-on-deactivate */ - if (!se->on_rq) - goto done; - - dequeue_entity(qcfs_rq, se, DEQUEUE_SLEEP); - - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta =3D cfs_rq->h_nr_running; - - qcfs_rq->h_nr_running -=3D task_delta; - qcfs_rq->idle_h_nr_running -=3D idle_task_delta; - - if (qcfs_rq->load.weight) { - /* Avoid re-evaluating load for this entity: */ - se =3D parent_entity(se); - break; - } - } - - for_each_sched_entity(se) { - struct cfs_rq *qcfs_rq =3D cfs_rq_of(se); - /* throttled entity or throttle-on-deactivate */ - if (!se->on_rq) - goto done; - - update_load_avg(qcfs_rq, se, 0); - se_update_runnable(se); - - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta =3D cfs_rq->h_nr_running; - - qcfs_rq->h_nr_running -=3D task_delta; - qcfs_rq->idle_h_nr_running -=3D idle_task_delta; - } - - /* At this point se is NULL and we are at root level*/ - sub_nr_running(rq, task_delta); - -done: - /* - * Note: distribution will already see us throttled via the - * throttled-list. rq->lock protects completion. - */ cfs_rq->throttled =3D 1; + SCHED_WARN_ON(cfs_rq->throttled_clock); if (cfs_rq->nr_running) cfs_rq->throttled_clock =3D rq_clock(rq); - return true; + + return false; } =20 void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) @@ -5922,25 +6152,17 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) goto unthrottle_throttle; } =20 - task_delta =3D cfs_rq->h_nr_running; - idle_task_delta =3D cfs_rq->idle_h_nr_running; - for_each_sched_entity(se) { - struct cfs_rq *qcfs_rq =3D cfs_rq_of(se); - - if (se->on_rq) - break; - enqueue_entity(qcfs_rq, se, ENQUEUE_WAKEUP); - - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta =3D cfs_rq->h_nr_running; + if (cfs_rq->throttle_count) + return; =20 - qcfs_rq->h_nr_running +=3D task_delta; - qcfs_rq->idle_h_nr_running +=3D idle_task_delta; + /* + * cfs_rq's below us may not have been fully emptied out, so we can't rely + * directly on ->h_nr_running. Use the stash instead. + */ + task_delta =3D cfs_rq->unthrottled_h_nr_running; + idle_task_delta =3D cfs_rq->unthrottled_idle_h_nr_running; =20 - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(qcfs_rq)) - goto unthrottle_throttle; - } + walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_clear_up, (void *)rq); =20 for_each_sched_entity(se) { struct cfs_rq *qcfs_rq =3D cfs_rq_of(se); @@ -5948,15 +6170,8 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) update_load_avg(qcfs_rq, se, UPDATE_TG); se_update_runnable(se); =20 - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta =3D cfs_rq->h_nr_running; - qcfs_rq->h_nr_running +=3D task_delta; qcfs_rq->idle_h_nr_running +=3D idle_task_delta; - - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(qcfs_rq)) - goto unthrottle_throttle; } =20 /* At this point se is NULL and we are at root level*/ @@ -6455,6 +6670,7 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq) cfs_rq->runtime_enabled =3D 0; INIT_LIST_HEAD(&cfs_rq->throttled_list); INIT_LIST_HEAD(&cfs_rq->throttled_csd_list); + INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list); } =20 void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b) @@ -6822,10 +7038,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct = *p, int flags) if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running =3D 1; =20 - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto enqueue_throttle; - flags =3D ENQUEUE_WAKEUP; } =20 @@ -6841,10 +7053,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct = *p, int flags) =20 if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running =3D 1; - - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto enqueue_throttle; } =20 /* At this point se is NULL and we are at root level*/ @@ -6867,7 +7075,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) if (!task_new) check_update_overutilized_status(rq); =20 -enqueue_throttle: assert_list_leaf_cfs_rq(rq); =20 hrtick_update(rq); @@ -6900,10 +7107,6 @@ static void dequeue_task_fair(struct rq *rq, struct = task_struct *p, int flags) if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running =3D 1; =20 - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto dequeue_throttle; - /* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) { /* Avoid re-evaluating load for this entity: */ @@ -6932,10 +7135,6 @@ static void dequeue_task_fair(struct rq *rq, struct = task_struct *p, int flags) if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running =3D 1; =20 - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto dequeue_throttle; - } =20 /* At this point se is NULL and we are at root level*/ @@ -6945,7 +7144,6 @@ static void dequeue_task_fair(struct rq *rq, struct t= ask_struct *p, int flags) if (unlikely(!was_sched_idle && sched_idle_rq(rq))) rq->next_balance =3D jiffies; =20 -dequeue_throttle: util_est_update(&rq->cfs, p, task_sleep); hrtick_update(rq); } @@ -12815,9 +13013,6 @@ static void propagate_entity_cfs_rq(struct sched_en= tity *se) { struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 - if (cfs_rq_throttled(cfs_rq)) - return; - if (!throttled_hierarchy(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); =20 @@ -12829,9 +13024,6 @@ static void propagate_entity_cfs_rq(struct sched_en= tity *se) =20 update_load_avg(cfs_rq, se, UPDATE_TG); =20 - if (cfs_rq_throttled(cfs_rq)) - break; - if (!throttled_hierarchy(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 943bca8263ffe..f4a00b1dd9505 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -690,8 +690,12 @@ struct cfs_rq { u64 throttled_clock_self_time; int throttled; int throttle_count; + /* Temp storage for updating the counts during unthrottling */ + unsigned int unthrottled_h_nr_running; + unsigned int unthrottled_idle_h_nr_running; struct list_head throttled_list; struct list_head throttled_csd_list; + struct list_head throttled_limbo_list; #endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ }; --=20 2.43.0