From nobody Wed Dec 17 19:16:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8ED3B16A36E for ; Tue, 25 Jun 2024 13:52:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323572; cv=none; b=hJbS/2t+c1nPKg6PhZm7KTSGOih/ylW+vkrrrc+2xmuq/vz2o88M+SFbWBw57I4ctLNReaTm/DHKBFg0ZfpmqhuDFbV/6uVz/l3tV99KiRRjODrr0ptUGPiGglng3bysM4Zw1dccW/o4VnlaUOGuZn9I1H8I5NGG4HHRzd/vPTY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323572; c=relaxed/simple; bh=VVQeerrLbO6LlQCdF2302Yt57InaLmIbhRY3uETpAj8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jnbi68F6mbFrnhX5znBGtnLm4RvKb50PvZWKKBRe6VyADqOXnljAWbemSjZDFAM1w3lNJTuqZNtKLsg+eeBNl0RP+XXp3O0uLvTG+7MinQT6g9XJZgbsl4joILOXgO3v0x/kxZxhLa6GxZCNVLCiBFI9OvNXP7BRfSIpnYmQEVc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Dvpibtpn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Dvpibtpn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1BDFC32781; Tue, 25 Jun 2024 13:52:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719323572; bh=VVQeerrLbO6LlQCdF2302Yt57InaLmIbhRY3uETpAj8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DvpibtpnDkesQ+tyacOfQjqqQ6LH+vYE+pK/mVCKETW8l+d2hpwKKAzL2PacTZHYi WRMdIJnnxVzpdnY7IGMfKIoYA2A2gkDKow+3F41hlgqLy6TtT5fJtwfm/2CnF97fNa ptqYJ9rMMrP3Xf2VuCqO5l0E51wl42bTKucXuojKKaZFJT/cLmMWD0+ptrr9hlEIP4 AA+TejVSt0WYvqP+R9qFJX9Qlqif3mb7UGtRdTKmOV8qmyYEQdo8fHjy5C+OqdMXzz azZi28HNYEFDgYMcoDmDVeQ/0gPSPZP0msEfkw7TKTes9oC3Ju4UWduBncfjppDWkf Am07vcZh58EfQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov Subject: [RFC PATCH 1/6] task_work: Provide means to check if a work is queued Date: Tue, 25 Jun 2024 15:52:39 +0200 Message-ID: <20240625135244.20227-2-frederic@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240625135244.20227-1-frederic@kernel.org> References: <20240625135244.20227-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some task work users implement their own ways to know if a callback is already queued on the current task while fiddling with the callback head internals. Provide instead a consolidated API to serve this very purpose. Signed-off-by: Frederic Weisbecker --- include/linux/task_work.h | 12 ++++++++++++ kernel/task_work.c | 1 + 2 files changed, 13 insertions(+) diff --git a/include/linux/task_work.h b/include/linux/task_work.h index 795ef5a68429..f2eae971b73a 100644 --- a/include/linux/task_work.h +++ b/include/linux/task_work.h @@ -5,12 +5,15 @@ #include #include =20 +#define TASK_WORK_DEQUEUED ((void *) -1UL) + typedef void (*task_work_func_t)(struct callback_head *); =20 static inline void init_task_work(struct callback_head *twork, task_work_func_t func) { twork->func =3D func; + twork->next =3D TASK_WORK_DEQUEUED; } =20 enum task_work_notify_mode { @@ -25,6 +28,15 @@ static inline bool task_work_pending(struct task_struct = *task) return READ_ONCE(task->task_works); } =20 +/* + * Check if a work is queued. Beware: this is inherently racy if the work = can + * be queued elsewhere than the current task. + */ +static inline bool task_work_queued(struct callback_head *twork) +{ + return twork->next !=3D TASK_WORK_DEQUEUED; +} + int task_work_add(struct task_struct *task, struct callback_head *twork, enum task_work_notify_mode mode); =20 diff --git a/kernel/task_work.c b/kernel/task_work.c index 95a7e1b7f1da..6e3bee0b7011 100644 --- a/kernel/task_work.c +++ b/kernel/task_work.c @@ -177,6 +177,7 @@ void task_work_run(void) =20 do { next =3D work->next; + work->next =3D TASK_WORK_DEQUEUED; work->func(work); work =3D next; cond_resched(); --=20 2.45.2 From nobody Wed Dec 17 19:16:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 898E81607B0 for ; Tue, 25 Jun 2024 13:52:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323576; cv=none; b=NUOkNBWTIXd0zysHU73zYN4GFy19ctUn5PS0shjh/c4wICBI+w5t7p2xeb0OwfupnxHaTM38K6PDmRBpccXKll8LPe/pHXzlRQm3ehRf+5LtA5cKhwVHjjk9kR2BurkK829WpXsZ/r/fmsRLgXsXj17R441tote47Qed5UcG8oc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323576; c=relaxed/simple; bh=ku8xq4mkvyOVgfWzpgtzoN0rLYNOKZuKetn2NDoCUSs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EZaaynqqH0J5/TFIurHAHHmwEnOxmdvLxQdX6XYQ1yVrZXemZ5S4WxJVb65k1QxsUdzBr7I3JcyeZUu/DtX8432cQwOveOru9KV3KkiXfuWVb3Jh67G4SPSVEsH/xs6Kat7qCaFFh2KMQNcJ/yz8O0Dl93cdcjswxuYTtfwhVWE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=C3f+cNs0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="C3f+cNs0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E7C06C32786; Tue, 25 Jun 2024 13:52:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719323576; bh=ku8xq4mkvyOVgfWzpgtzoN0rLYNOKZuKetn2NDoCUSs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C3f+cNs0CJ6m1YlCQXcy1Leh2t4bIXi63yq905R2Up5s7E+Q3et4XpbCFSWDpMEus ogivtCtNLGLFrqL3BXUXRBfN6DudgsSDNrI3loJAzN86jpNC1/C2I+nmSd/3Jv1gQC 8T0fNrK2vLZ2gExC01nOcc5AvhQ3Avw00W7socnThgl2rsYU+M3hLzMzPRo8neEjPQ 7+E1eNzdzZzhDNC05QTGoMUw9Y5OhukcW69JsXnnTdmQNLNT7znCFsMCqaZWkherKB QtOWRs3XRgpjFSRvlYGxdylPJT9qX7lJJLT1YNum5D1Ks5lBCggC/ohG+WFm8RJsmo Xp3o/BRk0QwBA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov Subject: [RFC PATCH 2/6] sched/fair: Use task_work_queued() on numa_work Date: Tue, 25 Jun 2024 15:52:40 +0200 Message-ID: <20240625135244.20227-3-frederic@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240625135244.20227-1-frederic@kernel.org> References: <20240625135244.20227-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the ad-hoc implementation of task_work_queued(). Signed-off-by: Frederic Weisbecker --- kernel/sched/fair.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8a5b1ae0aa55..619ef8bd1486 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3213,7 +3213,6 @@ static void task_numa_work(struct callback_head *work) =20 SCHED_WARN_ON(p !=3D container_of(work, struct task_struct, numa_work)); =20 - work->next =3D work; /* * Who cares about NUMA placement when they're dying. * @@ -3456,7 +3455,6 @@ void init_numa_balancing(unsigned long clone_flags, s= truct task_struct *p) p->numa_scan_period =3D sysctl_numa_balancing_scan_delay; p->numa_migrate_retry =3D 0; /* Protect against double add, see task_tick_numa and task_numa_work */ - p->numa_work.next =3D &p->numa_work; p->numa_faults =3D NULL; p->numa_pages_migrated =3D 0; p->total_numa_faults =3D 0; @@ -3497,7 +3495,7 @@ static void task_tick_numa(struct rq *rq, struct task= _struct *curr) /* * We don't care about NUMA placement if we don't have memory. */ - if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next = !=3D work) + if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || task_work_q= ueued(work)) return; =20 /* --=20 2.45.2 From nobody Wed Dec 17 19:16:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 891F916C68F for ; Tue, 25 Jun 2024 13:52:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323579; cv=none; b=oe3Unb9kBiooNXeSb1JXb4QLovWbQXZtdjnuHnJBTggb3w/NwE+71euKRoJ8aXMQzM3InmplR5Z88ulEJ8CQ0GrOJ+rgjOwT5dnVydoga0Qo3oj0yOb+ac9d2i9tKajlfyYkUqmdRumAFSZHBs/giVEaZLrD2loMaW3+yKvaP/Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323579; c=relaxed/simple; bh=1Lmu4cLSQd3WIs+Qp9CQZmWu+Bdz0TB/eweAMF+fREs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bjJLLotF6ihKRHBHw3MNOyO8BccRDyYPVtzyCi5mT71COdbr8csfz1U3HRevtbcB2uM9l/qMLcxm9NWA0Rd+p/p38aO7sv8fyk3JJHiNH9OrBO0L7t9ThTYaXZxoaG2/3y9bJ/TLWpdPkEH7pOpXN2Gtz5PiDOEe66MWwEC20VQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tMpGwrrs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tMpGwrrs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A2487C4AF10; Tue, 25 Jun 2024 13:52:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719323579; bh=1Lmu4cLSQd3WIs+Qp9CQZmWu+Bdz0TB/eweAMF+fREs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tMpGwrrs8lNBF6+IyzFFzHnboXv74SEp00R5GHX9cijPU3UyznfXy/D5o4eiuMJu2 3+gPpot5RrOwwJ/cEqn0Z7DcJncm/JiYV0gkT3eqm1uo6ObLGIUmnq++QYNGuwX9rW F/WmyqxJAV42SKJbW0aW+J5fW9TGCApj0IwUD06I2D1C1aWCadPHark+1iK7dF23gA qoPqYDwLgoqKsAr9PEqp6t7Y5bB0a6mANVONasJrxa1zkYsaSYD1xMBhqPXs9caA8X CUD2k2hPj1yTGeNlZoBpPzwUHklQa1n7sxEc6+OqW7p8+MML4AvPts13Fi1WayKqAz ApYyFfuSx6AOA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov Subject: [RFC PATCH 3/6] sched: Use task_work_queued() on cid_work Date: Tue, 25 Jun 2024 15:52:41 +0200 Message-ID: <20240625135244.20227-4-frederic@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240625135244.20227-1-frederic@kernel.org> References: <20240625135244.20227-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the ad-hoc implementation of task_work_queued() Signed-off-by: Frederic Weisbecker Reviewed-by: Valentin Schneider --- kernel/sched/core.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index bcf2c4cc0522..f01979b600e8 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -11930,7 +11930,6 @@ static void task_mm_cid_work(struct callback_head *= work) =20 SCHED_WARN_ON(t !=3D container_of(work, struct task_struct, cid_work)); =20 - work->next =3D work; /* Prevent double-add */ if (t->flags & PF_EXITING) return; mm =3D t->mm; @@ -11974,7 +11973,6 @@ void init_sched_mm_cid(struct task_struct *t) if (mm_users =3D=3D 1) mm->mm_cid_next_scan =3D jiffies + msecs_to_jiffies(MM_CID_SCAN_DELAY); } - t->cid_work.next =3D &t->cid_work; /* Protect against double add */ init_task_work(&t->cid_work, task_mm_cid_work); } =20 @@ -11983,8 +11981,7 @@ void task_tick_mm_cid(struct rq *rq, struct task_st= ruct *curr) struct callback_head *work =3D &curr->cid_work; unsigned long now =3D jiffies; =20 - if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || - work->next !=3D work) + if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || task_work_q= ueued(work)) return; if (time_before(now, READ_ONCE(curr->mm->mm_cid_next_scan))) return; --=20 2.45.2 From nobody Wed Dec 17 19:16:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 456B916C87A for ; Tue, 25 Jun 2024 13:53:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323582; cv=none; b=s4yhmwm6rUC2BJyqLPsBezGE20FuSIUMMwHYDgfwwWFn7rpMZGI9v/O4kxBV01ZEu2UbT2NsIOkfpf6j2/dWgtChMOSbqpBcMhPkuiQljljPsghqBIJz+JdgKnuYZvmLush/RyYQOTJBt2OwopUAluOIFsxQFkJwfdfI67ak/C8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323582; c=relaxed/simple; bh=xcwSoktWtb2GRPN0qb4RAhRIXNrCFWUhJrYeNl3lG5c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e8TglDWpBN+cRo6FMN97i/B6uTJsbTxlnvyb9O5OEoKofX4kILigYZH7HyerGaTTVWECnKbNDabwlWXwTl2IiX/jV0eZHFXoDj6vOY35fgapFR/TLN5bPkg3rXtpSOkKEXeM39w22UdILxH9i58SeKr8sMcpSfSXaqItp3VHfsc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Hgt2fyT9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Hgt2fyT9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C613C4AF09; Tue, 25 Jun 2024 13:52:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719323581; bh=xcwSoktWtb2GRPN0qb4RAhRIXNrCFWUhJrYeNl3lG5c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Hgt2fyT92tOPNdLavtHVDgjO5wk805rmWMAbRmSYeCjCPjs/Xwtz6/qvRFHptYbML gu07aPk6N4Svolo1Lzy7PBVDtOrZY+QpbeCDpwdvs1rYBo5KwcoV1yYlPCMV8xSkT1 aS7VzWxcZcRsmvheTWCiB71O+7Iu85baTVSQoHxmAPZAyxTvar/ptQLGJ/D9PBBFYN x7ACrgkxXykgHhPJtWOJm20tWxBbKzIIX17i+7R0LSIjkerL7lgy4YqQLJa1rSQ6mW S3xD6IShUAvedq2dDaiaiKU4xK0+jXhzzzgET3LDY1uIMNQSXo/UBTA451u7s4fpnH D2/UjzGxQ8IDA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov Subject: [RFC PATCH 4/6] tick/nohz: Move nohz_full related fields out of hot task struct's places Date: Tue, 25 Jun 2024 15:52:42 +0200 Message-ID: <20240625135244.20227-5-frederic@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240625135244.20227-1-frederic@kernel.org> References: <20240625135244.20227-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" nohz_full is a feature that only fits into rare and very corner cases. Yet distros enable it by default and therefore the related fields are always reserved in the task struct. Those task fields are stored in the middle of cacheline hot places such as cputime accounting and context switch counting, which doesn't make any sense for a feature that is disabled most of the time. Move the nohz_full storage to colder places. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 61591ac6eab6..d531b610c410 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1039,13 +1039,7 @@ struct task_struct { #endif u64 gtime; struct prev_cputime prev_cputime; -#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN - struct vtime vtime; -#endif =20 -#ifdef CONFIG_NO_HZ_FULL - atomic_t tick_dep_mask; -#endif /* Context switch counts: */ unsigned long nvcsw; unsigned long nivcsw; @@ -1350,6 +1344,14 @@ struct task_struct { struct task_delay_info *delays; #endif =20 +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN + struct vtime vtime; +#endif + +#ifdef CONFIG_NO_HZ_FULL + atomic_t tick_dep_mask; +#endif + #ifdef CONFIG_FAULT_INJECTION int make_it_fail; unsigned int fail_nth; --=20 2.45.2 From nobody Wed Dec 17 19:16:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC59516D4C5 for ; Tue, 25 Jun 2024 13:53:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323585; cv=none; b=gUiWqBCm+z82oLzfiLIIQs2rjlFB+1nT6QbWmubbLbAVvTz8O+6zKUv38se3dHcYiHtNwx6I7UQ34GN5r+B6134ui8RagbakiLD9IJ162R/BjxbNajClOGm61FCAHa0yxEQnHA4s0t2JmmrKYmBS5ihyGeUKKv3HUOjVF9zc+n8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323585; c=relaxed/simple; bh=jsbe59F+45c+ldudiJuMxz+R4Pww7gQC2ECwzseICnw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oXlQFrg4Qq6sceExevC2Lw0+taqQddTwBh2WUG5VBk1Lb8gK+2prfneJeua02baUmoUhgG4s+/dog0r5IR9oL/9X+v5p98hGtYe/fM5ABS7ocLoPDMhCuZX1g1295ItRfY16KW7Y/azJe17wt5iEiO3m+VmIF+gY2uSUWwmrYMU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pFuYuusr; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pFuYuusr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D979C4AF07; Tue, 25 Jun 2024 13:53:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719323584; bh=jsbe59F+45c+ldudiJuMxz+R4Pww7gQC2ECwzseICnw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pFuYuusrrokX4bozFJb3gokF7224w3g7XKbz5BhHFPbjypLcXEuLjqADsAUkWNiXv uzn0YCSsp7NNjfZ7IKeoLbNGkPIy1tP7GeTSKIDFNIpMvngPVX5Jl3wAxfBRKrmfsX jL6pvIfVA/Ku1FWinT1LQOoPM58QVgvWonuUhe/752u1puf94Jk5qHWac7kpDtx1b3 EN8yqZvvy8TYHE+UdT6lRi3ELnKm3Osaxw0lzUthWo669tFROB8abV649Tc5j4BeTv WmY4/ERQbBDOC649uELSHbYi0jE5SsMZdrZ9UkXIBD6OTgSESedaejAaIwZ2JRNd2B NYVRoTp02E/Pw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov Subject: [RFC PATCH 5/6] sched/isolation: Introduce isolated task work Date: Tue, 25 Jun 2024 15:52:43 +0200 Message-ID: <20240625135244.20227-6-frederic@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240625135244.20227-1-frederic@kernel.org> References: <20240625135244.20227-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some asynchronous kernel work may be pending upon resume to userspace and execute later on. On isolated workload this becomes problematic once the process is done with preparatory work involving syscalls and wants to run in userspace without being interrupted. Provide an infrastructure to queue a work to be executed from the current isolated task context right before resuming to userspace. This goes with the assumption that isolated tasks are pinned to a single nohz_full CPU. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 1 + include/linux/sched/isolation.h | 17 +++++++++++++++++ kernel/sched/core.c | 1 + kernel/sched/isolation.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 5 files changed, 51 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index d531b610c410..f6df21866055 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1349,6 +1349,7 @@ struct task_struct { #endif =20 #ifdef CONFIG_NO_HZ_FULL + struct callback_head nohz_full_work; atomic_t tick_dep_mask; #endif =20 diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolatio= n.h index 2b461129d1fa..e69ec5ed1d70 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -72,4 +72,21 @@ static inline bool cpu_is_isolated(int cpu) cpuset_cpu_is_isolated(cpu); } =20 +#if defined(CONFIG_NO_HZ_FULL) +extern int __isolated_task_work_queue(void); + +static inline int isolated_task_work_queue(void) +{ + if (!housekeeping_test_cpu(raw_smp_processor_id(), HK_TYPE_TICK)) + return -ENOTSUPP; + + return __isolated_task_work_queue(); +} + +extern void isolated_task_work_init(struct task_struct *tsk); +#else +static inline int isolated_task_work_queue(void) { } +static inline void isolated_task_work_init(struct task_struct *tsk) { } +#endif /* CONFIG_NO_HZ_FULL */ + #endif /* _LINUX_SCHED_ISOLATION_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index f01979b600e8..01960434dbfd 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4566,6 +4566,7 @@ static void __sched_fork(unsigned long clone_flags, s= truct task_struct *p) p->migration_pending =3D NULL; #endif init_sched_mm_cid(p); + isolated_task_work_init(p); } =20 DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 5891e715f00d..410df1fedc9d 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -253,3 +253,34 @@ static int __init housekeeping_isolcpus_setup(char *st= r) return housekeeping_setup(str, flags); } __setup("isolcpus=3D", housekeeping_isolcpus_setup); + +#if defined(CONFIG_NO_HZ_FULL) +static void isolated_task_work(struct callback_head *head) +{ +} + +int __isolated_task_work_queue(void) +{ + unsigned long flags; + int ret; + + if (current->flags & PF_KTHREAD) + return 0; + + local_irq_save(flags); + if (task_work_queued(¤t->nohz_full_work)) { + ret =3D 0; + goto out; + } + + ret =3D task_work_add(current, ¤t->nohz_full_work, TWA_RESUME); +out: + local_irq_restore(flags); + return ret; +} + +void isolated_task_work_init(struct task_struct *tsk) +{ + init_task_work(&tsk->nohz_full_work, isolated_task_work); +} +#endif /* CONFIG_NO_HZ_FULL */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a831af102070..24653f5879cc 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -60,6 +60,7 @@ #include #include #include +#include #include #include #include --=20 2.45.2 From nobody Wed Dec 17 19:16:26 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7AFC16E863 for ; Tue, 25 Jun 2024 13:53:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323587; cv=none; b=HVTnriE5kBInQFDyZXkrYIWBDLJmVLzqwOUBm95YOLJJ5JmMdP+TkOv0ZM3glbELHOdcgbOMWvtCewinQ1RdMiYsq957NpTpQ7L+UfftRTzmXIBYkSv/vXa5iR1XQLBZdUmVBu4ezCoxq8eMoU0QylD+PkxuuTbdsySMq3uHnOs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719323587; c=relaxed/simple; bh=DFr4TRqKLmfPFbF2NSC8zCtiAY10KyWLCkq+RiflglI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AOSfFwoGIGUZuPXIYvvHq91oUj0KzCvnPTw19BMJt9U89VF2i7chSczkw74GIWn5eiAVMNYp4hjDyH4A+ESg9r2rNpu1OWnNl96HoVPLfUIH5FTNqhrf2Wv/kc85azvLNXK6EXcuWdhNsW9J8wS4OojDC7GyPicdM2Pcrapsjqk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LzwXSsVt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LzwXSsVt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 365EEC32786; Tue, 25 Jun 2024 13:53:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719323587; bh=DFr4TRqKLmfPFbF2NSC8zCtiAY10KyWLCkq+RiflglI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LzwXSsVt2voH3OzqD4MFTzuqtShyjCtLb16dy5HNsoUN5HmLY2v1hdFbu3jG2+c70 PO4HNhVlGAVHA/sUkr2H6CEnI49KwAzLn9DSQgjC8mIrDOe8RAJ2DpRgnYac4P+JTP l2bW6ZTJg3XLEaAwrlEWJsNJzurXlMn9+FTOgV+4MxAOUemX5/XputUuuygW3Y81xz mNMTh2us0uqpF+3CSYPZm6auUY8cqFShmA0DjtXS6XUSIf8d9Uc3mh1ormmR6s3eqr T8AmNwQbu6vDs1/nYuyZsm+4KwDoGm9Na41rriSfWi0WUREiBl3U2whvATQh+i/9De URkGWbQuD25mg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov Subject: [RFC PATCH 6/6] mm: Drain LRUs upon resume to userspace on nohz_full CPUs Date: Tue, 25 Jun 2024 15:52:44 +0200 Message-ID: <20240625135244.20227-7-frederic@kernel.org> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240625135244.20227-1-frederic@kernel.org> References: <20240625135244.20227-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" LRUs can be drained through several ways. One of them may add disturbances to isolated workloads while queuing a work at any time to any target, whether running in nohz_full mode or not. Prevent from that on isolated tasks with draining LRUs upon resuming to userspace using the isolated task work framework. It's worth noting that this is inherently racy against lru_add_drain_all() remotely queueing the per CPU drain work and therefore it prevents from the undesired disturbance only *most of the time*. Signed-off-by: Frederic Weisbecker --- include/linux/swap.h | 1 + kernel/sched/isolation.c | 1 + mm/swap.c | 5 ++++- 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index bd450023b9a4..bd6169c9cc14 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -393,6 +393,7 @@ extern void lru_add_drain(void); extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_cpu_zone(struct zone *zone); extern void lru_add_drain_all(void); +extern void lru_add_and_bh_lrus_drain(void); void folio_deactivate(struct folio *folio); void folio_mark_lazyfree(struct folio *folio); extern void swap_setup(void); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 410df1fedc9d..68c70bea99e7 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -257,6 +257,7 @@ __setup("isolcpus=3D", housekeeping_isolcpus_setup); #if defined(CONFIG_NO_HZ_FULL) static void isolated_task_work(struct callback_head *head) { + lru_add_and_bh_lrus_drain(); } =20 int __isolated_task_work_queue(void) diff --git a/mm/swap.c b/mm/swap.c index 67786cb77130..a4d7e3dc2a66 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -37,6 +37,7 @@ #include #include #include +#include =20 #include "internal.h" =20 @@ -521,6 +522,8 @@ void folio_add_lru(struct folio *folio) fbatch =3D this_cpu_ptr(&cpu_fbatches.lru_add); folio_batch_add_and_move(fbatch, folio, lru_add_fn); local_unlock(&cpu_fbatches.lock); + + isolated_task_work_queue(); } EXPORT_SYMBOL(folio_add_lru); =20 @@ -765,7 +768,7 @@ void lru_add_drain(void) * the same cpu. It shouldn't be a problem in !SMP case since * the core is only one and the locks will disable preemption. */ -static void lru_add_and_bh_lrus_drain(void) +void lru_add_and_bh_lrus_drain(void) { local_lock(&cpu_fbatches.lock); lru_add_drain_cpu(smp_processor_id()); --=20 2.45.2