From nobody Wed Dec 17 19:17:03 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6349E241130 for ; Sun, 9 Feb 2025 22:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140216; cv=none; b=D4605DrOeC6bF1C4DRMh3Zobd/f+pZlbcTA5/FqZJ/0TGaUl9uV0gCam2KlF6pYup8RVNcDmPCHIXLVqW5lydSjKD3fPVOpzkG0wuBnfHJE76pgIThUJubRpAsxhqmKDZFd770cw1p8Aa3d8njHLvwGq1cAh64ZCUeJbG99Krlo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140216; c=relaxed/simple; bh=Wqntl1ggAaL74aa0p3y2a++/5r9dcdOK0C0/zF8B/OM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gNICrKQVFA5u2roOcjeoXc+N3etORVBAoKLni2BmtrphT3cB1KC0RDgsdCgqZNsvnK1wUxJ0ZkD6fTGd2mqRUAnTb4T+tY8cbGUzbokjSz/2JM7mK0tzkKbdzPQocbs9yFpzii2qC1IybPTGFPqQlGTac/hX4P3eJfx9KN2PA9Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KAPDlktu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KAPDlktu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A12FBC4CEDF; Sun, 9 Feb 2025 22:30:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140216; bh=Wqntl1ggAaL74aa0p3y2a++/5r9dcdOK0C0/zF8B/OM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KAPDlktuV4x4lOY7dt1MX379eFc7iOHZYEvyz50M+V1vMYLZ7fvi/DNn8BHL0uWC0 cXq25hvD0Trn40F1pWIwz9IczyCePognvXVQCGaIAKDvM2xUvzI6+hvU03b5Cy6676 LrZm7F5c4pvdz6EE7y7SE2gv1dj+bDk7jWASzAw/FMf9AVm6+e4zikrFJvFmLs4lOd 5iO3Oxaf7du0MevTy+0WWhJdedrBIABnQJ9GZ2QAeADelMF09h3j9edOOY4Ltj8gEI 9LAaHzsaa8DCVFuBbzU19CGwd0mRHpbQgpwddFBVCMCemNmiHRz8OEsri+qzxovymK LVdtWCPkeLQxA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 1/6 v2] task_work: Provide means to check if a work is queued Date: Sun, 9 Feb 2025 23:29:59 +0100 Message-ID: <20250209223005.11519-2-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some task work users implement their own ways to know if a callback is already queued on the current task while fiddling with the callback head internals. Provide instead a consolidated API to serve this very purpose. Signed-off-by: Frederic Weisbecker Reviewed-by: Oleg Nesterov Reviewed-by: Valentin Schneider --- include/linux/task_work.h | 12 ++++++++++++ kernel/task_work.c | 11 +++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/include/linux/task_work.h b/include/linux/task_work.h index 0646804860ff..31caf12c1313 100644 --- a/include/linux/task_work.h +++ b/include/linux/task_work.h @@ -5,12 +5,15 @@ #include #include =20 +#define TASK_WORK_DEQUEUED ((void *) -1UL) + typedef void (*task_work_func_t)(struct callback_head *); =20 static inline void init_task_work(struct callback_head *twork, task_work_func_t func) { twork->func =3D func; + twork->next =3D TASK_WORK_DEQUEUED; } =20 enum task_work_notify_mode { @@ -26,6 +29,15 @@ static inline bool task_work_pending(struct task_struct = *task) return READ_ONCE(task->task_works); } =20 +/* + * Check if a work is queued. Beware: this is inherently racy if the work = can + * be queued elsewhere than the current task. + */ +static inline bool task_work_queued(struct callback_head *twork) +{ + return twork->next !=3D TASK_WORK_DEQUEUED; +} + int task_work_add(struct task_struct *task, struct callback_head *twork, enum task_work_notify_mode mode); =20 diff --git a/kernel/task_work.c b/kernel/task_work.c index d1efec571a4a..0d7b04095753 100644 --- a/kernel/task_work.c +++ b/kernel/task_work.c @@ -56,6 +56,8 @@ int task_work_add(struct task_struct *task, struct callba= ck_head *work, { struct callback_head *head; =20 + work->next =3D TASK_WORK_DEQUEUED; + if (notify =3D=3D TWA_NMI_CURRENT) { if (WARN_ON_ONCE(task !=3D current)) return -EINVAL; @@ -67,8 +69,10 @@ int task_work_add(struct task_struct *task, struct callb= ack_head *work, =20 head =3D READ_ONCE(task->task_works); do { - if (unlikely(head =3D=3D &work_exited)) + if (unlikely(head =3D=3D &work_exited)) { + work->next =3D TASK_WORK_DEQUEUED; return -ESRCH; + } work->next =3D head; } while (!try_cmpxchg(&task->task_works, &head, work)); =20 @@ -129,8 +133,10 @@ task_work_cancel_match(struct task_struct *task, if (!match(work, data)) { pprev =3D &work->next; work =3D READ_ONCE(*pprev); - } else if (try_cmpxchg(pprev, &work, work->next)) + } else if (try_cmpxchg(pprev, &work, work->next)) { + work->next =3D TASK_WORK_DEQUEUED; break; + } } raw_spin_unlock_irqrestore(&task->pi_lock, flags); =20 @@ -224,6 +230,7 @@ void task_work_run(void) =20 do { next =3D work->next; + work->next =3D TASK_WORK_DEQUEUED; work->func(work); work =3D next; cond_resched(); --=20 2.46.0 From nobody Wed Dec 17 19:17:03 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99AB9241130 for ; Sun, 9 Feb 2025 22:30:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140219; cv=none; b=HgwI9cJx2GuyHBOBdOEx+M6N3YPJFiTdl/P5GFmg5UTXexrPTBlAvcz0+nAXwPQA/H1ZwdeXcQlHk2K90Te5IFfybzdV7ORiMvgdRPE9u/cSf9PEz0lz9XzbCfold/nsw5SydhZHtgJku84mQogTgrV0IOTuzxHcAAcmmLLyY3Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140219; c=relaxed/simple; bh=ZSwI4jjw4GwkwAYOaWT3SF4KhLf9zm4bvh5Jlfu1Q2E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=djV+rk6YkB2cP0v26Ze1USYKcui1Wnt5XHQUGo4RLv73DeKFClsU4zJUZ7BZL4Vihv4IiHNVHUE1kI43jc5ek1UrwasO2Eor7oxfGChHbo2/Uo3sGPb6EhMhXdnJ+CgoQRCSB8ocs8iP7mcaM60SMZg7PeIRyMiHG4orjcGiIMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EwO/kDlD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EwO/kDlD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 960E4C4CEE5; Sun, 9 Feb 2025 22:30:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140219; bh=ZSwI4jjw4GwkwAYOaWT3SF4KhLf9zm4bvh5Jlfu1Q2E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EwO/kDlDiWWnwTYOvc8Wn8BZKybBl8uluDwXw+J8N/hg8bPkRKhsgmNiGp96wemhW pK9lDV47kMS4n9Q9dVw7HhKnYg5h6s+YZTZw9Cfo23iUx0wgHGiMeUmOm/kLmSjmCH EtXXDe6Oyuvv8wTXW2OEfMWsrIiCM0JlEpKMRyoc5ERxfiBW/wZQ31kenSphOApFmF 96QgMhFEQuNq0gGlKvCZauQLWQVQZd9rwHiRQdoUwM2sdD+gY+1H+O44p8J7VokeEt jDbHxzNwUMnuFEm9LfXRfs4n5e9xWmozs7DdkyXijWLGxrhjbp0fX0Rax8A5Y9SpFn wYbMkbB7Vm0Bw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 2/6 v2] sched/fair: Use task_work_queued() on numa_work Date: Sun, 9 Feb 2025 23:30:00 +0100 Message-ID: <20250209223005.11519-3-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the ad-hoc implementation of task_work_queued(). Signed-off-by: Frederic Weisbecker Reviewed-by: Oleg Nesterov Reviewed-by: Valentin Schneider --- kernel/sched/fair.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ce2e94ccad0c..6a2f45821dc0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3303,7 +3303,6 @@ static void task_numa_work(struct callback_head *work) =20 SCHED_WARN_ON(p !=3D container_of(work, struct task_struct, numa_work)); =20 - work->next =3D work; /* * Who cares about NUMA placement when they're dying. * @@ -3551,8 +3550,6 @@ void init_numa_balancing(unsigned long clone_flags, s= truct task_struct *p) p->numa_scan_seq =3D mm ? mm->numa_scan_seq : 0; p->numa_scan_period =3D sysctl_numa_balancing_scan_delay; p->numa_migrate_retry =3D 0; - /* Protect against double add, see task_tick_numa and task_numa_work */ - p->numa_work.next =3D &p->numa_work; p->numa_faults =3D NULL; p->numa_pages_migrated =3D 0; p->total_numa_faults =3D 0; @@ -3593,7 +3590,7 @@ static void task_tick_numa(struct rq *rq, struct task= _struct *curr) /* * We don't care about NUMA placement if we don't have memory. */ - if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || work->next = !=3D work) + if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || task_work_q= ueued(work)) return; =20 /* --=20 2.46.0 From nobody Wed Dec 17 19:17:03 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FA42241130 for ; Sun, 9 Feb 2025 22:30:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140222; cv=none; b=k2NeekwXoc4yE75JXt4D4isjH1Yn0zkqPwZCfRC7C205s5gyglJCx8MVgNVZIbWtQKUuLYlpW9u7GOx3aeJPMeW1SqG/xcN+xaufIY9gkiNSNAc3eCcjNDnShDa17y4oGT4FWvdOqgJJVuuMnvyKkC5wZtkycE55rKgEYayYL4g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140222; c=relaxed/simple; bh=FqNJ6UqFojjpII8sBMn0id8eG6MSR2NFISi/uXoXOtU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uSqrrWB7DpOMBbI6ysDqtj2b3DuQ/cZcquWtH9JuCLpGFOz227oXnJGAyrpSuIBqkEpedegFY3zCQz1xiZS4/VmbqZu6Kf4fAbnfd7t2EKLag7fLRP+gYlc8hdWugRGVbqoFf0+9pzCMG48fIJn/grg7bt2WCN90zFlxWb4aDDo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=i3yc1Tcl; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="i3yc1Tcl" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7DBEFC4CEDF; Sun, 9 Feb 2025 22:30:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140222; bh=FqNJ6UqFojjpII8sBMn0id8eG6MSR2NFISi/uXoXOtU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i3yc1TclMFUBV7la89jwOBPVVcbWq9j9foLUc+l8S0j49dsWRkglTHczvWZ8vSUbr wD8rxcG6eePf4kB/0zJICVTMcSta23HMamwJHE6fo8Fp0v0A/ZkcOUtU9XTzHwlNZy +Gyep4bt1xDGZTNwnAkA2n28MRQnXc6BcSBuVaqslF+JGVWoaypjRynftsYAbPdiKf C7J0qpFDBdeJpw12H1SK8BUhLmUyc5msm80v3jZ7Ryi0H+ejgxjDYhgQGw+HwxKjQo x7TmecEPgKUGc1ZAqYnJ6g+Y9xrKsHJG2JEsdQx1RPMLftIKiFUFxpU+TT64Gd/iyV 6KMEvGDKLmi5Q== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 3/6 v2] sched: Use task_work_queued() on cid_work Date: Sun, 9 Feb 2025 23:30:01 +0100 Message-ID: <20250209223005.11519-4-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the ad-hoc implementation of task_work_queued() Reviewed-by: Valentin Schneider Signed-off-by: Frederic Weisbecker Reviewed-by: Oleg Nesterov --- kernel/sched/core.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 165c90ba64ea..606f596a6e0d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -10538,7 +10538,6 @@ static void task_mm_cid_work(struct callback_head *= work) =20 SCHED_WARN_ON(t !=3D container_of(work, struct task_struct, cid_work)); =20 - work->next =3D work; /* Prevent double-add */ if (t->flags & PF_EXITING) return; mm =3D t->mm; @@ -10582,7 +10581,6 @@ void init_sched_mm_cid(struct task_struct *t) if (mm_users =3D=3D 1) mm->mm_cid_next_scan =3D jiffies + msecs_to_jiffies(MM_CID_SCAN_DELAY); } - t->cid_work.next =3D &t->cid_work; /* Protect against double add */ init_task_work(&t->cid_work, task_mm_cid_work); } =20 @@ -10591,8 +10589,7 @@ void task_tick_mm_cid(struct rq *rq, struct task_st= ruct *curr) struct callback_head *work =3D &curr->cid_work; unsigned long now =3D jiffies; =20 - if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || - work->next !=3D work) + if (!curr->mm || (curr->flags & (PF_EXITING | PF_KTHREAD)) || task_work_q= ueued(work)) return; if (time_before(now, READ_ONCE(curr->mm->mm_cid_next_scan))) return; --=20 2.46.0 From nobody Wed Dec 17 19:17:03 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3A4822F380 for ; Sun, 9 Feb 2025 22:30:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140225; cv=none; b=nQROg/PKDw0GRriqfoUoGBXKPHivKqhbLsU3Kgp2RxGuY5CL2HRwMdLj+Hbwrdn2gV56tHPTUrOhvF6PdVuc8z3MFno5Rl2//FtBrvtvFhwKdlqZ4L6oJ30jLQeoj3AbMl//mMDnwQILu3o5szcWZ+qD5ViVRVmOV/hdpGmar4I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140225; c=relaxed/simple; bh=TXaR3PXscB0IR2zu2ykA6NDQs55O4UuGlAU5yWSXoiE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ve8XutkxwJFQE531ipCFGmKaeguc54NVCpXnfMpB6mQzTIlMcLpBxJXVwcHeDHKKd1QZmfJy72r7mRLzaom7+A+kzFWdIa+8JuVkQo3wJ9eTRYf2GVdBQzWsqvSXhxuUwBvbm/zcPRtyPhb8cpjo6KdunOJuJ87095SMoEu6VrQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GZIidSr/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GZIidSr/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8DE7BC4CEDD; Sun, 9 Feb 2025 22:30:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140225; bh=TXaR3PXscB0IR2zu2ykA6NDQs55O4UuGlAU5yWSXoiE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GZIidSr/P6waR4fDq5bCpnMDJ/9CJnNUWMNJm2VsqNQoj5A+V+IecPyfvvCnFJedQ uWCQySMec2pbxlzt6O3nszier5u2Q+MhW9zH3TeiMwbTiCDWRfw/XwHw3bNvSV07nT jv1IUqSyzNgavbpRXwQVNK3kpXQJlEM2E1O6k7UsmMzeAXDxEdDooxhJtVZdCBp5e3 75BhwkH7VaYwBbSZneZEvagRZTWxuJ1rgjsyXEWTEDxRKmBNuM+GfxL+ON9wYrEroQ jrp4RdT8Zf3ab8qNiYvPZRVUQ/ghWDBWrSO4qEyw8aF8L/Nf9r2hdzQlcn7LgoI4Ih YxlqbBf8xueMw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 4/6 v2] tick/nohz: Move nohz_full related fields out of hot task struct's places Date: Sun, 9 Feb 2025 23:30:02 +0100 Message-ID: <20250209223005.11519-5-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" nohz_full is a feature that only fits into rare and very corner cases. Yet distros enable it by default and therefore the related fields are always reserved in the task struct. Those task fields are stored in the middle of cacheline hot places such as cputime accounting and context switch counting, which doesn't make any sense for a feature that is disabled most of the time. Move the nohz_full storage to colder places. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 9632e3318e0d..10a9aa41b43a 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1088,13 +1088,7 @@ struct task_struct { #endif u64 gtime; struct prev_cputime prev_cputime; -#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN - struct vtime vtime; -#endif =20 -#ifdef CONFIG_NO_HZ_FULL - atomic_t tick_dep_mask; -#endif /* Context switch counts: */ unsigned long nvcsw; unsigned long nivcsw; @@ -1411,6 +1405,14 @@ struct task_struct { struct task_delay_info *delays; #endif =20 +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN + struct vtime vtime; +#endif + +#ifdef CONFIG_NO_HZ_FULL + atomic_t tick_dep_mask; +#endif + #ifdef CONFIG_FAULT_INJECTION int make_it_fail; unsigned int fail_nth; --=20 2.46.0 From nobody Wed Dec 17 19:17:03 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D156C22F380 for ; Sun, 9 Feb 2025 22:30:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140228; cv=none; b=AWdrCEifnpUTaeW+oAzhVbQ5Qdq7q+jtxM3tADXsBXsWzwCydhcgOBjsTf+z0tnqI5FqyTq9kKhkwSnVckVm/AZT5x+L4BBSOdVyf7dV8xisjFM8+f3JrsRu/O0CTmCicPaDRsrt2npfrc4DpjYErBILQX1VTW9+cGxuhWqX0hc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140228; c=relaxed/simple; bh=hpX5Isxts6HZwFzDaFwdkHMytRYO5OiHbV2gKBTOr1s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bUEH2w3zFYVLeGSNi3Q/nCaRPwYl/jt5M++Es774FzSEQ/ArxssP/LYe5rTpI5iFpq1CvXtc4lm5M+27zKd895hQLm0VLdckAfNeGzKycFvZPqNaHOknesj5pFw+RZsYFrkeJTbWpwUW83LKFq8ebRuflYsxXCejItSwkS+33IY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DZCOzR8n; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DZCOzR8n" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 99927C4CEE7; Sun, 9 Feb 2025 22:30:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140228; bh=hpX5Isxts6HZwFzDaFwdkHMytRYO5OiHbV2gKBTOr1s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DZCOzR8nKDpcLHHG+pUeg6XDkn23fPlzAQdzNGIId4ORHo4fyoa7vHVWO+9GkTzkQ rOAL8y9B5yI2fOD/bhpjqte9Ww4uACLQTLAGJ7r1DQtJK++Vu0v3ysxhrN0ae+AgmQ YOyoQmsrC1+06KU7jI2g+OOWGuThvARFCN82GCvxhURurB94W68F854JlfHd+gjI/a 0/Stpzb4hrHV3Z5A+rddkQ5H4Tgmv46ELAZEuc240GNCyulpPTbhMuHJ6crLPOE0lC nHBhpPNlvMIz9X+ja6w5uCnxjlAC7wDmN5gdKhX0MbzYNyOsJ0RJfSxRqIBmfDVKDm lGvlFEzyz5cCw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 5/6 v2] sched/isolation: Introduce isolated task work Date: Sun, 9 Feb 2025 23:30:03 +0100 Message-ID: <20250209223005.11519-6-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some asynchronous kernel work may be pending upon resume to userspace and execute later on. On isolated workload this becomes problematic once the process is done with preparatory work involving syscalls and wants to run in userspace without being interrupted. Provide an infrastructure to queue a work to be executed from the current isolated task context right before resuming to userspace. This goes with the assumption that isolated tasks are pinned to a single nohz_full CPU. Signed-off-by: Frederic Weisbecker --- include/linux/sched.h | 1 + include/linux/sched/isolation.h | 17 +++++++++++++++++ kernel/sched/core.c | 1 + kernel/sched/isolation.c | 31 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 5 files changed, 51 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 10a9aa41b43a..82827f962745 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1410,6 +1410,7 @@ struct task_struct { #endif =20 #ifdef CONFIG_NO_HZ_FULL + struct callback_head nohz_full_work; atomic_t tick_dep_mask; #endif =20 diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolatio= n.h index d8501f4709b5..74da4324b984 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -77,4 +77,21 @@ static inline bool cpu_is_isolated(int cpu) cpuset_cpu_is_isolated(cpu); } =20 +#if defined(CONFIG_NO_HZ_FULL) +extern int __isolated_task_work_queue(void); + +static inline int isolated_task_work_queue(void) +{ + if (!housekeeping_cpu(raw_smp_processor_id(), HK_TYPE_KERNEL_NOISE)) + return -ENOTSUPP; + + return __isolated_task_work_queue(); +} + +extern void isolated_task_work_init(struct task_struct *tsk); +#else +static inline int isolated_task_work_queue(void) { return -ENOTSUPP; } +static inline void isolated_task_work_init(struct task_struct *tsk) { } +#endif /* CONFIG_NO_HZ_FULL */ + #endif /* _LINUX_SCHED_ISOLATION_H */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 606f596a6e0d..78b4b996f85d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4525,6 +4525,7 @@ static void __sched_fork(unsigned long clone_flags, s= truct task_struct *p) p->migration_pending =3D NULL; #endif init_sched_mm_cid(p); + isolated_task_work_init(p); } =20 DEFINE_STATIC_KEY_FALSE(sched_numa_balancing); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 81bc8b329ef1..f25a5cb33c0d 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -249,3 +249,34 @@ static int __init housekeeping_isolcpus_setup(char *st= r) return housekeeping_setup(str, flags); } __setup("isolcpus=3D", housekeeping_isolcpus_setup); + +#if defined(CONFIG_NO_HZ_FULL) +static void isolated_task_work(struct callback_head *head) +{ +} + +int __isolated_task_work_queue(void) +{ + unsigned long flags; + int ret; + + if (current->flags & PF_KTHREAD) + return 0; + + local_irq_save(flags); + if (task_work_queued(¤t->nohz_full_work)) { + ret =3D 0; + goto out; + } + + ret =3D task_work_add(current, ¤t->nohz_full_work, TWA_RESUME); +out: + local_irq_restore(flags); + return ret; +} + +void isolated_task_work_init(struct task_struct *tsk) +{ + init_task_work(&tsk->nohz_full_work, isolated_task_work); +} +#endif /* CONFIG_NO_HZ_FULL */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 38e0e323dda2..f80dc1cad219 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -60,6 +60,7 @@ #include #include #include +#include #include #include #include --=20 2.46.0 From nobody Wed Dec 17 19:17:03 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 915D4244E9A for ; Sun, 9 Feb 2025 22:30:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140231; cv=none; b=D+BKVbDk9FJ0Yc1Sdl5ra206X47Tf9ttqMcHQpOLogrevy/Om6gME/mTp5Yp9guZx+QbDWVrzpivqx0G1tAqVncRX+vlAoEwijiE+ZUiQIUI42kpypG+hHQBUudH8Zvwgvkl3QNbh3cTs6OevABd/o67gnwLrfk8DsnukWS5jRQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739140231; c=relaxed/simple; bh=jJC9e1iQUws78AISrOzytLZgqafLyxgOcIGGwKvAOvw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vdb9qCDsPnzP4sPNuubpUAKzt4+qUWEc+yg+XIZgA2twmLeLyJ0AAbN1FjAharhviudaHYdc0EegLE2yWaPK4uqS3FZcn5N3r1HSsrztFvt8lIBMCRH4JxqfSvGGLJ7cOyKtin0Kq9R+R4rvZwC7QZfXLY+oTU67nqiKYTC31zg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OiznD3kz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OiznD3kz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id ADF1BC4CEDD; Sun, 9 Feb 2025 22:30:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1739140231; bh=jJC9e1iQUws78AISrOzytLZgqafLyxgOcIGGwKvAOvw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OiznD3kzJiFVXddt6RYzafH2Caz7uSr1dddZCrIu/u3/8mczDP7ZEKGEDjOVW/9yB btHfhsaIgyvr6xWqi13Em7VXQZrpIGjktay/irh2ACH0rVTRyVFuM/cwBj9P3YzoS2 HMYFygOMnWMB+fi00pTr3JpElh/O2dNQg8UHryyWUEt5erxMlSLx3vAWejp1Gl10L3 xA5YNFPcDwqlP5fBFm2Bie4loz+E0Sn1Zn5Yqu8KeIyjGU8ahnvAufLyvVx2ooEKYm kmKPw2Hf1FhX5g2//fFphPUfZoIcVTp+CopdTmBHa2lkSa3i/yegVGmp+lerJO4ZiS B3jytGQj0Lgew== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Ingo Molnar , Valentin Schneider , Marcelo Tosatti , Vlastimil Babka , Andrew Morton , Michal Hocko , Thomas Gleixner , Oleg Nesterov , linux-mm@kvack.org Subject: [PATCH 6/6 v2] mm: Drain LRUs upon resume to userspace on nohz_full CPUs Date: Sun, 9 Feb 2025 23:30:04 +0100 Message-ID: <20250209223005.11519-7-frederic@kernel.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250209223005.11519-1-frederic@kernel.org> References: <20250209223005.11519-1-frederic@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" LRUs can be drained through several ways. One of them may add disturbances to isolated workloads while queuing a work at any time to any target, whether running in nohz_full mode or not. Prevent from that on isolated tasks with defering LRUs drains upon resuming to userspace using the isolated task work framework. Signed-off-by: Frederic Weisbecker --- include/linux/swap.h | 1 + kernel/sched/isolation.c | 3 +++ mm/swap.c | 8 +++++++- 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index b13b72645db3..a6fdcc04403e 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -406,6 +406,7 @@ extern void lru_add_drain(void); extern void lru_add_drain_cpu(int cpu); extern void lru_add_drain_cpu_zone(struct zone *zone); extern void lru_add_drain_all(void); +extern void lru_add_and_bh_lrus_drain(void); void folio_deactivate(struct folio *folio); void folio_mark_lazyfree(struct folio *folio); extern void swap_setup(void); diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index f25a5cb33c0d..1f9ec201864c 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -8,6 +8,8 @@ * */ =20 +#include + enum hk_flags { HK_FLAG_DOMAIN =3D BIT(HK_TYPE_DOMAIN), HK_FLAG_MANAGED_IRQ =3D BIT(HK_TYPE_MANAGED_IRQ), @@ -253,6 +255,7 @@ __setup("isolcpus=3D", housekeeping_isolcpus_setup); #if defined(CONFIG_NO_HZ_FULL) static void isolated_task_work(struct callback_head *head) { + lru_add_and_bh_lrus_drain(); } =20 int __isolated_task_work_queue(void) diff --git a/mm/swap.c b/mm/swap.c index fc8281ef4241..da1e569ee3ce 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -37,6 +37,7 @@ #include #include #include +#include =20 #include "internal.h" =20 @@ -376,6 +377,8 @@ static void __lru_cache_activate_folio(struct folio *fo= lio) } =20 local_unlock(&cpu_fbatches.lock); + + isolated_task_work_queue(); } =20 #ifdef CONFIG_LRU_GEN @@ -738,7 +741,7 @@ void lru_add_drain(void) * the same cpu. It shouldn't be a problem in !SMP case since * the core is only one and the locks will disable preemption. */ -static void lru_add_and_bh_lrus_drain(void) +void lru_add_and_bh_lrus_drain(void) { local_lock(&cpu_fbatches.lock); lru_add_drain_cpu(smp_processor_id()); @@ -769,6 +772,9 @@ static bool cpu_needs_drain(unsigned int cpu) { struct cpu_fbatches *fbatches =3D &per_cpu(cpu_fbatches, cpu); =20 + if (!housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE)) + return false; + /* Check these in order of likelihood that they're not zero */ return folio_batch_count(&fbatches->lru_add) || folio_batch_count(&fbatches->lru_move_tail) || --=20 2.46.0