From nobody Sat Sep 26 21:59:09 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14EA839DBD6; Fri, 28 Aug 2026 22:23:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955833; cv=none; b=XmboS/gDRRenXWYfIYS+kdQlEe1aX7SWp091cyr4+nsZCryLtfd+ZfuA7gSvQmjiXEdVhCiG1JMdNvjCHq4n51+xL9AsUam2n+8l7RCc0fcFuVdrUH3YY2RuCUpyAiPxBgNmoDwskneNiw1ergbGKEYnBCrLMGnTIEH3lz9xCDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955833; c=relaxed/simple; bh=8Oi9LnmhOpikFPJVdlsU5Egxb9il/A1QSK2LrPmao+I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=s7YsOdYcwn4Q5akjB6vhHFlE25F418PPMTfz3+BiGivxX8mkq9FvdsOGUbmgb3nncj3Brfx7gwYKb+iei00XT1vkMAc95btZmvSTM31OsjVilqBH6DaCItI07lzAu1g6dqB/XsnHXm/IPFpfLO0ymojFadbFWetAwN5sU/sXsnE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=GIfNWY+6; arc=none smtp.client-ip=192.198.163.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="GIfNWY+6" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1787955831; x=1819491831; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8Oi9LnmhOpikFPJVdlsU5Egxb9il/A1QSK2LrPmao+I=; b=GIfNWY+6Z3cc/fvJyFaTPMDX5gKoW1P5LHqt6ywavo4CPqko3YcR4964 q7WQ44H1sq/41PIXi1jtOCXjLZH8HUcbiRcAhw3upEO6q77Gtdw6O+DHB VqbJbPOy9W9Gnzj+J72LC/SQujtXLCx6oIALXek5QFI3g9rlLGpiDwZrm dAxLz+CF1NP0XliX1APyZnSmQjaI6e86dEiY0vpYBdLspeve00zRg4GOo 9wBY8ha3y5Q63BkDHeGZr4wg7U5r/Iy9HokR3XqcmEA9cpjHHcguQIzVj Kasg0om9+Nr42R3S7WyxOVTQEavxsqcl3T3AbwLrH7UctvAqWbSYP8onQ g==; X-CSE-ConnectionGUID: XUT5HnIbTOa3ktPZVX4F+w== X-CSE-MsgGUID: etDAabgSRbq1RqenosIN6w== X-IronPort-AV: E=McAfee;i="6800,10657,11889"; a="87412282" X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="87412282" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2026 15:23:50 -0700 X-CSE-ConnectionGUID: iR49H/2iS02RAiBq8pUpLQ== X-CSE-MsgGUID: pSqkzZ2rTQiCHxlhSDr/Iw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="264540266" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa010.fm.intel.com with ESMTP; 28 Aug 2026 15:23:49 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar Cc: Tim Chen , Vincent Guittot , Qais Yousef , K Prateek Nayak , Juri Lelli , Dietmar Eggemann , Valentin Schneider , Madadi Vineeth Reddy , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Luo Gengkun , Gavin Guo , Yi Lai , Ricardo Neri , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org Subject: [RFC PATCH 1/7] sched/cache: Decouple sched_cache_group from mm Date: Fri, 28 Aug 2026 15:29:08 -0700 Message-Id: <4f82fe28ecdda1728e3ad29bd44f5287226a8d73.1787955777.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently the sched cache grouping is by mm. But in the future, the grouping could be by cgroup, cookie group, numa_group or others. Rename sched_cache_stat to sched_cache_group and turn it into a refcounted object allocated from mm_struct. The mm_struct now holds a pointer (sched_cache_grp) to this object instead of embedding it. Introduce kernel/sched/cache_sched.c to host the cache aware scheduling helpers and define sched_cache_group_put() there. Meanwhile skip kthreads in account_mm_sched(), consistently with task_tick_cache(). Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen --- include/linux/mm_types.h | 15 ++--- include/linux/sched.h | 8 ++- kernel/exit.c | 6 +- kernel/sched/build_utility.c | 4 ++ kernel/sched/cache_sched.c | 20 +++++++ kernel/sched/fair.c | 110 +++++++++++++++++++++-------------- 6 files changed, 105 insertions(+), 58 deletions(-) create mode 100644 kernel/sched/cache_sched.c diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index b18c2b2e7d2c..2b109cdaaad8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1211,7 +1211,7 @@ struct mm_struct { struct mm_mm_cid mm_cid; =20 /* sched_cache related statistics */ - struct sched_cache_stat sc_stat; + struct sched_cache_group *sched_cache_grp; #ifdef CONFIG_MMU atomic_long_t pgtables_bytes; /* size of all page tables */ #endif @@ -1609,8 +1609,9 @@ static inline unsigned int mm_cid_size(void) #endif /* CONFIG_SCHED_MM_CID */ =20 #ifdef CONFIG_SCHED_CACHE -void mm_init_sched(struct mm_struct *mm, - struct sched_cache_time __percpu *pcpu_sched); +int mm_init_sched(struct mm_struct *mm, + struct sched_cache_time __percpu *pcpu_sched); +void mm_destroy_sched(struct mm_struct *mm); =20 static inline int mm_alloc_sched_noprof(struct mm_struct *mm) { @@ -1620,17 +1621,11 @@ static inline int mm_alloc_sched_noprof(struct mm_s= truct *mm) if (!pcpu_sched) return -ENOMEM; =20 - mm_init_sched(mm, pcpu_sched); - return 0; + return mm_init_sched(mm, pcpu_sched); } =20 #define mm_alloc_sched(...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) =20 -static inline void mm_destroy_sched(struct mm_struct *mm) -{ - free_percpu(mm->sc_stat.pcpu_sched); - mm->sc_stat.pcpu_sched =3D NULL; -} #else /* !CONFIG_SCHED_CACHE */ =20 static inline int mm_alloc_sched(struct mm_struct *mm) { return 0; } diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..1974420e7bf2 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2390,7 +2390,7 @@ struct sched_cache_time { unsigned long epoch; }; =20 -struct sched_cache_stat { +struct sched_cache_group { struct sched_cache_time __percpu *pcpu_sched; raw_spinlock_t lock; unsigned long epoch; @@ -2398,11 +2398,15 @@ struct sched_cache_stat { unsigned long next_scan; unsigned long footprint; int cpu; + refcount_t refcnt; + struct rcu_head rcu; } ____cacheline_aligned_in_smp; =20 +void sched_cache_group_put(struct sched_cache_group *grp); + #else =20 -struct sched_cache_stat { }; +struct sched_cache_group { }; =20 #endif =20 diff --git a/kernel/exit.c b/kernel/exit.c index 2c0b1c02920f..ebe9a6145b35 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -561,12 +561,12 @@ static void exit_mm_sched_cache(struct mm_struct *mm) return; /* * No lock protection due to performance considerations. - * Make sure mm->sc_stat.footprint does not become + * Make sure the group footprint does not become * negative. */ - fp =3D READ_ONCE(mm->sc_stat.footprint); + fp =3D READ_ONCE(mm->sched_cache_grp->footprint); sub =3D min(fp, current->total_numa_faults); - WRITE_ONCE(mm->sc_stat.footprint, fp - sub); + WRITE_ONCE(mm->sched_cache_grp->footprint, fp - sub); } #else static inline void exit_mm_sched_cache(struct mm_struct *mm) diff --git a/kernel/sched/build_utility.c b/kernel/sched/build_utility.c index e2cf3b08d4e9..24202893b262 100644 --- a/kernel/sched/build_utility.c +++ b/kernel/sched/build_utility.c @@ -89,6 +89,10 @@ # include "core_sched.c" #endif =20 +#ifdef CONFIG_SCHED_CACHE +# include "cache_sched.c" +#endif + #ifdef CONFIG_PSI # include "psi.c" #endif diff --git a/kernel/sched/cache_sched.c b/kernel/sched/cache_sched.c new file mode 100644 index 000000000000..d492df55f9d5 --- /dev/null +++ b/kernel/sched/cache_sched.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include "sched.h" + +static void sched_cache_group_free_rcu(struct rcu_head *rcu) +{ + struct sched_cache_group *grp =3D + container_of(rcu, struct sched_cache_group, rcu); + + /* free_percpu() may be called from atomic context. */ + free_percpu(grp->pcpu_sched); + kfree(grp); +} + +void sched_cache_group_put(struct sched_cache_group *grp) +{ + if (!grp || !refcount_dec_and_test(&grp->refcnt)) + return; + + call_rcu(&grp->rcu, sched_cache_group_free_rcu); +} diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1..014a8826ac14 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1450,7 +1450,7 @@ static bool exceed_llc_capacity(struct mm_struct *mm,= int cpu) * excluded. */ llc =3D sd->llc_bytes; - footprint =3D READ_ONCE(mm->sc_stat.footprint); + footprint =3D READ_ONCE(mm->sched_cache_grp->footprint); =20 /* * Scale the LLC size by 256*llc_aggr_tolerance @@ -1495,7 +1495,7 @@ static bool invalid_llc_nr(struct mm_struct *mm, stru= ct task_struct *p, if (scale =3D=3D INT_MAX) return false; =20 - return !fits_capacity((mm->sc_stat.nr_running_avg * cpu_smt_num_threads), + return !fits_capacity((mm->sched_cache_grp->nr_running_avg * cpu_smt_num_= threads), (scale * per_cpu(sd_llc_size, cpu))); } =20 @@ -1572,12 +1572,19 @@ static void account_llc_dequeue(struct rq *rq, stru= ct task_struct *p) } } =20 -void mm_init_sched(struct mm_struct *mm, - struct sched_cache_time __percpu *_pcpu_sched) +int mm_init_sched(struct mm_struct *mm, + struct sched_cache_time __percpu *_pcpu_sched) { + struct sched_cache_group *grp; unsigned long epoch =3D 0; int i; =20 + grp =3D kzalloc_obj(*grp); + if (!grp) { + free_percpu(_pcpu_sched); + return -ENOMEM; + } + for_each_possible_cpu(i) { struct sched_cache_time *pcpu_sched =3D per_cpu_ptr(_pcpu_sched, i); struct rq *rq =3D cpu_rq(i); @@ -1588,18 +1595,28 @@ void mm_init_sched(struct mm_struct *mm, epoch =3D rq->cpu_epoch; } =20 - raw_spin_lock_init(&mm->sc_stat.lock); - mm->sc_stat.epoch =3D epoch; - mm->sc_stat.cpu =3D -1; - mm->sc_stat.next_scan =3D jiffies; - mm->sc_stat.nr_running_avg =3D 0; - mm->sc_stat.footprint =3D 0; + raw_spin_lock_init(&grp->lock); + grp->epoch =3D epoch; + grp->cpu =3D -1; + grp->next_scan =3D jiffies; + grp->nr_running_avg =3D 0; + grp->footprint =3D 0; + refcount_set(&grp->refcnt, 1); + mm->sched_cache_grp =3D grp; /* - * The update to mm->sc_stat should not be reordered - * before initialization to mm's other fields, in case + * The update to grp->pcpu_sched should not be reordered + * before initialization to grp's other fields, in case * the readers may get invalid mm_sched_epoch, etc. */ - smp_store_release(&mm->sc_stat.pcpu_sched, _pcpu_sched); + smp_store_release(&grp->pcpu_sched, _pcpu_sched); + return 0; +} + +void mm_destroy_sched(struct mm_struct *mm) +{ + if (mm->sched_cache_grp) + sched_cache_group_put(mm->sched_cache_grp); + mm->sched_cache_grp =3D NULL; } =20 /* because why would C be fully specified */ @@ -1657,7 +1674,7 @@ static int get_pref_llc(struct task_struct *p, struct= mm_struct *mm) if (!mm) return -1; =20 - mm_sched_cpu =3D READ_ONCE(mm->sc_stat.cpu); + mm_sched_cpu =3D READ_ONCE(mm->sched_cache_grp->cpu); if (mm_sched_cpu !=3D -1) { mm_sched_llc =3D llc_id(mm_sched_cpu); =20 @@ -1700,11 +1717,15 @@ void account_mm_sched(struct rq *rq, struct task_st= ruct *p, s64 delta_exec) /* * init_task, kthreads and user thread created * by user_mode_thread() don't have mm. + * + * A kthread can temporarily adopt an mm via kthread_use_mm(), + * so p->mm alone does not imply a user task. */ - if (!mm || !mm->sc_stat.pcpu_sched) + if (!mm || p->flags & PF_KTHREAD || !mm->sched_cache_grp || + !mm->sched_cache_grp->pcpu_sched) return; =20 - pcpu_sched =3D per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu_of(rq)); + pcpu_sched =3D per_cpu_ptr(mm->sched_cache_grp->pcpu_sched, cpu_of(rq)); =20 scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) { __update_mm_sched(rq, pcpu_sched); @@ -1717,11 +1738,11 @@ void account_mm_sched(struct rq *rq, struct task_st= ruct *p, s64 delta_exec) * If this process hasn't hit task_cache_work() for a while invalidate * its preferred state. */ - if ((long)(epoch - READ_ONCE(mm->sc_stat.epoch)) > llc_epoch_affinity_tim= eout || + if ((long)(epoch - READ_ONCE(mm->sched_cache_grp->epoch)) > llc_epoch_aff= inity_timeout || invalid_llc_nr(mm, p, cpu_of(rq)) || exceed_llc_capacity(mm, cpu_of(rq))) { - if (READ_ONCE(mm->sc_stat.cpu) !=3D -1) - WRITE_ONCE(mm->sc_stat.cpu, -1); + if (READ_ONCE(mm->sched_cache_grp->cpu) !=3D -1) + WRITE_ONCE(mm->sched_cache_grp->cpu, -1); } =20 mm_sched_llc =3D get_pref_llc(p, mm); @@ -1745,19 +1766,19 @@ static void task_tick_cache(struct rq *rq, struct t= ask_struct *p) return; =20 if (!mm || p->flags & PF_KTHREAD || - !mm->sc_stat.pcpu_sched) + !mm->sched_cache_grp->pcpu_sched) return; =20 epoch =3D rq->cpu_epoch; /* avoid moving backwards */ - if (time_after_eq(mm->sc_stat.epoch, epoch)) + if (time_after_eq(mm->sched_cache_grp->epoch, epoch)) return; =20 - guard(raw_spinlock)(&mm->sc_stat.lock); + guard(raw_spinlock)(&mm->sched_cache_grp->lock); =20 if (work->next =3D=3D work) { task_work_add(p, work, TWA_RESUME); - WRITE_ONCE(mm->sc_stat.epoch, epoch); + WRITE_ONCE(mm->sched_cache_grp->epoch, epoch); } } =20 @@ -1769,7 +1790,7 @@ static void get_scan_cpumasks(cpumask_var_t cpus, str= uct task_struct *p) if (!static_branch_likely(&sched_numa_balancing)) goto out; =20 - cpu =3D READ_ONCE(p->mm->sc_stat.cpu); + cpu =3D READ_ONCE(p->mm->sched_cache_grp->cpu); if (cpu !=3D -1) nid =3D cpu_to_node(cpu); curr_cpu =3D task_cpu(p); @@ -1841,12 +1862,12 @@ static void task_cache_work(struct callback_head *w= ork) if (p->flags & PF_EXITING) return; =20 - next_scan =3D READ_ONCE(mm->sc_stat.next_scan); + next_scan =3D READ_ONCE(mm->sched_cache_grp->next_scan); if (time_before(now, next_scan)) return; =20 /* only 1 thread is allowed to scan */ - if (!try_cmpxchg(&mm->sc_stat.next_scan, &next_scan, + if (!try_cmpxchg(&mm->sched_cache_grp->next_scan, &next_scan, now + max_t(unsigned long, READ_ONCE(llc_epoch_period), 1))) return; @@ -1854,8 +1875,8 @@ static void task_cache_work(struct callback_head *wor= k) curr_cpu =3D task_cpu(p); if (invalid_llc_nr(mm, p, curr_cpu) || exceed_llc_capacity(mm, curr_cpu)) { - if (READ_ONCE(mm->sc_stat.cpu) !=3D -1) - WRITE_ONCE(mm->sc_stat.cpu, -1); + if (READ_ONCE(mm->sched_cache_grp->cpu) !=3D -1) + WRITE_ONCE(mm->sched_cache_grp->cpu, -1); =20 return; } @@ -1878,8 +1899,10 @@ static void task_cache_work(struct callback_head *wo= rk) continue; =20 for_each_cpu(i, sched_domain_span(sd)) { + struct sched_cache_group *grp =3D mm->sched_cache_grp; + occ =3D fraction_mm_sched(cpu_rq(i), - per_cpu_ptr(mm->sc_stat.pcpu_sched, i)); + per_cpu_ptr(grp->pcpu_sched, i)); a_occ +=3D occ; if (occ > m_occ) { m_occ =3D occ; @@ -1912,7 +1935,7 @@ static void task_cache_work(struct callback_head *wor= k) m_a_cpu =3D m_cpu; } =20 - if (llc_id(cpu) =3D=3D llc_id(READ_ONCE(mm->sc_stat.cpu))) + if (llc_id(cpu) =3D=3D llc_id(READ_ONCE(mm->sched_cache_grp->cpu))) curr_m_a_occ =3D a_occ; =20 cpumask_andnot(cpus, cpus, sched_domain_span(sd)); @@ -1921,7 +1944,7 @@ static void task_cache_work(struct callback_head *wor= k) =20 if (m_a_occ > (2 * curr_m_a_occ)) { /* - * Avoid switching sc_stat.cpu too fast. + * Avoid switching sched_cache_grp->cpu too fast. * The reason to choose 2X is because: * 1. It is better to keep the preferred LLC stable, * rather than changing it frequently and cause migrations @@ -1930,10 +1953,10 @@ static void task_cache_work(struct callback_head *w= ork) * 3. 2X is chosen based on test results, as it delivers * the optimal performance gain so far. */ - WRITE_ONCE(mm->sc_stat.cpu, m_a_cpu); + WRITE_ONCE(mm->sched_cache_grp->cpu, m_a_cpu); } =20 - update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running); + update_avg_scale(&mm->sched_cache_grp->nr_running_avg, nr_running); free_cpumask_var(cpus); } =20 @@ -3731,18 +3754,19 @@ static void task_numa_placement(struct task_struct = *p) * heuristic and occasional lost updates are tolerable. * * If a task exits, its corresponding footprint must - * be subtracted from the mm->sc_stat.footprint, otherwise - * the mm->sc_stat.footprint will not converge: - * the exiting thread's footprint remains unchanged/undecayed - * in mm->sc_stat.footprint. See exit_mm(). + * be subtracted from the mm->sched_cache_grp->footprint, + * otherwise the mm->sched_cache_grp->footprint will not + * converge: the exiting thread's footprint remains + * unchanged/undecayed in mm->sched_cache_grp->footprint. + * See exit_mm(). * * Lost updates and unsynchronized subtraction * in exit_mm() can cause footprint + diff to * go negative. Clamp to zero to prevent the * unsigned footprint from wrapping. */ - new_fp =3D (long)READ_ONCE(p->mm->sc_stat.footprint) + diff; - WRITE_ONCE(p->mm->sc_stat.footprint, + new_fp =3D (long)READ_ONCE(p->mm->sched_cache_grp->footprint) + diff; + WRITE_ONCE(p->mm->sched_cache_grp->footprint, max(new_fp, 0L)); #endif } @@ -10575,18 +10599,18 @@ static enum llc_mig can_migrate_llc_task(int src_= cpu, int dst_cpu, int cpu; =20 mm =3D p->mm; - if (!mm) + if (!mm || !mm->sched_cache_grp) return mig_unrestricted; =20 - cpu =3D READ_ONCE(mm->sc_stat.cpu); + cpu =3D READ_ONCE(mm->sched_cache_grp->cpu); if (cpu < 0 || cpus_share_cache(src_cpu, dst_cpu)) return mig_unrestricted; =20 /* skip cache aware load balance for too many threads */ if (invalid_llc_nr(mm, p, dst_cpu) || exceed_llc_capacity(mm, dst_cpu)) { - if (READ_ONCE(mm->sc_stat.cpu) !=3D -1) - WRITE_ONCE(mm->sc_stat.cpu, -1); + if (READ_ONCE(mm->sched_cache_grp->cpu) !=3D -1) + WRITE_ONCE(mm->sched_cache_grp->cpu, -1); return mig_unrestricted; } =20 --=20 2.32.0 From nobody Sat Sep 26 21:59:09 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95AD63A9616; Fri, 28 Aug 2026 22:23:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955834; cv=none; b=M8vAVZsJqcE9bf6sN8KunvZF7wyscu+hvmPFYJumbiSIPtTY86jhMGy766aCh/kQFxjt12jEIzVNzqk8kWH4ULKcBVQvfw7IYW3kOLbUOW2D5x/qCUtSPR26CS737rwMYfv6MmZ/AmPjTPoqzdtNP7aYPwv7sOC54aSLPzUjacM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955834; c=relaxed/simple; bh=AmGlCTkEB67QPsfomKOuLYcrViVuuWmPwkLdlIr6ZJ8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e5yzCKVB6Pp57t3CiOTVGr6BFkE9vWhr7Oz6nlQTPikj682CxHOxqEKrtruS37gNGUe5rGIFEG/cJHi584fdOHsKcZivKTdBEHOYOD6b7z+q+ANlIJsrYjPOWtJ3ban4OMVycpFoB8d33YhxNjIFvEUD9OPgCmQK6ZgLw5Snqsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Fh0sbO5v; arc=none smtp.client-ip=192.198.163.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Fh0sbO5v" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1787955832; x=1819491832; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AmGlCTkEB67QPsfomKOuLYcrViVuuWmPwkLdlIr6ZJ8=; b=Fh0sbO5vEeAHNv6OCwnWR4Vyl6HBkMYFC5aE+I/GkgODAvXjZWI3MFPf YyRtR6Bt6HTaoUwaPbmq3JzJAs2vPCR5i9wp0xZutJ9AGwMjAoPwGVhF7 CsjvnW0WLslh7iiJTwvzkm+RZ/5GL/9GdVXrUG+SQ3b90BPEl99cGbenN gqfAAWoJLXPwDyuVvnDToXwpMyW/AXWekaN91/YEMDE+caZqGZ4mD7Y7U z951NcMt13+OzYr1DPPVsjNob/3RE1+fRish5nfIP4DL3lqlyNuhCY9ne m0OeMlh8GWsO8nseLAKflt0x8G28SNzsvS7doWBlpeLspRtEF/2sTbWoY A==; X-CSE-ConnectionGUID: nrWoOJdpTRq7uwY//B4LeA== X-CSE-MsgGUID: PI5Tmg9OT8+ueBU313869w== X-IronPort-AV: E=McAfee;i="6800,10657,11889"; a="87412301" X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="87412301" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2026 15:23:51 -0700 X-CSE-ConnectionGUID: 8xOq6l/LTj6MwA1Qi3eVVA== X-CSE-MsgGUID: gjq9vpz1Tt+WBUEfUjpUJg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="264540270" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa010.fm.intel.com with ESMTP; 28 Aug 2026 15:23:51 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar Cc: Tim Chen , Vincent Guittot , Qais Yousef , K Prateek Nayak , Juri Lelli , Dietmar Eggemann , Valentin Schneider , Madadi Vineeth Reddy , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Luo Gengkun , Gavin Guo , Yi Lai , Ricardo Neri , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org Subject: [RFC PATCH 2/7] sched/cache: Introduce task_struct->sched_cache_grp Date: Fri, 28 Aug 2026 15:29:09 -0700 Message-Id: <865e1e28ee8325d481c681a43401edd34e9a4141.1787955777.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a sched_cache_grp pointer to task_struct so that scheduler code can access the cache group directly via the task, without going through mm->sched_cache_grp. This decouples the scheduler's hot-path accesses from the mm_struct. Each task holds its own refcount on the sched_cache_group, separate from the reference held by its mm_struct. The reference is acquired in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm(). Convert all scheduler code in fair.c and exit.c to use p->sched_cache_grp instead of p->mm->sched_cache_grp. Add sched_cache_group_get() to kernel/sched/cache_sched.c. Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen --- fs/exec.c | 14 ++++ include/linux/sched.h | 3 + kernel/exit.c | 26 +++++-- kernel/fork.c | 23 +++++++ kernel/sched/cache_sched.c | 19 ++++++ kernel/sched/fair.c | 135 ++++++++++++++++++++----------------- kernel/sched/sched.h | 3 + 7 files changed, 157 insertions(+), 66 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index c7b8f2d6366c..a501e2ec84a0 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -880,6 +880,20 @@ static int exec_mmap(struct linux_binprm *bprm) active_mm =3D tsk->active_mm; tsk->active_mm =3D mm; tsk->mm =3D mm; +#ifdef CONFIG_SCHED_CACHE + { + struct sched_cache_group *old_grp, *new_grp; + + old_grp =3D rcu_dereference_protected(tsk->sched_cache_grp, true); + + /* Acquire the reference before publishing the pointer. */ + new_grp =3D sched_cache_group_get(mm->sched_cache_grp); + + rcu_assign_pointer(tsk->sched_cache_grp, new_grp); + if (old_grp) + sched_cache_group_put(old_grp); + } +#endif mm_init_cid(mm, tsk); exec_state =3D task_exec_state_replace(tsk, exec_state); /* diff --git a/include/linux/sched.h b/include/linux/sched.h index 1974420e7bf2..e7253cb332fd 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1422,6 +1422,7 @@ struct task_struct { #ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; int preferred_llc; + struct sched_cache_group __rcu *sched_cache_grp; /* 1: task was enqueued to its preferred LLC, 0 otherwise */ int pref_llc_queued; #endif @@ -2403,6 +2404,8 @@ struct sched_cache_group { } ____cacheline_aligned_in_smp; =20 void sched_cache_group_put(struct sched_cache_group *grp); +struct sched_cache_group *sched_cache_group_get(struct sched_cache_group *= grp); +struct sched_cache_group *task_cache_group_get(struct task_struct *p); =20 #else =20 diff --git a/kernel/exit.c b/kernel/exit.c index ebe9a6145b35..83fa28b3416b 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -553,23 +553,25 @@ void mm_update_next_owner(struct mm_struct *mm) * Subtract the memory footprint of the current task from * mm. */ -static void exit_mm_sched_cache(struct mm_struct *mm) +static void exit_mm_sched_cache(void) { + struct sched_cache_group *grp =3D + rcu_dereference_protected(current->sched_cache_grp, true); unsigned long fp, sub; =20 - if (!current->total_numa_faults) + if (!grp || !current->total_numa_faults) return; /* * No lock protection due to performance considerations. * Make sure the group footprint does not become * negative. */ - fp =3D READ_ONCE(mm->sched_cache_grp->footprint); + fp =3D READ_ONCE(grp->footprint); sub =3D min(fp, current->total_numa_faults); - WRITE_ONCE(mm->sched_cache_grp->footprint, fp - sub); + WRITE_ONCE(grp->footprint, fp - sub); } #else -static inline void exit_mm_sched_cache(struct mm_struct *mm) +static inline void exit_mm_sched_cache(void) { } #endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */ @@ -586,7 +588,19 @@ static void exit_mm(void) if (!mm) return; =20 - exit_mm_sched_cache(mm); + exit_mm_sched_cache(); + +#ifdef CONFIG_SCHED_CACHE + { + struct sched_cache_group *grp =3D + rcu_dereference_protected(current->sched_cache_grp, true); + + rcu_assign_pointer(current->sched_cache_grp, NULL); + + if (grp) + sched_cache_group_put(grp); + } +#endif =20 mmap_read_lock(mm); mmgrab_lazy_tlb(mm); diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5..195b7807ddbb 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1599,6 +1599,19 @@ static int copy_mm(u64 clone_flags, struct task_stru= ct *tsk) =20 tsk->mm =3D mm; tsk->active_mm =3D mm; +#ifdef CONFIG_SCHED_CACHE + { + /* + * A task holds its own reference on the group, separate from + * the reference held by its mm_struct. Acquire it before + * publishing the pointer. + */ + struct sched_cache_group *grp =3D + sched_cache_group_get(mm->sched_cache_grp); + + rcu_assign_pointer(tsk->sched_cache_grp, grp); + } +#endif return 0; } =20 @@ -2581,6 +2594,16 @@ __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_namespaces: exit_nsproxy_namespaces(p); bad_fork_cleanup_mm: +#ifdef CONFIG_SCHED_CACHE + /* + * copy_mm() took a task reference on the cache group; a failed fork + * never reaches exit_mm(), so release it here to avoid leaking the + * group and its per-CPU buffer. + */ + sched_cache_group_put(rcu_dereference_protected(p->sched_cache_grp, true)= ); + RCU_INIT_POINTER(p->sched_cache_grp, NULL); +#endif + if (p->mm) { mm_clear_owner(p->mm, p); mmput(p->mm); diff --git a/kernel/sched/cache_sched.c b/kernel/sched/cache_sched.c index d492df55f9d5..99d07e1e067c 100644 --- a/kernel/sched/cache_sched.c +++ b/kernel/sched/cache_sched.c @@ -1,6 +1,25 @@ // SPDX-License-Identifier: GPL-2.0-only #include "sched.h" =20 +struct sched_cache_group *sched_cache_group_get(struct sched_cache_group *= grp) +{ + /* + * refcount_inc_not_zero() is the acquire primitive for lockless + * (RCU) lookups; plain refcount_inc() would scribble the count if + * it already reached zero. Return NULL in that case. + */ + if (grp && !refcount_inc_not_zero(&grp->refcnt)) + grp =3D NULL; + + return grp; +} + +struct sched_cache_group *task_cache_group_get(struct task_struct *p) +{ + guard(rcu)(); + return sched_cache_group_get(rcu_dereference(p->sched_cache_grp)); +} + static void sched_cache_group_free_rcu(struct rcu_head *rcu) { struct sched_cache_group *grp =3D diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 014a8826ac14..281b4c896bf4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1431,7 +1431,7 @@ static inline int get_sched_cache_scale(int mul) return (1 + (tol - 1) * mul); } =20 -static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) +static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu) { #ifdef CONFIG_NUMA_BALANCING unsigned long llc, footprint; @@ -1450,7 +1450,7 @@ static bool exceed_llc_capacity(struct mm_struct *mm,= int cpu) * excluded. */ llc =3D sd->llc_bytes; - footprint =3D READ_ONCE(mm->sched_cache_grp->footprint); + footprint =3D READ_ONCE(grp->footprint); =20 /* * Scale the LLC size by 256*llc_aggr_tolerance @@ -1479,7 +1479,7 @@ static bool exceed_llc_capacity(struct mm_struct *mm,= int cpu) return false; } =20 -static bool invalid_llc_nr(struct mm_struct *mm, struct task_struct *p, +static bool invalid_llc_nr(struct sched_cache_group *grp, struct task_stru= ct *p, int cpu) { int scale; @@ -1495,7 +1495,7 @@ static bool invalid_llc_nr(struct mm_struct *mm, stru= ct task_struct *p, if (scale =3D=3D INT_MAX) return false; =20 - return !fits_capacity((mm->sched_cache_grp->nr_running_avg * cpu_smt_num_= threads), + return !fits_capacity((grp->nr_running_avg * cpu_smt_num_threads), (scale * per_cpu(sd_llc_size, cpu))); } =20 @@ -1667,14 +1667,14 @@ static unsigned long fraction_mm_sched(struct rq *r= q, return div64_u64(NICE_0_LOAD * pcpu_sched->runtime, rq->cpu_runtime + 1); } =20 -static int get_pref_llc(struct task_struct *p, struct mm_struct *mm) +static int get_pref_llc(struct task_struct *p, struct sched_cache_group *g= rp) { int mm_sched_llc =3D -1, mm_sched_cpu; =20 - if (!mm) + if (!grp) return -1; =20 - mm_sched_cpu =3D READ_ONCE(mm->sched_cache_grp->cpu); + mm_sched_cpu =3D READ_ONCE(grp->cpu); if (mm_sched_cpu !=3D -1) { mm_sched_llc =3D llc_id(mm_sched_cpu); =20 @@ -1704,8 +1704,8 @@ static unsigned int task_running_on_cpu(int cpu, stru= ct task_struct *p); static inline void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) { + struct sched_cache_group *grp =3D rcu_dereference_all(p->sched_cache_grp); struct sched_cache_time *pcpu_sched; - struct mm_struct *mm =3D p->mm; int mm_sched_llc =3D -1; unsigned long epoch; =20 @@ -1716,16 +1716,12 @@ void account_mm_sched(struct rq *rq, struct task_st= ruct *p, s64 delta_exec) return; /* * init_task, kthreads and user thread created - * by user_mode_thread() don't have mm. - * - * A kthread can temporarily adopt an mm via kthread_use_mm(), - * so p->mm alone does not imply a user task. + * by user_mode_thread() don't have a cache group. */ - if (!mm || p->flags & PF_KTHREAD || !mm->sched_cache_grp || - !mm->sched_cache_grp->pcpu_sched) + if (!grp || p->flags & PF_KTHREAD || !grp->pcpu_sched) return; =20 - pcpu_sched =3D per_cpu_ptr(mm->sched_cache_grp->pcpu_sched, cpu_of(rq)); + pcpu_sched =3D per_cpu_ptr(grp->pcpu_sched, cpu_of(rq)); =20 scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) { __update_mm_sched(rq, pcpu_sched); @@ -1738,14 +1734,14 @@ void account_mm_sched(struct rq *rq, struct task_st= ruct *p, s64 delta_exec) * If this process hasn't hit task_cache_work() for a while invalidate * its preferred state. */ - if ((long)(epoch - READ_ONCE(mm->sched_cache_grp->epoch)) > llc_epoch_aff= inity_timeout || - invalid_llc_nr(mm, p, cpu_of(rq)) || - exceed_llc_capacity(mm, cpu_of(rq))) { - if (READ_ONCE(mm->sched_cache_grp->cpu) !=3D -1) - WRITE_ONCE(mm->sched_cache_grp->cpu, -1); + if ((long)(epoch - READ_ONCE(grp->epoch)) > llc_epoch_affinity_timeout || + invalid_llc_nr(grp, p, cpu_of(rq)) || + exceed_llc_capacity(grp, cpu_of(rq))) { + if (READ_ONCE(grp->cpu) !=3D -1) + WRITE_ONCE(grp->cpu, -1); } =20 - mm_sched_llc =3D get_pref_llc(p, mm); + mm_sched_llc =3D get_pref_llc(p, grp); =20 /* task not on rq accounted later in account_entity_enqueue() */ if (task_running_on_cpu(rq->cpu, p) && @@ -1758,31 +1754,32 @@ void account_mm_sched(struct rq *rq, struct task_st= ruct *p, s64 delta_exec) =20 static void task_tick_cache(struct rq *rq, struct task_struct *p) { + struct sched_cache_group *grp =3D rcu_dereference_all(p->sched_cache_grp); struct callback_head *work =3D &p->cache_work; - struct mm_struct *mm =3D p->mm; unsigned long epoch; =20 if (!sched_cache_enabled()) return; =20 - if (!mm || p->flags & PF_KTHREAD || - !mm->sched_cache_grp->pcpu_sched) + if (!grp || p->flags & PF_KTHREAD || + !grp->pcpu_sched) return; =20 epoch =3D rq->cpu_epoch; /* avoid moving backwards */ - if (time_after_eq(mm->sched_cache_grp->epoch, epoch)) + if (time_after_eq(grp->epoch, epoch)) return; =20 - guard(raw_spinlock)(&mm->sched_cache_grp->lock); + guard(raw_spinlock)(&grp->lock); =20 if (work->next =3D=3D work) { task_work_add(p, work, TWA_RESUME); - WRITE_ONCE(mm->sched_cache_grp->epoch, epoch); + WRITE_ONCE(grp->epoch, epoch); } } =20 -static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p) +static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p, + struct sched_cache_group *grp) { #ifdef CONFIG_NUMA_BALANCING int cpu, curr_cpu, nid, pref_nid; @@ -1790,7 +1787,7 @@ static void get_scan_cpumasks(cpumask_var_t cpus, str= uct task_struct *p) if (!static_branch_likely(&sched_numa_balancing)) goto out; =20 - cpu =3D READ_ONCE(p->mm->sched_cache_grp->cpu); + cpu =3D READ_ONCE(grp->cpu); if (cpu !=3D -1) nid =3D cpu_to_node(cpu); curr_cpu =3D task_cpu(p); @@ -1851,9 +1848,7 @@ static void task_cache_work(struct callback_head *wor= k) unsigned long next_scan, now =3D jiffies; struct task_struct *p =3D current, *cur; unsigned long curr_m_a_occ =3D 0; - struct mm_struct *mm =3D p->mm; unsigned long m_a_occ =3D 0; - cpumask_var_t cpus; =20 WARN_ON_ONCE(work !=3D &p->cache_work); =20 @@ -1862,32 +1857,44 @@ static void task_cache_work(struct callback_head *w= ork) if (p->flags & PF_EXITING) return; =20 - next_scan =3D READ_ONCE(mm->sched_cache_grp->next_scan); + /* + * A reference makes sure grp is not released by others. The rcu + * lock can not be held till after zalloc_cpumask_var() below, + * because the latter might sleep. + */ + struct sched_cache_group *grp __free(sched_cache_group_put) =3D + task_cache_group_get(p); + if (!grp) + return; + + next_scan =3D READ_ONCE(grp->next_scan); if (time_before(now, next_scan)) return; =20 /* only 1 thread is allowed to scan */ - if (!try_cmpxchg(&mm->sched_cache_grp->next_scan, &next_scan, + if (!try_cmpxchg(&grp->next_scan, &next_scan, now + max_t(unsigned long, READ_ONCE(llc_epoch_period), 1))) return; =20 curr_cpu =3D task_cpu(p); - if (invalid_llc_nr(mm, p, curr_cpu) || - exceed_llc_capacity(mm, curr_cpu)) { - if (READ_ONCE(mm->sched_cache_grp->cpu) !=3D -1) - WRITE_ONCE(mm->sched_cache_grp->cpu, -1); + if (invalid_llc_nr(grp, p, curr_cpu) || + exceed_llc_capacity(grp, curr_cpu)) { + if (READ_ONCE(grp->cpu) !=3D -1) + WRITE_ONCE(grp->cpu, -1); =20 return; } =20 + cpumask_var_t cpus __free(free_cpumask_var) =3D CPUMASK_VAR_NULL; + if (!zalloc_cpumask_var(&cpus, GFP_KERNEL)) return; =20 scoped_guard (cpus_read_lock) { guard(rcu)(); =20 - get_scan_cpumasks(cpus, p); + get_scan_cpumasks(cpus, p, grp); =20 for_each_cpu(cpu, cpus) { /* XXX sched_cluster_active */ @@ -1899,8 +1906,6 @@ static void task_cache_work(struct callback_head *wor= k) continue; =20 for_each_cpu(i, sched_domain_span(sd)) { - struct sched_cache_group *grp =3D mm->sched_cache_grp; - occ =3D fraction_mm_sched(cpu_rq(i), per_cpu_ptr(grp->pcpu_sched, i)); a_occ +=3D occ; @@ -1909,9 +1914,13 @@ static void task_cache_work(struct callback_head *wo= rk) m_cpu =3D i; } =20 + /* + * rcu_access_pointer() is used because the + * pointer is only compared, never dereferenced. + */ cur =3D rcu_dereference_all(cpu_rq(i)->curr); if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) && - cur->mm =3D=3D mm) + rcu_access_pointer(cur->sched_cache_grp) =3D=3D grp) nr_running++; } =20 @@ -1935,7 +1944,7 @@ static void task_cache_work(struct callback_head *wor= k) m_a_cpu =3D m_cpu; } =20 - if (llc_id(cpu) =3D=3D llc_id(READ_ONCE(mm->sched_cache_grp->cpu))) + if (llc_id(cpu) =3D=3D llc_id(READ_ONCE(grp->cpu))) curr_m_a_occ =3D a_occ; =20 cpumask_andnot(cpus, cpus, sched_domain_span(sd)); @@ -1953,11 +1962,10 @@ static void task_cache_work(struct callback_head *w= ork) * 3. 2X is chosen based on test results, as it delivers * the optimal performance gain so far. */ - WRITE_ONCE(mm->sched_cache_grp->cpu, m_a_cpu); + WRITE_ONCE(grp->cpu, m_a_cpu); } =20 - update_avg_scale(&mm->sched_cache_grp->nr_running_avg, nr_running); - free_cpumask_var(cpus); + update_avg_scale(&grp->nr_running_avg, nr_running); } =20 void init_sched_mm(struct task_struct *p) @@ -3754,10 +3762,9 @@ static void task_numa_placement(struct task_struct *= p) * heuristic and occasional lost updates are tolerable. * * If a task exits, its corresponding footprint must - * be subtracted from the mm->sched_cache_grp->footprint, - * otherwise the mm->sched_cache_grp->footprint will not - * converge: the exiting thread's footprint remains - * unchanged/undecayed in mm->sched_cache_grp->footprint. + * be subtracted from p->sched_cache_grp->footprint, + * otherwise the footprint will not converge: the + * exiting thread's footprint remains unchanged/undecayed. * See exit_mm(). * * Lost updates and unsynchronized subtraction @@ -3765,9 +3772,17 @@ static void task_numa_placement(struct task_struct *= p) * go negative. Clamp to zero to prevent the * unsigned footprint from wrapping. */ - new_fp =3D (long)READ_ONCE(p->mm->sched_cache_grp->footprint) + diff; - WRITE_ONCE(p->mm->sched_cache_grp->footprint, - max(new_fp, 0L)); + { + struct sched_cache_group *grp; + + guard(rcu)(); + grp =3D rcu_dereference(p->sched_cache_grp); + + if (grp) { + new_fp =3D (long)READ_ONCE(grp->footprint) + diff; + WRITE_ONCE(grp->footprint, max(new_fp, 0L)); + } + } #endif } =20 @@ -10594,23 +10609,23 @@ static enum llc_mig can_migrate_llc(int src_cpu, = int dst_cpu, static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu, struct task_struct *p) { - struct mm_struct *mm; + struct sched_cache_group *grp; bool to_pref; int cpu; =20 - mm =3D p->mm; - if (!mm || !mm->sched_cache_grp) + grp =3D rcu_dereference_all(p->sched_cache_grp); + if (!grp) return mig_unrestricted; =20 - cpu =3D READ_ONCE(mm->sched_cache_grp->cpu); + cpu =3D READ_ONCE(grp->cpu); if (cpu < 0 || cpus_share_cache(src_cpu, dst_cpu)) return mig_unrestricted; =20 /* skip cache aware load balance for too many threads */ - if (invalid_llc_nr(mm, p, dst_cpu) || - exceed_llc_capacity(mm, dst_cpu)) { - if (READ_ONCE(mm->sched_cache_grp->cpu) !=3D -1) - WRITE_ONCE(mm->sched_cache_grp->cpu, -1); + if (invalid_llc_nr(grp, p, dst_cpu) || + exceed_llc_capacity(grp, dst_cpu)) { + if (READ_ONCE(grp->cpu) !=3D -1) + WRITE_ONCE(grp->cpu, -1); return mig_unrestricted; } =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56acf502ba26..7dbe070979d7 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -4106,6 +4106,9 @@ static inline bool sched_cache_enabled(void) return static_branch_unlikely(&sched_cache_active); } =20 +DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *, + sched_cache_group_put(_T)); + extern void sched_cache_active_set(void); =20 #endif --=20 2.32.0 From nobody Sat Sep 26 21:59:09 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CAFB3AA504; Fri, 28 Aug 2026 22:23:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955839; cv=none; b=PBCqxoZtClyqj/NhMSPFzbH01nvUdLhMJknf4Uk7dhYd0O5ji67I7h1wZolzj5vZQ/AlUs0K+TobLCpR+OvzW/GKkkhojq+QEmUqjg7mh3lwzH4TCJiG2XcaQikerEJDfx8ytjuo90bJoMHKHF2cEQdg/9JYi63QVt6CPvAbP10= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955839; c=relaxed/simple; bh=mGdivKR88nJONRsPiRnL4DiGPu+1f4won+EAVrdUSTY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RdfecNMktxRvRgT0PQAzY0qhBQ2r0XvfI+fqloyY3t4PdNbhzv9LRsIo1s9N6TwQqCSipS6/qJyMI5HzZAn94h4G6SuMZvxywlfGIBSmOGux4ar0VfWkI8bHFBW+vG42/j4PijnKseDVQ11668rsGVnQ3JX6HJmJFheEmnMcLIM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=PanimF4w; arc=none smtp.client-ip=192.198.163.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="PanimF4w" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1787955834; x=1819491834; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mGdivKR88nJONRsPiRnL4DiGPu+1f4won+EAVrdUSTY=; b=PanimF4wEbRu86u1sA/YFcxsGGp8V5aZZvtpeEHRwiRQjCyou6ka2XrE nvbve15tugYJS4CMM8lC1qHUCJnPONH3iCQDjKilMYsYSOp3JdNTIBdqP Rr1kg4PihvWVdJ00TXaTE0fPNvHW9dW9I/dackZGkgobkNLk4NF5S+tsJ D8qphlzwxEvfEesyXO1TfN92YtxUl9LgmttBdXdKlAASZrQmZVOrZtp43 MkkmGZoXEyaimfsM30s7Kcf3LQn0caqTSSsY0ZLoyYJUJQurzUuu2p2sP /UstlwnKOTLGbDEs9dzj9v8uDfrlA6hGCoaOVTa8ZIdfAYDok30BT5z/3 A==; X-CSE-ConnectionGUID: JAwvw9jLRGm+chJ56eYKeA== X-CSE-MsgGUID: F93bPWNrQ0ynqfjmxVVGuw== X-IronPort-AV: E=McAfee;i="6800,10657,11889"; a="87412321" X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="87412321" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2026 15:23:53 -0700 X-CSE-ConnectionGUID: /VlTN/ymR9CI6AehwruSiA== X-CSE-MsgGUID: Ey9yUyPYTliV5E7/NDGYWg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="264540274" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa010.fm.intel.com with ESMTP; 28 Aug 2026 15:23:52 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar Cc: Tim Chen , Vincent Guittot , Qais Yousef , K Prateek Nayak , Juri Lelli , Dietmar Eggemann , Valentin Schneider , Madadi Vineeth Reddy , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Luo Gengkun , Gavin Guo , Yi Lai , Ricardo Neri , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org Subject: [RFC PATCH 3/7] sched/cache: Extract sched_cache_alloc_group() helper Date: Fri, 28 Aug 2026 15:29:10 -0700 Message-Id: X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract sched_cache_alloc_group() into kernel/sched/cache_sched.c as a shared helper. Convert mm_init_sched() to use the new helper instead of open-coding the allocation and initialization. This prepares for a second caller in the upcoming prctl CREATE path. No functional change. Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen --- include/linux/sched.h | 2 ++ kernel/sched/cache_sched.c | 46 ++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 32 ++------------------------ 3 files changed, 50 insertions(+), 30 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index e7253cb332fd..e25347aa2cc5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2406,6 +2406,8 @@ struct sched_cache_group { void sched_cache_group_put(struct sched_cache_group *grp); struct sched_cache_group *sched_cache_group_get(struct sched_cache_group *= grp); struct sched_cache_group *task_cache_group_get(struct task_struct *p); +struct sched_cache_group * +sched_cache_alloc_group(struct sched_cache_time __percpu *pcpu_sched); =20 #else =20 diff --git a/kernel/sched/cache_sched.c b/kernel/sched/cache_sched.c index 99d07e1e067c..860209a47931 100644 --- a/kernel/sched/cache_sched.c +++ b/kernel/sched/cache_sched.c @@ -37,3 +37,49 @@ void sched_cache_group_put(struct sched_cache_group *grp) =20 call_rcu(&grp->rcu, sched_cache_group_free_rcu); } + +static void sched_cache_group_init(struct sched_cache_group *grp, + struct sched_cache_time __percpu *_pcpu_sched) +{ + unsigned long epoch =3D 0; + int i; + + for_each_possible_cpu(i) { + struct sched_cache_time *pcpu_sched =3D per_cpu_ptr(_pcpu_sched, i); + struct rq *rq =3D cpu_rq(i); + + pcpu_sched->runtime =3D 0; + /* a slightly stale cpu epoch is acceptible */ + pcpu_sched->epoch =3D rq->cpu_epoch; + epoch =3D rq->cpu_epoch; + } + + raw_spin_lock_init(&grp->lock); + grp->epoch =3D epoch; + grp->cpu =3D -1; + grp->next_scan =3D jiffies; + grp->nr_running_avg =3D 0; + grp->footprint =3D 0; + refcount_set(&grp->refcnt, 1); + /* + * The update to grp->pcpu_sched should not be reordered + * before initialization to grp's other fields, in case + * the readers may get invalid mm_sched_epoch, etc. + */ + smp_store_release(&grp->pcpu_sched, _pcpu_sched); +} + +struct sched_cache_group * +sched_cache_alloc_group(struct sched_cache_time __percpu *_pcpu_sched) +{ + struct sched_cache_group *grp; + + grp =3D kzalloc_obj(*grp); + if (!grp) { + free_percpu(_pcpu_sched); + return NULL; + } + + sched_cache_group_init(grp, _pcpu_sched); + return grp; +} diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 281b4c896bf4..be1f3568c3aa 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1575,40 +1575,12 @@ static void account_llc_dequeue(struct rq *rq, stru= ct task_struct *p) int mm_init_sched(struct mm_struct *mm, struct sched_cache_time __percpu *_pcpu_sched) { - struct sched_cache_group *grp; - unsigned long epoch =3D 0; - int i; + struct sched_cache_group *grp =3D sched_cache_alloc_group(_pcpu_sched); =20 - grp =3D kzalloc_obj(*grp); - if (!grp) { - free_percpu(_pcpu_sched); + if (!grp) return -ENOMEM; - } - - for_each_possible_cpu(i) { - struct sched_cache_time *pcpu_sched =3D per_cpu_ptr(_pcpu_sched, i); - struct rq *rq =3D cpu_rq(i); - - pcpu_sched->runtime =3D 0; - /* a slightly stale cpu epoch is acceptible */ - pcpu_sched->epoch =3D rq->cpu_epoch; - epoch =3D rq->cpu_epoch; - } =20 - raw_spin_lock_init(&grp->lock); - grp->epoch =3D epoch; - grp->cpu =3D -1; - grp->next_scan =3D jiffies; - grp->nr_running_avg =3D 0; - grp->footprint =3D 0; - refcount_set(&grp->refcnt, 1); mm->sched_cache_grp =3D grp; - /* - * The update to grp->pcpu_sched should not be reordered - * before initialization to grp's other fields, in case - * the readers may get invalid mm_sched_epoch, etc. - */ - smp_store_release(&grp->pcpu_sched, _pcpu_sched); return 0; } =20 --=20 2.32.0 From nobody Sat Sep 26 21:59:09 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 575C53A9D84; Fri, 28 Aug 2026 22:23:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955840; cv=none; b=LB2ChQiacZTH+Uvu1XWlMjXxWYrBhTeOaSB0Heb4BO5uBU0+h3lzZIctn2y8lB4As4LtEl3vp2l9o+PKkPbkIjTmAMCuFBjqbFBwGuh/ZY/fkaTxH8CgOWS5DedhJ5/TVkTQWEiTiL7iS9nlRhTUiHBKtP3RDyiOGoafYn8enmI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955840; c=relaxed/simple; bh=ZWqheJd1NEAZN8PnzaImedNHmcoUsnwyF5Cxa3W7GQY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RNahqh0Dxpl8zqai46EfHXdBowu6OFUc9gz8EgVmrbcSYxfERQxXryR+JJqUfHWbKfoKkqICnZ0tEiK25ee8sXlxnbZf4cDjd7WRx7wU7MTWbHL0eZW4HwOZ4lOPWDBzon9nwiC+qBBv8nIoVKUPFPBDCfG50TAQQXbXq66KlS0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=O5gjeQ22; arc=none smtp.client-ip=192.198.163.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="O5gjeQ22" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1787955835; x=1819491835; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZWqheJd1NEAZN8PnzaImedNHmcoUsnwyF5Cxa3W7GQY=; b=O5gjeQ2209lTnQHGep1xBqq1/JqPyF63B/lB5Zh+gaPJwQV/GomyIzXV h76CeQBOx5biDVswcIZ+6nhgeHJ3g2u2VvmGK1TAcEaoChZpomeIBirdQ dhetcHKdjTBp/gMCTo/ZH/LvLGiXqkKuy6NgT0sTEJLTLa43BUoI9s32L IqqAzDDipT7+EkKfSs48y42vUxlfTszR74mWxuLNDTzxVfSwXLT8cYzA2 YBrKTOqVt4EODiapaucbk1Ai1FAs4mbOdLmniYLUV4x02ZxCV3rQCLDHP 5NQ36WQNwpFuvYSo4lxyZVcYFe3vCbwgiUvrfo6y/fedsNma3XG5QwRaA w==; X-CSE-ConnectionGUID: p/f/T1h3TuGPCI72V/kVmw== X-CSE-MsgGUID: hbqu2CDeQNyrRk423dOVDQ== X-IronPort-AV: E=McAfee;i="6800,10657,11889"; a="87412343" X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="87412343" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2026 15:23:54 -0700 X-CSE-ConnectionGUID: ZszoZ9BlRk6RNbzAJMi23Q== X-CSE-MsgGUID: /Q6ss92zSOy48RZSb+67BQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="264540277" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa010.fm.intel.com with ESMTP; 28 Aug 2026 15:23:54 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar Cc: Tim Chen , Vincent Guittot , Qais Yousef , K Prateek Nayak , Juri Lelli , Dietmar Eggemann , Valentin Schneider , Madadi Vineeth Reddy , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Luo Gengkun , Gavin Guo , Yi Lai , Ricardo Neri , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org Subject: [RFC PATCH 4/7] sched/cache: Add prctl to manage per process cache scheduling groups Date: Fri, 28 Aug 2026 15:29:11 -0700 Message-Id: <50fe2db1a62ea2376a87d0c14778b1ff456d11ec.1787955777.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Derived from core scheduling's prctl interface. Implement the controls for cache aware scheduling on a process: int prctl(PR_SCHED_CACHE, unsigned long subop, pid_t pid, unsigned long arg4, unsigned long type); pid argument: the PID of the task the operation applies to, that is the destination of the group assignment. 0 means "the calling task." pid_type : PIDTYPE_PID targets the single thread, PIDTYPE_TGID the whole thread group and PIDTYPE_PGID the process group of that task. PR_SCHED_CACHE_GET only accepts PIDTYPE_PID. arg4 argument: the address to store the cookie for PR_SCHED_CACHE_GET, the pid to take the group from for PR_SCHED_CACHE_SHARE_FROM, where 0 means "the calling task". The second argument (subop) selects one of the following operations: 1. PR_SCHED_CACHE_GET Retrieve the cache aware scheduling cookie of a task. u64 cookie; prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, pid, (unsigned long)&cookie, PIDTYPE_PID); 2. PR_SCHED_CACHE_CREATE Create a new cache scheduling group and assign it to the target pid, prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_CREATE, pid, 0, PIDTYPE_TGID); 3. PR_SCHED_CACHE_SHARE_FROM Copy the group of the task given in arg4 to the target task. /* pull 1234's group onto the caller */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, 0, 1234, PIDTYPE_PID); /* push the caller's group onto the thread group of 1234 */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, 1234, 0, PIDTYPE_TGID); /* let the thread group of 1234 share 5678's group */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, 1234, 5678, PIDTYPE_TGID= ); Unlike the core scheduling's PR_SCHED_CORE_SHARE_FROM and PR_SCHED_CORE_SHARE_TO, there is no need to involve the current task: a daemon(scheqos, eg) can handle two tasks without joining their group, which would pollute the group's per LLC occupancy statistics. Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen --- fs/exec.c | 18 ++- include/linux/sched.h | 4 + include/uapi/linux/prctl.h | 7 + kernel/exit.c | 13 +- kernel/fork.c | 25 +++- kernel/sched/cache_sched.c | 274 +++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 8 +- kernel/sys.c | 5 + 8 files changed, 341 insertions(+), 13 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index a501e2ec84a0..4448ea123481 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -884,12 +884,22 @@ static int exec_mmap(struct linux_binprm *bprm) { struct sched_cache_group *old_grp, *new_grp; =20 - old_grp =3D rcu_dereference_protected(tsk->sched_cache_grp, true); - - /* Acquire the reference before publishing the pointer. */ + /* + * Acquire the reference before publishing the pointer: once + * tsk->sched_cache_grp is visible, a concurrent + * prctl(PR_SCHED_CACHE) writer may pick the group up as its + * old_grp and drop a reference we have not taken yet. + * + * pi_lock serializes the exchange against such a writer. IRQs + * are already disabled here, so a plain raw_spin_lock() + * suffices. + */ new_grp =3D sched_cache_group_get(mm->sched_cache_grp); =20 - rcu_assign_pointer(tsk->sched_cache_grp, new_grp); + raw_spin_lock(&tsk->pi_lock); + old_grp =3D sched_cache_grp_replace(tsk, new_grp); + raw_spin_unlock(&tsk->pi_lock); + if (old_grp) sched_cache_group_put(old_grp); } diff --git a/include/linux/sched.h b/include/linux/sched.h index e25347aa2cc5..79f0079c3aa1 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2407,7 +2407,11 @@ void sched_cache_group_put(struct sched_cache_group = *grp); struct sched_cache_group *sched_cache_group_get(struct sched_cache_group *= grp); struct sched_cache_group *task_cache_group_get(struct task_struct *p); struct sched_cache_group * +sched_cache_grp_replace(struct task_struct *p, struct sched_cache_group *g= rp); +struct sched_cache_group * sched_cache_alloc_group(struct sched_cache_time __percpu *pcpu_sched); +int sched_cache_prctl(int option, unsigned long arg2, unsigned long arg3, + unsigned long arg4, unsigned long arg5); =20 #else =20 diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index b6ec6f693719..fed7bb028f9a 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -416,4 +416,11 @@ struct prctl_mm_map { # define PR_CFI_DISABLE _BITUL(1) # define PR_CFI_LOCK _BITUL(2) =20 +/* Cache-aware scheduling */ +#define PR_SCHED_CACHE 82 +# define PR_SCHED_CACHE_GET 0 +# define PR_SCHED_CACHE_CREATE 1 +# define PR_SCHED_CACHE_SHARE_FROM 2 +# define PR_SCHED_CACHE_MAX 3 + #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/exit.c b/kernel/exit.c index 83fa28b3416b..d140429046d2 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -592,10 +592,17 @@ static void exit_mm(void) =20 #ifdef CONFIG_SCHED_CACHE { - struct sched_cache_group *grp =3D - rcu_dereference_protected(current->sched_cache_grp, true); + struct sched_cache_group *grp; + unsigned long flags; =20 - rcu_assign_pointer(current->sched_cache_grp, NULL); + /* + * pi_lock serializes the clear against a concurrent + * prctl(PR_SCHED_CACHE) writer targeting this task, so the + * reference is dropped exactly once. + */ + raw_spin_lock_irqsave(¤t->pi_lock, flags); + grp =3D sched_cache_grp_replace(current, NULL); + raw_spin_unlock_irqrestore(¤t->pi_lock, flags); =20 if (grp) sched_cache_group_put(grp); diff --git a/kernel/fork.c b/kernel/fork.c index 195b7807ddbb..9399f5e2070e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1601,13 +1601,28 @@ static int copy_mm(u64 clone_flags, struct task_str= uct *tsk) tsk->active_mm =3D mm; #ifdef CONFIG_SCHED_CACHE { + struct sched_cache_group *grp; + /* - * A task holds its own reference on the group, separate from - * the reference held by its mm_struct. Acquire it before - * publishing the pointer. + * For CLONE_VM (threads): inherit the parent's + * sched_cache_grp, which may be a cookie-based group different + * from mm->sched_cache_grp. For kernel thread, don't assign + * it a sched cache group. + * + * For new processes (own mm): use the new mm's group. The mm + * has just been created and is not visible to anyone else, so + * there is no race against a concurrent writer and the mm's + * own reference keeps the group alive. Take the refcount + * directly, without RCU. */ - struct sched_cache_group *grp =3D - sched_cache_group_get(mm->sched_cache_grp); + if (clone_flags & CLONE_VM) { + if (tsk->flags & PF_KTHREAD) + grp =3D NULL; + else + grp =3D task_cache_group_get(current); + } else { + grp =3D sched_cache_group_get(mm->sched_cache_grp); + } =20 rcu_assign_pointer(tsk->sched_cache_grp, grp); } diff --git a/kernel/sched/cache_sched.c b/kernel/sched/cache_sched.c index 860209a47931..d1932f0c5ee8 100644 --- a/kernel/sched/cache_sched.c +++ b/kernel/sched/cache_sched.c @@ -83,3 +83,277 @@ sched_cache_alloc_group(struct sched_cache_time __percp= u *_pcpu_sched) sched_cache_group_init(grp, _pcpu_sched); return grp; } + +static struct sched_cache_group *sched_cache_alloc_all(void) +{ + struct sched_cache_time __percpu *pcpu_sched; + + pcpu_sched =3D alloc_percpu(struct sched_cache_time); + if (!pcpu_sched) + return NULL; + + return sched_cache_alloc_group(pcpu_sched); +} + +#ifdef CONFIG_NUMA_BALANCING +/* + * When prctl() moves a task between groups, move its footprint estimate t= oo: + * otherwise the group it leaves over-estimates its footprint forever, and= the + * group it joins under-estimates it once the task exits. + */ +static void sched_cache_xfer_footprint(struct task_struct *p, + struct sched_cache_group *old, + struct sched_cache_group *new) +{ + unsigned long fp, sub; + + if (!old || !new || old =3D=3D new) + return; + + sub =3D READ_ONCE(p->total_numa_faults); + if (!sub) + return; + + fp =3D READ_ONCE(old->footprint); + sub =3D min(fp, sub); + WRITE_ONCE(old->footprint, fp - sub); + + fp =3D READ_ONCE(new->footprint); + WRITE_ONCE(new->footprint, fp + sub); +} +#else +static inline void sched_cache_xfer_footprint(struct task_struct *p, + struct sched_cache_group *old, + struct sched_cache_group *new) +{ +} +#endif /* CONFIG_NUMA_BALANCING */ + +/* Swap a task's cache group pointer and return the previous one. */ +struct sched_cache_group * +sched_cache_grp_replace(struct task_struct *p, struct sched_cache_group *g= rp) +{ + struct sched_cache_group *old; + + lockdep_assert_held(&p->pi_lock); + old =3D rcu_dereference_protected(p->sched_cache_grp, + lockdep_is_held(&p->pi_lock)); + rcu_assign_pointer(p->sched_cache_grp, grp); + + return old; +} + +static void __sched_cache_set(struct task_struct *p, + struct sched_cache_group *grp) +{ + struct sched_cache_group *old_grp; + unsigned long flags; + + grp =3D sched_cache_group_get(grp); + + /* + * p->pi_lock serializes the exchange against concurrent writers + * (other prctl callers as well as exec_mmap()/exit_mm()). Without + * it two writers could fetch the same old pointer and each drop a + * reference, over-decrementing the refcount. + */ + raw_spin_lock_irqsave(&p->pi_lock, flags); + + /* + * Avoid increasing the refcount for an exiting task, + * otherwise the grp can not be put after the task exits, + * and cause memory leak: + * CPU0 (prctl) CPU1 (task exiting) + * ---- ---- + * exit_mm() + * put(p->sched_cache_grp) + * free(old_grp) + * + * __sched_cache_set(p, new_grp) + * + * free_task() + * free(p) + * the "freed" p is unable to put new_grp + * + * PF_EXITING is set in exit_signals() without p->pi_lock held, so this + * test on its own would only narrow the window. What closes it is that + * exit_mm() takes p->pi_lock to clear the pointer, and exit_signals() + * runs before exit_mm(). So of the two possible orders: + * + * - we get pi_lock before exit_mm() does: we install new_grp and its + * reference, and exit_mm() subsequently drops it, because it puts + * whatever sched_cache_grp_replace() hands back. + * + * - we get pi_lock after exit_mm() released it: the lock's release/ + * acquire ordering makes the PF_EXITING store visible to us, so the + * test below fires and we install nothing. + * + * Either way the reference is dropped exactly once. The test is thus + * about not leaving a reference behind on a task that is already past + * exit_mm(), not about racing with the PF_EXITING store itself. + */ + if (p->flags & PF_EXITING) { + raw_spin_unlock_irqrestore(&p->pi_lock, flags); + if (grp) + sched_cache_group_put(grp); + return; + } + + old_grp =3D sched_cache_grp_replace(p, grp); + raw_spin_unlock_irqrestore(&p->pi_lock, flags); + + /* Carry this task's footprint estimate to the group it just joined. */ + sched_cache_xfer_footprint(p, old_grp, grp); + + if (old_grp) + sched_cache_group_put(old_grp); +} + +static struct task_struct *sched_cache_find_get_task(unsigned long vpid) +{ + struct task_struct *p; + + guard(rcu)(); + p =3D vpid ? find_task_by_vpid(vpid) : current; + if (p) + get_task_struct(p); + + return p; +} + +/* + * arg2: subcommand, + * arg3: destination pid, SHARE_FROM copies the group of arg4 to arg3 + * arg4: source pid, or cookie out ptr for GET + * arg5: pid type, the scope of the destination + * + */ +int sched_cache_prctl(int option, unsigned long arg2, unsigned long arg3, + unsigned long arg4, unsigned long arg5) +{ + struct task_struct *dst =3D NULL, *src =3D NULL, *p; + struct sched_cache_group *grp =3D NULL; + enum pid_type type =3D arg5; + struct pid *pid_grp; + int err =3D 0; + + if (arg2 >=3D PR_SCHED_CACHE_MAX || arg3 > INT_MAX || + arg5 > PIDTYPE_PGID) + return -EINVAL; + + /* only GET and SHARE_FROM take a 4th argument */ + if (arg4 && arg2 !=3D PR_SCHED_CACHE_GET && + arg2 !=3D PR_SCHED_CACHE_SHARE_FROM) + return -EINVAL; + + dst =3D sched_cache_find_get_task(arg3); + if (!dst) + return -ESRCH; + + if (dst->flags & PF_KTHREAD) { + err =3D -EINVAL; + goto out_task; + } + + if (!ptrace_may_access(dst, PTRACE_MODE_READ_REALCREDS)) { + err =3D -EPERM; + goto out_task; + } + + switch (arg2) { + case PR_SCHED_CACHE_GET: { + unsigned long id =3D 0; + + if (type !=3D PIDTYPE_PID || arg4 & 7) { + err =3D -EINVAL; + goto out_task; + } + + grp =3D task_cache_group_get(dst); + if (grp) + ptr_to_hashval((void *)grp, &id); + + if (arg4) + err =3D put_user((u64)id, (u64 __user *)arg4); + + goto out_group; + } + + case PR_SCHED_CACHE_CREATE: + /* + * Allocator owns ref 1, __sched_cache_set() acquires ref 2. + * The sched_cache_group_put() at out_group: drops ref 1, leaving + * ref 1 held by the task. + */ + grp =3D sched_cache_alloc_all(); + if (!grp) { + err =3D -ENOMEM; + goto out_task; + } + break; + + case PR_SCHED_CACHE_SHARE_FROM: + if (arg4 > INT_MAX) { + err =3D -EINVAL; + goto out_task; + } + + src =3D sched_cache_find_get_task(arg4); + if (!src) { + err =3D -ESRCH; + goto out_task; + } + + if (src->flags & PF_KTHREAD) { + err =3D -EINVAL; + goto out_task; + } + + if (!ptrace_may_access(src, PTRACE_MODE_READ_REALCREDS)) { + err =3D -EPERM; + goto out_task; + } + + /* copy the group of src to dst */ + grp =3D task_cache_group_get(src); + if (!grp) { + err =3D -ENOENT; + goto out_task; + } + break; + + default: + err =3D -EINVAL; + goto out_task; + } + + if (type =3D=3D PIDTYPE_PID) { + __sched_cache_set(dst, grp); + goto out_group; + } + + read_lock(&tasklist_lock); + pid_grp =3D task_pid_type(dst, type); + + do_each_pid_thread(pid_grp, type, p) { + if (!ptrace_may_access(p, PTRACE_MODE_READ_REALCREDS)) { + err =3D -EPERM; + goto out_tasklist; + } + } while_each_pid_thread(pid_grp, type, p); + + do_each_pid_thread(pid_grp, type, p) { + __sched_cache_set(p, grp); + } while_each_pid_thread(pid_grp, type, p); +out_tasklist: + read_unlock(&tasklist_lock); + +out_group: + sched_cache_group_put(grp); +out_task: + if (src) + put_task_struct(src); + if (dst) + put_task_struct(dst); + return err; +} diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index be1f3568c3aa..d422b62ba987 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1484,7 +1484,13 @@ static bool invalid_llc_nr(struct sched_cache_group = *grp, struct task_struct *p, { int scale; =20 - if (get_nr_threads(p) <=3D 1) + /* + * Single threaded process that is not grouped with other threads + * do not need cache aware scheduling. + * A single-threaded process has a refcount of 2: from the mm and task + * respectively. + */ + if (refcount_read(&grp->refcnt) <=3D 2 && get_nr_threads(p) <=3D 1) return true; =20 /* diff --git a/kernel/sys.c b/kernel/sys.c index df69bd71de03..f13f1904654b 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2907,6 +2907,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, a= rg2, unsigned long, arg3, if (arg3 & PR_CFI_LOCK && !(arg3 & PR_CFI_DISABLE)) error =3D arch_prctl_lock_branch_landing_pad_state(me); break; +#ifdef CONFIG_SCHED_CACHE + case PR_SCHED_CACHE: + error =3D sched_cache_prctl(option, arg2, arg3, arg4, arg5); + break; +#endif default: trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5); error =3D -EINVAL; --=20 2.32.0 From nobody Sat Sep 26 21:59:09 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DC70384CDD; Fri, 28 Aug 2026 22:23:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955840; cv=none; b=Les51BU5OuxYGOiC1IG4eBtM9ufY7NeJYTlO56dRZljmLzBMNaohoHqvyK7ZiIRDXIYar6X8wO4hXmyW24ccvcGoAS6m89pXArfqf/h8yx403w6Cw3qkHDH3hv3WPTBKfZu+ggHSDMT9qbgxeCH7IySZMSRNeaaqOoXLYpw+5Ko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955840; c=relaxed/simple; bh=EathXJ0ODYDs+CPUE4zzzr3s7RHPjcYP7FySau1gdaU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UBlDB19YQUPly5Iuwd7W76FVd2VmjBQ6yuE0WNGjur4P62OvJc69ghiKkwvqI8vpfuq0tzsnsJI/mAENghulkJoY9BE0ol92bKlNuyPjLoYP2/dAnwEMzQPtIxbQvIEFeILr2O39mJ4zOO1E+HBVRnKHX9ljjyOge4vtW0uPAUo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XxdCkudF; arc=none smtp.client-ip=192.198.163.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XxdCkudF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1787955838; x=1819491838; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EathXJ0ODYDs+CPUE4zzzr3s7RHPjcYP7FySau1gdaU=; b=XxdCkudFtjVfAs515FlZwhBG/to7x2Gt+RRGAyMI1KUBUn38HcKacAoB J6rotPaf1DdT48y89bylP3JNhQu4O/c7f4dFwhfj6h66vfXGp1PYnc0ny AE4UeBETiKs/MKOOTk4V3dAB8NPR9VTyZgQS0WoFn5Bv+sO3WvqSZWq1t UN1hHBeCXoQu01DKeVxMaEom5fMmeGlZeBq3hpFyA2svi9ujVcA6BX1kz 9Ll36cE6c5HY319Y1BratqXREldQS5cJ97alFR0Oyx26a2QEq7DQNJ+8H WMzL9x4v8HNdEGrT1NyQXwEf1BWEnRCDX3uZitt5liP73X/tInO8nVWJu Q==; X-CSE-ConnectionGUID: traEC4DARHCAOUy3kN4kdg== X-CSE-MsgGUID: quhGKK1iRUatloP60NqStg== X-IronPort-AV: E=McAfee;i="6800,10657,11889"; a="87412360" X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="87412360" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2026 15:23:56 -0700 X-CSE-ConnectionGUID: Ak2S1tTATEabCtKMAI+WlA== X-CSE-MsgGUID: ek2yFjTkT7SJyBt2zkQciw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="264540284" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa010.fm.intel.com with ESMTP; 28 Aug 2026 15:23:55 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar Cc: Tim Chen , Vincent Guittot , Qais Yousef , K Prateek Nayak , Juri Lelli , Dietmar Eggemann , Valentin Schneider , Madadi Vineeth Reddy , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Luo Gengkun , Gavin Guo , Yi Lai , Ricardo Neri , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org Subject: [RFC PATCH 5/7] sched/cache: Allow a process to enable cache aware scheduling via prctl Date: Fri, 28 Aug 2026 15:29:12 -0700 Message-Id: <14772dcf31c7eae3cec060fd6884579ed1c0f03b.1787955777.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the PR_SCHED_CACHE_ENABLE and PR_SCHED_CACHE_DISABLE subops to the PR_SCHED_CACHE prctl interface, allowing a process to turn cache aware scheduling on or off. int prctl(PR_SCHED_CACHE, unsigned long subop, pid_t pid, unsigned long cookie, unsigned long type); /* disable cache aware scheduling for this task */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_DISABLE, 0, 0, PIDTYPE_PID); /* enable cache aware scheduling for this task */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_ENABLE, 0, 0, PIDTYPE_PID); pid 0 targets the calling task; otherwise the group of the given pid is used. Both return -ENOENT when the task has no cache scheduling group. Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen --- include/linux/sched.h | 1 + include/uapi/linux/prctl.h | 4 +++- kernel/sched/cache_sched.c | 15 +++++++++++++++ kernel/sched/fair.c | 6 ++++++ 4 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 79f0079c3aa1..1529730c91a5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2399,6 +2399,7 @@ struct sched_cache_group { unsigned long next_scan; unsigned long footprint; int cpu; + int enabled; refcount_t refcnt; struct rcu_head rcu; } ____cacheline_aligned_in_smp; diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index fed7bb028f9a..3fb31c4ab7b5 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -421,6 +421,8 @@ struct prctl_mm_map { # define PR_SCHED_CACHE_GET 0 # define PR_SCHED_CACHE_CREATE 1 # define PR_SCHED_CACHE_SHARE_FROM 2 -# define PR_SCHED_CACHE_MAX 3 +# define PR_SCHED_CACHE_DISABLE 3 +# define PR_SCHED_CACHE_ENABLE 4 +# define PR_SCHED_CACHE_MAX 5 =20 #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/sched/cache_sched.c b/kernel/sched/cache_sched.c index d1932f0c5ee8..c4ec6c553cca 100644 --- a/kernel/sched/cache_sched.c +++ b/kernel/sched/cache_sched.c @@ -60,6 +60,7 @@ static void sched_cache_group_init(struct sched_cache_gro= up *grp, grp->next_scan =3D jiffies; grp->nr_running_avg =3D 0; grp->footprint =3D 0; + grp->enabled =3D 1; refcount_set(&grp->refcnt, 1); /* * The update to grp->pcpu_sched should not be reordered @@ -261,6 +262,20 @@ int sched_cache_prctl(int option, unsigned long arg2, = unsigned long arg3, } =20 switch (arg2) { + case PR_SCHED_CACHE_DISABLE: + case PR_SCHED_CACHE_ENABLE: + /* + * Setting a single task is OK, because the sched_cache_group is + * shared by multiple tasks, setting one equals to setting all. + */ + grp =3D task_cache_group_get(dst); + if (!grp) { + err =3D -ENOENT; + goto out_task; + } + WRITE_ONCE(grp->enabled, arg2 =3D=3D PR_SCHED_CACHE_ENABLE); + + goto out_group; case PR_SCHED_CACHE_GET: { unsigned long id =3D 0; =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d422b62ba987..e7c8b031946c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1652,6 +1652,9 @@ static int get_pref_llc(struct task_struct *p, struct= sched_cache_group *grp) if (!grp) return -1; =20 + if (!READ_ONCE(grp->enabled)) + return -1; + mm_sched_cpu =3D READ_ONCE(grp->cpu); if (mm_sched_cpu !=3D -1) { mm_sched_llc =3D llc_id(mm_sched_cpu); @@ -1743,6 +1746,9 @@ static void task_tick_cache(struct rq *rq, struct tas= k_struct *p) !grp->pcpu_sched) return; =20 + if (!READ_ONCE(grp->enabled)) + return; + epoch =3D rq->cpu_epoch; /* avoid moving backwards */ if (time_after_eq(grp->epoch, epoch)) --=20 2.32.0 From nobody Sat Sep 26 21:59:09 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21BB9358D32; Fri, 28 Aug 2026 22:24:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955846; cv=none; b=V4UkGASXChplL6U5ziCid8lC6L6R0WNiLPWMgSp+C4z1c1meQq0/mvwGarIp5O9lBSpVcUfZiRvKTODkPVnhIO21oGuXKKY/NBMEljzdCLngB5Ti83EEMeufxe3+FRPrhIl3oVD9WFNVmkVp6C/CzFopoWpi9gdIIwj7j7GvSn8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955846; c=relaxed/simple; bh=hxZbtaZ63m/+A+zqLGDwlA1agw5ugsOGWVeXCl/82Mk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pPecLwbgZoziuZNWfK9Ic9ime7U7ia71c8Kab7gnL8VRVuqqx0G5BXSlF4BX+VnpgkmJtDne4ODsQC47ihc9mgmxTNNdToR1eZQZxQSBOQ/Ilq/n+EVu+OjUlkvjLkXcDiSu53E9d8nIHaI/ygWOgBIimhdmymjgJ1bFQiFPiwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=YmKT6iqf; arc=none smtp.client-ip=192.198.163.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="YmKT6iqf" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1787955840; x=1819491840; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hxZbtaZ63m/+A+zqLGDwlA1agw5ugsOGWVeXCl/82Mk=; b=YmKT6iqfJbtFCtx9Zrh8cB4rixc+Pz6gUBYtMSBrIxObYgjARbvP6FK0 0OqCvoU/P1XiWTBD86ZzHyFpPvCGdFHjXc+8a4zuTM6kWYNOscToBvUe2 GD25vXt60m35rkwyGUEvNw4deJrciiYlVJ3WwetAB/wS9hJ9UjsVrrWtB h3FCy+Bq6rLh7XGNRn4XOc//EaSxG+HRVKzmoJru+kxWVnyqcvHugWuHt 4wGUj6N1ayu6AKpnY8koo9HhqZ7cU4rHA0hR5tsAw1hWa/btRgBTEOhk7 3MfloOrhRYeRVcCbQyd2WzEw16qNfkHG2KG/p5Q3zsjn4hj2N9yJETaNz w==; X-CSE-ConnectionGUID: b3hl+6ycQm6nn7AFmKh7jg== X-CSE-MsgGUID: EiCI/gB5RLq0Ta2HWtJqEg== X-IronPort-AV: E=McAfee;i="6800,10657,11889"; a="87412379" X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="87412379" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2026 15:23:58 -0700 X-CSE-ConnectionGUID: V3IRnfwvR1a36rUdKHkk5Q== X-CSE-MsgGUID: Rod7V+T+RB2mi4mg9oS5IA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="264540288" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa010.fm.intel.com with ESMTP; 28 Aug 2026 15:23:57 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar Cc: Chen Yu , Vincent Guittot , Qais Yousef , K Prateek Nayak , Juri Lelli , Dietmar Eggemann , Valentin Schneider , Madadi Vineeth Reddy , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Luo Gengkun , Gavin Guo , Yi Lai , Ricardo Neri , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Tim Chen Subject: [RFC PATCH 6/7] sched/cache: Extend the enabled debugfs to more modes Date: Fri, 28 Aug 2026 15:29:13 -0700 Message-Id: X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chen Yu A knob named "enabled" under debugfs llc_balancing is used to allow the user to turn off and on the cache aware scheduling at runtime. Since there is fine-gain control for per task cache-aware scheduling control via prctl, enhance the "enabled" knob in to a three-value mode, derived from the design of THP: always, advise, never. The design matrix is as followed: prctl ENABLE prctl DISABLE always Y Y advise Y N never N N Y: cache aware scheduling for the task is enabled N: cache aware scheduling for the task is disabled The default mode is "always". For backward compatibility, writing "1" or "0" is still accepted and is treated as "always" or "never" respectively. The same modes can also be selected at boot time via the sched_cache=3D kernel command line parameter. Signed-off-by: Chen Yu Signed-off-by: Tim Chen --- .../admin-guide/kernel-parameters.txt | 3 + kernel/sched/cache_sched.c | 2 +- kernel/sched/debug.c | 63 +++++++++++++++++-- kernel/sched/fair.c | 7 ++- kernel/sched/sched.h | 21 ++++++- kernel/sched/topology.c | 32 ++++++++-- 6 files changed, 112 insertions(+), 16 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index b5493a7f8f22..33b965d4ba19 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6751,6 +6751,9 @@ Kernel parameters solution to mutex-based priority inversion. Format: =20 + sched_cache=3D [KNL] Set the cache aware scheduling mode. + Format: { "always" | "advise" | "never" } + sched_verbose [KNL,EARLY] Enables verbose scheduler debug messages. =20 schedstats=3D [KNL,X86] Enable or disable scheduled statistics. diff --git a/kernel/sched/cache_sched.c b/kernel/sched/cache_sched.c index c4ec6c553cca..c50eb3bebec3 100644 --- a/kernel/sched/cache_sched.c +++ b/kernel/sched/cache_sched.c @@ -60,7 +60,7 @@ static void sched_cache_group_init(struct sched_cache_gro= up *grp, grp->next_scan =3D jiffies; grp->nr_running_avg =3D 0; grp->footprint =3D 0; - grp->enabled =3D 1; + grp->enabled =3D 0; refcount_set(&grp->refcnt, 1); /* * The update to grp->pcpu_sched should not be reordered diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 40584b27ea0c..ddefbaf48d47 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -211,19 +211,58 @@ static const struct file_operations sched_scaling_fop= s =3D { }; =20 #ifdef CONFIG_SCHED_CACHE + +static const char * const sc_modes_names[] =3D { + [SC_ENABLED_ALWAYS] =3D "always", + [SC_ENABLED_ADVISE] =3D "advise", + [SC_ENABLED_NEVER] =3D "never", +}; + +static int sched_cache_set_mode(const char *str) +{ + int mode; + bool val; + + mode =3D match_string(sc_modes_names, ARRAY_SIZE(sc_modes_names), str); + if (mode < 0) { + if (kstrtobool(str, &val)) + return mode; + + mode =3D val ? SC_ENABLED_ALWAYS : SC_ENABLED_NEVER; + } + + WRITE_ONCE(sysctl_sched_cache_mode, mode); + + return 0; +} + +static int __init setup_sched_cache_mode(char *str) +{ + return sched_cache_set_mode(str) ? 0 : 1; +} + +__setup("sched_cache=3D", setup_sched_cache_mode); + static ssize_t sched_cache_enable_write(struct file *filp, const char __user *ubuf, size_t cnt, loff_t *ppos) { - bool val; + char buf[16]; int ret; =20 - ret =3D kstrtobool_from_user(ubuf, cnt, &val); - if (ret) - return ret; + if (cnt > sizeof(buf) - 1) + return -EINVAL; =20 - sysctl_sched_cache_user =3D val; + if (copy_from_user(buf, ubuf, cnt)) + return -EFAULT; =20 + buf[cnt] =3D 0; + /* 1. parse user provide mode */ + ret =3D sched_cache_set_mode(strstrip(buf)); + if (ret < 0) + return ret; + + /* 2. adjust the static keys */ sched_cache_active_set(); =20 *ppos +=3D cnt; @@ -233,7 +272,19 @@ sched_cache_enable_write(struct file *filp, const char= __user *ubuf, =20 static int sched_cache_enable_show(struct seq_file *m, void *v) { - seq_printf(m, "%d\n", sysctl_sched_cache_user); + int mode =3D READ_ONCE(sysctl_sched_cache_mode); + int i; + + for (i =3D 0; i < SC_ENABLED_NR; i++) { + if (i) + seq_putc(m, ' '); + if (i =3D=3D mode) + seq_printf(m, "[%s]", sc_modes_names[i]); + else + seq_puts(m, sc_modes_names[i]); + } + seq_putc(m, '\n'); + return 0; } =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e7c8b031946c..cf83b24e925d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1652,7 +1652,7 @@ static int get_pref_llc(struct task_struct *p, struct= sched_cache_group *grp) if (!grp) return -1; =20 - if (!READ_ONCE(grp->enabled)) + if (!sched_cache_group_enabled(grp)) return -1; =20 mm_sched_cpu =3D READ_ONCE(grp->cpu); @@ -1746,7 +1746,7 @@ static void task_tick_cache(struct rq *rq, struct tas= k_struct *p) !grp->pcpu_sched) return; =20 - if (!READ_ONCE(grp->enabled)) + if (!sched_cache_group_enabled(grp)) return; =20 epoch =3D rq->cpu_epoch; @@ -10601,6 +10601,9 @@ static enum llc_mig can_migrate_llc_task(int src_cp= u, int dst_cpu, if (!grp) return mig_unrestricted; =20 + if (!sched_cache_group_enabled(grp)) + return mig_unrestricted; + cpu =3D READ_ONCE(grp->cpu); if (cpu < 0 || cpus_share_cache(src_cpu, dst_cpu)) return mig_unrestricted; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 7dbe070979d7..d013850dd253 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -4092,9 +4092,17 @@ static inline void mm_cid_switch_to(struct task_stru= ct *prev, struct task_struct #endif /* !CONFIG_SCHED_MM_CID */ =20 #ifdef CONFIG_SCHED_CACHE +enum sc_modes { + SC_ENABLED_ALWAYS =3D 0, + SC_ENABLED_ADVISE =3D 1, + SC_ENABLED_NEVER =3D 2, + SC_ENABLED_NR, +}; + DECLARE_STATIC_KEY_FALSE(sched_cache_present); DECLARE_STATIC_KEY_FALSE(sched_cache_active); -extern int sysctl_sched_cache_user; +DECLARE_STATIC_KEY_FALSE(sched_cache_adv); +extern int sysctl_sched_cache_mode; extern unsigned int llc_aggr_tolerance; extern unsigned int llc_epoch_period; extern unsigned int llc_epoch_affinity_timeout; @@ -4106,6 +4114,17 @@ static inline bool sched_cache_enabled(void) return static_branch_unlikely(&sched_cache_active); } =20 +static inline bool sched_cache_group_enabled(struct sched_cache_group *grp) +{ + if (!static_branch_unlikely(&sched_cache_active)) + return false; + + if (!static_branch_likely(&sched_cache_adv)) + return true; + + return READ_ONCE(grp->enabled); +} + DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *, sched_cache_group_put(_T)); =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 622e2e01974c..0ae9d270ebf8 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -853,8 +853,12 @@ DEFINE_STATIC_KEY_FALSE(sched_cache_present); * is active, used by the scheduler. */ DEFINE_STATIC_KEY_FALSE(sched_cache_active); -/* user wants cache aware scheduling [0 or 1] */ -int sysctl_sched_cache_user =3D 1; +DEFINE_STATIC_KEY_FALSE(sched_cache_adv); +/* + * User provided cache aware scheduling modes: + * always, advise, never. + */ +int sysctl_sched_cache_mode =3D SC_ENABLED_ALWAYS; =20 /* * Get the effective LLC size in bytes that @cpu's bottom sched_domain @@ -952,6 +956,7 @@ static void _sched_cache_active_set(void) /* hardware does not support */ if (!static_branch_likely(&sched_cache_present)) { static_branch_disable_cpuslocked(&sched_cache_active); + static_branch_disable_cpuslocked(&sched_cache_adv); if (sched_debug()) pr_info("%s: cache aware scheduling not supported on this platform\n", = __func__); return; @@ -963,14 +968,29 @@ static void _sched_cache_active_set(void) * It is not in the critical path, leave as-is * for now. */ - if (sysctl_sched_cache_user) { + switch (READ_ONCE(sysctl_sched_cache_mode)) { + case SC_ENABLED_ALWAYS: + static_branch_disable_cpuslocked(&sched_cache_adv); static_branch_enable_cpuslocked(&sched_cache_active); if (sched_debug()) - pr_info("%s: enabling cache aware scheduling\n", __func__); - } else { + pr_info("%s: cache aware scheduling switch to [always]\n", __func__); + break; + case SC_ENABLED_ADVISE: + static_branch_enable_cpuslocked(&sched_cache_adv); + static_branch_enable_cpuslocked(&sched_cache_active); + if (sched_debug()) + pr_info("%s: cache aware scheduling switch to [advise]\n", __func__); + break; + case SC_ENABLED_NEVER: static_branch_disable_cpuslocked(&sched_cache_active); + static_branch_disable_cpuslocked(&sched_cache_adv); if (sched_debug()) - pr_info("%s: disabling cache aware scheduling\n", __func__); + pr_info("%s: cache aware scheduling switch to [never]\n", __func__); + break; + default: + if (sched_debug()) + pr_info("%s: Invalid cache aware scheduling mode\n", __func__); + break; } } =20 --=20 2.32.0 From nobody Sat Sep 26 21:59:09 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56B363AA1A9; Fri, 28 Aug 2026 22:24:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.19 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955852; cv=none; b=Rx0H40DUKpEocdgv/Z6GX+M+vZxsrB++FAawiGD+wpH+OCaXbePMAXTaDT0rQaYorz+Z7sBe7AOGewLgn3gFEQr/X91wxTUheSpDKr4mi51mNQ3kj22ms//ZhtsFBmE396LN4le0f1jkwq0opqMKBnZvufA/AGpvfHQ++MPMdPw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787955852; c=relaxed/simple; bh=FE9HEPduTEIxPTgkPLJpb410zPHdhLbUT4oqpU/8vxs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=l/ilYpmqC2TWQt1Ec0/KroEPYCmxwjBxYqE4w5w+Il+hZjzKHBfo6BjBPf43Jlay2R850A9bPvKROFqI47U5R6oNWbaha7AyQaEISSwojysVvVzEgeKbNxOcmpZER8Nq/J1titmXbi0DEaEizGYHl7CMx5DKtEpFWJO5EOxDpt8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=l807Imck; arc=none smtp.client-ip=192.198.163.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="l807Imck" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1787955840; x=1819491840; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FE9HEPduTEIxPTgkPLJpb410zPHdhLbUT4oqpU/8vxs=; b=l807Imck+cGxMHLkH/5j9TSuo/7Sh5EK0B9KMIykDSd3vAGVyeE68ELf ntXafwejLUqlb9dYRUeC2P1fRygqdbet1LpZ+RrdpIFEG/azopEyMqq6i 84hdgg0IUdKaiDTo8xlg7ZEPnyZadJN9gSP6/GZUmHE8cNlMt2ikV1JDS 1CSINH72MLCKyZCscH7ie4WN/BhIeySSH7w+AVNVusA7uMxMXmD6qcF7p 0kkNreAz/TzJOErGvbbQReUpb9SRRxsbtFmfbscopdh/8ONJoz5o9+Hte fzJxYoH9BapNSeGQjiuyc5d7FUebX76dBWq1ykmxgxrYCXt3psQpVtkgD A==; X-CSE-ConnectionGUID: c/1a1mt1THCvg9tOvYPVEg== X-CSE-MsgGUID: ZT5GjM49QxOoFU9hby+Pmw== X-IronPort-AV: E=McAfee;i="6800,10657,11889"; a="87412398" X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="87412398" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa113.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2026 15:23:59 -0700 X-CSE-ConnectionGUID: t4Vn67LLSsqheMt730Ee7A== X-CSE-MsgGUID: lTjHApJRQdOcqrCVWEZs2g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.25,249,1779174000"; d="scan'208";a="264540295" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa010.fm.intel.com with ESMTP; 28 Aug 2026 15:23:59 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar Cc: Tim Chen , Vincent Guittot , Qais Yousef , K Prateek Nayak , Juri Lelli , Dietmar Eggemann , Valentin Schneider , Madadi Vineeth Reddy , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Luo Gengkun , Gavin Guo , Yi Lai , Ricardo Neri , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org Subject: [RFC PATCH 7/7] sched/cache: Documentation: document the PR_SCHED_CACHE prctl Date: Fri, 28 Aug 2026 15:29:14 -0700 Message-Id: <86909723689d9c496b5a67ee2a0ecfd0d63e8712.1787955777.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add Documentation/scheduler/sched-cache.rst describing cache aware scheduling and the PR_SCHED_CACHE prctl() interface: the GET/CREATE/SHARE_FROM/DISABLE/ENABLE sub-operations, their arguments, the ptrace_may_access() permission model, the return values and how the per-task hint composes with the always/advise/never debugfs policy. Hook it into the scheduler documentation toctree. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Tim Chen --- Documentation/scheduler/index.rst | 1 + Documentation/scheduler/sched-cache.rst | 175 ++++++++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 Documentation/scheduler/sched-cache.rst diff --git a/Documentation/scheduler/index.rst b/Documentation/scheduler/in= dex.rst index 17ce8d76befc..5c575593723a 100644 --- a/Documentation/scheduler/index.rst +++ b/Documentation/scheduler/index.rst @@ -23,5 +23,6 @@ Scheduler sched-stats sched-ext sched-debug + sched-cache =20 text_files diff --git a/Documentation/scheduler/sched-cache.rst b/Documentation/schedu= ler/sched-cache.rst new file mode 100644 index 000000000000..3144934ffa88 --- /dev/null +++ b/Documentation/scheduler/sched-cache.rst @@ -0,0 +1,175 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Cache Aware Scheduling +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Overview +=3D=3D=3D=3D=3D=3D=3D=3D + +On a machine with several last level caches (LLCs), it can pay off to keep +a set of cooperating tasks on CPUs that share one LLC, so that the data +they pass between each other stays cache hot instead of bouncing across +the interconnect. + +The scheduler tracks, per group of tasks, how much runtime the group spends +on each LLC and nudges the group's members towards the LLC where it is most +active, as long as that LLC is not already overcommitted. This is enabled = by +CONFIG_SCHED_CACHE. + +The unit that is aggregated is a *cache group* (struct sched_cache_group). +By default every address space (mm) gets its own cache group, so the threa= ds +of a process are aggregated together and nothing else is. That default is a +good fit for a classic multi-threaded process, but not for every workload: + + - A workload split across cooperating *processes* rather than threads - a + database with a process per connection, a browser with a renderer per + site, a server and its worker helpers - shares data through shared mem= ory + or pipes but never shares an mm, so the default never aggregates it. + + - A process whose threads do not actually share data is aggregated anywa= y, + just because the threads live in one address space. + +To cover those cases a process can manage cache group membership explicitly +through prctl(2). + +The prctl() interface +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +:: + + int prctl(int option, unsigned long subop, unsigned long pid, + unsigned long arg4, unsigned long pid_type); + +with ``option`` set to ``PR_SCHED_CACHE``. The remaining arguments are: + +``subop`` + Which operation to perform (see below). + +``pid`` + The task the operation applies to. ``0`` means the calling task. + +``arg4`` + Only used by ``PR_SCHED_CACHE_GET`` (output pointer) and + ``PR_SCHED_CACHE_SHARE_FROM`` (source pid). Must be ``0`` for every + other sub-operation. + +``pid_type`` + One of ``PIDTYPE_PID``, ``PIDTYPE_TGID`` or ``PIDTYPE_PGID`` (0, 1, 2). + It selects whether the operation affects just the named thread, its wh= ole + thread group, or its process group. ``PR_SCHED_CACHE_GET`` requires + ``PIDTYPE_PID``. + +The cache group itself is a kernel object. User space never invents or +passes a group id; it only names tasks by pid and asks the kernel to create +a group or to copy one task's group onto another. This follows core +scheduling, where the cookie is a kernel object and only an obfuscated +identifier is returned by ``PR_SCHED_CACHE_GET``. + +Sub-operations +-------------- + +``PR_SCHED_CACHE_GET`` + Read back the cache group id of ``pid``. ``arg4`` is a + ``__u64 __user *`` that must be 8-byte aligned; the kernel writes an + obfuscated hash of the task's cache group there (0 if the task current= ly + has no group). ``pid_type`` must be ``PIDTYPE_PID``. + +``PR_SCHED_CACHE_CREATE`` + Allocate a fresh cache group and install it on the target task(s). This + operation is *not* idempotent: each call creates a new group. A caller + that wants several tasks in one group should ``CREATE`` once and then + ``SHARE_FROM`` for the rest. + +``PR_SCHED_CACHE_SHARE_FROM`` + Copy the cache group of the task named by ``arg4`` (the source) onto t= he + task(s) named by ``pid``. This is how a task joins an existing group. + +``PR_SCHED_CACHE_DISABLE`` / ``PR_SCHED_CACHE_ENABLE`` + Opt the target's cache group out of / back into LLC aggregation. Becau= se + the flag lives on the (shared) group, changing it for one member chang= es + it for every task in the group. + +Permissions +----------- + +The caller must be able to ``ptrace_may_access(PTRACE_MODE_READ_REALCREDS)= `` +every task it touches. For the ``PIDTYPE_TGID`` and ``PIDTYPE_PGID`` scopes +the access of *all* tasks in the group is checked before *any* task is +changed, so the operation either applies to the whole group or fails with +-EPERM without touching anyone. ``PR_SCHED_CACHE_SHARE_FROM`` additionally +requires access to the source task. + +Kernel threads cannot be targeted. + +Return value +------------ + +Returns 0 on success. On error one of: + +``-EINVAL`` + Unknown sub-operation, out-of-range ``pid``/``pid_type``, ``arg4`` giv= en + for a sub-operation that does not take one, misaligned ``GET`` pointer, + ``GET`` with a ``pid_type`` other than ``PIDTYPE_PID``, or the target = is + a kernel thread. + +``-ESRCH`` + The target (or, for ``SHARE_FROM``, the source) task does not exist. + +``-EPERM`` + The caller is not allowed to access the target (or source) task. + +``-ENOENT`` + The source task (``SHARE_FROM``) or target task (``DISABLE``/``ENABLE`= `) + has no cache group. + +``-ENOMEM`` + ``CREATE`` could not allocate a new group. + +``-EFAULT`` + ``GET`` could not write to the ``arg4`` pointer. + +Interaction with the system-wide policy +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +A system-wide policy composes with the per-task hint the same way THP does. +It is set through debugfs:: + + /sys/kernel/debug/sched/llc_balancing/enabled + +and takes one of three modes: + +``always`` + Always aggregate, ignoring the per-group disable flag (the default). + +``advise`` + Honour the per-group flag set through ``PR_SCHED_CACHE_DISABLE`` / + ``PR_SCHED_CACHE_ENABLE``. + +``never`` + Disable cache aware scheduling entirely. + +For backward compatibility the knob still accepts the old boolean spelling= on +write (``1``/``y``/``on`` map to ``always``, ``0``/``n``/``off`` to +``never``); reads show ``[always] advise never`` with the active mode in +brackets. + +The same three modes can be selected at boot time, before any process has +had a chance to use the prctl, with the kernel command line parameter:: + + sched_cache=3D{always|advise|never} + +The debugfs knob can still be written afterwards to change the mode at +runtime. + +Example +=3D=3D=3D=3D=3D=3D=3D + +Put a helper process into the same cache group as its parent:: + + /* In the parent, once per instance. */ + prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_CREATE, 0, 0, PIDTYPE_TGID); + + /* In (or on behalf of) each helper. */ + prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, + helper_pid, parent_tgid, PIDTYPE_PID); --=20 2.32.0