From nobody Fri Dec 19 17:37:16 2025 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2068.outbound.protection.outlook.com [40.107.94.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 525622561CE for ; Wed, 9 Apr 2025 11:16:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.68 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197383; cv=fail; b=h+kuveMUWOZL6Go+7iBuMbQWzffiz4WI/bmWPesBSsqdWC1B6Cajz2QUp81hVNm9ZDV12lM55hFqdnJ/GXAjpIbP3Yx/ehxqyhysw+KT9Z9CVjXqZ1gHilviGulK92x+ekG/JPkGjji8DIfwGAM8crp4EZ9rQeVW4Maku2SgbLg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197383; c=relaxed/simple; bh=K4QmAXWan/Gfh4IaOG/Go2kfyw+qoo0me0o6FHn61WY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KhV7zaqSu7GC1qxNkfPG0PqtP+kX9GIK1pT87GK7vXupNCzN0xdDEGjiBG8YzlV6l6fJSKMQ7ufzIvn2Lbxj0zK+vR1MikwZHEA3UWEJisI6lAhz+A/mNiEK+I6O9CCOI4pmXyejyerLcKG4zkbacQa/QeYFZ8AcagHVqUHOmQU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=neZ87bgR; arc=fail smtp.client-ip=40.107.94.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="neZ87bgR" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=F/8vv0ocioFKVbDF6yZ8LiXJ8zPgXvSh6lJpW0LqFY+/bWl+VyG/RoOnTa+HMlCgXzrCWCoSrQsIQXSvL+pw9j2Pm4MNDaYSxPhEf9YKuVzVY5/PbGY+8LCxUY9+tmYQkNLJwFK7c9r1ZdqeLvyHTSpEuQi7+HxwJyzNrmJRmtIA4/CIF+MOP8pEGx62rdxiutVUm4j/dfttzxh1rlHQnucQiHOCFAzrf/AogQla/UeEsgZRp89aOt4nhXBGV76zX82Duoob84eqEd+S42FjSJTx5gBGWt8jNU468DdywIgPSs4k/kLjkVlPXgARHk+rm1Xlh692y1gE7IET1l8i7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kPVaxz21b+77MaWG9eBfmcoXKISJNR++9KE9ZeXUHHw=; b=AmzRrpg10NJPmsDo09K9tRx2Dw3eB3CjMNKcnqBZrJI1dgOTIFxj2EkyS9KOAofkFJcocyCNBX1nWe0LYHVxGyhGhXPOKQU/iYQRxWDlISxxTheCXVeUM2KxOabOA17vmXcMxOnU1wTBPmmvqPVRRLKy7U5hEQsuht3vdAvVf89jPDaL8L6sJLVi/YtEue1+6An866b+sx1B3e6b6lrTAjgcrNn/XKJIO4TqDwZ6hJmvjL2NBbbnxAb9y+nKODikGBD+jPG3x48bv3g92eoB2HuYxifMTV2oESHt5U0QszSHAZk8wqtpRGjOX+plrHzCbjQwPALG4+C2Svsz1Xrv0Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kPVaxz21b+77MaWG9eBfmcoXKISJNR++9KE9ZeXUHHw=; b=neZ87bgReiSZckgPbuD4r5sa+xxI3a8TjX5Et6UNEpiRLzXjLudljyxawKIQSYlif+K3llv2Nb9s2nKieOrgCxjtDukV5POoTSS1dhKpQ34pkOFLtfhBgn/67CHh5THDZWOohOcLv/czqqjyPFd6JAn4Efe+IexvlCXP0QuJoJQ= Received: from CY5PR16CA0017.namprd16.prod.outlook.com (2603:10b6:930:10::6) by PH7PR12MB7115.namprd12.prod.outlook.com (2603:10b6:510:1ee::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8606.33; Wed, 9 Apr 2025 11:16:18 +0000 Received: from CY4PEPF0000FCC2.namprd03.prod.outlook.com (2603:10b6:930:10:cafe::ca) by CY5PR16CA0017.outlook.office365.com (2603:10b6:930:10::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8632.22 via Frontend Transport; Wed, 9 Apr 2025 11:16:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000FCC2.mail.protection.outlook.com (10.167.242.104) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8606.22 via Frontend Transport; Wed, 9 Apr 2025 11:16:17 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 9 Apr 2025 06:16:12 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Gautham R. Shenoy" , Swapnil Sapkal , "K Prateek Nayak" Subject: [RFC PATCH 1/5] sched/fair: Add push task framework Date: Wed, 9 Apr 2025 11:15:35 +0000 Message-ID: <20250409111539.23791-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250409111539.23791-1-kprateek.nayak@amd.com> References: <20250409111539.23791-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCC2:EE_|PH7PR12MB7115:EE_ X-MS-Office365-Filtering-Correlation-Id: 52c57013-712e-46a7-f199-08dd7757f300 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?FiJo86WTMzdqjtw8VtlRltDI1mxluBlTos1eCb1O7UXU1jZK5OofhT9Z5Sdj?= =?us-ascii?Q?jQaIPAC0A9/lTPkrAPscOk44++HPcijChJujDIaKAS+Cv3TltR1w8zSEy7DF?= =?us-ascii?Q?VbFOPX70/hkzC6CrjwDyudLRUXaH++h4rICl1SI9nY8JrpzbvSdzp25TXLC0?= =?us-ascii?Q?9cLvxBWEStrh2DE4MEmmIs5tcmKX3rvaH3Y8Hz5aPeGL6RVPEQbdUC+i4CRt?= =?us-ascii?Q?jcG/THORVYGbso+NeOcUkmrCXi5WLygWvJWtn4SgJSVzky+3Wu1B1rxr0Vnz?= =?us-ascii?Q?1F/IpUPyGeHRZZdrIMDZSSTn0T9SRlvv/CymkgoX0rHNXSsPRkIPdJ0adPr6?= =?us-ascii?Q?u8KPlDLDi+kJR6QRGaPGapWeWU4qiro1DOSOC6jHWxGGHL9qzoYyX0jIGHrW?= =?us-ascii?Q?sfvfKkyKbyggG2H/cTTllLz49EI9ynoZ9Mew6baSznMXfHw9zU9IHjVLXQxt?= =?us-ascii?Q?uXOcKGysu3WhMBo3ktGgKJy0Z0qvPvioJjkSer8he27u3MjlUgKSJBFgGx2q?= =?us-ascii?Q?wD7TDxw7mAVaJK7oq057eD68+swMSGwFfANOYmLCwFrwKTRq3a9Z/aHfh5oa?= =?us-ascii?Q?rYyBtmKufV4IVYlnngQPbuXe7nDUx794Ps1F6rORnoxevMdiz+T8hILBiHcX?= =?us-ascii?Q?A71f7IH6otKzcdgbvft8zMVN/9yHBkuPhNmUenGPJM9ZStPUqHQXiP3RIRqY?= =?us-ascii?Q?Q8wRGm4JfBwdXkc8qCWMbbfwBZh85bu87bGftaa7gyNS0aNyS6MjUJP8+xDJ?= =?us-ascii?Q?yHMIWNKE1/E1QwuhsJ0DzhrSCIwW9/X6JpW2+tUKF7v8RKmd8pufPXStA2VJ?= =?us-ascii?Q?OoQomA5usXb8BXmePH7TrPss89TzA1Jjb8Ub4enYoIDeJtsVbcdD4nHYQ+iW?= =?us-ascii?Q?kC3WqDJfJmbGe4AcaBbLRE5L5lBY4VUtgMRKY/C8fsidAdSOzxN5E+aDkOUp?= =?us-ascii?Q?G8JlcNEZG3N1lu6S/609ESk/yRm2qA7pMsdxR9Zy2Lg8LQbwVYK4lW4jpPx6?= =?us-ascii?Q?1luFWjmor1Dog7uSn10uIZxPF7OxmV8aNQolWY3P31K9AbYZZN1N7OKXpedR?= =?us-ascii?Q?i0z+BWvC3HQulB2mcp2Q/9gSjfFytAxZ75bDz4Od+wRZtMF5BIwlaYTGZwZS?= =?us-ascii?Q?ZXhcKHJ833+Ju66f5IDW9wrzfhBQhJa+Jk62ebaiQTGIsL/jHvEZBRihdOza?= =?us-ascii?Q?O9dlFiwNye6Mr0HfcwcTYNlA4fJzDF0igVv/Bka5w/Up3Uc+/3nErvPHY51N?= =?us-ascii?Q?22wIgVqmv8g25jV07aaK9nVQaAMyqF+tlmmpYCRrbLAyRXbnQ+WX3KmvEUSH?= =?us-ascii?Q?iiQXDdqBIMAJWvdg6jv6BczIqd1Eku7I0y6gRBoui3NY21CNUu1e7eRLew0i?= =?us-ascii?Q?wGtnjyq9XxyFeKHmfFuRrRh+WtjyAuYQeyRdUKDdzpodNS5iX81PHk19ecJZ?= =?us-ascii?Q?9R2tErBiDz33NOce6fFu23JF6r79/VbJVGeWYwKQtv+Ev0OmgNu8ulxynzSt?= =?us-ascii?Q?GVC/D9p2GRhKCHjHzHHbd6M9xeSfjyrpuJTPzixhv/oo72drl+v1AFm08w?= =?us-ascii?Q?=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2025 11:16:17.7701 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 52c57013-712e-46a7-f199-08dd7757f300 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCC2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7115 Content-Type: text/plain; charset="utf-8" From: Vincent Guittot Add the skeleton for push task infrastructure. The empty push_fair_task() prototype will be used to implement proactive idle balancing in subsequent commits. [ prateek: Broke off relevant bits from [1] ] Link: https://lore.kernel.org/all/20250302210539.1563190-6-vincent.guittot@= linaro.org/ [1] Signed-off-by: Vincent Guittot Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 85 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 2 ++ 2 files changed, 87 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0c19459c8042..98d3ed2078cd 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7044,6 +7044,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) hrtick_update(rq); } =20 +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p= ); static void set_next_buddy(struct sched_entity *se); =20 /* @@ -7074,6 +7075,8 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) h_nr_idle =3D task_has_idle_policy(p); if (task_sleep || task_delayed || !se->sched_delayed) h_nr_runnable =3D 1; + + fair_remove_pushable_task(rq, p); } else { cfs_rq =3D group_cfs_rq(se); slice =3D cfs_rq_min_slice(cfs_rq); @@ -8556,6 +8559,64 @@ static int find_energy_efficient_cpu(struct task_str= uct *p, int prev_cpu) return target; } =20 +static inline bool fair_push_task(struct task_struct *p) +{ + if (!task_on_rq_queued(p)) + return false; + + if (p->se.sched_delayed) + return false; + + if (p->nr_cpus_allowed =3D=3D 1) + return false; + + return true; +} + +static inline int has_pushable_tasks(struct rq *rq) +{ + return !plist_head_empty(&rq->cfs.pushable_tasks); +} + +/* + * See if the non running fair tasks on this rq can be sent on other CPUs + * that fits better with their profile. + */ +static bool push_fair_task(struct rq *rq) +{ + return false; +} + +static void push_fair_tasks(struct rq *rq) +{ + /* push_fair_task() will return true if it moved a fair task */ + while (push_fair_task(rq)) + ; +} + +static DEFINE_PER_CPU(struct balance_callback, fair_push_head); + +static inline void fair_queue_pushable_tasks(struct rq *rq) +{ + if (!has_pushable_tasks(rq)) + return; + + queue_balance_callback(rq, &per_cpu(fair_push_head, rq->cpu), push_fair_t= asks); +} +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p) +{ + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); +} + +static void fair_add_pushable_task(struct rq *rq, struct task_struct *p) +{ + if (fair_push_task(p)) { + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); + plist_node_init(&p->pushable_tasks, p->prio); + plist_add(&p->pushable_tasks, &rq->cfs.pushable_tasks); + } +} + /* * select_task_rq_fair: Select target runqueue for the waking task in doma= ins * that have the relevant SD flag set. In practice, this is SD_BALANCE_WAK= E, @@ -8725,6 +8786,9 @@ balance_fair(struct rq *rq, struct task_struct *prev,= struct rq_flags *rf) return sched_balance_newidle(rq, rf) !=3D 0; } #else +static inline void fair_queue_pushable_tasks(struct rq *rq) {} +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p= ) {} +static inline void fair_add_pushable_task(struct rq *rq, struct task_struc= t *p) {} static inline void set_task_max_allowed_capacity(struct task_struct *p) {} #endif /* CONFIG_SMP */ =20 @@ -8914,6 +8978,12 @@ pick_next_task_fair(struct rq *rq, struct task_struc= t *prev, struct rq_flags *rf put_prev_entity(cfs_rq, pse); set_next_entity(cfs_rq, se); =20 + /* + * The previous task might be eligible for being pushed on + * another cpu if it is still active. + */ + fair_add_pushable_task(rq, prev); + __set_next_task_fair(rq, p, true); } =20 @@ -8986,6 +9056,13 @@ static void put_prev_task_fair(struct rq *rq, struct= task_struct *prev, struct t cfs_rq =3D cfs_rq_of(se); put_prev_entity(cfs_rq, se); } + + /* + * The previous task might be eligible for being pushed on another cpu + * if it is still active. + */ + fair_add_pushable_task(rq, prev); + } =20 /* @@ -13260,6 +13337,8 @@ static void __set_next_task_fair(struct rq *rq, str= uct task_struct *p, bool firs { struct sched_entity *se =3D &p->se; =20 + fair_remove_pushable_task(rq, p); + #ifdef CONFIG_SMP if (task_on_rq_queued(p)) { /* @@ -13277,6 +13356,11 @@ static void __set_next_task_fair(struct rq *rq, st= ruct task_struct *p, bool firs if (hrtick_enabled_fair(rq)) hrtick_start_fair(rq, p); =20 + /* + * Try to push prev task before checking misfit for next task as + * the migration of prev can make next fitting the CPU + */ + fair_queue_pushable_tasks(rq); update_misfit_status(p, rq); sched_fair_update_stop_tick(rq, p); } @@ -13307,6 +13391,7 @@ void init_cfs_rq(struct cfs_rq *cfs_rq) cfs_rq->tasks_timeline =3D RB_ROOT_CACHED; cfs_rq->min_vruntime =3D (u64)(-(1LL << 20)); #ifdef CONFIG_SMP + plist_head_init(&cfs_rq->pushable_tasks); raw_spin_lock_init(&cfs_rq->removed.lock); #endif } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index c5a6a503eb6d..aa92c0d75de7 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -716,6 +716,8 @@ struct cfs_rq { struct list_head leaf_cfs_rq_list; struct task_group *tg; /* group that "owns" this runqueue */ =20 + struct plist_head pushable_tasks; + /* Locally cached copy of our task_group's idle value */ int idle; =20 --=20 2.34.1 From nobody Fri Dec 19 17:37:16 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2078.outbound.protection.outlook.com [40.107.223.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 604C92561D3 for ; Wed, 9 Apr 2025 11:16:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.78 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197401; cv=fail; b=AZ8Pdg9x5rDJv46WmcDh6ML1nIytJaMfUAu43KmLkxzQDSnw3s72TUw1CB6uJMSq4m8ilK8Ql0U3kuE7deZOJ6rPvtCKkLunwPhs4L6H0QjSwty71hmWe0scB8VBOUV4qUo7KJ8XAiDLGKzQBgGEJQFjZxseiPmxjQ8QxGs57x0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197401; c=relaxed/simple; bh=DhCV7UvZubMo5Yn7uYLodz9lrKpzjW39fx8Z/6bN1bM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FAPV1OkorrTeTSIDe+99bsPS3vq37c/gWQPnVBr8IzutceOhoaW+htEizjF9zoJpewyy9xCqkdeC7Jwcqdto5kDIyerlOXD424id6rK9Yr5BfX5u4pKqjU583JRYwpuqkBGvFhVHgNVzrIcJQR4tJxMtVvthfBHzbqZgE03kFTo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=uDaf0Y+L; arc=fail smtp.client-ip=40.107.223.78 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="uDaf0Y+L" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OohTrRN2pdmcEbgYzO42rLfu8bMqdhDTx8mta71vMh+fQIWkE+nvl6kh++EruiJqFuddstZnfQiKkYensiZ/IEy6/7hD4vHGwaS5zhNugxL2Ka9T0R2j1km0iVrIzU6ZMZE45aPPGoWiZ3Q0P5d17Kb7LDBoSEkOrGCYt+rRy/pHCjzta7VAp9+QWkYdjtXpm3tKhSpYKOmkbli8coDg2W0Neip8Xr02ecE5Nusc+8GjpUdMe7+bQnvPf+LYmdP2rCNsPC9AfnmyN5BMekCKo89FXO4fwSBomQV3dIUngHtduhMtsrk3L3fal15A/kua+8JgRAi0jljrUCUPQcmaJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=te1g3Y8fwXuE/MHIGpEY/XmraX83wwpTWzsQjrF/yQw=; b=OBP9UTTcJ0+a1jKjlOM7ZL1+hvNIX48QjYVYEYqL1r1yDynO5N5uDvdqZvBGouOY0kdQbcVRyrbvrrd9HiuzebhrF0llWhOBbHcVFUO2QW51VgAuZOfsETRs/AzsAOVuctJIN47hHku4qk3djY9k2UP7C3HXvRnDx3mgjo0+BHtghkCmMxB9DWlSlgY7Z+LPV4Lx1yoRvgRHUMrSTL2Bfm4Q6iSDwKt7AVOUwE1gb0kJ/8xrJ/HMIqDp9VWRcPXK5vBZZRgs2s4rQ/C77K5BQHAdyCTf4LfPVppVt0gmbIW/fDimoU+IuyiM0CcAFg4xvPAvarhkwc8Gm/kXv5dOXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=te1g3Y8fwXuE/MHIGpEY/XmraX83wwpTWzsQjrF/yQw=; b=uDaf0Y+LqUsRzGywssXEVh97uO3Fupj4VEsDXSPDP63KDgMg18fcbBWAOavk/mzRR8tM+/wO89whIbH/fKx+2qgedu8ZEi3Gb3z/lcurVStC0abaiq4XboIiKpE34xulvk07l8qzZk70IfVfUQR61xv2oCV/vhCNJsnRA+wPbUY= Received: from CY8PR19CA0039.namprd19.prod.outlook.com (2603:10b6:930:6::25) by DM6PR12MB4202.namprd12.prod.outlook.com (2603:10b6:5:219::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.22; Wed, 9 Apr 2025 11:16:36 +0000 Received: from CY4PEPF0000FCC3.namprd03.prod.outlook.com (2603:10b6:930:6:cafe::7f) by CY8PR19CA0039.outlook.office365.com (2603:10b6:930:6::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8606.37 via Frontend Transport; Wed, 9 Apr 2025 11:16:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000FCC3.mail.protection.outlook.com (10.167.242.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8606.22 via Frontend Transport; Wed, 9 Apr 2025 11:16:36 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 9 Apr 2025 06:16:30 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Gautham R. Shenoy" , Swapnil Sapkal , "K Prateek Nayak" Subject: [RFC PATCH 2/5] sched/fair: Introduce overloaded_mask in sched_domain_shared Date: Wed, 9 Apr 2025 11:15:36 +0000 Message-ID: <20250409111539.23791-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250409111539.23791-1-kprateek.nayak@amd.com> References: <20250409111539.23791-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCC3:EE_|DM6PR12MB4202:EE_ X-MS-Office365-Filtering-Correlation-Id: ac0a48cd-2146-4fc0-f511-08dd7757fe2c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vn8vMhEpeYydEAzwFfLGl6TFTZA7eJksNWj3IfR2vz0LzY9cLmY+rcfdeL9b?= =?us-ascii?Q?SSLQqApIVTj0Tl/e5wTHbsTPBo536bd5FwaTECcyQSFpYeo8Wr434oalOxeu?= =?us-ascii?Q?4uqnXDeZ0/Y1+saETprLyvjL53Q+fDhkVMHxcNRhJX/FWa4YQQypP2RzbJfD?= =?us-ascii?Q?5ArDyR+zbrxWLQBoBdCwZuwHKoPYO1CFudGThaV3sBcaQhnWQ1LnTU7ISnV8?= =?us-ascii?Q?yCgxp2qoJM94hjTYOhBKHKqpRfNnAUtJxJkN1Brb43WYMMv5N9Y+/vkLykb8?= =?us-ascii?Q?BfCePRUxOqJP8y9KZUWGT9/uGuc/zycfGL87wRX87xNkQxG/S+xr4bUezJ9f?= =?us-ascii?Q?LC71HyN84zVw97tCs48JoVKQB420Qg3auORLBDROLb19dxwj5zRET5/tZ2nc?= =?us-ascii?Q?LaZYYeMDupVt1Yaas1Wd+xZrN2chZoI0XdEIAOZ8E7k4NvHlwxExNnKQqw5f?= =?us-ascii?Q?uvVD7Oyw5JgGIzIhOqdGsSnW5MQnoC2lTvgDZaYfoX7Mxld91JEqIiizIopd?= =?us-ascii?Q?LOk6xGb5lyvse+KyVn0oheMFQplsUW8TBwD4u3hw7vOiCBtJ1nTV34RZHJfm?= =?us-ascii?Q?O9k8iXMvA2xk+VfOmGh6lnSZLbNBbnUDWIIPXusRT7Q9ebqKVrK5Qp2y7LAx?= =?us-ascii?Q?SZNrMJRfE8wuJH2oIbA2NLCDTVBg+1OeufEnF+8AkhghPi0sj/O8ZXFWuLPM?= =?us-ascii?Q?DKxIJT27zpHLCFVU4wCnLnr5iHj4syfw8YXECJh68P/+nLUHiS5MbUB09+kK?= =?us-ascii?Q?WOkq/xqYbRPdlMsHHRkR4/gA4SZBu+kluBQ2JAUPYAFIeCNj3Ng7xO3o8Rgh?= =?us-ascii?Q?53mkgPu2S28yESW8Blz7uybeddQnJLp3hAAs8l7Fq2u2jYfR42xIM7QJJqJH?= =?us-ascii?Q?TkOte2Se6sj9pyPvJ5l4WwM8Eg2bEsji25uAWQb/+Br+soGkUYTooktGmX1R?= =?us-ascii?Q?k9Ldm7YRyH7kNjmi/Ndgzpn94mKnLHWN76+7+RvohEcItkETyq0J0yPONlFG?= =?us-ascii?Q?9L9e5r2WAsU28iSXU8nOHmAm7kqJNpeAb7M0cXDB4Xb9kqDy8Wf8v9uxJ4MJ?= =?us-ascii?Q?xc4ufgTd3kZ+hUA9zD+AEGBgLnOIfXgOTiq7kpSxop+7oF3gJRoevD/fr7ni?= =?us-ascii?Q?WexE3W6rQWW6w440lRkOdsBu2OU3y2Ab75j/ty7Fx0InSoog1z5KSG7VwI0n?= =?us-ascii?Q?JZTtT3FLUxxJ0CVNUoMWRrKOwAOyfeRnRyHUiokRdfLgMSgXYFPh29M3ZQ/7?= =?us-ascii?Q?wUxU8L/7vdddaATr+XAPCgraZQk55GEn8WkxuzSG42n7pL9NSQ5IQzm9d3Z7?= =?us-ascii?Q?08qbAw0QGgkrp3fwEyyT16AATsL2d/rQUTwOxioyyJDhJ4IGDlIK3utKP97k?= =?us-ascii?Q?iCd+KN+Bf+Y2RnG1H9RxbH/Pr+EzhckpmiA163P3kcJOh1877KB2sDcDaRl1?= =?us-ascii?Q?vW6eP7zeR1Z0AZ19WgHmHXYOOn6+H+DlUqg9CRoMF48dgdgo98VZ4Oe+IGTn?= =?us-ascii?Q?jWjLSLFbCt/kqVo=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2025 11:16:36.5135 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ac0a48cd-2146-4fc0-f511-08dd7757fe2c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCC3.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4202 Content-Type: text/plain; charset="utf-8" Introduce a new cpumask member "overloaded_mask" in sched_domain_shared. This mask will be used to keep track of overloaded CPUs with pushable tasks on them and will be later used by newidle balance to only scan through the overloaded CPUs to pull a task to it. Signed-off-by: K Prateek Nayak --- include/linux/sched/topology.h | 1 + kernel/sched/topology.c | 25 ++++++++++++++++++------- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 7b4301b7235f..2fc3794fd719 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -78,6 +78,7 @@ struct sched_domain_shared { atomic_t nr_busy_cpus; int has_idle_cores; int nr_idle_scan; + cpumask_var_t overloaded_mask; }; =20 struct sched_domain { diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index bbc2fc2c7c22..6b1ef953b571 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -638,8 +638,10 @@ static void destroy_sched_domain(struct sched_domain *= sd) */ free_sched_groups(sd->groups, 1); =20 - if (sd->shared && atomic_dec_and_test(&sd->shared->ref)) + if (sd->shared && atomic_dec_and_test(&sd->shared->ref)) { + free_cpumask_var(sd->shared->overloaded_mask); kfree(sd->shared); + } kfree(sd); } =20 @@ -2239,27 +2241,31 @@ static int __sdt_alloc(const struct cpumask *cpu_ma= p) return -ENOMEM; =20 for_each_cpu(j, cpu_map) { + int node =3D cpu_to_node(j); struct sched_domain *sd; struct sched_domain_shared *sds; struct sched_group *sg; struct sched_group_capacity *sgc; =20 sd =3D kzalloc_node(sizeof(struct sched_domain) + cpumask_size(), - GFP_KERNEL, cpu_to_node(j)); + GFP_KERNEL, node); if (!sd) return -ENOMEM; =20 *per_cpu_ptr(sdd->sd, j) =3D sd; =20 sds =3D kzalloc_node(sizeof(struct sched_domain_shared), - GFP_KERNEL, cpu_to_node(j)); + GFP_KERNEL, node); if (!sds) return -ENOMEM; =20 + if (!zalloc_cpumask_var_node(&sds->overloaded_mask, GFP_KERNEL, node)) + return -ENOMEM; + *per_cpu_ptr(sdd->sds, j) =3D sds; =20 sg =3D kzalloc_node(sizeof(struct sched_group) + cpumask_size(), - GFP_KERNEL, cpu_to_node(j)); + GFP_KERNEL, node); if (!sg) return -ENOMEM; =20 @@ -2268,7 +2274,7 @@ static int __sdt_alloc(const struct cpumask *cpu_map) *per_cpu_ptr(sdd->sg, j) =3D sg; =20 sgc =3D kzalloc_node(sizeof(struct sched_group_capacity) + cpumask_size= (), - GFP_KERNEL, cpu_to_node(j)); + GFP_KERNEL, node); if (!sgc) return -ENOMEM; =20 @@ -2299,8 +2305,13 @@ static void __sdt_free(const struct cpumask *cpu_map) kfree(*per_cpu_ptr(sdd->sd, j)); } =20 - if (sdd->sds) - kfree(*per_cpu_ptr(sdd->sds, j)); + if (sdd->sds) { + struct sched_domain_shared *sds =3D *per_cpu_ptr(sdd->sds, j); + + if (sds) + free_cpumask_var(sds->overloaded_mask); + kfree(sds); + } if (sdd->sg) kfree(*per_cpu_ptr(sdd->sg, j)); if (sdd->sgc) --=20 2.34.1 From nobody Fri Dec 19 17:37:16 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 332D62561D6 for ; Wed, 9 Apr 2025 11:16:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197420; cv=fail; b=bCx9RqOYv4aiJd2lC/OXnh+P8Bva18BH+1A1X+77iaW+cmVFm5efHwIvCWlvhpdGTXW4ypuiDwwdXhEoEfi1y46IoRmCtHmLXHPdjqMe3scf5BmYjY2FupPjadw/crxAkroGKEPRxFv7MVqF5vQNjA3Q/77cu5WEPilzvKA9w9k= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197420; c=relaxed/simple; bh=4nSn+GAO9m8AFD0xsJD8XZ0MBgk/A0cDRBNWDtcvkcU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=PzsqVHs8SJGoD5gHjPDsOShA8mYXMQhHS0qPIy1FUK06cQOibS6xNTdw76wDlJ4Fhn7UWEYGBZpbhtwtHSWWhKxxqR1PIdMkv32GSwEqjsvwoecmMMMasOzVRw2k0QJ2S5pa/S5GzOCiIj+Gb0y/j//12KiRMUKgxM+2cRSCaJk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=DbVteIn2; arc=fail smtp.client-ip=40.107.237.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="DbVteIn2" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zSZzgLG9SPJsrnwHryAp2GkFdwx3NzT1y66S/lFIrSWV55O6BUjAMWcCzBWBAvc1iohcb3Pd/SVtiiQ0hQ7TyyBIQ0x5vRZPzixkTOY/fTojslfnGV/l3movVZbAfHpxWCAQWY6dcEFKxRyWtmphbrZYVQQN7ckIHEtey+8TYAE78fGlTvrOplULdn4sMZGjqCTwb1tAcEtk6rX+eXAB7u3q+cnEDz9A3TifwyRogPE1upd3NZwrMHJDGoA1EPFd4euhMzUTsU3TVgdEBceYEJznWFDikjczhdx5kIhLyxk4aeP6Box6SFB5glSRlqU9CMAz7DJsRHFU6S1OeWIhqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dG4j9+b2ocjbwFW1eF/eGqXt241uQykSvmzqc6AQO4s=; b=fjXPDJ0bIVlF0uRQckfEMhQC06BsY3R2Fj15D7l91aI8q45FKw9R/C2PL6/H+dgBuZj9f0tDjVf+ftrKQ5iTt4sPfJwl0zFfNiIYHnSpttfAPi8J8sXjfSlkoSpKUJc/pvlZY5SSHc3/nbnEgD8+TxeHIyzqEQBPpTo2c+OZ9cjRlbLdQZyPLzj2K3xmpUt4ZS2gCwdA8xXvfjQuLYRgvyqxceWjQz18Mi6eC01Wmcb9xvoVA9uaU1mfyIVGpw9rgOZ5jGlKwrMotSq6lMDzfm6P3RU/EQw+BG1lfm6kTyMOsP5LLvwmT/lZyxuEx25eY7KcKdjaNjYg8V2rcX/pCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dG4j9+b2ocjbwFW1eF/eGqXt241uQykSvmzqc6AQO4s=; b=DbVteIn2JOFv4H2A9+hCBzXOASILekByZgx0boeVLqpF9CehvNM1UavlajLbPiCdAXma64Q4lzgQtPthYQdsZrJjlEQ832w29xJEV/GhwKC2/1yRiDaePtLIVEqhkg0maZJESJcTpgEE4s+WpB5Pz+Sy2MJey3B8bpjzZI7Z2+I= Received: from BY3PR05CA0022.namprd05.prod.outlook.com (2603:10b6:a03:254::27) by CY5PR12MB9053.namprd12.prod.outlook.com (2603:10b6:930:37::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.22; Wed, 9 Apr 2025 11:16:56 +0000 Received: from SJ1PEPF00002314.namprd03.prod.outlook.com (2603:10b6:a03:254:cafe::ee) by BY3PR05CA0022.outlook.office365.com (2603:10b6:a03:254::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8632.19 via Frontend Transport; Wed, 9 Apr 2025 11:16:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ1PEPF00002314.mail.protection.outlook.com (10.167.242.168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8606.22 via Frontend Transport; Wed, 9 Apr 2025 11:16:55 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 9 Apr 2025 06:16:49 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Gautham R. Shenoy" , Swapnil Sapkal , "K Prateek Nayak" Subject: [RFC PATCH 3/5] sched/fair: Update overloaded mask in presence of pushable task Date: Wed, 9 Apr 2025 11:15:37 +0000 Message-ID: <20250409111539.23791-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250409111539.23791-1-kprateek.nayak@amd.com> References: <20250409111539.23791-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002314:EE_|CY5PR12MB9053:EE_ X-MS-Office365-Filtering-Correlation-Id: e54c587e-d4d9-4eeb-7356-08dd77580973 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?4WmP52plH08zwmMUKj71kyEOPXVb+gELOgzlCLjnxNJhbZGwSYCi/4FMn1d8?= =?us-ascii?Q?IaTFWqljMsUpESLqTWb3ho+4DrObx2V6vRnRCyKHegv574OQwiBGF/alfHMh?= =?us-ascii?Q?yFb5jIW4G3MJOajjcuKA/2+yXZLbEx8wvnQB2092UV6xONXjSNUuz9TQNBv+?= =?us-ascii?Q?+WBQ8nZVQmV83TXUQex2SNatzWQvT71IbjZ3RF6kPnOh6U/aozKKG9YmH/Xw?= =?us-ascii?Q?jmurOawqSHPGmKNwMvbJUfruRlF3QOGm/XXn7cPHn/PIMIF8oDX0CAx28z6q?= =?us-ascii?Q?2EZt8vbft1/yWANI3FHj4/UERTb8kEi+WDtS75PzFo7xla7lHftuEwIHXBWA?= =?us-ascii?Q?4aLumeSWVtm0gJSr/GYJGAnzifa4h8yWC25y3tX4mnMfofMyCTF8KUnATf65?= =?us-ascii?Q?On1vS52g4QqhibquR1mCWMW0VkrAMJ8kuNIAO8/cVQmcS1dQA/atF8MQkSdn?= =?us-ascii?Q?xZnQYOjrbWD/aloOWw+f3UrPguAubsUHAAIQoCwXDX6uH98AEdxPxDJI7Ven?= =?us-ascii?Q?bg+L5nVGZkkKFNKJl34YbAQ33T6lLib26tSLHu+zAWnuDITiRwnTSWD54u/+?= =?us-ascii?Q?ZPdExQwWuSLnNcsLkw7FUhTvU03iozQgN1CTgtVwwi4SWy8hAoWVGYx7JBEl?= =?us-ascii?Q?SBy9e9SJUllAxKD3qX0gdW0pdc/dGMShLCdseo7wlmR9Gxte8ddfnkVi5/fQ?= =?us-ascii?Q?5DVleaHMDhzM0ufvzR12z5VKkWs0jWHVpiUuYzYmvH9Hcjaj654QkbPJoVrZ?= =?us-ascii?Q?7reLNQePXSjRx9Q7TyrZqcI6+IFngq1q9Ig0MRXLV5DQvN9TYhQ7KhGkz0fF?= =?us-ascii?Q?qoKQpNTM9x0Rfjb34AzCpxgwRdkmFKxc/d0xHZxpfpPzXqNd+KDhxBnUAXPB?= =?us-ascii?Q?L4rjQcMiMkKKCqbNWO3PiZ1yH7lZmv0NOITaQiSFAL/J3HUUfhXLp/Yw7l21?= =?us-ascii?Q?mUQh0dB3U25+0iNY60/oB2a5dTvoY+5BvwsyRCuyvAEln03CU1lFaoNt61Zy?= =?us-ascii?Q?K6q6QPjKMggB6bwIQLV/6OLvLbf5mEZVuXK8M4qZw5NblaUz2cPFh3a4NfM0?= =?us-ascii?Q?8esjOTxWj9lNwnw0XsQzw7HvgoWk+8b/pfxhgBKkgoFnwbnEyWqYl4plOX3h?= =?us-ascii?Q?NRVb9zzfy7Al/mg6h02BZuw06suoOLZAbPTrcTKxgr74/sB5mylSdZHTvAqx?= =?us-ascii?Q?s0qhZ5FgTAA9UR3vPKuByzk3/e7XsbtDMytt5Q8pEAkDKBSERh6/X3048hwS?= =?us-ascii?Q?O/jBHaSEG2IvzSkaIFdqJRaw0E40qdXfyr/cdxz/Q+m1RqdTLaJCq2lnroSw?= =?us-ascii?Q?nUbPe4SPoJ7SH4Qtyj9rlplpmLzLODa9LN2QVorou8UNRYb8OUROkyYux6ue?= =?us-ascii?Q?QJpxhihqVz1mHfWLmn0pw7f/j+DvNPdD9l9zakrpGBOSjU7uQpQ+RxF5p9JW?= =?us-ascii?Q?G5EVunfvpQK9y04WDDkGacVGaYRa5Yq8sm3FbgzZKvLPiCXtQsIMGBlIcaHa?= =?us-ascii?Q?UtOv+YloUeWTorDWi3dZ65ymo6CfXL+yxMqf?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2025 11:16:55.4460 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e54c587e-d4d9-4eeb-7356-08dd77580973 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002314.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB9053 Content-Type: text/plain; charset="utf-8" In presence of pushable tasks on the CPU, set it on the newly introduced "overloaded+mask" in sched_domain_shared struct. This will be used by the newidle balance to limit the scanning to these overloaded CPUs since they contain tasks that could be run on the newly idle target. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 98d3ed2078cd..834fcdd15cac 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8559,6 +8559,24 @@ static int find_energy_efficient_cpu(struct task_str= uct *p, int prev_cpu) return target; } =20 +static inline void update_overloaded_mask(int cpu, bool contains_pushable) +{ + struct sched_domain_shared *sd_share =3D rcu_dereference(per_cpu(sd_llc_s= hared, cpu)); + cpumask_var_t overloaded_mask; + + if (!sd_share) + return; + + overloaded_mask =3D sd_share->overloaded_mask; + if (!overloaded_mask) + return; + + if (contains_pushable) + cpumask_set_cpu(cpu, overloaded_mask); + else + cpumask_clear_cpu(cpu, overloaded_mask); +} + static inline bool fair_push_task(struct task_struct *p) { if (!task_on_rq_queued(p)) @@ -8606,11 +8624,17 @@ static inline void fair_queue_pushable_tasks(struct= rq *rq) static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p) { plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); + + if (!has_pushable_tasks(rq)) + update_overloaded_mask(rq->cpu, false); } =20 static void fair_add_pushable_task(struct rq *rq, struct task_struct *p) { if (fair_push_task(p)) { + if (!has_pushable_tasks(rq)) + update_overloaded_mask(rq->cpu, true); + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); plist_node_init(&p->pushable_tasks, p->prio); plist_add(&p->pushable_tasks, &rq->cfs.pushable_tasks); --=20 2.34.1 From nobody Fri Dec 19 17:37:16 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2046.outbound.protection.outlook.com [40.107.243.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 37ED3255255 for ; Wed, 9 Apr 2025 11:17:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197443; cv=fail; b=ClaWy0znjiChnW9yrC76dZWMvLH36tB2bWHg+bfb7pwyC7zDEJySk/l6/4ma9ru2Lh3bBn7ZmmHvYNVKlFGW+5sbYixz3J3tWR1mdHwfGar9DWRWwKU0GcdzK7+AH2eVBQ/9USW1VgIRYcIT/s7iWXPxFifTC0sWzLf0+M6HwaU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197443; c=relaxed/simple; bh=qIwYxhRon9WtBcT/lz9dO1Cc4SOnOXWVnbn6XpiuXxU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=g9/J10jGCQkCNLi35caVyrGoZlEWrOKDVEj6mgC3jrdmflTIg5up8Mu8kzhtZJISIToiM39BqkDD/mjAgVha/MDNgmObrn6CpNk4kStv05Hj758rtYaIUaWC/J6y35ZVZaXb143bWvixf+cOtHHnX+tXju28jzElEA5gn6cHMFo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=yyR9xG+l; arc=fail smtp.client-ip=40.107.243.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="yyR9xG+l" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=T4oxhYLgPp0L9jHK5f8jdnqesRD5DlgEcmqg345Bh9emGoP5vIGYbLglqDX5wCVmUId5vh0e9qOU1BPv9FtxWbwvZ6ZjcEu37Wn95XJlNY05v2CqXgwiE4DhsDLVYqdPpDYJC7x01vXxvqYXsIgOOMeDWt7z4GVOlXl8nWTXAQryRrNEFIn9D/KDF08D2yFvBS9KYqD6/VXLH8XJmBWrj8Ejf50ZchAmsTs7WtNX2R3gCuWfG8XPcuyt/tz5qAb+e3/k1MShQKNtDW1sHKFagBq73aV7h/FIX4soxA10FlHT0BPJJwgTwbOisuRK0yKherFVC8QuVzQap6zW+7iizg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yy/LaKZc1iEDiA9PIm7E4I9O97NIIJzAUR3c861G43o=; b=wbZV0wMYYyJzyOts5n1TLzkkiIGBEMn0R/FV2MgqdRfJpFYKEvtaK4Gh2ARrc2CC+l8h4lVqbTNh4BI9+rCnC5Ux48ydcikSq+ZI/mY8umIHXmszRuS00d9HUm0dd6otPy1TsEedqTh83z5M0wG8VZOl/GDEpSxQXu4G2Ptj06+6F+QFNcIqNqK9Qjk2CZqbG3mzfghJqt8o/aC9FY1AYYpE8rWMy7tBFyQ5YHEg9vvXGn0T0npUNtadO0q29Y2yb7OJFzYkD5339YeuHjNqB2cU1psDQxLtgQnzDXjuT9Bi8QfhrjyxCy8QeRkhJihd8foi1YaPIsPRKUzYFNIB4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yy/LaKZc1iEDiA9PIm7E4I9O97NIIJzAUR3c861G43o=; b=yyR9xG+l9+EISzfXdnL5b6vE58JNVZ5HLJlhdaEZQFpeJozJzq+7m0jNR7EvFk2JoHiIn+MfzpcV2yYvSy+KHxaH8mHAF+AYRFOuuXrQc5eQsikkZtdYfF8rPjeLhDhiQeZzIuUA7uCsEnY4LhdodB46ZNavemMO6IVwF6VLNbA= Received: from BYAPR05CA0043.namprd05.prod.outlook.com (2603:10b6:a03:74::20) by PH8PR12MB6724.namprd12.prod.outlook.com (2603:10b6:510:1cf::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.20; Wed, 9 Apr 2025 11:17:19 +0000 Received: from SJ1PEPF00002313.namprd03.prod.outlook.com (2603:10b6:a03:74:cafe::16) by BYAPR05CA0043.outlook.office365.com (2603:10b6:a03:74::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8632.19 via Frontend Transport; Wed, 9 Apr 2025 11:17:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ1PEPF00002313.mail.protection.outlook.com (10.167.242.167) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8606.22 via Frontend Transport; Wed, 9 Apr 2025 11:17:19 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 9 Apr 2025 06:17:08 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Gautham R. Shenoy" , Swapnil Sapkal , "K Prateek Nayak" Subject: [RFC PATCH 4/5] sched/fair: Rework inter-NUMA newidle balancing Date: Wed, 9 Apr 2025 11:15:38 +0000 Message-ID: <20250409111539.23791-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250409111539.23791-1-kprateek.nayak@amd.com> References: <20250409111539.23791-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002313:EE_|PH8PR12MB6724:EE_ X-MS-Office365-Filtering-Correlation-Id: af03fa58-efc4-4b96-728f-08dd77581788 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|7416014|36860700013|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ZM098wXqRsjM863ZqBB7Dxn+Ehxq2bZ66SFmuLYlPJbRc+oJt8QF/ZhI7/qK?= =?us-ascii?Q?uG4EigBv0iEoaDy+ZWTdOieKPJ1ovwmKkjxxytJRyBYGF3vjdZhqyhuflMQt?= =?us-ascii?Q?iiep5yxWE/BZuXjMvJiCocbesJNg0ojv+A5DJJ+SP152cUB4miL9BFNJJG/N?= =?us-ascii?Q?jdx5OmLNqX887oOkoTW5TQXjmZAC6SJ1ljEo1Oog7tUEiJpZs0UxwlOZKHlx?= =?us-ascii?Q?WDG6+AGso8s1e98o1+Jsn++49CkpPziTyF4y65ol71TwwC2oAcdsfE+JUYX7?= =?us-ascii?Q?+YfG1rYWCLhA6x7ui+aBRPakSkVgN0q+sygCqi5YoehPDMQQdQNFk+wPRKkQ?= =?us-ascii?Q?fcSjUJI8sfYu986T/BLADX2Gzd//Yp1Hr/9EgPVwzfSVWXc3IbKP26OIti4c?= =?us-ascii?Q?1nCSPyEhpOOBZgUyujOCTGteObzD3j4gaIvR1DC8k0ILOqUpW1uT74cIqnyj?= =?us-ascii?Q?M8Dlw2Lq9RC7010DMwD9tQrq56kCvU5dpJaOqLhenlX4b6/+PWeYZcEllxmW?= =?us-ascii?Q?ggr0SsDdQ7JyX/AAMQ649AzRUZHdf9rrP8xH+x+jGoo3jmKg5tu8//EQuhNE?= =?us-ascii?Q?ZDxErufEsZlMzsTe84xpcXQeHKLczysNnbKHs62TC7Alui56hB4s2rQZkN3Q?= =?us-ascii?Q?+Yydt+w1CGsrrr4UCSZoeE6TPPulxRhycRM5Jln9BZNNMpS2Ok14xAX5qKih?= =?us-ascii?Q?aaTdKakAwyx5Ga5IlxWyojy9eUyOqqLT4dFFoSEaLYKSmsip+YCLXpTm0MNx?= =?us-ascii?Q?tpmtorYwfdrdAvI8o20218p6/whfnGiF8vizovhA95Bn/OXJqcebx478sovG?= =?us-ascii?Q?QiLjQu+3ze5rpxJ3rVnStEiHw6EMXYQ0eVMrss1XMBTS+JtDT4DGedCsaZ97?= =?us-ascii?Q?D7gnYV/glbptXVyfvcqJRa4d8SAFImXgyOUZeDxGiUBQpMhwJEyA4HlXJva0?= =?us-ascii?Q?UQRxH1HObHjgbjpwKniKPM9ZR6pzYIio4EzG80I8XJRym7nx5GUkApFR970N?= =?us-ascii?Q?PFFCCI98dEASIgDek5aVtOFpdLxaWzcULLEMovfLywPAfrOswl3qz7GuhZP0?= =?us-ascii?Q?d4r2TizQIUD/J4GOxpyCuulTlBOquSqeygv99RUS8fmakFlhrPy4GEF0U1bA?= =?us-ascii?Q?tSNJHs1PHCGf0SgcCI8NK8fXd0U9v+fgfsXvo2dJgYluVRJQF76yICcz7EVI?= =?us-ascii?Q?BU3kwd/qsAid6YoeobBR89FawPxlxKyXMUtBQqXpiJ2lHtLtTlN1Z3p5dqbM?= =?us-ascii?Q?F9xqaXoRFRTxI7M40aMaUaqzm8qJEiw8RIRCFb5GDxTwk1iASPmwM9eXBgdy?= =?us-ascii?Q?evs8AhkIaDwdXp0BVaGvQJCOw62H9/nrlPslAFVZVHTA5jnhah+HlkbQ/eK8?= =?us-ascii?Q?5jPfayYfK7+hQDi1JRCfkoL4vhSGOACaDTDYeXAyXMX9ijlKdY8QF0JtvQGD?= =?us-ascii?Q?X9u9sipWXUS/kobVoGhPhzsXWk2Lw75hDehYREaxabDYzzw8sjK5/A=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(7416014)(36860700013)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2025 11:17:19.0717 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: af03fa58-efc4-4b96-728f-08dd77581788 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002313.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6724 Content-Type: text/plain; charset="utf-8" With the introduction of "overloaded_mask" in sched_domain_shared struct, it is now possible to scan through the CPUs that contain pushable tasks that could be run on the CPU going newly idle. Redesign the inter-NUMA newidle balancing to opportunistically pull a task to the CPU going idle from the overloaded CPUs only. The search starts from sd_llc and moves up until sd_numa. Since "overloaded_mask" is per-LLC, each LLC domain is visited individually using per-CPU sd_llc struct shared by all CPUs in an LLC. Once visited for one, all CPUs in the LLC are marked visited and the search resumes for the LLCs of CPUs that remain to be visited. detach_one_task() was used in instead of pick_next_pushable_fair_task() since detach_one_task() also considers the CPU affinity of the task being pulled as opposed to pick_next_pushable_fair_task() which returns the first pushable task. Since each iteration of overloaded_mask rechecks the idle state of the CPU doing newidle balance, the initial gating factor based on "rq->avg_idle" has been removed. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 129 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 117 insertions(+), 12 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 834fcdd15cac..93f180b67899 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12864,6 +12864,100 @@ static inline bool nohz_idle_balance(struct rq *t= his_rq, enum cpu_idle_type idle static inline void nohz_newidle_balance(struct rq *this_rq) { } #endif /* CONFIG_NO_HZ_COMMON */ =20 +static inline bool sched_newidle_continue_balance(struct rq *rq) +{ + return !rq->nr_running && !rq->ttwu_pending; +} + +static inline int sched_newidle_pull_overloaded(struct sched_domain *sd, + struct rq *this_rq, + int *continue_balancing) +{ + struct cpumask *cpus =3D this_cpu_cpumask_var_ptr(load_balance_mask); + int cpu, this_cpu =3D cpu_of(this_rq); + struct sched_domain *sd_parent; + struct lb_env env =3D { + .dst_cpu =3D this_cpu, + .dst_rq =3D this_rq, + .idle =3D CPU_NEWLY_IDLE, + }; + + + cpumask_and(cpus, sched_domain_span(sd), cpu_active_mask); + +next_domain: + env.sd =3D sd; + /* Allow migrating cache_hot tasks too. */ + sd->nr_balance_failed =3D sd->cache_nice_tries + 1; + + for_each_cpu_wrap(cpu, cpus, this_cpu) { + struct sched_domain_shared *sd_share; + struct cpumask *overloaded_mask; + struct sched_domain *cpu_llc; + int overloaded_cpu; + + cpu_llc =3D rcu_dereference(per_cpu(sd_llc, cpu)); + if (!cpu_llc) + break; + + sd_share =3D cpu_llc->shared; + if (!sd_share) + break; + + overloaded_mask =3D sd_share->overloaded_mask; + if (!overloaded_mask) + break; + + for_each_cpu_wrap(overloaded_cpu, overloaded_mask, this_cpu + 1) { + struct rq *overloaded_rq =3D cpu_rq(overloaded_cpu); + struct task_struct *p =3D NULL; + + if (sched_newidle_continue_balance(this_rq)) { + *continue_balancing =3D 0; + return 0; + } + + /* Quick peek to find if pushable tasks exist. */ + if (!has_pushable_tasks(overloaded_rq)) + continue; + + scoped_guard (rq_lock, overloaded_rq) { + update_rq_clock(overloaded_rq); + + if (!has_pushable_tasks(overloaded_rq)) + break; + + env.src_cpu =3D overloaded_cpu; + env.src_rq =3D overloaded_rq; + + p =3D detach_one_task(&env); + } + + if (!p) + continue; + + attach_one_task(this_rq, p); + return 1; + } + + cpumask_andnot(cpus, cpus, sched_domain_span(cpu_llc)); + } + + if (sched_newidle_continue_balance(this_rq)) { + *continue_balancing =3D 0; + return 0; + } + + sd_parent =3D sd->parent; + if (sd_parent && !(sd_parent->flags & SD_NUMA)) { + cpumask_andnot(cpus, sched_domain_span(sd_parent), sched_domain_span(sd)= ); + sd =3D sd_parent; + goto next_domain; + } + + return 0; +} + /* * sched_balance_newidle is called by schedule() if this_cpu is about to b= ecome * idle. Attempts to pull tasks from other CPUs. @@ -12881,6 +12975,7 @@ static int sched_balance_newidle(struct rq *this_rq= , struct rq_flags *rf) u64 t0, t1, curr_cost =3D 0; struct sched_domain *sd; int pulled_task =3D 0; + u64 domain_cost; =20 update_misfit_status(NULL, this_rq); =20 @@ -12913,28 +13008,34 @@ static int sched_balance_newidle(struct rq *this_= rq, struct rq_flags *rf) rq_unpin_lock(this_rq, rf); =20 rcu_read_lock(); - sd =3D rcu_dereference_check_sched_domain(this_rq->sd); - - if (!get_rd_overloaded(this_rq->rd) || - (sd && this_rq->avg_idle < sd->max_newidle_lb_cost)) { - - if (sd) - update_next_balance(sd, &next_balance); + if (!get_rd_overloaded(this_rq->rd)) { rcu_read_unlock(); - goto out; } rcu_read_unlock(); =20 raw_spin_rq_unlock(this_rq); =20 + rcu_read_lock(); t0 =3D sched_clock_cpu(this_cpu); - sched_balance_update_blocked_averages(this_cpu); =20 - rcu_read_lock(); - for_each_domain(this_cpu, sd) { - u64 domain_cost; + sd =3D rcu_dereference(per_cpu(sd_llc, this_cpu)); + if (sd) { + pulled_task =3D sched_newidle_pull_overloaded(sd, this_rq, &continue_bal= ancing); + + t1 =3D sched_clock_cpu(this_cpu); + domain_cost =3D t1 - t0; + curr_cost +=3D domain_cost; + t0 =3D t1; =20 + if (pulled_task || !continue_balancing) + goto skip_numa; + } + + sched_balance_update_blocked_averages(this_cpu); + + sd =3D rcu_dereference(per_cpu(sd_numa, this_cpu)); + while (sd) { update_next_balance(sd, &next_balance); =20 if (this_rq->avg_idle < curr_cost + sd->max_newidle_lb_cost) @@ -12960,7 +13061,11 @@ static int sched_balance_newidle(struct rq *this_r= q, struct rq_flags *rf) */ if (pulled_task || !continue_balancing) break; + + sd =3D sd->parent; } + +skip_numa: rcu_read_unlock(); =20 raw_spin_rq_lock(this_rq); --=20 2.34.1 From nobody Fri Dec 19 17:37:16 2025 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2048.outbound.protection.outlook.com [40.107.95.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31E7E201270 for ; Wed, 9 Apr 2025 11:17:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.48 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197457; cv=fail; b=idygOAyqnYA+zi4cTFCn70YBgaxRKnjrndYCIsopeQm9SNuA1usjWomnmxvMoKimukSqXQSDPWbX16i8OLJcP3Z2jBGAE5Y9GOi9pzeMK4C3WGNc6etQJXcRO8Kw92T/FDJRycudTTA1PF4/4sxG5H23DlNaGB5btNUrYF29160= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744197457; c=relaxed/simple; bh=W+O2IL7i3QUjqsHTGvLV/M4QESRELmet3e6Clo79x5Q=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=U9Wlew/1sdShKhmAMbT3F2BPdTGDr9FhlPJu4k4BNCe8JPh0Geh8HHxdAKwslrGgYzbc+Ge2+jO2e1G7+jNxoB6S2no1YbP9nb/9n3GReT1vDrOknPDSQxVp2+WwLfk4EZCRgGUxGFY7zmdSuTAkjIxGfQZGfAYeu4zlmWOGLKU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=ZKUhvn/V; arc=fail smtp.client-ip=40.107.95.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="ZKUhvn/V" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=sOUu1c2JH29TQYloEgDH3HzQiXRNVYYRJ2BBRe8q6idnWCnwYWRTKq8WaNSkcGi4lupKGBOgvRW+8scLDndnFEbZWhd+6mf18Rp2x5Kk8BNMda/gNY08qr9u/fViQtNH6yGPDiZhriAI7so4B1xcJ5X/vTJK7tRMlzb/ET4Bo/tuhwSY73liAe8N6DAaSWAsllb4DR0ic6rMT20hVvB3ggGA/YdvNE58P1GotOjNiMkAYc1lzd1rKkm/Leui9qrJWhZ99sIHWpP1oshqfqxqG0A8AWEm/RMb+mWYZwvPdHRCYz7ZbJ4+nOWm600qojznb5AUnNFFaxz4alP/IRQKgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TyHCs102wN12l/Q+mT7p6NHp6uL/MCZiVN4LT5dCC3Y=; b=mg18bQsVm654s8chnolrItNZpvmS+JKR6ecJBOi9/N0fd9OfW1wkLwdpcHsNOV+wBKHxxnSz011gjx67oVy6dMTICpky9ouedYmrbpdzJGkL1OiOsrkTqGfox7ArFDF3CpoUMRItIfg2rQh72BuqknbWbCZ6dwF/Fi0aEccIp1UB8a5dBp1wwMMNArW8ZYHIoI4AdxpOAHLUhuwbM9uuXFcgBMAgR6YOLhAFSyYhDp9XhmLsEkQDRi31aVQ0KPw70rWM9wqjsB+GAqx+ubw/LGwczuCvYz0rRqL56P1WAVt/5n3Jn8sog/W5oERrzDjxmgK0V/eD8Zb113QHpb3PiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TyHCs102wN12l/Q+mT7p6NHp6uL/MCZiVN4LT5dCC3Y=; b=ZKUhvn/Vqzc+48F+jpsG80tYyAYt8HGVsTrShU4A+rn5L2Jmo5jiIhYH5ydHMfs53uAOJlftS3k6eQUxz06a/1iN4C1mZ34x0rdNfbpuAZsrDM8STLvIhceH5NodqH/WEHbd91GO9chVgJ4WEFILphFvI3552aeyFi8InaUXJ0A= Received: from SJ0PR03CA0187.namprd03.prod.outlook.com (2603:10b6:a03:2ef::12) by CY8PR12MB8299.namprd12.prod.outlook.com (2603:10b6:930:6c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8606.37; Wed, 9 Apr 2025 11:17:33 +0000 Received: from SJ1PEPF00002317.namprd03.prod.outlook.com (2603:10b6:a03:2ef:cafe::4c) by SJ0PR03CA0187.outlook.office365.com (2603:10b6:a03:2ef::12) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8606.36 via Frontend Transport; Wed, 9 Apr 2025 11:17:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SJ1PEPF00002317.mail.protection.outlook.com (10.167.242.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8606.22 via Frontend Transport; Wed, 9 Apr 2025 11:17:32 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 9 Apr 2025 06:17:26 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Gautham R. Shenoy" , Swapnil Sapkal , "K Prateek Nayak" Subject: [RFC PATCH 5/5] sched/fair: Proactive idle balance using push mechanism Date: Wed, 9 Apr 2025 11:15:39 +0000 Message-ID: <20250409111539.23791-6-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250409111539.23791-1-kprateek.nayak@amd.com> References: <20250409111539.23791-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002317:EE_|CY8PR12MB8299:EE_ X-MS-Office365-Filtering-Correlation-Id: 72e6b74e-7085-4388-b662-08dd77581fc1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|36860700013|82310400026|13003099007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ufErXVD5lQfMCPX+O/EsE1+bETI2cHdgs15L40UVNk2bAD8HWz9TkvQZ/60f?= =?us-ascii?Q?ktFjqtJtZV6p1DI/ECwjIZP8jOKuLGkJHr0YoY3bmqvMp0lIfu5N4uD+eqvT?= =?us-ascii?Q?YLp2T213MIFcF+22W5OrgH4w7Wk4S2j5rnWxEFTjEjbbRICiswfLXwjiIaXN?= =?us-ascii?Q?iZoPaGaUFrIFmnQ3eQUBWO9ZFRGiFV7w+9GTf3FQ9CcTL0gJKtyY4i4KflTw?= =?us-ascii?Q?c9hWIpToTAcaQjj+PwVk6R/58v4CCr9QaV1xAZF5KfkPEBGytDBjwhqxdZ37?= =?us-ascii?Q?ADBXl8gaidAtSute5S2es8V7IZS0lxwx+2vZ3N1AA+PebMzVZwz+ggwuynwh?= =?us-ascii?Q?pHdA1lewC1o5pIVgcQ76m1T+j0/7ZIR91IeKaXJmuGKPh61iLPs4UcvxGacr?= =?us-ascii?Q?Q8QaPZUOCQUrqLnMbjSa6Wvq/CSbERnIxk8JIZdC2k5M3vknxYDQu4I9TfB2?= =?us-ascii?Q?urRs1k5PF+uuY0C9DtInG4aLYvP/YylvRPgkiMIYot9TTeC2CRq16uoJw7YM?= =?us-ascii?Q?dumRkvRGqsWl6KbXshVIksR6eAqdAGz+SnaCmOCn4JJzGFmV234FlrG+aoF7?= =?us-ascii?Q?MYQISKWq46P2WY4FEBW8iuh50KYXye2ESGxj/Ot8X/XBWOiITJmD+sSThLQO?= =?us-ascii?Q?xqa2F3kD6Ha+Xez98TdWLlhIATq1g1H3cjp+sKMAqXJINptimbfgGpwN0RZc?= =?us-ascii?Q?StqcPCgLBXX3c2LXragMFmxVzLI4MZHRsF9PbN/dw1Sfbukhhs33gn2VqoH3?= =?us-ascii?Q?9uLCuj0nUorBgjms7/1/Bmp1RQ88CnQaK2+sNZ+gr5eDzmE4NAmHcFmWxW20?= =?us-ascii?Q?n3vRhMa3urrrISsb1gjTblM+4I+z884BGZ6mcnu/lIPloomvW4cIGD7skhkH?= =?us-ascii?Q?T37yRC+kKe49j0jFOf0/WEcdxL3v65Wt6nr7tZjcN7m+rM5uue7mDzUa4h7z?= =?us-ascii?Q?b6whWKwXj4hK0igxZ+pqaqi5LBDekvyMWpmx/oH4HafQsdXnIJ97Wh07+ri9?= =?us-ascii?Q?RHZSReqfCCqqIgqADQ3Yk0BUPlm9E3vpvu03uWDRVz6MdQw+P7krLO9aJM7X?= =?us-ascii?Q?lRtY4g3Dq3QrorLgFho/nJzub+4IH6+MSTrPx8qL3swTgC6rh/rxe879hCpd?= =?us-ascii?Q?VxHaRnpZRgP32+68HGBjdfcL9uWL+R7Fuh+W36vRvNapdLbc5cSCe6dl6xro?= =?us-ascii?Q?Rz1gC2Cbo5Po59zhKDPLLWr3NCJnEuS+pmLDlsfX4UXSyjye9F0MIQxvhH59?= =?us-ascii?Q?OoaCVTVbEisly7qND8r9nJvCSpKcIpDPRwEZwpNvCSEwpdMHHpDmkeFdmtjf?= =?us-ascii?Q?2XI5hDMqFD21GD4J9tmrJDkbCS/MIhXuHaQ1uHi3Mfh5eo1WVNZRZNqycAHH?= =?us-ascii?Q?HkKu9zu6p3EuWbSP6u1dBGhsFG1oMphm8DrzH/yLOARH+LJfXCxK4cWnkTZi?= =?us-ascii?Q?OxZ88jrhsAlxYq9dIo2QPWvILpoo0u2gwLQ+7tqeZIDkibsFDVx2SuJf/gd+?= =?us-ascii?Q?bj7wp3oluYD/EXRv7kHUXYxKp6PxpErdUUp+?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(36860700013)(82310400026)(13003099007);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2025 11:17:32.8532 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 72e6b74e-7085-4388-b662-08dd77581fc1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002317.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8299 Content-Type: text/plain; charset="utf-8" Proactively try to push tasks to one of the CPUs set in the "nohz.idle_cpus_mask" from the push callback. pick_next_pushable_fair_task() is taken from Vincent's series [1] as is but the locking rules in push_fair_task() has been relaxed to release the local rq lock after dequeuing the task and reacquiring it after pushing it to the idle target. double_lock_balance() used in RT seems necessary to maintain strict priority ordering however that may not be necessary for fair tasks. Link: https://lore.kernel.org/all/20250302210539.1563190-6-vincent.guittot@= linaro.org/ [1] Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 59 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 93f180b67899..b2b316e75ad0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8596,12 +8596,71 @@ static inline int has_pushable_tasks(struct rq *rq) return !plist_head_empty(&rq->cfs.pushable_tasks); } =20 +static struct task_struct *pick_next_pushable_fair_task(struct rq *rq) +{ + struct task_struct *p; + + if (!has_pushable_tasks(rq)) + return NULL; + + p =3D plist_first_entry(&rq->cfs.pushable_tasks, + struct task_struct, pushable_tasks); + + WARN_ON_ONCE(rq->cpu !=3D task_cpu(p)); + WARN_ON_ONCE(task_current(rq, p)); + WARN_ON_ONCE(p->nr_cpus_allowed <=3D 1); + WARN_ON_ONCE(!task_on_rq_queued(p)); + + /* + * Remove task from the pushable list as we try only once after that + * the task has been put back in enqueued list. + */ + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); + + return p; +} + +static void fair_add_pushable_task(struct rq *rq, struct task_struct *p); +static void attach_one_task(struct rq *rq, struct task_struct *p); + /* * See if the non running fair tasks on this rq can be sent on other CPUs * that fits better with their profile. */ static bool push_fair_task(struct rq *rq) { + struct cpumask *cpus =3D this_cpu_cpumask_var_ptr(load_balance_mask); + struct task_struct *p =3D pick_next_pushable_fair_task(rq); + int cpu, this_cpu =3D cpu_of(rq); + + if (!p) + return false; + + if (!cpumask_and(cpus, nohz.idle_cpus_mask, housekeeping_cpumask(HK_TYPE_= KERNEL_NOISE))) + goto requeue; + + if (!cpumask_and(cpus, cpus, p->cpus_ptr)) + goto requeue; + + for_each_cpu_wrap(cpu, cpus, this_cpu + 1) { + struct rq *target_rq; + + if (!idle_cpu(cpu)) + continue; + + target_rq =3D cpu_rq(cpu); + deactivate_task(rq, p, 0); + set_task_cpu(p, cpu); + raw_spin_rq_unlock(rq); + + attach_one_task(target_rq, p); + raw_spin_rq_lock(rq); + + return true; + } + +requeue: + fair_add_pushable_task(rq, p); return false; } =20 --=20 2.34.1