From nobody Sun Dec 14 21:43:38 2025 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2076.outbound.protection.outlook.com [40.107.93.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 766EE184E for ; Mon, 23 Dec 2024 04:41:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734928894; cv=fail; b=uv12sTOHV69YXVZ9AVIgxRJpEr4nI9lcu3F8Fyzo6fimVKHyM/AIxjtx7fImwov6q3CTKgrM4RFoqvt3sykA+34PrEgvWrXND5lkL0QqTmMnMknnaysPQZgeyoUbkdJm4D+qcxFrJeZd5fn80kN3fiNkKwdcQd2cRDKCcRudq2I= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734928894; c=relaxed/simple; bh=gN1U6DBHgwDGVHX/EdBl3dw0lPUjb2E4YS+51Pcg1Jo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MnaCuRI1LJUsJVVJKy7DE4B/afe+ZaSbSadC+ifIIkEEYDrhCIUsubj72KyK55UxCrLic5evD5UZEHqM4xS3nJ7uEPcO9JZDdAc+WgU7wrolVipHtZO7rR4ULGC11e3d12AC0KLoN0nzFTeXv+8OwbY58LevhDi1/pkKd4Y4I/M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=VrVW957W; arc=fail smtp.client-ip=40.107.93.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="VrVW957W" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ayjMhuJ7waltUiXEL1qqjhfdOKciIvA8w3fLQ5D1MDB3rY400D2GnQZsV9+p6S/8snzV6U/FmxxrdW9LXYoxpyOq96NJgTp05PKEECrAbgKJbTc7fcTkwIzv5yuUekLCZjztVDeSya5Y1t5dKROKctRHKUriWPTDAgWseXpSzztIINesizcEHkmZ7XP79t83Os5DoRcMmqxNqVQYcJE7eZSkArFakND5DVj1KikoMGIhlF7oEl7nIWVxdaEgx0MmHZSJQa8D6D5LWHlgT9HdixBXsJE7HFwwseExfw7yEp4vcn5HSQUUDr6ePYSA6HeqXfv+KK7fY8gdFJFtghd8Tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BixxblLDKqQJxFWIqT8tTUhjt0vilXurJ1jZwrjGes0=; b=Wjjddlb+wYde48gBIG+bz8XYQ7SQJa/SeIfP1HN6QgOa66T4b8BaMcQLBFeHlhB04bgcG75kiE7SXRe/+qUDY1buTKdABoYp5sasCE8fKFHQwtLUfIDhfBOQqMVBMahg9pzuQIbMkUncAwp5fmfAkjXXZi3sQfkBcTpw5a+X0BjFA/uGziy8/JlLqqjlbmIJpUD2AYyTgBMwUW2JEUtkR0JavzmeQ1fW2oblvIXkAh+6R+RF8fGP1BbzRYnl8juTrfQ/u/qV6YuHdPVCREbufaPRc0Tai2u0u2AHOZksU+3vsNtnjnPec5LJzx9Y2L3+K0ZKEK1bK2F4mXwIHr23Cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linutronix.de smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BixxblLDKqQJxFWIqT8tTUhjt0vilXurJ1jZwrjGes0=; b=VrVW957WQuiBwg/B4NfO4AbcNQHwpMly/7JKktyouie+PeA5k0J2NrkMPyb9WoAv9sWfjcSAsS2mPp7YOX+5fd/v6doPwJWCvLJssGh6EX0qIM4OGmW45a3DoM3eIC2w3wKAUWy/womsTJY4ejr/oQYtNayn8WuFfJBk63R/nYQ= Received: from SJ0PR13CA0128.namprd13.prod.outlook.com (2603:10b6:a03:2c6::13) by CH3PR12MB7595.namprd12.prod.outlook.com (2603:10b6:610:14c::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8272.20; Mon, 23 Dec 2024 04:41:24 +0000 Received: from MWH0EPF000989EC.namprd02.prod.outlook.com (2603:10b6:a03:2c6:cafe::4d) by SJ0PR13CA0128.outlook.office365.com (2603:10b6:a03:2c6::13) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8272.12 via Frontend Transport; Mon, 23 Dec 2024 04:41:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by MWH0EPF000989EC.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8293.12 via Frontend Transport; Mon, 23 Dec 2024 04:41:23 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 22 Dec 2024 22:41:13 -0600 From: K Prateek Nayak To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Peter Zijlstra , Juri Lelli , Vincent Guittot , , CC: "H. Peter Anvin" , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , "Rafael J. Wysocki" , Ricardo Neri , Tim Chen , Shrikanth Hegde , "Mario Limonciello" , Meng Li , Huang Rui , "Gautham R. Shenoy" , "K Prateek Nayak" Subject: [PATCH RFC v2 8/8] sched/fair: Uncache asym_prefer_cpu and find it during update_sd_lb_stats() Date: Mon, 23 Dec 2024 04:34:07 +0000 Message-ID: <20241223043407.1611-9-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241223043407.1611-1-kprateek.nayak@amd.com> References: <20241223043407.1611-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000989EC:EE_|CH3PR12MB7595:EE_ X-MS-Office365-Filtering-Correlation-Id: 2c69ff96-1112-40cf-9609-08dd230c0db9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ydoGaZs4e1sgrftLrinaD6SGqDy5mnrF5Gsp8lyJVMlLpW5ozzUXLbhBuuqS?= =?us-ascii?Q?oc4aZBnmZyioKbW3dlb7naBXKWjvVA+YQNDwXx6an65fsa4mQF0mq54Kgx7Y?= =?us-ascii?Q?hDhXUWEDUx0bEz4B2NPkHb50+j8cHtoDfIBIn51I/S5Is2TK7N2Us+Pt0kT3?= =?us-ascii?Q?QWrhcU9WCYHTyCIxcSty7EoQYQ4zQVJSJtqq5PFTRn7MqHbD2hzyZ+tCSZLD?= =?us-ascii?Q?XYp9ZIf/Tq1d/tekpFJjniLCrbJ8vHpk9A+T3fKJErVTrg+dvYehx/H6DGLL?= =?us-ascii?Q?erjQlV8rXkZHR8J8bbPrdDaHGFYLqCxW+yn6GKQejVOTLabDZLvEXGXsvAFr?= =?us-ascii?Q?1iImNmine3XpVAK8WyalqAdKi2jcFnHNR97VAcA/FfVV826sGSh8qKVSfHu1?= =?us-ascii?Q?4hsiDi5eq5/BRDbnfxyBmHpUeWPebZMHeff3G557pcO0vQTm9SJLN7tU/ZIG?= =?us-ascii?Q?1GoSyGIQEGe8TrKK+CrAt/IRpi1mk/LxB9JbXng3WaJZLrGm7SDkjhtel1fM?= =?us-ascii?Q?p2S6LIIaFRhETkI5Gp2MXkKtxpARcjRRwsaW4G2sjFDpqZ8L1iXtnq1yPCZb?= =?us-ascii?Q?/nuHjMd9OOm6sfIVtssdOnl1/+clYozzgJMwrPB8HvR2R+kr8yNNiTx7vsF5?= =?us-ascii?Q?HOB1Tc1aVE/+06JANoz75J7cskI4Jw+Uz/pIO4J1v1rTVASvk07ahHh/C+fB?= =?us-ascii?Q?hPwPtERu1UTl4odPKQ9VAO07t5OC/FTrEpwM013wF7tYWhD3om95yagKSaYC?= =?us-ascii?Q?pO0OT6rnfGLBLMR//xIaUePmRx9IWaJLW67Vi8m0kuPAXRiTAO3WtakKaivl?= =?us-ascii?Q?8/F46K6G8yNa1unph9xV4EtoSV1nh+vo5mAre6g9Qyfvh0azgQZoP/YU/AAc?= =?us-ascii?Q?cL/apF97LtnuCp/vuDQJVnCSjPRg4huHnO28Zk9J29bO6SuIehuHRIz2ibWb?= =?us-ascii?Q?by9zQm0e1eF5SpLtv64Bkn/gMoPMTNB8F+ffdsKmOwM6bXve0sv+tVNaL8Dy?= =?us-ascii?Q?vhPtSSNyTNTYUeHRDJ8onN+OqC7pMBKzExmSNc/aZd5rl9LUyemGi1I775Q7?= =?us-ascii?Q?LEGEo0bAxE2sO2Ngt8U71vSPFfmJuhDgdo+15HmqZdVajVIurx24Q2wDTxo1?= =?us-ascii?Q?RSRlQx7BEdw0KZz15jj3B9qZihyeFlFk006DcDCwmenIuj/3GV6d5OG6QCTQ?= =?us-ascii?Q?3h7+64UXKt0ulFh2NjN8uWb/V4Fq2tztGmxmFjeAqoVUHaVSYFBDvCodyGpa?= =?us-ascii?Q?XeCcrPh23uppsVxYuIa9rrKOAV31bIC/vrKeeTYcvMu0ZtVagx/jLEmsm5QZ?= =?us-ascii?Q?LNs/JmSThltw8cSAsbW1bu1RtN/84YHAzK6+/5V6RPce3n3THaa+uJ6zI4Ho?= =?us-ascii?Q?N0hH3wblI/hXNNVogIyFoxc5iIVsjhKqKNDAlKXuadB4iheyDuxKI5LUVUlp?= =?us-ascii?Q?EP/e+4WQEEt0cpgZTZd8Vh+/ylSj75cSn8GFNXgmEwGuUvExygj6BglMek2/?= =?us-ascii?Q?YB6XCiSrJYJPmcY=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Dec 2024 04:41:23.1762 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2c69ff96-1112-40cf-9609-08dd230c0db9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000989EC.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7595 Content-Type: text/plain; charset="utf-8" On AMD processors supporting dynamic preferred core ranking, the asym_prefer_cpu cached in sched_group can change dynamically. Since asym_prefer_cpu is cached when the sched domain hierarchy is built, updating the cached value across the system would require rebuilding the sched domain which is prohibitively expensive. All the asym_prefer_cpu comparisons in the load balancing path are only carried out post the sched group stats have been updated after iterating all the CPUs in the group. Uncache the asym_prefer_cpu and compute it while sched group statistics are being updated as a part of sg_lb_stats. Fixes: f3a052391822 ("cpufreq: amd-pstate: Enable amd-pstate preferred core= support") Signed-off-by: K Prateek Nayak --- v1..v2: o Modified the layout of "struct sg_lb_stats" to keep all the SD_ASYM_PACKING fields together (Srikanth) --- kernel/sched/fair.c | 21 +++++++++++++++++++-- kernel/sched/sched.h | 1 - kernel/sched/topology.c | 15 +-------------- 3 files changed, 20 insertions(+), 17 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f27d7d36a78d..f088dd4a527c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9920,6 +9920,8 @@ struct sg_lb_stats { unsigned int group_weight; enum group_type group_type; unsigned int group_asym_packing; /* Tasks should be moved to preferred CP= U */ + unsigned int asym_prefer_cpu; /* Group CPU with highest asym priority */ + int highest_asym_prio; /* Asym priority of asym_prefer_cpu */ unsigned int group_smt_balance; /* Task on busy SMT be moved */ unsigned long group_misfit_task_load; /* A CPU has a task too big for its= capacity */ #ifdef CONFIG_NUMA_BALANCING @@ -10249,7 +10251,7 @@ sched_group_asym(struct lb_env *env, struct sg_lb_s= tats *sgs, struct sched_group (sgs->group_weight - sgs->idle_cpus !=3D 1)) return false; =20 - return sched_asym(env->sd, env->dst_cpu, group->asym_prefer_cpu); + return sched_asym(env->sd, env->dst_cpu, sgs->asym_prefer_cpu); } =20 /* One group has more than one SMT CPU while the other group does not */ @@ -10330,6 +10332,17 @@ sched_reduced_capacity(struct rq *rq, struct sched= _domain *sd) return check_cpu_capacity(rq, sd); } =20 +static inline void +update_sg_pick_asym_prefer(struct sg_lb_stats *sgs, int cpu) +{ + int asym_prio =3D arch_asym_cpu_priority(cpu); + + if (asym_prio > sgs->highest_asym_prio) { + sgs->asym_prefer_cpu =3D cpu; + sgs->highest_asym_prio =3D asym_prio; + } +} + /** * update_sg_lb_stats - Update sched_group's statistics for load balancing. * @env: The load balancing environment. @@ -10352,6 +10365,7 @@ static inline void update_sg_lb_stats(struct lb_env= *env, memset(sgs, 0, sizeof(*sgs)); =20 local_group =3D group =3D=3D sds->local; + sgs->highest_asym_prio =3D INT_MIN; =20 for_each_cpu_and(i, sched_group_span(group), env->cpus) { struct rq *rq =3D cpu_rq(i); @@ -10365,6 +10379,9 @@ static inline void update_sg_lb_stats(struct lb_env= *env, nr_running =3D rq->nr_running; sgs->sum_nr_running +=3D nr_running; =20 + if (sd_flags & SD_ASYM_PACKING) + update_sg_pick_asym_prefer(sgs, i); + if (cpu_overutilized(i)) *sg_overutilized =3D 1; =20 @@ -10486,7 +10503,7 @@ static bool update_sd_pick_busiest(struct lb_env *e= nv, =20 case group_asym_packing: /* Prefer to move from lowest priority CPU's work */ - return sched_asym_prefer(sds->busiest->asym_prefer_cpu, sg->asym_prefer_= cpu); + return sched_asym_prefer(busiest->asym_prefer_cpu, sgs->asym_prefer_cpu); =20 case group_misfit_task: /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index aef716c41edb..a3f0d326bd11 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2047,7 +2047,6 @@ struct sched_group { unsigned int group_weight; unsigned int cores; struct sched_group_capacity *sgc; - int asym_prefer_cpu; /* CPU of highest priority in group */ int flags; =20 /* diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 9c405f0e7b26..20aa087710f0 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1302,7 +1302,7 @@ static void init_sched_groups_capacity(int cpu, struc= t sched_domain *sd) WARN_ON(!sg); =20 do { - int cpu, cores =3D 0, max_cpu =3D -1; + int cpu, cores =3D 0; =20 sg->group_weight =3D cpumask_weight(sched_group_span(sg)); =20 @@ -1314,19 +1314,6 @@ static void init_sched_groups_capacity(int cpu, stru= ct sched_domain *sd) #endif } sg->cores =3D cores; - - if (!(sd->flags & SD_ASYM_PACKING)) - goto next; - - for_each_cpu(cpu, sched_group_span(sg)) { - if (max_cpu < 0) - max_cpu =3D cpu; - else if (sched_asym_prefer(cpu, max_cpu)) - max_cpu =3D cpu; - } - sg->asym_prefer_cpu =3D max_cpu; - -next: sg =3D sg->next; } while (sg !=3D sd->groups); =20 --=20 2.43.0