From nobody Tue Apr 7 20:08:12 2026 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011033.outbound.protection.outlook.com [52.101.57.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C0D62C0F6F for ; Thu, 12 Mar 2026 04:46:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.33 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773290769; cv=fail; b=e5zn6TeEqce1jWPtOfkW2cYtxcMSZlwysqxNo6FkdL7ARJA3P2zLt4FX1HVq8Ut4aCVWTAOlC+WKhZQ+xaQ5n/W4qVCskZMpNnI2dOzymprr4tR4oOig4s5mTp/0VwvBrNdfGHs73AHRHDwTLsvwasxBmizeQTNfhxrF52nGpX0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773290769; c=relaxed/simple; bh=p3uBEeUqyZKjonLGMoKue49qOA/3G9jJwDK9psnr/SQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=k6dRGElZL1ugQR3P6ELslQvb+d9wGKekXypBxeTJE8ao8oXMTY3NODTWpF8Ko/32kS5AQaYaGeh5PI3FlvZ+ru1ZUUre8YalzxnRB9m7FrvISqIDmySe0TP/VMENkgOAOs8bx9WomHhWo46jxlGYlysVontDykICplFzJwF35G4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=x24cqun9; arc=fail smtp.client-ip=52.101.57.33 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="x24cqun9" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wFmUAI7LprAdi27TOqnaaf1376hjfZ5RcZmqBNPVKpqLYJFkZfbBsS+htylRct8pBzhnuTGCmmVsy6lyEK9Gduv9iRr1BarQsuz5XhHSbV3cn7c3cS4eP6eGVLfP+vVO4AGY81x1gzbZT83HWTvL/o1LSjEsPvplNcHi2V56Nxr8HuGt2gtD4Er9OzfngFpBxCMU9KuGAeSGp8i2x0swNT1Wnj2bPano9C6wplWaR5gHxBGoRk9wfbtqCzyVJ2dMIkFC5B5Kq/mI/tHoYiBSnKc3ofKTzObdILohGWoNeAcJTNIzUTUhd7+Y9DvO0cfyCMt+XdCoZeoy4nSSOWSmbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3h1acG/U+8bWFA699D5Axzf2oLX1SjhQTHmRxsQWaVM=; b=Kvri8maX60zTkV8gHQ+afu5gY32t0tYdwibgoZTnZj1uA4MGk5Ny+TuOZOt9zB2hcYPug5KA+DsuC1qpAbXL44bc4uj2gRygf5bLfQAOc+VubZLbfrzz3oh9narUP2rr3+DHSR8sM0m9AxnBv6Uidt3eJ3LkcBShEYEioEJA4uxB0bnEbjZRTTXBQqzRIIToqRZXgHdHGzREyPzdPwlWk2+E/cwuZDFQMib8Ah25/b5EH81UFEsUTbGW1Y4oKipR2tIXN1hXlXPopLI+iwgfu0wMZ7J+T5HR/zZX3QR//fgLZmhkDlTDZ9MAI25yt8mNWwK0Sp/nDoe4rue9VHtfaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3h1acG/U+8bWFA699D5Axzf2oLX1SjhQTHmRxsQWaVM=; b=x24cqun9H+dwG1ARB1ky8GUsNrK/U6W/usuCz+/eMufYnLtUaX6XnKSPHGQtwHIQ1y7Bo7WMJBKMNbudM404O9IRQ9W8bSA1RzdhR6bsS2KLnkpucQk4fwU0SOgi2yZcwQhGkcnZ49ukgrYSSOEL8++bn9Xt+o3o76vU2OVIO4Q= Received: from BY3PR05CA0058.namprd05.prod.outlook.com (2603:10b6:a03:39b::33) by DS0PR12MB7747.namprd12.prod.outlook.com (2603:10b6:8:138::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9723.6; Thu, 12 Mar 2026 04:46:02 +0000 Received: from MWH0EPF000C6187.namprd02.prod.outlook.com (2603:10b6:a03:39b:cafe::c6) by BY3PR05CA0058.outlook.office365.com (2603:10b6:a03:39b::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9678.24 via Frontend Transport; Thu, 12 Mar 2026 04:46:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MWH0EPF000C6187.mail.protection.outlook.com (10.167.249.119) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9678.18 via Frontend Transport; Thu, 12 Mar 2026 04:46:02 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Wed, 11 Mar 2026 23:45:56 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Valentin Schneider , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Chen Yu , Shrikanth Hegde , Li Chen , "Gautham R. Shenoy" , K Prateek Nayak Subject: [PATCH v4 4/9] sched/topology: Switch to assigning "sd->shared" from s_data Date: Thu, 12 Mar 2026 04:44:29 +0000 Message-ID: <20260312044434.1974-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260312044434.1974-1-kprateek.nayak@amd.com> References: <20260312044434.1974-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000C6187:EE_|DS0PR12MB7747:EE_ X-MS-Office365-Filtering-Correlation-Id: 06349fca-5f73-4d9b-d44a-08de7ff24386 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700016|82310400026|7416014|376014|22082099003|56012099003|18002099003; X-Microsoft-Antispam-Message-Info: Xw+ibPFyQIl6ccrjxTKtV8VTCmMhCGBCAIxSOy4/1/nvs+s7Ntv6bqJQW23GQ1CZDj5XZU8Cv0dcXHg6LoY/f82UCkoEMueZ+pAH6aScqnjWGdKnTOyI8VXZZoRzlNm5eaAbCILgi0tL/MJlwDDSyCflYepG41TmVeFybYeXAfqIoqwWtRqjShwOmU9bj15fy5n5TBPHh343h3xqKjT4FlX/BIVvAuI3B/FtSPn5qQYL1J+uXeC1KMAxP/ddJr9sLJedHMDjYFzxOUgWWhIVczJTvsrMbTAf8bIa3dxXDKzpaQ2LtaCgz6vAIiQ79AO9hg45Mo8RIHh1j2/hk4U6Qze+8zORJBIbGpW+KsSAhz7mx8aLO7o2KbEchYHZBBkQEAX81xN9SqbPyJi3ckE1IEr5j+POV/JkqhWHGNRSRcE6K1+IV3VJK3O7S87bCob4J9iTvXX26dbzdw4nX+yXkWbHaOjuXCtZG6uYLERbbX6gXXwInyASvaz9gTOnZzjzwGJJiH3azMbJQo9XdnJRMEEpW8DDJWMVwfoa4JOzsuDpets1E7ctQbYVCPAWdtKTEt3rkKzQg7NfzwkW5vOCZqe7zsy4dpGRP3RfCHkAMjBTVDvvoottvLHRnBUQNlyuYHoJpSKj6PWMC72YHQUN22smn0gcLR/xq3QHeYUkATI4cNZ3p4CwBUWTRfFvda5uHS5/aaLGTySbn/jmA1ryrLd7ANsZVVI6Pvo/FDnTRp+ZQLq3+kKUWixc6fRrJM96TsgwS7qQY4vL8G/uKscQ0w== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700016)(82310400026)(7416014)(376014)(22082099003)(56012099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ZXZkBLRDQ7CTRur6lqmhRbTs6FIrfCv91QjPpSpjgtxwTfdcr+XtkhQ7LY3fki01L3xPm/H3QtS3RMpcHUE/cB+gofN1P7c6eMbX5nOfDxJPQz/v3dCuftqir+1DhlJFJQD1QLLgxMvi35mwbnERFVBDAG12F689mIEQbHi0URWc09w5EyI44AfEZsP7tetOyRCP2m2Nu2aNoT3V9EPBQWpbtz6Kf/p+ckWF0ePiFwwEEMn+QhTjQBPPHQxUxfLkQmY+RoOcZjy3uRrSz7QEDmhE5qzzuHklFQc5DH/jzxLx366oVEYJ6xkmMHRnTHY/AiSNwbN6tHqRq+AU1+Asehljy2u5WsXv6b1BAp4cxALXFJRgO/PAFRZFlO7qCzmV8wPpjbTDOnhH8bgKrkyO2S18QSU8nul1qnXVWthK7qFu8d6NoXDlx7gLXeExSfCX X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Mar 2026 04:46:02.3571 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 06349fca-5f73-4d9b-d44a-08de7ff24386 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000C6187.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7747 Content-Type: text/plain; charset="utf-8" Use the "sched_domain_shared" object allocated in s_data for "sd->shared" assignments. Assign "sd->shared" for the topmost SD_SHARE_LLC domain before degeneration and rely on the degeneration path to correctly pass down the shared object to "sd_llc". sd_degenerate_parent() ensures degenerating domains must have the same sched_domain_span() which ensures 1:1 passing down of the shared object. If the topmost SD_SHARE_LLC domain degenerates, the shared object is freed from destroy_sched_domain() when the last reference is dropped. claim_allocations() NULLs out the objects that have been assigned as "sd->shared" and the unassigned ones are freed from the __sds_free() path. To keep all the claim_allocations() bits in one place, claim_allocations() has been extended to accept "s_data" and iterate the domains internally to free both "sched_domain_shared" and the per-topology-level data for the particular CPU in one place. Post cpu_attach_domain(), all reclaims of "sd->shared" are handled via call_rcu() on the sched_domain object via destroy_sched_domains_rcu(). Signed-off-by: K Prateek Nayak --- Changelog v3..v4: o Moved claiming the per-CPU "d.sds" reference into claim_allocations() to keep everything in one place. (Shrikanth) o Slightly different diff as a result of moving the "imb_numa_nr" calculation into a separate helper in Patch 2. --- kernel/sched/topology.c | 73 +++++++++++++++++++++++++---------------- 1 file changed, 44 insertions(+), 29 deletions(-) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index f0541c6511fa..ebd955faab40 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -685,6 +685,9 @@ static void update_top_cache_domain(int cpu) if (sd) { id =3D cpumask_first(sched_domain_span(sd)); size =3D cpumask_weight(sched_domain_span(sd)); + + /* If sd_llc exists, sd_llc_shared should exist too. */ + WARN_ON_ONCE(!sd->shared); sds =3D sd->shared; } =20 @@ -733,6 +736,13 @@ cpu_attach_domain(struct sched_domain *sd, struct root= _domain *rd, int cpu) if (sd_parent_degenerate(tmp, parent)) { tmp->parent =3D parent->parent; =20 + /* Pick reference to parent->shared. */ + if (parent->shared) { + WARN_ON_ONCE(tmp->shared); + tmp->shared =3D parent->shared; + parent->shared =3D NULL; + } + if (parent->parent) { parent->parent->child =3D tmp; parent->parent->groups->flags =3D tmp->flags; @@ -1586,21 +1596,28 @@ __visit_domain_allocation_hell(struct s_data *d, co= nst struct cpumask *cpu_map) * sched_group structure so that the subsequent __free_domain_allocs() * will not free the data we're using. */ -static void claim_allocations(int cpu, struct sched_domain *sd) +static void claim_allocations(int cpu, struct s_data *d) { - struct sd_data *sdd =3D sd->private; + struct sched_domain *sd; + + if (atomic_read(&(*per_cpu_ptr(d->sds, cpu))->ref)) + *per_cpu_ptr(d->sds, cpu) =3D NULL; =20 - WARN_ON_ONCE(*per_cpu_ptr(sdd->sd, cpu) !=3D sd); - *per_cpu_ptr(sdd->sd, cpu) =3D NULL; + for (sd =3D *per_cpu_ptr(d->sd, cpu); sd; sd =3D sd->parent) { + struct sd_data *sdd =3D sd->private; =20 - if (atomic_read(&(*per_cpu_ptr(sdd->sds, cpu))->ref)) - *per_cpu_ptr(sdd->sds, cpu) =3D NULL; + WARN_ON_ONCE(*per_cpu_ptr(sdd->sd, cpu) !=3D sd); + *per_cpu_ptr(sdd->sd, cpu) =3D NULL; =20 - if (atomic_read(&(*per_cpu_ptr(sdd->sg, cpu))->ref)) - *per_cpu_ptr(sdd->sg, cpu) =3D NULL; + if (atomic_read(&(*per_cpu_ptr(sdd->sds, cpu))->ref)) + *per_cpu_ptr(sdd->sds, cpu) =3D NULL; =20 - if (atomic_read(&(*per_cpu_ptr(sdd->sgc, cpu))->ref)) - *per_cpu_ptr(sdd->sgc, cpu) =3D NULL; + if (atomic_read(&(*per_cpu_ptr(sdd->sg, cpu))->ref)) + *per_cpu_ptr(sdd->sg, cpu) =3D NULL; + + if (atomic_read(&(*per_cpu_ptr(sdd->sgc, cpu))->ref)) + *per_cpu_ptr(sdd->sgc, cpu) =3D NULL; + } } =20 #ifdef CONFIG_NUMA @@ -1740,16 +1757,6 @@ sd_init(struct sched_domain_topology_level *tl, sd->cache_nice_tries =3D 1; } =20 - /* - * For all levels sharing cache; connect a sched_domain_shared - * instance. - */ - if (sd->flags & SD_SHARE_LLC) { - sd->shared =3D *per_cpu_ptr(sdd->sds, sd_id); - atomic_inc(&sd->shared->ref); - atomic_set(&sd->shared->nr_busy_cpus, sd_weight); - } - sd->private =3D sdd; =20 return sd; @@ -2731,12 +2738,20 @@ build_sched_domains(const struct cpumask *cpu_map, = struct sched_domain_attr *att while (sd->parent && (sd->parent->flags & SD_SHARE_LLC)) sd =3D sd->parent; =20 - /* - * In presence of higher domains, adjust the - * NUMA imbalance stats for the hierarchy. - */ - if (IS_ENABLED(CONFIG_NUMA) && (sd->flags & SD_SHARE_LLC) && sd->parent) - adjust_numa_imbalance(sd); + if (sd->flags & SD_SHARE_LLC) { + int sd_id =3D cpumask_first(sched_domain_span(sd)); + + sd->shared =3D *per_cpu_ptr(d.sds, sd_id); + atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); + atomic_inc(&sd->shared->ref); + + /* + * In presence of higher domains, adjust the + * NUMA imbalance stats for the hierarchy. + */ + if (IS_ENABLED(CONFIG_NUMA) && sd->parent) + adjust_numa_imbalance(sd); + } } =20 /* Calculate CPU capacity for physical packages and nodes */ @@ -2744,10 +2759,10 @@ build_sched_domains(const struct cpumask *cpu_map, = struct sched_domain_attr *att if (!cpumask_test_cpu(i, cpu_map)) continue; =20 - for (sd =3D *per_cpu_ptr(d.sd, i); sd; sd =3D sd->parent) { - claim_allocations(i, sd); + claim_allocations(i, &d); + + for (sd =3D *per_cpu_ptr(d.sd, i); sd; sd =3D sd->parent) init_sched_groups_capacity(i, sd); - } } =20 /* Attach the domains */ --=20 2.34.1