From nobody Thu Dec 18 07:19:55 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2043.outbound.protection.outlook.com [40.107.223.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 88BC9261398 for ; Thu, 13 Mar 2025 09:38:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.43 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858712; cv=fail; b=lXpMkP+H3esSDMk7mvuEvRwiS1mHjzj1uYJKiO3DJwSdsROInXtu1dKBGKKF8LbbjxhIK6+ghtCTjTPakoPzCJ+EH9vF3JTif/ZI2HfRJYWOVzR1xF+CE+Xp8s5kbJ5qbftlXrN04tRYJNgcwosrlsLPUJA7Svc+X0MOKiQJyNI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858712; c=relaxed/simple; bh=hMracf4IVfkIta8/vH3i+2iTox5gMxK6J6BsYLu1hPA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=h35bUicDM5YPLi6+YGsfJ/lmxPY1Nzj4hLMQjawHjCrwWFR5cR32KICHAVQBH9tD2ra5rRozS7hLgA05XIzskHlb4ywgocXR43Y57onExeGogDSDCzp/Tbw2fzCDk0RJ4J3k4D/LPJ417NvA/+kXfYo3kk6EiiM+HCY8cqXFEvA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=PieYz8Ry; arc=fail smtp.client-ip=40.107.223.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="PieYz8Ry" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=w1syEnKUfF6RIdYrGdc4TckxN/IxgTi9zP0ZUkPDtKGEisuPolb1rIZLFINssKWoy/rOfmo+rwPXBK5dwhlVPcPLZ/tkBSIAV4e6UpXaykZtWLViW+3wWAC7PzzzdnVZoUJ9Cbgn070nsmMjvOvtkdrYrfYqJRi0MsfODTSoPXMVbexbTTRLaMZ8Yfn7RBRPKePeGThOStWIajyjOIrhn6c1uR1TyCpneMn3RJ/hCqTMDdPx5j1aEvcCFzwoOnGpfVP3qw6SsRGkN7TQqB9Hidmx3b60G2W8WFTru6OMaiojHSM+g2uTl6ObJIhOMsP7X8I/4sS3bTTKCOI8giBOwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vPbogZ8WRr8yFGfyNLdX3Jm1z7KxNTmPs3zOkHDfaXA=; b=kS2FSJ8dQ2+xMuBrrt1mblE/MEUuBPXfU/Jfkzj4jHR/Rxsbq8GXD7Uta1kSWLFwmwueJYwgG2Kl4y1MyB/xK+NMa8Oi0KBmlpov4Jk1BLm1G4jZX8pTDgNdYSwV/pTAo3JtDgBJfVT0rYAXPyNLZurcuybzyWCm+Wpzkc45KK6TLSD6nvUENfmIrjuht87twPmUwNXpEtkk+4/2in/Tyw1GyzWtTM+DwNIOR6ipVdZf3LZEx4Psn6l0WvrYsD6jZuoL75oTYxiHoCeV/VwOXiw3IuLVvxK/dB1Xxgok1weQbBKiXMiblaV9c7LXEYmH/Q8lvoSV0VsA3EOOIlIr+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vPbogZ8WRr8yFGfyNLdX3Jm1z7KxNTmPs3zOkHDfaXA=; b=PieYz8RyMliYcU6qcOCDvv20wp745rGaI3KsZEJ1ryzyeLZdF0ie5eVfXoBnC1hhIN7kcx1VO75nNu82BGn1vZ/xC4gC9+VYnqMM3WjxnFJb4f6ifuf9dQzj0EIInmncfbmYKXaIM5d2bk5xd/+NWWuDzaRhhzyNW+Dah0UVldk= Received: from CY5PR15CA0236.namprd15.prod.outlook.com (2603:10b6:930:66::8) by CH2PR12MB4054.namprd12.prod.outlook.com (2603:10b6:610:a6::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.22; Thu, 13 Mar 2025 09:38:25 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2603:10b6:930:66:cafe::24) by CY5PR15CA0236.outlook.office365.com (2603:10b6:930:66::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8511.26 via Frontend Transport; Thu, 13 Mar 2025 09:38:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:38:25 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:38:20 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 1/8] sched/topology: Assign sd_share for all non NUMA sched domains Date: Thu, 13 Mar 2025 09:37:39 +0000 Message-ID: <20250313093746.6760-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|CH2PR12MB4054:EE_ X-MS-Office365-Filtering-Correlation-Id: 481d6b76-08f8-4747-b896-08dd6212cdab X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|36860700013|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ahfu0gDLAu29zi8PscIcvi1BeNiiEqHxfxG1BAlpZTZ+kYMe3Oh1BzivMV4Y?= =?us-ascii?Q?70cgs6ScvzgUejCvKOR8PRtHvBtv1JR0nd7rOwYUmkPKYLBuqAAm30j7vRyQ?= =?us-ascii?Q?ajfBrlXS6JXvm6n1sSm7tI3OZu1BefoIqbLwE5RMQRlnFoZNgmL1zxalo7Gb?= =?us-ascii?Q?58veaj8tiGyOBUJon2gFRLX4XAKMLZHNn9qvfOtx7T9OUV/plQ9If78Qa+14?= =?us-ascii?Q?GmsKkz+qDWAbpAQ9JsNastYWCcoQ9Of0aWFNLcKrPDJ1GKNyy/42kjM5b0pq?= =?us-ascii?Q?gi1AE2GIj2xwLk7t9Ui+FVuEGVwMF1kZDsZDrW8mlycnIINqDaFDaU63E8JB?= =?us-ascii?Q?6STLezTA3kMoTkvEOgXmq6Jq7clfjeIQ1rFRIM1kqubktSp2Vo5i7kNavikb?= =?us-ascii?Q?wjrRgXidhrTCkTZTsJlhnWfWDpqOUkw5HJlp5W7iGPdXktPiBDF/mICtTcOv?= =?us-ascii?Q?b8kVEENe/C1kJRf6WKetbjyJBP5Iv2+XhkP2X5ltYU5cLVFcWRGlgCsjBFAd?= =?us-ascii?Q?edpQeqfWQwfhBwIQGv7tqKOfA1ZeG2HB+LB1uFLNBD1D4iXowIYCxOSNWkZv?= =?us-ascii?Q?Zym+CGfekHzKIuM4Kv7jfQLJ1V3+IOwieq6Tgx7xeeKEZLdiVbQeIuhR2tK5?= =?us-ascii?Q?o8IHkxaIradN3dq/KpPmnwNJopJDMywzxvIEDMO7rg34+IpvjvYbQkY2kaDB?= =?us-ascii?Q?brRkfqfF8StBYC7Jq08uKrl82JY2rH4jBqYBFmvhsGG3vhRzVWmq+RKkIF1u?= =?us-ascii?Q?GOJdPRKfiju2L9eKRlzaLa98ZGa8hYRfJHRjScautL9gaLPV0Q2DZ7uNSGrE?= =?us-ascii?Q?pfXN5f8k5BUAQwVQ6d15aqq0rpSgFQZw+NBLZEHs2wn9GPDjqvwxjBilbn0z?= =?us-ascii?Q?6NJwXjoZFvrzkIPuLBuai9DK6ABKc9MaZbRpfQIEpjyMiSy7yPg+TBpoqEc2?= =?us-ascii?Q?wba34zl75OpkQnAoIgZhU3+/ZvS0CJqVNm0WrAzr2Ca/w3cSgZSCS56Y+7oq?= =?us-ascii?Q?mpgnuWfLy0AIxkROPwstQbNSEWwraujJRgLPq51kxNMOit8TS5CCsvRbgO8j?= =?us-ascii?Q?FNniNh5N7z/5YlpF1Ru65BE+jcBTjp9Hzc5kFNUx7r13EEhWYL2ysuuKCwzi?= =?us-ascii?Q?IXJYHy5cY8CNgC1/FkK/PkGeCOKBlyFke/Pz7ANNPmo+h7OVy5wR725zX8sq?= =?us-ascii?Q?ckb7GexiE6pXX5UE++eTp/B/Q+BrVWtni44hdcLrA4htjPotoYdEe/RxZ9hq?= =?us-ascii?Q?Wr7qwlxM08tZG9K64TGT8NNhveIjW2qO8Z64bOj8U8ttZ7AaNshEQIQQrSUg?= =?us-ascii?Q?bSGS3fTCgU+Q6xa8DEdMxNYFoEBemhPfBHVbZD0V1nJzEX2ctLPYrr6MK+g0?= =?us-ascii?Q?t8Z2owJj3Udr4WE0vTklilMTFRS462ONRvb7uiFO1I3zG+H30HLxBHo+WQry?= =?us-ascii?Q?Du1k18Q45qB0zwlmgw1CCh+Tw9JnmHdXQceoOOC00OU6im8vheWr0A=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(36860700013)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:38:25.4432 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 481d6b76-08f8-4747-b896-08dd6212cdab X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4054 Content-Type: text/plain; charset="utf-8" From: Chen Yu Currently, only the domain with SD_SHARE_PKG_RESOURCES flag would share 1 sd_share for every CPU in this domain. Remove this restriction and extend it for other sched domains under NUMA domain. This shared field will be used by a later patch which optimizes newidle balancing. Suggested-by: "Gautham R. Shenoy" Suggested-by: Peter Zijlstra (Intel) Signed-off-by: Chen Yu Signed-off-by: K Prateek Nayak --- kernel/sched/topology.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index c49aea8c1025..815474823b3f 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1680,10 +1680,10 @@ sd_init(struct sched_domain_topology_level *tl, } =20 /* - * For all levels sharing cache; connect a sched_domain_shared + * For all levels except for NUMA; connect a sched_domain_shared * instance. */ - if (sd->flags & SD_SHARE_LLC) { + if (!(sd->flags & SD_NUMA)) { sd->shared =3D *per_cpu_ptr(sdd->sds, sd_id); atomic_inc(&sd->shared->ref); atomic_set(&sd->shared->nr_busy_cpus, sd_weight); --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2044.outbound.protection.outlook.com [40.107.237.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15C2D264FAF for ; Thu, 13 Mar 2025 09:38:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.44 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858728; cv=fail; b=GBBykzzl0fQg0QuMyPAE182Nx55FOaMLTo11kuawAe2oNhk3/nJArpt4ZcZUZ59wvgKlRtJMQGcH779EEHC7JyPcJ+F6FZcCXgYclR13YJZjJrLfifZgRKm8uNAOq1l6J75s5L+oRVDIFNFVjdzB1q5GezOXv7Qrm398lqfCrcA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858728; c=relaxed/simple; bh=beoNR2DjWWdAZMCEaieA0dVGdwpnNn5+3YntxgTzqUE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=S++IDHh8Uo3/9ZivWhlQCZpTkwM5mwaqKIKphg+Uet8Du0Z2W9FtDcQh1iQ6uL5Q8uCtJpFxYyo3P5pCwJqc+awnWuYkoXfCsotURd1yMA6sTIqD603cXBq2WCgaKdTDHYOsKRnwHssgbAxo47jG67kspiVAuO5paFPNanQFokM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=nR2LqVIq; arc=fail smtp.client-ip=40.107.237.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="nR2LqVIq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gLeF0l4Va2herFYteQoRIxe0wBYkUrnu4r9Cz192zXJJgrGc9Vaa/PKBGgJgeUJYOim4bP3uBr3qky+AC/+8z0e+YSbwmKxsu5y1WYubCO7yjsEfX0OX8aBb/0gZPyvrIGp6oAlHEm+ZgwkO/zG9qVRauk28jAxb6LZgmhfLpQB9+7607g05FNo/6t61mumhbtiTtvHVjv0d8qovjkSHhLnMI40reCDus/L/9fc5VSbcFZSuzoPl2P098s49rK7v6rJo5IadmEyTDAg8aRFVVRSDXJOuXCUjFPcYVvFls1dJZSEWy0mnJSEAwV2SBVm1V+IiY5X11VXccY0bo77ZfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IR1T1f8MV2sZYH0x+6dfeth/EPpxARwjEU5Ut2W1Wbc=; b=dc1YB3+ih9bPP5MC+jUgNf9qfQYY7GJwON5Jy29U4bVRrnX7N5i/ABGqoX5hGU85Edl4jD+T6Kfk2j+K8aTPA8s4tfS/pOR4F+SyaZ4SxrLGtTYHvvSNx4KNll1t6Hur0mELsuyyJGOgTszhThCiyAB54Eg+GnhwHbgPjh98JKRNuUIJgufrCbKFQpU2cRnCzgASNPsQvEYGe6Up8hPqITCjiASRhvomAnm2TpeWfZlivQ/SNQXAkxDObNxFEbj81jkRYGkO1EnKxPjZcvgdBW+Gb627gArwTca/uIDYkDlFJ2Q/CGqby4Gx2964E3DxX59WO7EF94CxI3sRtlyAwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IR1T1f8MV2sZYH0x+6dfeth/EPpxARwjEU5Ut2W1Wbc=; b=nR2LqVIqyGiCUs7GVkgK+3Ymdii2H/3nuysKmPl8UhtbyhABx8crYujIJAu6Vhc3iu3qT1eCNApbykEUaPPfqI/rpSE9etXV34wDnO/WCruFgbnVOPdLPZEkBQtjQG2jrfRCyobgAXfNouLy5UrZAXzV4m7JIghz0xqIpAuT/Lw= Received: from CY5PR13CA0031.namprd13.prod.outlook.com (2603:10b6:930:11::16) by CH0PR12MB8577.namprd12.prod.outlook.com (2603:10b6:610:18b::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.27; Thu, 13 Mar 2025 09:38:43 +0000 Received: from CY4PEPF0000E9D9.namprd05.prod.outlook.com (2603:10b6:930:11:cafe::94) by CY5PR13CA0031.outlook.office365.com (2603:10b6:930:11::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.26 via Frontend Transport; Thu, 13 Mar 2025 09:38:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000E9D9.mail.protection.outlook.com (10.167.241.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:38:42 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:38:37 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 2/8] sched/topology: Introduce sg->shared Date: Thu, 13 Mar 2025 09:37:40 +0000 Message-ID: <20250313093746.6760-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D9:EE_|CH0PR12MB8577:EE_ X-MS-Office365-Filtering-Correlation-Id: aee2aa5b-b773-4e13-1af9-08dd6212d81a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|36860700013|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?IF+lWFupTc5ALRop3Zg5h0IpHdI7q6sTXs32FgTuJofMW2aS+plbK19qAMgQ?= =?us-ascii?Q?LQyKHnD3F+AI6M10K9etmJeChT6coWVRJB4tBwRNEjM1DAnKIJfMzn2qEBJj?= =?us-ascii?Q?jQcf3GUaQyAWCgiy4QSK/7pL1aTL9N+vA1BS+Auka2irVDcOaB7e2j3DkxGE?= =?us-ascii?Q?fiq3tgwefcdJcU+suH1rBwRpv2SLhI0Hh3YP6vK14Wn9EsoPrETcB8liKJHq?= =?us-ascii?Q?XRd7ijtobgd9J0N77Sqdf+kkWFeRw0F+KfciiUo28W+HU0Q44UK1i67Y9ZWY?= =?us-ascii?Q?9GOvdCBlGg2qAMFouAvhgzVhNUU7+JJ472k5phlNqJuWD4li9wovOiOZc+MX?= =?us-ascii?Q?znkAfrTHJfTwuJywFxFRGXaY5/cGynT0mDkM0t2YP3Ctq1jS14w3L+XufBnE?= =?us-ascii?Q?M7mVUrkIFfvhbPnXw7A2ipWFRS+h/3riIch8kAR1BlYTpwt4ZWq1I0e08f0O?= =?us-ascii?Q?WQZTdzPRGpnTHXAkoMxUfV6fSdXsxp9maG2M6YyxHli8myQbOzJlNKwk7ra0?= =?us-ascii?Q?AhiYgzONxs8x6FtNaHPgt5YDzN8ySocFqO1TGb6GOabxcucS2FC3d2c7Xh+O?= =?us-ascii?Q?xIjRzJbGXFh5rHxfbOd7VVHyBrtlWZv91yaikCqtp1xyOp1vMwakd0lZe87F?= =?us-ascii?Q?mTjdLQnJ89g5WIyy1FwCYi5kjCNnb+aElwmmNKxUd7WmbsY1xmMMuY0/kAFD?= =?us-ascii?Q?9odSNefPHs15KmG9mQL9DaWrjWQtvu3hNR7zHIJAQvcu/muJttxxQCMP/eYy?= =?us-ascii?Q?2LH6COL4qNCnY63UUNpZPlLWmMZgx6YwNImjGYDIKXKSpxWz4+A0ri7iy9RT?= =?us-ascii?Q?GIfEgI35XXTi0mzxtP99pilOXvMUdEvjB8kf50bNGAELLN7fz1Yz0ap61p6M?= =?us-ascii?Q?/wlykOCiLtBV7J+XIKI1hb9vkNEPAwlXAviyL/ZKJ3+3rTKqu7XW1pyGGpm2?= =?us-ascii?Q?Lil0fyYkcFSvh+lTj+3LG94eleDUmRTNMF1osJ6nzPTNeH4pYhXJRexB7dEg?= =?us-ascii?Q?gcbyasYGTmb2Tm+h5REDnu5Ew3ttcMLRNrNzKwcux5uispOlrj9zkWoqSIjv?= =?us-ascii?Q?cCw1vhLQVC4RGtxE+9OneySwRrsseyCVbhs1SP05BbrJL7/JEaPc/Eamzc2U?= =?us-ascii?Q?5Gv/oXyZ7B3eC2HXnQQtv2gd3LhqeJ9N/6+a1cJ4uIYo1HdRTZBjL2M1BIcD?= =?us-ascii?Q?9JIDDpMnwPEk0Zuwf1ISOB5zrLBHpvQIdJADj/8e952Y80Gjo/WZaP3mtc1G?= =?us-ascii?Q?EGibjvu11cLbvYT08z6ANVy8ANth1rkH4k5/9ZoAXtnP/n5VWSE03gNG7TwK?= =?us-ascii?Q?Xj1KBX3LeKmNJS89bvyH2+KFRo5fhKCd2bAs8k8E6oIDSMH0I22zh7UjI4X4?= =?us-ascii?Q?eCWy3VT8jSi7acHiOqWfSc2G7BdM8/kn0eGu0ZX84W6wnSoIwYfEUwD4YQ2i?= =?us-ascii?Q?y7D0UOzD9OKDjeDwUXPMdJjYDXISBtCLuSSMdfGzoYomV2lEq+nQDw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(36860700013)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:38:42.9751 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aee2aa5b-b773-4e13-1af9-08dd6212d81a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D9.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8577 Content-Type: text/plain; charset="utf-8" sched_group(s) of a particular sched_domain are created using the sched_domain struct of the child domain. Attach the sched_domain_shared struct from the corresponding child domain to the sched_group. This shared struct will be used to propagate the sched group stats up the sched domain hierarchy to optimize load balancing in subsequent commits. Signed-off-by: K Prateek Nayak --- kernel/sched/sched.h | 3 +++ kernel/sched/topology.c | 27 +++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 023b844159c9..38aa4cba5d1f 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2089,6 +2089,9 @@ struct sched_group { int asym_prefer_cpu; /* CPU of highest priority in group */ int flags; =20 + /* sd->shared of the domain from which this group was created */ + struct sched_domain_shared *shared; + /* * The CPUs this group covers. * diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 815474823b3f..508ee8aa492b 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -612,6 +612,23 @@ static struct root_domain *alloc_rootdomain(void) return rd; } =20 +static void link_sg_shared(struct sched_group *sg, struct sched_domain_sha= red *sds) +{ + if (!sds) + return; + + sg->shared =3D sds; + atomic_inc(&sds->ref); +} + +static void free_sg_shared(struct sched_group *sg) +{ + if (sg->shared && atomic_dec_and_test(&sg->shared->ref)) + kfree(sg->shared); + + sg->shared =3D NULL; +} + static void free_sched_groups(struct sched_group *sg, int free_sgc) { struct sched_group *tmp, *first; @@ -626,6 +643,8 @@ static void free_sched_groups(struct sched_group *sg, i= nt free_sgc) if (free_sgc && atomic_dec_and_test(&sg->sgc->ref)) kfree(sg->sgc); =20 + free_sg_shared(sg); + if (atomic_dec_and_test(&sg->ref)) kfree(sg); sg =3D tmp; @@ -746,6 +765,9 @@ cpu_attach_domain(struct sched_domain *sd, struct root_= domain *rd, int cpu) if (parent->parent) { parent->parent->child =3D tmp; parent->parent->groups->flags =3D tmp->flags; + + free_sg_shared(parent->parent->groups); + link_sg_shared(parent->parent->groups, tmp->shared); } =20 /* @@ -773,6 +795,7 @@ cpu_attach_domain(struct sched_domain *sd, struct root_= domain *rd, int cpu) * the child is being destroyed. */ do { + free_sg_shared(sg); sg->flags =3D 0; } while (sg !=3D sd->groups); =20 @@ -972,10 +995,12 @@ build_group_from_child_sched_domain(struct sched_doma= in *sd, int cpu) if (!sg) return NULL; =20 + sg->shared =3D NULL; sg_span =3D sched_group_span(sg); if (sd->child) { cpumask_copy(sg_span, sched_domain_span(sd->child)); sg->flags =3D sd->child->flags; + link_sg_shared(sg, sd->child->shared); } else { cpumask_copy(sg_span, sched_domain_span(sd)); } @@ -1225,9 +1250,11 @@ static struct sched_group *get_group(int cpu, struct= sd_data *sdd) if (already_visited) return sg; =20 + sg->shared =3D NULL; if (child) { cpumask_copy(sched_group_span(sg), sched_domain_span(child)); cpumask_copy(group_balance_mask(sg), sched_group_span(sg)); + link_sg_shared(sg, child->shared); sg->flags =3D child->flags; } else { cpumask_set_cpu(cpu, sched_group_span(sg)); --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2042.outbound.protection.outlook.com [40.107.243.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB5292641E1 for ; Thu, 13 Mar 2025 09:39:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858746; cv=fail; b=gZj+xVR22ThGGqnHpmy/Lg4fqRo8OJXOwgTxfxBU1Q5Q9QGl2B9/VahQWUvO/lmuN0dSH048MdzcVU8zNfxvSMVVQs1dB5EqPY1SqcLT+bIC/LkAItG/VyQR0DrrAmKaCnB2y6EPqZbimjXx9Y6yo2N8ZpkVJ0gI3eAsTHdkepY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858746; c=relaxed/simple; bh=IWPX1jFAVG0dZRgyxTo3T1gUkHa8LndcS9c3lH/p/t4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=O7vxcGg6/HIDoqHntFzMb2pnVR/cJ6ZE0I3IVQPlMJMvcgzYO8ey4Gw/ivHcVK7ne6Yik1+M+grkDNcJ9ZEp3xKm6Yzw3B++lYhcr9rv/GrjvWf6PI9lF4ma4SmgBd7+hbe6ADm5nAlvJz6n9IQHqIAbs1PtjZK8f3rNTxudpy8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=MEmnSbXQ; arc=fail smtp.client-ip=40.107.243.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="MEmnSbXQ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=D1SgoKFiIc04GUYMjkksFrB/oiGvZ9Hn3cdx9jvgFc5OVZ7MIYRypfM+wELNEWZo0p2qayy4sfCN7Z56W91Xd+Zqys9ORAMn3OEvUQepctKuGHVbjfBGmy6y2rYu6UBgJX8VnRUugnrZvUjy8eh6oURtHQ3WkmZoApfLWeirJNKH6nklU7UBcjuZmOu3d2hQySyg0vZSuGiiHZ5s2a4PclaWgFjapzZIixrQ2oCoqfxE8mQqil+fDGX+IhkMbHc/uK0m/RlmxOIcNoaebnYkVSSnuj3XoBu1WQr/0xQrdW0QzQGcOit/s/8BDDs2r8tbbUNuPogDRgyBojWSf8G90Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AZHix5/tRJ0uPmqG1bVpzoGeTvzhyuIC+pdwRqUs+P8=; b=OxXaW8Elp7V1/ABMfw52umMuSfG0am0vwDZwL5MeI4lTkLwul09QQZwKyuqqUSHSjorIKxjosFY0RZAiasmqPyOAovpRSLWqVzljzFhtHQDOFv3Va4OpkNR+DlO80Fw42QIdglEQvgs1WDsjD/I5kufwkoZ5MAjNn0qYQVr2SJ68qaMJ1fCdGGwmqi+ogMFxnDJFwgUi6c19XtMzbbZ2g9jf+g+gSCkU3uGG02AbcXn0v61DMM9EyUDt0zO6SEmuhGHhhaJWwvJZxov937RZu/m7QaKpeUzCmOnG93d7oX3dwG4irSxMNRGmsctMPJPhHS64ol2Pv4wRj9cWA7Lhmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AZHix5/tRJ0uPmqG1bVpzoGeTvzhyuIC+pdwRqUs+P8=; b=MEmnSbXQuPRUhYRYoIZTC4IxKqd0z9x8+Ahqjs6s69OLEZFCi9sA3Ma9hZLd5uQ7fZjcPe0l3MbgrWOrxPbFcnI/KeWpOTby3Kp+PeIFKJqtLrL6KlYQNMX/j6RwU9DQ+qm/Z+OvrOiwNC/qR1tMIhY24PDhJFy3ndsa4FCTzhg= Received: from CY5PR15CA0254.namprd15.prod.outlook.com (2603:10b6:930:66::28) by SA5PPF7D510B798.namprd12.prod.outlook.com (2603:10b6:80f:fc04::8d0) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.28; Thu, 13 Mar 2025 09:39:00 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2603:10b6:930:66:cafe::9d) by CY5PR15CA0254.outlook.office365.com (2603:10b6:930:66::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8511.27 via Frontend Transport; Thu, 13 Mar 2025 09:39:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:39:00 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:38:55 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 3/8] sched/fair: Move "struct sg_lb_stats" and its dependencies to sched.h Date: Thu, 13 Mar 2025 09:37:41 +0000 Message-ID: <20250313093746.6760-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|SA5PPF7D510B798:EE_ X-MS-Office365-Filtering-Correlation-Id: aff0814d-7daa-407b-a465-08dd6212e2a9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?e7Tp7ri2wA1bfKFcvQuOknAXj21QZt96zOQD0pldOpbpt2Ax/RM3SZfYJAaG?= =?us-ascii?Q?GCJVvlvrukYswiFqtaYzaxfxbKMbnfN1AhkjsU70nXTCaJHzQuB6qMA/9OFs?= =?us-ascii?Q?7rlrui1XBDnXofE1eiaj5eAuvs33FzRRcCGCPUyQ/Q2v02ZpNc2cGXcE6xBw?= =?us-ascii?Q?X416GA8xT9V7Oo4j5VYLSh8X0vsC+kOjerJ8VheMufeXK35Yie4JzVlOYiCn?= =?us-ascii?Q?kzMVMnbVHN9ArBYgT0iC1sY0dQVOstMoivuJVS9vMy/apnFNrgUMNby4vtXk?= =?us-ascii?Q?HvSdUwkktoDm/LJOSO5oCk3ptNkLBpIs0/fjOdIulPze3wWkz6FvFPimMdxf?= =?us-ascii?Q?cW4BT9pyMzE3gu2xspa0Q6ppPO0lwhknAm6VeuX3Z8Vmb8C7P5LlK3NKY6TN?= =?us-ascii?Q?eAiHSAMaLe+RGnTkdvaKtR2w6I9QxXzzFIqCdDZy41wO41WoVuacBiDKk0NQ?= =?us-ascii?Q?y3blqh+twnXs55mpyhLzCbksgqjnrqNqye5vq3l0syMaZiciwJ/x/FTcBngR?= =?us-ascii?Q?14nXnb2JhkrKB5J3CAgIOsDdcczeMY/6I+Qtg1Oyvkql04UlL3gaaDtEcgNZ?= =?us-ascii?Q?lrp9T/O++hMbfmpHRcS2+cvGCxBlqlsAubLQMNtDyQTcDJu1VApveNr8uyju?= =?us-ascii?Q?SqNrylNX0c/+496F8onJ9KcAyTbkwMa168bCcuv3Cb7lURzwW8GmfJL2jhM+?= =?us-ascii?Q?5k2z5KSUAl+U4k+vnjJIKDo73nOiaKMPt46AMVPcHaeOj68mxS8v5H3/Vv7b?= =?us-ascii?Q?s1Cac63t9wTESwIwnJ4FRfUz2mvXcegn0Cg4sdqr9YEUuaXivU0RCp1IW6RW?= =?us-ascii?Q?DLv1YO52zQk11YvQJEOa/rf3KRmd11cQqOTb4Se0+i4pxVZHH1aGFS4s1ro5?= =?us-ascii?Q?d7ae3yP6WCuvyv4+gkIgFY2o2v1hlzwxU6ie872UHs+eiaFQwRZ7XPOqdpnB?= =?us-ascii?Q?AVJU7SKC76GzaqWcv0ygyUR7NqWZfyZOkUlDKjezbesMpgtoziuV5GS8GNZu?= =?us-ascii?Q?AHLHB5bGHhtg2/HNanTZh7XEypVJK5e0EhlEXBAa12lC76O7+xMztq8VGD87?= =?us-ascii?Q?yHgqedAIAr37SEiBWn2P5f6idfuGS/wxvRRvlxVMUtbFWTAHj4pYSpDrvUdH?= =?us-ascii?Q?c4UscbRWXbS5hK8u0j6pwmoC1EDAaEYRdMu1X9px25Mx389lRNE9e3DqtXAC?= =?us-ascii?Q?Jr/Bju0v4CdoSsa3aYlRSsRgHg07VxSguFy9ITnMEo7jVAKxonDdxsZ80PYU?= =?us-ascii?Q?ivQdgQoJRV6Teo2Iv24kw4OOpzwNFNi7nbQ4xk0wJ+l948CTfqToE9lj62ey?= =?us-ascii?Q?WldWajJN9js/uT50z+4bRHYHZf3cGTB9HCgKV1GneYY9WUeOZEhX9up22F4y?= =?us-ascii?Q?ApM0NYcAp3rD/Vhj+zlbWrWNYLVgNQ+OmdtijV8iNz1QQlTmFf1c8Sm0Js9l?= =?us-ascii?Q?3EI3m35NvigYWUeksQx1iQyW+ikdS2Aa9XI+JBUsbGhR70USbUo2tg=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:39:00.6617 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aff0814d-7daa-407b-a465-08dd6212e2a9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA5PPF7D510B798 Content-Type: text/plain; charset="utf-8" "struct sg_lb_stats" will be embedded into "struct sched_domain_shared" to propagate load balancing information up the sched domain hierarchy in the subsequent commits. Move it, and the internal types in depends on from fair.c to sched.h Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 66 -------------------------------------------- kernel/sched/sched.h | 66 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 66 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9dafb374d76d..39bee40dde27 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9168,49 +9168,6 @@ static unsigned long __read_mostly max_load_balance_= interval =3D HZ/10; =20 enum fbq_type { regular, remote, all }; =20 -/* - * 'group_type' describes the group of CPUs at the moment of load balancin= g. - * - * The enum is ordered by pulling priority, with the group with lowest pri= ority - * first so the group_type can simply be compared when selecting the busie= st - * group. See update_sd_pick_busiest(). - */ -enum group_type { - /* The group has spare capacity that can be used to run more tasks. */ - group_has_spare =3D 0, - /* - * The group is fully used and the tasks don't compete for more CPU - * cycles. Nevertheless, some tasks might wait before running. - */ - group_fully_busy, - /* - * One task doesn't fit with CPU's capacity and must be migrated to a - * more powerful CPU. - */ - group_misfit_task, - /* - * Balance SMT group that's fully busy. Can benefit from migration - * a task on SMT with busy sibling to another CPU on idle core. - */ - group_smt_balance, - /* - * SD_ASYM_PACKING only: One local CPU with higher capacity is available, - * and the task should be migrated to it instead of running on the - * current CPU. - */ - group_asym_packing, - /* - * The tasks' affinity constraints previously prevented the scheduler - * from balancing the load across the system. - */ - group_imbalanced, - /* - * The CPU is overloaded and can't provide expected CPU cycles to all - * tasks. - */ - group_overloaded -}; - enum migration_type { migrate_load =3D 0, migrate_util, @@ -9916,29 +9873,6 @@ static void sched_balance_update_blocked_averages(in= t cpu) =20 /********** Helpers for sched_balance_find_src_group *********************= ***/ =20 -/* - * sg_lb_stats - stats of a sched_group required for load-balancing: - */ -struct sg_lb_stats { - unsigned long avg_load; /* Avg load over the CPUs of the gro= up */ - unsigned long group_load; /* Total load over the CPUs of the gr= oup */ - unsigned long group_capacity; /* Capacity over the CPUs of th= e group */ - unsigned long group_util; /* Total utilization over the CPUs of the gr= oup */ - unsigned long group_runnable; /* Total runnable time over the CPUs of th= e group */ - unsigned int sum_nr_running; /* Nr of all tasks running in the group */ - unsigned int sum_h_nr_running; /* Nr of CFS tasks running in the group */ - unsigned int idle_cpus; /* Nr of idle CPUs in the= group */ - unsigned int group_weight; - enum group_type group_type; - unsigned int group_asym_packing; /* Tasks should be moved to preferred CP= U */ - unsigned int group_smt_balance; /* Task on busy SMT be moved */ - unsigned long group_misfit_task_load; /* A CPU has a task too big for its= capacity */ -#ifdef CONFIG_NUMA_BALANCING - unsigned int nr_numa_running; - unsigned int nr_preferred_running; -#endif -}; - /* * sd_lb_stats - stats of a sched_domain required for load-balancing: */ diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 38aa4cba5d1f..dc9d6e4c704b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2102,6 +2102,72 @@ struct sched_group { unsigned long cpumask[]; }; =20 +/* + * 'group_type' describes the group of CPUs at the moment of load balancin= g. + * + * The enum is ordered by pulling priority, with the group with lowest pri= ority + * first so the group_type can simply be compared when selecting the busie= st + * group. See update_sd_pick_busiest(). + */ +enum group_type { + /* The group has spare capacity that can be used to run more tasks. */ + group_has_spare =3D 0, + /* + * The group is fully used and the tasks don't compete for more CPU + * cycles. Nevertheless, some tasks might wait before running. + */ + group_fully_busy, + /* + * One task doesn't fit with CPU's capacity and must be migrated to a + * more powerful CPU. + */ + group_misfit_task, + /* + * Balance SMT group that's fully busy. Can benefit from migration + * a task on SMT with busy sibling to another CPU on idle core. + */ + group_smt_balance, + /* + * SD_ASYM_PACKING only: One local CPU with higher capacity is available, + * and the task should be migrated to it instead of running on the + * current CPU. + */ + group_asym_packing, + /* + * The tasks' affinity constraints previously prevented the scheduler + * from balancing the load across the system. + */ + group_imbalanced, + /* + * The CPU is overloaded and can't provide expected CPU cycles to all + * tasks. + */ + group_overloaded +}; + +/* + * sg_lb_stats - stats of a sched_group required for load-balancing: + */ +struct sg_lb_stats { + unsigned long avg_load; /* Avg load over the CPUs of the gro= up */ + unsigned long group_load; /* Total load over the CPUs of the gr= oup */ + unsigned long group_capacity; /* Capacity over the CPUs of th= e group */ + unsigned long group_util; /* Total utilization over the CPUs of the gr= oup */ + unsigned long group_runnable; /* Total runnable time over the CPUs of th= e group */ + unsigned int sum_nr_running; /* Nr of all tasks running in the group */ + unsigned int sum_h_nr_running; /* Nr of CFS tasks running in the group */ + unsigned int idle_cpus; /* Nr of idle CPUs in the= group */ + unsigned int group_weight; + enum group_type group_type; + unsigned int group_asym_packing; /* Tasks should be moved to preferred CP= U */ + unsigned int group_smt_balance; /* Task on busy SMT be moved */ + unsigned long group_misfit_task_load; /* A CPU has a task too big for its= capacity */ +#ifdef CONFIG_NUMA_BALANCING + unsigned int nr_numa_running; + unsigned int nr_preferred_running; +#endif +}; + static inline struct cpumask *sched_group_span(struct sched_group *sg) { return to_cpumask(sg->cpumask); --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2046.outbound.protection.outlook.com [40.107.236.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BAAD264A7C for ; Thu, 13 Mar 2025 09:39:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858764; cv=fail; b=VezVFZSbVbAz9kYb4uThCFVpdmRfkhmHnDs2Ss6KqsVB6qq7Gdh50qxJKY0fIc+IGjJcs4KnrZ8FNjEhjQViSO661aCtFqYhxKho3djx2YbVH90mvDOeTat9aEfnp/YmxIGUQ69NxveRkMkGB+aZr9Jswdm73ZM+i7qav2q9vJc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858764; c=relaxed/simple; bh=azqloHhRipsP/t/aybbzJgRwh4C2dDTEMfbmpHJ4mBc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=k9ZXRD8w58EIQkoQ6JFMOuYH0GJxu11axcP9yiL+o26xZwnySVESgM3XL4QgncHuuVpD27Hi5qponDDSHHuc2ASaxnt3M/ZrX+4weN/8pCieuzZvvp6ylMXD+3LeLPRdvkvaPWx31WEnptoPSATXuXGoEUEOmZEQZAGEGXl5EnQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=JOQgdK3r; arc=fail smtp.client-ip=40.107.236.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="JOQgdK3r" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=jlhMSwn82EKiPBoly6BfJPYt70uZZYcoDOInUOKVNpZN4moojw+3thIavPOX0EEVX8pbdzgQ1cc0qqjpdvKdgCZPFqNagg1E9mwz8DxMMuVKd1legZHr86a3OO2SVnIiAnhNLqsJmUXX+mmXdmXjALGKpSEpsstjAuUNCmICisd5OGujPsCG44md526Qqyajni66BUnce0NIZ81rNTPk1T8dD8PoFF6UMMnt7688Wod8EdSpyVxox61aUOS9paZwC71JnF0EUmxkoDiKaQGBR7g7YTjMpbS0A+jIs9tVm5YmBd85QJJ0blupDbfnCOy93R5voJfbpc3ffyRGQJ4TRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zQga1b8EBcWCeOnkEzr9HFTRrZrpSoTZQOooNme1C4w=; b=TevpHQtmQiKTTlMTEEVI2iZzZJEGpJPf/zGfeEWTyyPv5icF5VCnBkxa2LCWHoDqlsM58EFeiUkfaqJvB5+OOiSU4DB0G7eYUI989O9s69g3egfFGb2ZcJ2xSbDLpt6kUEwGzLBqRNVyTXPGQYZkNUfzEhiVKqYgwsvFy8NZsvoxU7qUOl6aABZVVOpclI/8SbsyedF053wWaNH0UHFlunh3rirGLtXGxgf1L0EWVBBe5l4/F9tOygfEt3tv4pEGCCdb+DJLPuG9ZJxVwkWUlsf+eRIGWmb/XV8294RRBlRfLmLJf9SkkZRvYb4JamhWLSPMcs/LKw46QTSYbpHvXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zQga1b8EBcWCeOnkEzr9HFTRrZrpSoTZQOooNme1C4w=; b=JOQgdK3r6N+zMO1mvIxF4HVCpMAB3ZleDl6q9z9ZRXzmpA7KvJdHOEP4+I054r5TXzzGDPWrCmOJQIeKbVP7EdfcjF8guYNqI7UZw0j6+VAsfGg1ggc4VJ8EqaS5EzCTUREINJZd7BS5pl4H2H71AByvKwcLrSXwjDs1neaUoF8= Received: from CY5PR15CA0114.namprd15.prod.outlook.com (2603:10b6:930:7::28) by LV2PR12MB5918.namprd12.prod.outlook.com (2603:10b6:408:174::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.28; Thu, 13 Mar 2025 09:39:18 +0000 Received: from CY4PEPF0000E9DB.namprd05.prod.outlook.com (2603:10b6:930:7:cafe::af) by CY5PR15CA0114.outlook.office365.com (2603:10b6:930:7::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.24 via Frontend Transport; Thu, 13 Mar 2025 09:39:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000E9DB.mail.protection.outlook.com (10.167.241.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:39:18 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:39:13 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 4/8] sched/fair: Move sg_{overloaded,overutilized} calculation to sg_lb_stats Date: Thu, 13 Mar 2025 09:37:42 +0000 Message-ID: <20250313093746.6760-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DB:EE_|LV2PR12MB5918:EE_ X-MS-Office365-Filtering-Correlation-Id: c9ca330a-d571-4774-08e7-08dd6212ed2a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|7416014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?09dvs2dKthOE9EQTxUXh4C8ZcsSejKNK7QimJz/PS52O+qkyagtXWmWrQBul?= =?us-ascii?Q?Z7z9KfKbMgfWfMiDyAiHHIXcF+4YE0pvCyY8ai9y208ZfeEeAF+gOFhiswSH?= =?us-ascii?Q?DZmcOXAFL8KXXCyvbJXYg9qSibv+IPtf3D5BGpcRkzL1POpeshu1Zte/GG33?= =?us-ascii?Q?Ir7diBjH3b/sZz8M4NpTAfQ6eNsGBV5jyvE4t59xUZd/4o6oovZRkGFYhs1V?= =?us-ascii?Q?457p+f+9wSVdz7A8bsRo6QFg3JrE5Bch20UR4juz3QObVhYXAhQgkKoxqtZN?= =?us-ascii?Q?d6a9ctJewnWRLzCNGFtUy6h2y3IW68cQgtHROwA9AqsnhCY03H9/SwQKeNNe?= =?us-ascii?Q?ihUUy25AUfLyWl6PVCk8OWqKHuqdNdFdTsSaXp3cVz6+boUxNJqhx5IygcWD?= =?us-ascii?Q?VMZPz5v2MuQok6O5EClSS7brRqYFbT5xmffJJO9tQ1UxQMhaPpngdZ9fMRez?= =?us-ascii?Q?RktiS9tfElSKNInyho9aLtMpmwZ17OsxXiqwUp5ypOEOBgqOGNw/vQItjL/y?= =?us-ascii?Q?/4dUd2749ubebYzdlFB5m7d0+80JZ6P4W28GT75O2lUIutTeGx2YI88HDviB?= =?us-ascii?Q?X3bEZMnI1xumSQuBK8GJh24gokeX98h2i/3D8aWFOKU4wJPx4SondyVdmnF8?= =?us-ascii?Q?KUyADVBHol4Fn6LZrrp8udHnx1NOo91sCxWACVmPNlgk9ujXKcvfDlNKVsTW?= =?us-ascii?Q?AsLSdLwqcKF77TJF3J2Gdj5VEnPdJBut0eslW5zkwLPlIkmlmGUmJA6A8vIL?= =?us-ascii?Q?McK9tZdOoqof2Z+uJWwIFQ07N/2Euw2eEOXJSQTf8/zKumUUpNYvj4bKdLsU?= =?us-ascii?Q?YlvDDpW8nOzad4mftU9XAQchq0lHDvL1rS7B5SWPmTo+IlNbYfo1Pdjp8vb1?= =?us-ascii?Q?hFNoGcFYdcCXycvNRYZLBJrN+DgiRCP3pX060iwkQFuMpbq+XFOQG1z6lh4g?= =?us-ascii?Q?GpvZ1s7+igmUV8qpV2iMaFPVtLtjDyObHxKqksPn4rYJiJP8/ihiX2ObF6H7?= =?us-ascii?Q?dKLoMecgvbKY6DRAqHBod+87hP3sdVwpL1gBwNNNYMbbLA8ncRxXXwTPcOad?= =?us-ascii?Q?xhGXj0ib+fBkzKruH99DriFSfcXmdGT53WH+Sa7d2K5URUB8EkLAdOtGlcJ4?= =?us-ascii?Q?CnEJCVQHVs/PjmRoAg9gVxneYBaz72otaphrd6sTC+hhEuM8NRU9y/rrwvdw?= =?us-ascii?Q?M/gQ1csJY5qWZKAfrKqeTosnMgMibWR4GkkvDJOEVlZ2bw6cp/6HVmLHq91W?= =?us-ascii?Q?+jyO3UR9l1zPg36aXHp3uHOMI+GPncphDZFp1ybl8OyK+OKsTFoq2Zf/f6Kp?= =?us-ascii?Q?Hkt2PEqDS6DcpJVff9fVx5rIaoY5auKJMvZIDEJ5Wm0qmwHYwbsfyoqYB0ka?= =?us-ascii?Q?as6UQBz9oWe8lIjui1FCQx3N5k7zqVr8UGmXrT+AI46daV+4AjZSzFg1xlbR?= =?us-ascii?Q?UN8AGrQyWcNXASvPcRXjFZUzX6MuBBGeDSLDbRIFqT5g6i5O+/vXyg=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(1800799024)(7416014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:39:18.3116 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c9ca330a-d571-4774-08e7-08dd6212ed2a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DB.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5918 Content-Type: text/plain; charset="utf-8" update_sg_lb_stats() used a pointer to update the group overloaded and overutilized status to propagate to root domain. Discard the pointer passing and use a flag in sg_lb_stats struct to indicate the overloaded and overutilized status. This will be used in subsequent commits to propagate the overloaded and overutilized status up the sched domain hierarchy and set these status at highest domain. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 14 +++++++------- kernel/sched/sched.h | 6 ++++-- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 39bee40dde27..3b1ed14e4b5e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10287,9 +10287,7 @@ sched_reduced_capacity(struct rq *rq, struct sched_= domain *sd) static inline void update_sg_lb_stats(struct lb_env *env, struct sd_lb_stats *sds, struct sched_group *group, - struct sg_lb_stats *sgs, - bool *sg_overloaded, - bool *sg_overutilized) + struct sg_lb_stats *sgs) { int i, nr_running, local_group, sd_flags =3D env->sd->flags; bool balancing_at_rd =3D !env->sd->parent; @@ -10311,7 +10309,7 @@ static inline void update_sg_lb_stats(struct lb_env= *env, sgs->sum_nr_running +=3D nr_running; =20 if (cpu_overutilized(i)) - *sg_overutilized =3D 1; + sgs->overutilized =3D 1; =20 /* * No need to call idle_cpu() if nr_running is not 0 @@ -10324,7 +10322,7 @@ static inline void update_sg_lb_stats(struct lb_env= *env, =20 /* Overload indicator is only updated at root domain */ if (balancing_at_rd && nr_running > 1) - *sg_overloaded =3D 1; + sgs->overloaded =3D 1; =20 #ifdef CONFIG_NUMA_BALANCING /* Only fbq_classify_group() uses this to classify NUMA groups */ @@ -10340,7 +10338,7 @@ static inline void update_sg_lb_stats(struct lb_env= *env, /* Check for a misfit task on the cpu */ if (sgs->group_misfit_task_load < rq->misfit_task_load) { sgs->group_misfit_task_load =3D rq->misfit_task_load; - *sg_overloaded =3D 1; + sgs->overloaded =3D 1; } } else if (env->idle && sched_reduced_capacity(rq, env->sd)) { /* Check for a task running on a CPU with reduced capacity */ @@ -10982,7 +10980,7 @@ static inline void update_sd_lb_stats(struct lb_env= *env, struct sd_lb_stats *sd update_group_capacity(env->sd, env->dst_cpu); } =20 - update_sg_lb_stats(env, sds, sg, sgs, &sg_overloaded, &sg_overutilized); + update_sg_lb_stats(env, sds, sg, sgs); =20 if (!local_group && update_sd_pick_busiest(env, sds, sg, sgs)) { sds->busiest =3D sg; @@ -10992,6 +10990,8 @@ static inline void update_sd_lb_stats(struct lb_env= *env, struct sd_lb_stats *sd /* Now, start updating sd_lb_stats */ sds->total_load +=3D sgs->group_load; sds->total_capacity +=3D sgs->group_capacity; + sg_overloaded |=3D sgs->overloaded; + sg_overutilized |=3D sgs->overutlizied; =20 sum_util +=3D sgs->group_util; sg =3D sg->next; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index dc9d6e4c704b..9372a75ab3cf 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2159,8 +2159,10 @@ struct sg_lb_stats { unsigned int idle_cpus; /* Nr of idle CPUs in the= group */ unsigned int group_weight; enum group_type group_type; - unsigned int group_asym_packing; /* Tasks should be moved to preferred CP= U */ - unsigned int group_smt_balance; /* Task on busy SMT be moved */ + unsigned char group_asym_packing; /* Tasks should be moved to preferred C= PU */ + unsigned char group_smt_balance; /* Task on busy SMT be moved */ + unsigned char overloaded; /* Contains at least one overloaded CPU */ + unsigned char overutilized; /* Contains at least one overutilized CPU */ unsigned long group_misfit_task_load; /* A CPU has a task too big for its= capacity */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running; --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2058.outbound.protection.outlook.com [40.107.237.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1B502641DF for ; Thu, 13 Mar 2025 09:39:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858783; cv=fail; b=gTjGcpOQM5vN0pBXk9wZo2D27OemjhMVwh9D5bRRnOtxQEvwoS7MWL+TuxKmjR+/o/q/MgHFzQVPbuCTzMOKtXM2sNuUFReBQYL5Q33y3guL4TRnYbQ6DnDOGPa2GDA3I9EztYxxQgmy1vOmEw5dxFu+OBjYEirFX2GFBisfBOI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858783; c=relaxed/simple; bh=ZX9E5u5YFODE1Da8tR5uU8WdJFzIBtlsolxhNQBUCPQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iX0ITj0Uj9USmxVT9CjBXYqJmeHckTPfH54LAvM3bR498ueiefrU5qRQ9WN1GCqjrcc33qpKU3ZLdLI2jg3PcttUKCRK2u6EHYE8FTG2oXN4nGXb85gGvH5T0vONzBAOwY9IXAqLTV+QT7kAOWzKBPAMm0ZZxnGj0WOhTvA62go= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=DVHINQW7; arc=fail smtp.client-ip=40.107.237.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="DVHINQW7" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QnYXo0YtZUWsMvYobagCz8EH2YoVq82JnA/2BR8lIQkvNa3Qn7OM4iLVXlcFAv52YlCWvGIyvmbFYhnAY/P6ug4ekjxO/97VSXCRsnKMtBvnVqTmIIMZEvLFyMk5zK+2U0Xn5cAaZ4hjMNEHn7ZZ0qoP1SwtvI6Xk6oWyhRcqy47P3Mb6+640oyp3Xi8MkvZRc59sRGhDHg8tH9GNUJAAXJn8+HPBo7Cd+7HxXgQhJR4oR0atj0r+CsFykqsyLLVCm7s+8Nw4kEfPgMTV+ldXIOGJal35FWkWaFTLu5gpaYNEi2er2w8xsy5J4ETFBT5sIplCbf9LpKMc8TAaHzy/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=q5MEss81ikQ4a1yPc7QlAkM9Ap7hvJpIRshYb2zErUk=; b=UGTI387h6qGGg+70y7gXzafjm1WmSYsbEtVlDYMvOjtzjIRU6KtXh41IHXpZu2KR3m2q9nFS0Svp86H/K8lhdl+c6fajzoLXOoi35NXy/bS7HSbrLMjgY8GdOzkCx62TmDRSLFoLi5O6Y94ZbruNrYFLIKOB+RSBnztE0aqD0Z+v23bZLB6pBS27q/UpsORt0zRa2RgVgMw+3vJ2ObQHLfG1BEzmh0keC8o1rXgwcAZJUa4bLVOYIa1jFTthA9e8eGlMlC99v4RvVeLP5tSC6Ks+DpgzIDJ6ZcvIWu1FrEEdDEGllUdBExRiFAVo0KFKP9cXIe/WKbVQYckyLvpoMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q5MEss81ikQ4a1yPc7QlAkM9Ap7hvJpIRshYb2zErUk=; b=DVHINQW7XUGXy+SEkGdrTzDTVYKIs4b/7brmTtWhIxYJeNGjId5jBhSuEqD/LqZHjUHugxjSW1MxsBrma1zD35nmucM8yMpDq+SrCQQXMsHlv0MFq+MtuCTe8KapgOOZKFurhCtQSKqgnvfRuGCTpmRUlKQ9FB2fuA+wrjPZm2c= Received: from DM6PR02CA0150.namprd02.prod.outlook.com (2603:10b6:5:332::17) by CY8PR12MB7217.namprd12.prod.outlook.com (2603:10b6:930:5b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.28; Thu, 13 Mar 2025 09:39:36 +0000 Received: from CY4PEPF0000E9DA.namprd05.prod.outlook.com (2603:10b6:5:332:cafe::3c) by DM6PR02CA0150.outlook.office365.com (2603:10b6:5:332::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.24 via Frontend Transport; Thu, 13 Mar 2025 09:39:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000E9DA.mail.protection.outlook.com (10.167.241.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:39:35 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:39:30 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 5/8] sched/topology: Define sg_lb_stats_prop and embed it inside sched_domain_shared Date: Thu, 13 Mar 2025 09:37:43 +0000 Message-ID: <20250313093746.6760-6-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DA:EE_|CY8PR12MB7217:EE_ X-MS-Office365-Filtering-Correlation-Id: e16bfeeb-ed39-4cea-49fa-08dd6212f7a9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|7416014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?6mUrQkYBH9wIDr3O4+VYBu4Y1ZHpoYft+YUrOKL2QzSTVggUfEsWHJqUXzqs?= =?us-ascii?Q?WhsLuD8ln1uaGIn5FyDhqXvaG/UiMzwJnn6IqNDcHWNvxSXbStFLxdkfseHT?= =?us-ascii?Q?lPZefx3rqDXxlLOWRnqX+snjqilQxipzRbGAohMQrKBnHW6aLL8t8kdWgPJ2?= =?us-ascii?Q?OihTmj4i743IoYXeArBi4uswkE4q7lh3YVNpH3OBIs5rb14qYLzW+1kJyWx9?= =?us-ascii?Q?SbR5fqOIrVzKNBk3c8jVhtF7xrY5o0VJ62P8RwRUmkXsxAT5RXsqRWWOggK1?= =?us-ascii?Q?Ku27EYphV6uQnEzCxY1trLV9hkGCniHePTzvdrRvABY/DI8diCPDAdo3G9WS?= =?us-ascii?Q?6SvKVfofcH9NrB65oiAicCIC+2uZ2z7nsqvIgEzsU0JTuQWI9dqN7m/CxruW?= =?us-ascii?Q?S1lciqxfOAV7Tp1LcW2JVOZgUDe5Zqq04vGXl4c3oHj5mJJNvmnQLrQFMu9k?= =?us-ascii?Q?awQAKCg9kQK1U6UjVuhRB/c6ut4d//37J0wDZyNSxxrQKxpEjj/RITkgrdD7?= =?us-ascii?Q?g+aObAms7Q4W9jBqOg/CbcqlqRDmcQbSP5+YhCXln1UFRfY+H4wSqNSEQ1Wc?= =?us-ascii?Q?7k53gHzf+yDQdgefFF1v33B8aQu++x/Ia42WwUk8C25cUz9ecfQpglBc8p6w?= =?us-ascii?Q?uBXlMCMo9WKnBduQFT7uVeasJXO8JaH81kRQt6pqqHgWoj036/txE+Iq+nXq?= =?us-ascii?Q?h5058lbEDQ1akuTeeZrQfXY/92bgWOprz2dZyxQ77NFVwFC6cUl4bT0JxiUm?= =?us-ascii?Q?oMMmXIkZqGKxLKK8HroZ3S0gAj9bcLT04GVaIO4283kocIrqUNbJwZtc93S6?= =?us-ascii?Q?comJzFY99Oy/f0lPQcDKsAdViZ9gecHSXbXK2eFyEvC9HDNj4axXbMPbrYIZ?= =?us-ascii?Q?gcVmzJsdS+mW2SgYceRafkOh8IcvPtsbr1oVTT8eZNmyCr2jTlBdaUQcRvO1?= =?us-ascii?Q?b5MH7whTE5Lg2Swn3esOHbr8dbZnQMyPUxNjAPodzE9aow9ajXzGP3BZp/u7?= =?us-ascii?Q?ByudwIqEeL1yowVdft4GRFo0e4WGRkqlBnrhJBhtbZPcSI5DBcRXUYUPgl8v?= =?us-ascii?Q?pIWT1XwyZfX5JeqPO2yMRWMMr8IjUVU9K5mFX4UQ24ASa4ZtzoNoHtQ53+B9?= =?us-ascii?Q?FUdBoyCPO3uJXDu9JDrlQYEgi31I6kmftM6bP95paFyk5K2U6tPeBvOhSQia?= =?us-ascii?Q?UsYb/OqhT0v5/Hf4XgnsKCb+ssE+y0kkMHTWHomls1/Q87Gb+CJN5w0icB6N?= =?us-ascii?Q?FWRs24G8bZO84kxbSdZiQRSs4xMtqveFHcHyh42TKiVAZ3JJp+naKkV+4600?= =?us-ascii?Q?7oenwr971zWy/y8G8mO/UuwJI1lyfGBi7PNl7JBYiY8lhNFNJeWQBbeO72AF?= =?us-ascii?Q?wu834rhqaqIuHb4qktSd5pcbgIsFrC1W7dsiqNmeoCuW8gfWkdm5ArzKmwlE?= =?us-ascii?Q?1uEz8vRxhqxsE0Zbd1HN6GjS7N6Htpq2KgOg4sQNqLu7wqS7BIPrpA=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(7416014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:39:35.9228 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e16bfeeb-ed39-4cea-49fa-08dd6212f7a9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DA.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7217 Content-Type: text/plain; charset="utf-8" "struct sg_lb_stats_prop" is a container around "sg_lb_stats" to help propagate the load balancing stats up the sched domain hierarchy. Embed the same in "sched_domain_shared" for concurrent load balancing instances to reuse the statistics collected for domains below. Signed-off-by: K Prateek Nayak --- include/linux/sched/topology.h | 9 +++++---- kernel/sched/sched.h | 11 +++++++++++ kernel/sched/topology.c | 26 +++++++++++++++++++++++--- 3 files changed, 39 insertions(+), 7 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 7f3dbafe1817..a16d7d9dd9d3 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -78,10 +78,11 @@ extern int sched_domain_level_max; struct sched_group; =20 struct sched_domain_shared { - atomic_t ref; - atomic_t nr_busy_cpus; - int has_idle_cores; - int nr_idle_scan; + atomic_t ref; + atomic_t nr_busy_cpus; + int has_idle_cores; + int nr_idle_scan; + void *private; /* lb stats propagation field */ }; =20 struct sched_domain { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9372a75ab3cf..391c4180eeb3 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2170,6 +2170,17 @@ struct sg_lb_stats { #endif }; =20 +/* + * sg_lb_stats_prop - Load balancer stats propagation container. + * This is embedded in sg->shared->private and is used to propagate + * sched_domain load balancing statistics up the hierarchy. + */ +struct sg_lb_stats_prop { + raw_spinlock_t stats_lock; /* Lock for updating the cached stats= */ + unsigned long last_update; /* Time when stats was last updated (jiffies)= */ + struct sg_lb_stats sg_stats; /* Cached sched_group stats */ +}; + static inline struct cpumask *sched_group_span(struct sched_group *sg) { return to_cpumask(sg->cpumask); diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 508ee8aa492b..aeb55f66e8d6 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -621,10 +621,19 @@ static void link_sg_shared(struct sched_group *sg, st= ruct sched_domain_shared *s atomic_inc(&sds->ref); } =20 +static void free_sched_domain_shared(struct sched_domain_shared *sd_shared) +{ + if (!sd_shared) + return; + + kfree(sd_shared->private); + kfree(sd_shared); +} + static void free_sg_shared(struct sched_group *sg) { if (sg->shared && atomic_dec_and_test(&sg->shared->ref)) - kfree(sg->shared); + free_sched_domain_shared(sg->shared); =20 sg->shared =3D NULL; } @@ -661,7 +670,7 @@ static void destroy_sched_domain(struct sched_domain *s= d) free_sched_groups(sd->groups, 1); =20 if (sd->shared && atomic_dec_and_test(&sd->shared->ref)) - kfree(sd->shared); + free_sched_domain_shared(sd->shared); kfree(sd); } =20 @@ -2273,6 +2282,7 @@ static int __sdt_alloc(const struct cpumask *cpu_map) struct sched_domain_shared *sds; struct sched_group *sg; struct sched_group_capacity *sgc; + struct sg_lb_stats_prop *sg_stats; =20 sd =3D kzalloc_node(sizeof(struct sched_domain) + cpumask_size(), GFP_KERNEL, cpu_to_node(j)); @@ -2288,6 +2298,16 @@ static int __sdt_alloc(const struct cpumask *cpu_map) =20 *per_cpu_ptr(sdd->sds, j) =3D sds; =20 + sg_stats =3D kzalloc_node(sizeof(struct sg_lb_stats_prop), + GFP_KERNEL, cpu_to_node(j)); + + if (!sg_stats) + return -ENOMEM; + + raw_spin_lock_init(&sg_stats->stats_lock); + sg_stats->last_update =3D 0; + sds->private =3D (void *)sg_stats; + sg =3D kzalloc_node(sizeof(struct sched_group) + cpumask_size(), GFP_KERNEL, cpu_to_node(j)); if (!sg) @@ -2332,7 +2352,7 @@ static void __sdt_free(const struct cpumask *cpu_map) } =20 if (sdd->sds) - kfree(*per_cpu_ptr(sdd->sds, j)); + free_sched_domain_shared(*per_cpu_ptr(sdd->sds, j)); if (sdd->sg) kfree(*per_cpu_ptr(sdd->sg, j)); if (sdd->sgc) --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2044.outbound.protection.outlook.com [40.107.92.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DED3264614 for ; Thu, 13 Mar 2025 09:39:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.44 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858800; cv=fail; b=qXfCRYAodbNeDSWq2Q2W4rUKaZ+L1fSrXWF7fHMowUUTbrbItBYsAg5gxXSx1UQV9aG0nqo7VF+l6Ea6Qkk4Z1WxfCvNItzIGgfPeMuUpOmM2OTceFoSJOKrXivwouB8kOV0HD/VJqCfdMt07EYXjGg6ZizAUyH9SXtxerJFOP0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858800; c=relaxed/simple; bh=8gHAAiJFTWF/5uCh9rllspDrOyQBv7+kaznkGXzY05Y=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HsESg6ukmvRIrE3JD08VHWpUbOhw9z2oH3k6yJJ9P8XQyDnX5CpckDBrOam8i99T67vVH025P1FxtYBcAw/ZT+WDl8gy4ZYhyrINsUF0Spk8SNKueZY+Zyzi+9X9QCBI8H73coapykgpee11sXFF39dI0NLSsFBUtEE8GsGXXCs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RnJHilNI; arc=fail smtp.client-ip=40.107.92.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RnJHilNI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=H4Y+vQoosjtC6suZRuFjS4fhW62GqgP4JqVsmGPWNDWFcJlAYrz1um6XiH1LqThAt2GKTvqmxmSXSbHXYIcJ8fT0vKBLKeHqk+WpC89pi3bcPtHcUpmKwPxkc5EuxZpqjVQwGh9xtiQqixI1FwIv+p35CV6Ck6aeuBnYplonjzDQhSc4a4xPJ8e7+MDFO8ROWhRSz4reDg3X1oRy+e+m/riQbJy0mxrz9+j+Z1u6q86+evZ48LKIFS2m+flbivm93KI374yBla072WBUZXnFgWQ5BzmZ2B+lz6cXJdUJCSmEXa0TmXc9OD1mFrSFH3foo595bn9KAl/TcZSkUzKm4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IUwVN6AlJq94CP3sCMcW4UPJp8D5mXUeUVcpBeK2w1Q=; b=Z2a0xqBz000jy+v9wSa1IHHMevqPCNOlRmEMDG8rJIOHSeEDHRqkEKZhODJdmudkB9JONnYWe0+H8H/pUyQ+g1t8AWGGe3hrCgpRfUVXU5STEpF7+NpY8fp+iaZwCC62sECXlL0Ur1tgLKvmVqRVpTdXlplj8B8En8c/aBDiolg/A240qd+evOxtidAJKwxJMXdK4dRywrM7WKL5s6ZBb+OFV7CgieJbor5NrGtGR6Hdga71DJ90wqbKJtSxYg4LHNK+jCfWjyWqX8oXzm3UgaOQRNW69ZtRcWMFneODp09azE/VCAt3aIBOE93naGOcq92gwAq3nR1SakJO4QizVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IUwVN6AlJq94CP3sCMcW4UPJp8D5mXUeUVcpBeK2w1Q=; b=RnJHilNI32sLWNsqjV4lSezKiEodY0BDYVbB6b/SdPSCQwwgU7QeVEAW4gfmCBie2tt/KwLNUOWS9VXWQE5g8yi44qgwzqfiYPKTKSR+43a67B2kyQYJDvlySDR71PC4oeba7bl1nsxr4aVnXOwtMHOQiBPOHEBEzA/f6O3Efq8= Received: from CY5PR15CA0248.namprd15.prod.outlook.com (2603:10b6:930:66::15) by MN2PR12MB4303.namprd12.prod.outlook.com (2603:10b6:208:198::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.28; Thu, 13 Mar 2025 09:39:54 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2603:10b6:930:66:cafe::96) by CY5PR15CA0248.outlook.office365.com (2603:10b6:930:66::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8511.27 via Frontend Transport; Thu, 13 Mar 2025 09:39:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:39:53 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:39:48 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 6/8] sched/fair: Increase probability of lb stats being reused Date: Thu, 13 Mar 2025 09:37:44 +0000 Message-ID: <20250313093746.6760-7-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|MN2PR12MB4303:EE_ X-MS-Office365-Filtering-Correlation-Id: 01e777bd-c9eb-490d-c0e8-08dd62130240 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?aZJHZlrJEk1KadugUU3vpskdvzZYq1WGUUeoX8NbK6PXblbpbgtQr+MAFc1y?= =?us-ascii?Q?Z8w/YdWCAZ4UMB2hr2GCAaMNRZfvV++iT/PiG+JTRFcQNYgnp5qggEI3iP1a?= =?us-ascii?Q?v6knMZnQZnYHN4IScurRKrnOzcGZvcPHgTt5ZmLlHrCz5zqbIOry+wDDwTZ2?= =?us-ascii?Q?TZwxAbvIUIJSB6DTRkqUjDHRiJHbwruRa+9/jZiBtA97xbVqtvQzPwUxGbOL?= =?us-ascii?Q?hKttlIwtKwKmBQqjqZMWEWqCclYEs062Wu4A1XJO3ikw3ziUMWG85DB5cSJD?= =?us-ascii?Q?IO6qjTw4wT/+cqFCtga8Gy4dOmiym2oXf+HbcfRYn3E8/hG0g74WWnJ+VyGk?= =?us-ascii?Q?TYb9qIYH+ZjFL7f8xn9QtRU12s/h9HT4rsWAah3p5mJw0cpTE0jsPEIaRBkj?= =?us-ascii?Q?tB6gC+jHoGPrQQ00LQij3UvJTVknCvdgS7q/yK2R0KdD+2PoVP9mjvspuDJl?= =?us-ascii?Q?zU7/2O5V8RNOek4fRLI1r5+KZOVn0obDJoVV7uTy20yzL6iyacziMDyt3s/j?= =?us-ascii?Q?lDs+VL3Umew3NZo7Zxe0uQfn+caazW2vAcaI9NsMTHswRMrJyPpyQP8t3CSR?= =?us-ascii?Q?QaapMuUepK/ukvaDXyEfNDr+1a4tCfeedtGlblYvcoLx41/CdSyYPwxqtOXD?= =?us-ascii?Q?0UcInOvtgOcke8NP+8VYWbeKTYYGj6yjeOkDMuP/I/Zj0RnDwi26nmuPSxdP?= =?us-ascii?Q?aDvBCKASiWW9ESIThfRCQnY2I/40hFEwY5Z2VTRJ4Mg7xn1AsGYdBvcQI5Pw?= =?us-ascii?Q?Iq0EJDgxvpz92+lJoy5mWzDEoCKYCkuh6f90tKDmiiRjOdiw3U9Ua3fJF24a?= =?us-ascii?Q?kxxCiQBfne5piR0pfDuCPnFoWVmkoSUlDs3sILAHk32J2Rl3f8o+mQ1H2SpM?= =?us-ascii?Q?CvfbCC6mA6TiYBMN7EOhp8AdNVmfXUm061l914stAwOlZjcbquo1BprIRbMD?= =?us-ascii?Q?ICUZqhfuOG/9Pd50l1R2yRm4fgJgn5C4ldAmU0qIjsopUjAUbaMzxcovdd1O?= =?us-ascii?Q?77LEKGUo0H5QwHYypfCjkKb0aktrpVQiTWH5ib9vf9l03fiZlk1YCZX8hhxV?= =?us-ascii?Q?p62qFfPg0+IyCVFNfDWUfdo4FGxcixRJM8+kzsZK6J8mZszIX1ehAY5xuBzx?= =?us-ascii?Q?A2tREjae4VMEgjN+vEOGYBjVhxE8aWk09N2LUdufRkoqMxuvyvtF23fGtrya?= =?us-ascii?Q?ps3SI5pChbuNHoz/bMR70tnJXMTR8qYLgiW/kraWe1DwtFeTt7x4D0MyXPi4?= =?us-ascii?Q?Ql1wYMFjxU7xTC7j63IrZQjHDGLcZGPbrmEk2oTLs2fGpGy3DmbUK13Y/Fqq?= =?us-ascii?Q?bNceI0QEwhnhOOy1FvcG5K2AK6lspnGujshURzdYDLvCJZaoXaM2KYYUcsO1?= =?us-ascii?Q?APDTLHzhXaLDNuy6KjUbE8mSWRn37lyx5IB2hVrK5aSxAs/wXkULBLgfQ4no?= =?us-ascii?Q?DWex/PpOCJtI0Bt9DkZfLo7KpRp9RNhRoH+cgNPwS3NpyjwUNb6MpQ=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:39:53.6770 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 01e777bd-c9eb-490d-c0e8-08dd62130240 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4303 Content-Type: text/plain; charset="utf-8" The load balancer will start caching the sg_lb_stats during load balancing and propagate it up the sched domain hierarchy in the subsequent commits. Increase the probability of load balancing intervals across domains to be aligned to improve the reuse efficiency of the propagated stats. Go one step further and proactively explore balancing at a higher domain if the next update time for a higher domain in before the next update time for its children. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3b1ed14e4b5e..60517a732c10 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11956,15 +11956,6 @@ get_sd_balance_interval(struct sched_domain *sd, i= nt cpu_busy) =20 /* scale ms to jiffies */ interval =3D msecs_to_jiffies(interval); - - /* - * Reduce likelihood of busy balancing at higher domains racing with - * balancing at lower domains by preventing their balancing periods - * from being multiples of each other. - */ - if (cpu_busy) - interval -=3D 1; - interval =3D clamp(interval, 1UL, max_load_balance_interval); =20 return interval; @@ -12126,7 +12117,7 @@ static void sched_balance_domains(struct rq *rq, en= um cpu_idle_type idle) int continue_balancing =3D 1; int cpu =3D rq->cpu; int busy =3D idle !=3D CPU_IDLE && !sched_idle_cpu(cpu); - unsigned long interval; + unsigned long interval, prev_sd_next_balance =3D 0; struct sched_domain *sd; /* Earliest time when we have to do rebalance again */ unsigned long next_balance =3D jiffies + 60*HZ; @@ -12136,6 +12127,8 @@ static void sched_balance_domains(struct rq *rq, en= um cpu_idle_type idle) =20 rcu_read_lock(); for_each_domain(cpu, sd) { + unsigned long next_interval; + /* * Decay the newidle max times here because this is a regular * visit to all the domains. @@ -12162,7 +12155,9 @@ static void sched_balance_domains(struct rq *rq, en= um cpu_idle_type idle) goto out; } =20 - if (time_after_eq(jiffies, sd->last_balance + interval)) { + next_interval =3D sd->last_balance + interval; + if (time_after_eq(jiffies, next_interval) || + (prev_sd_next_balance && time_after(prev_sd_next_balance, next_inter= val))) { if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) { /* * The LBF_DST_PINNED logic could have changed @@ -12174,6 +12169,7 @@ static void sched_balance_domains(struct rq *rq, en= um cpu_idle_type idle) } sd->last_balance =3D jiffies; interval =3D get_sd_balance_interval(sd, busy); + prev_sd_next_balance =3D sd->last_balance + interval; } if (need_serialize) atomic_set_release(&sched_balance_running, 0); --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2055.outbound.protection.outlook.com [40.107.93.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40FDA2641D5 for ; Thu, 13 Mar 2025 09:40:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.55 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858819; cv=fail; b=epxP8mTJU+pkzfZc7QR4Nm4V9uV/xZuR5zvpntyoPDYc7RZpUGp2HOMns4xlKHjYCfgoR5m3XFYNEqWgIMWWYiPXjoY5MWIbXoakUbxvaVourV/uKCbhUzXeBRP9/pilA55ac6Ma+zNWyXZCPxVCebkI7ToMIalgnBIP2cm5zIU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858819; c=relaxed/simple; bh=es9pVx2HhSueI0Mwksv+y1j3/Hm7M0odaToMzS28rtI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pN4WIlW/BO/Q5hvlbg4ekFMVN+4l+CZDxjnRcaSZpS63lyddHxutaAIIURAKoB8H6R4SMUb1qvWyEARA+dZcgfS6JJRAp+gh3bNlF0tL7k2bql67AtN49gbJ3rFhuB9ITp0y656nYhRw5cLBeevtl2g5z+lTJeUCF/RXSgXZGpo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=s2H+CjOh; arc=fail smtp.client-ip=40.107.93.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="s2H+CjOh" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=j15vRzY9V3zjj1DeD0HsRRsUHjyvv44pLrXgKyvhiobKVvqdmQ2MP/esNWK8mTgs23Bas5PUpUG+XXx/Q9A28DJE4qD9UAcHXHaa1MSebkBW8pS45vF54q6LcR/y4LZIj0hrWH7JktKvb6PNu1jPt5u+4IuTJX1TfY8a1+8EGJ2eKm9zwVQE1PI7wboU5dPjQrHXktAtfazTWDXFqK/1tcHoN4AZxm61u8G3SqZGTT/lOZWMncy8W31UGAKWFknsf4UMWhwVZoFwKhovAgKyoBjK8497iQlQ3R+zkGtL2C/aNU7LNLgMJj9ueJmljWogVq2mQAaE95EHQZjIVmh9kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0gJNKOqMGALhKZBgfD/6DKGI47PKEPHqifwKQMim4Fc=; b=PU9YTtFC9DEfha/wmfm9EF5DPYeeSI0jSXAkj3SgFnial0OeLcyIFrs9qP/A4DkHhhtMkPbGriu6YtQ+yy2M4qYOsnoX1qpt0PhrwqQkk3hNG+nno16Db7cxz2z0b1WMHaVzVWGxLTPSrcTGkAR8IfO4Y8I9nRzWk5LyGRFlX4uBhDAyVALm/QcQxiu9OFhoagVNOAyfUdMo+DOwQuMqXKi4naER8mUm7vtERfMtfpJeHf0ff0zVmdKASPgEUdMWn4buw3PYteEdIOqsh/ZjGBx/xRuJHcBWBwexfWUWTI7reppYV4HNcPYzqdnltJndc+vVONb+Q0aL8CjLFrWaLg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0gJNKOqMGALhKZBgfD/6DKGI47PKEPHqifwKQMim4Fc=; b=s2H+CjOh8DEj5ps7P6nfm4cWoC30VfrJnd/xBqTGYJW/LXfurABoJtoFnx1WkJ1Cx7Lrnc1xVtRbyvWzRazkL25mOdYzxs2giFDP0f+px7p6YmAY3WawyxOy8MJQy+WgEj6PWrQeryFwa+sFZXt3nkt5tjbrp8B1jivKot42Itg= Received: from MW4P223CA0021.NAMP223.PROD.OUTLOOK.COM (2603:10b6:303:80::26) by CH2PR12MB4104.namprd12.prod.outlook.com (2603:10b6:610:a4::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.28; Thu, 13 Mar 2025 09:40:13 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:303:80:cafe::ae) by MW4P223CA0021.outlook.office365.com (2603:10b6:303:80::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.26 via Frontend Transport; Thu, 13 Mar 2025 09:40:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:40:12 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:40:06 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 7/8] sched/fair: Retrieve cached group stats from sg_lb_stats_prop Date: Thu, 13 Mar 2025 09:37:45 +0000 Message-ID: <20250313093746.6760-8-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|CH2PR12MB4104:EE_ X-MS-Office365-Filtering-Correlation-Id: 74c35184-9c45-49a2-9cff-08dd62130d88 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?60l77rToGuxyzt/C/GCDm+ozCqnDvgjkC4LftdKI+4bKCQp5RZ/iwrTtjzOx?= =?us-ascii?Q?Z5hIIR/EpTv3L/fqxEbiofKdoLENMYKQNjz5TyClr+YbOL3fd9fUKtTkUk1q?= =?us-ascii?Q?KUnzMvbZG0STI+V5npJPiH9CwAAXo3OQh2NRLgo33MmlnRrf39cM1wipIDnU?= =?us-ascii?Q?YeLzHik/hAzy24EOz15SzSYFdh6VCdZ9vMhZLulVRzedmWdwiF7tPaMTtjRV?= =?us-ascii?Q?T0jyj4S7rmSdW3oyI+XsNlP+OojFUaSncN8OQGU5HFzvIrFF0lUmY6TjrpVu?= =?us-ascii?Q?OsyiAhQX1Nx2EYj3//s2pYY5KLeQND+3kpC0UqZImrUQ7ySsIg17gESkakdn?= =?us-ascii?Q?Ce7HSPI/e0rwhzjBWsILhCR35q4KW1RkMdakNcytnD4Q5xVtWfogOTAAsq8e?= =?us-ascii?Q?BdNOxSV8SwFpnRR7G2Lr6Zhdo75ahunszyMoDESefFdJ9eS7ukIJ1R0XZy2m?= =?us-ascii?Q?C9vZBWLFagZ0zceWVYIpialKNqGTj2puSh7gVM2aPiMBvlD9H5hZ6PKUjtkm?= =?us-ascii?Q?aJXwEFU2v7I7H+5duOAplfuS/9evGmeD1D5KaQKVjzS+mNHBWNiLjW04z3U7?= =?us-ascii?Q?/ZvfdIpSIDD7wPZLR0ZgoN57ulN6sSBVEJNYPaKhtMevlBiy1RHY2il28Ve+?= =?us-ascii?Q?m46o4jZvIgtgPytwEBn4+3nHgiVYzV8q+t74aVU1Cu3nATrRu+G3ba5NzR+B?= =?us-ascii?Q?AeEp8SI/fne5fvt3CbiVFRENWX36V/Xi84cusrEgXrVb0kXAXjzQBkf1pvEA?= =?us-ascii?Q?Gy10Fox+rjOWVNahWzX6DAEwVOfrTyu5SdQ0IZu8KSamP/NLeHE8UnCVsKWe?= =?us-ascii?Q?552m4OPTVrK/pOEi8e1i5gdP5umIn6FOLwC/ZyYZ1AT7BX119plXa1720Cf/?= =?us-ascii?Q?lVCuNV8SThk7LLhMRM55+Trhqxx2/cUzcDwiQfa9c/v1Hb2mOhctj626niIK?= =?us-ascii?Q?3c7CWQfyTuIWhUbfqfGG92DTfpgaCQyBOIBLD5i2QSopnbR8UOYl6DrwyYkh?= =?us-ascii?Q?9+DguIhJwNWlJM/jgLxCFQhxNxzGUbvaGGXl69NDFXPCv6QYtBHfgBUNVLyU?= =?us-ascii?Q?VsGxVBOry8YcIwq8H1g3SPoEV3vsV+S7ik/8soGi/k4NUJawCmVhwcG2uNvZ?= =?us-ascii?Q?+2jFUvtcO1ec69hbJplOeTA6F15vzDi9zkUa6up1mFNKq4j+90Rs9FqmOPjC?= =?us-ascii?Q?zHZbT6jslmA18/F6l8bREx69kBQAH3APvPm/USEr8JiEMdP1FT3GYIyRZUKm?= =?us-ascii?Q?du+q63dHNeeKSzHI/NZ8LRnxsU2DXrEmBvr2FAqwG0tQqUw4zaGWe4sLgXs1?= =?us-ascii?Q?9Wp2ZmmpMMnPz/C1aFZYeafK7jk3tqeEnq/yRawhqzassSpLOPX114bdpsvW?= =?us-ascii?Q?yALSxIZ3z0EtGWmr5DB1MlCeoi4G3B4NN11pYpPv7fD9VIcJbthRHNkm19rU?= =?us-ascii?Q?XOVMVEfLOzkjsn8h9++sE2PkwPZ0hfqkMuGzVhB1Q/SUZIsJPt5gnA=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:40:12.6654 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 74c35184-9c45-49a2-9cff-08dd62130d88 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4104 Content-Type: text/plain; charset="utf-8" Allow update_sg_lb_stats() to retrieve the group stats cached in sg_lb_stats_prop saved by another CPU performing load balancing around the same time (same jiffy) Current implementation without invalidation of cached stats have few limitations namely that the stats reuse is limited to busy load balancing since stats can only be updated once a jiffy. Newidle Balance can happen frequently and concurrently on many CPUs which can result in readers seeing inconsitent values for the propagated stats. For this iteration, the focus is to reduce the time taken for busy load balancing allowing the busy CPU to resume renning the task as quickly as possible. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 83 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 81 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 60517a732c10..3b402f294f0b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10275,6 +10275,75 @@ sched_reduced_capacity(struct rq *rq, struct sched= _domain *sd) return check_cpu_capacity(rq, sd); } =20 +static inline int can_retrieve_stats(struct sched_domain *sd, enum cpu_idl= e_type idle) +{ + /* + * Only under perioric load balancing can we ensure that no concurrent + * CPUs modifies the stats being propagated upwards since + * should_we_balance() can allow multiple concurrent newidle balance + * to progress and an idle -> busy transition for idle balance will + * require the stats to be recomputed since idleness metrics will + * change with migration. + */ + if (idle) + return 0; + + /* + * If individual groups are separate NUMA domains, migrations can cause + * preferred task statistics to change and will require recomputing of + * stats. + */ + if (sd->child && (sd->child->flags & SD_NUMA)) + return 0; + + /* + * misfit_task_load requires recalculation on SD_ASYM_CPUCAPACITY + * domains. Skip caching stats for them. + */ + if (sd->flags & SD_ASYM_CPUCAPACITY) + return 0; + + /* + * TODO: For CPU_IDLE case, invalidate stats for an idle -> busy + * transition but for the time being, save some cycles during busy + * load balancing. + */ + return 1; +} + +static inline int retrieve_cached_stats(struct sched_group *group, struct = sg_lb_stats *sg_stats) +{ + struct sched_domain_shared *sg_share =3D group->shared; + unsigned long current_jiffy =3D jiffies; + struct sg_lb_stats_prop *lb_prop; + + if (!sg_share) + return 0; + + lb_prop =3D (struct sg_lb_stats_prop *)sg_share->private; + if (!lb_prop) + return 0; + + /* Stale stats */ + if (READ_ONCE(lb_prop->last_update) !=3D current_jiffy) + return 0; + + /* + * Pairs against the update to sgs_prop->last_update to + * prevent readers from seeing an inconsistent value of + * the propagated stats from a concurrent update. + */ + smp_rmb(); + *sg_stats =3D lb_prop->sg_stats; + + /* + * If stats were read in the same interval, it cannot + * read an inconsistent state since stats are only + * updated once per jiffy. + */ + return time_before_eq(jiffies, current_jiffy); +} + /** * update_sg_lb_stats - Update sched_group's statistics for load balancing. * @env: The load balancing environment. @@ -10292,10 +10361,19 @@ static inline void update_sg_lb_stats(struct lb_e= nv *env, int i, nr_running, local_group, sd_flags =3D env->sd->flags; bool balancing_at_rd =3D !env->sd->parent; =20 - memset(sgs, 0, sizeof(*sgs)); - local_group =3D group =3D=3D sds->local; =20 + /* + * If stats can be retrieved, we are doing a busy load balancing. + * Skip right ahead to group_classify() since group_asym_packing and + * group_smt_balance is not possible under busy load balancing. + */ + if (can_retrieve_stats(env->sd, env->idle) && + retrieve_cached_stats(group, sgs)) + goto group_classify; + + memset(sgs, 0, sizeof(*sgs)); + for_each_cpu_and(i, sched_group_span(group), env->cpus) { struct rq *rq =3D cpu_rq(i); unsigned long load =3D cpu_load(rq); @@ -10360,6 +10438,7 @@ static inline void update_sg_lb_stats(struct lb_env= *env, if (!local_group && smt_balance(env, sgs, group)) sgs->group_smt_balance =3D 1; =20 +group_classify: sgs->group_type =3D group_classify(env->sd->imbalance_pct, group, sgs); =20 /* Computing avg_load makes sense only when group is overloaded */ --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2087.outbound.protection.outlook.com [40.107.244.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1C15263C8A for ; Thu, 13 Mar 2025 09:40:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858837; cv=fail; b=XYSLHE1CZ28QKTDQbyvZBrIk8vKwmtuJE/2ncrzPekr0xN7oH2R7IEnj2MnrSLaywT0kIR3evyKN2In6AHtXeZ/Wr2u3fqEbEpwZsT7RPktV8k+Sqyxb1zqeylFvyBVHfJr7N9LG0IpAAEWU42Y2H+Lf6rTTPX6Tl5NEolRPLV0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741858837; c=relaxed/simple; bh=ywuBgz505gv67Lr/Rj+8b6AlhISifiJ+qhKfoBYa4VM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=H/XPXZi/HoL+5Lq4Q2BLZbrDeEySFeq8aqTJQH2Ijc/ZXHr35ltPidnfBAohe92gVA/AosUO+MYhDxV1hrYqhdYc19KaNf69jVWVlZDCNFSj+fuzqtXdFHyjRfy+iwdSN+OByU1N1i10D1xHx8vx0fEQFwM2JqU2895/v7SsICs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=5omgTrrB; arc=fail smtp.client-ip=40.107.244.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="5omgTrrB" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=I8hrvMqrRDl4ZAVcKVZITSfgXOu8DaGdG+Fz/GoPDZyZ8vBek1V0u0uXF4J3FNnPkwUED7UomZJ2VoAQWpWbb9gbO/vmorW7sof2sCzJlDySTIup1L/ikYZYQH5ClZT8zk5jPxRaJzJKUzp9C8YNVSjUZUYLipfTOT+MMChxMs4DpAvX2A8OfE7wRGjGigy2ODHshJAqp0vMSf5biW3h/uR6DTxp9SiHJcOWXaikI3OmKAGi9TEnw22K8RMt2oOKFNSWyOkhBgUsByUoep43kxOPsqDtfazAiaOVJ2mL++66Rhz2misnI591jATH9EyMqsOvHDSJIk3ZT4xe1Iwj7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=twTnnaVI2hCo6fVQINPkV6z5gtBgDtebuMwOQd8tPLA=; b=i62lSYy1mzCgPhiLjq3LBwwe6eq6JuYfSYtI8U2PcHIrUzhG1+5r07ZIE/Ix48H8SybBl7kqFt3irM/w0t+vkEMoSQSFMvmiGPjmsom/9mtvNdVEkQ3UONcP3XOoObLFc+QaQJQtGsa/w2ouc2QXcGproK8SjkxbCESbdaYT+djGqRC5/bZsbc8rVrUDOKastcOVYfZqR99Da6ZX3vjknjPCPTlHkUpxzr9InAjNvNzUp0aXOssO7P8Neu6NqWndeMnrTRZUQ8L/KpZYU6YBX+J3AKohutxDYj0rVs/AR7a7fV5Mss+ZZRoRs8b0l2Aqp7d3npHS83audC4Wzoh6MQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=twTnnaVI2hCo6fVQINPkV6z5gtBgDtebuMwOQd8tPLA=; b=5omgTrrB5XfwV/YNq2w0vhE+41FTnqX/cwzLiK/KK3VOm7sQj6OPM8t8biaG4vh3vNhYQtyB47ukx97Xf/PcWxN4Z0AaOhIAHI0rxHWGR84mdZDNLA+5v+MH48OkDZtc0zv6n9rsayfuSpwftZZaJqmaw4sfThxyA84JE0RGvK0= Received: from PH1PEPF000132F7.NAMP220.PROD.OUTLOOK.COM (2603:10b6:518:1::3c) by SN7PR12MB6669.namprd12.prod.outlook.com (2603:10b6:806:26f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.27; Thu, 13 Mar 2025 09:40:30 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2a01:111:f403:f912::5) by PH1PEPF000132F7.outlook.office365.com (2603:1036:903:47::3) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.24 via Frontend Transport; Thu, 13 Mar 2025 09:40:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Thu, 13 Mar 2025 09:40:29 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 13 Mar 2025 04:40:23 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 8/8] sched/fair: Update stats for sched_domain using the sched_group stats Date: Thu, 13 Mar 2025 09:37:46 +0000 Message-ID: <20250313093746.6760-9-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|SN7PR12MB6669:EE_ X-MS-Office365-Filtering-Correlation-Id: 103b308e-a23a-4442-ce4d-08dd621317a2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?FU6Rv/zvcxU+ivk5e9Jd3cNjBPBnHkqYJhZWF+CqCnwdWQb8Mx48I9s8GLYM?= =?us-ascii?Q?dPcctaxnBjeCNNWgg9DUMxFMkPeNvp5Wau6zsvrQqeePvBEXYlL1HNvjEVbU?= =?us-ascii?Q?ZIzWDLED7K0HQGxJPtCOEJ7KjHKmxZfLY8YlB9K3uI7H33JcUcBL3nr0ViOn?= =?us-ascii?Q?Wx2WNMXkXXwc4nImEaCvB+t4NDM8V28jh6+L2iMTDsQJbITjhY+CwA1teOkb?= =?us-ascii?Q?Ucxu1ZgJp5QGNBCN4T+Fmk1Ly7Hh6zbjburY6ZPVwyWLRBP1HDyhMr2jQgQX?= =?us-ascii?Q?riFJOxTHrUnOu3rGCx+5/5xrDDX2wI42BAeFIwIvtZrjq3vGFz4Uya4cHaQx?= =?us-ascii?Q?p9r/vpNKc8uo5R+Dj8mbWmEes6tRdQB9B9I0Cc5iEVvp5X5YmjWAd9hVOb+D?= =?us-ascii?Q?VNYqBkUmNa96Ix8156NBM48M7PH5yJGvX0SfxWNn/bRmlXqVsLHEROd5XKqD?= =?us-ascii?Q?nnSQySsOhwcQrcJVY9VAt6BRWUfWNudJBf/ej4V2Nbp2pZavFVYz358RF7pp?= =?us-ascii?Q?2+bD81m5o9FK6/7ZMku0sDmTt7kjVJEMbjuJMvnwSXMXlB3G+fgcpFHbFr8J?= =?us-ascii?Q?g8sUa6Wt4b+/z9HFJJ6nKINwfmfXspGOcSV1BiZ/GPtAQfSQ9Pg00q1ekB4o?= =?us-ascii?Q?cznf5Lvjx5HkAK0YACSvHwoIIAJ8YfN155B6XtEz28zliKs9gK3q1969Eevl?= =?us-ascii?Q?xPGSiYMREKOVm+kpwPV8d/XY6ypR7Q3xIxwZvr9dwlWQm4fAZcNINihq5MHw?= =?us-ascii?Q?xj4nOtQmPjomsZolsIFVUh+GDb7SSr/e4VE8Ip33cIirAMdeC4nUFhYc6sGq?= =?us-ascii?Q?dWd4EhyKmLevGZhjf5mt5iBF78TIMJN2EOCYQupWUBIowNXPWWeVe0NasdCY?= =?us-ascii?Q?aS7DmSShMQ2wg4YLCKCuh2Kbh08JE6VNCpdrm+l7MKb6YcKSZYhY9UJ257ST?= =?us-ascii?Q?VrnsNaw20JKhxeP8YUbvRLmob6tzNuOaucnUsaKPmR1zkMbh74RM0jXVqeTU?= =?us-ascii?Q?/0dttq6ESzzR5UqS9nUZa0M1Xricm41hGeWfcF+g5o6DNfl3Qqk/0lH+bJIB?= =?us-ascii?Q?MYXAa9g1KQjyII6xybZRPCG0Wj2ze5b8czWDFFxoEhhQZER2lnOOp5q+PLNG?= =?us-ascii?Q?1ywpiNivWocj9TQNdZjbgvvqCjoXd4iQcnISU6eJP/hcK4kSQLD6IHI+/sqM?= =?us-ascii?Q?Y/uFjKpqHGTv8vtUfIrMHKPHB5Zc1HJI7SAplM/e9RYIMYYSFPhe5DNK1GPV?= =?us-ascii?Q?QWeqYDkzZSmCwXXaTkZ0wpxn2Reoz6ZTlNiShp7GJqQx0ZlOTEWg2G29CJrh?= =?us-ascii?Q?HuQTH9ZJdU+dZKFGcuil95NgQ0XpM6+hK+Qh+JYyq2O786ts32PE74b6WkoY?= =?us-ascii?Q?ziB5ZI8zMq8OBrtB9VSlHZOhL2Jvmf8mamFuC7By+VjtR8+0Pvv8hHErbDP8?= =?us-ascii?Q?yP1+QuljetTDJlSM2AK/xDetAtKY2l6I/RtsT65CRZASwh6uFFYAcg=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Mar 2025 09:40:29.5362 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 103b308e-a23a-4442-ce4d-08dd621317a2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6669 Content-Type: text/plain; charset="utf-8" Aggregate the individual sched_group stats to compute the stat for the entire sched_domain. Cache this in sd->shared which the sg->shared also points to for the corresponding sched_group of sd for its parent. This ensures that the stats are readily available for the higher domains if the load balancing continues. With the new infrastructure in place, following are the benchmark numbers: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : hackbench Units : Normalized time in seconds Interpretation: Lower is better Statistic : AMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Case: tip[pct imp](CV) stats_prop[pct imp](CV) 1-groups 1.00 [ -0.00](10.12) 1.09 [ -9.11](11.93) 2-groups 1.00 [ -0.00]( 6.92) 1.00 [ -0.22]( 4.57) 4-groups 1.00 [ -0.00]( 3.14) 0.99 [ 0.83]( 1.77) 8-groups 1.00 [ -0.00]( 1.35) 1.00 [ -0.31]( 2.24) 16-groups 1.00 [ -0.00]( 1.32) 0.99 [ 0.84]( 0.67) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : tbench Units : Normalized throughput Interpretation: Higher is better Statistic : AMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Clients: tip[pct imp](CV) stats_prop[pct imp](CV) 1 1.00 [ 0.00]( 0.43) 0.99 [ -0.87]( 1.34) 2 1.00 [ 0.00]( 0.58) 1.02 [ 2.14]( 0.29) 4 1.00 [ 0.00]( 0.54) 1.01 [ 1.24]( 0.82) 8 1.00 [ 0.00]( 0.49) 1.01 [ 0.62]( 0.97) 16 1.00 [ 0.00]( 1.06) 1.01 [ 0.94]( 0.70) 32 1.00 [ 0.00]( 1.27) 0.99 [ -1.24]( 1.38) 64 1.00 [ 0.00]( 1.54) 1.00 [ -0.43]( 0.36) 128 1.00 [ 0.00]( 0.38) 1.00 [ -0.01]( 1.22) 256 1.00 [ 0.00]( 1.85) 1.02 [ 1.58]( 0.90) 512 1.00 [ 0.00]( 0.31) 1.01 [ 0.76]( 1.19) 1024 1.00 [ 0.00]( 0.19) 1.00 [ 0.44]( 0.35) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : stream-10 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test: tip[pct imp](CV) stats_prop[pct imp](CV) Copy 1.00 [ 0.00](11.31) 1.02 [ 1.69]( 6.44) Scale 1.00 [ 0.00]( 6.62) 1.01 [ 0.80]( 5.37) Add 1.00 [ 0.00]( 7.06) 1.02 [ 1.54]( 6.72) Triad 1.00 [ 0.00]( 8.91) 1.01 [ 1.36]( 6.73) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : stream-100 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test: tip[pct imp](CV) stats_prop[pct imp](CV) Copy 1.00 [ 0.00]( 2.01) 0.98 [ -1.55]( 2.15) Scale 1.00 [ 0.00]( 1.49) 1.00 [ 0.23]( 0.58) Add 1.00 [ 0.00]( 2.67) 1.01 [ 0.65]( 1.95) Triad 1.00 [ 0.00]( 2.19) 1.01 [ 0.61]( 1.37) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : netperf Units : Normalized Througput Interpretation: Higher is better Statistic : AMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Clients: tip[pct imp](CV) stats_prop[pct imp](CV) 1-clients 1.00 [ 0.00]( 1.43) 1.00 [ 0.17]( 0.32) 2-clients 1.00 [ 0.00]( 1.02) 1.01 [ 1.00]( 0.44) 4-clients 1.00 [ 0.00]( 0.83) 1.01 [ 0.62]( 0.36) 8-clients 1.00 [ 0.00]( 0.73) 1.00 [ -0.11]( 0.65) 16-clients 1.00 [ 0.00]( 0.97) 1.00 [ 0.49]( 0.77) 32-clients 1.00 [ 0.00]( 0.88) 1.00 [ 0.30]( 0.94) 64-clients 1.00 [ 0.00]( 1.49) 1.00 [ 0.36]( 1.57) 128-clients 1.00 [ 0.00]( 1.05) 1.00 [ 0.14]( 1.46) 256-clients 1.00 [ 0.00]( 3.85) 1.00 [ -0.04]( 4.85) 512-clients 1.00 [ 0.00](59.63) 1.00 [ -0.02](62.28) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : schbench Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) stats_prop[pct imp](CV) 1 1.00 [ -0.00]( 6.67) 0.76 [ 24.44](35.80) 2 1.00 [ -0.00](10.18) 0.87 [ 13.04](10.38) 4 1.00 [ -0.00]( 4.49) 1.04 [ -4.26]( 3.14) 8 1.00 [ -0.00]( 6.68) 0.98 [ 1.89]( 8.07) 16 1.00 [ -0.00]( 1.87) 1.03 [ -3.28]( 5.21) 32 1.00 [ -0.00]( 4.01) 0.98 [ 2.20]( 1.31) 64 1.00 [ -0.00]( 3.21) 1.00 [ -0.00]( 3.23) 128 1.00 [ -0.00](44.13) 1.06 [ -6.43](113.66) 256 1.00 [ -0.00](14.46) 1.04 [ -3.52]( 8.43) 512 1.00 [ -0.00]( 1.95) 1.02 [ -1.80]( 1.14) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : new-schbench-requests-per-second Units : Normalized Requests per second Interpretation: Higher is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) stats_prop[pct imp](CV) 1 1.00 [ 0.00]( 0.46) 1.00 [ 0.00]( 0.55) 2 1.00 [ 0.00]( 0.15) 0.99 [ -0.88]( 0.26) 4 1.00 [ 0.00]( 0.15) 0.99 [ -0.59]( 0.15) 8 1.00 [ 0.00]( 0.15) 0.99 [ -0.88]( 0.26) 16 1.00 [ 0.00]( 0.00) 1.00 [ -0.29]( 0.15) 32 1.00 [ 0.00]( 3.40) 1.07 [ 6.59]( 0.16) 64 1.00 [ 0.00]( 7.09) 1.00 [ -0.38]( 0.96) 128 1.00 [ 0.00]( 0.00) 1.00 [ 0.00]( 0.20) 256 1.00 [ 0.00]( 1.12) 1.00 [ -0.30]( 1.50) 512 1.00 [ 0.00]( 0.22) 1.05 [ 4.86]( 0.71) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : new-schbench-wakeup-latency Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) stats_prop[pct imp](CV) 1 1.00 [ -0.00](19.72) 0.85 [ 15.38](16.75) 2 1.00 [ -0.00](15.96) 1.00 [ -0.00]( 0.00) 4 1.00 [ -0.00]( 3.87) 1.00 [ -0.00]( 4.08) 8 1.00 [ -0.00]( 8.15) 1.00 [ -0.00](11.71) 16 1.00 [ -0.00]( 3.87) 0.92 [ 7.69]( 4.19) 32 1.00 [ -0.00](12.99) 0.73 [ 26.67]( 0.00) 64 1.00 [ -0.00]( 6.20) 1.12 [-12.50]( 9.94) 128 1.00 [ -0.00]( 0.96) 0.98 [ 1.55]( 0.95) 256 1.00 [ -0.00]( 2.76) 0.99 [ 1.45]( 1.38) 512 1.00 [ -0.00]( 0.20) 1.20 [-20.42]( 0.00) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : new-schbench-request-latency Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) stats_prop[pct imp](CV) 1 1.00 [ -0.00]( 1.07) 1.02 [ -2.08]( 0.13) 2 1.00 [ -0.00]( 0.14) 1.04 [ -3.97]( 0.13) 4 1.00 [ -0.00]( 1.39) 1.03 [ -3.15]( 0.13) 8 1.00 [ -0.00]( 0.36) 1.03 [ -3.16]( 0.00) 16 1.00 [ -0.00]( 1.18) 1.02 [ -1.59]( 0.75) 32 1.00 [ -0.00]( 8.42) 0.81 [ 19.08]( 0.25) 64 1.00 [ -0.00]( 4.85) 1.01 [ -1.10]( 2.58) 128 1.00 [ -0.00]( 0.28) 1.00 [ -0.21]( 0.38) 256 1.00 [ -0.00](10.52) 0.95 [ 4.74]( 6.94) 512 1.00 [ -0.00]( 0.69) 1.09 [ -8.99]( 0.27) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : Various longer running benchmarks Units : %diff in throughput reported Interpretation: Higher is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Benchmarks: %diff ycsb-cassandra -0.54% ycsb-mongodb 0.09% deathstarbench-1x -0.30% deathstarbench-2x 2.38% deathstarbench-3x 0.58% deathstarbench-6x 0.62% hammerdb+mysql 16VU 0.76% hammerdb+mysql 64VU 0.74% * The tail latencies reported by schbench increases possibly due to the sync in load balancing across multiple domains however it remains to be investigated. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 99 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 92 insertions(+), 7 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3b402f294f0b..212bee3e9f35 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10275,6 +10275,38 @@ sched_reduced_capacity(struct rq *rq, struct sched= _domain *sd) return check_cpu_capacity(rq, sd); } =20 +static inline void cache_sd_stats(struct sched_domain *sd, struct sg_lb_st= ats *sd_stats) +{ + struct sched_domain_shared *sd_share =3D sd->shared; + unsigned long current_jiffy =3D jiffies; + struct sg_lb_stats_prop *lb_prop; + + if (!sd_share) + return; + + lb_prop =3D (struct sg_lb_stats_prop *)sd_share->private; + if (!lb_prop) + return; + + /* Concurrent load balancing instance already updated the stats */ + if (READ_ONCE(lb_prop->last_update) =3D=3D current_jiffy) + return; + + scoped_guard(raw_spinlock_irqsave_try, &lb_prop->stats_lock) { + if (READ_ONCE(lb_prop->last_update) =3D=3D current_jiffy) + break; + + lb_prop->sg_stats =3D *sd_stats; + + /* + * Pairs against readers checking the last_update + * before reading the cached stats. + */ + smp_wmb(); + WRITE_ONCE(lb_prop->last_update, current_jiffy); + } +} + static inline int can_retrieve_stats(struct sched_domain *sd, enum cpu_idl= e_type idle) { /* @@ -10344,6 +10376,35 @@ static inline int retrieve_cached_stats(struct sch= ed_group *group, struct sg_lb_ return time_before_eq(jiffies, current_jiffy); } =20 +/** + * aggregate_sd_prop_stats - Compute sched domains's stats from group stat= s. + * @env: The load balancing environment. + * @sgs_prop: variable to hold the statistics to propagate for the sd + * @sgs: Group stat that was computed or retrieved + */ +static inline void aggregate_sd_stats(struct lb_env *env, + struct sg_lb_stats *sd_stats, + struct sg_lb_stats *sg_stats) +{ + sd_stats->group_load +=3D sg_stats->group_load; + sd_stats->group_util +=3D sg_stats->group_util; + sd_stats->group_runnable +=3D sg_stats->group_runnable; + sd_stats->sum_h_nr_running +=3D sg_stats->sum_h_nr_running; + sd_stats->sum_nr_running +=3D sg_stats->sum_nr_running; + sd_stats->idle_cpus +=3D sg_stats->idle_cpus; + sd_stats->group_capacity +=3D sg_stats->group_capacity; + sd_stats->group_weight +=3D sg_stats->group_weight; + sd_stats->overloaded |=3D sg_stats->overloaded; + sd_stats->overutilized |=3D sg_stats->overutilized; + +#ifdef CONFIG_NUMA_BALANCING + if (env->sd->flags & SD_NUMA) { + sd_stats->nr_numa_running +=3D sg_stats->nr_numa_running; + sd_stats->nr_preferred_running +=3D sg_stats->nr_preferred_running; + } +#endif +} + /** * update_sg_lb_stats - Update sched_group's statistics for load balancing. * @env: The load balancing environment. @@ -11041,9 +11102,18 @@ static inline void update_sd_lb_stats(struct lb_en= v *env, struct sd_lb_stats *sd { struct sched_group *sg =3D env->sd->groups; struct sg_lb_stats *local =3D &sds->local_stat; - struct sg_lb_stats tmp_sgs; - unsigned long sum_util =3D 0; bool sg_overloaded =3D 0, sg_overutilized =3D 0; + struct sg_lb_stats tmp_sgs, sd_stats; + unsigned long sum_util =3D 0; + bool should_prop =3D false; + + /* + * If a parent domain exists and the cached stats can be retrieved when + * load balancing there, aggregate the statistics at current domain + * to be retrieved when load balancing at parent. + */ + if (env->sd->parent && can_retrieve_stats(env->sd->parent, env->idle)) + should_prop =3D true; =20 do { struct sg_lb_stats *sgs =3D &tmp_sgs; @@ -11061,21 +11131,36 @@ static inline void update_sd_lb_stats(struct lb_e= nv *env, struct sd_lb_stats *sd =20 update_sg_lb_stats(env, sds, sg, sgs); =20 + if (should_prop) + aggregate_sd_stats(env, &sd_stats, sgs); + if (!local_group && update_sd_pick_busiest(env, sds, sg, sgs)) { sds->busiest =3D sg; sds->busiest_stat =3D *sgs; } =20 /* Now, start updating sd_lb_stats */ - sds->total_load +=3D sgs->group_load; - sds->total_capacity +=3D sgs->group_capacity; - sg_overloaded |=3D sgs->overloaded; - sg_overutilized |=3D sgs->overutlizied; + if (!should_prop) { + sds->total_load +=3D sgs->group_load; + sds->total_capacity +=3D sgs->group_capacity; + sg_overloaded |=3D sgs->overloaded; + sg_overutilized |=3D sgs->overutilized; + sum_util +=3D sgs->group_util; + } =20 - sum_util +=3D sgs->group_util; sg =3D sg->next; } while (sg !=3D env->sd->groups); =20 + if (should_prop) { + sds->total_load =3D sd_stats.group_load; + sds->total_capacity =3D sd_stats.group_capacity; + sg_overloaded =3D sd_stats.overloaded; + sg_overutilized =3D sd_stats.overutilized; + sum_util =3D sd_stats.group_util; + + cache_sd_stats(env->sd, &sd_stats); + } + /* * Indicate that the child domain of the busiest group prefers tasks * go to a child's sibling domains first. NB the flags of a sched group --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2069.outbound.protection.outlook.com [40.107.243.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15D2C19D071 for ; Sun, 16 Mar 2025 10:29:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.69 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742120999; cv=fail; b=l8RY5Op15J8LJ19kl4nQF+523qlKnM1tYweqmrGi2eOH4wuwSeKx3r2gwzLzXQJxwllBCtf7h8ftHrcPSdapMAn/krWLUTt2Vcwawc9Y021bl6xXZ9ha3h0APx9v2W7bzfrhzxHgLUNusRU0pzpP8NqoRPERG9WeLmSTWNnvP6o= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742120999; c=relaxed/simple; bh=+b86qjF0Bm+ClJbxnvxpXfXvSVbJaKd/Q3xk+sZxwao=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=jjxibiX5gvV2QFilqWjxY/S48lWB4vr3lejGJOWelPG0phcGck0hFjtC8DkK4+eZl6OlLULTFKRFBsqGJmdt6b7Dt3PltqUkibruWD7sYVPt980/DArGC/b4z1nKXGwBe+151dJlgHI11VMFdyfHvYV+OVeM0l72wUPyadIoKJc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=3eO3Dcmv; arc=fail smtp.client-ip=40.107.243.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="3eO3Dcmv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Fxi5fd8FmAExXdUry2HqSdiSMmScpR/9OtS+m/xait0XcRt2AJLuNow0CdVHehwABH+/bzHUO8NrJpzrJEq7ERtYIbGmeji4eeeeFYCK9naeItqJ7ZE+R9rpoUd6qvCoQq6IdVYUyIuG/o/s4aebVLvKGKAe6fAXwhvDMXgzpfvicXoqGynQMHAnK4UoMzI60bJ4nJNZom0rmUIUxYHXtpK5YUESbE+iVyHKTbrJDmvlnXI65qIDxkuZSAL03vVKDFBUVD4u/+Sl+VF8oecADk6BmqXDsCNBx8ZcG65ENqB3wnbjQGbDMv5z+QxgmxUIrsfkTFjdGO3Tw5hM57bxYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eNTY6AKIOQdkAkihFXXqrPaTa0A/RIvLWqcc9RPyn34=; b=un8A94wvhy0LKsk77tNA1WMR/jcx0bU+SLY7VDrlcdBvpDDQz0mpyZAwREZd6J2hCG464sCt2MneGG9Nc9eTLDcQro7+VwfvTTN+D98b0aY7NloBkqHKEeaWFwAoj4pook46qH/BZrBCkNZb8BCjVZqW6YDMia12ej/N4y7rHImU2VKJ7c3Cn49PRB9sjfZJPpoOw1gSms4pfQgZ4Zyyv+N5SEnoBXr84A0cjIWDDRjopK9Yq4FS8byXkzjceteeuDA3dtSk1Fud68wElBc9dfK/ZXdqoErwfN73SiIfJpinJ5hxHxnktyIlwxiXMoIVO3eQN4TmghOvXbL+8ei2rg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eNTY6AKIOQdkAkihFXXqrPaTa0A/RIvLWqcc9RPyn34=; b=3eO3DcmvHh9LPlG8E1RSIUsjFiRPaUcIyachglYlRWBIvZy+Viwt1MaVzHrweNEeO6FvDLZB7Vs+Sw/cuQTZ6ZWniz/ftvfDXazU7c4r3LQm2u8jlGjEIhZCzYCDYfuMDY38F9+qIc3Whl3G4IdS7gHmIz0nDcLwitx+iWUdqw0= Received: from BLAPR03CA0100.namprd03.prod.outlook.com (2603:10b6:208:32a::15) by CH3PR12MB8852.namprd12.prod.outlook.com (2603:10b6:610:17d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.28; Sun, 16 Mar 2025 10:29:55 +0000 Received: from BL02EPF00021F69.namprd02.prod.outlook.com (2603:10b6:208:32a:cafe::9e) by BLAPR03CA0100.outlook.office365.com (2603:10b6:208:32a::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.29 via Frontend Transport; Sun, 16 Mar 2025 10:29:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL02EPF00021F69.mail.protection.outlook.com (10.167.249.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Sun, 16 Mar 2025 10:29:55 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 16 Mar 2025 05:29:50 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 10/08] sched/fair: Compute nr_{numa,preferred}_running for non-NUMA domains Date: Sun, 16 Mar 2025 10:29:11 +0000 Message-ID: <20250316102916.10614-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F69:EE_|CH3PR12MB8852:EE_ X-MS-Office365-Filtering-Correlation-Id: c294e2e2-1834-4be2-965d-08dd64757e5f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?KlEqe3gtsJ34PeVAQfjsfViDSPpV6XmkksmPoOuyTPGmo9pooVq0ki2K+z1D?= =?us-ascii?Q?tWyUusj7dfm6lpGbfALfjMAeua9R+UHWzxZvFm+9jaUn1z/SsoCawHuZyAd+?= =?us-ascii?Q?+5O7OIGdtd6rnYgtKvCLnxMvLv9XiyeuL8eX2t4RUVjyEHBulzODANImPQGE?= =?us-ascii?Q?HLx5dgbmkxTUg7peLr6iJA/0jXBJOTF+czaBNee7sasAiwfuKdigYPeUAFwX?= =?us-ascii?Q?X+04eTGc7luVfutvHdhfxe+n0KrSbBvUbRNy59SVItjlSPZfiYEAge5ooldl?= =?us-ascii?Q?UbTvbrYgJr/nPI5nx8pynXN8Nwx7upekUXaNuyH4PpyNDSfbj36hEtNw9jMM?= =?us-ascii?Q?QRqbRgl4teXWMB0mD2IRhC+YUb3J/rEQvm8eEDvu2z4mohLwljUaxlZwt8Jx?= =?us-ascii?Q?VHq6Ct6836XYhgRWvGhDwLJOSqtNhQH/vGh+FaxaBzJB/hmIUbfonROPOqZL?= =?us-ascii?Q?pcksjItOlmhrHzZXiaI5n1J/bnV//R6Q43c3fibDQkk7fG/DGM3Srw4xTnUQ?= =?us-ascii?Q?mMZjvSdCkTq+wdNI82/WSTEIzd7l3jlgAlqqhIonVAd0PxAZByDJVom88hR9?= =?us-ascii?Q?SBbHUSgHTtqxRE3G5rPPIDfRP5SgZ9E5uchqz+5bT1cD99Z/pt7epbV4lPr2?= =?us-ascii?Q?PifB+u3m68MzCWK1BtLFrSq9mkbUvzabqrppyy0Ix/1Q/JgsUV9YIq+/TzAU?= =?us-ascii?Q?xgOuKGQf19mWpN2zlMdyM9wMe6c8izhlVhOteceR5NJPf7BN8MufX638hgdz?= =?us-ascii?Q?j6WFbPPXSS6vf8RxgJ9g59XDawW85LxReKKrZiXNhcwOw1o5YCw2F5OEqkoA?= =?us-ascii?Q?AwbK8jj8p5LOKBMFuFq9dSR0AUy8vNfZ11G4Trkt0vj+sL2Afkz30owC/h16?= =?us-ascii?Q?knFOAUMQAkHjkSRhkjPZs3XW9MB/8kRsU4WukI+FfeYlFCtcHGODjwzRq48t?= =?us-ascii?Q?3VIH1hHQPa90ohwnEmuqrW2NhNAlOATWA7213EKFDMywa32ddTao/cS2KWT7?= =?us-ascii?Q?wbv0l3TNxfHDaquhw56zBHEieHNeIZnJON77LM/utcvnHTPENv+DGXBGt/8q?= =?us-ascii?Q?GfwDeWwrHs5Ap9nLY19/mvSn0JkebwTgXGRIyleEzgyx8XGtMbRVzMINgE/B?= =?us-ascii?Q?IE5UkxiixOGDSUfGDhj/fVC2jH7YxQJ67M23wTFUY+DzX9D/XNg9F/AqEEDa?= =?us-ascii?Q?b3gqAONHChwuo+qpixFb1B7fiLI6Jc1BdGiSTXzQxjSAolKxN/hfSLIkk/zC?= =?us-ascii?Q?ngtimKwJxnTM4GIJa7Q7qQsv66jALGp1bzEnNOQNawcENKLg5v8zO31mJgcu?= =?us-ascii?Q?qX5X4ftWcLMMqfOUZL15KXT0zCvuVMBR0OGcR487F4Devd/ldkI5dM9N2+74?= =?us-ascii?Q?a/IZphIKPNQUSF9sr7zvlsEpBdP3RKVgnkChiG/uv0Zj7h2CW2NnryMXK6XR?= =?us-ascii?Q?KM6Ki/6JM5nA61wzk+OjqIMbs+sZXTb33Xn5UbKWymIOaCLYQjIx2w=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2025 10:29:55.0310 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c294e2e2-1834-4be2-965d-08dd64757e5f X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F69.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8852 Content-Type: text/plain; charset="utf-8" Migrations within a NUMA domain will not change nr_{numa,preferred}_running stats. Compute it for non-NUMA groups for it to be propagated and reused for the first NUMA domain when it exists. While at it, also clear sd_stats before aggregation. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 212bee3e9f35..d09f900a3107 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10398,10 +10398,8 @@ static inline void aggregate_sd_stats(struct lb_en= v *env, sd_stats->overutilized |=3D sg_stats->overutilized; =20 #ifdef CONFIG_NUMA_BALANCING - if (env->sd->flags & SD_NUMA) { - sd_stats->nr_numa_running +=3D sg_stats->nr_numa_running; - sd_stats->nr_preferred_running +=3D sg_stats->nr_preferred_running; - } + sd_stats->nr_numa_running +=3D sg_stats->nr_numa_running; + sd_stats->nr_preferred_running +=3D sg_stats->nr_preferred_running; #endif } =20 @@ -10464,11 +10462,8 @@ static inline void update_sg_lb_stats(struct lb_en= v *env, sgs->overloaded =3D 1; =20 #ifdef CONFIG_NUMA_BALANCING - /* Only fbq_classify_group() uses this to classify NUMA groups */ - if (sd_flags & SD_NUMA) { - sgs->nr_numa_running +=3D rq->nr_numa_running; - sgs->nr_preferred_running +=3D rq->nr_preferred_running; - } + sgs->nr_numa_running +=3D rq->nr_numa_running; + sgs->nr_preferred_running +=3D rq->nr_preferred_running; #endif if (local_group) continue; @@ -11112,8 +11107,10 @@ static inline void update_sd_lb_stats(struct lb_en= v *env, struct sd_lb_stats *sd * load balancing there, aggregate the statistics at current domain * to be retrieved when load balancing at parent. */ - if (env->sd->parent && can_retrieve_stats(env->sd->parent, env->idle)) + if (env->sd->parent && can_retrieve_stats(env->sd->parent, env->idle)) { + memset(&sd_stats, 0, sizeof(sd_stats)); should_prop =3D true; + } =20 do { struct sg_lb_stats *sgs =3D &tmp_sgs; --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2087.outbound.protection.outlook.com [40.107.94.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6876F188587 for ; Sun, 16 Mar 2025 10:30:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121020; cv=fail; b=oRq2cM7jprkET86S23feF+0kD761k+QUOAmA05tNTqCs0Xanp7QU9SLg9ejetIBWwPORw/Vh+OcxyagzasZSMfFrrWQ+ucd6BRheYVqe6ZfkjfcFq1oZMcXXsfDIwIwYtPcT0TRw6/OeZFIjvzXVaMJ7W0JlsHqPVYRtxjaLDVc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121020; c=relaxed/simple; bh=Af30TauJFBz70w5kUkaNKtxSL7QmhsiLcXaJlT8Tozk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TmPMJvnuqJIWVj9GyJ+2edoAFqb0uiGVzpKwD+vAfqF9iZ8fnLQVUpZItOTH5eCaR4Tz+k+ep+MfKk2AXQ3IBbq2tlrK12Rhg4JmBbmxnqnv+OSR7Tc81cVipBOXwkePrkDI3aGNvBJeD04oNvZHYbV3Qc7gHSne39M3PhQa50c= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=e5VDxqFi; arc=fail smtp.client-ip=40.107.94.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="e5VDxqFi" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cTQuDxDbxaTgkVrzQIdZaaprSpQEmXZ4gJObyP3m6F9dp7fFKGJrOSsoCjKRn1lYAaiZAPIdBN8uV0H4/eMoXZe+k8PwIIsMq8HhMN9ZsxeoavT9Jcl04rdY+MxbJ4fRuAyjfxLJVIblQAb8lTS87qOQtU3vGZcdfm7Fnz4V6oU1Jh9MZ4bqrhri3KxcNGnTMzb975WfLZHmKQXl1pYOUbNSNka/69Y5K9lNlZ28r1od7gNI8p44/ARums0obAGIWsTwbkC40PSSFA8HNdOur93APm+gepogOpMUi0Z87rO5ExgRgSTqHoPsLN0f3b835aPMuvXI9kCT+tqjTbGHcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pOfC8Nv4ppL5FLK5g0TCRM06/pbruq9zDQmqRF2qliE=; b=N2dLLDAUOXEBhPxPk5IXbBHPsdmdjr5B2NUOTbV6Y16f5cm042pt8gkPKbJSJ5wgSrom7K0NcW2Izggz5iGrpFlyziuk0ps+NRoAkX8kjwiAH/HLH8Aa7D08o2tZz7O0npJf1LUPZU85HTNuPMKazSlB8kGd3f5uFKxPU1I4fJw93bf0pwocUhF7g+XvYyeTUWbMYG3d+yiTNTZ7VYKdS4tpDV1h+g1Ym1RSR9HkuSJlfG/pU+Y+3ScFAfoOnnLsbSzPk4cIPERbrKxl20Oba3S06BhKw4Jsio/+eQH0sIheMHWM3082rGag0ADaNtboCb04O35rVGAK7Uhrq3NFqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pOfC8Nv4ppL5FLK5g0TCRM06/pbruq9zDQmqRF2qliE=; b=e5VDxqFiwzSq/bLJ58GIzxIqmHTf0JnH0IOHoVw4VoqbpOBXSqCDjFXstxVo005sfrROB8re3po9A+i588DWc/WDt+FiECdx5gbFaWV7hwxDIegpYlvUNmBAXZmHApUO7rzipEWTrqfH/3t/98d14zJQtDMqI6IFcdD91kZf8Z4= Received: from MN2PR20CA0057.namprd20.prod.outlook.com (2603:10b6:208:235::26) by CY5PR12MB6082.namprd12.prod.outlook.com (2603:10b6:930:2a::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.28; Sun, 16 Mar 2025 10:30:13 +0000 Received: from BL02EPF00021F6C.namprd02.prod.outlook.com (2603:10b6:208:235:cafe::63) by MN2PR20CA0057.outlook.office365.com (2603:10b6:208:235::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Sun, 16 Mar 2025 10:30:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL02EPF00021F6C.mail.protection.outlook.com (10.167.249.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Sun, 16 Mar 2025 10:30:12 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 16 Mar 2025 05:30:07 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 11/08] sched/fair: Move from "last_update" to stats versioning Date: Sun, 16 Mar 2025 10:29:12 +0000 Message-ID: <20250316102916.10614-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F6C:EE_|CY5PR12MB6082:EE_ X-MS-Office365-Filtering-Correlation-Id: f19ce577-6961-4347-0ea0-08dd647588eb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|82310400026|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?NyTZT1mqHlDqYhIKeZ+9Xn7g9A19XzE7BzECkuw1kKAGeKpGSG+KpVj74qts?= =?us-ascii?Q?MbhwpRvFbuahPshJnCP/AZKwd8MT9kNtUMLNVI0ldMVOywRcyCXIiWMKouYG?= =?us-ascii?Q?R7MdHuevZvTex5VlhP4wtkHvGFUuMcN3b/yHR15PEGV/aCtN47dEDbe86YtC?= =?us-ascii?Q?awJP4ec3Pz0oefuk4c5lUfUplm2jnZRIHHwE/WsCsa1wLtT+CSgmjkxeYppE?= =?us-ascii?Q?DeRbp+euwm1yO6bWtcW6ljzf4q03CmEAHW7abfg6cbTRYfWzBNzgaFDSZkkd?= =?us-ascii?Q?yrRL9DwdZbJtm6rfmNvdxUMcZuaREQZICuwJN4mnNtpPEksyjHdc0/LNQZFU?= =?us-ascii?Q?Xy7GnwzbomeRsJAWF5DieFSg30k8dIEIIV/XUFUz+x/sER4TMvlgdKfc5ldr?= =?us-ascii?Q?l3UZdXog0/IK8QWCBhX+8JyQDNpRTKrpseJbJaGqs13SwQ/ua1Uv70QEWsvx?= =?us-ascii?Q?HmQYgIllU8eZFYyoL1pvwCmJ46aHdQeHcRZDnpIDamfdVfV64LTpi6aXVNmo?= =?us-ascii?Q?u8f21NTB8yop/gzn6e5zMDg6lzsYScfef55E8fuJS1lZVVVJU56Hoa/Aad1G?= =?us-ascii?Q?mTmUnlyVIhgPdBchYaubYe/fBWfdH5reS66c0OwbWsq1XEZtYPIzIhuypqQP?= =?us-ascii?Q?V/Qspewv4rZaz0N4LYa9VR31Gs3ORYJYfBLNFSLy4AIwn02Wu6oWKEJO+V8S?= =?us-ascii?Q?ExkNnfymMRppjf0QaGK0PoC7cG0W7UuR1cephiflygmXpZY4PRVh0qHRP7gA?= =?us-ascii?Q?dmc/S82c9Wiqc31hxhUXc4As2/puTmwv8qgxSvTnSuBPt/iGwXoWTjMUJDRd?= =?us-ascii?Q?rulZ50LUJMOZILgezhsIV3H+KRD//Dq9spGNReQ6KiYddb+RDBA3rzCEX63a?= =?us-ascii?Q?/48ax/LOwA4puU7tII331oP6GypCldtpINfOxnLWs2YyPAJR60pwwXQR7rkM?= =?us-ascii?Q?KPnoIXHbQkbJhT2wrG85/M4ffaWRgR9A8+gRmyzcxXjHsCkJwLG4fbOc52L2?= =?us-ascii?Q?rOhGyUjSCnNPExMGR40+fqg/r48ZqzFpKL5NnIevWJWg9b+xlmp97rlUueTL?= =?us-ascii?Q?HLMd/ImxjkxKa4yDKLdfwAM4iuRj09d6FkG+D5Ll8dL51i+BfW7KrUIYPvIT?= =?us-ascii?Q?XuZ42dYX4K5whupS53WbsSTjLCOEVGmw4yX+iAjjDaCKLyIf2E6qPt3kvhwk?= =?us-ascii?Q?2KCtAvyn5uLVsxYvwQni/Q79mBcEtuMUaFu6LxyJTrI3AQhE0caKwur2lfjI?= =?us-ascii?Q?kirqsU5G1eVxQObNFC0LYbAsjKJn4OTmid8a0iyb6/5oHO95kwnP6t4uxLh7?= =?us-ascii?Q?/2xF5tW+tPiOLlCvWPa7VBuqCg6mZPgqDd7c/sT1HHzE/6h0fWWotp040UhW?= =?us-ascii?Q?0/FWsv5U7FxAY6BeAhciFGw4L+lrzb9PnRD5QDjlwZQy+8ORZJ4vSpScokl8?= =?us-ascii?Q?ckFrGOWK8LvpGLMCsqOSbDdBJpQcnV9Khpea6HaBvETpLkDcBUvL5CK6cGeg?= =?us-ascii?Q?lH9gmzyp2rY3W6U=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(82310400026)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2025 10:30:12.7067 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f19ce577-6961-4347-0ea0-08dd647588eb X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F6C.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6082 Content-Type: text/plain; charset="utf-8" The combination of "stats_lock" and jiffy based "last_update" is not scalable for newidle balance. Instead move to a versioning-based scheme where the version number helps with both readers reading consistent data without the need for a lock and writers using the version for both locking and indicating stats freshness. Additional semantics have been added for the writers to update state stats if the time elapsed since last update has crossed the 50us threshold. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 83 +++++++++++++++++++++++++++-------------- kernel/sched/sched.h | 22 ++++++++++- kernel/sched/topology.c | 3 +- 3 files changed, 77 insertions(+), 31 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d09f900a3107..6c486e194a9d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10275,11 +10275,13 @@ sched_reduced_capacity(struct rq *rq, struct sche= d_domain *sd) return check_cpu_capacity(rq, sd); } =20 -static inline void cache_sd_stats(struct sched_domain *sd, struct sg_lb_st= ats *sd_stats) +static inline void cache_sd_stats(struct lb_env *env, struct sg_lb_stats *= sd_stats) { - struct sched_domain_shared *sd_share =3D sd->shared; - unsigned long current_jiffy =3D jiffies; + struct sched_domain_shared *sd_share =3D env->sd->shared; struct sg_lb_stats_prop *lb_prop; + int cpu, retry_limit =3D 3; + u64 time, lock; + long version; =20 if (!sd_share) return; @@ -10288,23 +10290,52 @@ static inline void cache_sd_stats(struct sched_do= main *sd, struct sg_lb_stats *s if (!lb_prop) return; =20 - /* Concurrent load balancing instance already updated the stats */ - if (READ_ONCE(lb_prop->last_update) =3D=3D current_jiffy) + version =3D atomic_long_read_acquire(&lb_prop->version); + if (version < 0) /* Raced with a concurrent update. */ return; =20 - scoped_guard(raw_spinlock_irqsave_try, &lb_prop->stats_lock) { - if (READ_ONCE(lb_prop->last_update) =3D=3D current_jiffy) - break; + guard(irqsave)(); /* Minimize interruptions. */ + + cpu =3D smp_processor_id(); + time =3D sched_clock_cpu(cpu); =20 - lb_prop->sg_stats =3D *sd_stats; + /* Version is still fresh, no need to be rude yet. */ + if (version > 0 && (s64)(time - version) <=3D 50 * NSEC_PER_USEC) + return; =20 +retry: + /* + * Try to grab the stats for update. If the cmpxchg fails, + * a concurrent writer succeeded to grab the stats before + * this load balancing instance did. The acquire ordering + * also pairs against readers checking the version after + * reading the stats to ensure consistent state. + */ + lock =3D atomic_long_cmpxchg_acquire(&lb_prop->version, version, LLONG_MI= N); + + /* Someone else grabbed the version. */ + if (lock !=3D version) { /* - * Pairs against readers checking the last_update - * before reading the cached stats. + * Version is up for grabs! Try again. If the CPU grabs + * the lock next time around lock =3D version =3D 0 and this + * is skipped. If it cannot grab the version, lock !=3D 0 + * and we return from here thus ensuring on a single + * retry. */ - smp_wmb(); - WRITE_ONCE(lb_prop->last_update, current_jiffy); + if (!lock) { + version =3D 0; + goto retry; + } + return; } + + lb_prop->sg_stats =3D *sd_stats; + + /* + * Pairs against readers checking the version + * before reading the stats. + */ + atomic_long_set_release(&lb_prop->version, time); } =20 static inline int can_retrieve_stats(struct sched_domain *sd, enum cpu_idl= e_type idle) @@ -10346,8 +10377,8 @@ static inline int can_retrieve_stats(struct sched_d= omain *sd, enum cpu_idle_type static inline int retrieve_cached_stats(struct sched_group *group, struct = sg_lb_stats *sg_stats) { struct sched_domain_shared *sg_share =3D group->shared; - unsigned long current_jiffy =3D jiffies; struct sg_lb_stats_prop *lb_prop; + long version; =20 if (!sg_share) return 0; @@ -10356,24 +10387,22 @@ static inline int retrieve_cached_stats(struct sc= hed_group *group, struct sg_lb_ if (!lb_prop) return 0; =20 - /* Stale stats */ - if (READ_ONCE(lb_prop->last_update) !=3D current_jiffy) - return 0; - /* - * Pairs against the update to sgs_prop->last_update to - * prevent readers from seeing an inconsistent value of - * the propagated stats from a concurrent update. + * Pairs with writer atomically updating version after + * writing the stats. */ - smp_rmb(); + version =3D atomic_long_read_acquire(&lb_prop->version); + if (version <=3D 0) /* Stats have gone stale / being updated. */ + return 0; + *sg_stats =3D lb_prop->sg_stats; =20 /* - * If stats were read in the same interval, it cannot - * read an inconsistent state since stats are only - * updated once per jiffy. + * Pairs with writer atomically invalidating a version + * before updating the stats. */ - return time_before_eq(jiffies, current_jiffy); + smp_rmb(); + return atomic_long_read(&lb_prop->version) =3D=3D version; } =20 /** @@ -11155,7 +11184,7 @@ static inline void update_sd_lb_stats(struct lb_env= *env, struct sd_lb_stats *sd sg_overutilized =3D sd_stats.overutilized; sum_util =3D sd_stats.group_util; =20 - cache_sd_stats(env->sd, &sd_stats); + cache_sd_stats(env, &sd_stats); } =20 /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 391c4180eeb3..64f7e013fd59 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2176,8 +2176,26 @@ struct sg_lb_stats { * sched_domain load balancing statistics up the hierarchy. */ struct sg_lb_stats_prop { - raw_spinlock_t stats_lock; /* Lock for updating the cached stats= */ - unsigned long last_update; /* Time when stats was last updated (jiffies)= */ + /* + * Stats version has the following semantics: + * + * When 0, stats are considered state. A writer can lock the + * stats by atomically changing it to LLONG_MIN. Once the + * stats are written, the version is atomically updated to the + * value returned by sched_clock_cpu(). + * + * If the reader finds a positive value for version, the stats + * are considered to be fresh and the reader will copy it for + * load balancing. The version seen before and after the read + * is compared to ensure the stats copied are consistent. + * + * Since invalidations under uncertain circumstances can take a + * long time, a rude writer can always attempt to take over the + * stats by atomically updating the version to LLONG_MIN if it + * finds a large difference betwwen a valid version and the + * value returned by sched_clock_cpu(). + */ + atomic_long_t version; struct sg_lb_stats sg_stats; /* Cached sched_group stats */ }; =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index aeb55f66e8d6..2e72ef8d8d8e 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2304,8 +2304,7 @@ static int __sdt_alloc(const struct cpumask *cpu_map) if (!sg_stats) return -ENOMEM; =20 - raw_spin_lock_init(&sg_stats->stats_lock); - sg_stats->last_update =3D 0; + atomic_long_set(&sg_stats->version, 0); sds->private =3D (void *)sg_stats; =20 sg =3D kzalloc_node(sizeof(struct sched_group) + cpumask_size(), --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2070.outbound.protection.outlook.com [40.107.220.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E12618C01D for ; Sun, 16 Mar 2025 10:30:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.70 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121037; cv=fail; b=N1weR3nDAaAmsOEY2oLMtM9Z4lFenWm4OpJOKfc4rm1+3nMaVRPJ6JwbUELlcVCMCU3ZFK7lyJk3ySYLkSFRtl5dZfP2w+R1SmpZX+dYE3tyM/4A58cIIPxZPMEoy7axPd2nx1VXzoMEcG/3CCoL/KQeaZGi/H3nI1x85I7RwIQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121037; c=relaxed/simple; bh=fHuJgymFQKFvuZonm6Uq/FmpkuZr+4NmZ5i3p2Qw1KQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=J4Gufo8iYEJECv8dxJY5UJ4/drr07F4K2J/X3t46jZfiQ0r51uX2qEbxI43sgw0QFHxbjy0HsAWi/ybkdZewossJKOQKBc5xE9p6YB4kzdo2ZkXUYUQLg9YvxSV+TgfsYTc0fkU5zKXydlQurtyzbMaRFbAC9Ogj8KpJGahOMn8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=c3vzwGve; arc=fail smtp.client-ip=40.107.220.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="c3vzwGve" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NQA5ebPqGemQWJLTn2MzCEWJh9Z3BMSFH94YrfE6ytDZOzVkqPAefRtNDZXSDHUoB4/Osq15KJ2cH6+T2eFtOVVrd5bZnx9SsXbxgji/QigJZdvt9/tJ7bznVs0Uw6x66E6RYk0Os8VssMXlOiz4ZR7ACq8f4LPAgAoxsD75N3u3ibEYI23DPTFYUljhvN2277qTEXuB4OADIDpLEZ4qYK3d5s2Dxkc0Gjp7vvYpwiiVcESnyEqnOmodoa73LpdOYy73opT4zVUb9k+YKix0oohJMcCDitTAveOzEym9bZjoKR0d2x4p8JumWgQr/GC8wQEIVy6f3Rysx4Mm1cIImQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1y1CQnQ6qMNxP0LUHHphXWekDRYraw4OXG44WKe5k5Q=; b=HGlecz+nTJkK6XLOV08BNgj5O7aaSR97HyGly4AtbNe8nXhBHpGnP0hYtYLEjBgmivwpgyAdKn9wBtmQHWJUMXegaW995FdDgRmee1Ir+6x3HDOyiDrhygEfl0r7OBi//5WevIPjekAWfQLkiVIn3/DVP6yJOZGBsvwKmIjIHOcS3S2FfZwvQC0HH60p4dOmaKXuXnHfoq2/z0KzXuL1j5e+34vSjSgfFmubEkMN4aSPcH8kDTsSnooRL1RTq+KDn1bOsGQGeo1fdEKIlozG37GnLKzuLED/GoGYemekG/Vyx+Zf2PdmJjJRfpiVpsV5tUQpKsbRpOfv01St6M+hEg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1y1CQnQ6qMNxP0LUHHphXWekDRYraw4OXG44WKe5k5Q=; b=c3vzwGveP0h0VtbHPZVxstIYWQo2umvIifz3rivlvCHVfp41ebwA+jLOzCLta0ZY3Ogxs2DHpgOg2YTwR8NdX2JqLYBEWzV7VY1mBR5fBR5QMrykyf8f3i/erfvvzlU/n8B9OVY9kW6h4mgFbLZgqqoyOLGLdr+sNM9hCvMpML0= Received: from LV3P220CA0008.NAMP220.PROD.OUTLOOK.COM (2603:10b6:408:234::31) by SA0PR12MB7075.namprd12.prod.outlook.com (2603:10b6:806:2d5::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Sun, 16 Mar 2025 10:30:30 +0000 Received: from BL02EPF00021F6E.namprd02.prod.outlook.com (2603:10b6:408:234:cafe::50) by LV3P220CA0008.outlook.office365.com (2603:10b6:408:234::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Sun, 16 Mar 2025 10:30:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL02EPF00021F6E.mail.protection.outlook.com (10.167.249.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Sun, 16 Mar 2025 10:30:30 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 16 Mar 2025 05:30:25 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 12/08] sched/fair: Record the cpu that updated the stats last Date: Sun, 16 Mar 2025 10:29:13 +0000 Message-ID: <20250316102916.10614-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F6E:EE_|SA0PR12MB7075:EE_ X-MS-Office365-Filtering-Correlation-Id: 4be81098-cc18-4792-292d-08dd6475936e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Aj7u/S6xDSZ/OyBYGGGLo2cOrk4hAndloKzIpRdmKldFQG2awPPto7W3oXKv?= =?us-ascii?Q?VGgjGTSscDKpSXE3XpmxPHxy4Ir52C8mmGCLQEU5vidita4MI2BFy30lbaHV?= =?us-ascii?Q?bCRkDpKeRRwgrlGyQIghyXFJzikX6rSO581q+wNPfYyMwxfoENGmtqFp0Jlm?= =?us-ascii?Q?5lrXwEsp0Z5phw1OGof5MHSM9aOz7igsqoUW871cQVnt/bhojpsZ14Z+JJY/?= =?us-ascii?Q?q9Z9VEKp8Blo2WBjzpNhWJD4h631pk8cFtTLTOf0IUjYY3mocbXkPF8hQT1d?= =?us-ascii?Q?r/t+lS5P3/4Vqb1D0hrMUPvNfRufE2R5gkLzHECD0oug8rk6KKseGCdKnRil?= =?us-ascii?Q?720IxvQtTAKa7Rfq6ho5vHTLU3SgtjAw31b9sNjQV49TAGtX1fvsJx8Y+RH4?= =?us-ascii?Q?DvWBlyoQsxmlB++dKx6EbrxmJp1rgXIJkZXBkXHIqERBPqtxVmgLoNURW/Aq?= =?us-ascii?Q?5AUG4pqRlgQOxk66imqOlpbuMj0uRC2OW7WoMz6l6wHy8HR3MjpK/9LISL5/?= =?us-ascii?Q?agfOJcIKsMwHQMpt3Yp1n0ImPDV9ImtKQmYBLiWAwcyB5rNTTjEndNK0j70y?= =?us-ascii?Q?NyJ1rvIorLmGynpx7udur3RVPSrbIuhOsZlikJZ0NnN7PpRC3Xevy1DX0hkx?= =?us-ascii?Q?7q76SNuRwVfciIPoUm+aiYMH9T8iHmxgkcDWCAsE5xaAd5nLfOrokzpOzOQG?= =?us-ascii?Q?rj5hPrLK6SEaUGXzdANPLbdeAXhYf2e0rREc4nDVUMUFBf06SvoNaT/wKJsy?= =?us-ascii?Q?d1Zg/yzxSEKpvtB6s4EkPb0rIPQwNUVs1fhPyzwznb028pyURXxfez4izwZH?= =?us-ascii?Q?wIUqAnXdQ8kx2bQKt5zxrQCPRlhElwXHdkaOjqJp/UPWGG7rnPkINJYDdOGK?= =?us-ascii?Q?gvJydkCu/jjtNEHG3h7zA9HKOhIHHNQ60SWodwNuoJouNP8ThIAVRCx7iyG+?= =?us-ascii?Q?g608oVKt1FYN7yUhgl+GmuMysnYHpDW7OYfT8FWB9AHXEHzb8qGhB4OTY/JB?= =?us-ascii?Q?19oaPOC1qhWTjv1ZaZgsDqzQpOCokorIHuxtPyThXNF5ytwYAEmH+iC6E1RY?= =?us-ascii?Q?fPdeyRkrFCQgqwx8n6wHMigY/O+PzpEv287U7Py3IkPw7EjoBGkWOdjUrVvI?= =?us-ascii?Q?FF877AvakD9X0lOX4Bf2fpYUrI+X+YZkyRPr4zOsFUYkXGQpRd8pqdnUSqv/?= =?us-ascii?Q?/0cvhxDhqRwNZMW1dwyzd+7wurYTJo53Knx1/tu+XhySbZtSpdiEGk/94l1q?= =?us-ascii?Q?hroEzsvXAiFPKyoh4ckcIt39rABrV7f3NJ/8ZoIpANyJwoZqVRV5ih4SRmGm?= =?us-ascii?Q?UXYk++Hnu/REyhhSr9kgZhR2q1zQ97dhJe0NoLHDzv7Zs2KPlJE4qMLgiwwe?= =?us-ascii?Q?gvL/Q65QGu2r7TUQ0Lj+IHL7t/ihZqFYuOKpDsSyzdJh6am02CW4uR0uvQ2E?= =?us-ascii?Q?XQ6DcDsHAz+9/qkezWTvHlQ+ASRNdIiWdxtq1/3PZsMx3NMk+2c7FM3MPZxV?= =?us-ascii?Q?FmGpZ+MPVkFokZg=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2025 10:30:30.3431 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4be81098-cc18-4792-292d-08dd6475936e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F6E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7075 Content-Type: text/plain; charset="utf-8" Record which CPU updated the stats last. This will be used to invalidate the stats in the following commits. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 5 +++-- kernel/sched/sched.h | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6c486e194a9d..2a34d73d824b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10279,9 +10279,9 @@ static inline void cache_sd_stats(struct lb_env *en= v, struct sg_lb_stats *sd_sta { struct sched_domain_shared *sd_share =3D env->sd->shared; struct sg_lb_stats_prop *lb_prop; - int cpu, retry_limit =3D 3; u64 time, lock; long version; + int cpu; =20 if (!sd_share) return; @@ -10319,7 +10319,7 @@ static inline void cache_sd_stats(struct lb_env *en= v, struct sg_lb_stats *sd_sta * Version is up for grabs! Try again. If the CPU grabs * the lock next time around lock =3D version =3D 0 and this * is skipped. If it cannot grab the version, lock !=3D 0 - * and we return from here thus ensuring on a single + * and we return from here thus ensuring only a single * retry. */ if (!lock) { @@ -10330,6 +10330,7 @@ static inline void cache_sd_stats(struct lb_env *en= v, struct sg_lb_stats *sd_sta } =20 lb_prop->sg_stats =3D *sd_stats; + lb_prop->update_cpu =3D cpu; =20 /* * Pairs against readers checking the version diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 64f7e013fd59..adf4fa2ed031 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2197,6 +2197,7 @@ struct sg_lb_stats_prop { */ atomic_long_t version; struct sg_lb_stats sg_stats; /* Cached sched_group stats */ + int update_cpu; /* CPU that updated the stats */ }; =20 static inline struct cpumask *sched_group_span(struct sched_group *sg) --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2065.outbound.protection.outlook.com [40.107.92.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFFEF18C01D for ; Sun, 16 Mar 2025 10:30:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121060; cv=fail; b=jdh7Wqr7ADC1elucrgTwt94k6M3hJ7f5AXH5x4AZ60sqVDRvce5MM/16FK/nATaLSEorT4f2vDOEzvDRtvwK1m5//UJswJ71T8oyBi4HWDsm5BkqhYtu4QbEeuRWjmB2YnfD+apl/k4N8D3g6kcjyjGlJyikSP5knZt/Eyl4y4g= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121060; c=relaxed/simple; bh=hAnmFazvNe+2DnMDnb/4TdczwV8unYTaSP1FOXIalQQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tkcpJPfTcxgiDS3XLLQf27oaIGwY4vAYPwyFiucdEdDAMSauC3KUU0fPFAHQNn8ORGZQnZlzf31j7gUJOkRo+qDf389XbBNJlMl4xNVkn9MT87TbInHMTKt8iCDyeMYyt5PW+vI0v1OM+sB4N5aDwgARi62B9NXWgXoIXkzX8cE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=rNetjpWV; arc=fail smtp.client-ip=40.107.92.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="rNetjpWV" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=rW8En3uT6AUZ6bZXARhNF8xfsqGu6xWp45To627zNRiNbDE91a2AbRfNuv0ttLimX0EPCft6XLyZtNKtUDBtGphccs1G5l719uEG9C/sUZbQ4QhtpFZL2OyOPs70BmdEtt9aHSAmDfis0eivnt0jdPQX43s1YgaNW9UH40xyr+69klXdbuUbdtCzNnlvZn3m99dT6E8OTn3sGEHkkD03YrGnevmv39bcoCuydOvofKY4jpazFr920c7bDLlJePnBuTM9OKe1ydokhAmV2A6Wyw4putbZiVU/zqEplv8cx2N7KJEq22qLCJFtgIwCRgBmUQwJ/oWukX187H3JQ3p+Dw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=825r1eewD7znbgg+SajXrYUwPqfcJYPYMVsC07dwyRk=; b=EuoE6Hu9a5l1p3G77YhV1t6EhUBQ2YUnzXRseRxZCqCu+bbUQALFCCD4vAcpjdsL5WRJlZBDB51Bz8T0OqMlAxornSFfN14qi/v10nee770AFC8MpU9FXpi8O/IVMeX8O9qnTwB7ZU+CR6kMoZYXhF+KK8tijzAIQknwgD43QUjO0h/GKAKrYEihZp7UJUJmnXquRpzYKSRxBSxjF2ush2NX1XIqEK/sNTNUk1IzVwJatOzS0l2U++tm4vIeCHMjKX5nOiCZkRitq+GdQcq53JV8A9sczUMlLBdgKL90ONzciZnnPDN4w0cwZbtm/zvC1mHSzX14VusrAVSDi7HFfw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=825r1eewD7znbgg+SajXrYUwPqfcJYPYMVsC07dwyRk=; b=rNetjpWVsATXi95kJyD1wOKstmKdUNAJqLzSjPxrgzxdHTerjBJXjfIlK22S1OvZZAwKPgwE4zIKitNU1R8WqIiWwFuVjBKPOmbjOpCMxa7TxeRKRPd2BnMt8Ioxe3pG3u/WQQPTzVSiejwUoFAF7f7Z8foU/XxPRJ8fmV1DDHE= Received: from BLAPR05CA0017.namprd05.prod.outlook.com (2603:10b6:208:36e::29) by SJ0PR12MB6904.namprd12.prod.outlook.com (2603:10b6:a03:483::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.27; Sun, 16 Mar 2025 10:30:49 +0000 Received: from BL02EPF00021F6F.namprd02.prod.outlook.com (2603:10b6:208:36e:cafe::c8) by BLAPR05CA0017.outlook.office365.com (2603:10b6:208:36e::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.27 via Frontend Transport; Sun, 16 Mar 2025 10:30:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL02EPF00021F6F.mail.protection.outlook.com (10.167.249.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Sun, 16 Mar 2025 10:30:48 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 16 Mar 2025 05:30:43 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 13/08] sched/fair: Invalidate stats once the load balancing instance is done Date: Sun, 16 Mar 2025 10:29:14 +0000 Message-ID: <20250316102916.10614-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F6F:EE_|SJ0PR12MB6904:EE_ X-MS-Office365-Filtering-Correlation-Id: 91a4f76a-1f2c-4bac-f8b9-08dd64759e32 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|82310400026|36860700013|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vrM6Oc5b7+Eku6jcICRMqvBC6jyIyB0oW2Ij/6YPAOt4fXPQ/rcJDD7klKzt?= =?us-ascii?Q?e4bPgva+Qa1y7c6rmgcv4IyzdrWYhpXo7xu14rMgoVwH37pIt3fjetp92Prr?= =?us-ascii?Q?1nQUAho/IlGlA6Bj2/6GVy9pSrqJPEPw+9qs6+txMa2v0dFvvm5FP8TEvtxo?= =?us-ascii?Q?8HZU51faw9lGOEZMy5BCPvmJTzZJDdqTI4D+47ygEAbp7qzSn5WSlsE23h+d?= =?us-ascii?Q?gKVRlU26r6nJIMe8Fh4dLlkkvundLLJEGaL+f25gxtIYwkxYn6xTrPLV96nR?= =?us-ascii?Q?Z8NQ2aKWeDNkYW7S5DA7jfz1ZTISvucArlRy6SISRrGuVaaM+8kD6/IPn1qI?= =?us-ascii?Q?pqm0jxcm+un9rUJ+LUc8CJDoWex3YADc2+51mOwZL5asxj0UQGjsa0kASWis?= =?us-ascii?Q?kulz0JmwHxwLJr7ZTU9X4q/YP0oF50etP2NXIJhTteky2o34yspXNVtPFgJT?= =?us-ascii?Q?EYUS298+8PcMEocFxL+TBLs7caFu+QCUbH3+0aGM5CbaznFcZV4+VolCz6fb?= =?us-ascii?Q?SJIY2zY0sX6j40oEopHM+fKof58b+YF3KzhhYHg4ztjcC9o+t3BheecezeND?= =?us-ascii?Q?+2d8i0bo/Q2yLtjnn721gCGbDioy74HCP8xexBwbDjH/7hjkhdz9TiXNSofm?= =?us-ascii?Q?+ermb7oV8xl98oOsNCt3yWbKmkt9582dUEO+tahsX/IrpUixIyRetJ8X8E8j?= =?us-ascii?Q?kZUdLuOrEix5N1agFqN6X8a7rQBIq//rbBXflAqruI8+JA/44w0nI1+9em1/?= =?us-ascii?Q?VdERAAL3Jml//hqXLe88Y8gopOg/Jwz3s15CBq1KHCzWaDUle12mAt0eVTt2?= =?us-ascii?Q?qOQ9anycX0Vx5Ynfl+3pyMFGWmDcoKe6+MMJg8d28iyNW3tBad8IE8tD2DUQ?= =?us-ascii?Q?UkkR3o8yzgA3MIEeumHcjXHj3G2pkdxwBTVmP1xmhmOJo5kFrsBlileReXNx?= =?us-ascii?Q?etwkUyuhUtbRMKPCeFAtyvRpeNFxlyrsQb3614qKC+uQopIbNVfmzEydK1kY?= =?us-ascii?Q?MEy1dX1Qcpvbl1Y1UV0CVBiap15rgLwKLKST8EamoA+vf0NdJJht6f6uK3xu?= =?us-ascii?Q?8u7T+itPMpWVr4exyVS/ocwj/SuzfKIRVvWlRX4XN9D1TmAlnSOry0A4RD+q?= =?us-ascii?Q?JP2+oI7CsnC46wKuskb4/jXrMsy1xntPqe5yCNH2Nh8sSZY+h3OPBZii8Zo6?= =?us-ascii?Q?uq6cwQzGIlWHI8iKF8XGI9k1AEFpLFGnKY/k5uJwTEt3IHOr5rZMjlWZ7e6P?= =?us-ascii?Q?ODeuw9imw3mC6SsG9b0RP5bX3EREb/JiAsfhqpbAyvC4bDW4rH1Rtjm/ZfRF?= =?us-ascii?Q?Db+7onPIiM7kd78Gse/W+qC4pVRHpaiNDKhRQ5NudfTnLGcjmvE0BRqLGpNZ?= =?us-ascii?Q?Fifs9Ct3qxMSawrkM0I6MoaOVMelHgA6nybQ53Ik9QzjkyzzLoIih+ExR030?= =?us-ascii?Q?nZ75w0QWWiGykFfzSQSecYzBlWfcDbetAsOlRx2djJe9Vphrx7dscDmjhDrF?= =?us-ascii?Q?Hz2xteNRIB8GgL0=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(82310400026)(36860700013)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2025 10:30:48.4101 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 91a4f76a-1f2c-4bac-f8b9-08dd64759e32 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F6F.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6904 Content-Type: text/plain; charset="utf-8" The CPU doing the load balancing propagates the stats bottom-up, reusing them as it traverses up the hierarchy. Once done, or if a decision is taken to migrate the tasks that affect the stats, the old version needs to be invalidated for a newer CPU with a recent view to recompute and cache it. Invalidate the old version once load balancing instance is done. Rudely take over the stats if another CPU sees that cached stats are older than 50us. This allows idle and newidle balance to propagate at the very least its local stats up the hierarchy. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : hackbench Units : Normalized time in seconds Interpretation: Lower is better Statistic : AMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Case: tip[pct imp](CV) versioning[pct imp](CV) 1-groups 1.00 [ -0.00](10.12) 1.00 [ 0.44](13.86) 2-groups 1.00 [ -0.00]( 6.92) 1.04 [ -4.32]( 3.00) 4-groups 1.00 [ -0.00]( 3.14) 1.00 [ -0.21]( 2.16) 8-groups 1.00 [ -0.00]( 1.35) 1.01 [ -1.25]( 1.32) 16-groups 1.00 [ -0.00]( 1.32) 1.01 [ -0.50]( 2.00) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : tbench Units : Normalized throughput Interpretation: Higher is better Statistic : AMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Clients: tip[pct imp](CV) versioning[pct imp](CV) 1 1.00 [ 0.00]( 0.43) 0.98 [ -1.65]( 0.15) 2 1.00 [ 0.00]( 0.58) 1.01 [ 1.27]( 0.49) 4 1.00 [ 0.00]( 0.54) 1.00 [ 0.47]( 0.40) 8 1.00 [ 0.00]( 0.49) 1.00 [ -0.44]( 1.18) 16 1.00 [ 0.00]( 1.06) 1.00 [ -0.07]( 1.14) 32 1.00 [ 0.00]( 1.27) 1.00 [ 0.02]( 0.11) 64 1.00 [ 0.00]( 1.54) 0.99 [ -1.12]( 1.09) 128 1.00 [ 0.00]( 0.38) 0.98 [ -2.43]( 1.00) 256 1.00 [ 0.00]( 1.85) 0.99 [ -0.50]( 0.94) 512 1.00 [ 0.00]( 0.31) 0.99 [ -1.03]( 0.35) 1024 1.00 [ 0.00]( 0.19) 0.99 [ -0.56]( 0.42) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : stream-10 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test: tip[pct imp](CV) versioning[pct imp](CV) Copy 1.00 [ 0.00](11.31) 1.08 [ 7.51]( 4.74) Scale 1.00 [ 0.00]( 6.62) 1.00 [ -0.31]( 7.45) Add 1.00 [ 0.00]( 7.06) 1.02 [ 2.50]( 7.34) Triad 1.00 [ 0.00]( 8.91) 1.08 [ 7.78]( 2.88) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : stream-100 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test: tip[pct imp](CV) versioning[pct imp](CV) Copy 1.00 [ 0.00]( 2.01) 1.02 [ 1.82]( 1.26) Scale 1.00 [ 0.00]( 1.49) 1.00 [ 0.26]( 0.80) Add 1.00 [ 0.00]( 2.67) 1.01 [ 0.98]( 1.29) Triad 1.00 [ 0.00]( 2.19) 1.02 [ 2.06]( 1.01) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : netperf Units : Normalized Througput Interpretation: Higher is better Statistic : AMean =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Clients: tip[pct imp](CV) versioning[pct imp](CV) 1-clients 1.00 [ 0.00]( 1.43) 0.99 [ -0.72]( 0.81) 2-clients 1.00 [ 0.00]( 1.02) 1.00 [ -0.09]( 1.11) 4-clients 1.00 [ 0.00]( 0.83) 1.00 [ 0.31]( 0.29) 8-clients 1.00 [ 0.00]( 0.73) 1.00 [ -0.25]( 0.61) 16-clients 1.00 [ 0.00]( 0.97) 1.00 [ -0.26]( 0.89) 32-clients 1.00 [ 0.00]( 0.88) 0.99 [ -0.61]( 0.82) 64-clients 1.00 [ 0.00]( 1.49) 0.99 [ -1.11]( 1.77) 128-clients 1.00 [ 0.00]( 1.05) 1.00 [ -0.03]( 1.13) 256-clients 1.00 [ 0.00]( 3.85) 1.00 [ -0.24]( 2.63) 512-clients 1.00 [ 0.00](59.63) 0.99 [ -0.74](59.01) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : schbench Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) versioning[pct imp](CV) 1 1.00 [ -0.00]( 6.67) 0.93 [ 6.67](15.25) 2 1.00 [ -0.00](10.18) 0.83 [ 17.39]( 7.15) 4 1.00 [ -0.00]( 4.49) 1.04 [ -4.26]( 6.12) 8 1.00 [ -0.00]( 6.68) 1.06 [ -5.66](12.98) 16 1.00 [ -0.00]( 1.87) 1.00 [ -0.00]( 3.38) 32 1.00 [ -0.00]( 4.01) 0.98 [ 2.20]( 4.79) 64 1.00 [ -0.00]( 3.21) 1.02 [ -1.68]( 0.84) 128 1.00 [ -0.00](44.13) 1.16 [-15.98](14.99) 256 1.00 [ -0.00](14.46) 0.90 [ 9.99](17.45) 512 1.00 [ -0.00]( 1.95) 0.98 [ 1.54]( 1.13) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : new-schbench-requests-per-second Units : Normalized Requests per second Interpretation: Higher is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) versioning[pct imp](CV) 1 1.00 [ 0.00]( 0.46) 1.00 [ 0.00]( 0.26) 2 1.00 [ 0.00]( 0.15) 1.00 [ -0.29]( 0.15) 4 1.00 [ 0.00]( 0.15) 1.00 [ -0.29]( 0.30) 8 1.00 [ 0.00]( 0.15) 1.00 [ -0.29]( 0.26) 16 1.00 [ 0.00]( 0.00) 1.00 [ 0.00]( 0.00) 32 1.00 [ 0.00]( 3.40) 1.06 [ 5.93]( 1.22) 64 1.00 [ 0.00]( 7.09) 1.00 [ 0.00]( 0.20) 128 1.00 [ 0.00]( 0.00) 0.98 [ -1.52]( 0.34) 256 1.00 [ 0.00]( 1.12) 0.98 [ -2.41]( 1.19) 512 1.00 [ 0.00]( 0.22) 1.00 [ 0.00]( 0.43) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : new-schbench-wakeup-latency Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) versioning[pct imp](CV) 1 1.00 [ -0.00](19.72) 1.00 [ -0.00]( 8.37) 2 1.00 [ -0.00](15.96) 1.09 [ -9.09](11.08) 4 1.00 [ -0.00]( 3.87) 1.15 [-15.38](17.44) 8 1.00 [ -0.00]( 8.15) 0.92 [ 8.33]( 8.85) 16 1.00 [ -0.00]( 3.87) 1.23 [-23.08]( 5.59) 32 1.00 [ -0.00](12.99) 0.73 [ 26.67](16.75) 64 1.00 [ -0.00]( 6.20) 1.25 [-25.00]( 2.63) 128 1.00 [ -0.00]( 0.96) 1.62 [-62.37]( 1.30) 256 1.00 [ -0.00]( 2.76) 0.82 [ 17.89](10.56) 512 1.00 [ -0.00]( 0.20) 1.00 [ -0.00]( 0.34) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : new-schbench-request-latency Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #workers: tip[pct imp](CV) versioning[pct imp](CV) 1 1.00 [ -0.00]( 1.07) 1.02 [ -2.34]( 0.13) 2 1.00 [ -0.00]( 0.14) 1.04 [ -3.97]( 0.13) 4 1.00 [ -0.00]( 1.39) 1.03 [ -3.15]( 0.13) 8 1.00 [ -0.00]( 0.36) 1.03 [ -3.43]( 0.66) 16 1.00 [ -0.00]( 1.18) 0.99 [ 0.79]( 1.22) 32 1.00 [ -0.00]( 8.42) 0.82 [ 18.29]( 9.02) 64 1.00 [ -0.00]( 4.85) 1.00 [ -0.44]( 1.61) 128 1.00 [ -0.00]( 0.28) 1.06 [ -5.64]( 1.10) 256 1.00 [ -0.00](10.52) 0.81 [ 19.18](12.55) 512 1.00 [ -0.00]( 0.69) 1.00 [ 0.33]( 1.27) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Test : Various longer running benchmarks Units : %diff in throughput reported Interpretation: Higher is better Statistic : Median =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Benchmarks: %diff ycsb-cassandra -0.76% ycsb-mongodb 0.49% deathstarbench-1x -2.37% deathstarbench-2x 0.12% deathstarbench-3x 2.30% deathstarbench-6x 1.88% hammerdb+mysql 16VU 3.85% hammerdb+mysql 64VU 0.27% Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 92 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 71 insertions(+), 21 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2a34d73d824b..31501b933d45 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10341,17 +10341,6 @@ static inline void cache_sd_stats(struct lb_env *e= nv, struct sg_lb_stats *sd_sta =20 static inline int can_retrieve_stats(struct sched_domain *sd, enum cpu_idl= e_type idle) { - /* - * Only under perioric load balancing can we ensure that no concurrent - * CPUs modifies the stats being propagated upwards since - * should_we_balance() can allow multiple concurrent newidle balance - * to progress and an idle -> busy transition for idle balance will - * require the stats to be recomputed since idleness metrics will - * change with migration. - */ - if (idle) - return 0; - /* * If individual groups are separate NUMA domains, migrations can cause * preferred task statistics to change and will require recomputing of @@ -10422,8 +10411,6 @@ static inline void aggregate_sd_stats(struct lb_env= *env, sd_stats->sum_h_nr_running +=3D sg_stats->sum_h_nr_running; sd_stats->sum_nr_running +=3D sg_stats->sum_nr_running; sd_stats->idle_cpus +=3D sg_stats->idle_cpus; - sd_stats->group_capacity +=3D sg_stats->group_capacity; - sd_stats->group_weight +=3D sg_stats->group_weight; sd_stats->overloaded |=3D sg_stats->overloaded; sd_stats->overutilized |=3D sg_stats->overutilized; =20 @@ -10431,6 +10418,52 @@ static inline void aggregate_sd_stats(struct lb_en= v *env, sd_stats->nr_numa_running +=3D sg_stats->nr_numa_running; sd_stats->nr_preferred_running +=3D sg_stats->nr_preferred_running; #endif + + if (env->idle && + sg_stats->group_misfit_task_load > sd_stats->group_misfit_task_load) + sd_stats->group_misfit_task_load =3D sg_stats->group_misfit_task_load; +} + +static inline void __invalidate_stats(struct sched_domain *sd) +{ + struct sched_domain_shared *sd_share =3D sd->shared; + struct sg_lb_stats_prop *lb_prop; + long version; + + if (!sd_share) + return; + + lb_prop =3D (struct sg_lb_stats_prop *)sd_share->private; + if (!lb_prop) + return; + + /* + * The acquire ordering pairs against the writer updating the + * "update_cpu" before setting a valid version. + */ + version =3D atomic_long_read_acquire(&lb_prop->version); + if (version <=3D 0) /* Stats are invalidated / being updated. */ + return; + + guard(irqsave)(); + + /* + * Stats were not updated by this CPU. Leave it to the + * update_cpu to clean it up. + */ + if (lb_prop->update_cpu !=3D smp_processor_id()) + return; + + /* Invalidate the stats. */ + atomic_long_cmpxchg_release(&lb_prop->version, version, 0); +} + +static inline void invalidate_below(struct sched_domain *sd) +{ + struct sched_domain *child; + + for (child =3D sd->child; child; child =3D child->child) + __invalidate_stats(child); } =20 /** @@ -10495,8 +10528,6 @@ static inline void update_sg_lb_stats(struct lb_env= *env, sgs->nr_numa_running +=3D rq->nr_numa_running; sgs->nr_preferred_running +=3D rq->nr_preferred_running; #endif - if (local_group) - continue; =20 if (sd_flags & SD_ASYM_CPUCAPACITY) { /* Check for a misfit task on the cpu */ @@ -10511,10 +10542,13 @@ static inline void update_sg_lb_stats(struct lb_e= nv *env, } } =20 +group_classify: sgs->group_capacity =3D group->sgc->capacity; - sgs->group_weight =3D group->group_weight; =20 + if (local_group || !env->idle) + sgs->group_misfit_task_load =3D 0; + /* Check if dst CPU is idle and preferred to this group */ if (!local_group && env->idle && sgs->sum_h_nr_running && sched_group_asym(env, sgs, group)) @@ -10524,7 +10558,6 @@ static inline void update_sg_lb_stats(struct lb_env= *env, if (!local_group && smt_balance(env, sgs, group)) sgs->group_smt_balance =3D 1; =20 -group_classify: sgs->group_type =3D group_classify(env->sd->imbalance_pct, group, sgs); =20 /* Computing avg_load makes sense only when group is overloaded */ @@ -11167,9 +11200,10 @@ static inline void update_sd_lb_stats(struct lb_en= v *env, struct sd_lb_stats *sd } =20 /* Now, start updating sd_lb_stats */ + sds->total_capacity +=3D sgs->group_capacity; + if (!should_prop) { sds->total_load +=3D sgs->group_load; - sds->total_capacity +=3D sgs->group_capacity; sg_overloaded |=3D sgs->overloaded; sg_overutilized |=3D sgs->overutilized; sum_util +=3D sgs->group_util; @@ -11180,7 +11214,6 @@ static inline void update_sd_lb_stats(struct lb_env= *env, struct sd_lb_stats *sd =20 if (should_prop) { sds->total_load =3D sd_stats.group_load; - sds->total_capacity =3D sd_stats.group_capacity; sg_overloaded =3D sd_stats.overloaded; sg_overutilized =3D sd_stats.overutilized; sum_util =3D sd_stats.group_util; @@ -11947,6 +11980,13 @@ static int sched_balance_rq(int this_cpu, struct r= q *this_rq, =20 if (cur_ld_moved) { attach_tasks(&env); + /* + * If tasks have moved to an idle CPU, the idleness + * metrics have changed. Invalidate stats for the + * next instance to compute them afresh. + */ + if (env.idle) + __invalidate_stats(env.sd); ld_moved +=3D cur_ld_moved; } =20 @@ -12308,12 +12348,12 @@ static void sched_balance_domains(struct rq *rq, = enum cpu_idle_type idle) int continue_balancing =3D 1; int cpu =3D rq->cpu; int busy =3D idle !=3D CPU_IDLE && !sched_idle_cpu(cpu); + int need_serialize, need_decay =3D 0, invalidate =3D 1; unsigned long interval, prev_sd_next_balance =3D 0; struct sched_domain *sd; /* Earliest time when we have to do rebalance again */ unsigned long next_balance =3D jiffies + 60*HZ; int update_next_balance =3D 0; - int need_serialize, need_decay =3D 0; u64 max_cost =3D 0; =20 rcu_read_lock(); @@ -12333,6 +12373,11 @@ static void sched_balance_domains(struct rq *rq, e= num cpu_idle_type idle) * actively. */ if (!continue_balancing) { + if (invalidate) { + invalidate_below(sd); + invalidate =3D 0; + } + if (need_decay) continue; break; @@ -12369,6 +12414,9 @@ static void sched_balance_domains(struct rq *rq, en= um cpu_idle_type idle) next_balance =3D sd->last_balance + interval; update_next_balance =3D 1; } + + if (!sd->parent) + invalidate_below(sd); } if (need_decay) { /* @@ -12987,8 +13035,10 @@ static int sched_balance_newidle(struct rq *this_r= q, struct rq_flags *rf) * Stop searching for tasks to pull if there are * now runnable tasks on this rq. */ - if (pulled_task || !continue_balancing) + if (pulled_task || !continue_balancing || !sd->parent) { + invalidate_below(sd); break; + } } rcu_read_unlock(); =20 --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2061.outbound.protection.outlook.com [40.107.243.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67D11188587 for ; Sun, 16 Mar 2025 10:31:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.61 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121077; cv=fail; b=P9LeWvrZpcskxvwVhtPcX/uskIo6zn6kDWR80MJ7XymKVsj/HJRbQiTPfg8sn+rnmfsA+VnJR3M5LgvmGJFOaZ6jxZqG4ZdX5otj4ljQX8X5wLO2oNzggSqTyZp/uFOji9Zv1iqK1V7ESx8OTBK5RnX98GLTQosQcNs6MJ6k/ck= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121077; c=relaxed/simple; bh=MinsjhYYMbWWT2d4h40ItM2KedtpUi+LpPU6x0XEe+k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TdMlJeKQLKrb7Dck+zLDd/v/CuY24DCIEoDHJudWkiiShc3ykmUuM4FtNPng03uaNqxglDTF2O2MJHRmk+3JEJOl2oCImhYdmcdx16G2uoKhIAkEceNCtPITc486imwA897+2EYvipTenVm2xpqths/hHXp6/Arv9BxF6qXq5cE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=lzRWuQAr; arc=fail smtp.client-ip=40.107.243.61 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="lzRWuQAr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GSXeGPmwy1bLcWXjxR8x+Wwb2PDG1d6JYEO0pzygiah9dZNtwAdSXiOaPmSsKSzYhbSLl+Co+YaGbFOpf5ea3ZJBKkOP4vfTOqs6TmK3RQETty6Stvamj2yRxLxK2eew3T7sGcu51ENQifh08p5Hns1/7OZysW1H0yAAyCHcChmhjoJid1ciaI2Q8+QGDfVeujhIrhZLtJaTJ4sCtW4IZLwJe2lMVMiK+jcUqd3QaIv6sWUPQx6uIme5JKG861K0S41AZLQRvmIfin9j5JwMYxrQdOTfNiefayE611lmK7UHVJ+u1ekEqNKKZvo1ZSWKQPni5WEbez9QmgV4zMUdFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wYgIuba+so9S7IdErF/MdN92/mVyNcfY/VobhY1kcV0=; b=TfrZGxvcAWlTwnHD72FbnR/qHKY1xoMnucOSUDn/yqMa2Sxs1bbzeERhlCAMmdUM7Kp5vVJDmfapPfFagCK/Ua+zXHa+BZzzWYQqaJRGaDeGrT8AO5e4cnwMfLF5HPOAA/puoWaykMDxDNHYE1MSsca0CI9oRylbCo3V3UfmEAenDegNQDtR86NKFgotVg4B+RK0Btb1V3oygxOEVbjw9gBVoBs90LGoteY8Q24ZycF8B3hiff6NFi5c5UjoQrciCi+jRMZbBdAx++XRhelUBBXJKeQcqT/OOJiSL+gRl5iKrcJDZZfsY1nJDlM39BI75phN6VUq4a4EdFvv/DjOGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wYgIuba+so9S7IdErF/MdN92/mVyNcfY/VobhY1kcV0=; b=lzRWuQArNcXf9QaBZBFgAjGaUzVhm9XYSaRdxgnOP96EC8bGNBiLGdieqoWl5rqqrI/YGGn4PyLW/9jSQ4WUUDlUo4Dq6IMpFBipVHZ+IRB6kkIVRkbhEInAoiKB8OkYYFuYjwaG8CGoxRNsEOUdUqCIcSoO6hwoRrG5FOYPFjQ= Received: from BL1PR13CA0186.namprd13.prod.outlook.com (2603:10b6:208:2be::11) by SJ0PR12MB5661.namprd12.prod.outlook.com (2603:10b6:a03:422::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.27; Sun, 16 Mar 2025 10:31:08 +0000 Received: from BL02EPF00021F68.namprd02.prod.outlook.com (2603:10b6:208:2be:cafe::be) by BL1PR13CA0186.outlook.office365.com (2603:10b6:208:2be::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Sun, 16 Mar 2025 10:31:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL02EPF00021F68.mail.protection.outlook.com (10.167.249.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Sun, 16 Mar 2025 10:31:06 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 16 Mar 2025 05:31:01 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 14/08] [DEBUG] sched/fair: Add more lb_stats around lb_time and stats reuse Date: Sun, 16 Mar 2025 10:29:15 +0000 Message-ID: <20250316102916.10614-6-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F68:EE_|SJ0PR12MB5661:EE_ X-MS-Office365-Filtering-Correlation-Id: dc699145-e295-429b-ab5c-08dd6475a8cf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?fIzAeAfdylAs1JGxK61rnBqK5cevZb1mEaVSfb2X3Cnkd8vlSC6blQaxSXgw?= =?us-ascii?Q?9W/ci1JYzHZ/kFOPnF2ByNRpYnEHPtTc+ABhIzbnpVijqIxHYkxinmI7T869?= =?us-ascii?Q?tT2Ft38EyfjO0Vk3Jvz3ypJefNK/QMH4+MpXTh9KlIZ3KBwAifWtg+QVPCaU?= =?us-ascii?Q?VaAe7QVANYM7WLRPJU/llcyqEWgC02E1BCgsMhm9hqz9sf6q9KBpJTA2paEj?= =?us-ascii?Q?tcPGjKXNlrNTftwabeLlMs3xT95tNyR7gljIPKdRRCjLgBYZ2F5jJHZvqu/Y?= =?us-ascii?Q?9AIk6C26o+0wiSeolikrV164Vs1ZQf8lf3Y/ckE5wk8GsfXJqThszpHmlyb8?= =?us-ascii?Q?1TO+8stgfmZR7zP19mB48nqlNS50JQ9oqssXfxZdNi/cq3j/AVFmECJmotYD?= =?us-ascii?Q?J6lU/htm77zHe+JrO6uVovVMN7iGL1YRrtYnxmXN6t0IlZlyPsZXKIWJdYCQ?= =?us-ascii?Q?0lg07iupKQUokxPgkPVkXon4JV+4GYG36/uQWzsW7EWnvH/cbqVFaT1AehrV?= =?us-ascii?Q?qBFfbo+EcsJZ/NzN+sCa8GWUn8Z7ocMN8OJLfcRRgPj2809Q/J7SmtVbvkco?= =?us-ascii?Q?9XZnP0d0iy1w4Or5hVoDyZZv26ifhne+PWuHAxlxCBiceKbxMsonrZ22IfXx?= =?us-ascii?Q?IstnayxjU6mtzBRXO/tvwo04PUXYLI1Wq4vZ+B0TnGJICXTtjB91MiRNyXFg?= =?us-ascii?Q?1DVx1AXOm3Hjv9Oz5yBstQ4bRQtgQZQAAQbkeQJ+EP6HHzBlAhXiOewVQUT+?= =?us-ascii?Q?zsBglUk5XGhrM1+4BitZUfw6ELLdXFTehNryu40SM7EOWD4C9IfGMSOpb931?= =?us-ascii?Q?/z1dV+0iXygRH9lIDpb4wBjAJtB3SCYLKPvpautc4gsvrXbReMZ9UXWwgd+x?= =?us-ascii?Q?0TjyEuY22bc3qJ9l3wGXtq0VIOWxOt+m5gqhceq96pPIM7Zis4qCwFcjMbdu?= =?us-ascii?Q?Nx5RRZp203XvjMuvh8oI19bslCuvuD1Sk/PNC0iMZwzrJw/jisUGxDItdZ/P?= =?us-ascii?Q?eRPvxgwGDUSbE1yvvhQhZjl9Qu1F3voj1wynZPfSWQ7lD8NG1gXFy4yIYUYK?= =?us-ascii?Q?rBB7AZMUn9wurkT8P65Oys0GkLGghl1IDQZ212N97t5f6/szX8Ww1eVPAj5A?= =?us-ascii?Q?gzVFoxCe5LLjbQ5OdkosJ9Vs7g5tPAKxXD6c7yC37x668KL6pL4V5Nfby+SY?= =?us-ascii?Q?YdYJUoPEtOxBlpYOTeoJJMYFz8VVmQ8bAjS5O9kj71pxOvUttFhK5iy1u0o7?= =?us-ascii?Q?Yn2GK6WhojWjBkQX2qcBhgAXqwJy221QY1HNhvvxcSxz7lAU33B/+HSjP8bI?= =?us-ascii?Q?HwO4Z9fqzLyaXAy/Af9CZHkKJE6g0+WOUxMXKMRObF2pJPBPHKHJFsOSg6sr?= =?us-ascii?Q?SqDKVbvPYjgpMOut3a90JbPijMlkuqm9CZIIieoZrSVQN24qRX+ySRMcOT7Y?= =?us-ascii?Q?4myoVhgpjuUA4uAIv9RsXAzRtOQ9KbW8MgRKMAd+rGAZ8lJYV3/hkPjo46vG?= =?us-ascii?Q?byz2WeTS5KwZJQA=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2025 10:31:06.2143 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: dc699145-e295-429b-ab5c-08dd6475a8cf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F68.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5661 Content-Type: text/plain; charset="utf-8" Add stats for load balancing time and stats reuse efficiency. Signed-off-by: K Prateek Nayak --- include/linux/sched/topology.h | 5 +++++ kernel/sched/fair.c | 21 ++++++++++++++++++++- kernel/sched/stats.c | 9 +++++++-- 3 files changed, 32 insertions(+), 3 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index a16d7d9dd9d3..dea65eb263c6 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -123,6 +123,11 @@ struct sched_domain { unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES]; unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES]; unsigned int lb_nobusyq[CPU_MAX_IDLE_TYPES]; + unsigned int lb_min_time[CPU_MAX_IDLE_TYPES]; + unsigned int lb_max_time[CPU_MAX_IDLE_TYPES]; + unsigned long lb_total_time[CPU_MAX_IDLE_TYPES]; + unsigned int lb_stats_reused[CPU_MAX_IDLE_TYPES]; + unsigned int lb_stats_recomputed[CPU_MAX_IDLE_TYPES]; =20 /* Active load balancing */ unsigned int alb_count; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 31501b933d45..bb7b21421415 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10491,10 +10491,13 @@ static inline void update_sg_lb_stats(struct lb_e= nv *env, * group_smt_balance is not possible under busy load balancing. */ if (can_retrieve_stats(env->sd, env->idle) && - retrieve_cached_stats(group, sgs)) + retrieve_cached_stats(group, sgs)) { + schedstat_inc(env->sd->lb_stats_reused[env->idle]); goto group_classify; + } =20 memset(sgs, 0, sizeof(*sgs)); + schedstat_inc(env->sd->lb_stats_recomputed[env->idle]); =20 for_each_cpu_and(i, sched_group_span(group), env->cpus) { struct rq *rq =3D cpu_rq(i); @@ -11901,6 +11904,7 @@ static int sched_balance_rq(int this_cpu, struct rq= *this_rq, { int ld_moved, cur_ld_moved, active_balance =3D 0; struct sched_domain *sd_parent =3D sd->parent; + u64 lb_start =3D sched_clock_cpu(this_cpu); struct sched_group *group; struct rq *busiest; struct rq_flags rf; @@ -12174,6 +12178,21 @@ static int sched_balance_rq(int this_cpu, struct r= q *this_rq, sd->balance_interval < sd->max_interval) sd->balance_interval *=3D 2; out: + if (schedstat_enabled()) { + u64 now =3D sched_clock_cpu(this_cpu); + u64 elapsed =3D now - lb_start; + + if (!schedstat_val(sd->lb_min_time[idle]) || + elapsed < schedstat_val(sd->lb_min_time[idle])) + __schedstat_set(sd->lb_min_time[idle], elapsed); + + if (!schedstat_val(sd->lb_max_time[idle]) || + elapsed > schedstat_val(sd->lb_max_time[idle])) + __schedstat_set(sd->lb_max_time[idle], elapsed); + + __schedstat_add(sd->lb_total_time[idle], elapsed); + } + return ld_moved; } =20 diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c index 4346fd81c31f..b2ace3c51062 100644 --- a/kernel/sched/stats.c +++ b/kernel/sched/stats.c @@ -141,7 +141,7 @@ static int show_schedstat(struct seq_file *seq, void *v) seq_printf(seq, "domain%d %s %*pb", dcount++, sd->name, cpumask_pr_args(sched_domain_span(sd))); for (itype =3D 0; itype < CPU_MAX_IDLE_TYPES; itype++) { - seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u", + seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u %u %lu %u %u", sd->lb_count[itype], sd->lb_balanced[itype], sd->lb_failed[itype], @@ -152,7 +152,12 @@ static int show_schedstat(struct seq_file *seq, void *= v) sd->lb_gained[itype], sd->lb_hot_gained[itype], sd->lb_nobusyq[itype], - sd->lb_nobusyg[itype]); + sd->lb_nobusyg[itype], + sd->lb_min_time[itype], + sd->lb_max_time[itype], + sd->lb_total_time[itype], + sd->lb_stats_reused[itype], + sd->lb_stats_recomputed[itype]); } seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u\n", --=20 2.43.0 From nobody Thu Dec 18 07:19:55 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2063.outbound.protection.outlook.com [40.107.223.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4E3E44C63 for ; Sun, 16 Mar 2025 10:31:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121090; cv=fail; b=EQuo6ZzHUREP9HLXHlT2ijbLJeUwIhgYRgPt4tHRw0g8th62A8oMDePu9ukcp9FnCOiQKaaz0RDEKOpZDY4rTjDgn9yXVDSjZHPMJMYkE0qDEgFKFT9QSeuYoiPgHRiZ3Z0l+wD4zVhZUXQnjMHfoVnXVrQU1k/m+7Wt96ZjN88= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742121090; c=relaxed/simple; bh=c8tFBihpZPYwDlU0uDSiHUTpRcnPAu3dlitSQ4pLCLs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=lcTZrW2XZEo5JQjabTfGZcBJcw9sW0kAGrivx4R7BP12zbkepacYbHOLiP9T6xjAdNDdqBKd6Gq+FIdlrzFluwF2buQY2ZqXY2GakKjg6s7EgE7gjOcWQ3YqvZASGgjBYJhdksH/JaKuUkU+/tKZ6+/74QC7u2xVlUhkJzxhWEM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=UnljLX8s; arc=fail smtp.client-ip=40.107.223.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="UnljLX8s" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=A+IwkiaCNg0F31VG1mqdScjAxaMu8VKHsnzmTTMqn1BC4bUMIJr4kGURAEWXWFOUtx0tOC69gyhtu6M7VB9+A2w+mXLLNMiSnFPo/6ux7ZFv4hkR5khi/2KYXJ2JTKewGx+PQIQk9CbQci3gmuOP1eS5vnmIeIo8cnTXlcDF4csJ9JrmiAyF3ORSFYCr1jUAsMaoFZTCMH++EwX1CruFjW43kQXXvGov9eSTtACzNBcNayRtKRHKvSFneUqbTHSBdG4WtXZcqGR7766aosduHuUbRysFL19Mn9oVKrIU/6oT0IHbjsonDH7vY8xNulSwLw5x5xO/HS4re13EMMkhSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BILs3xOKiqaYjA6DD429UlwOfdiL47kGZLs5V0XjnZ4=; b=R0NvunVxhx+FckDzLlJ9bJiPLUz2nKAdk4E3wVCjEEzgh2haVIzjsPI6kEPd3wWi+Mg1HnflTWIwQKC7PYAcjNxUkzA/kVKWMfjFq7zU8cCtQneMQCgpq5Eg3r3XqvDSAh58eDVXVYzm7JBIrRj+AC9lZJb6IxScKU/BER4REVME7DnIhAQ1z+GBGctjzf9vFpZx7Q85nbzk7OHr7/17UtlQcTzW8jCUXvZ92gZcshCp7+tvNm5stxO3q3k1I14fL4uAMXIpSve0WRI2VyXvhkaXzHlmGw7inv+LTyjVxMIbfhChnY7lRKnTN0ZpGLERUDKNl2TGuO6hdBLI4jmZUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BILs3xOKiqaYjA6DD429UlwOfdiL47kGZLs5V0XjnZ4=; b=UnljLX8sHhHdX/Xvcal7YW4GfKmPi82RfwLEPM66F7AyW8YEdlU2ITBovz/+NYPpbM5omnda+QZbcn28bnc716HOoemexrUFXZon7i4SE9QOssGqjgViBYmsiztS0ZwQtwYYXdFxUNK8iQMJmwNUjJByF6JSCTFN3lGjSK93CtM= Received: from BLAPR03CA0100.namprd03.prod.outlook.com (2603:10b6:208:32a::15) by DS7PR12MB5767.namprd12.prod.outlook.com (2603:10b6:8:76::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.31; Sun, 16 Mar 2025 10:31:25 +0000 Received: from BL02EPF00021F69.namprd02.prod.outlook.com (2603:10b6:208:32a:cafe::e) by BLAPR03CA0100.outlook.office365.com (2603:10b6:208:32a::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.29 via Frontend Transport; Sun, 16 Mar 2025 10:31:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL02EPF00021F69.mail.protection.outlook.com (10.167.249.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Sun, 16 Mar 2025 10:31:24 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 16 Mar 2025 05:31:18 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Chen Yu , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , David Vernet , "Gautham R. Shenoy" , "Swapnil Sapkal" , Shrikanth Hegde , "K Prateek Nayak" Subject: [RFC PATCH 15/08] [DEBUG] tools/lib/perf: Extend schedstats v17 headers to include the new debug fields Date: Sun, 16 Mar 2025 10:29:16 +0000 Message-ID: <20250316102916.10614-7-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250313093746.6760-1-kprateek.nayak@amd.com> References: <20250313093746.6760-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F69:EE_|DS7PR12MB5767:EE_ X-MS-Office365-Filtering-Correlation-Id: e0600b0c-1018-48d6-296e-08dd6475b3a5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|36860700013|1800799024|82310400026|13003099007; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?MROUl7FsYt1wLPxKGt/Zj4BYjU9e2fSbl4njsO82RHPsai+49xtE82FAcdYa?= =?us-ascii?Q?zsyeDV0rycaNz+4O6yGKWpJ1FC0GNnkXWhVqvFGIuY7erboQ7gd/0MpQ9hpq?= =?us-ascii?Q?cS2AgYRja0IIdc9t1Rz8oma0m7MTb1BeCqfes6aUJst3Nh1WD1oz4bqvEv5i?= =?us-ascii?Q?z5+j+JFhOPOr2IVXInCg38rVkCxdA0jIUmCVbRzKxk+lesbagUQtM/ApG0Tn?= =?us-ascii?Q?nGNqHqZTP0idl0u/eoBmhV+0LvF5tZ2Bnpsj80YKjUu5JvY+Q3SN2sFZUb82?= =?us-ascii?Q?QrVboBbX06Y5KfZ83j7bfgDW3opSNJEKzh/oxxX+1IHqyxxV7iJbBRQyuRAY?= =?us-ascii?Q?J0iaK8tag2D1jAnjO1hClrb++9GEEz2JNNQQmP/Ab3w3G8GTYLus+J2ah5yU?= =?us-ascii?Q?w7RtL0sPymjA9PwNcP4WxAoj3Xny5vxYlIab6iN3I1IkRisCB6rRnfkzjNiP?= =?us-ascii?Q?cR9rulIEft9FL65bidd6YjYxV4Prxu7zNbwIZe2ZvCKzFTvJMsQ3kDyplJot?= =?us-ascii?Q?uFUElGmt5G7Jc1hT1EWfuNyjc5Z3/GLDrEEzdcI/Ts+mR4MJ2NwO0fbvyVBB?= =?us-ascii?Q?7k7sj2/HXCKuagT5uRfNLh8kqRFB3ljv6XsCBg5tHwhIFucZ+UR2O7QG1B3v?= =?us-ascii?Q?HLt8r7jUICX+4DG2YGyijqTPdP0Wogni115COgrjQfQQtIQF6qiMRvU+cv40?= =?us-ascii?Q?g67CjMbdu0gCEv6cLbLNn0Cz1WVOtb5nzdjKiuSCfEhRBd5+CimDyL6ApXVk?= =?us-ascii?Q?ApeqpsbJ06qAGM3Z1glQ3fre2cgUa2C3SFMzaDXX8trt/EUXs9LFsPU8f6y6?= =?us-ascii?Q?nnjLeoqgGdBYlHNHUgqZRw+yQ38OnJuS2rpsMGnGCdilTNfwxaUxIlHzzsEc?= =?us-ascii?Q?pI5WP07Mn9pKSFti5prRs6EolS54wJkwT8tZAVPYdVb6+ZJxSt6GhU5gNNV8?= =?us-ascii?Q?l8O2T7I7Xyhwu3L3u/uMF5utnTH1N4krM0saudxal8rZtgKGfcEtrgxQD0UI?= =?us-ascii?Q?0O4o5yl+XnQ0IRcbxvExxcTesjqUJsVl8HdUbZ4/PpI7Gy4a2Exh64KnG4DP?= =?us-ascii?Q?9Szc5GaqyQFT0BAlvaUymf1TDvfkZVQFE+1r7bNeYPy+xDodQ1xJRKbU0lfR?= =?us-ascii?Q?cML7v3hEHfSUtACfGvsjBFvT8gAMWXAf/6F7sTI1hbzut5OTIfjY8Gpmkz4d?= =?us-ascii?Q?crMP6G4uw9uSS943wm0fKlx4qttji67tioiqHnTh3xx9i/CkUkhL0O9Xl3Ez?= =?us-ascii?Q?YRkohnw01+0ifx9Xs5or4TbGM1xzMhql9zUC2VnB1mR/hXJWclLvrak6mqMf?= =?us-ascii?Q?X6V2lKEI0eeulUvHdMT4TOkyPvpxg+nrQ1V5bmCSNcFahFNiCna1B9r656zL?= =?us-ascii?Q?dbrgMmqnggYRYXmKVvdZY4zodMuYZqgirkUPAmhUrmgOpDfhRQTh4Afw1XjO?= =?us-ascii?Q?J5Ii6Io0FA2dkQgBSEQ/DmB1yMEPDsJtI6mfAugeWIKfRWUD33MmgcOjchRJ?= =?us-ascii?Q?/6lKimcWPa0TqSk=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(36860700013)(1800799024)(82310400026)(13003099007);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2025 10:31:24.4062 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e0600b0c-1018-48d6-296e-08dd6475b3a5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F69.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5767 Content-Type: text/plain; charset="utf-8" The previous commit hacked up schedstats v17 to add more fields. Extend the header file for perf sched stats for analysis. These changes depend on perf sched stats tools being developed in [1]. Link: https://lore.kernel.org/lkml/20250311120230.61774-1-swapnil.sapkal@am= d.com/ [1] Signed-off-by: K Prateek Nayak --- tools/lib/perf/include/perf/schedstat-v17.h | 30 +++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/tools/lib/perf/include/perf/schedstat-v17.h b/tools/lib/perf/i= nclude/perf/schedstat-v17.h index 00009bd5f006..888dfa982a55 100644 --- a/tools/lib/perf/include/perf/schedstat-v17.h +++ b/tools/lib/perf/include/perf/schedstat-v17.h @@ -47,6 +47,16 @@ DOMAIN_FIELD(__u32, busy_lb_nobusyq, "load_balance() failed to find busier queue on cpu busy", "%11u", tr= ue, v17); DOMAIN_FIELD(__u32, busy_lb_nobusyg, "load_balance() failed to find busier group on cpu busy", "%11u", tr= ue, v17); +DOMAIN_FIELD(__u32, busy_lb_min_time, + "load_balance() min time to balance on busy", "%11u", true, v17); +DOMAIN_FIELD(__u32, busy_lb_max_time, + "load_balance() max time to balance on busy", "%11u", true, v17); +DOMAIN_FIELD(unsigned long, busy_lb_total_time, + "load_balance() total time to balance on busy", "%11lu", true, v17); +DOMAIN_FIELD(__u32, busy_lb_stats_reused, + "load_balance() stats reused on busy", "%11u", true, v17); +DOMAIN_FIELD(__u32, busy_lb_stats_recompute, + "load_balance() stats recomputed on busy", "%11u", true, v17); #ifdef DERIVED_CNT_FIELD DERIVED_CNT_FIELD("load_balance() success count on cpu busy", "%11u", busy_lb_count, busy_lb_balanced, busy_lb_failed, v17); @@ -80,6 +90,16 @@ DOMAIN_FIELD(__u32, idle_lb_nobusyq, "load_balance() failed to find busier queue on cpu idle", "%11u", tr= ue, v17); DOMAIN_FIELD(__u32, idle_lb_nobusyg, "load_balance() failed to find busier group on cpu idle", "%11u", tr= ue, v17); +DOMAIN_FIELD(__u32, idle_lb_min_time, + "load_balance() min time to balance on idle", "%11u", true, v17); +DOMAIN_FIELD(__u32, idle_lb_max_time, + "load_balance() max time to balance on idle", "%11u", true, v17); +DOMAIN_FIELD(unsigned long, idle_lb_total_time, + "load_balance() total time to balance on idle", "%11lu", true, v17); +DOMAIN_FIELD(__u32, idle_lb_stats_reused, + "load_balance() stats reused on idle", "%11u", true, v17); +DOMAIN_FIELD(__u32, idle_lb_stats_recompute, + "load_balance() stats recomputed on idle", "%11u", true, v17); #ifdef DERIVED_CNT_FIELD DERIVED_CNT_FIELD("load_balance() success count on cpu idle", "%11u", idle_lb_count, idle_lb_balanced, idle_lb_failed, v17); @@ -113,6 +133,16 @@ DOMAIN_FIELD(__u32, newidle_lb_nobusyq, "load_balance() failed to find busier queue on cpu newly idle", "%11= u", true, v17); DOMAIN_FIELD(__u32, newidle_lb_nobusyg, "load_balance() failed to find busier group on cpu newly idle", "%11= u", true, v17); +DOMAIN_FIELD(__u32, newidle_lb_min_time, + "load_balance() min time to balance on newly idle", "%11u", true, v1= 7); +DOMAIN_FIELD(__u32, newidle_lb_max_time, + "load_balance() max time to balance on newly idle", "%11u", true, v1= 7); +DOMAIN_FIELD(unsigned long, newidle_lb_total_time, + "load_balance() total time to balance on newly idle", "%11lu", true,= v17); +DOMAIN_FIELD(__u32, newidle_lb_stats_reused, + "load_balance() stats reused on newly idle", "%11u", true, v17); +DOMAIN_FIELD(__u32, newidle_lb_stats_recompute, + "load_balance() stats recomputed on newly idle", "%11u", true, v17); #ifdef DERIVED_CNT_FIELD DERIVED_CNT_FIELD("load_balance() success count on cpu newly idle", "%11u", newidle_lb_count, newidle_lb_balanced, newidle_lb_failed, v17); --=20 2.43.0