From nobody Sat Feb 7 10:08:18 2026 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 719AF1422DE for ; Mon, 25 Mar 2024 05:45:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711345532; cv=none; b=U6/MN0rHTEYNo4AzYmF3VG0qjebCi/8SPurKI/ozjHyZ5H3ymn11d4m/pl2G1s71fcLseqsxmGNr6LljE0/P0XSKbXoiAP/MR0ibwvvoHjUs1rpLg0i7koDrmEQ/sqGcs2R3ZS7untw67zrEqFFauIkhl1Mq1e6DOblPuy9puOM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711345532; c=relaxed/simple; bh=3003hjwxlKjpdM/HTmVW3IjXQTa9b3lBarRdaHkd+Ak=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VBBwwenRfoFxPOvyqhGOc6AEce+2t/eRDw7aeiZ2+jnMYdqoQZgNxKmsqpWf3+wKJtarVJVgEJAUAMFPLOek2IKpPp0ZC9ohoLQxjsSGMX0wYtZB3POnw2gnzVZ5QHTGOE4NT6omtDqPomuIr58phdWEWlumN7mc8fcJ2pp0NcA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=Def8TlyZ; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Def8TlyZ" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 42ONoMxG030563; Mon, 25 Mar 2024 05:45:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=YlGXSj5DSsi3seogjbHk3vlyDbRmRCvlpJOwGGPesu4=; b=Def8TlyZH4eVrCQJUWHbfBj6SY+kQVKgR74D5iDwMODCFTbL6ikpYXruGFZr4Y5XcqKD cc9yh/RSZrh1ZnQvR0E1PbyGPFAntD+m4AmgmHV1fvVo9l6R/J37zVYz7tN0Z2yrBlz2 ePGHlZo6pUID5rHz3fszE1uipySmX5nHSuzbfzy2BZx3ihU0UwZlVvfW6Axgn1piKrdQ TIgfZwuklNB45VF4U7BSFah1P0DmQfi1rpuahNrgqd8faEJ1hVTPuZTHCDCCiz4ORPf4 9j46fO+8XeFHNIUdS040SSabiYAuQbdGGQ9xY15ZxhLklBJI2VwzJTK3mpdUXGLa90R9 uw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3x2h6t1hbg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 25 Mar 2024 05:45:19 +0000 Received: from m0353725.ppops.net (m0353725.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 42P5jJNN024195; Mon, 25 Mar 2024 05:45:19 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3x2h6t1hbe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 25 Mar 2024 05:45:19 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 42P3giWA011233; Mon, 25 Mar 2024 05:45:18 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3x2bmkpp1d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 25 Mar 2024 05:45:18 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 42P5jECW27984134 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Mar 2024 05:45:16 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AC5702004B; Mon, 25 Mar 2024 05:45:14 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0E52920043; Mon, 25 Mar 2024 05:45:13 +0000 (GMT) Received: from li-c1fdab4c-355a-11b2-a85c-ef242fe9efb4.in.ibm.com (unknown [9.109.201.126]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Mon, 25 Mar 2024 05:45:12 +0000 (GMT) From: Shrikanth Hegde To: mingo@kernel.org, peterz@infradead.org, vincent.guittot@linaro.org Cc: sshegde@linux.ibm.com, dietmar.eggemann@arm.com, qyousef@layalina.io, linux-kernel@vger.kernel.org, vschneid@redhat.com Subject: [PATCH v3 1/2] sched/fair: Check rd->overload value before update Date: Mon, 25 Mar 2024 11:15:04 +0530 Message-Id: <20240325054505.201995-2-sshegde@linux.ibm.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240325054505.201995-1-sshegde@linux.ibm.com> References: <20240325054505.201995-1-sshegde@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: zL1M1edLB8lmiCnn293hjeN7-TR5UjCG X-Proofpoint-ORIG-GUID: B63sQfFj_90BstVCm4XSra9KUhUIbVXY X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-03-25_02,2024-03-21_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 priorityscore=1501 malwarescore=0 adultscore=0 suspectscore=0 clxscore=1015 phishscore=0 bulkscore=0 lowpriorityscore=0 spamscore=0 mlxscore=0 mlxlogscore=646 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2403210000 definitions=main-2403250029 Content-Type: text/plain; charset="utf-8" Overload is an indicator used to inform if there is any rq in the root domain has 2 or more running tasks. Root domain is a global structure per cpuset island. Overload status is updated when adding a task to the runqueue. It is cleared in update_sd_lb_stats during load balance, if none of the rq has 2 or more running task. It is used during to newidle balance to see if its worth doing load balance. If it is set, then newidle balance will try aggressively to pull a task. Since commit 630246a06ae2 ("sched/fair: Clean-up update_sg_lb_stats parameters"), overload is being updated unconditionally. The change in value of this depends on the workload. On typical workloads, it is likely that this value doesn't change often. Write causes the cacheline to be invalidated. This would cause true sharing when the same variable is accessed in sched_balance_newidle. On large systems this would cause more CPU bus traffic. Perf probe stats for reference. Detailed info on perf probe can be found in the cover letter. With Patch while running hackbench. 1M probe:sched_balance_newidle_L38 139 probe:update_sd_lb_stats_L53 129K probe:add_nr_running_L12 74 probe:add_nr_running_L13 54K probe:update_sd_lb_stats_L50 Perf probes prove that actual change in value is less often. L50 vs L53. So it would be better to check the value before updating it. CPU bus traffic reduces with the patch. Signed-off-by: Shrikanth Hegde Reviewed-by: Qais Yousef Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 02d4d224b436..eeebadd7d9ae 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10621,7 +10621,8 @@ static inline void update_sd_lb_stats(struct lb_env= *env, struct sd_lb_stats *sd if (!env->sd->parent) { /* update overload indicator if we are at root domain */ - WRITE_ONCE(env->dst_rq->rd->overload, sg_status & SG_OVERLOAD); + if (READ_ONCE(env->dst_rq->rd->overload) !=3D (sg_status & SG_OVERLOAD)) + WRITE_ONCE(env->dst_rq->rd->overload, sg_status & SG_OVERLOAD); /* Update over-utilization (tipping point, U >=3D 0) indicator */ set_rd_overutilized_status(env->dst_rq->rd, -- 2.39.3 From nobody Sat Feb 7 10:08:18 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 925721422DF for ; Mon, 25 Mar 2024 05:45:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711345538; cv=none; b=E1X3UZsx8U0GgGWNM/rsLrBBiKr/uc1mmH0i79ryOO4NwZO1flxh72OvDI6r70qBBV3YBjAOHrkGnbAqkQ+fDxn/DoGxZoml7vHd0iu4lXtBD21i00tb21bxNs9fXsBCMzVvNXSusqUYTxYD6rFBlH6l55PHAJHD8Or2YpdX+Bc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711345538; c=relaxed/simple; bh=Nf9/+absmF2p9Y3Nt6wAdkzbW99RzEk9jOUSMk1Hpj8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Mj17qUn19gwVLejzZhV3yd+PV9HzBDdUV9sQ1yZI95DMxeGA9VLGVcMrVor0mV+wIY0iM+p0fpDcgyDXO3MRZGjBZJfi5ads6Gn1/NMIeNNA1iQgGSTTMjLrV1HIjx7s6nsirvbyIRDzOdQa01hyibNSBf/eVij63MuVbdlbalI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=G8/2feiV; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="G8/2feiV" Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 42P3AFfv011154; Mon, 25 Mar 2024 05:45:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=/Cdxic8Oiq1h4/eSkqGMxCD/okNBm3CJhlovnrT+fWA=; b=G8/2feiVkyZOOkAJoKGJjHIM8FXLaZEknIdj5s0hdKM3W+uxW2KIxFhAqXVCVzrQyI+V 1luRFrZtvUhsjbgQTnGTFXbRugkA1IjAV6qHeJSDIQoP9gXZ0wxQUV/1G2xZm0fxpooG 1ZJqQ3EocbEh3jhjqeT0wx3hNQq/9dvvNfLW2h3mNfCF+Y//sybT5gICVlRcBvF6yOSh PLQ71Yew0p8ta8pzYZ8c+sYskKcPPKGzbL8csT2VP2OxAxRsmYu49HVptZnKC29jGmvH n9cps3wSSBmTPDVwee4aXC8KlRcxQnUvCs5TJ4CpAoQm6LoGJACDqgaTYo9a/RWIpb19 pQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3x28fhtejp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 25 Mar 2024 05:45:24 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 42P5jOxm000615; Mon, 25 Mar 2024 05:45:24 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3x28fhtejm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 25 Mar 2024 05:45:24 +0000 Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 42P5dG27025481; Mon, 25 Mar 2024 05:45:22 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3x2awmexch-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 25 Mar 2024 05:45:22 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 42P5jINf36176230 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 25 Mar 2024 05:45:20 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B0C912004D; Mon, 25 Mar 2024 05:45:18 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1262A2004B; Mon, 25 Mar 2024 05:45:17 +0000 (GMT) Received: from li-c1fdab4c-355a-11b2-a85c-ef242fe9efb4.in.ibm.com (unknown [9.109.201.126]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Mon, 25 Mar 2024 05:45:16 +0000 (GMT) From: Shrikanth Hegde To: mingo@kernel.org, peterz@infradead.org, vincent.guittot@linaro.org Cc: sshegde@linux.ibm.com, dietmar.eggemann@arm.com, qyousef@layalina.io, linux-kernel@vger.kernel.org, vschneid@redhat.com Subject: [PATCH v3 2/2] sched/fair: Use helper functions to access rd->overload Date: Mon, 25 Mar 2024 11:15:05 +0530 Message-Id: <20240325054505.201995-3-sshegde@linux.ibm.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240325054505.201995-1-sshegde@linux.ibm.com> References: <20240325054505.201995-1-sshegde@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: Lc3uyEvxYz45w_H31MU8eWGqDiRwmpWQ X-Proofpoint-GUID: _vsCfY0cPeSjIzc2oxU0yB5_4HlhpATB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-03-25_03,2024-03-21_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 malwarescore=0 suspectscore=0 phishscore=0 spamscore=0 clxscore=1015 mlxlogscore=771 bulkscore=0 impostorscore=0 adultscore=0 priorityscore=1501 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2403210000 definitions=main-2403250030 Content-Type: text/plain; charset="utf-8" rd->overload is accessed at multiple places. Instead it could use helper get/set functions. This would make the code more readable and easy to maintain. No change in functionality intended. Suggested-by: Qais Yousef Signed-off-by: Shrikanth Hegde Reviewed-by: Qais Yousef Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 5 ++--- kernel/sched/sched.h | 14 ++++++++++++-- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index eeebadd7d9ae..f00cb66cc479 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10621,8 +10621,7 @@ static inline void update_sd_lb_stats(struct lb_env= *env, struct sd_lb_stats *sd if (!env->sd->parent) { /* update overload indicator if we are at root domain */ - if (READ_ONCE(env->dst_rq->rd->overload) !=3D (sg_status & SG_OVERLOAD)) - WRITE_ONCE(env->dst_rq->rd->overload, sg_status & SG_OVERLOAD); + set_rd_overload_status(env->dst_rq->rd, sg_status & SG_OVERLOAD); /* Update over-utilization (tipping point, U >=3D 0) indicator */ set_rd_overutilized_status(env->dst_rq->rd, @@ -12344,7 +12343,7 @@ static int sched_balance_newidle(struct rq *this_rq= , struct rq_flags *rf) rcu_read_lock(); sd =3D rcu_dereference_check_sched_domain(this_rq->sd); - if (!READ_ONCE(this_rq->rd->overload) || + if (!is_rd_overloaded(this_rq->rd) || (sd && this_rq->avg_idle < sd->max_newidle_lb_cost)) { if (sd) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 41024c1c49b4..bed5ad8a1827 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -918,6 +918,17 @@ extern void rq_attach_root(struct rq *rq, struct root_= domain *rd); extern void sched_get_rd(struct root_domain *rd); extern void sched_put_rd(struct root_domain *rd); +static inline int is_rd_overloaded(struct root_domain *rd) +{ + return READ_ONCE(rd->overload); +} + +static inline void set_rd_overload_status(struct root_domain *rd, int stat= us) +{ + if (is_rd_overloaded(rd) !=3D status) + WRITE_ONCE(rd->overload, status); +} + #ifdef HAVE_RT_PUSH_IPI extern void rto_push_irq_work_func(struct irq_work *work); #endif @@ -2518,8 +2529,7 @@ static inline void add_nr_running(struct rq *rq, unsi= gned count) #ifdef CONFIG_SMP if (prev_nr < 2 && rq->nr_running >=3D 2) { - if (!READ_ONCE(rq->rd->overload)) - WRITE_ONCE(rq->rd->overload, 1); + set_rd_overload_status(rq->rd, SG_OVERLOAD); } #endif -- 2.39.3