From nobody Sun Feb 8 05:35:00 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DA3DBA20 for ; Fri, 22 Nov 2024 15:29:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732289346; cv=none; b=CCD+2nACrOzZrLy/YhcL8GhiQy3iOQjwIrPSB2T1CVvLvkpl8eozRA4q0gMiJrkDOfpsPShwOC0u9d+PqExWKmZPG7fy6rGt7PB2K0bVRqGKg2kdZ8hkA5X0wmwiyBwN30d1cZxgIQgxSx6OunxcT9lQ0IG2pp5NfL9y58sS154= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732289346; c=relaxed/simple; bh=CvWQomH5tQ6WDIkRGoA7mGBzdXNqYounsIFzF/JVQoY=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=I3jKTihwQrdvO5MO1Xpft2ILoubdzi3h5O4c9eEJkQAxg13co2N83HadlMi7uInPvO/elMwexaK6U3/REDvZqxYS9tu1KhQsfGuvaBv7G2mxcZT/L7r7jhos6nGXSYs1Cqxnelabu2KnYEUrd6m2IdgcroliSf28PfUVoG1lw8w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=fv1XDMqg; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=fv1XDMqg; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="fv1XDMqg"; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="fv1XDMqg" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 3E1391F37E; Fri, 22 Nov 2024 15:29:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1732289342; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=dctTZVVrxmtSkDVloqVLA9R/1bskoOXpW2WzccMFbw0=; b=fv1XDMqglSodbAkVca1VFf/PFrCypquUYX+Kp0bW2T2QUx89yiP91WxG9Q5cGoZxiGQSYO pwNZz8cpuMoLIZIfxkJ5rBXJpb2Yqcd3jsiWVroT//JH1+vdRe1JFwa0USWx/HEhWhmmAT wnIuOxX2KysyE9vyoICRcNeIULT5Ta4= Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.com header.s=susede1 header.b=fv1XDMqg DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1732289342; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=dctTZVVrxmtSkDVloqVLA9R/1bskoOXpW2WzccMFbw0=; b=fv1XDMqglSodbAkVca1VFf/PFrCypquUYX+Kp0bW2T2QUx89yiP91WxG9Q5cGoZxiGQSYO pwNZz8cpuMoLIZIfxkJ5rBXJpb2Yqcd3jsiWVroT//JH1+vdRe1JFwa0USWx/HEhWhmmAT wnIuOxX2KysyE9vyoICRcNeIULT5Ta4= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 1709813998; Fri, 22 Nov 2024 15:29:02 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id HrbgBD6jQGfqewAAD6G6ig (envelope-from ); Fri, 22 Nov 2024 15:29:02 +0000 From: Daniel Vacek To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider Cc: Daniel Vacek , linux-kernel@vger.kernel.org Subject: [PATCH] sched/fair: properly serialize the cfs_rq h_load calculation Date: Fri, 22 Nov 2024 16:28:55 +0100 Message-ID: <20241122152856.3533625-1-neelx@suse.com> X-Mailer: git-send-email 2.45.2 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 3E1391F37E X-Spam-Level: X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.com:s=susede1]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.com:+]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_SEVEN(0.00)[11]; ASN(0.00)[asn:25478, ipnet:::/0, country:RU]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email,suse.com:dkim,suse.com:mid,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns] X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Rspamd-Action: no action X-Spam-Score: -3.01 X-Spam-Flag: NO Content-Type: text/plain; charset="utf-8" Make sure the given cfs_rq's h_load is always correctly updated. This prevents a race between more CPUs eventually updating the same hierarchy of h_load_next return pointers. Signed-off-by: Daniel Vacek --- kernel/sched/fair.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2d16c8545c71..50794ba0db75 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9786,6 +9786,8 @@ static bool __update_blocked_fair(struct rq *rq, bool= *done) return decayed; } =20 +static DEFINE_PER_CPU(raw_spinlock_t, h_load_lock); + /* * Compute the hierarchical load factor for cfs_rq and all its ascendants. * This needs to be done in a top-down fashion because the load of a child @@ -9793,18 +9795,26 @@ static bool __update_blocked_fair(struct rq *rq, bo= ol *done) */ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) { - struct rq *rq =3D rq_of(cfs_rq); - struct sched_entity *se =3D cfs_rq->tg->se[cpu_of(rq)]; + int cpu =3D cpu_of(rq_of(cfs_rq)); + struct sched_entity *se =3D cfs_rq->tg->se[cpu]; + raw_spinlock_t * lock; unsigned long now =3D jiffies; unsigned long load; =20 if (cfs_rq->last_h_load_update =3D=3D now) return; =20 - WRITE_ONCE(cfs_rq->h_load_next, NULL); + /* Protects cfs_rq->h_load_next and cfs_rq->last_h_load_update */ + raw_spin_lock(lock =3D &per_cpu(h_load_lock, cpu)); + + now =3D jiffies; + if (cfs_rq->last_h_load_update =3D=3D now) + goto unlock; + + cfs_rq->h_load_next =3D NULL; for_each_sched_entity(se) { cfs_rq =3D cfs_rq_of(se); - WRITE_ONCE(cfs_rq->h_load_next, se); + cfs_rq->h_load_next =3D se; if (cfs_rq->last_h_load_update =3D=3D now) break; } @@ -9814,7 +9824,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_r= q) cfs_rq->last_h_load_update =3D now; } =20 - while ((se =3D READ_ONCE(cfs_rq->h_load_next)) !=3D NULL) { + while ((se =3D cfs_rq->h_load_next) !=3D NULL) { load =3D cfs_rq->h_load; load =3D div64_ul(load * se->avg.load_avg, cfs_rq_load_avg(cfs_rq) + 1); @@ -9822,6 +9832,8 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_r= q) cfs_rq->h_load =3D load; cfs_rq->last_h_load_update =3D now; } +unlock: + raw_spin_unlock(lock); } =20 static unsigned long task_h_load(struct task_struct *p) @@ -13665,6 +13677,9 @@ __init void init_sched_fair_class(void) zalloc_cpumask_var_node(&per_cpu(should_we_balance_tmpmask, i), GFP_KERNEL, cpu_to_node(i)); =20 +#ifdef CONFIG_FAIR_GROUP_SCHED + raw_spin_lock_init(&per_cpu(h_load_lock, i)); +#endif #ifdef CONFIG_CFS_BANDWIDTH INIT_CSD(&cpu_rq(i)->cfsb_csd, __cfsb_csd_unthrottle, cpu_rq(i)); INIT_LIST_HEAD(&cpu_rq(i)->cfsb_csd_list); --=20 2.45.2