From nobody Sat Feb 7 15:40:16 2026 Received: from mail-dl1-f43.google.com (mail-dl1-f43.google.com [74.125.82.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0689D33EB1A for ; Fri, 23 Jan 2026 01:39:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769132400; cv=none; b=H3C8fcfOUwgCPTArOHeErvfGXOwkQAZNmrjEHNZuoBIuGGSqIfIk6abXtA4ergg1dLCfqOyl0WAWKu5tWSbN5jchB+LzJAoiO907HXFj2KJLbXAohSPwxun/G/ectimaZmsmbYrxWF35nhJCqipBum06UEVCtj5emPuyYxAejEc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769132400; c=relaxed/simple; bh=hDSS0KwiuGng70TQcFhS+Bw9czWCOXlb5JsSVNXtH0M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sE35w9Xqhf57iz0SccUl21m2wvLazWo+yYxLbWKUAxo2Yxk1eSdEuLdC3MRs15zVHSsGu8fsKGFI3rfYFi9O49tlu0sRUAGNulwqugS3cGZZMdLY99mZUAORp7sRP4iUA+wrbFTsagPWGrfRrmPi3pLEHqruMn/SSr0Ftbx+5PY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EXVgTCvJ; arc=none smtp.client-ip=74.125.82.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EXVgTCvJ" Received: by mail-dl1-f43.google.com with SMTP id a92af1059eb24-12336f33098so1733197c88.0 for ; Thu, 22 Jan 2026 17:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1769132382; x=1769737182; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l6AZQMCv5PtCuHmhgpdPTXPnDHTEWLDP0NYzh6NR3To=; b=EXVgTCvJqoDWMh6MTr7t60a7J4Sn2dFdqP24Ceru5fyARlh3lyPqbedB/Svy6u+jGK 4rTJ39vzGt4iLZUmlK5IZn+liXpFOdHSzDZXpWMbzZiGoECxy3Ht7OZ7jIntlXWBgsPc zG2HiKazY7ed2zNM+MavCkNySIJQqPaTIaKxFSvGXJ9dZ+UTXuMmcjGOO9s3SD+qJ2zH fkhUtttH5/zOKMWeQ6Bb7dMWW689/dao7rp/kkdMRmz5SwZvuZkEHPFKkRGd23SA+nxs txr6IGzPAnUeGjpBSqEXq5remogQWfjToKFArgGiXEIm0c06EyFho++UkJ/qbwHUzvcv 69yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769132382; x=1769737182; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=l6AZQMCv5PtCuHmhgpdPTXPnDHTEWLDP0NYzh6NR3To=; b=kh3O/lsn1TrpNuFyDNgv/+dmGIGV2Q//ka+QBvd6YmwPICEYmUg4oCY4YLV5I+Whee P2s11cEe+9q7a5Qp7hojzTVACGNFfsCStMUAJKa0dt9nxuds23ivT9NZTK4W9gY04LAx 7+Xd8AFDcZyPl6lAbWMxXPbEFKBazycYI3chU44jWdWdPVoFq1WaHDGsOBBNjYD3b3mZ bTAwQLu+ynjEcX/D6sdzIhXNSxyEc+CE0mWnYj8spebuKNeBK/j+n3jRlB+KZLFSo/nA vTFAechj409NXCS5vA3RLMBmQT9UfyAu1qknUnG7LRV8ffSa3fyGun51z22hmLUJT3+t iZ6w== X-Forwarded-Encrypted: i=1; AJvYcCWufya9sCTnoK/u5AppINRF2YzT1QvsgApqyYUL0SndDi9Lzxe0P+d9yQVZTIPWLwn54aQDYSLYjdcTcl0=@vger.kernel.org X-Gm-Message-State: AOJu0YxSQZuFV83IU5r8QE7mMv+GM3GH2Ncer0SGSW/5iDqdn9jmQTQx m4wyp3KNTVWHRbwj8KAFihKfGaw3IEX0q7L49JkkEX4YGGqqz07u3oem X-Gm-Gg: AZuq6aKWrUhHPwj/lDupbwyyyDTzQgZlYbqA6U96ZLH20HtpEAfVjMvku9rGrlyqwt/ tKO4+mk2C5gs9fYcyINbu5CA+QpfVBIifH/t4OG4Jt9/Bq2x45Cu9wXAel9MiE/qrk8iJ4SjB3u /ddRNcs6XPMsmRzCeS1BYwbbCJ5u5h4rOKAU5XDhhAi09+fcpgLlURXU21uxxcBHgLi1YpF40yT D7LllhBOuSliKFROBl7RJOM1i3eHnUTD7fYVC6zhJYqS33tEQa8Ps8Mr3R3gPEawAccSCTX868s ktOdy2KNVi70LhcRQ7iJfA4DFiKOCHqPMt9G5oNZbbqB52QsK78LNcMPNUPRApFvcBrF4qNtzBS Q90kTYwXfNc4TIuulhLE9aFcUQvpbWHkaNYU6KmRLuoZVPnE3S5ISZGgOVxNxDBnSEicvbHzjMs 8Q7G4= X-Received: by 2002:a05:7022:2219:b0:119:e56b:91da with SMTP id a92af1059eb24-1247dbb319amr743656c88.11.1769132381824; Thu, 22 Jan 2026 17:39:41 -0800 (PST) Received: from debian ([74.48.213.230]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-1247d90ce0dsm1827743c88.4.2026.01.22.17.39.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jan 2026 17:39:41 -0800 (PST) From: Qiliang Yuan To: juri.lelli@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org Cc: realwujing@gmail.com, bsegall@google.com, dietmar.eggemann@arm.com, linux-kernel@vger.kernel.org, mgorman@suse.de, mingo@redhat.com, rostedt@goodmis.org, vschneid@redhat.com, Qiliang Yuan Subject: [PATCH v2] sched/fair: Cache NUMA node statistics to avoid O(N) scanning Date: Thu, 22 Jan 2026 20:39:33 -0500 Message-ID: <20260123013933.195263-1-realwujing@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260122161647.142704-1-realwujing@gmail.com> References: <20260122161647.142704-1-realwujing@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Optimize update_numa_stats() by leveraging pre-calculated group statistics from the load balancer hierarchy. This reduces the complexity of NUMA balancing overhead from O(CPUs_per_node) to O(1) in the hot path when stats are fresh. Signed-off-by: Qiliang Yuan Signed-off-by: Qiliang Yuan --- kernel/sched/fair.c | 35 +++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 7 +++++++ 2 files changed, 42 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e71302282671..dc46262bd227 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2099,11 +2099,36 @@ static void update_numa_stats(struct task_numa_env = *env, bool find_idle) { int cpu, idle_core =3D -1; + struct sched_domain *sd; + struct sched_group *sg; =20 memset(ns, 0, sizeof(*ns)); ns->idle_cpu =3D -1; =20 rcu_read_lock(); + /* Algorithmic Optimization: Avoid O(N) scan by using cached stats from l= oad balancer */ + sd =3D rcu_dereference(per_cpu(sd_numa, env->src_cpu)); + if (sd && !find_idle) { + sg =3D sd->groups; + do { + /* Check if this group corresponds to the node we are interested in */ + if (cpumask_test_cpu(cpumask_first(cpumask_of_node(nid)), sched_group_s= pan(sg))) { + /* Use cached stats if they are recent enough (e.g. within 10ms) */ + if (time_before(jiffies, sg->sgc->stats_update + msecs_to_jiffies(10))= ) { + ns->load =3D sg->sgc->load; + ns->runnable =3D sg->sgc->runnable; + ns->util =3D sg->sgc->util; + ns->nr_running =3D sg->sgc->nr_running; + ns->compute_capacity =3D sg->sgc->capacity; + rcu_read_unlock(); + goto skip_scan; + } + break; + } + sg =3D sg->next; + } while (sg !=3D sd->groups); + } + for_each_cpu(cpu, cpumask_of_node(nid)) { struct rq *rq =3D cpu_rq(cpu); =20 @@ -2126,6 +2151,7 @@ static void update_numa_stats(struct task_numa_env *e= nv, } rcu_read_unlock(); =20 +skip_scan: ns->weight =3D cpumask_weight(cpumask_of_node(nid)); =20 ns->node_type =3D numa_classify(env->imbalance_pct, ns); @@ -10488,6 +10514,15 @@ static inline void update_sg_lb_stats(struct lb_en= v *env, if (sgs->group_type =3D=3D group_overloaded) sgs->avg_load =3D (sgs->group_load * SCHED_CAPACITY_SCALE) / sgs->group_capacity; + + /* Algorithmic Optimization: Cache group stats for O(1) NUMA lookups */ + if (env->sd->flags & SD_NUMA) { + group->sgc->nr_running =3D sgs->sum_h_nr_running; + group->sgc->load =3D sgs->group_load; + group->sgc->util =3D sgs->group_util; + group->sgc->runnable =3D sgs->group_runnable; + WRITE_ONCE(group->sgc->stats_update, jiffies); + } } =20 /** diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index d30cca6870f5..81160790993e 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2105,6 +2105,13 @@ struct sched_group_capacity { =20 int id; =20 + /* O(1) NUMA stats cache */ + unsigned long nr_running; + unsigned long load; + unsigned long util; + unsigned long runnable; + unsigned long stats_update; + unsigned long cpumask[]; /* Balance mask */ }; =20 --=20 2.51.0