From nobody Sun Feb 8 05:35:37 2026 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4BA914AD30 for ; Tue, 20 Feb 2024 20:25:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708460739; cv=none; b=dPcGHHa3PfY4mGjKQ8rhSTDpP6P288C/zGazArq3h4lK6He9UBrW+8I5R6jnEd5sRfYIU0kMK4pkKVdGlCo7LyVHKC3sdOKcg9z0eag1ZxKSxJkLCrhOj2jlaQzIE03kxNYH5yb1xpbeOJHjvJ9Rxklfsb9GxUvoIfIxd6uMTG4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708460739; c=relaxed/simple; bh=xXscJzLuofLsErQ12WqLPIQ0oQ+LmPxgWF+KcXzMm+g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sOMilL8z2bd9Bi9iv0sZx40r/VAdk9+MXmK3PK8MGxPqXlvqnINx6KhgHUyP+KWh53DmbHP2yP4rxkN3VtLPvTKlMeRIok1o430Hsbyx8LkH+TIzja9ZSYZaczK12+Bs6tIIwTouK8X5VGo2eSJqOCx8SkzEa/hRnAoCfV+tE0Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fCW97FD4; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fCW97FD4" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-1dc1ff697f9so6516555ad.0 for ; Tue, 20 Feb 2024 12:25:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708460737; x=1709065537; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZBnklkS7WMFU168TcI+v2P140FGO/f8vdLXL7Ok+vL4=; b=fCW97FD4yu0udtb5AbnrJN47SKClRpOVbgYTqOeIGEVoL5HrOHyt4D+EwLtpRw3MiN LCEI33+xfJGoNq+0pTEzuuZJxDY70W4sH/wwq/YOvj8NVJfrjmKrvYGM95Uaa4A+2kBT uu5jU1E2pboHXl51UsEXZ4WDshDnYHaluGVAN4pIbIL7UDcuT1o3K1rWQPp1GOSH7mPJ 5kmX9mzHhxTedLDKN6dFihK2zc3vSsiNMXYdf+c/kYC2D9D4kVs+yNpIzhWTqcYAx1Tl gxtG4Y9zZyn05dGLpOcZMp5iydqu3j/pYzm7m7JLFhKUptNegd/v8ROgVGcnYYXVmRqp 5UMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708460737; x=1709065537; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZBnklkS7WMFU168TcI+v2P140FGO/f8vdLXL7Ok+vL4=; b=Lxtq2NLtW93qliOwazEdJwPgqWemhGrWA7athqNzRVZBBL6MWQGDCoQFhbxhY89yJd WMn2PrzIUKqOi4k1W4tjMN3rQkhHCC2gyK6wUGqglgoIxapbmJNz3gtgwaTIDZ9hasBZ 0pk5OCMXX882fGtCO+N5xQrwBZ5YZnWzC239SbpF+OvkzkcQNidrCnYl03yYqeLIIDF4 bwk9PZNlTddX0AqMhX3s56jYYEZBd6iyrXIWpQmaExel6upRt8I/AuQL3WYn7TeGPAWY Y2l2uQSF10IExJ/6T7XS8KYDnHI2HHOtJT6AmkPRKz9Yf+NrHEM0Ho15tjn5U7LhrFHT pLbw== X-Gm-Message-State: AOJu0YzVh3Ddh34s6GvTImRMC7DVVephpTmhE3urbvKxw598Aq9dc2Pj Nh+WamO26BfO51sfh1bOE/XgcaReancxM+L5EE8nFLT+eLOL0QLVxh8rEfYiXOWr X-Google-Smtp-Source: AGHT+IHkA4V6qwKf4cOC/uOz/uS+q2KycSuLitxGvr0jW9ZC5rlBdfn8y7exgV7lyAyk5hf55NW/PA== X-Received: by 2002:a17:902:eb8c:b0:1db:e245:8c35 with SMTP id q12-20020a170902eb8c00b001dbe2458c35mr6191582plg.30.1708460736845; Tue, 20 Feb 2024 12:25:36 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id je13-20020a170903264d00b001db40c0ed33sm6696678plb.61.2024.02.20.12.25.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Feb 2024 12:25:36 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, ying.huang@intel.com, hannes@cmpxchg.org, dan.j.williams@intel.com, dave.jiang@intel.com, Gregory Price Subject: [RFC 1/1] mm/mempolicy: introduce system default interleave weights Date: Tue, 20 Feb 2024 15:25:29 -0500 Message-Id: <20240220202529.2365-2-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240220202529.2365-1-gregory.price@memverge.com> References: <20240220202529.2365-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Startup and hotplug code may register HMAT data for memory devices. Utilize this data to generate reasonable default weighted interleave values. Introduce `mempolicy_set_node_perf()`. A function which can be invoked from node and CXL code to have mempolicy rebalance the system default interleave weights. mempolicy_set_node_perf() cache's each node's bandwidth (in this patch: min(read_bw, write_bw)), and recalculates the weight associated with each node. After weights are calculated, we use gcd() to reduce these weights to the smallest amount possible in and effort to more aggressively interleave on smaller intervals. For example, a 1-socket system with a CXL memory expander which exposes 224GB/s and 64GB/s of bandwidth respectively will end up with a weight array of [7,2]. The downside of this approach is that some distributes may experience large default values if they happen to a bandwidth distribution that includes an unfortunate prime number, or if any two values are co-prime. Signed-off-by: Gregory Price --- drivers/acpi/numa/hmat.c | 1 + drivers/base/node.c | 7 +++ include/linux/mempolicy.h | 4 ++ mm/mempolicy.c | 129 ++++++++++++++++++++++++++++++-------- 4 files changed, 116 insertions(+), 25 deletions(-) diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c index d6b85f0f6082..7935d387e001 100644 --- a/drivers/acpi/numa/hmat.c +++ b/drivers/acpi/numa/hmat.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/base/node.c b/drivers/base/node.c index 1c05640461dd..30458df504b4 100644 --- a/drivers/base/node.c +++ b/drivers/base/node.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -214,6 +215,12 @@ void node_set_perf_attrs(unsigned int nid, struct acce= ss_coordinate *coord, break; } } + + /* When setting CPU access coordinates, update mempolicy */ + if (access =3D=3D ACCESS_COORDINATE_CPU) { + if (mempolicy_set_node_perf(nid, coord)) + pr_info("failed to set node%d mempolicy attrs\n", nid); + } } =20 /** diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index 931b118336f4..d564e9e893ea 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -177,6 +178,9 @@ static inline bool mpol_is_preferred_many(struct mempol= icy *pol) =20 extern bool apply_policy_zone(struct mempolicy *policy, enum zone_type zon= e); =20 +extern int mempolicy_set_node_perf(unsigned int node, + struct access_coordinate *coords); + #else =20 struct mempolicy {}; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index ba0b2b81bd08..0a82aa51e497 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -109,6 +109,7 @@ #include #include #include +#include =20 #include #include @@ -139,31 +140,114 @@ static struct mempolicy default_policy =3D { static struct mempolicy preferred_node_policy[MAX_NUMNODES]; =20 /* - * iw_table is the sysfs-set interleave weight table, a value of 0 denotes - * system-default value should be used. A NULL iw_table also denotes that - * system-default values should be used. Until the system-default table - * is implemented, the system-default is always 1. + * The interleave weight tables denote what weights should be used with + * the weighted interleave policy. There are two tables: + * - iw_table : the sysfs-set interleave weight table + * - default_iw_table : the system default interleave weight table. * - * iw_table is RCU protected + * If the iw_table is NULL, default_iw_table values are used. + * If both tables are NULL, a minimum weight of 1 is always used. + * A value of 0 in the iw_table means the system default value will be use= d. + * + * iw_table, and default_iw_table are RCU protected + * node_bw_table is protected by default_iwt_lock + * + * system startup and hotplug code may register node performance informati= on + * via mempolicy_set_node_attributes() */ +static unsigned long *node_bw_table; +static u8 __rcu *default_iw_table; +static DEFINE_MUTEX(default_iwt_lock); + static u8 __rcu *iw_table; static DEFINE_MUTEX(iw_table_lock); =20 static u8 get_il_weight(int node) { - u8 *table; + u8 *table, *default_table; u8 weight; =20 rcu_read_lock(); table =3D rcu_dereference(iw_table); - /* if no iw_table, use system default */ - weight =3D table ? table[node] : 1; - /* if value in iw_table is 0, use system default */ - weight =3D weight ? weight : 1; + default_table =3D rcu_dereference(default_iw_table); + /* if no table pointers or value is 0, use system default or 1 */ + weight =3D table ? table[node] : 0; + weight =3D weight ? weight : (default_table ? default_table[node] : 1); rcu_read_unlock(); return weight; } =20 +int mempolicy_set_node_perf(unsigned int node, struct access_coordinate *c= oords) +{ + unsigned long *old_bw, *new_bw; + unsigned long gcd_val; + u8 *old_iw, *new_iw; + uint64_t ttl_bw =3D 0; + int i; + + new_bw =3D kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); + if (!new_bw) + return -ENOMEM; + + new_iw =3D kzalloc(nr_node_ids, GFP_KERNEL); + if (!new_iw) { + kfree(new_bw); + return -ENOMEM; + } + + mutex_lock(&default_iwt_lock); + old_bw =3D node_bw_table; + old_iw =3D rcu_dereference_protected(default_iw_table, + lockdep_is_held(&default_iwt_lock)); + + if (old_bw) + memcpy(new_bw, old_bw, nr_node_ids*sizeof(unsigned long)); + new_bw[node] =3D min(coords->read_bandwidth, coords->write_bandwidth); + + /* New recalculate the bandwidth distribution given the new info */ + for (i =3D 0; i < nr_node_ids; i++) + ttl_bw +=3D new_bw[i]; + + /* If node is not set or has < 1% of total bw, use minimum value of 1 */ + for (i =3D 0; i < nr_node_ids; i++) { + if (new_bw[i]) + new_iw[i] =3D max((100 * new_bw[i] / ttl_bw), 1); + else + new_iw[i] =3D 1; + } + /* + * Now attempt to aggressively reduce the interleave weights by GCD + * We want smaller interleave intervals to have a better distribution + * of memory, even on smaller memory regions. If weights are divisible + * by each other, we can do some quick math to aggresively squash them. + */ +reduce: + gcd_val =3D new_iw[i]; + for (i =3D 0; i < nr_node_ids; i++) { + /* Skip nodes that haven't been set */ + if (!new_bw[i]) + continue; + gcd_val =3D gcd(gcd_val, new_iw[i]); + if (gcd_val =3D=3D 1) + goto leave; + } + for (i =3D 0; i < nr_node_ids; i++) { + if (!new_bw[i]) + continue; + new_iw[i] /=3D gcd_val; + } + /* repeat until we get a gcd of 1 */ + goto reduce; +leave: + node_bw_table =3D new_bw; + rcu_assign_pointer(default_iw_table, new_iw); + mutex_unlock(&default_iwt_lock); + synchronize_rcu(); + kfree(old_bw); + kfree(old_iw); + return 0; +} + /** * numa_nearest_node - Find nearest node by state * @node: Node id to start the search @@ -1983,7 +2067,7 @@ static unsigned int weighted_interleave_nid(struct me= mpolicy *pol, pgoff_t ilx) { nodemask_t nodemask; unsigned int target, nr_nodes; - u8 *table; + u8 *table, *default_table; unsigned int weight_total =3D 0; u8 weight; int nid; @@ -1994,11 +2078,13 @@ static unsigned int weighted_interleave_nid(struct = mempolicy *pol, pgoff_t ilx) =20 rcu_read_lock(); table =3D rcu_dereference(iw_table); + default_table =3D rcu_dereference(default_iw_table); /* calculate the total weight */ for_each_node_mask(nid, nodemask) { /* detect system default usage */ - weight =3D table ? table[nid] : 1; - weight =3D weight ? weight : 1; + weight =3D table ? table[nid] : 0; + weight =3D weight ? weight : + (default_table ? default_table[nid] : 1); weight_total +=3D weight; } =20 @@ -2007,8 +2093,9 @@ static unsigned int weighted_interleave_nid(struct me= mpolicy *pol, pgoff_t ilx) nid =3D first_node(nodemask); while (target) { /* detect system default usage */ - weight =3D table ? table[nid] : 1; - weight =3D weight ? weight : 1; + weight =3D table ? table[nid] : 0; + weight =3D weight ? weight : + (default_table ? default_table[nid] : 1); if (target < weight) break; target -=3D weight; @@ -2391,7 +2478,7 @@ static unsigned long alloc_pages_bulk_array_weighted_= interleave(gfp_t gfp, unsigned long nr_allocated =3D 0; unsigned long rounds; unsigned long node_pages, delta; - u8 *table, *weights, weight; + u8 *weights, weight; unsigned int weight_total =3D 0; unsigned long rem_pages =3D nr_pages; nodemask_t nodes; @@ -2440,16 +2527,8 @@ static unsigned long alloc_pages_bulk_array_weighted= _interleave(gfp_t gfp, if (!weights) return total_allocated; =20 - rcu_read_lock(); - table =3D rcu_dereference(iw_table); - if (table) - memcpy(weights, table, nr_node_ids); - rcu_read_unlock(); - - /* calculate total, detect system default usage */ for_each_node_mask(node, nodes) { - if (!weights[node]) - weights[node] =3D 1; + weights[node] =3D get_il_weight(node); weight_total +=3D weights[node]; } =20 --=20 2.39.1