From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C62111F60B; Wed, 3 Jan 2024 22:42:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mAMqZIfI" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-1d480c6342dso44451775ad.2; Wed, 03 Jan 2024 14:42:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321751; x=1704926551; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o5MKWIUvYvBzFl4/zY3kO9lIA0w6d0GnkdKyiozKias=; b=mAMqZIfI2kRMSMKRbZnGeQ65sS0vFdaVoV2BW3GcL0iywpOcDrTUKRN08nbRmnUjBu nJ9X9RcnNsYZtr9r+rzjYIOXjwy+HB0OD4wG065NGunklH9111IKi09OH3QZ6GVb9CpK xo06x/9SJx55oT7nPpJDHIL56vv66TqLGeeOO99t20u6LHJlkkGvyArXyDqwefA/wIw0 98N28AH+Ot8scrtrd0aJUOz5pg/QWdqpXx78cYUbRaD6DbgQBsa+J9qY5+mGUCtyUp4P bAzfN1e2NYvIpnPWMNH+QB69C1L08eO52GG6pM66UbyjZJyM/ajUn5VZuWi0n7Z68KBO 1zIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321751; x=1704926551; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o5MKWIUvYvBzFl4/zY3kO9lIA0w6d0GnkdKyiozKias=; b=i3P3WV8BRssr2RiY+phVnAho7PZyYB5ZlcK1P09Xv2xJ22Vle232tdEivwHY/s/One eu3fRH7XNP2MrcxKMESwIL+cj42dwt92pAxNxLuXBQlLv496D62SckujHTNNqRYpUudg 7Xwqwozv8/BM1apLCX/V5GKTmkegUtrOFESYgN+deQ/lSWs1kOGO5pz5WCwGCYhjjqBn 7qhINAWGN6EI8rGyQc6eCAuUJN0fXnu6Jn7KCRDbSutOg05lAykXp7Kd2VgFbuD7VpXT WLYiGeI24bSLFTndAqr7C/K5a/Ab8xFvr38UfVLna0/ZGBRai+k1efW2nT7F0akuHnA+ 24Rg== X-Gm-Message-State: AOJu0YzfmXIg5rK5WQprQWFdtuEAjX8Hj3u/aVcOH45Jley+hJCb0Yyt g2RWL9xCqfiZ8vY6X+x/q90pHS5TMzCseaQ= X-Google-Smtp-Source: AGHT+IETjmZI/H5jzuTEDLcLeTtQG732PMB/ir8hLWjOSjN9vjly0mA8Vcj8KqCQX+cOsD5NH6lN8g== X-Received: by 2002:a17:902:e5d2:b0:1d4:d5bb:5d7c with SMTP id u18-20020a170902e5d200b001d4d5bb5d7cmr1022739plf.110.1704321750996; Wed, 03 Jan 2024 14:42:30 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.42.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:42:30 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v6 01/12] mm/mempolicy: implement the sysfs-based weighted_interleave interface Date: Wed, 3 Jan 2024 17:41:58 -0500 Message-Id: <20240103224209.2541-2-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Rakie Kim This patch provides a way to set interleave weight information under sysfs at /sys/kernel/mm/mempolicy/weighted_interleave/nodeN The sysfs structure is designed as follows. $ tree /sys/kernel/mm/mempolicy/ /sys/kernel/mm/mempolicy/ [1] =E2=94=94=E2=94=80=E2=94=80 weighted_interleave [2] =E2=94=9C=E2=94=80=E2=94=80 node0 [3] =E2=94=94=E2=94=80=E2=94=80 node1 Each file above can be explained as follows. [1] mm/mempolicy: configuration interface for mempolicy subsystem [2] weighted_interleave/: config interface for weighted interleave policy [3] weighted_interleave/nodeN: weight for nodeN Internally, there is a secondary table `default_iw_table`, which holds kernel-internal default interleave weights for each possible node. If the value for a node is set to `0`, the default value will be used. If sysfs is disabled in the config, interleave weights will default to use `default_iw_table`. Suggested-by: Huang Ying Signed-off-by: Rakie Kim Signed-off-by: Honggyu Kim Co-developed-by: Gregory Price Signed-off-by: Gregory Price Co-developed-by: Hyeongtak Ji Signed-off-by: Hyeongtak Ji Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- .../ABI/testing/sysfs-kernel-mm-mempolicy | 4 + ...fs-kernel-mm-mempolicy-weighted-interleave | 26 +++ mm/mempolicy.c | 178 ++++++++++++++++++ 3 files changed, 208 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-wei= ghted-interleave diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy b/Document= ation/ABI/testing/sysfs-kernel-mm-mempolicy new file mode 100644 index 000000000000..2dcf24f4384a --- /dev/null +++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy @@ -0,0 +1,4 @@ +What: /sys/kernel/mm/mempolicy/ +Date: December 2023 +Contact: Linux memory management mailing list +Description: Interface for Mempolicy diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-i= nterleave b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-in= terleave new file mode 100644 index 000000000000..e6a38139bf0f --- /dev/null +++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interlea= ve @@ -0,0 +1,26 @@ +What: /sys/kernel/mm/mempolicy/weighted_interleave/ +Date: December 2023 +Contact: Linux memory management mailing list +Description: Configuration Interface for the Weighted Interleave policy + +What: /sys/kernel/mm/mempolicy/weighted_interleave/nodeN +Date: December 2023 +Contact: Linux memory management mailing list +Description: Weight configuration interface for nodeN + + The interleave weight for a memory node (N). These weights are + utilized by processes which have set their mempolicy to + MPOL_WEIGHTED_INTERLEAVE and have opted into global weights by + omitting a task-local weight array. + + These weights only affect new allocations, and changes at runtime + will not cause migrations on already allocated pages. + + The minimum weight for a node is always 1. + + Minimum weight: 1 + Maximum weight: 255 + + Writing an empty string or `0` will reset the weight to the + system default. The system default may be set by the kernel + or drivers at boot or during hotplug events. diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 10a590ee1c89..30da1a1be707 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -131,6 +131,23 @@ static struct mempolicy default_policy =3D { =20 static struct mempolicy preferred_node_policy[MAX_NUMNODES]; =20 +/* + * default_iw_table is the kernel-internal default value interleave + * weight table. It is to be set by driver code capable of reading + * HMAT/CDAT information, and to provide mempolicy a sane set of + * default weight values for WEIGHTED_INTERLEAVE mode. + * + * By default, prior to HMAT/CDAT information being consumed, the + * default weight of all nodes is 1. The default weight of any + * node can only be in the range 1-255. A 0-weight is not allowed. + */ +static u8 default_iw_table[MAX_NUMNODES]; +/* + * iw_table is the sysfs-set interleave weight table, a value of 0 + * denotes that the default_iw_table value should be used. + */ +static u8 iw_table[MAX_NUMNODES]; + /** * numa_nearest_node - Find nearest node by state * @node: Node id to start the search @@ -3067,3 +3084,164 @@ void mpol_to_str(char *buffer, int maxlen, struct m= empolicy *pol) p +=3D scnprintf(p, buffer + maxlen - p, ":%*pbl", nodemask_pr_args(&nodes)); } + +#ifdef CONFIG_SYSFS +struct iw_node_attr { + struct kobj_attribute kobj_attr; + int nid; +}; + +static ssize_t node_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct iw_node_attr *node_attr; + u8 weight; + + node_attr =3D container_of(attr, struct iw_node_attr, kobj_attr); + weight =3D iw_table[node_attr->nid]; + if (!weight) + weight =3D default_iw_table[node_attr->nid]; + return sysfs_emit(buf, "%d\n", weight); +} + +static ssize_t node_store(struct kobject *kobj, struct kobj_attribute *att= r, + const char *buf, size_t count) +{ + struct iw_node_attr *node_attr; + u8 weight =3D 0; + + node_attr =3D container_of(attr, struct iw_node_attr, kobj_attr); + /* If no input, revert to default weight */ + if (count =3D=3D 0 || sysfs_streq(buf, "")) + weight =3D 0; + else if (kstrtou8(buf, 0, &weight)) + return -EINVAL; + + iw_table[node_attr->nid] =3D weight; + return count; +} + +static struct iw_node_attr *node_attrs[MAX_NUMNODES]; + +static void sysfs_wi_node_release(struct iw_node_attr *node_attr, + struct kobject *parent) +{ + if (!node_attr) + return; + sysfs_remove_file(parent, &node_attr->kobj_attr.attr); + kfree(node_attr->kobj_attr.attr.name); + kfree(node_attr); +} + +static void sysfs_mempolicy_release(struct kobject *mempolicy_kobj) +{ + int i; + + for (i =3D 0; i < MAX_NUMNODES; i++) + sysfs_wi_node_release(node_attrs[i], mempolicy_kobj); + kobject_put(mempolicy_kobj); +} + +static const struct kobj_type mempolicy_ktype =3D { + .sysfs_ops =3D &kobj_sysfs_ops, + .release =3D sysfs_mempolicy_release, +}; + +static int add_weight_node(int nid, struct kobject *wi_kobj) +{ + struct iw_node_attr *node_attr; + char *name; + + node_attr =3D kzalloc(sizeof(*node_attr), GFP_KERNEL); + if (!node_attr) + return -ENOMEM; + + name =3D kasprintf(GFP_KERNEL, "node%d", nid); + if (!name) { + kfree(node_attr); + return -ENOMEM; + } + + sysfs_attr_init(&node_attr->kobj_attr.attr); + node_attr->kobj_attr.attr.name =3D name; + node_attr->kobj_attr.attr.mode =3D 0644; + node_attr->kobj_attr.show =3D node_show; + node_attr->kobj_attr.store =3D node_store; + node_attr->nid =3D nid; + + if (sysfs_create_file(wi_kobj, &node_attr->kobj_attr.attr)) { + kfree(node_attr->kobj_attr.attr.name); + kfree(node_attr); + pr_err("failed to add attribute to weighted_interleave\n"); + return -ENOMEM; + } + + node_attrs[nid] =3D node_attr; + return 0; +} + +static int add_weighted_interleave_group(struct kobject *root_kobj) +{ + struct kobject *wi_kobj; + int nid, err; + + wi_kobj =3D kzalloc(sizeof(struct kobject), GFP_KERNEL); + if (!wi_kobj) + return -ENOMEM; + + err =3D kobject_init_and_add(wi_kobj, &mempolicy_ktype, root_kobj, + "weighted_interleave"); + if (err) { + kfree(wi_kobj); + return err; + } + + memset(node_attrs, 0, sizeof(node_attrs)); + for_each_node_state(nid, N_POSSIBLE) { + err =3D add_weight_node(nid, wi_kobj); + if (err) { + pr_err("failed to add sysfs [node%d]\n", nid); + break; + } + } + if (err) + kobject_put(wi_kobj); + return 0; +} + +static int __init mempolicy_sysfs_init(void) +{ + int err; + struct kobject *root_kobj; + + memset(&default_iw_table, 1, sizeof(default_iw_table)); + memset(&iw_table, 0, sizeof(iw_table)); + + root_kobj =3D kobject_create_and_add("mempolicy", mm_kobj); + if (!root_kobj) { + pr_err("failed to add mempolicy kobject to the system\n"); + return -ENOMEM; + } + + err =3D add_weighted_interleave_group(root_kobj); + + if (err) + kobject_put(root_kobj); + return err; + +} +#else +static int __init mempolicy_sysfs_init(void) +{ + /* + * if sysfs is not enabled MPOL_WEIGHTED_INTERLEAVE defaults to + * MPOL_INTERLEAVE behavior, but is still defined separately to + * allow task-local weighted interleave and system-defaults to + * operate as intended. + */ + memset(&default_iw_table, 1, sizeof(default_iw_table)); + memset(&iw_table, 0, sizeof(iw_table)); + return 0; +} +#endif /* CONFIG_SYSFS */ +late_initcall(mempolicy_sysfs_init); --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A3BB200AE; Wed, 3 Jan 2024 22:42:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lYznkNce" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-1d44200b976so40017325ad.2; Wed, 03 Jan 2024 14:42:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321761; x=1704926561; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R+2gqGxCQVVJrCooIPs7Fc2JoxPo3Bt1g3/hDzEpuj8=; b=lYznkNce6ro0tKZ/nS3v/LdaUx3GTp2uCQwf1C9nAnofEWdc1dmOfsqxhT0srQveMe /cAIizN8d9zgY5yUjDyH4p4vGpvwGQ4VphEoTtkeMv1WYazk8NhBndnEwW2KkVLE2Nr2 VeoXQUCqRWIZ0XCfhrcf5D11KXCH7t2JxoTCzny7yw8tMtcEg5f3pI3MvFTg8PZSMlZi G8yYizm62wHIxYc6v3oPr8eX/kl0fsgOrl2o+IIkVIhe63qKe6c2WZMIQRYQIe/SjR2O NMg5DykfGv7ojTdcchT1sh1WJjf88rNIfPMuJi8gB5Y3LG0/KpLOAG0XZVzR6iH1pUme xR0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321761; x=1704926561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R+2gqGxCQVVJrCooIPs7Fc2JoxPo3Bt1g3/hDzEpuj8=; b=XrTLmHY78TZCtNfzKbk+51chuhT1MCSOsHvs6WtqveBLD9vv+34H03whIGL/u+mE49 nLMoIG2D+G9Z6ckTpQ2gox6Ty/fOSrB+NlLMETjMfhNObl4Rrjy7XFAZDNa/+Hr5dhok T1GregYNkAl44m14UXLOJgp1zOxBgH8aDfLrL1FBzVNdjJtFauMIrIqVNh89Hrbi3/iF koawDKNLG0qJYjqhRyCbxM0ZWYC0ce4FpVw+MQCkE+ZGsEHDHoNQ0Djq6/26T57Y35Gl 4ulKJlsqdSy1z6B0AdaC7lQghNhpMXFwofrRvEEd23YVE2PgivmJZtAL5ykzS4BNLuJ0 GM8g== X-Gm-Message-State: AOJu0YxvivVdiGsxYuKskOw+fgB1kol7T6lKQi42DxhWQZLoNzlvzhf1 PLpyWeSlUtKt8Kfyrnq/5w== X-Google-Smtp-Source: AGHT+IFnOwL88aJ7puwINYSQH+GK5aFsJocJT3yRJ9rZ9oLHhwfKfEgPNmLvb6n4zqZvaSRmEBRslg== X-Received: by 2002:a17:902:ecce:b0:1d4:35ad:41cd with SMTP id a14-20020a170902ecce00b001d435ad41cdmr7934219plh.108.1704321760933; Wed, 03 Jan 2024 14:42:40 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.42.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:42:40 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v6 02/12] mm/mempolicy: refactor a read-once mechanism into a function for re-use Date: Wed, 3 Jan 2024 17:41:59 -0500 Message-Id: <20240103224209.2541-3-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" move the use of barrier() to force policy->nodemask onto the stack into a function read_once_policy_nodemask so that it may be re-used. Suggested-by: Huang Ying Signed-off-by: Gregory Price Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- mm/mempolicy.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 30da1a1be707..6cdb00acb86b 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1900,6 +1900,20 @@ unsigned int mempolicy_slab_node(void) } } =20 +static unsigned int read_once_policy_nodemask(struct mempolicy *pol, + nodemask_t *mask) +{ + /* + * barrier stabilizes the nodemask locally so that it can be iterated + * over safely without concern for changes. Allocators validate node + * selection does not violate mems_allowed, so this is safe. + */ + barrier(); + __builtin_memcpy(mask, &pol->nodes, sizeof(nodemask_t)); + barrier(); + return nodes_weight(*mask); +} + /* * Do static interleaving for interleave index @ilx. Returns the ilx'th * node in pol->nodes (starting from ilx=3D0), wrapping around if ilx @@ -1907,20 +1921,12 @@ unsigned int mempolicy_slab_node(void) */ static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx) { - nodemask_t nodemask =3D pol->nodes; + nodemask_t nodemask; unsigned int target, nnodes; int i; int nid; - /* - * The barrier will stabilize the nodemask in a register or on - * the stack so that it will stop changing under the code. - * - * Between first_node() and next_node(), pol->nodes could be changed - * by other threads. So we put pol->nodes in a local stack. - */ - barrier(); =20 - nnodes =3D nodes_weight(nodemask); + nnodes =3D read_once_policy_nodemask(pol, &nodemask); if (!nnodes) return numa_node_id(); target =3D ilx % nnodes; --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D70CD200C7; Wed, 3 Jan 2024 22:42:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l124v6Rl" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-1d4414ec9c7so28437025ad.0; Wed, 03 Jan 2024 14:42:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321769; x=1704926569; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=REqFU5ZxtHyflU6mIkcuYZVFroNR18NafHasOjXOpiU=; b=l124v6RliIzTkkjlHCebixGp/qdtfy9QCmxC76ZJUeQxKqjUxtQQEqdtYPalZtG+cs tUdcFRRSyHDa9YW4ntTRIAXCLpgfPxAb56UaZvxuFXLEhMIxiwi43lPJA/5BJw+NpvEy WomEOWQvVPSv7Jx23y5iUCHDpLm/z4u1X8c9u8Nk0WYs3NhXtj1iip4JLb/1lhCOSYaS s8bDiSDjjsPV/T1hgvj0J5BCXNhJsVyMQv4QRB8cbhbhr8ijJloaIEYg+xNbGyqPwwnG tDZa/YPq2KZdi5PcSwaKRQtWE0pyK925YJ0bQJyb4EJcRxss2FnALcOfZuy0HyKyLJCS G9Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321769; x=1704926569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=REqFU5ZxtHyflU6mIkcuYZVFroNR18NafHasOjXOpiU=; b=ItpP0IGZKmMs9W5ePjxrMeV9pJPPvKrl+B5W8v46AzwrIWlFWudCXNSVM3dCE/f7S1 CKkMS90VUYEHA8jyPpflzx38mFHGMSCF2SRdeU4IZ2cfPks+/vpIdXBMIRXrYg7cXdVn BrCSwALufeSCRj0Wwm5pez4XbJ8iDy+YRaytJWgFG1hOtraw/dSh3BPEsqAMlwj6vPjB vYI6qx9RSkTLJESOhUlJ1klSPkkLsnJomuvHjujXHQjxyYIgmi+mt4OMBf1/+SUkOM4Y 7wSCdZDH57FfqeNFAR40L9jAqIXxA8+cWP+a23eRnP7QR0S4w2+dYb0MgTjCg8q+iUP2 fb4Q== X-Gm-Message-State: AOJu0YzXLZVYEW8+21tRGtQAwkwMrQjXxJYarEPH5gMa3Q7xleeUhHhY YL0HxTTT1nkhd7bxXVct0A== X-Google-Smtp-Source: AGHT+IEch49DwGawQBn0E3k8RD3KwVjuH9SY2ca7uybqAfQ2DMlZPCK55U6uetT1xyJ87hwmWjjubg== X-Received: by 2002:a17:902:ce84:b0:1d4:5c6c:2089 with SMTP id f4-20020a170902ce8400b001d45c6c2089mr9352928plg.14.1704321769076; Wed, 03 Jan 2024 14:42:49 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.42.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:42:48 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com, Srinivasulu Thanneeru Subject: [PATCH v6 03/12] mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving Date: Wed, 3 Jan 2024 17:42:00 -0500 Message-Id: <20240103224209.2541-4-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When a system has multiple NUMA nodes and it becomes bandwidth hungry, the current MPOL_INTERLEAVE could be an wise option. However, if those NUMA nodes consist of different types of memory such as having local DRAM and CXL memory together, the current round-robin based interleaving policy doesn't maximize the overall bandwidth because of their different bandwidth characteristics. Instead, the interleaving can be more efficient when the allocation policy follows each NUMA nodes' bandwidth weight rather than having 1:1 round-robin allocation. This patch introduces a new memory policy, MPOL_WEIGHTED_INTERLEAVE, which enables weighted interleaving between NUMA nodes. Weighted interleave allows for a proportional distribution of memory across multiple numa nodes, preferablly apportioned to match the bandwidth capacity of each node from the perspective of the accessing node. For example, if a system has 1 CPU node (0), and 2 memory nodes (0,1), with a relative bandwidth of (100GB/s, 50GB/s) respectively, the appropriate weight distribution is (2:1). Weights will be acquired from the global weight matrix exposed by the sysfs extension: /sys/kernel/mm/mempolicy/weighted_interleave/ or from the `default_iw_table`, which will be extended to allow defaults to be registered by core/cxl drivers in the future. The policy will then allocate the number of pages according to the set weights. For example, if the weights are (2,1), then 2 pages will be allocated on node0 for every 1 page allocated on node1. The new flag MPOL_WEIGHTED_INTERLEAVE can be used in set_mempolicy(2) and mbind(2). There are 3 integration points: weighted_interleave_nodes: Counts the number of allocations as they occur, and applies the weight for the current node. When the weight reaches 0, switch to the next node. Applied by `mempolicy_slab_node()` and `policy_nodemask()` weighted_interleave_nid: Gets the total weight of the nodemask as well as each individual node weight, then calculates the node based on the given index. Applied by `policy_nodemask()` and `mpol_misplaced()` bulk_array_weighted_interleave: Gets the total weight of the nodemask as well as each individual node weight, then calculates the number of "interleave rounds" as well as any delta ("partial round"). Calculates the number of pages for each node and allocates them. If a node was scheduled for interleave via interleave_nodes, the current weight (pol->cur_weight) will be allocated first, before the remaining bulk calculation is done. This simplifies the calculation at the cost of an additional allocation call. One piece of complexity is the interaction between a recent refactor which split the logic to acquire the "ilx" (interleave index) of an allocation and the actually application of the interleave. The calculation of the `interleave index` is done by `get_vma_policy()`, while the actual selection of the node will be later appliex by the relevant weighted_interleave function. Suggested-by: Hasan Al Maruf Signed-off-by: Gregory Price Co-developed-by: Rakie Kim Signed-off-by: Rakie Kim Co-developed-by: Honggyu Kim Signed-off-by: Honggyu Kim Co-developed-by: Hyeongtak Ji Signed-off-by: Hyeongtak Ji Co-developed-by: Srinivasulu Thanneeru Signed-off-by: Srinivasulu Thanneeru Co-developed-by: Ravi Jonnalagadda Signed-off-by: Ravi Jonnalagadda Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- .../admin-guide/mm/numa_memory_policy.rst | 9 + include/linux/mempolicy.h | 6 + include/uapi/linux/mempolicy.h | 1 + mm/mempolicy.c | 198 +++++++++++++++++- 4 files changed, 211 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Document= ation/admin-guide/mm/numa_memory_policy.rst index eca38fa81e0f..a70f20ce1ffb 100644 --- a/Documentation/admin-guide/mm/numa_memory_policy.rst +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -250,6 +250,15 @@ MPOL_PREFERRED_MANY can fall back to all existing numa nodes. This is effectively MPOL_PREFERRED allowed for a mask rather than a single node. =20 +MPOL_WEIGHTED_INTERLEAVE + This mode operates the same as MPOL_INTERLEAVE, except that + interleaving behavior is executed based on weights set in + /sys/kernel/mm/mempolicy/weighted_interleave/ + + Weighted interleave allocates pages on nodes according to a + weight. For example if nodes [0,1] are weighted [5,2], 5 pages + will be allocated on node0 for every 2 pages allocated on node1. + NUMA memory policy supports the following optional mode flags: =20 MPOL_F_STATIC_NODES diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index 931b118336f4..fae903b1d3de 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -54,6 +54,12 @@ struct mempolicy { nodemask_t cpuset_mems_allowed; /* relative to these nodes */ nodemask_t user_nodemask; /* nodemask passed by user */ } w; + + /* Weighted interleave settings */ + struct { + u8 cur_weight; + u8 scratch_weights[MAX_NUMNODES]; /* Used to avoid allocations */ + } wil; }; =20 /* diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h index a8963f7ef4c2..1f9bb10d1a47 100644 --- a/include/uapi/linux/mempolicy.h +++ b/include/uapi/linux/mempolicy.h @@ -23,6 +23,7 @@ enum { MPOL_INTERLEAVE, MPOL_LOCAL, MPOL_PREFERRED_MANY, + MPOL_WEIGHTED_INTERLEAVE, MPOL_MAX, /* always last member of enum */ }; =20 diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 6cdb00acb86b..46e6b6f36a10 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -320,6 +320,7 @@ static struct mempolicy *mpol_new(unsigned short mode, = unsigned short flags, policy->mode =3D mode; policy->flags =3D flags; policy->home_node =3D NUMA_NO_NODE; + policy->wil.cur_weight =3D 0; =20 return policy; } @@ -432,6 +433,10 @@ static const struct mempolicy_operations mpol_ops[MPOL= _MAX] =3D { .create =3D mpol_new_nodemask, .rebind =3D mpol_rebind_preferred, }, + [MPOL_WEIGHTED_INTERLEAVE] =3D { + .create =3D mpol_new_nodemask, + .rebind =3D mpol_rebind_nodemask, + }, }; =20 static bool migrate_folio_add(struct folio *folio, struct list_head *folio= list, @@ -853,7 +858,8 @@ static long do_set_mempolicy(unsigned short mode, unsig= ned short flags, =20 old =3D current->mempolicy; current->mempolicy =3D new; - if (new && new->mode =3D=3D MPOL_INTERLEAVE) + if (new && (new->mode =3D=3D MPOL_INTERLEAVE || + new->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE)) current->il_prev =3D MAX_NUMNODES-1; task_unlock(current); mpol_put(old); @@ -879,6 +885,7 @@ static void get_policy_nodemask(struct mempolicy *pol, = nodemask_t *nodes) case MPOL_INTERLEAVE: case MPOL_PREFERRED: case MPOL_PREFERRED_MANY: + case MPOL_WEIGHTED_INTERLEAVE: *nodes =3D pol->nodes; break; case MPOL_LOCAL: @@ -963,6 +970,13 @@ static long do_get_mempolicy(int *policy, nodemask_t *= nmask, } else if (pol =3D=3D current->mempolicy && pol->mode =3D=3D MPOL_INTERLEAVE) { *policy =3D next_node_in(current->il_prev, pol->nodes); + } else if (pol =3D=3D current->mempolicy && + (pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE)) { + if (pol->wil.cur_weight) + *policy =3D current->il_prev; + else + *policy =3D next_node_in(current->il_prev, + pol->nodes); } else { err =3D -EINVAL; goto out; @@ -1792,7 +1806,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struc= t *vma, pol =3D __get_vma_policy(vma, addr, ilx); if (!pol) pol =3D get_task_policy(current); - if (pol->mode =3D=3D MPOL_INTERLEAVE) { + if (pol->mode =3D=3D MPOL_INTERLEAVE || + pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE) { *ilx +=3D vma->vm_pgoff >> order; *ilx +=3D (addr - vma->vm_start) >> (PAGE_SHIFT + order); } @@ -1842,6 +1857,29 @@ bool apply_policy_zone(struct mempolicy *policy, enu= m zone_type zone) return zone >=3D dynamic_policy_zone; } =20 +static unsigned int weighted_interleave_nodes(struct mempolicy *policy) +{ + unsigned int next; + struct task_struct *me =3D current; + + next =3D next_node_in(me->il_prev, policy->nodes); + if (next =3D=3D MAX_NUMNODES) + return next; + + if (!policy->wil.cur_weight) { + u8 next_weight =3D iw_table[next]; + + if (!next_weight) + next_weight =3D default_iw_table[next]; + policy->wil.cur_weight =3D next_weight; + } + + policy->wil.cur_weight--; + if (!policy->wil.cur_weight) + me->il_prev =3D next; + return next; +} + /* Do dynamic interleaving for a process */ static unsigned int interleave_nodes(struct mempolicy *policy) { @@ -1876,6 +1914,9 @@ unsigned int mempolicy_slab_node(void) case MPOL_INTERLEAVE: return interleave_nodes(policy); =20 + case MPOL_WEIGHTED_INTERLEAVE: + return weighted_interleave_nodes(policy); + case MPOL_BIND: case MPOL_PREFERRED_MANY: { @@ -1914,6 +1955,52 @@ static unsigned int read_once_policy_nodemask(struct= mempolicy *pol, return nodes_weight(*mask); } =20 +/* places active weights in pol->wil.scratch_weights and return total */ +static unsigned int read_once_interleave_weights(struct mempolicy *pol, + nodemask_t *mask) +{ + int nid; + unsigned int weight_total =3D 0; + + /* Similar issue to read_once_policy_nodemask */ + barrier(); + for_each_node_mask(nid, *mask) { + u8 weight =3D iw_table[nid]; + + if (!weight) + weight =3D default_iw_table[nid]; + weight_total +=3D weight; + pol->wil.scratch_weights[nid] =3D weight; + } + barrier(); + return weight_total; +} + +static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t= ilx) +{ + nodemask_t nodemask; + unsigned int target, weight_total; + int nid; + u8 weight; + + read_once_policy_nodemask(pol, &nodemask); + weight_total =3D read_once_interleave_weights(pol, &nodemask); + if (!weight_total) + return numa_node_id(); + + /* Finally, calculate the node offset based on totals */ + target =3D ilx % weight_total; + nid =3D first_node(nodemask); + while (target) { + weight =3D pol->wil.scratch_weights[nid]; + if (target < weight) + break; + target -=3D weight; + nid =3D next_node_in(nid, nodemask); + } + return nid; +} + /* * Do static interleaving for interleave index @ilx. Returns the ilx'th * node in pol->nodes (starting from ilx=3D0), wrapping around if ilx @@ -1974,6 +2061,11 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct= mempolicy *pol, *nid =3D (ilx =3D=3D NO_INTERLEAVE_INDEX) ? interleave_nodes(pol) : interleave_nid(pol, ilx); break; + case MPOL_WEIGHTED_INTERLEAVE: + *nid =3D (ilx =3D=3D NO_INTERLEAVE_INDEX) ? + weighted_interleave_nodes(pol) : + weighted_interleave_nid(pol, ilx); + break; } =20 return nodemask; @@ -2035,6 +2127,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask) case MPOL_PREFERRED_MANY: case MPOL_BIND: case MPOL_INTERLEAVE: + case MPOL_WEIGHTED_INTERLEAVE: *mask =3D mempolicy->nodes; break; =20 @@ -2134,7 +2227,8 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int= order, * If the policy is interleave or does not allow the current * node in its nodemask, we allocate the standard way. */ - if (pol->mode !=3D MPOL_INTERLEAVE && + if ((pol->mode !=3D MPOL_INTERLEAVE && + pol->mode !=3D MPOL_WEIGHTED_INTERLEAVE) && (!nodemask || node_isset(nid, *nodemask))) { /* * First, try to allocate THP only on local node, but @@ -2270,6 +2364,91 @@ static unsigned long alloc_pages_bulk_array_interlea= ve(gfp_t gfp, return total_allocated; } =20 +static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp, + struct mempolicy *pol, unsigned long nr_pages, + struct page **page_array) +{ + struct task_struct *me =3D current; + unsigned long total_allocated =3D 0; + unsigned long nr_allocated; + unsigned long rounds; + unsigned long node_pages, delta; + u8 weight; + unsigned int weight_total =3D 0; + unsigned long rem_pages =3D nr_pages; + nodemask_t nodes; + int nnodes, node; + int prev_node =3D NUMA_NO_NODE; + int i; + + nnodes =3D read_once_policy_nodemask(pol, &nodes); + weight_total =3D read_once_interleave_weights(pol, &nodes); + + /* Continue allocating from most recent node and adjust the nr_pages */ + if (pol->wil.cur_weight) { + node =3D next_node_in(me->il_prev, nodes); + node_pages =3D pol->wil.cur_weight; + if (node_pages > rem_pages) + node_pages =3D rem_pages; + nr_allocated =3D __alloc_pages_bulk(gfp, node, NULL, node_pages, + NULL, page_array); + page_array +=3D nr_allocated; + total_allocated +=3D nr_allocated; + /* if that's all the pages, no need to interleave */ + if (rem_pages <=3D pol->wil.cur_weight) { + pol->wil.cur_weight -=3D rem_pages; + return total_allocated; + } + /* Otherwise we adjust nr_pages down, and continue from there */ + rem_pages -=3D pol->wil.cur_weight; + pol->wil.cur_weight =3D 0; + prev_node =3D node; + } + + /* Now we can continue allocating as if from 0 instead of an offset */ + rounds =3D rem_pages / weight_total; + delta =3D rem_pages % weight_total; + for (i =3D 0; i < nnodes; i++) { + node =3D next_node_in(prev_node, nodes); + weight =3D pol->wil.scratch_weights[node]; + node_pages =3D weight * rounds; + if (delta) { + if (delta > weight) { + node_pages +=3D weight; + delta -=3D weight; + } else { + node_pages +=3D delta; + delta =3D 0; + } + } + nr_allocated =3D __alloc_pages_bulk(gfp, node, NULL, node_pages, + NULL, page_array); + page_array +=3D nr_allocated; + total_allocated +=3D nr_allocated; + if (total_allocated =3D=3D nr_pages) + break; + prev_node =3D node; + } + + /* + * Finally, we need to update me->il_prev and pol->wil.cur_weight + * if there were overflow pages, but not equivalent to the node + * weight, set the cur_weight to node_weight - delta and the + * me->il_prev to the previous node. Otherwise if it was perfect + * we can simply set il_prev to node and cur_weight to 0 + */ + if (node_pages) { + me->il_prev =3D prev_node; + node_pages %=3D weight; + pol->wil.cur_weight =3D weight - node_pages; + } else { + me->il_prev =3D node; + pol->wil.cur_weight =3D 0; + } + + return total_allocated; +} + static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int = nid, struct mempolicy *pol, unsigned long nr_pages, struct page **page_array) @@ -2310,6 +2489,11 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t= gfp, return alloc_pages_bulk_array_interleave(gfp, pol, nr_pages, page_array); =20 + if (pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE) + return alloc_pages_bulk_array_weighted_interleave(gfp, pol, + nr_pages, + page_array); + if (pol->mode =3D=3D MPOL_PREFERRED_MANY) return alloc_pages_bulk_array_preferred_many(gfp, numa_node_id(), pol, nr_pages, page_array); @@ -2385,6 +2569,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempoli= cy *b) case MPOL_INTERLEAVE: case MPOL_PREFERRED: case MPOL_PREFERRED_MANY: + case MPOL_WEIGHTED_INTERLEAVE: return !!nodes_equal(a->nodes, b->nodes); case MPOL_LOCAL: return true; @@ -2521,6 +2706,10 @@ int mpol_misplaced(struct folio *folio, struct vm_ar= ea_struct *vma, polnid =3D interleave_nid(pol, ilx); break; =20 + case MPOL_WEIGHTED_INTERLEAVE: + polnid =3D weighted_interleave_nid(pol, ilx); + break; + case MPOL_PREFERRED: if (node_isset(curnid, pol->nodes)) goto out; @@ -2895,6 +3084,7 @@ static const char * const policy_modes[] =3D [MPOL_PREFERRED] =3D "prefer", [MPOL_BIND] =3D "bind", [MPOL_INTERLEAVE] =3D "interleave", + [MPOL_WEIGHTED_INTERLEAVE] =3D "weighted interleave", [MPOL_LOCAL] =3D "local", [MPOL_PREFERRED_MANY] =3D "prefer (many)", }; @@ -2954,6 +3144,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol) } break; case MPOL_INTERLEAVE: + case MPOL_WEIGHTED_INTERLEAVE: /* * Default to online nodes with memory if no nodelist */ @@ -3064,6 +3255,7 @@ void mpol_to_str(char *buffer, int maxlen, struct mem= policy *pol) case MPOL_PREFERRED_MANY: case MPOL_BIND: case MPOL_INTERLEAVE: + case MPOL_WEIGHTED_INTERLEAVE: nodes =3D pol->nodes; break; default: --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C91322032B; Wed, 3 Jan 2024 22:42:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Pwyi7Tax" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-1d4a2526a7eso22819625ad.3; Wed, 03 Jan 2024 14:42:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321774; x=1704926574; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jOV0W7hivfDnJ4gd5lhXDZU7r1NQAk/nZjV64KhDSvs=; b=Pwyi7TaxhJsSPk3aBmbjdsmWL5GG0JKucPXkgrgfGrNBZ8+WGdUGj6WQ5mAkkzn5OA BUJ7dJoUY0wNAqzGcC8idy5HXGdfZEayk29VsRMOFPsEJdwtx4kPc/gyycVFpqIfrRo5 PMKPTYMyetszNcHMOuqhkB6Qkbp8Kg6L4Y1MXnnrj5lggJU152wASxYDVh/CEAQcO5tm 4CSeJBvfd2Jta7Tt412InbytUBD5QR9CQ0anS9GljvOe8Z2vmzkWqrASerCHFoTCuxay U0mNJiwF3umZYMpkq5KBFGMnfV5y4ZyXqaOoWDD6oOQ0EwrLffFeYCcCpS0Rc4cpGPic hlcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321774; x=1704926574; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jOV0W7hivfDnJ4gd5lhXDZU7r1NQAk/nZjV64KhDSvs=; b=s8n2d3ZSAdonl/RcyqDAo82Iarfl09P5jOSKaRrmydwh6p8dTxLtn6rxFIJ+XPOQaZ S7ns0i4i4MznggpFzz658TuH19EODOFVJQboflMNcQSPDh7CSWkOp90Fy4LBemx5CMOL +Ca7C8tgyFoRTyJhk+j8tOajsGsEeRaZ3XYVFHiwEZwN21ZV4eKarS5jXh7ErjHKJ//j ssw+QEZrqQTxbbQdDahei6Xb4a+ctNtLQiBidIgSXREf/C2fCXgc+jmP4Z9ZfOlcTvYM dbJFtFj9BqVXnxFvnS+s/8oarpbzbkEkXZr2Z/lC0/OIbbt/1QTu3IUf7mXUxxv9BRM5 mzFQ== X-Gm-Message-State: AOJu0YzvfQ2NKIDUTaLDRPpT2b7xRsxslvNl4SzGvrc+HAS14fDAX/3a DBzXxoVMBqs6t81QKzL6uA== X-Google-Smtp-Source: AGHT+IFh+qcxXaiogx+miqhx0SFpy/RI3hbA28LBl0CE6Pvd/A72N4h+GBcyXApCM4Y4r2iGXqWlmQ== X-Received: by 2002:a17:903:2450:b0:1d4:ab6b:f102 with SMTP id l16-20020a170903245000b001d4ab6bf102mr3418120pls.85.1704321773969; Wed, 03 Jan 2024 14:42:53 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.42.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:42:53 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v6 04/12] mm/mempolicy: refactor sanitize_mpol_flags for reuse Date: Wed, 3 Jan 2024 17:42:01 -0500 Message-Id: <20240103224209.2541-5-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" split sanitize_mpol_flags into sanitize and validate. Sanitize is used by set_mempolicy to split (int mode) into mode and mode_flags, and then validates them. Validate validates already split flags. Validate will be reused for new syscalls that accept already split mode and mode_flags. Signed-off-by: Gregory Price Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- mm/mempolicy.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 46e6b6f36a10..6e2ea94c0f31 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1478,24 +1478,35 @@ static int copy_nodes_to_user(unsigned long __user = *mask, unsigned long maxnode, return copy_to_user(mask, nodes_addr(*nodes), copy) ? -EFAULT : 0; } =20 -/* Basic parameter sanity check used by both mbind() and set_mempolicy() */ -static inline int sanitize_mpol_flags(int *mode, unsigned short *flags) +/* + * Basic parameter sanity check used by mbind/set_mempolicy + * May modify flags to include internal flags (e.g. MPOL_F_MOF/F_MORON) + */ +static inline int validate_mpol_flags(unsigned short mode, unsigned short = *flags) { - *flags =3D *mode & MPOL_MODE_FLAGS; - *mode &=3D ~MPOL_MODE_FLAGS; - - if ((unsigned int)(*mode) >=3D MPOL_MAX) + if ((unsigned int)(mode) >=3D MPOL_MAX) return -EINVAL; if ((*flags & MPOL_F_STATIC_NODES) && (*flags & MPOL_F_RELATIVE_NODES)) return -EINVAL; if (*flags & MPOL_F_NUMA_BALANCING) { - if (*mode !=3D MPOL_BIND) + if (mode !=3D MPOL_BIND) return -EINVAL; *flags |=3D (MPOL_F_MOF | MPOL_F_MORON); } return 0; } =20 +/* + * Used by mbind/set_memplicy to split and validate mode/flags + */ +static inline int sanitize_mpol_flags(int *mode, unsigned short *flags) +{ + *flags =3D *mode & MPOL_MODE_FLAGS; + *mode =3D *mode & ~MPOL_MODE_FLAGS; + + return validate_mpol_flags(*mode, flags); +} + static long kernel_mbind(unsigned long start, unsigned long len, unsigned long mode, const unsigned long __user *nmask, unsigned long maxnode, unsigned int flags) --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3ECCD208BB; Wed, 3 Jan 2024 22:42:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YxKtrbMj" Received: by mail-pg1-f195.google.com with SMTP id 41be03b00d2f7-5ceb3fe708eso625649a12.3; Wed, 03 Jan 2024 14:42:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321778; x=1704926578; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HZKG0+8bOrcMVz8DZz+ELjEjEoDwyx2ZdYAuvWnw0cE=; b=YxKtrbMjHA7VzQkGftdxZwS0PNvgsHCUJvsRoywI7pUztQIMBzltEEhefuN4tSbGMc WOW/S7S1fUuYTSjxMjFvZ987FJCLpc7z/uv+RbEtYaAIq4v8ALY9juYnsw7FIEBLe863 XMljYXRR90ri/J/x+I9LLD+JfQ9tLTqE9/5gy7MFrB77sRSx5DfFp7n8lQVtBSby8l4p L64VK0d3Ig/g6EGKxPPXGYa7GK9SPsWWWnsmggcy2djTAGjNRPv7XOWnk9PlmkIguidI Ko9bvXB39E1D5vPZVn9BvJxLqM/EvUO/6S1N9oI7xCW2zl/5gkRwCGz4sRo3D3IImZHf +RMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321778; x=1704926578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HZKG0+8bOrcMVz8DZz+ELjEjEoDwyx2ZdYAuvWnw0cE=; b=RIQj24kEiGfimS1IVst8vGw40KpySX4a7Fnc+j4TBw4dmosnu0EbVJF7QIuzX8nKrf NGQcPnn5gsnmd7yc1CVZznPK/+0TuFE7c66kp0a0elD28cjr7hyjwzuWierAy6KOMbGf PJiTqfQDcLDe2Q1PQwYbzH70QiT/dXm6Z6M6rf8MLzFTydhGXlDwwXBB8gN1EA/Eq8jT rhJxmziPxP8V1Hn8NGbSejn2xxd63K2rKF+hxHoM/mMFMDXF5QUMIJ1xUmeBn9/htqOE DCRZGnCmqAQiXP4jjffrgE/F3BfYOCF4l18ui/FKICGQ+04PxXOWcSY0RIuBdk+o/2gw fCPg== X-Gm-Message-State: AOJu0YzEvsvZ0c3xabq9I1L2oZFIVG+Q4hAbWArg17KQxV1qoIdlRypE c2fq9NbKkQlb7Xk427dTmw== X-Google-Smtp-Source: AGHT+IEjgc4CjelPkG95hlPm6CS+AolxPwp3LIumI7aUMfOsB138oQK6JNgNBpHDs3udC29zWbYE5w== X-Received: by 2002:a17:90a:e98c:b0:28b:c1ad:9cf5 with SMTP id v12-20020a17090ae98c00b0028bc1ad9cf5mr7508460pjy.26.1704321778498; Wed, 03 Jan 2024 14:42:58 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.42.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:42:58 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v6 05/12] mm/mempolicy: create struct mempolicy_param for creating new mempolicies Date: Wed, 3 Jan 2024 17:42:02 -0500 Message-Id: <20240103224209.2541-6-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch adds a new kernel structure `struct mempolicy_param`, intended to be used for an extensible get/set_mempolicy interface. This implements the fields required to support the existing syscall interfaces interfaces, but does not expose any user-facing arg structure. mpol_new is refactored to take the argument structure so that future mempolicy extensions can all be managed in the mempolicy constructor. The get_mempolicy and mbind syscalls are refactored to utilize the new argument structure, as are all the callers of mpol_new() and do_set_mempolicy. Signed-off-by: Gregory Price Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- include/linux/mempolicy.h | 11 +++++++ mm/mempolicy.c | 69 +++++++++++++++++++++++++++++---------- 2 files changed, 62 insertions(+), 18 deletions(-) diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index fae903b1d3de..e6795e2d0cc2 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -62,6 +62,17 @@ struct mempolicy { } wil; }; =20 +/* + * Describes settings of a mempolicy during set/get syscalls and + * kernel internal calls to do_set_mempolicy() + */ +struct mempolicy_param { + unsigned short mode; /* policy mode */ + unsigned short mode_flags; /* policy mode flags */ + int home_node; /* mbind: use MPOL_MF_HOME_NODE */ + nodemask_t *policy_nodes; /* get/set/mbind */ +}; + /* * Support for managing mempolicy data objects (clone, copy, destroy) * The default fast path of a NULL MPOL_DEFAULT policy is always inlined. diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 6e2ea94c0f31..1f6f19b5d157 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -280,10 +280,12 @@ static int mpol_set_nodemask(struct mempolicy *pol, * This function just creates a new policy, does some check and simple * initialization. You must invoke mpol_set_nodemask() to set nodes. */ -static struct mempolicy *mpol_new(unsigned short mode, unsigned short flag= s, - nodemask_t *nodes) +static struct mempolicy *mpol_new(struct mempolicy_param *param) { struct mempolicy *policy; + unsigned short mode =3D param->mode; + unsigned short flags =3D param->mode_flags; + nodemask_t *nodes =3D param->policy_nodes; =20 if (mode =3D=3D MPOL_DEFAULT) { if (nodes && !nodes_empty(*nodes)) @@ -832,8 +834,7 @@ static int mbind_range(struct vma_iterator *vmi, struct= vm_area_struct *vma, } =20 /* Set the process memory policy */ -static long do_set_mempolicy(unsigned short mode, unsigned short flags, - nodemask_t *nodes) +static long do_set_mempolicy(struct mempolicy_param *param) { struct mempolicy *new, *old; NODEMASK_SCRATCH(scratch); @@ -842,14 +843,14 @@ static long do_set_mempolicy(unsigned short mode, uns= igned short flags, if (!scratch) return -ENOMEM; =20 - new =3D mpol_new(mode, flags, nodes); + new =3D mpol_new(param); if (IS_ERR(new)) { ret =3D PTR_ERR(new); goto out; } =20 task_lock(current); - ret =3D mpol_set_nodemask(new, nodes, scratch); + ret =3D mpol_set_nodemask(new, param->policy_nodes, scratch); if (ret) { task_unlock(current); mpol_put(new); @@ -1247,8 +1248,7 @@ static struct folio *alloc_migration_target_by_mpol(s= truct folio *src, #endif =20 static long do_mbind(unsigned long start, unsigned long len, - unsigned short mode, unsigned short mode_flags, - nodemask_t *nmask, unsigned long flags) + struct mempolicy_param *mparam, unsigned long flags) { struct mm_struct *mm =3D current->mm; struct vm_area_struct *vma, *prev; @@ -1268,7 +1268,7 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, if (start & ~PAGE_MASK) return -EINVAL; =20 - if (mode =3D=3D MPOL_DEFAULT) + if (mparam->mode =3D=3D MPOL_DEFAULT) flags &=3D ~MPOL_MF_STRICT; =20 len =3D PAGE_ALIGN(len); @@ -1279,7 +1279,7 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, if (end =3D=3D start) return 0; =20 - new =3D mpol_new(mode, mode_flags, nmask); + new =3D mpol_new(mparam); if (IS_ERR(new)) return PTR_ERR(new); =20 @@ -1296,7 +1296,8 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, NODEMASK_SCRATCH(scratch); if (scratch) { mmap_write_lock(mm); - err =3D mpol_set_nodemask(new, nmask, scratch); + err =3D mpol_set_nodemask(new, mparam->policy_nodes, + scratch); if (err) mmap_write_unlock(mm); } else @@ -1310,7 +1311,7 @@ static long do_mbind(unsigned long start, unsigned lo= ng len, * Lock the VMAs before scanning for pages to migrate, * to ensure we don't miss a concurrently inserted page. */ - nr_failed =3D queue_pages_range(mm, start, end, nmask, + nr_failed =3D queue_pages_range(mm, start, end, mparam->policy_nodes, flags | MPOL_MF_INVERT | MPOL_MF_WRLOCK, &pagelist); =20 if (nr_failed < 0) { @@ -1511,6 +1512,7 @@ static long kernel_mbind(unsigned long start, unsigne= d long len, unsigned long mode, const unsigned long __user *nmask, unsigned long maxnode, unsigned int flags) { + struct mempolicy_param mparam; unsigned short mode_flags; nodemask_t nodes; int lmode =3D mode; @@ -1525,7 +1527,12 @@ static long kernel_mbind(unsigned long start, unsign= ed long len, if (err) return err; =20 - return do_mbind(start, len, lmode, mode_flags, &nodes, flags); + memset(&mparam, 0, sizeof(mparam)); + mparam.mode =3D lmode; + mparam.mode_flags =3D mode_flags; + mparam.policy_nodes =3D &nodes; + + return do_mbind(start, len, &mparam, flags); } =20 SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned lo= ng, len, @@ -1606,6 +1613,7 @@ SYSCALL_DEFINE6(mbind, unsigned long, start, unsigned= long, len, static long kernel_set_mempolicy(int mode, const unsigned long __user *nma= sk, unsigned long maxnode) { + struct mempolicy_param param; unsigned short mode_flags; nodemask_t nodes; int lmode =3D mode; @@ -1619,7 +1627,12 @@ static long kernel_set_mempolicy(int mode, const uns= igned long __user *nmask, if (err) return err; =20 - return do_set_mempolicy(lmode, mode_flags, &nodes); + memset(¶m, 0, sizeof(param)); + param.mode =3D lmode; + param.mode_flags =3D mode_flags; + param.policy_nodes =3D &nodes; + + return do_set_mempolicy(¶m); } =20 SYSCALL_DEFINE3(set_mempolicy, int, mode, const unsigned long __user *, nm= ask, @@ -2908,6 +2921,7 @@ static int shared_policy_replace(struct shared_policy= *sp, pgoff_t start, void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *m= pol) { int ret; + struct mempolicy_param mparam; =20 sp->root =3D RB_ROOT; /* empty tree =3D=3D default mempolicy */ rwlock_init(&sp->lock); @@ -2920,8 +2934,12 @@ void mpol_shared_policy_init(struct shared_policy *s= p, struct mempolicy *mpol) if (!scratch) goto put_mpol; =20 + memset(&mparam, 0, sizeof(mparam)); + mparam.mode =3D mpol->mode; + mparam.mode_flags =3D mpol->flags; + mparam.policy_nodes =3D &mpol->w.user_nodemask; /* contextualize the tmpfs mount point mempolicy to this file */ - npol =3D mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask); + npol =3D mpol_new(&mparam); if (IS_ERR(npol)) goto free_scratch; /* no valid nodemask intersection */ =20 @@ -3029,6 +3047,7 @@ static inline void __init check_numabalancing_enable(= void) =20 void __init numa_policy_init(void) { + struct mempolicy_param param; nodemask_t interleave_nodes; unsigned long largest =3D 0; int nid, prefer =3D 0; @@ -3074,7 +3093,11 @@ void __init numa_policy_init(void) if (unlikely(nodes_empty(interleave_nodes))) node_set(prefer, interleave_nodes); =20 - if (do_set_mempolicy(MPOL_INTERLEAVE, 0, &interleave_nodes)) + memset(¶m, 0, sizeof(param)); + param.mode =3D MPOL_INTERLEAVE; + param.policy_nodes =3D &interleave_nodes; + + if (do_set_mempolicy(¶m)) pr_err("%s: interleaving failed\n", __func__); =20 check_numabalancing_enable(); @@ -3083,7 +3106,12 @@ void __init numa_policy_init(void) /* Reset policy of current process to default */ void numa_default_policy(void) { - do_set_mempolicy(MPOL_DEFAULT, 0, NULL); + struct mempolicy_param param; + + memset(¶m, 0, sizeof(param)); + param.mode =3D MPOL_DEFAULT; + + do_set_mempolicy(¶m); } =20 /* @@ -3113,6 +3141,7 @@ static const char * const policy_modes[] =3D */ int mpol_parse_str(char *str, struct mempolicy **mpol) { + struct mempolicy_param mparam; struct mempolicy *new =3D NULL; unsigned short mode_flags; nodemask_t nodes; @@ -3199,7 +3228,11 @@ int mpol_parse_str(char *str, struct mempolicy **mpo= l) goto out; } =20 - new =3D mpol_new(mode, mode_flags, &nodes); + memset(&mparam, 0, sizeof(mparam)); + mparam.mode =3D mode; + mparam.mode_flags =3D mode_flags; + mparam.policy_nodes =3D &nodes; + new =3D mpol_new(&mparam); if (IS_ERR(new)) goto out; =20 --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93F3E208DE; Wed, 3 Jan 2024 22:43:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CHsonS2n" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-1d4d5b37670so5266915ad.0; Wed, 03 Jan 2024 14:43:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321783; x=1704926583; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6l+ciXKl7o03KQrjkXlHTiaF400Iug/LgpVj8Iwr0q8=; b=CHsonS2nbHdGigUCmmUjVFescCh1eOelX4rzqGu3GEI1/0ofW2hk2rAdWrgJbsUhtI UQNDIAIcLQl0fXUcND2GbMrl2muBIzDSpr++k+zUetr6YdghaLEVtUz9RYp5+nJAgOwp hV4GNSdbFwOcquGvOETDeoyYcB2aYFgoQe/ongmCZ7+n5iQLrGcdPSjHhNg+oCAYgf4F g4LputwWihK3nSu4UaGsKglk3ii/smp/Tjd7CVV0eRnni9CXvz/DIWt8wkRRdeunIIfs l3oSMjT4HwBa0p6cfPIz1OZq2mUpeBKRMnd6xGV6sHDfE5dt/XIbwA8nVOLXuASmXZLT DGaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321783; x=1704926583; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6l+ciXKl7o03KQrjkXlHTiaF400Iug/LgpVj8Iwr0q8=; b=Z6HFFocBFIFpGmMZY2/WWasaTw2DfMZRYAz3ai1MjByDStBBHCqeqzw1d/4w/+iok0 TTtGmpMtqGA4q7sgWs4hhKUiIsnw8m2sBqhKEO83a0k6ygxffwN4Y++yIZtPzjSUXjAD FLkk1X0xz4tkIN6odrdOu2fqAi4XJT1TUGkNUWUOClVgwucxmvQ0DzTr+08EiFyESs5B fWtHiJnzBqxy5AhkkIxf94mzOUJqzElJWUF/g/EaZPSaP4FNRfKxJ9XcA3C3nxakEZP9 D76Hq4GVEz3X93YuS9tcz6Z9YOrB/O848kbTchuxnmhDhbC00Ek9sRLSAzsb27C6ybqM 3TnQ== X-Gm-Message-State: AOJu0YwGS22qJd+CSS8fW9vq54k3XG2s69kgLGJxgXT5qTWiv87rX/VB CFKb5czTwk9m+Mefhg+Hb8HMcZTJ7SKqQjk= X-Google-Smtp-Source: AGHT+IHkz4DcjxxMztCbR+oS+wc++YJJKxyTPqHeIcSKYmPs8augJTuDCINITO+c5RZXPtTATceGWg== X-Received: by 2002:a17:902:b696:b0:1d4:786a:20d4 with SMTP id c22-20020a170902b69600b001d4786a20d4mr5871563pls.78.1704321782858; Wed, 03 Jan 2024 14:43:02 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.42.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:43:02 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v6 06/12] mm/mempolicy: refactor kernel_get_mempolicy for code re-use Date: Wed, 3 Jan 2024 17:42:03 -0500 Message-Id: <20240103224209.2541-7-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pull operation flag checking from inside do_get_mempolicy out to kernel_get_mempolicy. This allows us to flatten the internal code, and break it into separate functions for future syscalls (get_mempolicy2, process_get_mempolicy) to re-use the code, even after additional extensions are made. The primary change is that the flag is treated as the multiplexer that it actually is. For get_mempolicy, the flags represents 3 different primary operations: if (flags & MPOL_F_MEMS_ALLOWED) return task->mems_allowed else if (flags & MPOL_F_ADDR) return vma mempolicy information else return task mempolicy information Plus the behavior modifying flag: if (flags & MPOL_F_NODE) change the return value of (int __user *policy) based on whether MPOL_F_ADDR was set. The original behavior of get_mempolicy is retained, but we utilize the new mempolicy_param structure to pass the operations down the stack. This will allow us to extend the internal functions without affecting the legacy behavior of get_mempolicy. Signed-off-by: Gregory Price Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- mm/mempolicy.c | 244 +++++++++++++++++++++++++++++++------------------ 1 file changed, 154 insertions(+), 90 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 1f6f19b5d157..db290cf540d7 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -910,106 +910,109 @@ static int lookup_node(struct mm_struct *mm, unsign= ed long addr) return ret; } =20 -/* Retrieve NUMA policy */ -static long do_get_mempolicy(int *policy, nodemask_t *nmask, - unsigned long addr, unsigned long flags) +/* Retrieve the mems_allowed for current task */ +static inline long do_get_mems_allowed(nodemask_t *nmask) { - int err; - struct mm_struct *mm =3D current->mm; - struct vm_area_struct *vma =3D NULL; - struct mempolicy *pol =3D current->mempolicy, *pol_refcount =3D NULL; + task_lock(current); + *nmask =3D cpuset_current_mems_allowed; + task_unlock(current); + return 0; +} =20 - if (flags & - ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED)) - return -EINVAL; +/* If the policy has additional node information to retrieve, return it */ +static long do_get_policy_node(struct mempolicy *pol) +{ + /* + * For MPOL_INTERLEAVE, the extended node information is the next + * node that will be selected for interleave. For weighted interleave + * we return the next node based on the current weight. + */ + if (pol =3D=3D current->mempolicy && pol->mode =3D=3D MPOL_INTERLEAVE) + return next_node_in(current->il_prev, pol->nodes); =20 - if (flags & MPOL_F_MEMS_ALLOWED) { - if (flags & (MPOL_F_NODE|MPOL_F_ADDR)) - return -EINVAL; - *policy =3D 0; /* just so it's initialized */ + if (pol =3D=3D current->mempolicy && + pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE) { + if (pol->wil.cur_weight) + return current->il_prev; + else + return next_node_in(current->il_prev, pol->nodes); + } + return -EINVAL; +} + +/* Handle user_nodemask condition when fetching nodemask for userspace */ +static void do_get_mempolicy_nodemask(struct mempolicy *pol, nodemask_t *n= mask) +{ + if (mpol_store_user_nodemask(pol)) { + *nmask =3D pol->w.user_nodemask; + } else { task_lock(current); - *nmask =3D cpuset_current_mems_allowed; + get_policy_nodemask(pol, nmask); task_unlock(current); - return 0; } +} =20 - if (flags & MPOL_F_ADDR) { - pgoff_t ilx; /* ignored here */ - /* - * Do NOT fall back to task policy if the - * vma/shared policy at addr is NULL. We - * want to return MPOL_DEFAULT in this case. - */ - mmap_read_lock(mm); - vma =3D vma_lookup(mm, addr); - if (!vma) { - mmap_read_unlock(mm); - return -EFAULT; - } - pol =3D __get_vma_policy(vma, addr, &ilx); - } else if (addr) - return -EINVAL; +/* Retrieve NUMA policy for a VMA assocated with a given address */ +static long do_get_vma_mempolicy(unsigned long addr, int *addr_node, + struct mempolicy_param *param) +{ + pgoff_t ilx; + struct mm_struct *mm =3D current->mm; + struct vm_area_struct *vma =3D NULL; + struct mempolicy *pol =3D NULL; =20 + mmap_read_lock(mm); + vma =3D vma_lookup(mm, addr); + if (!vma) { + mmap_read_unlock(mm); + return -EFAULT; + } + pol =3D __get_vma_policy(vma, addr, &ilx); if (!pol) - pol =3D &default_policy; /* indicates default behavior */ + pol =3D &default_policy; + else + mpol_get(pol); + mmap_read_unlock(mm); =20 - if (flags & MPOL_F_NODE) { - if (flags & MPOL_F_ADDR) { - /* - * Take a refcount on the mpol, because we are about to - * drop the mmap_lock, after which only "pol" remains - * valid, "vma" is stale. - */ - pol_refcount =3D pol; - vma =3D NULL; - mpol_get(pol); - mmap_read_unlock(mm); - err =3D lookup_node(mm, addr); - if (err < 0) - goto out; - *policy =3D err; - } else if (pol =3D=3D current->mempolicy && - pol->mode =3D=3D MPOL_INTERLEAVE) { - *policy =3D next_node_in(current->il_prev, pol->nodes); - } else if (pol =3D=3D current->mempolicy && - (pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE)) { - if (pol->wil.cur_weight) - *policy =3D current->il_prev; - else - *policy =3D next_node_in(current->il_prev, - pol->nodes); - } else { - err =3D -EINVAL; - goto out; - } - } else { - *policy =3D pol =3D=3D &default_policy ? MPOL_DEFAULT : - pol->mode; - /* - * Internal mempolicy flags must be masked off before exposing - * the policy to userspace. - */ - *policy |=3D (pol->flags & MPOL_MODE_FLAGS); - } + /* Fetch the node for the given address */ + if (addr_node) + *addr_node =3D lookup_node(mm, addr); =20 - err =3D 0; - if (nmask) { - if (mpol_store_user_nodemask(pol)) { - *nmask =3D pol->w.user_nodemask; - } else { - task_lock(current); - get_policy_nodemask(pol, nmask); - task_unlock(current); - } + param->mode =3D pol =3D=3D &default_policy ? MPOL_DEFAULT : pol->mode; + param->mode_flags =3D (pol->flags & MPOL_MODE_FLAGS); + param->home_node =3D pol->home_node; + + if (param->policy_nodes) + do_get_mempolicy_nodemask(pol, param->policy_nodes); + + if (pol !=3D &default_policy) { + mpol_put(pol); + mpol_cond_put(pol); } =20 - out: - mpol_cond_put(pol); - if (vma) - mmap_read_unlock(mm); - if (pol_refcount) - mpol_put(pol_refcount); - return err; + return 0; +} + +/* Retrieve NUMA policy for the current task */ +static long do_get_task_mempolicy(struct mempolicy_param *param, int *pol_= node) +{ + struct mempolicy *pol =3D current->mempolicy; + + if (!pol) + pol =3D &default_policy; /* indicates default behavior */ + + param->mode =3D pol =3D=3D &default_policy ? MPOL_DEFAULT : pol->mode; + /* Internal flags must be masked off before exposing to userspace */ + param->mode_flags =3D (pol->flags & MPOL_MODE_FLAGS); + param->home_node =3D NUMA_NO_NODE; + + if (pol_node) + *pol_node =3D do_get_policy_node(pol); + + if (param->policy_nodes) + do_get_mempolicy_nodemask(pol, param->policy_nodes); + + return 0; } =20 #ifdef CONFIG_MIGRATION @@ -1742,16 +1745,77 @@ static int kernel_get_mempolicy(int __user *policy, unsigned long addr, unsigned long flags) { + struct mempolicy_param param; int err; - int pval; + int address_node =3D NUMA_NO_NODE; + int pval =3D 0; + int pol_node =3D 0; nodemask_t nodes; =20 if (nmask !=3D NULL && maxnode < nr_node_ids) return -EINVAL; =20 - addr =3D untagged_addr(addr); + if (flags & + ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED)) + return -EINVAL; =20 - err =3D do_get_mempolicy(&pval, &nodes, addr, flags); + /* Ensure any data that may be copied to userland is initialized */ + memset(¶m, 0, sizeof(param)); + param.policy_nodes =3D &nodes; + + /* + * set_mempolicy was originally multiplexed based on 3 flags: + * MPOL_F_MEMS_ALLOWED: fetch task->mems_allowed + * MPOL_F_ADDR : operate on vma->mempolicy + * MPOL_F_NODE : change return value of *policy + * + * Split this behavior out here, rather than internal functions, + * so that the internal functions can be re-used by future + * get_mempolicy2 interfaces and the arg structure made extensible + */ + if (flags & MPOL_F_MEMS_ALLOWED) { + if (flags & (MPOL_F_NODE|MPOL_F_ADDR)) + return -EINVAL; + pval =3D 0; /* just so it's initialized */ + err =3D do_get_mems_allowed(&nodes); + } else if (flags & MPOL_F_ADDR) { + /* If F_ADDR, we operation on a vma policy (or default) */ + err =3D do_get_vma_mempolicy(untagged_addr(addr), + &address_node, ¶m); + if (err) + return err; + /* if (F_ADDR | F_NODE), *pval is the address' node */ + if (flags & MPOL_F_NODE) { + /* if we failed to fetch, that's likely an EFAULT */ + if (address_node < 0) + return address_node; + pval =3D address_node; + } else + pval =3D param.mode | param.mode_flags; + } else { + /* if not F_ADDR and addr !=3D null, EINVAL */ + if (addr) + return -EINVAL; + + err =3D do_get_task_mempolicy(¶m, &pol_node); + if (err) + return err; + /* + * if F_NODE was set and mode was MPOL_INTERLEAVE + * *pval is equal to next interleave node. + * + * if pol_node < 0, this means the mode did not have a + * a compatible policy. This presently emulates the + * original behavior of (F_NODE) & (!MPOL_INTERLEAVE) + * producing -EINVAL + */ + if (flags & MPOL_F_NODE) { + if (pol_node < 0) + return pol_node; + pval =3D pol_node; + } else + pval =3D param.mode | param.mode_flags; + } =20 if (err) return err; --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAB4C20B1E; Wed, 3 Jan 2024 22:43:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TXfB9/au" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-1d43df785c2so29626565ad.1; Wed, 03 Jan 2024 14:43:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321787; x=1704926587; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p3MMRetX85ZbA15jPANVawmPvaaHqiRw+Z96q91FHVo=; b=TXfB9/auQvbjsB07xSHn3entFn3oHz1t5zaJ3yw4pQ3jHX4xAc9FzP7fd10F8EL9Wf jsgIt5/MBihYUDVG4aVJqZBgaHyejw8NuAF2Vr4EG2e9U+tUQVlADkxp7caKmF2ukLDO CVNiXcXQKXf0SSmE9okgFNw34/z+Kt8Ngrl+ogZQ+viiXl2p7JlsCxAwap2h3rwqGHdk PT2VvrWu+hIsoLSAkI/nLvbUxpJwvpMl1QG8E4C4Zwc4oNKPP9R165+39u5PsVRtxndN IytgRVeyNjGm+D70I1aQcbwL7CmkkpqVorXg16QUQ6C62sz4aY3lXM7QBHQzMVX9mxxB pjTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321787; x=1704926587; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p3MMRetX85ZbA15jPANVawmPvaaHqiRw+Z96q91FHVo=; b=Y3WO7yA51kSGLXVZ1QGXFS1NQy0E8taqDfko3/Ond+NKjIt5G7J08eizfToO/VX6W3 dX5daAU6LJ6723jL8xqQqMTKpSce+Fj194UPMlNRzBx8/MT6WuxMOqExXahWHQHVqj69 /fE6F/8pqYV9KVmHd4y5cJ4LIF+IZ53evomVgSj6YsMZiwMgl+fED73FPP5szHb7sfwY E4KgEqjHZ20hrx/0vEDR4c+If0zLFxCr4jZ0BKHi3236CLGRMVowh2AiWkZWMOtLsreU rhUXz7zs1aC8+c38oDbmKDlj90YjDuCKcDxs/VsvMKkPCD68UMI/ZarO6wEVeLd7V6/e DQfA== X-Gm-Message-State: AOJu0YyUsj1+0lzdi1MCnE57SyqFV4/frPecUzaMQxpZ200vYa7b4fbn 5bRtFogORt2fZ945421SOQ== X-Google-Smtp-Source: AGHT+IEk+VNPDcr69ZjxoMb3QEMmNUTZNO2GzmWL5L5QjmMOV28LtEm1X3yzYpnuphwNA+gyM8/kwQ== X-Received: by 2002:a17:902:db04:b0:1d4:ac2b:7d9 with SMTP id m4-20020a170902db0400b001d4ac2b07d9mr3356598plx.129.1704321787158; Wed, 03 Jan 2024 14:43:07 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.43.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:43:06 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v6 07/12] mm/mempolicy: allow home_node to be set by mpol_new Date: Wed, 3 Jan 2024 17:42:04 -0500 Message-Id: <20240103224209.2541-8-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch adds the plumbing into mpol_new() to allow the argument structure's home_node field to be set during mempolicy creation. The syscall sys_set_mempolicy_home_node was added to allow a home node to be registered for a vma. For set_mempolicy2 and mbind2 syscalls, it would be useful to add this as an extension to allow the user to submit a fully formed mempolicy configuration in a single call, rather than require multiple calls to configure a mempolicy. This will become particularly useful if/when pidfd interfaces to change process mempolicies from outside the task appear, as each call to change the mempolicy does an atomic swap of that policy in the task, rather than mutate the policy. Signed-off-by: Gregory Price Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- mm/mempolicy.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index db290cf540d7..4a566341cf43 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -321,7 +321,7 @@ static struct mempolicy *mpol_new(struct mempolicy_para= m *param) atomic_set(&policy->refcnt, 1); policy->mode =3D mode; policy->flags =3D flags; - policy->home_node =3D NUMA_NO_NODE; + policy->home_node =3D param->home_node; policy->wil.cur_weight =3D 0; =20 return policy; @@ -1634,6 +1634,7 @@ static long kernel_set_mempolicy(int mode, const unsi= gned long __user *nmask, param.mode =3D lmode; param.mode_flags =3D mode_flags; param.policy_nodes =3D &nodes; + param.home_node =3D NUMA_NO_NODE; =20 return do_set_mempolicy(¶m); } @@ -3002,6 +3003,8 @@ void mpol_shared_policy_init(struct shared_policy *sp= , struct mempolicy *mpol) mparam.mode =3D mpol->mode; mparam.mode_flags =3D mpol->flags; mparam.policy_nodes =3D &mpol->w.user_nodemask; + mparam.home_node =3D NUMA_NO_NODE; + /* contextualize the tmpfs mount point mempolicy to this file */ npol =3D mpol_new(&mparam); if (IS_ERR(npol)) @@ -3160,6 +3163,7 @@ void __init numa_policy_init(void) memset(¶m, 0, sizeof(param)); param.mode =3D MPOL_INTERLEAVE; param.policy_nodes =3D &interleave_nodes; + param.home_node =3D NUMA_NO_NODE; =20 if (do_set_mempolicy(¶m)) pr_err("%s: interleaving failed\n", __func__); @@ -3174,6 +3178,7 @@ void numa_default_policy(void) =20 memset(¶m, 0, sizeof(param)); param.mode =3D MPOL_DEFAULT; + param.home_node =3D NUMA_NO_NODE; =20 do_set_mempolicy(¶m); } @@ -3296,6 +3301,8 @@ int mpol_parse_str(char *str, struct mempolicy **mpol) mparam.mode =3D mode; mparam.mode_flags =3D mode_flags; mparam.policy_nodes =3D &nodes; + mparam.home_node =3D NUMA_NO_NODE; + new =3D mpol_new(&mparam); if (IS_ERR(new)) goto out; --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D349820DCE; Wed, 3 Jan 2024 22:43:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eyg52bOi" Received: by mail-pl1-f193.google.com with SMTP id d9443c01a7336-1d3e6c86868so84051535ad.1; Wed, 03 Jan 2024 14:43:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321792; x=1704926592; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RUHKocyjko4JE9NI2Z1eJ5WM6U4OAc9XzLqGoGJ7CFY=; b=eyg52bOiw4mTVvjV72SkXROesTBkAlvz+KsCpor4lvz4B6cIxpbt13cSjHEpAhuiRU Smjkl+hYd607nyhRQrrqUXSGJE7Nfy8wErEixjnxxaC1ppOvQDjsUPghV6yb+PJE/CLy +VFXh8fH8PNsnXen4kQcgzmL5cbMcO4nBf7oahu808lzbFitojaS9RUv/BepwWhtmZyD pU3aiJplJh7YH5gBGoLCtMhKbGq8Kty9DtNhxM3VjLTlR3fMOaRwJbIjRMoB7NxDHunM vyVHtJfDy+rVYwAFbq2Z6Flo0jiRX3SjAE9XYBcgiMAm6t8pL8OffYAkuKPVbJCgSdjQ Xyhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321792; x=1704926592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RUHKocyjko4JE9NI2Z1eJ5WM6U4OAc9XzLqGoGJ7CFY=; b=sS2bGPmjb7+FT6wLmk8yYuzu5ozYcC+xV445wvy3t2kOfHYi3KYyolaixhb4iBFJ3M 4I/46xZBOd4d9u2nm8k+6p/YnFFZGnKCYNYQLSnTean6GPqYOhhAM9QwVSp3Fe7jb82P nSWuSzfiCJcKtxzmQ3uhEPZQHXin1yXokU/BI3mVunUHijNqCUezf+SlePPBWKj294cf q9F2PbJ9QtktK+vlY951UmnMkO3pAE3zwt6ey4x2cfqOt5hFp7XWGI7ubUnHWipAFWcu 1cUANRxSf40wfrqOJnYSvxk0/eVlDX14/PBKNPR8UpnPVv2qujbSaeXazY2Mr+m1D7QK pQtA== X-Gm-Message-State: AOJu0Yx6c5vDTre6FXcbAjeLOgTeWxXfJqHP35vlyjcRM/an8wg9wuou tOHp4G3qzp7nbmDG+V9tsA== X-Google-Smtp-Source: AGHT+IE6dzpobsqtQII2rv9CF4X53MepsPxqTJ4TnvU9EOom4dXLh7CLzaRiTQ2kttaU5pcRlRYB3A== X-Received: by 2002:a17:902:d503:b0:1d4:576e:91cc with SMTP id b3-20020a170902d50300b001d4576e91ccmr19465094plg.7.1704321792227; Wed, 03 Jan 2024 14:43:12 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:43:11 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com, Frank van der Linden Subject: [PATCH v6 08/12] mm/mempolicy: add userland mempolicy arg structure Date: Wed, 3 Jan 2024 17:42:05 -0500 Message-Id: <20240103224209.2541-9-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch adds the new user-api argument structure intended for set_mempolicy2 and mbind2. struct mpol_param { __u16 mode; __u16 mode_flags; __s32 home_node; /* mbind2: policy home node */ __u16 pol_maxnodes; __u8 resv[6]; __aligned_u64 *pol_nodes; }; This structure is intended to be extensible as new mempolicy extensions are added. For example, set_mempolicy_home_node was added to allow vma mempolicies to have a preferred/home node assigned. This structure allows the user to set the home node at the time mempolicy is created, rather than requiring an additional syscalls. Full breakdown of arguments as of this patch: mode: Mempolicy mode (MPOL_DEFAULT, MPOL_INTERLEAVE) mode_flags: Flags previously or'd into mode in set_mempolicy (e.g.: MPOL_F_STATIC_NODES, MPOL_F_RELATIVE_NODES) home_node: for mbind2. Allows the setting of a policy's home with the use of MPOL_MF_HOME_NODE pol_maxnodes: Max number of nodes in the policy nodemask pol_nodes: Policy nodemask The reserved field accounts explicitly for a potential memory hole in the structure. Suggested-by: Frank van der Linden Suggested-by: Vinicius Tavares Petrucci Suggested-by: Hasan Al Maruf Signed-off-by: Gregory Price Co-developed-by: Vinicius Tavares Petrucci Signed-off-by: Vinicius Tavares Petrucci Suggested-by: Dan Williams Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- .../admin-guide/mm/numa_memory_policy.rst | 17 +++++++++++++++++ include/linux/syscalls.h | 1 + include/uapi/linux/mempolicy.h | 9 +++++++++ 3 files changed, 27 insertions(+) diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Document= ation/admin-guide/mm/numa_memory_policy.rst index a70f20ce1ffb..cbfc5f65ed77 100644 --- a/Documentation/admin-guide/mm/numa_memory_policy.rst +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -480,6 +480,23 @@ closest to which page allocation will come from. Speci= fying the home node overri the default allocation policy to allocate memory close to the local node f= or an executing CPU. =20 +Extended Mempolicy Arguments:: + + struct mpol_param { + __u16 mode; + __u16 mode_flags; + __s32 home_node; /* mbind2: set home node */ + __u64 pol_maxnodes; + __aligned_u64 pol_nodes; /* nodemask pointer */ + }; + +The extended mempolicy argument structure is defined to allow the mempolicy +interfaces future extensibility without the need for additional system cal= ls. + +The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to +all interfaces relative to their non-extended counterparts. Each additional +field may only apply to specific extended interfaces. See the respective +extended interface man page for more details. =20 Memory Policy Command Line Interface =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index fd9d12de7e92..fb0b4b2b9bea 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -74,6 +74,7 @@ struct landlock_ruleset_attr; enum landlock_rule_type; struct cachestat_range; struct cachestat; +struct mpol_param; =20 #include #include diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h index 1f9bb10d1a47..109788c8be92 100644 --- a/include/uapi/linux/mempolicy.h +++ b/include/uapi/linux/mempolicy.h @@ -27,6 +27,15 @@ enum { MPOL_MAX, /* always last member of enum */ }; =20 +struct mpol_param { + __u16 mode; + __u16 mode_flags; + __s32 home_node; /* mbind2: policy home node */ + __u16 pol_maxnodes; + __u8 resv[6]; + __aligned_u64 pol_nodes; +}; + /* Flags for set_mempolicy */ #define MPOL_F_STATIC_NODES (1 << 15) #define MPOL_F_RELATIVE_NODES (1 << 14) --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97D3B20DCE; Wed, 3 Jan 2024 22:43:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SgfqcaTh" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-1d4c3393f99so16722245ad.0; Wed, 03 Jan 2024 14:43:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321797; x=1704926597; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xVh9SfnHzOtZ+DBgaTgeEedVzFLOpOnB7SI7Ca2v4UM=; b=SgfqcaTh6zPSb3yIpFLvDCVIl4GCkQW7B5GNR/REXrPpUcwul5GlDemVhSeuIapDJj D5VKB87whXwYI+0GaBDZbozo4FP6BlGNJwGC7CuUVuzqJe+qMKScjFW8O8ADP2VXuDuf WTcCuQbOawiihcF/jWS50QtSzW2nv8x1UYmk33y9qmvS+3rcolPqn/K1nAg8Mk74tVFN +i3V4OUDAizzXlHS8S3SMxt223VqPqGWXTIMB/QN9L9lhX94c9Xv0fbYFxbMWckB1WSR YZ+pmyJy2uAQix8i8876jEg67Wm70wPV4KkYAum89OPLDL1x7gKAc86K0OCTFndqc00q HFkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321797; x=1704926597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xVh9SfnHzOtZ+DBgaTgeEedVzFLOpOnB7SI7Ca2v4UM=; b=Znluiy9JxGpVVkxezXYoNOFNUxctU5A2VrshTCNDWecgkJAHa7kB852N1esBN68Cmh itDz+rTZ14lJr3xJsgPZvjG4kSBjglGvPFsoh2K5a9wsB0ywFl+ohOSUAVZkmkBUsPqd gLYgyHkP53h81wQQxNqeXdc1pOsm+OSjF71DGvKJNh8gA2MkYZ2MC4XapCarSSWqyS99 vl+1mC2tiO5AAL8djM8ZyKPqImuOYXs50J1Yt06bY81AuH8XEBMhUQUpfpUe8fBbfbh8 jHapqDR6ZViW1PMbPrYtw3RJxLmN2EMrdPo1R6ITBPDkC5IQfh6Pq3fiW16HD4ZIW6pN 005Q== X-Gm-Message-State: AOJu0YzPixNmE0ia41lLSFfMfyQqDO5nG0u9G6MrZYpzPQmE6C6yJvgn 0LtPoCmb7zZzurXdOZf+CA== X-Google-Smtp-Source: AGHT+IEDNwZiLvgSl5Cc64vp9lWc4m4y7JJYana5z+woE417su8nxbcEBBHituoNrgSM8C/dtWInVQ== X-Received: by 2002:a17:903:64d:b0:1d3:f112:2c03 with SMTP id kh13-20020a170903064d00b001d3f1122c03mr19098150plb.53.1704321796882; Wed, 03 Jan 2024 14:43:16 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.43.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:43:16 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com, Michal Hocko , Geert Uytterhoeven Subject: [PATCH v6 09/12] mm/mempolicy: add set_mempolicy2 syscall Date: Wed, 3 Jan 2024 17:42:06 -0500 Message-Id: <20240103224209.2541-10-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" set_mempolicy2 is an extensible set_mempolicy interface which allows a user to set the per-task memory policy. Defined as: set_mempolicy2(struct mpol_param *param, size_t usize, unsigned long flags); relevant mpol_param fields include the following: mode: The MPOL_* policy (DEFAULT, INTERLEAVE, etc.) mode_flags: The MPOL_F_* flags that were previously passed in or'd into the mode. This was split to hopefully allow future extensions additional mode/flag space. home_node: ignored (see note below) pol_nodes: the nodemask to apply for the memory policy pol_maxnodes: The max number of nodes described by pol_nodes The usize arg is intended for the user to pass in sizeof(mpol_param) to allow forward/backward compatibility whenever possible. The flags argument is intended to future proof the syscall against future extensions which may require interpreting the arguments in the structure differently. Semantics of `set_mempolicy` are otherwise the same as `set_mempolicy` as of this patch. As of this patch, setting the home node of a task-policy is not supported, as this functionality was not supported by set_mempolicy. Additional research should be done to determine whether adding this functionality is safe, but doing so would only require setting MPOL_MF_HOME_NODE and providing a valid home node value. Suggested-by: Michal Hocko Signed-off-by: Gregory Price Acked-by: Geert Uytterhoeven Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- .../admin-guide/mm/numa_memory_policy.rst | 10 ++++++ arch/alpha/kernel/syscalls/syscall.tbl | 1 + arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 ++ arch/m68k/kernel/syscalls/syscall.tbl | 1 + arch/microblaze/kernel/syscalls/syscall.tbl | 1 + arch/mips/kernel/syscalls/syscall_n32.tbl | 1 + arch/mips/kernel/syscalls/syscall_o32.tbl | 1 + arch/parisc/kernel/syscalls/syscall.tbl | 1 + arch/powerpc/kernel/syscalls/syscall.tbl | 1 + arch/s390/kernel/syscalls/syscall.tbl | 1 + arch/sh/kernel/syscalls/syscall.tbl | 1 + arch/sparc/kernel/syscalls/syscall.tbl | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/xtensa/kernel/syscalls/syscall.tbl | 1 + include/linux/syscalls.h | 2 ++ include/uapi/asm-generic/unistd.h | 4 ++- kernel/sys_ni.c | 1 + mm/mempolicy.c | 36 +++++++++++++++++++ .../arch/mips/entry/syscalls/syscall_n64.tbl | 1 + .../arch/powerpc/entry/syscalls/syscall.tbl | 1 + .../perf/arch/s390/entry/syscalls/syscall.tbl | 1 + .../arch/x86/entry/syscalls/syscall_64.tbl | 1 + 25 files changed, 73 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Document= ation/admin-guide/mm/numa_memory_policy.rst index cbfc5f65ed77..62a4ea14c646 100644 --- a/Documentation/admin-guide/mm/numa_memory_policy.rst +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -430,6 +430,8 @@ Set [Task] Memory Policy:: =20 long set_mempolicy(int mode, const unsigned long *nmask, unsigned long maxnode); + long set_mempolicy2(struct mpol_param *param, size_t size, + unsigned long flags); =20 Set's the calling task's "task/process memory policy" to mode specified by the 'mode' argument and the set of nodes defined by @@ -438,6 +440,12 @@ specified by the 'mode' argument and the set of nodes = defined by 'mode' argument with the flag (for example: MPOL_INTERLEAVE | MPOL_F_STATIC_NODES). =20 +set_mempolicy2() is an extended version of set_mempolicy() capable +of setting a mempolicy which requires more information than can be +passed via get_mempolicy(). For example, weighted interleave with +task-local weights requires a weight array to be passed via the +'mpol_param->il_weights' parameter. + See the set_mempolicy(2) man page for more details =20 =20 @@ -493,6 +501,8 @@ Extended Mempolicy Arguments:: The extended mempolicy argument structure is defined to allow the mempolicy interfaces future extensibility without the need for additional system cal= ls. =20 +Extended interfaces (set_mempolicy2) use this argument structure. + The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to all interfaces relative to their non-extended counterparts. Each additional field may only apply to specific extended interfaces. See the respective diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/sys= calls/syscall.tbl index 18c842ca6c32..0dc288a1118a 100644 --- a/arch/alpha/kernel/syscalls/syscall.tbl +++ b/arch/alpha/kernel/syscalls/syscall.tbl @@ -496,3 +496,4 @@ 564 common futex_wake sys_futex_wake 565 common futex_wait sys_futex_wait 566 common futex_requeue sys_futex_requeue +567 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl index 584f9528c996..50172ec0e1f5 100644 --- a/arch/arm/tools/syscall.tbl +++ b/arch/arm/tools/syscall.tbl @@ -470,3 +470,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/arm64/include/asm/unistd.h b/arch/arm64/include/asm/unist= d.h index 531effca5f1f..298313d2e0af 100644 --- a/arch/arm64/include/asm/unistd.h +++ b/arch/arm64/include/asm/unistd.h @@ -39,7 +39,7 @@ #define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5) #define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800) =20 -#define __NR_compat_syscalls 457 +#define __NR_compat_syscalls 458 #endif =20 #define __ARCH_WANT_SYS_CLONE diff --git a/arch/arm64/include/asm/unistd32.h b/arch/arm64/include/asm/uni= std32.h index 9f7c1bf99526..cee8d669c342 100644 --- a/arch/arm64/include/asm/unistd32.h +++ b/arch/arm64/include/asm/unistd32.h @@ -919,6 +919,8 @@ __SYSCALL(__NR_futex_wake, sys_futex_wake) __SYSCALL(__NR_futex_wait, sys_futex_wait) #define __NR_futex_requeue 456 __SYSCALL(__NR_futex_requeue, sys_futex_requeue) +#define __NR_set_mempolicy2 457 +__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2) =20 /* * Please add new compat syscalls above this comment and update diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/sysca= lls/syscall.tbl index 7a4b780e82cb..839d90c535f2 100644 --- a/arch/m68k/kernel/syscalls/syscall.tbl +++ b/arch/m68k/kernel/syscalls/syscall.tbl @@ -456,3 +456,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/= kernel/syscalls/syscall.tbl index 5b6a0b02b7de..567c8b883735 100644 --- a/arch/microblaze/kernel/syscalls/syscall.tbl +++ b/arch/microblaze/kernel/syscalls/syscall.tbl @@ -462,3 +462,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/s= yscalls/syscall_n32.tbl index a842b41c8e06..cc0640e16f2f 100644 --- a/arch/mips/kernel/syscalls/syscall_n32.tbl +++ b/arch/mips/kernel/syscalls/syscall_n32.tbl @@ -395,3 +395,4 @@ 454 n32 futex_wake sys_futex_wake 455 n32 futex_wait sys_futex_wait 456 n32 futex_requeue sys_futex_requeue +457 n32 set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/s= yscalls/syscall_o32.tbl index 525cc54bc63b..f7262fde98d9 100644 --- a/arch/mips/kernel/syscalls/syscall_o32.tbl +++ b/arch/mips/kernel/syscalls/syscall_o32.tbl @@ -444,3 +444,4 @@ 454 o32 futex_wake sys_futex_wake 455 o32 futex_wait sys_futex_wait 456 o32 futex_requeue sys_futex_requeue +457 o32 set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/s= yscalls/syscall.tbl index a47798fed54e..e10f0e8bd064 100644 --- a/arch/parisc/kernel/syscalls/syscall.tbl +++ b/arch/parisc/kernel/syscalls/syscall.tbl @@ -455,3 +455,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel= /syscalls/syscall.tbl index 7fab411378f2..4f03f5f42b78 100644 --- a/arch/powerpc/kernel/syscalls/syscall.tbl +++ b/arch/powerpc/kernel/syscalls/syscall.tbl @@ -543,3 +543,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/sysca= lls/syscall.tbl index 86fec9b080f6..f98dadc2e9df 100644 --- a/arch/s390/kernel/syscalls/syscall.tbl +++ b/arch/s390/kernel/syscalls/syscall.tbl @@ -459,3 +459,4 @@ 454 common futex_wake sys_futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/= syscall.tbl index 363fae0fe9bf..f47ba9f2d05d 100644 --- a/arch/sh/kernel/syscalls/syscall.tbl +++ b/arch/sh/kernel/syscalls/syscall.tbl @@ -459,3 +459,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/sys= calls/syscall.tbl index 7bcaa3d5ea44..53fb16616728 100644 --- a/arch/sparc/kernel/syscalls/syscall.tbl +++ b/arch/sparc/kernel/syscalls/syscall.tbl @@ -502,3 +502,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscal= ls/syscall_32.tbl index c8fac5205803..4b4dc41b24ee 100644 --- a/arch/x86/entry/syscalls/syscall_32.tbl +++ b/arch/x86/entry/syscalls/syscall_32.tbl @@ -461,3 +461,4 @@ 454 i386 futex_wake sys_futex_wake 455 i386 futex_wait sys_futex_wait 456 i386 futex_requeue sys_futex_requeue +457 i386 set_mempolicy2 sys_set_mempolicy2 diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscal= ls/syscall_64.tbl index 8cb8bf68721c..1bc2190bec27 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -378,6 +378,7 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 =20 # # Due to a historical design error, certain syscalls are numbered differen= tly diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/s= yscalls/syscall.tbl index 06eefa9c1458..e26dc89399eb 100644 --- a/arch/xtensa/kernel/syscalls/syscall.tbl +++ b/arch/xtensa/kernel/syscalls/syscall.tbl @@ -427,3 +427,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index fb0b4b2b9bea..b37ea6715456 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -823,6 +823,8 @@ asmlinkage long sys_get_mempolicy(int __user *policy, unsigned long addr, unsigned long flags); asmlinkage long sys_set_mempolicy(int mode, const unsigned long __user *nm= ask, unsigned long maxnode); +asmlinkage long sys_set_mempolicy2(struct mpol_param __user *param, size_t= size, + unsigned long flags); asmlinkage long sys_migrate_pages(pid_t pid, unsigned long maxnode, const unsigned long __user *from, const unsigned long __user *to); diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/u= nistd.h index 756b013fb832..55486aba099f 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -828,9 +828,11 @@ __SYSCALL(__NR_futex_wake, sys_futex_wake) __SYSCALL(__NR_futex_wait, sys_futex_wait) #define __NR_futex_requeue 456 __SYSCALL(__NR_futex_requeue, sys_futex_requeue) +#define __NR_set_mempolicy2 457 +__SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2) =20 #undef __NR_syscalls -#define __NR_syscalls 457 +#define __NR_syscalls 458 =20 /* * 32 bit systems traditionally used different diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index 9a846439b36a..fa1373c8bff8 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -189,6 +189,7 @@ COND_SYSCALL(remap_file_pages); COND_SYSCALL(mbind); COND_SYSCALL(get_mempolicy); COND_SYSCALL(set_mempolicy); +COND_SYSCALL(set_mempolicy2); COND_SYSCALL(migrate_pages); COND_SYSCALL(move_pages); COND_SYSCALL(set_mempolicy_home_node); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 4a566341cf43..84d877195deb 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1645,6 +1645,42 @@ SYSCALL_DEFINE3(set_mempolicy, int, mode, const unsi= gned long __user *, nmask, return kernel_set_mempolicy(mode, nmask, maxnode); } =20 +SYSCALL_DEFINE3(set_mempolicy2, struct mpol_param __user *, uparam, + size_t, usize, unsigned long, flags) +{ + struct mpol_param kparam; + struct mempolicy_param mparam; + int err; + nodemask_t policy_nodemask; + unsigned long __user *nodes_ptr; + + if (flags) + return -EINVAL; + + err =3D copy_struct_from_user(&kparam, sizeof(kparam), uparam, usize); + if (err) + return err; + + err =3D validate_mpol_flags(kparam.mode, &kparam.mode_flags); + if (err) + return err; + + memset(&mparam, 0, sizeof(mparam)); + mparam.mode =3D kparam.mode; + mparam.mode_flags =3D kparam.mode_flags; + if (kparam.pol_nodes) { + nodes_ptr =3D u64_to_user_ptr(kparam.pol_nodes); + err =3D get_nodes(&policy_nodemask, nodes_ptr, + kparam.pol_maxnodes); + if (err) + return err; + mparam.policy_nodes =3D &policy_nodemask; + } else + mparam.policy_nodes =3D NULL; + + return do_set_mempolicy(&mparam); +} + static int kernel_migrate_pages(pid_t pid, unsigned long maxnode, const unsigned long __user *old_nodes, const unsigned long __user *new_nodes) diff --git a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl b/tools/pe= rf/arch/mips/entry/syscalls/syscall_n64.tbl index 116ff501bf92..bb1351df51d9 100644 --- a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl +++ b/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl @@ -371,3 +371,4 @@ 454 n64 futex_wake sys_futex_wake 455 n64 futex_wait sys_futex_wait 456 n64 futex_requeue sys_futex_requeue +457 n64 set_mempolicy2 sys_set_mempolicy2 diff --git a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl b/tools/per= f/arch/powerpc/entry/syscalls/syscall.tbl index 7fab411378f2..4f03f5f42b78 100644 --- a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl +++ b/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl @@ -543,3 +543,4 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 diff --git a/tools/perf/arch/s390/entry/syscalls/syscall.tbl b/tools/perf/a= rch/s390/entry/syscalls/syscall.tbl index 86fec9b080f6..f98dadc2e9df 100644 --- a/tools/perf/arch/s390/entry/syscalls/syscall.tbl +++ b/tools/perf/arch/s390/entry/syscalls/syscall.tbl @@ -459,3 +459,4 @@ 454 common futex_wake sys_futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2 diff --git a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl b/tools/perf= /arch/x86/entry/syscalls/syscall_64.tbl index 8cb8bf68721c..21f2579679d4 100644 --- a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl +++ b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl @@ -378,6 +378,7 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common set_mempolicy2 sys_set_mempolicy2 =20 # # Due to a historical design error, certain syscalls are numbered differen= tly --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48FD5210FD; Wed, 3 Jan 2024 22:43:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Kf4eTr/x" Received: by mail-pg1-f196.google.com with SMTP id 41be03b00d2f7-5c6bd3100fcso3233212a12.3; Wed, 03 Jan 2024 14:43:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321802; x=1704926602; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+Qqdj1NnYTIDFJ6uY9J8ib53vu7GVR0gDgEnOcK2G8U=; b=Kf4eTr/xCCZD77R4nn6jUt8rXiSPfi4LhCHlPSoSmitbrTl45V7BuZG+zWzbAjWI6S /h0myx1nold7c4PrBxmnOHqdHtJ8TDvrNIRbJJn7id+1CkOhqlnzdnTPeQMILD4+alPD WNGAdTLKFWOsLQIAVcpV8hZyhEycD0PMW0lhFVuuFLSvlLUu3JSET8+iCqOqo/Xk/eXP 6NgW7V7usJuOrCz8CW01lkm3Ov9/95iRbrmWWI0/jroep3mnCfLInMfEcVnUUyq6VzlZ 81WYxKuxzDmrB9jkdQ6GLA10fM1mx2u/pIct6prcqhp+0VPN3630lKnRQgFYDEEgcJuA p0TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321802; x=1704926602; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+Qqdj1NnYTIDFJ6uY9J8ib53vu7GVR0gDgEnOcK2G8U=; b=lhBX78mqf0FIr51ne2Y8Hlj1JWJXnHeKb0LhGFTl43/a3so5i3tb6Ba8ZhfhWyA3P/ KCr0riR9IhxOJXngNw4Whn2ENeBvRkL1quN3PFO1GJDyd48Eiutp9KY/LgVL+Xja+wV4 aGjw3yOs5tjco5a1Kpahbi+/qnZT+erjqN7ee1aylXOul+wgr3WM8C5380qIqv4LAGTv 5Rpgfv8EAsYcL5BPyc2/xVAp147uWK/FJkvkrbV0zGTWi8PiWVPNpYai87Co35pT4K/f kjMOmNmyBH/OtikoB0AGNT0kmKc3866ik06R6pxjOovN5HcPo4svPFCKnK4FKki0BwLl tzjQ== X-Gm-Message-State: AOJu0Yx07GcfiMCXzwQoCXuzyf9tkc5GAs2dlXQ5Ef8C7oVrcHmBt9XK daIH5kCRNh4GlPLN3rN/AA== X-Google-Smtp-Source: AGHT+IGS84r7imvtLLcpZ/L3HLh7Z6Adh2a2nzXMjkCA8aD2pALvXvexEWC9PmlzIh8LVvOvL7EGXg== X-Received: by 2002:a17:902:db02:b0:1d4:caea:5fad with SMTP id m2-20020a170902db0200b001d4caea5fadmr2459374plx.33.1704321801625; Wed, 03 Jan 2024 14:43:21 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.43.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:43:21 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com, Michal Hocko , Geert Uytterhoeven Subject: [PATCH v6 10/12] mm/mempolicy: add get_mempolicy2 syscall Date: Wed, 3 Jan 2024 17:42:07 -0500 Message-Id: <20240103224209.2541-11-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" get_mempolicy2 is an extensible get_mempolicy interface which allows a user to retrieve the memory policy for a task or address. Defined as: get_mempolicy2(struct mpol_param *param, size_t size, unsigned long addr, unsigned long flags) Top level input values: mpol_param: The field which collects information about the mempolicy returned to userspace. addr: if MPOL_F_ADDR is passed in `flags`, this address will be used to return the mempolicy details of the vma the address belongs to flags: if MPOL_F_ADDR, return mempolicy info vma containing addr else, returns task mempolicy information Input values include the following fields of mpol_param: pol_maxnodes: if pol_nodes is set, must describe max number of nodes to be copied to pol_nodes pol_nodes: if set, the nodemask of the policy returned here Output values include the following fields of mpol_param: mode: mempolicy mode mode_flags: mempolicy mode flags home_node: policy home node will be returned here, or -1 if not. pol_nodes: if set, the nodemask for the mempolicy MPOL_F_NODE has been dropped from get_mempolicy2 (EINVAL). MPOL_F_NODE originally allowed for either the ability to acquire the node the page of `addr` is presently allocated on - or, if addr is not provided and the policy mode is MPOL_INTERLEAVE - it would return the node that would be used for the next allocation. Neither of these capabilities were pulled forward into get_mempolicy2 a) both are still possible to acquire via get_mempolicy() b) both of pieces of data are racey by definition and have not proven useful. c) Should such a use be identified, it can be easily added back into mempolicy2 as an extension. MPOL_F_MEMS_ALLOWED has been dropped from get_mempolicy2 (EINVAL). MPOL_F_MEMS_ALLOWED originally returned the task->mems_allowed nodemask in the nodemask parameter instead of the task or vma nodemask. a) this is still accessible from get_mempolicy() b) task->mems_allowed is not technically part of mempolicy, though it is related. c) should this warrant bringing forward (in the scenario where get_mempolicy is deprecated), it can be added as an explicit extension. Or more smartly: implemented as its own syscall. Suggested-by: Michal Hocko Signed-off-by: Gregory Price Acked-by: Geert Uytterhoeven Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Hyeongtak Ji Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Vinicius Tavares Petrucci Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- .../admin-guide/mm/numa_memory_policy.rst | 11 ++++- arch/alpha/kernel/syscalls/syscall.tbl | 1 + arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 + arch/m68k/kernel/syscalls/syscall.tbl | 1 + arch/microblaze/kernel/syscalls/syscall.tbl | 1 + arch/mips/kernel/syscalls/syscall_n32.tbl | 1 + arch/mips/kernel/syscalls/syscall_o32.tbl | 1 + arch/parisc/kernel/syscalls/syscall.tbl | 1 + arch/powerpc/kernel/syscalls/syscall.tbl | 1 + arch/s390/kernel/syscalls/syscall.tbl | 1 + arch/sh/kernel/syscalls/syscall.tbl | 1 + arch/sparc/kernel/syscalls/syscall.tbl | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/xtensa/kernel/syscalls/syscall.tbl | 1 + include/linux/syscalls.h | 2 + include/uapi/asm-generic/unistd.h | 4 +- kernel/sys_ni.c | 1 + mm/mempolicy.c | 42 +++++++++++++++++++ .../arch/mips/entry/syscalls/syscall_n64.tbl | 1 + .../arch/powerpc/entry/syscalls/syscall.tbl | 1 + .../perf/arch/s390/entry/syscalls/syscall.tbl | 1 + .../arch/x86/entry/syscalls/syscall_64.tbl | 1 + 25 files changed, 79 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Document= ation/admin-guide/mm/numa_memory_policy.rst index 62a4ea14c646..a2ff6e89e48b 100644 --- a/Documentation/admin-guide/mm/numa_memory_policy.rst +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -454,11 +454,20 @@ Get [Task] Memory Policy or Related Information:: long get_mempolicy(int *mode, const unsigned long *nmask, unsigned long maxnode, void *addr, int flags); + long get_mempolicy2(struct mpol_param *param, size_t size, + unsigned long addr, unsigned long flags); =20 Queries the "task/process memory policy" of the calling task, or the policy or location of a specified virtual address, depending on the 'flags' argument. =20 +get_mempolicy2() is an extended version of get_mempolicy() capable of +acquiring extended information about a mempolicy, including those +that can only be set via set_mempolicy2() or mbind2(). + +MPOL_F_NODE functionality has been removed from get_mempolicy2(), +but can still be accessed via get_mempolicy(). + See the get_mempolicy(2) man page for more details =20 =20 @@ -501,7 +510,7 @@ Extended Mempolicy Arguments:: The extended mempolicy argument structure is defined to allow the mempolicy interfaces future extensibility without the need for additional system cal= ls. =20 -Extended interfaces (set_mempolicy2) use this argument structure. +Extended interfaces (set_mempolicy2 and get_mempolicy2) use this structure. =20 The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to all interfaces relative to their non-extended counterparts. Each additional diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/sys= calls/syscall.tbl index 0dc288a1118a..0301a8b0a262 100644 --- a/arch/alpha/kernel/syscalls/syscall.tbl +++ b/arch/alpha/kernel/syscalls/syscall.tbl @@ -497,3 +497,4 @@ 565 common futex_wait sys_futex_wait 566 common futex_requeue sys_futex_requeue 567 common set_mempolicy2 sys_set_mempolicy2 +568 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl index 50172ec0e1f5..771a33446e8e 100644 --- a/arch/arm/tools/syscall.tbl +++ b/arch/arm/tools/syscall.tbl @@ -471,3 +471,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/arm64/include/asm/unistd.h b/arch/arm64/include/asm/unist= d.h index 298313d2e0af..b63f870debaf 100644 --- a/arch/arm64/include/asm/unistd.h +++ b/arch/arm64/include/asm/unistd.h @@ -39,7 +39,7 @@ #define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5) #define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800) =20 -#define __NR_compat_syscalls 458 +#define __NR_compat_syscalls 459 #endif =20 #define __ARCH_WANT_SYS_CLONE diff --git a/arch/arm64/include/asm/unistd32.h b/arch/arm64/include/asm/uni= std32.h index cee8d669c342..f8d01007aee0 100644 --- a/arch/arm64/include/asm/unistd32.h +++ b/arch/arm64/include/asm/unistd32.h @@ -921,6 +921,8 @@ __SYSCALL(__NR_futex_wait, sys_futex_wait) __SYSCALL(__NR_futex_requeue, sys_futex_requeue) #define __NR_set_mempolicy2 457 __SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2) +#define __NR_get_mempolicy2 458 +__SYSCALL(__NR_get_mempolicy2, sys_get_mempolicy2) =20 /* * Please add new compat syscalls above this comment and update diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/sysca= lls/syscall.tbl index 839d90c535f2..048a409e684c 100644 --- a/arch/m68k/kernel/syscalls/syscall.tbl +++ b/arch/m68k/kernel/syscalls/syscall.tbl @@ -457,3 +457,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/= kernel/syscalls/syscall.tbl index 567c8b883735..327b01bd6793 100644 --- a/arch/microblaze/kernel/syscalls/syscall.tbl +++ b/arch/microblaze/kernel/syscalls/syscall.tbl @@ -463,3 +463,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/s= yscalls/syscall_n32.tbl index cc0640e16f2f..921d58e1da23 100644 --- a/arch/mips/kernel/syscalls/syscall_n32.tbl +++ b/arch/mips/kernel/syscalls/syscall_n32.tbl @@ -396,3 +396,4 @@ 455 n32 futex_wait sys_futex_wait 456 n32 futex_requeue sys_futex_requeue 457 n32 set_mempolicy2 sys_set_mempolicy2 +458 n32 get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/s= yscalls/syscall_o32.tbl index f7262fde98d9..9271c83c9993 100644 --- a/arch/mips/kernel/syscalls/syscall_o32.tbl +++ b/arch/mips/kernel/syscalls/syscall_o32.tbl @@ -445,3 +445,4 @@ 455 o32 futex_wait sys_futex_wait 456 o32 futex_requeue sys_futex_requeue 457 o32 set_mempolicy2 sys_set_mempolicy2 +458 o32 get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/s= yscalls/syscall.tbl index e10f0e8bd064..0654f3f89fc7 100644 --- a/arch/parisc/kernel/syscalls/syscall.tbl +++ b/arch/parisc/kernel/syscalls/syscall.tbl @@ -456,3 +456,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel= /syscalls/syscall.tbl index 4f03f5f42b78..ac11d2064e7a 100644 --- a/arch/powerpc/kernel/syscalls/syscall.tbl +++ b/arch/powerpc/kernel/syscalls/syscall.tbl @@ -544,3 +544,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/sysca= lls/syscall.tbl index f98dadc2e9df..1cdcafe1ccca 100644 --- a/arch/s390/kernel/syscalls/syscall.tbl +++ b/arch/s390/kernel/syscalls/syscall.tbl @@ -460,3 +460,4 @@ 455 common futex_wait sys_futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/= syscall.tbl index f47ba9f2d05d..f71742024c29 100644 --- a/arch/sh/kernel/syscalls/syscall.tbl +++ b/arch/sh/kernel/syscalls/syscall.tbl @@ -460,3 +460,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/sys= calls/syscall.tbl index 53fb16616728..2fbf5dbe0620 100644 --- a/arch/sparc/kernel/syscalls/syscall.tbl +++ b/arch/sparc/kernel/syscalls/syscall.tbl @@ -503,3 +503,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscal= ls/syscall_32.tbl index 4b4dc41b24ee..0af813b9a118 100644 --- a/arch/x86/entry/syscalls/syscall_32.tbl +++ b/arch/x86/entry/syscalls/syscall_32.tbl @@ -462,3 +462,4 @@ 455 i386 futex_wait sys_futex_wait 456 i386 futex_requeue sys_futex_requeue 457 i386 set_mempolicy2 sys_set_mempolicy2 +458 i386 get_mempolicy2 sys_get_mempolicy2 diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscal= ls/syscall_64.tbl index 1bc2190bec27..0b777876fc15 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -379,6 +379,7 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 =20 # # Due to a historical design error, certain syscalls are numbered differen= tly diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/s= yscalls/syscall.tbl index e26dc89399eb..4536c9a4227d 100644 --- a/arch/xtensa/kernel/syscalls/syscall.tbl +++ b/arch/xtensa/kernel/syscalls/syscall.tbl @@ -428,3 +428,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index b37ea6715456..c4dc5069bae7 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -821,6 +821,8 @@ asmlinkage long sys_get_mempolicy(int __user *policy, unsigned long __user *nmask, unsigned long maxnode, unsigned long addr, unsigned long flags); +asmlinkage long sys_get_mempolicy2(struct mpol_param __user *param, size_t= size, + unsigned long addr, unsigned long flags); asmlinkage long sys_set_mempolicy(int mode, const unsigned long __user *nm= ask, unsigned long maxnode); asmlinkage long sys_set_mempolicy2(struct mpol_param __user *param, size_t= size, diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/u= nistd.h index 55486aba099f..719accc731db 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -830,9 +830,11 @@ __SYSCALL(__NR_futex_wait, sys_futex_wait) __SYSCALL(__NR_futex_requeue, sys_futex_requeue) #define __NR_set_mempolicy2 457 __SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2) +#define __NR_get_mempolicy2 458 +__SYSCALL(__NR_get_mempolicy2, sys_get_mempolicy2) =20 #undef __NR_syscalls -#define __NR_syscalls 458 +#define __NR_syscalls 459 =20 /* * 32 bit systems traditionally used different diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index fa1373c8bff8..6afbd3a41319 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -188,6 +188,7 @@ COND_SYSCALL(process_mrelease); COND_SYSCALL(remap_file_pages); COND_SYSCALL(mbind); COND_SYSCALL(get_mempolicy); +COND_SYSCALL(get_mempolicy2); COND_SYSCALL(set_mempolicy); COND_SYSCALL(set_mempolicy2); COND_SYSCALL(migrate_pages); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 84d877195deb..0b2e31d8636d 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1873,6 +1873,48 @@ SYSCALL_DEFINE5(get_mempolicy, int __user *, policy, return kernel_get_mempolicy(policy, nmask, maxnode, addr, flags); } =20 +SYSCALL_DEFINE4(get_mempolicy2, struct mpol_param __user *, uparam, size_t= , usize, + unsigned long, addr, unsigned long, flags) +{ + struct mpol_param kparam; + struct mempolicy_param mparam; + int err; + nodemask_t policy_nodemask; + unsigned long __user *nodes_ptr; + + if (flags & ~(MPOL_F_ADDR)) + return -EINVAL; + + /* initialize any memory liable to be copied to userland */ + memset(&mparam, 0, sizeof(mparam)); + + err =3D copy_struct_from_user(&kparam, sizeof(kparam), uparam, usize); + if (err) + return -EINVAL; + + mparam.policy_nodes =3D kparam.pol_nodes ? &policy_nodemask : NULL; + if (flags & MPOL_F_ADDR) + err =3D do_get_vma_mempolicy(untagged_addr(addr), NULL, &mparam); + else + err =3D do_get_task_mempolicy(&mparam, NULL); + + if (err) + return err; + + kparam.mode =3D mparam.mode; + kparam.mode_flags =3D mparam.mode_flags; + kparam.home_node =3D mparam.home_node; + if (kparam.pol_nodes) { + nodes_ptr =3D u64_to_user_ptr(kparam.pol_nodes); + err =3D copy_nodes_to_user(nodes_ptr, kparam.pol_maxnodes, + mparam.policy_nodes); + if (err) + return err; + } + + return copy_to_user(uparam, &kparam, usize) ? -EFAULT : 0; +} + bool vma_migratable(struct vm_area_struct *vma) { if (vma->vm_flags & (VM_IO | VM_PFNMAP)) diff --git a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl b/tools/pe= rf/arch/mips/entry/syscalls/syscall_n64.tbl index bb1351df51d9..c34c6877379e 100644 --- a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl +++ b/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl @@ -372,3 +372,4 @@ 455 n64 futex_wait sys_futex_wait 456 n64 futex_requeue sys_futex_requeue 457 n64 set_mempolicy2 sys_set_mempolicy2 +458 n64 get_mempolicy2 sys_get_mempolicy2 diff --git a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl b/tools/per= f/arch/powerpc/entry/syscalls/syscall.tbl index 4f03f5f42b78..ac11d2064e7a 100644 --- a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl +++ b/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl @@ -544,3 +544,4 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 diff --git a/tools/perf/arch/s390/entry/syscalls/syscall.tbl b/tools/perf/a= rch/s390/entry/syscalls/syscall.tbl index f98dadc2e9df..1cdcafe1ccca 100644 --- a/tools/perf/arch/s390/entry/syscalls/syscall.tbl +++ b/tools/perf/arch/s390/entry/syscalls/syscall.tbl @@ -460,3 +460,4 @@ 455 common futex_wait sys_futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 sys_get_mempolicy2 diff --git a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl b/tools/perf= /arch/x86/entry/syscalls/syscall_64.tbl index 21f2579679d4..edf338f32645 100644 --- a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl +++ b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl @@ -379,6 +379,7 @@ 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 +458 common get_mempolicy2 sys_get_mempolicy2 =20 # # Due to a historical design error, certain syscalls are numbered differen= tly --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2580221345; Wed, 3 Jan 2024 22:43:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KjFD6hnM" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-1d3ea5cc137so84265515ad.0; Wed, 03 Jan 2024 14:43:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321806; x=1704926606; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JGON5uNofvfE+VQKTV04yG0OYqE17b/v8cI1b6yghm8=; b=KjFD6hnMGhef2DG2P/8Et3CcZEUS/3me4TO6CnfPiAt8gvpEg0ppG80GABdrmD6BX1 hfigSGhf3PAHBghJctFgZ+WFQgkbNBsDYuGZXaL26HXouuhs/JL3STq6fnWIk/Q3aGTK WSbwNnIQeRJB+eZ5Upz7hVpoBy5yQ7m70FEJ9RzRpvmqgsGhEuwgU3wqOTQgbFrsaTp+ Wa6XoES2+zzlWXXywkzcniqiO9ZATo7XtNDVOnnMlohGFyHbKHZnUsh3LtBvp2GPmA7r OzG33TA2f7Dm3wvU6obIfrs8gJnqXTShLly9mWMGXyta9aNUUrvD7x8eB/8nndEKcIBB JH2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321806; x=1704926606; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JGON5uNofvfE+VQKTV04yG0OYqE17b/v8cI1b6yghm8=; b=EWvuUd0IQvtnIlH9QVi/aqYHdDrPn5GTCGq9xbw4sagGjchbTxxdyDfzkzIahNnysk ZmzTV9PCANHyM8f7kpWtHl0X+m/gKm1i6o2gW2x01i6mfNpN+9POa7RnmzUrd/efq7iu qqJ5n/HphX6olhHAZph+FZ9lGcJItA9MV+7vnVj3tSjc1P8iPt4brqdwlRsrwJnOZTjM sztcmCLKIgnw64E3cczS40nIOxw62qxLuDfjs5vPu7wlSt8s/kjBM0X6zxejueDz9i6N w2ANch4ahRTL0PQDPk4zZJB1SwPuRHx0nJWizrWrSL4yXABcxnTG3LzMxnEQUbK+38Xp tg6A== X-Gm-Message-State: AOJu0Yz4zPhSAH3leCU+lC5IR2ek+o7bgrfaMcWC799tHJ/4ZXkO0AZS b+q25rZ07+59S5fzOBtLnAdpY2WjbKYbkVA= X-Google-Smtp-Source: AGHT+IGMfKOlH2P1OArn75s/+EDYjmbM+5QEPd1KuUWMUTxPnmPVUk0d/iZKbPi93hOtG88McABlbQ== X-Received: by 2002:a17:902:e80b:b0:1d4:c98d:4032 with SMTP id u11-20020a170902e80b00b001d4c98d4032mr3120550plg.23.1704321806467; Wed, 03 Jan 2024 14:43:26 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.43.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:43:26 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com, Michal Hocko , Frank van der Linden , Geert Uytterhoeven Subject: [PATCH v6 11/12] mm/mempolicy: add the mbind2 syscall Date: Wed, 3 Jan 2024 17:42:08 -0500 Message-Id: <20240103224209.2541-12-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" mbind2 is an extensible mbind interface which allows a user to set the mempolicy for one or more address ranges. Defined as: mbind2(unsigned long addr, unsigned long len, struct mpol_param *param, size_t size, unsigned long flags) addr: address of the memory range to operate on len: length of the memory range flags: MPOL_MF_HOME_NODE + original mbind() flags Input values include the following fields of mpol_param: mode: The MPOL_* policy (DEFAULT, INTERLEAVE, etc.) mode_flags: The MPOL_F_* flags that were previously passed in or'd into the mode. This was split to hopefully allow future extensions additional mode/flag space. home_node: if (flags & MPOL_MF_HOME_NODE), set home node of policy to this otherwise it is ignored. pol_maxnodes: The max number of nodes described by pol_nodes pol_nodes: the nodemask to apply for the memory policy The semantics are otherwise the same as mbind(), except that the home_node can be set. Suggested-by: Michal Hocko Suggested-by: Frank van der Linden Suggested-by: Vinicius Tavares Petrucci Suggested-by: Rakie Kim Suggested-by: Hyeongtak Ji Suggested-by: Honggyu Kim Signed-off-by: Gregory Price Co-developed-by: Vinicius Tavares Petrucci Acked-by: Geert Uytterhoeven Suggested-by: Dan Williams Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- .../admin-guide/mm/numa_memory_policy.rst | 12 +++++- arch/alpha/kernel/syscalls/syscall.tbl | 1 + arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 + arch/m68k/kernel/syscalls/syscall.tbl | 1 + arch/microblaze/kernel/syscalls/syscall.tbl | 1 + arch/mips/kernel/syscalls/syscall_n32.tbl | 1 + arch/mips/kernel/syscalls/syscall_o32.tbl | 1 + arch/parisc/kernel/syscalls/syscall.tbl | 1 + arch/powerpc/kernel/syscalls/syscall.tbl | 1 + arch/s390/kernel/syscalls/syscall.tbl | 1 + arch/sh/kernel/syscalls/syscall.tbl | 1 + arch/sparc/kernel/syscalls/syscall.tbl | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/xtensa/kernel/syscalls/syscall.tbl | 1 + include/linux/syscalls.h | 3 ++ include/uapi/asm-generic/unistd.h | 4 +- include/uapi/linux/mempolicy.h | 5 ++- kernel/sys_ni.c | 1 + mm/mempolicy.c | 43 +++++++++++++++++++ .../arch/mips/entry/syscalls/syscall_n64.tbl | 1 + .../arch/powerpc/entry/syscalls/syscall.tbl | 1 + .../perf/arch/s390/entry/syscalls/syscall.tbl | 1 + .../arch/x86/entry/syscalls/syscall_64.tbl | 1 + 26 files changed, 85 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Document= ation/admin-guide/mm/numa_memory_policy.rst index a2ff6e89e48b..66a778d58899 100644 --- a/Documentation/admin-guide/mm/numa_memory_policy.rst +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -476,12 +476,18 @@ Install VMA/Shared Policy for a Range of Task's Addre= ss Space:: long mbind(void *start, unsigned long len, int mode, const unsigned long *nmask, unsigned long maxnode, unsigned flags); + long mbind2(void* start, unsigned long len, struct mpol_param *param, + size_t size, unsigned long flags); =20 mbind() installs the policy specified by (mode, nmask, maxnodes) as a VMA policy for the range of the calling task's address space specified by the 'start' and 'len' arguments. Additional actions may be requested via the 'flags' argument. =20 +mbind2() is an extended version of mbind() capable of setting extended +mempolicy features. For example, one can set the home node for the memory +policy without an additional call to set_mempolicy_home_node(). + See the mbind(2) man page for more details. =20 Set home node for a Range of Task's Address Spacec:: @@ -497,6 +503,9 @@ closest to which page allocation will come from. Specif= ying the home node overri the default allocation policy to allocate memory close to the local node f= or an executing CPU. =20 +mbind2() also provides a way for the home node to be set at the time the +mempolicy is set. See the mbind(2) man page for more details. + Extended Mempolicy Arguments:: =20 struct mpol_param { @@ -510,7 +519,8 @@ Extended Mempolicy Arguments:: The extended mempolicy argument structure is defined to allow the mempolicy interfaces future extensibility without the need for additional system cal= ls. =20 -Extended interfaces (set_mempolicy2 and get_mempolicy2) use this structure. +Extended interfaces (set_mempolicy2, get_mempolicy2, and mbind2) use this +this argument structure. =20 The core arguments (mode, mode_flags, pol_nodes, and pol_maxnodes) apply to all interfaces relative to their non-extended counterparts. Each additional diff --git a/arch/alpha/kernel/syscalls/syscall.tbl b/arch/alpha/kernel/sys= calls/syscall.tbl index 0301a8b0a262..e8239293c35a 100644 --- a/arch/alpha/kernel/syscalls/syscall.tbl +++ b/arch/alpha/kernel/syscalls/syscall.tbl @@ -498,3 +498,4 @@ 566 common futex_requeue sys_futex_requeue 567 common set_mempolicy2 sys_set_mempolicy2 568 common get_mempolicy2 sys_get_mempolicy2 +569 common mbind2 sys_mbind2 diff --git a/arch/arm/tools/syscall.tbl b/arch/arm/tools/syscall.tbl index 771a33446e8e..a3f39750257a 100644 --- a/arch/arm/tools/syscall.tbl +++ b/arch/arm/tools/syscall.tbl @@ -472,3 +472,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/arch/arm64/include/asm/unistd.h b/arch/arm64/include/asm/unist= d.h index b63f870debaf..abe10a833fcd 100644 --- a/arch/arm64/include/asm/unistd.h +++ b/arch/arm64/include/asm/unistd.h @@ -39,7 +39,7 @@ #define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5) #define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800) =20 -#define __NR_compat_syscalls 459 +#define __NR_compat_syscalls 460 #endif =20 #define __ARCH_WANT_SYS_CLONE diff --git a/arch/arm64/include/asm/unistd32.h b/arch/arm64/include/asm/uni= std32.h index f8d01007aee0..446b7f034332 100644 --- a/arch/arm64/include/asm/unistd32.h +++ b/arch/arm64/include/asm/unistd32.h @@ -923,6 +923,8 @@ __SYSCALL(__NR_futex_requeue, sys_futex_requeue) __SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2) #define __NR_get_mempolicy2 458 __SYSCALL(__NR_get_mempolicy2, sys_get_mempolicy2) +#define __NR_mbind2 459 +__SYSCALL(__NR_mbind2, sys_mbind2) =20 /* * Please add new compat syscalls above this comment and update diff --git a/arch/m68k/kernel/syscalls/syscall.tbl b/arch/m68k/kernel/sysca= lls/syscall.tbl index 048a409e684c..9a12dface18e 100644 --- a/arch/m68k/kernel/syscalls/syscall.tbl +++ b/arch/m68k/kernel/syscalls/syscall.tbl @@ -458,3 +458,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/arch/microblaze/kernel/syscalls/syscall.tbl b/arch/microblaze/= kernel/syscalls/syscall.tbl index 327b01bd6793..6cb740123137 100644 --- a/arch/microblaze/kernel/syscalls/syscall.tbl +++ b/arch/microblaze/kernel/syscalls/syscall.tbl @@ -464,3 +464,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/arch/mips/kernel/syscalls/syscall_n32.tbl b/arch/mips/kernel/s= yscalls/syscall_n32.tbl index 921d58e1da23..52cf720f8ae2 100644 --- a/arch/mips/kernel/syscalls/syscall_n32.tbl +++ b/arch/mips/kernel/syscalls/syscall_n32.tbl @@ -397,3 +397,4 @@ 456 n32 futex_requeue sys_futex_requeue 457 n32 set_mempolicy2 sys_set_mempolicy2 458 n32 get_mempolicy2 sys_get_mempolicy2 +459 n32 mbind2 sys_mbind2 diff --git a/arch/mips/kernel/syscalls/syscall_o32.tbl b/arch/mips/kernel/s= yscalls/syscall_o32.tbl index 9271c83c9993..fd37c5301a48 100644 --- a/arch/mips/kernel/syscalls/syscall_o32.tbl +++ b/arch/mips/kernel/syscalls/syscall_o32.tbl @@ -446,3 +446,4 @@ 456 o32 futex_requeue sys_futex_requeue 457 o32 set_mempolicy2 sys_set_mempolicy2 458 o32 get_mempolicy2 sys_get_mempolicy2 +459 o32 mbind2 sys_mbind2 diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/s= yscalls/syscall.tbl index 0654f3f89fc7..fcd67bc405b1 100644 --- a/arch/parisc/kernel/syscalls/syscall.tbl +++ b/arch/parisc/kernel/syscalls/syscall.tbl @@ -457,3 +457,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel= /syscalls/syscall.tbl index ac11d2064e7a..89715417014c 100644 --- a/arch/powerpc/kernel/syscalls/syscall.tbl +++ b/arch/powerpc/kernel/syscalls/syscall.tbl @@ -545,3 +545,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/arch/s390/kernel/syscalls/syscall.tbl b/arch/s390/kernel/sysca= lls/syscall.tbl index 1cdcafe1ccca..c8304e0d0aa7 100644 --- a/arch/s390/kernel/syscalls/syscall.tbl +++ b/arch/s390/kernel/syscalls/syscall.tbl @@ -461,3 +461,4 @@ 456 common futex_requeue sys_futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 sys_mbind2 diff --git a/arch/sh/kernel/syscalls/syscall.tbl b/arch/sh/kernel/syscalls/= syscall.tbl index f71742024c29..e5c51b6c367f 100644 --- a/arch/sh/kernel/syscalls/syscall.tbl +++ b/arch/sh/kernel/syscalls/syscall.tbl @@ -461,3 +461,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/arch/sparc/kernel/syscalls/syscall.tbl b/arch/sparc/kernel/sys= calls/syscall.tbl index 2fbf5dbe0620..74527f585500 100644 --- a/arch/sparc/kernel/syscalls/syscall.tbl +++ b/arch/sparc/kernel/syscalls/syscall.tbl @@ -504,3 +504,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscal= ls/syscall_32.tbl index 0af813b9a118..be2e2aa17dd8 100644 --- a/arch/x86/entry/syscalls/syscall_32.tbl +++ b/arch/x86/entry/syscalls/syscall_32.tbl @@ -463,3 +463,4 @@ 456 i386 futex_requeue sys_futex_requeue 457 i386 set_mempolicy2 sys_set_mempolicy2 458 i386 get_mempolicy2 sys_get_mempolicy2 +459 i386 mbind2 sys_mbind2 diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscal= ls/syscall_64.tbl index 0b777876fc15..6e2347eb8773 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -380,6 +380,7 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 =20 # # Due to a historical design error, certain syscalls are numbered differen= tly diff --git a/arch/xtensa/kernel/syscalls/syscall.tbl b/arch/xtensa/kernel/s= yscalls/syscall.tbl index 4536c9a4227d..f00a21317dc0 100644 --- a/arch/xtensa/kernel/syscalls/syscall.tbl +++ b/arch/xtensa/kernel/syscalls/syscall.tbl @@ -429,3 +429,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index c4dc5069bae7..02f5c1e94ae5 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -817,6 +817,9 @@ asmlinkage long sys_mbind(unsigned long start, unsigned= long len, const unsigned long __user *nmask, unsigned long maxnode, unsigned flags); +asmlinkage long sys_mbind2(unsigned long start, unsigned long len, + const struct mpol_param __user *param, size_t usize, + unsigned long flags); asmlinkage long sys_get_mempolicy(int __user *policy, unsigned long __user *nmask, unsigned long maxnode, diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/u= nistd.h index 719accc731db..cd31599bb9cc 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -832,9 +832,11 @@ __SYSCALL(__NR_futex_requeue, sys_futex_requeue) __SYSCALL(__NR_set_mempolicy2, sys_set_mempolicy2) #define __NR_get_mempolicy2 458 __SYSCALL(__NR_get_mempolicy2, sys_get_mempolicy2) +#define __NR_mbind2 459 +__SYSCALL(__NR_mbind2, sys_mbind2) =20 #undef __NR_syscalls -#define __NR_syscalls 459 +#define __NR_syscalls 460 =20 /* * 32 bit systems traditionally used different diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h index 109788c8be92..7c7c384479fc 100644 --- a/include/uapi/linux/mempolicy.h +++ b/include/uapi/linux/mempolicy.h @@ -53,13 +53,14 @@ struct mpol_param { #define MPOL_F_ADDR (1<<1) /* look up vma using address */ #define MPOL_F_MEMS_ALLOWED (1<<2) /* return allowed memories */ =20 -/* Flags for mbind */ +/* Flags for mbind/mbind2 */ #define MPOL_MF_STRICT (1<<0) /* Verify existing pages in the mapping */ #define MPOL_MF_MOVE (1<<1) /* Move pages owned by this process to conform to policy */ #define MPOL_MF_MOVE_ALL (1<<2) /* Move every page to conform to policy */ #define MPOL_MF_LAZY (1<<3) /* UNSUPPORTED FLAG: Lazy migrate on fault */ -#define MPOL_MF_INTERNAL (1<<4) /* Internal flags start here */ +#define MPOL_MF_HOME_NODE (1<<4) /* mbind2: set home node */ +#define MPOL_MF_INTERNAL (1<<5) /* Internal flags start here */ =20 #define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \ diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index 6afbd3a41319..2483b5afa99f 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -187,6 +187,7 @@ COND_SYSCALL(process_madvise); COND_SYSCALL(process_mrelease); COND_SYSCALL(remap_file_pages); COND_SYSCALL(mbind); +COND_SYSCALL(mbind2); COND_SYSCALL(get_mempolicy); COND_SYSCALL(get_mempolicy2); COND_SYSCALL(set_mempolicy); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 0b2e31d8636d..53301e173c90 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1612,6 +1612,49 @@ SYSCALL_DEFINE6(mbind, unsigned long, start, unsigne= d long, len, return kernel_mbind(start, len, mode, nmask, maxnode, flags); } =20 +SYSCALL_DEFINE5(mbind2, unsigned long, start, unsigned long, len, + const struct mpol_param __user *, uparam, size_t, usize, + unsigned long, flags) +{ + struct mpol_param kparam; + struct mempolicy_param mparam; + nodemask_t policy_nodes; + unsigned long __user *nodes_ptr; + int err; + + if (!start || !len) + return -EINVAL; + + err =3D copy_struct_from_user(&kparam, sizeof(kparam), uparam, usize); + if (err) + return -EINVAL; + + err =3D validate_mpol_flags(kparam.mode, &kparam.mode_flags); + if (err) + return err; + + mparam.mode =3D kparam.mode; + mparam.mode_flags =3D kparam.mode_flags; + + /* if home node given, validate it is online */ + if (flags & MPOL_MF_HOME_NODE) { + if ((kparam.home_node >=3D MAX_NUMNODES) || + !node_online(kparam.home_node)) + return -EINVAL; + mparam.home_node =3D kparam.home_node; + } else + mparam.home_node =3D NUMA_NO_NODE; + flags &=3D ~MPOL_MF_HOME_NODE; + + nodes_ptr =3D u64_to_user_ptr(kparam.pol_nodes); + err =3D get_nodes(&policy_nodes, nodes_ptr, kparam.pol_maxnodes); + if (err) + return err; + mparam.policy_nodes =3D &policy_nodes; + + return do_mbind(untagged_addr(start), len, &mparam, flags); +} + /* Set the process memory policy */ static long kernel_set_mempolicy(int mode, const unsigned long __user *nma= sk, unsigned long maxnode) diff --git a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl b/tools/pe= rf/arch/mips/entry/syscalls/syscall_n64.tbl index c34c6877379e..4fd9f742d903 100644 --- a/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl +++ b/tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl @@ -373,3 +373,4 @@ 456 n64 futex_requeue sys_futex_requeue 457 n64 set_mempolicy2 sys_set_mempolicy2 458 n64 get_mempolicy2 sys_get_mempolicy2 +459 n64 mbind2 sys_mbind2 diff --git a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl b/tools/per= f/arch/powerpc/entry/syscalls/syscall.tbl index ac11d2064e7a..89715417014c 100644 --- a/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl +++ b/tools/perf/arch/powerpc/entry/syscalls/syscall.tbl @@ -545,3 +545,4 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 diff --git a/tools/perf/arch/s390/entry/syscalls/syscall.tbl b/tools/perf/a= rch/s390/entry/syscalls/syscall.tbl index 1cdcafe1ccca..c8304e0d0aa7 100644 --- a/tools/perf/arch/s390/entry/syscalls/syscall.tbl +++ b/tools/perf/arch/s390/entry/syscalls/syscall.tbl @@ -461,3 +461,4 @@ 456 common futex_requeue sys_futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 sys_mbind2 diff --git a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl b/tools/perf= /arch/x86/entry/syscalls/syscall_64.tbl index edf338f32645..3fc74241da5d 100644 --- a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl +++ b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl @@ -380,6 +380,7 @@ 456 common futex_requeue sys_futex_requeue 457 common set_mempolicy2 sys_set_mempolicy2 458 common get_mempolicy2 sys_get_mempolicy2 +459 common mbind2 sys_mbind2 =20 # # Due to a historical design error, certain syscalls are numbered differen= tly --=20 2.39.1 From nobody Fri Dec 19 09:47:54 2025 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35AC121375; Wed, 3 Jan 2024 22:43:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mWIyoQM/" Received: by mail-pg1-f195.google.com with SMTP id 41be03b00d2f7-5c229dabbb6so3045456a12.0; Wed, 03 Jan 2024 14:43:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704321811; x=1704926611; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RTh6xtZGVzP3PdSvCf1bvrsloFYLRrQIaYq034Jk50c=; b=mWIyoQM/MmPyjyOnYICtxyFjABf2yl28me2enOwWU3ljEAM+XJnHAlrRdwSZLDKMV1 k4argNxhpwbhLp/pxdKBhoHazDDKTqf5EDdRBawTG76M6Deb25IfRv7h0INKnhdIEFZV Mwa2ust0W5Fi9TT7Za9Blj3BNwqZj+TcIt1VRCMTtVcJYLzXcA7X2vyPmtLqmvFL8+Lz RIi89z3In4frG80IJm71hjt9nB7dnYHql6F+J5+cuYTYDdj9adulboj8PLxc9pXtOb0P nIKbzy2wBmE3F+QKCEoQ6ymsB+hH8Fc1u5Ucvxh4/bF14upN0wuCfkJbK9qNpro9/djG M6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704321811; x=1704926611; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RTh6xtZGVzP3PdSvCf1bvrsloFYLRrQIaYq034Jk50c=; b=JYXpsE/t85Z+VKmJMKFhAO+x/bM/cE3WHr1Rp+eMHTCrC6lui+wBygCtox+otK4n6a 2fIzQgXjUp6D92Nzx6FJUPlsVDYpamniLc4eBJR4OZI1iyIm08+qhyetPnvsdaAwpJaE uqM+fxwlGhQGX0nPwbKE3c6nWUPBFFmrUND7TNqlor8iPvwBs3GbePQ7J/Ag84n3Bmfa mjbJLpLTR8dX5acnZqkBRcw/L2tVCHOvohOL8aDxs6Ty2lVLbrMwKCXHecYQMIXGFohE z1fyWQAssdHp599aaK5tp7fZ1YNWEywSIUxV/JDng8UNw/wMTY5KkulMOsRnRXL/gxCP Yf6A== X-Gm-Message-State: AOJu0Yy/TH0LWI6vGw941C2Wqyuxr+0DqixrNazzRUvEZOOgsnxELIVV fXGytkxV9xt+A4WMkgUaGe90dT1nuGDq8jM= X-Google-Smtp-Source: AGHT+IHj4IdnrLjJKicXSl2ez9+yiEio1lEnIuJiRk/U8wWoKnVHnsybkLZIFe48jEEc9NQFScbm2w== X-Received: by 2002:a17:902:8b8a:b0:1d0:6ffe:1e6f with SMTP id ay10-20020a1709028b8a00b001d06ffe1e6fmr8043944plb.82.1704321811547; Wed, 03 Jan 2024 14:43:31 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id g1-20020a170902fe0100b001d36df58ba2sm24269426plj.308.2024.01.03.14.43.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 14:43:31 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v6 12/12] mm/mempolicy: extend mempolicy2 and mbind2 to support weighted interleave Date: Wed, 3 Jan 2024 17:42:09 -0500 Message-Id: <20240103224209.2541-13-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240103224209.2541-1-gregory.price@memverge.com> References: <20240103224209.2541-1-gregory.price@memverge.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extend mempolicy2 and mbind2 to support weighted interleave, and demonstrate the extensibility of the mpol_param structure. To support weighted interleave we add interleave weight fields to the following structures: Kernel Internal: (include/linux/mempolicy.h) struct mempolicy { /* task-local weights to apply to weighted interleave */ u8 weights[MAX_NUMNODES]; } struct mempolicy_param { /* Optional: interleave weights for MPOL_WEIGHTED_INTERLEAVE */ u8 *il_weights; /* of size MAX_NUMNODES */ } UAPI: (/include/uapi/linux/mempolicy.h) struct mpol_param { /* Optional: interleave weights for MPOL_WEIGHTED_INTERLEAVE */ __u8 *il_weights; /* of size pol_maxnodes */ } The minimum weight of a node is always 1. If the user desires 0 allocations on a node, the node should be removed from the nodemask. If the user does not provide weights (il_weights =3D=3D NULL), global weights will be used during allocation. Changes made to global weights will be reflected in future allocations. If the user provides weights and a weight is set to 0, the weight for that node will be initialized to the global value. If a user provides weights and a node is not set in the node mask, the weight for that node will be set to the globally defined weight. This is so a reasonable default value can be expected if the nodemask changes (e.g. cgroups causes a migration or mems_allowed change). Local weights are never updated when a global weight is updated. Examples: global weights: [4,4,2,2] Set: Nodes-0,1,2,3 Weights: NULL [global weights] are used. Set: Nodes-0,1,2,3 Weights: [1,2,3,4] local_weights =3D [1,2,3,4] Set: Nodes-0,2 Weights: [2,0,2,0] local_weights =3D [2,4,1,2] Basic logic during allocation is as follows: weight =3D pol->wil.weights[node] /* if no local weight, use sysfs weight */ if (!weight) weight =3D iw_table[weight] /* if no sysfs weight, use system default */ if (!weight) weight =3D default_iw_table[weight] To simplify creations and duplication of mempolicies, the weights are added as a structure directly within mempolicy. This allows the existing logic in __mpol_dup to copy the weights without additional allocations: if (old =3D=3D current->mempolicy) { task_lock(current); *new =3D *old; task_unlock(current); } else *new =3D *old Suggested-by: Rakie Kim Suggested-by: Hyeongtak Ji Suggested-by: Honggyu Kim Suggested-by: Vinicius Tavares Petrucci Suggested-by: Huang Ying Signed-off-by: Gregory Price Co-developed-by: Rakie Kim Signed-off-by: Rakie Kim Co-developed-by: Hyeongtak Ji Signed-off-by: Hyeongtak Ji Co-developed-by: Honggyu Kim Signed-off-by: Honggyu Kim Co-developed-by: Vinicius Tavares Petrucci Signed-off-by: Vinicius Tavares Petrucci Suggested-by: Dan Williams Suggested-by: Frank van der Linden Suggested-by: Gregory Price Suggested-by: Hao Wang Suggested-by: Hasan Al Maruf Suggested-by: Johannes Weiner Suggested-by: John Groves Suggested-by: Jonathan Cameron Suggested-by: Michal Hocko Suggested-by: Ravi Jonnalagadda Suggested-by: Srinivasulu Thanneeru Suggested-by: Ying Huang Suggested-by: Zhongkun He Suggested-by: tj --- .../admin-guide/mm/numa_memory_policy.rst | 12 ++ include/linux/mempolicy.h | 2 + include/uapi/linux/mempolicy.h | 1 + mm/mempolicy.c | 134 ++++++++++++++++-- 4 files changed, 141 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Document= ation/admin-guide/mm/numa_memory_policy.rst index 66a778d58899..620b54ff2cef 100644 --- a/Documentation/admin-guide/mm/numa_memory_policy.rst +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -254,11 +254,22 @@ MPOL_WEIGHTED_INTERLEAVE This mode operates the same as MPOL_INTERLEAVE, except that interleaving behavior is executed based on weights set in /sys/kernel/mm/mempolicy/weighted_interleave/ + when configured to utilize global weights, or based on task-local + weights configured with set_mempolicy2(2) or mbind2(2). =20 Weighted interleave allocates pages on nodes according to a weight. For example if nodes [0,1] are weighted [5,2], 5 pages will be allocated on node0 for every 2 pages allocated on node1. =20 + When utilizing task-local weights, if node's is not set in the + nodemask, or its weight was set to 0, the local weight will be + set to the system default. Updates to system default weights + will not be refleted in local weights. + + The minimum weight for a node set in the policy nodemask is + always 1. If no allocations on a node, the node should be + removed from the nodemask. + NUMA memory policy supports the following optional mode flags: =20 MPOL_F_STATIC_NODES @@ -514,6 +525,7 @@ Extended Mempolicy Arguments:: __s32 home_node; /* mbind2: set home node */ __u64 pol_maxnodes; __aligned_u64 pol_nodes; /* nodemask pointer */ + __aligned_u64 il_weights; /* u8 buf of size pol_maxnodes */ }; =20 The extended mempolicy argument structure is defined to allow the mempolicy diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index e6795e2d0cc2..9854790a9aac 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -58,6 +58,7 @@ struct mempolicy { /* Weighted interleave settings */ struct { u8 cur_weight; + u8 weights[MAX_NUMNODES]; u8 scratch_weights[MAX_NUMNODES]; /* Used to avoid allocations */ } wil; }; @@ -71,6 +72,7 @@ struct mempolicy_param { unsigned short mode_flags; /* policy mode flags */ int home_node; /* mbind: use MPOL_MF_HOME_NODE */ nodemask_t *policy_nodes; /* get/set/mbind */ + u8 *il_weights; /* for mode MPOL_WEIGHTED_INTERLEAVE */ }; =20 /* diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h index 7c7c384479fc..06e0fc2bb29b 100644 --- a/include/uapi/linux/mempolicy.h +++ b/include/uapi/linux/mempolicy.h @@ -34,6 +34,7 @@ struct mpol_param { __u16 pol_maxnodes; __u8 resv[6]; __aligned_u64 pol_nodes; + __aligned_u64 il_weights; /* size: pol_maxnodes * sizeof(__u8) */ }; =20 /* Flags for set_mempolicy */ diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 53301e173c90..78e7614e0cd4 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -286,6 +286,7 @@ static struct mempolicy *mpol_new(struct mempolicy_para= m *param) unsigned short mode =3D param->mode; unsigned short flags =3D param->mode_flags; nodemask_t *nodes =3D param->policy_nodes; + int node; =20 if (mode =3D=3D MPOL_DEFAULT) { if (nodes && !nodes_empty(*nodes)) @@ -323,6 +324,23 @@ static struct mempolicy *mpol_new(struct mempolicy_par= am *param) policy->flags =3D flags; policy->home_node =3D param->home_node; policy->wil.cur_weight =3D 0; + memset(policy->wil.weights, 0, MAX_NUMNODES); + + /* If user provides weights, ensure all weights are set to something */ + if (policy->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE && param->il_weights) { + for (node =3D 0; node < MAX_NUMNODES; node++) { + u8 weight =3D 0; + + if (node_isset(node, *nodes)) + weight =3D param->il_weights[node]; + /* If a user sets a weight to 0, use global default */ + if (!weight) + weight =3D iw_table[node]; + if (!weight) + weight =3D default_iw_table[node]; + policy->wil.weights[node] =3D weight; + } + } =20 return policy; } @@ -952,6 +970,26 @@ static void do_get_mempolicy_nodemask(struct mempolicy= *pol, nodemask_t *nmask) } } =20 +static void do_get_mempolicy_il_weights(struct mempolicy *pol, + u8 weights[MAX_NUMNODES]) +{ + int i =3D 0; + + if (pol->mode !=3D MPOL_WEIGHTED_INTERLEAVE) { + memset(weights, 0, MAX_NUMNODES); + return; + } + for (i =3D 0; i < MAX_NUMNODES; i++) { + u8 weight =3D pol->wil.weights[i]; + + if (!weight) + weight =3D iw_table[i]; + if (!weight) + weight =3D default_iw_table[i]; + weights[i] =3D weight; + } +} + /* Retrieve NUMA policy for a VMA assocated with a given address */ static long do_get_vma_mempolicy(unsigned long addr, int *addr_node, struct mempolicy_param *param) @@ -985,6 +1023,9 @@ static long do_get_vma_mempolicy(unsigned long addr, i= nt *addr_node, if (param->policy_nodes) do_get_mempolicy_nodemask(pol, param->policy_nodes); =20 + if (param->il_weights) + do_get_mempolicy_il_weights(pol, param->il_weights); + if (pol !=3D &default_policy) { mpol_put(pol); mpol_cond_put(pol); @@ -1012,6 +1053,9 @@ static long do_get_task_mempolicy(struct mempolicy_pa= ram *param, int *pol_node) if (param->policy_nodes) do_get_mempolicy_nodemask(pol, param->policy_nodes); =20 + if (param->il_weights) + do_get_mempolicy_il_weights(pol, param->il_weights); + return 0; } =20 @@ -1620,6 +1664,8 @@ SYSCALL_DEFINE5(mbind2, unsigned long, start, unsigne= d long, len, struct mempolicy_param mparam; nodemask_t policy_nodes; unsigned long __user *nodes_ptr; + u8 *weights =3D NULL; + u8 __user *weights_ptr; int err; =20 if (!start || !len) @@ -1652,7 +1698,27 @@ SYSCALL_DEFINE5(mbind2, unsigned long, start, unsign= ed long, len, return err; mparam.policy_nodes =3D &policy_nodes; =20 - return do_mbind(untagged_addr(start), len, &mparam, flags); + if (kparam.mode =3D=3D MPOL_WEIGHTED_INTERLEAVE) { + weights_ptr =3D u64_to_user_ptr(kparam.il_weights); + if (weights_ptr) { + weights =3D kzalloc(MAX_NUMNODES, + GFP_KERNEL | __GFP_NORETRY); + if (!weights) + return -ENOMEM; + err =3D copy_struct_from_user(weights, + MAX_NUMNODES, + weights_ptr, + kparam.pol_maxnodes); + if (err) + goto leave_weights; + } + } + mparam.il_weights =3D weights; + + err =3D do_mbind(untagged_addr(start), len, &mparam, flags); +leave_weights: + kfree(weights); + return err; } =20 /* Set the process memory policy */ @@ -1696,6 +1762,8 @@ SYSCALL_DEFINE3(set_mempolicy2, struct mpol_param __u= ser *, uparam, int err; nodemask_t policy_nodemask; unsigned long __user *nodes_ptr; + u8 *weights =3D NULL; + u8 __user *weights_ptr; =20 if (flags) return -EINVAL; @@ -1721,7 +1789,24 @@ SYSCALL_DEFINE3(set_mempolicy2, struct mpol_param __= user *, uparam, } else mparam.policy_nodes =3D NULL; =20 - return do_set_mempolicy(&mparam); + if (kparam.mode =3D=3D MPOL_WEIGHTED_INTERLEAVE && kparam.il_weights) { + weights =3D kzalloc(MAX_NUMNODES, GFP_KERNEL | __GFP_NORETRY); + if (!weights) + return -ENOMEM; + weights_ptr =3D u64_to_user_ptr(kparam.il_weights); + err =3D copy_struct_from_user(weights, + MAX_NUMNODES, + weights_ptr, + kparam.pol_maxnodes); + if (err) + goto leave_weights; + } + mparam.il_weights =3D weights; + + err =3D do_set_mempolicy(&mparam); +leave_weights: + kfree(weights); + return err; } =20 static int kernel_migrate_pages(pid_t pid, unsigned long maxnode, @@ -1924,6 +2009,8 @@ SYSCALL_DEFINE4(get_mempolicy2, struct mpol_param __u= ser *, uparam, size_t, usiz int err; nodemask_t policy_nodemask; unsigned long __user *nodes_ptr; + u8 __user *weights_ptr; + u8 *weights =3D NULL; =20 if (flags & ~(MPOL_F_ADDR)) return -EINVAL; @@ -1935,6 +2022,13 @@ SYSCALL_DEFINE4(get_mempolicy2, struct mpol_param __= user *, uparam, size_t, usiz if (err) return -EINVAL; =20 + if (kparam.il_weights) { + weights =3D kzalloc(MAX_NUMNODES, GFP_KERNEL | __GFP_NORETRY); + if (!weights) + return -ENOMEM; + } + mparam.il_weights =3D weights; + mparam.policy_nodes =3D kparam.pol_nodes ? &policy_nodemask : NULL; if (flags & MPOL_F_ADDR) err =3D do_get_vma_mempolicy(untagged_addr(addr), NULL, &mparam); @@ -1942,7 +2036,7 @@ SYSCALL_DEFINE4(get_mempolicy2, struct mpol_param __u= ser *, uparam, size_t, usiz err =3D do_get_task_mempolicy(&mparam, NULL); =20 if (err) - return err; + goto leave_weights; =20 kparam.mode =3D mparam.mode; kparam.mode_flags =3D mparam.mode_flags; @@ -1952,10 +2046,21 @@ SYSCALL_DEFINE4(get_mempolicy2, struct mpol_param _= _user *, uparam, size_t, usiz err =3D copy_nodes_to_user(nodes_ptr, kparam.pol_maxnodes, mparam.policy_nodes); if (err) - return err; + goto leave_weights; + } + + if (kparam.mode =3D=3D MPOL_WEIGHTED_INTERLEAVE && kparam.il_weights) { + weights_ptr =3D u64_to_user_ptr(kparam.il_weights); + if (copy_to_user(weights_ptr, weights, kparam.pol_maxnodes)) { + err =3D -EFAULT; + goto leave_weights; + } } =20 - return copy_to_user(uparam, &kparam, usize) ? -EFAULT : 0; + err =3D copy_to_user(uparam, &kparam, usize) ? -EFAULT : 0; +leave_weights: + kfree(weights); + return err; } =20 bool vma_migratable(struct vm_area_struct *vma) @@ -2077,8 +2182,10 @@ static unsigned int weighted_interleave_nodes(struct= mempolicy *policy) return next; =20 if (!policy->wil.cur_weight) { - u8 next_weight =3D iw_table[next]; + u8 next_weight =3D policy->wil.weights[next]; =20 + if (!next_weight) + next_weight =3D iw_table[next]; if (!next_weight) next_weight =3D default_iw_table[next]; policy->wil.cur_weight =3D next_weight; @@ -2175,8 +2282,10 @@ static unsigned int read_once_interleave_weights(str= uct mempolicy *pol, /* Similar issue to read_once_policy_nodemask */ barrier(); for_each_node_mask(nid, *mask) { - u8 weight =3D iw_table[nid]; + u8 weight =3D pol->wil.weights[nid]; =20 + if (!weight) + weight =3D iw_table[nid]; if (!weight) weight =3D default_iw_table[nid]; weight_total +=3D weight; @@ -3115,21 +3224,28 @@ void mpol_shared_policy_init(struct shared_policy *= sp, struct mempolicy *mpol) if (mpol) { struct sp_node *sn; struct mempolicy *npol; + u8 *weights =3D NULL; NODEMASK_SCRATCH(scratch); =20 if (!scratch) goto put_mpol; =20 + weights =3D kzalloc(MAX_NUMNODES, GFP_KERNEL | __GFP_NORETRY); + if (!weights) + goto free_scratch; + memcpy(weights, mpol->wil.weights, sizeof(weights)); + memset(&mparam, 0, sizeof(mparam)); mparam.mode =3D mpol->mode; mparam.mode_flags =3D mpol->flags; mparam.policy_nodes =3D &mpol->w.user_nodemask; mparam.home_node =3D NUMA_NO_NODE; + mparam.il_weights =3D weights; =20 /* contextualize the tmpfs mount point mempolicy to this file */ npol =3D mpol_new(&mparam); if (IS_ERR(npol)) - goto free_scratch; /* no valid nodemask intersection */ + goto free_weights; /* no valid nodemask intersection */ =20 task_lock(current); ret =3D mpol_set_nodemask(npol, &mpol->w.user_nodemask, scratch); @@ -3143,6 +3259,8 @@ void mpol_shared_policy_init(struct shared_policy *sp= , struct mempolicy *mpol) sp_insert(sp, sn); put_npol: mpol_put(npol); /* drop initial ref on file's npol */ +free_weights: + kfree(weights); free_scratch: NODEMASK_SCRATCH_FREE(scratch); put_mpol: --=20 2.39.1