From nobody Fri Dec 19 13:14:22 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6E89C10F05 for ; Sat, 9 Dec 2023 07:00:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234404AbjLIHAG (ORCPT ); Sat, 9 Dec 2023 02:00:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234422AbjLIG7q (ORCPT ); Sat, 9 Dec 2023 01:59:46 -0500 Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 129A9172A; Fri, 8 Dec 2023 22:59:46 -0800 (PST) Received: by mail-qk1-x744.google.com with SMTP id af79cd13be357-77f552d4179so62881385a.1; Fri, 08 Dec 2023 22:59:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702105185; x=1702709985; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HyO37uEt2s7CfDc9j6MDVN1pIs7702j+Cd0isrN3eeg=; b=NNeaphrkJL/NzbDz5k2bPGGJP/pV3lk3HsAxNGJ9m5Ax14lKzonV2uY95IHmigQXOV f+VEql9zfjrkE8ADdgCiGOnAfyrp1HnXh65bEZK8Z6FxI+WBlzVJPUHMN5706k6YA9TC gC3YG7sBnDChRp9eBNJXRRpz1UC0XX7+gG01MnBMQ+x1WKwZDuPVrqriJUZJ+td+icld 3dt7t22Ev/j8BFwOGpwpaSi9xG+25FP98kcgiKP7GHnfzsP0Cy1sAk0+X2nNqd8VhXGR 0hu1Du1ahcuwJcS/goTzgj1oPnCL691IeQt16ps4beeYBYrxCmoBDccHdeT/dP+Rjxcr XSyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702105185; x=1702709985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HyO37uEt2s7CfDc9j6MDVN1pIs7702j+Cd0isrN3eeg=; b=QuTBth7R9lZo6DUb/0VhIT+M9LzvWGoAwpagTytX7G+bWuQn6JI7GglNY2E6kJq/iq zxVaOa+AQcRCfFiVJxiNdZVYvW/JgB6/UUIkAlVs4PTzWVy4zvEd8QhcCv42E6orsa4t Lj4fNKAVQAeBzwpKXhdjY/DWuTW7tthWkqKHfntSJQwRLX/8u+4d3p99NT91a2E7CCE6 IzjMfj9RwKZ0EWW/oXliwu1oPJclh22xtI8nWSxRyvdAhCvxWM5zZDInfAaMRP+UkjMY 5W6YG1WTsssM/5pnrkMUW1vUsMRzzNjEEc18QR18CGSkchQbG8w3EkgZp8Gaccxb7jql cOJw== X-Gm-Message-State: AOJu0YwBdzPrMDvMgxQCS+hJMpn8ZPuRxjr+gIghDJkULE02PS+0InrZ meSLMilL5cduoOWR6O9Lyw== X-Google-Smtp-Source: AGHT+IFUsOfBH9W6yp5ojwYyZpUt8WzKjfVdhDUOYCzr9GQ8Voczka7QjJSa/kK3LqyrPGhITVuFVw== X-Received: by 2002:a37:ad0a:0:b0:77e:fba3:937b with SMTP id f10-20020a37ad0a000000b0077efba3937bmr1328328qkm.93.1702105184733; Fri, 08 Dec 2023 22:59:44 -0800 (PST) Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id x8-20020a81b048000000b005df5d592244sm326530ywk.78.2023.12.08.22.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 22:59:44 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, arnd@arndb.de, tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, mhocko@kernel.org, tj@kernel.org, ying.huang@intel.com, gregory.price@memverge.com, corbet@lwn.net, rakie.kim@sk.com, hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com, peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com Subject: [PATCH v2 05/11] mm/mempolicy: refactor kernel_get_mempolicy for code re-use Date: Sat, 9 Dec 2023 01:59:25 -0500 Message-Id: <20231209065931.3458-6-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231209065931.3458-1-gregory.price@memverge.com> References: <20231209065931.3458-1-gregory.price@memverge.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Pull operation flag checking from inside do_get_mempolicy out to kernel_get_mempolicy. This allows us to flatten the internal code, and break it into separate functions for future syscalls (get_mempolicy2, process_get_mempolicy) to re-use the code, even after additional extensions are made. The primary change is that the flag is treated as the multiplexer that it actually is. For get_mempolicy, the flags represents 3 different primary operations: if (flags & MPOL_F_MEMS_ALLOWED) return task->mems_allowed else if (flags & MPOL_F_ADDR) return vma mempolicy information else return task mempolicy information Plus the behavior modifying flag: if (flags & MPOL_F_NODE) change the return value of (int __user *policy) based on whether MPOL_F_ADDR was set. The original behavior of get_mempolicy is retained, but we utilize the new mempolicy_args structure to pass the operations down the stack. This will allow us to extend the internal functions without affecting the legacy behavior of get_mempolicy. Signed-off-by: Gregory Price --- mm/mempolicy.c | 240 ++++++++++++++++++++++++++++++------------------- 1 file changed, 150 insertions(+), 90 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 324dbf1782df..ce5b7963e9b5 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -895,106 +895,107 @@ static int lookup_node(struct mm_struct *mm, unsign= ed long addr) return ret; } =20 -/* Retrieve NUMA policy */ -static long do_get_mempolicy(int *policy, nodemask_t *nmask, - unsigned long addr, unsigned long flags) +/* Retrieve the mems_allowed for current task */ +static inline long do_get_mems_allowed(nodemask_t *nmask) { - int err; - struct mm_struct *mm =3D current->mm; - struct vm_area_struct *vma =3D NULL; - struct mempolicy *pol =3D current->mempolicy, *pol_refcount =3D NULL; + task_lock(current); + *nmask =3D cpuset_current_mems_allowed; + task_unlock(current); + return 0; +} =20 - if (flags & - ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED)) - return -EINVAL; +/* If the policy has additional node information to retrieve, return it */ +static long do_get_policy_node(struct mempolicy *pol) +{ + /* + * For MPOL_INTERLEAVE, the extended node information is the next + * node that will be selected for interleave. For weighted interleave + * we return the next node based on the current weight. + */ + if (pol =3D=3D current->mempolicy && pol->mode =3D=3D MPOL_INTERLEAVE) + return next_node_in(current->il_prev, pol->nodes); =20 - if (flags & MPOL_F_MEMS_ALLOWED) { - if (flags & (MPOL_F_NODE|MPOL_F_ADDR)) - return -EINVAL; - *policy =3D 0; /* just so it's initialized */ + if (pol =3D=3D current->mempolicy && + pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE) { + if (pol->wil.cur_weight) + return current->il_prev; + else + return next_node_in(current->il_prev, pol->nodes); + } + return -EINVAL; +} + +/* Handle user_nodemask condition when fetching nodemask for userspace */ +static void do_get_mempolicy_nodemask(struct mempolicy *pol, nodemask_t *n= mask) +{ + if (mpol_store_user_nodemask(pol)) { + *nmask =3D pol->w.user_nodemask; + } else { task_lock(current); - *nmask =3D cpuset_current_mems_allowed; + get_policy_nodemask(pol, nmask); task_unlock(current); - return 0; } +} =20 - if (flags & MPOL_F_ADDR) { - pgoff_t ilx; /* ignored here */ - /* - * Do NOT fall back to task policy if the - * vma/shared policy at addr is NULL. We - * want to return MPOL_DEFAULT in this case. - */ - mmap_read_lock(mm); - vma =3D vma_lookup(mm, addr); - if (!vma) { - mmap_read_unlock(mm); - return -EFAULT; - } - pol =3D __get_vma_policy(vma, addr, &ilx); - } else if (addr) - return -EINVAL; +/* Retrieve NUMA policy for a VMA assocated with a given address */ +static long do_get_vma_mempolicy(struct mempolicy_args *args) +{ + pgoff_t ilx; + struct mm_struct *mm =3D current->mm; + struct vm_area_struct *vma =3D NULL; + struct mempolicy *pol =3D NULL; =20 + mmap_read_lock(mm); + vma =3D vma_lookup(mm, args->addr); + if (!vma) { + mmap_read_unlock(mm); + return -EFAULT; + } + pol =3D __get_vma_policy(vma, args->addr, &ilx); if (!pol) - pol =3D &default_policy; /* indicates default behavior */ + pol =3D &default_policy; + /* this may cause a double-reference, resolved by a put+cond_put */ + mpol_get(pol); + mmap_read_unlock(mm); =20 - if (flags & MPOL_F_NODE) { - if (flags & MPOL_F_ADDR) { - /* - * Take a refcount on the mpol, because we are about to - * drop the mmap_lock, after which only "pol" remains - * valid, "vma" is stale. - */ - pol_refcount =3D pol; - vma =3D NULL; - mpol_get(pol); - mmap_read_unlock(mm); - err =3D lookup_node(mm, addr); - if (err < 0) - goto out; - *policy =3D err; - } else if (pol =3D=3D current->mempolicy && - pol->mode =3D=3D MPOL_INTERLEAVE) { - *policy =3D next_node_in(current->il_prev, pol->nodes); - } else if (pol =3D=3D current->mempolicy && - (pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE)) { - if (pol->wil.cur_weight) - *policy =3D current->il_prev; - else - *policy =3D next_node_in(current->il_prev, - pol->nodes); - } else { - err =3D -EINVAL; - goto out; - } - } else { - *policy =3D pol =3D=3D &default_policy ? MPOL_DEFAULT : - pol->mode; - /* - * Internal mempolicy flags must be masked off before exposing - * the policy to userspace. - */ - *policy |=3D (pol->flags & MPOL_MODE_FLAGS); - } + /* Fetch the node for the given address */ + args->addr_node =3D lookup_node(mm, args->addr); =20 - err =3D 0; - if (nmask) { - if (mpol_store_user_nodemask(pol)) { - *nmask =3D pol->w.user_nodemask; - } else { - task_lock(current); - get_policy_nodemask(pol, nmask); - task_unlock(current); - } + args->mode =3D pol =3D=3D &default_policy ? MPOL_DEFAULT : pol->mode; + args->mode_flags =3D (pol->flags & MPOL_MODE_FLAGS); + + /* If this policy has extra node info, fetch that */ + args->policy_node =3D do_get_policy_node(pol); + + if (args->policy_nodes) + do_get_mempolicy_nodemask(pol, args->policy_nodes); + + if (pol !=3D &default_policy) { + mpol_put(pol); + mpol_cond_put(pol); } =20 - out: - mpol_cond_put(pol); - if (vma) - mmap_read_unlock(mm); - if (pol_refcount) - mpol_put(pol_refcount); - return err; + return 0; +} + +/* Retrieve NUMA policy for the current task */ +static long do_get_task_mempolicy(struct mempolicy_args *args) +{ + struct mempolicy *pol =3D current->mempolicy; + + if (!pol) + pol =3D &default_policy; /* indicates default behavior */ + + args->mode =3D pol =3D=3D &default_policy ? MPOL_DEFAULT : pol->mode; + /* Internal flags must be masked off before exposing to userspace */ + args->mode_flags =3D (pol->flags & MPOL_MODE_FLAGS); + + args->policy_node =3D do_get_policy_node(pol); + + if (args->policy_nodes) + do_get_mempolicy_nodemask(pol, args->policy_nodes); + + return 0; } =20 #ifdef CONFIG_MIGRATION @@ -1731,16 +1732,75 @@ static int kernel_get_mempolicy(int __user *policy, unsigned long addr, unsigned long flags) { + struct mempolicy_args args; int err; - int pval; + int pval =3D 0; nodemask_t nodes; =20 if (nmask !=3D NULL && maxnode < nr_node_ids) return -EINVAL; =20 - addr =3D untagged_addr(addr); + if (flags & + ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED)) + return -EINVAL; =20 - err =3D do_get_mempolicy(&pval, &nodes, addr, flags); + /* Ensure any data that may be copied to userland is initialized */ + memset(&args, 0, sizeof(args)); + args.policy_nodes =3D &nodes; + args.addr =3D untagged_addr(addr); + + /* + * set_mempolicy was originally multiplexed based on 3 flags: + * MPOL_F_MEMS_ALLOWED: fetch task->mems_allowed + * MPOL_F_ADDR : operate on vma->mempolicy + * MPOL_F_NODE : change return value of *policy + * + * Split this behavior out here, rather than internal functions, + * so that the internal functions can be re-used by future + * get_mempolicy2 interfaces and the arg structure made extensible + */ + if (flags & MPOL_F_MEMS_ALLOWED) { + if (flags & (MPOL_F_NODE|MPOL_F_ADDR)) + return -EINVAL; + pval =3D 0; /* just so it's initialized */ + err =3D do_get_mems_allowed(&nodes); + } else if (flags & MPOL_F_ADDR) { + /* If F_ADDR, we operation on a vma policy (or default) */ + err =3D do_get_vma_mempolicy(&args); + if (err) + return err; + /* if (F_ADDR | F_NODE), *pval is the address' node */ + if (flags & MPOL_F_NODE) { + /* if we failed to fetch, that's likely an EFAULT */ + if (args.addr_node < 0) + return args.addr_node; + pval =3D args.addr_node; + } else + pval =3D args.mode | args.mode_flags; + } else { + /* if not F_ADDR and addr !=3D null, EINVAL */ + if (addr) + return -EINVAL; + + err =3D do_get_task_mempolicy(&args); + if (err) + return err; + /* + * if F_NODE was set and mode was MPOL_INTERLEAVE + * *pval is equal to next interleave node. + * + * if args.policy_node < 0, this means the mode did + * not have a policy. This presently emulates the + * original behavior of (F_NODE) & (!MPOL_INTERLEAVE) + * producing -EINVAL + */ + if (flags & MPOL_F_NODE) { + if (args.policy_node < 0) + return args.policy_node; + pval =3D args.policy_node; + } else + pval =3D args.mode | args.mode_flags; + } =20 if (err) return err; --=20 2.39.1