From nobody Wed Dec 17 12:48:32 2025 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABF96237A5E for ; Tue, 10 Dec 2024 21:38:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866694; cv=none; b=pDq6jw0258pQVMYnKGNlQATw0o7ZsvIh7dBSQG8oMGG2CqjXxlTxCIYa3weK2gmyFPqi2ahUTj/7jsLLocX87rkRK2IQ1Pj4GXby9yzedojfpytcW7YyUb938hExBprrb3d2bpWAA5H7/8gmqG4pJ46jGCU2EdetKFcTR6p8d3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866694; c=relaxed/simple; bh=4ESIYq/H6qTs9FLOoJEIeCHtazL9GkHE2RwwJuHRH7M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bYjrZ/TRZ24qF2IC0/TpgLfhLde+oB8mEsnXpbTBoY9y6CVsWytA5Bq6047uu3KzvXpMng6gjr1ZpivatPJ3Oz5l4P/GgVWOeu5ANScbI/ylAABZPJShgmHjH9AdlD600o4F9OLoko61YuLMHomnOIHsyskK/EHNRmqg10XVEpE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=qo9co1FA; arc=none smtp.client-ip=209.85.160.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="qo9co1FA" Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-46760d9e4daso22873011cf.0 for ; Tue, 10 Dec 2024 13:38:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1733866691; x=1734471491; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OQZEugtHND2zHmh8pPoKkZgS6JzSbLVFT2/QKBK/z+4=; b=qo9co1FA7LMKg2SpmL16pBIEPGNcf3CXTSOL9BBnhZ3ytZE0s3toZSYbZlH/JVIERT 9JV0ibUudXWZzJHpcq3Hs8uKLrNSXlylThIGgBA6ZrSTXjaYVmX5bStcRDdrSYMbcLbc JNpWJ9a8PRWCzANJZ3n30SFv66obPWZL9ok9kMXiihnxykrM2wbnqew9ZQeKqnE6/bBm fmxvmdVw87PNsj4IsYPB0piGPBIyYr7+CIKIN22BzETk8YNYzVb1RE8ry51/WReQk4rz qvuyqsbUy2c6CtzQ3J7tQJeq0gXk51h0+kH1VxEp2vvNv3MwHm8JGZhFcZyupNr0aBY4 V/hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733866691; x=1734471491; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OQZEugtHND2zHmh8pPoKkZgS6JzSbLVFT2/QKBK/z+4=; b=YIOdYN8W36UCuGSXRUtp+OMpTRrtWFofdufcNtIsUnA5fsvzvLo8OvBQpXN59PEzv3 /mp3lTPIfK4nb7sDWUVQzHRmXL0tpBhXNaYA9c7Twi53A0PqfZDmWiJvxMMLthO7WFWZ wPo7zXeUWMkkzkgmTGgjD/Kl6b27zqXe+gdWb4lfx+kx1begFTRfjpCW6SJVbYsp/Ewr Bw2BFmPFisBxBoOdc8lmeIA3cgNPkf7REAIFjfB7I70eAayW3zq688OSCCP98x0OAdei JhUjrW0ffkr0H+daKhNSziIEhkG5jj3yVlSa6vJqPGe0+Y7pALhgo3e7NA/3SY4Bwsuk yBjA== X-Gm-Message-State: AOJu0Yy8CEtWRsp7enCfJuAUjxU83j0cZ+hPYPG7yyI8bcXJuDaaXxXz o3SYAf1o2Ic+18sQ8J/IGjrvVQj5G5DheK5q6QMlqqZorkJJ9er3/xC+gJSnRkM= X-Gm-Gg: ASbGncuOzRDoaBoEgmvfhU00NH6RqZnwIc2FJYqEi/FEa1G1f8vkAX1wtDX/qs9aZar pyNSe2n6CbJwUsMupPLwQ9Jmav5ejRXoA2jeucJtTku0sU93guon4yvVWOhT1rIf+rQn8wNT1Kv 354LLgqjPjDso6x+Oq47y/z94ZeGRH5aaQaAQJCmKY2wG1k7OZZNNLrO2lx5HrKKIBSFGqROu8C Xi75wLP+PFp8/cQMdPv0zfVqplIQ9t2HRQuzpuC4HI/o+g3+LIdDAXZY3DmowNfrtoAGfy7OmoG 3VbWZbS75bLjieE8MgQjb1srnfJ6J65IRoW9fEE= X-Google-Smtp-Source: AGHT+IFYbR7aXy36yPVN9o7PP/Nyz3CEWrs6BuemC8qkn+p43Ye5cXgiGx1LfhBNIjaaOLXxBrl2Ig== X-Received: by 2002:ac8:5947:0:b0:466:9197:b503 with SMTP id d75a77b69052e-46789379413mr8570201cf.46.1733866691608; Tue, 10 Dec 2024 13:38:11 -0800 (PST) Received: from PC2K9PVX.TheFacebook.com (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-467296f61f6sm65978991cf.43.2024.12.10.13.38.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 13:38:10 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com Subject: [RFC v2 PATCH 1/5] migrate: Allow migrate_misplaced_folio_prepare() to accept a NULL VMA. Date: Tue, 10 Dec 2024 16:37:40 -0500 Message-ID: <20241210213744.2968-2-gourry@gourry.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241210213744.2968-1-gourry@gourry.net> References: <20241210213744.2968-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" migrate_misplaced_folio_prepare() may be called on a folio without a VMA, and so it must be made to accept a NULL VMA. Suggested-by: Johannes Weiner Signed-off-by: Gregory Price Suggested-by: Feng Tang Suggested-by: Huang Ying Suggested-by: Keith Busch --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index e9e00d1d1d19..af07b399060b 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2632,7 +2632,7 @@ int migrate_misplaced_folio_prepare(struct folio *fol= io, * See folio_likely_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ - if ((vma->vm_flags & VM_EXEC) && + if (vma && (vma->vm_flags & VM_EXEC) && folio_likely_mapped_shared(folio)) return -EACCES; =20 --=20 2.43.0 From nobody Wed Dec 17 12:48:32 2025 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40FEC23872F for ; Tue, 10 Dec 2024 21:38:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866695; cv=none; b=Fjc6ITIAbM0GNk7afa4/pBcc3/9xhR95bVVa3Zc3fwqegk9tllt7M9G/B+0TvhqDmxYNGkISuRh4N2v6kUJ/Gw+cnH6RJfS7e+cCu+AK7sXQAINhujj71PcLncJlmAVKN0y+9IJmlEfFUsoFdTqzH5o8tXvhWl6OVJeuN0KJiCs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866695; c=relaxed/simple; bh=blwGRn4D8exPkZTkDVFRUh0NPIGr3Pmq8/aaGVNtNe8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CeNpG9bCNjjuJdx4dlHWKcJO+8jY3FIQfqNRjRzkC7RmcoLKTDJNcCsGraEy2i5jfz0I+IgOsXJH8YnVT2vyigg5I+m+mE0s2Nf5ydSG9X+ioGUMxVjfBNyAmQBAmJxTmmtBls9U90sMIyEx631IgrpLtLUxXZ9S0KaxfzJVFOY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=FJ9oKrwM; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="FJ9oKrwM" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7b6e9db19c8so43419085a.3 for ; Tue, 10 Dec 2024 13:38:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1733866693; x=1734471493; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7SLO2zP3WSxmf0tbBTHWC3uKgLqzLgdzgwGcqHy1nF8=; b=FJ9oKrwM/IKwLA2PqTrVyEl0QOWc74PuHB2XTuNmPiRK0P+3wJGMHsbe2mPg0tLt0/ uaKv5lnIHXLFrwXOKEEHE4HLzGSTA4e8X3XBawo5A4LG1OeQk0rId1DSVcmGt6CtQwGw q0MnxkmsAZdYqOqTxd47DdU7G6wP06oVqNEk5pgRBMnVU11knJgxOZIuKhUAWLyVAmaP vHpCBXa5Y41Qv/MtlwGZhgL1LNoCmlKGvsJ4U4H6fUzgpXQLltjo2qkiayk8tXXkdlhu 1le1Tp1ywAuBkwLOyYyhKQSXaFcK0pOnyLM96MdhCgms5HwQByB9Gi8xWLGVDIJymfPv ydnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733866693; x=1734471493; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7SLO2zP3WSxmf0tbBTHWC3uKgLqzLgdzgwGcqHy1nF8=; b=Ey5lumjVJQukWLEETLdqz4jdbYnmfZDadEAd3t9U9gYEYm20vhLvBAoPJqHXRTokZi AjrUXANGgjbbjU/PmOXMfAFbqLkorv6NPYPPtAVuBiWI7seYDWcz+XuUYIoC5ke2R/cY 6JFT+UR2w83VNkwdGY/VZMoDNaQ+gxcWUuxlTFTPma/9hegUFfgLvVtpiLwfht39GiBp zOqGyE5QMzb/MWmQLJp0W8FVgE91bX67vW3035zQviME1KBPSSAqJwZh4oDISfDeSBbt CfZ/9FBxD60sXsOQlLvceRROIbBDWwUnPSIR03K72hEGrUpyXfwxQnuVEN+6tQQdwEZZ okqA== X-Gm-Message-State: AOJu0YzdoNPy34Zqpt49uCPLK2kW/i8QieBkuqzZuKVbvZN2R+aA3BDa GjC09D5iOtWyntIOIdNaWZvNB7x+lZNvjF/v09znwYUIsX0GF1SUqiS9I/zWam4= X-Gm-Gg: ASbGncvFMQFoKKonMqia3GZgru5nAz25KsstAKGjDyU7N9hJHwxnd93QCo7fRNdP5qo vVPSMcSBpdmaV1UJl5e0KMwxbUMDAm+QXQtXkKpnBmVtWCFFAGNZnLy+Jlsfaw51qdkC0lCukw+ z0GaMFX8mfO/I57OM4Sskaa0C4ncvEICL0HbZAmkN10qWyM19oqK4kSxK61I+eEOjVxLwKzrmXR HUPIjX3QPVp+sp1ltVZnrDBgD1Rf4vhSS9NjofKHWpuTOtKYl32UVai9SvBOpE6v6/GCFG6UzsO K5Lq1VB9Yrx8BdcOeei8VRcubVgXPKZjPt55oSo= X-Google-Smtp-Source: AGHT+IGR91VwyQ9Xd3L5CFowo5m6Bq9C98LuxwXRSibAvLZ+jNUO5jNdqjrNPT9txkkOjxePaZAJ4Q== X-Received: by 2002:a05:620a:4014:b0:7b6:d3b3:575e with SMTP id af79cd13be357-7b6eb4d1730mr126961285a.47.1733866693178; Tue, 10 Dec 2024 13:38:13 -0800 (PST) Received: from PC2K9PVX.TheFacebook.com (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-467296f61f6sm65978991cf.43.2024.12.10.13.38.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 13:38:12 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com Subject: [RFC v2 PATCH 2/5] memory: move conditionally defined enums use inside ifdef tags Date: Tue, 10 Dec 2024 16:37:41 -0500 Message-ID: <20241210213744.2968-3-gourry@gourry.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241210213744.2968-1-gourry@gourry.net> References: <20241210213744.2968-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" NUMA_HINT_FAULTS and NUMA_HINT_FAULTS_LOCAL are only defined if CONFIG_NUMA_BALANCING is defined, but are used outside the tags in numa_migrate_check(). Fix this. TNF_SHARED is only used if CONFIG_NUMA_BALANCING is enabled, so moving this line inside the ifdef is also safe - despite use of TNF_* elsewhere in the function. TNF_* are not conditionally defined. Signed-off-by: Gregory Price Suggested-by: Feng Tang Suggested-by: Huang Ying Suggested-by: Johannes Weiner Suggested-by: Keith Busch --- mm/memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 83fd35c034d7..6ad7616918c4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5573,14 +5573,14 @@ int numa_migrate_check(struct folio *folio, struct = vm_fault *vmf, /* Record the current PID acceesing VMA */ vma_set_access_pid_bit(vma); =20 - count_vm_numa_event(NUMA_HINT_FAULTS); #ifdef CONFIG_NUMA_BALANCING + count_vm_numa_event(NUMA_HINT_FAULTS); count_memcg_folio_events(folio, NUMA_HINT_FAULTS, 1); -#endif if (folio_nid(folio) =3D=3D numa_node_id()) { count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); *flags |=3D TNF_FAULT_LOCAL; } +#endif =20 return mpol_misplaced(folio, vmf, addr); } --=20 2.43.0 From nobody Wed Dec 17 12:48:32 2025 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A99B238758 for ; Tue, 10 Dec 2024 21:38:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866698; cv=none; b=geVu8QGdoEJdJEqFkz4yUrlXf4tiNHV2AocMW42ZWqtIqHbV/Hrv2Qb0XIIfC5BDgsjrnkt+4FL1iEQScnYT89OX9IJVSYcDvAlCOyDz9AgZMgexsaBg6s1wBrQVfBYixnA9xnVytkBKXS0X6fzmzfGZCBA2+qR6z3YgEfVIm9M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866698; c=relaxed/simple; bh=g1DsUKkKE5QxoofU8cGp0UwxnURYT/y1aUuxGbuiN/E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IVZw2nlQNnzN0mFt7ulWuy1pYVQgnUyg5HwjnbAmZAWRDeGlNyqTEi3jJZrv1Dai83uPyMfYPplJLU8yoZgKwwTpzTxe6ElqmeLUSHonZCApxgXH3IMy3yTTJBO4DT5DpZ9+Z5pGxkrrwslZ4FguewA3dtEyqi882+QrT3VbCHI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=bO0QsdyT; arc=none smtp.client-ip=209.85.160.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="bO0QsdyT" Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-46741855f9bso31076961cf.2 for ; Tue, 10 Dec 2024 13:38:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1733866695; x=1734471495; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rD13RBY30Rrvkd04V+Adk5FJ3Ta4BXdJd5R/HBJ/aEU=; b=bO0QsdyTYyC12fQsU3XL9PAvenHlcaRvNktzEEF82wptUPS7c7HguPE9uR0cPrrfWd cmEgt2PdS93C1qteQAoNRxT5d8++ooQrJDi6rtXUs9QWAwcAOzsCspBOYCmlQBd53SYs P5v/zRsmE0cNo0923Cfkr8tpAVNJ9F6pbeNiLcj7Ibl4WUpUU6BJz/cbPiBsEa8W2iFW m/7+xtbgyTVK5S3m1iQZy8Zlyi/b9+cn3JrKJBB3mhxI/mk3jcHPn1+BTvvycLBLMagz nOVQfvNS2nQ9X3ntoPegfJmJQ0a9EmrGC4oZDirigNB7vrAmWXH8LO8N7m4DGT72XqDI nmiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733866695; x=1734471495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rD13RBY30Rrvkd04V+Adk5FJ3Ta4BXdJd5R/HBJ/aEU=; b=QPi1o3GSkdpELI/pPyVnOacNDzzf5hTYDezR+ttPPDIlKYAQUbtDXh6zrXc07ALFPn 0x7od1AM33cV2DXDcP2Nm6jNbG6q+rQX2HepRdBluUr4EX688zE6DKEPV2tqQ7y9NA10 LEsLAPPVV73XwypwaUNuH/69ediW+wni6ahL9IciTymb34h1/3R8GqncRXAOFNpudNki LeMUeL0/XtfBwKNFfGlYLU0X5CAMLS9NrtnWcdEDC6XklloXq2yY7TWF85ITUJrZnU4t haMr5WD2IikRqgP/QW50JSYaXUPWFFYKvSd5CyvpR835Pg032MeM/7JQrLClFVSANkQ7 PH+A== X-Gm-Message-State: AOJu0Yz6hmCwSxItrNTLGuD63qNbPKUGYWoxZvEpxraO5uY50wR/iuzx FkRotzlThdq+MPJ9vpcjfLsV1bCzoftoPjy0wLLo5uf9dxOcTabPzpMC+ZD8AnM= X-Gm-Gg: ASbGncuS6CfoEqQqxGPJmPh4sFtYHWDZ/+Ho5S5Px0SqZ1mCbehgy/72pRf7D3NadZW 5V0+es5WgI0l6Bpyp6XVrWm6zz4CTRFoqEBgXAi9sXYtdIS9lTyWCmTy/U19nZnCAcY5PRdBagi q+Zr6lgwRWamPzAy6RVsiM6SwKjsDomC9JCUBE3NfvtfbPWVlz+gROtZEFZPCLt/aVxYxrDbNxX nBujaA14B7w5JkTxAT6w7d5HqeiDcGwICAKbuGgDvixNCOFmAT16/k+x/sldKCNuuhu3+ZEUVAo zAsueyTpajW9BiVcBLDR6UMUdj1Ew7z+Ss+ukCc= X-Google-Smtp-Source: AGHT+IFsnNfg3615eI3lmfgEmFA2vm6QMKjB1yR8PQ1ci2ihqRLidFlxPlnZADQfcm/FjEoxGxlntQ== X-Received: by 2002:a05:622a:15c8:b0:466:77d0:5941 with SMTP id d75a77b69052e-4678929f162mr8994041cf.10.1733866695082; Tue, 10 Dec 2024 13:38:15 -0800 (PST) Received: from PC2K9PVX.TheFacebook.com (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-467296f61f6sm65978991cf.43.2024.12.10.13.38.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 13:38:14 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com Subject: [RFC v2 PATCH 3/5] memory: allow non-fault migration in numa_migrate_check path Date: Tue, 10 Dec 2024 16:37:42 -0500 Message-ID: <20241210213744.2968-4-gourry@gourry.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241210213744.2968-1-gourry@gourry.net> References: <20241210213744.2968-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" numa_migrate_check and mpol_misplaced presume callers are in the fault path with access to a VMA. To enable migrations from page cache, re-using the same logic to handle migration prep is preferable. Mildly refactor numa_migrate_check and mpol_misplaced so that they may be called with (vmf =3D NULL) from non-faulting paths. Signed-off-by: Gregory Price Suggested-by: Feng Tang Suggested-by: Huang Ying Suggested-by: Johannes Weiner Suggested-by: Keith Busch --- mm/memory.c | 24 ++++++++++++++---------- mm/mempolicy.c | 25 +++++++++++++++++-------- 2 files changed, 31 insertions(+), 18 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6ad7616918c4..af7ba56a4e1e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5542,7 +5542,20 @@ int numa_migrate_check(struct folio *folio, struct v= m_fault *vmf, unsigned long addr, int *flags, bool writable, int *last_cpupid) { - struct vm_area_struct *vma =3D vmf->vma; + if (vmf) { + struct vm_area_struct *vma =3D vmf->vma; + const vm_flags_t vmflags =3D vma->vm_flags; + + /* + * Flag if the folio is shared between multiple address spaces. + * This used later when determining whether to group tasks. + */ + if (folio_likely_mapped_shared(folio)) + *flags |=3D vmflags & VM_SHARED ? TNF_SHARED : 0; + + /* Record the current PID acceesing VMA */ + vma_set_access_pid_bit(vma); + } =20 /* * Avoid grouping on RO pages in general. RO pages shouldn't hurt as @@ -5555,12 +5568,6 @@ int numa_migrate_check(struct folio *folio, struct v= m_fault *vmf, if (!writable) *flags |=3D TNF_NO_GROUP; =20 - /* - * Flag if the folio is shared between multiple address spaces. This - * is later used when determining whether to group tasks together - */ - if (folio_likely_mapped_shared(folio) && (vma->vm_flags & VM_SHARED)) - *flags |=3D TNF_SHARED; /* * For memory tiering mode, cpupid of slow memory page is used * to record page access time. So use default value. @@ -5570,9 +5577,6 @@ int numa_migrate_check(struct folio *folio, struct vm= _fault *vmf, else *last_cpupid =3D folio_last_cpupid(folio); =20 - /* Record the current PID acceesing VMA */ - vma_set_access_pid_bit(vma); - #ifdef CONFIG_NUMA_BALANCING count_vm_numa_event(NUMA_HINT_FAULTS); count_memcg_folio_events(folio, NUMA_HINT_FAULTS, 1); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 88eef9776bb0..77a123fa71b0 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2746,12 +2746,16 @@ static void sp_free(struct sp_node *n) * mpol_misplaced - check whether current folio node is valid in policy * * @folio: folio to be checked - * @vmf: structure describing the fault + * @vmf: structure describing the fault (NULL if called outside fault path) * @addr: virtual address in @vma for shared policy lookup and interleave = policy + * Ignored if vmf is NULL. * * Lookup current policy node id for vma,addr and "compare to" folio's - * node id. Policy determination "mimics" alloc_page_vma(). - * Called from fault path where we know the vma and faulting address. + * node id - or task's policy node id if vmf is NULL. Policy determination + * "mimics" alloc_page_vma(). + * + * vmf must be non-NULL if called from fault path where we know the vma and + * faulting address. The PTL must be held by caller if vmf is not NULL. * * Return: NUMA_NO_NODE if the page is in a node that is valid for this * policy, or a suitable node ID to allocate a replacement folio from. @@ -2763,7 +2767,6 @@ int mpol_misplaced(struct folio *folio, struct vm_fau= lt *vmf, pgoff_t ilx; struct zoneref *z; int curnid =3D folio_nid(folio); - struct vm_area_struct *vma =3D vmf->vma; int thiscpu =3D raw_smp_processor_id(); int thisnid =3D numa_node_id(); int polnid =3D NUMA_NO_NODE; @@ -2773,18 +2776,24 @@ int mpol_misplaced(struct folio *folio, struct vm_f= ault *vmf, * Make sure ptl is held so that we don't preempt and we * have a stable smp processor id */ - lockdep_assert_held(vmf->ptl); - pol =3D get_vma_policy(vma, addr, folio_order(folio), &ilx); + if (vmf) { + lockdep_assert_held(vmf->ptl); + pol =3D get_vma_policy(vmf->vma, addr, folio_order(folio), &ilx); + } else { + pol =3D get_task_policy(current); + } if (!(pol->flags & MPOL_F_MOF)) goto out; =20 switch (pol->mode) { case MPOL_INTERLEAVE: - polnid =3D interleave_nid(pol, ilx); + polnid =3D vmf ? interleave_nid(pol, ilx) : + interleave_nodes(pol); break; =20 case MPOL_WEIGHTED_INTERLEAVE: - polnid =3D weighted_interleave_nid(pol, ilx); + polnid =3D vmf ? weighted_interleave_nid(pol, ilx) : + weighted_interleave_nodes(pol); break; =20 case MPOL_PREFERRED: --=20 2.43.0 From nobody Wed Dec 17 12:48:32 2025 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71D9623978C for ; Tue, 10 Dec 2024 21:38:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866699; cv=none; b=XKekTvgqzvmozf+qH7vcwUBchPGAhhleAlEym0wkXmAyS+LBlrSRuEMssJn9CUJwSKyjqo04r/RG32eqg6CZqD8X5o6Dpg0yo4aWZxJogjtPZCskJOnUtiGkv+7TPwIMjHYOJlxys3lBMr5w+vP3p7lTh8Yz08djSTHavG+p5z8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866699; c=relaxed/simple; bh=tYVwqyB74v30wxub/2XwQXi7EDQ2PRrHyv5WyH8ynQg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=USHE92pWJnPZayLuPWIdKcuLd6LIt8jtdC5ZE32QTK8t9P2/0WOlvBk1lbLB38kErFsZyeJusKzmX/Z3HQl5VQ1TVdma7dRYlE5xyGc4s6WGgrCATQzhFo0bgcgU5t0leP7dvv/JDvheHnmFHnz1Qu8mBmeBAjfW9uUng0C242s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=PFAhwcXE; arc=none smtp.client-ip=209.85.160.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="PFAhwcXE" Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-46779ae3139so10392791cf.2 for ; Tue, 10 Dec 2024 13:38:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1733866696; x=1734471496; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Jq0f3dJTpHJXJDokS6aQHyPKu/LFNDC+TZCLMRgrQ0=; b=PFAhwcXEhQxXFFgBa0kvrfu6L6MzOX/zNZNHJhGOh8DzE38tI+/rb4JV/RaYOep9lC fH81H1nZM7BoJSXcBwmW8md3bj71wU9UlAmBjDY8XPOm8zVuylc6/9dGzz6eXlvndlYv CbFsKMqgo/DUXHYx1khyJe6ZAB3AudT252F126X3qskdNdUOvaEl6I75wj6LmvoggAi3 t9YjE5vCvoVkkMW6FPlR10bj/pyPnxldoavYFxL7ReOuHn3SwjP8zYRBbIvviJrfhWFQ 6vQQ044F1OFuMunoiKkrZj+XHpm2AG6lUXyU8vTDTc8fzkQtPQqIPPK8tYjLrsz2E30A RYTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733866696; x=1734471496; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Jq0f3dJTpHJXJDokS6aQHyPKu/LFNDC+TZCLMRgrQ0=; b=GbtF6haFNHJKXdB7FKorX+CHxkhe710yFmiJZ6a8RHnHemvcAza3ZfLXbP4GG/8H3q H2bs8VM8GLyhYkhmshIx1w7Xii4xA9C45Kp65hARjAsLM6SeuRrF5Y7UQx0sHpGVzkvv 9HBxYK6h0xx1Hrahok/hVNDToaGOMsOL+HCGEgTu4RUaNIZSlYOXkSBmmkFtdx5XxOuL 1O3PM2X9FtYsLqEeliXHAviXZkIrKIPhGhPLULW5Ua6U6oeXAUzEs8NOeAp3Y6zztbhk Hi4b8kJFFkuAxG5MuuFeYlJu1ORxA3WiAWCwXrhhJMYXyQrJ9LGVnU7lYgeyqy1MwjyV xPrw== X-Gm-Message-State: AOJu0YyizqyR1BL0g0HxBwnxGklypUf5prAx+mt5usKSHmPiwC3FVGyg z2elpCDhZzne3zi8K66tU+ZWcWkuPzQk4XwtKFe323BT1hXdlUV4HJWMv/xq0N4= X-Gm-Gg: ASbGncvbxLCAO5R3HVQk8zjJU4uPHWIoDFCTfHCMYsD/ESewyU+yAPjsPTd0LzcK3ro uWSD5Ip/vU+DZyu1/UnJxk3JRez6UZmjFjCk2qIcDpOPjwkQJMZ7afo7tEB9nxT1g/BBcrD/Fv6 BpluREoQGiLLOUhGTmYyNyxfs7/SYqd0Z2MzMh9B8RreKwlBfaipE6adhJu08uQcy2fj2Ea14Ub ZeQqyHGVOmP+XRMfoj9jhVhc5zpl4DZAMiR1Fq+3xDumR5ux71QkDEjAt/1o+f0E1rjGB+woP2T NeE3YwgXVa64OJ004x4fqd6vzuIFqtcvKKDeZbY= X-Google-Smtp-Source: AGHT+IHxInG7lq5H3xVJww7rlvFoTxqlg+CA0d/WG83+KpCA77jzrX+fsIGQc0RjdUSsv/+VccPkKg== X-Received: by 2002:a05:622a:4d45:b0:466:a584:69f8 with SMTP id d75a77b69052e-467893bcc14mr7050891cf.43.1733866696431; Tue, 10 Dec 2024 13:38:16 -0800 (PST) Received: from PC2K9PVX.TheFacebook.com (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-467296f61f6sm65978991cf.43.2024.12.10.13.38.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 13:38:16 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com Subject: [RFC v2 PATCH 4/5] vmstat: add page-cache numa hints Date: Tue, 10 Dec 2024 16:37:43 -0500 Message-ID: <20241210213744.2968-5-gourry@gourry.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241210213744.2968-1-gourry@gourry.net> References: <20241210213744.2968-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Count non-page-fault events as page-cache numa hints instead of fault hints in vmstat. Add a define to select the hint type to keep the code clean. Signed-off-by: Gregory Price Suggested-by: Feng Tang Suggested-by: Huang Ying Suggested-by: Johannes Weiner Suggested-by: Keith Busch --- include/linux/vm_event_item.h | 8 ++++++++ mm/memory.c | 6 +++--- mm/vmstat.c | 2 ++ 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f70d0958095c..c5abb0f7cca7 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -63,6 +63,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HUGE_PTE_UPDATES, NUMA_HINT_FAULTS, NUMA_HINT_FAULTS_LOCAL, + NUMA_HINT_PAGE_CACHE, + NUMA_HINT_PAGE_CACHE_LOCAL, NUMA_PAGE_MIGRATE, #endif #ifdef CONFIG_MIGRATION @@ -185,6 +187,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NR_VM_EVENT_ITEMS }; =20 +#ifdef CONFIG_NUMA_BALANCING +#define NUMA_HINT_TYPE(vmf) (vmf ? NUMA_HINT_FAULTS : NUMA_HINT_PAGE_CACHE) +#define NUMA_HINT_TYPE_LOCAL(vmf) (vmf ? NUMA_HINT_FAULTS_LOCAL : \ + NUMA_HINT_PAGE_CACHE_LOCAL) +#endif + #ifndef CONFIG_TRANSPARENT_HUGEPAGE #define THP_FILE_ALLOC ({ BUILD_BUG(); 0; }) #define THP_FILE_FALLBACK ({ BUILD_BUG(); 0; }) diff --git a/mm/memory.c b/mm/memory.c index af7ba56a4e1e..620e2045af7b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5578,10 +5578,10 @@ int numa_migrate_check(struct folio *folio, struct = vm_fault *vmf, *last_cpupid =3D folio_last_cpupid(folio); =20 #ifdef CONFIG_NUMA_BALANCING - count_vm_numa_event(NUMA_HINT_FAULTS); - count_memcg_folio_events(folio, NUMA_HINT_FAULTS, 1); + count_vm_numa_event(NUMA_HINT_TYPE(vmf)); + count_memcg_folio_events(folio, NUMA_HINT_TYPE(vmf), 1); if (folio_nid(folio) =3D=3D numa_node_id()) { - count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); + count_vm_numa_event(NUMA_HINT_TYPE_LOCAL(vmf)); *flags |=3D TNF_FAULT_LOCAL; } #endif diff --git a/mm/vmstat.c b/mm/vmstat.c index 4d016314a56c..bcd9be11e957 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1338,6 +1338,8 @@ const char * const vmstat_text[] =3D { "numa_huge_pte_updates", "numa_hint_faults", "numa_hint_faults_local", + "numa_hint_page_cache", + "numa_hint_page_cache_local", "numa_pages_migrated", #endif #ifdef CONFIG_MIGRATION --=20 2.43.0 From nobody Wed Dec 17 12:48:32 2025 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 171722397A6 for ; Tue, 10 Dec 2024 21:38:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866701; cv=none; b=YeOQvwzTQLhh2CWj4sZW/UWU8mI4mpwCaC4P4op7obgeZsUkuobyMHwTNyUvKEXUE+w7pkYqbWJiYWvK11hqbu8StgQpH3cOvYwaEsFFkbNSq1F3wWrt3W6Sn5OsYR3HjunqSd7sQhroOKf23fCtj4t3AA3aCJPdE4DzmXMZyhk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733866701; c=relaxed/simple; bh=hE9bjV2sBCy6yetf6MGYbjGzkjGv3G8CKtCSWbCTqi0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cwkxV1Gm+9p+m6J5WKCOARIr5VTJEX3WFgW/JQkS9wxJqXx3IrjgFy4GJbIgqBe1LTztmfv2Td+fBlorkmquKig0xOyodgr3jtCVYDFicAhzNdOgDEXeB7rzxeBYto3BCr24WhIEauQamObi8rK9YwX0Pt8YM2FIkvKcum8k9ZI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=KyA4V8al; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="KyA4V8al" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-46741855f9bso31077241cf.2 for ; Tue, 10 Dec 2024 13:38:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1733866698; x=1734471498; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+9y/Pmi/E+qcbGw7qvBfkROEFTzGsGqs1ClRtYT08n0=; b=KyA4V8alZaa7RNJr2jA/wGP8j99SA+DrnnwcECWJCKDY5NP9otzUyHHCzviejvWLpc wn5UeVGRUZ7Ul8d91a/P0yUbtc1C6E3w4PbyxOtIh7PfJKqHRrDguGgl2qB96tVrsWMG 6MfL5ZOFtV8VKuUgGS1xpiw8Iu4ZCrmdSD05fnEAmpb0yegf78dcvp8bd5tP256Rn2z/ sMsbaKlPAad6O3DRoLvo18EfvHE+ztICxmnrabP+oogITnUdCTvLC4bzyLC1uA92W8TZ Ll07W7MxupuMEIB05qZ6Pf7Z6snDL+QthuzCcx9WYTVsWPvjNZWAr03+4DBxQjsdUTrU K54A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733866698; x=1734471498; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+9y/Pmi/E+qcbGw7qvBfkROEFTzGsGqs1ClRtYT08n0=; b=XeV/r3BffAkEmSL/RdssPIdSmECDGzZ0XixYWmxijAV6ImhtlrWPc2uoKDOSEPrN0c V43h1d18SzWaz3anfP39nUiWgz9LuZd23OU+bTZtOB1zz7vXimCb6hw7ddjDaAp6eEvY n1c+GmOw4yOegdtwu+R3wSf2FYu08w8fXf1b/bEuLrZAqBSwh7QsG/By6DQNWOXVa4Dp aHeR4pOVR5x1H5coDS8ORhwbma4BwhlZPH0DlMXzJaGR7vGFugT6mxDJzXdV3am3r1Co DnPM5+l1odjovDs1Aodz7qMSVo/ocPVTF4My/SefJopqhWpm/737PHMeGkhnXpEr5g0u UWug== X-Gm-Message-State: AOJu0Ywn2R6MZiPcVgW6kbFkSAtllJTwIb7jfPOuuxT802oHG/HrJ7FK WctgIb8gedmpsQaW4AMLW87ogWvdQxFAltzvmwPapuY2gU0vMJMpIDosEgI4Sf4= X-Gm-Gg: ASbGncu+r3wrLzNgRuO0eTu0p5N+UD3m4+oQ3+pvS7VfoRzudX9pt+6keZCeKAjKgOp lFnH8R3t8OWcXuvBLYoz1u+nVZRSCU/m9cL4iZoqexZg4dBvOsPPUezfC2hgOB9iI0dNmuwftv/ BKNmtdtFmYaejBPn95gneLjcI3aIh9m0gKFnOTRGfb22+M5eU5CKkmH0SDACCEkih6tuff7NJNR zfL0WBPoEbc3GYmUb/HyYNc3F4z9GRdSDTeRBa2m+Vrt/kxp9JJ9H93xQFlcN9Hj7590ZVz3FAf EiKVsEdPRyWMDKESlambnYCiMTB2sRpV1pL/xgw= X-Google-Smtp-Source: AGHT+IFxNZ06/2veak16EjkYa8clNKlgcdi7wLb2OYdhSxCOFJs32hkfuiKlgIR8eXNwOyd59R311Q== X-Received: by 2002:a05:622a:1815:b0:467:7208:8eeb with SMTP id d75a77b69052e-467893769a8mr9097551cf.37.1733866697811; Tue, 10 Dec 2024 13:38:17 -0800 (PST) Received: from PC2K9PVX.TheFacebook.com (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-467296f61f6sm65978991cf.43.2024.12.10.13.38.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2024 13:38:17 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, gourry@gourry.net, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com, ying.huang@linux.alibaba.com Subject: [RFC v2 PATCH 5/5] migrate,sysfs: add pagecache promotion Date: Tue, 10 Dec 2024 16:37:44 -0500 Message-ID: <20241210213744.2968-6-gourry@gourry.net> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241210213744.2968-1-gourry@gourry.net> References: <20241210213744.2968-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" adds /sys/kernel/mm/numa/pagecache_promotion_enabled When page cache lands on lower tiers, there is no way for promotion to occur unless it becomes memory-mapped and exposed to NUMA hint faults. Just adding a mechanism to promote pages unconditionally, however, opens up significant possibility of performance regressions. Similar to the `demotion_enabled` sysfs entry, provide a sysfs toggle to enable and disable page cache promotion. This option will enable opportunistic promotion of unmapped page cache during syscall access. This option is intended for operational conditions where demoted page cache will eventually contain memory which becomes hot - and where said memory likely to cause performance issues due to being trapped on the lower tier of memory. A Page Cache folio is considered a promotion candidates when: 0) tiering and pagecache-promotion are enabled 1) the folio reside on a node not in the top tier 2) the folio is already marked referenced and active. 3) Multiple accesses in (referenced & active) state occur quickly. Since promotion is not safe to execute unconditionally from within folio_mark_accessed, we defer promotion to a new task_work captured in the task_struct. This ensures that the task doing the access has some hand in promoting pages - even among deduplicated read only files. We use numa_hint_fault_latency to help identify when a folio is accessed multiple times in a short period. Along with folio flag checks, this helps us minimize promoting pages on the first few accesses. The promotion node is always the local node of the promoting cpu. Suggested-by: Johannes Weiner Signed-off-by: Gregory Price Suggested-by: Feng Tang Suggested-by: Huang Ying Suggested-by: Keith Busch --- .../ABI/testing/sysfs-kernel-mm-numa | 20 +++++++ include/linux/memory-tiers.h | 2 + include/linux/migrate.h | 2 + include/linux/sched.h | 3 + include/linux/sched/numa_balancing.h | 5 ++ init/init_task.c | 1 + kernel/sched/fair.c | 26 +++++++- mm/memory-tiers.c | 27 +++++++++ mm/migrate.c | 59 +++++++++++++++++++ mm/swap.c | 3 + 10 files changed, 147 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-numa b/Documentation= /ABI/testing/sysfs-kernel-mm-numa index 77e559d4ed80..b846e7d80cba 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-numa +++ b/Documentation/ABI/testing/sysfs-kernel-mm-numa @@ -22,3 +22,23 @@ Description: Enable/disable demoting pages during reclaim the guarantees of cpusets. This should not be enabled on systems which need strict cpuset location guarantees. + +What: /sys/kernel/mm/numa/pagecache_promotion_enabled +Date: November 2024 +Contact: Linux memory management mailing list +Description: Enable/disable promoting pages during file access + + Page migration during file access is intended for systems + with tiered memory configurations that have significant + unmapped file cache usage. By default, file cache memory + on slower tiers will not be opportunistically promoted by + normal NUMA hint faults, because the system has no way to + track them. This option enables opportunistic promotion + of pages that are accessed via syscall (e.g. read/write) + if multiple accesses occur in quick succession. + + It may move data to a NUMA node that does not fall into + the cpuset of the allocating process which might be + construed to violate the guarantees of cpusets. This + should not be enabled on systems which need strict cpuset + location guarantees. diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h index 0dc0cf2863e2..fa96a67b8996 100644 --- a/include/linux/memory-tiers.h +++ b/include/linux/memory-tiers.h @@ -37,6 +37,7 @@ struct access_coordinate; =20 #ifdef CONFIG_NUMA extern bool numa_demotion_enabled; +extern bool numa_pagecache_promotion_enabled; extern struct memory_dev_type *default_dram_type; extern nodemask_t default_dram_nodes; struct memory_dev_type *alloc_memory_type(int adistance); @@ -76,6 +77,7 @@ static inline bool node_is_toptier(int node) #else =20 #define numa_demotion_enabled false +#define numa_pagecache_promotion_enabled false #define default_dram_type NULL #define default_dram_nodes NODE_MASK_NONE /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 29919faea2f1..cf58a97d4216 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -145,6 +145,7 @@ const struct movable_operations *page_movable_ops(struc= t page *page) int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); +void promotion_candidate(struct folio *folio); #else static inline int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node) @@ -155,6 +156,7 @@ static inline int migrate_misplaced_folio(struct folio = *folio, int node) { return -EAGAIN; /* can't migrate now */ } +static inline void promotion_candidate(struct folio *folio) { } #endif /* CONFIG_NUMA_BALANCING */ =20 #ifdef CONFIG_MIGRATION diff --git a/include/linux/sched.h b/include/linux/sched.h index d380bffee2ef..faa84fb7a756 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1356,6 +1356,9 @@ struct task_struct { unsigned long numa_faults_locality[3]; =20 unsigned long numa_pages_migrated; + + struct callback_head numa_promo_work; + struct list_head promo_list; #endif /* CONFIG_NUMA_BALANCING */ =20 #ifdef CONFIG_RSEQ diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/num= a_balancing.h index 52b22c5c396d..cc7750d754ff 100644 --- a/include/linux/sched/numa_balancing.h +++ b/include/linux/sched/numa_balancing.h @@ -32,6 +32,7 @@ extern void set_numabalancing_state(bool enabled); extern void task_numa_free(struct task_struct *p, bool final); bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, int src_nid, int dst_cpu); +int numa_hint_fault_latency(struct folio *folio); #else static inline void task_numa_fault(int last_node, int node, int pages, int flags) @@ -52,6 +53,10 @@ static inline bool should_numa_migrate_memory(struct tas= k_struct *p, { return true; } +static inline int numa_hint_fault_latency(struct folio *folio) +{ + return 0; +} #endif =20 #endif /* _LINUX_SCHED_NUMA_BALANCING_H */ diff --git a/init/init_task.c b/init/init_task.c index e557f622bd90..f831980748c4 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -187,6 +187,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = =3D { .numa_preferred_nid =3D NUMA_NO_NODE, .numa_group =3D NULL, .numa_faults =3D NULL, + .promo_list =3D LIST_HEAD_INIT(init_task.promo_list), #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) .kasan_depth =3D 1, diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a59ae2e23daf..047f02091773 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -1842,7 +1843,7 @@ static bool pgdat_free_space_enough(struct pglist_dat= a *pgdat) * The smaller the hint page fault latency, the higher the possibility * for the page to be hot. */ -static int numa_hint_fault_latency(struct folio *folio) +int numa_hint_fault_latency(struct folio *folio) { int last_time, time; =20 @@ -3534,6 +3535,27 @@ static void task_numa_work(struct callback_head *wor= k) } } =20 +static void task_numa_promotion_work(struct callback_head *work) +{ + struct task_struct *p =3D current; + struct list_head *promo_list =3D &p->promo_list; + struct folio *folio, *tmp; + int nid =3D numa_node_id(); + + SCHED_WARN_ON(p !=3D container_of(work, struct task_struct, numa_promo_wo= rk)); + + work->next =3D work; + + if (list_empty(promo_list)) + return; + + list_for_each_entry_safe(folio, tmp, promo_list, lru) { + list_del_init(&folio->lru); + migrate_misplaced_folio(folio, nid); + } +} + + void init_numa_balancing(unsigned long clone_flags, struct task_struct *p) { int mm_users =3D 0; @@ -3558,8 +3580,10 @@ void init_numa_balancing(unsigned long clone_flags, = struct task_struct *p) RCU_INIT_POINTER(p->numa_group, NULL); p->last_task_numa_placement =3D 0; p->last_sum_exec_runtime =3D 0; + INIT_LIST_HEAD(&p->promo_list); =20 init_task_work(&p->numa_work, task_numa_work); + init_task_work(&p->numa_promo_work, task_numa_promotion_work); =20 /* New address space, reset the preferred nid */ if (!(clone_flags & CLONE_VM)) { diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c index fc14fe53e9b7..4c44598e485e 100644 --- a/mm/memory-tiers.c +++ b/mm/memory-tiers.c @@ -935,6 +935,7 @@ static int __init memory_tier_init(void) subsys_initcall(memory_tier_init); =20 bool numa_demotion_enabled =3D false; +bool numa_pagecache_promotion_enabled; =20 #ifdef CONFIG_MIGRATION #ifdef CONFIG_SYSFS @@ -957,11 +958,37 @@ static ssize_t demotion_enabled_store(struct kobject = *kobj, return count; } =20 +static ssize_t pagecache_promotion_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%s\n", + numa_pagecache_promotion_enabled ? "true" : "false"); +} + +static ssize_t pagecache_promotion_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + ssize_t ret; + + ret =3D kstrtobool(buf, &numa_pagecache_promotion_enabled); + if (ret) + return ret; + + return count; +} + + static struct kobj_attribute numa_demotion_enabled_attr =3D __ATTR_RW(demotion_enabled); =20 +static struct kobj_attribute numa_pagecache_promotion_enabled_attr =3D + __ATTR_RW(pagecache_promotion_enabled); + static struct attribute *numa_attrs[] =3D { &numa_demotion_enabled_attr.attr, + &numa_pagecache_promotion_enabled_attr.attr, NULL, }; =20 diff --git a/mm/migrate.c b/mm/migrate.c index af07b399060b..320258a1aaba 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -44,6 +44,8 @@ #include #include #include +#include +#include =20 #include =20 @@ -2710,5 +2712,62 @@ int migrate_misplaced_folio(struct folio *folio, int= node) BUG_ON(!list_empty(&migratepages)); return nr_remaining ? -EAGAIN : 0; } + +/** + * promotion_candidate() - report a promotion candidate folio + * + * @folio: The folio reported as a candidate + * + * Records folio access time and places the folio on the task promotion li= st + * if access time is less than the threshold. The folio will be isolated f= rom + * LRU if selected, and task_work will putback the folio on promotion fail= ure. + * + * If selected, takes a folio reference to be released in task work. + */ +void promotion_candidate(struct folio *folio) +{ + struct task_struct *task =3D current; + struct list_head *promo_list =3D &task->promo_list; + struct callback_head *work =3D &task->numa_promo_work; + struct address_space *mapping =3D folio_mapping(folio); + bool write =3D mapping ? mapping->gfp_mask & __GFP_WRITE : false; + int nid =3D folio_nid(folio); + int flags, last_cpupid; + + /* + * Only do this work if: + * 1) tiering and pagecache promotion are enabled + * 2) the page can actually be promoted + * 3) The hint-fault latency is relatively hot + * 4) the folio is not already isolated + * 5) This is not a kernel thread context + */ + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + !numa_pagecache_promotion_enabled || + node_is_toptier(nid) || + numa_hint_fault_latency(folio) >=3D PAGE_ACCESS_TIME_MASK || + folio_test_isolated(folio) || + (current->flags & PF_KTHREAD)) { + return; + } + + nid =3D numa_migrate_check(folio, NULL, 0, &flags, write, &last_cpupid); + if (nid =3D=3D NUMA_NO_NODE) + return; + + if (migrate_misplaced_folio_prepare(folio, NULL, nid)) + return; + + /* + * Ensure task can schedule work, otherwise we'll leak folios. + * If the list is not empty, task work has already been scheduled. + */ + if (list_empty(promo_list) && task_work_add(task, work, TWA_RESUME)) { + folio_putback_lru(folio); + return; + } + list_add(&folio->lru, promo_list); +} +EXPORT_SYMBOL(promotion_candidate); #endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */ diff --git a/mm/swap.c b/mm/swap.c index 320b959b74c6..57909c349388 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -37,6 +37,7 @@ #include #include #include +#include =20 #include "internal.h" =20 @@ -469,6 +470,8 @@ void folio_mark_accessed(struct folio *folio) __lru_cache_activate_folio(folio); folio_clear_referenced(folio); workingset_activation(folio); + } else { + promotion_candidate(folio); } if (folio_test_idle(folio)) folio_clear_idle(folio); --=20 2.43.0