From nobody Fri Oct 10 21:13:59 2025 Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com [209.85.219.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2465D29B766; Thu, 12 Jun 2025 18:15:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752108; cv=none; b=EpxNrRzp4zmO1+e6RjgGwIZMwdsIpfCBWIeveyWosDmo3umnQtpPXDmPWlCwJksEnM7byON+wdjkysUePVeuvGFsIXj2qLojHF3Boq295QQrkAd+PI3Jl30lW8+/j/ZYG/DbMlC67JxqiEkJMUwyPsqRi8HmGBbjq+G1FOgXjvA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752108; c=relaxed/simple; bh=oyqZmRXZ4nv9NU3aggfqss2LA4D+iQG3/UH+ZUIJS9U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZIR/cg/CPvk3E4B9mCaXRv5xK0/Ahy3zhwzyKAJxKfNpWzTDKAbmKrsoWutZ7Em+m8C6tidzJyU9uNAf8+blLejD+d2MvtQkeXtdHTtpYTjpZu4XXlFIeu1Gh0Tn2yytAXlWYZ+1+DbJCI0miE8caSYucSTdgbQTZlwVoN9I+rY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=e4U2VbFC; arc=none smtp.client-ip=209.85.219.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="e4U2VbFC" Received: by mail-yb1-f181.google.com with SMTP id 3f1490d57ef6-e7b4ba530feso1114790276.1; Thu, 12 Jun 2025 11:15:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749752106; x=1750356906; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V++6cv3qI3iZESOVkUblHQI5RcqouiBvkHuiKma2hsk=; b=e4U2VbFCV4ZhapRm0OOnx3kASwLpLCIf/1Y9QHpXA5r12AdfcShDlK2vGUcfVfsaCK 9KBrdabl8Wh/7FwKvW06LEOf53X97Jxb0wctMOqg5uOWQXK1DtrwP4nhEtBCoPgtidTW rynilKCxhd8KzPGD1k8vImH/fdR1KqIuDV8PWE6syc74z3i6uGRGg7mW+ytkF3Z7JF6u hVYKje/TBLHTCb2p+C+7sR6FWbFnUf8keajx5+A3v/tgvmvdTTZABuEWachhYcwTDYqA p7P/rDqM3g1Y9/FGaKGVs9OJtsMiN+AZmH9MOO1ML0Jlmf+iEWx5cbeMGS4evaw+Cn90 0PHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749752106; x=1750356906; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V++6cv3qI3iZESOVkUblHQI5RcqouiBvkHuiKma2hsk=; b=TliRPEOb4JsJD9pG/BuXAhxUMMHDEFFFRnp1nhzPlBohKRfBZpNwvWn1VMA+1HEpMt 4iZ0KcEYqxhMZD6rZ809t4+OnuKxdOEPPKRADk+XvcdQhWpGuaAX7HeFbMVaAr3v6moF 3Dz26gpRECaQYfKgsjraHs0lf8DLUlFRwwd+ZuxhqkacwKYHZkFsIswYPNwDsIlar7P3 iDnhz1n1jJA61JatVp12ivAT2EL9Cg+T7zMcIZx9gbiYrR3oPHhmGsjdezFA+5+VNeLc /hOHY631AlwRzq31E23VJ1H6ybxS50Rl6gpP8TDAmGvtYJg/m+kGmPeTvo1ne1WX3xJL Od2Q== X-Forwarded-Encrypted: i=1; AJvYcCWDmyugdIUvIYmHl/Z5LvikPlS4nwV9J3jrRZ6Kg16Q5y0x/A2T7DXZNLhlqlDFGlr7J+Bg6DVsQdRI1M1V@vger.kernel.org, AJvYcCXaOaiMn7caa9bk+DCvcpJ38QCBjIE/relN3Ibi4beXAuuYUPr747h+nCiFxhTX4v7QBY5kDBq4jmA=@vger.kernel.org X-Gm-Message-State: AOJu0YxtnroQcn3BDvAAka4sSj1L27V+lFxV2mytIr6fq+tbA8T0oHoC 0D/0oF7I8HITIl30mN8W0wT7enIVZqmVRgpDcKz/NKYg2ibupRG+qmpkr4uiSmQtjww= X-Gm-Gg: ASbGncsh0xH1Lw26DIeHJjEMkwetRCTCZURiuAF38TLlrXZ/qY9qPpGG1d6cpO3TYjB P40wMNnTZItplDsLcMRRa0Q/2C8Izp6FQePr68r1Oklh+HWZZLkdIFoMYCYolB98nOAf/7W2zDd 9UMPAS5YsjYh7+Wg0+dNhHfJD3h9BifRMS4E70ZSABQvAlCwI3mmfFmURMPWi9KU7VwtezWKYiT ZGrE1LmEiks2Bt7jfEZAxzBRvX2DMd/QPRxRpdqQ32VDSVN8QlV7tkKP6LgHkGI7+tdZvCZWyMc qY2GEzAvvHH2qSa6DUGN8vqco3HGf/pTpn0vY7xcxikbEx+M6dQQtmapsM2zErXaQu85kYgUFxj j7POmDV4= X-Google-Smtp-Source: AGHT+IHBLjOPvosv/hB5QLaCCYrNVuZy979I3J7L41vsNM7eevjRMEzQ+OyB6twZbnVhhLkQ5/HufA== X-Received: by 2002:a05:6902:200f:b0:e81:32c9:f9c7 with SMTP id 3f1490d57ef6-e820b64cc40mr6125353276.5.1749752105996; Thu, 12 Jun 2025 11:15:05 -0700 (PDT) Received: from bijan-laptop.attlocal.net ([2600:1700:680e:c000:dd1b:d4ae:15de:11db]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e820e312452sm592480276.40.2025.06.12.11.15.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jun 2025 11:15:05 -0700 (PDT) From: Bijan Tabatabai To: damon@lists.linux.com, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: [RFC PATCH 1/4] mm/mempolicy: Expose policy_nodemask() in include/linux/mempolicy.h Date: Thu, 12 Jun 2025 13:13:27 -0500 Message-ID: <20250612181330.31236-2-bijan311@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250612181330.31236-1-bijan311@gmail.com> References: <20250612181330.31236-1-bijan311@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Bijan Tabatabai This patch is to allow DAMON to call policy_nodemask() so it can determine where to place a page for interleaving. Signed-off-by: Bijan Tabatabai --- include/linux/mempolicy.h | 9 +++++++++ mm/mempolicy.c | 4 +--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h index 0fe96f3ab3ef..e96bf493ff7a 100644 --- a/include/linux/mempolicy.h +++ b/include/linux/mempolicy.h @@ -133,6 +133,8 @@ struct mempolicy *__get_vma_policy(struct vm_area_struc= t *vma, struct mempolicy *get_vma_policy(struct vm_area_struct *vma, unsigned long addr, int order, pgoff_t *ilx); bool vma_policy_mof(struct vm_area_struct *vma); +nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, + pgoff_t ilx, int *nid); =20 extern void numa_default_policy(void); extern void numa_policy_init(void); @@ -232,6 +234,13 @@ static inline struct mempolicy *get_vma_policy(struct = vm_area_struct *vma, return NULL; } =20 +static inline nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, + pgoff_t ilx, int *nid) +{ + *nid =3D NUMA_NO_NODE; + return NULL; +} + static inline int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) { diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 3b1dfd08338b..54f539497e20 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -596,8 +596,6 @@ static const struct mempolicy_operations mpol_ops[MPOL_= MAX] =3D { =20 static bool migrate_folio_add(struct folio *folio, struct list_head *folio= list, unsigned long flags); -static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, - pgoff_t ilx, int *nid); =20 static bool strictly_unmovable(unsigned long flags) { @@ -2195,7 +2193,7 @@ static unsigned int interleave_nid(struct mempolicy *= pol, pgoff_t ilx) * Return a nodemask representing a mempolicy for filtering nodes for * page allocation, together with preferred node id (or the input node id). */ -static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, +nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, pgoff_t ilx, int *nid) { nodemask_t *nodemask =3D NULL; --=20 2.43.5 From nobody Fri Oct 10 21:13:59 2025 Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8262329A327; Thu, 12 Jun 2025 18:15:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752116; cv=none; b=UGaWES3ZbdxOb198ujEfUCPtwT1VOUT9GcK7LOzUm9zxtjTWcKmm4WXeU6+kfX/KHllIjxrlboJfoCWe2L61ihwe8q7EKHAvpKYdUU6SNiNeics+IAeafK+QzFZLZResFHayEvwyigTa9yX/L18vNwpAdgTv0ZuFre1V/cSZfr8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752116; c=relaxed/simple; bh=LuA5TzrXw8UwY+IQG/F7IxJh4SrAKbIGWRkyONKmnbk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QGtsB54wuOAxfRfWJA0R1LUt/LmpAApVIU2gZinfN18YVRMPnPj9kbsPPPuDJ2XfG27INURJTPEWKvRqlKgcl2iSwXrpnr9Zj1uSl87DwFr8xlHm5y9g0vUBYPQDA3ZK4TGSe9gJUDE7nHt7NLChnuvjsjHi7BDK+jxk8qdaFMg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f0oAbB6C; arc=none smtp.client-ip=209.85.219.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f0oAbB6C" Received: by mail-yb1-f172.google.com with SMTP id 3f1490d57ef6-e733e25bfc7so1121147276.3; Thu, 12 Jun 2025 11:15:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749752113; x=1750356913; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rI4RhBSsXuuVnpj1aLVVp/cqvhqfeHIgQ1Bp+6ronfs=; b=f0oAbB6CQQRakReSJh9DwPxzPgCuwJ+KLRi1L26MYJ/3MDb5pfHb9XOTUnD/TegTBa 58rF759xHJwN7lNWpI+FVkbxleKirP71ahb9smxRbooTkxiQfDX6woI3WG1A/7i+Zlf2 Rn4ziizcrnrzmKsmimV/tNAbmzZudul4NlJnf3opO4lGog3R7bKVgHwkrIrDMhMmko7T sfB5yxam3A5tePSLSihlQ28F6QJznzDGvHQUm2usu6/fzDs8kfTYq3ke912BFiTBCYoR rcyfbApaqHAByXCoUPEPLnh5oKuLFlP2NyLARGDhYo73zFUOvOxKvsWAu7hjRDTyWApd VzFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749752113; x=1750356913; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rI4RhBSsXuuVnpj1aLVVp/cqvhqfeHIgQ1Bp+6ronfs=; b=cElNiYzaGOEjzCX3pGRzS1TtzYUdi4MfDoDqG5VTSK4lf1TNmPoG2q9IN1GSN7r0X+ q7LuSTgqcpU8S1Cw6BtaSXo6BO9l+F9i5D/Qpw+LUcZb3I6V4mPmE/71WDi4ntgvO/ed fSFvNDSZH4kouD4czCkS4XVrHmjVhyewW528hIcmNBFnN//ej8xAddyZbwP+/Idt1fKQ gESB9PBo7sW8Fp9nhCaXIiis0cQPvXxYk3DF7QKDrlNlp+eL6l3q4EZohIWIDntuYl86 cKKOlGAjSlqUyck6uxbC1L5cN5lg/nLLVH1AoVVhxXHptyCKtN6SNrWkyZN2y92S43+u rDTw== X-Forwarded-Encrypted: i=1; AJvYcCV+yd2owgJiyFwsMVjalNAeuXYDs9cbX6PBbJ+80Zi3RQybDhvaJ5PyaV2mz6SK9xN3bwEF/rvr5mo=@vger.kernel.org, AJvYcCWAoIuy2z6i2l6FBUMextUnYwQleW2pseNTierkm+j5MN/cKSnB+Zf7XeZMqWBHPFm7wESrbz7c5I1ea3MH@vger.kernel.org X-Gm-Message-State: AOJu0Yz9s2+5RBcTm9DmcbaZlYDTVUQr8KJUTYclz/L/mUUe7XrCvIxA 0G49szeIqzk3NvCpA71xw/w4PDiqRdwobLTZp/PmHqqpbqkmrzmoASP4 X-Gm-Gg: ASbGncs+I3u84oG6Sz/r5cX3vfPViw/IXx7JBq5+xTfzI9DWLnzCf98Mq3cL5OVexDO upyZi7/G3OiJIUXypcU41TVKCWGZXrsEH7lVPypqdDptq1fzD5arCH5xRxgyUrTXGEQrsrBzuTb T6ZoN8nH9Q6ydwHZcMS1VlSJ9csFFobdXtXOIgbGi88JJa/1iDg7MtniHwmLSYzSiuwhgB6DqVL ama/vK9ycgPMLz6IiRMwgOvBF4cyOSguQbLf2JSSr2vfY4KkEtYIrLucuiQ6v47w874lJ9eBsg1 N7yv1f3sK+QQgy7OWquLmElZwzhffYQ/XsEeCM1C8gI1NLYyx8Cc337M02/ddmqJqu1LF55HJpd jHedN5GIfOFBbaOLHfQ== X-Google-Smtp-Source: AGHT+IFRULeRqZvjiyXhrkg+34BsFqcLiAM8WQUjj7GW8kCxanbKyVwN1t/1QqcXNLvmk8wOV/g/ow== X-Received: by 2002:a05:6902:220c:b0:e81:d976:ce25 with SMTP id 3f1490d57ef6-e820b64c93cmr6416416276.3.1749752113416; Thu, 12 Jun 2025 11:15:13 -0700 (PDT) Received: from bijan-laptop.attlocal.net ([2600:1700:680e:c000:dd1b:d4ae:15de:11db]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e820e312452sm592480276.40.2025.06.12.11.15.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jun 2025 11:15:13 -0700 (PDT) From: Bijan Tabatabai To: damon@lists.linux.com, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: [RFC PATCH 2/4] mm/damon/paddr: Add DAMOS_INTERLEAVE action Date: Thu, 12 Jun 2025 13:13:28 -0500 Message-ID: <20250612181330.31236-3-bijan311@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250612181330.31236-1-bijan311@gmail.com> References: <20250612181330.31236-1-bijan311@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Bijan Tabatabai This patch adds the DAMOS_INTERLEAVE action. It interleaves pages inside of a given region according to the weights in the iw_table. To reuse existing interleaving code, the target nid for a folio is determined by calling policy_nodemask, therefore only folios belonging to processes using the MPOL_WEIGHTED_INTERLEAVE policy will have their pages migrated. Below is an example of its usage where pages are initially interleaved at a 1:1 ratio and then changed to be interleaved at a 2:1 ratio. The alloc_data program simply allocates 1GB of data then sleeps. $ cd /sys/kernel/mm/damon/admin/kdamonds/0 $ sudo cat ./contexts/0/schemes/0/action interleave $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node0 $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node1 $ numactl -w 0,1 ~/alloc_data 1G & $ numastat -c -p alloc_data Per-node process memory usage (in MBs) for PID 18473 (alloc_data) Node 0 Node 1 Total ------ ------ ----- Huge 0 0 0 Heap 0 0 0 Stack 0 0 0 Private 514 514 1027 ------- ------ ------ ----- Total 514 514 1028 $ echo 2 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node0 $ numastat -c -p alloc_data Per-node process memory usage (in MBs) for PID 18473 (alloc_data) Node 0 Node 1 Total ------ ------ ----- Huge 0 0 0 Heap 0 0 0 Stack 0 0 0 Private 684 343 1027 ------- ------ ------ ----- Total 684 343 1027 Signed-off-by: Bijan Tabatabai --- Documentation/mm/damon/design.rst | 2 + include/linux/damon.h | 2 + mm/damon/paddr.c | 112 ++++++++++++++++++++++++++++++ mm/damon/sysfs-schemes.c | 1 + 4 files changed, 117 insertions(+) diff --git a/Documentation/mm/damon/design.rst b/Documentation/mm/damon/des= ign.rst index ddc50db3afa4..c50d2105cea0 100644 --- a/Documentation/mm/damon/design.rst +++ b/Documentation/mm/damon/design.rst @@ -455,6 +455,8 @@ that supports each action are as below. Supported by ``paddr`` operations set. - ``migrate_cold``: Migrate the regions prioritizing colder regions. Supported by ``paddr`` operations set. + - ``interleave``: Interleave the regions according to the weighted interl= eave weights. + Supported by ``paddr`` operations set. - ``stat``: Do nothing but count the statistics. Supported by all operations sets. =20 diff --git a/include/linux/damon.h b/include/linux/damon.h index a4011726cb3b..81d26a203337 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -117,6 +117,7 @@ struct damon_target { * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists. * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions. * @DAMOS_MIGRATE_COLD: Migrate the regions prioritizing colder regions. + * @DAMOS_INTERLEAVE: Interleave the regions by the weighted interleave ra= tio * @DAMOS_STAT: Do nothing but count the stat. * @NR_DAMOS_ACTIONS: Total number of DAMOS actions * @@ -136,6 +137,7 @@ enum damos_action { DAMOS_LRU_DEPRIO, DAMOS_MIGRATE_HOT, DAMOS_MIGRATE_COLD, + DAMOS_INTERLEAVE, DAMOS_STAT, /* Do nothing but only record the stat */ NR_DAMOS_ACTIONS, }; diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 4102a8c5f992..e989464635cd 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -535,6 +535,114 @@ static unsigned long damon_pa_migrate(struct damon_re= gion *r, struct damos *s, return applied * PAGE_SIZE; } =20 +#if defined(CONFIG_MEMCG) && defined(CONFIG_NUMA) +struct damos_interleave_private { + struct list_head *folio_migration_list; + bool putback_lru; +}; + +static bool damon_pa_interleave_rmap(struct folio *folio, struct vm_area_s= truct *vma, + unsigned long addr, void *arg) +{ + struct mempolicy *pol; + struct task_struct *task; + pgoff_t ilx; + int target_nid; + struct damos_interleave_private *priv =3D arg; + + task =3D rcu_dereference(vma->vm_mm->owner); + if (!task) + return true; + + pol =3D get_task_policy(task); + if (!pol) + return true; + + /* Getting the interleave weights only makes sense with MPOL_WEIGHTED_INT= ERLEAVE */ + if (pol->mode !=3D MPOL_WEIGHTED_INTERLEAVE) { + mpol_cond_put(pol); + return true; + } + + ilx =3D vma->vm_pgoff >> folio_order(folio); + ilx +=3D (addr - vma->vm_start) >> (PAGE_SHIFT + folio_order(folio)); + policy_nodemask(0, pol, ilx, &target_nid); + + if (target_nid !=3D NUMA_NO_NODE && folio_nid(folio) !=3D target_nid) { + list_add(&folio->lru, &priv->folio_migration_list[target_nid]); + priv->putback_lru =3D false; + } + + mpol_cond_put(pol); + return false; +} + +static unsigned long damon_pa_interleave(struct damon_region *r, struct da= mos *s, + unsigned long *sz_filter_passed) +{ + struct damos_interleave_private priv; + struct rmap_walk_control rwc; + unsigned long addr, applied; + struct folio *folio; + + priv.folio_migration_list =3D kmalloc_array(nr_node_ids, sizeof(struct li= st_head), + GFP_KERNEL); + if (!priv.folio_migration_list) + return 0; + + for (int i =3D 0; i < nr_node_ids; i++) + INIT_LIST_HEAD(&priv.folio_migration_list[i]); + + memset(&rwc, 0, sizeof(struct rmap_walk_control)); + rwc.rmap_one =3D damon_pa_interleave_rmap; + rwc.arg =3D &priv; + + addr =3D r->ar.start; + while (addr < r->ar.end) { + folio =3D damon_get_folio(PHYS_PFN(addr)); + + if (damon_pa_invalid_damos_folio(folio, s)) { + addr +=3D PAGE_SIZE; + continue; + } + + if (damos_pa_filter_out(s, folio)) + goto put_folio; + else + *sz_filter_passed +=3D folio_size(folio); + + if (!folio_isolate_lru(folio)) + goto put_folio; + + priv.putback_lru =3D true; + rmap_walk(folio, &rwc); + + if (priv.putback_lru) + folio_putback_lru(folio); + +put_folio: + addr +=3D folio_size(folio); + folio_put(folio); + } + + applied =3D 0; + for (int i =3D 0; i < nr_node_ids; i++) { + applied +=3D damon_pa_migrate_pages(&priv.folio_migration_list[i], i); + cond_resched(); + } + + kfree(priv.folio_migration_list); + s->last_applied =3D folio; + return applied * PAGE_SIZE; +} +#else +static unsigned long damon_pa_interleave(struct damon_region *r, struct da= mos *s, + unsigned long *sz_filter_passed) +{ + return 0; +} +#endif /* defined(CONFIG_MEMCG) && defined(CONFIG_NUMA) */ + static bool damon_pa_scheme_has_filter(struct damos *s) { struct damos_filter *f; @@ -584,6 +692,8 @@ static unsigned long damon_pa_apply_scheme(struct damon= _ctx *ctx, case DAMOS_MIGRATE_HOT: case DAMOS_MIGRATE_COLD: return damon_pa_migrate(r, scheme, sz_filter_passed); + case DAMOS_INTERLEAVE: + return damon_pa_interleave(r, scheme, sz_filter_passed); case DAMOS_STAT: return damon_pa_stat(r, scheme, sz_filter_passed); default: @@ -608,6 +718,8 @@ static int damon_pa_scheme_score(struct damon_ctx *cont= ext, return damon_hot_score(context, r, scheme); case DAMOS_MIGRATE_COLD: return damon_cold_score(context, r, scheme); + case DAMOS_INTERLEAVE: + return damon_hot_score(context, r, scheme); default: break; } diff --git a/mm/damon/sysfs-schemes.c b/mm/damon/sysfs-schemes.c index 0f6c9e1fec0b..12a32e066d06 100644 --- a/mm/damon/sysfs-schemes.c +++ b/mm/damon/sysfs-schemes.c @@ -1597,6 +1597,7 @@ static const char * const damon_sysfs_damos_action_st= rs[] =3D { "lru_deprio", "migrate_hot", "migrate_cold", + "interleave", "stat", }; =20 --=20 2.43.5 From nobody Fri Oct 10 21:13:59 2025 Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F95C1632DD; Thu, 12 Jun 2025 18:15:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752125; cv=none; b=jNHl85uvkfd4O9lFu565aplJRLQlt79SqCgq/5L2BEnZnQjaAAYDtbAgBCnbvWM1+BY5HcFixdQMc77LDRUCm3wbX5MIkaQkrR6kyuGo26lzbrxSbbhzoiKwds3dJ0ECCuuAUdF0NQtgerBA+R0oThArfFrXqXT9wYWRS7ndM78= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752125; c=relaxed/simple; bh=4ECuvmuOJa1WL+atdNsEmrWyTpz1tzuFAE4YXPmPJXw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eNarTpFAgT6GjD5BARVfeR7B+lGFdXa/3F/p5PzBqw3dRw5xVwEUGQMgy7DVsD6Da31cub8O7OUPNy4xM4Z9aU7M6kNUQPHteqaPxMCAXL+dlYx0kbHBtjDIwvmJ00GO5lcAA7VWWLB2MqjyJjtZVqetTpsNpV1vbIGnWZZkJDk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bwd6g1vT; arc=none smtp.client-ip=209.85.219.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bwd6g1vT" Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-e7db5c13088so1152437276.1; Thu, 12 Jun 2025 11:15:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749752122; x=1750356922; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LArn2/NqwWy0wVHKXQau9kT4miDpoQMz65raO0syifs=; b=bwd6g1vTbLDVG/nGXzGhB7aTVM7x8vbmPX//5HpKEGz9ApsF5bAbgJSo4FY2gVe2f4 WyRqLzX2ygUhtf3Tdk87myrG025bBfog9qvFjySagnZxWNge+T6mSXAaCtzDLvmIs5Mt rn2WoR3mLLuYXiTUDb4hwC4ysmKKIl9H85wKREN4DAExf3suVv1NhpX/ZvK/FN4xEowG 5/xlcVBWdKOC91uWRR5MxeZZMavUlZmod5N5IHjyW4ZKR1wizEcTwBlO/nvuYMH+sjRR llnLxejTnjEWp4iuQshTl55iWgzYgw/bzlTBkF3h8reBh5/EXCkYZ77KrgfSdagomeJ/ THpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749752122; x=1750356922; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LArn2/NqwWy0wVHKXQau9kT4miDpoQMz65raO0syifs=; b=oDAwZ3rGaQo8rFZ+eAMMNPyyFbFlsinxTJclL6+kUGj7/aldqvgN32XK09KxQG3Nvt 7Rtc1RtBi+Yp3BikYpLW4Ewcn1iqPNzs18Vc8r75J8blQQzpwI5SrfCWqVYM1j1ZqUew FZnx01jeWICwlzLhM4d7JagYyK6lTIjSYClWKHClHgSuuMSb7DHkN9AJtT3vqJ+pUO+s h70xMOteFx0Slr3oqgNBLQ/TiA1YrB2yGE/oD3T6RaZx/X5pEqauEkiUMJ79nj17cPA8 76SPLanIkPUPQsX8Hst98pgWJDYkGSrzVpMDMAVVWYO1zbqrvhoODhN3AbLI7gG6p7pR K59Q== X-Forwarded-Encrypted: i=1; AJvYcCWIBIYoK0DuYFALRsZ8FX2ei62UGqVjX6cT+t6KJT+LirH/BeQUuottZJjcgkluXYM2HQVos/NGy6p1bwRv@vger.kernel.org, AJvYcCXU9RNRojvefs5C9803PH+FMpOHzhdJxV3dH8LPQvvZBmtibazq0EoNSVWCx+S/aKYNXI07K0r+fyU=@vger.kernel.org X-Gm-Message-State: AOJu0YwnrYXxDXiSS0wUvJxZQYaQO+wwDmzKt/48hD25GeUt40wgsxGu xG0cJcx4Gp8LGPo5S05ty5z1G5G4ME8Ggc9v3ft3OmyvcylvrC5I4kg2tUo0AX80zRk= X-Gm-Gg: ASbGncvb+GyZc8axHPoiQaVMp9YeMJ/P/ZUGvIxyxQpu9lde6MM6AaZlP8+yYBufWtj DgRQibpEbW7K3FAs9zErUoHK2pa/E3C+JQ2ZL2gVKV77kcb+p5M2b9rDYtmtfqTdEWQ4lrBKDCl agm+QLODXzJvbi2e4DtzXs4BWgO5zGdo/C3fjH1+o7QZeVTdOuloP3ORkfjtafQxKgUlHylW9VD hOqHq8jtRHMBieBcqhnnaBWj9wIVNjBJ9h6tDREJc1SLXIJ6Tz54OgEGf9RUaMwfKEO1MW9/FWK 7GZ/iMSXGgO3ZV9iZWfp7ANA1/fDpyJjv8+DxB33Ojy3YWvDRkA0JruX5yYifI1+0vfQTKQIw/d 4cxULYks= X-Google-Smtp-Source: AGHT+IH8StR2oqDE+bGgZZWyYluYInSYI54x04NKJ85FqXoRUgJbr/xsBUEHZcj6UiB3fcMUvqy1wA== X-Received: by 2002:a05:6902:220c:b0:e81:b5c8:3d71 with SMTP id 3f1490d57ef6-e820baf86edmr5215637276.17.1749752122004; Thu, 12 Jun 2025 11:15:22 -0700 (PDT) Received: from bijan-laptop.attlocal.net ([2600:1700:680e:c000:dd1b:d4ae:15de:11db]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e820e312452sm592480276.40.2025.06.12.11.15.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jun 2025 11:15:21 -0700 (PDT) From: Bijan Tabatabai To: damon@lists.linux.com, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: [RFC PATCH 3/4] mm/damon: Move damon_pa_migrate_pages to ops-common Date: Thu, 12 Jun 2025 13:13:29 -0500 Message-ID: <20250612181330.31236-4-bijan311@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250612181330.31236-1-bijan311@gmail.com> References: <20250612181330.31236-1-bijan311@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Bijan Tabatabai This patch moves damon_pa_migrate_pages from mm/damon/paddr.c to mm/damon/ops-common.c and renames it damon_migrate_pages. This gives vaddr based actions the ability to migrate pages as well. Signed-off-by: Bijan Tabatabai --- mm/damon/ops-common.c | 123 +++++++++++++++++++++++++++++++++++++++++ mm/damon/ops-common.h | 2 + mm/damon/paddr.c | 125 +----------------------------------------- 3 files changed, 127 insertions(+), 123 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index b43620fee6bb..2c4fb274b7f6 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -5,6 +5,7 @@ * Author: SeongJae Park */ =20 +#include #include #include #include @@ -12,6 +13,7 @@ #include #include =20 +#include "../internal.h" #include "ops-common.h" =20 /* @@ -138,3 +140,124 @@ int damon_cold_score(struct damon_ctx *c, struct damo= n_region *r, /* Return coldness of the region */ return DAMOS_MAX_SCORE - hotness; } + +static unsigned int __damon_migrate_folio_list( + struct list_head *migrate_folios, struct pglist_data *pgdat, + int target_nid) +{ + unsigned int nr_succeeded =3D 0; + nodemask_t allowed_mask =3D NODE_MASK_NONE; + struct migration_target_control mtc =3D { + /* + * Allocate from 'node', or fail quickly and quietly. + * When this happens, 'page' will likely just be discarded + * instead of migrated. + */ + .gfp_mask =3D (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | + __GFP_NOWARN | __GFP_NOMEMALLOC | GFP_NOWAIT, + .nid =3D target_nid, + .nmask =3D &allowed_mask + }; + + if (pgdat->node_id =3D=3D target_nid || target_nid =3D=3D NUMA_NO_NODE) + return 0; + + if (list_empty(migrate_folios)) + return 0; + + /* Migration ignores all cpuset and mempolicy settings */ + migrate_pages(migrate_folios, alloc_migrate_folio, NULL, + (unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON, + &nr_succeeded); + + return nr_succeeded; +} + +static unsigned int damon_migrate_folio_list(struct list_head *folio_list, + struct pglist_data *pgdat, + int target_nid) +{ + unsigned int nr_migrated =3D 0; + struct folio *folio; + LIST_HEAD(ret_folios); + LIST_HEAD(migrate_folios); + + while (!list_empty(folio_list)) { + struct folio *folio; + + cond_resched(); + + folio =3D lru_to_folio(folio_list); + list_del(&folio->lru); + + if (!folio_trylock(folio)) + goto keep; + + /* Relocate its contents to another node. */ + list_add(&folio->lru, &migrate_folios); + folio_unlock(folio); + continue; +keep: + list_add(&folio->lru, &ret_folios); + } + /* 'folio_list' is always empty here */ + + /* Migrate folios selected for migration */ + nr_migrated +=3D __damon_migrate_folio_list( + &migrate_folios, pgdat, target_nid); + /* + * Folios that could not be migrated are still in @migrate_folios. Add + * those back on @folio_list + */ + if (!list_empty(&migrate_folios)) + list_splice_init(&migrate_folios, folio_list); + + try_to_unmap_flush(); + + list_splice(&ret_folios, folio_list); + + while (!list_empty(folio_list)) { + folio =3D lru_to_folio(folio_list); + list_del(&folio->lru); + folio_putback_lru(folio); + } + + return nr_migrated; +} + +unsigned long damon_migrate_pages(struct list_head *folio_list, + int target_nid) +{ + int nid; + unsigned long nr_migrated =3D 0; + LIST_HEAD(node_folio_list); + unsigned int noreclaim_flag; + + if (list_empty(folio_list)) + return nr_migrated; + + noreclaim_flag =3D memalloc_noreclaim_save(); + + nid =3D folio_nid(lru_to_folio(folio_list)); + do { + struct folio *folio =3D lru_to_folio(folio_list); + + if (nid =3D=3D folio_nid(folio)) { + list_move(&folio->lru, &node_folio_list); + continue; + } + + nr_migrated +=3D damon_migrate_folio_list(&node_folio_list, + NODE_DATA(nid), + target_nid); + nid =3D folio_nid(lru_to_folio(folio_list)); + } while (!list_empty(folio_list)); + + nr_migrated +=3D damon_migrate_folio_list(&node_folio_list, + NODE_DATA(nid), + target_nid); + + memalloc_noreclaim_restore(noreclaim_flag); + + return nr_migrated; +} diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index cc9f5da9c012..54209a7e70e6 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -16,3 +16,5 @@ int damon_cold_score(struct damon_ctx *c, struct damon_re= gion *r, struct damos *s); int damon_hot_score(struct damon_ctx *c, struct damon_region *r, struct damos *s); + +unsigned long damon_migrate_pages(struct list_head *folio_list, int target= _nid); diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index e989464635cd..722d69f26e37 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -381,127 +381,6 @@ static unsigned long damon_pa_deactivate_pages(struct= damon_region *r, sz_filter_passed); } =20 -static unsigned int __damon_pa_migrate_folio_list( - struct list_head *migrate_folios, struct pglist_data *pgdat, - int target_nid) -{ - unsigned int nr_succeeded =3D 0; - nodemask_t allowed_mask =3D NODE_MASK_NONE; - struct migration_target_control mtc =3D { - /* - * Allocate from 'node', or fail quickly and quietly. - * When this happens, 'page' will likely just be discarded - * instead of migrated. - */ - .gfp_mask =3D (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | - __GFP_NOWARN | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid =3D target_nid, - .nmask =3D &allowed_mask - }; - - if (pgdat->node_id =3D=3D target_nid || target_nid =3D=3D NUMA_NO_NODE) - return 0; - - if (list_empty(migrate_folios)) - return 0; - - /* Migration ignores all cpuset and mempolicy settings */ - migrate_pages(migrate_folios, alloc_migrate_folio, NULL, - (unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON, - &nr_succeeded); - - return nr_succeeded; -} - -static unsigned int damon_pa_migrate_folio_list(struct list_head *folio_li= st, - struct pglist_data *pgdat, - int target_nid) -{ - unsigned int nr_migrated =3D 0; - struct folio *folio; - LIST_HEAD(ret_folios); - LIST_HEAD(migrate_folios); - - while (!list_empty(folio_list)) { - struct folio *folio; - - cond_resched(); - - folio =3D lru_to_folio(folio_list); - list_del(&folio->lru); - - if (!folio_trylock(folio)) - goto keep; - - /* Relocate its contents to another node. */ - list_add(&folio->lru, &migrate_folios); - folio_unlock(folio); - continue; -keep: - list_add(&folio->lru, &ret_folios); - } - /* 'folio_list' is always empty here */ - - /* Migrate folios selected for migration */ - nr_migrated +=3D __damon_pa_migrate_folio_list( - &migrate_folios, pgdat, target_nid); - /* - * Folios that could not be migrated are still in @migrate_folios. Add - * those back on @folio_list - */ - if (!list_empty(&migrate_folios)) - list_splice_init(&migrate_folios, folio_list); - - try_to_unmap_flush(); - - list_splice(&ret_folios, folio_list); - - while (!list_empty(folio_list)) { - folio =3D lru_to_folio(folio_list); - list_del(&folio->lru); - folio_putback_lru(folio); - } - - return nr_migrated; -} - -static unsigned long damon_pa_migrate_pages(struct list_head *folio_list, - int target_nid) -{ - int nid; - unsigned long nr_migrated =3D 0; - LIST_HEAD(node_folio_list); - unsigned int noreclaim_flag; - - if (list_empty(folio_list)) - return nr_migrated; - - noreclaim_flag =3D memalloc_noreclaim_save(); - - nid =3D folio_nid(lru_to_folio(folio_list)); - do { - struct folio *folio =3D lru_to_folio(folio_list); - - if (nid =3D=3D folio_nid(folio)) { - list_move(&folio->lru, &node_folio_list); - continue; - } - - nr_migrated +=3D damon_pa_migrate_folio_list(&node_folio_list, - NODE_DATA(nid), - target_nid); - nid =3D folio_nid(lru_to_folio(folio_list)); - } while (!list_empty(folio_list)); - - nr_migrated +=3D damon_pa_migrate_folio_list(&node_folio_list, - NODE_DATA(nid), - target_nid); - - memalloc_noreclaim_restore(noreclaim_flag); - - return nr_migrated; -} - static unsigned long damon_pa_migrate(struct damon_region *r, struct damos= *s, unsigned long *sz_filter_passed) { @@ -529,7 +408,7 @@ static unsigned long damon_pa_migrate(struct damon_regi= on *r, struct damos *s, addr +=3D folio_size(folio); folio_put(folio); } - applied =3D damon_pa_migrate_pages(&folio_list, s->target_nid); + applied =3D damon_migrate_pages(&folio_list, s->target_nid); cond_resched(); s->last_applied =3D folio; return applied * PAGE_SIZE; @@ -627,7 +506,7 @@ static unsigned long damon_pa_interleave(struct damon_r= egion *r, struct damos *s =20 applied =3D 0; for (int i =3D 0; i < nr_node_ids; i++) { - applied +=3D damon_pa_migrate_pages(&priv.folio_migration_list[i], i); + applied +=3D damon_migrate_pages(&priv.folio_migration_list[i], i); cond_resched(); } =20 --=20 2.43.5 From nobody Fri Oct 10 21:13:59 2025 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81EAC1A3154; Thu, 12 Jun 2025 18:15:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752134; cv=none; b=X5RX2NUoReX5KF8kJzjlrrD60hqJcHzNMChZT4nRAaLYNGZg0hcU3yymBeXaoxG3GCVKkj7cbCdMs0QrS8LKPelNd9hu5gE+BzyupE/eCJ1S6qMGorAAohma6pnzta0Ndble8VGZLvQoQoGnW7XGSHZbaN4UOo9bk1a+/MKR+eA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1749752134; c=relaxed/simple; bh=vj2LbrYA9jkg17qAZiV25qjeVHmAcoe1KQIetmpaIQk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gynzwTbXjWlt9/Sv32wjrGcei1GFBW7fnNE3wMhxW7vpWwYYqn/jv5WmkZzJwGBKNm43NNKh5Fx8rj511D+clMB8blA+jZJgxdFhLEUihh3ZUXeq8sejfeFnLe4M9m9Zfv/F+xmIhAlX+9olfkWcY7ftLH547rSUfNJmhlvWf34= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GSUhBZg3; arc=none smtp.client-ip=209.85.219.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GSUhBZg3" Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-e7b4ba530feso1115182276.1; Thu, 12 Jun 2025 11:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749752131; x=1750356931; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yJgUDGFlTqLrMXyweY6vXETHiIGKHN448RSe2G/RQd4=; b=GSUhBZg3FLX0/Ej/Q+swaM7fATpS2iObtFTBatP3FFwVjJAIyOF0N7GmGewzNMX4sV GRQsK+wqNVUAFGFDpRB+udxVRmE8kdsnr5OkHP2lQ5moaid85/FlZVL2mdZn8XD1WlQM H8D9ZNsNfC6JAonbhMMuKV4wi+10kwiH75kmJX0XOCBqwRbwxp0YsrXXjFp7EvKMCczb KsUAoaePw5Z4AGAgfwS0jJBwSvc0V2sg8UwiMktxuxqmJD1P68TD+OKPt8EJd8fEYZip wdLur6t5MLja3s/3PIcEhwy0n4r+UMqquXBxPmou6fcDklHcW5uq0pAxbOyr6kljCn9u AMIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749752131; x=1750356931; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yJgUDGFlTqLrMXyweY6vXETHiIGKHN448RSe2G/RQd4=; b=X0qs60JOOIGvkeGS4nzQLOQf3n+2L7gz8fCMYPZaVOT3fy2HOd9bHLQDACPcgOIdEv p7uHdyX/BygkzU2qh8cc69MLUZsp4u9OX58mK7qxPTM+01ruk8DMaa7OUzI6U6BBUeQF Wyngf1Q4EVVl0dK8MxzyS2doJNYMHw7utZq20ejOrfjNxNoWqj1c+hjn2Ro3xcDh8FHQ lRWMXGRHdB48tGq3v+U15hRxF2EM1ftyjVq5uUqjQA69USFvn0tdDI7JWWXqlZtxWLzd 1isWMbW0VYpoYYG0fBX0cMsIJDXDjIgSCzAZyOutWraQ03DkAj76nfzWb1Sh34tSS6t2 CPyA== X-Forwarded-Encrypted: i=1; AJvYcCUy0BhP2k6HIQVRXvuuuzbkuzw5iJp5BRMIj5sQWP2Bn9H7U4yG6B7b/Bkv5v3VtRiMNcdVTte4zHQpANQU@vger.kernel.org, AJvYcCWAkCwXXqreix8wfvUhVojIpK4LBPm4IRTxjuWyG9Z9pHMOcFTWPZZXy2AjqsYaQm8utkjJDxhfh2M=@vger.kernel.org X-Gm-Message-State: AOJu0Yz9zpn5DdShCq45HUiDMNcvEp0NzpdD9LthSz/vrPs6Z+YlEsvE qIY4EydKwgcBm5jTKMoni2g8/4MaMezqxL6hAlKP1crmQOs9zHJCYeGh X-Gm-Gg: ASbGncsNUR7CI/QKzJvmGmQT7D3b6lombwLnypqb9vS+avTNKbU+GhL63lFQdm6EPBs noKw8HtdEScdURtO/Zs6uu1SKKlwSZ9+jyJPSuhvpKuDJR3aY6iHetVHN88XFMIdyhSb6RskQQR 0ftUzpMmo5g4vSAAOSZfqFEMuZXQSlwC7OpGW8fKTLF6aFHJrMl9RQqL4l+C3Z13I/bmVHypbl6 LeY/fjtSvsZi2uzX6re8TCggR0yYaV9YdVTD5IcinqaGu52A3rw2atH4UPrqXh/MfsOnnMajz1t 5PWhtjZiWx0w6qrW1ilY0zHquUug7aCSxeNPT3YXG94wkVkCKRvbA+G+vydfdgLiDGjPIU9Zk/g XGSRsmbY= X-Google-Smtp-Source: AGHT+IGmai82U6it5jDvpRdZAOh/erEuNSBZ/HFRj2fOcXstR7G50ztztivaNFHDIHN9lND4uANkQQ== X-Received: by 2002:a05:6902:4790:b0:e7d:d162:f15a with SMTP id 3f1490d57ef6-e820b719f47mr6337384276.40.1749752131219; Thu, 12 Jun 2025 11:15:31 -0700 (PDT) Received: from bijan-laptop.attlocal.net ([2600:1700:680e:c000:dd1b:d4ae:15de:11db]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e820e312452sm592480276.40.2025.06.12.11.15.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jun 2025 11:15:30 -0700 (PDT) From: Bijan Tabatabai To: damon@lists.linux.com, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: [RFC PATCH 4/4] mm/damon/vaddr: Add vaddr version of DAMOS_INTERLEAVE Date: Thu, 12 Jun 2025 13:13:30 -0500 Message-ID: <20250612181330.31236-5-bijan311@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250612181330.31236-1-bijan311@gmail.com> References: <20250612181330.31236-1-bijan311@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Bijan Tabatabai This patch adds a vaddr implementation of the DAMOS_INTERLEAVE action. Below is an example of its usage where pages are initially interleaved at a 1:1 ratio and then changed to be interleaved at a 2:1 ratio. The alloc_data program simply allocates 1GB of data then sleeps. $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node0 $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node1 $ numactl -w 0,1 ./alloc_data 1G& [1] 11447 $ cat interleave_vaddr.yaml kdamonds: - state: null pid: null contexts: - ops: vaddr targets: - pid: 11447 regions: [] intervals: sample_us: 200 ms aggr_us: 5 s ops_update_us: 10 s nr_regions: min: 200 max: 500 schemes: - action: interleave access_pattern: sz_bytes: min: 0 B max: max nr_accesses: min: 0 % max: 100 % age: min: 0 ns max: max $ sudo ./damo/damo start interleave_vaddr.yaml $ numastat -c -p 11447 Per-node process memory usage (in MBs) for PID 11447 (alloc_data) Node 0 Node 1 Total ------ ------ ----- Huge 0 0 0 Heap 0 0 0 Stack 0 0 0 Private 514 514 1027 ------- ------ ------ ----- Total 514 514 1027 $ echo 2 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node0 $ numastat -c -p 11447 Per-node process memory usage (in MBs) for PID 11447 (alloc_data) Node 0 Node 1 Total ------ ------ ----- Huge 0 0 0 Heap 0 0 0 Stack 0 0 0 Private 684 343 1027 ------- ------ ------ ----- Total 684 343 1027 Signed-off-by: Bijan Tabatabai --- Documentation/mm/damon/design.rst | 2 +- mm/damon/ops-common.c | 13 +++ mm/damon/ops-common.h | 2 + mm/damon/paddr.c | 11 +-- mm/damon/vaddr.c | 135 ++++++++++++++++++++++++++++++ 5 files changed, 155 insertions(+), 8 deletions(-) diff --git a/Documentation/mm/damon/design.rst b/Documentation/mm/damon/des= ign.rst index c50d2105cea0..a79ba62f820b 100644 --- a/Documentation/mm/damon/design.rst +++ b/Documentation/mm/damon/design.rst @@ -456,7 +456,7 @@ that supports each action are as below. - ``migrate_cold``: Migrate the regions prioritizing colder regions. Supported by ``paddr`` operations set. - ``interleave``: Interleave the regions according to the weighted interl= eave weights. - Supported by ``paddr`` operations set. + Supported by ``vaddr``, ``fvaddr`` and ``paddr`` operations set. - ``stat``: Do nothing but count the statistics. Supported by all operations sets. =20 diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 2c4fb274b7f6..59d92404fc8f 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -261,3 +261,16 @@ unsigned long damon_migrate_pages(struct list_head *fo= lio_list, =20 return nr_migrated; } + +int damon_interleave_target_nid(unsigned long addr, struct vm_area_struct = *vma, + struct mempolicy *pol, struct folio *folio) +{ + pgoff_t ilx; + int target_nid; + + ilx =3D vma->vm_pgoff >> folio_order(folio); + ilx +=3D (addr - vma->vm_start) >> (PAGE_SHIFT + folio_order(folio)); + policy_nodemask(0, pol, ilx, &target_nid); + + return target_nid; +} diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index 54209a7e70e6..bacb4de92dc9 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -18,3 +18,5 @@ int damon_hot_score(struct damon_ctx *c, struct damon_reg= ion *r, struct damos *s); =20 unsigned long damon_migrate_pages(struct list_head *folio_list, int target= _nid); +int damon_interleave_target_nid(unsigned long addr, struct vm_area_struct = *vma, + struct mempolicy *pol, struct folio *folio); diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 722d69f26e37..93e3c72b54c7 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -415,7 +415,7 @@ static unsigned long damon_pa_migrate(struct damon_regi= on *r, struct damos *s, } =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_NUMA) -struct damos_interleave_private { +struct damos_pa_interleave_private { struct list_head *folio_migration_list; bool putback_lru; }; @@ -425,9 +425,8 @@ static bool damon_pa_interleave_rmap(struct folio *foli= o, struct vm_area_struct { struct mempolicy *pol; struct task_struct *task; - pgoff_t ilx; int target_nid; - struct damos_interleave_private *priv =3D arg; + struct damos_pa_interleave_private *priv =3D arg; =20 task =3D rcu_dereference(vma->vm_mm->owner); if (!task) @@ -443,9 +442,7 @@ static bool damon_pa_interleave_rmap(struct folio *foli= o, struct vm_area_struct return true; } =20 - ilx =3D vma->vm_pgoff >> folio_order(folio); - ilx +=3D (addr - vma->vm_start) >> (PAGE_SHIFT + folio_order(folio)); - policy_nodemask(0, pol, ilx, &target_nid); + target_nid =3D damon_interleave_target_nid(addr, vma, pol, folio); =20 if (target_nid !=3D NUMA_NO_NODE && folio_nid(folio) !=3D target_nid) { list_add(&folio->lru, &priv->folio_migration_list[target_nid]); @@ -459,7 +456,7 @@ static bool damon_pa_interleave_rmap(struct folio *foli= o, struct vm_area_struct static unsigned long damon_pa_interleave(struct damon_region *r, struct da= mos *s, unsigned long *sz_filter_passed) { - struct damos_interleave_private priv; + struct damos_pa_interleave_private priv; struct rmap_walk_control rwc; unsigned long addr, applied; struct folio *folio; diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 46554e49a478..1d1170f49317 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -9,12 +9,14 @@ =20 #include #include +#include #include #include #include #include #include =20 +#include "../internal.h" #include "ops-common.h" =20 #ifdef CONFIG_DAMON_VADDR_KUNIT_TEST @@ -653,6 +655,137 @@ static unsigned long damos_madvise(struct damon_targe= t *target, } #endif /* CONFIG_ADVISE_SYSCALLS */ =20 +#ifdef CONFIG_NUMA +struct damos_va_interleave_private { + struct list_head *folio_migration_list; + struct mempolicy *pol; +}; + +static void damos_va_interleave_folio(unsigned long addr, struct folio *fo= lio, + struct vm_area_struct *vma, struct damos_va_interleave_private *priv) +{ + int target_nid; + + if (!folio_isolate_lru(folio)) + return; + + target_nid =3D damon_interleave_target_nid(addr, vma, priv->pol, folio); + + if (target_nid !=3D NUMA_NO_NODE && folio_nid(folio) !=3D target_nid) + list_add(&folio->lru, &priv->folio_migration_list[target_nid]); + else + folio_putback_lru(folio); + +} + +static int damos_va_interleave_pmd(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + struct damos_va_interleave_private *priv =3D walk->private; + struct folio *folio; + spinlock_t *ptl; + pmd_t pmde; + + ptl =3D pmd_lock(walk->mm, pmd); + pmde =3D pmdp_get(pmd); + + if (!pmd_present(pmde) || !pmd_trans_huge(pmde)) + goto unlock; + + folio =3D damon_get_folio(pmd_pfn(pmde)); + if (!folio) + goto unlock; + + damos_va_interleave_folio(addr, folio, walk->vma, priv); + + folio_put(folio); +unlock: + spin_unlock(ptl); + return 0; +} + +static int damos_va_interleave_pte(pte_t *pte, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + struct damos_va_interleave_private *priv =3D walk->private; + struct folio *folio; + + if (pte_none(*pte) || !pte_present(*pte)) + return 0; + + folio =3D vm_normal_folio(walk->vma, addr, *pte); + if (!folio) + return 0; + folio_get(folio); + + damos_va_interleave_folio(addr, folio, walk->vma, priv); + + folio_put(folio); + return 0; +} + +static unsigned long damos_va_interleave(struct damon_target *target, + struct damon_region *r, struct damos *s) +{ + struct damos_va_interleave_private priv; + struct task_struct *task; + struct mm_struct *mm; + int ret; + unsigned long applied =3D 0; + struct mm_walk_ops walk_ops =3D { + .pmd_entry =3D damos_va_interleave_pmd, + .pte_entry =3D damos_va_interleave_pte, + }; + + task =3D damon_get_task_struct(target); + if (!task) + return 0; + + priv.pol =3D get_task_policy(task); + if (!priv.pol) + goto put_task; + + if (priv.pol->mode !=3D MPOL_WEIGHTED_INTERLEAVE) + goto put_pol; + + priv.folio_migration_list =3D kmalloc_array(nr_node_ids, sizeof(struct li= st_head), + GFP_KERNEL); + if (!priv.folio_migration_list) + goto put_pol; + + for (int i =3D 0; i < nr_node_ids; i++) + INIT_LIST_HEAD(&priv.folio_migration_list[i]); + + mm =3D damon_get_mm(target); + if (!mm) + goto free_folio_list; + + mmap_read_lock(mm); + ret =3D walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv); + mmap_read_unlock(mm); + mmput(mm); + + for (int i =3D 0; i < nr_node_ids; i++) { + applied +=3D damon_migrate_pages(&priv.folio_migration_list[i], i); + cond_resched(); + } + +free_folio_list: + kfree(priv.folio_migration_list); +put_pol: + mpol_cond_put(priv.pol); +put_task: + put_task_struct(task); + return applied * PAGE_SIZE; +} +#else +static unsigned long damos_va_interleave(struct damon_target *target, + struct damon_region *r, struct damos *s) +{ + return 0; +} +#endif /* CONFIG_NUMA */ + static unsigned long damon_va_apply_scheme(struct damon_ctx *ctx, struct damon_target *t, struct damon_region *r, struct damos *scheme, unsigned long *sz_filter_passed) @@ -675,6 +808,8 @@ static unsigned long damon_va_apply_scheme(struct damon= _ctx *ctx, case DAMOS_NOHUGEPAGE: madv_action =3D MADV_NOHUGEPAGE; break; + case DAMOS_INTERLEAVE: + return damos_va_interleave(t, r, scheme); case DAMOS_STAT: return 0; default: --=20 2.43.5