From nobody Sat Apr 18 01:57:16 2026 Received: from mail-dy1-f196.google.com (mail-dy1-f196.google.com [74.125.82.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0F80134AB for ; Thu, 12 Feb 2026 04:51:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770871893; cv=none; b=LlLoPUuDtC5rJlJchQyUoX3vrjWIfKkOyQu32tQ8jQaaKokjaXu3rlyX26Xux6rNz+n7w2V8xR6R3bmX4DG8Sl/DPy0TdNoI/MT/K1zVGZgxkDqr7GhIAgBZZAkZSEQIuXdQbWhNe0t7NjZNDRYNEwYMk1aK+a1FaitRRPm+eOk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770871893; c=relaxed/simple; bh=3KDDnkAfe9/mVhflKpeuOU6R545jzogvk/ExLONqcck=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tI0FpULF2KzSZS/sqQT70siJUBmjh4YEdhcndutRdS+Jn/quSNNWcQrSg5+oqOLMnVt3QXHpwjXQzEj0C9AdGp6fW3uB22hJXL060hCupLdPBU6Lt0tql69hnaHMgToKE4RZhH2l8HY5MwvAPxRj3y79Xdilp8g4MvfT1alro+c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KV23WAps; arc=none smtp.client-ip=74.125.82.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KV23WAps" Received: by mail-dy1-f196.google.com with SMTP id 5a478bee46e88-2ba6aa57d5fso4466398eec.1 for ; Wed, 11 Feb 2026 20:51:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1770871891; x=1771476691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3624LeowZDDBkH5ZBXE4RT1ghzAQISXfFg8F+edSqSU=; b=KV23WAps2qEPln42T408fdhEfWmTQOkKPlwkxqxsj+SKL/tlUt6cA9BjK1NpCIt0b5 hcqgbMQCmkXe2c+nvRZLxmlG4EUoWw+0gKnlQfpF6OU8yVTgpaDeP9daQs0CU5x6gFDa E05+16jjqzSVyClupLkWaNdUgzeK9kBkTzN9HVNnbMEv6Rv7c5TFkiijfHUH4lznpdhJ ydbQDC2yiEM1RMQ+PRH0CwD+IPkJ8th+d1UBht1O+lgWxMYf4NqsNEjAvD/M+7vXZ8qF QO5kK+tpaqZkeNi+FKhMtXYs8L8CGGMOmdbM4kW7uJ+u7fY6HhzTepLWiTRHQG6uBaLN hHYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770871891; x=1771476691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3624LeowZDDBkH5ZBXE4RT1ghzAQISXfFg8F+edSqSU=; b=rqd65+1zs2nJB9MsBP7V0zIevkLOrDIW/rjfJHyEnJhCALqn9PsMBtPznevPVjhN9I d1T8yeVrvzqxr2bojeEXQlaEVF60nv/yha1+L7hdkQD5+jFCoCoEDtTjFGb9gZEFqsyE 0GO1e/BBu5wLAyjONm/KDG8Hy4u0nQuHuFJHTpp0POXpZGtFBnXr04Z6kaDucXp/doa+ C8/xb7PpbTmk42vRf7DodZ3RGZJkNttaW0gLNpRzBm+fHMTY5AT20z/oP4KESvkQgduI S7eooS0DNbbpB29+cWu0Y6OXiePPZ9SJWo3RI+OLwLAsbRjPdIGrfAGTsDzsrw5yEUUK V+Hw== X-Forwarded-Encrypted: i=1; AJvYcCVLVMsmP3XRCFc6pLymS1V98LbW4mvCTjkpOfhuTdeL4LQux028HhOgqvoyaaBUMH5Gj+6cQiiA3UUDbgM=@vger.kernel.org X-Gm-Message-State: AOJu0YwdiZp3UfWW3Jz14fHlSynodspo/VfEaIWN3M6A2kg0Fe2tIHYs n+q75L+mcMmxgXUn1GNZ1u6P06o3ZBOTBVhxtBbP8kG5qLC7SDF5JMfn X-Gm-Gg: AZuq6aLN4299/FGBGzWZZz7jlxd9QoOU86dF6ykbLfD7pkn8TH+xO+Tt8OEuQriqMoP jlh6Ppke77m8kopd3WLoAj2R+hSXHLKHNb9K64hOFYVQzK6Ii95xCq61OkB6ChA+NB2oDBP/+zc 7C5aOswNY/GfjL73stBMVWfsVSO7qHYSJWl4AvFcAvFkpKLyYqJNIYGeZW03rRihMUm7sYZtrM+ o9d2RfXfuKKer5t5HMa4Rot/d+dfa6h9sZ0lbBPYbjnfmNgTGSlvHyhydmTwohwtm/xjNyjdvu7 vEAluVZDNKEtLwQbaS2wG07MLxhdSGdNxSdU0K974T9cxGDGpNgAIysH91Q2bqJUVqYrOfPMOGQ bZ7AGhn75YPPARjtnXIKuq8WRBEQouNvjHaDH9iaDu9nl/vdsE4mQTisL0UW2BylS94U5JiYQy3 +qT7cvSAcUkbLrG6FSwh4UfUSS5OO5m/JFa5mAogn8tMky6eWGSQ== X-Received: by 2002:a05:7301:2f91:b0:2ba:871f:796d with SMTP id 5a478bee46e88-2baa8091399mr577872eec.30.1770871891051; Wed, 11 Feb 2026 20:51:31 -0800 (PST) Received: from jpkobryn-fedora-PF5CFKNC.lan ([73.222.117.172]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-2ba9daa6151sm2878699eec.0.2026.02.11.20.51.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Feb 2026 20:51:30 -0800 (PST) From: JP Kobryn To: linux-mm@kvack.org Cc: apopple@nvidia.com, akpm@linux-foundation.org, axelrasmussen@google.com, byungchul@sk.com, cgroups@vger.kernel.org, david@kernel.org, eperezma@redhat.com, gourry@gourry.net, jasowang@redhat.com, hannes@cmpxchg.org, joshua.hahnjy@gmail.com, Liam.Howlett@oracle.com, linux-kernel@vger.kernel.org, lorenzo.stoakes@oracle.com, matthew.brost@intel.com, mst@redhat.com, mhocko@suse.com, rppt@kernel.org, muchun.song@linux.dev, zhengqi.arch@bytedance.com, rakie.kim@sk.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, surenb@google.com, virtualization@lists.linux.dev, vbabka@suse.cz, weixugc@google.com, xuanzhuo@linux.alibaba.com, ying.huang@linux.alibaba.com, yuanchu@google.com, ziy@nvidia.com, kernel-team@meta.com Subject: [PATCH 1/2] mm/mempolicy: track page allocations per mempolicy Date: Wed, 11 Feb 2026 20:51:08 -0800 Message-ID: <20260212045109.255391-2-inwardvessel@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260212045109.255391-1-inwardvessel@gmail.com> References: <20260212045109.255391-1-inwardvessel@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It would be useful to see a breakdown of allocations to understand which NUMA policies are driving them. For example, when investigating memory pressure, having policy-specific counts could show that allocations were bound to the affected node (via MPOL_BIND). Add per-policy page allocation counters as new node stat items. These counters can provide correlation between a mempolicy and pressure on a given node. Signed-off-by: JP Kobryn Suggested-by: Johannes Weiner --- include/linux/mmzone.h | 9 +++++++++ mm/mempolicy.c | 30 ++++++++++++++++++++++++++++-- mm/vmstat.c | 9 +++++++++ 3 files changed, 46 insertions(+), 2 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index fc5d6c88d2f0..762609d5f0af 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -255,6 +255,15 @@ enum node_stat_item { PGDEMOTE_DIRECT, PGDEMOTE_KHUGEPAGED, PGDEMOTE_PROACTIVE, +#ifdef CONFIG_NUMA + PGALLOC_MPOL_DEFAULT, + PGALLOC_MPOL_PREFERRED, + PGALLOC_MPOL_BIND, + PGALLOC_MPOL_INTERLEAVE, + PGALLOC_MPOL_LOCAL, + PGALLOC_MPOL_PREFERRED_MANY, + PGALLOC_MPOL_WEIGHTED_INTERLEAVE, +#endif #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, #endif diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 68a98ba57882..3c64784af761 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -217,6 +217,21 @@ static void reduce_interleave_weights(unsigned int *bw= , u8 *new_iw) new_iw[nid] /=3D iw_gcd; } =20 +#define CHECK_MPOL_NODE_STAT_OFFSET(mpol) \ + BUILD_BUG_ON(PGALLOC_##mpol - mpol !=3D PGALLOC_MPOL_DEFAULT) + +static enum node_stat_item mpol_node_stat(unsigned short mode) +{ + CHECK_MPOL_NODE_STAT_OFFSET(MPOL_PREFERRED); + CHECK_MPOL_NODE_STAT_OFFSET(MPOL_BIND); + CHECK_MPOL_NODE_STAT_OFFSET(MPOL_INTERLEAVE); + CHECK_MPOL_NODE_STAT_OFFSET(MPOL_LOCAL); + CHECK_MPOL_NODE_STAT_OFFSET(MPOL_PREFERRED_MANY); + CHECK_MPOL_NODE_STAT_OFFSET(MPOL_WEIGHTED_INTERLEAVE); + + return PGALLOC_MPOL_DEFAULT + mode; +} + int mempolicy_set_node_perf(unsigned int node, struct access_coordinate *c= oords) { struct weighted_interleave_state *new_wi_state, *old_wi_state =3D NULL; @@ -2446,8 +2461,14 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsi= gned int order, =20 nodemask =3D policy_nodemask(gfp, pol, ilx, &nid); =20 - if (pol->mode =3D=3D MPOL_PREFERRED_MANY) - return alloc_pages_preferred_many(gfp, order, nid, nodemask); + if (pol->mode =3D=3D MPOL_PREFERRED_MANY) { + page =3D alloc_pages_preferred_many(gfp, order, nid, nodemask); + if (page) + __mod_node_page_state(page_pgdat(page), + mpol_node_stat(MPOL_PREFERRED_MANY), 1 << order); + + return page; + } =20 if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && /* filter "hugepage" allocation, unless from alloc_pages() */ @@ -2472,6 +2493,9 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsig= ned int order, page =3D __alloc_frozen_pages_noprof( gfp | __GFP_THISNODE | __GFP_NORETRY, order, nid, NULL); + if (page) + __mod_node_page_state(page_pgdat(page), + mpol_node_stat(pol->mode), 1 << order); if (page || !(gfp & __GFP_DIRECT_RECLAIM)) return page; /* @@ -2484,6 +2508,8 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsig= ned int order, } =20 page =3D __alloc_frozen_pages_noprof(gfp, order, nid, nodemask); + if (page) + __mod_node_page_state(page_pgdat(page), mpol_node_stat(pol->mode), 1 << = order); =20 if (unlikely(pol->mode =3D=3D MPOL_INTERLEAVE || pol->mode =3D=3D MPOL_WEIGHTED_INTERLEAVE) && page) { diff --git a/mm/vmstat.c b/mm/vmstat.c index 65de88cdf40e..74e0ddde1e93 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1291,6 +1291,15 @@ const char * const vmstat_text[] =3D { [I(PGDEMOTE_DIRECT)] =3D "pgdemote_direct", [I(PGDEMOTE_KHUGEPAGED)] =3D "pgdemote_khugepaged", [I(PGDEMOTE_PROACTIVE)] =3D "pgdemote_proactive", +#ifdef CONFIG_NUMA + [I(PGALLOC_MPOL_DEFAULT)] =3D "pgalloc_mpol_default", + [I(PGALLOC_MPOL_PREFERRED)] =3D "pgalloc_mpol_preferred", + [I(PGALLOC_MPOL_BIND)] =3D "pgalloc_mpol_bind", + [I(PGALLOC_MPOL_INTERLEAVE)] =3D "pgalloc_mpol_interleave", + [I(PGALLOC_MPOL_LOCAL)] =3D "pgalloc_mpol_local", + [I(PGALLOC_MPOL_PREFERRED_MANY)] =3D "pgalloc_mpol_preferred_many", + [I(PGALLOC_MPOL_WEIGHTED_INTERLEAVE)] =3D "pgalloc_mpol_weighted_interlea= ve", +#endif #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] =3D "nr_hugetlb", #endif --=20 2.47.3 From nobody Sat Apr 18 01:57:16 2026 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE9F719CC14 for ; Thu, 12 Feb 2026 06:40:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770878415; cv=none; b=WFeKQSlToSsk+dLMhOyEZsf7b2OCWqPysNwU3t3AT8Vvzt2Ut0BcbAPoj4b8qjllHT1FIBYm600aot4f5x3hyIeqXbeiI7xEHWcObVfokk4+2LpGEeecevqdKG9rJQTuvlRw4GcBwamUO2iLw6y1PNsC3Icq7+JnzDLl2DPbD1Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770878415; c=relaxed/simple; bh=plx+CtPhEZ+Jyrnv1Cj+j5hR9mCGL1R7sdnicJWoz0U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tlRhP8aLZtHn83m69QjHLObhZjDVW+Skfg6lET//ELW1ECsIVoyA3H5onTLrvPFZoYNDR+C/vvabjuPU+ylc1M59JC2oeR2EHs32man96378o7bfEWI3spfPURp13fu8K/hD1lCJpUoDDwS3T8Fz7Gld+VgJI4dlFv7NlUCrNaU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VKMV5u8E; arc=none smtp.client-ip=209.85.219.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VKMV5u8E" Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-8947404b367so83046816d6.3 for ; Wed, 11 Feb 2026 22:40:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1770878413; x=1771483213; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A3fhgkTTV9eq12h9CvoXojKy73Yh8kGYUPsNNOceHZA=; b=VKMV5u8EE2k9xZr6194zJ3UeRYaDA+Jc7nu2OjqYKQiXPQBfhaX0AHVLR9xr9AdW/o +WOWxsK4vBGj66lduO/uCwS6R/KyXDHDePH9ia1pL2HKWtXl/ODHQ7giAdSZzb3TjKLl nH121g5PFX0if7qrUvJ2RugZx9zJddawPvV0nqtynyhTTQmybXz9sEvFW5eIipJpHeIo ReW0KOZsYn/E+3kakJunleBtWLBXxBEI9+hbKVFbHMitQaDmVh7ru3ywLdMoPIOBksCG 73g++nNWo8RE6FrFGtLfpbRQcPl8OfW2uNELSRr8qS4vHMRIotJJUdcAyOkLQXi9hPj2 F7Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770878413; x=1771483213; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=A3fhgkTTV9eq12h9CvoXojKy73Yh8kGYUPsNNOceHZA=; b=BSBcC4A8sIzAIA8MHlD2/NSSN7qKiy/Qs91By98qgbwN8WKxw51tAWbEgkHnBZDVvC gdpfj4iBGU61AuS/u8qssox2jg/OnGuwxHcODLguyRMHkeTEuDr8qgf+QCzh23O0nEYd 4865GPsjv+oEKjAal7x3CtPRkPeBHwJQ51GqI65fMR8jJxSyq/xbQM16mWJC7geSGHqS wcheecA0OTM4AveqVMXEFyDwB8dWFJ81SsADw2pwdTZ0xj4FYV+WsrhODAWVMgxekzKH 1Ch4us3WZSKZaysqn/3GVHGPZEInP65rrErFIaKKiRkOrnK3hYIU41dakSOy+duk0EIG ilyA== X-Forwarded-Encrypted: i=1; AJvYcCVPKCX29vRBQbhP/oMdFDsBUtJLeevKhbCc6Q0JrZHCvm/oX5V399esg1t0UnGUd3yqun263pint+DHrRE=@vger.kernel.org X-Gm-Message-State: AOJu0Ywdqp158tRZSvee9QjL90C36Y224nRMzxCfHHf3LEykoGK1iU5N LByOUWqginLjy9vY/wzZZPp0edPnnCU0X5jRt2DuUIyrV4kiXCkmmx+iwlsL4Q== X-Gm-Gg: AZuq6aKcWQOLSW2tURJchLJYK711I0IRPPlEXKgp2PSrJpsLg/CuyQJvt/Kt1nbgqwl mDjlLlS1iWuCDjcYW54J5cNK+yS6wyA9SBM8OVOxw7GLDLp4940Q3Rewt9wA18LqC3ppEspxvpc QbxaCam9q/jfw4jp0OMlsCtnXUVyCweotdmljfL2nWCSgYHZsj+uLmmuU/uEi9fQFPlQthJR+2M SVdMZHTRpchoTptGylNbWz3FAUI+YLAe9qlcaaaV/QahV9kp8hdSjEYxVg8n2MjBio+Ks21d+NP KaTFnxidAvqCjwwuGYqSu8sM2q1hUMF96+6XQAl8HstezvO4JlJt8+ANyd672HNI4mjZjG4fQ85 2Zta7FkhzkgJDLd07cUIn480kdHUX0q1Xf/6UFAywXSHJhixbQ4H0qOfdmpTc2dg6psCCrgGl46 wvevuKzl6Pd+IiQA8MoVwdifMgC2tiz+Y9vnufhkDtdz36ZOfyJw== X-Received: by 2002:a05:693c:2c15:b0:2ba:73db:3e81 with SMTP id 5a478bee46e88-2baa80a790cmr705679eec.33.1770871892999; Wed, 11 Feb 2026 20:51:32 -0800 (PST) Received: from jpkobryn-fedora-PF5CFKNC.lan ([73.222.117.172]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-2ba9daa6151sm2878699eec.0.2026.02.11.20.51.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Feb 2026 20:51:32 -0800 (PST) From: JP Kobryn To: linux-mm@kvack.org Cc: apopple@nvidia.com, akpm@linux-foundation.org, axelrasmussen@google.com, byungchul@sk.com, cgroups@vger.kernel.org, david@kernel.org, eperezma@redhat.com, gourry@gourry.net, jasowang@redhat.com, hannes@cmpxchg.org, joshua.hahnjy@gmail.com, Liam.Howlett@oracle.com, linux-kernel@vger.kernel.org, lorenzo.stoakes@oracle.com, matthew.brost@intel.com, mst@redhat.com, mhocko@suse.com, rppt@kernel.org, muchun.song@linux.dev, zhengqi.arch@bytedance.com, rakie.kim@sk.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, surenb@google.com, virtualization@lists.linux.dev, vbabka@suse.cz, weixugc@google.com, xuanzhuo@linux.alibaba.com, ying.huang@linux.alibaba.com, yuanchu@google.com, ziy@nvidia.com, kernel-team@meta.com Subject: [PATCH 2/2] mm: move pgscan and pgsteal to node stats Date: Wed, 11 Feb 2026 20:51:09 -0800 Message-ID: <20260212045109.255391-3-inwardvessel@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260212045109.255391-1-inwardvessel@gmail.com> References: <20260212045109.255391-1-inwardvessel@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It would be useful to narrow down reclaim to specific nodes. Provide per-node reclaim visibility by changing the pgscan and pgsteal stats from global vm_event_item's to node_stat_item's. Note this change has the side effect of now tracking these stats on a per-memcg basis. Signed-off-by: JP Kobryn Suggested-by: Johannes Weiner --- drivers/virtio/virtio_balloon.c | 8 ++++---- include/linux/mmzone.h | 12 +++++++++++ include/linux/vm_event_item.h | 12 ----------- mm/memcontrol.c | 36 ++++++++++++++++++--------------- mm/vmscan.c | 32 +++++++++++------------------ mm/vmstat.c | 24 +++++++++++----------- 6 files changed, 60 insertions(+), 64 deletions(-) diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloo= n.c index 74fe59f5a78c..1341d9d1a2a1 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -374,13 +374,13 @@ static inline unsigned int update_balloon_vm_stats(st= ruct virtio_balloon *vb) update_stat(vb, idx++, VIRTIO_BALLOON_S_ALLOC_STALL, stall); =20 update_stat(vb, idx++, VIRTIO_BALLOON_S_ASYNC_SCAN, - pages_to_bytes(events[PGSCAN_KSWAPD])); + pages_to_bytes(global_node_page_state(PGSCAN_KSWAPD))); update_stat(vb, idx++, VIRTIO_BALLOON_S_DIRECT_SCAN, - pages_to_bytes(events[PGSCAN_DIRECT])); + pages_to_bytes(global_node_page_state(PGSCAN_DIRECT))); update_stat(vb, idx++, VIRTIO_BALLOON_S_ASYNC_RECLAIM, - pages_to_bytes(events[PGSTEAL_KSWAPD])); + pages_to_bytes(global_node_page_state(PGSTEAL_KSWAPD))); update_stat(vb, idx++, VIRTIO_BALLOON_S_DIRECT_RECLAIM, - pages_to_bytes(events[PGSTEAL_DIRECT])); + pages_to_bytes(global_node_page_state(PGSTEAL_DIRECT))); =20 #ifdef CONFIG_HUGETLB_PAGE update_stat(vb, idx++, VIRTIO_BALLOON_S_HTLB_PGALLOC, diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 762609d5f0af..fc39c107a4b5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -255,6 +255,18 @@ enum node_stat_item { PGDEMOTE_DIRECT, PGDEMOTE_KHUGEPAGED, PGDEMOTE_PROACTIVE, + PGSTEAL_KSWAPD, + PGSTEAL_DIRECT, + PGSTEAL_KHUGEPAGED, + PGSTEAL_PROACTIVE, + PGSTEAL_ANON, + PGSTEAL_FILE, + PGSCAN_KSWAPD, + PGSCAN_DIRECT, + PGSCAN_KHUGEPAGED, + PGSCAN_PROACTIVE, + PGSCAN_ANON, + PGSCAN_FILE, #ifdef CONFIG_NUMA PGALLOC_MPOL_DEFAULT, PGALLOC_MPOL_PREFERRED, diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 92f80b4d69a6..6f1787680658 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -40,19 +40,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, PGLAZYFREED, PGREFILL, PGREUSE, - PGSTEAL_KSWAPD, - PGSTEAL_DIRECT, - PGSTEAL_KHUGEPAGED, - PGSTEAL_PROACTIVE, - PGSCAN_KSWAPD, - PGSCAN_DIRECT, - PGSCAN_KHUGEPAGED, - PGSCAN_PROACTIVE, PGSCAN_DIRECT_THROTTLE, - PGSCAN_ANON, - PGSCAN_FILE, - PGSTEAL_ANON, - PGSTEAL_FILE, #ifdef CONFIG_NUMA PGSCAN_ZONE_RECLAIM_SUCCESS, PGSCAN_ZONE_RECLAIM_FAILED, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 86f43b7e5f71..bde0b6536be6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -328,6 +328,18 @@ static const unsigned int memcg_node_stat_items[] =3D { PGDEMOTE_DIRECT, PGDEMOTE_KHUGEPAGED, PGDEMOTE_PROACTIVE, + PGSTEAL_KSWAPD, + PGSTEAL_DIRECT, + PGSTEAL_KHUGEPAGED, + PGSTEAL_PROACTIVE, + PGSTEAL_ANON, + PGSTEAL_FILE, + PGSCAN_KSWAPD, + PGSCAN_DIRECT, + PGSCAN_KHUGEPAGED, + PGSCAN_PROACTIVE, + PGSCAN_ANON, + PGSCAN_FILE, #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, #endif @@ -441,14 +453,6 @@ static const unsigned int memcg_vm_event_stat[] =3D { #endif PSWPIN, PSWPOUT, - PGSCAN_KSWAPD, - PGSCAN_DIRECT, - PGSCAN_KHUGEPAGED, - PGSCAN_PROACTIVE, - PGSTEAL_KSWAPD, - PGSTEAL_DIRECT, - PGSTEAL_KHUGEPAGED, - PGSTEAL_PROACTIVE, PGFAULT, PGMAJFAULT, PGREFILL, @@ -1496,15 +1500,15 @@ static void memcg_stat_format(struct mem_cgroup *me= mcg, struct seq_buf *s) =20 /* Accumulated memory events */ seq_buf_printf(s, "pgscan %lu\n", - memcg_events(memcg, PGSCAN_KSWAPD) + - memcg_events(memcg, PGSCAN_DIRECT) + - memcg_events(memcg, PGSCAN_PROACTIVE) + - memcg_events(memcg, PGSCAN_KHUGEPAGED)); + memcg_page_state(memcg, PGSCAN_KSWAPD) + + memcg_page_state(memcg, PGSCAN_DIRECT) + + memcg_page_state(memcg, PGSCAN_PROACTIVE) + + memcg_page_state(memcg, PGSCAN_KHUGEPAGED)); seq_buf_printf(s, "pgsteal %lu\n", - memcg_events(memcg, PGSTEAL_KSWAPD) + - memcg_events(memcg, PGSTEAL_DIRECT) + - memcg_events(memcg, PGSTEAL_PROACTIVE) + - memcg_events(memcg, PGSTEAL_KHUGEPAGED)); + memcg_page_state(memcg, PGSTEAL_KSWAPD) + + memcg_page_state(memcg, PGSTEAL_DIRECT) + + memcg_page_state(memcg, PGSTEAL_PROACTIVE) + + memcg_page_state(memcg, PGSTEAL_KHUGEPAGED)); =20 for (i =3D 0; i < ARRAY_SIZE(memcg_vm_event_stat); i++) { #ifdef CONFIG_MEMCG_V1 diff --git a/mm/vmscan.c b/mm/vmscan.c index 614ccf39fe3f..16a0f21e3ea1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1977,7 +1977,7 @@ static unsigned long shrink_inactive_list(unsigned lo= ng nr_to_scan, unsigned long nr_taken; struct reclaim_stat stat; bool file =3D is_file_lru(lru); - enum vm_event_item item; + enum node_stat_item item; struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); bool stalled =3D false; =20 @@ -2003,10 +2003,8 @@ static unsigned long shrink_inactive_list(unsigned l= ong nr_to_scan, =20 __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); item =3D PGSCAN_KSWAPD + reclaimer_offset(sc); - if (!cgroup_reclaim(sc)) - __count_vm_events(item, nr_scanned); - count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned); - __count_vm_events(PGSCAN_ANON + file, nr_scanned); + mod_lruvec_state(lruvec, item, nr_scanned); + mod_lruvec_state(lruvec, PGSCAN_ANON + file, nr_scanned); =20 spin_unlock_irq(&lruvec->lru_lock); =20 @@ -2023,10 +2021,8 @@ static unsigned long shrink_inactive_list(unsigned l= ong nr_to_scan, stat.nr_demoted); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); item =3D PGSTEAL_KSWAPD + reclaimer_offset(sc); - if (!cgroup_reclaim(sc)) - __count_vm_events(item, nr_reclaimed); - count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed); - __count_vm_events(PGSTEAL_ANON + file, nr_reclaimed); + mod_lruvec_state(lruvec, item, nr_reclaimed); + mod_lruvec_state(lruvec, PGSTEAL_ANON + file, nr_reclaimed); =20 lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); @@ -4536,7 +4532,7 @@ static int scan_folios(unsigned long nr_to_scan, stru= ct lruvec *lruvec, { int i; int gen; - enum vm_event_item item; + enum node_stat_item item; int sorted =3D 0; int scanned =3D 0; int isolated =3D 0; @@ -4595,13 +4591,11 @@ static int scan_folios(unsigned long nr_to_scan, st= ruct lruvec *lruvec, } =20 item =3D PGSCAN_KSWAPD + reclaimer_offset(sc); - if (!cgroup_reclaim(sc)) { - __count_vm_events(item, isolated); + if (!cgroup_reclaim(sc)) __count_vm_events(PGREFILL, sorted); - } - count_memcg_events(memcg, item, isolated); + mod_lruvec_state(lruvec, item, isolated); count_memcg_events(memcg, PGREFILL, sorted); - __count_vm_events(PGSCAN_ANON + type, isolated); + mod_lruvec_state(lruvec, PGSCAN_ANON + type, isolated); trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, scan_batch, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); @@ -4686,7 +4680,7 @@ static int evict_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, LIST_HEAD(clean); struct folio *folio; struct folio *next; - enum vm_event_item item; + enum node_stat_item item; struct reclaim_stat stat; struct lru_gen_mm_walk *walk; bool skip_retry =3D false; @@ -4750,10 +4744,8 @@ static int evict_folios(unsigned long nr_to_scan, st= ruct lruvec *lruvec, stat.nr_demoted); =20 item =3D PGSTEAL_KSWAPD + reclaimer_offset(sc); - if (!cgroup_reclaim(sc)) - __count_vm_events(item, reclaimed); - count_memcg_events(memcg, item, reclaimed); - __count_vm_events(PGSTEAL_ANON + type, reclaimed); + mod_lruvec_state(lruvec, item, reclaimed); + mod_lruvec_state(lruvec, PGSTEAL_ANON + type, reclaimed); =20 spin_unlock_irq(&lruvec->lru_lock); =20 diff --git a/mm/vmstat.c b/mm/vmstat.c index 74e0ddde1e93..e4b259989d58 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1291,6 +1291,18 @@ const char * const vmstat_text[] =3D { [I(PGDEMOTE_DIRECT)] =3D "pgdemote_direct", [I(PGDEMOTE_KHUGEPAGED)] =3D "pgdemote_khugepaged", [I(PGDEMOTE_PROACTIVE)] =3D "pgdemote_proactive", + [I(PGSTEAL_KSWAPD)] =3D "pgsteal_kswapd", + [I(PGSTEAL_DIRECT)] =3D "pgsteal_direct", + [I(PGSTEAL_KHUGEPAGED)] =3D "pgsteal_khugepaged", + [I(PGSTEAL_PROACTIVE)] =3D "pgsteal_proactive", + [I(PGSTEAL_ANON)] =3D "pgsteal_anon", + [I(PGSTEAL_FILE)] =3D "pgsteal_file", + [I(PGSCAN_KSWAPD)] =3D "pgscan_kswapd", + [I(PGSCAN_DIRECT)] =3D "pgscan_direct", + [I(PGSCAN_KHUGEPAGED)] =3D "pgscan_khugepaged", + [I(PGSCAN_PROACTIVE)] =3D "pgscan_proactive", + [I(PGSCAN_ANON)] =3D "pgscan_anon", + [I(PGSCAN_FILE)] =3D "pgscan_file", #ifdef CONFIG_NUMA [I(PGALLOC_MPOL_DEFAULT)] =3D "pgalloc_mpol_default", [I(PGALLOC_MPOL_PREFERRED)] =3D "pgalloc_mpol_preferred", @@ -1344,19 +1356,7 @@ const char * const vmstat_text[] =3D { =20 [I(PGREFILL)] =3D "pgrefill", [I(PGREUSE)] =3D "pgreuse", - [I(PGSTEAL_KSWAPD)] =3D "pgsteal_kswapd", - [I(PGSTEAL_DIRECT)] =3D "pgsteal_direct", - [I(PGSTEAL_KHUGEPAGED)] =3D "pgsteal_khugepaged", - [I(PGSTEAL_PROACTIVE)] =3D "pgsteal_proactive", - [I(PGSCAN_KSWAPD)] =3D "pgscan_kswapd", - [I(PGSCAN_DIRECT)] =3D "pgscan_direct", - [I(PGSCAN_KHUGEPAGED)] =3D "pgscan_khugepaged", - [I(PGSCAN_PROACTIVE)] =3D "pgscan_proactive", [I(PGSCAN_DIRECT_THROTTLE)] =3D "pgscan_direct_throttle", - [I(PGSCAN_ANON)] =3D "pgscan_anon", - [I(PGSCAN_FILE)] =3D "pgscan_file", - [I(PGSTEAL_ANON)] =3D "pgsteal_anon", - [I(PGSTEAL_FILE)] =3D "pgsteal_file", =20 #ifdef CONFIG_NUMA [I(PGSCAN_ZONE_RECLAIM_SUCCESS)] =3D "zone_reclaim_success", --=20 2.47.3