From nobody Fri Sep 25 04:07:46 2026 Received: from mail-oi2-f13.google.com (mail-oi2-f13.google.com [74.125.231.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EEF24E1C7A for ; Wed, 16 Sep 2026 21:05:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.231.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592764; cv=none; b=jpSwKfLAW9cYW/JAWMkoeCK1mNTi7koi6RMxh+YuPu32InMz33pmCneb6C/AQ3I8Wi3UGLkbVAgFxsedwjcjUtWs5F+JK1SI4RObI9WyRiql4G266DsKODF7Ysr39YDx4FeCYPa5rRSeEvcdsVhZZEXYkUU5dt06HWoM3Ue45eU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592764; c=relaxed/simple; bh=IQNpo0voZl44flDTr/7bJrTih9RnNbJtVb25BrCwX4A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dlrUn8LaD7xFjY2Bmg4ysDamJrUfRXYJ9NFCacG/TO/Dp7+PxdjJzyYtpZLkckTtmt2XfbaUpkU/Udeslu6eQVLG9cPT3T4KJghqGIu771bff+QuKb23nt741KnUt1amfjNd04gLepZKBXg8DqYc1AT1QCQAXfGZjklvUWJsdCA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aHPK4bAu; arc=none smtp.client-ip=74.125.231.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aHPK4bAu" Received: by mail-oi2-f13.google.com with SMTP id 5614622812f47-4b37a3a9768so157374b6e.0 for ; Wed, 16 Sep 2026 14:05:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789592757; x=1790197557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KzgD7tpJ/TV5FYPH6ulvzWsXqYOY3DYINqe8nFmUw9M=; b=aHPK4bAurX8scni84dKccg0rLHmPsoLmjjaVuhlPtnrHrP1Rko8oJF3wSb15S0yMcW AlAlcHXG3KoTvbmi78oLTlkFhwWeMdc0DxHLc56WPvrdRnb0LbjtAGtKauHFtgjx5zAG W8FXOB3CttFaKBpaWmRw4n4gwDQR5Q3qnnzqjqvUZE97gs31dBjswkrvdEaLFmkvZz6m LYzfqNui6IUWxlBJUZWPWf9mHTPHYy9Q5mRpdiaHClVSqEP7Go/6A8C2muguFQZbuod+ +TKahYvAsVSaaYqD+NZ1abceH4BWiMGAnWUSHz7VX0AUMPV8hv+t+CfWPeO7N2jAlF9f +dvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789592757; x=1790197557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=KzgD7tpJ/TV5FYPH6ulvzWsXqYOY3DYINqe8nFmUw9M=; b=QjU7t+74QRzjSxphE7nnbbE+sv4fKUg1DG/K7lctu0W27kJx2sjr6xKNL+So2nbaWU 0ofLD03zrwJ0UZ1FEXCbEfG7XD4xSvRFfPwl3cv1os2RCzyAOg7qaWOsqdediRB7Putx D+cjssdxX3XBwkAGrhkWoEDW/43PvGqFADWdlyycvYfd7Ip+mRHuWaQSHXsQm6kuFFd3 AaD3TjkKfr0J8aYXquArqjGFeeDdeqxw8SBLEWFGYdOKb+fsUb67kZ6RxWKpovaQHkn+ UFwU3A+QAvM6PuZMnVMy3RHsL+nDn4UmmggBTp1Q7Q97FYm2ljxPlZpAqFsIv0IpZNKL Nn7A== X-Forwarded-Encrypted: i=1; AKwUvBx/X5cvMm8ubuxvZkmcqcpv1ueKHAcQsYA2s6s7L1LvsgLtmFcS+2KeH5g5HBySu8KDHgqocxbLMHncyYk=@vger.kernel.org X-Gm-Message-State: AFuF++lZM6bv3TFb5Rd/K4q+BUBCcQwhoU7Yi+j9YqIyIluDi7aJR9lR Tf1MkvPH+RbR3kDSeP6Iu4nsmEGGG06uR90DD83r2/FuRiQkSveEP4Um X-Gm-Gg: AYBFou2ObBIkOTrwqx7yJR/pZGKbPY5SZzGuqHt5z3V+yRYH93eShb+l25Odd6jycCs FmQ9Dhikk++cG3j1ju7q0zld9vfNYV+/SQ3e3bhm5j5y41becic9ezDHdgdvQFGjW6fQUEOUCJG SpjtgSyNNyt9AOHjNT/Uf/bBDZfFTkTxjzWJ94RYK2EqckJlkUQbaXgxvyUZ/qpRJTVobob++LS y//rhQvG8SF2Rg3MgTlv1kUd5O0mnK2dziGTSZsPOMWTLX1eJA5zO9NhqUGmpMIGwqYRhOpS60L 61s1KSqDVZjOZSoXt4RPXm8Io9R9Y2uHtbbLowxiKqXZYJC4JWWrcXGpNL+HfpWUChtXFSPqrmw dZOsQFNga73mRXEMY7+Nb5yRyL9y38r0Pj/3ADjirSNVKqomXBSP+sdJedC4wxh1pa0LuMKQGoB cgssNQzbgk1y43NKxGrBAcTn05C5GhTiIAqcY2TtVDs3kv+hhxAljds4QHlvuQ30WQfd9UMZC1t CIzn1rg9+Pr/HS4UVv/iCmX7OUs5Sfg394QimRf X-Received: by 2002:a05:6808:1a0e:b0:4c3:7695:1177 with SMTP id 5614622812f47-4ca497b2b58mr4346226b6e.6.1789592756726; Wed, 16 Sep 2026 14:05:56 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:4c::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-4cb6d796b64sm891126b6e.15.2026.09.16.14.05.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Sep 2026 14:05:56 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko , Shakeel Butt Cc: Roman Gushchin , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Maarten Lankhorst , Maxime Ripard , Natalie Vock , Tejun Heo , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Oscar Salvador , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v6 1/5] mm/memcontrol: flatten try_charge_memcg control flow Date: Wed, 16 Sep 2026 14:05:47 -0700 Message-ID: <20260916210552.891730-2-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260916210552.891730-1-joshua.hahnjy@gmail.com> References: <20260916210552.891730-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor try_charge_memcg by flattening the nested memsw/memory page_counter operations to separate the logic between the two. When page_counter_try_charge is made stock-aware, this flattening makes the control flow easier to follow since each page counter now has its own success/failure paths. No functional changes intended. Acked-by: Shakeel Butt Signed-off-by: Joshua Hahn --- mm/memcontrol.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1460cba53588e..48c475909e6bb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2707,18 +2707,21 @@ static int try_charge_memcg(struct mem_cgroup *memc= g, gfp_t gfp_mask, batch =3D nr_pages; =20 reclaim_options =3D MEMCG_RECLAIM_MAY_SWAP; - if (!do_memsw_account() || - page_counter_try_charge(&memcg->memsw, batch, &counter)) { - if (page_counter_try_charge(&memcg->memory, batch, &counter)) - goto done_restock; - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, batch); - mem_over_limit =3D mem_cgroup_from_counter(counter, memory); - } else { + if (do_memsw_account() && + !page_counter_try_charge(&memcg->memsw, batch, &counter)) { mem_over_limit =3D mem_cgroup_from_counter(counter, memsw); reclaim_options &=3D ~MEMCG_RECLAIM_MAY_SWAP; + goto reclaim; } =20 + if (page_counter_try_charge(&memcg->memory, batch, &counter)) + goto done_restock; + + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, batch); + mem_over_limit =3D mem_cgroup_from_counter(counter, memory); + +reclaim: if (batch > nr_pages) { batch =3D nr_pages; goto retry; --=20 2.53.0-Meta From nobody Fri Sep 25 04:07:46 2026 Received: from mail-oi2-f43.google.com (mail-oi2-f43.google.com [74.125.231.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7ED314E1C78 for ; Wed, 16 Sep 2026 21:06:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.231.235 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592774; cv=none; b=oWIDLO3JKcnDY8TCNUIoiZ6x4NCWVUKHlw+hgCiZsMbH5ZORWuebNxteJRYPBUJuzpVSzC83Ti5VDhitnaJCn/a9uE2jjbAJcWI/u2L3qqj+YPJ12oGzW395hwPXkzAqvl5TlMGmr6oxJbrHkpuikrVKsXq+C4En5B5QoRF1b48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592774; c=relaxed/simple; bh=aYack9iQ/NLabz3h4yWk2IS2sT6rmARUEn3hue6kgyE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Or/SEWcqx62MKzFKd7GHqLWQgt2W9FX9a1bentE+AyoLSugTe1YGH+ySxzJzEvPSSlmDlELz71/9gWtL+wVV7as/I5HURpQAToNUoP48m7VqOyc+iv2G4wVohh8XEwaFHEWOmPv33JCWpy0qSb+hlY4eUnWY0AW+oo0t8PQfvsY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Bgd428WP; arc=none smtp.client-ip=74.125.231.235 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Bgd428WP" Received: by mail-oi2-f43.google.com with SMTP id 46e09a7af769-805bf8c2661so111458a34.3 for ; Wed, 16 Sep 2026 14:06:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789592758; x=1790197558; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=OpeT7E+4SgQjmsGt8zdgsDPzGh6zBoDDRVsBk7NbP6k=; b=Bgd428WP+qUACtEX9SUjR/X8O82e0YP2zDh8OMDFjZMYoQpQv7qBDp9UV2MzZkGWcD BC9RahzlNZjAxJCRxsSJMPSefR6DWDmOhCsOBEjp3ble2Nu4iHZBXbVq5qjfYd0JnyDj Ei9H3Z1UbKvETZNVYevuQdJ87/qCrojkHd15SjqDHkmafY+7BgY4/yrH8wB4NNts+uR4 7h8AOCwsv5XvTaZvW+esJpRW1mvtEXLSXITOxZCJLUiazhRsVK8Ll3y/Cv30aeXLUEa+ KnwtKh+VKhIIPolHr7ucdnMWC1g3qJ00AkAcoMOBfNQfdSREeHGhJ2ux2kBinoRl85j2 kMRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789592758; x=1790197558; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=OpeT7E+4SgQjmsGt8zdgsDPzGh6zBoDDRVsBk7NbP6k=; b=B7fCSRF68u3LjVY5yev+vTvSdPCgfym4ySwq1q631qB3phRidLKSx1qf1/uL1Ga0jv bkaFwaw4Gqn7Y8POmqRfAJb/OJ3p39E7SFr0knP0YRxwnrCNARqrZioq1RjRvJdmW0lK QG7qjjB2iOYMbpBtnFyLZ/27SUhHQcO4w7qX2rzuKg23GIvqW36qJAZutDjpDJp/4ii6 b6yf0taJXJswo44F6pidsfYUlhNhRW5bkypanmxpF+8qQBEXyQORHlSppn8TbXbAwEz7 cxr+oAyG6+OJG6NoewLzKL6xjb4Xot0ch7SFdyM20uflxxyWfzyi2htMWzED8nnDxfqT J3OQ== X-Forwarded-Encrypted: i=1; AKwUvBzakMbfMJuw3cUTeLxhZ+EzPAEOskR3opm227bjljRWSeKo3nKCWaA3TehMBMC4VlWR9TdIv65PDbW+2lg=@vger.kernel.org X-Gm-Message-State: AFuF++l3Y8FTmhmMH793oBO6DP9X9zA3HTDH6BeFuR6cPKdQ+Xa2EPCG fHnh7WM0o4C5NuTk0yOKYCnYgdFhiuDqkStRcLZc1++zr/4gpclKl80H X-Gm-Gg: AYBFou3JFenHuMFnU7BxorJhN9/OMmr1UxZA8QXv+VrQW6j6xVTFmvr69tYVbr1j5xF 9JKsmo6WgXCKHBwx2RupgvbC4jdmH9MTZw5QuF23cCf7So4qyagQUxyEMZI42PpgvRcbKIn4E/J lmTGg7q5Kd2l6Pn+pwMJ9LGCuPlO5H13v8JyWmB3qpWSGl0PAadZ80hxp8eS1Y0BOZZbG+FX6l+ aOuMHGgh/+DqKyI3DxZU57wJIwPRMVLzCEDfS4ZLAScC4VNH61vD089Tu45IiLPW5ox2B5I40DH 0vGnS3UkAcgONq9wuZmnBoBy7au+edfmagaoZzjj1Wwj5a1pXQukmKK1mGFQtAwzxz+4AqI++CO ijqwMnteoQAM1KyMGe24QTOuCIVZTNiaiBrvdWZv5lBE49lmLN2QPpLvjjKuuXpreYlMGJ5I8+0 TNu9VrzoqqffjJaBnj77LZU/MYdBQHin8r/OHK5pwV4YNh6XIST3Xrr6h/aCQwZwIUNxNh4RhF3 FOcATD4Q01wWX3SiQVJNcIPQjBV5Q== X-Received: by 2002:a05:6808:1921:b0:4b9:a8ac:480 with SMTP id 5614622812f47-4ca4c6bab9emr5238137b6e.30.1789592758099; Wed, 16 Sep 2026 14:05:58 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:51::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-4cb6d796b64sm891175b6e.15.2026.09.16.14.05.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Sep 2026 14:05:57 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko , Shakeel Butt Cc: Roman Gushchin , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Maarten Lankhorst , Maxime Ripard , Natalie Vock , Tejun Heo , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Oscar Salvador , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v6 2/5] mm/page_counter: introduce per-CPU stock Date: Wed, 16 Sep 2026 14:05:48 -0700 Message-ID: <20260916210552.891730-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260916210552.891730-1-joshua.hahnjy@gmail.com> References: <20260916210552.891730-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a copy of the seven-slot per-CPU stock representation from memcg in the page_counter layer. struct page_counter_stock_pcp preserves everything from struct memcg_stock_pcp, but adds a new backpointer to the base of the percpu stock, since there will be multiple percpu stock base pointers later in the series (one for memory, one for memsw). struct page_counter also gets a pointer to the base of the percpu stock, as well as a struct cgroup_subsys_state pointer to pin its owning CSS as long as the cached counter remains reachable. Let's also copy over the drain, refill, and "flush required" functions, preserving all behaviors from the memcg equivalent. In this patch they are not connected. No functional changes intended. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- include/linux/page_counter.h | 31 +++++++++ mm/page_counter.c | 127 +++++++++++++++++++++++++++++++++++ 2 files changed, 158 insertions(+) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index 07b7cb12249c7..9cb5612fe190f 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -5,8 +5,30 @@ #include #include #include +#include +#include +#include #include =20 +/* + * The value of NR_PAGE_COUNTER_STOCK is selected to keep the cached count= ers + * and their nr_pages in a single cacheline. This may change in the future. + */ +#define NR_PAGE_COUNTER_STOCK 7 +#define PAGE_COUNTER_STOCK_BATCH 64UL +struct cgroup_subsys_state; +struct page_counter; +struct page_counter_stock_pcp { + local_trylock_t lock; + u8 nr_pages[NR_PAGE_COUNTER_STOCK]; + struct page_counter *cached[NR_PAGE_COUNTER_STOCK]; + + struct page_counter_stock_pcp __percpu *base; + struct work_struct work; + unsigned long flags; + u8 drain_idx; +}; + struct page_counter { /* * Make sure 'usage' does not share cacheline with any other field in @@ -41,6 +63,8 @@ struct page_counter { unsigned long high; unsigned long max; struct page_counter *parent; + struct page_counter_stock_pcp __percpu *stock; + struct cgroup_subsys_state *stock_css; } ____cacheline_internodealigned_in_smp; =20 #if BITS_PER_LONG =3D=3D 32 @@ -61,6 +85,8 @@ static inline void page_counter_init(struct page_counter = *counter, counter->parent =3D parent; counter->protection_support =3D protection_support; counter->track_failcnt =3D false; + counter->stock =3D NULL; + counter->stock_css =3D NULL; } =20 static inline unsigned long page_counter_read(struct page_counter *counter) @@ -74,6 +100,11 @@ void page_counter_charge(struct page_counter *counter, = unsigned long nr_pages); bool page_counter_try_charge(struct page_counter *counter, unsigned long nr_pages, struct page_counter **fail); +void page_counter_refill_stock(struct page_counter *counter, + unsigned long nr_pages); +void page_counter_drain_stock_fully(struct page_counter_stock_pcp *stock); +bool page_counter_stock_flush_required(struct page_counter_stock_pcp *stoc= k, + struct cgroup_subsys_state *root_css); void page_counter_uncharge(struct page_counter *counter, unsigned long nr_= pages); void page_counter_set_min(struct page_counter *counter, unsigned long nr_p= ages); void page_counter_set_low(struct page_counter *counter, unsigned long nr_p= ages); diff --git a/mm/page_counter.c b/mm/page_counter.c index 98322803941a7..480a447bd7265 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -7,6 +7,7 @@ =20 #include #include +#include #include #include #include @@ -14,6 +15,14 @@ #include #include =20 +/* + * Watermarks for a charge stock slot, in the spirit of pcp->high and + * pcp->batch: PAGE_COUNTER_STOCK_HIGH is the high watermark at which a sl= ot is + * trimmed down to PAGE_COUNTER_STOCK_LOW rather than emptied. + */ +#define PAGE_COUNTER_STOCK_LOW (PAGE_COUNTER_STOCK_BATCH / 2) +#define PAGE_COUNTER_STOCK_HIGH PAGE_COUNTER_STOCK_BATCH + static bool track_protection(struct page_counter *c) { return c->protection_support; @@ -192,6 +201,124 @@ bool page_counter_try_charge(struct page_counter *cou= nter, return false; } =20 +static void page_counter_drain_stock(struct page_counter_stock_pcp *stock, + int i) +{ + struct page_counter *counter =3D READ_ONCE(stock->cached[i]); + u8 nr_pages; + + if (!counter) + return; + + nr_pages =3D READ_ONCE(stock->nr_pages[i]); + if (nr_pages) { + page_counter_uncharge(counter, nr_pages); + WRITE_ONCE(stock->nr_pages[i], 0); + } + css_put(counter->stock_css); + WRITE_ONCE(stock->cached[i], NULL); +} + +void page_counter_drain_stock_fully(struct page_counter_stock_pcp *stock) +{ + int i; + + for (i =3D 0; i < NR_PAGE_COUNTER_STOCK; i++) + page_counter_drain_stock(stock, i); +} + +bool page_counter_stock_flush_required(struct page_counter_stock_pcp *stoc= k, + struct cgroup_subsys_state *root_css) +{ + struct cgroup_subsys_state *css; + struct page_counter *counter; + bool flush =3D false; + int i; + + rcu_read_lock(); + for (i =3D 0; i < NR_PAGE_COUNTER_STOCK; i++) { + counter =3D READ_ONCE(stock->cached[i]); + if (!counter) + continue; + css =3D READ_ONCE(counter->stock_css); + + if (READ_ONCE(stock->nr_pages[i]) && + cgroup_is_descendant(css->cgroup, root_css->cgroup)) { + flush =3D true; + break; + } + } + rcu_read_unlock(); + return flush; +} + +/** + * page_counter_refill_stock - return pages to a counter's stock + * @counter: counter to return pages to + * @nr_pages: number of pages to return + * + * If the stock cannot accept the pages, uncharge them from the hierarchy. + */ +void page_counter_refill_stock(struct page_counter *counter, + unsigned long nr_pages) +{ + struct page_counter_stock_pcp __percpu *stock =3D counter->stock; + struct page_counter_stock_pcp *pcp_stock; + unsigned int stock_pages; + int empty_slot =3D -1; + int i; + + /* + * nr_pages[] is a u8 and a slot is capped at PAGE_COUNTER_STOCK_HIGH. + * Raising PAGE_COUNTER_STOCK_BATCH beyond 127 would need careful + * handling of nr_pages[] in struct page_counter_stock_pcp. + */ + BUILD_BUG_ON(PAGE_COUNTER_STOCK_BATCH > S8_MAX); + BUILD_BUG_ON(PAGE_COUNTER_STOCK_HIGH > U8_MAX); + + if (!stock || nr_pages > PAGE_COUNTER_STOCK_BATCH || + !local_trylock(&stock->lock)) { + /* + * For a larger-than-batch refill or an unlikely failure to lock + * the per-CPU stock, uncharge the hierarchy directly. + */ + page_counter_uncharge(counter, nr_pages); + return; + } + + pcp_stock =3D this_cpu_ptr(stock); + for (i =3D 0; i < NR_PAGE_COUNTER_STOCK; i++) { + struct page_counter *cached =3D READ_ONCE(pcp_stock->cached[i]); + + if (!cached && empty_slot =3D=3D -1) + empty_slot =3D i; + if (counter !=3D cached) + continue; + + stock_pages =3D READ_ONCE(pcp_stock->nr_pages[i]) + nr_pages; + if (stock_pages > PAGE_COUNTER_STOCK_HIGH) { + page_counter_uncharge(counter, + stock_pages - PAGE_COUNTER_STOCK_LOW); + stock_pages =3D PAGE_COUNTER_STOCK_LOW; + } + WRITE_ONCE(pcp_stock->nr_pages[i], stock_pages); + local_unlock(&stock->lock); + return; + } + + i =3D empty_slot; + if (i =3D=3D -1) { + i =3D pcp_stock->drain_idx++; + if (pcp_stock->drain_idx =3D=3D NR_PAGE_COUNTER_STOCK) + pcp_stock->drain_idx =3D 0; + page_counter_drain_stock(pcp_stock, i); + } + css_get(counter->stock_css); + WRITE_ONCE(pcp_stock->cached[i], counter); + WRITE_ONCE(pcp_stock->nr_pages[i], nr_pages); + local_unlock(&stock->lock); +} + /** * page_counter_uncharge - hierarchically uncharge pages * @counter: counter --=20 2.53.0-Meta From nobody Fri Sep 25 04:07:46 2026 Received: from mail-oa2-f12.google.com (mail-oa2-f12.google.com [74.125.231.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9681B4E1C7E for ; Wed, 16 Sep 2026 21:06:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.231.76 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592778; cv=none; b=IM40ajRhLKmj2yAYJmgEdT33tOolvADye46U5QvVfjmzaSRnwWE+mx0vLBdxeoi5Rsqx0hJ1rB2Noibwy4A0cBWdjUUIyKPUOQfInILBIag5j5PYvv2NPvHBLU1N2fJufJlmve6O/aboTuSMaT9mmtciSoOkEl7suNVA0a4lNGM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592778; c=relaxed/simple; bh=jQYqPGmyomyUKJxHMNQLhim9L5ANd3ptwKSMbWAB9iM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Cbi9l51S3Hsy6NRpOxT/34ZuYXvk+iOynWPylo+FU6jImuagAJR3xx3IfwSSufzELLy51579/zzlZYi43BIJViV6PdZUUPBnKcf54E5CejbMq69TngZVNFbFlTdNpbp4vmIBhhx3ATZ3XVYEVmNs3GfNpTwsAPwwLAP14WXBeiE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=qoi27KPc; arc=none smtp.client-ip=74.125.231.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qoi27KPc" Received: by mail-oa2-f12.google.com with SMTP id 586e51a60fabf-466ccdd77e2so22599fac.3 for ; Wed, 16 Sep 2026 14:06:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789592759; x=1790197559; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iqYH4Dmno3zIyHsVcbhXkDT5EaBSu0ei+LzQmvQECZs=; b=qoi27KPcYeOFVh9tA71saQUBlGtZR3QraQBJHp2NRMKWORhnMvCP7pRJmYkZvM8isT uDofQO5iFeWmoY241BJbkc5vcQIk41nWOF1ID+DjcItlgaYsjDL851QizSogkQSlqrOs V+FJ7g+tJAVit48N7F0QAs56Gu+EbbDhRfzw5jtjpvk3nklPhEnmyQAvhVt5UNaGQQCg YlJYrVoKjqq3169HSB8QdEhgNtVDcDtLxep+oLXDw7d0baROZDjqIzn9sowd0UaLJK3P SlW+8WMJD3OD+2t3i5OvbdUZMelwxQaMHfUkSgxznKn3h6SrBWl8U5gSHsU6JZ9103JJ MqmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789592759; x=1790197559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iqYH4Dmno3zIyHsVcbhXkDT5EaBSu0ei+LzQmvQECZs=; b=IeSOqF5dlqiWDO3NQn2BB2ikiIftSViR+yYvuKtZJjI1FMC0UL8rXaz9nmMHDQfknw 4qw5QmN7HZji0ghtte81DtaD6qSVakZnUOeiJSOD9BkFAtyWvx/+g0E1fEURS+Bze+hl lEcWsx2jzjBGCcWzFWQlsXvRkRrW2MUTJrXhxXuqqBD11ILPYREJEYhxGjS9ILjmyYeJ 6hlS/v6UI39goDUgvr7iA5G2MBvfhoITcDW6sVMqmdQCjSI2huGoij56bTs3Hk9JKxSn le/PddvJY2NaO/EiktQqbu2zlETZQ1K2PySv5ueysEq9QqodN2UpmQiU3NmRK8J5nGl8 mwaA== X-Forwarded-Encrypted: i=1; AKwUvBwn+AM8UZuhrZVlrOODWbUdgwialFAuWFex6zLYrViKkRcd3ntqNOokb/EO9ejuuduVJ9zmP/Sv9OMdVOQ=@vger.kernel.org X-Gm-Message-State: AFuF++lSmP4pqIIgJ/un6sZlUDSKthD3jzIjCNEjVTZeEOMSehHmIs3J MMd86iTBA53u5ruYvVihN3Zpf76UY8mILhWMWH11Bo8cEMyEgBiPWoQM X-Gm-Gg: AYBFou3SqkTApPPUfmYW9+wuroErmLjQ1F5knX5WEYMIxZN3fYD9LQK/W87+QjhJFR9 p6Y44kX3n2uOPOaLlE+pj0FzQosND2hKDP0r6mM7SFCYcZrqqljwC9Y6C7P3KoAKbzAPK87ksUw OpKgkRxEjmZBaVCMat9oLAzK7bRic7K9rG1UvP3DYTFmCDVePTHx5X4v/7HT/3m8arcp+ASwt3V 28TiLiJRxFnWNzFCsXFmM7tBQIbXrvu3qBgnMLtYtJhAsDf3J4eiYlTZxYPnsZFGI6z1Ny6QSxm FzB8oLMCp7O6etVksmCvVhkUbuLrFLXp4tBy270Ip7FJVNNzazN8T5Jx/CKBu/6ky7Xs+sZypq7 K1gZFjAWqrYIwxYp4fG5vp1Rm9DL6T7ZAduV4Imreb+0MNpgM/Kb4xB8ZBcc+w0qTB9AKtdK7pL o6ApxRTUJmTidv3KsCaa3BjkcQ7w5cT8p6F7PQkBBSdqmSaiWWx+s1NwvZ9Md+wuZBVXz3QBYxu XnPbo7p0mm0rCLiYSrXoDnOAas61sYyQqeFC0Y= X-Received: by 2002:a05:6820:4c17:b0:6b7:46e9:9706 with SMTP id 006d021491bc7-6c7d45e67c4mr3656381eaf.54.1789592759507; Wed, 16 Sep 2026 14:05:59 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:7::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-80c43e5385csm1117267a34.1.2026.09.16.14.05.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Sep 2026 14:05:59 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko , Shakeel Butt Cc: Roman Gushchin , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Maarten Lankhorst , Maxime Ripard , Natalie Vock , Tejun Heo , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Oscar Salvador , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v6 3/5] mm/page_counter: make page_counter_try_charge() stock-aware Date: Wed, 16 Sep 2026 14:05:49 -0700 Message-ID: <20260916210552.891730-4-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260916210552.891730-1-joshua.hahnjy@gmail.com> References: <20260916210552.891730-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make page_counter_try_charge() consume stock transparently to callers while preserving the same semantics as try_charge_memcg's greedy charge attempt, refill, and !allow_spinning special case. page_counter_try_charge gets two new parameters, may_batch and nr_charged. may_batch is intended to preserve the behavior of !allow_spinning scenarios in try_charge_memcg, where the goal is to charge as quickly as possible, without evicting other stock slots or making a greedy charge to refill the stock. nr_charged is used to reflect the size of the successful hierarchy charge, preserving the existing batch-based memory.high accounting for current->memcg_nr_pages_over_high. As of this patch, no page_counter has stock yet, so there are no functional changes intended. Signed-off-by: Joshua Hahn --- include/linux/page_counter.h | 4 +- kernel/cgroup/dmem.c | 2 +- mm/hugetlb_cgroup.c | 2 +- mm/memcontrol-v1.c | 3 +- mm/memcontrol.c | 8 ++-- mm/page_counter.c | 82 +++++++++++++++++++++++++++++++----- 6 files changed, 83 insertions(+), 18 deletions(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index 9cb5612fe190f..a5a5a789d002c 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -98,8 +98,8 @@ long page_counter_margin(struct page_counter *counter); void page_counter_cancel(struct page_counter *counter, unsigned long nr_pa= ges); void page_counter_charge(struct page_counter *counter, unsigned long nr_pa= ges); bool page_counter_try_charge(struct page_counter *counter, - unsigned long nr_pages, - struct page_counter **fail); + unsigned long nr_pages, struct page_counter **fail, + bool may_batch, unsigned long *nr_charged); void page_counter_refill_stock(struct page_counter *counter, unsigned long nr_pages); void page_counter_drain_stock_fully(struct page_counter_stock_pcp *stock); diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c index 4683f3d680226..569307aa4bd62 100644 --- a/kernel/cgroup/dmem.c +++ b/kernel/cgroup/dmem.c @@ -736,7 +736,7 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *r= egion, u64 size, goto err; } =20 - if (!page_counter_try_charge(&pool->cnt, size, &fail)) { + if (!page_counter_try_charge(&pool->cnt, size, &fail, false, NULL)) { if (ret_limit_pool) { *ret_limit_pool =3D container_of(fail, struct dmem_cgroup_pool_state, c= nt); css_get(&(*ret_limit_pool)->cs->css); diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index ecb6e0b7819a0..5b8d9f0e25535 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -274,7 +274,7 @@ static int __hugetlb_cgroup_charge_cgroup(int idx, unsi= gned long nr_pages, =20 if (!page_counter_try_charge( __hugetlb_cgroup_counter_from_cgroup(h_cg, idx, rsvd), - nr_pages, &counter)) { + nr_pages, &counter, false, NULL)) { ret =3D -ENOMEM; hugetlb_event(h_cg, idx, HUGETLB_MAX); css_put(&h_cg->css); diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index bf2c7d53b01b1..aba9e3b851235 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -2194,7 +2194,8 @@ bool memcg1_charge_skmem(struct mem_cgroup *memcg, un= signed int nr_pages, { struct page_counter *fail; =20 - if (page_counter_try_charge(&memcg->tcpmem, nr_pages, &fail)) { + if (page_counter_try_charge(&memcg->tcpmem, nr_pages, &fail, + false, NULL)) { memcg->tcpmem_pressure =3D 0; return true; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 48c475909e6bb..04ab7355c6d2d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2708,13 +2708,14 @@ static int try_charge_memcg(struct mem_cgroup *memc= g, gfp_t gfp_mask, =20 reclaim_options =3D MEMCG_RECLAIM_MAY_SWAP; if (do_memsw_account() && - !page_counter_try_charge(&memcg->memsw, batch, &counter)) { + !page_counter_try_charge(&memcg->memsw, batch, &counter, false, + NULL)) { mem_over_limit =3D mem_cgroup_from_counter(counter, memsw); reclaim_options &=3D ~MEMCG_RECLAIM_MAY_SWAP; goto reclaim; } =20 - if (page_counter_try_charge(&memcg->memory, batch, &counter)) + if (page_counter_try_charge(&memcg->memory, batch, &counter, false, NULL)) goto done_restock; =20 if (do_memsw_account()) @@ -5965,7 +5966,8 @@ int __mem_cgroup_try_charge_swap(struct folio *folio) rcu_read_unlock(); =20 if (!mem_cgroup_is_root(memcg) && - !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { + !page_counter_try_charge(&memcg->swap, nr_pages, &counter, false, + NULL)) { memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); mem_cgroup_private_id_put(memcg, nr_pages); diff --git a/mm/page_counter.c b/mm/page_counter.c index 480a447bd7265..3cd8601961673 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -136,23 +136,76 @@ void page_counter_charge(struct page_counter *counter= , unsigned long nr_pages) } } =20 +/* + * Consume the cached charge if enough nr_pages are present, otherwise ret= urn + * failure. Also return failure for charge requests larger than + * PAGE_COUNTER_STOCK_BATCH or if the local lock is already taken. + */ +static bool page_counter_consume_stock(struct page_counter *counter, + unsigned long nr_pages) +{ + struct page_counter_stock_pcp __percpu *stock =3D counter->stock; + struct page_counter_stock_pcp *pcp_stock; + u8 stock_pages; + bool ret =3D false; + int i; + + if (nr_pages > PAGE_COUNTER_STOCK_BATCH || + !local_trylock(&stock->lock)) + return false; + + pcp_stock =3D this_cpu_ptr(stock); + for (i =3D 0; i < NR_PAGE_COUNTER_STOCK; i++) { + if (counter !=3D READ_ONCE(pcp_stock->cached[i])) + continue; + + stock_pages =3D READ_ONCE(pcp_stock->nr_pages[i]); + if (stock_pages >=3D nr_pages) { + stock_pages -=3D nr_pages; + WRITE_ONCE(pcp_stock->nr_pages[i], stock_pages); + if (!stock_pages) { + css_put(counter->stock_css); + WRITE_ONCE(pcp_stock->cached[i], NULL); + } + ret =3D true; + } + break; + } + local_unlock(&stock->lock); + + return ret; +} + /** - * page_counter_try_charge - try to hierarchically charge pages + * page_counter_try_charge - try to hierarchically charge pages using stock * @counter: counter - * @nr_pages: number of pages to charge - * @fail: points first counter to hit its limit, if any + * @nr_pages: number of pages requested + * @fail: points to the first counter to hit its limit, if any + * @may_batch: whether a stock miss may trigger a batch charge + * @nr_charged: optional; set to the hierarchy charge size on success * - * Returns %true on success, or %false and @fail if the counter or one - * of its ancestors has hit its configured limit. + * Return: %true if the request was satisfied. A failed batch charge may u= pdate + * @fail before an exact retry succeeds. */ bool page_counter_try_charge(struct page_counter *counter, - unsigned long nr_pages, - struct page_counter **fail) + unsigned long nr_pages, struct page_counter **fail, + bool may_batch, unsigned long *nr_charged) { + unsigned long charge =3D nr_pages; struct page_counter *c; bool protection =3D track_protection(counter); bool track_failcnt =3D counter->track_failcnt; =20 + if (counter->stock && may_batch) + charge =3D max(nr_pages, PAGE_COUNTER_STOCK_BATCH); + +retry: + if (counter->stock && page_counter_consume_stock(counter, nr_pages)) { + if (nr_charged) + *nr_charged =3D 0; + return true; + } + for (c =3D counter; c; c =3D c->parent) { long new; /* @@ -169,9 +222,9 @@ bool page_counter_try_charge(struct page_counter *count= er, * we either see the new limit or the setter sees the * counter has changed and retries. */ - new =3D atomic_long_add_return(nr_pages, &c->usage); + new =3D atomic_long_add_return(charge, &c->usage); if (new > c->max) { - atomic_long_sub(nr_pages, &c->usage); + atomic_long_sub(charge, &c->usage); /* * This is racy, but we can live with some * inaccuracy in the failcnt which is only used @@ -192,11 +245,20 @@ bool page_counter_try_charge(struct page_counter *cou= nter, WRITE_ONCE(c->watermark, new); } } + if (charge > nr_pages) + page_counter_refill_stock(counter, charge - nr_pages); + if (nr_charged) + *nr_charged =3D charge; return true; =20 failed: for (c =3D counter; c !=3D *fail; c =3D c->parent) - page_counter_cancel(c, nr_pages); + page_counter_cancel(c, charge); + + if (charge > nr_pages) { + charge =3D nr_pages; + goto retry; + } =20 return false; } --=20 2.53.0-Meta From nobody Fri Sep 25 04:07:46 2026 Received: from mail-oi2-f43.google.com (mail-oi2-f43.google.com [74.125.231.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CE474E4308 for ; Wed, 16 Sep 2026 21:06:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.231.235 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592782; cv=none; b=tF3m4rMFcBT3AQ0ZsMFZhsCGUfDf4GDyKxM38UksHfxcdqN52plasGDZQY4ckRT+7yn3GsAhb0Rb5bTBxEIq58zTpHJxFu82/FI7fnS2mXrkcs4/29moh6oUvgVVp3jmxBrB0cq18dWIQhxjcvLcPfn//sM4B+rfc6N9Ex6X++4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592782; c=relaxed/simple; bh=ZProGhW8PrqhlbImgxPr7aE6DxRvaRhCeL/rnTHyH8w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ENmbXDVMwJLROQLdsQMXK1cdl/NZn9KiznlK6lPVKF/1Um2UP78QWUfAa7gie+4ptg0M495tpDBGM6Fxj4XIIP7RH2ejK8mvswiUQkRf/Ppnmz+1TGfbOq+uHsQR8OBV6BnJVA2CdZp4uzsC+r66mjqNAv0Pit1k/4+Ib7URCaU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VZIVRp4p; arc=none smtp.client-ip=74.125.231.235 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VZIVRp4p" Received: by mail-oi2-f43.google.com with SMTP id 46e09a7af769-80032c08611so103384a34.3 for ; Wed, 16 Sep 2026 14:06:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789592761; x=1790197561; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=QPow959ZPjGUlrz9e897hy2PeMPVtPPWVlAib704wP4=; b=VZIVRp4phBioAJsp5CDDUccsu7tttbXwvTyx3X2ymcdfGC3fWvQ19H6CvHEOgDniQl xRppLCSmRbjo4OHAhOYxpRGXowAr8p1jAfIBda8GR04CeLQ/TibQBcQNI8wuSFdMbG+4 hfqDQJPwqu9+1Eu41lNb/PLQp+Zq54InLpylwTjyLF+TO8pZAIvERbY1zrxyC6xaNRDy acEZruXgh06S2keFXA6uO7OG+3xeBe6np3Toam3VhBXRHDxLkyij7hhYMrhqejR+KVZM hTPk4/GcRQctMaLFS+9rc7NKGR2M1o0lm34NLbxAhVdjLE7ZewWjmuM1XxdNW9wLBp0N xSaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789592761; x=1790197561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=QPow959ZPjGUlrz9e897hy2PeMPVtPPWVlAib704wP4=; b=bctMHjzGJj+VoWBTn0hGLn+n9kKf8mBCIw73YQQGNMrOFZpiM70hKzqwElco6rLM2T RNa18GJDpWtLAYnLDhgqW8pUaOUDFi0Z5UV3twi73mUTORVrOCJgsw6dysm4oe4Q5pjX 1KUgcwLmcgoNNrtzUrGAC7MIhb0ll2NRFk14/ii5HEs0N8eDB/Yu2YYShTnW1aJQgPAK fVKiCsZjelXD3Rce9X8BRsSGt1xbOOO6dARx4S9q56E5HR+YMA8B/R8vjJsWN9p3luy3 LssXW3ee7HFGUbzU37IlRbgCebFQt9xTGzf0FFIoRcQ7v1H0VxHYMjhoU3c75NzzsAa8 vx4g== X-Forwarded-Encrypted: i=1; AKwUvBzt/EHREMgBC/0A1TFcm63sdHDpT63j+WMVmYRkJewOTnw/CyZ1Bv3g2wKGssSuc8JPuvNLNzE8eWkCacc=@vger.kernel.org X-Gm-Message-State: AFuF++mU8v2tKAXOT9KnMkfX1Q0uz1/xPX3kUj82+8BQnQX2gBrs/dLO a54KbpOm76xNpt5FMmCtMdkkaW4b9+4cU/ZsEYWZEW2LVCe2LtPVy9uy X-Gm-Gg: AYBFou01aNC4oxllNmtB+UhfwP/R2Xi7XQpOCi/KP8NcDOwTa6TrKkVkaxialOCdunS 9I+D6hJdK3FXd/cI+QaWNaux8pbBt/ReWdYFiLYvf79UjK5gREWTkQw2n/e20ocjWn0Uf3Q0qlF WJNEbfFa7bvy+SqsyZDVPLH/IEqfN7F2qi8vejZ2Tf92OsQhKMOrvAlyMLS9IVjSq/0tsRCY/WY J1fWpX/QQDhRhKvOK3K651RYw2p1fdW9AlkdCTU/Wg8yfgzchnl/JwWMltMC8JF0qYT7xS08jCn 0yxxFnk4lK3UFj2T81DYYS6I2KoCgvb476g/nYwJTRWVrsuN1jX9y9MFxOvV3gLkEta6va2T50F qbrcWXavdFw8wCoy2JExjv6un2QWUYAVBvCxhJTJjj+s49yQ7iFp1QpF31Gsjvk3l/euXEsLV57 MeTOJry2c8dxCs/dsI8+K+WnVsnLgdNt7WvwE6eCwZukBjrglcn+dIyU8RssnkNrm7+SJ5kpm6v Pe33r4CRa+9HL1VZBFSS5QsS7Mvw28= X-Received: by 2002:a05:6830:6503:b0:7e6:d0ad:5254 with SMTP id 46e09a7af769-80b2cfc865fmr8183323a34.8.1789592760843; Wed, 16 Sep 2026 14:06:00 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:37::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-80c46199e7csm901867a34.7.2026.09.16.14.06.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Sep 2026 14:06:00 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko , Shakeel Butt Cc: Roman Gushchin , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Maarten Lankhorst , Maxime Ripard , Natalie Vock , Tejun Heo , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Oscar Salvador , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v6 4/5] mm/memcontrol: move memory stock to page counters Date: Wed, 16 Sep 2026 14:05:50 -0700 Message-ID: <20260916210552.891730-5-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260916210552.891730-1-joshua.hahnjy@gmail.com> References: <20260916210552.891730-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Transition memcg to use the page_counter_stock for the memory page_counter instead of relying on a memcg-wide stock. One aspect that remains non-transparent to memcg is the uncharge path. This is intentional, as the caller is responsible for managing the batching. Cacheable releases refill the stock, while already-batched frees (i.e. uncharge_gather), rollbacks, and accounting transfers uncharge the hierarchy directly. The refill helper itself already falls back to a raw uncharge when the stock cannot accept the pages anyways. Because the memory and memsw counters no longer share one stock, their raw values can temporarily diverge. Preserve the legacy user-visible memory <=3D memory+swap invariant by reporting the larger raw value for memory.memsw.usage_in_bytes. This masks the temporary inversions caused by the decoupling of the single memcg stock. Note that this remains a bounded stock-related overestimate, consistent with the existing fuzzy usage reporting for usage. With this transition, remove all memcg code that is no longer used. After all of this, there should be no functional change for cgroup v2 users. All v2 behaviors from memcg are preserved, just moved from memcg to page_counter code, so that future work can introduce additional page_counters without removing the fast path. Explicitly, the preserved behaviors are: - 7-slot stock - drain policy works locally and remotely through the memcg_wq - check whether a stock requires flushing before taking action - exact charging for non-spinning callers - report hierarchy growth for memory.high overage accounting As of this patch, this leaves memsw un-stocked and always taking the slow path (raw hierarchy charge). The next patch will make memsw stocked, which will close the fast path gap for legacy cgroup users. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- mm/memcontrol-v1.c | 9 +- mm/memcontrol.c | 279 ++++++++------------------------------------- 2 files changed, 56 insertions(+), 232 deletions(-) diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index aba9e3b851235..22822e7a85b49 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -122,10 +122,13 @@ static unsigned long mem_cgroup_usage(struct mem_cgro= up *memcg, bool swap) if (swap) val +=3D total_swap_pages - get_nr_swap_pages(); } else { - if (!swap) + if (!swap) { val =3D page_counter_read(&memcg->memory); - else - val =3D page_counter_read(&memcg->memsw); + } else { + /* Preserve the user-visible memory <=3D memsw invariant. */ + val =3D max(page_counter_read(&memcg->memory), + page_counter_read(&memcg->memsw)); + } } return val; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 04ab7355c6d2d..1a209ad535540 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2049,33 +2049,11 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *= memcg) pr_cont(" are going to be killed due to memory.oom.group set\n"); } =20 -/* - * The value of NR_MEMCG_STOCK is selected to keep the cached memcgs and t= heir - * nr_pages in a single cacheline. This may change in future. - */ -#define NR_MEMCG_STOCK 7 - -/* - * Watermarks for a charge stock slot, in the spirit of pcp->high and - * pcp->batch: MEMCG_STOCK_HIGH is the high watermark at which a slot is - * trimmed, and it is trimmed down to MEMCG_STOCK_LOW rather than emptied. - */ -#define MEMCG_STOCK_LOW (MEMCG_CHARGE_BATCH / 2) -#define MEMCG_STOCK_HIGH (MEMCG_CHARGE_BATCH) - #define FLUSHING_CACHED_CHARGE 0 -struct memcg_stock_pcp { - local_trylock_t lock; - uint8_t nr_pages[NR_MEMCG_STOCK]; - struct mem_cgroup *cached[NR_MEMCG_STOCK]; =20 - struct work_struct work; - unsigned long flags; - uint8_t drain_idx; -}; - -static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) =3D { +static DEFINE_PER_CPU_ALIGNED(struct page_counter_stock_pcp, memory_stock)= =3D { .lock =3D INIT_LOCAL_TRYLOCK(lock), + .base =3D &memory_stock, }; =20 /* @@ -2125,52 +2103,6 @@ static void drain_obj_stock(struct obj_stock_pcp *st= ock); static bool obj_stock_flush_required(struct obj_stock_pcp *stock, struct mem_cgroup *root_memcg); =20 -/** - * consume_stock: Try to consume stocked charge on this cpu. - * @memcg: memcg to consume from. - * @nr_pages: how many pages to charge. - * - * Consume the cached charge if enough nr_pages are present otherwise retu= rn - * failure. Also return failure for charge request larger than - * MEMCG_CHARGE_BATCH or if the local lock is already taken. - * - * returns true if successful, false otherwise. - */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) -{ - struct memcg_stock_pcp *stock; - uint8_t stock_pages; - bool ret =3D false; - int i; - - if (nr_pages > MEMCG_CHARGE_BATCH || - !local_trylock(&memcg_stock.lock)) - return ret; - - stock =3D this_cpu_ptr(&memcg_stock); - - for (i =3D 0; i < NR_MEMCG_STOCK; ++i) { - if (memcg !=3D READ_ONCE(stock->cached[i])) - continue; - - stock_pages =3D READ_ONCE(stock->nr_pages[i]); - if (stock_pages >=3D nr_pages) { - stock_pages -=3D nr_pages; - WRITE_ONCE(stock->nr_pages[i], stock_pages); - if (!stock_pages) { - css_put(&memcg->css); - WRITE_ONCE(stock->cached[i], NULL); - } - ret =3D true; - } - break; - } - - local_unlock(&memcg_stock.lock); - - return ret; -} - static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages) { page_counter_uncharge(&memcg->memory, nr_pages); @@ -2178,49 +2110,22 @@ static void memcg_uncharge(struct mem_cgroup *memcg= , unsigned int nr_pages) page_counter_uncharge(&memcg->memsw, nr_pages); } =20 -/* - * Returns stocks cached in percpu and reset cached information. - */ -static void drain_stock(struct memcg_stock_pcp *stock, int i) -{ - struct mem_cgroup *old =3D READ_ONCE(stock->cached[i]); - uint8_t stock_pages; - - if (!old) - return; - - stock_pages =3D READ_ONCE(stock->nr_pages[i]); - if (stock_pages) { - memcg_uncharge(old, stock_pages); - WRITE_ONCE(stock->nr_pages[i], 0); - } - - css_put(&old->css); - WRITE_ONCE(stock->cached[i], NULL); -} - -static void drain_stock_fully(struct memcg_stock_pcp *stock) +static void drain_local_stock(struct work_struct *work) { - int i; - - for (i =3D 0; i < NR_MEMCG_STOCK; ++i) - drain_stock(stock, i); -} - -static void drain_local_memcg_stock(struct work_struct *dummy) -{ - struct memcg_stock_pcp *stock; + struct page_counter_stock_pcp *pcp_stock; + struct page_counter_stock_pcp __percpu *stock; =20 if (WARN_ONCE(!in_task(), "drain in non-task context")) return; =20 - local_lock(&memcg_stock.lock); + stock =3D container_of(work, struct page_counter_stock_pcp, work)->base; + local_lock(&stock->lock); =20 - stock =3D this_cpu_ptr(&memcg_stock); - drain_stock_fully(stock); - clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); + pcp_stock =3D this_cpu_ptr(stock); + page_counter_drain_stock_fully(pcp_stock); + clear_bit(FLUSHING_CACHED_CHARGE, &pcp_stock->flags); =20 - local_unlock(&memcg_stock.lock); + local_unlock(&stock->lock); } =20 static void drain_local_obj_stock(struct work_struct *dummy) @@ -2239,92 +2144,6 @@ static void drain_local_obj_stock(struct work_struct= *dummy) local_unlock(&obj_stock.lock); } =20 -static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) -{ - struct memcg_stock_pcp *stock; - struct mem_cgroup *cached; - unsigned int stock_pages; - bool success =3D false; - int empty_slot =3D -1; - int i; - - /* - * nr_pages[] is a uint8_t and a slot's count is capped at - * MEMCG_STOCK_HIGH. Raising MEMCG_CHARGE_BATCH beyond 127 would need - * more careful handling of nr_pages[] in struct memcg_stock_pcp. - */ - BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S8_MAX); - BUILD_BUG_ON(MEMCG_STOCK_HIGH > U8_MAX); - - VM_WARN_ON_ONCE(mem_cgroup_is_root(memcg)); - - if (nr_pages > MEMCG_CHARGE_BATCH || - !local_trylock(&memcg_stock.lock)) { - /* - * In case of larger than batch refill or unlikely failure to - * lock the percpu memcg_stock.lock, uncharge memcg directly. - */ - memcg_uncharge(memcg, nr_pages); - return; - } - - stock =3D this_cpu_ptr(&memcg_stock); - for (i =3D 0; i < NR_MEMCG_STOCK; ++i) { - cached =3D READ_ONCE(stock->cached[i]); - if (!cached && empty_slot =3D=3D -1) - empty_slot =3D i; - if (memcg =3D=3D READ_ONCE(stock->cached[i])) { - stock_pages =3D READ_ONCE(stock->nr_pages[i]) + nr_pages; - if (stock_pages > MEMCG_STOCK_HIGH) { - memcg_uncharge(memcg, - stock_pages - MEMCG_STOCK_LOW); - stock_pages =3D MEMCG_STOCK_LOW; - } - WRITE_ONCE(stock->nr_pages[i], stock_pages); - success =3D true; - break; - } - } - - if (!success) { - i =3D empty_slot; - if (i =3D=3D -1) { - i =3D stock->drain_idx++; - if (stock->drain_idx =3D=3D NR_MEMCG_STOCK) - stock->drain_idx =3D 0; - drain_stock(stock, i); - } - css_get(&memcg->css); - WRITE_ONCE(stock->cached[i], memcg); - WRITE_ONCE(stock->nr_pages[i], nr_pages); - } - - local_unlock(&memcg_stock.lock); -} - -static bool is_memcg_drain_needed(struct memcg_stock_pcp *stock, - struct mem_cgroup *root_memcg) -{ - struct mem_cgroup *memcg; - bool flush =3D false; - int i; - - rcu_read_lock(); - for (i =3D 0; i < NR_MEMCG_STOCK; ++i) { - memcg =3D READ_ONCE(stock->cached[i]); - if (!memcg) - continue; - - if (READ_ONCE(stock->nr_pages[i]) && - mem_cgroup_is_descendant(memcg, root_memcg)) { - flush =3D true; - break; - } - } - rcu_read_unlock(); - return flush; -} - static bool schedule_drain_work(int cpu, struct work_struct *work) { /* @@ -2356,23 +2175,25 @@ void drain_all_stock(struct mem_cgroup *root_memcg) * Notify other cpus that system-wide "drain" is running * We do not care about races with the cpu hotplug because cpu down * as well as workers from this path always operate on the local - * per-cpu data. CPU up doesn't touch memcg_stock at all. + * per-cpu data. CPU up doesn't touch the stocks at all. */ migrate_disable(); curcpu =3D smp_processor_id(); for_each_online_cpu(cpu) { - struct memcg_stock_pcp *memcg_st =3D &per_cpu(memcg_stock, cpu); + struct page_counter_stock_pcp *memory_st =3D + per_cpu_ptr(&memory_stock, cpu); struct obj_stock_pcp *obj_st =3D &per_cpu(obj_stock, cpu); =20 - if (!test_bit(FLUSHING_CACHED_CHARGE, &memcg_st->flags) && - is_memcg_drain_needed(memcg_st, root_memcg) && + if (!test_bit(FLUSHING_CACHED_CHARGE, &memory_st->flags) && + page_counter_stock_flush_required(memory_st, + &root_memcg->css) && !test_and_set_bit(FLUSHING_CACHED_CHARGE, - &memcg_st->flags)) { + &memory_st->flags)) { if (cpu =3D=3D curcpu) - drain_local_memcg_stock(&memcg_st->work); - else if (!schedule_drain_work(cpu, &memcg_st->work)) + drain_local_stock(&memory_st->work); + else if (!schedule_drain_work(cpu, &memory_st->work)) clear_bit(FLUSHING_CACHED_CHARGE, - &memcg_st->flags); + &memory_st->flags); } =20 if (!test_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags) && @@ -2392,12 +2213,14 @@ void drain_all_stock(struct mem_cgroup *root_memcg) =20 static int memcg_hotplug_cpu_dead(unsigned int cpu) { - struct memcg_stock_pcp *memcg_st =3D &per_cpu(memcg_stock, cpu); + struct page_counter_stock_pcp *stock; struct obj_stock_pcp *obj_st =3D &per_cpu(obj_stock, cpu); =20 /* no need for the local lock */ drain_obj_stock(obj_st); - drain_stock_fully(memcg_st); + stock =3D per_cpu_ptr(&memory_stock, cpu); + page_counter_drain_stock_fully(stock); + clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); =20 /* * A drain work queued before the CPU went away is executed by an @@ -2405,7 +2228,6 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) * clear the flags here to make these stocks drainable again once * the CPU comes back online. */ - clear_bit(FLUSHING_CACHED_CHARGE, &memcg_st->flags); clear_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags); =20 return 0; @@ -2685,10 +2507,10 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask) static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned int nr_pages) { - unsigned int batch =3D max(MEMCG_CHARGE_BATCH, nr_pages); int nr_retries =3D MAX_RECLAIM_RETRIES; struct mem_cgroup *mem_over_limit; struct page_counter *counter; + unsigned long nr_charged; unsigned long nr_reclaimed; bool passed_oom =3D false; unsigned int reclaim_options; @@ -2696,37 +2518,30 @@ static int try_charge_memcg(struct mem_cgroup *memc= g, gfp_t gfp_mask, bool raised_max_event =3D false; unsigned long pflags; bool allow_spinning =3D gfpflags_allow_spinning(gfp_mask); + bool may_batch =3D allow_spinning; int ret =3D 0; =20 retry: - if (consume_stock(memcg, nr_pages)) - return ret; - - if (!allow_spinning) - /* Avoid the refill and flush of the older stock */ - batch =3D nr_pages; - reclaim_options =3D MEMCG_RECLAIM_MAY_SWAP; if (do_memsw_account() && - !page_counter_try_charge(&memcg->memsw, batch, &counter, false, + !page_counter_try_charge(&memcg->memsw, nr_pages, &counter, false, NULL)) { mem_over_limit =3D mem_cgroup_from_counter(counter, memsw); reclaim_options &=3D ~MEMCG_RECLAIM_MAY_SWAP; goto reclaim; } =20 - if (page_counter_try_charge(&memcg->memory, batch, &counter, false, NULL)) - goto done_restock; + if (page_counter_try_charge(&memcg->memory, nr_pages, &counter, + may_batch, &nr_charged)) + goto check_high; =20 if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, batch); + page_counter_uncharge(&memcg->memsw, nr_pages); mem_over_limit =3D mem_cgroup_from_counter(counter, memory); =20 reclaim: - if (batch > nr_pages) { - batch =3D nr_pages; - goto retry; - } + /* Do not retry speculative batch charges after the first miss. */ + may_batch =3D false; =20 /* * Prevent unbounded recursion when reclaim operations need to @@ -2839,10 +2654,9 @@ static int try_charge_memcg(struct mem_cgroup *memcg= , gfp_t gfp_mask, =20 return ret; =20 -done_restock: - if (batch > nr_pages) - refill_stock(memcg, batch - nr_pages); - +check_high: + if (!nr_charged) + return ret; /* * If the hierarchy is above the normal consumption range, schedule * reclaim on returning to userland. We can perform reclaim here @@ -2882,7 +2696,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg,= gfp_t gfp_mask, * and distribute reclaim work and delay penalties * based on how much each task is actually allocating. */ - current->memcg_nr_pages_over_high +=3D batch; + current->memcg_nr_pages_over_high +=3D nr_charged; set_notify_resume(current); break; } @@ -3187,8 +3001,11 @@ static void obj_cgroup_uncharge_pages(struct obj_cgr= oup *objcg, =20 account_kmem_nmi_safe(memcg, -nr_pages); memcg1_account_kmem(memcg, -nr_pages); - if (!mem_cgroup_is_root(memcg)) - refill_stock(memcg, nr_pages); + if (!mem_cgroup_is_root(memcg)) { + page_counter_refill_stock(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); + } =20 css_put(&memcg->css); } @@ -4287,6 +4104,8 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *pare= nt_css) page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX); if (parent) { page_counter_init(&memcg->memory, &parent->memory, memcg_on_dfl); + memcg->memory.stock =3D &memory_stock; + memcg->memory.stock_css =3D &memcg->css; page_counter_init(&memcg->swap, &parent->swap, false); #ifdef CONFIG_MEMCG_V1 WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); @@ -5754,7 +5573,7 @@ void mem_cgroup_sk_uncharge(const struct sock *sk, un= signed int nr_pages) =20 mod_memcg_state(memcg, MEMCG_SOCK, -nr_pages); =20 - refill_stock(memcg, nr_pages); + page_counter_refill_stock(&memcg->memory, nr_pages); } =20 void mem_cgroup_flush_workqueue(void) @@ -5902,6 +5721,8 @@ int __init mem_cgroup_init(void) * exceed S32_MAX / PAGE_SIZE. */ BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S32_MAX / PAGE_SIZE); + /* Batched page-counter charges feed memcg's memory.high accounting. */ + BUILD_BUG_ON(MEMCG_CHARGE_BATCH !=3D PAGE_COUNTER_STOCK_BATCH); =20 memcg_struct_check(); =20 @@ -5912,8 +5733,8 @@ int __init mem_cgroup_init(void) WARN_ON(!memcg_wq); =20 for_each_possible_cpu(cpu) { - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_memcg_stock); + INIT_WORK(&per_cpu_ptr(&memory_stock, cpu)->work, + drain_local_stock); INIT_WORK(&per_cpu_ptr(&obj_stock, cpu)->work, drain_local_obj_stock); } --=20 2.53.0-Meta From nobody Fri Sep 25 04:07:46 2026 Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA5C24E2F3E for ; Wed, 16 Sep 2026 21:06:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592782; cv=none; b=Yb1kHVPEe24c/vXqKtKBiFIIvS1rQSnD9jjAupb05CBy6UkV98W/p+xn5ubwTYy5zR5YtHWZf1wTZf0e4jdzmR1ne+tHku6swamic9fuu++gocg7oRteV3VdLBrta7o2J/4EsJc/rCIqfR7P+x/mbI6cjSYRgpVDDes16v/tMPI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789592782; c=relaxed/simple; bh=4iNlQRuHJ3/AP4thfCwYHevvDqFDf2gRy5/msQNXKiU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BBwfC4kj1PWY/gDH5lobZ6meGw0U7InrO0ZIMKZipU21SHqZXk17whoQvzKbTDnF7q3F2UOyDQeUqwvuzpZEPSDo/F9ZL6+019qKda3Zmu6hnGHmM6T6dDmAMycpdP9rkhpGHqO1RASuKPlE/QX2RW8Pwi95VCjgtp2QkZ28tP8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WVJkKTre; arc=none smtp.client-ip=209.85.210.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WVJkKTre" Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-7f4df360cc9so78508a34.1 for ; Wed, 16 Sep 2026 14:06:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789592762; x=1790197562; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=HaJwiI/6dGGbCvd8Rz5XOkB+OlzyBFXfmU7W/ZpuGJs=; b=WVJkKTre4aCj15aYOsoOM6QO3yqYX7djhgTpKS4/D4fvYcrrLsGWd0OtdxH+KrTN0z lUItTKad+25FSMAx+NyFQ5CaM28We44s0YrpE1PVoPJ9hMQ4BoSgDENApkxpFAyzeiwD q5KLvIvRyhR+8GBMhSSC5DXTGU0kxrghStqPQ0vRIKoa8uZBFr51pFF15VH1yrdFvXfq MSiOfB8T3WOPhQYnnZ0PskVFnKFixZRC51dNpCky7KuWaxiS8Dxge1d5LEeINTNZrj+3 xh3iUeaE5scl8y3kxfh6CmmdsDDMOEzxypp0KOp3GJK5PT7+qyTcgYMVwlVfJslIUBiq XgZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789592762; x=1790197562; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=HaJwiI/6dGGbCvd8Rz5XOkB+OlzyBFXfmU7W/ZpuGJs=; b=K4LVUuHRKhRpZs3T4ChPoAng9lpRaYlt50tJkbIpjnB+mjKJtVcOAmnec1dDbx2UVD Jtim2qm0c6j/IE1S6RVWW4UTgDg4zCOMsfRZYnw11HY3Rap+nm/v/bkeaIW07H/RHOtS 69kE4Oe8G821rvZ4SDX80M8A2iboqcTXk6G0yTO+PZv6/iuqTrkfhZDarKXNnteRNjZc 9fT6YluSWQEmi5uvaAHgMoaSD9wRwjL7+vNmz/zTbwGbZypcLUdVlP0YlwhysPdu+TOC TnFyuqhXRFFXzk0WECqbq3NufMOCEbwpLhZJzP0FUnbBtTRJIz8ax5vld17PoGLh/mZ2 i/Nw== X-Forwarded-Encrypted: i=1; AKwUvBzoivPLYaP+lVYiL11Sx598swanUv25QeJQzRK5pj1KRyPZDujgPQcQXtzCTlhNRr1yKg7V/+QrkSxJEeo=@vger.kernel.org X-Gm-Message-State: AFuF++lWutBLovgX351EhxpbaX7nS6JL4tU44b9OfjkSBAFS4WcQhoqO /+JZm/2WYHRTF0fX2ZOuo6Ks/biwUUmmS+wRk4YTmYmgpWwn9D6kGs95 X-Gm-Gg: AYBFou1l/JURMNvuMLVak3MbI1luSlYkhcDcbz5Z1jPInUJ89NeRscSaHvr18Xao072 n7Z50fO6hY+3kZ5xcElPFMlQHZ9YtIK5zFhzwFwB0I1SdBbzzx8sdUbN/pDoenach37PMUuowpA oTmUlCkXyRzgIZdaeO1fnGHMPAu+4xMH1TSqg9GmljcWI3uUyAWEWDNai/MlC2IkrDDQMUwfjfZ Suj5gvC9HQkug9fXZgqd/DiBe47L+uu5g5ktRg5/0lDYMXPqXckvswrhSOa5IXeOJ8a7xKoPOm1 P7tiRiKPKtIHK+j0Y8hEoc04Pmczi8dZnS8ffKj3bnQdkYG0S5Qwmv8dgrsotLfqLs6kS6wf/A1 wODCgK11lmZMgQy0aVJGQpN1fB8INSbU3HkCUidgs3+y6J6fXzVFiLiUHmu77tTP2j30fPNBrm/ DfeWYk1RDH48d42gvif0tZwbtXCa34nykx6InSaBVaor+ONtdrRQF4M0GenzBooC+nYEumg+iFX toaE4QgNS6lQdaxnwIxOf1dHYpLOw== X-Received: by 2002:a05:6830:a1d0:10b0:805:5bf7:4ab with SMTP id 46e09a7af769-80c4bd14b16mr715864a34.1.1789592762077; Wed, 16 Sep 2026 14:06:02 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:2e::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-80c466a2dcesm1019834a34.11.2026.09.16.14.06.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Sep 2026 14:06:01 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Michal Hocko , Shakeel Butt Cc: Roman Gushchin , Muchun Song , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Maarten Lankhorst , Maxime Ripard , Natalie Vock , Tejun Heo , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Oscar Salvador , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v6 5/5] mm/memcontrol: add stock to the memsw page counter Date: Wed, 16 Sep 2026 14:05:51 -0700 Message-ID: <20260916210552.891730-6-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260916210552.891730-1-joshua.hahnjy@gmail.com> References: <20260916210552.891730-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Attach legacy memsw page counters to their own seven-slot per-CPU stock. Charge, refill, drain, and hotplug handling now operate on the memory and memsw banks independently. Factor the common drain scheduling into schedule_stock_drain() now that both stocks use it. Keep the existing direct memsw rollback when the memory charge fails, ensuring that failed allocations do not replenish the newly attached memsw stock. The separate banks can hit, contend, evict, and drain independently, so their raw counters can temporarily drift by their cached amounts. The previous patch preserves the user-visible cgroup-v1 invariant by reporting memory.memsw.usage_in_bytes as the larger raw value. Signed-off-by: Joshua Hahn --- mm/memcontrol.c | 62 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 46 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1a209ad535540..7d5b2539c5699 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2056,6 +2056,13 @@ static DEFINE_PER_CPU_ALIGNED(struct page_counter_st= ock_pcp, memory_stock) =3D { .base =3D &memory_stock, }; =20 +#ifdef CONFIG_MEMCG_V1 +static DEFINE_PER_CPU_ALIGNED(struct page_counter_stock_pcp, memsw_stock) = =3D { + .lock =3D INIT_LOCAL_TRYLOCK(lock), + .base =3D &memsw_stock, +}; +#endif + /* * NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp * (lock, accounting metadata, nr_bytes[] and cached[]) fits within a @@ -2160,12 +2167,30 @@ static bool schedule_drain_work(int cpu, struct wor= k_struct *work) return true; } =20 +static void schedule_stock_drain(struct page_counter_stock_pcp __percpu *s= tock, + struct cgroup_subsys_state *root_css, + int cpu, int curcpu) +{ + struct page_counter_stock_pcp *pcp_stock =3D per_cpu_ptr(stock, cpu); + + if (test_bit(FLUSHING_CACHED_CHARGE, &pcp_stock->flags) || + !page_counter_stock_flush_required(pcp_stock, root_css) || + test_and_set_bit(FLUSHING_CACHED_CHARGE, &pcp_stock->flags)) + return; + + if (cpu =3D=3D curcpu) + drain_local_stock(&pcp_stock->work); + else if (!schedule_drain_work(cpu, &pcp_stock->work)) + clear_bit(FLUSHING_CACHED_CHARGE, &pcp_stock->flags); +} + /* * Drains all per-CPU charge caches for given root_memcg resp. subtree * of the hierarchy under it. */ void drain_all_stock(struct mem_cgroup *root_memcg) { + struct cgroup_subsys_state *root_css =3D &root_memcg->css; int cpu, curcpu; =20 /* If someone's already draining, avoid adding running more workers. */ @@ -2180,21 +2205,13 @@ void drain_all_stock(struct mem_cgroup *root_memcg) migrate_disable(); curcpu =3D smp_processor_id(); for_each_online_cpu(cpu) { - struct page_counter_stock_pcp *memory_st =3D - per_cpu_ptr(&memory_stock, cpu); struct obj_stock_pcp *obj_st =3D &per_cpu(obj_stock, cpu); =20 - if (!test_bit(FLUSHING_CACHED_CHARGE, &memory_st->flags) && - page_counter_stock_flush_required(memory_st, - &root_memcg->css) && - !test_and_set_bit(FLUSHING_CACHED_CHARGE, - &memory_st->flags)) { - if (cpu =3D=3D curcpu) - drain_local_stock(&memory_st->work); - else if (!schedule_drain_work(cpu, &memory_st->work)) - clear_bit(FLUSHING_CACHED_CHARGE, - &memory_st->flags); - } + schedule_stock_drain(&memory_stock, root_css, cpu, curcpu); +#ifdef CONFIG_MEMCG_V1 + if (do_memsw_account()) + schedule_stock_drain(&memsw_stock, root_css, cpu, curcpu); +#endif =20 if (!test_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags) && obj_stock_flush_required(obj_st, root_memcg) && @@ -2221,6 +2238,11 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) stock =3D per_cpu_ptr(&memory_stock, cpu); page_counter_drain_stock_fully(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); +#ifdef CONFIG_MEMCG_V1 + stock =3D per_cpu_ptr(&memsw_stock, cpu); + page_counter_drain_stock_fully(stock); + clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); +#endif =20 /* * A drain work queued before the CPU went away is executed by an @@ -2524,8 +2546,8 @@ static int try_charge_memcg(struct mem_cgroup *memcg,= gfp_t gfp_mask, retry: reclaim_options =3D MEMCG_RECLAIM_MAY_SWAP; if (do_memsw_account() && - !page_counter_try_charge(&memcg->memsw, nr_pages, &counter, false, - NULL)) { + !page_counter_try_charge(&memcg->memsw, nr_pages, &counter, + may_batch, NULL)) { mem_over_limit =3D mem_cgroup_from_counter(counter, memsw); reclaim_options &=3D ~MEMCG_RECLAIM_MAY_SWAP; goto reclaim; @@ -3004,7 +3026,7 @@ static void obj_cgroup_uncharge_pages(struct obj_cgro= up *objcg, if (!mem_cgroup_is_root(memcg)) { page_counter_refill_stock(&memcg->memory, nr_pages); if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); + page_counter_refill_stock(&memcg->memsw, nr_pages); } =20 css_put(&memcg->css); @@ -4108,6 +4130,10 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *par= ent_css) memcg->memory.stock_css =3D &memcg->css; page_counter_init(&memcg->swap, &parent->swap, false); #ifdef CONFIG_MEMCG_V1 + if (!memcg_on_dfl) { + memcg->memsw.stock =3D &memsw_stock; + memcg->memsw.stock_css =3D &memcg->css; + } WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); memcg->memory.track_failcnt =3D !memcg_on_dfl; memcg->memsw.track_failcnt =3D !memcg_on_dfl; @@ -5735,6 +5761,10 @@ int __init mem_cgroup_init(void) for_each_possible_cpu(cpu) { INIT_WORK(&per_cpu_ptr(&memory_stock, cpu)->work, drain_local_stock); +#ifdef CONFIG_MEMCG_V1 + INIT_WORK(&per_cpu_ptr(&memsw_stock, cpu)->work, + drain_local_stock); +#endif INIT_WORK(&per_cpu_ptr(&obj_stock, cpu)->work, drain_local_obj_stock); } --=20 2.53.0-Meta