From nobody Sun Apr 5 21:35:14 2026 Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A14972C11D6 for ; Mon, 23 Feb 2026 22:38:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771886318; cv=none; b=Rt9Mh+L27TCN7bePZXbmjTNHBg2wiuSj5VA8aWXNnUBagmryNmzQaVyrL2Jz6726lETmEGjgUITEuzy6N3ePKfha8rHXxFyTIQc5ekyYI0jL+hYtlh0uoGiOAU9FfKtImb2GVZ1LEjxX6/CkXV8ozGLEaE9wJ4wZqqNBLdtQoFw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771886318; c=relaxed/simple; bh=CaHKfHzTnSEHysdxeu/IeDFU49MuR1XQ3fwEJyAveAQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qc+ov16Cf7JGAmVG9GBDoLDyGMPVF9IL7zRIyP0nTcHzvsnGOyHO/J5XWwl7+ybPj1FHNSS4SmgralNKTzmBamE+it8TGn6afZ5lJbeM9PSZFJWzjw8gD7YiGdiNnAE6DjusCLKZNY930WAFN2IjegBhwj41JjM5Hmdm81OdoQk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mPjEDf68; arc=none smtp.client-ip=209.85.210.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mPjEDf68" Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-7d4c65d772cso4470483a34.1 for ; Mon, 23 Feb 2026 14:38:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1771886316; x=1772491116; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yQzkmrIS51etVi0n5gq17TgjaIpr+hzH+jF14jX4sdo=; b=mPjEDf68l6hiid2EqfuyhHlTz9lrGl+PBYPGHQ1en0dIe5lvtm0LOs62bAOS4YOfeu HML0WyfVtO/UqJ8XW2UNq9YciWUpNdaSxVmau5iQV9uA8EHcMsbXzUq+B3JOD0Q6Af+e 3slCfmkOeW6AB9u3nOmytd4n9703GagipNDPaFCZes1Xz8X1Ge3yuz3IMjP6ctjpYIM5 6LnhFWfQDQija+JRiIqG7PH1iDpwF4DqNN7VJhYWZXosAk5fYffmSuQKVwriPOBlwWdn TXcffbl2FkfBuN2kDoJMJ1Cp7qmxV+8Dxjmk37CF2mUFP4udyb79LNBAPJrYG3J2d5Wm 6bpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771886316; x=1772491116; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=yQzkmrIS51etVi0n5gq17TgjaIpr+hzH+jF14jX4sdo=; b=gtOnGzbc9Yb7R8qQGhYqMvHR+4fYbt6/2V7gzatTq+K59C+2e84lo8f96azDh5uE7G VSKxMPa9WTBdU7CemV6iK+daXZ1V4vPih4rTH53oHt3HKdWr+MrW04C7K8Uos9ZptEDl l4GR/moQdEG2hKayb1LqUpkjBeRFcsh4HH8BP5lv5Udd156tLapO9N3Y8UGsM8gaiiig ivv8AZjE7UNm1ib7gRRXzns7uuIq7rq21xhku7+RcKX47I8+8t8UMoiaMyXxVFoPjU50 ekKJgDALX/jUEtU0sg9xLTRXBgGkBdleuNikUvGfzjN80rGZ/9VXyBHMC2JNz7lIDElH in2A== X-Forwarded-Encrypted: i=1; AJvYcCW7LM5u+UbU4S7Kv4394qKQr3YZ05KpqAvklmIs9FTG30a78ZXGJ4MOcPwswqjKyJLjfffbuax6sRkId9c=@vger.kernel.org X-Gm-Message-State: AOJu0Yy0kV0gtDbm96PyCFm8HQOm61t8YjVju7lE1fZD8ycwxY0WjcNf I5DCOTk2KhAjphTSP+aXwfyoCqXhKEId8xHQuCXSsrgT3CPgx7hGCsiO X-Gm-Gg: AZuq6aJ+/vrPD+xL52HbZWLxpMT45Ixu88kP7Z1oaVFgNsgyMtStdGe2AnOYZ0Cw9nq dWocaanAdukt4vBaYYVCMVkQru4jQQTYFxrSPWUBSDiZYluqSFfiQ0Th+olzGCNCbk3cOwiCD/X CI0kf352g2hX4eYjmyHaSL5CziVMybBPyB2N+dPodqUHqx0MDr+s6K7Be1cWk8XQK9RmWkjAESt oCOrU4ICRNqLIvq0i9a5YrBa2eMVe9V4Xhnu0/ha1jqPstvjR9lR0Juk+xlH80xbzD2sRNsiCUp vZiJQhBe1pZ1oUxgdGYSVfpr6dv+WcqGjpSnGhpk7+w83frGWk7OYtp7uH+oqYZ+vFeaKpgBqvS tk32huPft2gwaGcFvuF1kwern/CBojIRpWLT+xY0rb5VG0OqMtPfTTsPiK+orFQaimYcjx24Ifg 6Q3+WKWHsAkYUBT9CPr4j+ X-Received: by 2002:a05:6830:369b:b0:7c5:2dbf:4a7d with SMTP id 46e09a7af769-7d52c1c04c4mr6267036a34.31.1771886316479; Mon, 23 Feb 2026 14:38:36 -0800 (PST) Received: from localhost ([2a03:2880:10ff:9::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d52cfa03bcsm8389594a34.10.2026.02.23.14.38.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Feb 2026 14:38:36 -0800 (PST) From: Joshua Hahn To: Joshua Hahn Cc: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [RFC PATCH 2/6] mm/page_counter: Introduce tiered memory awareness to page_counter Date: Mon, 23 Feb 2026 14:38:25 -0800 Message-ID: <20260223223830.586018-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260223223830.586018-1-joshua.hahnjy@gmail.com> References: <20260223223830.586018-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On systems with tiered memory, there is currently no tracking of memory at the tier-memcg granularity. While per-memcg-lruvec serves at a finer granularity that can be accumulated to give us the desired per-tier-memcg accounting, relying on these lruvec stats for limit checking can prove touch too many hot paths too frequently and can introduce increased latency for other memcg users. Instead, add a new cacheline in struct page_counter to track toptier memcg limits and usage, as well as cached capacity values. This cacheline is only used by the mem_cgroup->memory page_counter. Also, introduce helpers that use these new fields to calculate proportional toptier high and low values, based on the system's toptier:total capacity ratio. Signed-off-by: Joshua Hahn --- include/linux/page_counter.h | 22 +++++++++++++++++++++- mm/page_counter.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index d649b6bbbc87..128c1272c88c 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -5,6 +5,7 @@ #include #include #include +#include #include =20 struct page_counter { @@ -31,9 +32,23 @@ struct page_counter { /* Latest cg2 reset watermark */ unsigned long local_watermark; =20 - /* Keep all the read most fields in a separete cacheline. */ + /* Keep all the tiered memory fields in a separate cacheline. */ CACHELINE_PADDING(_pad2_); =20 + atomic_long_t toptier_usage; + + /* effective toptier-proportional low protection */ + unsigned long etoptier_low; + atomic_long_t toptier_low_usage; + atomic_long_t children_toptier_low_usage; + + /* Cached toptier capacity for proportional limit calculations */ + unsigned long toptier_capacity; + unsigned long total_capacity; + + /* Keep all the read most fields in a separate cacheline. */ + CACHELINE_PADDING(_pad3_); + bool protection_support; bool track_failcnt; unsigned long min; @@ -61,6 +76,9 @@ static inline void page_counter_init(struct page_counter = *counter, counter->parent =3D parent; counter->protection_support =3D protection_support; counter->track_failcnt =3D false; + counter->toptier_usage =3D (atomic_long_t)ATOMIC_LONG_INIT(0); + counter->toptier_capacity =3D 0; + counter->total_capacity =3D 0; } =20 static inline unsigned long page_counter_read(struct page_counter *counter) @@ -103,6 +121,8 @@ static inline void page_counter_reset_watermark(struct = page_counter *counter) void page_counter_calculate_protection(struct page_counter *root, struct page_counter *counter, bool recursive_protection); +unsigned long page_counter_toptier_high(struct page_counter *counter); +unsigned long page_counter_toptier_low(struct page_counter *counter); #else static inline void page_counter_calculate_protection(struct page_counter *= root, struct page_counter *counter, diff --git a/mm/page_counter.c b/mm/page_counter.c index 661e0f2a5127..5ec97811c418 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -462,4 +462,38 @@ void page_counter_calculate_protection(struct page_cou= nter *root, atomic_long_read(&parent->children_low_usage), recursive_protection)); } + +unsigned long page_counter_toptier_high(struct page_counter *counter) +{ + unsigned long high =3D READ_ONCE(counter->high); + unsigned long toptier_cap, total_cap; + + if (high =3D=3D PAGE_COUNTER_MAX) + return PAGE_COUNTER_MAX; + + toptier_cap =3D counter->toptier_capacity; + total_cap =3D counter->total_capacity; + + if (!total_cap) + return PAGE_COUNTER_MAX; + + return mult_frac(high, toptier_cap, total_cap); +} + +unsigned long page_counter_toptier_low(struct page_counter *counter) +{ + unsigned long low =3D READ_ONCE(counter->low); + unsigned long toptier_cap, total_cap; + + if (!low) + return 0; + + toptier_cap =3D counter->toptier_capacity; + total_cap =3D counter->total_capacity; + + if (!total_cap) + return 0; + + return mult_frac(low, toptier_cap, total_cap); +} #endif /* CONFIG_MEMCG || CONFIG_CGROUP_DMEM */ --=20 2.47.3