From nobody Thu Apr 2 15:37:42 2026 Received: from mail-oi1-f169.google.com (mail-oi1-f169.google.com [209.85.167.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4648C29B789 for ; Fri, 27 Mar 2026 19:19:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774639179; cv=none; b=NRp2/dVpelfdNWWWfNwL0o0PbwhYJl5josPZiiQfNlcIjDZ3NSvbaSytRS9qU8YRXqpva92lm4Gl3pveXKapRajRJ8wxooPM/akUnoaPu0VKsu093ij8FJ9v3JQG1xHh6dqXGACb0ls1hhX2A7qneSgpoQGTyCJ7bYP7zEg+OI8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774639179; c=relaxed/simple; bh=mHiZ1XBdo4Ah4SJDDYH+/xbdYWRu1m/OZcr1uMyQ5Bg=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=NWbZ6jw4Mi93HULB05BJsvfmp8Yb2pgYckUa2/HYBm4i1KqdvBg2fibazBWEujhXwS6KDYSjKanrstWNMwGAqiU6hTl7HdV/JJP/AOk9aD2xMtab42nR30PCFIudEkOjcgXUc0gjaCBGm/bfc3XSChblLnWU89lXLKW/RaemRLM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hpzc+MGj; arc=none smtp.client-ip=209.85.167.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hpzc+MGj" Received: by mail-oi1-f169.google.com with SMTP id 5614622812f47-46a9ae3f857so144079b6e.0 for ; Fri, 27 Mar 2026 12:19:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1774639177; x=1775243977; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=g36eDtHxaYoJojvs9L9tY/Cu0/ANeoyQ3rXoQcxOHvE=; b=hpzc+MGjaXS9gyOd1MzCgbdbOEOp5azwCY6SoQQY5yrTLrcGwdmoWO135Ce5ACh6V8 /YSDt3yLKHPZhepexbeSOybrW3SXxJvrW9NIny4+WrokVLnP1ZD12TvGnMNz6TB60sCW P17/UwKkFJbqoNGHbSvg4IKM3xTI5b5vs4/kGYjRICisEgx5Q2qT2nLpOYpe5QTuVE6n FQFrQ6b3MsE6Nu52RDokmbSjoubm9A/Z94NFuV1EpjkSSAu/4D6ZM+RrA0O7UJcPzqeV TaW/cQ4PLvhsZl4iVYuWWDtAGmJKZ7Kbpx2/gTH1hSpHtW7pilRbQrGSar1TkNXgZhrR Ap3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774639177; x=1775243977; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=g36eDtHxaYoJojvs9L9tY/Cu0/ANeoyQ3rXoQcxOHvE=; b=nZ17gzlPUlT99pwiXi/iTDi0qIpxFeh9aE6T8DFqGrfsxqTcIPMcQ9p7bwKE5CXO+H QYq92sK3cToFSOnqtEhBvXPdhotmYHC83CEA54Lze1TxQmKBqhtkmrawve27mYUsgND7 CR+JQHLshkM5uzrn5uMQ1TeuXCJETO2IYFqq1F51330bx2eGHC5/Y06HJBNCiq3gwYwa K1hLTLgz+O090YmfQ4QT1yfM0RbGtEt4fhvQeqc4pgr6a7YhPYxVhStfrE8y2Cdis4/j 0ql2jtzbH6W9ZIQh/7n2XczJ+1IPgINkMuUE6AFbpP4L0PZ3GFj8Yx4n/eK0q0JUYNmF aTDg== X-Forwarded-Encrypted: i=1; AJvYcCVkcg9WZC64QgL6gi8ozfYuY4HM5l2EjYqLEUAw66gxzSO6XMXXXIW7LwIhdLY94W8wvrNxIfvJ6srFOGQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yy0K80Xn68i8fBAebI3Qgi9Za8QebZLCeyZW62TO4EIVXMQhtIA 2I0CRtuxj3V/Ba9Pup0ZpWSR7l4KCtZGaYZtmN3QzC0qekoJ09xV6pq/ X-Gm-Gg: ATEYQzzBVG9O1pzazW6wxQlJaf03Y3Yn4TIBknksWop4VKcSp+I/nQEOAXP8qjBTDBW EGtAI3v59akOEQttGpfVoUQ5baE3gx3u3rRNEOVMa/hE2Zfz3sXAErDQ+3KAHX4C9P1d8iw1KD3 NfDQwuOXiKsyIJFximE9kyOaZLTZg+1X5Gx7wqjRcoRTT6GQOw9yhtG56XD5yjOqtTIKMTahtqq P7xsVUvx2AxRk6GMUGFquOJvIRxMhQVFPns0IaPsPzs8sWo4k3jLPWr7kICD+iVcjmHZLRU9xlb eiFN+6fd3AkWzyrIxwfS0PKw4soQNWyUi2smlb6xKiksmPdesQv7XrcQ6TIrVdTnuaAm+/rRoyb zzZmMBussRQvL+9l7szFMvfXdKfQYJIoZo+57a9HCUOHaY6dsB1NIyqefXDNuI60rVuULTHp31F yLzX87mezqjaf2Z26ZyQ== X-Received: by 2002:a05:6808:894c:b0:467:2be4:9e39 with SMTP id 5614622812f47-46a8a5a37a8mr1494347b6e.39.1774639177209; Fri, 27 Mar 2026 12:19:37 -0700 (PDT) Received: from localhost ([2a03:2880:10ff::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-41d04cf9ceesm138877fac.15.2026.03.27.12.19.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Mar 2026 12:19:36 -0700 (PDT) From: Joshua Hahn To: Johannes Weiner , Andrew Morton Cc: Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , David Hildenbrand , Lorenzo Stoakes , Vlastimil Babka , Dennis Zhou , Tejun Heo , Christoph Lameter , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH] mm/percpu, memcontrol: Per-memcg-lruvec percpu accounting Date: Fri, 27 Mar 2026 12:19:35 -0700 Message-ID: <20260327191936.1980054-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert MEMCG_PERCPU_B from a memcg_stat_item to a memcg_node_stat_item to give visibility into per-node breakdowns for percpu allocations and turn it into NR_PERCPU_B. Because percpu memory is accounted at a sub-PAGE_SIZE level, we must account node level statistics (accounted in PAGE_SIZE units) and memcg-lruvec statistics separately. Account node statistics when the pcpu pages are allocated, and account memcg-lruvec statistics when pcpu objects are handed out. To do account these separately, expose mod_memcg_lruvec_state to be used outside of memcontrol. One functional change is that we do not account the 8 byte objcg pointer per-memcg-lruvec. Since the objcg membership is tracked per-memcg and not percpu, there is no appropriate lruvec to charge this memory to (see pcpu_obj_full_size). Instead of adding additional mechanisms to detect which lruvec the 8 byte pointer belongs to, let's just simplify and account the pcpu objects' size. Limit-checking is still done with the additional 8 bytes. Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 4 +++- include/linux/mmzone.h | 4 +++- mm/memcontrol.c | 12 ++++++------ mm/percpu-vm.c | 14 ++++++++++++-- mm/percpu.c | 24 ++++++++++++++++++++---- mm/vmstat.c | 1 + 6 files changed, 45 insertions(+), 14 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 086158969529..96dae769c60d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -34,7 +34,6 @@ struct kmem_cache; enum memcg_stat_item { MEMCG_SWAP =3D NR_VM_NODE_STAT_ITEMS, MEMCG_SOCK, - MEMCG_PERCPU_B, MEMCG_KMEM, MEMCG_ZSWAP_B, MEMCG_ZSWAPPED, @@ -909,6 +908,9 @@ struct mem_cgroup *mem_cgroup_get_oom_group(struct task= _struct *victim, struct mem_cgroup *oom_domain); void mem_cgroup_print_oom_group(struct mem_cgroup *memcg); =20 +void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val); + /* idx can be of type enum memcg_stat_item or node_stat_item */ void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, int val); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 7bd0134c241c..e38d8fe8552b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -328,6 +328,7 @@ enum node_stat_item { #endif NR_BALLOON_PAGES, NR_KERNEL_FILE_PAGES, + NR_PERCPU_B, NR_VM_NODE_STAT_ITEMS }; =20 @@ -365,7 +366,8 @@ static __always_inline bool vmstat_item_in_bytes(int id= x) * byte-precise. */ return (idx =3D=3D NR_SLAB_RECLAIMABLE_B || - idx =3D=3D NR_SLAB_UNRECLAIMABLE_B); + idx =3D=3D NR_SLAB_UNRECLAIMABLE_B || + idx =3D=3D NR_PERCPU_B); } =20 /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a47fb68dd65f..b320b6a42696 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -377,6 +377,7 @@ static const unsigned int memcg_node_stat_items[] =3D { NR_UNEVICTABLE, NR_SLAB_RECLAIMABLE_B, NR_SLAB_UNRECLAIMABLE_B, + NR_PERCPU_B, WORKINGSET_REFAULT_ANON, WORKINGSET_REFAULT_FILE, WORKINGSET_ACTIVATE_ANON, @@ -428,7 +429,6 @@ static const unsigned int memcg_node_stat_items[] =3D { static const unsigned int memcg_stat_items[] =3D { MEMCG_SWAP, MEMCG_SOCK, - MEMCG_PERCPU_B, MEMCG_KMEM, MEMCG_ZSWAP_B, MEMCG_ZSWAPPED, @@ -920,9 +920,8 @@ static void __mod_memcg_lruvec_state(struct mem_cgroup_= per_node *pn, put_cpu(); } =20 -static void mod_memcg_lruvec_state(struct lruvec *lruvec, - enum node_stat_item idx, - int val) +void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val) { struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); struct mem_cgroup_per_node *pn; @@ -936,6 +935,7 @@ static void mod_memcg_lruvec_state(struct lruvec *lruve= c, =20 get_non_dying_memcg_end(); } +EXPORT_SYMBOL(mod_memcg_lruvec_state); =20 /** * mod_lruvec_state - update lruvec memory statistics @@ -1535,7 +1535,7 @@ static const struct memory_stat memory_stats[] =3D { { "kernel_stack", NR_KERNEL_STACK_KB }, { "pagetables", NR_PAGETABLE }, { "sec_pagetables", NR_SECONDARY_PAGETABLE }, - { "percpu", MEMCG_PERCPU_B }, + { "percpu", NR_PERCPU_B }, { "sock", MEMCG_SOCK }, { "vmalloc", NR_VMALLOC }, { "shmem", NR_SHMEM }, @@ -1597,7 +1597,7 @@ static const struct memory_stat memory_stats[] =3D { static int memcg_page_state_unit(int item) { switch (item) { - case MEMCG_PERCPU_B: + case NR_PERCPU_B: case MEMCG_ZSWAP_B: case NR_SLAB_RECLAIMABLE_B: case NR_SLAB_UNRECLAIMABLE_B: diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index 4f5937090590..e36b639f521d 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -55,7 +55,8 @@ static void pcpu_free_pages(struct pcpu_chunk *chunk, struct page **pages, int page_start, int page_end) { unsigned int cpu; - int i; + int nr_pages =3D page_end - page_start; + int i, nid; =20 for_each_possible_cpu(cpu) { for (i =3D page_start; i < page_end; i++) { @@ -65,6 +66,10 @@ static void pcpu_free_pages(struct pcpu_chunk *chunk, __free_page(page); } } + + for_each_node(nid) + mod_node_page_state(NODE_DATA(nid), NR_PERCPU_B, + -1L * nr_pages * nr_cpus_node(nid) * PAGE_SIZE); } =20 /** @@ -84,7 +89,8 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk, gfp_t gfp) { unsigned int cpu, tcpu; - int i; + int nr_pages =3D page_end - page_start; + int i, nid; =20 gfp |=3D __GFP_HIGHMEM; =20 @@ -97,6 +103,10 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk, goto err; } } + + for_each_node(nid) + mod_node_page_state(NODE_DATA(nid), NR_PERCPU_B, + nr_pages * nr_cpus_node(nid) * PAGE_SIZE); return 0; =20 err: diff --git a/mm/percpu.c b/mm/percpu.c index b0676b8054ed..4ad3b9739eb9 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1632,6 +1632,24 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, g= fp_t gfp, return true; } =20 +static void pcpu_mod_memcg_lruvec(struct obj_cgroup *objcg, int charge) +{ + struct mem_cgroup *memcg; + int nid; + + memcg =3D obj_cgroup_memcg(objcg); + for_each_node(nid) { + struct lruvec *lruvec; + unsigned int nr_cpus =3D nr_cpus_node(nid); + + if (!nr_cpus) + continue; + + lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(nid)); + mod_memcg_lruvec_state(lruvec, NR_PERCPU_B, nr_cpus * charge); + } +} + static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, struct pcpu_chunk *chunk, int off, size_t size) @@ -1644,8 +1662,7 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgr= oup *objcg, chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup =3D objcg; =20 rcu_read_lock(); - mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, - pcpu_obj_full_size(size)); + pcpu_mod_memcg_lruvec(objcg, size); rcu_read_unlock(); } else { obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); @@ -1667,8 +1684,7 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *c= hunk, int off, size_t size) obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); =20 rcu_read_lock(); - mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, - -pcpu_obj_full_size(size)); + pcpu_mod_memcg_lruvec(objcg, -size); rcu_read_unlock(); =20 obj_cgroup_put(objcg); diff --git a/mm/vmstat.c b/mm/vmstat.c index b33097ab9bc8..d73c3355be71 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1296,6 +1296,7 @@ const char * const vmstat_text[] =3D { #endif [I(NR_BALLOON_PAGES)] =3D "nr_balloon_pages", [I(NR_KERNEL_FILE_PAGES)] =3D "nr_kernel_file_pages", + [I(NR_PERCPU_B)] =3D "nr_percpu", #undef I =20 /* system-wide enum vm_stat_item counters */ --=20 2.52.0