From nobody Sat Feb 7 08:45:16 2026 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4461D262D0B for ; Fri, 23 Jan 2026 15:01:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769180510; cv=none; b=cOAk3eVhdE+0QIrbtdOkm78/zvTOvOHsMCvOadk3iILvhpONGM3aFy6xfdi1NSUcDOLKJNf/vokMoYSRadlhO8+6kJzm+bllo55y6YPYUg18u+5bdD66v5mrPNZbQwBeUblPgCnZIxfYZkUYSOjJfLdqJsR7WLRRe2/ki7uFUng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769180510; c=relaxed/simple; bh=98z08aP2SJAE58imTYc7ee2Bfn47u8ghp+7FoXEjgls=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SwVPsVfy1JCnov7BSiPTKAkKPnWQ4/lDs+gjDbVGedQcWrTB/baynbmh98dP9dV6ZReQvKkVFVWtLcyl8Koz8Xn2YwSEuuRXE7RFDYeI2b7AaIhxywGYBnl/QIPBBtzK6BTsoKDpz9X9lfMtxQmF5L9QWKNik+9Nk/I/SkxpPVM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=S5MrvUue; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="S5MrvUue" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-2a0ac29fca1so16951455ad.2 for ; Fri, 23 Jan 2026 07:01:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1769180509; x=1769785309; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Set9gO/ypIAbsARkMT+qR5jxMJl3CCIAGoFSSuhdA4A=; b=S5MrvUueE8pha9YI+/L2za/1cXRlhMC/nPJLE1p8OZvpfO58S5/KL77jTL8/dZePBA AIfy/ix/Pf+wdmsW+09zPmOkk3fKCfn6ihW6f0vmIaZKgR2mQVKRlIjCwLVijxyRN2ud VHY4eU3I5PfesashKugjnCjT5ALhV1R2laabBdLoxa9AE7V+GeHTBfb3ZKAjlbs2q+k0 MmkEy/D6Afh2lOsMqRws0q1Csz7CdyWXDQ07k9RyXroXjfKJ71Du7I/P1ivTA5k2gwnH e5U/a5iLLutD+W4fTzEFSA/3hn+O/ueDjgANs9OhmXqvr4DK5uGM+AbO1nSsUds7iKcY 1j1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769180509; x=1769785309; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Set9gO/ypIAbsARkMT+qR5jxMJl3CCIAGoFSSuhdA4A=; b=Atnu3AWC245egrMMO9ymJzbE4aH+TACGYlHKRfZ4IMPBkwH2UH4+RaYPd7+nLvzd83 VJe5EuEz+0+sgdIg13Fxy2vLlD2WQZ6VzH/JoRqLTtQizMqIRCQe9LR54arOm0n+JJF/ AS0eyN/JuVB4ollmKVPRlsYcSysP/JpWmxP6yHzl35p9OEjN4l8YeeOz+K+QdPCrrRHF CtuTGlktAfMnrlmrecPYLoJx7qQmZnxLD8awJIkc1uJZmhAOf270ZrPXXO2KTVHEWRTY puTQSqe8gbppgsEQXUP0qfCdpqVzu/+6rSaAR/8aCOVCYBdK6MRP9ryLgKIMbtJTZKU6 bCRQ== X-Forwarded-Encrypted: i=1; AJvYcCW7ZVdA70oCVri5hv9OQ6b0TtV83inLTjqBR+3VaTkqLxjxSC+OgFRCLIA0tv24s8QzKE9HLsxZ9uYDBRI=@vger.kernel.org X-Gm-Message-State: AOJu0YxMSkp8pjZDjySVKxHX7S98+RomYDfqAvcNiVliQmmM/KtBzi54 EIYhWbhmK91143QKP2HlY2WTBOJvHxRqBoAQPHXBfglH7QguDY/PTQ1Notnisv7F X-Gm-Gg: AZuq6aIOXB5Xhhf4qv92w+YnRe5JOpPtoqXDG29SvsqL/Oo9+54849LYhoosvDpiw+A 1ooRfcDJ6abpsvNGcIU+ab6phcOCJIjwJTHcCS5vvWfjNpsaJfH4DKD2ZUgp+OyF4Lum7wdA7uj pv0mxww69eAvUq/42EWE2lqT9WhDGzC8erdUD8yXDblH+nesB4E2GqFzogXUS6SYLkyyQhZcBby GQbEPO82/Bs5BhJza0arBe976+YzAWSY3m2NK2mpqtc3QsEG9zHPqTnIxVFo5Xoe9gFT79MuATE mycpSLxGmrUxzqteRxbq5dzUjcP10rWOko6eLVrXctWH4P2/2Q68Te4BJVPf4ENDDZpsyTzkevh vtz2kk6Ad8Cbz0RYlfWL798c+0sUIECHpsq+BDJBgdw4Ty9LFF5FyZCy5tNFUVqGC1WMrWbNwy1 4SNassOWOQ80VRcjUq/4gLWZYrFfE= X-Received: by 2002:a17:902:f647:b0:295:86a1:5008 with SMTP id d9443c01a7336-2a7fe62504dmr26429375ad.38.1769180506366; Fri, 23 Jan 2026 07:01:46 -0800 (PST) Received: from NV-J4GCB44.nvidia.com ([103.74.125.162]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a802fae9b5sm22297685ad.80.2026.01.23.07.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jan 2026 07:01:45 -0800 (PST) From: Jianyue Wu To: akpm@linux-foundation.org Cc: shakeel.butt@linux.dev, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, inwardvessel@gmail.com, Jianyue Wu Subject: [PATCH v4 1/1] mm: optimize stat output for 11% sys time reduce Date: Fri, 23 Jan 2026 23:01:08 +0800 Message-ID: <20260123150108.43443-2-wujianyue000@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260123150108.43443-1-wujianyue000@gmail.com> References: <20260122114242.72139-1-wujianyue000@gmail.com> <20260123150108.43443-1-wujianyue000@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Replace seq_buf_printf() calls in memcg stat formatting with a lightweight helper function and replace seq_printf() in numa stat output with direct seq operations to avoid printf format parsing overhead. Key changes: - Add memcg_seq_buf_print_stat() helper that embeds separator and newline directly in the number buffer to reduce function calls - Replace seq_buf_printf() in memcg_stat_format() with memcg_seq_buf_print_stat() - Replace seq_printf() in memory_numa_stat_show() and memcg_numa_stat_show() with seq_puts() and seq_put_decimal_ull() - Update memory_stat_format() and related v1 stats formatting Performance improvement (1M reads of memory.stat + memory.numa_stat): - Before: real 0m9.663s, user 0m4.840s, sys 0m4.823s - After: real 0m8.909s, user 0m4.661s, sys 0m4.247s - Result: ~11% sys time reduction Test script: for ((i=3D1; i<=3D1000000; i++)); do : > /dev/null < /sys/fs/cgroup/memory.stat : > /dev/null < /sys/fs/cgroup/memory.numa_stat done Suggested-by: JP Kobryn Acked-by: Shakeel Butt Tested-by: JP Kobryn Signed-off-by: Jianyue Wu --- mm/memcontrol-v1.c | 84 ++++++++++++++++++++++++------------------- mm/memcontrol-v1.h | 4 +++ mm/memcontrol.c | 88 +++++++++++++++++++++++++++++++++++++--------- 3 files changed, 123 insertions(+), 53 deletions(-) diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c index 6eed14bff742..5efea38da69e 100644 --- a/mm/memcontrol-v1.c +++ b/mm/memcontrol-v1.c @@ -10,7 +10,7 @@ #include #include #include -#include +#include =20 #include "internal.h" #include "swap.h" @@ -1794,26 +1794,39 @@ static int memcg_numa_stat_show(struct seq_file *m,= void *v) =20 mem_cgroup_flush_stats(memcg); =20 + /* + * Output format: "stat_name=3Dvalue N0=3Dvalue0 N1=3Dvalue1 ...\n" + * Use seq_puts and seq_put_decimal_ull to avoid printf format + * parsing overhead in this hot path. + */ for (stat =3D stats; stat < stats + ARRAY_SIZE(stats); stat++) { - seq_printf(m, "%s=3D%lu", stat->name, - mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, - false)); - for_each_node_state(nid, N_MEMORY) - seq_printf(m, " N%d=3D%lu", nid, - mem_cgroup_node_nr_lru_pages(memcg, nid, - stat->lru_mask, false)); + seq_puts(m, stat->name); + seq_put_decimal_ull(m, "=3D", + mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, + false)); + for_each_node_state(nid, N_MEMORY) { + seq_put_decimal_ull(m, " N", nid); + seq_put_decimal_ull(m, "=3D", + mem_cgroup_node_nr_lru_pages(memcg, nid, + stat->lru_mask, + false)); + } seq_putc(m, '\n'); } =20 for (stat =3D stats; stat < stats + ARRAY_SIZE(stats); stat++) { - - seq_printf(m, "hierarchical_%s=3D%lu", stat->name, - mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, - true)); - for_each_node_state(nid, N_MEMORY) - seq_printf(m, " N%d=3D%lu", nid, - mem_cgroup_node_nr_lru_pages(memcg, nid, - stat->lru_mask, true)); + seq_puts(m, "hierarchical_"); + seq_puts(m, stat->name); + seq_put_decimal_ull(m, "=3D", + mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, + true)); + for_each_node_state(nid, N_MEMORY) { + seq_put_decimal_ull(m, " N", nid); + seq_put_decimal_ull(m, "=3D", + mem_cgroup_node_nr_lru_pages(memcg, nid, + stat->lru_mask, + true)); + } seq_putc(m, '\n'); } =20 @@ -1879,17 +1892,17 @@ void memcg1_stat_format(struct mem_cgroup *memcg, s= truct seq_buf *s) unsigned long nr; =20 nr =3D memcg_page_state_local_output(memcg, memcg1_stats[i]); - seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr); + memcg_seq_buf_print_stat(s, NULL, memcg1_stat_names[i], ' ', (u64)nr); } =20 for (i =3D 0; i < ARRAY_SIZE(memcg1_events); i++) - seq_buf_printf(s, "%s %lu\n", vm_event_name(memcg1_events[i]), - memcg_events_local(memcg, memcg1_events[i])); + memcg_seq_buf_print_stat(s, NULL, vm_event_name(memcg1_events[i]), + ' ', memcg_events_local(memcg, memcg1_events[i])); =20 for (i =3D 0; i < NR_LRU_LISTS; i++) - seq_buf_printf(s, "%s %lu\n", lru_list_name(i), - memcg_page_state_local(memcg, NR_LRU_BASE + i) * - PAGE_SIZE); + memcg_seq_buf_print_stat(s, NULL, lru_list_name(i), ' ', + memcg_page_state_local(memcg, NR_LRU_BASE + i) * + PAGE_SIZE); =20 /* Hierarchical information */ memory =3D memsw =3D PAGE_COUNTER_MAX; @@ -1897,28 +1910,27 @@ void memcg1_stat_format(struct mem_cgroup *memcg, s= truct seq_buf *s) memory =3D min(memory, READ_ONCE(mi->memory.max)); memsw =3D min(memsw, READ_ONCE(mi->memsw.max)); } - seq_buf_printf(s, "hierarchical_memory_limit %llu\n", - (u64)memory * PAGE_SIZE); - seq_buf_printf(s, "hierarchical_memsw_limit %llu\n", - (u64)memsw * PAGE_SIZE); + memcg_seq_buf_print_stat(s, NULL, "hierarchical_memory_limit", ' ', + (u64)memory * PAGE_SIZE); + memcg_seq_buf_print_stat(s, NULL, "hierarchical_memsw_limit", ' ', + (u64)memsw * PAGE_SIZE); =20 for (i =3D 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; =20 nr =3D memcg_page_state_output(memcg, memcg1_stats[i]); - seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i], - (u64)nr); + memcg_seq_buf_print_stat(s, "total_", memcg1_stat_names[i], ' ', + (u64)nr); } =20 for (i =3D 0; i < ARRAY_SIZE(memcg1_events); i++) - seq_buf_printf(s, "total_%s %llu\n", - vm_event_name(memcg1_events[i]), - (u64)memcg_events(memcg, memcg1_events[i])); + memcg_seq_buf_print_stat(s, "total_", vm_event_name(memcg1_events[i]), + ' ', (u64)memcg_events(memcg, memcg1_events[i])); =20 for (i =3D 0; i < NR_LRU_LISTS; i++) - seq_buf_printf(s, "total_%s %llu\n", lru_list_name(i), - (u64)memcg_page_state(memcg, NR_LRU_BASE + i) * - PAGE_SIZE); + memcg_seq_buf_print_stat(s, "total_", lru_list_name(i), ' ', + (u64)memcg_page_state(memcg, NR_LRU_BASE + i) * + PAGE_SIZE); =20 #ifdef CONFIG_DEBUG_VM { @@ -1933,8 +1945,8 @@ void memcg1_stat_format(struct mem_cgroup *memcg, str= uct seq_buf *s) anon_cost +=3D mz->lruvec.anon_cost; file_cost +=3D mz->lruvec.file_cost; } - seq_buf_printf(s, "anon_cost %lu\n", anon_cost); - seq_buf_printf(s, "file_cost %lu\n", file_cost); + memcg_seq_buf_print_stat(s, NULL, "anon_cost", ' ', (u64)anon_cost); + memcg_seq_buf_print_stat(s, NULL, "file_cost", ' ', (u64)file_cost); } #endif } diff --git a/mm/memcontrol-v1.h b/mm/memcontrol-v1.h index 6358464bb416..b1015cbe858f 100644 --- a/mm/memcontrol-v1.h +++ b/mm/memcontrol-v1.h @@ -4,6 +4,7 @@ #define __MM_MEMCONTROL_V1_H =20 #include +#include =20 /* Cgroup v1 and v2 common declarations */ =20 @@ -33,6 +34,9 @@ int memory_stat_show(struct seq_file *m, void *v); void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n); struct mem_cgroup *mem_cgroup_id_get_online(struct mem_cgroup *memcg); =20 +void memcg_seq_buf_print_stat(struct seq_buf *s, const char *prefix, + const char *name, char sep, u64 val); + /* Cgroup v1-specific declarations */ #ifdef CONFIG_MEMCG_V1 =20 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 86f43b7e5f71..136c08462cd1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -1460,6 +1461,53 @@ static bool memcg_accounts_hugetlb(void) } #endif /* CONFIG_HUGETLB_PAGE */ =20 +/* Max 2^64 - 1 =3D 18446744073709551615 (20 digits) */ +#define MEMCG_DEC_U64_MAX_LEN 20 + +/** + * memcg_seq_buf_print_stat - Write a name-value pair to a seq_buf with ne= wline + * @s: The seq_buf to write to + * @prefix: Optional prefix string (can be NULL or "") + * @name: The name string to write + * @sep: Separator character between name and value (typically ' ' or '=3D= ') + * @val: The u64 value to write + * + * This helper efficiently formats and writes "\= n" + * to a seq_buf. It manually converts the value to a string using num_to_s= tr + * and embeds the separator and newline in the buffer to minimize function + * calls for better performance. + * + * The function checks for overflow at each step and returns early if any + * operation would cause the buffer to overflow. + * + * Example: memcg_seq_buf_print_stat(s, "total_", "cache", ' ', 1048576) + * Output: "total_cache 1048576\n" + */ +void memcg_seq_buf_print_stat(struct seq_buf *s, const char *prefix, + const char *name, char sep, u64 val) +{ + char num_buf[MEMCG_DEC_U64_MAX_LEN + 2]; /* +2 for separator and newline= */ + int num_len; + + /* Embed separator at the beginning */ + num_buf[0] =3D sep; + + /* Convert number starting at offset 1 */ + num_len =3D num_to_str(num_buf + 1, sizeof(num_buf) - 2, val, 0); + if (num_len <=3D 0) + return; + + /* Embed newline at the end */ + num_buf[num_len + 1] =3D '\n'; + + if (prefix && *prefix && seq_buf_puts(s, prefix)) + return; + if (seq_buf_puts(s, name)) + return; + /* Output separator, value, and newline in one call */ + seq_buf_putmem(s, num_buf, num_len + 2); +} + static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) { int i; @@ -1485,26 +1533,26 @@ static void memcg_stat_format(struct mem_cgroup *me= mcg, struct seq_buf *s) continue; #endif size =3D memcg_page_state_output(memcg, memory_stats[i].idx); - seq_buf_printf(s, "%s %llu\n", memory_stats[i].name, size); + memcg_seq_buf_print_stat(s, NULL, memory_stats[i].name, ' ', size); =20 if (unlikely(memory_stats[i].idx =3D=3D NR_SLAB_UNRECLAIMABLE_B)) { size +=3D memcg_page_state_output(memcg, NR_SLAB_RECLAIMABLE_B); - seq_buf_printf(s, "slab %llu\n", size); + memcg_seq_buf_print_stat(s, NULL, "slab", ' ', size); } } =20 /* Accumulated memory events */ - seq_buf_printf(s, "pgscan %lu\n", - memcg_events(memcg, PGSCAN_KSWAPD) + - memcg_events(memcg, PGSCAN_DIRECT) + - memcg_events(memcg, PGSCAN_PROACTIVE) + - memcg_events(memcg, PGSCAN_KHUGEPAGED)); - seq_buf_printf(s, "pgsteal %lu\n", - memcg_events(memcg, PGSTEAL_KSWAPD) + - memcg_events(memcg, PGSTEAL_DIRECT) + - memcg_events(memcg, PGSTEAL_PROACTIVE) + - memcg_events(memcg, PGSTEAL_KHUGEPAGED)); + memcg_seq_buf_print_stat(s, NULL, "pgscan", ' ', + memcg_events(memcg, PGSCAN_KSWAPD) + + memcg_events(memcg, PGSCAN_DIRECT) + + memcg_events(memcg, PGSCAN_PROACTIVE) + + memcg_events(memcg, PGSCAN_KHUGEPAGED)); + memcg_seq_buf_print_stat(s, NULL, "pgsteal", ' ', + memcg_events(memcg, PGSTEAL_KSWAPD) + + memcg_events(memcg, PGSTEAL_DIRECT) + + memcg_events(memcg, PGSTEAL_PROACTIVE) + + memcg_events(memcg, PGSTEAL_KHUGEPAGED)); =20 for (i =3D 0; i < ARRAY_SIZE(memcg_vm_event_stat); i++) { #ifdef CONFIG_MEMCG_V1 @@ -1512,9 +1560,9 @@ static void memcg_stat_format(struct mem_cgroup *memc= g, struct seq_buf *s) memcg_vm_event_stat[i] =3D=3D PGPGOUT) continue; #endif - seq_buf_printf(s, "%s %lu\n", - vm_event_name(memcg_vm_event_stat[i]), - memcg_events(memcg, memcg_vm_event_stat[i])); + memcg_seq_buf_print_stat(s, NULL, + vm_event_name(memcg_vm_event_stat[i]), ' ', + memcg_events(memcg, memcg_vm_event_stat[i])); } } =20 @@ -4544,7 +4592,12 @@ static int memory_numa_stat_show(struct seq_file *m,= void *v) if (memory_stats[i].idx >=3D NR_VM_NODE_STAT_ITEMS) continue; =20 - seq_printf(m, "%s", memory_stats[i].name); + /* + * Output format: "stat_name N0=3Dvalue0 N1=3Dvalue1 ...\n" + * Use seq_puts and seq_put_decimal_ull to avoid printf + * format parsing overhead in this hot path. + */ + seq_puts(m, memory_stats[i].name); for_each_node_state(nid, N_MEMORY) { u64 size; struct lruvec *lruvec; @@ -4552,7 +4605,8 @@ static int memory_numa_stat_show(struct seq_file *m, = void *v) lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(nid)); size =3D lruvec_page_state_output(lruvec, memory_stats[i].idx); - seq_printf(m, " N%d=3D%llu", nid, size); + seq_put_decimal_ull(m, " N", nid); + seq_put_decimal_ull(m, "=3D", size); } seq_putc(m, '\n'); } --=20 2.43.0