From nobody Tue Apr 7 18:51:21 2026 Received: from mail-oa1-f52.google.com (mail-oa1-f52.google.com [209.85.160.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89B5538237E for ; Wed, 11 Mar 2026 19:52:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258730; cv=none; b=QTJVmawpxFPYMvR2/egMwQ6PKv2yDKD1cSIJDnPRCN9P6mTjO+/aRzWjJYCe3C9IKQFCK+zcSr5MQhDorCC/5wh9Yzu6GV3cOOjOmP6QfKYiRqSGvdOfGRC32pKbrjFNqEqd9tZ2Kh8DtTSxGlPTEyptKEcswAIDuG52DiUhazc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258730; c=relaxed/simple; bh=ltWZnUnXeRVqYVTEBtTy0eYCoDdqibyFZCtqJ7qUlq8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rIwnPe3uGMfqUsyXNOrdh9Z/gfEmmyXfQcOyGir5VXdo+f3ckjXhD/DFBGSnq39Ja+pl0zaySAwi6Tvw4gwpNqO2axMJQYQkbDKZBNBD0u3dS1CJvY7unZ+NhEdl8sFds014UvN+vSzsNDxSvrDpdHfyWyjopVtzxjDBcE3J6lo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=foAkeCW2; arc=none smtp.client-ip=209.85.160.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="foAkeCW2" Received: by mail-oa1-f52.google.com with SMTP id 586e51a60fabf-41708f6c3feso189564fac.3 for ; Wed, 11 Mar 2026 12:52:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258724; x=1773863524; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8u5o092L3ULCpmoipnWOnXiIPpIcwoMHmpMXH/cA6Oo=; b=foAkeCW2rLG48Jt1YjeSANAiyq3bdHTJ6/zTw+Sp677+qvV0xiPJ9PKZgczYzHi8NC e6ZQtHTwqyOaL4gEXamXx/JzgULF/0zGxwKfmqo2iEQ48VEo4hvVzUnV2VnFSAvEX47+ 4eBiEIP00POoi2ATzv4xQDaVA3s+CdtbqqEBspnJM5se/YZ60Be158amBJHcE38W2/UI ZU4Rs/QbF5TIlNTeg/ugHB9ShVS5lUO4mDK4zoSCwYvXg2sgfQ44825xzs9rLrrh6pqm igf02XDUUHdA22HqjVXojxGx0mN99w+tu0gSD8RwQX+Nu/ld7HvLsbb7DUP8V23ZcDi6 FZWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258724; x=1773863524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8u5o092L3ULCpmoipnWOnXiIPpIcwoMHmpMXH/cA6Oo=; b=NmNgTyh53kt1bTe5ZjipvqnqEYrxsG29bASfVOfkxyC5Qdo+9DABK6zbCkzHKsaq3T CP8edMk2WOGWCXJNx9YhCldAG37t9fetx4GJ05/NRHZ57IIcoqR65XnvhC5UJoaNEdyC JnMmHexCt9h1WqHq9M2WyCOP+qUASJfaVL+v++Mo9ZnF3RxsqUQO20i+ouDYg1x9bQXg 5NUWg/6NYTvIGwnr+rglphHxswulT82AHNHNReJhmx+1ey8BbBy7JEDi+hOsITxJ5qXa ZZ1JwNsifoauAoU89oRWELe4i+ORA6lrxmbVJSUF7hBvJtHxWVoZJok0ieYZKMlxf9zG RdjA== X-Forwarded-Encrypted: i=1; AJvYcCU9MOzwe9qebdp/Mi1eedjprj8Alt1ST2iqbi+Y8ygAJZF+O6FnJXwXfDTgx2KDUmBU8dJPWOM5ok87w0A=@vger.kernel.org X-Gm-Message-State: AOJu0YwXxq765xucR7CdHXeWZ2mIksYot1f8A7VialfbGgsmAqMAo892 rqvVGHJJivJ/64L9j0YePY3/zarHbRvTBABdOmkgyE8j5hGk6bJaa1t8 X-Gm-Gg: ATEYQzxUKCzo+7z0b//HIcHK80A3eijeHM0ogSyjaTE3YlXoS6knO5+mOEZrB9oYW/a iGtFBT9UITbI+s5CCTeaMphjkhTV8dziBLfFLw1TQjfJQ4bUaapupuofX30CkxriHYdn679hLUM VVv50+Dn85dqD0rbOjHthh3PgtjxKXU0Yuy+FQyY7lgKZ8+SDi4rYWvYkPewy0975JiMriZ+hlr d8VJyDKvKqthhD1xSbUlqh52BrkMRMETXMctSdihhmGTJkgFz0CPAkLJiefNuwYdpQNc3EixeHW 4c4S7wZ1QGX0xZ5d7aVT5+fWPj+I8SgSGloJOoNt5HCJT7e9eAWDvRACFMtoTxaReP3FeFEcSZr LyE+AI6wUBvHzPX7kDg+tMvf3w7RF3mL4kGZ7QuJRNui7oF3iXWusxtftcKZqqNyCZGo7QAj4y3 hAzYNFhSUp3wmwwkdIiF1PWQ== X-Received: by 2002:a05:6870:169d:b0:3fa:eb8:4c92 with SMTP id 586e51a60fabf-4177c9c1dd0mr2187825fac.50.1773258724198; Wed, 11 Mar 2026 12:52:04 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:46::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4177e6e82cdsm3116391fac.18.2026.03.11.12.52.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:03 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 07/11] mm/zsmalloc, zswap: Handle objcg charging and lifetime in zsmalloc Date: Wed, 11 Mar 2026 12:51:44 -0700 Message-ID: <20260311195153.4013476-8-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that zswap_entries do not directly track obj_cgroups of the entries, handle the lifetime management and charging of these entries into the zsmalloc layer. One functional change is that zswap entries are now no longer accounted by the size of the compressed object, but by the size of the size_class slot they occupy. This brings charging one step closer to an accurate representation of the memory consumed in the zpdesc; even if a compressed object doesn't consume the entirety of a obj slot, we should account the entirety of the compressed object slot that the object makes unusable. While at it, also remove an unnecessary newline in obj_free. Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 10 ------ mm/memcontrol.c | 54 ++----------------------------- mm/zsmalloc.c | 65 ++++++++++++++++++++++++++++++++++++-- mm/zswap.c | 8 ----- 4 files changed, 66 insertions(+), 71 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0652db4ff2d5..701d9ab6fef1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1851,22 +1851,12 @@ static inline bool memcg_is_dying(struct mem_cgroup= *memcg) =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP) bool obj_cgroup_may_zswap(struct obj_cgroup *objcg); -void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size); -void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size); bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg); #else static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) { return true; } -static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, - size_t size) -{ -} -static inline void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, - size_t size) -{ -} static inline bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *m= emcg) { /* if zswap is disabled, do not block pages going to the swapping device = */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a52da3a5e4fd..68139be66a4f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -716,6 +716,7 @@ void mod_memcg_state(struct mem_cgroup *memcg, enum mem= cg_stat_item idx, =20 put_cpu(); } +EXPORT_SYMBOL(mod_memcg_state); =20 #ifdef CONFIG_MEMCG_V1 /* idx can be of type enum memcg_stat_item or node_stat_item. */ @@ -3169,11 +3170,13 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp= _t gfp, size_t size) { return obj_cgroup_charge_account(objcg, gfp, size, NULL, 0); } +EXPORT_SYMBOL(obj_cgroup_charge); =20 void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) { refill_obj_stock(objcg, size, true, 0, NULL, 0); } +EXPORT_SYMBOL(obj_cgroup_uncharge); =20 static inline size_t obj_full_size(struct kmem_cache *s) { @@ -5488,57 +5491,6 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) return ret; } =20 -/** - * obj_cgroup_charge_zswap - charge compression backend memory - * @objcg: the object cgroup - * @size: size of compressed object - * - * This forces the charge after obj_cgroup_may_zswap() allowed - * compression and storage in zswap for this cgroup to go ahead. - */ -void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size) -{ - struct mem_cgroup *memcg; - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return; - - VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC)); - - /* PF_MEMALLOC context, charging must succeed */ - if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) - VM_WARN_ON_ONCE(1); - - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1); - rcu_read_unlock(); -} - -/** - * obj_cgroup_uncharge_zswap - uncharge compression backend memory - * @objcg: the object cgroup - * @size: size of compressed object - * - * Uncharges zswap memory on page in. - */ -void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) -{ - struct mem_cgroup *memcg; - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return; - - obj_cgroup_uncharge(objcg, size); - - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, -size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, -1); - rcu_read_unlock(); -} - bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg) { /* if zswap is disabled, do not block pages going to the swapping device = */ diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a94ca8c26ad9..291194572a09 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1028,6 +1028,59 @@ static bool zspage_empty(struct zspage *zspage) return get_zspage_inuse(zspage) =3D=3D 0; } =20 +#ifdef CONFIG_MEMCG +static void zs_charge_objcg(struct zs_pool *pool, struct obj_cgroup *objcg, + int size) +{ + struct mem_cgroup *memcg; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC)); + WARN_ON_ONCE(!pool->memcg_aware); + + /* PF_MEMALLOC context, charging must succeed */ + if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) + VM_WARN_ON_ONCE(1); + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + mod_memcg_state(memcg, pool->compressed_stat, size); + mod_memcg_state(memcg, pool->uncompressed_stat, 1); + rcu_read_unlock(); +} + +static void zs_uncharge_objcg(struct zs_pool *pool, struct obj_cgroup *obj= cg, + int size) +{ + struct mem_cgroup *memcg; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + WARN_ON_ONCE(!pool->memcg_aware); + + obj_cgroup_uncharge(objcg, size); + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + mod_memcg_state(memcg, pool->compressed_stat, -size); + mod_memcg_state(memcg, pool->uncompressed_stat, -1); + rcu_read_unlock(); +} +#else +static void zs_charge_objcg(struct zs_pool *pool, struct obj_cgroup *objcg, + int size) +{ +} + +static void zs_uncharge_objcg(struct zs_pool *pool, struct obj_cgroup *obj= cg, + int size) +{ +} +#endif + /** * zs_lookup_class_index() - Returns index of the zsmalloc &size_class * that hold objects of the provided size. @@ -1244,6 +1297,8 @@ void zs_obj_write(struct zs_pool *pool, unsigned long= handle, if (objcg) { WARN_ON_ONCE(!pool->memcg_aware); zspage->objcgs[obj_idx] =3D objcg; + obj_cgroup_get(objcg); + zs_charge_objcg(pool, objcg, class->size); } =20 if (!ZsHugePage(zspage)) @@ -1409,17 +1464,23 @@ static void obj_free(int class_size, unsigned long = obj) struct link_free *link; struct zspage *zspage; struct zpdesc *f_zpdesc; + struct zs_pool *pool; unsigned long f_offset; unsigned int f_objidx; void *vaddr; =20 - obj_to_location(obj, &f_zpdesc, &f_objidx); f_offset =3D offset_in_page(class_size * f_objidx); zspage =3D get_zspage(f_zpdesc); + pool =3D zspage->pool; + + if (pool->memcg_aware && zspage->objcgs[f_objidx]) { + struct obj_cgroup *objcg =3D zspage->objcgs[f_objidx]; =20 - if (zspage->pool->memcg_aware) + zs_uncharge_objcg(pool, objcg, class_size); + obj_cgroup_put(objcg); zspage->objcgs[f_objidx] =3D NULL; + } =20 vaddr =3D kmap_local_zpdesc(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); diff --git a/mm/zswap.c b/mm/zswap.c index 436066965413..bca29a6e18f3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -711,10 +711,6 @@ static void zswap_entry_free(struct zswap_entry *entry) zswap_lru_del(&zswap_list_lru, entry, objcg); zs_free(entry->pool->zs_pool, entry->handle); zswap_pool_put(entry->pool); - if (objcg) { - obj_cgroup_uncharge_zswap(objcg, entry->length); - obj_cgroup_put(objcg); - } if (entry->length =3D=3D PAGE_SIZE) atomic_long_dec(&zswap_stored_incompressible_pages); zswap_entry_cache_free(entry); @@ -1437,10 +1433,6 @@ static bool zswap_store_page(struct page *page, * when the entry is removed from the tree. */ zswap_pool_get(pool); - if (objcg) { - obj_cgroup_get(objcg); - obj_cgroup_charge_zswap(objcg, entry->length); - } atomic_long_inc(&zswap_stored_pages); if (entry->length =3D=3D PAGE_SIZE) atomic_long_inc(&zswap_stored_incompressible_pages); --=20 2.52.0