From nobody Tue Apr 7 18:46:54 2026 Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EC0C383C9C for ; Wed, 11 Mar 2026 19:52:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; cv=none; b=CEgiJQjV/d2gvSNxynvc9WJnASz2nDuruvSQUZmbCtj1ww98Mq/gO7jabKPpmNIIEetbAaEapC9J6x04VvQod85TpVVhp6C9MBmpqPfwtV1VfWY++pymjUorTfaWoIqH5xjIMwtLNB8NsyQYaA0C/roMOgMq9ZyZbM4+x5Oh73Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; c=relaxed/simple; bh=bHJG3+YkzZlvrN13LSWBugwv+mqzY0z+0JYsWV64PVA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jo0dyoGb0nSUGV3Kir0jbrEn4aUs4ISJM3LI5nZ6oPqg8ECPuyHDjJLbAhx5hQteqU4lYIjMokbnH3qCtSF0PdYOrdeHerrCsbCe8liaT/N3MNoY7a386kZskhJJRESZNSZ1TifgZYnguHcegc5XgW8DzAn9akBvyw7jfUsXWto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m120IXrF; arc=none smtp.client-ip=209.85.161.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m120IXrF" Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-67bb39a1122so174634eaf.3 for ; Wed, 11 Mar 2026 12:52:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258728; x=1773863528; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lInkB9M3FDEV05WLG9mmxw0Eg6pj3q7yDvK5tck5dXo=; b=m120IXrF6QXv0Gn2wdeZl59k+OAuq7Q9l2OzioGV+FMn7SGoxG/yVf1tPoWjgjUdfs QD6dIS2YRwPq0UhRbTk9qdSqEmMRsVuLs63tDdX+vF/OIxc1yDRYkWJPCaJaHxQKdOmc jXgpl6S9ekomKEDacVw19sr/woa9kbIlKob8uM3ubasgEemWDXNe7ZAL5iI+uJppCZAp A14mEPzBhj63rbn9vjmxylLBwt1QEM+YvIZsb1LSuSaAubzYZ4mx266ZHQdzOiQ24VR2 h+1qETCCYhSeVvjypQ+zQ/sWaCuZX18r1dJoJzALt7C6mQ/OXHwLONrueG2Tsmrn87yl sJyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258728; x=1773863528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=lInkB9M3FDEV05WLG9mmxw0Eg6pj3q7yDvK5tck5dXo=; b=qSht0JUTAzDvOZOfwCFtQ/uzg+QwVz06PYjVLYJm4qPHjJoeehBVU7QBcW9TadMYlf wdRiRvZm5Afgxkx7PHrIkO0SnSu5tEf1zfSAbRrHPfRPp/iPKgxJQmSq1tB21iC+QHVW l63816SdRbRSvhv6OGxdUIOCo/XJi5Eli6kXcS5xECchKAB9w1qft0ALxppwyiTflVSl w4bBQxwrxu+++3bJIchdpa3tnyztuaQIxGnG+kwVoxpaZ0QEd4ivHPoNn7O0gDkZwobS waOH92Ub5PAd9B8s7Uhtf5k/RqpDmy79ZuJOLIFZH020c1OpvLd2+f4j6fyeS/NT0Pva ZaGw== X-Forwarded-Encrypted: i=1; AJvYcCUpeN8aR12pRJSgxQ720c/EO/bF7zBWzWrQSksczis9LdseHsrfYkfCTZS/4bsH2Uq41szK6rY9lcZ8HGg=@vger.kernel.org X-Gm-Message-State: AOJu0YzwAuLxPZlgxdJ8MtqXyB8rWLXTWjQJc81wG59m4UIPIgESBVpM WqouO9plP1il1++HarYpf5O7HHrnTlZhKWP5zkAInq8jL91dd1gdMbxJ X-Gm-Gg: ATEYQzwxhYJAVCtO/LR0TystqQ8wHL4F1UjLYj4VStQFwbSh1WZbjtBYIKGqMb9HLB2 48PBM8C2FU5rj01wmZCM3vtbQTU9QVtXUTV3dlY10oP61Bsw7hiHK4rBKD5KYGm1CzoUEKTs8Rd np6AIMybGY6/1ZQYtie4n0dOgqZ4WsMcHKC92RloE5DsrMYKYaW6W7S782FLat1m55X7C45BDE0 J3NQzZPT9Gael+KqEIJuDCVAf4c6aIoN4j+k60ZEhRk1QEi6IsFl0mzQnPexP987o+cyqU3uObS vYHYgG9yB9qc9hrIP3P/6xoap3KtoKbnE4ZpegSR7ip+ql7OZrSSe8d9Y1Ta/JzbwZIqZp+/QoC xQ3iLLu0AjNsGhP6ff6VtquURBjZyM/Aqn/9ZQrWaUUdqGRekhyWruIUg0hHsjhQWHvl2wCpchv wkiUyzO04QkqK7T9DDBBIF0Q== X-Received: by 2002:a05:6820:1629:b0:67b:b627:349b with SMTP id 006d021491bc7-67bc8a92e0amr2511581eaf.64.1773258728114; Wed, 11 Mar 2026 12:52:08 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:5e::]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-67bc9118bc7sm2041227eaf.6.2026.03.11.12.52.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:07 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Harry Yoo , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 10/11] mm/zsmalloc: Handle single object charge migration in migrate_zspage Date: Wed, 11 Mar 2026 12:51:47 -0700 Message-ID: <20260311195153.4013476-11-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In zsmalloc, there are two types of migrations: Migrations of single compressed objects from one zspage to another, and substitutions of zpdescs from zspages. In both of these migrations, memcg association for the compressed objects do not change. However, the physical location of the compressed objects may change, which alters their lruvec association. In this patch, handle the single compressed object migration and transfer lruvec and node statistics across the affected lruvecs / nodes. Zsmalloc compressed objects, like slab objects, can span two pages. When a spanning object is migrated, possibly to another zspage where it spans two zpdescs, up to 4 nodes can be touched. Instead of enumerating all possible combinations of node migrations, simply uncharge entirely from the source (1 or 2 nodes) and charge entirely to the destination (1 or 2 nodes). s_off d_off v v ----------+ +---- -----+ +--------- ... ooo ooo xx| |x oo ... --> ... ooo x| |xx ooo oo ... ----------+ +---- -----+ +--------- pg1 pg2 pg3 pg4 s_zspage d_zspage To do this, calculate how much of the compressed object lives on each page and perform up to 4 uncharge-charges. Note that these operations cannot call the existing zs_{charge, uncharge}_objcg functions we introduced, since we are holding the class spin lock and obj_cgroup_charge can sleep. Signed-off-by: Joshua Hahn --- mm/zsmalloc.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 70 insertions(+), 4 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index ab085961b0e2..f3508ff8b3ab 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1684,15 +1684,81 @@ static unsigned long find_alloced_obj(struct size_c= lass *class, return handle; } =20 +#ifdef CONFIG_MEMCG static void zs_migrate_objcg(struct zspage *s_zspage, struct zspage *d_zsp= age, - unsigned long used_obj, unsigned long free_obj) + unsigned long used_obj, unsigned long free_obj, + struct zs_pool *pool, int size) { - unsigned int s_idx =3D used_obj & OBJ_INDEX_MASK; - unsigned int d_idx =3D free_obj & OBJ_INDEX_MASK; + struct zpdesc *s_zpdesc, *d_zpdesc; + struct obj_cgroup *objcg; + struct mem_cgroup *memcg; + struct lruvec *l; + unsigned int s_idx, d_idx; + unsigned int s_off, d_off; + int charges[4], nids[4], partial; + int s_bytes_in_page, d_bytes_in_page; + int i; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + goto out; + + obj_to_location(used_obj, &s_zpdesc, &s_idx); + obj_to_location(free_obj, &d_zpdesc, &d_idx); + + objcg =3D s_zspage->objcgs[s_idx]; + if (!objcg) + goto out; + + /* + * The object migration here can touch up to 4 nodes. + * Instead of breaking down all possible combinations of node changes, + * just uncharge entirely from the source and charge entirely to the + * destination, even if there is are node overlaps between src and dst. + */ + s_off =3D (s_idx * size) % PAGE_SIZE; + d_off =3D (d_idx * size) % PAGE_SIZE; + s_bytes_in_page =3D min_t(int, size, PAGE_SIZE - s_off); + d_bytes_in_page =3D min_t(int, size, PAGE_SIZE - d_off); + + charges[0] =3D -s_bytes_in_page; + nids[0] =3D page_to_nid(zpdesc_page(s_zpdesc)); + charges[1] =3D -(size - s_bytes_in_page); /* 0 if object doesn't span */ + if (charges[1]) + nids[1] =3D page_to_nid(zpdesc_page(get_next_zpdesc(s_zpdesc))); + + charges[2] =3D d_bytes_in_page; + nids[2] =3D page_to_nid(zpdesc_page(d_zpdesc)); + charges[3] =3D size - d_bytes_in_page; /* 0 if object doesn't span */ + if (charges[3]) + nids[3] =3D page_to_nid(zpdesc_page(get_next_zpdesc(d_zpdesc))); =20 + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + for (i =3D 0; i < 4; i++) { + if (!charges[i]) + continue; + + l =3D mem_cgroup_lruvec(memcg, NODE_DATA(nids[i])); + partial =3D (PAGE_SIZE * charges[i]) / size; + mod_memcg_lruvec_state(l, pool->compressed_stat, charges[i]); + mod_memcg_lruvec_state(l, pool->uncompressed_stat, partial); + } + rcu_read_unlock(); + + dec_node_page_state(zpdesc_page(s_zpdesc), pool->uncompressed_stat); + inc_node_page_state(zpdesc_page(d_zpdesc), pool->uncompressed_stat); + +out: d_zspage->objcgs[d_idx] =3D s_zspage->objcgs[s_idx]; s_zspage->objcgs[s_idx] =3D NULL; } +#else +static void zs_migrate_objcg(struct zspage *s_zspage, struct zspage *d_zsp= age, + unsigned long used_obj, unsigned long free_obj, + struct zs_pool *pool, int size) +{ +} +#endif =20 static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage, struct zspage *dst_zspage) @@ -1719,7 +1785,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, =20 if (pool->memcg_aware) zs_migrate_objcg(src_zspage, dst_zspage, - used_obj, free_obj); + used_obj, free_obj, pool, class->size); =20 obj_idx++; obj_free(class->size, used_obj); --=20 2.52.0